Re: Btrfs/RAID5 became unmountable after SATA cable fault

[Austin S Hemmelgarn <ahferroin7@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4556 bytes --]

On 2015-11-04 13:01, Janos Toth F. wrote:
> But the worst part is that there are some ISO files which were
> seemingly copied without errors but their external checksums (the one
> which I can calculate with md5sum and compare to the one supplied by
> the publisher of the ISO file) don't match!
> Well... this, I cannot understand.
> How could these files become corrupt from a single disk failure? And
> more importantly: how could these files be copied without errors? Why
> didn't Btrfs gave a read error when the checksums didn't add up?
If you can prove that there was a checksum mismatch and BTRFS returned 
invalid data instead of a read error or going to the other disk, then 
that is a very serious bug that needs to be fixed.  You need to keep in 
mind also however that it's completely possible that the data was bad 
before you wrote it to the filesystem, and if that's the case, there's 
nothing any filesystem can do to fix it for you.
>
> Isn't Btrfs supposed to constantly check the integrity of the file
> data during any normal read operations and give an error instead of
> spitting out corrupt data as if it was perfectly legit?
> I thought that's how it is supposed to work.
Assuming that all of your hardware is working exactly like it's supposed 
to, yes it should work that way.  If however, you have something that 
corrupts the data in RAM before or while BTRFS is computing the checksum 
prior to writing the data, the it's fully possible for bad data to get 
written to disk and still have a perfectly correct checksum.  Bad RAM 
may also explain your issues mentioned above with not being able to copy 
stuff off of the filesystem.

Also, if you're using NOCOW files (or just the mount option), those very 
specifically do not store checksums for the blocks, because there is no 
way to do it without significant risk of data corruption.
> What's the point of full data checksuming if only an explicitly
> requested scrub operation might look for errors? I thought's it's the
> logical thing to do if checksum verification happens during every
> single read operation and passing that check is mandatory in order to
> get any data out of the filesystem (might be excluding the Direct-I/O
> mode but I never use that on Btrfs - if that's even actually
> supported, I don't know).
>
>
> Now I am really considering to move from Linux to Windows and from
> Btrfs RAID-5 to Storage Spaces RAID-1 + ReFS (the only limitation is
> that ReFS is only "self-healing" on RAID-1, not RAID-5, so I need a
> new motherboard with more native SATA connectors and an extra HDD).
> That one seemed to actually do what it promises (abort any read
> operations upon checksum errors [which always happens seamlessly on
> every read] but look at the redundant data first and seamlessly
> "self-heal" if possible). The only thing which made Btrfs to look as a
> better alternative was the RAID-5 support. But I recently experienced
> two cases of 1 drive failing of 3 and it always tuned out as a smaller
> or bigger disaster (completely lost data or inconsistent data).
Have you considered looking into ZFS?  I hate to suggest it as an 
alternative to BTRFS, but it's a much more mature and well tested 
technology than ReFS, and has many of the same features as BTRFS (and 
even has the option for triple parity instead of the double you get with 
RAID6).  If you do consider ZFS, make a point to look at FreeBSD in 
addition to the Linux version, the BSD one was a much better written 
port of the original Solaris drivers, and has better performance in many 
cases (and as much as I hate to admit it, BSD is way more reliable than 
Linux in most use cases).

You should also seriously consider whether the convenience of having a 
filesystem that fixes internal errors itself with no user intervention 
is worth the risk of it corrupting your data.  Returning correct data 
whenever possible is one thing, being 'self-healing' is completely 
different.  When you start talking about things that automatically fix 
internal errors without user intervention is when most seasoned system 
administrators start to get really nervous.  Self correcting systems 
have just as much chance to make things worse as they do to make things 
better, and most of them depend on the underlying hardware working 
correctly to actually provide any guarantee of reliability.  I cannot 
count the number of stories I've heard of 'self-healing' hardware RAID 
controllers destroying data.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]