Re: BTRFS Data at Rest File Corruption

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2016-05-16 23:42, Chris Murphy wrote:
> On Mon, May 16, 2016 at 5:44 PM, Richard A. Lochner <lochner@clone1.com> wrote:
>> Chris,
>>
>> It has actually happened to me three times that I know of in ~7mos.,
>> but your point about the "larger footprint" for data corruption is a
>> good one.  No doubt I have silently experienced that too.
>
> I dunno three is a lot to have the exact same corruption only in
> memory then written out into two copies with valid node checksums; and
> yet not have other problems, like a node item, or uuid, or xattr or
> any number of other item or object types all of which get checksummed.
> I suppose if the file system contains large files, the % of metadata
> that's csums could be the 2nd largest footprint. But still.
Assuming that the workload on the volume is mostly backup images like 
the file that originally sparked this discussion, then inodes, xattrs, 
and even UUID's would be nowhere near as common as metadata blocks just 
containing checksums.  The fact that this hasn't hit any metadata 
checksums is unusual, but not impossible.
>
> Three times in 7 months, if it's really the same vector, is just short
> of almost reproducible. Ha. It seems like if you merely balanced this
> file system a few times, you'd eventually stumble on this. And if
> that's true, then it's time for debug options and see if it can be
> caught in action, and whether there's a hardware or software
> explanation for it.
>
>
>> And, as you
>> suggest, there is no way to prevent those errors.  If the memory to be
>> written to disk gets corrupted before its checksum is calculated, the
>> data will be silently corrupted, period.
>
> Well, no way in the present design, maybe.
If the RAM is bad, there is no way we can completely protect user data, 
period.  We can try to mitigate certain situations, but we cannot 
protect against all forms of memory corruption.
>
>>
>> Clearly, I won't rely on this machine to produce any data directly that
>> I would consider important at this point.
>>
>> One odd thing to me is that if this is really due to undetected memory
>> errors, I'd think this system would crash fairly often due to detected
>> "parity errors."  This system rarely crashes.  It often runs for
>> several months without an indication of problems.
>
> I think you'd have other problems. Only data csums are being corrupt
> after they're read in, but before the node csum is computed? Three
> times?  Pretty wonky.
Running regularly for several months without ECC RAM may be part of the 
issue.  Minute electrical instabilities build up over time, as do 
instabilities caused by background radiation, and beyond a certain point 
(which is based on more factors than are practical to compute), you end 
up almost certain to have at least a single bit error.

On that note, I'd actually be curious to see how far off the checksum is 
(how many bits aren't correct).  Given that there are no other visible 
issues with the system, I'd expect it to only be one or at most two bits 
that are incorrect.