Re: RFC: log record CRC validation

["William J. Earl" <earl@agami.com>]

David Chinner wrote:
> On Thu, Jul 26, 2007 at 10:53:02AM -0700, Michael Nishimoto wrote:
>   
> ...
>> Is CRC checking being added to xfs log data?
>>     
>
> Yes. it's a little used debug option right now, and I'm
> planning on making it default behaviour.
>
>   
>> If so, what data has been collected to show that this needs to be added?
>>     
>
> The size of high-end filesystems are now at the same order of
> magnitude as the bit error rate of the storage hardware. e.g. 1PB =
> 10^16 bits. The bit error rate of high end FC drives? 1 in 10^16
> bits.  For "enterprise" SATA drives? 1 in 10^15 bits. For desktop
> SATA drives it's 1 in 10^14 bits (i.e. 1 in 10TB).
>
> We've got filesystems capable of moving > 2 x 10^16 bits of data
> *per day* and we see lots of instances of multi-TB arrays made out
> of desktop SATA disks. Given the recent studies of long-term disk
> reliability, these vendor figures are likely to be the best error
> rates we can hope for.....
>
> IOWs, we don't need evidence to justify this sort of error detection
> detection because simple maths says there is going to be errors.  We
> have to deal with that, and hence we are going to be adding CRC
> checking to on-disk metadata structures so we can detect bit errors
> that would otherwise go undetected and result in filesystem
> corruption.
>
> This means that instead of getting shutdown reports for some strange
> and unreproducable btree corruption, we'll get a shutdown for a CRC
> failure on the btree block. It is very likely that this will occur
> much earlier than a subtle btree corruption would otherwise be
> detected and hence we'll be less likely to propagate errors around
> the fileystem.
> ...
         Mike Nishimoto pointed out this thread to me, and suggested I 
reply, since I have worked on the analyzing the disk failure modes.  

       First, note that the claimed bit error rates are rates of 
reported bad blocks, not rates of silent data corruption.   The latter, 
while not quoted, are far lower.

       With large modern disks, it is unrealistic to not use RAID 1, 
RAID 5, or RAID 6 to mask reported disk bit errors.   A CRC in a 
filesystem data structure, however, is not required to detect disk 
errors, even without RAID protection, since the disks report error 
correction failures.   The rates of 1 in 10**14 (desktop SATA), 1 in 
10**15 (Enterprise SATA and some FC), and 1 in 10**16 (some FC) bits 
read are detected and reported error rates.   From conversations with 
drive vendors, these are actually fairly conservative, and assume you 
write the data once and read it after 10 years without rewriting it, as 
in an archive, which is the worst case, since you have ten years of 
accumulated deterioration.  

       With RAID protection, seeing reported errors which are not masked 
by the RAID layer is extremely unlikely, except in the case of a drive 
failure, where it is necessary to read all of the surviving drives to 
reconstruct the contents of the lost drive.    With 500 GB drives and a 
7D+1P RAID 5 group, we need to read 350 GB to rebuild the RAID array 
using a replacement drive, which is to say about 2.8 * 10**12 bits.   
This implies we would see one block not reconstructed in every 35 
rebuilds on desktop SATA drives, if the data were archive data written 
once and the rebuild happened after 10 years.     We would expect about 
one rebuild in 10 years.

       With more RAID groups, the chance of data loss grows very 
high.    With 100 groups, we would see a rebuild every  month or so, so 
we would expect an unmasked read error every few years.    With better 
drives, the chance is of course reduced, but is still significant in 
multi-PB storage systems.   RAID 6 largely eliminates the chance of 
seeing a read error, at some cost in performance.

       Note that some people have reported both much higher annual 
failure rates for drives (which increases the frequency of RAID 
reconstructions and hence the chance of data loss) and higher read error 
rates.    Based on personal experience with a large number of drives, I 
believe that both of these are a consequence of systems (including 
software and disk host bus adapters) dealing poorly with common 
transient disk problems, not actual errors on the surface of the disk.   
For example, if the drive firmware gets confused and stops talking, the 
controller will treat it as a read timeout, and the software may simply 
report "read failure", which in turn may be interpreted as "drive 
failed", even though an adequate drive reset would return the drive to 
service with no harm done to the data.  

     In addition, many people have taken up using desktop drives for 
primary storage, for which they are not designed.   Desktop drives are 
typically rated at 800,000 to  1,000,000 hours MTBF at 30% duty 
cycle.    Using them at 100% duty cycle drastically decreases their 
MTBF, which in turn drastically increases the rate of unmasked read 
errors, as a consequence of the extra RAID reconstructions.

      Lastly, quite a few "white box" vendors have shipped chassis which 
do not adequately cool and power the drives, and excessive heat can also 
drastically reduce the MTBF.

       In a well-designed chassis, with software which correctly 
recovers from transient drive issues, I have observed higher than 
claimed MTBF and much lower than claimed bit error rates.    The 
undetected error rate from a modern drive is not quoted publicly, but 
the block ECC is quite strong (since it has to mask raw bit error rates 
as high as 1 in 10**3) and hence can detect most error scenarios and 
correct many of them.

       None of the above, however, implies that we need CRCs on 
filesystem data structures.   That is, if you get EIO on a disk read, 
then you don't need a CRC to know the block is bad.    Other concerns, 
however, can motivate having CRCs.   In particular, if the path between 
drive and memory can corrupt the data, then CRCs can help us recover to 
some extent.    This has been a recurring problem with various 
technologies, but was particularly common on IDE (PATA) drives with 
their vulnerable cables, where mishandling of the cable could lead to 
silent data corruption.    CRCs on just filesystem data structures, 
however, only help with metadata integrity, leaving file data integrity 
in the dark.   Some vendors at various times,   such as Tandem and 
NetApp, have added a checksum to each block, usually when they were 
having data integrity issues which turned out in the end to be bad 
cables or bad software, but which could be masked by RAID recovery, if 
detected by the block checksum.   It is usually cost-effective, however, 
to simply select a reliable disk subsystem.   

       With SATA, SAS, and FC, which have link-level integrity checks, 
silent data corruption on the link is unlikely.   This leaves mainly the 
host bus adapter, the main memory, and the path between them.   If those 
are bad, however, it is hard to see how much the filesystem can help.

      In conclusion, I doubt that CRCs are worth the added 
complexity.    If I wanted to mask flaky hardware, I would look at using 
RAID 6 and validating parity on all reads and doing RAID recovery on any 
errors, but doing offline testing of the hardware and repairing or 
replacing any broken bits would be yet simpler.