Scrub not fixing checksum errors on RAID6

Scrub not fixing checksum errors on RAID6

[Jussi Kansanen]

Hello,

Scrub found some checksum errors on my RAID6, errors are said to be fixed but
every scrub results in identical errors. Any ideas why this is happening? The
HDD seems to be working OK, no IO errors, no SMART errors and it passed SMART
long-test.

Latest scrub was run with 4.6.2 kernel & 4.6 btrfs progs. Earlier runs where
done with 4.5.x & 4.6 with up to date btrfs progs (at the time).

This is after 3rd attempt (previous errors are identical):

BTRFS warning (device dm-6): checksum error at logical 10994876289024 on dev
/dev/mapper/crypt_sdk1, sector 3131104392, root 8265, inode 138347, offset
16846848, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876289024 on dev
/dev/mapper/crypt_sdk1, sector 3131104392, root 8260, inode 138347, offset
16846848, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876289024 on dev
/dev/mapper/crypt_sdk1, sector 3131104392, root 5, inode 138347, offset
16846848, length 4096, links 1 (path: path/file1)
BTRFS error (device dm-6): bdev /dev/mapper/crypt_sdk1 errs: wr 0, rd 0, flush
0, corrupt 11, gen 0
BTRFS error (device dm-6): fixed up error at logical 10994876289024 on dev
/dev/mapper/crypt_sdk1

BTRFS warning (device dm-6): checksum error at logical 10994876293120 on dev
/dev/mapper/crypt_sdk1, sector 3131104400, root 8265, inode 138347, offset
16850944, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876293120 on dev
/dev/mapper/crypt_sdk1, sector 3131104400, root 8260, inode 138347, offset
16850944, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876293120 on dev
/dev/mapper/crypt_sdk1, sector 3131104400, root 5, inode 138347, offset
16850944, length 4096, links 1 (path: path/file1)
BTRFS error (device dm-6): bdev /dev/mapper/crypt_sdk1 errs: wr 0, rd 0, flush
0, corrupt 12, gen 0
BTRFS error (device dm-6): fixed up error at logical 10994876293120 on dev
/dev/mapper/crypt_sdk1

BTRFS warning (device dm-6): checksum error at logical 10994876297216 on dev
/dev/mapper/crypt_sdk1, sector 3131104408, root 8265, inode 138347, offset
16855040, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876297216 on dev
/dev/mapper/crypt_sdk1, sector 3131104408, root 8260, inode 138347, offset
16855040, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876297216 on dev
/dev/mapper/crypt_sdk1, sector 3131104408, root 5, inode 138347, offset
16855040, length 4096, links 1 (path: path/file1)
BTRFS error (device dm-6): bdev /dev/mapper/crypt_sdk1 errs: wr 0, rd 0, flush
0, corrupt 13, gen 0
BTRFS error (device dm-6): fixed up error at logical 10994876297216 on dev
/dev/mapper/crypt_sdk1

BTRFS warning (device dm-6): checksum error at logical 10994876301312 on dev
/dev/mapper/crypt_sdk1, sector 3131104416, root 8265, inode 138347, offset
16859136, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876301312 on dev
/dev/mapper/crypt_sdk1, sector 3131104416, root 8260, inode 138347, offset
16859136, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876301312 on dev
/dev/mapper/crypt_sdk1, sector 3131104416, root 5, inode 138347, offset
16859136, length 4096, links 1 (path: path/file1)
BTRFS error (device dm-6): bdev /dev/mapper/crypt_sdk1 errs: wr 0, rd 0, flush
0, corrupt 14, gen 0
BTRFS error (device dm-6): fixed up error at logical 10994876301312 on dev
/dev/mapper/crypt_sdk1

BTRFS warning (device dm-6): checksum error at logical 10994876305408 on dev
/dev/mapper/crypt_sdk1, sector 3131104424, root 8265, inode 138347, offset
16863232, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876305408 on dev
/dev/mapper/crypt_sdk1, sector 3131104424, root 8260, inode 138347, offset
16863232, length 4096, links 1 (path: path/file1)
BTRFS warning (device dm-6): checksum error at logical 10994876305408 on dev
/dev/mapper/crypt_sdk1, sector 3131104424, root 5, inode 138347, offset
16863232, length 4096, links 1 (path: path/file1)
BTRFS error (device dm-6): bdev /dev/mapper/crypt_sdk1 errs: wr 0, rd 0, flush
0, corrupt 15, gen 0
BTRFS error (device dm-6): fixed up error at logical 10994876305408 on dev
/dev/mapper/crypt_sdk1


btrfs dev stats /storage/
[/dev/mapper/crypt_sde1].write_io_errs   0
[/dev/mapper/crypt_sde1].read_io_errs    0
[/dev/mapper/crypt_sde1].flush_io_errs   0
[/dev/mapper/crypt_sde1].corruption_errs 0
[/dev/mapper/crypt_sde1].generation_errs 0
[/dev/mapper/crypt_sdb1].write_io_errs   0
[/dev/mapper/crypt_sdb1].read_io_errs    0
[/dev/mapper/crypt_sdb1].flush_io_errs   0
[/dev/mapper/crypt_sdb1].corruption_errs 0
[/dev/mapper/crypt_sdb1].generation_errs 0
[/dev/mapper/crypt_sdc1].write_io_errs   0
[/dev/mapper/crypt_sdc1].read_io_errs    0
[/dev/mapper/crypt_sdc1].flush_io_errs   0
[/dev/mapper/crypt_sdc1].corruption_errs 0
[/dev/mapper/crypt_sdc1].generation_errs 0
[/dev/mapper/crypt_sdk1].write_io_errs   0
[/dev/mapper/crypt_sdk1].read_io_errs    0
[/dev/mapper/crypt_sdk1].flush_io_errs   0
[/dev/mapper/crypt_sdk1].corruption_errs 15
[/dev/mapper/crypt_sdk1].generation_errs 0
[/dev/mapper/crypt_sdd1].write_io_errs   0
[/dev/mapper/crypt_sdd1].read_io_errs    0
[/dev/mapper/crypt_sdd1].flush_io_errs   0
[/dev/mapper/crypt_sdd1].corruption_errs 0
[/dev/mapper/crypt_sdd1].generation_errs 0
[/dev/mapper/crypt_sdf1].write_io_errs   0
[/dev/mapper/crypt_sdf1].read_io_errs    0
[/dev/mapper/crypt_sdf1].flush_io_errs   0
[/dev/mapper/crypt_sdf1].corruption_errs 0
[/dev/mapper/crypt_sdf1].generation_errs 0


/dev/mapper/crypt_sde1 on /storage type btrfs
(rw,noatime,compress=lzo,noacl,space_cache,subvolid=5,subvol=/)

Re: Scrub not fixing checksum errors on RAID6

[Duncan]

Jussi Kansanen posted on Thu, 23 Jun 2016 18:17:46 +0300 as excerpted:

> Scrub found some checksum errors on my RAID6, errors are said to be
> fixed but every scrub results in identical errors. Any ideas why this is
> happening? The HDD seems to be working OK, no IO errors, no SMART errors
> and it passed SMART long-test.
> 
> Latest scrub was run with 4.6.2 kernel & 4.6 btrfs progs. Earlier runs
> where done with 4.5.x & 4.6 with up to date btrfs progs (at the time).

Short non-technical FYI...

Btrfs raid56 mode is unfortunately not yet upto current general btrfs 
stability[1].  There are a number of known raid56 mode bugs and reports 
such as this that need to be fixed first before it should be considered 
generally usable for anything but testing.

It wasn't clear from your post whether you know this and were simply 
reporting a problem that appeared in your testing so it can be fixed, or 
if you were trying to use raid56 mode for daily use and ran into this 
problem as a result.  I'm simply saying such issues are to be expected at 
this point, and that raid56 mode simply isn't ready for normal use yet, 
as there are a number of known problems such as this still around.

Btrfs raid1 mode, and to a slightly lessor extent, raid10 mode, is rather 
more mature and considered up to general btrfs stability status.  (I'm 
using raid1 mode here.  Raid10 mode basically works but may be slower 
than expected as it's not optimized yet.  Btrfs raid1 mode layered on top 
of a pair of md/dm-raid0s will likely be faster than btrfs raid10 without 
loss of the additional btrfs functionality.)

---
[1]  Current general btrfs stability status:  Stabilizing, but not yet 
fully stable and mature.  Keep backups and be prepared for occasional 
bugs that may force you to use them, but btrfs can be reasonably used for 
daily use provided you are doing so.  



-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman