checksum error...

checksum error...

[Scott E. Blomquist]

Hi All,

The weekend btrfs scrub/balance came back with this following...

[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 62378, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 270894, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271453, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271449, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271435, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271432, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271421, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271411, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271401, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271391, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271345, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271253, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271163, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271074, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 270983, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[Sun Apr  7 06:57:10 2019] BTRFS error (device sdb1): bdev /dev/sda1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Sun Apr  7 06:57:10 2019] BTRFS error (device sdb1): unable to fixup (regular) error at logical 274820497408 on dev /dev/sda1

Here is what I have...

    root@cbmm-fsb:~# uname -a
    Linux cbmm-fsb 4.14.24-custom #1 SMP Mon Mar 5 10:10:39 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
    
    root@cbmm-fsb:~# btrfs --version
    btrfs-progs v4.15.1
    
    root@cbmm-fsb:~# btrfs fi show
    Label: none  uuid: d83b1e28-db27-4035-8638-d4b2eb824ff2
           Total devices 2 FS bytes used 80.09TiB
           devid    1 size 76.40TiB used 62.49TiB path /dev/sda1
           devid    2 size 32.74TiB used 18.83TiB path /dev/sdb1
    
    root@cbmm-fsb:~# btrfs fi df /home/cbcl
    Data, single: total=79.80TiB, used=79.80TiB
    System, RAID1: total=32.00MiB, used=9.09MiB
    Metadata, RAID1: total=757.00GiB, used=281.34GiB
    Metadata, DUP: total=22.50GiB, used=19.27GiB
    GlobalReserve, single: total=512.00MiB, used=0.00B
    
sda and sdb are megaraid raid6 with BBU and both are optimal.

Any tips?  Thanks.

sb. Scott Blomquist

Re: checksum error...

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2185 bytes --]

On Mon, Apr 08, 2019 at 11:48:03AM -0400, Scott E. Blomquist wrote:
> 
> Hi All,
> 
> The weekend btrfs scrub/balance came back with this following...
> 
> [Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
[snip]

   Since there doesn't seem to be anything else wrong (no messages
without a filename, which would imply metadata corruption), this is
most likely a simple case of on-device corruption.

   Delete yyyy/yyyyy and restore it from backups. At least, do so in
the working copy; The snapshots of it can safely remain until they get
rotated out normally.

   Check your SMART statistics and see if anything looks wrong there
on the hardware side. Also check dmesg and earlier kernel logs for
signs of the hardware showing an error on read -- it may have tried
several times to read that location before giving up and/or returning
bad data.

   Hugo.

> 
> Here is what I have...
> 
>     root@cbmm-fsb:~# uname -a
>     Linux cbmm-fsb 4.14.24-custom #1 SMP Mon Mar 5 10:10:39 EST 2018 x86_64 x86_64 x86_64 GNU/Linux
>     
>     root@cbmm-fsb:~# btrfs --version
>     btrfs-progs v4.15.1
>     
>     root@cbmm-fsb:~# btrfs fi show
>     Label: none  uuid: d83b1e28-db27-4035-8638-d4b2eb824ff2
>            Total devices 2 FS bytes used 80.09TiB
>            devid    1 size 76.40TiB used 62.49TiB path /dev/sda1
>            devid    2 size 32.74TiB used 18.83TiB path /dev/sdb1
>     
>     root@cbmm-fsb:~# btrfs fi df /home/cbcl
>     Data, single: total=79.80TiB, used=79.80TiB
>     System, RAID1: total=32.00MiB, used=9.09MiB
>     Metadata, RAID1: total=757.00GiB, used=281.34GiB
>     Metadata, DUP: total=22.50GiB, used=19.27GiB
>     GlobalReserve, single: total=512.00MiB, used=0.00B
>     
> sda and sdb are megaraid raid6 with BBU and both are optimal.
> 
> Any tips?  Thanks.
> 
> sb. Scott Blomquist
> 

-- 
Hugo Mills             | If it ain't broke, hit it again.
hugo@... carfax.org.uk |
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                                  Foon

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: checksum error...

[Scott E. Blomquist]

Hugo Mills writes:
 > On Mon, Apr 08, 2019 at 11:48:03AM -0400, Scott E. Blomquist wrote:
 > > 
 > > Hi All,
 > > 
 > > The weekend btrfs scrub/balance came back with this following...
 > > 
 > > [Sun Apr  7 06:57:10 2019] BTRFS warning (device sdb1): checksum error at logical 274820497408 on dev /dev/sda1, sector 536758784, root 271471, inode 109421914, offset 491520, length 4096, links 1 (path: yyyy/yyyyy)
 > [snip]
 > 
 >    Since there doesn't seem to be anything else wrong (no messages
 > without a filename, which would imply metadata corruption), this is
 > most likely a simple case of on-device corruption.
 > 
 >    Delete yyyy/yyyyy and restore it from backups. At least, do so in
 > the working copy; The snapshots of it can safely remain until they get
 > rotated out normally.
 > 
 >    Check your SMART statistics and see if anything looks wrong there
 > on the hardware side. Also check dmesg and earlier kernel logs for
 > signs of the hardware showing an error on read -- it may have tried
 > several times to read that location before giving up and/or returning
 > bad data.
 > 
 >    Hugo.

Thanks, Hugo.  Very helpful.

Turns out event log from MegaCli is showing some unexpected sense in
the eventlog.

Cheers,

sb. Scott Blomquist

Re: checksum error...

[Patrik Lundquist]

On Mon, 8 Apr 2019 at 18:27, Scott E. Blomquist <sb@techsquare.com> wrote:
>
>     root@cbmm-fsb:~# btrfs fi df /home/cbcl
>     Data, single: total=79.80TiB, used=79.80TiB
>     System, RAID1: total=32.00MiB, used=9.09MiB
>     Metadata, RAID1: total=757.00GiB, used=281.34GiB
>     Metadata, DUP: total=22.50GiB, used=19.27GiB
>     GlobalReserve, single: total=512.00MiB, used=0.00B
>
> sda and sdb are megaraid raid6 with BBU and both are optimal.

You should convert the remaining "Metadata, DUP" to RAID1.

btrfs balance start -v -mconvert=raid1,soft /home/cbcl