Corrupted data, failed drive(s)

Corrupted data, failed drive(s)

[Gaardiolor]

Hello,

I could use some help with some issues I'm having with my drives.

I've got 4 disks in raid1.
--
[17:59:07]root@kiwi:/storage/samba/storage# btrfs filesystem df /storage/
Data, RAID1: total=4.39TiB, used=4.38TiB
System, RAID1: total=32.00MiB, used=720.00KiB
Metadata, RAID1: total=6.00GiB, used=4.66GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

[17:59:10]root@kiwi:/storage/samba/storage# btrfs filesystem show
Label: none  uuid: 8ce9e167-57ea-4cf8-8678-3049ba028c12
         Total devices 4 FS bytes used 4.38TiB
         devid    1 size 3.64TiB used 3.10TiB path /dev/sdc
         devid    2 size 3.64TiB used 3.14TiB path /dev/sdb
         devid    3 size 1.82TiB used 1.32TiB path /dev/sda
         devid    4 size 1.82TiB used 1.21TiB path /dev/sdd
--

I'm having some issues with faulty disk(s). /dev/sdd is bad for sure, 
SMART is complaining.
--
# smartctl -aq errorsonly /dev/sdd
ATA Error Count: 108 (device log contains only the most recent five errors)
Error 108 occurred at disk power-on lifetime: 47563 hours (1981 days + 
19 hours)
Error 107 occurred at disk power-on lifetime: 47563 hours (1981 days + 
19 hours)
Error 106 occurred at disk power-on lifetime: 47563 hours (1981 days + 
19 hours)
Error 105 occurred at disk power-on lifetime: 47563 hours (1981 days + 
19 hours)
Error 104 occurred at disk power-on lifetime: 47563 hours (1981 days + 
19 hours)
--

Also in /var/log/messages:
--
Jun  3 17:47:21 kiwi smartd[1112]: Device: /dev/sdd [SAT], 3088 
Currently unreadable (pending) sectors
Jun  3 17:47:21 kiwi smartd[1112]: Device: /dev/sdd [SAT], 3088 Offline 
uncorrectable sectors
--

However, the other disks also generate errors.
--
[18:00:35]root@kiwi:/storage/samba/storage# btrfs device stats /dev/sda
[/dev/sda].write_io_errs    0
[/dev/sda].read_io_errs     0
[/dev/sda].flush_io_errs    0
[/dev/sda].corruption_errs  408
[/dev/sda].generation_errs  0
[18:00:39]root@kiwi:/storage/samba/storage# btrfs device stats /dev/sdb
[/dev/sdb].write_io_errs    0
[/dev/sdb].read_io_errs     0
[/dev/sdb].flush_io_errs    0
[/dev/sdb].corruption_errs  322
[/dev/sdb].generation_errs  0
[18:00:42]root@kiwi:/storage/samba/storage# btrfs device stats /dev/sdc
[/dev/sdc].write_io_errs    0
[/dev/sdc].read_io_errs     0
[/dev/sdc].flush_io_errs    0
[/dev/sdc].corruption_errs  1283
[/dev/sdc].generation_errs  0
[18:00:43]root@kiwi:/storage/samba/storage# btrfs device stats /dev/sdd
[/dev/sdd].write_io_errs    0
[/dev/sdd].read_io_errs     1582
[/dev/sdd].flush_io_errs    0
[/dev/sdd].corruption_errs  1310
[/dev/sdd].generation_errs  0
-

/dev/sdd is the only one with read_io_errs.

I've tried unpacking a .tar.gz from /storage to another filesystem, but 
the tar.gz was obviously corrupt. Very strange filenames which were, 
because of the name, pretty difficult to remove. I will not post the 
filenames here, it'd probably crash the internet. I'm also getting:
--
gzip: stdin: invalid compressed data--crc error
tar: Child returned status 1
tar: Error is not recoverable: exiting now
--

I can't btrfs remove /dev/sdd . The command below ran for a while (I 
could see the allocated space of /dev/sdd decrease with btrfs fi us 
/storage/), but then errored:
--
root@kiwi:~# btrfs device remove /dev/sdd /storage/
  ERROR: error removing device '/dev/sdd': Input/output error
--


I have a couple of questions:

1) Unpacking some .tar.gz files from /storage resulted in files with 
weird names, data was unusable. But, it's raid1. Why is my data corrupt, 
I've read that BTRFS checks the checksum on read ?
2) Are all my 4 drives faulty because of the corruption_errs ? If so, 4 
faulty drives is somewhat unusual. Any other possibilities ?
3) Given that
- I can't 'btrfs device remove' the device
- I do not have a free SATA port
- I'd prefer a method that doesn't unnecessarily take a very long time

What's the best way to migrate to a different device ? I'm guessing, 
after doing some reading:
- shutdown
- physically remove faulty disk
- boot
- verify /dev/sdd is missing, and that I've removed the correct disk
- shutdown
- connect new disk, it will also be /dev/sdd, because I have no other 
free SATA port
- boot
- check that the new disk is /dev/sdd
- mount -o degraded /dev/sda /storage
- btrfs replace start 4 /dev/sdd /storage
- btrfs balance /storage

Is this correct, should this also check / fix errors, if not, what's the 
best approach.. Thanks!

Gaardiolor

Re: Corrupted data, failed drive(s)

[Chris Murphy]

On Thu, Jun 3, 2021 at 10:51 AM Gaardiolor <gaardiolor@gmail.com> wrote:
>
> Hello,
>
> I could use some help with some issues I'm having with my drives.
>
> I've got 4 disks in raid1.
> --
> [17:59:07]root@kiwi:/storage/samba/storage# btrfs filesystem df /storage/
> Data, RAID1: total=4.39TiB, used=4.38TiB
> System, RAID1: total=32.00MiB, used=720.00KiB
> Metadata, RAID1: total=6.00GiB, used=4.66GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> [17:59:10]root@kiwi:/storage/samba/storage# btrfs filesystem show
> Label: none  uuid: 8ce9e167-57ea-4cf8-8678-3049ba028c12
>          Total devices 4 FS bytes used 4.38TiB
>          devid    1 size 3.64TiB used 3.10TiB path /dev/sdc
>          devid    2 size 3.64TiB used 3.14TiB path /dev/sdb
>          devid    3 size 1.82TiB used 1.32TiB path /dev/sda
>          devid    4 size 1.82TiB used 1.21TiB path /dev/sdd
> --
>
> I'm having some issues with faulty disk(s). /dev/sdd is bad for sure,
> SMART is complaining.
> --
> # smartctl -aq errorsonly /dev/sdd
> ATA Error Count: 108 (device log contains only the most recent five errors)
> Error 108 occurred at disk power-on lifetime: 47563 hours (1981 days +
> 19 hours)
> Error 107 occurred at disk power-on lifetime: 47563 hours (1981 days +
> 19 hours)
> Error 106 occurred at disk power-on lifetime: 47563 hours (1981 days +
> 19 hours)
> Error 105 occurred at disk power-on lifetime: 47563 hours (1981 days +
> 19 hours)
> Error 104 occurred at disk power-on lifetime: 47563 hours (1981 days +
> 19 hours)
> --
>
> Also in /var/log/messages:
> --
> Jun  3 17:47:21 kiwi smartd[1112]: Device: /dev/sdd [SAT], 3088
> Currently unreadable (pending) sectors
> Jun  3 17:47:21 kiwi smartd[1112]: Device: /dev/sdd [SAT], 3088 Offline
> uncorrectable sectors
> --
>
> However, the other disks also generate errors.
> --
> [18:00:35]root@kiwi:/storage/samba/storage# btrfs device stats /dev/sda
> [/dev/sda].write_io_errs    0
> [/dev/sda].read_io_errs     0
> [/dev/sda].flush_io_errs    0
> [/dev/sda].corruption_errs  408
> [/dev/sda].generation_errs  0
> [18:00:39]root@kiwi:/storage/samba/storage# btrfs device stats /dev/sdb
> [/dev/sdb].write_io_errs    0
> [/dev/sdb].read_io_errs     0
> [/dev/sdb].flush_io_errs    0
> [/dev/sdb].corruption_errs  322
> [/dev/sdb].generation_errs  0
> [18:00:42]root@kiwi:/storage/samba/storage# btrfs device stats /dev/sdc
> [/dev/sdc].write_io_errs    0
> [/dev/sdc].read_io_errs     0
> [/dev/sdc].flush_io_errs    0
> [/dev/sdc].corruption_errs  1283
> [/dev/sdc].generation_errs  0
> [18:00:43]root@kiwi:/storage/samba/storage# btrfs device stats /dev/sdd
> [/dev/sdd].write_io_errs    0
> [/dev/sdd].read_io_errs     1582
> [/dev/sdd].flush_io_errs    0
> [/dev/sdd].corruption_errs  1310
> [/dev/sdd].generation_errs  0
> -
>
> /dev/sdd is the only one with read_io_errs.
>
> I've tried unpacking a .tar.gz from /storage to another filesystem, but
> the tar.gz was obviously corrupt. Very strange filenames which were,
> because of the name, pretty difficult to remove. I will not post the
> filenames here, it'd probably crash the internet. I'm also getting:
> --
> gzip: stdin: invalid compressed data--crc error
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
> --
>
> I can't btrfs remove /dev/sdd . The command below ran for a while (I
> could see the allocated space of /dev/sdd decrease with btrfs fi us
> /storage/), but then errored:
> --
> root@kiwi:~# btrfs device remove /dev/sdd /storage/
>   ERROR: error removing device '/dev/sdd': Input/output error
> --
>
>
> I have a couple of questions:
>
> 1) Unpacking some .tar.gz files from /storage resulted in files with
> weird names, data was unusable. But, it's raid1. Why is my data corrupt,
> I've read that BTRFS checks the checksum on read ?

It suggests an additional problem, but we kinda need full dmesg to
figure it out I think. If it were just one device having either
partial or full failure, you'd get a bunch of messages indicating
those failure or csum mismatches as  well as fixup attempts which then
either succeed or fail. But no EIO. That there's EIO  suggests both
copies are somehow bad, so it could be two independent problems. That
there's four drives with a small number of reported corruptions could
mean some common problem affecting all of them: cabling or power
supply.


> 2) Are all my 4 drives faulty because of the corruption_errs ? If so, 4
> faulty drives is somewhat unusual. Any other possibilities ?
> 3) Given that
> - I can't 'btrfs device remove' the device
> - I do not have a free SATA port
> - I'd prefer a method that doesn't unnecessarily take a very long time

You really don't want to remove a drive unless it's on fire. Partial
failures, you're better off leaving it in, until ready to be replaced.
And even then it is officially left in until replace completes. Btrfs
is AOK with partially failing drives, it can unambiguously determine
when any block is untrustworthy. But the partial failure case also
means possibly quite a lot of *good* blocks that you might need in
order to recover from this situation, so you don't want to throw the
baby out with the bath water, so to speak.



>
> What's the best way to migrate to a different device ? I'm guessing,
> after doing some reading:
> - shutdown
> - physically remove faulty disk
> - boot
> - verify /dev/sdd is missing, and that I've removed the correct disk
> - shutdown
> - connect new disk, it will also be /dev/sdd, because I have no other
> free SATA port
> - boot
> - check that the new disk is /dev/sdd
> - mount -o degraded /dev/sda /storage
> - btrfs replace start 4 /dev/sdd /storage
> - btrfs balance /storage

You can but again this throws away quite a lot of good blocks, both
data and metadata. Only if all three of the other drives are perfect
is this a good idea and there's some evidence it isn't.

I'd say your top priority is freshen the backups of most important
things you cannot stand to lose from this file system in case it gets
much worse. Then try to figure out what's wrong and fix that. The
direct causes need to be discovered and fixed, and the above sequence
doesn't identify multiple problems; it just assumes it's this one
drive. And the available evidence suggests more than one thing is
going on. If this is cable or dirty/noisy power supply related, the
recovery process itself can be negatively affected and make things
worse (more corruptions).

I think a better approach, as finicky as they can be, is a USB SATA
enclosure connected to an externally powered hub. Really you want a
fifth SATA port, even if it's eSATA. But barring that, I think it's
less risky to keep all four drives together, to do the replacement.



-- 
Chris Murphy

Re: Corrupted data, failed drive(s)

[Gaardiolor]

Hi Chris,

Thanks for your reply. Just noticed I forgot to mention I'm running 
kernel 5.12.8-300.fc34.x86_64 with btrfs-progs-5.12.1-1.fc34.x86_64 .


>> I have a couple of questions:
>>
>> 1) Unpacking some .tar.gz files from /storage resulted in files with
>> weird names, data was unusable. But, it's raid1. Why is my data corrupt,
>> I've read that BTRFS checks the checksum on read ?
> 
> It suggests an additional problem, but we kinda need full dmesg to
> figure it out I think. If it were just one device having either
> partial or full failure, you'd get a bunch of messages indicating
> those failure or csum mismatches as  well as fixup attempts which then
> either succeed or fail. But no EIO. That there's EIO  suggests both
> copies are somehow bad, so it could be two independent problems. That
> there's four drives with a small number of reported corruptions could
> mean some common problem affecting all of them: cabling or power
> supply.
> 

First try on unpacking the .tar.gz worked. Second try on the same 
.tar.gz now results in:

gzip: stdin: Input/output error
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

dmesg:
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247860736 csum 0x545eef4e expected csum 0x2cd08f83 
mirror 1
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs: 
wr 0, rd 0, flush 0, corrupt 323, gen 0
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247864832 csum 0x79b50174 expected csum 0xa744e4f5 
mirror 1
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs: 
wr 0, rd 0, flush 0, corrupt 324, gen 0
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247864832 csum 0x79b50174 expected csum 0xa744e4f5 
mirror 2
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sda errs: 
wr 0, rd 0, flush 0, corrupt 409, gen 0
[Fri Jun  4 09:53:03 2021] repair_io_failure: 326 callbacks suppressed
[Fri Jun  4 09:53:03 2021] BTRFS info (device sdc): read error 
corrected: ino 5114941 off 5247860736 (dev /dev/sdb sector 6674359360)
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247864832 csum 0x79b50174 expected csum 0xa744e4f5 
mirror 1
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs: 
wr 0, rd 0, flush 0, corrupt 325, gen 0
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247864832 csum 0x81568248 expected csum 0xa744e4f5 
mirror 2
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sda errs: 
wr 0, rd 0, flush 0, corrupt 410, gen 0
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247864832 csum 0x81568248 expected csum 0xa744e4f5 
mirror 1
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs: 
wr 0, rd 0, flush 0, corrupt 326, gen 0
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247864832 csum 0x81568248 expected csum 0xa744e4f5 
mirror 2
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sda errs: 
wr 0, rd 0, flush 0, corrupt 411, gen 0
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247864832 csum 0x79b50174 expected csum 0xa744e4f5 
mirror 1
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs: 
wr 0, rd 0, flush 0, corrupt 327, gen 0
[Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root 
5 ino 5114941 off 5247864832 csum 0x81568248 expected csum 0xa744e4f5 
mirror 2
[Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sda errs: 
wr 0, rd 0, flush 0, corrupt 412, gen 0

No weird filenames though, and no sdd errors this time. I also see these 
errors in /var/log/messages (on a different filesystem), but I don't see 
any "csum failed" errors in the messages log of yesterday, when the 
strange filenames appeared..

> 
>> 2) Are all my 4 drives faulty because of the corruption_errs ? If so, 4
>> faulty drives is somewhat unusual. Any other possibilities ?
>> 3) Given that
>> - I can't 'btrfs device remove' the device
>> - I do not have a free SATA port
>> - I'd prefer a method that doesn't unnecessarily take a very long time
> 
> You really don't want to remove a drive unless it's on fire. Partial
> failures, you're better off leaving it in, until ready to be replaced.
> And even then it is officially left in until replace completes. Btrfs
> is AOK with partially failing drives, it can unambiguously determine
> when any block is untrustworthy. But the partial failure case also
> means possibly quite a lot of *good* blocks that you might need in
> order to recover from this situation, so you don't want to throw the
> baby out with the bath water, so to speak.
>
I think we're mixing 'btrfs device remove' with physically remove. I did 
not plan on physically remove, but I might be forced because the 
graceful 'btrfs device remove' results in an I/O error. Or is there a 
better way ? Can 'btrfs device remove' ignore errors and continue with 
the good blocks?

My guess was that btrfs remove would take a very long time, I'd have a 
new drive before it would be finished. I had enough free space available 
to remove this device without adding a new drive. Did at the time not 
realize the other 3 drives had issues as well though.

>>
>> What's the best way to migrate to a different device ? I'm guessing,
>> after doing some reading:
>> - shutdown
>> - physically remove faulty disk
>> - boot
>> - verify /dev/sdd is missing, and that I've removed the correct disk
>> - shutdown
>> - connect new disk, it will also be /dev/sdd, because I have no other
>> free SATA port
>> - boot
>> - check that the new disk is /dev/sdd
>> - mount -o degraded /dev/sda /storage
>> - btrfs replace start 4 /dev/sdd /storage
>> - btrfs balance /storage
> 
> You can but again this throws away quite a lot of good blocks, both
> data and metadata. Only if all three of the other drives are perfect
> is this a good idea and there's some evidence it isn't.
> 
> I'd say your top priority is freshen the backups of most important
> things you cannot stand to lose from this file system in case it gets
> much worse. Then try to figure out what's wrong and fix that. The
> direct causes need to be discovered and fixed, and the above sequence
> doesn't identify multiple problems; it just assumes it's this one
> drive. And the available evidence suggests more than one thing is
> going on. If this is cable or dirty/noisy power supply related, the
> recovery process itself can be negatively affected and make things
> worse (more corruptions).
> 
> I think a better approach, as finicky as they can be, is a USB SATA
> enclosure connected to an externally powered hub. Really you want a
> fifth SATA port, even if it's eSATA. But barring that, I think it's
> less risky to keep all four drives together, to do the replacement.
> 

Yes, a fifth SATA port is a good idea. I did plan for this too, I 
actually have 6 SATA ports. What I didn't realize though is that 2 are 
disabled because I I installed 2 NVME drives. Should have read the fm :)

Apart from the general problem I might have (PSU for example), I might 
be able to hook up the new drive temporary via USB3 ? But, what would be 
the approach then ? I'd still need to 'btrfs device remove' sdd right, 
to evict data gracefully and replace it with the new one. But, btrfs 
remove results in an I/O error.

Turns out my drives aren't very cool though. 2 have >45k hours, 2 have 
 >12k which should be kinda ok, but are SMR. Just might be that they are 
all failing.. any idea how plausible that scenario could be ?

sda
   Model Family:     Seagate Barracuda 7200.14 (AF)
   Device Model:     ST2000DM001-1ER164
   TBR: 1350.26 TB
   TBW: 30.4776 TB
   Power_On_Hours 47582

sdb
   Model Family:     Seagate BarraCuda 3.5 (SMR)
   Device Model:     ST4000DM004-2CV104
   TBR: 6.71538 TB
   TBW: 32.5086 TB
   Power_On_Hours 12079

sdc
   Model Family:     Seagate BarraCuda 3.5 (SMR)
   Device Model:     ST4000DM004-2CV104
   TBR: 9.48872 TB
   TBW: 34.1534 TB
   Power_On_Hours 12079

sdd
   Model Family:     Seagate Barracuda 7200.14 (AF)
   Device Model:     ST2000DM001-1ER164
   TBR: 863.043 TB
   TBW: 28.6935 TB
   Power_On_Hours 47583

Thanks

Re: Corrupted data, failed drive(s)

[Chris Murphy]

On Fri, Jun 4, 2021 at 3:27 AM Gaardiolor <gaardiolor@gmail.com> wrote:
>
> Hi Chris,
>
> Thanks for your reply. Just noticed I forgot to mention I'm running
> kernel 5.12.8-300.fc34.x86_64 with btrfs-progs-5.12.1-1.fc34.x86_64 .
>
>
> >> I have a couple of questions:
> >>
> >> 1) Unpacking some .tar.gz files from /storage resulted in files with
> >> weird names, data was unusable. But, it's raid1. Why is my data corrupt,
> >> I've read that BTRFS checks the checksum on read ?
> >
> > It suggests an additional problem, but we kinda need full dmesg to
> > figure it out I think. If it were just one device having either
> > partial or full failure, you'd get a bunch of messages indicating
> > those failure or csum mismatches as  well as fixup attempts which then
> > either succeed or fail. But no EIO. That there's EIO  suggests both
> > copies are somehow bad, so it could be two independent problems. That
> > there's four drives with a small number of reported corruptions could
> > mean some common problem affecting all of them: cabling or power
> > supply.
> >
>
> First try on unpacking the .tar.gz worked. Second try on the same
> .tar.gz now results in:
>
> gzip: stdin: Input/output error
> tar: Unexpected EOF in archive
> tar: Unexpected EOF in archive
> tar: Error is not recoverable: exiting now
>
> dmesg:
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247860736 csum 0x545eef4e expected csum 0x2cd08f83
> mirror 1
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs:
> wr 0, rd 0, flush 0, corrupt 323, gen 0
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247864832 csum 0x79b50174 expected csum 0xa744e4f5
> mirror 1
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs:
> wr 0, rd 0, flush 0, corrupt 324, gen 0
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247864832 csum 0x79b50174 expected csum 0xa744e4f5
> mirror 2
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sda errs:
> wr 0, rd 0, flush 0, corrupt 409, gen 0
> [Fri Jun  4 09:53:03 2021] repair_io_failure: 326 callbacks suppressed
> [Fri Jun  4 09:53:03 2021] BTRFS info (device sdc): read error
> corrected: ino 5114941 off 5247860736 (dev /dev/sdb sector 6674359360)
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247864832 csum 0x79b50174 expected csum 0xa744e4f5
> mirror 1
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs:
> wr 0, rd 0, flush 0, corrupt 325, gen 0
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247864832 csum 0x81568248 expected csum 0xa744e4f5
> mirror 2
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sda errs:
> wr 0, rd 0, flush 0, corrupt 410, gen 0
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247864832 csum 0x81568248 expected csum 0xa744e4f5
> mirror 1
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs:
> wr 0, rd 0, flush 0, corrupt 326, gen 0
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247864832 csum 0x81568248 expected csum 0xa744e4f5
> mirror 2
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sda errs:
> wr 0, rd 0, flush 0, corrupt 411, gen 0
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247864832 csum 0x79b50174 expected csum 0xa744e4f5
> mirror 1
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sdb errs:
> wr 0, rd 0, flush 0, corrupt 327, gen 0
> [Fri Jun  4 09:53:03 2021] BTRFS warning (device sdc): csum failed root
> 5 ino 5114941 off 5247864832 csum 0x81568248 expected csum 0xa744e4f5
> mirror 2
> [Fri Jun  4 09:53:03 2021] BTRFS error (device sdc): bdev /dev/sda errs:
> wr 0, rd 0, flush 0, corrupt 412, gen 0


This is data corruption being detected. We know this because of the
formatting "root X ino Y" which translates as subvolume ID inode
number. Scrub is a bit different because it will not only show path to
file, but also shows all instances of a bad block in common with every
snapshot/reflink. So it can be quite noisy (repetitive).

In the above sequence there is one inode, thus one file. But two
corrupt blocks. One was fixed from a good copy. The other was not. But
also some messages might be missing, which is indicated in this
message:

>[Fri Jun  4 09:53:03 2021] repair_io_failure: 326 callbacks suppressed

I'm not sure how to stop this suppression to make sure we've got a
complete picture of what's going on. Possibly it just means 326 of the
exact same prior message, but I'm not 100% certain of the meaning.


But there's more...

One block, both copies are bad. But mirror 2 is consistently bad,
whereas with mirror 1 there is an inconsistency. While three read
attempts are all wrong (csum does not match expected), 2 reads match
each other and 1 read is different. So there is even a transient
problem reading from this sector as it is, in addition to being wrong
all three times.

Is this system ever crashing or doing other weird things?


>
> No weird filenames though, and no sdd errors this time. I also see these
> errors in /var/log/messages (on a different filesystem), but I don't see
> any "csum failed" errors in the messages log of yesterday, when the
> strange filenames appeared..
>
> >
> >> 2) Are all my 4 drives faulty because of the corruption_errs ? If so, 4
> >> faulty drives is somewhat unusual. Any other possibilities ?
> >> 3) Given that
> >> - I can't 'btrfs device remove' the device
> >> - I do not have a free SATA port
> >> - I'd prefer a method that doesn't unnecessarily take a very long time
> >
> > You really don't want to remove a drive unless it's on fire. Partial
> > failures, you're better off leaving it in, until ready to be replaced.
> > And even then it is officially left in until replace completes. Btrfs
> > is AOK with partially failing drives, it can unambiguously determine
> > when any block is untrustworthy. But the partial failure case also
> > means possibly quite a lot of *good* blocks that you might need in
> > order to recover from this situation, so you don't want to throw the
> > baby out with the bath water, so to speak.
> >
> I think we're mixing 'btrfs device remove' with physically remove. I did
> not plan on physically remove, but I might be forced because the
> graceful 'btrfs device remove' results in an I/O error. Or is there a
> better way ? Can 'btrfs device remove' ignore errors and continue with
> the good blocks?

I recommend neither physical device removal nor btrfs device remove.
You really want to avoid that and use btrfs replace instead. If you do
a 'btrfs device remove' to go from 4x to 3x drives, it is a shrink
operation, involving reading every block on the entire array and
writing all new blocks to make a 3x array. It's expensive and slow.
Use btrfs replace instead. It's faster and safer.

> Apart from the general problem I might have (PSU for example), I might
> be able to hook up the new drive temporary via USB3 ? But, what would be
> the approach then ? I'd still need to 'btrfs device remove' sdd right,
> to evict data gracefully and replace it with the new one. But, btrfs
> remove results in an I/O error.

You still want to use 'btrfs replace' not 'btrfs dev add/remove'.
Btrfs replace can tolerate one bad copy and fix it up while also
replicating the new copy to the new drive. But btrfs remove should
tolerate it too if there is a good copy. If there is no good copy,
it's a huge fail whale and i expect btrfs just stops.

> Turns out my drives aren't very cool though. 2 have >45k hours, 2 have
>  >12k which should be kinda ok, but are SMR. Just might be that they are
> all failing.. any idea how plausible that scenario could be ?

I can't compute the probability. It does seem unlikely. Even if it's
something weird like all four drives are the same make/model/firmware
and are hitting some kind of firmware bug that's in common to them?
Rare but not impossible, perhaps even plausible. I think though, if
you have memory bitflips, this will show up elsewhere like in-memory
corruption unrelated to the file system, and cause weird behaviors
including random crashing. Same if it's power induced. I guess it
could be a SATA controller in common for all four drives. If that
becomes a real suspicion, moving two drives to USB SATA enclosures
might help isolate it. Not ideal but the key I've found to USB being
mostly reliable rather than mostly unreliable is the externally
powered USB hub - rather than direct connection to ports on the logic
board.


-- 
Chris Murphy

Re: Re: Corrupted data, failed drive(s)

[Graham Cobb]

On 05/06/2021 00:22, Chris Murphy wrote:
> On Fri, Jun 4, 2021 at 3:27 AM Gaardiolor <gaardiolor@gmail.com> wrote:
.
.
.
>> Turns out my drives aren't very cool though. 2 have >45k hours, 2 have
>>  >12k which should be kinda ok, but are SMR. Just might be that they are
>> all failing.. any idea how plausible that scenario could be ?
> 
> I can't compute the probability. It does seem unlikely. Even if it's
> something weird like all four drives are the same make/model/firmware
> and are hitting some kind of firmware bug that's in common to them?
> Rare but not impossible, perhaps even plausible. I think though, if
> you have memory bitflips, this will show up elsewhere like in-memory
> corruption unrelated to the file system, and cause weird behaviors
> including random crashing. Same if it's power induced. 

I wouldn't dismiss power problems. I had some problems on a small system
where I had added several big disks over time and had some (2 or 3)
sharing a single PSU connection. The problems went away completely when
I recabled each disk to use a separate port on the PSU.

In my case the problems looked like SCSI cable problems (command
communication errors resulting in frequent link resets) rather than data
corruption (but didn't go away when I replaced the cables and didn't
move when I swapped the data cables around) but I could easily believe
other errors could have been seen. And, of course, they were load
related - if all the disks were working at once.

I recommend avoiding sharing PSU connections. Particularly between
heavily used disks.

Re: Corrupted data, failed drive(s)

[Gaardiolor]

ok.. the memtest looks kinda bad. I'll replace the memory.

memtester version 4.3.0 (64-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 20480MB (21474836480 bytes)
got  20480MB (21474836480 bytes), trying mlock ...locked.
Loop 1/10:
   Stuck Address       : testing   1FAILURE: possible bad address line 
at offset 0x123547370.
Skipping to next test...
   Random Value        : FAILURE: 0x6f77d4f5f77d938f != 
0x6f77dcf5f77d938f at offset 0x12186fd48.
FAILURE: 0x3c98d79170ce341b != 0x3c98df9170ce341b at offset 0x12186f570.
FAILURE: 0xb4825de59a0a714f != 0xb48255e59a0a714f at offset 0x12186fd48.
FAILURE: 0x2d9a7833948a2798 != 0x2d9a7033948a2798 at offset 0x123547370.
FAILURE: 0x6a2a69b98088822b != 0x6a2a61b98088822b at offset 0x123547470.
   Compare XOR         : FAILURE: 0xa1403b114589b90a != 
0xa14043114589b90a at offset 0x12186d068.
FAILURE: 0x3daddd6882cfbe3a != 0x3dadd56882cfbe3a at offset 0x12186f570.
FAILURE: 0xb59753bcac0bfb6e != 0xb5974bbcac0bfb6e at offset 0x12186fd48.
FAILURE: 0x2eaf6e0aa68bb1b7 != 0x2eaf660aa68bb1b7 at offset 0x123547370.
FAILURE: 0x6b3f5f90928a0c4a != 0x6b3f5790928a0c4a at offset 0x123547470.
   Compare SUB         : FAILURE: 0xde4d93737a22df84 != 
0xde4d83737a22df84 at offset 0x1526bde8.
FAILURE: 0x27a753236cf59e0a != 0xa72b5b236cf59e0a at offset 0x12186d068.
FAILURE: 0x37a9842273eb3b3a != 0xb8257c2273eb3b3a at offset 0x12186f570.
FAILURE: 0x5b272e785db2d26e != 0xdba326785db2d26e at offset 0x12186fd48.
FAILURE: 0x525adbd6a6f91d37 != 0xd2d6d3d6a6f91d37 at offset 0x123547370.
FAILURE: 0x54acab2f98eb914a != 0xd528a32f98eb914a at offset 0x123547470.
   Compare MUL         : FAILURE: 0x00000000 != 0x00000001 at offset 
0x12186d068.
FAILURE: 0x00000000 != 0x00000001 at offset 0x12186f570.
FAILURE: 0x00000000 != 0x00000001 at offset 0x12186fd48.
FAILURE: 0x00000000 != 0x00000001 at offset 0x123547370.
FAILURE: 0x00000000 != 0x00000001 at offset 0x123547470.
   Compare DIV         : FAILURE: 0x7dbef883f67eecb7 != 
0x7dbee883f67eecb7 at offset 0x1526bde8.
   Compare OR          : FAILURE: 0x7db63883667e6c91 != 
0x7db62883667e6c91 at offset 0x1526bde8.
   Compare AND         : FAILURE: 0x7ee6329401ea8682 != 
0x7ee6229401ea8682 at offset 0x1526bde8.
   Sequential Increment:   Solid Bits          : testing   1FAILURE: 
0xffffffffffffffff != 0xffffefffffffffff at offset 0x1526bde8.
FAILURE: 0x80000000000 != 0x00000000 at offset 0x123547370.
FAILURE: 0x80000000000 != 0x00000000 at offset 0x123547470.
   Block Sequential    : testing  16FAILURE: 0x1010101010101010 != 
0x1010001010101010 at offset 0x1526bde8.
   Checkerboard        : testing   0FAILURE: 0x55555d5555555555 != 
0x5555555555555555 at offset 0x123547370.
FAILURE: 0x55555d5555555555 != 0x5555555555555555 at offset 0x123547470.
   Bit Spread          : testing   0FAILURE: 0xfffffffffffffffa != 
0xffffeffffffffffa at offset 0x1526bde8.
FAILURE: 0x80000000005 != 0x00000005 at offset 0x123547370.
FAILURE: 0x80000000005 != 0x00000005 at offset 0x123547470.
   Bit Flip            : testing   1FAILURE: 0xfffffffffffffffe != 
0xffffeffffffffffe at offset 0x1526bde8.
FAILURE: 0x80000000001 != 0x00000001 at offset 0x123547370.
FAILURE: 0x80000000001 != 0x00000001 at offset 0x123547470.
   Walking Ones        : testing   0FAILURE: 0xfffffffffffffffe != 
0xffffeffffffffffe at offset 0x1526bde8.
   Walking Zeroes      : testing  44FAILURE: 0x100000000000 != 
0x00000000 at offset 0x1526bde8.
   8-bit Writes        : /FAILURE: 0xedd92db03fefa26d != 
0xedd925b03fefa26d at offset 0x123547470.
FAILURE: 0xf9f71c7056ff1fdf != 0xf9f7147056ff1fdf at offset 0x123547b30.
FAILURE: 0x8df1a5abeafa39de != 0x8df1b5abeafa39de at offset 0x1238d27e0.
   16-bit Writes       : /FAILURE: 0x7fbaade47b7d6b03 != 
0x7fbaa5e47b7d6b03 at offset 0x12186f570.