Re: uncorrectable errors in Raid 10

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4711 bytes --]



On 2017年11月23日 00:38, Steffen Sindzinski wrote:
> Hello,
> 
> I did btrfs check --readonly on both disk without finding any error. To
> reconfirm I did a scrub again which still has found 2 uncorrectable

Still metadata corruption?

> errors. I had to boot into Arch Linux 4.13.12-1-ARCH
> btrfs-progs v4.13 to run btrfs check.

Well, btrfs-progs v4.13 from Arch is out-of-data for a while.
Even ignoring the latest v4.14 release, there is no v4.13.x releases for
Arch.

> 
> 
> Checking filesystem on /dev/sde2
> UUID: 4fafd0d4-7dd9-4dcc-9a33-5f1ad9555358
> checking extents
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> found 1598669742080 bytes used, no error found
> total csum bytes: 1540740308
> total tree bytes: 19391266816
> total fs tree bytes: 9844703232
> total extent tree bytes: 6808731648
> btree space waste bytes: 3471512438
> file data blocks allocated: 2124703174656
>  referenced 1454353633280
> sudo btrfs check --readonly /dev/sde2  2708,20s user 594,58s system 15%
> cpu 5:45:40,36 total
> ***********************************************************************
> sudo btrfs check --readonly /dev/sdd2
> Checking filesystem on /dev/sdd2
> UUID: 4fafd0d4-7dd9-4dcc-9a33-5f1ad9555358
> checking extents
> checking free space cache
> checking fs roots
> checking csums
> checking root refs
> found 1598669742080 bytes used, no error found
> total csum bytes: 1540740308
> total tree bytes: 19391266816
> total fs tree bytes: 9844703232
> total extent tree bytes: 6808731648
> btree space waste bytes: 3471512438
> file data blocks allocated: 2124703174656
>  referenced 1454353633280
> sudo btrfs check --readonly /dev/sdd2  2655,23s user 626,48s system 15%
> cpu 5:56:16,85 total

Just as Chris mentioned, if the scrub is reporting metadata corruption,
please try "btrfs check --mode=lowmem" to see if lowmem mode can detect
such corruption.

> 
> 
> I mentioned that I cannot access 2 directories anymore. Permission is
> denied, not even as root I have permission nor I can change ownership.
> This happens in Ubuntu 17.10, kernel 4.13.0-17-generic. It looks like this:
> 
> ls -la The*
> The-Wire:
> ls: Zugriff auf 'The-Wire/.' nicht möglich: Keine Berechtigung
> ls: Zugriff auf 'The-Wire/..' nicht möglich: Keine Berechtigung
> <...>
> insgesamt 0
> d????????? ? ? ? ?            ? .
> d????????? ? ? ? ?            ? ..
> -????????? ? ? ? ?            ? The Wire S1E10.m4v
> -????????? ? ? ? ?            ? The Wire S1E11.m4v

Normally this means DIR_INDEX/DIR_ITEM missing or points to incorrect inode.

> <...>
> 
> Oddly in Arch Linux, 4.13.12-1-ARCH, btrfs-progs v4.13, I can access
> this same directory, permission is set correctly.>
> The debug trees for both drives you may find attached to this mail. They
> were done in Ubunutu.

Not really helping.
I don't know how you take the dump, but it is only a leaf of subvolume,
full of EXTENT_DATA without any useful info.

Thanks,
Qu

> 
> For reference here the scrub report on Arch linux.
> 
> 
> Steffen
> 
> 
> 
> Am 20.11.2017 um 21:26 schrieb Chris Murphy:
>> On Mon, Nov 20, 2017 at 1:36 AM, Steffen Sindzinski
>> <stesind@gmail.com> wrote:
>>> Hi,
>>>
>>> I did btrfs-debug-tree for this block on both devices. The result is
>>> attached to this mail.
>>>
>>> It is really weird, same block, different drives, different sector. I
>>> have
>>> no problem with bit rod. Btrfs worked perfectly fine with this both
>>> HDDs for
>>> so long on Raid1. The drive sdf1 which I attached to form a Raid 10
>>> was also
>>> in a different Btrfs in the same machine for years flawlessly.
>>>
>>> I have not found any other checksum errors than the ones from this
>>> scrub.
>>>
>>> Is there no way to just safely recreate the checksum of this particular
>>> block from the disk contents?
>>
>> I'll cc Qu because I don't understand what's going on. It's the 2nd
>> case of both copies of metadata being bad in as many days, which could
>> just be coincidence.
>>
>> I also don't understand the specific error "checksum/header error"
>> which sounds to me like Btrfs knows the leaf is otherwise OK, but
>> there is some kind of problem with either the leaf csum or its header.
>> In which case I'd like to think that btrfs check --repair can fix this
>> kind of problem.
>>
>> What do you get for btrfs check without --repair?
>>
>> Curiously, your scrub complains about this "checksum/header error" but
>> btrfs-debug-tree gives no indication that leaf has any problem at all.
>>
>>
>>
>>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 520 bytes --]