* ERROR: failed to read block groups: Input/output error
@ 2021-01-13 23:09 Dāvis Mosāns
2021-01-13 23:39 ` Dāvis Mosāns
2021-02-19 19:29 ` Zygo Blaxell
0 siblings, 2 replies; 10+ messages in thread
From: Dāvis Mosāns @ 2021-01-13 23:09 UTC (permalink / raw)
To: Btrfs BTRFS
Hi,
I've 6x 3TB HDD RAID1 BTRFS filesystem where HBA card failed and
caused some corruption.
When I try to mount it I get
$ mount /dev/sdt /mnt
mount: /mnt/: wrong fs type, bad option, bad superblock on /dev/sdt,
missing codepage or helper program, or other error
$ dmesg | tail -n 9
[ 617.158962] BTRFS info (device sdt): disk space caching is enabled
[ 617.158965] BTRFS info (device sdt): has skinny extents
[ 617.756924] BTRFS info (device sdt): bdev /dev/sdl errs: wr 0, rd
0, flush 0, corrupt 473, gen 0
[ 617.756929] BTRFS info (device sdt): bdev /dev/sdj errs: wr 31626,
rd 18765, flush 178, corrupt 5841, gen 0
[ 617.756933] BTRFS info (device sdt): bdev /dev/sdg errs: wr 6867,
rd 2640, flush 178, corrupt 1066, gen 0
[ 631.353725] BTRFS warning (device sdt): sdt checksum verify failed
on 21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
[ 631.376024] BTRFS warning (device sdt): sdt checksum verify failed
on 21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
[ 631.376038] BTRFS error (device sdt): failed to read block groups: -5
[ 631.422811] BTRFS error (device sdt): open_ctree failed
$ uname -r
5.9.14-arch1-1
$ btrfs --version
btrfs-progs v5.9
$ btrfs check /dev/sdt
Opening filesystem to check...
checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
checksum verify failed on 21057101103104 found 0000009C wanted 00000075
checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
Csum didn't match
ERROR: failed to read block groups: Input/output error
ERROR: cannot open file system
$ btrfs filesystem show
Label: 'RAID' uuid: 8aef11a9-beb6-49ea-9b2d-7876611a39e5
Total devices 6 FS bytes used 4.69TiB
devid 1 size 2.73TiB used 1.71TiB path /dev/sdt
devid 2 size 2.73TiB used 1.70TiB path /dev/sdl
devid 3 size 2.73TiB used 1.71TiB path /dev/sdj
devid 4 size 2.73TiB used 1.70TiB path /dev/sds
devid 5 size 2.73TiB used 1.69TiB path /dev/sdg
devid 6 size 2.73TiB used 1.69TiB path /dev/sdc
My guess is that some drives dropped out while kernel was still
writing to rest thus causing inconsistency.
There should be some way to find out which drives has the most
up-to-date info and assume those are correct.
I tried to mount with
$ mount -o ro,degraded,rescue=usebackuproot /dev/sdt /mnt
but that didn't make any difference
So any idea how to fix this filesystem?
Thanks!
Best regards,
Dāvis
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-01-13 23:09 ERROR: failed to read block groups: Input/output error Dāvis Mosāns
@ 2021-01-13 23:39 ` Dāvis Mosāns
2021-02-19 3:03 ` Dāvis Mosāns
2021-02-19 19:29 ` Zygo Blaxell
1 sibling, 1 reply; 10+ messages in thread
From: Dāvis Mosāns @ 2021-01-13 23:39 UTC (permalink / raw)
To: Btrfs BTRFS
>
> Hi,
>
> I've 6x 3TB HDD RAID1 BTRFS filesystem where HBA card failed and
> caused some corruption.
> When I try to mount it I get
> $ mount /dev/sdt /mnt
> mount: /mnt/: wrong fs type, bad option, bad superblock on /dev/sdt,
> missing codepage or helper program, or other error
> $ dmesg | tail -n 9
> [ 617.158962] BTRFS info (device sdt): disk space caching is enabled
> [ 617.158965] BTRFS info (device sdt): has skinny extents
> [ 617.756924] BTRFS info (device sdt): bdev /dev/sdl errs: wr 0, rd
> 0, flush 0, corrupt 473, gen 0
> [ 617.756929] BTRFS info (device sdt): bdev /dev/sdj errs: wr 31626,
> rd 18765, flush 178, corrupt 5841, gen 0
> [ 617.756933] BTRFS info (device sdt): bdev /dev/sdg errs: wr 6867,
> rd 2640, flush 178, corrupt 1066, gen 0
> [ 631.353725] BTRFS warning (device sdt): sdt checksum verify failed
> on 21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
> [ 631.376024] BTRFS warning (device sdt): sdt checksum verify failed
> on 21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
> [ 631.376038] BTRFS error (device sdt): failed to read block groups: -5
> [ 631.422811] BTRFS error (device sdt): open_ctree failed
>
> $ uname -r
> 5.9.14-arch1-1
> $ btrfs --version
> btrfs-progs v5.9
> $ btrfs check /dev/sdt
> Opening filesystem to check...
> checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> checksum verify failed on 21057101103104 found 0000009C wanted 00000075
> checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> Csum didn't match
> ERROR: failed to read block groups: Input/output error
> ERROR: cannot open file system
>
> $ btrfs filesystem show
> Label: 'RAID' uuid: 8aef11a9-beb6-49ea-9b2d-7876611a39e5
> Total devices 6 FS bytes used 4.69TiB
> devid 1 size 2.73TiB used 1.71TiB path /dev/sdt
> devid 2 size 2.73TiB used 1.70TiB path /dev/sdl
> devid 3 size 2.73TiB used 1.71TiB path /dev/sdj
> devid 4 size 2.73TiB used 1.70TiB path /dev/sds
> devid 5 size 2.73TiB used 1.69TiB path /dev/sdg
> devid 6 size 2.73TiB used 1.69TiB path /dev/sdc
>
>
> My guess is that some drives dropped out while kernel was still
> writing to rest thus causing inconsistency.
> There should be some way to find out which drives has the most
> up-to-date info and assume those are correct.
> I tried to mount with
> $ mount -o ro,degraded,rescue=usebackuproot /dev/sdt /mnt
> but that didn't make any difference
>
> So any idea how to fix this filesystem?
>
> Thanks!
>
> Best regards,
> Dāvis
By the way
$ btrfs-find-root /dev/sdt
ERROR: failed to read block groups: Input/output error
Superblock thinks the generation is 2262739
Superblock thinks the level is 1
Found tree root at 21057011679232 gen 2262739 level 1
Well block 21056933724160(gen: 2262738 level: 1) seems good, but
generation/level doesn't match, want gen: 2262739 level: 1
Well block 21056867319808(gen: 2262737 level: 1) seems good, but
generation/level doesn't match, want gen: 2262739 level: 1
Well block 21056855900160(gen: 2262736 level: 1) seems good, but
generation/level doesn't match, want gen: 2262739 level: 1
Well block 21056850739200(gen: 2120504 level: 0) seems good, but
generation/level doesn't match, want gen: 2262739 level: 1
$ btrfs restore -l /dev/sdt
tree key (EXTENT_TREE ROOT_ITEM 0) 21057008975872 level 3
tree key (DEV_TREE ROOT_ITEM 0) 21056861863936 level 1
tree key (FS_TREE ROOT_ITEM 0) 21063463993344 level 1
tree key (CSUM_TREE ROOT_ITEM 0) 21057010728960 level 3
tree key (UUID_TREE ROOT_ITEM 0) 21061425545216 level 0
tree key (262 ROOT_ITEM 0) 21063533002752 level 0
tree key (263 ROOT_ITEM 0) 21058890629120 level 2
tree key (418 ROOT_ITEM 0) 21057902198784 level 2
tree key (421 ROOT_ITEM 0) 21060222500864 level 2
tree key (427 ROOT_ITEM 0) 21061262114816 level 2
tree key (428 ROOT_ITEM 0) 21061278040064 level 2
tree key (440 ROOT_ITEM 0) 21061362417664 level 2
tree key (451 ROOT_ITEM 0) 21061017174016 level 2
tree key (454 ROOT_ITEM 0) 21559581114368 level 1
tree key (455 ROOT_ITEM 0) 21079314776064 level 1
tree key (456 ROOT_ITEM 0) 21058026831872 level 2
tree key (457 ROOT_ITEM 0) 21060907909120 level 3
tree key (497 ROOT_ITEM 0) 21058120990720 level 2
tree key (571 ROOT_ITEM 0) 21058195668992 level 2
tree key (599 ROOT_ITEM 0) 21058818015232 level 2
tree key (635 ROOT_ITEM 0) 21056973766656 level 2
tree key (638 ROOT_ITEM 0) 21061023072256 level 0
tree key (676 ROOT_ITEM 0) 21061314330624 level 2
tree key (3937 ROOT_ITEM 0) 21061408686080 level 0
tree key (3938 ROOT_ITEM 0) 21079315841024 level 1
tree key (3957 ROOT_ITEM 0) 21061419139072 level 2
tree key (6128 ROOT_ITEM 0) 21061400018944 level 1
tree key (8575 ROOT_ITEM 0) 21061023055872 level 0
tree key (18949 ROOT_ITEM 1728623) 21080421875712 level 1
tree key (18950 ROOT_ITEM 1728624) 21080424726528 level 2
tree key (18951 ROOT_ITEM 1728625) 21080424824832 level 2
tree key (18952 ROOT_ITEM 1728626) 21080426004480 level 3
tree key (18953 ROOT_ITEM 1728627) 21080422105088 level 2
tree key (18954 ROOT_ITEM 1728628) 21080424497152 level 2
tree key (18955 ROOT_ITEM 1728629) 21080426332160 level 2
tree key (18956 ROOT_ITEM 1728631) 21080423645184 level 2
tree key (18957 ROOT_ITEM 1728632) 21080425316352 level 2
tree key (18958 ROOT_ITEM 1728633) 21080423972864 level 2
tree key (18959 ROOT_ITEM 1728634) 21080422400000 level 2
tree key (18960 ROOT_ITEM 1728635) 21080422662144 level 2
tree key (18961 ROOT_ITEM 1728636) 21080423153664 level 2
tree key (18962 ROOT_ITEM 1728637) 21080425414656 level 2
tree key (18963 ROOT_ITEM 1728638) 21080421171200 level 1
tree key (18964 ROOT_ITEM 1728639) 21080423481344 level 2
tree key (19721 ROOT_ITEM 0) 21076937326592 level 2
checksum verify failed on 21057125580800 found 00000026 wanted 00000035
checksum verify failed on 21057108082688 found 00000074 wanted FFFFFFC5
checksum verify failed on 21057108082688 found 000000ED wanted FFFFFFC5
checksum verify failed on 21057108082688 found 00000074 wanted FFFFFFC5
Csum didn't match
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-01-13 23:39 ` Dāvis Mosāns
@ 2021-02-19 3:03 ` Dāvis Mosāns
2021-02-19 5:16 ` Chris Murphy
0 siblings, 1 reply; 10+ messages in thread
From: Dāvis Mosāns @ 2021-02-19 3:03 UTC (permalink / raw)
To: Btrfs BTRFS
ceturtd., 2021. g. 14. janv., plkst. 01:39 — lietotājs Dāvis Mosāns
(<davispuh@gmail.com>) rakstīja:
>
> >
> > Hi,
> >
> > I've 6x 3TB HDD RAID1 BTRFS filesystem where HBA card failed and
> > caused some corruption.
> > When I try to mount it I get
> > $ mount /dev/sdt /mnt
> > mount: /mnt/: wrong fs type, bad option, bad superblock on /dev/sdt,
> > missing codepage or helper program, or other error
> > $ dmesg | tail -n 9
> > [ 617.158962] BTRFS info (device sdt): disk space caching is enabled
> > [ 617.158965] BTRFS info (device sdt): has skinny extents
> > [ 617.756924] BTRFS info (device sdt): bdev /dev/sdl errs: wr 0, rd
> > 0, flush 0, corrupt 473, gen 0
> > [ 617.756929] BTRFS info (device sdt): bdev /dev/sdj errs: wr 31626,
> > rd 18765, flush 178, corrupt 5841, gen 0
> > [ 617.756933] BTRFS info (device sdt): bdev /dev/sdg errs: wr 6867,
> > rd 2640, flush 178, corrupt 1066, gen 0
> > [ 631.353725] BTRFS warning (device sdt): sdt checksum verify failed
> > on 21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
> > [ 631.376024] BTRFS warning (device sdt): sdt checksum verify failed
> > on 21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
> > [ 631.376038] BTRFS error (device sdt): failed to read block groups: -5
> > [ 631.422811] BTRFS error (device sdt): open_ctree failed
> >
> > $ uname -r
> > 5.9.14-arch1-1
> > $ btrfs --version
> > btrfs-progs v5.9
> > $ btrfs check /dev/sdt
> > Opening filesystem to check...
> > checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> > checksum verify failed on 21057101103104 found 0000009C wanted 00000075
> > checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> > Csum didn't match
> > ERROR: failed to read block groups: Input/output error
> > ERROR: cannot open file system
> >
> > $ btrfs filesystem show
> > Label: 'RAID' uuid: 8aef11a9-beb6-49ea-9b2d-7876611a39e5
> > Total devices 6 FS bytes used 4.69TiB
> > devid 1 size 2.73TiB used 1.71TiB path /dev/sdt
> > devid 2 size 2.73TiB used 1.70TiB path /dev/sdl
> > devid 3 size 2.73TiB used 1.71TiB path /dev/sdj
> > devid 4 size 2.73TiB used 1.70TiB path /dev/sds
> > devid 5 size 2.73TiB used 1.69TiB path /dev/sdg
> > devid 6 size 2.73TiB used 1.69TiB path /dev/sdc
> >
> >
> > My guess is that some drives dropped out while kernel was still
> > writing to rest thus causing inconsistency.
> > There should be some way to find out which drives has the most
> > up-to-date info and assume those are correct.
> > I tried to mount with
> > $ mount -o ro,degraded,rescue=usebackuproot /dev/sdt /mnt
> > but that didn't make any difference
> >
> > So any idea how to fix this filesystem?
> >
> > Thanks!
> >
> > Best regards,
> > Dāvis
>
> By the way
>
> $ btrfs-find-root /dev/sdt
> ERROR: failed to read block groups: Input/output error
> Superblock thinks the generation is 2262739
> Superblock thinks the level is 1
> Found tree root at 21057011679232 gen 2262739 level 1
> Well block 21056933724160(gen: 2262738 level: 1) seems good, but
> generation/level doesn't match, want gen: 2262739 level: 1
> Well block 21056867319808(gen: 2262737 level: 1) seems good, but
> generation/level doesn't match, want gen: 2262739 level: 1
> Well block 21056855900160(gen: 2262736 level: 1) seems good, but
> generation/level doesn't match, want gen: 2262739 level: 1
> Well block 21056850739200(gen: 2120504 level: 0) seems good, but
> generation/level doesn't match, want gen: 2262739 level: 1
>
> $ btrfs restore -l /dev/sdt
> tree key (EXTENT_TREE ROOT_ITEM 0) 21057008975872 level 3
> tree key (DEV_TREE ROOT_ITEM 0) 21056861863936 level 1
> tree key (FS_TREE ROOT_ITEM 0) 21063463993344 level 1
> tree key (CSUM_TREE ROOT_ITEM 0) 21057010728960 level 3
> tree key (UUID_TREE ROOT_ITEM 0) 21061425545216 level 0
> tree key (262 ROOT_ITEM 0) 21063533002752 level 0
> tree key (263 ROOT_ITEM 0) 21058890629120 level 2
> tree key (418 ROOT_ITEM 0) 21057902198784 level 2
> tree key (421 ROOT_ITEM 0) 21060222500864 level 2
> tree key (427 ROOT_ITEM 0) 21061262114816 level 2
> tree key (428 ROOT_ITEM 0) 21061278040064 level 2
> tree key (440 ROOT_ITEM 0) 21061362417664 level 2
> tree key (451 ROOT_ITEM 0) 21061017174016 level 2
> tree key (454 ROOT_ITEM 0) 21559581114368 level 1
> tree key (455 ROOT_ITEM 0) 21079314776064 level 1
> tree key (456 ROOT_ITEM 0) 21058026831872 level 2
> tree key (457 ROOT_ITEM 0) 21060907909120 level 3
> tree key (497 ROOT_ITEM 0) 21058120990720 level 2
> tree key (571 ROOT_ITEM 0) 21058195668992 level 2
> tree key (599 ROOT_ITEM 0) 21058818015232 level 2
> tree key (635 ROOT_ITEM 0) 21056973766656 level 2
> tree key (638 ROOT_ITEM 0) 21061023072256 level 0
> tree key (676 ROOT_ITEM 0) 21061314330624 level 2
> tree key (3937 ROOT_ITEM 0) 21061408686080 level 0
> tree key (3938 ROOT_ITEM 0) 21079315841024 level 1
> tree key (3957 ROOT_ITEM 0) 21061419139072 level 2
> tree key (6128 ROOT_ITEM 0) 21061400018944 level 1
> tree key (8575 ROOT_ITEM 0) 21061023055872 level 0
> tree key (18949 ROOT_ITEM 1728623) 21080421875712 level 1
> tree key (18950 ROOT_ITEM 1728624) 21080424726528 level 2
> tree key (18951 ROOT_ITEM 1728625) 21080424824832 level 2
> tree key (18952 ROOT_ITEM 1728626) 21080426004480 level 3
> tree key (18953 ROOT_ITEM 1728627) 21080422105088 level 2
> tree key (18954 ROOT_ITEM 1728628) 21080424497152 level 2
> tree key (18955 ROOT_ITEM 1728629) 21080426332160 level 2
> tree key (18956 ROOT_ITEM 1728631) 21080423645184 level 2
> tree key (18957 ROOT_ITEM 1728632) 21080425316352 level 2
> tree key (18958 ROOT_ITEM 1728633) 21080423972864 level 2
> tree key (18959 ROOT_ITEM 1728634) 21080422400000 level 2
> tree key (18960 ROOT_ITEM 1728635) 21080422662144 level 2
> tree key (18961 ROOT_ITEM 1728636) 21080423153664 level 2
> tree key (18962 ROOT_ITEM 1728637) 21080425414656 level 2
> tree key (18963 ROOT_ITEM 1728638) 21080421171200 level 1
> tree key (18964 ROOT_ITEM 1728639) 21080423481344 level 2
> tree key (19721 ROOT_ITEM 0) 21076937326592 level 2
> checksum verify failed on 21057125580800 found 00000026 wanted 00000035
> checksum verify failed on 21057108082688 found 00000074 wanted FFFFFFC5
> checksum verify failed on 21057108082688 found 000000ED wanted FFFFFFC5
> checksum verify failed on 21057108082688 found 00000074 wanted FFFFFFC5
> Csum didn't match
From what I understand it seems that some EXTENT_ITEM is corrupted and
when mount tries to read block groups it encounters csum mismatch for
it and immediatly aborts.
Is there some tool I could use to check this EXTENT_ITEM and see if it
can be fixed or maybe just removed?
Basically I guess I need to find physical location on disk from this
block number.
Also I think ignoring csum for btrfs inspect would be useful.
$ btrfs inspect dump-tree -b 21057050689536 /dev/sda
btrfs-progs v5.10.1
node 21057050689536 level 1 items 281 free space 212 generation
2262739 owner EXTENT_TREE
node 21057050689536 flags 0x1(WRITTEN) backref revision 1
fs uuid 8aef11a9-beb6-49ea-9b2d-7876611a39e5
chunk uuid 4ffec48c-28ed-419d-ba87-229c0adb2ab9
[...]
key (19264654909440 EXTENT_ITEM 524288) block 21057101103104 gen 2262739
[...]
$ btrfs inspect dump-tree -b 21057101103104 /dev/sda
btrfs-progs v5.10.1
checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
checksum verify failed on 21057101103104 found 0000009C wanted 00000075
checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
Csum didn't match
ERROR: failed to read tree block 21057101103104
Thanks!
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-02-19 3:03 ` Dāvis Mosāns
@ 2021-02-19 5:16 ` Chris Murphy
0 siblings, 0 replies; 10+ messages in thread
From: Chris Murphy @ 2021-02-19 5:16 UTC (permalink / raw)
To: Dāvis Mosāns; +Cc: Btrfs BTRFS
On Thu, Feb 18, 2021 at 8:08 PM Dāvis Mosāns <davispuh@gmail.com> wrote:
>
> ceturtd., 2021. g. 14. janv., plkst. 01:39 — lietotājs Dāvis Mosāns
> (<davispuh@gmail.com>) rakstīja:
> >
> > >
> > > Hi,
> > >
> > > I've 6x 3TB HDD RAID1 BTRFS filesystem where HBA card failed and
> > > caused some corruption.
> > > When I try to mount it I get
> > > $ mount /dev/sdt /mnt
> > > mount: /mnt/: wrong fs type, bad option, bad superblock on /dev/sdt,
> > > missing codepage or helper program, or other error
> > > $ dmesg | tail -n 9
> > > [ 617.158962] BTRFS info (device sdt): disk space caching is enabled
> > > [ 617.158965] BTRFS info (device sdt): has skinny extents
> > > [ 617.756924] BTRFS info (device sdt): bdev /dev/sdl errs: wr 0, rd
> > > 0, flush 0, corrupt 473, gen 0
> > > [ 617.756929] BTRFS info (device sdt): bdev /dev/sdj errs: wr 31626,
> > > rd 18765, flush 178, corrupt 5841, gen 0
> > > [ 617.756933] BTRFS info (device sdt): bdev /dev/sdg errs: wr 6867,
> > > rd 2640, flush 178, corrupt 1066, gen 0
> > > [ 631.353725] BTRFS warning (device sdt): sdt checksum verify failed
> > > on 21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
> > > [ 631.376024] BTRFS warning (device sdt): sdt checksum verify failed
> > > on 21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
> > > [ 631.376038] BTRFS error (device sdt): failed to read block groups: -5
> > > [ 631.422811] BTRFS error (device sdt): open_ctree failed
> > >
> > > $ uname -r
> > > 5.9.14-arch1-1
> > > $ btrfs --version
> > > btrfs-progs v5.9
> > > $ btrfs check /dev/sdt
> > > Opening filesystem to check...
> > > checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> > > checksum verify failed on 21057101103104 found 0000009C wanted 00000075
> > > checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> > > Csum didn't match
> > > ERROR: failed to read block groups: Input/output error
> > > ERROR: cannot open file system
> > >
> > > $ btrfs filesystem show
> > > Label: 'RAID' uuid: 8aef11a9-beb6-49ea-9b2d-7876611a39e5
> > > Total devices 6 FS bytes used 4.69TiB
> > > devid 1 size 2.73TiB used 1.71TiB path /dev/sdt
> > > devid 2 size 2.73TiB used 1.70TiB path /dev/sdl
> > > devid 3 size 2.73TiB used 1.71TiB path /dev/sdj
> > > devid 4 size 2.73TiB used 1.70TiB path /dev/sds
> > > devid 5 size 2.73TiB used 1.69TiB path /dev/sdg
> > > devid 6 size 2.73TiB used 1.69TiB path /dev/sdc
> > >
> > >
> > > My guess is that some drives dropped out while kernel was still
> > > writing to rest thus causing inconsistency.
> > > There should be some way to find out which drives has the most
> > > up-to-date info and assume those are correct.
> > > I tried to mount with
> > > $ mount -o ro,degraded,rescue=usebackuproot /dev/sdt /mnt
> > > but that didn't make any difference
> > >
> > > So any idea how to fix this filesystem?
> > >
> > > Thanks!
> > >
> > > Best regards,
> > > Dāvis
> >
> > By the way
> >
> > $ btrfs-find-root /dev/sdt
> > ERROR: failed to read block groups: Input/output error
> > Superblock thinks the generation is 2262739
> > Superblock thinks the level is 1
> > Found tree root at 21057011679232 gen 2262739 level 1
> > Well block 21056933724160(gen: 2262738 level: 1) seems good, but
> > generation/level doesn't match, want gen: 2262739 level: 1
> > Well block 21056867319808(gen: 2262737 level: 1) seems good, but
> > generation/level doesn't match, want gen: 2262739 level: 1
> > Well block 21056855900160(gen: 2262736 level: 1) seems good, but
> > generation/level doesn't match, want gen: 2262739 level: 1
> > Well block 21056850739200(gen: 2120504 level: 0) seems good, but
> > generation/level doesn't match, want gen: 2262739 level: 1
> >
> > $ btrfs restore -l /dev/sdt
> > tree key (EXTENT_TREE ROOT_ITEM 0) 21057008975872 level 3
> > tree key (DEV_TREE ROOT_ITEM 0) 21056861863936 level 1
> > tree key (FS_TREE ROOT_ITEM 0) 21063463993344 level 1
> > tree key (CSUM_TREE ROOT_ITEM 0) 21057010728960 level 3
> > tree key (UUID_TREE ROOT_ITEM 0) 21061425545216 level 0
> > tree key (262 ROOT_ITEM 0) 21063533002752 level 0
> > tree key (263 ROOT_ITEM 0) 21058890629120 level 2
> > tree key (418 ROOT_ITEM 0) 21057902198784 level 2
> > tree key (421 ROOT_ITEM 0) 21060222500864 level 2
> > tree key (427 ROOT_ITEM 0) 21061262114816 level 2
> > tree key (428 ROOT_ITEM 0) 21061278040064 level 2
> > tree key (440 ROOT_ITEM 0) 21061362417664 level 2
> > tree key (451 ROOT_ITEM 0) 21061017174016 level 2
> > tree key (454 ROOT_ITEM 0) 21559581114368 level 1
> > tree key (455 ROOT_ITEM 0) 21079314776064 level 1
> > tree key (456 ROOT_ITEM 0) 21058026831872 level 2
> > tree key (457 ROOT_ITEM 0) 21060907909120 level 3
> > tree key (497 ROOT_ITEM 0) 21058120990720 level 2
> > tree key (571 ROOT_ITEM 0) 21058195668992 level 2
> > tree key (599 ROOT_ITEM 0) 21058818015232 level 2
> > tree key (635 ROOT_ITEM 0) 21056973766656 level 2
> > tree key (638 ROOT_ITEM 0) 21061023072256 level 0
> > tree key (676 ROOT_ITEM 0) 21061314330624 level 2
> > tree key (3937 ROOT_ITEM 0) 21061408686080 level 0
> > tree key (3938 ROOT_ITEM 0) 21079315841024 level 1
> > tree key (3957 ROOT_ITEM 0) 21061419139072 level 2
> > tree key (6128 ROOT_ITEM 0) 21061400018944 level 1
> > tree key (8575 ROOT_ITEM 0) 21061023055872 level 0
> > tree key (18949 ROOT_ITEM 1728623) 21080421875712 level 1
> > tree key (18950 ROOT_ITEM 1728624) 21080424726528 level 2
> > tree key (18951 ROOT_ITEM 1728625) 21080424824832 level 2
> > tree key (18952 ROOT_ITEM 1728626) 21080426004480 level 3
> > tree key (18953 ROOT_ITEM 1728627) 21080422105088 level 2
> > tree key (18954 ROOT_ITEM 1728628) 21080424497152 level 2
> > tree key (18955 ROOT_ITEM 1728629) 21080426332160 level 2
> > tree key (18956 ROOT_ITEM 1728631) 21080423645184 level 2
> > tree key (18957 ROOT_ITEM 1728632) 21080425316352 level 2
> > tree key (18958 ROOT_ITEM 1728633) 21080423972864 level 2
> > tree key (18959 ROOT_ITEM 1728634) 21080422400000 level 2
> > tree key (18960 ROOT_ITEM 1728635) 21080422662144 level 2
> > tree key (18961 ROOT_ITEM 1728636) 21080423153664 level 2
> > tree key (18962 ROOT_ITEM 1728637) 21080425414656 level 2
> > tree key (18963 ROOT_ITEM 1728638) 21080421171200 level 1
> > tree key (18964 ROOT_ITEM 1728639) 21080423481344 level 2
> > tree key (19721 ROOT_ITEM 0) 21076937326592 level 2
> > checksum verify failed on 21057125580800 found 00000026 wanted 00000035
> > checksum verify failed on 21057108082688 found 00000074 wanted FFFFFFC5
> > checksum verify failed on 21057108082688 found 000000ED wanted FFFFFFC5
> > checksum verify failed on 21057108082688 found 00000074 wanted FFFFFFC5
> > Csum didn't match
>
> From what I understand it seems that some EXTENT_ITEM is corrupted and
> when mount tries to read block groups it encounters csum mismatch for
> it and immediatly aborts.
> Is there some tool I could use to check this EXTENT_ITEM and see if it
> can be fixed or maybe just removed?
> Basically I guess I need to find physical location on disk from this
> block number.
> Also I think ignoring csum for btrfs inspect would be useful.
>
> $ btrfs inspect dump-tree -b 21057050689536 /dev/sda
> btrfs-progs v5.10.1
> node 21057050689536 level 1 items 281 free space 212 generation
> 2262739 owner EXTENT_TREE
> node 21057050689536 flags 0x1(WRITTEN) backref revision 1
> fs uuid 8aef11a9-beb6-49ea-9b2d-7876611a39e5
> chunk uuid 4ffec48c-28ed-419d-ba87-229c0adb2ab9
> [...]
> key (19264654909440 EXTENT_ITEM 524288) block 21057101103104 gen 2262739
> [...]
>
>
> $ btrfs inspect dump-tree -b 21057101103104 /dev/sda
> btrfs-progs v5.10.1
> checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> checksum verify failed on 21057101103104 found 0000009C wanted 00000075
> checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> Csum didn't match
> ERROR: failed to read tree block 21057101103104
>
>
> Thanks!
What do you get for
btrfs rescue super -v /dev/
btrfs check -b /dev/
You might try kernel 5.11 which has a new mount option that will skip
bad roots and csums. It's 'mount -o ro,rescue=all' and while it won't
let you fix it, in the off chance it mounts, it'll let you get data
out before trying to repair the file system, which sometimes makes
things worse.
--
Chris Murphy
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-01-13 23:09 ERROR: failed to read block groups: Input/output error Dāvis Mosāns
2021-01-13 23:39 ` Dāvis Mosāns
@ 2021-02-19 19:29 ` Zygo Blaxell
2021-02-20 23:45 ` Dāvis Mosāns
1 sibling, 1 reply; 10+ messages in thread
From: Zygo Blaxell @ 2021-02-19 19:29 UTC (permalink / raw)
To: Dāvis Mosāns; +Cc: Btrfs BTRFS
On Thu, Jan 14, 2021 at 01:09:40AM +0200, Dāvis Mosāns wrote:
> Hi,
>
> I've 6x 3TB HDD RAID1 BTRFS filesystem where HBA card failed and
> caused some corruption.
> When I try to mount it I get
> $ mount /dev/sdt /mnt
> mount: /mnt/: wrong fs type, bad option, bad superblock on /dev/sdt,
> missing codepage or helper program, or other error
> $ dmesg | tail -n 9
> [ 617.158962] BTRFS info (device sdt): disk space caching is enabled
> [ 617.158965] BTRFS info (device sdt): has skinny extents
> [ 617.756924] BTRFS info (device sdt): bdev /dev/sdl errs: wr 0, rd
> 0, flush 0, corrupt 473, gen 0
> [ 617.756929] BTRFS info (device sdt): bdev /dev/sdj errs: wr 31626,
> rd 18765, flush 178, corrupt 5841, gen 0
> [ 617.756933] BTRFS info (device sdt): bdev /dev/sdg errs: wr 6867,
> rd 2640, flush 178, corrupt 1066, gen 0
You have write errors on 2 disks, read errors on 3 disks, and raid1
tolerates only 1 disk failure, so successful recovery is unlikely.
> [ 631.353725] BTRFS warning (device sdt): sdt checksum verify failed
> on 21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
> [ 631.376024] BTRFS warning (device sdt): sdt checksum verify failed
> on 21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
Both copies of this metadata block are corrupted, differently.
This is consistent with some kinds of HBA failure: every outgoing block
from the host is potentially corrupted, usually silently. Due to the HBA
failure, there is no indication of failure available to the filesystem
until after several corrupt blocks are written to disk. By the time
failure is detected, damage is extensive, especially for metadata where
overwrites are frequent.
This is failure mode that you need backups to recover from (or mirror
disks on separate, non-failing HBA hardware).
> [ 631.376038] BTRFS error (device sdt): failed to read block groups: -5
> [ 631.422811] BTRFS error (device sdt): open_ctree failed
>
> $ uname -r
> 5.9.14-arch1-1
> $ btrfs --version
> btrfs-progs v5.9
> $ btrfs check /dev/sdt
> Opening filesystem to check...
> checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> checksum verify failed on 21057101103104 found 0000009C wanted 00000075
> checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> Csum didn't match
> ERROR: failed to read block groups: Input/output error
> ERROR: cannot open file system
>
> $ btrfs filesystem show
> Label: 'RAID' uuid: 8aef11a9-beb6-49ea-9b2d-7876611a39e5
> Total devices 6 FS bytes used 4.69TiB
> devid 1 size 2.73TiB used 1.71TiB path /dev/sdt
> devid 2 size 2.73TiB used 1.70TiB path /dev/sdl
> devid 3 size 2.73TiB used 1.71TiB path /dev/sdj
> devid 4 size 2.73TiB used 1.70TiB path /dev/sds
> devid 5 size 2.73TiB used 1.69TiB path /dev/sdg
> devid 6 size 2.73TiB used 1.69TiB path /dev/sdc
>
>
> My guess is that some drives dropped out while kernel was still
> writing to rest thus causing inconsistency.
> There should be some way to find out which drives has the most
> up-to-date info and assume those are correct.
Neither available copy is correct, so the kernel's self-healing mechanism
doesn't work. Thousands of pages are damaged, possibly only with minor
errors, but multiply a minor error by a thousand and it's no longer minor.
At this point it is a forensic recovery exercise.
> I tried to mount with
> $ mount -o ro,degraded,rescue=usebackuproot /dev/sdt /mnt
> but that didn't make any difference
>
> So any idea how to fix this filesystem?
Before you can mount the filesystem read-write again, you would need to
rebuild the extent tree from the surviving pages of the subvol trees.
All other metadata pages on the filesystem must be scanned, any excess
reference items must be deleted, and any missing reference items must
be inserted. Once the metadata references are correct, btrfs can
rebuild the free space maps, and then you can scrub and delete/replace
any damaged data files.
'btrfs check --repair' might work if only a handful of blocks are
corrupted (it takes a few short cuts and can repair minor damage)
but according to your dev stats you have thousands of corrupted blocks,
so the filesystem is probably beyond the capabilities of this tool.
'btrfs check --repair --init-extent-tree' is a brute-force operation that
will more or less rebuild the entire filesystem by scraping metadata
leaf pages off the disks. This is your only hope here, and it's not a
good one.
Both methods are likely to fail in the presence of so much corruption
and they may take so long to run that mkfs + restore from backups could
be significantly faster. Definitely extract any data from the filesystem
that you want to keep _before_ attempting any of these operations.
It might be possible to recover by manually inspecting the corrupted
metadata blocks and making guesses and adjustments, but that could take
even longer than check --repair if there are thousands of damaged pages.
> Thanks!
>
> Best regards,
> Dāvis
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-02-19 19:29 ` Zygo Blaxell
@ 2021-02-20 23:45 ` Dāvis Mosāns
2021-02-21 1:03 ` Dāvis Mosāns
2021-02-22 5:22 ` Zygo Blaxell
0 siblings, 2 replies; 10+ messages in thread
From: Dāvis Mosāns @ 2021-02-20 23:45 UTC (permalink / raw)
To: Zygo Blaxell, Chris Murphy; +Cc: Btrfs BTRFS
piektd., 2021. g. 19. febr., plkst. 07:16 — lietotājs Chris Murphy
(<lists@colorremedies.com>) rakstīja:
> [...]
> What do you get for
>
> btrfs rescue super -v /dev/
>
That seems to be all good
$ btrfs rescue super -v /dev/sda
All Devices:
Device: id = 2, name = /dev/sdt
Device: id = 4, name = /dev/sdj
Device: id = 3, name = /dev/sdg
Device: id = 6, name = /dev/sdb
Device: id = 1, name = /dev/sdl
Device: id = 5, name = /dev/sda
Before Recovering:
[All good supers]:
device name = /dev/sdt
superblock bytenr = 65536
device name = /dev/sdt
superblock bytenr = 67108864
device name = /dev/sdt
superblock bytenr = 274877906944
device name = /dev/sdj
superblock bytenr = 65536
device name = /dev/sdj
superblock bytenr = 67108864
device name = /dev/sdj
superblock bytenr = 274877906944
device name = /dev/sdg
superblock bytenr = 65536
device name = /dev/sdg
superblock bytenr = 67108864
device name = /dev/sdg
superblock bytenr = 274877906944
device name = /dev/sdb
superblock bytenr = 65536
device name = /dev/sdb
superblock bytenr = 67108864
device name = /dev/sdb
superblock bytenr = 274877906944
device name = /dev/sdl
superblock bytenr = 65536
device name = /dev/sdl
superblock bytenr = 67108864
device name = /dev/sdl
superblock bytenr = 274877906944
device name = /dev/sda
superblock bytenr = 65536
device name = /dev/sda
superblock bytenr = 67108864
device name = /dev/sda
superblock bytenr = 274877906944
[All bad supers]:
All supers are valid, no need to recover
$ btrfs inspect dump-super -f /dev/sda
superblock: bytenr=65536, device=/dev/sda
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xf72e6634 [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 8aef11a9-beb6-49ea-9b2d-7876611a39e5
metadata_uuid 8aef11a9-beb6-49ea-9b2d-7876611a39e5
label RAID
generation 2262739
root 21057011679232
sys_array_size 129
chunk_root_generation 2205349
root_level 1
chunk_root 21056798736384
chunk_root_level 1
log_root 0
log_root_transid 0
log_root_level 0
total_bytes 18003557892096
bytes_used 5154539671552
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 6
compat_flags 0x0
compat_ro_flags 0x0
incompat_flags 0x36b
( MIXED_BACKREF |
DEFAULT_SUBVOL |
COMPRESS_LZO |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 2262739
uuid_tree_generation 1807368
dev_item.uuid 098e5987-adf9-4a37-aad0-dff0819c6588
dev_item.fsid 8aef11a9-beb6-49ea-9b2d-7876611a39e5 [match]
dev_item.type 0
dev_item.total_bytes 3000592982016
dev_item.bytes_used 1860828135424
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 5
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 21056797999104)
length 33554432 owner 2 stripe_len 65536 type SYSTEM|RAID1
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 5 offset 4329570304
dev_uuid 098e5987-adf9-4a37-aad0-dff0819c6588
stripe 1 devid 6 offset 4329570304
dev_uuid 7036ea10-4dce-48c6-b6d5-66378ba54b03
backup_roots[4]:
backup 0:
backup_tree_root: 21056867319808 gen: 2262737 level: 1
backup_chunk_root: 21056798736384 gen: 2205349 level: 1
backup_extent_root: 21056867106816 gen: 2262737 level: 3
backup_fs_root: 21063463993344 gen: 2095377 level: 1
backup_dev_root: 21056861863936 gen: 2262736 level: 1
backup_csum_root: 21056868122624 gen: 2262737 level: 3
backup_total_bytes: 18003557892096
backup_bytes_used: 5154539933696
backup_num_devices: 6
backup 1:
backup_tree_root: 21056933724160 gen: 2262738 level: 1
backup_chunk_root: 21056798736384 gen: 2205349 level: 1
backup_extent_root: 21056867762176 gen: 2262738 level: 3
backup_fs_root: 21063463993344 gen: 2095377 level: 1
backup_dev_root: 21056861863936 gen: 2262736 level: 1
backup_csum_root: 21056944685056 gen: 2262738 level: 3
backup_total_bytes: 18003557892096
backup_bytes_used: 5154548318208
backup_num_devices: 6
backup 2:
backup_tree_root: 21057011679232 gen: 2262739 level: 1
backup_chunk_root: 21056798736384 gen: 2205349 level: 1
backup_extent_root: 21057133690880 gen: 2262740 level: 3
backup_fs_root: 21063463993344 gen: 2095377 level: 1
backup_dev_root: 21056861863936 gen: 2262736 level: 1
backup_csum_root: 21057139916800 gen: 2262740 level: 3
backup_total_bytes: 18003557892096
backup_bytes_used: 5154540572672
backup_num_devices: 6
backup 3:
backup_tree_root: 21056855900160 gen: 2262736 level: 1
backup_chunk_root: 21056798736384 gen: 2205349 level: 1
backup_extent_root: 21056854228992 gen: 2262736 level: 3
backup_fs_root: 21063463993344 gen: 2095377 level: 1
backup_dev_root: 21056861863936 gen: 2262736 level: 1
backup_csum_root: 21056857341952 gen: 2262736 level: 3
backup_total_bytes: 18003557892096
backup_bytes_used: 5154539933696
backup_num_devices: 6
> btrfs check -b /dev/
>
This gives lots of errors and not sure if main superblock can be fixed
with less errors.
$ btrfs check -b /dev/sda
Opening filesystem to check...
Checking filesystem on /dev/sda
UUID: 8aef11a9-beb6-49ea-9b2d-7876611a39e5
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space cache
block group 20733568155648 has wrong amount of free space, free space
cache has 696320 block group has 729088
failed to load free space cache for block group 20733568155648
[4/7] checking fs roots
root 457 inode 682022 errors 1040, bad file extent, some csum missing
root 457 inode 2438260 errors 1040, bad file extent, some csum missing
root 599 inode 228661 errors 1040, bad file extent, some csum missing
root 18950 inode 2298187 errors 1040, bad file extent, some csum missing
[...] 21068 entries like: root 18950 inode X errors 1040, bad file
extent, some csum missing
root 18950 inode 13845002 errors 1040, bad file extent, some csum missing
root 18952 inode 682022 errors 1040, bad file extent, some csum missing
root 18952 inode 2438161 errors 1040, bad file extent, some csum missing
root 18952 inode 2438162 errors 1040, bad file extent, some csum missing
root 18952 inode 2438166 errors 1040, bad file extent, some csum missing
root 18952 inode 2438167 errors 1040, bad file extent, some csum missing
root 18952 inode 2438170 errors 1040, bad file extent, some csum missing
root 18952 inode 2438187 errors 1040, bad file extent, some csum missing
root 18952 inode 2438260 errors 1040, bad file extent, some csum missing
root 18955 inode 228661 errors 1040, bad file extent, some csum missing
root 28874 inode 682022 errors 1040, bad file extent, some csum missing
root 28874 inode 2438162 errors 1040, bad file extent, some csum missing
root 28874 inode 2438187 errors 1040, bad file extent, some csum missing
root 28874 inode 2438260 errors 1040, bad file extent, some csum missing
root 28877 inode 228661 errors 1040, bad file extent, some csum missing
root 29405 inode 682022 errors 1040, bad file extent, some csum missing
root 29405 inode 2438260 errors 1040, bad file extent, some csum missing
root 29408 inode 228661 errors 1040, bad file extent, some csum missing
root 29581 inode 682022 errors 1040, bad file extent, some csum missing
root 29581 inode 2438260 errors 1040, bad file extent, some csum missing
root 29584 inode 228661 errors 1040, bad file extent, some csum missing
root 29597 inode 682022 errors 1040, bad file extent, some csum missing
root 29597 inode 2438260 errors 1040, bad file extent, some csum missing
root 29600 inode 228661 errors 1040, bad file extent, some csum missing
root 29613 inode 682022 errors 1040, bad file extent, some csum missing
root 29613 inode 2438260 errors 1040, bad file extent, some csum missing
root 29616 inode 228661 errors 1040, bad file extent, some csum missing
root 29629 inode 682022 errors 1040, bad file extent, some csum missing
root 29629 inode 2438260 errors 1040, bad file extent, some csum missing
root 29632 inode 228661 errors 1040, bad file extent, some csum missing
root 29645 inode 682022 errors 1040, bad file extent, some csum missing
root 29645 inode 2438260 errors 1040, bad file extent, some csum missing
root 29648 inode 228661 errors 1040, bad file extent, some csum missing
root 29661 inode 682022 errors 1040, bad file extent, some csum missing
root 29661 inode 2438260 errors 1040, bad file extent, some csum missing
root 29664 inode 228661 errors 1040, bad file extent, some csum missing
ERROR: errors found in fs roots
found 5152420646912 bytes used, error(s) found
total csum bytes: 4748365652
total tree bytes: 22860578816
total fs tree bytes: 15688564736
total extent tree bytes: 1642725376
btree space waste bytes: 3881167880
file data blocks allocated: 24721653870592
referenced 7836810440704
I also tried
$ btrfs check -r 21056933724160 /dev/sda
but the output was exactly same so seems it doesn't really make difference.
So I think btrfs check -b --repair should be able to fix most of things.
> You might try kernel 5.11 which has a new mount option that will skip
> bad roots and csums. It's 'mount -o ro,rescue=all' and while it won't
> let you fix it, in the off chance it mounts, it'll let you get data
> out before trying to repair the file system, which sometimes makes
> things worse.
>
>
It doesn't make any difference, still doesn't mount
$ uname -r
5.11.0-arch2-1
$ sudo mount -o ro,rescue=all /dev/sda ./RAID
mount: /mnt/RAID: wrong fs type, bad option, bad superblock on
/dev/sda, missing codepage or helper program, or other error.
BTRFS warning (device sdl): sdl checksum verify failed on
21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
BTRFS warning (device sdl): sdl checksum verify failed on
21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
BTRFS error (device sdl): failed to read block groups: -5
BTRFS error (device sdl): open_ctree failed
It seems there should be a way to mount with backup tree root like I
did for check but strangly usebackuproot doesn't do that...
piektd., 2021. g. 19. febr., plkst. 21:29 — lietotājs Zygo Blaxell
(<ce3g8jdj@umail.furryterror.org>) rakstīja:
>
> On Thu, Jan 14, 2021 at 01:09:40AM +0200, Dāvis Mosāns wrote:
> > Hi,
> >
> > I've 6x 3TB HDD RAID1 BTRFS filesystem where HBA card failed and
> > caused some corruption.
> > When I try to mount it I get
> > $ mount /dev/sdt /mnt
> > mount: /mnt/: wrong fs type, bad option, bad superblock on /dev/sdt,
> > missing codepage or helper program, or other error
> > $ dmesg | tail -n 9
> > [ 617.158962] BTRFS info (device sdt): disk space caching is enabled
> > [ 617.158965] BTRFS info (device sdt): has skinny extents
> > [ 617.756924] BTRFS info (device sdt): bdev /dev/sdl errs: wr 0, rd
> > 0, flush 0, corrupt 473, gen 0
> > [ 617.756929] BTRFS info (device sdt): bdev /dev/sdj errs: wr 31626,
> > rd 18765, flush 178, corrupt 5841, gen 0
> > [ 617.756933] BTRFS info (device sdt): bdev /dev/sdg errs: wr 6867,
> > rd 2640, flush 178, corrupt 1066, gen 0
>
> You have write errors on 2 disks, read errors on 3 disks, and raid1
> tolerates only 1 disk failure, so successful recovery is unlikely.
>
Those wr/rd/corrupt error counts are inflated/misleading, in past when
some HDD drops out I've had them increase in huge numbers, but after
running scrub usually it was able to fix almost everything except like
few files that could be just deleted. Only now it's possible that it
failed while scrub was running making it a lot worse.
> > [ 631.353725] BTRFS warning (device sdt): sdt checksum verify failed
> > on 21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
> > [ 631.376024] BTRFS warning (device sdt): sdt checksum verify failed
> > on 21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
>
> Both copies of this metadata block are corrupted, differently.
>
> This is consistent with some kinds of HBA failure: every outgoing block
> from the host is potentially corrupted, usually silently. Due to the HBA
> failure, there is no indication of failure available to the filesystem
> until after several corrupt blocks are written to disk. By the time
> failure is detected, damage is extensive, especially for metadata where
> overwrites are frequent.
>
I don't think it's that bad here. My guess is that it failed while
updating extent tree and some part of it didn't got written to disk. I
want to check how it looks like on disk, is there some tool to map
block number to offset in disk?
> This is failure mode that you need backups to recover from (or mirror
> disks on separate, non-failing HBA hardware).
>
I don't know how btrfs decides in which disks it stores copies or if
it's always in same disk. To prevent such failure in future I could
split RAID1 across 2 different HBAs but it's not clear which disks
would need to be seperated.
> > [ 631.376038] BTRFS error (device sdt): failed to read block groups: -5
> > [ 631.422811] BTRFS error (device sdt): open_ctree failed
> >
> > $ uname -r
> > 5.9.14-arch1-1
> > $ btrfs --version
> > btrfs-progs v5.9
> > $ btrfs check /dev/sdt
> > Opening filesystem to check...
> > checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> > checksum verify failed on 21057101103104 found 0000009C wanted 00000075
> > checksum verify failed on 21057101103104 found 000000B9 wanted 00000075
> > Csum didn't match
> > ERROR: failed to read block groups: Input/output error
> > ERROR: cannot open file system
> >
> > $ btrfs filesystem show
> > Label: 'RAID' uuid: 8aef11a9-beb6-49ea-9b2d-7876611a39e5
> > Total devices 6 FS bytes used 4.69TiB
> > devid 1 size 2.73TiB used 1.71TiB path /dev/sdt
> > devid 2 size 2.73TiB used 1.70TiB path /dev/sdl
> > devid 3 size 2.73TiB used 1.71TiB path /dev/sdj
> > devid 4 size 2.73TiB used 1.70TiB path /dev/sds
> > devid 5 size 2.73TiB used 1.69TiB path /dev/sdg
> > devid 6 size 2.73TiB used 1.69TiB path /dev/sdc
> >
> >
> > My guess is that some drives dropped out while kernel was still
> > writing to rest thus causing inconsistency.
> > There should be some way to find out which drives has the most
> > up-to-date info and assume those are correct.
>
> Neither available copy is correct, so the kernel's self-healing mechanism
> doesn't work. Thousands of pages are damaged, possibly only with minor
> errors, but multiply a minor error by a thousand and it's no longer minor.
>
> At this point it is a forensic recovery exercise.
>
> > I tried to mount with
> > $ mount -o ro,degraded,rescue=usebackuproot /dev/sdt /mnt
> > but that didn't make any difference
> >
> > So any idea how to fix this filesystem?
>
> Before you can mount the filesystem read-write again, you would need to
> rebuild the extent tree from the surviving pages of the subvol trees.
> All other metadata pages on the filesystem must be scanned, any excess
> reference items must be deleted, and any missing reference items must
> be inserted. Once the metadata references are correct, btrfs can
> rebuild the free space maps, and then you can scrub and delete/replace
> any damaged data files.
>
> 'btrfs check --repair' might work if only a handful of blocks are
> corrupted (it takes a few short cuts and can repair minor damage)
> but according to your dev stats you have thousands of corrupted blocks,
> so the filesystem is probably beyond the capabilities of this tool.
>
> 'btrfs check --repair --init-extent-tree' is a brute-force operation that
> will more or less rebuild the entire filesystem by scraping metadata
> leaf pages off the disks. This is your only hope here, and it's not a
> good one.
>
I don't want to use --init-extent-tree because I don't want to reset
everything, but only corrupted things. Also btrfs check --repair
doesn't work as it aborts too quickly, only using with -b flag could
fix it I think.
> Both methods are likely to fail in the presence of so much corruption
> and they may take so long to run that mkfs + restore from backups could
> be significantly faster. Definitely extract any data from the filesystem
> that you want to keep _before_ attempting any of these operations.
>
It's not really about time, I rather want to reduce possilble data
loss as much as possible.
I can't mount it even read-only so it seems the only way to get data
out is by using btrfs restore which seems to work fine but does it
verify file checksums? It looks like it doesn't... I have some files
where it said:
We seem to be looping a lot on ./something, do you want to keep going
on ? (y/N/a)
When I checked this file I see that it's corrupted. Basically I want
restore only files with valid checksums and then have a list of
corrupted files. From currupted files there are few I want to see if
they can be recovered. I have lot of of snapshots but even the oldest
ones are corrupted in exactly same way - they're identical. It seems I
need to find previous copy of this file if it exsists at all... Any
idea how to find previous version of file?
I tried
$ btrfs restore -u 2 -t 21056933724160
with different superblocks/tree roots but they all give same corrupted file.
The file looks like this
$ hexdump -C file | head -n 5
00000000 27 47 10 00 6d 64 61 74 7b 7b 7b 7b 7b 7b 7b 7b |'G..mdat{{{{{{{{|
00000010 7a 7a 79 79 7a 7a 7a 7a 7b 7b 7b 7c 7c 7c 7b 7b |zzyyzzzz{{{|||{{|
00000020 7c 7c 7c 7b 7b 7b 7b 7b 7c 7c 7c 7c 7c 7b 7b 7b ||||{{{{{|||||{{{|
00000030 7b 7b 7b 7b 7b 7b 7b 7b 7b 7a 7a 7a 7a 79 79 7a |{{{{{{{{{zzzzyyz|
00000040 7b 7b 7b 7b 7a 7b 7b 7c 7b 7c 7c 7b 7c 7c 7b 7b |{{{{z{{|{||{||{{|
Those repeated 7a/b/c is wrong data. Also I'm not sure if these files
have been corrupted now or more in past... So I need to check if
checksum matches.
> It might be possible to recover by manually inspecting the corrupted
> metadata blocks and making guesses and adjustments, but that could take
> even longer than check --repair if there are thousands of damaged pages.
>
I want to look into this but not sure if there are any tools with
which it would be easy to inspect data. dump-tree is nice but it
doesn't work when checksum is incorrect.
My current plan is:
1. btrfs restore only valid files, how? I don't want to mix good files
with corrupted ones together
2. look into how exactly extent tree is corrupted
3. try to see if few of corrupted files can be recovered in some way
4. do btrfs check -b --repair (maybe if extent tree can be fixed then
wouldn't need to use -b flag)
5. try to mount and btrfs scrub
6. maybe wipe and recreate new fielsystem
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-02-20 23:45 ` Dāvis Mosāns
@ 2021-02-21 1:03 ` Dāvis Mosāns
2021-02-21 1:08 ` Qu Wenruo
2021-02-22 5:22 ` Zygo Blaxell
1 sibling, 1 reply; 10+ messages in thread
From: Dāvis Mosāns @ 2021-02-21 1:03 UTC (permalink / raw)
To: Zygo Blaxell, Chris Murphy; +Cc: Btrfs BTRFS
I just found something really strange, it seems pointers for extent
tree and csum tree have somehow gotten swapped...
$ btrfs inspect dump-super -f /dev/sda | grep backup_extent_root
backup_extent_root: 21056867106816 gen: 2262737 level: 3
backup_extent_root: 21056867762176 gen: 2262738 level: 3
backup_extent_root: 21057133690880 gen: 2262740 level: 3 <<
points to CSUM_TREE
backup_extent_root: 21056854228992 gen: 2262736 level: 3
$ btrfs inspect dump-super -f /dev/sda | grep backup_csum_root
backup_csum_root: 21056868122624 gen: 2262737 level: 3
backup_csum_root: 21056944685056 gen: 2262738 level: 3
backup_csum_root: 21057139916800 gen: 2262740 level: 3 <<
points to EXTENT_TREE
backup_csum_root: 21056857341952 gen: 2262736 level: 3
$ btrfs inspect dump-tree -b 21057133690880 /dev/sda | head -n 2
btrfs-progs v5.10.1
node 21057133690880 level 1 items 316 free space 177 generation
2262698 owner CSUM_TREE
$ btrfs inspect dump-tree -b 21057139916800 /dev/sda | head -n 2
btrfs-progs v5.10.1
leaf 21057139916800 items 166 free space 6367 generation 2262696 owner
EXTENT_TREE
Previous gen is fine
$ btrfs inspect dump-tree -b 21056867762176 /dev/sda | head -n 2
btrfs-progs v5.10.1
node 21056867762176 level 3 items 2 free space 491 generation 2262738
owner EXTENT_TREE
$ btrfs inspect dump-tree -b 21056944685056 /dev/sda | head -n 2
btrfs-progs v5.10.1
node 21056944685056 level 3 items 5 free space 488 generation 2262738
owner CSUM_TREE
Also generation specified in backup root doesn't match with one in
block so seems like latest gen wasn't written to disk or something
like that.
In root tree there is different extent tree used than one specified in
backup root.
$ btrfs inspect dump-tree -b 21057011679232 /dev/sda | head -n 6
btrfs-progs v5.10.1
node 21057011679232 level 1 items 126 free space 367 generation
2262739 owner ROOT_TREE
node 21057011679232 flags 0x1(WRITTEN) backref revision 1
fs uuid 8aef11a9-beb6-49ea-9b2d-7876611a39e5
chunk uuid 4ffec48c-28ed-419d-ba87-229c0adb2ab9
key (EXTENT_TREE ROOT_ITEM 0) block 21057018363904 gen 2262739
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-02-21 1:03 ` Dāvis Mosāns
@ 2021-02-21 1:08 ` Qu Wenruo
2021-02-21 2:21 ` Dāvis Mosāns
0 siblings, 1 reply; 10+ messages in thread
From: Qu Wenruo @ 2021-02-21 1:08 UTC (permalink / raw)
To: Dāvis Mosāns, Zygo Blaxell, Chris Murphy; +Cc: Btrfs BTRFS
On 2021/2/21 上午9:03, Dāvis Mosāns wrote:
> I just found something really strange, it seems pointers for extent
> tree and csum tree have somehow gotten swapped...
Only the latest 2 backup roots are supposed to be correct, older ones
are no longer ensured to be correct.
This is not strange.
Thanks,
Qu
>
> $ btrfs inspect dump-super -f /dev/sda | grep backup_extent_root
> backup_extent_root: 21056867106816 gen: 2262737 level: 3
> backup_extent_root: 21056867762176 gen: 2262738 level: 3
> backup_extent_root: 21057133690880 gen: 2262740 level: 3 <<
> points to CSUM_TREE
> backup_extent_root: 21056854228992 gen: 2262736 level: 3
>
> $ btrfs inspect dump-super -f /dev/sda | grep backup_csum_root
> backup_csum_root: 21056868122624 gen: 2262737 level: 3
> backup_csum_root: 21056944685056 gen: 2262738 level: 3
> backup_csum_root: 21057139916800 gen: 2262740 level: 3 <<
> points to EXTENT_TREE
> backup_csum_root: 21056857341952 gen: 2262736 level: 3
>
> $ btrfs inspect dump-tree -b 21057133690880 /dev/sda | head -n 2
> btrfs-progs v5.10.1
> node 21057133690880 level 1 items 316 free space 177 generation
> 2262698 owner CSUM_TREE
>
> $ btrfs inspect dump-tree -b 21057139916800 /dev/sda | head -n 2
> btrfs-progs v5.10.1
> leaf 21057139916800 items 166 free space 6367 generation 2262696 owner
> EXTENT_TREE
>
>
> Previous gen is fine
>
> $ btrfs inspect dump-tree -b 21056867762176 /dev/sda | head -n 2
> btrfs-progs v5.10.1
> node 21056867762176 level 3 items 2 free space 491 generation 2262738
> owner EXTENT_TREE
>
> $ btrfs inspect dump-tree -b 21056944685056 /dev/sda | head -n 2
> btrfs-progs v5.10.1
> node 21056944685056 level 3 items 5 free space 488 generation 2262738
> owner CSUM_TREE
>
> Also generation specified in backup root doesn't match with one in
> block so seems like latest gen wasn't written to disk or something
> like that.
>
> In root tree there is different extent tree used than one specified in
> backup root.
> $ btrfs inspect dump-tree -b 21057011679232 /dev/sda | head -n 6
> btrfs-progs v5.10.1
> node 21057011679232 level 1 items 126 free space 367 generation
> 2262739 owner ROOT_TREE
> node 21057011679232 flags 0x1(WRITTEN) backref revision 1
> fs uuid 8aef11a9-beb6-49ea-9b2d-7876611a39e5
> chunk uuid 4ffec48c-28ed-419d-ba87-229c0adb2ab9
> key (EXTENT_TREE ROOT_ITEM 0) block 21057018363904 gen 2262739
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-02-21 1:08 ` Qu Wenruo
@ 2021-02-21 2:21 ` Dāvis Mosāns
0 siblings, 0 replies; 10+ messages in thread
From: Dāvis Mosāns @ 2021-02-21 2:21 UTC (permalink / raw)
To: Qu Wenruo; +Cc: Zygo Blaxell, Chris Murphy, Btrfs BTRFS
svētd., 2021. g. 21. febr., plkst. 03:08 — lietotājs Qu Wenruo
(<quwenruo.btrfs@gmx.com>) rakstīja:
>
>
>
> On 2021/2/21 上午9:03, Dāvis Mosāns wrote:
> > I just found something really strange, it seems pointers for extent
> > tree and csum tree have somehow gotten swapped...
>
> Only the latest 2 backup roots are supposed to be correct, older ones
> are no longer ensured to be correct.
>
> This is not strange.
>
Well here it is latest backup root, it has the highest generation but
looks like it's not issue as it's not used.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: ERROR: failed to read block groups: Input/output error
2021-02-20 23:45 ` Dāvis Mosāns
2021-02-21 1:03 ` Dāvis Mosāns
@ 2021-02-22 5:22 ` Zygo Blaxell
1 sibling, 0 replies; 10+ messages in thread
From: Zygo Blaxell @ 2021-02-22 5:22 UTC (permalink / raw)
To: Dāvis Mosāns; +Cc: Chris Murphy, Btrfs BTRFS
On Sun, Feb 21, 2021 at 01:45:10AM +0200, Dāvis Mosāns wrote:
> piektd., 2021. g. 19. febr., plkst. 07:16 — lietotājs Chris Murphy
> (<lists@colorremedies.com>) rakstīja:
[...]
> > btrfs check -b /dev/
> So I think btrfs check -b --repair should be able to fix most of things.
Why do you think that? If the HBA was failing for more than a minute
before the filesystem detected a failure and went read-only, it will
have damaged metadata all over the tree.
> > You might try kernel 5.11 which has a new mount option that will skip
> > bad roots and csums. It's 'mount -o ro,rescue=all' and while it won't
> > let you fix it, in the off chance it mounts, it'll let you get data
> > out before trying to repair the file system, which sometimes makes
> > things worse.
rescue=all (or rescue=ignorebadroots) only takes effect if the extent
tree root cannot be found at all. In this case, you have an available
extent tree root, but some of the leaves are damaged, so the filesystem
fails later on.
There's an opportunity for a patch here that doesn't even try to use the
extent tree, just pretends it wasn't found. There is similar logic in the
patch for ignoredatacsums that can be used as a reference. Maybe that's
what rescue=ignorebadroots should always do...I mean if we think the
non-essential roots are bad, why would we even try to use any of them?
On the other hand, such a patch would only get past this problem so
we can encounter the next problem. You say you want to check csums,
so you need to read the csum tree, but the csum tree would probably be
similarly damaged. Large numbers of csums may not be available without
rebuilding the csum tree. Note this is not the same as recomputing the
csums--we just want to scrape up all the surviving csum tree metadata
leaf pages and build a new csum tree out of them. I'm not sure if any
of the existing tools do that.
> It doesn't make any difference, still doesn't mount
> $ uname -r
> 5.11.0-arch2-1
> $ sudo mount -o ro,rescue=all /dev/sda ./RAID
> mount: /mnt/RAID: wrong fs type, bad option, bad superblock on
> /dev/sda, missing codepage or helper program, or other error.
>
> BTRFS warning (device sdl): sdl checksum verify failed on
> 21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
> BTRFS warning (device sdl): sdl checksum verify failed on
> 21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
> BTRFS error (device sdl): failed to read block groups: -5
> BTRFS error (device sdl): open_ctree failed
>
> It seems there should be a way to mount with backup tree root like I
> did for check but strangly usebackuproot doesn't do that...
All but the last few backup trees are normally destroyed within a few
minutes by writes from new transactions.
> piektd., 2021. g. 19. febr., plkst. 21:29 — lietotājs Zygo Blaxell
> (<ce3g8jdj@umail.furryterror.org>) rakstīja:
> >
> > On Thu, Jan 14, 2021 at 01:09:40AM +0200, Dāvis Mosāns wrote:
> > > Hi,
> > >
> > > I've 6x 3TB HDD RAID1 BTRFS filesystem where HBA card failed and
> > > caused some corruption.
> > > When I try to mount it I get
> > > $ mount /dev/sdt /mnt
> > > mount: /mnt/: wrong fs type, bad option, bad superblock on /dev/sdt,
> > > missing codepage or helper program, or other error
> > > $ dmesg | tail -n 9
> > > [ 617.158962] BTRFS info (device sdt): disk space caching is enabled
> > > [ 617.158965] BTRFS info (device sdt): has skinny extents
> > > [ 617.756924] BTRFS info (device sdt): bdev /dev/sdl errs: wr 0, rd
> > > 0, flush 0, corrupt 473, gen 0
> > > [ 617.756929] BTRFS info (device sdt): bdev /dev/sdj errs: wr 31626,
> > > rd 18765, flush 178, corrupt 5841, gen 0
> > > [ 617.756933] BTRFS info (device sdt): bdev /dev/sdg errs: wr 6867,
> > > rd 2640, flush 178, corrupt 1066, gen 0
> >
> > You have write errors on 2 disks, read errors on 3 disks, and raid1
> > tolerates only 1 disk failure, so successful recovery is unlikely.
>
> Those wr/rd/corrupt error counts are inflated/misleading, in past when
> some HDD drops out I've had them increase in huge numbers, but after
> running scrub usually it was able to fix almost everything except like
> few files that could be just deleted.
That is very bad.
With recoverable failures on raid1, scrub should be able to fix every
error, every time (to the extent that btrfs can detect errors, and there
is a known issue with writing to pages in memory while doing direct IO at
the same time). If scrub fails to recover all errors then your system
was likely exceeding the RAID1 failure tolerances (at most 1 failure
per mirrored block) and if that is not fixed then loss of filesystem
is inevitable.
Healthy disks should never drop off the SATA bus. If that is happening
then problem needs to be identified and resolved, i.e. find the bad cable,
bad PSU, bad HBA, bad disk, bad disk firmware, mismatched kernel/SCTERC
timeouts, failing disk, BIOS misconfiguration, etc. and replace the bad
hardware or fix the configuration.
> Only now it's possible that it
> failed while scrub was running making it a lot worse.
If the HBA is mangling command headers (turning reads into writes, or
changing the LBA of a write) then scrub could do some damage: corrupted
reads would trigger self-repair which would try to overwrite the
affected blocks with correct data. If the HBA corrupts the next write
command address then that data could end up written somewhere it's not
supposed to be. The error rate would have to be very high or bursty
before scrub could damage metadata--~99% of btrfs scrub IO is data,
so metadata corruption caused by scrub is very rare.
Scrub will heat up the HBA chip which could increase the error rate
temporarily or trigger a more permanent failure if it was already faulty.
That theory is consistent with the observations so far.
The HBA could have been randomly corrupting data and dropping drives off
the bus the whole time, and finally the corruption landed on a metadata
block recently. That theory is also consistent with the observations
so far.
> > > [ 631.353725] BTRFS warning (device sdt): sdt checksum verify failed
> > > on 21057101103104 wanted 0x753cdd5f found 0x9c0ba035 level 0
> > > [ 631.376024] BTRFS warning (device sdt): sdt checksum verify failed
> > > on 21057101103104 wanted 0x753cdd5f found 0xb908effa level 0
> >
> > Both copies of this metadata block are corrupted, differently.
> >
> > This is consistent with some kinds of HBA failure: every outgoing block
> > from the host is potentially corrupted, usually silently. Due to the HBA
> > failure, there is no indication of failure available to the filesystem
> > until after several corrupt blocks are written to disk. By the time
> > failure is detected, damage is extensive, especially for metadata where
> > overwrites are frequent.
>
> I don't think it's that bad here. My guess is that it failed while
> updating extent tree and some part of it didn't got written to disk.
"some part of [the extent tree] didn't get written to disk" is the worst
case scenario. It doesn't matter if the filesystem lost 1 extent tree
page or 1000. If there are any missing interior nodes then the trees
need to be rebuilt before they are usable again.
If metadata interior nodes are missing, then the leaf pages behind them
are no longer accessible without brute-force search of all metadata
chunks. There's no other way to find them because they could be anywhere
in a metadata chunk. If leaf pages are missing then reference counts
are no longer reliable and safely writing to the filesystem is not
possible.
It's theoretically possible to scan for just the leaf pages and rebuild
the interior nodes so that read-only access to the data is possible,
but I don't know of an existing tool that does only that. Check will
read from the subvol trees and try to rebuild the extent tree, which
is not quite the same thing.
> I
> want to check how it looks like on disk, is there some tool to map
> block number to offset in disk?
'btrfs-map-logical' does the translation internally, but it will go ahead
and read the block for you, it doesn't tell you the translation result.
> > This is failure mode that you need backups to recover from (or mirror
> > disks on separate, non-failing HBA hardware).
>
> I don't know how btrfs decides in which disks it stores copies or if
> it's always in same disk. To prevent such failure in future I could
> split RAID1 across 2 different HBAs but it's not clear which disks
> would need to be seperated.
Ideally in any RAID system every disk gets its own separate controller.
There would be isolated hardware for every disk, including HBA and
power supply. This isn't very practical or cost-effective (except
on purpose-built NAS and server hardware), so most people backup the
filesystem content on another host instead.
Using a separate host means you get easy and effective isolation from
failures in CPU, RAM, HBA, PCI bridges, power supply, cooling, kernel,
and any of a dozen other points of failure that could break a filesystem.
> I don't want to use --init-extent-tree because I don't want to reset
> everything, but only corrupted things.
Same thing.
For read-write mounts, you can remove all the corrupted things,
but after that some of the reference data is missing from the tree,
so you have to verify the entire tree to remove the inconsistencies.
The easiest way to do that is to build a new consistent tree out of the
old metadata items so you can query it for duplicate and missing items,
but if you've already built a new consistent tree then you might as well
keep it and throw away the old one.
For read-only mounts, you will need to search for all the leaf pages of
the tree where an interior node is damaged and no longer points to the
leaf page.
> Also btrfs check --repair
> doesn't work as it aborts too quickly, only using with -b flag could
> fix it I think.
The difference between --repair and --init-extent-tree is that --repair
can make small fixes to trees that are otherwise intact. If --repair
encounters something that cannot be repaired in place (like missing
interior nodes of the tree) then it will abort. In such cases you will
need to do --init-extent-tree to make the filesystem writable again
(or even readable in some cases).
I would not advise trying to make this filesystem writable again.
I'd copy the data off, then mkfs and copy it back (or swap disks with
the copy). The HBA failure has likely corrupted recently written data
too, so I'd stick with trying to recover older files and assume all new
files are bad. Old files can be compared with backups to see if they
were damaged by misaddressed writes.
I wouldn't advise using this filesystem as a btrfs check --repair
test case. It's easy to build a randomly corrupted filesystem for
development testing, and check --repair needs more development work to
cope with much gentler corruption cases.
> > Both methods are likely to fail in the presence of so much corruption
> > and they may take so long to run that mkfs + restore from backups could
> > be significantly faster. Definitely extract any data from the filesystem
> > that you want to keep _before_ attempting any of these operations.
>
> It's not really about time, I rather want to reduce possilble data
> loss as much as possible.
> I can't mount it even read-only so it seems the only way to get data
> out is by using btrfs restore which seems to work fine but does it
> verify file checksums? It looks like it doesn't...
btrfs restore does not verify checksums (nor is it confused by metadata
inconsistency or corruption in the csum tree). You get the data that
is on disk in all its corrupt glory.
There's another patch opportunity: teach btrfs restore how to read the
csum tree.
Note that if you have damaged metadata, the file data checksums may no
longer be available, or a brute-force search may be required to locate
them, so even if you get csum verification working, it might not be
very useful.
> I have some files
> where it said:
> We seem to be looping a lot on ./something, do you want to keep going
> on ? (y/N/a)
That's normal, the looping detection threshold is very low. Most people
hit 'a' here. It doesn't affect correctness of output. It only prevents
getting stuck in infinite loops when trying to locate the metadata page
for the next extent in the file.
> When I checked this file I see that it's corrupted. Basically I want
> restore only files with valid checksums and then have a list of
> corrupted files. From currupted files there are few I want to see if
> they can be recovered. I have lot of of snapshots but even the oldest
> ones are corrupted in exactly same way - they're identical. It seems I
> need to find previous copy of this file if it exsists at all... Any
> idea how to find previous version of file?
If they were snapshots then the files share physical storage, i.e. there
are not two distinct versions of the shared content, they are two
references to the same content, and corruption in one will appear in
the other.
> I tried
> $ btrfs restore -u 2 -t 21056933724160
> with different superblocks/tree roots but they all give same corrupted file.
> The file looks like this
> $ hexdump -C file | head -n 5
> 00000000 27 47 10 00 6d 64 61 74 7b 7b 7b 7b 7b 7b 7b 7b |'G..mdat{{{{{{{{|
> 00000010 7a 7a 79 79 7a 7a 7a 7a 7b 7b 7b 7c 7c 7c 7b 7b |zzyyzzzz{{{|||{{|
> 00000020 7c 7c 7c 7b 7b 7b 7b 7b 7c 7c 7c 7c 7c 7b 7b 7b ||||{{{{{|||||{{{|
> 00000030 7b 7b 7b 7b 7b 7b 7b 7b 7b 7a 7a 7a 7a 79 79 7a |{{{{{{{{{zzzzyyz|
> 00000040 7b 7b 7b 7b 7a 7b 7b 7c 7b 7c 7c 7b 7c 7c 7b 7b |{{{{z{{|{||{||{{|
All tree roots will ultimately point to the same data blocks unless the
file was written in between. Tree roots share all pages except for the
unique pages that are introduced in the tree root's transaction.
> Those repeated 7a/b/c is wrong data. Also I'm not sure if these files
> have been corrupted now or more in past... So I need to check if
> checksum matches.
>
> > It might be possible to recover by manually inspecting the corrupted
> > metadata blocks and making guesses and adjustments, but that could take
> > even longer than check --repair if there are thousands of damaged pages.
> >
>
> I want to look into this but not sure if there are any tools with
> which it would be easy to inspect data. dump-tree is nice but it
> doesn't work when checksum is incorrect.
It also doesn't work if an interior tree page is destroyed. In that
case, nothing less than --init-extent-tree can find the page--and even
init-extent-tree needs to read a few pages to work.
> My current plan is:
> 1. btrfs restore only valid files, how? I don't want to mix good files
> with corrupted ones together
> 2. look into how exactly extent tree is corrupted
> 3. try to see if few of corrupted files can be recovered in some way
> 4. do btrfs check -b --repair (maybe if extent tree can be fixed then
> wouldn't need to use -b flag)
> 5. try to mount and btrfs scrub
> 6. maybe wipe and recreate new fielsystem
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2021-02-22 5:23 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-01-13 23:09 ERROR: failed to read block groups: Input/output error Dāvis Mosāns
2021-01-13 23:39 ` Dāvis Mosāns
2021-02-19 3:03 ` Dāvis Mosāns
2021-02-19 5:16 ` Chris Murphy
2021-02-19 19:29 ` Zygo Blaxell
2021-02-20 23:45 ` Dāvis Mosāns
2021-02-21 1:03 ` Dāvis Mosāns
2021-02-21 1:08 ` Qu Wenruo
2021-02-21 2:21 ` Dāvis Mosāns
2021-02-22 5:22 ` Zygo Blaxell
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).