linux-btrfs.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Filesystem corruption after convert-to-block-group-tree
@ 2023-10-03 23:17 Alexander Duscheleit
  2023-10-04  3:27 ` Qu Wenruo
  2023-10-04  7:42 ` Alexander Duscheleit
  0 siblings, 2 replies; 5+ messages in thread
From: Alexander Duscheleit @ 2023-10-03 23:17 UTC (permalink / raw)
  To: linux-btrfs

[-- Attachment #1: Type: text/plain, Size: 1639 bytes --]

Hi all,
earlier today I tried to convert my BTRFS filesystem to block-group-tree and the operation seemed successful at first glance.
(I unmounted, converted and mounted the fs without any error.)
Some time later I tried to access a file and got an I/O error.

after some updates, reboots and troubleshooting I ended up in the following situation:

The fs cannot be mounted normally, but it mounts (consistently) with
-o rescue=all,ro (see attached dmesg.log).

No data _appears_ to be missing or corrupt.

btrfs-find-root throws many errors concerning corrupt leaf blocks but does find the curren tree root. (Again, see attached log.)

Is there any way to bring this fs back to a useable state without starting over from scratch?


System Data:
# uname -a
Linux hera 6.5.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 23 Sep 2023 22:55:13 +0000 x86_64 GNU/Linux

# btrfs --version                               
btrfs-progs v6.5.1

(Note: The conversion to block group tree was done with btrfs-progs 6.3.3 and Kernel 6.4.12.arch1-1)

# btrfs fi show                   
Label: 'hera-storage'  uuid: a71011f9-d79c-40e8-85fb-60b6f2af0637
	Total devices 4 FS bytes used 8.36TiB
	devid    1 size 4.55TiB used 4.19TiB path /dev/sdb1
	devid    2 size 4.55TiB used 4.19TiB path /dev/sdd1
	devid    3 size 4.55TiB used 4.19TiB path /dev/sdc1
	devid    4 size 4.55TiB used 4.19TiB path /dev/sde1

# btrfs fi df /mnt/btrfs_storage 
Data, RAID10: total=8.34TiB, used=8.34TiB
System, RAID1C4: total=8.00MiB, used=912.00KiB
Metadata, RAID1C4: total=24.00GiB, used=23.93GiB
GlobalReserve, single: total=512.00MiB, used=0.00B

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dmesg.log --]
[-- Type: text/x-log; name="dmesg.log", Size: 2790 bytes --]


[    3.301052] Btrfs loaded, zoned=yes, fsverity=yes
[    3.302315] BTRFS: device label hera-system devid 1 transid 2585590 /dev/sda3 scanned by (udev-worker) (143)
[    3.313122] BTRFS: device label hera-storage devid 3 transid 41302 /dev/sdc1 scanned by (udev-worker) (145)
[    3.335675] BTRFS: device label hera-storage devid 1 transid 41302 /dev/sdb1 scanned by (udev-worker) (142)
[    3.388224] BTRFS: device label hera-storage devid 4 transid 41302 /dev/sde1 scanned by (udev-worker) (131)
[    3.394259] BTRFS: device label hera-storage devid 2 transid 41302 /dev/sdd1 scanned by (udev-worker) (145)

[    3.536658] BTRFS info (device sda3): using crc32c (crc32c-intel) checksum algorithm
[    3.536670] BTRFS info (device sda3): using free space tree
[    3.554865] BTRFS info (device sda3): enabling ssd optimizations
[    3.554868] BTRFS info (device sda3): auto enabling async discard

[    4.216302] BTRFS info (device sda3: state M): force zstd compression, level 3

[  678.909602] BTRFS info (device sdb1): using xxhash64 (xxhash64-generic) checksum algorithm
[  678.909611] BTRFS info (device sdb1): force zstd compression, level 3
[  678.909614] BTRFS info (device sdb1): using free space tree
[  679.734658] BTRFS error (device sdb1): level verify failed on logical 7868411314176 mirror 2 wanted 0 found 1
[  679.754569] BTRFS error (device sdb1): level verify failed on logical 7868411314176 mirror 1 wanted 0 found 1
[  679.770766] BTRFS error (device sdb1): level verify failed on logical 7868411314176 mirror 3 wanted 0 found 1
[  679.843966] BTRFS error (device sdb1): level verify failed on logical 7868411314176 mirror 4 wanted 0 found 1
[  679.844022] BTRFS error (device sdb1): failed to read block groups: -5
[  679.847667] BTRFS error (device sdb1): open_ctree failed
[  696.027389] BTRFS info (device sdb1): using xxhash64 (xxhash64-generic) checksum algorithm
[  696.027399] BTRFS info (device sdb1): force zstd compression, level 3
[  696.027405] BTRFS info (device sdb1): enabling all of the rescue options
[  696.027406] BTRFS info (device sdb1): ignoring data csums
[  696.027407] BTRFS info (device sdb1): ignoring bad roots
[  696.027409] BTRFS info (device sdb1): disabling log replay at mount time
[  696.027410] BTRFS info (device sdb1): using free space tree
[  696.345008] BTRFS error (device sdb1: state C): level verify failed on logical 7868411314176 mirror 3 wanted 0 found 1
[  696.345731] BTRFS error (device sdb1: state C): level verify failed on logical 7868411314176 mirror 1 wanted 0 found 1
[  696.345964] BTRFS error (device sdb1: state C): level verify failed on logical 7868411314176 mirror 2 wanted 0 found 1
[  696.346616] BTRFS error (device sdb1: state C): level verify failed on logical 7868411314176 mirror 4 wanted 0 found 1

[-- Attachment #3: find-root.log --]
[-- Type: text/plain, Size: 3071 bytes --]

parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
ERROR: failed to read block groups: Input/output error
corrupt leaf: block=230911688704 slot=9 extent bytenr=5816833572864 len=16384 invalid generation, have 41341 expect (0, 41303]
corrupt leaf: block=230911688704 slot=9 extent bytenr=5816833572864 len=16384 invalid generation, have 41341 expect (0, 41303]

[ ...many more of these ]

corrupt leaf: block=7868567764992 slot=1 extent bytenr=9396362510336 len=16384 invalid generation, have 41355 expect (0, 41303]
corrupt leaf: root=1 block=7868568403968 slot=0, invalid root generation, have 41368 expect (0, 41303]
corrupt leaf: root=1 block=7868568403968 slot=0, invalid root generation, have 41368 expect (0, 41303]
corrupt leaf: root=1 block=7868568403968 slot=0, invalid root generation, have 41368 expect (0, 41303]
corrupt leaf: root=1 block=7868568403968 slot=0, invalid root generation, have 41368 expect (0, 41303]
corrupt leaf: block=7868568469504 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]

[ ...more of these ]

corrupt leaf: block=7868570451968 slot=11 extent bytenr=7868424683520 len=16384 invalid generation, have 41336 expect (0, 41303]
corrupt leaf: root=1 block=7868570468352 slot=0, invalid root generation, have 41369 expect (0, 41303]
corrupt leaf: root=1 block=7868570468352 slot=0, invalid root generation, have 41369 expect (0, 41303]
corrupt leaf: root=1 block=7868570468352 slot=0, invalid root generation, have 41369 expect (0, 41303]
corrupt leaf: root=1 block=7868570468352 slot=0, invalid root generation, have 41369 expect (0, 41303]
corrupt leaf: block=7868570484736 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
corrupt leaf: block=7868570484736 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
corrupt leaf: block=7868570484736 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
corrupt leaf: block=7868570484736 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
Superblock thinks the generation is 41302
Superblock thinks the level is 0
Found tree root at 7868986654720 gen 41302 level 0
Well block 7868881141760(gen: 41295 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0

[ ...more ]

Well block 230998687744(gen: 41235 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0
Well block 486801408(gen: 41233 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0
Well block 403341312(gen: 41232 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0
Well block 3776035717120(gen: 40791 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0

^ permalink raw reply	[flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-10-04 15:13 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-03 23:17 Filesystem corruption after convert-to-block-group-tree Alexander Duscheleit
2023-10-04  3:27 ` Qu Wenruo
2023-10-04  7:42 ` Alexander Duscheleit
2023-10-04  8:18   ` Qu Wenruo
2023-10-04 15:13   ` Alexander Duscheleit

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).