* Filesystem corruption after convert-to-block-group-tree
@ 2023-10-03 23:17 Alexander Duscheleit
2023-10-04 3:27 ` Qu Wenruo
2023-10-04 7:42 ` Alexander Duscheleit
0 siblings, 2 replies; 5+ messages in thread
From: Alexander Duscheleit @ 2023-10-03 23:17 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1639 bytes --]
Hi all,
earlier today I tried to convert my BTRFS filesystem to block-group-tree and the operation seemed successful at first glance.
(I unmounted, converted and mounted the fs without any error.)
Some time later I tried to access a file and got an I/O error.
after some updates, reboots and troubleshooting I ended up in the following situation:
The fs cannot be mounted normally, but it mounts (consistently) with
-o rescue=all,ro (see attached dmesg.log).
No data _appears_ to be missing or corrupt.
btrfs-find-root throws many errors concerning corrupt leaf blocks but does find the curren tree root. (Again, see attached log.)
Is there any way to bring this fs back to a useable state without starting over from scratch?
System Data:
# uname -a
Linux hera 6.5.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 23 Sep 2023 22:55:13 +0000 x86_64 GNU/Linux
# btrfs --version
btrfs-progs v6.5.1
(Note: The conversion to block group tree was done with btrfs-progs 6.3.3 and Kernel 6.4.12.arch1-1)
# btrfs fi show
Label: 'hera-storage' uuid: a71011f9-d79c-40e8-85fb-60b6f2af0637
Total devices 4 FS bytes used 8.36TiB
devid 1 size 4.55TiB used 4.19TiB path /dev/sdb1
devid 2 size 4.55TiB used 4.19TiB path /dev/sdd1
devid 3 size 4.55TiB used 4.19TiB path /dev/sdc1
devid 4 size 4.55TiB used 4.19TiB path /dev/sde1
# btrfs fi df /mnt/btrfs_storage
Data, RAID10: total=8.34TiB, used=8.34TiB
System, RAID1C4: total=8.00MiB, used=912.00KiB
Metadata, RAID1C4: total=24.00GiB, used=23.93GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: dmesg.log --]
[-- Type: text/x-log; name="dmesg.log", Size: 2790 bytes --]
[ 3.301052] Btrfs loaded, zoned=yes, fsverity=yes
[ 3.302315] BTRFS: device label hera-system devid 1 transid 2585590 /dev/sda3 scanned by (udev-worker) (143)
[ 3.313122] BTRFS: device label hera-storage devid 3 transid 41302 /dev/sdc1 scanned by (udev-worker) (145)
[ 3.335675] BTRFS: device label hera-storage devid 1 transid 41302 /dev/sdb1 scanned by (udev-worker) (142)
[ 3.388224] BTRFS: device label hera-storage devid 4 transid 41302 /dev/sde1 scanned by (udev-worker) (131)
[ 3.394259] BTRFS: device label hera-storage devid 2 transid 41302 /dev/sdd1 scanned by (udev-worker) (145)
[ 3.536658] BTRFS info (device sda3): using crc32c (crc32c-intel) checksum algorithm
[ 3.536670] BTRFS info (device sda3): using free space tree
[ 3.554865] BTRFS info (device sda3): enabling ssd optimizations
[ 3.554868] BTRFS info (device sda3): auto enabling async discard
[ 4.216302] BTRFS info (device sda3: state M): force zstd compression, level 3
[ 678.909602] BTRFS info (device sdb1): using xxhash64 (xxhash64-generic) checksum algorithm
[ 678.909611] BTRFS info (device sdb1): force zstd compression, level 3
[ 678.909614] BTRFS info (device sdb1): using free space tree
[ 679.734658] BTRFS error (device sdb1): level verify failed on logical 7868411314176 mirror 2 wanted 0 found 1
[ 679.754569] BTRFS error (device sdb1): level verify failed on logical 7868411314176 mirror 1 wanted 0 found 1
[ 679.770766] BTRFS error (device sdb1): level verify failed on logical 7868411314176 mirror 3 wanted 0 found 1
[ 679.843966] BTRFS error (device sdb1): level verify failed on logical 7868411314176 mirror 4 wanted 0 found 1
[ 679.844022] BTRFS error (device sdb1): failed to read block groups: -5
[ 679.847667] BTRFS error (device sdb1): open_ctree failed
[ 696.027389] BTRFS info (device sdb1): using xxhash64 (xxhash64-generic) checksum algorithm
[ 696.027399] BTRFS info (device sdb1): force zstd compression, level 3
[ 696.027405] BTRFS info (device sdb1): enabling all of the rescue options
[ 696.027406] BTRFS info (device sdb1): ignoring data csums
[ 696.027407] BTRFS info (device sdb1): ignoring bad roots
[ 696.027409] BTRFS info (device sdb1): disabling log replay at mount time
[ 696.027410] BTRFS info (device sdb1): using free space tree
[ 696.345008] BTRFS error (device sdb1: state C): level verify failed on logical 7868411314176 mirror 3 wanted 0 found 1
[ 696.345731] BTRFS error (device sdb1: state C): level verify failed on logical 7868411314176 mirror 1 wanted 0 found 1
[ 696.345964] BTRFS error (device sdb1: state C): level verify failed on logical 7868411314176 mirror 2 wanted 0 found 1
[ 696.346616] BTRFS error (device sdb1: state C): level verify failed on logical 7868411314176 mirror 4 wanted 0 found 1
[-- Attachment #3: find-root.log --]
[-- Type: text/plain, Size: 3071 bytes --]
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
ERROR: failed to read block groups: Input/output error
corrupt leaf: block=230911688704 slot=9 extent bytenr=5816833572864 len=16384 invalid generation, have 41341 expect (0, 41303]
corrupt leaf: block=230911688704 slot=9 extent bytenr=5816833572864 len=16384 invalid generation, have 41341 expect (0, 41303]
[ ...many more of these ]
corrupt leaf: block=7868567764992 slot=1 extent bytenr=9396362510336 len=16384 invalid generation, have 41355 expect (0, 41303]
corrupt leaf: root=1 block=7868568403968 slot=0, invalid root generation, have 41368 expect (0, 41303]
corrupt leaf: root=1 block=7868568403968 slot=0, invalid root generation, have 41368 expect (0, 41303]
corrupt leaf: root=1 block=7868568403968 slot=0, invalid root generation, have 41368 expect (0, 41303]
corrupt leaf: root=1 block=7868568403968 slot=0, invalid root generation, have 41368 expect (0, 41303]
corrupt leaf: block=7868568469504 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
[ ...more of these ]
corrupt leaf: block=7868570451968 slot=11 extent bytenr=7868424683520 len=16384 invalid generation, have 41336 expect (0, 41303]
corrupt leaf: root=1 block=7868570468352 slot=0, invalid root generation, have 41369 expect (0, 41303]
corrupt leaf: root=1 block=7868570468352 slot=0, invalid root generation, have 41369 expect (0, 41303]
corrupt leaf: root=1 block=7868570468352 slot=0, invalid root generation, have 41369 expect (0, 41303]
corrupt leaf: root=1 block=7868570468352 slot=0, invalid root generation, have 41369 expect (0, 41303]
corrupt leaf: block=7868570484736 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
corrupt leaf: block=7868570484736 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
corrupt leaf: block=7868570484736 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
corrupt leaf: block=7868570484736 slot=65 extent bytenr=30507008 len=16384 invalid generation, have 41308 expect (0, 41303]
Superblock thinks the generation is 41302
Superblock thinks the level is 0
Found tree root at 7868986654720 gen 41302 level 0
Well block 7868881141760(gen: 41295 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0
[ ...more ]
Well block 230998687744(gen: 41235 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0
Well block 486801408(gen: 41233 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0
Well block 403341312(gen: 41232 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0
Well block 3776035717120(gen: 40791 level: 0) seems good, but generation/level doesn't match, want gen: 41302 level: 0
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Filesystem corruption after convert-to-block-group-tree
2023-10-03 23:17 Filesystem corruption after convert-to-block-group-tree Alexander Duscheleit
@ 2023-10-04 3:27 ` Qu Wenruo
2023-10-04 7:42 ` Alexander Duscheleit
1 sibling, 0 replies; 5+ messages in thread
From: Qu Wenruo @ 2023-10-04 3:27 UTC (permalink / raw)
To: Alexander Duscheleit, linux-btrfs
On 2023/10/4 09:47, Alexander Duscheleit wrote:
> Hi all,
> earlier today I tried to convert my BTRFS filesystem to block-group-tree and the operation seemed successful at first glance.
> (I unmounted, converted and mounted the fs without any error.)
> Some time later I tried to access a file and got an I/O error.
The error from the dmesg shows a level mismatch in block group tree.
However the block group tree itself is always fully read at each mount,
thus it means at your first mount, the block group tree is good, or you
can not even mount the fs (unless go with -o rescue=all).
So I'm afraid the corruption is not caused by the conversion, but
something else, after your initial mount.
I'm more interested in what happened between the initial mount and "some
time later".
>
> after some updates, reboots and troubleshooting I ended up in the following situation:
>
> The fs cannot be mounted normally, but it mounts (consistently) with
> -o rescue=all,ro (see attached dmesg.log).
>
> No data _appears_ to be missing or corrupt.
>
> btrfs-find-root throws many errors concerning corrupt leaf blocks but does find the curren tree root. (Again, see attached log.)
>
> Is there any way to bring this fs back to a useable state without starting over from scratch?
It looks like block group tree is corrupted, but without a full "btrfs
check --read-only" it's hard to say.
Extra dump on the bg tree can also help (needs both stderr and stdout):
# btrfs ins dump-tree -t 11 <device>
# btrfs ins dump-tree -b 7868411314176 <device>
BTW, btrfs-find-root is not that useful, thus it should only be adviced
by developers, but at least it does no harm.
Thanks,
Qu
>
>
> System Data:
> # uname -a
> Linux hera 6.5.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 23 Sep 2023 22:55:13 +0000 x86_64 GNU/Linux
>
> # btrfs --version
> btrfs-progs v6.5.1
>
> (Note: The conversion to block group tree was done with btrfs-progs 6.3.3 and Kernel 6.4.12.arch1-1)
>
> # btrfs fi show
> Label: 'hera-storage' uuid: a71011f9-d79c-40e8-85fb-60b6f2af0637
> Total devices 4 FS bytes used 8.36TiB
> devid 1 size 4.55TiB used 4.19TiB path /dev/sdb1
> devid 2 size 4.55TiB used 4.19TiB path /dev/sdd1
> devid 3 size 4.55TiB used 4.19TiB path /dev/sdc1
> devid 4 size 4.55TiB used 4.19TiB path /dev/sde1
>
> # btrfs fi df /mnt/btrfs_storage
> Data, RAID10: total=8.34TiB, used=8.34TiB
> System, RAID1C4: total=8.00MiB, used=912.00KiB
> Metadata, RAID1C4: total=24.00GiB, used=23.93GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Filesystem corruption after convert-to-block-group-tree
2023-10-03 23:17 Filesystem corruption after convert-to-block-group-tree Alexander Duscheleit
2023-10-04 3:27 ` Qu Wenruo
@ 2023-10-04 7:42 ` Alexander Duscheleit
2023-10-04 8:18 ` Qu Wenruo
2023-10-04 15:13 ` Alexander Duscheleit
1 sibling, 2 replies; 5+ messages in thread
From: Alexander Duscheleit @ 2023-10-04 7:42 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 4149 bytes --]
4 October 2023 at 05:27, "Qu Wenruo" <quwenruo.btrfs@gmx.com> wrote:
>
> On 2023/10/4 09:47, Alexander Duscheleit wrote:
>
> >
> > Hi all,
> > earlier today I tried to convert my BTRFS filesystem to block-group-tree and the operation seemed successful at first glance.
> > (I unmounted, converted and mounted the fs without any error.)
> > Some time later I tried to access a file and got an I/O error.
> >
>
> The error from the dmesg shows a level mismatch in block group tree.
>
> However the block group tree itself is always fully read at each mount,
> thus it means at your first mount, the block group tree is good, or you
> can not even mount the fs (unless go with -o rescue=all).
>
> So I'm afraid the corruption is not caused by the conversion, but
> something else, after your initial mount.
>
> I'm more interested in what happened between the initial mount and "some
> time later".
As far as I can tell, nothing. The array contains mostly media files, but
also some minor backups (borg-backup). I didn't put any new media on it
during the time in question and no backups were running.
After the conversion I mounted the fs, restartet nfs, re-mounted the nfs
share on the htpc and then left it alone. I didn't do any operations on it
besides a ls to confirm it's "there".
I noticed the error when I wanted to play some media file and the player threw
an error.
>
> >
> > after some updates, reboots and troubleshooting I ended up in the following situation:
> >
> > The fs cannot be mounted normally, but it mounts (consistently) with
> > -o rescue=all,ro (see attached dmesg.log).
> >
> > No data _appears_ to be missing or corrupt.
> >
> > btrfs-find-root throws many errors concerning corrupt leaf blocks but does find the curren tree root. (Again, see attached log.)
> >
> > Is there any way to bring this fs back to a useable state without starting over from scratch?
> >
>
> It looks like block group tree is corrupted, but without a full "btrfs
> check --read-only" it's hard to say.
# btrfs check --readonly /dev/sdb1
Opening filesystem to check...
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
parent transid verify failed on 7868411314176 wanted 41293 found 41370
Ignoring transid failure
ERROR: child eb corrupted: parent bytenr=7868939862016 item=336 parent level=1 child bytenr=7868411314176 child level=1
ERROR: failed to read block groups: Input/output error
ERROR: cannot open file system
>
> Extra dump on the bg tree can also help (needs both stderr and stdout):
>
> # btrfs ins dump-tree -t 11 <device>
yields "ERROR: cannot print block group tree, invalid pointer" for each of the 4 devices.
>
> # btrfs ins dump-tree -b 7868411314176 <device>
Attached
>
> BTW, btrfs-find-root is not that useful, thus it should only be adviced
> by developers, but at least it does no harm.
>
> Thanks,
> Qu
>
> >
> > System Data:
> > # uname -a
> > Linux hera 6.5.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 23 Sep 2023 22:55:13 +0000 x86_64 GNU/Linux
> >
> > # btrfs --version
> > btrfs-progs v6.5.1
> >
> > (Note: The conversion to block group tree was done with btrfs-progs 6.3.3 and Kernel 6.4.12.arch1-1)
> >
> > # btrfs fi show
> > Label: 'hera-storage' uuid: a71011f9-d79c-40e8-85fb-60b6f2af0637
> > Total devices 4 FS bytes used 8.36TiB
> > devid 1 size 4.55TiB used 4.19TiB path /dev/sdb1
> > devid 2 size 4.55TiB used 4.19TiB path /dev/sdd1
> > devid 3 size 4.55TiB used 4.19TiB path /dev/sdc1
> > devid 4 size 4.55TiB used 4.19TiB path /dev/sde1
> >
> > # btrfs fi df /mnt/btrfs_storage
> > Data, RAID10: total=8.34TiB, used=8.34TiB
> > System, RAID1C4: total=8.00MiB, used=912.00KiB
> > Metadata, RAID1C4: total=24.00GiB, used=23.93GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> >
>
Thanks,
Alex
[-- Attachment #2: dump_tree.log --]
[-- Type: text/plain, Size: 26969 bytes --]
# btrfs ins dump-tree -b 7868411314176 /dev/sdb1
btrfs-progs v6.5.1
node 7868411314176 level 1 items 411 free space 82 generation 41370 owner EXTENT_TREE
node 7868411314176 flags 0x1(WRITTEN) backref revision 1
fs uuid a71011f9-d79c-40e8-85fb-60b6f2af0637
chunk uuid e9379230-3981-4b42-9605-ab297c06ff6a
key (4468457472 EXTENT_ITEM 4096) block 9396682113024 gen 36019
key (4469997568 EXTENT_ITEM 8192) block 786825216 gen 101
key (4471267328 EXTENT_ITEM 8192) block 653279232 gen 56
key (4477804544 EXTENT_ITEM 4096) block 231155974144 gen 127
key (4483133440 EXTENT_ITEM 45056) block 653295616 gen 56
key (4495466496 EXTENT_ITEM 102400) block 637943808 gen 55
key (4509626368 EXTENT_ITEM 69632) block 786890752 gen 101
key (4514025472 EXTENT_ITEM 4096) block 994459648 gen 83
key (4516372480 EXTENT_ITEM 81920) block 1017479168 gen 86
key (4518547456 EXTENT_ITEM 12288) block 9396682129408 gen 36019
key (4520587264 EXTENT_ITEM 28672) block 9396770684928 gen 36021
key (4525461504 EXTENT_ITEM 90112) block 9397125709824 gen 39762
key (4528177152 EXTENT_ITEM 4096) block 994476032 gen 83
key (4537344000 EXTENT_ITEM 102400) block 638107648 gen 55
key (4540297216 EXTENT_ITEM 4096) block 9396770717696 gen 36021
key (4542210048 EXTENT_ITEM 4096) block 9396682211328 gen 36019
key (4543131648 EXTENT_ITEM 4096) block 9396682227712 gen 36019
key (4544966656 EXTENT_ITEM 8192) block 9396682244096 gen 36019
key (4546826240 EXTENT_ITEM 16384) block 669401088 gen 57
key (4547911680 EXTENT_ITEM 16384) block 95633408 gen 28
key (4550500352 EXTENT_ITEM 8192) block 1017593856 gen 86
key (4553515008 EXTENT_ITEM 12288) block 9396726775808 gen 36020
key (4555333632 EXTENT_ITEM 16384) block 653492224 gen 56
key (4556091392 EXTENT_ITEM 4096) block 653475840 gen 56
key (4557496320 EXTENT_ITEM 8192) block 638304256 gen 55
key (4559028224 EXTENT_ITEM 8192) block 638271488 gen 55
key (4559794176 EXTENT_ITEM 4096) block 638320640 gen 55
key (4560556032 EXTENT_ITEM 8192) block 638287872 gen 55
key (4562206720 EXTENT_ITEM 8192) block 95748096 gen 28
key (4563947520 EXTENT_ITEM 8192) block 1017659392 gen 86
key (4565712896 EXTENT_ITEM 8192) block 1017626624 gen 86
key (4566577152 EXTENT_ITEM 8192) block 1017675776 gen 86
key (4567441408 EXTENT_ITEM 8192) block 1017643008 gen 86
key (4569190400 EXTENT_ITEM 16384) block 95813632 gen 28
key (4571041792 EXTENT_ITEM 4096) block 638369792 gen 55
key (4573585408 EXTENT_ITEM 20480) block 669515776 gen 57
key (4574826496 EXTENT_ITEM 8192) block 9396682440704 gen 36019
key (4575920128 EXTENT_ITEM 12288) block 9396682457088 gen 36019
key (4577693696 EXTENT_ITEM 4096) block 9396726792192 gen 36020
key (4579495936 EXTENT_ITEM 24576) block 669646848 gen 57
key (4580556800 EXTENT_ITEM 16384) block 669630464 gen 57
key (4583555072 EXTENT_ITEM 20480) block 95911936 gen 28
key (4586459136 EXTENT_ITEM 16384) block 95928320 gen 28
key (4588883968 EXTENT_ITEM 8192) block 9396682522624 gen 36019
key (4590178304 EXTENT_ITEM 12288) block 687112192 gen 58
key (4591378432 EXTENT_ITEM 4096) block 95961088 gen 28
key (4593541120 EXTENT_ITEM 8192) block 95977472 gen 28
key (4595793920 EXTENT_ITEM 12288) block 95993856 gen 28
key (4597719040 EXTENT_ITEM 4096) block 1017757696 gen 86
key (4599504896 EXTENT_ITEM 16384) block 96026624 gen 28
key (4601450496 EXTENT_ITEM 4096) block 96043008 gen 28
key (4603678720 EXTENT_ITEM 8192) block 96059392 gen 28
key (4619288576 EXTENT_ITEM 94208) block 96075776 gen 28
key (4635693056 EXTENT_ITEM 102400) block 96092160 gen 28
key (4651212800 EXTENT_ITEM 73728) block 96108544 gen 28
key (4668284928 EXTENT_ITEM 102400) block 9396682539008 gen 36019
key (4679827456 EXTENT_ITEM 4096) block 9396682571776 gen 36019
key (4680417280 EXTENT_ITEM 4096) block 9396682588160 gen 36019
key (4682432512 EXTENT_ITEM 4096) block 9396682620928 gen 36019
key (4683227136 EXTENT_ITEM 4096) block 9396682637312 gen 36019
key (4684341248 EXTENT_ITEM 4096) block 9396682653696 gen 36019
key (4685266944 EXTENT_ITEM 4096) block 9396682752000 gen 36019
key (4685721600 EXTENT_ITEM 4096) block 9396683227136 gen 36019
key (4686344192 EXTENT_ITEM 4096) block 96223232 gen 28
key (4687216640 EXTENT_ITEM 4096) block 9396683341824 gen 36019
key (4688338944 EXTENT_ITEM 4096) block 9396683390976 gen 36019
key (4689268736 EXTENT_ITEM 4096) block 9396726890496 gen 36020
key (4689985536 EXTENT_ITEM 12288) block 786956288 gen 101
key (4692430848 EXTENT_ITEM 16384) block 9396726906880 gen 36020
key (4695109632 EXTENT_ITEM 20480) block 301498368 gen 4335
key (4696961024 EXTENT_ITEM 8192) block 669908992 gen 57
key (4698869760 EXTENT_ITEM 4096) block 9396726939648 gen 36020
key (4701495296 EXTENT_ITEM 8192) block 1017856000 gen 86
key (4703854592 EXTENT_ITEM 8192) block 1017823232 gen 86
key (4705173504 EXTENT_ITEM 8192) block 1017872384 gen 86
key (4706164736 EXTENT_ITEM 4096) block 1017839616 gen 86
key (4708089856 EXTENT_ITEM 12288) block 9396726956032 gen 36020
key (4710158336 EXTENT_ITEM 12288) block 1017905152 gen 86
key (4711268352 EXTENT_ITEM 4096) block 96419840 gen 28
key (4712189952 EXTENT_ITEM 4096) block 96436224 gen 28
key (4713074688 EXTENT_ITEM 4096) block 638926848 gen 55
key (4713955328 EXTENT_ITEM 4096) block 9396726972416 gen 36020
key (4716351488 EXTENT_ITEM 24576) block 9396770734080 gen 36021
key (4719550464 EXTENT_ITEM 4096) block 9396727054336 gen 36020
key (4722884608 EXTENT_ITEM 4096) block 9396727382016 gen 36020
key (4726890496 EXTENT_ITEM 98304) block 670056448 gen 57
key (4732628992 EXTENT_ITEM 4096) block 231292944384 gen 28531
key (4734210048 EXTENT_ITEM 77824) block 639041536 gen 55
key (4740845568 EXTENT_ITEM 102400) block 639008768 gen 55
key (4750446592 EXTENT_ITEM 86016) block 96567296 gen 28
key (4758401024 EXTENT_ITEM 102400) block 9396778450944 gen 37522
key (4765986816 EXTENT_ITEM 20480) block 7868714041344 gen 41295
key (4767555584 EXTENT_ITEM 4096) block 7868638625792 gen 41294
key (4768800768 EXTENT_ITEM 4096) block 7868714762240 gen 41295
key (4770930688 EXTENT_ITEM 4096) block 2536061288448 gen 41254
key (4773249024 EXTENT_ITEM 4096) block 9397087469568 gen 41211
key (4775387136 EXTENT_ITEM 8192) block 9396848607232 gen 41195
key (4778356736 EXTENT_ITEM 28672) block 942625062912 gen 41247
key (4780965888 EXTENT_ITEM 4096) block 942625112064 gen 41247
key (4782530560 EXTENT_ITEM 4096) block 3775344148480 gen 41261
key (4784771072 EXTENT_ITEM 16384) block 942625243136 gen 41247
key (4787290112 EXTENT_ITEM 8192) block 7868936814592 gen 41296
key (4789653504 EXTENT_ITEM 4096) block 942625357824 gen 41247
key (4791791616 EXTENT_ITEM 8192) block 3775344312320 gen 41261
key (4793933824 EXTENT_ITEM 98304) block 5816509218816 gen 40812
key (4796350464 EXTENT_ITEM 28672) block 9396366884864 gen 41156
key (4799774720 EXTENT_ITEM 8192) block 942625374208 gen 41247
key (4802109440 EXTENT_ITEM 20480) block 7448950226944 gen 41291
key (4803289088 EXTENT_ITEM 16384) block 7868937043968 gen 41296
key (4805685248 EXTENT_ITEM 4096) block 7868937175040 gen 41296
key (4809154560 EXTENT_ITEM 4096) block 9397294366720 gen 41223
key (4811177984 EXTENT_ITEM 40960) block 942625947648 gen 41247
key (4813615104 EXTENT_ITEM 8192) block 4589373095936 gen 40655
key (4816318464 EXTENT_ITEM 24576) block 3776074383360 gen 40147
key (4818489344 EXTENT_ITEM 12288) block 8639672598528 gen 40870
key (4822265856 EXTENT_ITEM 4096) block 3776061538304 gen 40791
key (4825964544 EXTENT_ITEM 32768) block 9396546158592 gen 40893
key (4829077504 EXTENT_ITEM 16384) block 7868715188224 gen 41295
key (4830892032 EXTENT_ITEM 8192) block 784449536 gen 40953
key (4833468416 EXTENT_ITEM 28672) block 7868937207808 gen 41296
key (4835475456 EXTENT_ITEM 4096) block 9396878082048 gen 41199
key (4838522880 EXTENT_ITEM 12288) block 9396603584512 gen 41169
key (4841701376 EXTENT_ITEM 32768) block 558690385920 gen 41237
key (4844453888 EXTENT_ITEM 4096) block 9396802961408 gen 40913
key (4848943104 EXTENT_ITEM 98304) block 9396838121472 gen 41194
key (4852142080 EXTENT_ITEM 98304) block 9396838154240 gen 41194
key (4857458688 EXTENT_ITEM 28672) block 9396809285632 gen 41191
key (4862119936 EXTENT_ITEM 4096) block 4589973487616 gen 41266
key (4863778816 EXTENT_ITEM 4096) block 9396714717184 gen 41182
key (4865093632 EXTENT_ITEM 8192) block 7868955361280 gen 41297
key (4866400256 EXTENT_ITEM 4096) block 7868955492352 gen 41297
key (4868026368 EXTENT_ITEM 4096) block 9397142306816 gen 41218
key (4871659520 EXTENT_ITEM 524288) block 7869437165568 gen 41294
key (4895887360 EXTENT_ITEM 16384) block 7868937256960 gen 41296
key (4910223360 EXTENT_ITEM 4096) block 8639690457088 gen 41124
key (4911898624 EXTENT_ITEM 4096) block 7868937306112 gen 41296
key (4936237056 EXTENT_ITEM 49152) block 7868716531712 gen 41295
key (4939005952 EXTENT_ITEM 24576) block 9396665958400 gen 41176
key (4942991360 EXTENT_ITEM 4096) block 9397134147584 gen 41216
key (4952170496 EXTENT_ITEM 16384) block 7448645550080 gen 40818
key (4956131328 EXTENT_ITEM 4096) block 3775344443392 gen 41261
key (4959563776 EXTENT_ITEM 4096) block 4590017527808 gen 41267
key (4963291136 EXTENT_ITEM 4096) block 3775344541696 gen 41261
key (4965392384 EXTENT_ITEM 4096) block 5816529928192 gen 41278
key (4968370176 EXTENT_ITEM 53248) block 942503280640 gen 39944
key (4970328064 EXTENT_ITEM 4096) block 9397138194432 gen 41217
key (4973715456 EXTENT_ITEM 12288) block 9396865826816 gen 41198
key (4976988160 EXTENT_ITEM 98304) block 9396882669568 gen 41200
key (4979671040 EXTENT_ITEM 28672) block 9396970242048 gen 41205
key (4984193024 EXTENT_ITEM 81920) block 9397067284480 gen 41209
key (4988620800 EXTENT_ITEM 4096) block 7868955508736 gen 41297
key (4991299584 EXTENT_ITEM 77824) block 9397134180352 gen 41216
key (4995235840 EXTENT_ITEM 28672) block 7868976021504 gen 41301
key (4997394432 EXTENT_ITEM 8192) block 7868978298880 gen 41301
key (4998549504 EXTENT_ITEM 4096) block 7868978348032 gen 41301
key (4999438336 EXTENT_ITEM 4096) block 7868978741248 gen 41301
key (5000491008 EXTENT_ITEM 8192) block 7868978839552 gen 41301
key (5002002432 EXTENT_ITEM 8192) block 7868978888704 gen 41301
key (5003468800 EXTENT_ITEM 8192) block 7868964487168 gen 41298
key (5005344768 EXTENT_ITEM 4096) block 7868959227904 gen 41299
key (5006823424 EXTENT_ITEM 8192) block 7868979331072 gen 41301
key (5008551936 EXTENT_ITEM 4096) block 942627520512 gen 41247
key (5010718720 EXTENT_ITEM 524288) block 4590226325504 gen 41272
key (5012951040 EXTENT_ITEM 4096) block 4590227636224 gen 41272
key (5014720512 EXTENT_ITEM 4096) block 7868717072384 gen 41295
key (5017251840 EXTENT_ITEM 4096) block 4590030864384 gen 41267
key (5019693056 EXTENT_ITEM 20480) block 9397294792704 gen 41223
key (5021507584 EXTENT_ITEM 24576) block 977780736 gen 40747
key (5024075776 EXTENT_ITEM 4096) block 4180998193152 gen 41264
key (5026807808 EXTENT_ITEM 16384) block 7448755437568 gen 41289
key (5029982208 EXTENT_ITEM 65536) block 7448755453952 gen 41289
key (5033459712 EXTENT_ITEM 4096) block 231498268672 gen 40956
key (5036089344 EXTENT_ITEM 98304) block 231498301440 gen 40956
key (5038014464 EXTENT_ITEM 4096) block 942627569664 gen 41247
key (5041418240 EXTENT_ITEM 4096) block 7868937699328 gen 41296
key (5044297728 EXTENT_ITEM 16384) block 7869441900544 gen 41294
key (5049479168 EXTENT_ITEM 4096) block 9396863270912 gen 41197
key (5053247488 EXTENT_ITEM 45056) block 9396470906880 gen 41158
key (5057945600 EXTENT_ITEM 32768) block 9396705083392 gen 41181
key (5062139904 EXTENT_ITEM 28672) block 5817906200576 gen 41281
key (5068361728 EXTENT_ITEM 24576) block 7868938158080 gen 41296
key (5073403904 EXTENT_ITEM 40960) block 7448755601408 gen 41289
key (5076926464 EXTENT_ITEM 53248) block 942627864576 gen 41247
key (5083922432 EXTENT_ITEM 28672) block 9396714995712 gen 41182
key (5088481280 EXTENT_ITEM 16384) block 7448755896320 gen 41289
key (5096517632 EXTENT_ITEM 98304) block 5816517230592 gen 40812
key (5102915584 EXTENT_ITEM 4096) block 105168896 gen 38049
key (5117231104 EXTENT_ITEM 8192) block 942057668608 gen 36283
key (5123514368 EXTENT_ITEM 524288) block 7448950554624 gen 41291
key (5129158656 EXTENT_ITEM 4096) block 7868938190848 gen 41296
key (5131296768 EXTENT_ITEM 8192) block 9397142536192 gen 41218
key (5133934592 EXTENT_ITEM 4096) block 7868955803648 gen 41297
key (5135691776 EXTENT_ITEM 4096) block 7868955901952 gen 41297
key (5137408000 EXTENT_ITEM 20480) block 7869441998848 gen 41294
key (5139275776 EXTENT_ITEM 4096) block 9397379121152 gen 40936
key (5141598208 EXTENT_ITEM 24576) block 9396369162240 gen 41156
key (5143535616 EXTENT_ITEM 4096) block 7869442015232 gen 41294
key (5145665536 EXTENT_ITEM 4096) block 7868938354688 gen 41296
key (5148573696 EXTENT_ITEM 4096) block 92143616 gen 41230
key (5151682560 EXTENT_ITEM 28672) block 9396471054336 gen 41158
key (5154365440 EXTENT_ITEM 8192) block 942224998400 gen 40645
key (5160894464 EXTENT_ITEM 4096) block 942628208640 gen 41247
key (5166387200 EXTENT_ITEM 8192) block 5406310367232 gen 41278
key (5170249728 EXTENT_ITEM 8192) block 7868717400064 gen 41295
key (5175058432 EXTENT_ITEM 20480) block 9396471169024 gen 41158
key (5180461056 EXTENT_ITEM 4096) block 7448756158464 gen 41289
key (5185339392 EXTENT_ITEM 126976) block 942509686784 gen 41245
key (5215932416 EXTENT_ITEM 163840) block 942628290560 gen 41247
key (5226233856 EXTENT_ITEM 8192) block 970440704 gen 39927
key (5248667648 EXTENT_ITEM 8192) block 309215232 gen 40949
key (5254905856 EXTENT_ITEM 98304) block 7868718563328 gen 41295
key (5262569472 EXTENT_ITEM 4096) block 7868955918336 gen 41297
key (5265358848 EXTENT_ITEM 16384) block 7868955951104 gen 41297
key (5266345984 EXTENT_ITEM 8192) block 7868964716544 gen 41298
key (5267578880 EXTENT_ITEM 4096) block 7868956000256 gen 41297
key (5269118976 EXTENT_ITEM 45056) block 7868964831232 gen 41298
key (5270089728 EXTENT_ITEM 49152) block 4589975011328 gen 41266
key (5271646208 EXTENT_ITEM 16384) block 942629044224 gen 41247
key (5274095616 EXTENT_ITEM 20480) block 7868938387456 gen 41296
key (5276610560 EXTENT_ITEM 4096) block 9396370227200 gen 41156
key (5279043584 EXTENT_ITEM 4096) block 9396370259968 gen 41156
key (5281902592 EXTENT_ITEM 12288) block 799604736 gen 40953
key (5283176448 EXTENT_ITEM 4096) block 7448756338688 gen 41289
key (5285687296 EXTENT_ITEM 8192) block 9396806483968 gen 40913
key (5288542208 EXTENT_ITEM 12288) block 942629289984 gen 41247
key (5290602496 EXTENT_ITEM 12288) block 3776076038144 gen 41262
key (5293809664 EXTENT_ITEM 49152) block 5816867209216 gen 41279
key (5296504832 EXTENT_ITEM 53248) block 9396370505728 gen 41156
key (5300219904 EXTENT_ITEM 36864) block 942629388288 gen 41247
key (5303939072 EXTENT_ITEM 4096) block 7448757108736 gen 41289
key (5307674624 EXTENT_ITEM 36864) block 4589225345024 gen 41264
key (5310844928 EXTENT_ITEM 8192) block 7869444489216 gen 41294
key (5316055040 EXTENT_ITEM 20480) block 942629535744 gen 41247
key (5326077952 EXTENT_ITEM 45056) block 9397067776000 gen 41209
key (5345366016 EXTENT_ITEM 24576) block 5406310531072 gen 41278
key (5348425728 EXTENT_ITEM 4096) block 5816867635200 gen 41279
key (5352050688 EXTENT_ITEM 167936) block 9396896940032 gen 41201
key (5356605440 EXTENT_ITEM 4096) block 942629732352 gen 41247
key (5360381952 EXTENT_ITEM 98304) block 7448757256192 gen 41289
key (5364903936 EXTENT_ITEM 20480) block 7868938420224 gen 41296
key (5369483264 EXTENT_ITEM 81920) block 7868421308416 gen 41370
key (5410738176 EXTENT_ITEM 4096) block 7448787648512 gen 39692
key (5502967808 EXTENT_ITEM 524288) block 3775344787456 gen 41261
key (5519265792 EXTENT_ITEM 98304) block 3775765921792 gen 41261
key (5533323264 EXTENT_ITEM 49152) block 7868719415296 gen 41295
key (5552054272 EXTENT_ITEM 28672) block 7868938469376 gen 41296
key (5561016320 EXTENT_ITEM 110592) block 3775765970944 gen 41261
key (5570375680 EXTENT_ITEM 98304) block 3775766102016 gen 41261
key (5589356544 EXTENT_ITEM 98304) block 7868938485760 gen 41296
key (5602086912 EXTENT_ITEM 98304) block 7868938502144 gen 41296
key (5672407040 EXTENT_ITEM 524288) block 5818311360512 gen 39108
key (5723496448 EXTENT_ITEM 524288) block 3775766200320 gen 41261
key (5766778880 EXTENT_ITEM 36864) block 4996213800960 gen 40996
key (5776162816 EXTENT_ITEM 98304) block 8278224125952 gen 40678
key (5784961024 EXTENT_ITEM 110592) block 7449169199104 gen 41293
key (5805203456 EXTENT_ITEM 98304) block 4996213932032 gen 40996
key (5814771712 EXTENT_ITEM 69632) block 3775344934912 gen 41261
key (5822115840 EXTENT_ITEM 98304) block 9396372127744 gen 41156
key (5833756672 EXTENT_ITEM 98304) block 7868938518528 gen 41296
key (5845352448 EXTENT_ITEM 98304) block 942505279488 gen 39944
key (5902315520 EXTENT_ITEM 524288) block 3776013942784 gen 36818
key (6039678976 EXTENT_ITEM 524288) block 942099382272 gen 35523
key (6094204928 EXTENT_ITEM 524288) block 3775766413312 gen 41261
key (6184697856 EXTENT_ITEM 524288) block 7869425303552 gen 37923
key (6270717952 EXTENT_ITEM 73728) block 9396854947840 gen 37326
key (6343368704 EXTENT_ITEM 524288) block 942504312832 gen 40769
key (6401916928 EXTENT_ITEM 524288) block 9396564049920 gen 40444
key (6489030656 EXTENT_ITEM 524288) block 482508800 gen 39926
key (6569422848 EXTENT_ITEM 8192) block 482525184 gen 39926
key (6629724160 EXTENT_ITEM 524288) block 8639742197760 gen 4797
key (6738776064 EXTENT_ITEM 524288) block 5816846073856 gen 4451
key (6844104704 EXTENT_ITEM 524288) block 9397189148672 gen 40922
key (6884216832 EXTENT_ITEM 98304) block 2943141330944 gen 40974
key (6892744704 EXTENT_ITEM 98304) block 3775766462464 gen 41261
key (6908829696 EXTENT_ITEM 126976) block 9396372455424 gen 41156
key (6915735552 EXTENT_ITEM 98304) block 3775345000448 gen 41261
key (6928158720 EXTENT_ITEM 32768) block 7868721168384 gen 41295
key (6937444352 EXTENT_ITEM 122880) block 3775345033216 gen 41261
key (6947274752 EXTENT_ITEM 98304) block 3775766478848 gen 41261
key (6973108224 EXTENT_ITEM 524288) block 9396684226560 gen 36019
key (7110107136 EXTENT_ITEM 524288) block 942100070400 gen 35523
key (7218696192 EXTENT_ITEM 524288) block 3776014385152 gen 36818
key (7271628800 EXTENT_ITEM 524288) block 7040438484992 gen 39109
key (7325204480 EXTENT_ITEM 524288) block 942182432768 gen 35523
key (7433043968 EXTENT_ITEM 524288) block 942182449152 gen 35523
key (7527575552 EXTENT_ITEM 114688) block 7868411461632 gen 41370
key (7594835968 EXTENT_ITEM 524288) block 8278230417408 gen 40685
key (7636733952 EXTENT_ITEM 524288) block 4590258159616 gen 40163
key (7734194176 EXTENT_ITEM 524288) block 8278371172352 gen 39707
key (7832891392 EXTENT_ITEM 98304) block 9397245100032 gen 40930
key (7893303296 EXTENT_ITEM 524288) block 7449139478528 gen 41022
key (7971422208 EXTENT_ITEM 12288) block 4996214308864 gen 40996
key (7983030272 EXTENT_ITEM 98304) block 9397260566528 gen 40279
key (8027660288 EXTENT_ITEM 524288) block 9396372930560 gen 41156
key (8132108288 EXTENT_ITEM 4096) block 9396372996096 gen 41156
key (8222445568 EXTENT_ITEM 524288) block 7868938534912 gen 41296
key (8298119168 EXTENT_ITEM 524288) block 942407598080 gen 35526
key (8403017728 EXTENT_ITEM 524288) block 7868721266688 gen 41295
key (8457416704 EXTENT_ITEM 524288) block 942407581696 gen 35526
key (8566042624 EXTENT_ITEM 98304) block 4996208443392 gen 41275
key (8601628672 EXTENT_ITEM 524288) block 9396398686208 gen 36503
key (8700424192 EXTENT_ITEM 524288) block 9396373209088 gen 41156
key (8747155456 EXTENT_ITEM 524288) block 8639667011584 gen 4789
key (8856207360 EXTENT_ITEM 524288) block 3776014958592 gen 36818
key (8892473344 EXTENT_ITEM 524288) block 3776014991360 gen 36818
key (8941182976 EXTENT_ITEM 524288) block 4996214407168 gen 40996
key (9045504000 EXTENT_ITEM 524288) block 942192705536 gen 35523
key (9154273280 EXTENT_ITEM 524288) block 4996214652928 gen 40996
key (9259773952 EXTENT_ITEM 4096) block 9396808155136 gen 39906
key (9354686464 EXTENT_ITEM 524288) block 3776015122432 gen 36818
key (9463738368 EXTENT_ITEM 524288) block 8278135226368 gen 40675
key (9517215744 EXTENT_ITEM 524288) block 3776015171584 gen 36818
key (9569591296 EXTENT_ITEM 524288) block 5818298400768 gen 4453
key (9678643200 EXTENT_ITEM 524288) block 7868466675712 gen 41369
key (9732280320 EXTENT_ITEM 524288) block 8277859418112 gen 4468
key (9786400768 EXTENT_ITEM 524288) block 8639389859840 gen 4684
key (9894985728 EXTENT_ITEM 524288) block 9396471332864 gen 41158
key (9926197248 EXTENT_ITEM 81920) block 9396471349248 gen 41158
key (9966399488 EXTENT_ITEM 24576) block 7448640667648 gen 41024
key (9974960128 EXTENT_ITEM 188416) block 7868721283072 gen 41295
key (10026110976 EXTENT_ITEM 73728) block 7868938551296 gen 41296
key (10049306624 EXTENT_ITEM 524288) block 9396775354368 gen 36021
key (10158358528 EXTENT_ITEM 524288) block 5818494631936 gen 4454
key (10266910720 EXTENT_ITEM 524288) block 5405501112320 gen 41277
key (10295099392 EXTENT_ITEM 98304) block 7868938567680 gen 41296
key (10304290816 EXTENT_ITEM 32768) block 5405501145088 gen 41277
key (10331996160 EXTENT_ITEM 524288) block 7448692178944 gen 41031
key (10436939776 EXTENT_ITEM 4096) block 7868938584064 gen 41296
key (10468483072 EXTENT_ITEM 32768) block 942510587904 gen 41245
key (10530668544 EXTENT_ITEM 98304) block 942510604288 gen 41245
key (10544721920 EXTENT_ITEM 98304) block 5817952567296 gen 40659
key (10583732224 EXTENT_ITEM 98304) block 7448606425088 gen 41033
key (10601713664 EXTENT_ITEM 32768) block 5817952583680 gen 40659
key (10628919296 EXTENT_ITEM 57344) block 7868938682368 gen 41296
key (10642010112 EXTENT_ITEM 32768) block 7868939075584 gen 41296
key (10684547072 EXTENT_ITEM 98304) block 9396471398400 gen 41158
key (10700075008 EXTENT_ITEM 98304) block 4590262255616 gen 40163
key (10720514048 EXTENT_ITEM 98304) block 8278282272768 gen 40673
key (10742046720 EXTENT_ITEM 524288) block 942225965056 gen 40645
key (10767659008 EXTENT_ITEM 524288) block 4996214734848 gen 40996
key (10795347968 EXTENT_ITEM 40960) block 9397260632064 gen 40279
key (10842365952 EXTENT_ITEM 98304) block 9397260713984 gen 40279
key (10906628096 EXTENT_ITEM 16384) block 4590268366848 gen 40163
key (10998546432 EXTENT_ITEM 524288) block 4590268383232 gen 40163
key (11030339584 EXTENT_ITEM 524288) block 3776073924608 gen 39956
key (11086778368 EXTENT_ITEM 8192) block 7868722692096 gen 41295
key (11130830848 EXTENT_ITEM 524288) block 3776074088448 gen 39956
key (11179741184 EXTENT_ITEM 524288) block 9396487946240 gen 36015
key (11288309760 EXTENT_ITEM 8192) block 942504411136 gen 40769
key (11314683904 EXTENT_ITEM 524288) block 7040428982272 gen 39108
key (11365765120 EXTENT_ITEM 524288) block 942621818880 gen 40646
key (11413032960 EXTENT_ITEM 524288) block 9396775829504 gen 36021
key (11478949888 EXTENT_ITEM 524288) block 9396775845888 gen 36021
key (11592572928 EXTENT_ITEM 12288) block 9396775862272 gen 36021
key (11643600896 EXTENT_ITEM 524288) block 8639827148800 gen 4812
key (11752652800 EXTENT_ITEM 524288) block 9396588888064 gen 36017
key (11811061760 EXTENT_ITEM 524288) block 7868466544640 gen 41369
key (11868700672 EXTENT_ITEM 524288) block 2129032593408 gen 40321
key (11913945088 EXTENT_ITEM 20480) block 9396374487040 gen 41156
key (11929706496 EXTENT_ITEM 102400) block 942504476672 gen 40769
key (11949088768 EXTENT_ITEM 16384) block 9396374962176 gen 41156
key (11962871808 EXTENT_ITEM 20480) block 305971200 gen 40744
key (11978194944 EXTENT_ITEM 163840) block 9396375142400 gen 41156
key (11989528576 EXTENT_ITEM 98304) block 9396375175168 gen 41156
key (12004290560 EXTENT_ITEM 8192) block 9396375240704 gen 41156
key (12023758848 EXTENT_ITEM 8192) block 9396375306240 gen 41156
key (12039868416 EXTENT_ITEM 98304) block 7869453352960 gen 40671
key (12062629888 EXTENT_ITEM 524288) block 3776077185024 gen 41262
key (12090339328 EXTENT_ITEM 45056) block 9396375552000 gen 41156
key (12109578240 EXTENT_ITEM 40960) block 7868939124736 gen 41296
key (12123807744 EXTENT_ITEM 57344) block 7868939223040 gen 41296
key (12154621952 EXTENT_ITEM 8192) block 9396471644160 gen 41158
key (12182827008 EXTENT_ITEM 77824) block 7868939255808 gen 41296
key (12202307584 EXTENT_ITEM 98304) block 9396472020992 gen 41158
key (12224954368 EXTENT_ITEM 524288) block 4996215029760 gen 40996
key (12257734656 EXTENT_ITEM 524288) block 8277561999360 gen 40399
key (12311920640 EXTENT_ITEM 102400) block 7868939354112 gen 41296
key (12322902016 EXTENT_ITEM 94208) block 4996215930880 gen 40996
key (12336902144 EXTENT_ITEM 94208) block 7868723052544 gen 41295
key (12371206144 EXTENT_ITEM 524288) block 7868723101696 gen 41295
key (12412854272 EXTENT_ITEM 20480) block 9396376502272 gen 41156
key (12481040384 EXTENT_ITEM 77824) block 7868939370496 gen 41296
key (12508151808 EXTENT_ITEM 106496) block 9396376567808 gen 41156
key (12557406208 EXTENT_ITEM 524288) block 9396833189888 gen 36022
key (12694769664 EXTENT_ITEM 524288) block 7868939386880 gen 41296
key (12732932096 EXTENT_ITEM 98304) block 7868939436032 gen 41296
key (12780756992 EXTENT_ITEM 98304) block 9396572504064 gen 40444
key (12802924544 EXTENT_ITEM 131072) block 394002432 gen 40745
key (12820791296 EXTENT_ITEM 53248) block 7448939151360 gen 40383
key (12847665152 EXTENT_ITEM 24576) block 231939375104 gen 40120
key (12894363648 EXTENT_ITEM 126976) block 7868939681792 gen 41296
key (12938608640 EXTENT_ITEM 126976) block 8278492938240 gen 40025
key (12951064576 EXTENT_ITEM 4096) block 8278492954624 gen 40025
key (12967153664 EXTENT_ITEM 4096) block 395182080 gen 40745
key (13098221568 EXTENT_ITEM 524288) block 5816522833920 gen 40812
key (13139005440 EXTENT_ITEM 98304) block 4997158699008 gen 39972
key (13156515840 EXTENT_ITEM 98304) block 5816526864384 gen 40658
key (13208715264 EXTENT_ITEM 122880) block 9396472643584 gen 41158
key (13227663360 EXTENT_ITEM 98304) block 7868939698176 gen 41296
key (13242699776 EXTENT_ITEM 524288) block 7868724723712 gen 41295
key (13278638080 EXTENT_ITEM 524288) block 4996216373248 gen 40996
key (13309706240 EXTENT_ITEM 4096) block 4996216602624 gen 40996
key (13319843840 EXTENT_ITEM 32768) block 7868725035008 gen 41295
key (13362659328 EXTENT_ITEM 524288) block 230915735552 gen 40303
key (13407723520 EXTENT_ITEM 524288) block 964149248 gen 40746
key (13440901120 EXTENT_ITEM 118784) block 3775345508352 gen 41261
key (13463625728 EXTENT_ITEM 4096) block 7448772083712 gen 40662
key (13489963008 EXTENT_ITEM 4096) block 7868939763712 gen 41296
key (13509861376 EXTENT_ITEM 12288) block 5816868290560 gen 41279
key (13531414528 EXTENT_ITEM 122880) block 3775604441088 gen 41260
key (13547933696 EXTENT_ITEM 131072) block 964231168 gen 40746
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Filesystem corruption after convert-to-block-group-tree
2023-10-04 7:42 ` Alexander Duscheleit
@ 2023-10-04 8:18 ` Qu Wenruo
2023-10-04 15:13 ` Alexander Duscheleit
1 sibling, 0 replies; 5+ messages in thread
From: Qu Wenruo @ 2023-10-04 8:18 UTC (permalink / raw)
To: Alexander Duscheleit, linux-btrfs
On 2023/10/4 18:12, Alexander Duscheleit wrote:
> 4 October 2023 at 05:27, "Qu Wenruo" <quwenruo.btrfs@gmx.com> wrote:
>
>>
>> On 2023/10/4 09:47, Alexander Duscheleit wrote:
>>
>>>
>>> Hi all,
>>> earlier today I tried to convert my BTRFS filesystem to block-group-tree and the operation seemed successful at first glance.
>>> (I unmounted, converted and mounted the fs without any error.)
>>> Some time later I tried to access a file and got an I/O error.
>>>
>>
>> The error from the dmesg shows a level mismatch in block group tree.
>>
>> However the block group tree itself is always fully read at each mount,
>> thus it means at your first mount, the block group tree is good, or you
>> can not even mount the fs (unless go with -o rescue=all).
>>
>> So I'm afraid the corruption is not caused by the conversion, but
>> something else, after your initial mount.
>>
>> I'm more interested in what happened between the initial mount and "some
>> time later".
>
> As far as I can tell, nothing. The array contains mostly media files, but
> also some minor backups (borg-backup). I didn't put any new media on it
> during the time in question and no backups were running.
>
> After the conversion I mounted the fs, restartet nfs, re-mounted the nfs
> share on the htpc and then left it alone. I didn't do any operations on it
> besides a ls to confirm it's "there".
> I noticed the error when I wanted to play some media file and the player threw
> an error. >
>>
>>>
>>> after some updates, reboots and troubleshooting I ended up in the following situation:
>>>
>>> The fs cannot be mounted normally, but it mounts (consistently) with
>>> -o rescue=all,ro (see attached dmesg.log).
>>>
>>> No data _appears_ to be missing or corrupt.
>>>
>>> btrfs-find-root throws many errors concerning corrupt leaf blocks but does find the curren tree root. (Again, see attached log.)
>>>
>>> Is there any way to bring this fs back to a useable state without starting over from scratch?
>>>
>>
>> It looks like block group tree is corrupted, but without a full "btrfs
>> check --read-only" it's hard to say.
>
> # btrfs check --readonly /dev/sdb1
> Opening filesystem to check...
> parent transid verify failed on 7868411314176 wanted 41293 found 41370
> parent transid verify failed on 7868411314176 wanted 41293 found 41370
> parent transid verify failed on 7868411314176 wanted 41293 found 41370
> parent transid verify failed on 7868411314176 wanted 41293 found 41370
> parent transid verify failed on 7868411314176 wanted 41293 found 41370
> Ignoring transid failure
> ERROR: child eb corrupted: parent bytenr=7868939862016 item=336 parent level=1 child bytenr=7868411314176 child level=1
OK, this mean is way worse than I initially thought.
With your dump tree (the only working one), this means one extent tree
block has written into the location owned by block group tree.
This breaks the very basis of btrfs metadata COW, thus a huge break to
the fs.
Unfortunately btrfs progs is not clever enough to ignore the block
groups tree corruption and continue, thus no further corruption analyze
from btrfs check.
From this stage, it's really hard to pin down the cause.
I can only recommend to salvage data first using "-o rescue=all,ro" for now.
Thanks,
Qu
> ERROR: failed to read block groups: Input/output error
> ERROR: cannot open file system
>
>>
>> Extra dump on the bg tree can also help (needs both stderr and stdout):
>>
>> # btrfs ins dump-tree -t 11 <device>
>
> yields "ERROR: cannot print block group tree, invalid pointer" for each of the 4 devices.
>
>>
>> # btrfs ins dump-tree -b 7868411314176 <device>
>
> Attached
>
>>
>> BTW, btrfs-find-root is not that useful, thus it should only be adviced
>> by developers, but at least it does no harm.
>>
>> Thanks,
>> Qu
>>
>>>
>>> System Data:
>>> # uname -a
>>> Linux hera 6.5.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 23 Sep 2023 22:55:13 +0000 x86_64 GNU/Linux
>>>
>>> # btrfs --version
>>> btrfs-progs v6.5.1
>>>
>>> (Note: The conversion to block group tree was done with btrfs-progs 6.3.3 and Kernel 6.4.12.arch1-1)
>>>
>>> # btrfs fi show
>>> Label: 'hera-storage' uuid: a71011f9-d79c-40e8-85fb-60b6f2af0637
>>> Total devices 4 FS bytes used 8.36TiB
>>> devid 1 size 4.55TiB used 4.19TiB path /dev/sdb1
>>> devid 2 size 4.55TiB used 4.19TiB path /dev/sdd1
>>> devid 3 size 4.55TiB used 4.19TiB path /dev/sdc1
>>> devid 4 size 4.55TiB used 4.19TiB path /dev/sde1
>>>
>>> # btrfs fi df /mnt/btrfs_storage
>>> Data, RAID10: total=8.34TiB, used=8.34TiB
>>> System, RAID1C4: total=8.00MiB, used=912.00KiB
>>> Metadata, RAID1C4: total=24.00GiB, used=23.93GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>>
>>
>
> Thanks,
> Alex
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Filesystem corruption after convert-to-block-group-tree
2023-10-04 7:42 ` Alexander Duscheleit
2023-10-04 8:18 ` Qu Wenruo
@ 2023-10-04 15:13 ` Alexander Duscheleit
1 sibling, 0 replies; 5+ messages in thread
From: Alexander Duscheleit @ 2023-10-04 15:13 UTC (permalink / raw)
To: linux-btrfs
4 October 2023 at 10:18, "Qu Wenruo" <quwenruo.btrfs@gmx.com> wrote:
>
> On 2023/10/4 18:12, Alexander Duscheleit wrote:
>
> >
> > 4 October 2023 at 05:27, "Qu Wenruo" <quwenruo.btrfs@gmx.com> wrote:
> >
> > >
> > > On 2023/10/4 09:47, Alexander Duscheleit wrote:
> > >
> >
> > Hi all,
> > earlier today I tried to convert my BTRFS filesystem to block-group-tree and the operation seemed successful at first glance.
> > (I unmounted, converted and mounted the fs without any error.)
> > Some time later I tried to access a file and got an I/O error.
> >
> > >
> > > The error from the dmesg shows a level mismatch in block group tree.
> > >
> > > However the block group tree itself is always fully read at each mount,
> > > thus it means at your first mount, the block group tree is good, or you
> > > can not even mount the fs (unless go with -o rescue=all).
> > >
> > > So I'm afraid the corruption is not caused by the conversion, but
> > > something else, after your initial mount.
> > >
> > > I'm more interested in what happened between the initial mount and "some
> > > time later".
> > >
> >
> > As far as I can tell, nothing. The array contains mostly media files, but
> > also some minor backups (borg-backup). I didn't put any new media on it
> > during the time in question and no backups were running.
> >
> > After the conversion I mounted the fs, restartet nfs, re-mounted the nfs
> > share on the htpc and then left it alone. I didn't do any operations on it
> > besides a ls to confirm it's "there".
> > I noticed the error when I wanted to play some media file and the player threw
> > an error. >
> >
> > >
> > >
> > >
> >
> > after some updates, reboots and troubleshooting I ended up in the following situation:
> >
> > The fs cannot be mounted normally, but it mounts (consistently) with
> > -o rescue=all,ro (see attached dmesg.log).
> >
> > No data _appears_ to be missing or corrupt.
> >
> > btrfs-find-root throws many errors concerning corrupt leaf blocks but does find the curren tree root. (Again, see attached log.)
> >
> > Is there any way to bring this fs back to a useable state without starting over from scratch?
> >
> > >
> > > It looks like block group tree is corrupted, but without a full "btrfs
> > > check --read-only" it's hard to say.
> > >
> >
> > # btrfs check --readonly /dev/sdb1
> > Opening filesystem to check...
> > parent transid verify failed on 7868411314176 wanted 41293 found 41370
> > parent transid verify failed on 7868411314176 wanted 41293 found 41370
> > parent transid verify failed on 7868411314176 wanted 41293 found 41370
> > parent transid verify failed on 7868411314176 wanted 41293 found 41370
> > parent transid verify failed on 7868411314176 wanted 41293 found 41370
> > Ignoring transid failure
> > ERROR: child eb corrupted: parent bytenr=7868939862016 item=336 parent level=1 child bytenr=7868411314176 child level=1
> >
>
> OK, this mean is way worse than I initially thought.
>
> With your dump tree (the only working one), this means one extent tree
> block has written into the location owned by block group tree.
>
> This breaks the very basis of btrfs metadata COW, thus a huge break to
> the fs.
>
> Unfortunately btrfs progs is not clever enough to ignore the block
> groups tree corruption and continue, thus no further corruption analyze
> from btrfs check.
If you think of any info that might be useful I'm happy to provide it.
>
> From this stage, it's really hard to pin down the cause.
> I can only recommend to salvage data first using "-o rescue=all,ro" for now.
>
Most data is backed up anyway. Exapnding the array was already underway, I'll
just start over from scratch with the new disks instead.
My main concern right now is how I can prevent something similar from happening
in the future, since I don't understand how it could have happened in the first
place.
Thanks,
Alex
> Thanks,
> Qu
>
> >
> > ERROR: failed to read block groups: Input/output error
> > ERROR: cannot open file system
> >
> > >
> > > Extra dump on the bg tree can also help (needs both stderr and stdout):
> > >
> > > # btrfs ins dump-tree -t 11 <device>
> > >
> >
> > yields "ERROR: cannot print block group tree, invalid pointer" for each of the 4 devices.
> >
> > >
> > > # btrfs ins dump-tree -b 7868411314176 <device>
> > >
> >
> > Attached
> >
> > >
> > > BTW, btrfs-find-root is not that useful, thus it should only be adviced
> > > by developers, but at least it does no harm.
> > >
> > > Thanks,
> > > Qu
> > >
> >
> > System Data:
> > # uname -a
> > Linux hera 6.5.5-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 23 Sep 2023 22:55:13 +0000 x86_64 GNU/Linux
> >
> > # btrfs --version
> > btrfs-progs v6.5.1
> >
> > (Note: The conversion to block group tree was done with btrfs-progs 6.3.3 and Kernel 6.4.12.arch1-1)
> >
> > # btrfs fi show
> > Label: 'hera-storage' uuid: a71011f9-d79c-40e8-85fb-60b6f2af0637
> > Total devices 4 FS bytes used 8.36TiB
> > devid 1 size 4.55TiB used 4.19TiB path /dev/sdb1
> > devid 2 size 4.55TiB used 4.19TiB path /dev/sdd1
> > devid 3 size 4.55TiB used 4.19TiB path /dev/sdc1
> > devid 4 size 4.55TiB used 4.19TiB path /dev/sde1
> >
> > # btrfs fi df /mnt/btrfs_storage
> > Data, RAID10: total=8.34TiB, used=8.34TiB
> > System, RAID1C4: total=8.00MiB, used=912.00KiB
> > Metadata, RAID1C4: total=24.00GiB, used=23.93GiB
> > GlobalReserve, single: total=512.00MiB, used=0.00B
> >
> > >
> > >
> > >
> >
> > Thanks,
> > Alex
> >
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-10-04 15:13 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-10-03 23:17 Filesystem corruption after convert-to-block-group-tree Alexander Duscheleit
2023-10-04 3:27 ` Qu Wenruo
2023-10-04 7:42 ` Alexander Duscheleit
2023-10-04 8:18 ` Qu Wenruo
2023-10-04 15:13 ` Alexander Duscheleit
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).