Re: BTRFS corruption after conversion to block group tree

Russell Haley
2023-12-02T02:45:20Z

[-- Attachment #1: Type: text/plain, Size: 2467 bytes --]

On 12/1/23 14:09, Qu Wenruo wrote:
>
>
> On 2023/12/1 22:16, Russell Haley wrote:
>> Distro: Fedora 39
>> kernel: 6.5.12
>> btrfs-progs: v6.5.1.
>>
>> To put everyone at ease, the data seems mostly okay. The disk is
>> mountable with ro,rescue=ingorebadroots:nologreplay, and I'm pulling off
>> what I can.  Furthermore, there's a good chance this is the result of a
>> marginally-unstable overclock. However, the symptoms are very similar to
>> a recent report from Alexander Duscheleit [1], so there may be a deeper
>> problem. I can provide requested metadata dumps if it would be useful,
>> and I also have a few questions.
>>
>> A few days ago I used btrfstune to convert two btrfs filesystems to
>> block-group-tree, after reading that it would reduce mount time. After
>> about 10 hours, while I was using Steam, the more heavily-used of the
>> two disks, which holds (among other things) the Steam game library,
>> experienced a transaction abort and went read-only.
>
> Any dmesg of that transaction abort? That's the most critical info here.

[80343.530191] BTRFS error (device dm-1): parent transid verify failed
on logical 31850496 mirror 1 wanted 123883 found 123907
[80343.587776] BTRFS error (device dm-1): parent transid verify failed
on logical 31850496 mirror 2 wanted 123883 found 123907
[80343.587836] BTRFS error (device dm-1): failed to run delayed ref for
logical 916815872 num_bytes 16384 type 176 action 1 ref_mod 1: -5
[80343.587844] BTRFS error (device dm-1: state A): Transaction aborted
(error -5)
[80343.587846] BTRFS: error (device dm-1: state A) in
btrfs_run_delayed_refs:2182: errno=-5 IO failure

>> 2. There's something extremely weird with the primary superblock. "btrfs
>>     inspect-internal dump-super" says that superblock 0 doesn't have the
>>     BLOCK_GROUP_TREE or NO_HOLES flags set, unlike superblocks 1 and 2,
>>     which are identical in every field except csum ("[match]", for 0, 1,
>>     and 2) and bytenr.
>
> This is not good at all, if csum of super blocks doesn't match, there
> must be something especially wrong.

Comparing to other btrfs disks I have access to, different checksum for
the backup superblocks seems normal? It does say [match] after the
checksum value. Just the value itself is different.

I dumped the 3 superblocks with btrfs inspect-internal dump-super -f and
attached them to this message. "Meld" for 3-way diff worked well for me.

Thanks for your help,

Russell Haley

[-- Attachment #2: super-0.txt --]
[-- Type: text/plain, Size: 3376 bytes --]

superblock: bytenr=65536, device=/dev/mapper/3tb_spinner
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x404bfde3 [match]
bytenr 65536
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 8cf4bcd8-76c7-4494-a2c8-2578136c0aa6
metadata_uuid 00000000-0000-0000-0000-000000000000
label
generation 123887
root 875036672
sys_array_size 129
chunk_root_generation 123187
root_level 0
chunk_root 28459008
chunk_root_level 1
log_root 920076288
log_root_transid (deprecated) 0
log_root_level 0
total_bytes 3000589352960
bytes_used 1951818739712
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0x3
( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID )
incompat_flags 0x171
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA )
cache_generation 0
uuid_tree_generation 123887
dev_item.uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
dev_item.fsid 8cf4bcd8-76c7-4494-a2c8-2578136c0aa6 [match]
dev_item.type 0
dev_item.total_bytes 3000589352960
dev_item.bytes_used 1981078831104
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 22020096
dev_uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
stripe 1 devid 1 offset 30408704
dev_uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
backup_roots[4]:
backup 0:
backup_tree_root: 287932416 gen: 123885 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 203538432 gen: 123885 level: 2
backup_fs_root: 185040896 gen: 123885 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 185630720 gen: 123885 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951818743808
backup_num_devices: 1

backup 1:
backup_tree_root: 805404672 gen: 123886 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 458981376 gen: 123886 level: 2
backup_fs_root: 291143680 gen: 123886 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 185630720 gen: 123885 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951818743808
backup_num_devices: 1

backup 2:
backup_tree_root: 875036672 gen: 123887 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 806912000 gen: 123887 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951818739712
backup_num_devices: 1

backup 3:
backup_tree_root: 146046976 gen: 123884 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 44793856 gen: 123884 level: 2
backup_fs_root: 185040896 gen: 123885 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 36765696 gen: 123884 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951779827712
backup_num_devices: 1

[-- Attachment #3: super-1.txt --]
[-- Type: text/plain, Size: 3432 bytes --]

superblock: bytenr=67108864, device=/dev/mapper/3tb_spinner
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0x076d5bba [match]
bytenr 67108864
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 8cf4bcd8-76c7-4494-a2c8-2578136c0aa6
metadata_uuid 00000000-0000-0000-0000-000000000000
label
generation 123917
root 1871598567424
sys_array_size 129
chunk_root_generation 123187
root_level 0
chunk_root 28459008
chunk_root_level 1
log_root 0
log_root_transid (deprecated) 0
log_root_level 0
total_bytes 3000589352960
bytes_used 1951820050432
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0xb
( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID |
BLOCK_GROUP_TREE )
incompat_flags 0x371
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 0
uuid_tree_generation 123887
dev_item.uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
dev_item.fsid 8cf4bcd8-76c7-4494-a2c8-2578136c0aa6 [match]
dev_item.type 0
dev_item.total_bytes 3000589352960
dev_item.bytes_used 1981078831104
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 22020096
dev_uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
stripe 1 devid 1 offset 30408704
dev_uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
backup_roots[4]:
backup 0:
backup_tree_root: 1871598567424 gen: 123917 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 30507008 gen: 123917 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951820050432
backup_num_devices: 1

backup 1:
backup_tree_root: 1871599681536 gen: 123914 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 30457856 gen: 123914 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951820050432
backup_num_devices: 1

backup 2:
backup_tree_root: 1871600844800 gen: 123915 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 30507008 gen: 123915 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951820050432
backup_num_devices: 1

backup 3:
backup_tree_root: 1871602515968 gen: 123916 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 30457856 gen: 123916 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951820050432
backup_num_devices: 1

[-- Attachment #4: super-2.txt --]
[-- Type: text/plain, Size: 3440 bytes --]

superblock: bytenr=274877906944, device=/dev/mapper/3tb_spinner
---------------------------------------------------------
csum_type 0 (crc32c)
csum_size 4
csum 0xfaea0d8b [match]
bytenr 274877906944
flags 0x1
( WRITTEN )
magic _BHRfS_M [match]
fsid 8cf4bcd8-76c7-4494-a2c8-2578136c0aa6
metadata_uuid 00000000-0000-0000-0000-000000000000
label
generation 123917
root 1871598567424
sys_array_size 129
chunk_root_generation 123187
root_level 0
chunk_root 28459008
chunk_root_level 1
log_root 0
log_root_transid (deprecated) 0
log_root_level 0
total_bytes 3000589352960
bytes_used 1951820050432
sectorsize 4096
nodesize 16384
leafsize (deprecated) 16384
stripesize 4096
root_dir 6
num_devices 1
compat_flags 0x0
compat_ro_flags 0xb
( FREE_SPACE_TREE |
FREE_SPACE_TREE_VALID |
BLOCK_GROUP_TREE )
incompat_flags 0x371
( MIXED_BACKREF |
COMPRESS_ZSTD |
BIG_METADATA |
EXTENDED_IREF |
SKINNY_METADATA |
NO_HOLES )
cache_generation 0
uuid_tree_generation 123887
dev_item.uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
dev_item.fsid 8cf4bcd8-76c7-4494-a2c8-2578136c0aa6 [match]
dev_item.type 0
dev_item.total_bytes 3000589352960
dev_item.bytes_used 1981078831104
dev_item.io_align 4096
dev_item.io_width 4096
dev_item.sector_size 4096
dev_item.devid 1
dev_item.dev_group 0
dev_item.seek_speed 0
dev_item.bandwidth 0
dev_item.generation 0
sys_chunk_array[2048]:
item 0 key (FIRST_CHUNK_TREE CHUNK_ITEM 22020096)
length 8388608 owner 2 stripe_len 65536 type SYSTEM|DUP
io_align 65536 io_width 65536 sector_size 4096
num_stripes 2 sub_stripes 1
stripe 0 devid 1 offset 22020096
dev_uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
stripe 1 devid 1 offset 30408704
dev_uuid ba0038ab-68b2-4646-8f30-7d40dbd1b4ab
backup_roots[4]:
backup 0:
backup_tree_root: 1871598567424 gen: 123917 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 30507008 gen: 123917 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951820050432
backup_num_devices: 1

backup 1:
backup_tree_root: 1871599681536 gen: 123914 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 30457856 gen: 123914 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951820050432
backup_num_devices: 1

backup 2:
backup_tree_root: 1871600844800 gen: 123915 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 30507008 gen: 123915 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951820050432
backup_num_devices: 1

backup 3:
backup_tree_root: 1871602515968 gen: 123916 level: 0
backup_chunk_root: 28459008 gen: 123187 level: 1
backup_extent_root: 30457856 gen: 123916 level: 2
backup_fs_root: 805961728 gen: 123887 level: 3
backup_dev_root: 146063360 gen: 123884 level: 1
csum_root: 806600704 gen: 123887 level: 2
backup_total_bytes: 3000589352960
backup_bytes_used: 1951820050432
backup_num_devices: 1