Re: BTRFS corruption after conversion to block group tree

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2023/12/2 13:15, Russell Haley wrote:
> On 12/1/23 14:09, Qu Wenruo wrote:
>>
>>
>> On 2023/12/1 22:16, Russell Haley wrote:
>>> Distro: Fedora 39
>>> kernel: 6.5.12
>>> btrfs-progs: v6.5.1.
>>>
>>> To put everyone at ease, the data seems mostly okay. The disk is
>>> mountable with ro,rescue=ingorebadroots:nologreplay, and I'm pulling off
>>> what I can.  Furthermore, there's a good chance this is the result of a
>>> marginally-unstable overclock. However, the symptoms are very similar to
>>> a recent report from Alexander Duscheleit [1], so there may be a deeper
>>> problem. I can provide requested metadata dumps if it would be useful,
>>> and I also have a few questions.
>>>
>>> A few days ago I used btrfstune to convert two btrfs filesystems to
>>> block-group-tree, after reading that it would reduce mount time. After
>>> about 10 hours, while I was using Steam, the more heavily-used of the
>>> two disks, which holds (among other things) the Steam game library,
>>> experienced a transaction abort and went read-only.
>>
>> Any dmesg of that transaction abort? That's the most critical info here.
>
> [80343.530191] BTRFS error (device dm-1): parent transid verify failed
> on logical 31850496 mirror 1 wanted 123883 found 123907
> [80343.587776] BTRFS error (device dm-1): parent transid verify failed
> on logical 31850496 mirror 2 wanted 123883 found 123907
> [80343.587836] BTRFS error (device dm-1): failed to run delayed ref for
> logical 916815872 num_bytes 16384 type 176 action 1 ref_mod 1: -5

This is weird, the corruption looks like to be inside extent tree, not
the new block group tree, but it still shows the same meaning, metadata
COW is already broken at that time.

Anyway considering there are already two reports here, I'm going to add
mandatory btrfs checks before and after the conversion just to be extra
safe.

> [80343.587844] BTRFS error (device dm-1: state A): Transaction aborted
> (error -5)
> [80343.587846] BTRFS: error (device dm-1: state A) in
> btrfs_run_delayed_refs:2182: errno=-5 IO failure
>
>>> 2. There's something extremely weird with the primary superblock. "btrfs
>>>      inspect-internal dump-super" says that superblock 0 doesn't have the
>>>      BLOCK_GROUP_TREE or NO_HOLES flags set, unlike superblocks 1 and 2,
>>>      which are identical in every field except csum ("[match]", for 0, 1,
>>>      and 2) and bytenr.
>>
>> This is not good at all, if csum of super blocks doesn't match, there
>> must be something especially wrong.
>
> Comparing to other btrfs disks I have access to, different checksum for
> the backup superblocks seems normal?
> It does say [match] after the
> checksum value. Just the value itself is different.

Oh, as long as there is the "[match]" suffix, it's fine.

Backup and primary csums are different due to the bytenr differences,
thus those super blocks are indeed different in their contents.

>
> I dumped the 3 superblocks with btrfs inspect-internal dump-super -f and
> attached them to this message. "Meld" for 3-way diff worked well for me.

Indeed the backup super blocks are newer than the primary, which is not
correct at all, and several generations apart.

It can be a sign or bad write order/cache.

If you want to be extra safe, I'd recommend to disable the write cache
for those involved disks in the future.
(Or it can be something with LUKS? I'm not sure, but one more layer just
means one more point of failure)

Unless you're using nobarrier mount option, there should never be a case
where the primary super blocks are written before the primary one, not
to mention a tree block is already overwritten before it's corresponding
super block.

I'd say for now, the only good way to continue is mount with
rescue=all,ro and grab any important data.

Converting back to old extent tree is not possible, since the fs is
already broken.

Thanks,
Qu
>
> Thanks for your help,
>
>    Russell Haley