Re: BTRFS corruption after conversion to block group tree

Qu Wenruo
2023-12-02T04:25:09Z

On 2023/12/2 14:43, Qu Wenruo wrote:
>
>
> On 2023/12/2 13:15, Russell Haley wrote:
>> On 12/1/23 14:09, Qu Wenruo wrote:
>>>
>>>
>>> On 2023/12/1 22:16, Russell Haley wrote:
>>>> Distro: Fedora 39
>>>> kernel: 6.5.12
>>>> btrfs-progs: v6.5.1.
>>>>
>>>> To put everyone at ease, the data seems mostly okay. The disk is
>>>> mountable with ro,rescue=ingorebadroots:nologreplay, and I'm pulling
>>>> off
>>>> what I can.  Furthermore, there's a good chance this is the result of a
>>>> marginally-unstable overclock. However, the symptoms are very
>>>> similar to
>>>> a recent report from Alexander Duscheleit [1], so there may be a deeper
>>>> problem. I can provide requested metadata dumps if it would be useful,
>>>> and I also have a few questions.
>>>>
>>>> A few days ago I used btrfstune to convert two btrfs filesystems to
>>>> block-group-tree, after reading that it would reduce mount time. After
>>>> about 10 hours, while I was using Steam, the more heavily-used of the
>>>> two disks, which holds (among other things) the Steam game library,
>>>> experienced a transaction abort and went read-only.
>>>
>>> Any dmesg of that transaction abort? That's the most critical info here.
>>
>> [80343.530191] BTRFS error (device dm-1): parent transid verify failed
>> on logical 31850496 mirror 1 wanted 123883 found 123907
>> [80343.587776] BTRFS error (device dm-1): parent transid verify failed
>> on logical 31850496 mirror 2 wanted 123883 found 123907
>> [80343.587836] BTRFS error (device dm-1): failed to run delayed ref for
>> logical 916815872 num_bytes 16384 type 176 action 1 ref_mod 1: -5
>
> This is weird, the corruption looks like to be inside extent tree, not
> the new block group tree, but it still shows the same meaning, metadata
> COW is already broken at that time.
>
> Anyway considering there are already two reports here, I'm going to add
> mandatory btrfs checks before and after the conversion just to be extra
> safe.
>
>> [80343.587844] BTRFS error (device dm-1: state A): Transaction aborted
>> (error -5)
>> [80343.587846] BTRFS: error (device dm-1: state A) in
>> btrfs_run_delayed_refs:2182: errno=-5 IO failure
>>
>>>> 2. There's something extremely weird with the primary superblock.
>>>> "btrfs
>>>>      inspect-internal dump-super" says that superblock 0 doesn't
>>>> have the
>>>>      BLOCK_GROUP_TREE or NO_HOLES flags set, unlike superblocks 1
>>>> and 2,
>>>>      which are identical in every field except csum ("[match]", for
>>>> 0, 1,
>>>>      and 2) and bytenr.
>>>
>>> This is not good at all, if csum of super blocks doesn't match, there
>>> must be something especially wrong.
>>
>> Comparing to other btrfs disks I have access to, different checksum for
>> the backup superblocks seems normal?
>> It does say [match] after the
>> checksum value. Just the value itself is different.
>
> Oh, as long as there is the "[match]" suffix, it's fine.
>
> Backup and primary csums are different due to the bytenr differences,
> thus those super blocks are indeed different in their contents.
>
>>
>> I dumped the 3 superblocks with btrfs inspect-internal dump-super -f and
>> attached them to this message. "Meld" for 3-way diff worked well for me.
>
> Indeed the backup super blocks are newer than the primary, which is not
> correct at all, and several generations apart.
>
> It can be a sign or bad write order/cache.
>
> If you want to be extra safe, I'd recommend to disable the write cache
> for those involved disks in the future.
> (Or it can be something with LUKS? I'm not sure, but one more layer just
> means one more point of failure)
>
> Unless you're using nobarrier mount option, there should never be a case
> where the primary super blocks are written before the primary one, not
> to mention a tree block is already overwritten before it's corresponding
> super block.
>
> I'd say for now, the only good way to continue is mount with
> rescue=all,ro and grab any important data.
>
> Converting back to old extent tree is not possible, since the fs is
> already broken.

Just one more question, is there any hibernation/suspension involved in
this particular corruption?

I have seen exactly such unexplainable writes-from-future cases, which a
lot of them have hibernation/suspension involved.
(Thus personally speaking, I never go hibernation/suspension on my own
devices)

Thanks,
Qu
>
> Thanks,
> Qu
>>
>> Thanks for your help,
>>
>>    Russell Haley
>