Re: dev extent physical offset [...] on devid 1 doesn't have corresponding chunk

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2024/12/15 07:30, Qu Wenruo 写道:
>
>
> 在 2024/12/15 04:09, Ben Millwood 写道:
>> On Sat, 14 Dec 2024 at 02:51, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>> Both kernel and btrfs-progs should go with metadata COW with transaction
>>> protection, so even something went wrong (power loss or Ctrl-C) we
>>> should only see the previous transaction, thus everything should be
>>> fine.
>>
>> Thanks for the reassurance, that is what I'd hoped would be true :)
>>
>>> 在 2024/12/14 12:47, Ben Millwood 写道:
>>>> While I'm waiting for the lowmem check to progress, are there any
>>>> other useful recovery / diagnosis steps I could try?
>>>
>>> If you do not want to waste too long time on btrfs check, please dump
>>> the device tree and chunk tree:
>>>
>>> # btrfs ins dump-tree -t chunk <device>
>>> # btrfs ins dump-tree -t dev <device>
>>>
>>> That's all the info we need to cross-check the result.
>>>
>>> Although `btrfs check --readonly --mode=lowmem` would be the best, as it
>>> will save me a lot of time to either manually verify the output or craft
>>> a script to do that.
>>
>> Well, the check is still going:
>>
>> root@vigilance:~# btrfs check --progress --mode lowmem /dev/
>> masterchef-vg/btrfs
>> Opening filesystem to check...
>> Checking filesystem on /dev/masterchef-vg/btrfs
>> UUID: a0ed3709-1490-4f2d-96b5-bb1fb22f0b45
>> [1/7] checking root items                      (0:46:43 elapsed,
>> 68928137 items checked)
>> [2/7] checking extents                         (14:31:49 elapsed,
>> 239591 items checked)
>>
>> I'll let it continue. In the meantime I'll e-mail you the trees you
>> asked for off-thread: they don't obviously look like they contain
>> private information, but I'd like to minimise the exposure anyway.
>> (Feel free to send them to other btrfs devs.)
>
> Those trees are completely anonymous, the only information that contains
> are:
>
> - How large your fs is
> - How many bytes and their ranges are allocated
> - The type of the allocated chunks
>
> So it should be very safe to share, unless you have some very
> confidential info hidden in the device size :)
>
> [...]
>>>
>>> That's all the info we need to cross-check the result.
>>>
>>> Although `btrfs check --readonly --mode=lowmem` would be the best, as it
>>> will save me a lot of time to either manually verify the output or craft
>>> a script to do that.
>>>
>>> My current assumption is a bitflip at runtime, but no proof yet.
>
> Unfortunately it doesn't look like this.
>
> I scanned the last several dev-extents and chunks, it turns out that
> it's very possible `btrfs clear-space-cache` is causing something wrong:
>
> - The offending dev-extent have chunk_tree_uuid set
>    This is not the kernel behavior, but progs specific one.
>    This means there are two chunks allocated during
>    `btrfs-progs clear-space-cache`, but one is missing.
>
> - One of chunk allocated by btrfs-progs is totally fine
>    And it's still in the chunk tree
>
> - The other (the offending one) points to a chunk that's beyond
>    the last known chunk
>
> So I guess either:
>
> - The btrfs-progs has a bug in the chunk creation code
>    So that a chunk and its dev-extent are not created in the same
>    transaction, and Ctrl-C breaks it, causing an orphan dev-extent
>
> - The btrfs-progs has a bug in the chunk deletion code
>    Similar but in the empty chunk cleanup code.
>
> Anyway I'll need to dig deeper to fix the bug.

Unfortunately I failed to find why the chunk is removed but free dev
extent is left.

Both kernel and progs are doing the proper chunk removal inside one
transaction to remove both the chunk item and dev-extent item, from the
very beginning.

The same is for the chunk allocation part.

So unless there is something totally wrong, I didn't see why progs or
kernel can cause such mismatch.

Do you still remember if there is any error message for the
clear-space-cache interruption and the next RW mount of it?

Thanks,
Qu
>
> Meanwhile I have created a branch for you to manually fix the bug:
> https://github.com/adam900710/btrfs-progs/tree/orphan_dev_extent_cleanup
>
> Since the lowmem is still running, you can prepare an environment to
> build btrfs-progs, so after the lowmem check finished, you can use that
> branch to delete the offending item by:
>
> # ./btrfs-corrupt-block -X <device>
>
> Thanks,
> Qu
>
>>>
>>> Thanks,
>>> Qu
>>>
>>>> smartctl appears
>>>> not to work with this disk, so I can't easily say whether the disk is
>>>> or is not healthy.
>>>>
>>>
>
>