Re: btrfs crashes during routine btrfs-balance-least-used

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

在 2024/7/15 01:43, Kai Krakow 写道:
> Hello btrfs list!
>
> (also reported in irc)
>
> Our btrfs pool crashed during a routine btrfs-balance-least-used.
> Maybe of interest: bees is also running on this filesystem, snapper
> takes hourly snapshots with retention policy.
>
> I'm currently still collecting diagnostics, "btrfs check" log is
> already 3 GB and growing.
>
> The btrfs runs on three devices vd{c,e,f}1 with data=single meta=raid1.
>
> Here's an excerpt from dmesg (full log https://gist.tnonline.net/TE):

Unfortunately the full log is not really full.

There should be extent leaf dump, and after that dump, showing the
reason why we believe it's a problem.

Is there any true full dmesg dump?

But overall, most of the errors inside __btrfs_free_extent() would be
extent tree corruption.

> [...]
>
> "btrfs check" can only run in lowmem mode, it will crash with "out of
> memory" (the system has 74G of RAM). Here's the beginning of the log:
>
> [1/7] checking root items
> [2/7] checking extents
> ERROR: shared extent 15929577472 referencer lost (parent: 1147747794944)

I believe that's the cause, some extent tree corruption.

> ERROR: shared extent 15929577472 referencer lost (parent: 1148095201280)
> ERROR: shared extent 15929577472 referencer lost (parent: 1175758274560)
> (repeating thousands of similar lines)
>
> Last gist: https://gist.tnonline.net/Z4 (meanwhile, this log is over
> 3GB, I can upload it somewhere later).
>
> We have backups (daily backups stored inside borg on a remote host).
>
> Is there anything we can do? Restoring from backup will probably take
> more than 24h (3 TB). The system runs web and mail hosts for more than
> 100 customers.
>
> We did not try to run "btrfs check --repair" yet, nor
> "--init-extent-tree". I'd rather try a quick repair before restoring.
> But OTOH, I don't want to make it worse and waste time by trying.

Considering the size of the metadata, I do not believe --repair nor
--init-extent-tree is going to fully fix the problem.

>
> Unfortunately, the btrfs has been mounted rw again after unmounting
> following the incident. This restarted the balance, and it seems it
> changed the first error "btrfs check" found. I'll try
> "ro,skip-balance" after btrfs-check finished. I think the file-system
> is still fully readable and we can take one last backup.
>
> Also, I happily provide the logs collected if a dev wanted to look into it.

I guess there is no real full dmesg of that incident?

The corrupted extent leaf has 260 items, but the dump only contains 36,
nor the final reason line.

The other thing is, does the server has ECC memory?
It's not uncommon to see bitflips causing various problems (almost
monthly reports).

If the machine doesn't have ECC memory, then a memtest would be preferable.

Thanks,
Qu
>
>
> Thanks in advance
> Kai
>