Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

[Zygo Blaxell <ce3g8jdj@umail.furryterror.org>]

On Tue, Apr 05, 2022 at 05:08:44PM -0700, Marc MERLIN wrote:
> On Tue, Apr 05, 2022 at 07:51:29PM -0400, Zygo Blaxell wrote:
> > Other possibilites include irreproducible reads coming from the bcache
> > layer if that's still active (e.g. if it was on mdadm raid1, and the
> 
> bcache is still active, correct.
> mdadm5 array is running with 4 drives out of 5.
> 
> I'm not sure if I full understand the tree root issue we've been having.

Based on the history, I'd expect the filesystem is missing some number
of tree nodes, from a few dozen to thousands, depending on how many
writes were dropped after the 2nd drive failure before it was detected.
Since the array was also degraded at that time, with 4 drives in raid5,
there's 3 data drives, and if one of them was offline then we'd have a 2/3
success rate reading metadata blocks and 1/3 garbage.  That's definitely
in the "we need to write new software to recover from this" territory.

Normally, I'd expect that once we dig through a few layers of simple
dropped write blocks, we'll start hitting metadata pages with bad csums
and trashed contents, since the parity blocks will be garbage in the raid5
stripes where the writes were lost.  One important data point against
this theory is that we have not seen a csum failure yet, so maybe this
is a different (possibly better) scenario.  Possibly some of the lost
writes on the raid5 are still stored in the bcache, so there's few or no
garbage blocks (though reading the array through the cache might evict
the last copy of usable data and make some damage permanent--you might
want to make a backup copy of the cache device).

Backup roots only work if writes are dropped only in the most recent
transaction, maybe two, because only these trees are guaranteed to be
intact on disk.  After that, previously occupied pages are fair game
for new write allocations, and old metadata will be lost.  Unlike other
filesystems, btrfs never writes metadata in the same place twice, so when
a write is dropped, there isn't an old copy of the data still available at
the location of the dropped write--that location contains some completely
unrelated piece of the metadata tree whose current version now lives
at some other location.  Later tree updates will overwrite old copies
of the updated page, destroying the data in the affected page forever.
Essentially there will be a set of metadata pages where you have two
versions of different ages, and another set of metadata pages where you
have zero versions, and (hopefully) most of the other pages are intact.

> We know from the dump that we have these backups
> backup_tree_root:       13577814573056  gen: 1602089    level: 1
> backup_tree_root:       13577775284224  gen: 1602086    level: 1
> backup_tree_root:       13577821667328  gen: 1602087    level: 1
> backup_tree_root:       13577779511296  gen: 1602088    level: 1
> 
> but they didn't seem to work, or are we sure that none of them don't?

The root pointers are just pointers--having a pointer to a tree root isn't
going to help if the trees were never written at the referenced location.
Recursively, if you have a tree node but you don't have the next-level
nodes the previous-level node points to, then the root isn't useful
because large parts of its tree are unreachable.  If you're missing a
leaf node then whatever data was in the leaf is gone.  Once you've found
a root node, you can try walking over the entire tree to see if it's all
there (e.g. with btrfs ins dump-tree -t N), but if anything's missing
in any part of the tree, then that tree root can't be used as-is.

> Then we did a custom scan, but it found one of the 4 backup_tree_root we
> already had, so that doesn't help, correct?
> Found tree root at 13577814573056 gen 1602089 level 1
> 
> Was the expectation that there are more backup_tree_root hiding
> somewhere and we've been trying to find them?

If we have a superblock, the chunk tree, and a subvol tree, we can
drop all the other trees and rebuild them (bonus points if the csum
tree survived, then we can verify all the data was recovered correctly;
otherwise, we can read all the files and make a new csum tree but it won't
detect any data corruption that might have happened in degraded mode).
This is roughly what 'btrfs check --init-extent-tree' does (though due
to implementation details it has a few extra dependencies that might
get in the way) and you can find subvol roots with btrfs-find-root.

If we don't have any intact subvol trees (or the subvol trees we really
want aren't intact), then we can't recover this way.  Instead we'd have
to scrape the disk looking for metadata leaf nodes, and try to reinsert
those into a new tree structure.  The trick here is that we'll have the
duplicated and inconsitent nodes and we won't have some nodes at all,
and we'll have to make sense of those (or pass it to the existing btrfs
check and hope it can cope with them).  I'm guessing that a simplified
version of this is what Josef is building at this point, or will be
building soon if we aren't extremely lucky and find an intact subvol tree.
After building an intact subvol tree (even with a few garbage items in it
as long as check can handle them) we can go back to the --init-extent-tree
step and rebuild the rest of the filesystem.

> Thanks,
> Marc
> -- 
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>  
> Home page: http://marc.merlins.org/  
>