Re: Rebuilding 24TB Raid5 array (was btrfs corruption: parent transid verify failed + open_ctree failed)

[Josef Bacik <josef@toxicpanda.com>]

On Thu, Mar 31, 2022 at 10:19:27AM -0700, Marc MERLIN wrote:
> I'm going to wait a few more days before I give up and restore from
> backup and will consider whether if I should go back to ext4 which for
> me has clearly been more resilient of disk misbehaviours like I get
> sometimes (of course, nothing will recover from a real double risk
> failure in raid5, but over the last 20 years, it's not been uncommon for
> me to see more than one drive kicked out of an array due to SCSI or sata
> issues, or a drive having a weird shutdown and being able to come back
> after a power cycle with all data except the last blocks written to it).
> 
> If I go back with btrfs (despite it being non resilient to any problem
> described above, btrfs send is great for backups of course), what are
> today's best recommended practices?
> 
> Kernel will be 5.16. Filesystem will be 24TB and contain mostly bigger
> files (100MB to 10GB).
> 
> 1) mdadm --create /dev/md7 --level=5 --consistency-policy=ppl --raid-devices=5 /dev/sd[abdef]1 --chunk=256 --bitmap=internal
> 2) echo 0fb96f02-d8da-45ce-aba7-070a1a8420e3 >  /sys/block/bcache64/bcache/attach 
>    gargamel:/dev# cat /sys/block/md7/bcache/cache_mode
>    [writethrough] writeback writearound none
> 3) cryptsetup luksFormat --align-payload=2048 -s 256 -c aes-xts-plain64  /dev/bcache64
> 4) cryptsetup luksOpen /dev/bcache64 dshelf1
> 5) mkfs.btrfs -m dup -L dshelf1 /dev/mapper/dshelf1
> 
> Any other btrfs options I should set for format to improve reliability
> first and performance second?
> I'm told I should use space_cache=v2, is it default now with btrfs-progs 5.10.1-2 ?
> 
> Anything else you recommend given that I indeed I have layers in the
> middle. I'm actually considering dropping bcache for this filesystem
> given that it could be another cause for corruption that btrfs can't
> deal with.
>

Sorry shit went real wrong this week.  Can you do

btrfs inspect-internal dump-super -f /dev/whatever

This is going to spit out the backup roots for everything.  I was looking at the
code and realized the backup root code doesn't actually work for the chunk root,
since it's special.  Can you then do

btrfs check --chunk-root <bytenr> /dev/whatever

from whatever the newest backup root is, and then keep rolling back until you
find one that works.  If you find one that does work hooray, I can write
something to swap out the chunk bytenr in your super block and we can carry on.

If not we can go with

btrfs rescue chunk-recover /dev/whatever

and see how that works out.  Thanks,

Josef