Re: Tried to replace a drive in a raid 1 and all hell broke loose

[efkf@firemail.cc]

thanks a lot for reading into this

On 2022-05-24 20:11, Chris Murphy wrote:
> I suggest mounting with "mount -o ro,rescue=all"
Thanks a lot, with this command I was able to mount the filesystem again 
and retrieve a lot more data!
Only a very small percentage of the files (that i could check) were 
corrupt.

> Do you have a complete dmesg that shows boot, mount, and the kernel
> errors while copying?
If you mean the copying of the data mentioned on my update email then 
yes, its the attached file on this one.


> From one of your attached files:
> 
>> Total devices 2 FS bytes used 772.76GiB
>> devid    2 size 1.82TiB used 334.00GiB path 
>> /dev/mapper/ST2000DL003-###############
>> devid    3 size 1.82TiB used 661.00GiB path 
>> /dev/mapper/ST3000VN007-###############
> 
> This doesn't list a 3rd device so it suggests it's a 2x device raid1. 
> However:

I had (i think) successfully removed the failing drive with devid 1

> 
>> #btrfs fi df /mnt/sd/
>> Data, RAID1: total=772.00GiB, used=771.22GiB
>> Data, single: total=1.00GiB, used=2.25MiB
>> System, RAID1: total=32.00MiB, used=96.00KiB
>> System, single: total=32.00MiB, used=48.00KiB
>> Metadata, RAID1: total=3.00GiB, used=1.54GiB
>> Metadata, single: total=1.00GiB, used=0.00B
> 
> This is not good. Some of the data and some of the metadata
> (specifically system profile which is the chunk tree) is only
> available on one drive
I had that issue with single chunks, run the command to make it all 
raid1, run a scrub some checks messed around a tiny bit, most likely 
mounted with -o degraded in the process and they appeared again ( on 
linux 5.17.0 ).


> Some older kernels would create single
> profile chunks when a raid1 file system was mounted in degraded,rw
> mode with a missing device. This happens silently.

I had mounted the fs with -o degraded and one drive a couple of times 
just as a sanity check to make sure the data really is in both drives, i 
assume this would mount it rw and fall into the category you described. 
The first chunks were created before having updated to debian testing 
under kernel 5.10.0-11 but the same thing happened after updating to 
testing
I think i had tried to mount ro,degraded and it failed but i'm not sure.

Once again thank you so much for suggesting -o rescue=all, i had 
previously managed to recover around 300G and now i think i got the full 
~800G with a very small amount of corrupted files !!