Trying to mount RAID1 degraded with removed disk -> open_ctree failed

Trying to mount RAID1 degraded with removed disk -> open_ctree failed

[Dirk Lutzebaeck]

Hi,

I have setup a RAID1 using 3 devices (500G each) on separate disks. 
After removing one disk physically the filesystem cannot be mounted in 
degraded nor in recovery mode. When putting the disk back in the 
filesystem can be mounted without errors. I did a cold-swap (powercycle 
after removal/insertion of the disk).

Here are the details:

- latest kernel 3.2.1 and btrfs-tools on xubuntu 11.10

# uname -a
Linux sol2 3.2.1-030201-generic #201201121644 SMP Thu Jan 12 21:45:24 
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

# btrfs device scan

# btrfs fi show btrfs-r1b
Label: 'btrfs-r1b'  uuid: 11ac3be1-62ce-4b1c-8069-63942f1354bd
         Total devices 3 FS bytes used 206.60GB
         devid    3 size 465.66GB used 140.00GB path /dev/sdc1
         devid    2 size 465.66GB used 139.02GB path /dev/sda2
         *** Some devices missing

# mount -o degraded /r1b
mount: /dev/sdc1: can't read superblock
[ 2156.393337] device label btrfs-r1b devid 3 transid 2441 /dev/sdc1
[ 2156.407816] btrfs: allowing degraded mounts
[ 2156.770745] parent transid verify failed on 448316575744 wanted 2427 
found 2425
[ 2156.770762] parent transid verify failed on 448316575744 wanted 2427 
found 2425
[ 2156.770769] parent transid verify failed on 448316575744 wanted 2427 
found 2425
[ 2156.770774] parent transid verify failed on 448316575744 wanted 2427 
found 2425
[ 2156.784722] btrfs warning page private not zero on page 448229711872
[ 2156.784842] btrfs: open_ctree failed


# mount -o recovery /r1b
[ 2286.282041] device label btrfs-r1b devid 3 transid 2441 /dev/sdc1
[ 2286.282664] btrfs: enabling auto recovery
[ 2286.284709] btrfs: failed to read chunk tree on sda2
[ 2286.285045] btrfs: open_ctree failed


# find-root /dev/sda2
warning devid 1 not found already
Failed to read: Illegal seek
warning, device 1 is missing
warning, device 1 is missing
warning, device 1 is missing
Super think's the tree root is at 448316571648, chunk root 20983808


# btrfsck /dev/sda2
warning devid 1 not found already
parent transid verify failed on 448316575744 wanted 2427 found 2425
parent transid verify failed on 448316575744 wanted 2427 found 2425
Ignoring transid failure
Check tree block failed, want=448316588032, have=11529940786129179172
Check tree block failed, want=448316588032, have=11529940786129179172
Check tree block failed, want=448316588032, have=11529940786129179172
read block failed check_tree_block


What is happening? RAID1 should be mountable degraded with one 
missing/removed device.


-Dirk

Re: Trying to mount RAID1 degraded with removed disk -> open_ctree failed

[Duncan]

Dirk Lutzebaeck posted on Sun, 22 Jan 2012 16:05:14 +0100 as excerpted:

> I have setup a RAID1 using 3 devices (500G each) on separate disks.
> After removing one disk physically the filesystem cannot be mounted in
> degraded nor in recovery mode.

> - latest kernel 3.2.1 and btrfs-tools on xubuntu 11.10

> What is happening? RAID1 should be mountable degraded with one
> missing/removed device.

Note that I'm only researching btrfs for my own systems at this point and 
am not using it yet.  However, because I *AM* researching it and already 
read thru most of the wiki documentation, it's fresh in mind.

Here's what the wiki says, tho of course it could be outdated:

https://btrfs.wiki.kernel.org/

>From the multiple devices page:

>> By default, metadata will be mirrored across two devices and data will
>> be striped across all of the devices present.

Question:  Did you specify -m raid1 -d raid1 when you did the mkfs.btrfs?

While the -m raid1 would be the default given multiple devices, the -d 
raid1 is not.  If you didn't specify -d raid1, you'll have raid0/striped 
data with only the metadata being raid1/mirrored, thus explaining the 
problem.


At least with all devices present, the following should show the raid 
level actually used (from the use cases page):

>> On a 2.6.37 or later kernel, use
>> 
>> btrfs fi df /mountpoint
>> 
>> The required support was broken accidentally in earlier kernels,
>> but has now been fixed.

Also note since you're running a 3-device btrfs-raid-1, tho it shouldn't 
affect a single device dropout, from the sysadmin guide page (near the 
bottom of the raid and data replication section):

>> With RAID-1 and RAID-10, only two copies of each byte of data are
>> written, regardless of how many block devices are actually in use
>> on the filesystem.

IOW, unlike standard or kernel/md raid-1, that 3-device btrfs-raid-1 will 
**NOT** protect you if two of the three devices go bad before you've had 
a chance to bring in and balance to a replacement for the first bad 
device.

As I said I'm just now researching my own btrfs upgrade, and don't know 
for sure whether that's true or not, but if it is, it's a HUGE negative 
for me, as I'm currently running 4-way kernel/md RAID-1 on an aging set 
of drives, and was hoping to upgrade to btrfs raid-1 for the checksummed 
integrity.  But given the age of the drives I really don't want to drop 
below dual redundancy (3 copies), and this two-copies-only (single 
redundancy) raid-1(-ish) no matter the number of devices, is 
disappointing indeed!

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman