Howto mount intact left-over device of a degraded btrfs-raid?

Howto mount intact left-over device of a degraded btrfs-raid?

[Mr. Tux]

Hi list

I tried to mount the intact device of a two memer btrfs raid array with one 
device missing:

# mount -o degraded /dev/sdb /mnt

this wouldn't work with kernel 2.6.32-2 - caused some sort of memory segfault

# btrfs-vol -r missing /mnt

complained and refused to mount the array degraded with only one intact array 
member left -> demanded two intact members

How can I mount this array anyway?

thanks!

Re: Howto mount intact left-over device of a degraded btrfs-raid?

[Josef Bacik]

On Wed, Jan 20, 2010 at 07:31:24PM +0100, Mr. Tux wrote:
> Hi list
> 
> I tried to mount the intact device of a two memer btrfs raid array with one 
> device missing:
> 
> # mount -o degraded /dev/sdb /mnt
> 
> this wouldn't work with kernel 2.6.32-2 - caused some sort of memory segfault
> 
> # btrfs-vol -r missing /mnt
> 
> complained and refused to mount the array degraded with only one intact array 
> member left -> demanded two intact members
> 
> How can I mount this array anyway?
> 

How did you setup this array to begin with?  I'm trying to reproduce this bug
but I haven't been able to.  Thanks,

Josef

Re: Howto mount intact left-over device of a degraded btrfs-raid?

[Mr. Tux]

On Monday 25 January 2010 19.43:27 Josef Bacik wrote:
>
> How did you setup this array to begin with?  I'm trying to reproduce this
> bug but I haven't been able to.  Thanks,
>

Hi Josef,

Thanks for your help!

Linux Kernel 2.6.32.2 from kernel.org (was marked as stable at that time,  I 
compiled it using make-kpkg with btrfs support inside the kernel -> not 
modular)

OS: Debian Lenny 5.03

I created the btrfs raid on a two sata-II drives like this:

mkfs.btrfs -d raid1 -m raid 1 /dev/sda4 /dev/sdb4

Then I mounted the btrfs with:

mount /dev/sda4 /mnt/

cat /proc/mounts and dmesg showed the raid was mounted successfully

Then I started to copy large amount of data into the raid1 from a file server. 
With the copying process ongoing I detached the data cable from one device to 
simulate hardware failure. Copying process went on as planed to the drive 
still attached.

Then I unmounted the btrfs raid like this:

umount /mnt/

and I shut down the system.

Now the goal was to boot and mount only the intact btrfs partition again as a 
degraded raid.

I followed the multiple-device-wiki of btrfs [1]

I didn't try to remove the faulty device - I shut down the system already and 
booted again without the faulty drive. So I tried this:

mount -o degraded /dev/sda /mnt/

This caused a memory segfault - the system still ran, but I couldn't use the 
array any more.

So I used the next command

btrfs-vol -r missing /mnt

Complained it couldn't mount with only one member present.

My assumption: Bad luck with the kernel - or did this memory segfault happen 
on earlier kernel versions as well?

[1] 
http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices#Replacing_Failed_Devices

Re: Howto mount intact left-over device of a degraded btrfs-raid?

[Josef Bacik]

On Mon, Jan 25, 2010 at 10:46:47PM +0100, Mr. Tux wrote:
> On Monday 25 January 2010 19.43:27 Josef Bacik wrote:
> >
> > How did you setup this array to begin with?  I'm trying to reproduce this
> > bug but I haven't been able to.  Thanks,
> >
> 
> Hi Josef,
> 
> Thanks for your help!
> 
> Linux Kernel 2.6.32.2 from kernel.org (was marked as stable at that time,  I 
> compiled it using make-kpkg with btrfs support inside the kernel -> not 
> modular)
> 
> OS: Debian Lenny 5.03
> 
> I created the btrfs raid on a two sata-II drives like this:
> 
> mkfs.btrfs -d raid1 -m raid 1 /dev/sda4 /dev/sdb4
> 
> Then I mounted the btrfs with:
> 
> mount /dev/sda4 /mnt/
> 
> cat /proc/mounts and dmesg showed the raid was mounted successfully
> 
> Then I started to copy large amount of data into the raid1 from a file server. 
> With the copying process ongoing I detached the data cable from one device to 
> simulate hardware failure. Copying process went on as planed to the drive 
> still attached.
> 
> Then I unmounted the btrfs raid like this:
> 
> umount /mnt/
> 
> and I shut down the system.
> 
> Now the goal was to boot and mount only the intact btrfs partition again as a 
> degraded raid.
> 
> I followed the multiple-device-wiki of btrfs [1]
> 
> I didn't try to remove the faulty device - I shut down the system already and 
> booted again without the faulty drive. So I tried this:
> 
> mount -o degraded /dev/sda /mnt/
> 
> This caused a memory segfault - the system still ran, but I couldn't use the 
> array any more.
> 
> So I used the next command
> 
> btrfs-vol -r missing /mnt
> 
> Complained it couldn't mount with only one member present.
> 
> My assumption: Bad luck with the kernel - or did this memory segfault happen 
> on earlier kernel versions as well?
> 
> [1] 
> http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices#Replacing_Failed_Devices
> 

Hmm well I'm trying to reproduce that here and it's working fine, course I can't
pull a cable on my disk since I only have one disk to test with.  Would you mind
sending me your dmesg after you try

mount -o degraded /dev/sda1 /mnt/

so I can see whats going on.  Thanks,

Josef

Re: Howto mount intact left-over device of a degraded btrfs-raid?

[Mr. Tux]

On Monday 25 January 2010 23:08:39 Josef Bacik wrote:

Hi Josef

>
> Hmm well I'm trying to reproduce that here and it's working fine, course I
> can't pull a cable on my disk since I only have one disk to test with. 
> Would you mind sending me your dmesg after you try
>
> mount -o degraded /dev/sda1 /mnt/
>
> so I can see whats going on.  Thanks,
>

Did you try 2.6.32.2 from kernel.org? Is it possible to use  a second test 
drive like I did? I know you can create a raid array with one device missing, 
but the segfault happened with a two member raid1.

I don't have the test environment any more, so I can't offer a backtrace right 
now - but I will set it up again in the next few days. If it happens again I 
will contact you and send you a backtrace.

Mr. Tux

Re: Howto mount intact left-over device of a degraded btrfs-raid?

[Josef Bacik]

On Tue, Jan 26, 2010 at 01:20:30PM +0100, Mr. Tux wrote:
> On Monday 25 January 2010 23:08:39 Josef Bacik wrote:
> 
> Hi Josef
> 
> >
> > Hmm well I'm trying to reproduce that here and it's working fine, course I
> > can't pull a cable on my disk since I only have one disk to test with. 
> > Would you mind sending me your dmesg after you try
> >
> > mount -o degraded /dev/sda1 /mnt/
> >
> > so I can see whats going on.  Thanks,
> >
> 
> Did you try 2.6.32.2 from kernel.org? Is it possible to use  a second test 
> drive like I did? I know you can create a raid array with one device missing, 
> but the segfault happened with a two member raid1.
> 
> I don't have the test environment any more, so I can't offer a backtrace right 
> now - but I will set it up again in the next few days. If it happens again I 
> will contact you and send you a backtrace.
>

Ok I think I've tracked down all the various problems that you were having with
this.  I've sent 3 patches to the list just now

Btrfs: check total number of devices when removing missing
Btrfs: check return value of open_bdev_exclusive properly
Btrfs: do not mark the chunk as readonly if in degraded mode

Make sure you run btrfsctl -a before running mount -o degraded.  I made that
mistake and things still didnt work.  Let me know if you have any problems.
Thanks,

Josef