replacing failed disks in RAID-1 (kernel BUG)?

replacing failed disks in RAID-1 (kernel BUG)?

[Tomasz Chmielewski]

How do I replace failed disks in RAID-1 mode?

I followed the http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices, but it triggered a kernel BUG (using 2.6.30.1 kernel):


# mount -o degraded /dev/sdb4 /mnt/btrfs/

# btrfs-vol -r missing /mnt/btrfs/
removing missing devices from /mnt/btrfs/
ioctl returns -1

# dmesg -c
btrfs: unable to go below two devices on raid1

# btrfs-vol -a /dev/sda4 /mnt/btrfs/
Speicherzugriffsfehler

# dmesg
btrfs allocation failed flags 18, wanted 4096
space_info has 8384512 free, is not full
space_info total=8388608, pinned=0, delalloc=0, may_use=0, used=4096
block group 20971520 has 8388608 bytes, 4096 used 0 pinned 0 reserved
entry offset 20975616, bytes 8384512
1 blocks of free space at or bigger than bytes is
------------[ cut here ]------------
kernel BUG at fs/btrfs/extent-tree.c:2930!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
CPU 1
Modules linked in: btrfs zlib_deflate crc32c libcrc32c cpufreq_stats tun af_packet fglrx(P) ipv6 coretemp binfmt_misc loop ext3 jbd joydev dm_mod hid_multilaser usbhid cpufreq_ondemand cpufreq_conservative cpufreq_powersave acpi_cpufreq freq_table hid kvm_intel kvm snd_hda_codec_atihdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_seq_dummy snd_hwdep snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd pcspkr soundcore e1000e iTCO_wdt iTCO_vendor_support serio_raw snd_page_alloc i2c_i801 rtc_cmos i2c_core evdev thermal processor button wmi ehci_hcd uhci_hcd sr_mod sg usbcore ata_piix ahci libata sd_mod scsi_mod crc_t10dif raid1 ext4 jbd2 crc16
Pid: 23996, comm: btrfs-vol Tainted: P           2.6.30.1-desktop-1mnb #1
RIP: 0010:[<ffffffffa06a5426>]  [<ffffffffa06a5426>] __btrfs_reserve_extent+0x296/0x310 [btrfs]
RSP: 0018:ffff8800bb441808  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff88023cd9da2c RCX: 000000000001ffff
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff8800bb4418b8 R08: 000000000000ddd9 R09: 000000000000000a
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88023d644c68
R13: 0000000000001000 R14: ffff88023d644d30 R15: ffff88023d644d18
FS:  00007ffc25a67740(0000) GS:ffff88002f21b000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f0645383ea4 CR3: 00000000be85c000 CR4: 00000000000026e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process btrfs-vol (pid: 23996, threadinfo ffff8800bb440000, task ffff88013a1a0000)
Stack:
 0000000000000000 ffff8800bb441988 0000000000000000 0000000000000000
 0000000000000012 ffff88023fc016d0 ffff8800bb4418a8 ffff880149532000
 0000000000000000 ffff8800bb441988 ffffffffffffffff 0000000000000000
Call Trace:
 [<ffffffffa06a550e>] btrfs_alloc_extent+0x6e/0x100 [btrfs]
 [<ffffffffa06a5619>] btrfs_alloc_free_block+0x79/0xd0 [btrfs]
 [<ffffffffa0693779>] __btrfs_cow_block+0x199/0x880 [btrfs]
 [<ffffffffa0693f7d>] btrfs_cow_block+0x11d/0x250 [btrfs]
 [<ffffffffa069a9b6>] btrfs_search_slot+0x476/0x8c0 [btrfs]
 [<ffffffffa069b5bd>] btrfs_insert_empty_items+0x8d/0x100 [btrfs]
 [<ffffffffa0691718>] ? btrfs_alloc_path+0x28/0x50 [btrfs]
 [<ffffffffa06d8d29>] btrfs_add_device+0x89/0x210 [btrfs]
 [<ffffffff80346db4>] ? kill_bdev+0x44/0x70
 [<ffffffffa06ddc4a>] btrfs_init_new_device+0x65a/0xb20 [btrfs]
 [<ffffffffa06e0b8d>] ? btrfs_ioctl+0x3cd/0x910 [btrfs]
 [<ffffffffa06e0baf>] btrfs_ioctl+0x3ef/0x910 [btrfs]
 [<ffffffff80325680>] vfs_ioctl+0x30/0xd0
 [<ffffffff8034de53>] ? inotify_inode_queue_event+0xf3/0x130
 [<ffffffff80213ece>] ? apic_timer_interrupt+0xe/0x20
 [<ffffffff80325e5b>] do_vfs_ioctl+0x31b/0x5b0
 [<ffffffff8027cd6c>] ? up_read+0x1c/0x40
 [<ffffffff80326189>] sys_ioctl+0x99/0xc0
 [<ffffffff80213342>] system_call_fastpath+0x16/0x1b
Code: 4c 8b 63 58 49 81 ec b0 00 00 00 49 8b 84 24 b0 00 00 00 0f 18 08 49 8d 84 24 b0 00 00 00 49 39 c7 75 27 4c 89 f7 e8 2a 79 bd df <0f> 0b eb fe be 5c 0b 00 00 48 c7 c7 9b d2 6e a0 e8 05 5d bb df
RIP  [<ffffffffa06a5426>] __btrfs_reserve_extent+0x296/0x310 [btrfs]
 RSP <ffff8800bb441808>
---[ end trace a51370eaca1eb6d6 ]---


-- 
Tomasz Chmielewski
http://wpkg.org

Re: replacing failed disks in RAID-1 (kernel BUG)?

[David Woodhouse]

On Mon, 2009-07-13 at 12:28 +0200, Tomasz Chmielewski wrote:
> How do I replace failed disks in RAID-1 mode?

I don't think you can. In theory you can remove the broken one, and you
can add a _new_ empty one -- I say 'in theory' because you seem to have
demonstrated both of those actions failing.

But I don't believe we have yet implemented anything to let you
_replace_ a failed disk and recreate its original contents.

I had that on my TODO list for some time after I get the basic RAID[56]
operation working.

-- 
David Woodhouse                            Open Source Technology Centre
David.Woodhouse@intel.com                              Intel Corporation

Re: replacing failed disks in RAID-1 (kernel BUG)?

[Tomasz Chmielewski]

David Woodhouse wrote:
> On Mon, 2009-07-13 at 12:28 +0200, Tomasz Chmielewski wrote:
>> How do I replace failed disks in RAID-1 mode?
> 
> I don't think you can. In theory you can remove the broken one, and you
> can add a _new_ empty one -- I say 'in theory' because you seem to have
> demonstrated both of those actions failing.
> 
> But I don't believe we have yet implemented anything to let you
> _replace_ a failed disk and recreate its original contents.
> 
> I had that on my TODO list for some time after I get the basic RAID[56]
> operation working.

It would be also interesting to have a tool to monitor the state of the 
RAID (i.e. similar to what /proc/mdstat provides for md).


I also tried to compare what happens when we do writes to md-raid and to 
btrfs-raid (RAID-1 in both cases) and it looks... strange for btrfs. Or 
perhaps this is how RAID-1 works in btrfs?

I used iostat to monitor the writes on both devices.


With md RAID-1, when we do:

# dd if=/dev/zero of=/mnt/md-raid-1/testfile

and

# iostat -dk 1


We can see the write speed on both devices is more or less the same.


With btrfs RAID-1, when we do the same, I can see that writes go to one 
drive, while the second drive receives 0 kb/s writes; then it changes 
(one drive is written to, the second isn't). Only sometimes, writes 
happen concurrently to both drives, like with md RAID-1.

Is it intended?


-- 
Tomasz Chmielewski
http://wpkg.org

Re: replacing failed disks in RAID-1 (kernel BUG)?

[Roger Bumgarner]

I have no idea if this would work with btrfs, but the first thing I
would do is use 'dd' to clone one drive over to the new drive. I've
done that to repair real failed RAID1 arrays.

-rb

On Mon, Jul 13, 2009 at 3:28 AM, Tomasz Chmielewski<mangoo@wpkg.org> wr=
ote:
> How do I replace failed disks in RAID-1 mode?
>
> I followed the
> http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devi=
ces,
> but it triggered a kernel BUG (using 2.6.30.1 kernel):
>
>
> # mount -o degraded /dev/sdb4 /mnt/btrfs/
>
> # btrfs-vol -r missing /mnt/btrfs/
> removing missing devices from /mnt/btrfs/
> ioctl returns -1
>
> # dmesg -c
> btrfs: unable to go below two devices on raid1
>
> # btrfs-vol -a /dev/sda4 /mnt/btrfs/
> Speicherzugriffsfehler
>
> # dmesg
> btrfs allocation failed flags 18, wanted 4096
> space_info has 8384512 free, is not full
> space_info total=3D8388608, pinned=3D0, delalloc=3D0, may_use=3D0, us=
ed=3D4096
> block group 20971520 has 8388608 bytes, 4096 used 0 pinned 0 reserved
> entry offset 20975616, bytes 8384512
> 1 blocks of free space at or bigger than bytes is
> ------------[ cut here ]------------
> kernel BUG at fs/btrfs/extent-tree.c:2930!
> invalid opcode: 0000 [#1] SMP
> last sysfs file: /sys/devices/system/cpu/cpu7/cache/index2/shared_cpu=
_map
> CPU 1
> Modules linked in: btrfs zlib_deflate crc32c libcrc32c cpufreq_stats =
tun
> af_packet fglrx(P) ipv6 coretemp binfmt_misc loop ext3 jbd joydev dm_=
mod
> hid_multilaser usbhid cpufreq_ondemand cpufreq_conservative
> cpufreq_powersave acpi_cpufreq freq_table hid kvm_intel kvm
> snd_hda_codec_atihdmi snd_hda_codec_realtek snd_hda_intel snd_hda_cod=
ec
> snd_seq_dummy snd_hwdep snd_seq_oss snd_seq_midi_event snd_seq
> snd_seq_device snd_pcm_oss snd_pcm snd_timer snd_mixer_oss snd pcspkr
> soundcore e1000e iTCO_wdt iTCO_vendor_support serio_raw snd_page_allo=
c
> i2c_i801 rtc_cmos i2c_core evdev thermal processor button wmi ehci_hc=
d
> uhci_hcd sr_mod sg usbcore ata_piix ahci libata sd_mod scsi_mod crc_t=
10dif
> raid1 ext4 jbd2 crc16
> Pid: 23996, comm: btrfs-vol Tainted: P =A0 =A0 =A0 =A0 =A0 2.6.30.1-d=
esktop-1mnb #1
> RIP: 0010:[<ffffffffa06a5426>] =A0[<ffffffffa06a5426>]
> __btrfs_reserve_extent+0x296/0x310 [btrfs]
> RSP: 0018:ffff8800bb441808 =A0EFLAGS: 00010246
> RAX: 0000000000000000 RBX: ffff88023cd9da2c RCX: 000000000001ffff
> RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246
> RBP: ffff8800bb4418b8 R08: 000000000000ddd9 R09: 000000000000000a
> R10: 0000000000000000 R11: 0000000000000000 R12: ffff88023d644c68
> R13: 0000000000001000 R14: ffff88023d644d30 R15: ffff88023d644d18
> FS: =A000007ffc25a67740(0000) GS:ffff88002f21b000(0000) knlGS:0000000=
000000000
> CS: =A00010 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 00007f0645383ea4 CR3: 00000000be85c000 CR4: 00000000000026e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process btrfs-vol (pid: 23996, threadinfo ffff8800bb440000, task
> ffff88013a1a0000)
> Stack:
> 0000000000000000 ffff8800bb441988 0000000000000000 0000000000000000
> 0000000000000012 ffff88023fc016d0 ffff8800bb4418a8 ffff880149532000
> 0000000000000000 ffff8800bb441988 ffffffffffffffff 0000000000000000
> Call Trace:
> [<ffffffffa06a550e>] btrfs_alloc_extent+0x6e/0x100 [btrfs]
> [<ffffffffa06a5619>] btrfs_alloc_free_block+0x79/0xd0 [btrfs]
> [<ffffffffa0693779>] __btrfs_cow_block+0x199/0x880 [btrfs]
> [<ffffffffa0693f7d>] btrfs_cow_block+0x11d/0x250 [btrfs]
> [<ffffffffa069a9b6>] btrfs_search_slot+0x476/0x8c0 [btrfs]
> [<ffffffffa069b5bd>] btrfs_insert_empty_items+0x8d/0x100 [btrfs]
> [<ffffffffa0691718>] ? btrfs_alloc_path+0x28/0x50 [btrfs]
> [<ffffffffa06d8d29>] btrfs_add_device+0x89/0x210 [btrfs]
> [<ffffffff80346db4>] ? kill_bdev+0x44/0x70
> [<ffffffffa06ddc4a>] btrfs_init_new_device+0x65a/0xb20 [btrfs]
> [<ffffffffa06e0b8d>] ? btrfs_ioctl+0x3cd/0x910 [btrfs]
> [<ffffffffa06e0baf>] btrfs_ioctl+0x3ef/0x910 [btrfs]
> [<ffffffff80325680>] vfs_ioctl+0x30/0xd0
> [<ffffffff8034de53>] ? inotify_inode_queue_event+0xf3/0x130
> [<ffffffff80213ece>] ? apic_timer_interrupt+0xe/0x20
> [<ffffffff80325e5b>] do_vfs_ioctl+0x31b/0x5b0
> [<ffffffff8027cd6c>] ? up_read+0x1c/0x40
> [<ffffffff80326189>] sys_ioctl+0x99/0xc0
> [<ffffffff80213342>] system_call_fastpath+0x16/0x1b
> Code: 4c 8b 63 58 49 81 ec b0 00 00 00 49 8b 84 24 b0 00 00 00 0f 18 =
08 49
> 8d 84 24 b0 00 00 00 49 39 c7 75 27 4c 89 f7 e8 2a 79 bd df <0f> 0b e=
b fe be
> 5c 0b 00 00 48 c7 c7 9b d2 6e a0 e8 05 5d bb df
> RIP =A0[<ffffffffa06a5426>] __btrfs_reserve_extent+0x296/0x310 [btrfs=
]
> RSP <ffff8800bb441808>
> ---[ end trace a51370eaca1eb6d6 ]---
>
>
> --
> Tomasz Chmielewski
> http://wpkg.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at =A0http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: replacing failed disks in RAID-1 (kernel BUG)?

[Tomasz Chmielewski]

Roger Bumgarner wrote:
> I have no idea if this would work with btrfs, but the first thing I
> would do is use 'dd' to clone one drive over to the new drive. I've
> done that to repair real failed RAID1 arrays.

I don't think it's a good idea.
Well, it could be a temporary workaround (_if_ it works) for btrfs until 
it stabilizes.

But certainly not for md RAID-1. _If_ it did work for you, you would 
have to take the whole array down as dd works, which is not good. md 
provides its own tools (mdadm) to replace/rebuild the RAID.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: replacing failed disks in RAID-1 (kernel BUG)?

[Chris Mason]

On Mon, Jul 13, 2009 at 12:28:09PM +0200, Tomasz Chmielewski wrote:
> How do I replace failed disks in RAID-1 mode?
>
> I followed the http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices, but it triggered a kernel BUG (using 2.6.30.1 kernel):
>
>
> # mount -o degraded /dev/sdb4 /mnt/btrfs/
>
> # btrfs-vol -r missing /mnt/btrfs/
> removing missing devices from /mnt/btrfs/
> ioctl returns -1
>
> # dmesg -c
> btrfs: unable to go below two devices on raid1
>
> # btrfs-vol -a /dev/sda4 /mnt/btrfs/
> Speicherzugriffsfehler

So, this should have worked.  I'll dig into why the btrfs-vol -a failed.

-chris

Re: replacing failed disks in RAID-1 (kernel BUG)?

[Tomasz Chmielewski]

Chris Mason wrote:
> On Mon, Jul 13, 2009 at 12:28:09PM +0200, Tomasz Chmielewski wrote:
>> How do I replace failed disks in RAID-1 mode?
>>
>> I followed the http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices, but it triggered a kernel BUG (using 2.6.30.1 kernel):
>>
>>
>> # mount -o degraded /dev/sdb4 /mnt/btrfs/
>>
>> # btrfs-vol -r missing /mnt/btrfs/
>> removing missing devices from /mnt/btrfs/
>> ioctl returns -1
>>
>> # dmesg -c
>> btrfs: unable to go below two devices on raid1
>>
>> # btrfs-vol -a /dev/sda4 /mnt/btrfs/
>> Speicherzugriffsfehler
> 
> So, this should have worked.  I'll dig into why the btrfs-vol -a failed.

Mind that I was using 2.6.30.1 kernel, so the code was not very fresh.


-- 
Tomasz Chmielewski
http://wpkg.org