Raid 5 on 2.5.50/2.5.51, dirty array, kernel panic

Raid 5 on 2.5.50/2.5.51, dirty array, kernel panic

[Eurijk]

I set up an IDE raid array as such:
  /dev/md1 hda1 hdc1 hde1 hdg1 RAID 1
  /dev/md0 hda2 hdc2 hde2 hdg2 RAID 5

Installed slack, set up lilo, and waited for rebuilding to finish.
Now to simulate a failure I shut it down and physically unhooked
(**boom hellfire fury**! :D) /dev/hda. From everything I understand,
the system should work just fine still.

Bios detects hdc is still there, and auto boots off that. Kernel loads,
good lilo did it's job :D, but...:

...
md: created md0
...
md: running: <hdg2><hde2><hdc2>
md: md0: raid array is not clean -- starting background reconstuction
...
raid5: device hdg2 operational as raid disk 3
raid5: device hde2 operational as raid disk 2
raid5: device hdc2 operational as raid disk 1
raid5: cannot start dirty degraded array for md0
RAID5 conf printout:
 --- rd:4 wd:3 fd:1
 disk 1, o:1, dev:hdc2
 disk 2, o:1, dev:hde2
 disk 3, o:1, dev:hdg2
raid5: failed to run raid set md0
md: pers->run() failed ...
md :do_md_run() returned -22
md: md0 still in use.
...
raid1: raid set md1 active with 3 out of 4 mirrors
...
md: ... autorun DONE.
Unable to handle kernel NULL pointer dereference at virtual address 00000088
 printing eip:
c0289977
(stack/register dump is here :D)
...
 <0>Kernel panic: Attempted to kill init!

The "cannot start dirty degraded array" was added in 2.5.34 I believe. I can't
test a pre 2.5.34 since my 20271 was added in 2.5.37. >;-) I can test it using
good ol' hda/hdb/hdc/hdd if it would be of some help.

Let me know if I'm off my rocker!

Thanks

-eurijk!

Re: Raid 5 on 2.5.50/2.5.51, dirty array, kernel panic

[Steven Dake]

Eurijk
Please run the dump through ksymoops -m System.map.  Without this, its 
quite difficult to tell where the failure occured.

Thanks
-steve

Eurijk wrote:

>I set up an IDE raid array as such:
>  /dev/md1 hda1 hdc1 hde1 hdg1 RAID 1
>  /dev/md0 hda2 hdc2 hde2 hdg2 RAID 5
>
>Installed slack, set up lilo, and waited for rebuilding to finish.
>Now to simulate a failure I shut it down and physically unhooked
>(**boom hellfire fury**! :D) /dev/hda. From everything I understand,
>the system should work just fine still.
>
>Bios detects hdc is still there, and auto boots off that. Kernel loads,
>good lilo did it's job :D, but...:
>
>...
>md: created md0
>...
>md: running: <hdg2><hde2><hdc2>
>md: md0: raid array is not clean -- starting background reconstuction
>...
>raid5: device hdg2 operational as raid disk 3
>raid5: device hde2 operational as raid disk 2
>raid5: device hdc2 operational as raid disk 1
>raid5: cannot start dirty degraded array for md0
>RAID5 conf printout:
> --- rd:4 wd:3 fd:1
> disk 1, o:1, dev:hdc2
> disk 2, o:1, dev:hde2
> disk 3, o:1, dev:hdg2
>raid5: failed to run raid set md0
>md: pers->run() failed ...
>md :do_md_run() returned -22
>md: md0 still in use.
>...
>raid1: raid set md1 active with 3 out of 4 mirrors
>...
>md: ... autorun DONE.
>Unable to handle kernel NULL pointer dereference at virtual address 00000088
> printing eip:
>c0289977
>(stack/register dump is here :D)
>...
> <0>Kernel panic: Attempted to kill init!
>
>The "cannot start dirty degraded array" was added in 2.5.34 I believe. I can't
>test a pre 2.5.34 since my 20271 was added in 2.5.37. >;-) I can test it using
>good ol' hda/hdb/hdc/hdd if it would be of some help.
>
>Let me know if I'm off my rocker!
>
>Thanks
>
>-eurijk!
>-
>To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
>  
>

Re: Raid 5 on 2.5.50/2.5.51, dirty array, kernel panic

[Eurijk]

> Please run the dump through ksymoops -m System.map.  Without this, its 
> quite difficult to tell where the failure occured.

I appear unable to link ksymoops :(, perhaps I can do something similar by hand... (see below)

Let me know, Thanks

-eurijk!

=== BREAK ===

eip c0289977              => c0289960 t make_request
EIP:	0060:[<c0289977>] => c0289880 t raid5_unplug_device

Call Trace Function Reference:
[<c014c90b>] c014c8a0 t bvec_alloc
[<c0249adc>] c02499c0 T generic_make_request
[<c0249b9d>] c0249b60 T submit_bio
[<c0149359>] c0149300 t __bread_slow
[<c01495c8>] c0149590 T __bread

[<c017d9d3>] c017d920 t ext2_fill_super
[<c01deeef>] c01deed0 T sprintf
[<c014dce5>] c014dcc0 T sb_set_blocksize
[<c014d613>] c014d400 T get_sb_bdev
[<c0160eec>] c0160e50 T alloc_vfsmnt
[<c017e46f>] c017e440 t ext2_get_sb

[<c017d920>] c017d920 t ext2_fill_super
[<c014d86f>] c014d810 T do_kern_mount
[<c0162125>] c0162090 t do_add_mount
[<c0162440>] c01622e0 T do_mount
[<c01e0ffc>] c01e0fb0 T copy_from_user
[<c0162828>] c0162760 T sys_mount

[<c010524b>] c01051e0 T prepare_namespace
[<c010507f>] c0105040 t init
[<c0105040>] c0105040 t init
[<c0107249>] c0107244 t kernel_thread_helper

=== BREAK ===

Unable to handle kernel NULL pointer dereference at virtual address 00000088 printing eip:
c0289977
*pde = 00000000
Oops: 0000
CPU:    0
EIP:    0060:[<c0289977>]    Not tainted
EFLAGS: 00010292
eax: f7d3a200   ebx: f7cfd660   ecx: c19e9680   edx: 00000000
esi: 00000002   edi: 00000000   ebp: f7fb09e0   esp: f7f9fd58
ds: 0068   es: 0068   ss: 0068
Process swapper (pid: 1, threadinfo=f7f9e000 task=f7f9c040)
Stack: f7f9fd54 f7f9fd78 00000000 00000000 f7fb09e0 00000000 00000000 00000010
       c014c90b f7ffee40 00000010 f7cfd660 00000002 f7d3a2a8 f7fb09e0 c0249adc
       f7d3a2a8 f7fb09e0 00000010 00000001 f7f9fdac 00000000 f7ff87f0 00000000
Call trace: [<c014c90b>] [<c0249adc>] [<c0249b9d>] [<c0149359>] [<c01495c8>]
[<c017d9d3>] [<c01deeef>] [<c014dce5>] [<c014d613>] [<c0160eec>] [<c017e46f>]
[<c017d920>] [<c014d86f>] [<c0162125>] [<c0162440>] [<c01e0ffc>] [<c0162828>]
[<c010524b>] [<c010507f>] [<c0105040>] [<c0107249>]
Code: 8b 87 88 00 00 00 89 44 24 20 48 89 44 24 1c 8b 55 00 8b 45
 <0>Kernel panic: Attempted to kill init!

Re: Raid 5 on 2.5.50/2.5.51, dirty array, kernel panic

[Eurijk]

>I set up an IDE raid array as such:
>  /dev/md1 hda1 hdc1 hde1 hdg1 RAID 1 (ext2)
>  /dev/md0 hda2 hdc2 hde2 hdg2 RAID 5 (reiserfs)
>... and physically remove a drive (hda) ...
> <0>Kernel panic: Attempted to kill init!

^^^ The above also applies if you put hda back in after
it has been marked bad by it not being there after the
goto abort (see below) has been removed.

This appears to be stemming from a 3.5.33/34? patch that included:

   if (mddev->degraded == 1 &&
	    !(mddev->state & (1<<MD_SB_CLEAN))) {
      printk(KERN_ERR "raid5: cannot start dirty degraded array for md%d\n", mdidx(mddev));
      goto abort;
   }

If you remove the goto abort; there this fixes the problem.

I imagine this needs to check to see if only one device is bad,
and if so, continue.

I would have expected an "unable to mount root" error rather then a kernel
panic so there must be something wrong (probably not very critical) elsewhere.

Can someone more familiar with linux-2.5.51/drivers/md/raid5.c and kernel
modules take a look at this please? Thanks!

-eurijk!