HELP! New disks being dropped from RAID 6 array on every reboot

HELP! New disks being dropped from RAID 6 array on every reboot

[Joshua Johnson]

Greetings, long time listener, first time caller.

I recently replaced a disk in my existing 8 disk RAID 6 array.
Previously, all disks were PATA drives connected to the motherboard
IDE and 3 promise Ultra 100/133 controllers.  I replaced one of the
Promise controllers with a Via 64xx based controller, which has 2 SATA
ports and one PATA port.  I connected a new SATA drive to the new
card, partitioned the drive and added it to the array.  After 5 or 6
hours the resyncing process finished and the array showed up complete.
 Upon rebooting I discovered that the new drive had not been added to
the array when it was assembled on boot.   I resynced it and tried
again -- still would not persist after a reboot.  I moved one of the
existing PATA drives to the new controller (so I could have the slot
for network), rebooted and rebuilt the array.  Now when I reboot BOTH
disks are missing from the array (sda and sdb).  Upon examining the
disks it appears they think they are part of the array, but for some
reason they are not being added when the array is being assembled.
For example, this is a disk on the new controller which was not added
to the array after rebooting:

# mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 00.90.03
           UUID : 63ee7d14:a0ac6a6e:aef6fe14:50e047a5
  Creation Time : Thu Sep 21 23:52:19 2006
     Raid Level : raid6
    Device Size : 191157248 (182.30 GiB 195.75 GB)
     Array Size : 1146943488 (1093.81 GiB 1174.47 GB)
   Raid Devices : 8
  Total Devices : 8
Preferred Minor : 0

    Update Time : Fri Nov 23 10:22:57 2007
          State : clean
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 50df590e - correct
         Events : 0.96419878

     Chunk Size : 256K

      Number   Major   Minor   RaidDevice State
this     6       8        1        6      active sync   /dev/sda1

   0     0       3        2        0      active sync   /dev/hda2
   1     1      57        2        1      active sync   /dev/hdk2
   2     2      33        2        2      active sync   /dev/hde2
   3     3      34        2        3      active sync   /dev/hdg2
   4     4      22        2        4      active sync   /dev/hdc2
   5     5      56        2        5      active sync   /dev/hdi2
   6     6       8        1        6      active sync   /dev/sda1
   7     7       8       17        7      active sync   /dev/sdb1


Everything there seems to be correct and current up to the last
shutdown.  But the disk is not being added on boot.  Examining a disk
that is currently running in the array shows:

# mdadm --examine /dev/hdc2
/dev/hdc2:
          Magic : a92b4efc
        Version : 00.90.03
           UUID : 63ee7d14:a0ac6a6e:aef6fe14:50e047a5
  Creation Time : Thu Sep 21 23:52:19 2006
     Raid Level : raid6
    Device Size : 191157248 (182.30 GiB 195.75 GB)
     Array Size : 1146943488 (1093.81 GiB 1174.47 GB)
   Raid Devices : 8
  Total Devices : 6
Preferred Minor : 0

    Update Time : Fri Nov 23 10:23:52 2007
          State : clean
 Active Devices : 6
Working Devices : 6
 Failed Devices : 2
  Spare Devices : 0
       Checksum : 50df5934 - correct
         Events : 0.96419880

     Chunk Size : 256K

      Number   Major   Minor   RaidDevice State
this     4      22        2        4      active sync   /dev/hdc2

   0     0       3        2        0      active sync   /dev/hda2
   1     1      57        2        1      active sync   /dev/hdk2
   2     2      33        2        2      active sync   /dev/hde2
   3     3      34        2        3      active sync   /dev/hdg2
   4     4      22        2        4      active sync   /dev/hdc2
   5     5      56        2        5      active sync   /dev/hdi2
   6     6       0        0        6      faulty removed
   7     7       0        0        7      faulty removed


Here is my /etc/mdadm/mdadm.conf:

DEVICE partitions
PROGRAM /bin/echo
MAILADDR <redacted>
ARRAY /dev/md0 level=raid6 num-devices=8
UUID=63ee7d14:a0ac6a6e:aef6fe14:50e047a5


Can anyone see anything that is glaringly wrong here?  Has anybody
experienced similar behavior?  I am running Debian using kernel
2.6.23.8.  All partitions are set to type 0xFD and it appears the
superblocks on the sd* disks were written, why wouldn't they be added
to the array on boot?  Any help is greatly appreciated!

Re: HELP! New disks being dropped from RAID 6 array on every reboot

[BERTRAND Joël]

Joshua Johnson wrote:
> Greetings, long time listener, first time caller.
> 
> I recently replaced a disk in my existing 8 disk RAID 6 array.
> Previously, all disks were PATA drives connected to the motherboard
> IDE and 3 promise Ultra 100/133 controllers.  I replaced one of the
> Promise controllers with a Via 64xx based controller, which has 2 SATA
> ports and one PATA port.  I connected a new SATA drive to the new
> card, partitioned the drive and added it to the array.  After 5 or 6
> hours the resyncing process finished and the array showed up complete.
>  Upon rebooting I discovered that the new drive had not been added to
> the array when it was assembled on boot.   I resynced it and tried
> again -- still would not persist after a reboot.  I moved one of the
> existing PATA drives to the new controller (so I could have the slot
> for network), rebooted and rebuilt the array.  Now when I reboot BOTH
> disks are missing from the array (sda and sdb).  Upon examining the
> disks it appears they think they are part of the array, but for some
> reason they are not being added when the array is being assembled.
> For example, this is a disk on the new controller which was not added
> to the array after rebooting:

	What is your partition system ? When I have tried to created a raid6 
array over a SunOS partition type, I have seen this bug. Never on PC system.

	Regards,

	JKB

Re: HELP! New disks being dropped from RAID 6 array on every reboot

[Dan Williams]

On Nov 23, 2007 11:19 AM, Joshua Johnson <enigma2175@gmail.com> wrote:
> Greetings, long time listener, first time caller.
>
> I recently replaced a disk in my existing 8 disk RAID 6 array.
> Previously, all disks were PATA drives connected to the motherboard
> IDE and 3 promise Ultra 100/133 controllers.  I replaced one of the
> Promise controllers with a Via 64xx based controller, which has 2 SATA
> ports and one PATA port.  I connected a new SATA drive to the new
> card, partitioned the drive and added it to the array.  After 5 or 6
> hours the resyncing process finished and the array showed up complete.
>  Upon rebooting I discovered that the new drive had not been added to
> the array when it was assembled on boot.   I resynced it and tried
> again -- still would not persist after a reboot.  I moved one of the
> existing PATA drives to the new controller (so I could have the slot
> for network), rebooted and rebuilt the array.  Now when I reboot BOTH
> disks are missing from the array (sda and sdb).  Upon examining the
> disks it appears they think they are part of the array, but for some
> reason they are not being added when the array is being assembled.
> For example, this is a disk on the new controller which was not added
> to the array after rebooting:
>
> # mdadm --examine /dev/sda1
> /dev/sda1:
>           Magic : a92b4efc
>         Version : 00.90.03
>            UUID : 63ee7d14:a0ac6a6e:aef6fe14:50e047a5
>   Creation Time : Thu Sep 21 23:52:19 2006
>      Raid Level : raid6
>     Device Size : 191157248 (182.30 GiB 195.75 GB)
>      Array Size : 1146943488 (1093.81 GiB 1174.47 GB)
>    Raid Devices : 8
>   Total Devices : 8
> Preferred Minor : 0
>
>     Update Time : Fri Nov 23 10:22:57 2007
>           State : clean
>  Active Devices : 8
> Working Devices : 8
>  Failed Devices : 0
>   Spare Devices : 0
>        Checksum : 50df590e - correct
>          Events : 0.96419878
>
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     6       8        1        6      active sync   /dev/sda1
>
>    0     0       3        2        0      active sync   /dev/hda2
>    1     1      57        2        1      active sync   /dev/hdk2
>    2     2      33        2        2      active sync   /dev/hde2
>    3     3      34        2        3      active sync   /dev/hdg2
>    4     4      22        2        4      active sync   /dev/hdc2
>    5     5      56        2        5      active sync   /dev/hdi2
>    6     6       8        1        6      active sync   /dev/sda1
>    7     7       8       17        7      active sync   /dev/sdb1
>
>
> Everything there seems to be correct and current up to the last
> shutdown.  But the disk is not being added on boot.  Examining a disk
> that is currently running in the array shows:
>
> # mdadm --examine /dev/hdc2
> /dev/hdc2:
>           Magic : a92b4efc
>         Version : 00.90.03
>            UUID : 63ee7d14:a0ac6a6e:aef6fe14:50e047a5
>   Creation Time : Thu Sep 21 23:52:19 2006
>      Raid Level : raid6
>     Device Size : 191157248 (182.30 GiB 195.75 GB)
>      Array Size : 1146943488 (1093.81 GiB 1174.47 GB)
>    Raid Devices : 8
>   Total Devices : 6
> Preferred Minor : 0
>
>     Update Time : Fri Nov 23 10:23:52 2007
>           State : clean
>  Active Devices : 6
> Working Devices : 6
>  Failed Devices : 2
>   Spare Devices : 0
>        Checksum : 50df5934 - correct
>          Events : 0.96419880
>
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     4      22        2        4      active sync   /dev/hdc2
>
>    0     0       3        2        0      active sync   /dev/hda2
>    1     1      57        2        1      active sync   /dev/hdk2
>    2     2      33        2        2      active sync   /dev/hde2
>    3     3      34        2        3      active sync   /dev/hdg2
>    4     4      22        2        4      active sync   /dev/hdc2
>    5     5      56        2        5      active sync   /dev/hdi2
>    6     6       0        0        6      faulty removed
>    7     7       0        0        7      faulty removed
>
>
> Here is my /etc/mdadm/mdadm.conf:
>
> DEVICE partitions
> PROGRAM /bin/echo
> MAILADDR <redacted>
> ARRAY /dev/md0 level=raid6 num-devices=8
> UUID=63ee7d14:a0ac6a6e:aef6fe14:50e047a5
>
>
> Can anyone see anything that is glaringly wrong here?  Has anybody
> experienced similar behavior?  I am running Debian using kernel
> 2.6.23.8.  All partitions are set to type 0xFD and it appears the
> superblocks on the sd* disks were written, why wouldn't they be added
> to the array on boot?  Any help is greatly appreciated!

I wonder if you are running into a driver load order problem where the
ide driver and md are coming up before the sata driver.  You can let
userspace do the assembly after everything is up and running.  Specify
'raid=noautodetect' on the kernel command line and then let Debian's
'/etc/init.d/mdadm-raid' initscript take care of the assembly based on
your configuration file, or just run 'mdadm --assemble --scan' by
hand.

--
Dan

Re: HELP! New disks being dropped from RAID 6 array on every reboot

[Bill Davidsen]

Joshua Johnson wrote:
> Greetings, long time listener, first time caller.
>
> I recently replaced a disk in my existing 8 disk RAID 6 array.
> Previously, all disks were PATA drives connected to the motherboard
> IDE and 3 promise Ultra 100/133 controllers.  I replaced one of the
> Promise controllers with a Via 64xx based controller, which has 2 SATA
> ports and one PATA port.  I connected a new SATA drive to the new
> card, partitioned the drive and added it to the array.  After 5 or 6
> hours the resyncing process finished and the array showed up complete.
>  Upon rebooting I discovered that the new drive had not been added to
> the array when it was assembled on boot.   I resynced it and tried
> again -- still would not persist after a reboot.  I moved one of the
> existing PATA drives to the new controller (so I could have the slot
> for network), rebooted and rebuilt the array.  Now when I reboot BOTH
> disks are missing from the array (sda and sdb).  Upon examining the
> disks it appears they think they are part of the array, but for some
> reason they are not being added when the array is being assembled.
> For example, this is a disk on the new controller which was not added
> to the array after rebooting:
>
> # mdadm --examine /dev/sda1
> /dev/sda1:
>           Magic : a92b4efc
>         Version : 00.90.03
>            UUID : 63ee7d14:a0ac6a6e:aef6fe14:50e047a5
>   Creation Time : Thu Sep 21 23:52:19 2006
>      Raid Level : raid6
>     Device Size : 191157248 (182.30 GiB 195.75 GB)
>      Array Size : 1146943488 (1093.81 GiB 1174.47 GB)
>    Raid Devices : 8
>   Total Devices : 8
> Preferred Minor : 0
>
>     Update Time : Fri Nov 23 10:22:57 2007
>           State : clean
>  Active Devices : 8
> Working Devices : 8
>  Failed Devices : 0
>   Spare Devices : 0
>        Checksum : 50df590e - correct
>          Events : 0.96419878
>
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     6       8        1        6      active sync   /dev/sda1
>
>    0     0       3        2        0      active sync   /dev/hda2
>    1     1      57        2        1      active sync   /dev/hdk2
>    2     2      33        2        2      active sync   /dev/hde2
>    3     3      34        2        3      active sync   /dev/hdg2
>    4     4      22        2        4      active sync   /dev/hdc2
>    5     5      56        2        5      active sync   /dev/hdi2
>    6     6       8        1        6      active sync   /dev/sda1
>    7     7       8       17        7      active sync   /dev/sdb1
>
>
> Everything there seems to be correct and current up to the last
> shutdown.  But the disk is not being added on boot.  Examining a disk
> that is currently running in the array shows:
>
> # mdadm --examine /dev/hdc2
> /dev/hdc2:
>           Magic : a92b4efc
>         Version : 00.90.03
>            UUID : 63ee7d14:a0ac6a6e:aef6fe14:50e047a5
>   Creation Time : Thu Sep 21 23:52:19 2006
>      Raid Level : raid6
>     Device Size : 191157248 (182.30 GiB 195.75 GB)
>      Array Size : 1146943488 (1093.81 GiB 1174.47 GB)
>    Raid Devices : 8
>   Total Devices : 6
> Preferred Minor : 0
>
>     Update Time : Fri Nov 23 10:23:52 2007
>           State : clean
>  Active Devices : 6
> Working Devices : 6
>  Failed Devices : 2
>   Spare Devices : 0
>        Checksum : 50df5934 - correct
>          Events : 0.96419880
>
>      Chunk Size : 256K
>
>       Number   Major   Minor   RaidDevice State
> this     4      22        2        4      active sync   /dev/hdc2
>
>    0     0       3        2        0      active sync   /dev/hda2
>    1     1      57        2        1      active sync   /dev/hdk2
>    2     2      33        2        2      active sync   /dev/hde2
>    3     3      34        2        3      active sync   /dev/hdg2
>    4     4      22        2        4      active sync   /dev/hdc2
>    5     5      56        2        5      active sync   /dev/hdi2
>    6     6       0        0        6      faulty removed
>    7     7       0        0        7      faulty removed
>
>
> Here is my /etc/mdadm/mdadm.conf:
>
> DEVICE partitions
> PROGRAM /bin/echo
> MAILADDR <redacted>
> ARRAY /dev/md0 level=raid6 num-devices=8
> UUID=63ee7d14:a0ac6a6e:aef6fe14:50e047a5
>
>
> Can anyone see anything that is glaringly wrong here?  Has anybody
> experienced similar behavior?  I am running Debian using kernel
> 2.6.23.8.  All partitions are set to type 0xFD and it appears the
> superblocks on the sd* disks were written, why wouldn't they be added
> to the array on boot?  Any help is greatly appreciated!

Does that match what's in the init files used at boot? By any chance 
does the information there explicitly list partitions by name? If you 
change to "PARTITIONS" in /etc/mdadm.conf it won't bite you until you 
change the detected partitions so they no longer match what was correct 
at install time.

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979

Re: HELP! New disks being dropped from RAID 6 array on every reboot

[Joshua Johnson]

On Nov 24, 2007 12:20 PM, Bill Davidsen <davidsen@tmr.com> wrote:
>
> Does that match what's in the init files used at boot? By any chance
> does the information there explicitly list partitions by name? If you
> change to "PARTITIONS" in /etc/mdadm.conf it won't bite you until you
> change the detected partitions so they no longer match what was correct
> at install time.

According to the man page, using 'partitions' as your DEVICE should
cause mdadm to read /proc/partitions and scan all partitions listed
there.  The sda*/sdb* partitions were in /proc/partitions (at least
after the machine fully booted) but for some reason when mdadm
assembled the array it was not adding those partitions.  Changing the
DEVICE to '/dev/hd* /dev/sd*' rather than 'partitions' resolved the
issue.

Re: HELP! New disks being dropped from RAID 6 array on every reboot

[Joshua Johnson]

On Nov 24, 2007 9:27 PM, Bill Davidsen <davidsen@tmr.com> wrote:

>  No, I think you read that backward. using PARTITIONS is the right way to do
> it, but I was suggesting that the boot mdadm.conf in initrd was still using
> the old deleted partition names. And I assume that the old drives were
> either physically removed or you used the zero-superblock option to prevent
> the old partitions from being found and confusing the issue.

I doubt a second mdadm.conf is the problem (unless I misunderstand
something about the boot process), as I am not using a initrd on this
kernel.  One PATA drive was physically removed and one was moved to
the new controller.  The new SATA drive took the place of the removed
PATA drive in the array.

>  I assume you made the old partitions go away in one of these ways, so
> PARTITIONS should work, and from what you said I had the impression it did
> work after boot, which would fit having a non-functional mdadm config in
> initrd.
>
>  Any of that match what you're doing? I've never had to use the explicit
> partitions except when I forgot to zap the old superblocks.
>
>  --
> Bill Davidsen <davidsen@tmr.com>

I'm not sure exactly why the array wasn't being assembled with the sd*
disks but I suspect that the md driver was being loaded before the
scsi disk driver was done with the partition scan.  At any rate, using
the wildcarded device names resolved the issue and the server is
humming along happily.  Thanks for the help!