--examine contradicts --create and --detail, again

--examine contradicts --create and --detail, again

[whollygoat]

I am on my 4th or 5th try at getting a raid5 array 
set up.  I did have one working quite nicely with 6 
(5 active, 1 spare) 40GB drives but have had a 
succession of failures since trying to grow the array
after putting bigger drives in the box.

Either I am misunderstanding something, or I have
a hardware issue I don't know how to diagnose.  Any 
suggestions on things to read or to try are welcome.

I've been told that zeroing the superblock is sufficient
but having had so many failures I wanted to be thourough


# dd if=/dev/zero of=/dev/hd{egikmo} (for brevity, each was
run separately)

# rm -R /dev/md0 /dev/md

# fdisk /dev/hd{egikmo}
 - creating a new, primary, partition starting cylinder 1, 80G
 - type da (non-fs data)
 
#mdadm -C /dev/md/0 -e 1.0 -v -l 5 -b internal -a yes\
  -n 5 /dev/hde1 /dev/hdg1 /dev/hdi1 /dev/hdk1 /dev/hdm1\
  -x 1 /dev/hdo1  --name=FlyFileServ_md

After waiting for the initial syncing of the array to complete

# mdadm -D /dev/md/0
/dev/md/0:
        Version : 01.00.03
  Creation Time : Tue Feb 10 14:45:39 2009
     Raid Level : raid5
     Array Size : 312501760 (298.02 GiB 320.00 GB)
    Device Size : 156250880 (74.51 GiB 80.00 GB)
   Raid Devices : 5
  Total Devices : 6
Preferred Minor : 0
    Persistence : Superblock is persistent

  Intent Bitmap : Internal
   
    Update Time : Tue Feb 10 16:41:48 2009
          State : active
 Active Devices : 5
Working Devices : 6
 Failed Devices : 0
  Spare Devices : 1
           
         Layout : left-symmetric
     Chunk Size : 64K
           
           Name : fly:FlyFileServ_md  (local to host fly)
           UUID : 684bf5f1:de2c0d2a:5a5ac88f:de7cf2d3
         Events : 2
    
    Number   Major   Minor   RaidDevice State
       0      33        1        0      active sync   /dev/hde1
       1      34        1        1      active sync   /dev/hdg1
       2      56        1        2      active sync   /dev/hdi1
       3      57        1        3      active sync   /dev/hdk1
       6      88        1        4      active sync   /dev/hdm1
         
       5      89        1        -      spare   /dev/hdo1

So, then, why oh why oh why does --examine, on any of the 
component devices, show slots for  7 devices, one failed, 
one empty?  I have recently changed RAM, mobo, system disk
and some IDE cables thinking each time that I had finally
come to the end of this.



# mdadm -E /dev/hde1
/dev/hde1:
          Magic : a92b4efc
        Version : 01
    Feature Map : 0x1
     Array UUID : 684bf5f1:de2c0d2a:5a5ac88f:de7cf2d3
           Name : fly:FlyFileServ_md  (local to host fly)
  Creation Time : Tue Feb 10 14:45:39 2009
     Raid Level : raid5
   Raid Devices : 5
     
    Device Size : 156250880 (74.51 GiB 80.00 GB)
     Array Size : 625003520 (298.02 GiB 320.00 GB)
   Super Offset : 156251008 sectors
          State : clean
    Device UUID : 88b0d67e:3e2cf8ee:83f58286:4040c5da

Internal Bitmap : 2 sectors from superblock
    Update Time : Tue Feb 10 16:41:48 2009
       Checksum : dae7896e - correct
         Events : 2

         Layout : left-symmetric
     Chunk Size : 64K

    Array Slot : 0 (0, 1, 2, 3, failed, empty, 4)
   Array State : Uuuuu 1 failed

Sorry for bringing up this new instance of the same old 
problem yet one more time.
-- 
  
  whollygoat@letterboxes.org

-- 
http://www.fastmail.fm - A no graphics, no pop-ups email service

Re: --examine contradicts --create and --detail, again

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2162 bytes --]

On Wed Feb 11, 2009 at 02:48:09AM -0800, whollygoat@letterboxes.org wrote:

> Either I am misunderstanding something, or I have
> a hardware issue I don't know how to diagnose.  Any 
> suggestions on things to read or to try are welcome.
> 
> # mdadm -D /dev/md/0
>     
>     Number   Major   Minor   RaidDevice State
>        0      33        1        0      active sync   /dev/hde1
>        1      34        1        1      active sync   /dev/hdg1
>        2      56        1        2      active sync   /dev/hdi1
>        3      57        1        3      active sync   /dev/hdk1
>        6      88        1        4      active sync   /dev/hdm1
>          
>        5      89        1        -      spare   /dev/hdo1
> 
> So, then, why oh why oh why does --examine, on any of the 
> component devices, show slots for  7 devices, one failed, 
> one empty?  I have recently changed RAM, mobo, system disk
> and some IDE cables thinking each time that I had finally
> come to the end of this.
> 
> # mdadm -E /dev/hde1
> 
>     Array Slot : 0 (0, 1, 2, 3, failed, empty, 4)
>    Array State : Uuuuu 1 failed
> 
> Sorry for bringing up this new instance of the same old 
> problem yet one more time.
>
I'd say this is just an artifact of how the array creation works,
Because it starts as a degraded array, initially one of the RAID devices
is flagged as missing/faulty (so you have member numbers 0, 1, 2 and 3
in the array, 4 flagged as missing/faulty, and 5 & 6 as spares).  When
it does the initial build, it uses member number 6, leaving 5 as the
spare.  This leaves you with 1 "failed" member and one "empty" (the
spare - it's not part of the array so doesn't have a Device Number).

Unfortunately I don't have any RAID5 arrays around now to check whether
I see the same here.  However, if the system's working fine otherwise
then I certainly wouldn't worry about it.

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: --examine contradicts --create and --detail, again

[whollygoat]

On Wed, 11 Feb 2009 13:59:17 +0000, "Robin Hill" <robin@robinhill.me.uk>
said:
> On Wed Feb 11, 2009 at 02:48:09AM -0800, whollygoat@letterboxes.org
> wrote:
> 
[snip]

> > # mdadm -E /dev/hde1
> > 
> >     Array Slot : 0 (0, 1, 2, 3, failed, empty, 4)
> >    Array State : Uuuuu 1 failed
> > 
> > Sorry for bringing up this new instance of the same old 
> > problem yet one more time.
> >
> I'd say this is just an artifact of how the array creation works,
> Because it starts as a degraded array, initially one of the RAID devices
> is flagged as missing/faulty (so you have member numbers 0, 1, 2 and 3
> in the array, 4 flagged as missing/faulty, and 5 & 6 as spares).  When
> it does the initial build, it uses member number 6, leaving 5 as the
> spare.  This leaves you with 1 "failed" member and one "empty" (the
> spare - it's not part of the array so doesn't have a Device Number).
> 
> Unfortunately I don't have any RAID5 arrays around now to check whether
> I see the same here.  However, if the system's working fine otherwise
> then I certainly wouldn't worry about it.

Thanks for the reply.  Unfortunately, I can't claim to be enjoying a
finely working system.  Since your reply, I've tried unplugging the ide
expansion cards from the pci but, and playing around with the ide
cables.

I've managed to pull off one create run where the end result showed no
"failed" or "empty" slots for --examine.  Unfortunately, I have not been
able to reproduce that "clean" run, even with the same card removed.  So
I 
am guessing that it is either an ide cable or the sole ide card I have
not
been able to remove (kernel oopses when this card is not plugged into
the 
bus).

Ta,

wg
some bad h/w somewhere.
-- 
  
  whollygoat@letterboxes.org

-- 
http://www.fastmail.fm - The way an email service should be