2.6.29 - raid10 issue - spares do not get synched automatically

2.6.29 - raid10 issue - spares do not get synched automatically

[Patrick Ringl]

Hello,

I am using 2.6.29 on a debian lenny. Anyway, I am having problems with a 
raid10 sandbox setup consisting of 4 loopback devices. To make a long 
story short:

I created a raid10 array with all these 4 devices (loop0-loop3)
then I set two devices faulty, removed them .. checked if I can still 
write to the array (which I could, so I/O errors, nothing in dmesg):

BUT - here it comes: once I am trying to (re)add them (I zero'ed the 
superblock before) they're only getting added as spares and the array 
will not sync.

Here's the logfile with mdadm -D /dev/md0 in between so one can see what 
is going on:

http://paste.debian.net/31758/


Any ideas?


PS: Please CC to me since I am not subscribed.


regards,
Patrick

2.6.29 - raid10 issue - spares do not get synched automatically

[Patrick Ringl]

Hello,

I am using 2.6.29 on a debian lenny. Anyway, I am having problems with a 
raid10 sandbox setup consisting of 4 loopback devices. To make a long 
story short:

I created a raid10 array with all these 4 devices (loop0-loop3)
then I set two devices faulty, removed them .. checked if I can still 
write to the array (which I could, so I/O errors, nothing in dmesg):

BUT - here it comes: once I am trying to (re)add them (I zero'ed the 
superblock before) they're only getting added as spares and the array 
will not sync.

Here's the logfile with mdadm -D /dev/md0 in between so one can see what 
is going on:

http://paste.debian.net/31758/


Any ideas?


PS: Please CC to me since I am not subscribed.


regards,
Patrick

Re: 2.6.29 - raid10 issue - spares do not get synched automatically

[NeilBrown]

On Sat, March 28, 2009 11:37 pm, Patrick Ringl wrote:
> Hello,
>
> I am using 2.6.29 on a debian lenny. Anyway, I am having problems with a
> raid10 sandbox setup consisting of 4 loopback devices. To make a long
> story short:
>
> I created a raid10 array with all these 4 devices (loop0-loop3)
> then I set two devices faulty, removed them .. checked if I can still
> write to the array (which I could, so I/O errors, nothing in dmesg):
>
> BUT - here it comes: once I am trying to (re)add them (I zero'ed the
> superblock before) they're only getting added as spares and the array
> will not sync.
>
> Here's the logfile with mdadm -D /dev/md0 in between so one can see what
> is going on:
>
> http://pa#ste.debian.net/31758/
>

Linux RAID10 with

     Raid Level : raid10
   Raid Devices : 4
         Layout : near=2, far=1

Will layout data like this:

   disk0   disk1   disk2   disk3
     A       A       B       B
     C       C       D       D
     E       E       F       F

You have removed disk0 and disk1, so A,C,E don't exist
any more.  You should get IO errors when you try to access
that data.
There is nothing that can be recovered, so the drives remain
as spares.

NeilBrown

Re: 2.6.29 - raid10 issue - spares do not get synched automatically

[Patrick Ringl]

Hello Neil,

NeilBrown wrote:
> Linux RAID10 with
>
>      Raid Level : raid10
>    Raid Devices : 4
>          Layout : near=2, far=1
>
> Will layout data like this:
>
>    disk0   disk1   disk2   disk3
>      A       A       B       B
>      C       C       D       D
>      E       E       F       F
>
> You have removed disk0 and disk1, so A,C,E don't exist
> any more.  You should get IO errors when you try to access
> that data.
>   
Sorry, but I removed loop0 and loop2 so that would be disk0 and disk2 - 
so ACE as well as BDF should still exist and recover. Could you explain 
this, I am I just getting you wrong?
> There is nothing that can be recovered, so the drives remain
> as spares.
>
> NeilBrown
>   
regards,
Patrick

Re: 2.6.29 - raid10 issue - spares do not get synched automatically

[Patrick Ringl]

Hello Neil,

NeilBrown wrote:
>
> Linux RAID10 with
>
>      Raid Level : raid10
>    Raid Devices : 4
>          Layout : near=2, far=1
>
> Will layout data like this:
>
>    disk0   disk1   disk2   disk3
>      A       A       B       B
>      C       C       D       D
>      E       E       F       F
>
> You have removed disk0 and disk1, so A,C,E don't exist
> any more.  You should get IO errors when you try to access
> that data.
> There is nothing that can be recovered, so the drives remain
> as spares.
>
> NeilBron
You're right - sorry. I better should read my own logfiles tho. Somehow 
removing a single disk and readding it mixed up with the raid device 
numbering - which I didnt notice at first.



regards,
Patrick

Re: 2.6.29 - raid10 issue - spares do not get synched automatically

[NeilBrown]

On Sun, March 29, 2009 9:21 am, Patrick Ringl wrote:
> Hello Neil,
>
> NeilBrown wrote:
>> Linux RAID10 with
>>
>>      Raid Level : raid10
>>    Raid Devices : 4
>>          Layout : near=2, far=1
>>
>> Will layout data like this:
>>
>>    disk0   disk1   disk2   disk3
>>      A       A       B       B
>>      C       C       D       D
>>      E       E       F       F
>>
>> You have removed disk0 and disk1, so A,C,E don't exist
>> any more.  You should get IO errors when you try to access
>> that data.
>>
> Sorry, but I removed loop0 and loop2 so that would be disk0 and disk2 -
> so ACE as well as BDF should still exist and recover. Could you explain
> this, I am I just getting you wrong?

From the Pastezone thing...
-----------------------------------------------------
    Number   Major   Minor   RaidDevice State
       0       7        2        0      active sync   /dev/loop2
       1       7        0        1      active sync   /dev/loop0
       2       7        1        2      active sync   /dev/loop1
       3       7        3        3      active sync   /dev/loop3
pari:/mnt/debian# mdadm --manage /dev/md0 -f /dev/loop2
mdadm: set /dev/loop2 faulty in /dev/md0
pari:/mnt/debian# mdadm --manage /dev/md0 -f /dev/loop0
mdadm: set /dev/loop0 faulty in /dev/md0
---------------------------------------------------

So while you did remove loop0 and loop2, these were disk1 and disk0.

NeilBrown

Re: 2.6.29 - raid10 issue - spares do not get synched automatically

[Dan Williams]

On Sat, Mar 28, 2009 at 4:55 AM, Patrick Ringl <patrick_@freenet.de> wrote:
> Hello,
>
> I am using 2.6.29 on a debian lenny. Anyway, I am having problems with a
> raid10 sandbox setup consisting of 4 loopback devices. To make a long story
> short:
>
> I created a raid10 array with all these 4 devices (loop0-loop3)
> then I set two devices faulty, removed them .. checked if I can still write
> to the array (which I could, so I/O errors, nothing in dmesg):
>
> BUT - here it comes: once I am trying to (re)add them (I zero'ed the
> superblock before) they're only getting added as spares and the array will
> not sync.
>
> Here's the logfile with mdadm -D /dev/md0 in between so one can see what is
> going on:
>
> http://paste.debian.net/31758/
>
>
> Any ideas?

You failed the whole array.  With 4 devices in the default layout the
array looks like stripe(mirror(slot0, slot1), mirror(slot2, slot3)).
In your case mdadm -D shows that you have the loop devices in the
following slots stripe(mirror(loop2, loop0), mirror(loop1, 3)).  A
raid10 is failed once all devices in a mirror set are failed, and a
failed array can't be rebuilt.

Regards,
Dan