mdadm and raid5 hot-rebuild problem on 2.5.67

mdadm and raid5 hot-rebuild problem on 2.5.67

[Stef Telford]

or rather not a 'rebuild'. 

The original problem was that one of the raid5 disks was
dead on waking up this morning, so, i used mdadm to set
it to fail, and then i set it removed. since it was software
raid5, i shut down the machine (sort of crazy with raid5
i know but) and then i put in a new 80gig drive and restarted
the machine, now, all i had to do was add back the drive
and then it should auto-rebuild. at least, thats the idea.



[root@survivor root]# mdadm --manage /dev/md0 -a
/dev/ide/host4/bus0/target0/lun0/part2
mdadm: add new device failed for /dev/ide/host4/bus0/target0/lun0/part2:
Device or resource busy

okay, so you probably would think that this device (or
/dev/hdi as its called short form) is being mounted or
used in someway, except i can assure you that its not.
this is the new disk and, yes, i have double checked
the actual device name ;>

this is under 2.5.67, under 2.5.62 it WOULD succeed however
the device would be inserted in location '0' in the raid
array rather than the empty (or -removed) slot. The output
from the raid5 is below:




[root@survivor root]# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Fri Dec 20 14:45:52 2002
     Raid Level : raid5
    Device Size : 58633216 (55.92 GiB 60.04 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Apr 15 17:45:24 2003
          State : dirty, no-errors
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

    Number   Major   Minor   RaidDevice State
       0      33        6        0      active sync  
/dev/ide/host2/bus0/target0/lun0/part6
       1      34        1        1      active sync  
/dev/ide/host2/bus1/target0/lun0/part1
       2       0        0       -1      removed
       3      57        1        3      active sync  
/dev/ide/host4/bus1/target0/lun0/part1
           UUID : 5bced80a:8f74bfa6:3e658b6a:46a5055d
         Events : 0.1895


and for good measure, here is /proc/mdstat

[root@survivor root]# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5]
md0 : inactive ide/host4/bus1/target0/lun0/part1[3]
ide/host2/bus1/target0/lun0/part1[1]
ide/host2/bus0/target0/lun0/part6[0]
      175900160 blocks
unused devices: <none>


any ideas/pointers ? i really would like the raid5
array back :D

regards 
Stef Telford <stef@chronozon.artofdns.com>

Re: mdadm and raid5 hot-rebuild problem on 2.5.67

[Neil Brown]

On  April 16, stef@chronozon.artofdns.com wrote:
> 
> [root@survivor root]# mdadm --manage /dev/md0 -a
> /dev/ide/host4/bus0/target0/lun0/part2
> mdadm: add new device failed for /dev/ide/host4/bus0/target0/lun0/part2:
> Device or resource busy

Do you get any kernel messages at the same time as this error?

NeilBrown

Re: mdadm and raid5 hot-rebuild problem on 2.5.67

[Stef Telford]

On Tue, 2003-04-15 at 21:48, Neil Brown wrote:
> On  April 16, stef@chronozon.artofdns.com wrote:
> > 
> > [root@survivor root]# mdadm --manage /dev/md0 -a
> > /dev/ide/host4/bus0/target0/lun0/part2
> > mdadm: add new device failed for /dev/ide/host4/bus0/target0/lun0/part2:
> > Device or resource busy
> 
> Do you get any kernel messages at the same time as this error?
> 

hello again neil, sorry to be yet more of a 'pain' ;)

this time my problem -doesnt- have any 'oops' or kernel fault.
that being said, please find attached the output from when
raidstart is re-run (after a raidstop of course ;)


md: md0 stopped.
md: unbind<ide/host4/bus1/target0/lun0/part1>
md: export_rdev(ide/host4/bus1/target0/lun0/part1)
md: unbind<ide/host2/bus1/target0/lun0/part1>
md: export_rdev(ide/host2/bus1/target0/lun0/part1)
md: unbind<ide/host2/bus0/target0/lun0/part6>
md: export_rdev(ide/host2/bus0/target0/lun0/part6)
md: autorun ...
md: considering ide/host4/bus1/target0/lun0/part1 ...
md:  adding ide/host4/bus1/target0/lun0/part1 ...
md:  adding ide/host2/bus1/target0/lun0/part1 ...
md:  adding ide/host2/bus0/target0/lun0/part6 ...
md: created md0
md: bind<ide/host2/bus0/target0/lun0/part6>
md: bind<ide/host2/bus1/target0/lun0/part1>
md: bind<ide/host4/bus1/target0/lun0/part1>
md: running:
<ide/host4/bus1/target0/lun0/part1><ide/host2/bus1/target0/lun0/part1><ide/host2/bus0/target0/lun0/part6>
md: md0: raid array is not clean -- starting background reconstruction
md0: max total readahead window set to 768k
md0: 3 data-disks, max readahead per data-disk: 256k
md0: setting max_sectors to 128, segment boundary to 32767
raid5: device ide/host4/bus1/target0/lun0/part1 operational as raid disk
3
raid5: device ide/host2/bus1/target0/lun0/part1 operational as raid disk
1
raid5: device ide/host2/bus0/target0/lun0/part6 operational as raid disk
0
raid5: cannot start dirty degraded array for md0
RAID5 conf printout:
 --- rd:4 wd:3 fd:1
 disk 0, o:1, dev:ide/host2/bus0/target0/lun0/part6
 disk 1, o:1, dev:ide/host2/bus1/target0/lun0/part1
 disk 3, o:1, dev:ide/host4/bus1/target0/lun0/part1
raid5: failed to run raid set md0
md: pers->run() failed ...
md :do_md_run() returned -22
md: md0 still in use.
md: ... autorun DONE.



is there anything else you need or would like ?

regards 
Stef Telford <stef@chronozon.artofdns.com>

Re: mdadm and raid5 hot-rebuild problem on 2.5.67

[Neil Brown]

On  April 16, stef@chronozon.artofdns.com wrote:
> On Tue, 2003-04-15 at 21:48, Neil Brown wrote:
> > On  April 16, stef@chronozon.artofdns.com wrote:
> > > 
> > > [root@survivor root]# mdadm --manage /dev/md0 -a
> > > /dev/ide/host4/bus0/target0/lun0/part2
> > > mdadm: add new device failed for /dev/ide/host4/bus0/target0/lun0/part2:
> > > Device or resource busy
> > 
> > Do you get any kernel messages at the same time as this error?
> > 
> 
> hello again neil, sorry to be yet more of a 'pain' ;)
> 
> this time my problem -doesnt- have any 'oops' or kernel fault.
> that being said, please find attached the output from when
> raidstart is re-run (after a raidstop of course ;)

raidstart!  Horrible tool.  Throw it away.

...
> md: running:
> <ide/host4/bus1/target0/lun0/part1><ide/host2/bus1/target0/lun0/part1><ide/host2/bus0/target0/lun0/part6>
> md: md0: raid array is not clean -- starting background reconstruction
> md0: max total readahead window set to 768k
> md0: 3 data-disks, max readahead per data-disk: 256k
> md0: setting max_sectors to 128, segment boundary to 32767
> raid5: device ide/host4/bus1/target0/lun0/part1 operational as raid disk
> 3
> raid5: device ide/host2/bus1/target0/lun0/part1 operational as raid disk
> 1
> raid5: device ide/host2/bus0/target0/lun0/part6 operational as raid disk
> 0
> raid5: cannot start dirty degraded array for md0

You will need
   mdadm --assemble --force ....
to start this array as it seems to have suffered an unclean shutdown
while degraded.

> 
> 
> is there anything else you need or would like ?

Well, an answer to my original question would be nice:-)  You should
have kernel logs of the time when you got "device busy" from "mdadm -a". 

NeilBrown

Re: mdadm and raid5 hot-rebuild problem on 2.5.67

[Stef Telford]

> raidstart!  Horrible tool.  Throw it away.
> 

consider it gone :)

> You will need
>    mdadm --assemble --force ....
> to start this array as it seems to have suffered an unclean shutdown
> while degraded.
> 

hrm, seems to not want to 'fly' to be honest...


[root@survivor stef]# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.00
  Creation Time : Fri Dec 20 14:45:52 2002
     Raid Level : raid5
    Device Size : 58633216 (55.92 GiB 60.04 GB)
   Raid Devices : 4
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Apr 15 17:45:24 2003
          State : dirty, no-errors
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

    Number   Major   Minor   RaidDevice State
       0      33        6        0      active sync  
/dev/ide/host2/bus0/target0/lun0/part6
       1      34        1        1      active sync  
/dev/ide/host2/bus1/target0/lun0/part1
       2       0        0       -1      removed
       3      57        1        3      active sync  
/dev/ide/host4/bus1/target0/lun0/part1
           UUID : 5bced80a:8f74bfa6:3e658b6a:46a5055d
         Events : 0.1895
[root@survivor stef]# mdadm --assemble --force /dev/md0
mdadm: device /dev/md0 already active - cannot assemble it
[root@survivor stef]# mdadm --manage /dev/md0 -a
/dev/ide/host4/bus0/target0/lun0/part2
mdadm: added /dev/ide/host4/bus0/target0/lun0/part2
[root@survivor stef]# mdadm --assemble --force /dev/md0
mdadm: device /dev/md0 already active - cannot assemble it
[root@survivor stef]# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.00
  Creation Time : Fri Dec 20 14:45:52 2002
     Raid Level : raid5
    Device Size : 58633216 (55.92 GiB 60.04 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue Apr 15 17:45:24 2003
          State : dirty, no-errors
 Active Devices : 3
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 64K

    Number   Major   Minor   RaidDevice State
       0      56        2        0       
/dev/ide/host4/bus0/target0/lun0/part2
       1      34        1        1      active sync  
/dev/ide/host2/bus1/target0/lun0/part1
       2       0        0       -1      removed
       3      57        1        3      active sync  
/dev/ide/host4/bus1/target0/lun0/part1
           UUID : 5bced80a:8f74bfa6:3e658b6a:46a5055d
         Events : 0.1895

surely, when i -a a drive into the array, it should
'fit into' the removed slot on item '2', or does it
do that already and i am reading the output wrong :\

> > is there anything else you need or would like ?
> 
> Well, an answer to my original question would be nice:-)  You should
> have kernel logs of the time when you got "device busy" from "mdadm -a". 
> 

ah this being the rub, there isnt any oops or anything
like that in the kernel log when i get the device busy,
not even a 'device currently locked' or 'missing device'
from what i can see. sorry :\

> regards 
> Stef Telford <stef@chronozon.artofdns.com>

Re: mdadm and raid5 hot-rebuild problem on 2.5.67

[Neil Brown]

On  April 16, stef@chronozon.artofdns.com wrote:
> > raidstart!  Horrible tool.  Throw it away.
> > 
> 
> consider it gone :)
> 
> > You will need
> >    mdadm --assemble --force ....
> > to start this array as it seems to have suffered an unclean shutdown
> > while degraded.
> > 
> 
> hrm, seems to not want to 'fly' to be honest...
> 

I think I misinterpreted the messages cause by raidstart.
You have given me lots of good information here.  I will try to
interpret it for you:

> 
> [root@survivor stef]# mdadm --detail /dev/md0
> /dev/md0:

to md0 has been sucessfully started...

>    Raid Devices : 4
it should have 4 devices...
...
> 
>     Number   Major   Minor   RaidDevice State
>        0      33        6        0      active sync   /dev/ide/host2/bus0/target0/lun0/part6
>        1      34        1        1      active sync   /dev/ide/host2/bus1/target0/lun0/part1
>        2       0        0       -1      removed
>        3      57        1        3      active sync   /dev/ide/host4/bus1/target0/lun0/part1

but only has 3, so it is degraded.

> [root@survivor stef]# mdadm --assemble --force /dev/md0
> mdadm: device /dev/md0 already active - cannot assemble it

Yep.  It is already started somehow so it cannot be started again.

> [root@survivor stef]# mdadm --manage /dev/md0 -a /dev/ide/host4/bus0/target0/lun0/part2
> mdadm: added /dev/ide/host4/bus0/target0/lun0/part2

We've successfully added a newdrive.  This should be installed as
RaidDevice 2, but not marked 'sync'.  A resync process should run, and
once that is finished, it will be marked sync.
Note that this *is*different* to 2.4 behaviour.
In 2.4 a spare would take up a virtual location outside the regular
array, be synced while there, and then swapped in.
In 2.5, when a spare is activated, it is swapped into position
straight away, then synced, then marked active.  This makes the
internal code cleaner.


> [root@survivor stef]# mdadm --assemble --force /dev/md0
> mdadm: device /dev/md0 already active - cannot assemble it

Still active, so cannot be assembled again.

> [root@survivor stef]# mdadm --detail /dev/md0
...
>     Number   Major   Minor   RaidDevice State
>        0      56        2        0           /dev/ide/host4/bus0/target0/lun0/part2
>        1      34        1        1      active sync   /dev/ide/host2/bus1/target0/lun0/part1
>        2       0        0       -1      removed
>        3      57        1        3      active sync   /dev/ide/host4/bus1/target0/lun0/part1

Ok, this is wrong.  Bigtime.
The spare has taken up postion 0 instead of 2.

I cannot immediately see what would be causing this.  The code looks
right, but obviously isn't.  I'll try to see if I can reproduce it
tomorrow.

> 
> surely, when i -a a drive into the array, it should
> 'fit into' the removed slot on item '2', or does it
> do that already and i am reading the output wrong :\
> 

Yes it should and no it isn't.

> > Well, an answer to my original question would be nice:-)  You should
> > have kernel logs of the time when you got "device busy" from "mdadm -a". 
> > 
> 
> ah this being the rub, there isnt any oops or anything
> like that in the kernel log when i get the device busy,
> not even a 'device currently locked' or 'missing device'
> from what i can see. sorry :\

Well that fact that were was definately no messages logged is helpful,
though I'm not sure yet what it means.
There aren't very many paths which return 'EBUSY'.

NeilBrown

Re: mdadm and raid5 hot-rebuild problem on 2.5.67

[Neil Brown]

On Wednesday April 16, neilb@cse.unsw.edu.au wrote:
> 
> Ok, this is wrong.  Bigtime.
> The spare has taken up postion 0 instead of 2.
> 
> I cannot immediately see what would be causing this.  The code looks
> right, but obviously isn't.  I'll try to see if I can reproduce it
> tomorrow.
> 

I haven't managed to reproduce it, but I'm faily sure that I know what
was happening.  I'll try to figure out the best way to fix it.

In the mean time, you can avoid the problem by not using raidstart at
all. 
Reboot your machine, with raidstart disabled (e.g. move /etc/raidtab
out of the way) and then us mdadm to start the array.  I think you
will need "mdadm --assemble --force" to start it.
That didn't work before because raidstart had already tried to started
it and failed and the array have been left in an inconsistant start.
I suggest the reboot to make sure the array is not in an inconsistant
state. 

NeilBrown

Re: mdadm and raid5 hot-rebuild problem on 2.5.67

[Gregory Leblanc]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On Wed, 2003-04-16 at 17:29, Neil Brown wrote:
> On Wednesday April 16, neilb@cse.unsw.edu.au wrote:
> > 
> > Ok, this is wrong.  Bigtime.
> > The spare has taken up postion 0 instead of 2.
> > 
> > I cannot immediately see what would be causing this.  The code looks
> > right, but obviously isn't.  I'll try to see if I can reproduce it
> > tomorrow.
> > 
> 
> I haven't managed to reproduce it, but I'm faily sure that I know what
> was happening.  I'll try to figure out the best way to fix it.
> 
> In the mean time, you can avoid the problem by not using raidstart at
> all. 
> Reboot your machine, with raidstart disabled (e.g. move /etc/raidtab
> out of the way) and then us mdadm to start the array.  I think you
> will need "mdadm --assemble --force" to start it.

On a related note, I've recently hacked on rc.sysinit so that it
recognizes /etc/mdadm.conf, in addition to /etc/raidtab.  I've sent to
patch upstream to my vendor of Linux, you can find it at
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88785
	Greg


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: mdadm and raid5 hot-rebuild problem on 2.5.67

[Stef Telford]

neil wrote
> I haven't managed to reproduce it, but I'm faily sure that I know what
> was happening.  I'll try to figure out the best way to fix it.
> 

okay, its solved at my end (as i jst informed the list
and yourself) but would rather no one else is 'bitten'
by this. again, it could be purely a semantical error
or display problem and there wasnt a 'real' error, 
but, i am skeptical of that (no offense meant)

> In the mean time, you can avoid the problem by not using raidstart at
> all. 
> Reboot your machine, with raidstart disabled (e.g. move /etc/raidtab
> out of the way) and then us mdadm to start the array.  I think you
> will need "mdadm --assemble --force" to start it.
> That didn't work before because raidstart had already tried to started
> it and failed and the array have been left in an inconsistant start.
> I suggest the reboot to make sure the array is not in an inconsistant
> state. 
> 

funnily enough, this is -exactly- what i did, i threw
away /etc/raidtab and even went so far as to mv the
raid* binaries into a directory under root ( go_away
i called it ;), once i jst stuck with using mdadm, the
actual operation was almost laughably easy. so, mea culpa :)

thanks and best wishes 
Stef Telford <stef@chronozon.artofdns.com>