* Re: mdadm and raid5 hot-rebuild problem on 2.5.67
2003-04-16 23:53 mdadm and raid5 hot-rebuild problem on 2.5.67 Stef Telford
@ 2003-04-16 1:48 ` Neil Brown
2003-04-17 1:51 ` Stef Telford
0 siblings, 1 reply; 9+ messages in thread
From: Neil Brown @ 2003-04-16 1:48 UTC (permalink / raw)
To: Stef Telford; +Cc: linux-raid
On April 16, stef@chronozon.artofdns.com wrote:
>
> [root@survivor root]# mdadm --manage /dev/md0 -a
> /dev/ide/host4/bus0/target0/lun0/part2
> mdadm: add new device failed for /dev/ide/host4/bus0/target0/lun0/part2:
> Device or resource busy
Do you get any kernel messages at the same time as this error?
NeilBrown
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mdadm and raid5 hot-rebuild problem on 2.5.67
2003-04-17 1:51 ` Stef Telford
@ 2003-04-16 1:57 ` Neil Brown
2003-04-16 2:13 ` Stef Telford
0 siblings, 1 reply; 9+ messages in thread
From: Neil Brown @ 2003-04-16 1:57 UTC (permalink / raw)
To: Stef Telford; +Cc: linux-raid
On April 16, stef@chronozon.artofdns.com wrote:
> On Tue, 2003-04-15 at 21:48, Neil Brown wrote:
> > On April 16, stef@chronozon.artofdns.com wrote:
> > >
> > > [root@survivor root]# mdadm --manage /dev/md0 -a
> > > /dev/ide/host4/bus0/target0/lun0/part2
> > > mdadm: add new device failed for /dev/ide/host4/bus0/target0/lun0/part2:
> > > Device or resource busy
> >
> > Do you get any kernel messages at the same time as this error?
> >
>
> hello again neil, sorry to be yet more of a 'pain' ;)
>
> this time my problem -doesnt- have any 'oops' or kernel fault.
> that being said, please find attached the output from when
> raidstart is re-run (after a raidstop of course ;)
raidstart! Horrible tool. Throw it away.
...
> md: running:
> <ide/host4/bus1/target0/lun0/part1><ide/host2/bus1/target0/lun0/part1><ide/host2/bus0/target0/lun0/part6>
> md: md0: raid array is not clean -- starting background reconstruction
> md0: max total readahead window set to 768k
> md0: 3 data-disks, max readahead per data-disk: 256k
> md0: setting max_sectors to 128, segment boundary to 32767
> raid5: device ide/host4/bus1/target0/lun0/part1 operational as raid disk
> 3
> raid5: device ide/host2/bus1/target0/lun0/part1 operational as raid disk
> 1
> raid5: device ide/host2/bus0/target0/lun0/part6 operational as raid disk
> 0
> raid5: cannot start dirty degraded array for md0
You will need
mdadm --assemble --force ....
to start this array as it seems to have suffered an unclean shutdown
while degraded.
>
>
> is there anything else you need or would like ?
Well, an answer to my original question would be nice:-) You should
have kernel logs of the time when you got "device busy" from "mdadm -a".
NeilBrown
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mdadm and raid5 hot-rebuild problem on 2.5.67
2003-04-16 1:57 ` Neil Brown
@ 2003-04-16 2:13 ` Stef Telford
2003-04-16 5:39 ` Neil Brown
0 siblings, 1 reply; 9+ messages in thread
From: Stef Telford @ 2003-04-16 2:13 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
> raidstart! Horrible tool. Throw it away.
>
consider it gone :)
> You will need
> mdadm --assemble --force ....
> to start this array as it seems to have suffered an unclean shutdown
> while degraded.
>
hrm, seems to not want to 'fly' to be honest...
[root@survivor stef]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.00
Creation Time : Fri Dec 20 14:45:52 2002
Raid Level : raid5
Device Size : 58633216 (55.92 GiB 60.04 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Apr 15 17:45:24 2003
State : dirty, no-errors
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
0 33 6 0 active sync
/dev/ide/host2/bus0/target0/lun0/part6
1 34 1 1 active sync
/dev/ide/host2/bus1/target0/lun0/part1
2 0 0 -1 removed
3 57 1 3 active sync
/dev/ide/host4/bus1/target0/lun0/part1
UUID : 5bced80a:8f74bfa6:3e658b6a:46a5055d
Events : 0.1895
[root@survivor stef]# mdadm --assemble --force /dev/md0
mdadm: device /dev/md0 already active - cannot assemble it
[root@survivor stef]# mdadm --manage /dev/md0 -a
/dev/ide/host4/bus0/target0/lun0/part2
mdadm: added /dev/ide/host4/bus0/target0/lun0/part2
[root@survivor stef]# mdadm --assemble --force /dev/md0
mdadm: device /dev/md0 already active - cannot assemble it
[root@survivor stef]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.00
Creation Time : Fri Dec 20 14:45:52 2002
Raid Level : raid5
Device Size : 58633216 (55.92 GiB 60.04 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Apr 15 17:45:24 2003
State : dirty, no-errors
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
0 56 2 0
/dev/ide/host4/bus0/target0/lun0/part2
1 34 1 1 active sync
/dev/ide/host2/bus1/target0/lun0/part1
2 0 0 -1 removed
3 57 1 3 active sync
/dev/ide/host4/bus1/target0/lun0/part1
UUID : 5bced80a:8f74bfa6:3e658b6a:46a5055d
Events : 0.1895
surely, when i -a a drive into the array, it should
'fit into' the removed slot on item '2', or does it
do that already and i am reading the output wrong :\
> > is there anything else you need or would like ?
>
> Well, an answer to my original question would be nice:-) You should
> have kernel logs of the time when you got "device busy" from "mdadm -a".
>
ah this being the rub, there isnt any oops or anything
like that in the kernel log when i get the device busy,
not even a 'device currently locked' or 'missing device'
from what i can see. sorry :\
> regards
> Stef Telford <stef@chronozon.artofdns.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mdadm and raid5 hot-rebuild problem on 2.5.67
2003-04-16 2:13 ` Stef Telford
@ 2003-04-16 5:39 ` Neil Brown
2003-04-17 0:29 ` Neil Brown
0 siblings, 1 reply; 9+ messages in thread
From: Neil Brown @ 2003-04-16 5:39 UTC (permalink / raw)
To: Stef Telford; +Cc: linux-raid
On April 16, stef@chronozon.artofdns.com wrote:
> > raidstart! Horrible tool. Throw it away.
> >
>
> consider it gone :)
>
> > You will need
> > mdadm --assemble --force ....
> > to start this array as it seems to have suffered an unclean shutdown
> > while degraded.
> >
>
> hrm, seems to not want to 'fly' to be honest...
>
I think I misinterpreted the messages cause by raidstart.
You have given me lots of good information here. I will try to
interpret it for you:
>
> [root@survivor stef]# mdadm --detail /dev/md0
> /dev/md0:
to md0 has been sucessfully started...
> Raid Devices : 4
it should have 4 devices...
...
>
> Number Major Minor RaidDevice State
> 0 33 6 0 active sync /dev/ide/host2/bus0/target0/lun0/part6
> 1 34 1 1 active sync /dev/ide/host2/bus1/target0/lun0/part1
> 2 0 0 -1 removed
> 3 57 1 3 active sync /dev/ide/host4/bus1/target0/lun0/part1
but only has 3, so it is degraded.
> [root@survivor stef]# mdadm --assemble --force /dev/md0
> mdadm: device /dev/md0 already active - cannot assemble it
Yep. It is already started somehow so it cannot be started again.
> [root@survivor stef]# mdadm --manage /dev/md0 -a /dev/ide/host4/bus0/target0/lun0/part2
> mdadm: added /dev/ide/host4/bus0/target0/lun0/part2
We've successfully added a newdrive. This should be installed as
RaidDevice 2, but not marked 'sync'. A resync process should run, and
once that is finished, it will be marked sync.
Note that this *is*different* to 2.4 behaviour.
In 2.4 a spare would take up a virtual location outside the regular
array, be synced while there, and then swapped in.
In 2.5, when a spare is activated, it is swapped into position
straight away, then synced, then marked active. This makes the
internal code cleaner.
> [root@survivor stef]# mdadm --assemble --force /dev/md0
> mdadm: device /dev/md0 already active - cannot assemble it
Still active, so cannot be assembled again.
> [root@survivor stef]# mdadm --detail /dev/md0
...
> Number Major Minor RaidDevice State
> 0 56 2 0 /dev/ide/host4/bus0/target0/lun0/part2
> 1 34 1 1 active sync /dev/ide/host2/bus1/target0/lun0/part1
> 2 0 0 -1 removed
> 3 57 1 3 active sync /dev/ide/host4/bus1/target0/lun0/part1
Ok, this is wrong. Bigtime.
The spare has taken up postion 0 instead of 2.
I cannot immediately see what would be causing this. The code looks
right, but obviously isn't. I'll try to see if I can reproduce it
tomorrow.
>
> surely, when i -a a drive into the array, it should
> 'fit into' the removed slot on item '2', or does it
> do that already and i am reading the output wrong :\
>
Yes it should and no it isn't.
> > Well, an answer to my original question would be nice:-) You should
> > have kernel logs of the time when you got "device busy" from "mdadm -a".
> >
>
> ah this being the rub, there isnt any oops or anything
> like that in the kernel log when i get the device busy,
> not even a 'device currently locked' or 'missing device'
> from what i can see. sorry :\
Well that fact that were was definately no messages logged is helpful,
though I'm not sure yet what it means.
There aren't very many paths which return 'EBUSY'.
NeilBrown
^ permalink raw reply [flat|nested] 9+ messages in thread
* mdadm and raid5 hot-rebuild problem on 2.5.67
@ 2003-04-16 23:53 Stef Telford
2003-04-16 1:48 ` Neil Brown
0 siblings, 1 reply; 9+ messages in thread
From: Stef Telford @ 2003-04-16 23:53 UTC (permalink / raw)
To: linux-raid
or rather not a 'rebuild'.
The original problem was that one of the raid5 disks was
dead on waking up this morning, so, i used mdadm to set
it to fail, and then i set it removed. since it was software
raid5, i shut down the machine (sort of crazy with raid5
i know but) and then i put in a new 80gig drive and restarted
the machine, now, all i had to do was add back the drive
and then it should auto-rebuild. at least, thats the idea.
[root@survivor root]# mdadm --manage /dev/md0 -a
/dev/ide/host4/bus0/target0/lun0/part2
mdadm: add new device failed for /dev/ide/host4/bus0/target0/lun0/part2:
Device or resource busy
okay, so you probably would think that this device (or
/dev/hdi as its called short form) is being mounted or
used in someway, except i can assure you that its not.
this is the new disk and, yes, i have double checked
the actual device name ;>
this is under 2.5.67, under 2.5.62 it WOULD succeed however
the device would be inserted in location '0' in the raid
array rather than the empty (or -removed) slot. The output
from the raid5 is below:
[root@survivor root]# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.01
Creation Time : Fri Dec 20 14:45:52 2002
Raid Level : raid5
Device Size : 58633216 (55.92 GiB 60.04 GB)
Raid Devices : 4
Total Devices : 3
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Tue Apr 15 17:45:24 2003
State : dirty, no-errors
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
0 33 6 0 active sync
/dev/ide/host2/bus0/target0/lun0/part6
1 34 1 1 active sync
/dev/ide/host2/bus1/target0/lun0/part1
2 0 0 -1 removed
3 57 1 3 active sync
/dev/ide/host4/bus1/target0/lun0/part1
UUID : 5bced80a:8f74bfa6:3e658b6a:46a5055d
Events : 0.1895
and for good measure, here is /proc/mdstat
[root@survivor root]# cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid5]
md0 : inactive ide/host4/bus1/target0/lun0/part1[3]
ide/host2/bus1/target0/lun0/part1[1]
ide/host2/bus0/target0/lun0/part6[0]
175900160 blocks
unused devices: <none>
any ideas/pointers ? i really would like the raid5
array back :D
regards
Stef Telford <stef@chronozon.artofdns.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mdadm and raid5 hot-rebuild problem on 2.5.67
2003-04-16 5:39 ` Neil Brown
@ 2003-04-17 0:29 ` Neil Brown
2003-04-17 0:52 ` Gregory Leblanc
2003-04-17 23:43 ` Stef Telford
0 siblings, 2 replies; 9+ messages in thread
From: Neil Brown @ 2003-04-17 0:29 UTC (permalink / raw)
To: Stef Telford, linux-raid
On Wednesday April 16, neilb@cse.unsw.edu.au wrote:
>
> Ok, this is wrong. Bigtime.
> The spare has taken up postion 0 instead of 2.
>
> I cannot immediately see what would be causing this. The code looks
> right, but obviously isn't. I'll try to see if I can reproduce it
> tomorrow.
>
I haven't managed to reproduce it, but I'm faily sure that I know what
was happening. I'll try to figure out the best way to fix it.
In the mean time, you can avoid the problem by not using raidstart at
all.
Reboot your machine, with raidstart disabled (e.g. move /etc/raidtab
out of the way) and then us mdadm to start the array. I think you
will need "mdadm --assemble --force" to start it.
That didn't work before because raidstart had already tried to started
it and failed and the array have been left in an inconsistant start.
I suggest the reboot to make sure the array is not in an inconsistant
state.
NeilBrown
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mdadm and raid5 hot-rebuild problem on 2.5.67
2003-04-17 0:29 ` Neil Brown
@ 2003-04-17 0:52 ` Gregory Leblanc
2003-04-17 23:43 ` Stef Telford
1 sibling, 0 replies; 9+ messages in thread
From: Gregory Leblanc @ 2003-04-17 0:52 UTC (permalink / raw)
To: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]
On Wed, 2003-04-16 at 17:29, Neil Brown wrote:
> On Wednesday April 16, neilb@cse.unsw.edu.au wrote:
> >
> > Ok, this is wrong. Bigtime.
> > The spare has taken up postion 0 instead of 2.
> >
> > I cannot immediately see what would be causing this. The code looks
> > right, but obviously isn't. I'll try to see if I can reproduce it
> > tomorrow.
> >
>
> I haven't managed to reproduce it, but I'm faily sure that I know what
> was happening. I'll try to figure out the best way to fix it.
>
> In the mean time, you can avoid the problem by not using raidstart at
> all.
> Reboot your machine, with raidstart disabled (e.g. move /etc/raidtab
> out of the way) and then us mdadm to start the array. I think you
> will need "mdadm --assemble --force" to start it.
On a related note, I've recently hacked on rc.sysinit so that it
recognizes /etc/mdadm.conf, in addition to /etc/raidtab. I've sent to
patch upstream to my vendor of Linux, you can find it at
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88785
Greg
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mdadm and raid5 hot-rebuild problem on 2.5.67
2003-04-16 1:48 ` Neil Brown
@ 2003-04-17 1:51 ` Stef Telford
2003-04-16 1:57 ` Neil Brown
0 siblings, 1 reply; 9+ messages in thread
From: Stef Telford @ 2003-04-17 1:51 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On Tue, 2003-04-15 at 21:48, Neil Brown wrote:
> On April 16, stef@chronozon.artofdns.com wrote:
> >
> > [root@survivor root]# mdadm --manage /dev/md0 -a
> > /dev/ide/host4/bus0/target0/lun0/part2
> > mdadm: add new device failed for /dev/ide/host4/bus0/target0/lun0/part2:
> > Device or resource busy
>
> Do you get any kernel messages at the same time as this error?
>
hello again neil, sorry to be yet more of a 'pain' ;)
this time my problem -doesnt- have any 'oops' or kernel fault.
that being said, please find attached the output from when
raidstart is re-run (after a raidstop of course ;)
md: md0 stopped.
md: unbind<ide/host4/bus1/target0/lun0/part1>
md: export_rdev(ide/host4/bus1/target0/lun0/part1)
md: unbind<ide/host2/bus1/target0/lun0/part1>
md: export_rdev(ide/host2/bus1/target0/lun0/part1)
md: unbind<ide/host2/bus0/target0/lun0/part6>
md: export_rdev(ide/host2/bus0/target0/lun0/part6)
md: autorun ...
md: considering ide/host4/bus1/target0/lun0/part1 ...
md: adding ide/host4/bus1/target0/lun0/part1 ...
md: adding ide/host2/bus1/target0/lun0/part1 ...
md: adding ide/host2/bus0/target0/lun0/part6 ...
md: created md0
md: bind<ide/host2/bus0/target0/lun0/part6>
md: bind<ide/host2/bus1/target0/lun0/part1>
md: bind<ide/host4/bus1/target0/lun0/part1>
md: running:
<ide/host4/bus1/target0/lun0/part1><ide/host2/bus1/target0/lun0/part1><ide/host2/bus0/target0/lun0/part6>
md: md0: raid array is not clean -- starting background reconstruction
md0: max total readahead window set to 768k
md0: 3 data-disks, max readahead per data-disk: 256k
md0: setting max_sectors to 128, segment boundary to 32767
raid5: device ide/host4/bus1/target0/lun0/part1 operational as raid disk
3
raid5: device ide/host2/bus1/target0/lun0/part1 operational as raid disk
1
raid5: device ide/host2/bus0/target0/lun0/part6 operational as raid disk
0
raid5: cannot start dirty degraded array for md0
RAID5 conf printout:
--- rd:4 wd:3 fd:1
disk 0, o:1, dev:ide/host2/bus0/target0/lun0/part6
disk 1, o:1, dev:ide/host2/bus1/target0/lun0/part1
disk 3, o:1, dev:ide/host4/bus1/target0/lun0/part1
raid5: failed to run raid set md0
md: pers->run() failed ...
md :do_md_run() returned -22
md: md0 still in use.
md: ... autorun DONE.
is there anything else you need or would like ?
regards
Stef Telford <stef@chronozon.artofdns.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: mdadm and raid5 hot-rebuild problem on 2.5.67
2003-04-17 0:29 ` Neil Brown
2003-04-17 0:52 ` Gregory Leblanc
@ 2003-04-17 23:43 ` Stef Telford
1 sibling, 0 replies; 9+ messages in thread
From: Stef Telford @ 2003-04-17 23:43 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
neil wrote
> I haven't managed to reproduce it, but I'm faily sure that I know what
> was happening. I'll try to figure out the best way to fix it.
>
okay, its solved at my end (as i jst informed the list
and yourself) but would rather no one else is 'bitten'
by this. again, it could be purely a semantical error
or display problem and there wasnt a 'real' error,
but, i am skeptical of that (no offense meant)
> In the mean time, you can avoid the problem by not using raidstart at
> all.
> Reboot your machine, with raidstart disabled (e.g. move /etc/raidtab
> out of the way) and then us mdadm to start the array. I think you
> will need "mdadm --assemble --force" to start it.
> That didn't work before because raidstart had already tried to started
> it and failed and the array have been left in an inconsistant start.
> I suggest the reboot to make sure the array is not in an inconsistant
> state.
>
funnily enough, this is -exactly- what i did, i threw
away /etc/raidtab and even went so far as to mv the
raid* binaries into a directory under root ( go_away
i called it ;), once i jst stuck with using mdadm, the
actual operation was almost laughably easy. so, mea culpa :)
thanks and best wishes
Stef Telford <stef@chronozon.artofdns.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2003-04-17 23:43 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-04-16 23:53 mdadm and raid5 hot-rebuild problem on 2.5.67 Stef Telford
2003-04-16 1:48 ` Neil Brown
2003-04-17 1:51 ` Stef Telford
2003-04-16 1:57 ` Neil Brown
2003-04-16 2:13 ` Stef Telford
2003-04-16 5:39 ` Neil Brown
2003-04-17 0:29 ` Neil Brown
2003-04-17 0:52 ` Gregory Leblanc
2003-04-17 23:43 ` Stef Telford
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox