* Need help recovering a raid5 array
@ 2006-10-23 15:21 eharney
2006-10-24 8:49 ` David Greaves
0 siblings, 1 reply; 3+ messages in thread
From: eharney @ 2006-10-23 15:21 UTC (permalink / raw)
To: linux-raid
Hello all,
I had a disk fail in a raid 5 array (4 disk array, no spares), and am
having trouble recovering it. I believe my data is still safe, but I
cannot tell what is going wrong here.
When I try to rebuild the array "mdadm --assemble /dev/md0 /dev/sda2
/dev/sdb2 /dev/sdc2 /dev/sdd2" I see "failed to RUN_ARRAY /dev/md0:
Input/output error".
dmesg shows the following:
md: bind<sdb2>
md: bind<sdc2>
md: bind<sdd2>
md: bind<sda2>
md: md0: raid array is not clean -- starting background reconstruction
raid5: device sda2 operational as raid disk 0
raid5: device sdc2 operational as raid disk 2
raid5: device sdb2 operational as raid disk 1
raid5: cannot start dirty degraded array for md0
RAID5 conf printout:
--- rd:4 wd:3 fd:1
disk 0, o:1, dev:sda2
disk 1, o:1, dev:sdb2
disk 2, o:1, dev:sdc2
raid5: failed to run raid set md0
md: pers->run() failed ...
/proc mdstat shows:
md0 : inactive sda2[0] sdd2[3](S) sdc2[2] sdb2[1]
This seems wrong, as sdd2 should not be a spare - I want it to be the
fourth disk.
The output of mdadm -E for each disk is as follows:
sda2:
/dev/sda2:
Magic : a92b4efc
Version : 00.90.00
UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
Creation Time : Thu Jun 1 21:13:58 2006
Raid Level : raid5
Device Size : 390555904 (372.46 GiB 399.93 GB)
Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Sun Oct 22 23:39:06 2006
State : active
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Checksum : 683f2f5c - correct
Events : 0.8831997
Layout : left-symmetric
Chunk Size : 256K
Number Major Minor RaidDevice State
this 0 8 2 0 active sync /dev/sda2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
4 4 8 50 4 spare /dev/sdd2
sdb2:
/dev/sdb2:
Magic : a92b4efc
Version : 00.90.00
UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
Creation Time : Thu Jun 1 21:13:58 2006
Raid Level : raid5
Device Size : 390555904 (372.46 GiB 399.93 GB)
Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Sun Oct 22 23:39:06 2006
State : active
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Checksum : 683f2f6e - correct
Events : 0.8831997
Layout : left-symmetric
Chunk Size : 256K
Number Major Minor RaidDevice State
this 1 8 18 1 active sync /dev/sdb2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
4 4 8 50 4 spare /dev/sdd2
sdc2:
/dev/sdc2:
Magic : a92b4efc
Version : 00.90.00
UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
Creation Time : Thu Jun 1 21:13:58 2006
Raid Level : raid5
Device Size : 390555904 (372.46 GiB 399.93 GB)
Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Sun Oct 22 23:39:06 2006
State : active
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Checksum : 683f2f80 - correct
Events : 0.8831997
Layout : left-symmetric
Chunk Size : 256K
Number Major Minor RaidDevice State
this 2 8 34 2 active sync /dev/sdc2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 0 0 3 faulty removed
4 4 8 50 4 spare /dev/sdd2
sdd2:
/dev/sdd2:
Magic : a92b4efc
Version : 00.90.00
UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
Creation Time : Thu Jun 1 21:13:58 2006
Raid Level : raid5
Device Size : 390555904 (372.46 GiB 399.93 GB)
Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 0
Update Time : Sun Oct 22 23:39:06 2006
State : active
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1
Checksum : 683f2fbf - correct
Events : 0.8831997
Layout : left-symmetric
Chunk Size : 256K
Number Major Minor RaidDevice State
this 3 8 50 -1 sync /dev/sdd2
0 0 8 2 0 active sync /dev/sda2
1 1 8 18 1 active sync /dev/sdb2
2 2 8 34 2 active sync /dev/sdc2
3 3 8 50 -1 sync /dev/sdd2
4 4 8 50 4 spare /dev/sdd2
Does anyone have any idea how to get this array back into good shape?
I'm not sure why it thinks sdd2 should be a spare, or how to get it back
to being a regular disk.
I would appreciate any help you can offer. (Also, am I right in thinking
my data is still good? I should still have 3 of the 4 disks working fine,
at any rate.)
Thanks,
Eric
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Need help recovering a raid5 array
[not found] <44029.130.127.44.164.1161616905.squirrel@wm.clemson.edu>
@ 2006-10-23 17:49 ` eharney
0 siblings, 0 replies; 3+ messages in thread
From: eharney @ 2006-10-23 17:49 UTC (permalink / raw)
To: eharney; +Cc: linux-raid
> Hello all,
>
> I had a disk fail in a raid 5 array (4 disk array, no spares), and am
> having trouble recovering it. I believe my data is still safe, but I
> cannot tell what is going wrong here.
>
> When I try to rebuild the array "mdadm --assemble /dev/md0 /dev/sda2
> /dev/sdb2 /dev/sdc2 /dev/sdd2" I see "failed to RUN_ARRAY /dev/md0:
> Input/output error".
>
> dmesg shows the following:
> md: bind<sdb2>
> md: bind<sdc2>
> md: bind<sdd2>
> md: bind<sda2>
> md: md0: raid array is not clean -- starting background reconstruction
> raid5: device sda2 operational as raid disk 0
> raid5: device sdc2 operational as raid disk 2
> raid5: device sdb2 operational as raid disk 1
> raid5: cannot start dirty degraded array for md0
> RAID5 conf printout:
> --- rd:4 wd:3 fd:1
> disk 0, o:1, dev:sda2
> disk 1, o:1, dev:sdb2
> disk 2, o:1, dev:sdc2
> raid5: failed to run raid set md0
> md: pers->run() failed ...
>
>
>
> /proc mdstat shows:
> md0 : inactive sda2[0] sdd2[3](S) sdc2[2] sdb2[1]
>
> This seems wrong, as sdd2 should not be a spare - I want it to be the
> fourth disk.
>
>
> The output of mdadm -E for each disk is as follows:
> sda2:
> /dev/sda2:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
> Creation Time : Thu Jun 1 21:13:58 2006
> Raid Level : raid5
> Device Size : 390555904 (372.46 GiB 399.93 GB)
> Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
> Raid Devices : 4
> Total Devices : 4
> Preferred Minor : 0
>
> Update Time : Sun Oct 22 23:39:06 2006
> State : active
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 683f2f5c - correct
> Events : 0.8831997
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 0 8 2 0 active sync /dev/sda2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> 4 4 8 50 4 spare /dev/sdd2
>
>
> sdb2:
> /dev/sdb2:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
> Creation Time : Thu Jun 1 21:13:58 2006
> Raid Level : raid5
> Device Size : 390555904 (372.46 GiB 399.93 GB)
> Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
> Raid Devices : 4
> Total Devices : 4
> Preferred Minor : 0
>
> Update Time : Sun Oct 22 23:39:06 2006
> State : active
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 683f2f6e - correct
> Events : 0.8831997
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 1 8 18 1 active sync /dev/sdb2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> 4 4 8 50 4 spare /dev/sdd2
>
>
> sdc2:
> /dev/sdc2:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
> Creation Time : Thu Jun 1 21:13:58 2006
> Raid Level : raid5
> Device Size : 390555904 (372.46 GiB 399.93 GB)
> Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
> Raid Devices : 4
> Total Devices : 4
> Preferred Minor : 0
>
> Update Time : Sun Oct 22 23:39:06 2006
> State : active
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 683f2f80 - correct
> Events : 0.8831997
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 2 8 34 2 active sync /dev/sdc2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> 4 4 8 50 4 spare /dev/sdd2
>
>
> sdd2:
> /dev/sdd2:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
> Creation Time : Thu Jun 1 21:13:58 2006
> Raid Level : raid5
> Device Size : 390555904 (372.46 GiB 399.93 GB)
> Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
> Raid Devices : 4
> Total Devices : 4
> Preferred Minor : 0
>
> Update Time : Sun Oct 22 23:39:06 2006
> State : active
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 683f2fbf - correct
> Events : 0.8831997
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 3 8 50 -1 sync /dev/sdd2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 8 50 -1 sync /dev/sdd2
> 4 4 8 50 4 spare /dev/sdd2
>
>
>
> Does anyone have any idea how to get this array back into good shape?
> I'm not sure why it thinks sdd2 should be a spare, or how to get it back
> to being a regular disk.
>
> I would appreciate any help you can offer. (Also, am I right in thinking
> my data is still good? I should still have 3 of the 4 disks working fine,
> at any rate.)
>
> Thanks,
> Eric
>
If it helps, I think I may have made the mistake of trying to readd (mdadm
--add) the fourth device to the array - probably related to why it thinks
it's a spare.
Knowing that, maybe there is a clean way to fix it all?
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Need help recovering a raid5 array
2006-10-23 15:21 Need help recovering a raid5 array eharney
@ 2006-10-24 8:49 ` David Greaves
0 siblings, 0 replies; 3+ messages in thread
From: David Greaves @ 2006-10-24 8:49 UTC (permalink / raw)
To: eharney; +Cc: linux-raid
eharney@CLEMSON.EDU wrote:
> Hello all,
Hi
First off, don't do anything else without reading up or talking on here :)
The list archive has got a lot of good material - 'help' is usually a good
search term!!!
>
> I had a disk fail in a raid 5 array (4 disk array, no spares), and am
> having trouble recovering it. I believe my data is still safe, but I
> cannot tell what is going wrong here.
There's some useful stuff but always include:
* kernel version
* mdadm version
* relevant dmesg or similar output
What went wrong?
Did /dev/sdd fail? If so then why are you adding it back to the array? Or is
this now a replacement?
You should be OK - I'll reply quickly now and see if I can make some suggestions
later (or sooner).
David
>
> When I try to rebuild the array "mdadm --assemble /dev/md0 /dev/sda2
> /dev/sdb2 /dev/sdc2 /dev/sdd2" I see "failed to RUN_ARRAY /dev/md0:
> Input/output error".
>
> dmesg shows the following:
> md: bind<sdb2>
> md: bind<sdc2>
> md: bind<sdd2>
> md: bind<sda2>
> md: md0: raid array is not clean -- starting background reconstruction
> raid5: device sda2 operational as raid disk 0
> raid5: device sdc2 operational as raid disk 2
> raid5: device sdb2 operational as raid disk 1
> raid5: cannot start dirty degraded array for md0
> RAID5 conf printout:
> --- rd:4 wd:3 fd:1
> disk 0, o:1, dev:sda2
> disk 1, o:1, dev:sdb2
> disk 2, o:1, dev:sdc2
> raid5: failed to run raid set md0
> md: pers->run() failed ...
>
>
>
> /proc mdstat shows:
> md0 : inactive sda2[0] sdd2[3](S) sdc2[2] sdb2[1]
>
> This seems wrong, as sdd2 should not be a spare - I want it to be the
> fourth disk.
>
>
> The output of mdadm -E for each disk is as follows:
> sda2:
> /dev/sda2:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
> Creation Time : Thu Jun 1 21:13:58 2006
> Raid Level : raid5
> Device Size : 390555904 (372.46 GiB 399.93 GB)
> Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
> Raid Devices : 4
> Total Devices : 4
> Preferred Minor : 0
>
> Update Time : Sun Oct 22 23:39:06 2006
> State : active
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 683f2f5c - correct
> Events : 0.8831997
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 0 8 2 0 active sync /dev/sda2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> 4 4 8 50 4 spare /dev/sdd2
>
>
> sdb2:
> /dev/sdb2:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
> Creation Time : Thu Jun 1 21:13:58 2006
> Raid Level : raid5
> Device Size : 390555904 (372.46 GiB 399.93 GB)
> Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
> Raid Devices : 4
> Total Devices : 4
> Preferred Minor : 0
>
> Update Time : Sun Oct 22 23:39:06 2006
> State : active
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 683f2f6e - correct
> Events : 0.8831997
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 1 8 18 1 active sync /dev/sdb2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> 4 4 8 50 4 spare /dev/sdd2
>
>
> sdc2:
> /dev/sdc2:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
> Creation Time : Thu Jun 1 21:13:58 2006
> Raid Level : raid5
> Device Size : 390555904 (372.46 GiB 399.93 GB)
> Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
> Raid Devices : 4
> Total Devices : 4
> Preferred Minor : 0
>
> Update Time : Sun Oct 22 23:39:06 2006
> State : active
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 683f2f80 - correct
> Events : 0.8831997
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 2 8 34 2 active sync /dev/sdc2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 0 0 3 faulty removed
> 4 4 8 50 4 spare /dev/sdd2
>
>
> sdd2:
> /dev/sdd2:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : c50a81fc:ef4323e6:438a7cb1:25ae35e5
> Creation Time : Thu Jun 1 21:13:58 2006
> Raid Level : raid5
> Device Size : 390555904 (372.46 GiB 399.93 GB)
> Array Size : 1171667712 (1117.39 GiB 1199.79 GB)
> Raid Devices : 4
> Total Devices : 4
> Preferred Minor : 0
>
> Update Time : Sun Oct 22 23:39:06 2006
> State : active
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 0
> Spare Devices : 1
> Checksum : 683f2fbf - correct
> Events : 0.8831997
>
> Layout : left-symmetric
> Chunk Size : 256K
>
> Number Major Minor RaidDevice State
> this 3 8 50 -1 sync /dev/sdd2
>
> 0 0 8 2 0 active sync /dev/sda2
> 1 1 8 18 1 active sync /dev/sdb2
> 2 2 8 34 2 active sync /dev/sdc2
> 3 3 8 50 -1 sync /dev/sdd2
> 4 4 8 50 4 spare /dev/sdd2
>
>
>
> Does anyone have any idea how to get this array back into good shape?
> I'm not sure why it thinks sdd2 should be a spare, or how to get it back
> to being a regular disk.
>
> I would appreciate any help you can offer. (Also, am I right in thinking
> my data is still good? I should still have 3 of the 4 disks working fine,
> at any rate.)
>
> Thanks,
> Eric
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2006-10-24 8:49 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2006-10-23 15:21 Need help recovering a raid5 array eharney
2006-10-24 8:49 ` David Greaves
[not found] <44029.130.127.44.164.1161616905.squirrel@wm.clemson.edu>
2006-10-23 17:49 ` eharney
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).