* Help recovering RAID6 failure
@ 2008-12-15 22:03 Kevin Shanahan
2008-12-15 22:15 ` Neil Brown
0 siblings, 1 reply; 7+ messages in thread
From: Kevin Shanahan @ 2008-12-15 22:03 UTC (permalink / raw)
To: linux-raid
Hi,
My server seems to have had a bad night, and so I'm not having the
best morning. Overnight three drives failed in my RAID6 array. Worse,
that's on top of one failure I already knew about, so 4 drives in
total.
I put this down to problems with the controller - I had been using
3ware 9650 controllers (in JBOD mode). This morning I've loaded the
drives into another server connected to the on board Intel AHCI SATA
ports and an extra Sil3132 controller.
My first naive attempt was to try re-assembling the array from the
eight most up-to-date member devices:
hermes:~# mdadm -A /dev/md5 /dev/sd[cefghijk]1
mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
I assume that's not an I/O error with the underlying device? I can't
see anything indicating that dmesg/syslog.
Output from "mdadm -E /dev/sd[cdefghijkl]1" included below.
Any advice would be much appreciated.
(and now, back to scouring the mail archives for clues).
Cheers,
Kevin.
--
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Tue Dec 16 05:35:46 2008
State : clean
Internal Bitmap : present
Active Devices : 6
Working Devices : 6
Failed Devices : 3
Spare Devices : 0
Checksum : e3df9eea - correct
Events : 176870
Chunk Size : 64K
Number Major Minor RaidDevice State
this 8 8 129 8 active sync /dev/sdi1
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 0 0 7 faulty removed
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
/dev/sdd1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Thu Nov 27 09:20:39 2008
State : clean
Internal Bitmap : present
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Checksum : e3c42f8b - correct
Events : 91974
Chunk Size : 64K
Number Major Minor RaidDevice State
this 9 8 145 9 active sync /dev/sdj1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
8 8 8 129 8 active sync /dev/sdi1
9 9 8 145 9 active sync /dev/sdj1
/dev/sde1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Tue Dec 16 00:28:14 2008
State : active
Internal Bitmap : present
Active Devices : 8
Working Devices : 8
Failed Devices : 2
Spare Devices : 0
Checksum : e3dca329 - correct
Events : 176870
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 1 0 active sync /dev/sda1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 0 0 7 faulty removed
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
/dev/sdf1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Tue Dec 16 00:29:00 2008
State : active
Internal Bitmap : present
Active Devices : 7
Working Devices : 7
Failed Devices : 2
Spare Devices : 0
Checksum : e3dca382 - correct
Events : 176870
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 17 1 active sync /dev/sdb1
0 0 0 0 0 removed
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 0 0 7 faulty removed
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
/dev/sdg1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Tue Dec 16 05:35:46 2008
State : clean
Internal Bitmap : present
Active Devices : 6
Working Devices : 6
Failed Devices : 3
Spare Devices : 0
Checksum : e3df9e7e - correct
Events : 176870
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 33 2 active sync /dev/sdc1
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 0 0 7 faulty removed
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
/dev/sdh1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Tue Dec 16 05:35:46 2008
State : clean
Internal Bitmap : present
Active Devices : 6
Working Devices : 6
Failed Devices : 3
Spare Devices : 0
Checksum : e3df9ec6 - correct
Events : 176870
Chunk Size : 64K
Number Major Minor RaidDevice State
this 6 8 97 6 active sync /dev/sdg1
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 0 0 7 faulty removed
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
/dev/sdi1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Tue Dec 16 05:35:46 2008
State : clean
Internal Bitmap : present
Active Devices : 6
Working Devices : 6
Failed Devices : 3
Spare Devices : 0
Checksum : e3df9eb4 - correct
Events : 176870
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 81 5 active sync /dev/sdf1
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 0 0 7 faulty removed
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
/dev/sdj1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Tue Dec 16 05:35:46 2008
State : clean
Internal Bitmap : present
Active Devices : 6
Working Devices : 6
Failed Devices : 3
Spare Devices : 0
Checksum : e3df9ea2 - correct
Events : 176870
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 65 4 active sync /dev/sde1
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 0 0 7 faulty removed
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
/dev/sdk1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Tue Dec 16 05:35:46 2008
State : clean
Internal Bitmap : present
Active Devices : 6
Working Devices : 6
Failed Devices : 3
Spare Devices : 0
Checksum : e3df9e90 - correct
Events : 176870
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 49 3 active sync /dev/sdd1
0 0 0 0 0 removed
1 1 0 0 1 faulty removed
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 0 0 7 faulty removed
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
/dev/sdl1:
Magic : a92b4efc
Version : 00.90.00
UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
Creation Time : Sun Jul 27 15:03:44 2008
Raid Level : raid6
Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 5
Update Time : Mon Dec 15 22:36:32 2008
State : active
Internal Bitmap : present
Active Devices : 9
Working Devices : 9
Failed Devices : 1
Spare Devices : 0
Checksum : e3dc8525 - correct
Events : 175787
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 8 113 7 active sync /dev/sdh1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
8 8 8 129 8 active sync /dev/sdi1
9 9 0 0 9 faulty removed
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Help recovering RAID6 failure
2008-12-15 22:03 Help recovering RAID6 failure Kevin Shanahan
@ 2008-12-15 22:15 ` Neil Brown
2008-12-15 22:25 ` Kevin Shanahan
0 siblings, 1 reply; 7+ messages in thread
From: Neil Brown @ 2008-12-15 22:15 UTC (permalink / raw)
To: Kevin Shanahan; +Cc: linux-raid
On Tuesday December 16, kmshanah@disenchant.net wrote:
> Hi,
>
> My server seems to have had a bad night, and so I'm not having the
> best morning. Overnight three drives failed in my RAID6 array. Worse,
> that's on top of one failure I already knew about, so 4 drives in
> total.
>
> I put this down to problems with the controller - I had been using
> 3ware 9650 controllers (in JBOD mode). This morning I've loaded the
> drives into another server connected to the on board Intel AHCI SATA
> ports and an extra Sil3132 controller.
>
> My first naive attempt was to try re-assembling the array from the
> eight most up-to-date member devices:
>
> hermes:~# mdadm -A /dev/md5 /dev/sd[cefghijk]1
> mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
Try adding --force to over-ride apparent errors, and --verbose to help
you see what is happening.
NeilBrown
>
> I assume that's not an I/O error with the underlying device? I can't
> see anything indicating that dmesg/syslog.
>
> Output from "mdadm -E /dev/sd[cdefghijkl]1" included below.
>
> Any advice would be much appreciated.
>
> (and now, back to scouring the mail archives for clues).
>
> Cheers,
> Kevin.
> --
>
> /dev/sdc1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9eea - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 8 8 129 8 active sync /dev/sdi1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdd1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Thu Nov 27 09:20:39 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : e3c42f8b - correct
> Events : 91974
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 9 8 145 9 active sync /dev/sdj1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 8 113 7 active sync /dev/sdh1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 145 9 active sync /dev/sdj1
> /dev/sde1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 00:28:14 2008
> State : active
> Internal Bitmap : present
> Active Devices : 8
> Working Devices : 8
> Failed Devices : 2
> Spare Devices : 0
> Checksum : e3dca329 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 0 8 1 0 active sync /dev/sda1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdf1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 00:29:00 2008
> State : active
> Internal Bitmap : present
> Active Devices : 7
> Working Devices : 7
> Failed Devices : 2
> Spare Devices : 0
> Checksum : e3dca382 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 1 8 17 1 active sync /dev/sdb1
>
> 0 0 0 0 0 removed
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdg1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9e7e - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 2 8 33 2 active sync /dev/sdc1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdh1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9ec6 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 6 8 97 6 active sync /dev/sdg1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdi1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9eb4 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 5 8 81 5 active sync /dev/sdf1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdj1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9ea2 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 4 8 65 4 active sync /dev/sde1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdk1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9e90 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 3 8 49 3 active sync /dev/sdd1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdl1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Mon Dec 15 22:36:32 2008
> State : active
> Internal Bitmap : present
> Active Devices : 9
> Working Devices : 9
> Failed Devices : 1
> Spare Devices : 0
> Checksum : e3dc8525 - correct
> Events : 175787
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 7 8 113 7 active sync /dev/sdh1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 8 113 7 active sync /dev/sdh1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Help recovering RAID6 failure
2008-12-15 22:15 ` Neil Brown
@ 2008-12-15 22:25 ` Kevin Shanahan
2008-12-15 22:37 ` Kevin Shanahan
0 siblings, 1 reply; 7+ messages in thread
From: Kevin Shanahan @ 2008-12-15 22:25 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On Tue, Dec 16, 2008 at 09:15:17AM +1100, Neil Brown wrote:
> On Tuesday December 16, kmshanah@disenchant.net wrote:
> > hermes:~# mdadm -A /dev/md5 /dev/sd[cefghijk]1
> > mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
>
> Try adding --force to over-ride apparent errors, and --verbose to help
> you see what is happening.
Hrm, force doesn't seem to help. Here's the verbose output:
hermes:~# mdadm -A --force --verbose /dev/md5 /dev/sd[cefghijk]1
mdadm: looking for devices for /dev/md5
mdadm: /dev/sdc1 is identified as a member of /dev/md5, slot 8.
mdadm: /dev/sde1 is identified as a member of /dev/md5, slot 0.
mdadm: /dev/sdf1 is identified as a member of /dev/md5, slot 1.
mdadm: /dev/sdg1 is identified as a member of /dev/md5, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md5, slot 6.
mdadm: /dev/sdi1 is identified as a member of /dev/md5, slot 5.
mdadm: /dev/sdj1 is identified as a member of /dev/md5, slot 4.
mdadm: /dev/sdk1 is identified as a member of /dev/md5, slot 3.
mdadm: added /dev/sdf1 to /dev/md5 as 1
mdadm: added /dev/sdg1 to /dev/md5 as 2
mdadm: added /dev/sdk1 to /dev/md5 as 3
mdadm: added /dev/sdj1 to /dev/md5 as 4
mdadm: added /dev/sdi1 to /dev/md5 as 5
mdadm: added /dev/sdh1 to /dev/md5 as 6
mdadm: no uptodate device for slot 7 of /dev/md5
mdadm: added /dev/sdc1 to /dev/md5 as 8
mdadm: no uptodate device for slot 9 of /dev/md5
mdadm: added /dev/sde1 to /dev/md5 as 0
mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
I don't know what to make of that. It looks like it's got 8/10
devices, which should be enough to re-assemble in degraded mode.
/proc/mdstat shows:
md5 : inactive sde1[0] sdc1[8] sdh1[6] sdi1[5] sdj1[4] sdk1[3] sdg1[2] sdf1[1]
7814079488 blocks
Cheers,
Kevin.
> > /dev/sdc1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Tue Dec 16 05:35:46 2008
> > State : clean
> > Internal Bitmap : present
> > Active Devices : 6
> > Working Devices : 6
> > Failed Devices : 3
> > Spare Devices : 0
> > Checksum : e3df9eea - correct
> > Events : 176870
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 8 8 129 8 active sync /dev/sdi1
> >
> > 0 0 0 0 0 removed
> > 1 1 0 0 1 faulty removed
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 0 0 7 faulty removed
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > /dev/sdd1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Thu Nov 27 09:20:39 2008
> > State : clean
> > Internal Bitmap : present
> > Active Devices : 10
> > Working Devices : 10
> > Failed Devices : 0
> > Spare Devices : 0
> > Checksum : e3c42f8b - correct
> > Events : 91974
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 9 8 145 9 active sync /dev/sdj1
> >
> > 0 0 8 1 0 active sync /dev/sda1
> > 1 1 8 17 1 active sync /dev/sdb1
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 8 113 7 active sync /dev/sdh1
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 8 145 9 active sync /dev/sdj1
> > /dev/sde1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Tue Dec 16 00:28:14 2008
> > State : active
> > Internal Bitmap : present
> > Active Devices : 8
> > Working Devices : 8
> > Failed Devices : 2
> > Spare Devices : 0
> > Checksum : e3dca329 - correct
> > Events : 176870
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 0 8 1 0 active sync /dev/sda1
> >
> > 0 0 8 1 0 active sync /dev/sda1
> > 1 1 8 17 1 active sync /dev/sdb1
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 0 0 7 faulty removed
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > /dev/sdf1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Tue Dec 16 00:29:00 2008
> > State : active
> > Internal Bitmap : present
> > Active Devices : 7
> > Working Devices : 7
> > Failed Devices : 2
> > Spare Devices : 0
> > Checksum : e3dca382 - correct
> > Events : 176870
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 1 8 17 1 active sync /dev/sdb1
> >
> > 0 0 0 0 0 removed
> > 1 1 8 17 1 active sync /dev/sdb1
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 0 0 7 faulty removed
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > /dev/sdg1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Tue Dec 16 05:35:46 2008
> > State : clean
> > Internal Bitmap : present
> > Active Devices : 6
> > Working Devices : 6
> > Failed Devices : 3
> > Spare Devices : 0
> > Checksum : e3df9e7e - correct
> > Events : 176870
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 2 8 33 2 active sync /dev/sdc1
> >
> > 0 0 0 0 0 removed
> > 1 1 0 0 1 faulty removed
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 0 0 7 faulty removed
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > /dev/sdh1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Tue Dec 16 05:35:46 2008
> > State : clean
> > Internal Bitmap : present
> > Active Devices : 6
> > Working Devices : 6
> > Failed Devices : 3
> > Spare Devices : 0
> > Checksum : e3df9ec6 - correct
> > Events : 176870
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 6 8 97 6 active sync /dev/sdg1
> >
> > 0 0 0 0 0 removed
> > 1 1 0 0 1 faulty removed
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 0 0 7 faulty removed
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > /dev/sdi1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Tue Dec 16 05:35:46 2008
> > State : clean
> > Internal Bitmap : present
> > Active Devices : 6
> > Working Devices : 6
> > Failed Devices : 3
> > Spare Devices : 0
> > Checksum : e3df9eb4 - correct
> > Events : 176870
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 5 8 81 5 active sync /dev/sdf1
> >
> > 0 0 0 0 0 removed
> > 1 1 0 0 1 faulty removed
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 0 0 7 faulty removed
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > /dev/sdj1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Tue Dec 16 05:35:46 2008
> > State : clean
> > Internal Bitmap : present
> > Active Devices : 6
> > Working Devices : 6
> > Failed Devices : 3
> > Spare Devices : 0
> > Checksum : e3df9ea2 - correct
> > Events : 176870
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 4 8 65 4 active sync /dev/sde1
> >
> > 0 0 0 0 0 removed
> > 1 1 0 0 1 faulty removed
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 0 0 7 faulty removed
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > /dev/sdk1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Tue Dec 16 05:35:46 2008
> > State : clean
> > Internal Bitmap : present
> > Active Devices : 6
> > Working Devices : 6
> > Failed Devices : 3
> > Spare Devices : 0
> > Checksum : e3df9e90 - correct
> > Events : 176870
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 3 8 49 3 active sync /dev/sdd1
> >
> > 0 0 0 0 0 removed
> > 1 1 0 0 1 faulty removed
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 0 0 7 faulty removed
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > /dev/sdl1:
> > Magic : a92b4efc
> > Version : 00.90.00
> > UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> > Creation Time : Sun Jul 27 15:03:44 2008
> > Raid Level : raid6
> > Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> > Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> > Raid Devices : 10
> > Total Devices : 10
> > Preferred Minor : 5
> >
> > Update Time : Mon Dec 15 22:36:32 2008
> > State : active
> > Internal Bitmap : present
> > Active Devices : 9
> > Working Devices : 9
> > Failed Devices : 1
> > Spare Devices : 0
> > Checksum : e3dc8525 - correct
> > Events : 175787
> >
> > Chunk Size : 64K
> >
> > Number Major Minor RaidDevice State
> > this 7 8 113 7 active sync /dev/sdh1
> >
> > 0 0 8 1 0 active sync /dev/sda1
> > 1 1 8 17 1 active sync /dev/sdb1
> > 2 2 8 33 2 active sync /dev/sdc1
> > 3 3 8 49 3 active sync /dev/sdd1
> > 4 4 8 65 4 active sync /dev/sde1
> > 5 5 8 81 5 active sync /dev/sdf1
> > 6 6 8 97 6 active sync /dev/sdg1
> > 7 7 8 113 7 active sync /dev/sdh1
> > 8 8 8 129 8 active sync /dev/sdi1
> > 9 9 0 0 9 faulty removed
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Help recovering RAID6 failure
2008-12-15 22:25 ` Kevin Shanahan
@ 2008-12-15 22:37 ` Kevin Shanahan
2008-12-15 22:39 ` Kevin Shanahan
2008-12-15 23:05 ` Neil Brown
0 siblings, 2 replies; 7+ messages in thread
From: Kevin Shanahan @ 2008-12-15 22:37 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On Tue, Dec 16, 2008 at 08:55:22AM +1030, Kevin Shanahan wrote:
> On Tue, Dec 16, 2008 at 09:15:17AM +1100, Neil Brown wrote:
> > On Tuesday December 16, kmshanah@disenchant.net wrote:
> > > hermes:~# mdadm -A /dev/md5 /dev/sd[cefghijk]1
> > > mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
> >
> > Try adding --force to over-ride apparent errors, and --verbose to help
> > you see what is happening.
>
> Hrm, force doesn't seem to help. Here's the verbose output:
>
> hermes:~# mdadm -A --force --verbose /dev/md5 /dev/sd[cefghijk]1
> mdadm: looking for devices for /dev/md5
> mdadm: /dev/sdc1 is identified as a member of /dev/md5, slot 8.
> mdadm: /dev/sde1 is identified as a member of /dev/md5, slot 0.
> mdadm: /dev/sdf1 is identified as a member of /dev/md5, slot 1.
> mdadm: /dev/sdg1 is identified as a member of /dev/md5, slot 2.
> mdadm: /dev/sdh1 is identified as a member of /dev/md5, slot 6.
> mdadm: /dev/sdi1 is identified as a member of /dev/md5, slot 5.
> mdadm: /dev/sdj1 is identified as a member of /dev/md5, slot 4.
> mdadm: /dev/sdk1 is identified as a member of /dev/md5, slot 3.
> mdadm: added /dev/sdf1 to /dev/md5 as 1
> mdadm: added /dev/sdg1 to /dev/md5 as 2
> mdadm: added /dev/sdk1 to /dev/md5 as 3
> mdadm: added /dev/sdj1 to /dev/md5 as 4
> mdadm: added /dev/sdi1 to /dev/md5 as 5
> mdadm: added /dev/sdh1 to /dev/md5 as 6
> mdadm: no uptodate device for slot 7 of /dev/md5
> mdadm: added /dev/sdc1 to /dev/md5 as 8
> mdadm: no uptodate device for slot 9 of /dev/md5
> mdadm: added /dev/sde1 to /dev/md5 as 0
> mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
Oh, and here's what gets added to dmesg after running that command:
md: bind<sdf1>
md: bind<sdg1>
md: bind<sdk1>
md: bind<sdj1>
md: bind<sdi1>
md: bind<sdh1>
md: bind<sdc1>
md: bind<sde1>
md: md5: raid array is not clean -- starting background reconstruction
raid5: device sde1 operational as raid disk 0
raid5: device sdc1 operational as raid disk 8
raid5: device sdh1 operational as raid disk 6
raid5: device sdi1 operational as raid disk 5
raid5: device sdj1 operational as raid disk 4
raid5: device sdk1 operational as raid disk 3
raid5: device sdg1 operational as raid disk 2
raid5: device sdf1 operational as raid disk 1
raid5: cannot start dirty degraded array for md5
RAID5 conf printout:
--- rd:10 wd:8
disk 0, o:1, dev:sde1
disk 1, o:1, dev:sdf1
disk 2, o:1, dev:sdg1
disk 3, o:1, dev:sdk1
disk 4, o:1, dev:sdj1
disk 5, o:1, dev:sdi1
disk 6, o:1, dev:sdh1
disk 8, o:1, dev:sdc1
raid5: failed to run raid set md5
md: pers->run() failed ...
This is run on Linux 2.6.26.9, mdadm 2.6.7.1 (Debian)
Cheers,
Kevin.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Help recovering RAID6 failure
2008-12-15 22:37 ` Kevin Shanahan
@ 2008-12-15 22:39 ` Kevin Shanahan
2008-12-15 23:05 ` Neil Brown
1 sibling, 0 replies; 7+ messages in thread
From: Kevin Shanahan @ 2008-12-15 22:39 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On Tue, Dec 16, 2008 at 09:07:53AM +1030, Kevin Shanahan wrote:
> On Tue, Dec 16, 2008 at 08:55:22AM +1030, Kevin Shanahan wrote:
> > On Tue, Dec 16, 2008 at 09:15:17AM +1100, Neil Brown wrote:
> > > On Tuesday December 16, kmshanah@disenchant.net wrote:
> > > > hermes:~# mdadm -A /dev/md5 /dev/sd[cefghijk]1
> > > > mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
> > >
> > > Try adding --force to over-ride apparent errors, and --verbose to help
> > > you see what is happening.
> >
> > Hrm, force doesn't seem to help. Here's the verbose output:
> >
> > hermes:~# mdadm -A --force --verbose /dev/md5 /dev/sd[cefghijk]1
> > mdadm: looking for devices for /dev/md5
> > mdadm: /dev/sdc1 is identified as a member of /dev/md5, slot 8.
> > mdadm: /dev/sde1 is identified as a member of /dev/md5, slot 0.
> > mdadm: /dev/sdf1 is identified as a member of /dev/md5, slot 1.
> > mdadm: /dev/sdg1 is identified as a member of /dev/md5, slot 2.
> > mdadm: /dev/sdh1 is identified as a member of /dev/md5, slot 6.
> > mdadm: /dev/sdi1 is identified as a member of /dev/md5, slot 5.
> > mdadm: /dev/sdj1 is identified as a member of /dev/md5, slot 4.
> > mdadm: /dev/sdk1 is identified as a member of /dev/md5, slot 3.
> > mdadm: added /dev/sdf1 to /dev/md5 as 1
> > mdadm: added /dev/sdg1 to /dev/md5 as 2
> > mdadm: added /dev/sdk1 to /dev/md5 as 3
> > mdadm: added /dev/sdj1 to /dev/md5 as 4
> > mdadm: added /dev/sdi1 to /dev/md5 as 5
> > mdadm: added /dev/sdh1 to /dev/md5 as 6
> > mdadm: no uptodate device for slot 7 of /dev/md5
> > mdadm: added /dev/sdc1 to /dev/md5 as 8
> > mdadm: no uptodate device for slot 9 of /dev/md5
> > mdadm: added /dev/sde1 to /dev/md5 as 0
> > mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
>
> Oh, and here's what gets added to dmesg after running that command:
>
> md: bind<sdf1>
> md: bind<sdg1>
> md: bind<sdk1>
> md: bind<sdj1>
> md: bind<sdi1>
> md: bind<sdh1>
> md: bind<sdc1>
> md: bind<sde1>
> md: md5: raid array is not clean -- starting background reconstruction
> raid5: device sde1 operational as raid disk 0
> raid5: device sdc1 operational as raid disk 8
> raid5: device sdh1 operational as raid disk 6
> raid5: device sdi1 operational as raid disk 5
> raid5: device sdj1 operational as raid disk 4
> raid5: device sdk1 operational as raid disk 3
> raid5: device sdg1 operational as raid disk 2
> raid5: device sdf1 operational as raid disk 1
> raid5: cannot start dirty degraded array for md5
> RAID5 conf printout:
> --- rd:10 wd:8
> disk 0, o:1, dev:sde1
> disk 1, o:1, dev:sdf1
> disk 2, o:1, dev:sdg1
> disk 3, o:1, dev:sdk1
> disk 4, o:1, dev:sdj1
> disk 5, o:1, dev:sdi1
> disk 6, o:1, dev:sdh1
> disk 8, o:1, dev:sdc1
> raid5: failed to run raid set md5
> md: pers->run() failed ...
>
> This is run on Linux 2.6.26.9, mdadm 2.6.7.1 (Debian)
Gah, that's 2.6.27.9. Sorry.
> Cheers,
> Kevin.
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Help recovering RAID6 failure
2008-12-15 22:37 ` Kevin Shanahan
2008-12-15 22:39 ` Kevin Shanahan
@ 2008-12-15 23:05 ` Neil Brown
2008-12-15 23:18 ` Kevin Shanahan
1 sibling, 1 reply; 7+ messages in thread
From: Neil Brown @ 2008-12-15 23:05 UTC (permalink / raw)
To: Kevin Shanahan; +Cc: linux-raid
On Tuesday December 16, kmshanah@disenchant.net wrote:
>
> Oh, and here's what gets added to dmesg after running that command:
>
> raid5: cannot start dirty degraded array for md5
I thought that might be the case. --force is meant to fix that -
remove the 'dirty' flag from the array.
>
> This is run on Linux 2.6.26.9, mdadm 2.6.7.1 (Debian)
Hmm.. and there goes that theory. There was a bug in mdadm prior to
2.6 which caused --force not to work for raid6 with 2 drives missing.
It looks like some of your devices are marks 'clean' and some are
'active'. mdadm is noticing one that is 'clean' and not bothering to
mark the others as 'clean'. The kernel is seeing one that is 'active'
and complaining.
The devices that are 'active' are sd[efl]1. Maybe if you list one of
those last it will work.
e.g.
mdadm -A --force --verbose /dev/md5 /dev/sd[cfghijk]1 /dev/sde1
If not, try listing it first.
I'll try to fix mdadm so that it gets this right.
Thanks,
NeilBrown
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: Help recovering RAID6 failure
2008-12-15 23:05 ` Neil Brown
@ 2008-12-15 23:18 ` Kevin Shanahan
0 siblings, 0 replies; 7+ messages in thread
From: Kevin Shanahan @ 2008-12-15 23:18 UTC (permalink / raw)
To: Neil Brown; +Cc: linux-raid
On Tue, Dec 16, 2008 at 10:05:19AM +1100, Neil Brown wrote:
> On Tuesday December 16, kmshanah@disenchant.net wrote:
> >
> > Oh, and here's what gets added to dmesg after running that command:
> >
>
> > raid5: cannot start dirty degraded array for md5
>
> I thought that might be the case. --force is meant to fix that -
> remove the 'dirty' flag from the array.
> >
> > This is run on Linux 2.6.26.9, mdadm 2.6.7.1 (Debian)
>
> Hmm.. and there goes that theory. There was a bug in mdadm prior to
> 2.6 which caused --force not to work for raid6 with 2 drives missing.
>
> It looks like some of your devices are marks 'clean' and some are
> 'active'. mdadm is noticing one that is 'clean' and not bothering to
> mark the others as 'clean'. The kernel is seeing one that is 'active'
> and complaining.
>
> The devices that are 'active' are sd[efl]1. Maybe if you list one of
> those last it will work.
> e.g.
>
> mdadm -A --force --verbose /dev/md5 /dev/sd[cfghijk]1 /dev/sde1
>
> If not, try listing it first.
Aha, you're a life saver Neil:
hermes:~# mdadm -A --force --verbose /dev/md5 /dev/sd[cfghijk]1 /dev/sde1
mdadm: looking for devices for /dev/md5
mdadm: /dev/sdc1 is identified as a member of /dev/md5, slot 8.
mdadm: /dev/sdf1 is identified as a member of /dev/md5, slot 1.
mdadm: /dev/sdg1 is identified as a member of /dev/md5, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md5, slot 6.
mdadm: /dev/sdi1 is identified as a member of /dev/md5, slot 5.
mdadm: /dev/sdj1 is identified as a member of /dev/md5, slot 4.
mdadm: /dev/sdk1 is identified as a member of /dev/md5, slot 3.
mdadm: /dev/sde1 is identified as a member of /dev/md5, slot 0.
mdadm: added /dev/sdf1 to /dev/md5 as 1
mdadm: added /dev/sdg1 to /dev/md5 as 2
mdadm: added /dev/sdk1 to /dev/md5 as 3
mdadm: added /dev/sdj1 to /dev/md5 as 4
mdadm: added /dev/sdi1 to /dev/md5 as 5
mdadm: added /dev/sdh1 to /dev/md5 as 6
mdadm: no uptodate device for slot 7 of /dev/md5
mdadm: added /dev/sdc1 to /dev/md5 as 8
mdadm: no uptodate device for slot 9 of /dev/md5
mdadm: added /dev/sde1 to /dev/md5 as 0
mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
hermes:~# mdadm -S /dev/md5
mdadm: stopped /dev/md5
hermes:~# mdadm -A --force --verbose /dev/md5 /dev/sde1 /dev/sd[cfghijk]1
mdadm: looking for devices for /dev/md5
mdadm: /dev/sde1 is identified as a member of /dev/md5, slot 0.
mdadm: /dev/sdc1 is identified as a member of /dev/md5, slot 8.
mdadm: /dev/sdf1 is identified as a member of /dev/md5, slot 1.
mdadm: /dev/sdg1 is identified as a member of /dev/md5, slot 2.
mdadm: /dev/sdh1 is identified as a member of /dev/md5, slot 6.
mdadm: /dev/sdi1 is identified as a member of /dev/md5, slot 5.
mdadm: /dev/sdj1 is identified as a member of /dev/md5, slot 4.
mdadm: /dev/sdk1 is identified as a member of /dev/md5, slot 3.
mdadm: added /dev/sdf1 to /dev/md5 as 1
mdadm: added /dev/sdg1 to /dev/md5 as 2
mdadm: added /dev/sdk1 to /dev/md5 as 3
mdadm: added /dev/sdj1 to /dev/md5 as 4
mdadm: added /dev/sdi1 to /dev/md5 as 5
mdadm: added /dev/sdh1 to /dev/md5 as 6
mdadm: no uptodate device for slot 7 of /dev/md5
mdadm: added /dev/sdc1 to /dev/md5 as 8
mdadm: no uptodate device for slot 9 of /dev/md5
mdadm: added /dev/sde1 to /dev/md5 as 0
mdadm: /dev/md5 has been started with 8 drives (out of 10).
Now to check my data is still okay.
> I'll try to fix mdadm so that it gets this right.
Cool - glad it wasn't just a lack of coffee on my part ;)
Cheers,
Kevin.
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2008-12-15 23:18 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-12-15 22:03 Help recovering RAID6 failure Kevin Shanahan
2008-12-15 22:15 ` Neil Brown
2008-12-15 22:25 ` Kevin Shanahan
2008-12-15 22:37 ` Kevin Shanahan
2008-12-15 22:39 ` Kevin Shanahan
2008-12-15 23:05 ` Neil Brown
2008-12-15 23:18 ` Kevin Shanahan
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).