* recovery from multiple failures
@ 2012-01-08 20:12 Keith Keller
2012-01-09 0:11 ` Phil Turmel
0 siblings, 1 reply; 8+ messages in thread
From: Keith Keller @ 2012-01-08 20:12 UTC (permalink / raw)
To: linux-raid
Hi all,
I recently had an experience very similar to what's posted on the wiki:
https://raid.wiki.kernel.org/articles/r/a/i/RAID_Recovery_d376.html
From what I can piece together from the logs, my controller went a
little crazy, and dropped a whole bunch of drives in a span of about
10 minutes. I attempted to look at the output of mdadm --examine, as
documented in the wiki, but the format appears to have changed, so I
am unclear where to go next.
I'm going to include the entire mdadm --examine output below, but as I
was looking at it, I was wondering if the analogous scenario to the wiki
situation is to look at the array slots:
$ grep Slot raid.status |cut -f1 -d '('
Array Slot : 0
Array Slot : 0
Array Slot : 13
Array Slot : 4
Array Slot : 10
Array Slot : 6
Array Slot : 7
Array Slot : 9
Array Slot : 8
Array Slot : 11
Array Slot : 2
Array Slot : 4
Array Slot : 12
There are two separate arrays on this box; the problematic one is
24363b01:90deb9b5:4b51e5df:68b8b6ea. Will I be able to recover this
array with an appropriate mdadm --create --assume-clean command, and if
so, how would I go about determining the correct order in which to
specify the drives? The big confusing part for me is, as a 9 device
RAID6, I'd expect to see device slots 0-8, but here I see slots 10
through 13, and I am unclear how to get the order exactly right.
My mdadm --examine (which I have also saved separately) is below. If
you need any more information let me know. Thanks!
--keith
/dev/sdb1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : active
Device UUID : 7abdc565:f4f29c91:c92d41be:c4fe1fa9
Update Time : Sat Jan 7 22:50:29 2012
Checksum : 55d050cb - correct
Events : 106059
Chunk Size : 64K
Array Slot : 0 (0, failed, failed, failed, 4, failed, 3, 7, 8, 5, 6, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fa
iled, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed
, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : Uuuuuuuuu 374 failed
/dev/sdc1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x4
Array UUID : d835beeb:ae980218:ea6a9702:b21d7b29
Name : 1
Creation Time : Wed Dec 7 15:22:06 2011
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 390624831 (186.26 GiB 200.00 GB)
Array Size : 1171808256 (558.76 GiB 599.97 GB)
Used Dev Size : 390602752 (186.25 GiB 199.99 GB)
Data Offset : 136 sectors
Super Offset : 0 sectors
State : clean
Device UUID : c3bb6c0a:4c470d1d:d8f113fc:3a8dfead
Reshape pos'n : 7691712 (7.34 GiB 7.88 GB)
Delta Devices : 1 (4->5)
Update Time : Mon Dec 12 13:14:25 2011
Checksum : 7713b627 - correct
Events : 384
Chunk Size : 64K
Array Slot : 0 (0, 1, 2, 3, 4, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, faile
d, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fai
led, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, f
ailed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : Uuuuu 379 failed
/dev/sdd1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
Device UUID : c1a0df18:c7114a25:28129700:62db72ae
Update Time : Sun Jan 8 09:36:19 2012
Checksum : e43bab70 - correct
Events : 106120
Chunk Size : 64K
Array Slot : 13 (failed, failed, failed, failed, 4, failed, failed, failed, failed, 5, failed, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed
, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fail
ed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : _Uu_uu___ 379 failed
/dev/sde1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
Device UUID : 07136a9d:e5206ada:4a54d22b:b90e6858
Update Time : Sun Jan 8 09:36:19 2012
Checksum : f316a245 - correct
Events : 106120
Chunk Size : 64K
Array Slot : 4 (failed, failed, failed, failed, 4, failed, failed, failed, failed, 5, failed, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, f
ailed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, faile
d, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : _uu_Uu___ 379 failed
/dev/sdf1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : active
Device UUID : 6079c394:2acfaf27:2dc87760:a2cb1800
Update Time : Sat Jan 7 22:50:29 2012
Checksum : 14105032 - correct
Events : 106059
Chunk Size : 64K
Array Slot : 10 (0, failed, failed, failed, 4, failed, 3, 7, 8, 5, 6, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, f
ailed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, faile
d, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : uuuuuuUuu 374 failed
/dev/sdg1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x2
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
Recovery Offset : 0 sectors
State : active
Device UUID : e3b05100:736369e4:d7cc777f:029d08d6
Update Time : Sat Jan 7 22:50:29 2012
Checksum : 3147f687 - correct
Events : 106059
Chunk Size : 64K
Array Slot : 6 (0, failed, failed, failed, 4, failed, 3, 7, 8, 5, 6, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fa
iled, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed
, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : uuuUuuuuu 374 failed
/dev/sdh1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : active
Device UUID : b5ec03cc:c659fcd8:1cbffa7c:7d99c2c2
Update Time : Sat Jan 7 22:50:29 2012
Checksum : dbca12f3 - correct
Events : 106059
Chunk Size : 64K
Array Slot : 7 (0, failed, failed, failed, 4, failed, 3, 7, 8, 5, 6, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fa
iled, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed
, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : uuuuuuuUu 374 failed
/dev/sdi1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
Device UUID : 6726eb21:46024127:bb8ef26a:181eefc7
Update Time : Sun Jan 8 09:36:19 2012
Checksum : 7315e1db - correct
Events : 106120
Chunk Size : 64K
Array Slot : 9 (failed, failed, failed, failed, 4, failed, failed, failed, failed, 5, failed, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, f
ailed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, faile
d, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : _uu_uU___ 379 failed
/dev/sdj1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : active
Device UUID : 1595f70a:1c1d5455:ca81c332:ae654260
Update Time : Sat Jan 7 22:50:29 2012
Checksum : ea5e0d7f - correct
Events : 106059
Chunk Size : 64K
Array Slot : 8 (0, failed, failed, failed, 4, failed, 3, 7, 8, 5, 6, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fa
iled, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed
, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : uuuuuuuuU 374 failed
/dev/sdk1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
Device UUID : 483df5eb:76a6af2a:55eafe1a:1b084b16
Update Time : Sun Jan 8 09:36:19 2012
Checksum : 3ef6e18b - correct
Events : 106120
Chunk Size : 64K
Array Slot : 11 (failed, failed, failed, failed, 4, failed, failed, failed, failed, 5, failed, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed
, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fail
ed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : _uU_uu___ 379 failed
/dev/sdl1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x4
Array UUID : d835beeb:ae980218:ea6a9702:b21d7b29
Name : 1
Creation Time : Wed Dec 7 15:22:06 2011
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 390624831 (186.26 GiB 200.00 GB)
Array Size : 1171808256 (558.76 GiB 599.97 GB)
Used Dev Size : 390602752 (186.25 GiB 199.99 GB)
Data Offset : 136 sectors
Super Offset : 0 sectors
State : clean
Device UUID : 67b71288:2113a126:6ddc3a2a:6c317bde
Reshape pos'n : 7691712 (7.34 GiB 7.88 GB)
Delta Devices : 1 (4->5)
Update Time : Mon Dec 12 13:14:25 2011
Checksum : 5cef3831 - correct
Events : 384
Chunk Size : 64K
Array Slot : 2 (0, 1, 2, 3, 4, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, faile
d, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fai
led, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, f
ailed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : uuUuu 379 failed
/dev/sdm1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x4
Array UUID : d835beeb:ae980218:ea6a9702:b21d7b29
Name : 1
Creation Time : Wed Dec 7 15:22:06 2011
Raid Level : raid6
Raid Devices : 5
Avail Dev Size : 390624831 (186.26 GiB 200.00 GB)
Array Size : 1171808256 (558.76 GiB 599.97 GB)
Used Dev Size : 390602752 (186.25 GiB 199.99 GB)
Data Offset : 136 sectors
Super Offset : 0 sectors
State : clean
Device UUID : fc8330af:564224ff:dfff62e8:476b9635
Reshape pos'n : 7691712 (7.34 GiB 7.88 GB)
Delta Devices : 1 (4->5)
Update Time : Mon Dec 12 13:14:25 2011
Checksum : 71d39a8b - correct
Events : 384
Chunk Size : 64K
Array Slot : 4 (0, 1, 2, 3, 4, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, faile
d, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fai
led, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, f
ailed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : uuuuU 379 failed
/dev/sdn1:
Magic : a92b4efc
Version : 1.1
Feature Map : 0x0
Array UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Name : 0
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Raid Devices : 9
Avail Dev Size : 3906227893 (1862.63 GiB 1999.99 GB)
Array Size : 27343594880 (13038.44 GiB 13999.92 GB)
Used Dev Size : 3906227840 (1862.63 GiB 1999.99 GB)
Data Offset : 264 sectors
Super Offset : 0 sectors
State : clean
Device UUID : 851b031b:faa4ee37:89c05950:b4746150
Update Time : Sun Jan 8 09:36:19 2012
Checksum : eab50119 - correct
Events : 106120
Chunk Size : 64K
Array Slot : 12 (failed, failed, failed, failed, 4, failed, failed, failed, failed, 5, failed, 2, empty, 1, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed,
failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed
, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, fail
ed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed, failed)
Array State : _uu_uu___ 379 failed
--
kkeller@wombat.san-francisco.ca.us
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovery from multiple failures
2012-01-08 20:12 recovery from multiple failures Keith Keller
@ 2012-01-09 0:11 ` Phil Turmel
2012-01-09 1:08 ` Mikael Abrahamsson
2012-01-09 1:43 ` Keith Keller
0 siblings, 2 replies; 8+ messages in thread
From: Phil Turmel @ 2012-01-09 0:11 UTC (permalink / raw)
To: Keith Keller; +Cc: linux-raid
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Keith,
On 01/08/2012 03:12 PM, Keith Keller wrote:
> Hi all,
>
> I recently had an experience very similar to what's posted on the wiki:
>
> https://raid.wiki.kernel.org/articles/r/a/i/RAID_Recovery_d376.html
This page describes the older --examine output format, fortunately, since
that is what you seem to have.
> From what I can piece together from the logs, my controller went a
> little crazy, and dropped a whole bunch of drives in a span of about
> 10 minutes. I attempted to look at the output of mdadm --examine, as
> documented in the wiki, but the format appears to have changed, so I
> am unclear where to go next.
>
> I'm going to include the entire mdadm --examine output below, but as I
> was looking at it, I was wondering if the analogous scenario to the wiki
> situation is to look at the array slots:
>
> $ grep Slot raid.status |cut -f1 -d '('
> Array Slot : 0
> Array Slot : 0
> Array Slot : 13
> Array Slot : 4
> Array Slot : 10
> Array Slot : 6
> Array Slot : 7
> Array Slot : 9
> Array Slot : 8
> Array Slot : 11
> Array Slot : 2
> Array Slot : 4
> Array Slot : 12
You are confusing "Slot" with "Role", aka "Raid Device". All of your devices
report their own role between 0 and 8, except for slot #12, which is "empty".
> There are two separate arrays on this box; the problematic one is
> 24363b01:90deb9b5:4b51e5df:68b8b6ea. Will I be able to recover this
> array with an appropriate mdadm --create --assume-clean command, and if
> so, how would I go about determining the correct order in which to
> specify the drives? The big confusing part for me is, as a 9 device
> RAID6, I'd expect to see device slots 0-8, but here I see slots 10
> through 13, and I am unclear how to get the order exactly right.
- From what I can see, you should use "--assemble --force". The wiki does
not recommend this, but is wrong. There is no advantage to "--create
- --assume-clean" in this situation, and opportunities for catastrophic
destruction. Only if "--assemble --force" fails, and not from "device in use"
reports, should you move to "--create".
Another word of warning: Your --examine output reports Data Offset == 264
on all of your devices. You cannot use "--create --assume-clean" with a
new version of mdadm, as it will create with the new default Data Offset of
2048.
> My mdadm --examine (which I have also saved separately) is below. If
> you need any more information let me know. Thanks!
This is very good. And clearly shows that "--assemble --force" should
succeed. You will probably want to run an fsck to deal with the ten minutes
of inconsistent data, but that should be the only losses. A "check" or
"repair" pass should also be run.
HTH,
Phil
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk8KMMoACgkQBP+iHzflm3CWmACfQqsqxua7Kp9q5ydpPV5Rtxih
Uc0An0rCW6p8ni4caecGLFoLDxin3wEE
=Gnmb
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovery from multiple failures
2012-01-09 0:11 ` Phil Turmel
@ 2012-01-09 1:08 ` Mikael Abrahamsson
2012-01-09 1:43 ` Keith Keller
1 sibling, 0 replies; 8+ messages in thread
From: Mikael Abrahamsson @ 2012-01-09 1:08 UTC (permalink / raw)
To: Phil Turmel; +Cc: Keith Keller, linux-raid
On Sun, 8 Jan 2012, Phil Turmel wrote:
> - From what I can see, you should use "--assemble --force". The wiki does
> not recommend this, but is wrong. There is no advantage to "--create
> - --assume-clean" in this situation, and opportunities for catastrophic
> destruction. Only if "--assemble --force" fails, and not from "device in use"
> reports, should you move to "--create".
I totally agree here. People generally go to --create way too early in the
recovery process, partly due to several recommendations out there on wikis
and other guides. I've seen numerous people wreck their data due to
getting this wrong (order or offset wrong, or forgetting --assume-clean
and forgetting to go read-only on fsck, etc).
Does someone have some history on the reason for this recommendation, and
do we have consensus on the recommendation that --assemble --force is
*always* better to try before going to --create --assume-clean?
--
Mikael Abrahamsson email: swmike@swm.pp.se
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovery from multiple failures
2012-01-09 0:11 ` Phil Turmel
2012-01-09 1:08 ` Mikael Abrahamsson
@ 2012-01-09 1:43 ` Keith Keller
2012-01-09 1:58 ` Keith Keller
2012-01-09 4:51 ` Keith Keller
1 sibling, 2 replies; 8+ messages in thread
From: Keith Keller @ 2012-01-09 1:43 UTC (permalink / raw)
To: linux-raid
On 2012-01-09, Phil Turmel <philip@turmel.org> wrote:
>
> On 01/08/2012 03:12 PM, Keith Keller wrote:
>>
>> I'm going to include the entire mdadm --examine output below, but as I
>> was looking at it, I was wondering if the analogous scenario to the wiki
>> situation is to look at the array slots:
>>
>> $ grep Slot raid.status |cut -f1 -d '('
>> Array Slot : 0
>> Array Slot : 0
>> Array Slot : 13
>> Array Slot : 4
>> Array Slot : 10
>> Array Slot : 6
>> Array Slot : 7
>> Array Slot : 9
>> Array Slot : 8
>> Array Slot : 11
>> Array Slot : 2
>> Array Slot : 4
>> Array Slot : 12
>
> You are confusing "Slot" with "Role", aka "Raid Device". All of your devices
> report their own role between 0 and 8, except for slot #12, which is "empty".
That brings up another question, then: how did you determine the role?
Is it the capital U in the Array State line, or is it something obvious
I'm missing, or something unobvious that I should look at? I mostly
just want to know what to look for in the future.
> - From what I can see, you should use "--assemble --force". The wiki does
> not recommend this, but is wrong. There is no advantage to "--create
> - --assume-clean" in this situation, and opportunities for catastrophic
> destruction. Only if "--assemble --force" fails, and not from "device in use"
> reports, should you move to "--create".
So, if a rebuild already started with new disks, will --force get
confused by the array's state? Or is md smart enough to look at the
last update times to assemble the disks that are most up to date, or are
otherwise smart enough not to assemble disks in a really bad way?
> Another word of warning: Your --examine output reports Data Offset == 264
> on all of your devices. You cannot use "--create --assume-clean" with a
> new version of mdadm, as it will create with the new default Data Offset of
> 2048.
Great, thanks for the pointer. I currently have version 2.6.9 of mdadm,
which IIRC is fairly old.
> This is very good. And clearly shows that "--assemble --force" should
> succeed. You will probably want to run an fsck to deal with the ten minutes
> of inconsistent data, but that should be the only losses. A "check" or
> "repair" pass should also be run.
Okay: here's what happened when I made the attempt:
# mdadm --assemble --scan --uuid=24363b01:90deb9b5:4b51e5df:68b8b6ea
# --config=mdadm.conf --force
/dev/md0: File exists
mdadm: forcing event count in /dev/sdb1(0) from 106059 upto 106120
mdadm: forcing event count in /dev/sdg1(3) from 106059 upto 106120
mdadm: forcing event count in /dev/sdf1(6) from 106059 upto 106120
mdadm: forcing event count in /dev/sdh1(7) from 106059 upto 106120
mdadm: forcing event count in /dev/sdj1(8) from 106059 upto 106120
mdadm: failed to RUN_ARRAY /dev/md/0: Input/output error
Here's what appeared in dmesg:
md/raid:md0: not clean -- starting background reconstruction
md/raid:md0: device sdb1 operational as raid disk 0
md/raid:md0: device sdj1 operational as raid disk 8
md/raid:md0: device sdh1 operational as raid disk 7
md/raid:md0: device sdf1 operational as raid disk 6
md/raid:md0: device sdi1 operational as raid disk 5
md/raid:md0: device sde1 operational as raid disk 4
md/raid:md0: device sdk1 operational as raid disk 2
md/raid:md0: device sdc1 operational as raid disk 1
md/raid:md0: allocated 9522kB
md/raid:md0: cannot start dirty degraded array.
RAID conf printout:
--- level:6 rd:9 wd:8
disk 0, o:1, dev:sdb1
disk 1, o:1, dev:sdc1
disk 2, o:1, dev:sdk1
disk 3, o:1, dev:sdg1
disk 4, o:1, dev:sde1
disk 5, o:1, dev:sdi1
disk 6, o:1, dev:sdf1
disk 7, o:1, dev:sdh1
disk 8, o:1, dev:sdj1
md/raid:md0: failed to run raid set.
md: pers->run() failed ...
And finally, mdadm -D:
# mdadm -D /dev/md0
/dev/md0:
Version : 1.01
Creation Time : Thu Sep 29 21:26:35 2011
Raid Level : raid6
Used Dev Size : 1953113920 (1862.63 GiB 1999.99 GB)
Raid Devices : 9
Total Devices : 10
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Sat Jan 7 22:50:29 2012
State : active, degraded, Not Started
Active Devices : 8
Working Devices : 10
Failed Devices : 0
Spare Devices : 2
Chunk Size : 64K
Name : 0
UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
Events : 106120
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
13 8 33 1 active sync /dev/sdc1
11 8 161 2 active sync /dev/sdk1
6 8 97 3 spare rebuilding /dev/sdg1
4 8 65 4 active sync /dev/sde1
9 8 129 5 active sync /dev/sdi1
10 8 81 6 active sync /dev/sdf1
7 8 113 7 active sync /dev/sdh1
8 8 145 8 active sync /dev/sdj1
12 8 177 - spare /dev/sdl1
Now I really don't know where to go from here. Any thoughts? Will
doing a check help at this point, or just make things worse?
--keith
--
kkeller@wombat.san-francisco.ca.us
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovery from multiple failures
2012-01-09 1:43 ` Keith Keller
@ 2012-01-09 1:58 ` Keith Keller
2012-01-09 4:51 ` Keith Keller
1 sibling, 0 replies; 8+ messages in thread
From: Keith Keller @ 2012-01-09 1:58 UTC (permalink / raw)
To: linux-raid
On 2012-01-09, Keith Keller <kkeller@wombat.san-francisco.ca.us> wrote:
> # mdadm -D /dev/md0
> /dev/md0:
> Version : 1.01
> Creation Time : Thu Sep 29 21:26:35 2011
> Raid Level : raid6
> Used Dev Size : 1953113920 (1862.63 GiB 1999.99 GB)
> Raid Devices : 9
> Total Devices : 10
> Preferred Minor : 0
> Persistence : Superblock is persistent
>
> Update Time : Sat Jan 7 22:50:29 2012
> State : active, degraded, Not Started
> Active Devices : 8
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 2
>
> Chunk Size : 64K
>
> Name : 0
> UUID : 24363b01:90deb9b5:4b51e5df:68b8b6ea
> Events : 106120
>
> Number Major Minor RaidDevice State
> 0 8 17 0 active sync /dev/sdb1
> 13 8 33 1 active sync /dev/sdc1
> 11 8 161 2 active sync /dev/sdk1
> 6 8 97 3 spare rebuilding /dev/sdg1
> 4 8 65 4 active sync /dev/sde1
> 9 8 129 5 active sync /dev/sdi1
> 10 8 81 6 active sync /dev/sdf1
> 7 8 113 7 active sync /dev/sdh1
> 8 8 145 8 active sync /dev/sdj1
>
> 12 8 177 - spare /dev/sdl1
I apologize for not pointing this out in my previous post: it is
possible that drives were relabeled from the last --examine to now, as I
removed the drives that were in the other array and rebooted (simply to
stop them from confusing me). If new mdadm --examine output is required
please let me know.
Thanks for your help!
--keith
--
kkeller@wombat.san-francisco.ca.us
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovery from multiple failures
2012-01-09 1:43 ` Keith Keller
2012-01-09 1:58 ` Keith Keller
@ 2012-01-09 4:51 ` Keith Keller
2012-01-09 5:13 ` NeilBrown
1 sibling, 1 reply; 8+ messages in thread
From: Keith Keller @ 2012-01-09 4:51 UTC (permalink / raw)
To: linux-raid
On 2012-01-09, Keith Keller <kkeller@wombat.san-francisco.ca.us> wrote:
> md/raid:md0: cannot start dirty degraded array.
Upon further reading (specifically man md), it looks like this is the
tail reason for the array not starting--the kernel will refuse to start
arrays in this condition. If I'm reading the docs and other posts
accurately, it seems like I can use a kernel parameter (since this isn't
my /, I should also be able to unload and re-load the md module) to
force the kernel to start the array, at which point in theory I should
be able to work with it (including, ultimately, an xfs_repair if
needed). But OTOH, it seems like this parameter applies only to / on
mdraid, and --force should have started the array, so I'm not really
sure what to make of it.
I've posted this before, but the kernel I'm using is fairly recent,
2.6.39-4.el5 from ELrepo.
--keith
--
kkeller@wombat.san-francisco.ca.us
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovery from multiple failures
2012-01-09 4:51 ` Keith Keller
@ 2012-01-09 5:13 ` NeilBrown
2012-01-09 6:23 ` Keith Keller
0 siblings, 1 reply; 8+ messages in thread
From: NeilBrown @ 2012-01-09 5:13 UTC (permalink / raw)
To: Keith Keller; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]
On Sun, 08 Jan 2012 20:51:06 -0800 Keith Keller
<kkeller@wombat.san-francisco.ca.us> wrote:
> On 2012-01-09, Keith Keller <kkeller@wombat.san-francisco.ca.us> wrote:
> > md/raid:md0: cannot start dirty degraded array.
>
> Upon further reading (specifically man md), it looks like this is the
> tail reason for the array not starting--the kernel will refuse to start
> arrays in this condition. If I'm reading the docs and other posts
> accurately, it seems like I can use a kernel parameter (since this isn't
> my /, I should also be able to unload and re-load the md module) to
> force the kernel to start the array, at which point in theory I should
> be able to work with it (including, ultimately, an xfs_repair if
> needed). But OTOH, it seems like this parameter applies only to / on
> mdraid, and --force should have started the array, so I'm not really
> sure what to make of it.
>
> I've posted this before, but the kernel I'm using is fairly recent,
> 2.6.39-4.el5 from ELrepo.
>
> --keith
>
--assemble --force certainly should start dirty/degraded array.
There was a bug a while back where it didn't. Maybe you should try a newer
mdadm.
Alternately:
echo 1 > /sys/module/md_mod/parameters/start_dirty_degraded
and try again. That is an easy way to set the module parameter after the
module is loaded.
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: recovery from multiple failures
2012-01-09 5:13 ` NeilBrown
@ 2012-01-09 6:23 ` Keith Keller
0 siblings, 0 replies; 8+ messages in thread
From: Keith Keller @ 2012-01-09 6:23 UTC (permalink / raw)
To: linux-raid
On 2012-01-09, NeilBrown <neilb@suse.de> wrote:
>
> --assemble --force certainly should start dirty/degraded array.
> There was a bug a while back where it didn't. Maybe you should try a newer
> mdadm.
I'll try this next time (if there is one).
> Alternately:
>=20
> echo 1 > /sys/module/md_mod/parameters/start_dirty_degraded=20
>
>
> and try again. That is an easy way to set the module parameter after the
> module is loaded.
Worked great, the array is rebuilding now. xfs_repair has found a small
number of errors, but otherwise the filesystem seems fine (at least for
now; haven't mounted it yet).
--keith
--
kkeller@wombat.san-francisco.ca.us
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2012-01-09 6:23 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-01-08 20:12 recovery from multiple failures Keith Keller
2012-01-09 0:11 ` Phil Turmel
2012-01-09 1:08 ` Mikael Abrahamsson
2012-01-09 1:43 ` Keith Keller
2012-01-09 1:58 ` Keith Keller
2012-01-09 4:51 ` Keith Keller
2012-01-09 5:13 ` NeilBrown
2012-01-09 6:23 ` Keith Keller
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).