From: Neil Brown <neilb@suse.de>
To: Kevin Shanahan <kmshanah@disenchant.net>
Cc: linux-raid@vger.kernel.org
Subject: Re: Help recovering RAID6 failure
Date: Tue, 16 Dec 2008 09:15:17 +1100 [thread overview]
Message-ID: <18758.55029.597319.376426@notabene.brown> (raw)
In-Reply-To: message from Kevin Shanahan on Tuesday December 16
On Tuesday December 16, kmshanah@disenchant.net wrote:
> Hi,
>
> My server seems to have had a bad night, and so I'm not having the
> best morning. Overnight three drives failed in my RAID6 array. Worse,
> that's on top of one failure I already knew about, so 4 drives in
> total.
>
> I put this down to problems with the controller - I had been using
> 3ware 9650 controllers (in JBOD mode). This morning I've loaded the
> drives into another server connected to the on board Intel AHCI SATA
> ports and an extra Sil3132 controller.
>
> My first naive attempt was to try re-assembling the array from the
> eight most up-to-date member devices:
>
> hermes:~# mdadm -A /dev/md5 /dev/sd[cefghijk]1
> mdadm: failed to RUN_ARRAY /dev/md5: Input/output error
Try adding --force to over-ride apparent errors, and --verbose to help
you see what is happening.
NeilBrown
>
> I assume that's not an I/O error with the underlying device? I can't
> see anything indicating that dmesg/syslog.
>
> Output from "mdadm -E /dev/sd[cdefghijkl]1" included below.
>
> Any advice would be much appreciated.
>
> (and now, back to scouring the mail archives for clues).
>
> Cheers,
> Kevin.
> --
>
> /dev/sdc1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9eea - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 8 8 129 8 active sync /dev/sdi1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdd1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Thu Nov 27 09:20:39 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 10
> Working Devices : 10
> Failed Devices : 0
> Spare Devices : 0
> Checksum : e3c42f8b - correct
> Events : 91974
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 9 8 145 9 active sync /dev/sdj1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 8 113 7 active sync /dev/sdh1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 8 145 9 active sync /dev/sdj1
> /dev/sde1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 00:28:14 2008
> State : active
> Internal Bitmap : present
> Active Devices : 8
> Working Devices : 8
> Failed Devices : 2
> Spare Devices : 0
> Checksum : e3dca329 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 0 8 1 0 active sync /dev/sda1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdf1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 00:29:00 2008
> State : active
> Internal Bitmap : present
> Active Devices : 7
> Working Devices : 7
> Failed Devices : 2
> Spare Devices : 0
> Checksum : e3dca382 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 1 8 17 1 active sync /dev/sdb1
>
> 0 0 0 0 0 removed
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdg1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9e7e - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 2 8 33 2 active sync /dev/sdc1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdh1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9ec6 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 6 8 97 6 active sync /dev/sdg1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdi1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9eb4 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 5 8 81 5 active sync /dev/sdf1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdj1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9ea2 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 4 8 65 4 active sync /dev/sde1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdk1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Tue Dec 16 05:35:46 2008
> State : clean
> Internal Bitmap : present
> Active Devices : 6
> Working Devices : 6
> Failed Devices : 3
> Spare Devices : 0
> Checksum : e3df9e90 - correct
> Events : 176870
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 3 8 49 3 active sync /dev/sdd1
>
> 0 0 0 0 0 removed
> 1 1 0 0 1 faulty removed
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 0 0 7 faulty removed
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> /dev/sdl1:
> Magic : a92b4efc
> Version : 00.90.00
> UUID : 77fa8675:444efef0:12f8dff0:9f609eb4
> Creation Time : Sun Jul 27 15:03:44 2008
> Raid Level : raid6
> Used Dev Size : 976759936 (931.51 GiB 1000.20 GB)
> Array Size : 7814079488 (7452.09 GiB 8001.62 GB)
> Raid Devices : 10
> Total Devices : 10
> Preferred Minor : 5
>
> Update Time : Mon Dec 15 22:36:32 2008
> State : active
> Internal Bitmap : present
> Active Devices : 9
> Working Devices : 9
> Failed Devices : 1
> Spare Devices : 0
> Checksum : e3dc8525 - correct
> Events : 175787
>
> Chunk Size : 64K
>
> Number Major Minor RaidDevice State
> this 7 8 113 7 active sync /dev/sdh1
>
> 0 0 8 1 0 active sync /dev/sda1
> 1 1 8 17 1 active sync /dev/sdb1
> 2 2 8 33 2 active sync /dev/sdc1
> 3 3 8 49 3 active sync /dev/sdd1
> 4 4 8 65 4 active sync /dev/sde1
> 5 5 8 81 5 active sync /dev/sdf1
> 6 6 8 97 6 active sync /dev/sdg1
> 7 7 8 113 7 active sync /dev/sdh1
> 8 8 8 129 8 active sync /dev/sdi1
> 9 9 0 0 9 faulty removed
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2008-12-15 22:15 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2008-12-15 22:03 Help recovering RAID6 failure Kevin Shanahan
2008-12-15 22:15 ` Neil Brown [this message]
2008-12-15 22:25 ` Kevin Shanahan
2008-12-15 22:37 ` Kevin Shanahan
2008-12-15 22:39 ` Kevin Shanahan
2008-12-15 23:05 ` Neil Brown
2008-12-15 23:18 ` Kevin Shanahan
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=18758.55029.597319.376426@notabene.brown \
--to=neilb@suse.de \
--cc=kmshanah@disenchant.net \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).