From: Fabian Knorr <knorrfab@fim.uni-passau.de>
To: linux-raid@vger.kernel.org
Subject: Recovering an Array with inconsistent Superblocks
Date: Sat, 04 Jan 2014 11:04:41 +0100 [thread overview]
Message-ID: <1388829881.16265.20.camel@vessel> (raw)
[-- Attachment #1: Type: text/plain, Size: 1540 bytes --]
Good morning folks,
I have a MD-RAID5 with 8 disks, 7 of them active, 1 spare. They are
connected to two SATA controllers, 4 disks each.
A few days ago, the disks connected to one of the controllers stopped
operating (some sort of controller hiccup, I guess). Now those disks are
marked as "possibly outdated" and the array refuses to assemble or
start, telling me there are only 4 out of 7 devices operational (See
attachment "assemble.status")
On boot, "/proc/mdstat" reports an inactive array with 7 spares, trying
to "mdadm --run" the array fails with the message mentioned above,
changing "/proc/mdstat" to now show an array of 4 disks.
I tried "--add" with a missing device, getting the message "re-added
device /dev/sd...", but failing for subsequent devices with the message
"/dev/md0 failed to start, not adding device ..., You should stop and
re-assemble the array".
Then I tried "--assemble --scan --force", which yielded the same result
as above.
The next thing I would try is recreating the array with the layout
stored in the superblocks, but I was surprised to find it to be
inconsistent between the disks. I attached the output of "--examine
--verbose" as "raid.status".
Could "--add"ing have changed one superblock, and can I safely try to
re-create the array with the layout given by /dev/sda and /dev/sdc?
Also, what parameters would I need to keep the layout (As mentioned in
the wiki at https://raid.wiki.kernel.org/index.php/RAID_Recovery)
consistent with the one I have now?
Thanks for your time!
Fabian
[-- Attachment #2: assemble.status --]
[-- Type: text/plain, Size: 2854 bytes --]
mdadm: looking for devices for /dev/md0
mdadm: /dev/dm-0 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/loop2 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/loop1 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/loop0 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdj2 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdj1 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdj is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdi is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdh is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdg is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdf is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sde is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdd1 is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdd is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sda is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdb is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdc is not one of /dev/sdi1,/dev/sdh1,/dev/sdg1,/dev/sdf1,/dev/sde1,/dev/sda1,/dev/sdb1,/dev/sdc1
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sda1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 5.
mdadm: added /dev/sdg1 to /dev/md0 as 1
mdadm: added /dev/sdh1 to /dev/md0 as 2
mdadm: added /dev/sdi1 to /dev/md0 as 3
mdadm: added /dev/sda1 to /dev/md0 as 4 (possibly out of date)
mdadm: added /dev/sdc1 to /dev/md0 as 5 (possibly out of date)
mdadm: added /dev/sde1 to /dev/md0 as 6 (possibly out of date)
mdadm: added /dev/sdf1 to /dev/md0 as 0
mdadm: /dev/md0 assembled from 4 drives - not enough to start the array.
[-- Attachment #3: raid.status --]
[-- Type: text/plain, Size: 9719 bytes --]
/dev/sda1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2a6 - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 4 8 1 4 active sync /dev/sda1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sdb1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 8
Preferred Minor : 0
Update Time : Sun Oct 27 15:00:43 2013
State : active
Active Devices : 7
Working Devices : 8
Failed Devices : 0
Spare Devices : 1
Checksum : f3d95119 - correct
Events : 24319
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 4 8 17 4 active sync /dev/sdb1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 17 4 active sync /dev/sdb1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
7 7 8 1 7 spare /dev/sda1
/dev/sdc1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2c8 - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 5 8 33 5 active sync /dev/sdc1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sde1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:51:28 2014
State : active
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Checksum : f430c2da - correct
Events : 24433
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 6 8 49 6 active sync /dev/sdd1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 8 1 4 active sync /dev/sda1
5 5 8 33 5 active sync /dev/sdc1
6 6 8 49 6 active sync /dev/sdd1
/dev/sdf1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f43123f2 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 0 8 65 0 active sync /dev/sde1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdg1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312404 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 1 8 81 1 active sync /dev/sdf1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdh1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312416 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 2 8 97 2 active sync /dev/sdg1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdi1:
Magic : a92b4efc
Version : 0.90.00
UUID : cb5a6670:1b5d391f:edd8d016:611227ea
Creation Time : Wed Jul 13 13:47:12 2011
Raid Level : raid5
Used Dev Size : 1953512448 (1863.01 GiB 2000.40 GB)
Array Size : 11721074688 (11178.09 GiB 12002.38 GB)
Raid Devices : 7
Total Devices : 7
Preferred Minor : 0
Update Time : Wed Jan 1 22:55:03 2014
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 3
Spare Devices : 0
Checksum : f4312428 - correct
Events : 24506
Layout : left-symmetric
Chunk Size : 1024K
Number Major Minor RaidDevice State
this 3 8 113 3 active sync /dev/sdh1
0 0 8 65 0 active sync /dev/sde1
1 1 8 81 1 active sync /dev/sdf1
2 2 8 97 2 active sync /dev/sdg1
3 3 8 113 3 active sync /dev/sdh1
4 4 0 0 4 faulty removed
5 5 0 0 5 faulty removed
6 6 0 0 6 faulty removed
/dev/sdj1:
MBR Magic : aa55
Partition[0] : 1069056 sectors at 0 (type 00)
Partition[1] : 63488 sectors at 95576 (type ef)
next reply other threads:[~2014-01-04 10:04 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-01-04 10:04 Fabian Knorr [this message]
2014-01-04 16:24 ` Recovering an Array with inconsistent Superblocks Phil Turmel
2014-01-04 17:59 ` Can Jeuleers
2014-01-04 19:16 ` Phil Turmel
2014-01-04 22:05 ` Fabian Knorr
2014-01-05 2:32 ` Phil Turmel
2014-01-05 9:07 ` Fabian Knorr
2014-01-05 9:56 ` NeilBrown
2014-01-05 10:40 ` Fabian Knorr
[not found] ` <1388918703.3591.20.camel@vessel>
2014-01-05 18:25 ` Phil Turmel
2014-01-05 23:50 ` NeilBrown
2014-01-06 14:00 ` Fabian Knorr
2014-01-07 0:26 ` NeilBrown
2014-01-14 8:54 ` David Brown
2014-01-04 22:08 ` Fabian Knorr
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1388829881.16265.20.camel@vessel \
--to=knorrfab@fim.uni-passau.de \
--cc=linux-raid@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.