From: Chris Allen <chris@cjx.com>
To: Neil Brown <neilb@suse.de>
Cc: linux-raid@vger.kernel.org
Subject: Re: Recovering a raid5 array with strange event count
Date: Fri, 13 Apr 2007 14:07:58 +0100 [thread overview]
Message-ID: <461F80AE.5080702@cjx.com> (raw)
In-Reply-To: <17951.29662.910442.896659@notabene.brown>
Neil Brown wrote:
> On Friday April 13, chris@cjx.com wrote:
>
>> Dear All,
>>
>> I have an 8-drive raid-5 array running under 2.6.11. This morning it
>> bombed out, and when I brought
>> it up again, two drives had incorrect event counts:
>>
>>
>> sda1: 0.8258715
>> sdb1: 0.8258715
>> sdc1: 0.8258715
>> sdd1: 0.8258715
>> sde1: 0.8258715
>> sdf1: 0.8258715
>> sdg1: 0.8258708
>> sdh1: 0.8258716
>>
>>
>> sdg1 is out of date (expected), but sdh1 has received an extra event.
>>
>> Any attempt to restart with mdadm --assemble --force, results in an an
>> un-startable array with an event count of 0.8258715.
>>
>> Can anybody advise on the correct command to use to get it started again?
>> I'm assuming I'll need to use mdadm --create --assume-clean - but I'm
>> not sure
>> which drives should be included/excluded when I do this.
>>
>
> A difference of 1 in event counts is not supposed to cause a problem.
> Have you tried simply assembling the array without including sdg1.
> e.g.
> mdadm -A /dev/md0 /dev/sd[abcdefh]1
>
>
>
# mdadm -A /dev/md0 /dev/sd[abcdefh]1
mdadm: /dev/md0 assembled from 7 drives - need all 8 to start it (use
--run to insist)
# mdadm -D /dev/md0
mdadm: md device /dev/md0 does not appear to be active.
mdadm --run /dev/md0
mdadm: failed to run array /dev/md0: invalid argument
I've attached the syslog, the dump for the assembled array, the dump for
each drive
and the contents of /proc/mdstat. Using --force makes no difference.
Apr 13 13:59:45 snap29 kernel: md: bind<sdb1>
Apr 13 13:59:45 snap29 kernel: md: bind<sdc1>
Apr 13 13:59:45 snap29 kernel: md: bind<sdd1>
Apr 13 13:59:45 snap29 kernel: md: bind<sde1>
Apr 13 13:59:45 snap29 kernel: md: bind<sdf1>
Apr 13 13:59:45 snap29 kernel: md: bind<sdh1>
Apr 13 13:59:45 snap29 kernel: md: bind<sda1>
Apr 13 14:00:01 snap29 kernel: md: md0: raid array is not clean --
starting background reconstruction
Apr 13 14:00:01 snap29 kernel: raid5: device sda1 operational as raid disk 0
Apr 13 14:00:01 snap29 kernel: raid5: device sdh1 operational as raid disk 7
Apr 13 14:00:01 snap29 kernel: raid5: device sdf1 operational as raid disk 5
Apr 13 14:00:01 snap29 kernel: raid5: device sde1 operational as raid disk 4
Apr 13 14:00:01 snap29 kernel: raid5: device sdd1 operational as raid disk 3
Apr 13 14:00:01 snap29 kernel: raid5: device sdc1 operational as raid disk 2
Apr 13 14:00:01 snap29 kernel: raid5: device sdb1 operational as raid disk 1
Apr 13 14:00:01 snap29 kernel: raid5: cannot start dirty degraded array
for md0
Apr 13 14:00:01 snap29 kernel: RAID5 conf printout:
Apr 13 14:00:01 snap29 kernel: --- rd:8 wd:7 fd:1
Apr 13 14:00:01 snap29 kernel: disk 0, o:1, dev:sda1
Apr 13 14:00:01 snap29 kernel: disk 1, o:1, dev:sdb1
Apr 13 14:00:01 snap29 kernel: disk 2, o:1, dev:sdc1
Apr 13 14:00:01 snap29 kernel: disk 3, o:1, dev:sdd1
Apr 13 14:00:01 snap29 kernel: disk 4, o:1, dev:sde1
Apr 13 14:00:01 snap29 kernel: disk 5, o:1, dev:sdf1
Apr 13 14:00:01 snap29 kernel: disk 7, o:1, dev:sdh1
Apr 13 14:00:01 snap29 kernel: raid5: failed to run raid set md0
Apr 13 14:00:01 snap29 kernel: md: pers->run() failed ...
/dev/md0:
Version : 00.90.01
Creation Time : Wed Apr 19 06:23:21 2006
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Raid Devices : 8
Total Devices : 7
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Fri Apr 13 10:11:15 2007
State : active, degraded, Not Started
Active Devices : 7
Working Devices : 7
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : 31b253f9:02049908:aa4bb1ab:753b8fda
Events : 0.8258715
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1
6 0 0 6 removed
7 8 113 7 active sync /dev/sdh1
/dev/sda1:
Magic : a92b4efc
Version : 00.90.01
UUID : 31b253f9:02049908:aa4bb1ab:753b8fda
Creation Time : Wed Apr 19 06:23:21 2006
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 3418687552 (3260.31 GiB 3500.74 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Apr 13 10:11:12 2007
State : clean
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : a469bd5a - correct
Events : 0.8258715
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 0 8 1 0 active sync /dev/sda1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
/dev/sdb1:
Magic : a92b4efc
Version : 00.90.01
UUID : 31b253f9:02049908:aa4bb1ab:753b8fda
Creation Time : Wed Apr 19 06:23:21 2006
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 3418687552 (3260.31 GiB 3500.74 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Apr 13 10:11:12 2007
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : a469bd6b - correct
Events : 0.8258715
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 1 8 17 1 active sync /dev/sdb1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.01
UUID : 31b253f9:02049908:aa4bb1ab:753b8fda
Creation Time : Wed Apr 19 06:23:21 2006
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 3418687552 (3260.31 GiB 3500.74 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Apr 13 10:11:12 2007
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : a469bd7d - correct
Events : 0.8258715
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 2 8 33 2 active sync /dev/sdc1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
/dev/sdd1:
Magic : a92b4efc
Version : 00.90.01
UUID : 31b253f9:02049908:aa4bb1ab:753b8fda
Creation Time : Wed Apr 19 06:23:21 2006
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 3418687552 (3260.31 GiB 3500.74 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Apr 13 10:11:12 2007
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : a469bd8f - correct
Events : 0.8258715
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 3 8 49 3 active sync /dev/sdd1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
/dev/sde1:
Magic : a92b4efc
Version : 00.90.01
UUID : 31b253f9:02049908:aa4bb1ab:753b8fda
Creation Time : Wed Apr 19 06:23:21 2006
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 3418687552 (3260.31 GiB 3500.74 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Apr 13 10:11:12 2007
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : a469bda1 - correct
Events : 0.8258715
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 4 8 65 4 active sync /dev/sde1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
/dev/sdf1:
Magic : a92b4efc
Version : 00.90.01
UUID : 31b253f9:02049908:aa4bb1ab:753b8fda
Creation Time : Wed Apr 19 06:23:21 2006
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 3418687552 (3260.31 GiB 3500.74 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Apr 13 10:11:12 2007
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : a469bdb3 - correct
Events : 0.8258715
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 5 8 81 5 active sync /dev/sdf1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
/dev/sdh1:
Magic : a92b4efc
Version : 00.90.01
UUID : 31b253f9:02049908:aa4bb1ab:753b8fda
Creation Time : Wed Apr 19 06:23:21 2006
Raid Level : raid5
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Array Size : 3418687552 (3260.31 GiB 3500.74 GB)
Raid Devices : 8
Total Devices : 8
Preferred Minor : 0
Update Time : Fri Apr 13 10:11:15 2007
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Checksum : a469bddb - correct
Events : 0.8258716
Layout : left-symmetric
Chunk Size : 64K
Number Major Minor RaidDevice State
this 7 8 113 7 active sync /dev/sdh1
0 0 8 1 0 active sync /dev/sda1
1 1 8 17 1 active sync /dev/sdb1
2 2 8 33 2 active sync /dev/sdc1
3 3 8 49 3 active sync /dev/sdd1
4 4 8 65 4 active sync /dev/sde1
5 5 8 81 5 active sync /dev/sdf1
6 6 8 97 6 active sync /dev/sdg1
7 7 8 113 7 active sync /dev/sdh1
Personalities : [raid5]
md0 : inactive sda1[0] sdh1[7] sdf1[5] sde1[4] sdd1[3] sdc1[2] sdb1[1]
3418687552 blocks
unused devices: <none>
next prev parent reply other threads:[~2007-04-13 13:07 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2007-04-13 10:14 Recovering a raid5 array with strange event count Chris Allen
2007-04-13 12:13 ` Neil Brown
2007-04-13 13:07 ` Chris Allen [this message]
2007-04-16 13:55 ` Chris Allen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=461F80AE.5080702@cjx.com \
--to=chris@cjx.com \
--cc=linux-raid@vger.kernel.org \
--cc=neilb@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).