* Q: how to move "spare" back into raid?
@ 2010-09-27 11:45 Henrik Holst
2010-10-03 0:05 ` Henrik Holst
0 siblings, 1 reply; 2+ messages in thread
From: Henrik Holst @ 2010-09-27 11:45 UTC (permalink / raw)
To: linux-raid
Hi linux-raid!
I have not been working with linux raid for a few years,
everything has been working great in my 12x1TB RAID5 system.
Last week smartctl gave me some information that two of my
discs where failing so I had to replace them. I used
dd_rescue to copy the entire discs into two new discs. And
then I tried to reassemble them. I do not know what I did
wrong but now the system is in a broken state.
My guess is that I have tried to reassemble the raid from
/dev/sd{...} instead of /dev/sd{...}1 as I was supposed to.
If anyone can give me some hints on how I can rescue this
mess I would be very glad. If the procedure is lengthy I
would be willing to pay you (Paypal?) something for your
time to type it up.
The raid state as-is now:
lurch:~# mdadm -A /dev/md0 -Rfv /dev/sd[b-m]1
mdadm: looking for devices for /dev/md0
mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 13.
mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 12.
mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 8.
mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 6.
mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 3.
mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 0.
mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 2.
mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 1.
mdadm: /dev/sdj1 is identified as a member of /dev/md0, slot 7.
mdadm: /dev/sdk1 is identified as a member of /dev/md0, slot 14.
mdadm: /dev/sdl1 is identified as a member of /dev/md0, slot 4.
mdadm: /dev/sdm1 is identified as a member of /dev/md0, slot 5.
mdadm: added /dev/sdi1 to /dev/md0 as 1
mdadm: added /dev/sdh1 to /dev/md0 as 2
mdadm: added /dev/sdf1 to /dev/md0 as 3
mdadm: added /dev/sdl1 to /dev/md0 as 4
mdadm: added /dev/sdm1 to /dev/md0 as 5
mdadm: added /dev/sde1 to /dev/md0 as 6
mdadm: added /dev/sdj1 to /dev/md0 as 7
mdadm: added /dev/sdd1 to /dev/md0 as 8
mdadm: no uptodate device for slot 9 of /dev/md0
mdadm: no uptodate device for slot 10 of /dev/md0
mdadm: no uptodate device for slot 11 of /dev/md0
mdadm: added /dev/sdc1 to /dev/md0 as 12
mdadm: added /dev/sdb1 to /dev/md0 as 13
mdadm: added /dev/sdk1 to /dev/md0 as 14
mdadm: added /dev/sdg1 to /dev/md0 as 0
mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
mdadm: Not enough devices to start the array.
lurch:~#
lurch:~# mdadm -D /dev/md0
/dev/md0:
Version : 00.90
Creation Time : Sun Aug 3 02:50:49 2008
Raid Level : raid5
Used Dev Size : 976562432 (931.32 GiB 1000.00 GB)
Raid Devices : 12
Total Devices : 12
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Fri Sep 24 22:12:18 2010
State : active, degraded, Not Started
Active Devices : 9
Working Devices : 12
Failed Devices : 0
Spare Devices : 3
Layout : left-symmetric
Chunk Size : 128K
UUID : 6ba6a0be:6f2da934:e368bf24:bd0fce41
Events : 0.68570
Number Major Minor RaidDevice State
0 8 97 0 active sync /dev/sdg1
1 8 129 1 active sync /dev/sdi1
2 8 113 2 active sync /dev/sdh1
3 8 81 3 active sync /dev/sdf1
4 8 177 4 active sync /dev/sdl1
5 8 193 5 active sync /dev/sdm1
6 8 65 6 active sync /dev/sde1
7 8 145 7 active sync /dev/sdj1
8 8 49 8 active sync /dev/sdd1
9 0 0 9 removed
10 0 0 10 removed
11 0 0 11 removed
12 8 33 - spare /dev/sdc1
13 8 17 - spare /dev/sdb1
14 8 161 - spare /dev/sdk1
lurch:~#
Any hints on how to proceed is welcome.
Thank you!
Henrik Holst
^ permalink raw reply [flat|nested] 2+ messages in thread* Re: Q: how to move "spare" back into raid?
2010-09-27 11:45 Q: how to move "spare" back into raid? Henrik Holst
@ 2010-10-03 0:05 ` Henrik Holst
0 siblings, 0 replies; 2+ messages in thread
From: Henrik Holst @ 2010-10-03 0:05 UTC (permalink / raw)
To: linux-raid
Hello again linux-raid!
I solved the problem by recreating the raid and by fixating
the 9 first drives and trying all 6 combinations of sdb1,
sdc1 and sdk1 as the last 3 drivers. And it worked!
The one-liner I used was:
mdadm -C /dev/md0 -l5 -c128 -z 976562432 -n12 --auto=yes
--assume-clean /dev/sdg1 /dev/sdi1 /dev/sdh1 /dev/sdf1 \
/dev/sdl1 /dev/sdm1 /dev/sde1 /dev/sdj1 /dev/sdd1 \
/dev/sdk1 /dev/sdc1 /dev/sdb1
in combination with cryptsetup isLuks /dev/md0
and fsck.jfs -n /dev/mapper/cmd0
It almost worked perfectly! But the file system was damaged
in the initial mistake. But I was able to mount the RAID
again (in R/O with the -oro option) and extract the latest
data which I did not have a backup on, yet.
linux-raid is still great and for sure very robust!
Henrik Holst
On Mon, Sep 27, 2010 at 01:45:20PM +0200, Henrik Holst wrote:
> Hi linux-raid!
>
> I have not been working with linux raid for a few years,
> everything has been working great in my 12x1TB RAID5 system.
> Last week smartctl gave me some information that two of my
> discs where failing so I had to replace them. I used
> dd_rescue to copy the entire discs into two new discs. And
> then I tried to reassemble them. I do not know what I did
> wrong but now the system is in a broken state.
>
> My guess is that I have tried to reassemble the raid from
> /dev/sd{...} instead of /dev/sd{...}1 as I was supposed to.
> If anyone can give me some hints on how I can rescue this
> mess I would be very glad. If the procedure is lengthy I
> would be willing to pay you (Paypal?) something for your
> time to type it up.
>
> The raid state as-is now:
>
> lurch:~# mdadm -A /dev/md0 -Rfv /dev/sd[b-m]1
> mdadm: looking for devices for /dev/md0
> mdadm: /dev/sdb1 is identified as a member of /dev/md0, slot 13.
> mdadm: /dev/sdc1 is identified as a member of /dev/md0, slot 12.
> mdadm: /dev/sdd1 is identified as a member of /dev/md0, slot 8.
> mdadm: /dev/sde1 is identified as a member of /dev/md0, slot 6.
> mdadm: /dev/sdf1 is identified as a member of /dev/md0, slot 3.
> mdadm: /dev/sdg1 is identified as a member of /dev/md0, slot 0.
> mdadm: /dev/sdh1 is identified as a member of /dev/md0, slot 2.
> mdadm: /dev/sdi1 is identified as a member of /dev/md0, slot 1.
> mdadm: /dev/sdj1 is identified as a member of /dev/md0, slot 7.
> mdadm: /dev/sdk1 is identified as a member of /dev/md0, slot 14.
> mdadm: /dev/sdl1 is identified as a member of /dev/md0, slot 4.
> mdadm: /dev/sdm1 is identified as a member of /dev/md0, slot 5.
> mdadm: added /dev/sdi1 to /dev/md0 as 1
> mdadm: added /dev/sdh1 to /dev/md0 as 2
> mdadm: added /dev/sdf1 to /dev/md0 as 3
> mdadm: added /dev/sdl1 to /dev/md0 as 4
> mdadm: added /dev/sdm1 to /dev/md0 as 5
> mdadm: added /dev/sde1 to /dev/md0 as 6
> mdadm: added /dev/sdj1 to /dev/md0 as 7
> mdadm: added /dev/sdd1 to /dev/md0 as 8
> mdadm: no uptodate device for slot 9 of /dev/md0
> mdadm: no uptodate device for slot 10 of /dev/md0
> mdadm: no uptodate device for slot 11 of /dev/md0
> mdadm: added /dev/sdc1 to /dev/md0 as 12
> mdadm: added /dev/sdb1 to /dev/md0 as 13
> mdadm: added /dev/sdk1 to /dev/md0 as 14
> mdadm: added /dev/sdg1 to /dev/md0 as 0
> mdadm: failed to RUN_ARRAY /dev/md0: Input/output error
> mdadm: Not enough devices to start the array.
> lurch:~#
> lurch:~# mdadm -D /dev/md0
> /dev/md0:
> Version : 00.90
> Creation Time : Sun Aug 3 02:50:49 2008
> Raid Level : raid5
> Used Dev Size : 976562432 (931.32 GiB 1000.00 GB)
> Raid Devices : 12
> Total Devices : 12
> Preferred Minor : 0
> Persistence : Superblock is persistent
>
> Update Time : Fri Sep 24 22:12:18 2010
> State : active, degraded, Not Started
> Active Devices : 9
> Working Devices : 12
> Failed Devices : 0
> Spare Devices : 3
>
> Layout : left-symmetric
> Chunk Size : 128K
>
> UUID : 6ba6a0be:6f2da934:e368bf24:bd0fce41
> Events : 0.68570
>
> Number Major Minor RaidDevice State
> 0 8 97 0 active sync /dev/sdg1
> 1 8 129 1 active sync /dev/sdi1
> 2 8 113 2 active sync /dev/sdh1
> 3 8 81 3 active sync /dev/sdf1
> 4 8 177 4 active sync /dev/sdl1
> 5 8 193 5 active sync /dev/sdm1
> 6 8 65 6 active sync /dev/sde1
> 7 8 145 7 active sync /dev/sdj1
> 8 8 49 8 active sync /dev/sdd1
> 9 0 0 9 removed
> 10 0 0 10 removed
> 11 0 0 11 removed
>
> 12 8 33 - spare /dev/sdc1
> 13 8 17 - spare /dev/sdb1
> 14 8 161 - spare /dev/sdk1
> lurch:~#
>
> Any hints on how to proceed is welcome.
>
> Thank you!
>
> Henrik Holst
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2010-10-03 0:05 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-09-27 11:45 Q: how to move "spare" back into raid? Henrik Holst
2010-10-03 0:05 ` Henrik Holst
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).