* raid5 with 2 disks not working correctly. Need help!
@ 2008-12-16 13:36 Roy Lim
2009-01-05 8:19 ` Roy Lim
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Roy Lim @ 2008-12-16 13:36 UTC (permalink / raw)
To: linux-raid
Hi
I have ran into a serious problem with my raid5 array and need expert
help on this one.
My array is with 5x500GB disks none spare, I've accidentally did a
unclean shutdown of the server. When it came up it gave me this error:
Dec 15 09:07:11 ares kernel: md: kicking non-fresh hdk1 from array!
Dec 15 09:07:11 ares kernel: md: unbind<hdk1>
Dec 15 09:07:11 ares kernel: md: export_rdev(hdk1)
Dec 15 09:07:11 ares kernel: md: md0: raid array is not clean --
starting background reconstruction
Dec 15 09:07:11 ares kernel: raid5: device hde1 operational as raid disk 0
Dec 15 09:07:11 ares kernel: raid5: device sda1 operational as raid disk 4
Dec 15 09:07:11 ares kernel: raid5: device hdi1 operational as raid disk 2
Dec 15 09:07:11 ares kernel: raid5: device hdg1 operational as raid disk 1
Dec 15 09:07:11 ares kernel: raid5: cannot start dirty degraded array for md0
However i managed to get this back up with remove and add, the system
was rebuilding the array. around 70-80% in the process i was going to
prepare to decrypt it (luks crypted) so I'm going to mount the usb
stick witch has the encryption key however when i was going to mount
it i tried to mount wrong device, resulting trying to mount a device
in the array (sda1) i hit repeatedly crtl+c, now taking a look at the
details on the array it looks like this:
/dev/md0:
Version : 00.90.03
Creation Time : Mon Feb 4 18:25:28 2008
Raid Level : raid5
Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
Device Size : 488383936 (465.76 GiB 500.11 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Mon Dec 15 16:02:54 2008
State : clean, degraded
Active Devices : 3
Working Devices : 4
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 64K
UUID : 856a6e4e:a1663e98:9efbd2b1:7507133b
Events : 0.74
Number Major Minor RaidDevice State
0 33 1 0 active sync /dev/hde1
1 34 1 1 active sync /dev/hdg1
2 56 1 2 active sync /dev/hdi1
3 0 0 3 removed
4 0 0 4 removed
5 57 1 - spare /dev/hdk1
6 8 1 - faulty spare /dev/sda1
a quick summary: hdk1 was the one failing in the first place, witch
was about to get rebuild, sda1 was the device i tried to mount while
reconstruction the data. As you can see, hdk1 is now marked as spare
and sda1 a faulty spare. I have not touch the array after this. Can
anyone help me out? How can i force mdadm to set the sda1 to active
sync so i can mount the array and start a backup? Can i use dd_rescue
to file of all of the disks and play with it?
All help is appreciated!
/Roy
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: raid5 with 2 disks not working correctly. Need help!
2008-12-16 13:36 raid5 with 2 disks not working correctly. Need help! Roy Lim
@ 2009-01-05 8:19 ` Roy Lim
2009-01-05 10:27 ` Neil Brown
2009-01-14 19:35 ` Roy Lim
2 siblings, 0 replies; 5+ messages in thread
From: Roy Lim @ 2009-01-05 8:19 UTC (permalink / raw)
To: linux-raid
Hi again,
Can anybody please help me out here?
Roy
On Tue, Dec 16, 2008 at 2:36 PM, Roy Lim <cableroy@gmail.com> wrote:
> Hi
> I have ran into a serious problem with my raid5 array and need expert
> help on this one.
>
> My array is with 5x500GB disks none spare, I've accidentally did a
> unclean shutdown of the server. When it came up it gave me this error:
>
> Dec 15 09:07:11 ares kernel: md: kicking non-fresh hdk1 from array!
> Dec 15 09:07:11 ares kernel: md: unbind<hdk1>
> Dec 15 09:07:11 ares kernel: md: export_rdev(hdk1)
> Dec 15 09:07:11 ares kernel: md: md0: raid array is not clean --
> starting background reconstruction
> Dec 15 09:07:11 ares kernel: raid5: device hde1 operational as raid disk 0
> Dec 15 09:07:11 ares kernel: raid5: device sda1 operational as raid disk 4
> Dec 15 09:07:11 ares kernel: raid5: device hdi1 operational as raid disk 2
> Dec 15 09:07:11 ares kernel: raid5: device hdg1 operational as raid disk 1
> Dec 15 09:07:11 ares kernel: raid5: cannot start dirty degraded array for md0
>
> However i managed to get this back up with remove and add, the system
> was rebuilding the array. around 70-80% in the process i was going to
> prepare to decrypt it (luks crypted) so I'm going to mount the usb
> stick witch has the encryption key however when i was going to mount
> it i tried to mount wrong device, resulting trying to mount a device
> in the array (sda1) i hit repeatedly crtl+c, now taking a look at the
> details on the array it looks like this:
>
> /dev/md0:
> Version : 00.90.03
> Creation Time : Mon Feb 4 18:25:28 2008
> Raid Level : raid5
> Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
> Device Size : 488383936 (465.76 GiB 500.11 GB)
> Raid Devices : 5
> Total Devices : 5
> Preferred Minor : 0
> Persistence : Superblock is persistent
>
> Update Time : Mon Dec 15 16:02:54 2008
> State : clean, degraded
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 1
> Spare Devices : 1
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> UUID : 856a6e4e:a1663e98:9efbd2b1:7507133b
> Events : 0.74
>
> Number Major Minor RaidDevice State
> 0 33 1 0 active sync /dev/hde1
> 1 34 1 1 active sync /dev/hdg1
> 2 56 1 2 active sync /dev/hdi1
> 3 0 0 3 removed
> 4 0 0 4 removed
>
> 5 57 1 - spare /dev/hdk1
> 6 8 1 - faulty spare /dev/sda1
>
> a quick summary: hdk1 was the one failing in the first place, witch
> was about to get rebuild, sda1 was the device i tried to mount while
> reconstruction the data. As you can see, hdk1 is now marked as spare
> and sda1 a faulty spare. I have not touch the array after this. Can
> anyone help me out? How can i force mdadm to set the sda1 to active
> sync so i can mount the array and start a backup? Can i use dd_rescue
> to file of all of the disks and play with it?
>
> All help is appreciated!
>
> /Roy
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: raid5 with 2 disks not working correctly. Need help!
2008-12-16 13:36 raid5 with 2 disks not working correctly. Need help! Roy Lim
2009-01-05 8:19 ` Roy Lim
@ 2009-01-05 10:27 ` Neil Brown
2009-01-14 19:35 ` Roy Lim
2 siblings, 0 replies; 5+ messages in thread
From: Neil Brown @ 2009-01-05 10:27 UTC (permalink / raw)
To: Roy Lim; +Cc: linux-raid
On Tuesday December 16, cableroy@gmail.com wrote:
> Hi
> I have ran into a serious problem with my raid5 array and need expert
> help on this one.
>
> My array is with 5x500GB disks none spare, I've accidentally did a
> unclean shutdown of the server. When it came up it gave me this error:
>
> Dec 15 09:07:11 ares kernel: md: kicking non-fresh hdk1 from array!
> Dec 15 09:07:11 ares kernel: md: unbind<hdk1>
> Dec 15 09:07:11 ares kernel: md: export_rdev(hdk1)
> Dec 15 09:07:11 ares kernel: md: md0: raid array is not clean --
> starting background reconstruction
> Dec 15 09:07:11 ares kernel: raid5: device hde1 operational as raid disk 0
> Dec 15 09:07:11 ares kernel: raid5: device sda1 operational as raid disk 4
> Dec 15 09:07:11 ares kernel: raid5: device hdi1 operational as raid disk 2
> Dec 15 09:07:11 ares kernel: raid5: device hdg1 operational as raid disk 1
> Dec 15 09:07:11 ares kernel: raid5: cannot start dirty degraded array for md0
>
> However i managed to get this back up with remove and add, the system
> was rebuilding the array. around 70-80% in the process i was going to
> prepare to decrypt it (luks crypted) so I'm going to mount the usb
> stick witch has the encryption key however when i was going to mount
> it i tried to mount wrong device, resulting trying to mount a device
> in the array (sda1) i hit repeatedly crtl+c, now taking a look at the
> details on the array it looks like this:
Trying to mount sda1 should have simply failed as the device was in
use. I wonder what really happened....
>
> /dev/md0:
> Version : 00.90.03
> Creation Time : Mon Feb 4 18:25:28 2008
> Raid Level : raid5
> Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
> Device Size : 488383936 (465.76 GiB 500.11 GB)
> Raid Devices : 5
> Total Devices : 5
> Preferred Minor : 0
> Persistence : Superblock is persistent
>
> Update Time : Mon Dec 15 16:02:54 2008
> State : clean, degraded
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 1
> Spare Devices : 1
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> UUID : 856a6e4e:a1663e98:9efbd2b1:7507133b
> Events : 0.74
>
> Number Major Minor RaidDevice State
> 0 33 1 0 active sync /dev/hde1
> 1 34 1 1 active sync /dev/hdg1
> 2 56 1 2 active sync /dev/hdi1
> 3 0 0 3 removed
> 4 0 0 4 removed
>
> 5 57 1 - spare /dev/hdk1
> 6 8 1 - faulty spare /dev/sda1
>
> a quick summary: hdk1 was the one failing in the first place, witch
> was about to get rebuild, sda1 was the device i tried to mount while
> reconstruction the data. As you can see, hdk1 is now marked as spare
> and sda1 a faulty spare. I have not touch the array after this. Can
> anyone help me out? How can i force mdadm to set the sda1 to active
> sync so i can mount the array and start a backup? Can i use dd_rescue
> to file of all of the disks and play with it?
>
> All help is appreciated!
Your only option at this stage is to re-create the array with the best
devices. Create it will one device missing so that it won't try a
resync.
e.g.
mdadm -C /dev/md0 -l5 -n5 /dev/hde1 /dev/hdg1 /dev/hdi1 missing /dev/sda1
then try to access the data to see if it looks right.
This will not modify the data on any devices (until you try writing to
md0), only the RAID metadata.
If you are happy that md0 looks OK, you can then
mdadm /dev/md0 --add /dev/hdk1
and it will recover data to hdk1 and you will have redundancy back
again.
Good luck,
NeilBrown
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: raid5 with 2 disks not working correctly. Need help!
2008-12-16 13:36 raid5 with 2 disks not working correctly. Need help! Roy Lim
2009-01-05 8:19 ` Roy Lim
2009-01-05 10:27 ` Neil Brown
@ 2009-01-14 19:35 ` Roy Lim
2009-01-17 20:59 ` Roy Lim
2 siblings, 1 reply; 5+ messages in thread
From: Roy Lim @ 2009-01-14 19:35 UTC (permalink / raw)
To: linux-raid
Call me coward but...
ares:~# mdadm -C /dev/md0 -l5 -n5 /dev/hde1 /dev/hdg1 /dev/hdi1
missing /dev/sda1
mdadm: /dev/hde1 appears to contain an ext2fs file system
size=1953535744K mtime=Mon Feb 4 20:57:17 2008
mdadm: /dev/hde1 appears to be part of a raid array:
level=raid5 devices=5 ctime=Mon Feb 4 18:25:28 2008
mdadm: /dev/hdg1 appears to be part of a raid array:
level=raid5 devices=5 ctime=Mon Feb 4 18:25:28 2008
Continue creating array?
hde1 appears to contain an ext2fs file system?? No i don't think so..
But experts... can't i create with assume-clean without hdk1 then? I
ask cause there is 1.5TB data there and i'm not any expert on raid :)
Roy
On Tue, Dec 16, 2008 at 2:36 PM, Roy Lim <cableroy@gmail.com> wrote:
> Hi
> I have ran into a serious problem with my raid5 array and need expert
> help on this one.
>
> My array is with 5x500GB disks none spare, I've accidentally did a
> unclean shutdown of the server. When it came up it gave me this error:
>
> Dec 15 09:07:11 ares kernel: md: kicking non-fresh hdk1 from array!
> Dec 15 09:07:11 ares kernel: md: unbind<hdk1>
> Dec 15 09:07:11 ares kernel: md: export_rdev(hdk1)
> Dec 15 09:07:11 ares kernel: md: md0: raid array is not clean --
> starting background reconstruction
> Dec 15 09:07:11 ares kernel: raid5: device hde1 operational as raid disk 0
> Dec 15 09:07:11 ares kernel: raid5: device sda1 operational as raid disk 4
> Dec 15 09:07:11 ares kernel: raid5: device hdi1 operational as raid disk 2
> Dec 15 09:07:11 ares kernel: raid5: device hdg1 operational as raid disk 1
> Dec 15 09:07:11 ares kernel: raid5: cannot start dirty degraded array for md0
>
> However i managed to get this back up with remove and add, the system
> was rebuilding the array. around 70-80% in the process i was going to
> prepare to decrypt it (luks crypted) so I'm going to mount the usb
> stick witch has the encryption key however when i was going to mount
> it i tried to mount wrong device, resulting trying to mount a device
> in the array (sda1) i hit repeatedly crtl+c, now taking a look at the
> details on the array it looks like this:
>
> /dev/md0:
> Version : 00.90.03
> Creation Time : Mon Feb 4 18:25:28 2008
> Raid Level : raid5
> Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
> Device Size : 488383936 (465.76 GiB 500.11 GB)
> Raid Devices : 5
> Total Devices : 5
> Preferred Minor : 0
> Persistence : Superblock is persistent
>
> Update Time : Mon Dec 15 16:02:54 2008
> State : clean, degraded
> Active Devices : 3
> Working Devices : 4
> Failed Devices : 1
> Spare Devices : 1
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> UUID : 856a6e4e:a1663e98:9efbd2b1:7507133b
> Events : 0.74
>
> Number Major Minor RaidDevice State
> 0 33 1 0 active sync /dev/hde1
> 1 34 1 1 active sync /dev/hdg1
> 2 56 1 2 active sync /dev/hdi1
> 3 0 0 3 removed
> 4 0 0 4 removed
>
> 5 57 1 - spare /dev/hdk1
> 6 8 1 - faulty spare /dev/sda1
>
> a quick summary: hdk1 was the one failing in the first place, witch
> was about to get rebuild, sda1 was the device i tried to mount while
> reconstruction the data. As you can see, hdk1 is now marked as spare
> and sda1 a faulty spare. I have not touch the array after this. Can
> anyone help me out? How can i force mdadm to set the sda1 to active
> sync so i can mount the array and start a backup? Can i use dd_rescue
> to file of all of the disks and play with it?
>
> All help is appreciated!
>
> /Roy
>
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: raid5 with 2 disks not working correctly. Need help!
2009-01-14 19:35 ` Roy Lim
@ 2009-01-17 20:59 ` Roy Lim
0 siblings, 0 replies; 5+ messages in thread
From: Roy Lim @ 2009-01-17 20:59 UTC (permalink / raw)
To: linux-raid
Hi!
Just wanted to say that the array is up and running and no data loss
so I'm quite happy now!
So thank you for the help on the command Neil Brown.
/Roy
On Wed, Jan 14, 2009 at 8:35 PM, Roy Lim <cableroy@gmail.com> wrote:
> Call me coward but...
> ares:~# mdadm -C /dev/md0 -l5 -n5 /dev/hde1 /dev/hdg1 /dev/hdi1
> missing /dev/sda1
> mdadm: /dev/hde1 appears to contain an ext2fs file system
> size=1953535744K mtime=Mon Feb 4 20:57:17 2008
> mdadm: /dev/hde1 appears to be part of a raid array:
> level=raid5 devices=5 ctime=Mon Feb 4 18:25:28 2008
> mdadm: /dev/hdg1 appears to be part of a raid array:
> level=raid5 devices=5 ctime=Mon Feb 4 18:25:28 2008
> Continue creating array?
>
>
> hde1 appears to contain an ext2fs file system?? No i don't think so..
>
> But experts... can't i create with assume-clean without hdk1 then? I
> ask cause there is 1.5TB data there and i'm not any expert on raid :)
>
>
> Roy
>
> On Tue, Dec 16, 2008 at 2:36 PM, Roy Lim <cableroy@gmail.com> wrote:
>> Hi
>> I have ran into a serious problem with my raid5 array and need expert
>> help on this one.
>>
>> My array is with 5x500GB disks none spare, I've accidentally did a
>> unclean shutdown of the server. When it came up it gave me this error:
>>
>> Dec 15 09:07:11 ares kernel: md: kicking non-fresh hdk1 from array!
>> Dec 15 09:07:11 ares kernel: md: unbind<hdk1>
>> Dec 15 09:07:11 ares kernel: md: export_rdev(hdk1)
>> Dec 15 09:07:11 ares kernel: md: md0: raid array is not clean --
>> starting background reconstruction
>> Dec 15 09:07:11 ares kernel: raid5: device hde1 operational as raid disk 0
>> Dec 15 09:07:11 ares kernel: raid5: device sda1 operational as raid disk 4
>> Dec 15 09:07:11 ares kernel: raid5: device hdi1 operational as raid disk 2
>> Dec 15 09:07:11 ares kernel: raid5: device hdg1 operational as raid disk 1
>> Dec 15 09:07:11 ares kernel: raid5: cannot start dirty degraded array for md0
>>
>> However i managed to get this back up with remove and add, the system
>> was rebuilding the array. around 70-80% in the process i was going to
>> prepare to decrypt it (luks crypted) so I'm going to mount the usb
>> stick witch has the encryption key however when i was going to mount
>> it i tried to mount wrong device, resulting trying to mount a device
>> in the array (sda1) i hit repeatedly crtl+c, now taking a look at the
>> details on the array it looks like this:
>>
>> /dev/md0:
>> Version : 00.90.03
>> Creation Time : Mon Feb 4 18:25:28 2008
>> Raid Level : raid5
>> Array Size : 1953535744 (1863.04 GiB 2000.42 GB)
>> Device Size : 488383936 (465.76 GiB 500.11 GB)
>> Raid Devices : 5
>> Total Devices : 5
>> Preferred Minor : 0
>> Persistence : Superblock is persistent
>>
>> Update Time : Mon Dec 15 16:02:54 2008
>> State : clean, degraded
>> Active Devices : 3
>> Working Devices : 4
>> Failed Devices : 1
>> Spare Devices : 1
>>
>> Layout : left-symmetric
>> Chunk Size : 64K
>>
>> UUID : 856a6e4e:a1663e98:9efbd2b1:7507133b
>> Events : 0.74
>>
>> Number Major Minor RaidDevice State
>> 0 33 1 0 active sync /dev/hde1
>> 1 34 1 1 active sync /dev/hdg1
>> 2 56 1 2 active sync /dev/hdi1
>> 3 0 0 3 removed
>> 4 0 0 4 removed
>>
>> 5 57 1 - spare /dev/hdk1
>> 6 8 1 - faulty spare /dev/sda1
>>
>> a quick summary: hdk1 was the one failing in the first place, witch
>> was about to get rebuild, sda1 was the device i tried to mount while
>> reconstruction the data. As you can see, hdk1 is now marked as spare
>> and sda1 a faulty spare. I have not touch the array after this. Can
>> anyone help me out? How can i force mdadm to set the sda1 to active
>> sync so i can mount the array and start a backup? Can i use dd_rescue
>> to file of all of the disks and play with it?
>>
>> All help is appreciated!
>>
>> /Roy
>>
>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2009-01-17 20:59 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2008-12-16 13:36 raid5 with 2 disks not working correctly. Need help! Roy Lim
2009-01-05 8:19 ` Roy Lim
2009-01-05 10:27 ` Neil Brown
2009-01-14 19:35 ` Roy Lim
2009-01-17 20:59 ` Roy Lim
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox