Am I dead?

Am I dead?

[Nick Lindsell]

Greetings list,
	My server died last night while copying 4Gb data over the network to
it's IDE raid5 array. While trying to boot it cannot mount root as the 
superblock
is bad for that raid device.
	I've moved the array to another machine with it's own disk so I may be
able to effect a repair by running the raidtools.
	I recreated /etc/raidtab and tried to start the arrays but I get this
in the syslog:

Dec 2 12:41:18 nexnix-192 kernel: [events: 00000018]
Dec 2 12:41:18 nexnix-192 kernel: [events: 0000001d]
Dec 2 12:41:18 nexnix-192 kernel: [events: 0000001c]
Dec 2 12:41:18 nexnix-192 kernel: md: autorun ...
Dec 2 12:41:18 nexnix-192 kernel: md: considering sdc2 ...
Dec 2 12:41:18 nexnix-192 kernel: md: adding sdc2 ...
Dec 2 12:41:18 nexnix-192 kernel: md: adding sdb2 ...
Dec 2 12:41:18 nexnix-192 kernel: md: adding sda2 ...
Dec 2 12:41:18 nexnix-192 kernel: md: created md2
Dec 2 12:41:18 nexnix-192 kernel: md: bind<sda2,1>
Dec 2 12:41:18 nexnix-192 kernel: md: bind<sdb2,2>
Dec 2 12:41:18 nexnix-192 kernel: md: bind<sdc2,3>
Dec 2 12:41:18 nexnix-192 kernel: md: running: <sdc2><sdb2><sda2>
Dec 2 12:41:18 nexnix-192 kernel: md: sdc2's event counter: 0000001c
Dec 2 12:41:18 nexnix-192 kernel: md: sdb2's event counter: 0000001d
Dec 2 12:41:18 nexnix-192 kernel: md: sda2's event counter: 00000018
Dec 2 12:41:18 nexnix-192 kernel: md: superblock update time inconsistency 
-- using the most recent one
Dec 2 12:41:18 nexnix-192 kernel: md: freshest: sdb2
Dec 2 12:41:18 nexnix-192 kernel: md: kicking non-fresh sda2 from array!
Dec 2 12:41:18 nexnix-192 kernel: md: unbind<sda2,2>
Dec 2 12:41:18 nexnix-192 kernel: md: export_rdev(sda2)
Dec 2 12:41:18 nexnix-192 kernel: md2: kicking faulty sdc2!
Dec 2 12:41:18 nexnix-192 kernel: md: unbind<sdc2,1>
Dec 2 12:41:18 nexnix-192 kernel: md: export_rdev(sdc2)
Dec 2 12:41:18 nexnix-192 kernel: md: md2: raid array is not clean -- 
starting background reconstruction
Dec 2 12:41:18 nexnix-192 kernel: md2: max total readahead window set to 512k
Dec 2 12:41:18 nexnix-192 kernel: md2: 2 data-disks, max readahead per 
data-disk: 256k
Dec 2 12:41:18 nexnix-192 kernel: raid5: device sdb2 operational as raid 
disk 1
Dec 2 12:41:18 nexnix-192 kernel: raid5: not enough operational devices for 
md2 (2/3 failed)
Dec 2 12:41:18 nexnix-192 kernel: RAID5 conf printout:
Dec 2 12:41:18 nexnix-192 kernel: --- rd:3 wd:1 fd:2
Dec 2 12:41:18 nexnix-192 kernel: disk 0, s:0, o:0, n:0 rd:0 us:1 dev:[dev 
00:00]
Dec 2 12:41:18 nexnix-192 kernel: disk 1, s:0, o:1, n:1 rd:1 us:1 dev:sdb2
Dec 2 12:41:18 nexnix-192 kernel: disk 2, s:0, o:0, n:2 rd:2 us:1 dev:[dev 
00:00]
Dec 2 12:41:18 nexnix-192 kernel: raid5: failed to run raid set md2
Dec 2 12:41:18 nexnix-192 kernel: md: pers->run() failed ...
Dec 2 12:41:18 nexnix-192 kernel: md :do_md_run() returned -22
Dec 2 12:41:18 nexnix-192 kernel: md: md2 stopped.
Dec 2 12:41:18 nexnix-192 kernel: md: unbind<sdb2,0>
Dec 2 12:41:18 nexnix-192 kernel: md: export_rdev(sdb2)
Dec 2 12:41:18 nexnix-192 kernel: md: ... autorun DONE.

Am I dead or can I "freshen" sda2 so I can at least have 2 devices in my 
array and therefore
have a chance of either backing off the data or rebulding with a new sdc ?

tia
nick

Re: Am I dead?

[Jakob Oestergaard]

On Mon, Dec 02, 2002 at 12:21:30PM +0000, Nick Lindsell wrote:
...
> Am I dead or can I "freshen" sda2 so I can at least have 2 devices in my 
> array and therefore
> have a chance of either backing off the data or rebulding with a new sdc ?

http://unthought.net/Software-RAID.HOWTO/Software-RAID.HOWTO.html

Section 6.1 "Recovery from Multiple Disk Failure"

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Am I dead?

[Gernot A. Weber]

On Mon, 2 Dec 2002, Jakob Oestergaard wrote:

> On Mon, Dec 02, 2002 at 12:21:30PM +0000, Nick Lindsell wrote:
> ...
> > Am I dead or can I "freshen" sda2 so I can at least have 2 devices in my
> > array and therefore
> > have a chance of either backing off the data or rebulding with a new sdc ?
>
> http://unthought.net/Software-RAID.HOWTO/Software-RAID.HOWTO.html
>
> Section 6.1 "Recovery from Multiple Disk Failure"

Thank you soooooooooooo much. I just restored my for weeks as dead
believed raid. It was just a coincidence that the disks weren't
reformatted in the meanwhile. So I was able to recover my home
directories, which had _no_ backup (I changed this last week...).

Bye,

   Gernot

Re: Am I dead?

[Nick Lindsell]

At 13:56 02/12/2002 +0100, you wrote:
>On Mon, Dec 02, 2002 at 12:42:06PM +0000, Nick Lindsell wrote:
> > At 13:32 02/12/2002 +0100, you wrote:
> >
> > Thanks, Jakob - I'll give that a shot.
> > Fingers crossed that I have my raidtab right...........
>
>Yep  ;)

Hmm - you wrote in the HOWTO that
"To get this to work, you'll need to have an up to date /etc/raidtab - if 
it doesn't EXACTLY match devices and ordering of the original disks this 
will not work"
I can work out which disk partition is associated with which array.
Do the arrays have to be listed in the exact same order in raidtab too?

Thanks for the HOWTO, btw.

ttfn
nick



>--
>................................................................
>:   jakob@unthought.net   : And I see the elder races,         :
>:.........................: putrid forms of man                :
>:   Jakob Østergaard      : See him rise and claim the earth,  :
>:        OZ9ABN           : his downfall is at hand.           :
>:.........................:............{Konkhra}...............:

-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Am I dead? NO! (yaay!)

[Nick Lindsell]

Thanks to Jakob I have my data back.  <phew>.
Just for the record I made a raidtab with the following:-

raiddev /dev/md5
	raid-level 5
	nr-raid-disks 3
	chunk-size 64k
	persistent-superblock 1
	nr-spare-disks 0
	device /dev/sda2
	raid-disk 0
	device /dev/sdb2
	raid-disk 1
	device /dev/sdc2
	raid-disk 2

and ran "raidstart /dev/md5". Looking in the syslog
showed that the raiddev was wrong - needed to be /dev/md2.
So I corrected that and ran "mkraid --really-force /dev/md2"
The array now corrected I was able to mount the root filesystem
and copy the original raidtab to my repair disk and run mkraid
on the other partitions. I now have my data.

Finally, what could have caused this? I'm pretty confident about the
hardware (I've not replaced anything yet) - could a power outage have
broken my arrays?

many thanks
nick.