Crashing a software raid 5 partition ( disk trouble )

Crashing a software raid 5 partition ( disk trouble )

[Nicolas Meaux]

Hi,

I have a big problems with my software RAID partition ( its a RAID 5 one ).
I am using a Debian woody on a linux 2.4.21-pre6 kernel.
It was working fine, when suddenly the message appears in syslog :

Jun 10 13:27:22 arda kernel: scsi0: ERROR on channel 0, id 3, lun 0, CDB:
Read (10) 00 02 66 fa 00 00 01 80 00
Jun 10 13:27:22 arda kernel: Info fld=0x266fa1d, Current sd08:30: sense key
Medium Error
Jun 10 13:27:22 arda kernel: Additional sense indicates Unrecovered read
error
Jun 10 13:27:22 arda kernel:  I/O error: dev 08:30, sector 40303128
Jun 10 13:27:22 arda kernel: md: recovery thread got woken up ...
Jun 10 13:27:22 arda kernel: md: updating md2 RAID superblock on device
Jun 10 13:27:22 arda kernel: sym53c896-0-<3,*>: FAST-40 SCSI 40.0 MB/s (25.0
ns, offset 31)
Jun 10 13:27:22 arda kernel: md: recovery thread finished ...
Jun 10 13:27:22 arda kernel: md: (skipping faulty sdd )
Jun 10 13:27:22 arda kernel: scsi0: ERROR on channel 0, id 3, lun 0, CDB:
Read (10) 00 02 66 fa 00 00 01 80 00
Jun 10 13:27:22 arda kernel: Info fld=0x266fa1d, Current sd08:30: sense key
Medium Error
Jun 10 13:27:22 arda kernel: Additional sense indicates Unrecovered read
error
Jun 10 13:27:22 arda kernel:  I/O error: dev 08:30, sector 40303128
Jun 10 13:27:22 arda kernel: raid5: Disk failure on sdd, disabling device.
Operation continuing on 5 devices
Jun 10 13:27:22 arda kernel: md: recovery thread got woken up ...
Jun 10 13:27:22 arda kernel: md: updating md2 RAID superblock on device
Jun 10 13:27:22 arda kernel: sym53c896-0-<3,*>: FAST-40 SCSI 40.0 MB/s (25.0
ns, offset 31)
Jun 10 13:27:22 arda kernel: md2: no spare disk to reconstruct array! --
continuing in degraded mode
Jun 10 13:27:22 arda kernel: md: recovery thread finished ...
Jun 10 13:27:22 arda kernel: md: (skipping faulty sdd )
Jun 10 13:32:03 arda kernel: EXT3-fs error (device md(9,2)): ext3_readdir:
directory #13533203 contains a hole at offset 0
Jun 10 13:32:36 arda kernel: EXT3-fs error (device md(9,2)): ext3_readdir:
directory #13533203 contains a hole at offset 0
Jun 10 13:36:04 arda kernel: EXT3-fs error (device md(9,2)): ext3_readdir:
directory #10846265 contains a hole at offset 0
[...]

It looks like that a disk have a hard problem.
After a reboot, my raid partition goes down.
And i try to restart her, whit :raidstart /dev/md2 :

Jun 10 15:48:30 arda kernel:  [events: 0000003e]
Jun 10 15:48:30 arda kernel:  [events: 00000057]
Jun 10 15:48:30 arda kernel:  [events: 00000057]
Jun 10 15:48:30 arda kernel:  [events: 00000055]
Jun 10 15:48:30 arda kernel:  [events: 00000057]
Jun 10 15:48:30 arda last message repeated 2 times
Jun 10 15:48:30 arda kernel:  [events: 0000003c]
Jun 10 15:48:30 arda kernel: md: autorun ...
Jun 10 15:48:30 arda kernel: md: considering sdb ...
Jun 10 15:48:30 arda kernel: md:  adding sdb ...
Jun 10 15:48:30 arda kernel: md:  adding sdg ...
Jun 10 15:48:30 arda kernel: md:  adding sdh ...
Jun 10 15:48:30 arda kernel: md:  adding sde ...
Jun 10 15:48:30 arda kernel: md:  adding sdd ...
Jun 10 15:48:30 arda kernel: md:  adding sdf ...
Jun 10 15:48:30 arda kernel: md:  adding sdc ...
Jun 10 15:48:30 arda kernel: md:  adding sda ...
Jun 10 15:48:30 arda kernel: md: created md2
Jun 10 15:48:30 arda kernel: md: bind<sda,1>
Jun 10 15:48:30 arda kernel: md: bind<sdc,2>
Jun 10 15:48:30 arda kernel: md: bind<sdf,3>
Jun 10 15:48:30 arda kernel: md: bind<sdd,4>
Jun 10 15:48:30 arda kernel: md: bind<sde,5>
Jun 10 15:48:30 arda kernel: md: bind<sdh,6>
Jun 10 15:48:30 arda kernel: md: bind<sdg,7>
Jun 10 15:48:30 arda kernel: md: bind<sdb,8>
Jun 10 15:48:30 arda kernel: md: running:
<sdb><sdg><sdh><sde><sdd><sdf><sdc><sda>
Jun 10 15:48:30 arda kernel: md: sdb's event counter: 0000003c
Jun 10 15:48:30 arda kernel: md: sdg's event counter: 00000057
Jun 10 15:48:30 arda kernel: md: sdh's event counter: 00000057
Jun 10 15:48:30 arda kernel: md: sde's event counter: 00000057
Jun 10 15:48:30 arda kernel: md: sdd's event counter: 00000055
Jun 10 15:48:30 arda kernel: md: sdf's event counter: 00000057
Jun 10 15:48:30 arda kernel: md: sdc's event counter: 00000057
Jun 10 15:48:30 arda kernel: md: sda's event counter: 0000003e
Jun 10 15:48:30 arda kernel: md: superblock update time inconsistency --
using the most recent one
Jun 10 15:48:30 arda kernel: md: freshest: sdg
Jun 10 15:48:30 arda kernel: md: kicking non-fresh sdb from array!
Jun 10 15:48:30 arda kernel: md: unbind<sdb,7>
Jun 10 15:48:30 arda kernel: md: export_rdev(sdb)
Jun 10 15:48:30 arda kernel: md: kicking non-fresh sdd from array!
Jun 10 15:48:30 arda kernel: md: unbind<sdd,6>
Jun 10 15:48:30 arda kernel: md: export_rdev(sdd)
Jun 10 15:48:30 arda kernel: md: kicking non-fresh sda from array!
Jun 10 15:48:30 arda kernel: md: unbind<sda,5>
Jun 10 15:48:30 arda kernel: md: export_rdev(sda)
Jun 10 15:48:30 arda kernel: md2: removing former faulty sdd!
Jun 10 15:48:30 arda kernel: md2: max total readahead window set to 1488k
Jun 10 15:48:30 arda kernel: md2: 6 data-disks, max readahead per data-disk:
248k
Jun 10 15:48:30 arda kernel: raid5: device sdg operational as raid disk 6
Jun 10 15:48:30 arda kernel: raid5: device sdh operational as raid disk 5
Jun 10 15:48:30 arda kernel: raid5: device sde operational as raid disk 4
Jun 10 15:48:30 arda kernel: raid5: device sdf operational as raid disk 2
Jun 10 15:48:30 arda kernel: raid5: device sdc operational as raid disk 1
Jun 10 15:48:30 arda kernel: raid5: not enough operational devices for md2
(2/7 failed)
Jun 10 15:48:30 arda kernel: RAID5 conf printout:
Jun 10 15:48:30 arda kernel:  --- rd:7 wd:5 fd:2
Jun 10 15:48:30 arda kernel:  disk 0, s:0, o:0, n:0 rd:0 us:1 dev:[dev
00:00]
Jun 10 15:48:30 arda kernel:  disk 1, s:0, o:1, n:1 rd:1 us:1 dev:sdc
Jun 10 15:48:30 arda kernel:  disk 2, s:0, o:1, n:2 rd:2 us:1 dev:sdf
Jun 10 15:48:30 arda kernel:  disk 3, s:0, o:0, n:3 rd:3 us:1 dev:[dev
00:00]
Jun 10 15:48:30 arda kernel:  disk 4, s:0, o:1, n:4 rd:4 us:1 dev:sde
Jun 10 15:48:30 arda kernel:  disk 5, s:0, o:1, n:5 rd:5 us:1 dev:sdh
Jun 10 15:48:30 arda kernel:  disk 6, s:0, o:1, n:6 rd:6 us:1 dev:sdg
Jun 10 15:48:30 arda kernel: raid5: failed to run raid set md2
Jun 10 15:48:30 arda kernel: md: pers->run() failed ...
Jun 10 15:48:30 arda kernel: md :do_md_run() returned -22
Jun 10 15:48:30 arda kernel: md: md2 stopped.
Jun 10 15:48:30 arda kernel: md: unbind<sdg,4>
Jun 10 15:48:30 arda kernel: md: export_rdev(sdg)
Jun 10 15:48:30 arda kernel: md: unbind<sdh,3>
Jun 10 15:48:30 arda kernel: md: export_rdev(sdh)
Jun 10 15:48:30 arda kernel: md: unbind<sde,2>
Jun 10 15:48:30 arda kernel: md: export_rdev(sde)
Jun 10 15:48:30 arda kernel: md: unbind<sdf,1>
Jun 10 15:48:30 arda kernel: md: export_rdev(sdf)
Jun 10 15:48:30 arda kernel: md: unbind<sdc,0>
Jun 10 15:48:30 arda kernel: md: export_rdev(sdc)
Jun 10 15:48:30 arda kernel: md: ... autorun DONE.

My /etc/raidtab, looks like this :
raiddev                         /dev/md2
        raid-level              5
        nr-raid-disks           7
        nr-spare-disks          1
        chunk-size              32
        persistent-superblock   1
        parity-algorithm        left-symmetric
        device                  /dev/sda
        raid-disk               0
        device                  /dev/sdb
        raid-disk               1
        device                  /dev/sdc
        raid-disk               2
        device                  /dev/sdd
        raid-disk               3
        device                  /dev/sde
        raid-disk               4
        device                  /dev/sdh
        raid-disk               5
        device                  /dev/sdg
        raid-disk               6
        device                  /dev/sdf
        spare-disk              0

When, i was making a cat /proc/mdstat before the problem appeared :

md2 : active raid5 sdg[6] sdh[5] sde[4] sdd[3] sdf[2] sdc[1]
      215061504 blocks level 5, 32k chunk, algorithm 2 [7/6] [_UUUUUU]

But unfortunately I dont take care off it.

In the FAQ i read :

In short: quite often you get a temporary failure of several disks at once;
afterwards the RAID superblocks are out of sync and you can no longer init
your RAID array.
One thing left: rewrite the RAID superblocks by mkraid --force


But when i type this command, i have :
DESTROYING the contents of /dev/md2 in 5 seconds, Ctrl-C if unsure!

And i am very afraid off this , i dont want to loose my data.
( my /etc/raidtab is conform as my raid architecture )

Finally, i would like to say, that i have backup the most important data,
but not all the data.

So, the question is : is there any way to restore my raid partition or
recover data on it ?
Or did i have loose everything ?

Thank you to have read me.

    Nicolas

Re: Crashing a software raid 5 partition ( disk trouble )

[Martin Hermanowski]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

On Thu, Jun 12, 2003 at 10:08:09AM +0200, Nicolas Meaux wrote:
[lost 2 members of a raid5]
> In the FAQ i read :
> 
> In short: quite often you get a temporary failure of several disks at once;
> afterwards the RAID superblocks are out of sync and you can no longer init
> your RAID array.
> One thing left: rewrite the RAID superblocks by mkraid --force
> 
> 
> But when i type this command, i have :
> DESTROYING the contents of /dev/md2 in 5 seconds, Ctrl-C if unsure!
> 
> And i am very afraid off this , i dont want to loose my data.
> ( my /etc/raidtab is conform as my raid architecture )
> 
> Finally, i would like to say, that i have backup the most important data,
> but not all the data.
> 
> So, the question is : is there any way to restore my raid partition or
> recover data on it ?
> Or did i have loose everything ?

I did a `mkraid --force' once, after two two hdds locked up nearly
simultaneously. It worked.

But I do not know what happens if one disc has been out of the raid for
a longer time.

LLAP, Martin

-- 
pgp-key: https://mh57.de/cc/view.php?file=/martin/pgp

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]

Re: Crashing a software raid 5 partition ( disk trouble )

[Neil Brown]

On Thursday June 12, nicolas.meaux@artabel.net wrote:
> Hi,
> 
> I have a big problems with my software RAID partition ( its a RAID 5
> one ).

Get mdadm
Use
   mdadm -A /dev/md2 /dev/sd[a-f]

NeilBrown

http://www.kernel.org/pub/linux/utils/raid/mdadm/

Re: Crashing a software raid 5 partition ( disk trouble )

[Neil Brown]

On Thursday June 12, neilb@cse.unsw.edu.au wrote:
> On Thursday June 12, nicolas.meaux@artabel.net wrote:
> > Hi,
> > 
> > I have a big problems with my software RAID partition ( its a RAID 5
> > one ).
> 
> Get mdadm
> Use
>    mdadm -A /dev/md2 /dev/sd[a-f]

Sorry, forgot the --force:

   mdadm -A --force /dev/md2 /dev/sd[a-f]

NB

> 
> NeilBrown
> 
> http://www.kernel.org/pub/linux/utils/raid/mdadm/
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Crashing a software raid 5 partition ( disk trouble )

[Nicolas Meaux]

Thanks for you quick respond Neil  ;)

>    mdadm -A --force /dev/md2 /dev/sd[a-f]

So, i type this command :

# mdadm -A --force /dev/md2 /dev/sd[a-h]
mdadm: /dev/md2 has been started with 7 drives.

( i do a [a-h] , because i have 8 scsi disk )

I try too mount md2, it work fine, but :
# ls -al
ls: nohup.out: Input/output error
ls: tape.sh: Input/output error
ls: .viminfo: Input/output error
ls: .bash_history: Input/output error
ls: .bashrc~: Input/output error
ls: .xauth: Input/output error

It seem that there is many errors on the partition.
So, in good newbie, i decided too reboot the machine, but now and since one
hour, he is blocking on these lines :

[...]
md: updating md2 RAID superblock device
md: sdg [events: 000005a]<6>write sdg's sb offset 35843584
md: sdh [events: 000005a]<6>write sdh's sb offset 35843584
md: sde [events: 000005a]<6>write sde's sb offset 35843584
md: sdd [events: 000005a]<6>write sdd's sb offset 35843584


But, there is disk activity on the first one.

It's normal ? does RAID reconsctruct itsef ? if so , why did not he say that
he is doing it ?


    Nicolas

Re: Crashing a software raid 5 partition ( disk trouble )

[Bernd Schubert]

On Thursday 12 June 2003 12:06, Nicolas Meaux wrote:
> Thanks for you quick respond Neil  ;)
>
> >    mdadm -A --force /dev/md2 /dev/sd[a-f]
>
> So, i type this command :
>
> # mdadm -A --force /dev/md2 /dev/sd[a-h]
> mdadm: /dev/md2 has been started with 7 drives.
>
> ( i do a [a-h] , because i have 8 scsi disk )
>
> I try too mount md2, it work fine, but :
> # ls -al
> ls: nohup.out: Input/output error
> ls: tape.sh: Input/output error
> ls: .viminfo: Input/output error
> ls: .bash_history: Input/output error
> ls: .bashrc~: Input/output error
> ls: .xauth: Input/output error
>
> It seem that there is many errors on the partition.
> So, in good newbie, i decided too reboot the machine, but now and since one
> hour, he is blocking on these lines :
>
> [...]
> md: updating md2 RAID superblock device
> md: sdg [events: 000005a]<6>write sdg's sb offset 35843584
> md: sdh [events: 000005a]<6>write sdh's sb offset 35843584
> md: sde [events: 000005a]<6>write sde's sb offset 35843584
> md: sdd [events: 000005a]<6>write sdd's sb offset 35843584
>

Hi,

why are you rebooting that often ? Even after sdd failed I wouldn't have 
rebooted that fast. Just do 'cat /proc/mdstat' and see in what state your 
raid-array is!

Bernd

maximum disk capacity?

[Donghui Wen]

Hi, I am setting up hardware raid with 7 harddrives on redhat linux.
I use 3ware + hardware raid 5, each harddrive is 200G. For
linux, this raid sould act as a 1.2TB scsi hard drive. But it seems
redhat could not reconize it. I am wondering may be a 1.2TB hard drive
is too large to linux. So my question is: what't
the maxium disk capacity Linux can support?

Thanks!


Donghui

Re: maximum disk capacity?

[Stephan van Hienen]

On Thu, 12 Jun 2003, Donghui Wen wrote:

> Hi, I am setting up hardware raid with 7 harddrives on redhat linux.
> I use 3ware + hardware raid 5, each harddrive is 200G. For
> linux, this raid sould act as a 1.2TB scsi hard drive. But it seems
> redhat could not reconize it. I am wondering may be a 1.2TB hard drive
> is too large to linux. So my question is: what't
> the maxium disk capacity Linux can support?

no problem till 2TB
if you need to go above to 2TB, you need the largedisk patch
which doesn't work with sw raid5 (so no 2TB+ with sw raid5)

i'm running with 13*180GB (+1 Hotspare) in sw raid5 (hd's
connected to 2*3ware 7850)

Re: maximum disk capacity?

[bmoon]

What is the reason that it can not cover more than 2TB?
By the way you are able to use more than 2TB, ~2.3TB???
Why?

----- Original Message ----- 
From: "Stephan van Hienen" <raid@a2000.nu>
To: "Donghui Wen" <dhwen@protegonetworks.com>
Cc: <linux-raid@vger.kernel.org>
Sent: Friday, June 13, 2003 12:51 AM
Subject: Re: maximum disk capacity?


> On Thu, 12 Jun 2003, Donghui Wen wrote:
> 
> > Hi, I am setting up hardware raid with 7 harddrives on redhat linux.
> > I use 3ware + hardware raid 5, each harddrive is 200G. For
> > linux, this raid sould act as a 1.2TB scsi hard drive. But it seems
> > redhat could not reconize it. I am wondering may be a 1.2TB hard drive
> > is too large to linux. So my question is: what't
> > the maxium disk capacity Linux can support?
> 
> no problem till 2TB
> if you need to go above to 2TB, you need the largedisk patch
> which doesn't work with sw raid5 (so no 2TB+ with sw raid5)
> 
> i'm running with 13*180GB (+1 Hotspare) in sw raid5 (hd's
> connected to 2*3ware 7850)
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: maximum disk capacity?

[Peter L. Ashford]

> What is the reason that it can not cover more than 2TB?

A 32-bit block number with 512-byte blocks.  This multiplies out to 2TB.

> By the way you are able to use more than 2TB, ~2.3TB???
> Why?

2TB binary is 2.2TB decimal (2,199,023,255,552).  You shouldn't be able to
get 2.3TB without the patch.

> > > Hi, I am setting up hardware raid with 7 harddrives on redhat linux.
> > > I use 3ware + hardware raid 5, each harddrive is 200G. For
> > > linux, this raid sould act as a 1.2TB scsi hard drive. But it seems
> > > redhat could not reconize it. I am wondering may be a 1.2TB hard drive
> > > is too large to linux. So my question is: what't
> > > the maxium disk capacity Linux can support?

There is a problem with the RedHat installation scripts.  When performing
a Kickstart install, and using the Grub boot loader, disks are limited to
1TB (binary).  Is this the problem you encountered?

Good luck.
				Peter Ashford

Re: maximum disk capacity?

[Donghui Wen]

I tried to use fdisk, sfdisk  to partition /dev/sdb which is 1.2TB.
It always complaint could not read sector 0.

I also tried dd if=/dev/sdb of=boot.sec bs=512 count=1,
it has the same error.

But if I tried to write using
   dd if=/dev/zero of=/dev/sdb bs=512 count=1

It works.

Also, the kernel found this disk, but report the size as a negative number.
I guess if using unsigned interger (32 bit), linux can support 2TB for one
block device.
But some time, the other tools only use (int, signed interger), then it can
only support 1TB.


Donghui

----- Original Message -----
From: "Peter L. Ashford" <ashford@sdsc.edu>
To: "bmoon" <bo@anthologysolutions.com>
Cc: "Stephan van Hienen" <raid@a2000.nu>; "Donghui Wen"
<dhwen@protegonetworks.com>; <linux-raid@vger.kernel.org>
Sent: Friday, June 13, 2003 10:39 AM
Subject: Re: maximum disk capacity?


> > What is the reason that it can not cover more than 2TB?
>
> A 32-bit block number with 512-byte blocks.  This multiplies out to 2TB.
>
> > By the way you are able to use more than 2TB, ~2.3TB???
> > Why?
>
> 2TB binary is 2.2TB decimal (2,199,023,255,552).  You shouldn't be able to
> get 2.3TB without the patch.
>
> > > > Hi, I am setting up hardware raid with 7 harddrives on redhat linux.
> > > > I use 3ware + hardware raid 5, each harddrive is 200G. For
> > > > linux, this raid sould act as a 1.2TB scsi hard drive. But it seems
> > > > redhat could not reconize it. I am wondering may be a 1.2TB hard
drive
> > > > is too large to linux. So my question is: what't
> > > > the maxium disk capacity Linux can support?
>
> There is a problem with the RedHat installation scripts.  When performing
> a Kickstart install, and using the Grub boot loader, disks are limited to
> 1TB (binary).  Is this the problem you encountered?
>
> Good luck.
> Peter Ashford
>
>

Re: maximum disk capacity?

[bmoon]

My question is from  Stephan van Hienen's email;

"
         if you need to go above to 2TB, you need the largedisk patch
         which doesn't work with sw raid5 (so no 2TB+ with sw raid5)

          i'm running with 13*180GB (+1 Hotspare) in sw raid5 (hd's
         connected to 2*3ware 7850)
"

What is the reason why it can not work with SW raid5 even with the largedisk
patch???



----- Original Message -----
From: "Peter L. Ashford" <ashford@sdsc.edu>
To: "bmoon" <bo@anthologysolutions.com>
Cc: "Stephan van Hienen" <raid@a2000.nu>; "Donghui Wen"
<dhwen@protegonetworks.com>; <linux-raid@vger.kernel.org>
Sent: Friday, June 13, 2003 10:39 AM
Subject: Re: maximum disk capacity?


> > What is the reason that it can not cover more than 2TB?
>
> A 32-bit block number with 512-byte blocks.  This multiplies out to 2TB.
>
> > By the way you are able to use more than 2TB, ~2.3TB???
> > Why?
>
> 2TB binary is 2.2TB decimal (2,199,023,255,552).  You shouldn't be able to
> get 2.3TB without the patch.
>
> > > > Hi, I am setting up hardware raid with 7 harddrives on redhat linux.
> > > > I use 3ware + hardware raid 5, each harddrive is 200G. For
> > > > linux, this raid sould act as a 1.2TB scsi hard drive. But it seems
> > > > redhat could not reconize it. I am wondering may be a 1.2TB hard
drive
> > > > is too large to linux. So my question is: what't
> > > > the maxium disk capacity Linux can support?
>
> There is a problem with the RedHat installation scripts.  When performing
> a Kickstart install, and using the Grub boot loader, disks are limited to
> 1TB (binary).  Is this the problem you encountered?
>
> Good luck.
> Peter Ashford
>
>

Re: maximum disk capacity?

[Stephan van Hienen]

On Fri, 13 Jun 2003, bmoon wrote:

> What is the reason why it can not work with SW raid5 even with the largedisk
> patch???
the largedisk patch is not for the sw raid5 code
(check the archive for my emails about this 'problem')

Re: maximum disk capacity?

[Stephan van Hienen]

On Fri, 13 Jun 2003, bmoon wrote:

> What is the reason that it can not cover more than 2TB?
32 bits raid5 code

> By the way you are able to use more than 2TB, ~2.3TB???
> Why?
check the archive from this mailinglist
i was able to format 2.3TB (15*180GB raid5)
but when i was writing to the 2TB+ i lost my filesystem

Re: Crashing a software raid 5 partition ( disk trouble )

[Neil Brown]

On Thursday June 12, nicolas.meaux@artabel.net wrote:
> Thanks for you quick respond Neil  ;)
> 
> >    mdadm -A --force /dev/md2 /dev/sd[a-f]
> 
> So, i type this command :
> 
> # mdadm -A --force /dev/md2 /dev/sd[a-h]
> mdadm: /dev/md2 has been started with 7 drives.
> 
> ( i do a [a-h] , because i have 8 scsi disk )
> 
> I try too mount md2, it work fine, but :
> # ls -al
> ls: nohup.out: Input/output error
> ls: tape.sh: Input/output error
> ls: .viminfo: Input/output error
> ls: .bash_history: Input/output error
> ls: .bashrc~: Input/output error
> ls: .xauth: Input/output error

Looks like filesystem corruption, which isn't overly surprising.
I would 'fsck' at this point, then copy off the data that you need (if
it is there) and recreate the array, the filesystem, and restore from
backups.


NeilBrown