HELP 13 disk raid 5 not starting

HELP 13 disk raid 5 not starting

[Jim Buttafuoco]

all,

I have a 13 disk raid 5 set with 4 disks marks as "clean" and the rest marked as dirty.  When I do the following command
to start the raid set (md0) I get an error.  Any ideas on how to recover?

This is a debian sarge system running kernel 2.6.8-1-686-smp and mdadm version v1.4.0 - 29 Oct 2003

Thanks
Jim

# mdadm --assemble --scan
mdadm: /dev/md0 assembled from 6 drives and 1 spare - not enough to start the array.

my /etc/mdadm/mdadm.conf looks like

DEVICE partitions
ARRAY /dev/md0 level=raid5 num-devices=12 UUID=c9669a6b:cdc5b9d4:1df0802c:86d53e17


and /proc/mdstat looks like

Personalities : [raid5] 
md0 : inactive sdb1[0] sdm1[12] sdn1[11] sdl1[10] sdk1[9] sdj1[8] sdi1[7] sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1]
      3173710592 blocks
unused devices: <none>

Re: HELP 13 disk raid 5 not starting

[Neil Brown]

On Tuesday November 15, jim@contactbda.com wrote:
> all,
> 
> I have a 13 disk raid 5 set with 4 disks marks as "clean" and the
> rest marked as dirty.

And important question to answer is 'how did this happen'?

> When I do the following command
> to start the raid set (md0) I get an error.  Any ideas on how to
> recover?

Add '--force' to the 'mdadm --assemble' command.  This tells mdadm to
try really hard to assemble the array, modifying info in the
super blocks if necessary.
Be aware that though doing this will normally give you a working
array, there may be data corruption within the array (it depends
somewhat on the answer to that first important question).
I would recommend at least an 'fsck' if that is practical.

The array will be assembled degraded.  You will need to add in a spare
if you are happy that the data is sufficiently intact.

NeilBrown

Re: HELP 13 disk raid 5 not starting

[Jim Buttafuoco]

I believe there were data access errors on the console (scrolling to fast to read).  I will try the force and see what 
happends.




---------- Original Message -----------
From: Neil Brown <neilb@suse.de>
To: jim@contactbda.com
Cc: linux-raid@vger.kernel.org
Sent: Wed, 16 Nov 2005 10:16:22 +1100
Subject: Re: HELP 13 disk raid 5 not starting

> On Tuesday November 15, jim@contactbda.com wrote:
> > all,
> > 
> > I have a 13 disk raid 5 set with 4 disks marks as "clean" and the
> > rest marked as dirty.
> 
> And important question to answer is 'how did this happen'?
> 
> > When I do the following command
> > to start the raid set (md0) I get an error.  Any ideas on how to
> > recover?
> 
> Add '--force' to the 'mdadm --assemble' command.  This tells mdadm to
> try really hard to assemble the array, modifying info in the
> super blocks if necessary.
> Be aware that though doing this will normally give you a working
> array, there may be data corruption within the array (it depends
> somewhat on the answer to that first important question).
> I would recommend at least an 'fsck' if that is practical.
> 
> The array will be assembled degraded.  You will need to add in a spare
> if you are happy that the data is sufficiently intact.
> 
> NeilBrown
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
------- End of Original Message -------

Re: HELP 13 disk raid 5 not starting

[Dan Stromberg]

On Wed, 2005-11-16 at 10:16 +1100, Neil Brown wrote:
> On Tuesday November 15, jim@contactbda.com wrote:
> > all,
> > 
> > I have a 13 disk raid 5 set with 4 disks marks as "clean" and the
> > rest marked as dirty.
> 
> And important question to answer is 'how did this happen'?
> 
> > When I do the following command
> > to start the raid set (md0) I get an error.  Any ideas on how to
> > recover?
> 
> Add '--force' to the 'mdadm --assemble' command.  This tells mdadm to
> try really hard to assemble the array, modifying info in the
> super blocks if necessary.
> Be aware that though doing this will normally give you a working
> array, there may be data corruption within the array (it depends
> somewhat on the answer to that first important question).
> I would recommend at least an 'fsck' if that is practical.
> 
> The array will be assembled degraded.  You will need to add in a spare
> if you are happy that the data is sufficiently intact.

Before you do anything that might make things worse (--force is a good
thing to try, I'm sure, but sometimes the good thing to try causes
further problems, EG sometimes fsck goes kerflooey on ya), you might
want to see about getting an image backup of all the drives in the RAID
array to some other system, if you have the space.  If your array wasn't
that full, you may be able to compress the image backups to good effect.

Re: HELP 13 disk raid 5 not starting

[Jim Buttafuoco]

Neil,

Thanks for the reply, the --force worked great,  md0 is syncing now, I will run testing against my database once the 
sync completes in 400 minutes.

Jim



---------- Original Message -----------
From: Neil Brown <neilb@suse.de>
To: jim@contactbda.com
Cc: linux-raid@vger.kernel.org
Sent: Wed, 16 Nov 2005 10:16:22 +1100
Subject: Re: HELP 13 disk raid 5 not starting

> On Tuesday November 15, jim@contactbda.com wrote:
> > all,
> > 
> > I have a 13 disk raid 5 set with 4 disks marks as "clean" and the
> > rest marked as dirty.
> 
> And important question to answer is 'how did this happen'?
> 
> > When I do the following command
> > to start the raid set (md0) I get an error.  Any ideas on how to
> > recover?
> 
> Add '--force' to the 'mdadm --assemble' command.  This tells mdadm to
> try really hard to assemble the array, modifying info in the
> super blocks if necessary.
> Be aware that though doing this will normally give you a working
> array, there may be data corruption within the array (it depends
> somewhat on the answer to that first important question).
> I would recommend at least an 'fsck' if that is practical.
> 
> The array will be assembled degraded.  You will need to add in a spare
> if you are happy that the data is sufficiently intact.
> 
> NeilBrown
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
------- End of Original Message -------

Re: HELP 13 disk raid 5 not starting

[Jim Buttafuoco]

The disks contain a ~2TB Postgresql database so i was unable to copy to another system.  The force worked as suggested 
by Neil.

Thanks for the reply
Jim



---------- Original Message -----------
From: Dan Stromberg <strombrg@dcs.nac.uci.edu>
To: Neil Brown <neilb@suse.de>
Cc: jim@contactbda.com, linux-raid@vger.kernel.org, strombrg@dcs.nac.uci.edu
Sent: Tue, 15 Nov 2005 15:27:04 -0800
Subject: Re: HELP 13 disk raid 5 not starting

> On Wed, 2005-11-16 at 10:16 +1100, Neil Brown wrote:
> > On Tuesday November 15, jim@contactbda.com wrote:
> > > all,
> > > 
> > > I have a 13 disk raid 5 set with 4 disks marks as "clean" and the
> > > rest marked as dirty.
> > 
> > And important question to answer is 'how did this happen'?
> > 
> > > When I do the following command
> > > to start the raid set (md0) I get an error.  Any ideas on how to
> > > recover?
> > 
> > Add '--force' to the 'mdadm --assemble' command.  This tells mdadm to
> > try really hard to assemble the array, modifying info in the
> > super blocks if necessary.
> > Be aware that though doing this will normally give you a working
> > array, there may be data corruption within the array (it depends
> > somewhat on the answer to that first important question).
> > I would recommend at least an 'fsck' if that is practical.
> > 
> > The array will be assembled degraded.  You will need to add in a spare
> > if you are happy that the data is sufficiently intact.
> 
> Before you do anything that might make things worse (--force is a good
> thing to try, I'm sure, but sometimes the good thing to try causes
> further problems, EG sometimes fsck goes kerflooey on ya), you might
> want to see about getting an image backup of all the drives in the RAID
> array to some other system, if you have the space.  If your array wasn't
> that full, you may be able to compress the image backups to good effect.
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
------- End of Original Message -------