differnet UUIDs and no of spares :(

differnet UUIDs and no of spares :(

[Eamonn Hamilton]

Hi Guys,

I'm looking at a server with a bunch of disks that had a raid 5 with two
spares, however, one of the spares failed, the system then started
rebuilding on the other and it crashed during the rebuild.

I'm now left in the following situation :


for a in a b c d e f g h i; do mdadm --examine --scan /dev/sd${a}; done
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
   spares=1
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
   spares=1
ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0


The system complains because of the different uuids, and refuses to
recreate the array.

Is it basically stuffed, or is there something I can do to recover the
2TB filesystem that's on there ?

Cheers,
Eamonn

Re: differnet UUIDs and no of spares :(

[Bill Davidsen]

Eamonn Hamilton wrote:
> Hi Guys,
>
> I'm looking at a server with a bunch of disks that had a raid 5 with two
> spares, however, one of the spares failed, the system then started
> rebuilding on the other and it crashed during the rebuild.
>
> I'm now left in the following situation :
>
>
> for a in a b c d e f g h i; do mdadm --examine --scan /dev/sd${a}; done
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
>    spares=1
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
>    spares=1
> ARRAY /dev/md0 level=raid5 num-devices=7 UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
>
>
> The system complains because of the different uuids, and refuses to
> recreate the array.
>
> Is it basically stuffed, or is there something I can do to recover the
> 2TB filesystem that's on there ?
>   

What can you tell us about how that happened? When (if ever) was it 
running, how was it created, etc, etc.

You could probably try some things like trying to start it read-only 
using --force, but don't do that yet, if you get it wrong you WILL be 
likely to lose data.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: differnet UUIDs and no of spares :(

[Eamonn Hamilton]

Hi Bill,

> 
> What can you tell us about how that happened? When (if ever) was it 
> running, how was it created, etc, etc.

It was running stably for a long time, with the raid5 being expanded
from 1TB to 2. It recently had two spare drives added to it, and again
was running fine, however  a new kernel on the box provoked some nasty
behaviour with the sata_nv driver.

This left the raid not starting at boot-up, as devmapper had claimed one
of the spares as it's own, so I started the array without that spare,
which provoked a re-sync for reasons I'm not clear on. Having cleared
the devmapper mapping on the other spare, it was added back in as a
spare to the array. 15 minutes into the resync, however, the biox fell
over, whether as a result of the array resync or something else I dont
know, there's nothing suspicious in the logs. Which brings us to where
we are today :(


> 
> You could probably try some things like trying to start it read-only 
> using --force, but don't do that yet, if you get it wrong you WILL be 
> likely to lose data.
> 

I was tempted to change the UUID on the nonstarting members, but the
number of spares being different gave me pause.

Re: differnet UUIDs and no of spares :(

[eamonn]

On Wed, January 14, 2009 10:50 pm, Bill Davidsen wrote:
> Eamonn Hamilton wrote:
>
>> Hi Guys,
>>
>>
>> I'm looking at a server with a bunch of disks that had a raid 5 with
>> two spares, however, one of the spares failed, the system then started
>> rebuilding on the other and it crashed during the rebuild.
>>
>> I'm now left in the following situation :
>>
>>
>>
>> for a in a b c d e f g h i; do mdadm --examine --scan /dev/sd${a}; done
>>  ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
>> ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
>> ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
>> ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
>> ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:5feed4c5:31c51eb2
>> ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
>> spares=1 ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
>> ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
>> spares=1 ARRAY /dev/md0 level=raid5 num-devices=7
>> UUID=e1e75e8b:f9f387cd:8d12a2d2:3188faf0
>>
>>
>>
>> The system complains because of the different uuids, and refuses to
>> recreate the array.
>>
>> Is it basically stuffed, or is there something I can do to recover the
>> 2TB filesystem that's on there ?
>>
>>
>
> What can you tell us about how that happened? When (if ever) was it
> running, how was it created, etc, etc.
>
> You could probably try some things like trying to start it read-only
> using --force, but don't do that yet, if you get it wrong you WILL be
> likely to lose data.
>
> --
> Bill Davidsen <davidsen@tmr.com>
> "Woe unto the statesman who makes war without a reason that will still
> be valid when the war is over..." Otto von Bismark
>
>
>


I don't suppose anybody else has any ideas? I've been holding off
attempting anything in the hope of somebody handing me a silver bullet,
but failing that ... ;)

Given that I still have the logs showing the order in which the drives
were assembled in these various phases, if I force a re-assembly in a
particular order, is there anything in the meta-data which would cause the
array to auto-magically continue where it left off?

Or should I simply take the afore-mentioned silver bullet, bite it and
recreate the whole lot as a RAID6, followed by the usual mad scrabbling
though old hard drives and tapes to try and get most of the content back?

Thanks in advance for any help,
Eamonn

Re: differnet UUIDs and no of spares :(

[Bryon Roche]

On Wed, 21 Jan 2009 11:06:10 +0000, eamonn wrote:

> I don't suppose anybody else has any ideas? I've been holding off
> attempting anything in the hope of somebody handing me a silver bullet,
> but failing that ... ;)
No silver bullets. your best chance is to use the logs like you mention 
to do a forced read-only assemble.  For this reason, it's also usually a 
good idea to save the arguments you used for mdadm --create somewhere, 
especially if you made the array with a non-default layout.  At least 
this will let you stream the data of the array onto something else.

A forced assembly necessarily destroys the superblock and any metadata 
that could be used to resume the build.  Your best bet here will be to 
make sure you got the device order correct, probably in accordance to the 
oldest list.  This assumes that you didn't start rebuilding onto any 
disks in such a way as to change that order.