RAID 1 partition with hot spare shows [UUU] ?

RAID 1 partition with hot spare shows [UUU] ?

[John Crisp]

Hi,

Sorry to trouble people, and I am sure you have better things to do, but
I can't find an answer to the following and don't know where else to go.

I have been struggling with this problem for weeks and having read all I
can I still don't know the answer.

I have a server running a version of CentOS 5.x   Yes, mdadm is old at
2.6.9 but it isn't possible to update it currently.

A year or so ago I clean installed with a software RAID 1 array using
/dev/sda & /dev/sdb and two partitions, md1 & md2 configure
automatically on install.

I restored data to the RAID and then added manually added a third drive
/dev/sdc as a spare.

All appeared hunky dory, but whilst trying to figure a slightly
different problem on a different machine, I went back to the first one
to check how it was configured. Although I am sure all looked normal
when I had last looked, this time is looked a bit strange.

Unfortunately I don't have an exact copy of things before I started
messing about but it looked something like this :


cat /proc/mdstat revealed :

Personalities : [raid1]
md1 : active raid1 sdc1[2] sdb1[1] sda1[0]
      104320 blocks [3/3] [UUU]

md2 : active raid1 sdc2(S) sdb2[1] sda2[0]
      244091520 blocks [2/2] [UU]

unused devices: <none>


I don't understand how md1 shows [UUU] ??


On my other machine which has a similar configuration it shows the
following which I expect :

Personalities : [raid1]
md1 : active raid1 sda1[2](S) hdc1[1] hda1[0]
      104320 blocks [2/2] [UU]

md2 : active raid1 sda2[2](S) hdc2[1] hda2[0]
      312464128 blocks [2/2] [UU]

unused devices: <none>

I thought I could fail and remove the drive, dd/fdisk/reformat, sfdisk
and then try to re add it back to the array effectively as a new drive.
No joy.

If I just fail and remove it md1 shows as [UU_]

I have tried checking mdadm.conf which has the following :

DEVICE partitions
ARRAY /dev/md1 level=raid1 num-devices=2
uuid=8833ba3d:ca592541:20c7be04:42cbbdf1 spares=1
ARRAY /dev/md2 level=raid1 num-devices=2
uuid=43a5b70d:9733da5c:7dd8d970:1e476a26 spares=1

Somewhere along the line the RAID is remembering the earlier
configuration but having changed stuff left right and Cambridge, I can't
seem to get it to forget.

I have tried different variations of mdadm.conf, and tried to rebuild
initrd but that didn't fix it and I am clean out of ideas where to go
next. mdadm.conf seems to be ignored.

Undoubtedly it will take some clever tweaking and I'm scared witless at
trashing the array as I am in a different country from the hardware and
would struggle to get back to fix it !

Any advice on how to put it back to RAID 1 with a 'hot' spare would be
appreciated.

B. Rgds
John

Re: RAID 1 partition with hot spare shows [UUU] ?

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 3912 bytes --]

On Wed, 18 Apr 2012 00:42:54 +0200 John Crisp <john@reetspetit.net> wrote:

> Hi,
> 
> Sorry to trouble people, and I am sure you have better things to do, but
> I can't find an answer to the following and don't know where else to go.
> 
> I have been struggling with this problem for weeks and having read all I
> can I still don't know the answer.
> 
> I have a server running a version of CentOS 5.x   Yes, mdadm is old at
> 2.6.9 but it isn't possible to update it currently.
> 
> A year or so ago I clean installed with a software RAID 1 array using
> /dev/sda & /dev/sdb and two partitions, md1 & md2 configure
> automatically on install.
> 
> I restored data to the RAID and then added manually added a third drive
> /dev/sdc as a spare.
> 
> All appeared hunky dory, but whilst trying to figure a slightly
> different problem on a different machine, I went back to the first one
> to check how it was configured. Although I am sure all looked normal
> when I had last looked, this time is looked a bit strange.
> 
> Unfortunately I don't have an exact copy of things before I started
> messing about but it looked something like this :
> 
> 
> cat /proc/mdstat revealed :
> 
> Personalities : [raid1]
> md1 : active raid1 sdc1[2] sdb1[1] sda1[0]
>       104320 blocks [3/3] [UUU]
> 
> md2 : active raid1 sdc2(S) sdb2[1] sda2[0]
>       244091520 blocks [2/2] [UU]
> 
> unused devices: <none>
> 
> 
> I don't understand how md1 shows [UUU] ??

You have a RAID1 with 3 devices.  What it difficult to comprehend about that.
Each block is written to all three devices.

You don't want that?  Change it.

   mdadm --grow /dev/md1 --raid-disks=4

now it has 4 devices - though one will be missing.

   mdadm --grow /dev/md1 --raid-disks=2

now it has 2 devices.  Actually that won't work until you mark one of the
devices as failed, so

   mdadm /dev/md1 --fail /dev/sdc1
   mdadm --grow /dev/md1 --raid-disks=2


But maybe you really wanted a 3-disk RAID1 before - it is a configuration
that certainly has a place.

Next you'll be telling me that a RAID5 cannot be made with just 2 devices!
:-)

NeilBrown


> 
> 
> On my other machine which has a similar configuration it shows the
> following which I expect :
> 
> Personalities : [raid1]
> md1 : active raid1 sda1[2](S) hdc1[1] hda1[0]
>       104320 blocks [2/2] [UU]
> 
> md2 : active raid1 sda2[2](S) hdc2[1] hda2[0]
>       312464128 blocks [2/2] [UU]
> 
> unused devices: <none>
> 
> I thought I could fail and remove the drive, dd/fdisk/reformat, sfdisk
> and then try to re add it back to the array effectively as a new drive.
> No joy.
> 
> If I just fail and remove it md1 shows as [UU_]
> 
> I have tried checking mdadm.conf which has the following :
> 
> DEVICE partitions
> ARRAY /dev/md1 level=raid1 num-devices=2
> uuid=8833ba3d:ca592541:20c7be04:42cbbdf1 spares=1
> ARRAY /dev/md2 level=raid1 num-devices=2
> uuid=43a5b70d:9733da5c:7dd8d970:1e476a26 spares=1
> 
> Somewhere along the line the RAID is remembering the earlier
> configuration but having changed stuff left right and Cambridge, I can't
> seem to get it to forget.
> 
> I have tried different variations of mdadm.conf, and tried to rebuild
> initrd but that didn't fix it and I am clean out of ideas where to go
> next. mdadm.conf seems to be ignored.
> 
> Undoubtedly it will take some clever tweaking and I'm scared witless at
> trashing the array as I am in a different country from the hardware and
> would struggle to get back to fix it !
> 
> Any advice on how to put it back to RAID 1 with a 'hot' spare would be
> appreciated.
> 
> B. Rgds
> John
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: RAID 1 partition with hot spare shows [UUU] ?

[David Brown]

On 18/04/2012 01:12, NeilBrown wrote:

> Next you'll be telling me that a RAID5 cannot be made with just 2 devices!
> :-)
>

Can't you make it with just one device, if you make a two-device raid5 
with one drive missing?

mvh.,

David

Re: RAID 1 partition with hot spare shows [UUU] ?

[John Crisp]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 18/04/12 01:12, NeilBrown wrote:
> On Wed, 18 Apr 2012 00:42:54 +0200 John Crisp wrote:
>> 
>> I don't understand how md1 shows [UUU] ??
> 
> You have a RAID1 with 3 devices.  What it difficult to comprehend
> about that.

Probably a misunderstanding of the terminology - in Hicksville where I
live mirrored usually just means a pair ;-)

> Each block is written to all three devices.

I didn't realise that a RAID 1 could have more than two devices. Most
documentation around always illustrates a RAID 1 with a pair of drives
and hence the confusion.

> 
> You don't want that?  Change it.
> 
> mdadm --grow /dev/md1 --raid-disks=4
> 
> now it has 4 devices - though one will be missing.
> 

Logical if you know it (RAID 1) can have more devices.

> mdadm --grow /dev/md1 --raid-disks=2

Illogical naming if what you actually want to do is shrink the
array............ like many things in life, once you know, you know.

> 
> now it has 2 devices.  Actually that won't work until you mark one
> of the devices as failed, so
> 
> mdadm /dev/md1 --fail /dev/sdc1 mdadm --grow /dev/md1
> --raid-disks=2
> 


And there lies the answer to my problem. For that I can't thank you
enough !

> 
> But maybe you really wanted a 3-disk RAID1 before - it is a
> configuration that certainly has a place.
> 

No I didn't, but I now understand.

> Next you'll be telling me that a RAID5 cannot be made with just 2
> devices! :-)

Me ? I know jack ...... though I know one more thing than I did
yesterday, and that is good enough for me !

Many thanks for the assistance.

B. Rgds
John
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJPjn5zAAoJEPGW9QpGvPXME0sH/0p9dpJkZvWDDxP2sMjjDlE5
0xO8v3BY8frdHT803EYaQUkxOStgtkgMd8+MaZIK3M5Z8DM2BpbDuGyz39FvZ7Ai
r+TWygB2lESxgnFI1PbSeZq+qZholIpS+TD8cJ25gDImXjD9mtkN8sx/cMmcVP7k
CHzPY5cPOGCHNPC1V1BOYAU82+fjGZP8asQJT/TjUTQ/DlWuGVkDSENYGkWNY84T
a7MuF40StY6/RG1X3ETiTlk/iq2+1j7V6I+tQlTIqWyREb9T82O2YJ2HzTeN5uN7
YXc/asVRjELmIQHode/Xq9WXkGZpQskdHqLAPyP1lKc1aNyKARsyaGHKvYZj/94=
=a+Ob
-----END PGP SIGNATURE-----

Re: RAID 1 partition with hot spare shows [UUU] ?

[David Brown]

On 18/04/2012 10:42, John Crisp wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 18/04/12 01:12, NeilBrown wrote:
>> On Wed, 18 Apr 2012 00:42:54 +0200 John Crisp wrote:
>>>
>>> I don't understand how md1 shows [UUU] ??
>>
>> You have a RAID1 with 3 devices.  What it difficult to comprehend
>> about that.
>
> Probably a misunderstanding of the terminology - in Hicksville where I
> live mirrored usually just means a pair ;-)
>
>> Each block is written to all three devices.
>
> I didn't realise that a RAID 1 could have more than two devices. Most
> documentation around always illustrates a RAID 1 with a pair of drives
> and hence the confusion.
>

A lot of hardware raid systems (and software systems, such as Window's 
"dynamic disks") are limited to 2 devices in a raid1 set.  The world of 
Linux md raid is very different - here flexibility is the key.  Amongst 
other things, you can have raid1 sets with any number of devices - 
you've seen it with 3 devices, but you can also make them with just one 
device.  And you can also use all sorts of different disks in the set, 
not just identical ones.

You might well ask what use a 1-disk raid1 "mirror" is.  The answer is 
not much use on its own - but very useful in that you can add and remove 
disks whenever you want.  Suppose, for example, you have a single disk 
in the machine, and you want to exchange it for a new, bigger disk.  If 
the disk is a 1-disk raid1 mirror, you can do it quickly and safely 
without going off-line (except if you need to power-off to do the actual 
disk swapping).  Attach your new, bigger disk.  Add it to the raid1 
array.  Wait for the sync to complete (using the disk normally all this 
time).  Fail and remove the original disk.  Your new disk is now a 
1-disk "mirror" with the same data.  Grow the raid to fill the new disk, 
then grow your filesystems (or physical volumes for LVM) to use the new 
space.

There are lots of cool things you can do when you are not limited to 
hardware raid mindsets!