RAID on partitions and partitions on RAID

RAID on partitions and partitions on RAID

[Cry Regarder]

Sentient Beings,

RAID on parititions vs. raw disks:

I built a raid-5 array with six 500gb SATA disks.  For some reason which I can
no longer remember but related I suspect to hubris, I built the array directly
on the drive devices rather than making a fullsize partition.  As a consequence,
I assemble the array from /dev/sd[bcdefg] instead of /dev/sd[bcdefg]1.  In
reality, because my motherboard seems to like to shuffle the order of
recognition, my boot drive isn't always /dev/sda so in practice I do:

mdadm -Ac partitions -m 0 /dev/md0

When I built the array, some of the drives were virgin, some had a factory NTFS
format, and some were previously partitioned by me for linux use.  The problem I
have now is twofold: 

1.  There are remnants of the partition tables still on the drives, despite
those partitions being meaningless.

2.  I'd actually like to have clean /dev/sd?1 partitions on each drive in the
array marked with the raid identifier.  

Is there away to non-destructively adjust the partition tables, or resize the
array to allow for 2.?  If not, is there a good way to clean up 1. so that there
are no longer any partition tables on the drive?

Partitions on RAID:

Back to when I constructed the array, I placed an ext3 partition directly on
/dev/md0.  Since construction, I have expanded the array by two disks, but have
not at this time resized the ext3 partition.  

What I would like to do is put a swap partition and XFS partition into that 1TB
free space.  However, I have no idea how to go about doing that.  If I had
originally used LVM, it would be easy, but I had read about some performance
hits re ext3 on lvm on raid so I decided to be "simple".  Any advice?

Thanks!

Cry

Re: RAID on partitions and partitions on RAID

[Michal Soltys]

Cry Regarder wrote:

> 
> Partitions on RAID:
> 
> Back to when I constructed the array, I placed an ext3 partition directly on
> /dev/md0.  Since construction, I have expanded the array by two disks, but have
> not at this time resized the ext3 partition.  
> 
> What I would like to do is put a swap partition and XFS partition into that 1TB
> free space.  However, I have no idea how to go about doing that.  If I had
> originally used LVM, it would be easy, but I had read about some performance
> hits re ext3 on lvm on raid so I decided to be "simple".  Any advice?
> 

When I was doing recently some tests, I haven't noticed any significant 
performance differences if lvm2 is between filesystem and raid, or not. Note 
  though, that my tests weren't too extensive (yet). Also, I use large lv 
extents  - 512MiB.

Either way, you can use partitionable raid arrays. Today I've set following 
config on one of the new machines: raid 5 from 4x500GB disks + one hot 
spare. Each disk is partitioned in the same way - 64mb boot partition 
identical on each disk, and each disk is bootable (sdX1), swap (sdX2), 
partitionable (sdX3). sd[abcde]3 raid has GPT partition - 1st one is used by 
LVM2 for the usual stuff (root,usr,var,home.. "just" 24GB assigned here), 
the 2nd one is big XFS for my space-hungry users. The XFS partition is 
carefully aligned to be always on the stride boundary, up to 9 disks (as I 
plan do add new disks and grow that XFS later, while having its su/sw 
properly set).

You will of course need initramfs to boot from something like that. 
Depending on your distro (and how much can you dig into its config files), 
YMMV. (small remark - udev and mdadm, depending on their rules, might get in 
one others way - check / adjust both of them carefully).

Re: RAID on partitions and partitions on RAID

[Michal Soltys]

Michal Soltys wrote:
> Cry Regarder wrote:
> 
>>
> one hot spare. Each disk is partitioned in the same way - 64mb boot 
> partition identical on each disk, and each disk is bootable (sdX1), swap 
> (sdX2), partitionable (sdX3). sd[abcde]3 raid has GPT partition - 1st 
> one is used by LVM2 for the usual stuff (root,usr,var,home.. "just" 24GB 
> 

Correction - The swap is also on raid (on partition governed by lvm). 
Initial idea was well, risky (i.e. what if a disk failed that had its swap 
actually used).

Re: RAID on partitions and partitions on RAID

[Cry Regarder]

Michal Soltys <nozo <at> ziu.info> writes:

> 
> Michal Soltys wrote:
> >>
> > one hot spare. Each disk is partitioned in the same way - 64mb boot 
> > partition identical on each disk, and each disk is bootable (sdX1), swap 
> > (sdX2), partitionable (sdX3). sd[abcde]3 raid has GPT partition - 1st 
> > one is used by LVM2 for the usual stuff (root,usr,var,home.. "just" 24GB 
> 

Thanks!  A couple questions:

1.  Are you sharing that spare with an other array?  If not, why not do a raid-6
instead of a raid-5?

2.  I noticed that you built your raid partitions on corresponding disk
partitions.  Why do that instead of making one monolithic raid volume and then
partitioning that into the desired pieces?

Since I already built my array, I don't want to dump, rebuild, and restore at
this time.  What I'd like to do is do two small (I don't know if they are small
actually) to my system:

1.  Adjust the partition table on each of the component disks so that I can
assemble the drives from /dev/sd?1 instead of /dev/sd?

2.  Adjust my /dev/md0 so that it is partitioned into /dev/md0_1 /dev/md0_2 or
something of that ilk.

I see now that I could have built it that way from the start, but is it too late
to convert?

Thanks!

Cry

Re: RAID on partitions and partitions on RAID

[Michal Soltys]

Cry Regarder wrote:
> 
> Thanks!  A couple questions:
> 
> 1.  Are you sharing that spare with an other array?  If not, why not do a raid-6
> instead of a raid-5?
> 

No, the spare is not shared. Now that I think about it and you reminded 
about raid-6...

I can think of one small plus of my setup - that spare is not spinning
(it's stopped with sdparm -C stop). So the disk should be in almost unused
condition if the need for it arrives. Excluding vibrations from the rest of 
the disks, and barely active electronics.

> 2.  I noticed that you built your raid partitions on corresponding disk
> partitions.  Why do that instead of making one monolithic raid volume and then
> partitioning that into the desired pieces?
> 

few reasons:

1) I wanted /boot out of any raid
2) I wanted LVM for some of the partitions
3) I prefered regular mbr on the whole disks.
4) I wanted properly aligned XFS, so I can use its su/sw options

Alternatives:

- create separate md0 (for lvm) md1 (for xfs), but it seemed wrong when 
there is support for partitionable arrays
- or I could just put everything on lvm and don't bother with xfs alignment.

All things considered, it looked like a decent compromise. Of course, I'm 
open for suggestions.

> 
> 1.  Adjust the partition table on each of the component disks so that I can
> assemble the drives from /dev/sd?1 instead of /dev/sd?
> 

No can do here, afaik. When you assemble from /dev/sd? , there're no 
partitions to start with on those disks. Either way, I'm not aware of any 
tools allowing - shrink raid, shift it, create fitting mbr ...

> 2.  Adjust my /dev/md0 so that it is partitioned into /dev/md0_1 /dev/md0_2 or
> something of that ilk.
> 

No idea if you can easily switch from non-partitionable to partitionable raid.

Is it possible to grow a RAID-10 array with mdadm?

[Tomas France]

Hi everyone,

Is it possible to add drives to an active RAID-10 array, using the "grow" 
switch with mdadm, just like it is possible with a RAID-5 array? Or perhaps 
there is another way?

I have been looking for this information for a long time but have been 
unable to find it anywhere. The man page for mdadm does not mention RAID-10 
at all so that didn't help either. Has anyone tried it?

Thanks for any information about this issue!

Tomas 

Re: Is it possible to grow a RAID-10 array with mdadm?

[Neil Brown]

On Sunday July 29, tomfra@centrum.cz wrote:
> Hi everyone,
> 
> Is it possible to add drives to an active RAID-10 array, using the "grow" 
> switch with mdadm, just like it is possible with a RAID-5 array? Or perhaps 
> there is another way?
> 
> I have been looking for this information for a long time but have been 
> unable to find it anywhere. The man page for mdadm does not mention RAID-10 
> at all so that didn't help either. Has anyone tried it?

The man page for mdadm does not mention it because it is not
supported.

There are several reshape options that I would like to implement
including
  - raid5 -> raid6
  - shrinking raid4/5/6
  - raid0 -> raid5
  - changing chunksize/layout of raid4/5/6
  - raid10 growing and layout change

unfortunately I haven't yet found/made the time.

Patches are always welcome :-)

NeilBrown

Re: Is it possible to grow a RAID-10 array with mdadm?

[Tomas France]

Thanks for the answer Neil!

>> The man page for mdadm does not mention it because it is not supported. 
>> <<

It doesn't actually even mention the possibility to create a RAID-10 array 
(without creating RAID-0 on top of RAID-1 pairs), yet from the info I found, 
a lot of people have been using it for quite a while. Almost as if it was a 
complete secret ;) As for the RAID-10 growing / layout change - I'd 
absolutely love to see that implemented in the (hopefully near) future. 
IMHO, RAID-10 is becoming very popular because of the falling hard drive 
prices.

Tomas


----- Original Message ----- 
From: "Neil Brown" <neilb@suse.de>
To: "Tomas France" <tomfra@centrum.cz>
Cc: <linux-raid@vger.kernel.org>
Sent: Monday, July 30, 2007 10:48 AM
Subject: Re: Is it possible to grow a RAID-10 array with mdadm?


> On Sunday July 29, tomfra@centrum.cz wrote:
>> Hi everyone,
>>
>> Is it possible to add drives to an active RAID-10 array, using the "grow"
>> switch with mdadm, just like it is possible with a RAID-5 array? Or 
>> perhaps
>> there is another way?
>>
>> I have been looking for this information for a long time but have been
>> unable to find it anywhere. The man page for mdadm does not mention 
>> RAID-10
>> at all so that didn't help either. Has anyone tried it?
>
> The man page for mdadm does not mention it because it is not
> supported.
>
> There are several reshape options that I would like to implement
> including
>  - raid5 -> raid6
>  - shrinking raid4/5/6
>  - raid0 -> raid5
>  - changing chunksize/layout of raid4/5/6
>  - raid10 growing and layout change
>
> unfortunately I haven't yet found/made the time.
>
> Patches are always welcome :-)
>
> NeilBrown
> -
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
> 

Re: Is it possible to grow a RAID-10 array with mdadm?

[Neil Brown]

On Monday July 30, tomfra@centrum.cz wrote:
> Thanks for the answer Neil!
> 
> >> The man page for mdadm does not mention it because it is not supported. 
> >> <<
> 
> It doesn't actually even mention the possibility to create a RAID-10 array 
> (without creating RAID-0 on top of RAID-1 pairs), yet from the info I found, 
> a lot of people have been using it for quite a while. Almost as if it was a 
> complete secret ;) As for the RAID-10 growing / layout change - I'd 
> absolutely love to see that implemented in the (hopefully near) future. 
> IMHO, RAID-10 is becoming very popular because of the falling hard drive 
> prices.

What version of mdadm do you have installed (the bottom of the man
page will tell you).
My v2.6.2 manpage mentions raid10 5 times, and "man md" mentions it 9
times. 
If you have a recent mdadm and there was some particular place in the
man page were you were looking and didn't find raid10, please let me
know and I will try to improve that part of the documentation.

Thanks,
NeilBrown

Re: Is it possible to grow a RAID-10 array with mdadm?

[Tomas France]

Oops... my faul Neil, sorry for that. I must have been reading the man page 
for an older version. It's all well documented in 2.6.2. I apologize 
again...

Tomas

P.S. And I almost forgot - thanks for creating mdadm! It's a powerful stuff 
for sure. I will soon be playing with it in a live environment...which, 
looking at all the options and reading the documentation, makes me somewhat 
scared :)


----- Original Message ----- 
From: "Neil Brown" <neilb@suse.de>
To: "Tomas France" <tomfra@centrum.cz>
Cc: <linux-raid@vger.kernel.org>
Sent: Monday, July 30, 2007 1:07 PM
Subject: Re: Is it possible to grow a RAID-10 array with mdadm?


> On Monday July 30, tomfra@centrum.cz wrote:
>> Thanks for the answer Neil!
>>
>> >> The man page for mdadm does not mention it because it is not 
>> >> supported.
>> >> <<
>>
>> It doesn't actually even mention the possibility to create a RAID-10 
>> array
>> (without creating RAID-0 on top of RAID-1 pairs), yet from the info I 
>> found,
>> a lot of people have been using it for quite a while. Almost as if it was 
>> a
>> complete secret ;) As for the RAID-10 growing / layout change - I'd
>> absolutely love to see that implemented in the (hopefully near) future.
>> IMHO, RAID-10 is becoming very popular because of the falling hard drive
>> prices.
>
> What version of mdadm do you have installed (the bottom of the man
> page will tell you).
> My v2.6.2 manpage mentions raid10 5 times, and "man md" mentions it 9
> times.
> If you have a recent mdadm and there was some particular place in the
> man page were you were looking and didn't find raid10, please let me
> know and I will try to improve that part of the documentation.
>
> Thanks,
> NeilBrown
>
> 

Re: Is it possible to grow a RAID-10 array with mdadm?

[Bill Davidsen]

Neil Brown wrote:
> On Sunday July 29, tomfra@centrum.cz wrote:
>   
>> Hi everyone,
>>
>> Is it possible to add drives to an active RAID-10 array, using the "grow" 
>> switch with mdadm, just like it is possible with a RAID-5 array? Or perhaps 
>> there is another way?
>>
>> I have been looking for this information for a long time but have been 
>> unable to find it anywhere. The man page for mdadm does not mention RAID-10 
>> at all so that didn't help either. Has anyone tried it?
>>     
>
> The man page for mdadm does not mention it because it is not
> supported.
>
> There are several reshape options that I would like to implement
> including
>   - raid5 -> raid6
>   - shrinking raid4/5/6
>   - raid0 -> raid5
>   - changing chunksize/layout of raid4/5/6
>   - raid10 growing and layout change
>   

While changing chunk size is probably the least generally useful, I sure 
would like to have it ;-)

-- 
bill davidsen <davidsen@tmr.com>
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979