Using LVM for RAID 1+0

Using LVM for RAID 1+0

[Ewan Grantham]

Had though of setting up several disks as a RAID-5 array, but for some
reason one of the disks would keep failing when I tried to build the
array. The same disk, however, would format just fine with EXT3, and
is quite happy about being put in a RAID-1 array with another disk.
Given that strangeness, I'm thinking a better way to go is to pair off
my disks into several RAID-1 arrays, and then use LVM to setup
striping across all of them to effectively give me a RAID 1+0 setup.
Since one pair of disks is a different size than the other pairs, I
presumed this would be better than actually creating a RAID-0 across
all the RAID-1 arrays. Is that correct?

Re: Using LVM for RAID 1+0

[Neil Brown]

On Tuesday April 18, ewan.grantham@gmail.com wrote:
> Had though of setting up several disks as a RAID-5 array, but for some
> reason one of the disks would keep failing when I tried to build the
> array. The same disk, however, would format just fine with EXT3, and
> is quite happy about being put in a RAID-1 array with another disk.
> Given that strangeness, I'm thinking a better way to go is to pair off
> my disks into several RAID-1 arrays, and then use LVM to setup
> striping across all of them to effectively give me a RAID 1+0 setup.
> Since one pair of disks is a different size than the other pairs, I
> presumed this would be better than actually creating a RAID-0 across
> all the RAID-1 arrays. Is that correct?


No, this is not correct.
md/raid0 has no problem with devices being of different sizes.  It
will make use of all the space doing as much striping as possible.


However it sounds like your drive or your controller is not to be
trusted.  There is a reason that it is failing.  Maybe raid5 triggers
that reason more often, but that doesn't mean that raid1 won't
trigger it at all.  I suggest you be very wary of your hardware!

NeilBrown

Re: Using LVM for RAID 1+0

[Ewan Grantham]

On 4/18/06, Neil Brown <neilb@suse.de> wrote:
> No, this is not correct.
> md/raid0 has no problem with devices being of different sizes.  It
> will make use of all the space doing as much striping as possible.

In that case, the other question would be if you can grow a RAID 0
array. Currently, one pair of disks have my remaining data. The idea
was to set the other pairs to RAID 1 arrays, link them with LVM (or
RAID 0) into one partition, copy the data, then convert the remaining
pair of disks to a RAID 1 array and grow the RAID 0 array to add the
additional RAID 1 array.

> However it sounds like your drive or your controller is not to be
> trusted.  There is a reason that it is failing.  Maybe raid5 triggers
> that reason more often, but that doesn't mean that raid1 won't
> trigger it at all.  I suggest you be very wary of your hardware!

Suspicious is the word for it. Which is why I'm not going back to RAID
5 using my current configuration. My drives are all external USB-2,
and split between some ports on front, some on back, and some on an
add-on card. My guess is that I had an arrangement that was swamping
the channels. I've played around with what is attached where in a
manner that appears to have "fixed' that.

FWIW,
Ewan

Re: Using LVM for RAID 1+0

[Neil Brown]

On Tuesday April 18, ewan.grantham@gmail.com wrote:
> On 4/18/06, Neil Brown <neilb@suse.de> wrote:
> > No, this is not correct.
> > md/raid0 has no problem with devices being of different sizes.  It
> > will make use of all the space doing as much striping as possible.
> 
> In that case, the other question would be if you can grow a RAID 0
> array. Currently, one pair of disks have my remaining data. The idea
> was to set the other pairs to RAID 1 arrays, link them with LVM (or
> RAID 0) into one partition, copy the data, then convert the remaining
> pair of disks to a RAID 1 array and grow the RAID 0 array to add the
> additional RAID 1 array.

No, you cannot grow an md/raid0 array.  If you need growth, then you
would need to consider LVM at this stage, though md/linear will
probably be able to grow soon.

> 
> > However it sounds like your drive or your controller is not to be
> > trusted.  There is a reason that it is failing.  Maybe raid5 triggers
> > that reason more often, but that doesn't mean that raid1 won't
> > trigger it at all.  I suggest you be very wary of your hardware!
> 
> Suspicious is the word for it. Which is why I'm not going back to RAID
> 5 using my current configuration. My drives are all external USB-2,
> and split between some ports on front, some on back, and some on an
> add-on card. My guess is that I had an arrangement that was swamping
> the channels. I've played around with what is attached where in a
> manner that appears to have "fixed' that.

I've always felt that raid over USB was a "courageous" design.  USB
has such a "temporary" feel to it.  However, if it work for you,
great.

NeilBrown

Re: Using LVM for RAID 1+0

[Ewan Grantham]

On 4/18/06, Neil Brown <neilb@suse.de> wrote:
> No, you cannot grow an md/raid0 array.  If you need growth, then you
> would need to consider LVM at this stage, though md/linear will
> probably be able to grow soon.

Any chance RAID-5 will be able to grow at some point also?

> I've always felt that raid over USB was a "courageous" design.  USB
> has such a "temporary" feel to it.  However, if it work for you,
> great.

Well, after playing with LVM for a while, I realized that there were a
few limitations I hadn't counted on... :-)

Since everything seemed to be working ok with the switch of cables and
ports, I decided to take the plunge, and am now (knocking on a large
virtual pile of wood) almost 60% through a build of the RAID-5 array
using the same drives I was having issues with earlier. Given that I
had the setup running for several months before Windows decided to get
"funny" with my embedded Linux solution, I have hope that this will
run for a good while. Of course that means my other pair of drives
will now have to be a seperate volume, but that should give me a good
setup for some of my more essential files (photos and home movies).

USB at least has the advantage over 1394 external drives of not
tending to decide to change dev names at each reboot. When I had a
firewire array I was always afraid to do a shutdown not knowing what
would happen when it came back up.

I agree that getting a large box with enough bays to put the disks
internal would probably be preferable - but if I had any extra money
at this point I'd be more likely to get a good UPS to replace my
current power strip with surge protector.

Thanks again for the help and advice,
Ewan

Re: Using LVM for RAID 1+0

[David Greaves]

Ewan Grantham wrote:
> On 4/18/06, Neil Brown <neilb@suse.de> wrote:
>   
>> No, you cannot grow an md/raid0 array.  If you need growth, then you
>> would need to consider LVM at this stage, though md/linear will
>> probably be able to grow soon.
>>     
>
> Any chance RAID-5 will be able to grow at some point also?
Truly courageous :)

Kidding!!


raid5 can grow already - both size of disks (ie replacing each existing
disk with a bigger one) and more recently, number of disks.

There's not been much traffic about it - but what there has been has
been good.

You'll need a very recent kernel/mdadm though... and a backup is always
recommended.

David
PS I'm not talking about raidreconf - that has generated some 'doh!'
messages

-- 

Re: Using LVM for RAID 1+0

[Neil Brown]

On Wednesday April 19, ewan.grantham@gmail.com wrote:
> On 4/18/06, Neil Brown <neilb@suse.de> wrote:
> > No, you cannot grow an md/raid0 array.  If you need growth, then you
> > would need to consider LVM at this stage, though md/linear will
> > probably be able to grow soon.
> 
> Any chance RAID-5 will be able to grow at some point also?

You haven't been following the list have you ..... ;-)

mdadm-2.4.1 plus linux-2.6.17 (or any -rc) can grow a raid5.
Just add a spare drive and
   mdadm --grow /dev/mdX --raid-disks=N+1

NeilBrown

Re: Using LVM for RAID 1+0

[Ewan Grantham]

On 4/19/06, Neil Brown <neilb@suse.de> wrote:
> You haven't been following the list have you ..... ;-)
>
> mdadm-2.4.1 plus linux-2.6.17 (or any -rc) can grow a raid5.
> Just add a spare drive and
>    mdadm --grow /dev/mdX --raid-disks=N+1

Actually I had, but I thought I understood this was extremely
experimental. Does this mean it's considered accepted/acceptable? In
which case, the fact that I'm using a 2.6.15 kernel and mdadm 1.12.0
(June 2005) (is that version number possibly right?) suggest I need to
do a little tool updating so I can try this out. I presume that I can
upgrade mdadm and then do the grow - or do I need to upgrade and
rebuild the array with the upgraded tool before I can use the grow
option? How much risk do I have of data loss doing this?

Thanks,
Ewan

Re: Using LVM for RAID 1+0

[Neil Brown]

On Wednesday April 19, ewan.grantham@gmail.com wrote:
> On 4/19/06, Neil Brown <neilb@suse.de> wrote:
> > You haven't been following the list have you ..... ;-)
> >
> > mdadm-2.4.1 plus linux-2.6.17 (or any -rc) can grow a raid5.
> > Just add a spare drive and
> >    mdadm --grow /dev/mdX --raid-disks=N+1
> 
> Actually I had, but I thought I understood this was extremely
> experimental. Does this mean it's considered accepted/acceptable? In
> which case, the fact that I'm using a 2.6.15 kernel and mdadm 1.12.0
> (June 2005) (is that version number possibly right?) suggest I need to
> do a little tool updating so I can try this out. I presume that I can
> upgrade mdadm and then do the grow - or do I need to upgrade and
> rebuild the array with the upgraded tool before I can use the grow
> option? How much risk do I have of data loss doing this?

I think it has transitioned from "Extremely experimental" to
"Experimental". 

If should be safe, and if anything does go wrong, I'm sure I can pick
up the pieces for you.

But yes, you'll need a tool upgrade.
1.12.0 was the last of the '1' series.
2.4.1 is the latest and is required for raid5 resize.
Also, you will need 2.6.17-rc1 or later. And you'll need to select
CONFIG_EXPERIMENTAL and the config option to enable raid5 reshaping.

You don't need to "upgrade" the array at all, or rebuild it or
anything.

Risk of data loss is very low.  I believe the code is correct and
reliable, but until a wide variety of people test it and report
success, it is hard to be certain.  And as I said, if you do have
problem (other than multiple drive failures), I'll do my best to
stitch things back together for you.

NeilBrown