[linux-lvm] General Q on adding new PVs to existing striped RAID set

[linux-lvm] General Q on adding new PVs to existing striped RAID set

[hansbkk]

According to the advice I've been given so far - I should use RAID
"under" my LVM configuration, not only for fault tolerance but also
for the performance benefits of striping, and leave LVM as a "top
layer" for disk space management efficiency and flexibility, plus of
course the benefits of snapshots.

One of LVM's biggest wins is the fact that if you're running out of
space you can just slap in more disks, add them to your VG and extend
your LVs. But I can see how this may result in performance or fault
tolerance issues.

Example - if I have say 6TB of RAID10 managed by LVM, and then find I
need to add another 2TB, I might add a RAID1 mirror of 2x 2TB drives
and extend my VG to get more space. However it seems for optimum
performance and fault tolerance I should  consider this a somewhat
temporary workaround. Ultimately I would want to get back to a
coherent single 8TB RAID set, and take the RAID1 mirror out.

This seems to be better from multiple POVs - more space efficient,
more consistently predictable performance (if not actually faster),
and most importantly more fault tolerant.

I realize if I was using RAID6 as many have advised I should, I could
grow the underlying RAID itself directly and then expand the VG, and
obviously this would be ideal. But I wanted to clarify regarding those
RAID flavors that can't be grown in place, as a principle of best
practice.

Please confirm if my thinking on this is on the right track.

General Q on adding new PVs to existing striped RAID set

[hansbkk]

According to the advice I've been given so far - I should use RAID
"under" my LVM configuration, not only for fault tolerance but also
for the performance benefits of striping, and leave LVM as a "top
layer" for disk space management efficiency and flexibility, plus of
course the benefits of snapshots.

One of LVM's biggest wins is the fact that if you're running out of
space you can just slap in more disks, add them to your VG and extend
your LVs. But I can see how this may result in performance or fault
tolerance issues.

Example - if I have say 6TB of RAID10 managed by LVM, and then find I
need to add another 2TB, I might add a RAID1 mirror of 2x 2TB drives
and extend my VG to get more space. However it seems for optimum
performance and fault tolerance I should  consider this a somewhat
temporary workaround. Ultimately I would want to get back to a
coherent single 8TB RAID set, and take the RAID1 mirror out.

This seems to be better from multiple POVs - more space efficient,
more consistently predictable performance (if not actually faster),
and most importantly more fault tolerant.

I realize if I was using RAID6 as many have advised I should, I could
grow the underlying RAID itself directly and then expand the VG, and
obviously this would be ideal. But I wanted to clarify regarding those
RAID flavors that can't be grown in place, as a principle of best
practice.

Please confirm if my thinking on this is on the right track.

Re: [linux-lvm] General Q on adding new PVs to existing striped RAID set

[Stuart D Gathman]

On 12/07/2010 04:15 AM, hansbkk@gmail.com wrote:
> Example - if I have say 6TB of RAID10 managed by LVM, and then find I
> need to add another 2TB, I might add a RAID1 mirror of 2x 2TB drives
> and extend my VG to get more space. However it seems for optimum
> performance and fault tolerance I should  consider this a somewhat
> temporary workaround. Ultimately I would want to get back to a
> coherent single 8TB RAID set, and take the RAID1 mirror out.
>
> This seems to be better from multiple POVs - more space efficient,
> more consistently predictable performance (if not actually faster),
> and most importantly more fault tolerant.
Raid1 is just as fault tolerant as raid10.  Raid10 just adds striping
for performance.  You are probably thinking that with more devices, the
probability of 2 devices failing close together is larger.  However,
going from 4 to 6 (or 8) devices does not significantly decrease
reliability compared to the increase provided by raid1 (with or without
striping).
> I realize if I was using RAID6 as many have advised I should, I could
> grow the underlying RAID itself directly and then expand the VG, and
> obviously this would be ideal. But I wanted to clarify regarding those
> RAID flavors that can't be grown in place, as a principle of best
> practice.
You can grow raid1/raid10 also.  Raid6 (and 5) has the read/modify/write
cycle issue that affects random write performance.
> Please confirm if my thinking on this is on the right track.
LVM supports raid0 and 1 (and other levels in testing).  The only reason
I use raid1 "underneath" is that LVM is still maturing in raid support
(compared to, say, AIX).  LVM raid0 is quite good, however.  Consider
making all your PVs raid1, and doing your striping in LVM.  That is the
equivalent of raid10 and gives you much more flexibility in growing your
VG than raid10 underneath.

Re: [linux-lvm] General Q on adding new PVs to existing striped RAID set

[hansbkk]

On Tue, Dec 7, 2010 at 11:58 PM, Stuart D Gathman <stuart@bmsi.com> wrote:
> On 12/07/2010 04:15 AM, hansbkk@gmail.com wrote:
>> Example - if I have say 6TB of RAID10 managed by LVM, and then find I
>> need to add another 2TB, I might add a RAID1 mirror of 2x 2TB drives
>> and extend my VG to get more space. However it seems for optimum
>> performance and fault tolerance I should �consider this a somewhat
>> temporary workaround. Ultimately I would want to get back to a
>> coherent single 8TB RAID set, and take the RAID1 mirror out.
>>
>> This seems to be better from multiple POVs - more space efficient,
>> more consistently predictable performance (if not actually faster),
>> and most importantly more fault tolerant.
>
> Raid1 is just as fault tolerant as raid10. �Raid10 just adds striping
> for performance. �You are probably thinking that with more devices, the
> probability of 2 devices failing close together is larger. �However,
> going from 4 to 6 (or 8) devices does not significantly decrease
> reliability compared to the increase provided by raid1 (with or without
> striping).

You're right of course. I wrote my first draft of the scenario with
RAID6, then realized the answer would be to grow the underlying RAID6
rather than linearly adding a disk set via LVM, so changed to RAID10
specifically because it doesn't allow for online growing, and wanted
to address the issue of linearly adding a mix of RAID sets.

<snip>

> You can grow raid1/raid10 also.

My understanding is only by replacing smaller disks with bigger ones,
but not by adding new disks.

> Consider making all your PVs raid1, and doing your striping in LVM. �That is the
> equivalent of raid10 and gives you much more flexibility in growing your
> VG than raid10 underneath.

<snip>

> Look into this possibility.  I've not tried it myself, so hopefully
> others can verify if it works or not, and if there are any performance
> issues.
>
> Create multiple mdadm 2 disk mirror sets (RAID 1) with all your drives
> (assuming they're the same size/rpm).  Create an LVM stripe across the
> resulting md devices.  Tune/tweak/test for performance and
> functionality, specifically expanding the LVM volume while maintaining
> the stripe correctly.  If everything looks good, format with your
> favorite FS, do more performance tests, then go.
>
> Poor man's RAID 10, basically.  More accurately, it's a multi-layer
> RAID, with, I think, the ability to expand via LVM without any gotchas.
>  Performance probably won't be quite as good as native mdadm RAID 10,
> but if you get expansion capability like you would with RAID5/6, it may
> be worth the performance hit.

Thanks Stan, great ideas. However in the meantime I found a statement
that perfectly addresses my feelings:

>In LVM 2, striped LVs can be extended by concatenating another set of devices onto the end of the first set. So you can get into a situation where your LV is a 2 stripe set concatenated with a linear set concatenated with a 4 stripe set. Are you confused yet?

Given my state of noob'ness, and the fact that I'm currently setting
up a single filer that will service dozens of hosts, virtual and
phsyical, clients and servers, without much ability to predict future
usage patterns. I'm not confident of my ability to keep the LVM
striping optimized over time, and will most likely elect to keep the
striping-for-performance at the underlying RAID level.

I also will be running some backup services whose disk space need to
be kept on a completely different set of spindles from the hosts' data
they are backing up. On the latter set, where I want maximum fault
tolerance, performance isn't such an issue but I need lots of space on
my limited budget. Therefore I believe RAID6 is the way to go. As
mentioned above, it seems this will also allow me to expand the disk
set as needed "under" the LVM level, maintaining consistent
performance and optimizing space usage and fault tolerance.

On the other, "working data" VG, I'll probably also start off using
plain RAID1, and therefore may well not run into the issue I've
raised, as adding additional disk pairs will be at least consistent.
If I keep a chunk of space empty on each set as I grow, I will then be
able to experiment with using LVM for the striping for a particular
host that needs the performance boost - or at least be able to do some
benchmarking.

If I find I generally need the extra performance of RAID10 (f2) down
the road, I do feel confirmed in the main thrust of my original
question, which is that I should treat inconsistent linear additions
of RAID1 pairs to existing RAID-striped sets as temporary, and plan to
migrate the VG back to a single coherent set when the chance comes.

Thanks very much for helping me to clarify my thinking - additional
feedback of course would be welcome (from anyone).

Re: [linux-lvm] General Q on adding new PVs to existing striped RAID set

[hansbkk]

On Tue, Dec 7, 2010 at 11:58 PM, Stuart D Gathman <stuart@bmsi.com> wrote:
> On 12/07/2010 04:15 AM, hansbkk@gmail.com wrote:
>> Example - if I have say 6TB of RAID10 managed by LVM, and then find I
>> need to add another 2TB, I might add a RAID1 mirror of 2x 2TB drives
>> and extend my VG to get more space. However it seems for optimum
>> performance and fault tolerance I should  consider this a somewhat
>> temporary workaround. Ultimately I would want to get back to a
>> coherent single 8TB RAID set, and take the RAID1 mirror out.
>>
>> This seems to be better from multiple POVs - more space efficient,
>> more consistently predictable performance (if not actually faster),
>> and most importantly more fault tolerant.
>
> Raid1 is just as fault tolerant as raid10.  Raid10 just adds striping
> for performance.  You are probably thinking that with more devices, the
> probability of 2 devices failing close together is larger.  However,
> going from 4 to 6 (or 8) devices does not significantly decrease
> reliability compared to the increase provided by raid1 (with or without
> striping).

You're right of course. I wrote my first draft of the scenario with
RAID6, then realized the answer would be to grow the underlying RAID6
rather than linearly adding a disk set via LVM, so changed to RAID10
specifically because it doesn't allow for online growing, and wanted
to address the issue of linearly adding a mix of RAID sets.

<snip>

> You can grow raid1/raid10 also.

My understanding is only by replacing smaller disks with bigger ones,
but not by adding new disks.

> Consider making all your PVs raid1, and doing your striping in LVM.  That is the
> equivalent of raid10 and gives you much more flexibility in growing your
> VG than raid10 underneath.

<snip>

> Look into this possibility.  I've not tried it myself, so hopefully
> others can verify if it works or not, and if there are any performance
> issues.
>
> Create multiple mdadm 2 disk mirror sets (RAID 1) with all your drives
> (assuming they're the same size/rpm).  Create an LVM stripe across the
> resulting md devices.  Tune/tweak/test for performance and
> functionality, specifically expanding the LVM volume while maintaining
> the stripe correctly.  If everything looks good, format with your
> favorite FS, do more performance tests, then go.
>
> Poor man's RAID 10, basically.  More accurately, it's a multi-layer
> RAID, with, I think, the ability to expand via LVM without any gotchas.
>  Performance probably won't be quite as good as native mdadm RAID 10,
> but if you get expansion capability like you would with RAID5/6, it may
> be worth the performance hit.

Thanks Stan, great ideas. However in the meantime I found a statement
that perfectly addresses my feelings:

>In LVM 2, striped LVs can be extended by concatenating another set of devices onto the end of the first set. So you can get into a situation where your LV is a 2 stripe set concatenated with a linear set concatenated with a 4 stripe set. Are you confused yet?

Given my state of noob'ness, and the fact that I'm currently setting
up a single filer that will service dozens of hosts, virtual and
phsyical, clients and servers, without much ability to predict future
usage patterns. I'm not confident of my ability to keep the LVM
striping optimized over time, and will most likely elect to keep the
striping-for-performance at the underlying RAID level.

I also will be running some backup services whose disk space need to
be kept on a completely different set of spindles from the hosts' data
they are backing up. On the latter set, where I want maximum fault
tolerance, performance isn't such an issue but I need lots of space on
my limited budget. Therefore I believe RAID6 is the way to go. As
mentioned above, it seems this will also allow me to expand the disk
set as needed "under" the LVM level, maintaining consistent
performance and optimizing space usage and fault tolerance.

On the other, "working data" VG, I'll probably also start off using
plain RAID1, and therefore may well not run into the issue I've
raised, as adding additional disk pairs will be at least consistent.
If I keep a chunk of space empty on each set as I grow, I will then be
able to experiment with using LVM for the striping for a particular
host that needs the performance boost - or at least be able to do some
benchmarking.

If I find I generally need the extra performance of RAID10 (f2) down
the road, I do feel confirmed in the main thrust of my original
question, which is that I should treat inconsistent linear additions
of RAID1 pairs to existing RAID-striped sets as temporary, and plan to
migrate the VG back to a single coherent set when the chance comes.

Thanks very much for helping me to clarify my thinking - additional
feedback of course would be welcome (from anyone).
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html