migrating from RAID5 to RAID10

migrating from RAID5 to RAID10

[Gilad Arnold]

Dear all,

I have been using RAID5 on a desktop machine using a couple of 500 GB
drives.  My original intent was to grow the array by adding more drives,
as the need arises.  I am now looking to expand the capacity (by adding
one more drive, for simplicity assume it's an identical one) but I'd
also like to switch to RAID10, which (I understand) has much better read
and write performance compared to RAID5 with more than two drives.  It
is my understanding that growing a RAID10 array is not yet supported,
but my hope is that by the time I need to expand again it will already
be supported ;-)  Finally, I'd like to do the migration on a "live"
machine, to the extent possible (I will use a remote backup, at least
for some of the data).

I'd like to hear your thoughts about whether or not the following is
a good way to perform the migration:

1. Backup data.

2. Remove one of the drives from the current array, causing it to work
   in degraded mode.

3. Start a new RAID10 array with two actual drives (the one removed from
   the existing array and the new one) and one "missing" drive.  I know
   that other RAID levels (e.g. 1 and 5) can be created with a missing
   drive, not sure about RAID10 though.  If this actually works, I'll
   get a 750 GB array working in degraded mode.

4. Copy the data from the old array to the new array.

5. Kill the old array entirely; add the remaining drive to the new array
   and watch it rebuild.

My main doubts are about step 3 above, and of course the fact that
a drive failure at any point between 2-5 (except when 4 has fully
completed and before going to 5) means losing data and having to resort
to backup.

A separate question concerns which layout I should be using for RAID10,
it sounds like f2 is the way to go, isn't it?

Your advice is much appreciated. Thanks!

Gilad

Re: migrating from RAID5 to RAID10

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 398 bytes --]

Hello!

On Wed, 9 Jun 2010 08:11:33 -0700
Gilad Arnold <arnold@cs.berkeley.edu> wrote:

> I have been using RAID5 on a desktop machine using a couple of 500 GB
> drives.  My original intent was to grow the array by adding more drives,
> as the need arises.

Do you currently run RAID5 with just 2 drives, in degraded mode, or maybe you
meant something else?

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: migrating from RAID5 to RAID10

[Gilad Arnold]

Thanks for your response, Roman.

On Fri, Jun 11, 2010 at 12:52:30AM +0600, Roman Mamedov wrote:
> > I have been using RAID5 on a desktop machine using a couple of 500
> > GB drives.  My original intent was to grow the array by adding more
> > drives, as the need arises.
> 
> Do you currently run RAID5 with just 2 drives, in degraded mode, or
> maybe you meant something else?

No, it is not degraded, it's a clean 2-drive RAID5. I know it doesn't
make much sense as it is ;-)  the intent was to grow the array later,
relying on mdadm's grow feature. Right now, I'm guessing that it
operates like a RAID1 for all practical purposes.

Gilad

Re: migrating from RAID5 to RAID10

[Drew]

>> Do you currently run RAID5 with just 2 drives, in degraded mode, or
>> maybe you meant something else?
>
> No, it is not degraded, it's a clean 2-drive RAID5. I know it doesn't
> make much sense as it is ;-)  the intent was to grow the array later,
> relying on mdadm's grow feature. Right now, I'm guessing that it
> operates like a RAID1 for all practical purposes.

Actually No. If it is RAID-5 as you claim, then currently it is
effectively a RAID-0.

RAID-5 is Striping with Distributed Parity and requires a *minimum* of
3 disks. Two disks, regardless of how you got there, is considered a
degraded state.


-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: migrating from RAID5 to RAID10

[Drew]

>> No, it is not degraded, it's a clean 2-drive RAID5. I know it doesn't
>> make much sense as it is ;-)  the intent was to grow the array later,
>> relying on mdadm's grow feature. Right now, I'm guessing that it
>> operates like a RAID1 for all practical purposes.
>
> Actually No. If it is RAID-5 as you claim, then currently it is
> effectively a RAID-0.
>
> RAID-5 is Striping with Distributed Parity and requires a *minimum* of
> 3 disks. Two disks, regardless of how you got there, is considered a
> degraded state.

Actually, I need to retract those statements. A clean two disk RAID-5
array can be created and is actually very similar to a RAID-1 array.

From this thread: http://ubuntuforums.org/archive/index.php/t-566803.html
---
If you're using just 2 disks in a non-degraded setup, then you'll have
the following structure
Drive1 Drive2
Block1: A1 P1
Block2: P2 B1
Block3: C1 P3
Block4: P4 D1

In this case, you're parity blocks will actually be identical to your
non-parity blocks. This is because P1 is calculated as P1 = A1 XOR ??
Since there is no other block to XOR your bits with, md falls back to
?? = 0, and we all know that the XOR of 0 is the identity. That is, P1
= A1 XOR 0 == A1
---

I stand corrected. :-)

-- 
Drew

"Nothing in life is to be feared. It is only to be understood."
--Marie Curie
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: migrating from RAID5 to RAID10

[Gilad Arnold]

On Thu, Jun 10, 2010 at 03:21:27PM -0700, Drew wrote:
> In this case, you're parity blocks will actually be identical to your
> non-parity blocks. This is because P1 is calculated as P1 = A1 XOR ??
> Since there is no other block to XOR your bits with, md falls back to
> ?? = 0, and we all know that the XOR of 0 is the identity. That is, P1
> = A1 XOR 0 == A1

I'm curious whether md actually computes the XOR or has a shortcut path
for the case where n=2.

> I stand corrected. :-)

No problem ;-)

Gilad

Re: migrating from RAID5 to RAID10

[Gilad Arnold]

On Thu, Jun 10, 2010 at 03:11:43PM -0700, Drew wrote:
> Actually No. If it is RAID-5 as you claim, then currently it is
> effectively a RAID-0.
> 
> RAID-5 is Striping with Distributed Parity and requires a *minimum* of
> 3 disks. Two disks, regardless of how you got there, is considered
> a degraded state.

Sounds to me that you can still stripe and distribute parity with two
drives, only that parity will amount to mirroring, and it can still
tolerate a single drive failure. I got there in the usual way, creating
a RAID5 with --raid-devices=2.


--8<------8<------8<------8<------8<------8<------8<------8<------8<----

$ mdadm --detail /dev/md4
/dev/md4:
        Version : 0.90
  Creation Time : Thu Nov 27 15:48:58 2008
     Raid Level : raid5
     Array Size : 484391808 (461.95 GiB 496.02 GB)
  Used Dev Size : 484391808 (461.95 GiB 496.02 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 4
    Persistence : Superblock is persistent

    Update Time : Thu Jun 10 15:16:45 2010
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           UUID : 1da37f10:3c14071d:40784aba:8675d418
         Events : 0.94324

    Number   Major   Minor   RaidDevice State
       0       8        4        0      active sync   /dev/sda4
       1       8       20        1      active sync   /dev/sdb4

--8<------8<------8<------8<------8<------8<------8<------8<------8<----


Gilad

Re: migrating from RAID5 to RAID10

[Bill Davidsen]

Gilad Arnold wrote:
> Thanks for your response, Roman.
>
> On Fri, Jun 11, 2010 at 12:52:30AM +0600, Roman Mamedov wrote:
>   
>>> I have been using RAID5 on a desktop machine using a couple of 500
>>> GB drives.  My original intent was to grow the array by adding more
>>> drives, as the need arises.
>>>       
>> Do you currently run RAID5 with just 2 drives, in degraded mode, or
>> maybe you meant something else?
>>     
>
> No, it is not degraded, it's a clean 2-drive RAID5. I know it doesn't
> make much sense as it is ;-)  the intent was to grow the array later,
> relying on mdadm's grow feature. Right now, I'm guessing that it
> operates like a RAID1 for all practical purposes.
>
>   
I have the feeling that you will then get a lot of read, alter, rewrite 
(RAW) operations on the parity chuck, where you would just get writes of 
a copy for raid{1,10} configuration. The performance of raid5 is poor, 
but with 4-5 drives it gives you some fault tolerance. With even three 
drives the write performance is poor by algorithm, and the read 
performance is poor by implementation (I see no reason for reading at 
the speed of just one drive).

I think your upgrade will fail, but feel free to see if a two drive 
raid5 will start with only one drive, I could be wrong.

If it were me, I would leave the raid up, install the new drive, and 
just copy the data to it. That way you have error tolerance on the 
original data, while your method doesn't. After doing that and reading 
the entire new drive to be sure it's valid, configure the two original 
drives as a degraded three drive raid10. After that's done AND TESTED to 
see that the data are still all valid, then you add the new drive to the 
raid10 and get full operation.

Actually, if it were me I'd have a backup, too. And be damn sure to run 
on a UPS.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein

Re: migrating from RAID5 to RAID10

[Gilad Arnold]

Hi Bill,

On Wed, Jun 16, 2010 at 11:54:06AM -0400, Bill Davidsen wrote:
> If it were me, I would leave the raid up, install the new drive, and
> just copy the data to it. That way you have error tolerance on the
> original data, while your method doesn't. After doing that and reading
> the entire new drive to be sure it's valid, configure the two original
> drives as a degraded three drive raid10. After that's done AND TESTED
> to see that the data are still all valid, then you add the new drive
> to the raid10 and get full operation.

Thanks for the thorough evaluation and thoughtful advice! Much
appreciated.  I agree it makes a lot more sense and I will follow your
suggested procedure (when I get to purchase the extra drive ;-) )

> Actually, if it were me I'd have a backup, too. And be damn sure to
> run on a UPS.

Great points. I will use a remote backup. I should get a UPS, really...
that'll be a good incentive to stop procrastinating this ;-)

Thanks again.

Gilad

Re: migrating from RAID5 to RAID10

[Gilad Arnold]

On Wed, Jun 16, 2010 at 11:30:31AM -0700, Gilad Arnold wrote:
> Thanks for the thorough evaluation and thoughtful advice! Much
> appreciated.  I agree it makes a lot more sense and I will follow your
> suggested procedure (when I get to purchase the extra drive ;-) )

PS, a related question: some measurements posted on the web suggest that
RAID10 in f2 mode is far superior to n2 (the default?) when it comes to
reading performance, and only mildly inferior when writing.  I tend to
go with f2, unless someone tells me it's a bad idea.

Thanks again for everyone's input.

Gilad

Re: migrating from RAID5 to RAID10

[Bill Davidsen]

Gilad Arnold wrote:
> On Wed, Jun 16, 2010 at 11:30:31AM -0700, Gilad Arnold wrote:
>   
>> Thanks for the thorough evaluation and thoughtful advice! Much
>> appreciated.  I agree it makes a lot more sense and I will follow your
>> suggested procedure (when I get to purchase the extra drive ;-) )
>>     
>
> PS, a related question: some measurements posted on the web suggest that
> RAID10 in f2 mode is far superior to n2 (the default?) when it comes to
> reading performance, and only mildly inferior when writing.  I tend to
> go with f2, unless someone tells me it's a bad idea.
>
>   
It is a GREAT idea, and will produce much better sustained read performance.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein

Re: migrating from RAID5 to RAID10

[Neil Brown]

On Wed, 16 Jun 2010 11:54:06 -0400
Bill Davidsen <davidsen@tmr.com> wrote:

> Gilad Arnold wrote:
> > Thanks for your response, Roman.
> >
> > On Fri, Jun 11, 2010 at 12:52:30AM +0600, Roman Mamedov wrote:
> >   
> >>> I have been using RAID5 on a desktop machine using a couple of 500
> >>> GB drives.  My original intent was to grow the array by adding more
> >>> drives, as the need arises.
> >>>       
> >> Do you currently run RAID5 with just 2 drives, in degraded mode, or
> >> maybe you meant something else?
> >>     
> >
> > No, it is not degraded, it's a clean 2-drive RAID5. I know it doesn't
> > make much sense as it is ;-)  the intent was to grow the array later,
> > relying on mdadm's grow feature. Right now, I'm guessing that it
> > operates like a RAID1 for all practical purposes.
> >
> >   
> I have the feeling that you will then get a lot of read, alter, rewrite 
> (RAW) operations on the parity chuck, where you would just get writes of 
> a copy for raid{1,10} configuration.

Nup.  When there are just two devices, every write will be a full-stripe write
so it will just calculate the parity and write.
If you try to write less than one (aligned) page it will need to pre-read to
get the rest of the page, but that is unlikely.  There will definitely be no
pre-reading of parity.

A RAID5 on 2 devices would be a little slower than RAID1 on two devices as
there is more copying of data around in memory, and there is no read
balancing, but it shouldn't be much slower.

However with recent mdadm and kernel you can trivially convert a 2 drive
RAID1 to a 2 drive RAID5 while the array is online, so it should be easy to
experiment and change you mind about how you want it configured.

>                                        The performance of raid5 is poor, 
> but with 4-5 drives it gives you some fault tolerance. With even three 
> drives the write performance is poor by algorithm, and the read 
> performance is poor by implementation (I see no reason for reading at 
> the speed of just one drive).
> 
> I think your upgrade will fail, but feel free to see if a two drive 
> raid5 will start with only one drive, I could be wrong.

A two-drive RAID5 will definitely work with just one drive present.

NeilBrown

> 
> If it were me, I would leave the raid up, install the new drive, and 
> just copy the data to it. That way you have error tolerance on the 
> original data, while your method doesn't. After doing that and reading 
> the entire new drive to be sure it's valid, configure the two original 
> drives as a degraded three drive raid10. After that's done AND TESTED to 
> see that the data are still all valid, then you add the new drive to the 
> raid10 and get full operation.
> 
> Actually, if it were me I'd have a backup, too. And be damn sure to run 
> on a UPS.
> 

Re: migrating from RAID5 to RAID10

[Gilad Arnold]

On Thu, Jun 17, 2010 at 07:26:31AM +1000, Neil Brown wrote:
> A RAID5 on 2 devices would be a little slower than RAID1 on two
> devices as there is more copying of data around in memory, and there
> is no read balancing, but it shouldn't be much slower.
> 
> However with recent mdadm and kernel you can trivially convert a 2 drive
> RAID1 to a 2 drive RAID5 while the array is online, so it should be easy to
> experiment and change you mind about how you want it configured.

Thanks for the interesting comments, Neil.

Preparing for the upgrade I've been cleaning up data and ended up
realizing that I won't need to add another drive in the near future ;-)
I would like to change my RAID level, though, to improve performance,
and the idea of online layout conversion is very appealing.

I understand that a 2-drive RAID5 can be converted to a RAID1 already,
but it is also my understanding that RAID1 currently does not benefit
from striped reading performance, either entirely (due to
implementation) or partly (compared to RAID10 in f2 mode).  This leads
me to think that even with 2 drives, RAID10/f2 is a better choice than
RAID1. Is this a fair assessment?  And, if it is the case, can RAID5 be
converted to RAID10/f2 on-the-fly, or would I have to take the longer
path?  (i.e.  degrade RAID5, start RAID10 in degraded mode, copy data,
kill RAID5 and rebuild RAID10)

Many thanks for all your help. I finally seem to be converging ;-)

Gilad

Re: migrating from RAID5 to RAID10

[Neil Brown]

On Thu, 17 Jun 2010 10:44:46 -0700
Gilad Arnold <arnold@cs.berkeley.edu> wrote:

> On Thu, Jun 17, 2010 at 07:26:31AM +1000, Neil Brown wrote:
> > A RAID5 on 2 devices would be a little slower than RAID1 on two
> > devices as there is more copying of data around in memory, and there
> > is no read balancing, but it shouldn't be much slower.
> > 
> > However with recent mdadm and kernel you can trivially convert a 2 drive
> > RAID1 to a 2 drive RAID5 while the array is online, so it should be easy to
> > experiment and change you mind about how you want it configured.
> 
> Thanks for the interesting comments, Neil.
> 
> Preparing for the upgrade I've been cleaning up data and ended up
> realizing that I won't need to add another drive in the near future ;-)
> I would like to change my RAID level, though, to improve performance,
> and the idea of online layout conversion is very appealing.
> 
> I understand that a 2-drive RAID5 can be converted to a RAID1 already,
> but it is also my understanding that RAID1 currently does not benefit
> from striped reading performance, either entirely (due to
> implementation) or partly (compared to RAID10 in f2 mode).  This leads
> me to think that even with 2 drives, RAID10/f2 is a better choice than
> RAID1. Is this a fair assessment?  And, if it is the case, can RAID5 be
> converted to RAID10/f2 on-the-fly, or would I have to take the longer
> path?  (i.e.  degrade RAID5, start RAID10 in degraded mode, copy data,
> kill RAID5 and rebuild RAID10)


2-drive RAID10 f2  would be expected to provide better read throughput
(possibly twice as fast) at some cost to write throughput.  For many people
this is a worthwhile trade-off.  So it might be better for you.
Read throughput would degraded down to write throughput (i.e. slower than
RAID1) if the RAID10 were degraded.

There is currently no support for converting RAID10 arrays in any way.  You
have to copy the data.
With 2 drives, you could make a degraded RAID1 and a degraded RAID10 and copy
the data across.  Then add the RAID1 device into the RAID10 array.

NeilBrown

Re: migrating from RAID5 to RAID10

[Keld Simonsen]

On Fri, Jun 18, 2010 at 08:30:53AM +1000, Neil Brown wrote:
> 
> 2-drive RAID10 f2  would be expected to provide better read throughput
> (possibly twice as fast) at some cost to write throughput.  For many people
> this is a worthwhile trade-off.  So it might be better for you.
> Read throughput would degraded down to write throughput (i.e. slower than
> RAID1) if the RAID10 were degraded.

The slowdown of write thruput is on the raw raid, and on file systems
without elevator algorithms. If you employ a file system with an
elevator algorithm, then there will be no noticeable slowdown for
writes, as witnessed by many benchmarks.

best regards
keld

Re: migrating from RAID5 to RAID10

[Neil Brown]

On Fri, 18 Jun 2010 10:40:28 +0200
Keld Simonsen <keld@keldix.com> wrote:

> On Fri, Jun 18, 2010 at 08:30:53AM +1000, Neil Brown wrote:
> > 
> > 2-drive RAID10 f2  would be expected to provide better read throughput
> > (possibly twice as fast) at some cost to write throughput.  For many people
> > this is a worthwhile trade-off.  So it might be better for you.
> > Read throughput would degraded down to write throughput (i.e. slower than
> > RAID1) if the RAID10 were degraded.
> 
> The slowdown of write thruput is on the raw raid, and on file systems
> without elevator algorithms. If you employ a file system with an
> elevator algorithm, then there will be no noticeable slowdown for
> writes, as witnessed by many benchmarks.

Filesystems don't implement the elevator algorithm.
It is the driver for the disk drive which does that.
So if you have a filesystem on RAID10 on 4 disk drives, then there are 4
instances of the elevator algorithm between the RAID10 and each disk drive.

With RAID10 f2, consecutive chunks needs to be written to widely different
locations on the device.  The elevator certainly helps here by sorting the
write requests so that all the even chunks go to one end of the device, then
all the odd chunks go to the other end.
The only difference that I can imagine between a filesystem and raw-device
access is that maybe one allows the queue to fill up with more write requests
before flushing them out, thus giving the elevator more opportunity to
improve things. However I cannot see that difference in the code (though I
might be missing it).

One difference between writing to a raw device and writing to a filesystem is
that when you close the raw device, all pending writes are flushed, while
when you close a file the data can stay 'dirty' in memory for several more
seconds.  So writing to a raw device might seem slower because the close is
slower.

NeilBrown

Re: migrating from RAID5 to RAID10

[Gilad Arnold]

On Thu, Jun 17, 2010 at 10:44:45AM -0700, Gilad Arnold wrote:
> This leads me to think that even with 2 drives, RAID10/f2 is a better
> choice than RAID1. Is this a fair assessment?

Okay, I found the benchmarks that support this hypothesis ;-)

> And, if it is the case, can RAID5 be converted to RAID10/f2
> on-the-fly, or would I have to take the longer path?  (i.e. degrade
> RAID5, start RAID10 in degraded mode, copy data, kill RAID5 and
> rebuild RAID10)

Some more reading leads me to think that on-the-fly RAID5 to RAID10/f2
conversion wouldn't work.  However, I'm tempted to believe that I can
avoid the data copy phase entirely and skip straight to RAID10 rebuild:

1. Shutdown the RAID5.

2. Create a RAID10/f2 on top of one drive + missing, ignoring mdadm's
   warnings about an existing array. If I'm not terribly wrong, this
   will give me a degraded RAID10 that still has all my data in place.

3. Add the second drive to the RAID10 and watch it rebuild.


Apologies for my petty questions ;-)  Advice appreciated.

Gilad

Re: migrating from RAID5 to RAID10

[Neil Brown]

On Thu, 17 Jun 2010 15:46:57 -0700
Gilad Arnold <arnold@cs.berkeley.edu> wrote:

> On Thu, Jun 17, 2010 at 10:44:45AM -0700, Gilad Arnold wrote:
> > This leads me to think that even with 2 drives, RAID10/f2 is a better
> > choice than RAID1. Is this a fair assessment?
> 
> Okay, I found the benchmarks that support this hypothesis ;-)
> 
> > And, if it is the case, can RAID5 be converted to RAID10/f2
> > on-the-fly, or would I have to take the longer path?  (i.e. degrade
> > RAID5, start RAID10 in degraded mode, copy data, kill RAID5 and
> > rebuild RAID10)
> 
> Some more reading leads me to think that on-the-fly RAID5 to RAID10/f2
> conversion wouldn't work.  However, I'm tempted to believe that I can
> avoid the data copy phase entirely and skip straight to RAID10 rebuild:
> 
> 1. Shutdown the RAID5.
> 
> 2. Create a RAID10/f2 on top of one drive + missing, ignoring mdadm's
>    warnings about an existing array. If I'm not terribly wrong, this
>    will give me a degraded RAID10 that still has all my data in place.

I cannot imagine why you would think that all your data would be in place.
I can assure that it would not.

NeilBrown

> 
> 3. Add the second drive to the RAID10 and watch it rebuild.
> 
> 
> Apologies for my petty questions ;-)  Advice appreciated.
> 
> Gilad

Re: migrating from RAID5 to RAID10

[Gilad Arnold]

On Fri, Jun 18, 2010 at 12:39:10PM +1000, Neil Brown wrote:
> > 1. Shutdown the RAID5.
> > 
> > 2. Create a RAID10/f2 on top of one drive + missing, ignoring mdadm's
> >    warnings about an existing array. If I'm not terribly wrong, this
> >    will give me a degraded RAID10 that still has all my data in place.
> 
> I cannot imagine why you would think that all your data would be in place.
> I can assure that it would not.

I was thinking it would be a similar case to the one mentioned here:

http://scott.wallace.sh/2007/04/14/converting-raid1-to-raid5-with-no-data-loss/

Guess I was wrong. Thanks for the advice.

Gilad