Raid 5 rebuild and Raid 6 reshape question

Raid 5 rebuild and Raid 6 reshape question

[Thomas Fjellstrom]

This is a potentially dumb question...

I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still 
seemed to work), and i decided to replace it. I actually bought two drives, 
thinking I'd keep one as a cold spare, but when i got to replacing the drive 
(after days of load testing to see if I'd gotten a dud), I figured maybe I'd 
use the spare as a hot spare, or make a Raid6 array. In the end I decided on a 
Raid6, as it is the recommended way to go these days for large disks.

When I did the add for the first disk, it started rebuilding the raid5, and 
said it'd only take like 5 hours, then I added the second disk and it 
skyrocketed to about 116 hours (7000s or so). I presume this is expected, due 
to having to re-stripe all of the data, and re-calculate the parity at the 
same time.

How much faster would this have finished if I had just waited for the first 
drive to finish rebuilding and done the reshape as a separate step? I decided 
to just let this reshape continue, its about 30% the way through, and this is 
actually the backup array for my main NAS, so it isn't actually a problem if 
it dies mid-reshape. I'm just curious about the alternatives to how I did it. 
I know i could have just copied the old disk to one of the new ones, and 
skipped the entire rebuild on one disk, but i didn't feel like it ;D

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: Raid 5 rebuild and Raid 6 reshape question

[Mikael Abrahamsson]

On Fri, 17 Apr 2015, Thomas Fjellstrom wrote:

> I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
> seemed to work), and i decided to replace it. I actually bought two drives,

If you have a not-ancient kernel, you should have issued the replace 
command. You should have added one new drive as spare, then (with a new 
mdadm) issued the --replace command. This would have started copying the 
drive you wanted replaced onto the spare. Then after all this was done, 
you could have added another spare and told it to re-shape to raid6. 
That's at least what I would have done.

http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-yet-failed-disk-in-a-linux-raid5-array

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Raid 5 rebuild and Raid 6 reshape question

[Thomas Fjellstrom]

On Sat 18 Apr 2015 03:13:52 PM Mikael Abrahamsson wrote:
> On Fri, 17 Apr 2015, Thomas Fjellstrom wrote:
> > I had a 5 disk Raid5 with 3TB disks. One disk threw some errors (but still
> > seemed to work), and i decided to replace it. I actually bought two
> > drives,
> 
> If you have a not-ancient kernel, you should have issued the replace
> command. You should have added one new drive as spare, then (with a new
> mdadm) issued the --replace command. This would have started copying the
> drive you wanted replaced onto the spare. Then after all this was done,
> you could have added another spare and told it to re-shape to raid6.
> That's at least what I would have done.
> 
> http://unix.stackexchange.com/questions/74924/how-to-safely-replace-a-not-ye
> t-failed-disk-in-a-linux-raid5-array

Yes, I could have. The machine's drive bays were full at the time, it wasnt 
till I had inserted the new drives that I realized i could have left the old 
one in and done a copy or replace. But then the machine was booting without it 
and probably not a great time to put the old disk back in (though an assemble 
force could have brought it back?). It's almost 60% done, so it's not a huge 
problem. I was just curious as to how long an actual raid5->raid6 reshape 
would normally be or this setup. I expect it normally can take a while as it 
has to rewrite a lot of data (11TiB worth).

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca

Re: Raid 5 rebuild and Raid 6 reshape question

[Mikael Abrahamsson]

On Sat, 18 Apr 2015, Thomas Fjellstrom wrote:

> problem. I was just curious as to how long an actual raid5->raid6 
> reshape would normally be or this setup. I expect it normally can take a 
> while as it has to rewrite a lot of data (11TiB worth).

In my experience it takes 5-15 times longer than a normal resync takes, to 
reshape from raid5->raid6. So for your 3TB disk size case, I'd say expect 
4-8 days or so.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Raid 5 rebuild and Raid 6 reshape question

[Thomas Fjellstrom]

On Sun 19 Apr 2015 10:43:32 AM Mikael Abrahamsson wrote:
> On Sat, 18 Apr 2015, Thomas Fjellstrom wrote:
> > problem. I was just curious as to how long an actual raid5->raid6
> > reshape would normally be or this setup. I expect it normally can take a
> > while as it has to rewrite a lot of data (11TiB worth).
> 
> In my experience it takes 5-15 times longer than a normal resync takes, to
> reshape from raid5->raid6. So for your 3TB disk size case, I'd say expect
> 4-8 days or so.

Ah. Good to know. mdstat was saying about 4-5 days. I kindof thought that was 
about normal given the amount of work involved to restripe all drives and 
calculate parity.

Thanks :)

-- 
Thomas Fjellstrom
thomas@fjellstrom.ca