Basic RAID5/6 reshape question

Basic RAID5/6 reshape question

[Brad Campbell]

G'day all,

I'm just doing some system upgrades and shuffling some arrays around as part of the process.
All my other current questions have been answered by experimentation, but one.

I'm going to start with a RAID 6 comprised of 7 x 1TB drives, and reshaping to 10 x 1TB.
I'm doing an experiment right now with re-shaping a RAID-5 of 2 drives into 3 drives which is taking 
about 20 hours, from that I'm assuming that a nice big re-shape will take a considerable amount of 
time. I have a reliable UPS, I'm more concerned about a drive or cable flaking out.

What happens if I'm in the middle of a reshape of a RAID 5 or 6 and lose a drive?

Will the array be able to continue to reshape in a degraded state?

Hoping of course that this does not occur, but just trying to plan ahead in case.

Regards,
Brad
-- 
Dolphins are so intelligent that within a few weeks they can
train Americans to stand at the edge of the pool and throw them
fish.

Re: Basic RAID5/6 reshape question

[Ryan Wagoner]

It takes 3 hours to resync my 3 x 1 TB drive RAID 5 array at a rate of
85MB/s. It shouldn't take 20 hours to write 3 x 1 TB drives. How fast
is it progressing? You might consider a PCI-X or PCI Express SATA or
SAS controller card. I have my drives connected to an Supermicro LSI
1068 PCI Express SAS card, which performs great.

The other option would be to create 2 RAID 5 arrays of 5 drives each.
Create the first array and use LVM with the filesystem of your choice.
Add the second array later and expand the LVM volume.

Ryan

On Thu, Dec 11, 2008 at 4:55 AM, Brad Campbell <brad@wasp.net.au> wrote:
> G'day all,
>
> I'm just doing some system upgrades and shuffling some arrays around as part
> of the process.
> All my other current questions have been answered by experimentation, but
> one.
>
> I'm going to start with a RAID 6 comprised of 7 x 1TB drives, and reshaping
> to 10 x 1TB.
> I'm doing an experiment right now with re-shaping a RAID-5 of 2 drives into
> 3 drives which is taking about 20 hours, from that I'm assuming that a nice
> big re-shape will take a considerable amount of time. I have a reliable UPS,
> I'm more concerned about a drive or cable flaking out.
>
> What happens if I'm in the middle of a reshape of a RAID 5 or 6 and lose a
> drive?
>
> Will the array be able to continue to reshape in a degraded state?
>
> Hoping of course that this does not occur, but just trying to plan ahead in
> case.
>
> Regards,
> Brad
> --
> Dolphins are so intelligent that within a few weeks they can
> train Americans to stand at the edge of the pool and throw them
> fish.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Basic RAID5/6 reshape question

[Brad Campbell]

Ryan Wagoner wrote:
> It takes 3 hours to resync my 3 x 1 TB drive RAID 5 array at a rate of
> 85MB/s. It shouldn't take 20 hours to write 3 x 1 TB drives. How fast
> is it progressing? You might consider a PCI-X or PCI Express SATA or
> SAS controller card. I have my drives connected to an Supermicro LSI
> 1068 PCI Express SAS card, which performs great.
>

I'm using 2 x Marvell 7042 based PCI-E x4 controllers and a SIL 3xxx (I don't recall but its PCI-E 
x1 and 2 ports). Bandwidth is not an issue.

A resync appears to be a completely different kettle of fish to a reshape. With re-shape it appears 
to write status data to the raid superblocks periodically. This appears to thrash the heads from the 
current position to the end of the disk and back. At the start of the reshape I was running about 
15-20MB/s to each drive. As the reshape progressed to the end of the disks I was looking at closer 
to 40MB/s. So obviously the final figure was less than the initial 20 hours, but a hell of a lot 
slower than a straight resync.

This blind stab at reshape behaviour is absolutely pure conjecture based on a gross guess at what 
was happening, however it does seem to bear out scrutiny.

> The other option would be to create 2 RAID 5 arrays of 5 drives each.
> Create the first array and use LVM with the filesystem of your choice.
> Add the second array later and expand the LVM volume.

Mmm thanks, but no.. to use a car analogy, "I'd like something I can fix with a screwdriver and a 
hammer". LVM is overcomplicating things way past what I require. In addition, I really like RAID-6. 
Over the last 4 years with my 15 drive RAID-6 I have had two double drive failures. Both times one 
drive died in a not so nice fashion, and one developed a grown defect during the rebuild. Raid-5 
just does not cut the mustard with me anymore.


Regards,
Brad
-- 
Dolphins are so intelligent that within a few weeks they can
train Americans to stand at the edge of the pool and throw them
fish.

Re: Basic RAID5/6 reshape question

[Justin Piszcz]

On Fri, 12 Dec 2008, Brad Campbell wrote:

> Ryan Wagoner wrote:
>> It takes 3 hours to resync my 3 x 1 TB drive RAID 5 array at a rate of
>> 85MB/s. It shouldn't take 20 hours to write 3 x 1 TB drives. How fast
>> is it progressing? You might consider a PCI-X or PCI Express SATA or
>> SAS controller card. I have my drives connected to an Supermicro LSI
>> 1068 PCI Express SAS card, which performs great.
>> 
>
> I'm using 2 x Marvell 7042 based PCI-E x4 controllers and a SIL 3xxx (I don't 
> recall but its PCI-E x1 and 2 ports). Bandwidth is not an issue.
>
> A resync appears to be a completely different kettle of fish to a reshape. 
> With re-shape it appears to write status data to the raid superblocks 
> periodically. This appears to thrash the heads from the current position to 
> the end of the disk and back. At the start of the reshape I was running about 
> 15-20MB/s to each drive. As the reshape progressed to the end of the disks I 
> was looking at closer to 40MB/s. So obviously the final figure was less than 
> the initial 20 hours, but a hell of a lot slower than a straight resync.
>
> This blind stab at reshape behaviour is absolutely pure conjecture based on a 
> gross guess at what was happening, however it does seem to bear out scrutiny.
>
>> The other option would be to create 2 RAID 5 arrays of 5 drives each.
>> Create the first array and use LVM with the filesystem of your choice.
>> Add the second array later and expand the LVM volume.
>
> Mmm thanks, but no.. to use a car analogy, "I'd like something I can fix with 
> a screwdriver and a hammer". LVM is overcomplicating things way past what I 
> require. In addition, I really like RAID-6. Over the last 4 years with my 15 
> drive RAID-6 I have had two double drive failures. Both times one drive died 
> in a not so nice fashion, and one developed a grown defect during the 
> rebuild. Raid-5 just does not cut the mustard with me anymore.
            ^^^^^^^^^^^^^

Agree with you there, even when you have the data backed up losing and 
having to rebuild everything is such a pain-- RAID-6 has served me quite 
well and I have only started using it recently, it did save me from one 
double-drive failure so far, I have not looked back since.

Justin.

Re: Basic RAID5/6 reshape question

[Roger Heflin]

Brad Campbell wrote:
> G'day all,
> 
> I'm just doing some system upgrades and shuffling some arrays around as 
> part of the process.
> All my other current questions have been answered by experimentation, 
> but one.
> 
> I'm going to start with a RAID 6 comprised of 7 x 1TB drives, and 
> reshaping to 10 x 1TB.
> I'm doing an experiment right now with re-shaping a RAID-5 of 2 drives 
> into 3 drives which is taking about 20 hours, from that I'm assuming 
> that a nice big re-shape will take a considerable amount of time. I have 
> a reliable UPS, I'm more concerned about a drive or cable flaking out.
> 
> What happens if I'm in the middle of a reshape of a RAID 5 or 6 and lose 
> a drive?
> 
> Will the array be able to continue to reshape in a degraded state?
> 
> Hoping of course that this does not occur, but just trying to plan ahead 
> in case.
> 
> Regards,
> Brad

Reshaping appears to be much much slower.

I can resync in just a 2-3 hours, but a reshape of a single drive 
takes 10-15 hours with nothing else happening, if any other work is 
being done by the disks the number gets much worse.

To make it look just like a normally built array of the same number of 
disks it is very likely that reshaping is moving lots and lots of data 
around to make the changes.

I have so far done 2 reshapes, and I would like to know also how 
dangerous an event would be if it happens during a reshape.

Re: Basic RAID5/6 reshape question

[Neil Brown]

On Thursday December 11, rogerheflin@gmail.com wrote:
> 
> Reshaping appears to be much much slower.

Yes.

> 
> I can resync in just a 2-3 hours, but a reshape of a single drive 
> takes 10-15 hours with nothing else happening, if any other work is 
> being done by the disks the number gets much worse.

With resync, the drives a streaming - reading sequential blocks.
With reshape, they are reading from one place, and writing to
another.   It tries to maximise the size of the reads and writes and
so minimise seeks, but there is a limit to that.  If you increase the
stripe_cache_size that should increase the size of each read and write
and so might improve reshape speed a bit.


> 
> To make it look just like a normally built array of the same number of 
> disks it is very likely that reshaping is moving lots and lots of data 
> around to make the changes.

True.

> 
> I have so far done 2 reshapes, and I would like to know also how 
> dangerous an event would be if it happens during a reshape.

A drive failure or a system crash should be handled correctly.

NeilBrown

Re: Basic RAID5/6 reshape question

[Neil Brown]

On Thursday December 11, brad@wasp.net.au wrote:
> G'day all,
> 
> I'm just doing some system upgrades and shuffling some arrays around as part of the process.
> All my other current questions have been answered by experimentation, but one.
> 
> I'm going to start with a RAID 6 comprised of 7 x 1TB drives, and reshaping to 10 x 1TB.
> I'm doing an experiment right now with re-shaping a RAID-5 of 2 drives into 3 drives which is taking 
> about 20 hours, from that I'm assuming that a nice big re-shape will take a considerable amount of 
> time. I have a reliable UPS, I'm more concerned about a drive or cable flaking out.
> 
> What happens if I'm in the middle of a reshape of a RAID 5 or 6 and lose a drive?

The reshape continues creating a degraded array.

> 
> Will the array be able to continue to reshape in a degraded state?

Yes.
The process is designed to survive losing a single drive, or a system
crash (but not both).

> 
> Hoping of course that this does not occur, but just trying to plan ahead in case.

Sensible!

NeilBrown