RAID 10 on Fusion IO cards problems

RAID 10 on Fusion IO cards problems

[Albert Pauw]

Hi guys,

I am trying to get a RAID 10 configuration working at work, but seem
to hit a performance wall after 20 minutes into a DB creation session.

Here's the setup:

OS: Oracle Linux 5.9 (effectively RHEL 5.9), kernel  2.6.32-400.29.2.el5uek.
All utilities updates, mdadm (2.6.9 latest through updates).

Setup:

Two Fusion IO Duo cards, each Fusion IO device 640 GB, so four in total.

Raid 10 set up as:

striped between the two IO devices on the same Fusion IO card, and
mirrored between the separate cards.

So, Fusion IO card 1, device fioa and fiob, Fusion IO card 2, device
fioc and fiod.

The two stripes are fioa/fiob and fioc/fiod, and a mirror between these devices:

mdadm --create --verbose /dev/md0 --level=10 --metadata=1.2
--chunk=512 --raid-devices=4 /dev/fioa /dev/fioc /dev/fiob /dev/fiod
--assume-clean -N md0

When the performance turned out bad, after about 20 minutes, the
process was stopped. I broke the mirror, so the md0 device is only
striped, but the performance hit after 20 minutes happened again.

The status of all cards are fine, no problems there. Then I created a
fs on only one device and have it run again. This time it worked fine.
The fs was in all cases ext3, no TRIM.

Any suggestions, experience with this kind of setup?

Thanks,

Albert

Re: RAID 10 on Fusion IO cards problems

[Roberto Spadim]

i use a raid10 far in revodrive cards, but i'm using kernel 3.10.7
with slackware and it runs fine

2013/8/29 Albert Pauw <albert.pauw@gmail.com>:
> Hi guys,
>
> I am trying to get a RAID 10 configuration working at work, but seem
> to hit a performance wall after 20 minutes into a DB creation session.
>
> Here's the setup:
>
> OS: Oracle Linux 5.9 (effectively RHEL 5.9), kernel  2.6.32-400.29.2.el5uek.
> All utilities updates, mdadm (2.6.9 latest through updates).
>
> Setup:
>
> Two Fusion IO Duo cards, each Fusion IO device 640 GB, so four in total.
>
> Raid 10 set up as:
>
> striped between the two IO devices on the same Fusion IO card, and
> mirrored between the separate cards.
>
> So, Fusion IO card 1, device fioa and fiob, Fusion IO card 2, device
> fioc and fiod.
>
> The two stripes are fioa/fiob and fioc/fiod, and a mirror between these devices:
>
> mdadm --create --verbose /dev/md0 --level=10 --metadata=1.2
> --chunk=512 --raid-devices=4 /dev/fioa /dev/fioc /dev/fiob /dev/fiod
> --assume-clean -N md0
>
> When the performance turned out bad, after about 20 minutes, the
> process was stopped. I broke the mirror, so the md0 device is only
> striped, but the performance hit after 20 minutes happened again.
>
> The status of all cards are fine, no problems there. Then I created a
> fs on only one device and have it run again. This time it worked fine.
> The fs was in all cases ext3, no TRIM.
>
> Any suggestions, experience with this kind of setup?
>
> Thanks,
>
> Albert
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Roberto Spadim

Re: RAID 10 on Fusion IO cards problems

[Albert Pauw]

Hi Roberto,

could you share your setup? What chunksize did you use, e.g. How did
you create it (if you can remember)?

Thanks,

Albert

On 29 August 2013 15:11, Roberto Spadim <rspadim@gmail.com> wrote:
> i use a raid10 far in revodrive cards, but i'm using kernel 3.10.7
> with slackware and it runs fine
>
> 2013/8/29 Albert Pauw <albert.pauw@gmail.com>:
>> Hi guys,
>>
>> I am trying to get a RAID 10 configuration working at work, but seem
>> to hit a performance wall after 20 minutes into a DB creation session.
>>
>> Here's the setup:
>>
>> OS: Oracle Linux 5.9 (effectively RHEL 5.9), kernel  2.6.32-400.29.2.el5uek.
>> All utilities updates, mdadm (2.6.9 latest through updates).
>>
>> Setup:
>>
>> Two Fusion IO Duo cards, each Fusion IO device 640 GB, so four in total.
>>
>> Raid 10 set up as:
>>
>> striped between the two IO devices on the same Fusion IO card, and
>> mirrored between the separate cards.
>>
>> So, Fusion IO card 1, device fioa and fiob, Fusion IO card 2, device
>> fioc and fiod.
>>
>> The two stripes are fioa/fiob and fioc/fiod, and a mirror between these devices:
>>
>> mdadm --create --verbose /dev/md0 --level=10 --metadata=1.2
>> --chunk=512 --raid-devices=4 /dev/fioa /dev/fioc /dev/fiob /dev/fiod
>> --assume-clean -N md0
>>
>> When the performance turned out bad, after about 20 minutes, the
>> process was stopped. I broke the mirror, so the md0 device is only
>> striped, but the performance hit after 20 minutes happened again.
>>
>> The status of all cards are fine, no problems there. Then I created a
>> fs on only one device and have it run again. This time it worked fine.
>> The fs was in all cases ext3, no TRIM.
>>
>> Any suggestions, experience with this kind of setup?
>>
>> Thanks,
>>
>> Albert
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> Roberto Spadim

Re: RAID 10 on Fusion IO cards problems

[Roberto Spadim]

i created it with
mdadm --create /dev/md0 --level=10 --layout=f2 --raid-devices=4
/dev/sda /dev/sdb /dev/sdc /dev/sdd

i didn't used the chunck option (512kb is the default right?)
i think you can omit the layout, maybe the n2 is better  but i didn't tested
i use the f2 in hdd configs when i want a 'big' read performace and a
write isn't important, since ssd don't have head, maybe a layout=n2
(or no layout option, just use the default) is better, but you must
test with your workload


2013/8/29 Albert Pauw <albert.pauw@gmail.com>:
> Hi Roberto,
>
> could you share your setup? What chunksize did you use, e.g. How did
> you create it (if you can remember)?
>
> Thanks,
>
> Albert
>
> On 29 August 2013 15:11, Roberto Spadim <rspadim@gmail.com> wrote:
>> i use a raid10 far in revodrive cards, but i'm using kernel 3.10.7
>> with slackware and it runs fine
>>
>> 2013/8/29 Albert Pauw <albert.pauw@gmail.com>:
>>> Hi guys,
>>>
>>> I am trying to get a RAID 10 configuration working at work, but seem
>>> to hit a performance wall after 20 minutes into a DB creation session.
>>>
>>> Here's the setup:
>>>
>>> OS: Oracle Linux 5.9 (effectively RHEL 5.9), kernel  2.6.32-400.29.2.el5uek.
>>> All utilities updates, mdadm (2.6.9 latest through updates).
>>>
>>> Setup:
>>>
>>> Two Fusion IO Duo cards, each Fusion IO device 640 GB, so four in total.
>>>
>>> Raid 10 set up as:
>>>
>>> striped between the two IO devices on the same Fusion IO card, and
>>> mirrored between the separate cards.
>>>
>>> So, Fusion IO card 1, device fioa and fiob, Fusion IO card 2, device
>>> fioc and fiod.
>>>
>>> The two stripes are fioa/fiob and fioc/fiod, and a mirror between these devices:
>>>
>>> mdadm --create --verbose /dev/md0 --level=10 --metadata=1.2
>>> --chunk=512 --raid-devices=4 /dev/fioa /dev/fioc /dev/fiob /dev/fiod
>>> --assume-clean -N md0
>>>
>>> When the performance turned out bad, after about 20 minutes, the
>>> process was stopped. I broke the mirror, so the md0 device is only
>>> striped, but the performance hit after 20 minutes happened again.
>>>
>>> The status of all cards are fine, no problems there. Then I created a
>>> fs on only one device and have it run again. This time it worked fine.
>>> The fs was in all cases ext3, no TRIM.
>>>
>>> Any suggestions, experience with this kind of setup?
>>>
>>> Thanks,
>>>
>>> Albert
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>>
>> --
>> Roberto Spadim



-- 
Roberto Spadim

Re: RAID 10 on Fusion IO cards problems

[Stan Hoeppner]

On 8/29/2013 4:20 AM, Albert Pauw wrote:
...
> OS: Oracle Linux 5.9 (effectively RHEL 5.9), kernel  2.6.32-400.29.2.el5uek.
> All utilities updates, mdadm (2.6.9 latest through updates).
...
> Two Fusion IO Duo cards, each Fusion IO device 640 GB, so four in total.
...
> mdadm --create --verbose /dev/md0 --level=10 --metadata=1.2
> --chunk=512 --raid-devices=4 /dev/fioa /dev/fioc /dev/fiob /dev/fiod
> --assume-clean -N md0
> 
> When the performance turned out bad, after about 20 minutes, the
> process was stopped. I broke the mirror, so the md0 device is only
> striped, but the performance hit after 20 minutes happened again.
> 
> The status of all cards are fine, no problems there. Then I created a
> fs on only one device and have it run again. This time it worked fine.
> The fs was in all cases ext3, no TRIM.

You've presented insufficient information to allow a definitive answer.
 That said, it's very likely that you're hitting the same wall many
folks do with SSDs.  All md/RAID personalities are limited to a single
write thread which limits you to one CPU of IO throughput.  When writing
to a single device without md/RAID, block IOs can be processed by all
CPUs in parallel.  The Fusion IO device is likely sufficiently fast that
a single md/RAID10 thread can't saturate the device, so you run out of
CPU before IOPS.  This is very common with SSD and md/RAID.  Shaohua Li
has been busily working on patches for quite some time now to eliminate
this CPU bottleneck in md.

The fact that a single Fusion IO device with EXT3 on it is faster than
md/RAID10 strongly suggests this may be the cause.  If you have multiple
application threads or processes writing to a single device the IOs will
be processed on the same CPU (core) as the thread, so you can have IOs
in flight from all CPUs in parallel.  When using md/RAID all of that IO
must be shuttled to the md driver which can only execute on a single CPU
(core).  To verify this, simply run your tests again and monitor CPU
burn of the md/RAID10 thread.  If that CPU is 100% at any time then this
is the problem.

If this is true, you can immediately mitigate it by using a layered
md/RAID0 over md/RAID1 setup.  Doing this will give you two md/RAID1
write threads, doubling the number of CPU cores you can put into play.
To do this and maintain the card<->card mirror layout you described, you
will create an md/RAID1 with fioa and fioc, and another md/RAID1 with
fiob and fiod.  Then you'll create an md/RAID0 across these two md/RAID1
devices.  The md/RAID0 and linear personalities don't use write threads
and are thus not limited to a single CPU core.

One final suggestion.  Use XFS instead of EXT3/4.  You should get
significantly better performance with a parallel database workload.  But
I'd strongly suggest moving up to a RHEL 6.2+ clone if you do.  5.9 is
ancient, and there are tons of performance and stability enhancements in
newer kernels, specifically related to XFS.

-- 
Stan

Re: RAID 10 on Fusion IO cards problems

[Stan Hoeppner]

On 8/29/2013 6:15 PM, Stan Hoeppner wrote:
> On 8/29/2013 4:20 AM, Albert Pauw wrote:

> I am trying to get a RAID 10 configuration working at work, but seem
> to hit a performance wall after 20 minutes into a DB creation session.

It may help if you explain what a "DB creation session" entails in this
case.  If this is a write heavy process from the beginning of the run,
the fact that you don't run into performance problems until 20 minutes
in would suggest the problem is garbage collection at the SSDs.

However, since the single device w/filesystem doesn't exhibit the
performance problem it would seem GC isn't the cause.  Thus it seems the
process likely doesn't begin heavy write IO until 20 minutes in, at
which point you hit the problem I described in my first reply below.
I'm making logical deductions by analyzing the information you've
presented.  They may not be wholly correct or there may be additional
information that would change the analysis.

A good description of the application's read/write profile would take a
lot of the guesswork/deduction out of the equation, and would be very
helpful in nailing down the root cause of this performance problem.

> ...
>> OS: Oracle Linux 5.9 (effectively RHEL 5.9), kernel  2.6.32-400.29.2.el5uek.
>> All utilities updates, mdadm (2.6.9 latest through updates).
> ...
>> Two Fusion IO Duo cards, each Fusion IO device 640 GB, so four in total.
> ...
>> mdadm --create --verbose /dev/md0 --level=10 --metadata=1.2
>> --chunk=512 --raid-devices=4 /dev/fioa /dev/fioc /dev/fiob /dev/fiod
>> --assume-clean -N md0
>>
>> When the performance turned out bad, after about 20 minutes, the
>> process was stopped. I broke the mirror, so the md0 device is only
>> striped, but the performance hit after 20 minutes happened again.
>>
>> The status of all cards are fine, no problems there. Then I created a
>> fs on only one device and have it run again. This time it worked fine.
>> The fs was in all cases ext3, no TRIM.
> 
> You've presented insufficient information to allow a definitive answer.
>  That said, it's very likely that you're hitting the same wall many
> folks do with SSDs.  All md/RAID personalities are limited to a single
> write thread which limits you to one CPU of IO throughput.  When writing
> to a single device without md/RAID, block IOs can be processed by all
> CPUs in parallel.  The Fusion IO device is likely sufficiently fast that
> a single md/RAID10 thread can't saturate the device, so you run out of
> CPU before IOPS.  This is very common with SSD and md/RAID.  Shaohua Li
> has been busily working on patches for quite some time now to eliminate
> this CPU bottleneck in md.
> 
> The fact that a single Fusion IO device with EXT3 on it is faster than
> md/RAID10 strongly suggests this may be the cause.  If you have multiple
> application threads or processes writing to a single device the IOs will
> be processed on the same CPU (core) as the thread, so you can have IOs
> in flight from all CPUs in parallel.  When using md/RAID all of that IO
> must be shuttled to the md driver which can only execute on a single CPU
> (core).  To verify this, simply run your tests again and monitor CPU
> burn of the md/RAID10 thread.  If that CPU is 100% at any time then this
> is the problem.
> 
> If this is true, you can immediately mitigate it by using a layered
> md/RAID0 over md/RAID1 setup.  Doing this will give you two md/RAID1
> write threads, doubling the number of CPU cores you can put into play.
> To do this and maintain the card<->card mirror layout you described, you
> will create an md/RAID1 with fioa and fioc, and another md/RAID1 with
> fiob and fiod.  Then you'll create an md/RAID0 across these two md/RAID1
> devices.  The md/RAID0 and linear personalities don't use write threads
> and are thus not limited to a single CPU core.
> 
> One final suggestion.  Use XFS instead of EXT3/4.  You should get
> significantly better performance with a parallel database workload.  But
> I'd strongly suggest moving up to a RHEL 6.2+ clone if you do.  5.9 is
> ancient, and there are tons of performance and stability enhancements in
> newer kernels, specifically related to XFS.

-- 
Stan

Re: RAID 10 on Fusion IO cards problems

[Albert Pauw]

Hi Stan,

thanks for your thorough explanation. Since the testing is out of our
scope (it's done by another group) I don't have anymore details then
they will give me,
and yes that's very annoying. But your explanation is very interesting
read, thanks for that.

As for the choice of RHEL 5.9, that was also their choice, not ours.
Also very frustrating, but that's what we have to deal with.

Thanks again, and we'll investigate further (latest claim from them is
that they also have problems on a single device, so I am a bit
clueless what they are actually doing).

Albert

On 30 August 2013 01:15, Stan Hoeppner <stan@hardwarefreak.com> wrote:
> On 8/29/2013 4:20 AM, Albert Pauw wrote:
> ...
>> OS: Oracle Linux 5.9 (effectively RHEL 5.9), kernel  2.6.32-400.29.2.el5uek.
>> All utilities updates, mdadm (2.6.9 latest through updates).
> ...
>> Two Fusion IO Duo cards, each Fusion IO device 640 GB, so four in total.
> ...
>> mdadm --create --verbose /dev/md0 --level=10 --metadata=1.2
>> --chunk=512 --raid-devices=4 /dev/fioa /dev/fioc /dev/fiob /dev/fiod
>> --assume-clean -N md0
>>
>> When the performance turned out bad, after about 20 minutes, the
>> process was stopped. I broke the mirror, so the md0 device is only
>> striped, but the performance hit after 20 minutes happened again.
>>
>> The status of all cards are fine, no problems there. Then I created a
>> fs on only one device and have it run again. This time it worked fine.
>> The fs was in all cases ext3, no TRIM.
>
> You've presented insufficient information to allow a definitive answer.
>  That said, it's very likely that you're hitting the same wall many
> folks do with SSDs.  All md/RAID personalities are limited to a single
> write thread which limits you to one CPU of IO throughput.  When writing
> to a single device without md/RAID, block IOs can be processed by all
> CPUs in parallel.  The Fusion IO device is likely sufficiently fast that
> a single md/RAID10 thread can't saturate the device, so you run out of
> CPU before IOPS.  This is very common with SSD and md/RAID.  Shaohua Li
> has been busily working on patches for quite some time now to eliminate
> this CPU bottleneck in md.
>
> The fact that a single Fusion IO device with EXT3 on it is faster than
> md/RAID10 strongly suggests this may be the cause.  If you have multiple
> application threads or processes writing to a single device the IOs will
> be processed on the same CPU (core) as the thread, so you can have IOs
> in flight from all CPUs in parallel.  When using md/RAID all of that IO
> must be shuttled to the md driver which can only execute on a single CPU
> (core).  To verify this, simply run your tests again and monitor CPU
> burn of the md/RAID10 thread.  If that CPU is 100% at any time then this
> is the problem.
>
> If this is true, you can immediately mitigate it by using a layered
> md/RAID0 over md/RAID1 setup.  Doing this will give you two md/RAID1
> write threads, doubling the number of CPU cores you can put into play.
> To do this and maintain the card<->card mirror layout you described, you
> will create an md/RAID1 with fioa and fioc, and another md/RAID1 with
> fiob and fiod.  Then you'll create an md/RAID0 across these two md/RAID1
> devices.  The md/RAID0 and linear personalities don't use write threads
> and are thus not limited to a single CPU core.
>
> One final suggestion.  Use XFS instead of EXT3/4.  You should get
> significantly better performance with a parallel database workload.  But
> I'd strongly suggest moving up to a RHEL 6.2+ clone if you do.  5.9 is
> ancient, and there are tons of performance and stability enhancements in
> newer kernels, specifically related to XFS.
>
> --
> Stan
>

Re: RAID 10 on Fusion IO cards problems

[Roberto Spadim]

maybe you can just upadte the kernel package... that's enought to get
newers update, i don't know if red hat enterprise have a source of
kernel to build it, but you could try

2013/8/30 Albert Pauw <albert.pauw@gmail.com>:
> Hi Stan,
>
> thanks for your thorough explanation. Since the testing is out of our
> scope (it's done by another group) I don't have anymore details then
> they will give me,
> and yes that's very annoying. But your explanation is very interesting
> read, thanks for that.
>
> As for the choice of RHEL 5.9, that was also their choice, not ours.
> Also very frustrating, but that's what we have to deal with.
>
> Thanks again, and we'll investigate further (latest claim from them is
> that they also have problems on a single device, so I am a bit
> clueless what they are actually doing).
>
> Albert
>
> On 30 August 2013 01:15, Stan Hoeppner <stan@hardwarefreak.com> wrote:
>> On 8/29/2013 4:20 AM, Albert Pauw wrote:
>> ...
>>> OS: Oracle Linux 5.9 (effectively RHEL 5.9), kernel  2.6.32-400.29.2.el5uek.
>>> All utilities updates, mdadm (2.6.9 latest through updates).
>> ...
>>> Two Fusion IO Duo cards, each Fusion IO device 640 GB, so four in total.
>> ...
>>> mdadm --create --verbose /dev/md0 --level=10 --metadata=1.2
>>> --chunk=512 --raid-devices=4 /dev/fioa /dev/fioc /dev/fiob /dev/fiod
>>> --assume-clean -N md0
>>>
>>> When the performance turned out bad, after about 20 minutes, the
>>> process was stopped. I broke the mirror, so the md0 device is only
>>> striped, but the performance hit after 20 minutes happened again.
>>>
>>> The status of all cards are fine, no problems there. Then I created a
>>> fs on only one device and have it run again. This time it worked fine.
>>> The fs was in all cases ext3, no TRIM.
>>
>> You've presented insufficient information to allow a definitive answer.
>>  That said, it's very likely that you're hitting the same wall many
>> folks do with SSDs.  All md/RAID personalities are limited to a single
>> write thread which limits you to one CPU of IO throughput.  When writing
>> to a single device without md/RAID, block IOs can be processed by all
>> CPUs in parallel.  The Fusion IO device is likely sufficiently fast that
>> a single md/RAID10 thread can't saturate the device, so you run out of
>> CPU before IOPS.  This is very common with SSD and md/RAID.  Shaohua Li
>> has been busily working on patches for quite some time now to eliminate
>> this CPU bottleneck in md.
>>
>> The fact that a single Fusion IO device with EXT3 on it is faster than
>> md/RAID10 strongly suggests this may be the cause.  If you have multiple
>> application threads or processes writing to a single device the IOs will
>> be processed on the same CPU (core) as the thread, so you can have IOs
>> in flight from all CPUs in parallel.  When using md/RAID all of that IO
>> must be shuttled to the md driver which can only execute on a single CPU
>> (core).  To verify this, simply run your tests again and monitor CPU
>> burn of the md/RAID10 thread.  If that CPU is 100% at any time then this
>> is the problem.
>>
>> If this is true, you can immediately mitigate it by using a layered
>> md/RAID0 over md/RAID1 setup.  Doing this will give you two md/RAID1
>> write threads, doubling the number of CPU cores you can put into play.
>> To do this and maintain the card<->card mirror layout you described, you
>> will create an md/RAID1 with fioa and fioc, and another md/RAID1 with
>> fiob and fiod.  Then you'll create an md/RAID0 across these two md/RAID1
>> devices.  The md/RAID0 and linear personalities don't use write threads
>> and are thus not limited to a single CPU core.
>>
>> One final suggestion.  Use XFS instead of EXT3/4.  You should get
>> significantly better performance with a parallel database workload.  But
>> I'd strongly suggest moving up to a RHEL 6.2+ clone if you do.  5.9 is
>> ancient, and there are tons of performance and stability enhancements in
>> newer kernels, specifically related to XFS.
>>
>> --
>> Stan
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Roberto Spadim