MD performance options: More CPU’s or more Hz’s?

MD performance options: More CPU’s or more Hz’s?

[mark delfman]

Hi... I am wondering if anyone can offer some advice on MD performance
related to CPU (speed and or cores).  The basic question (probably too
basic) is “for more MD performance are you better with more cpu’s or a
faster single cpu”.  (in an ideal world we would have lots of very
fast CPUs, but given we never have enough money....).

Is there any grounding to the following logic:

Presuming that a RAID0 will deliver 1.5GBsec and a RAID6 circa
700MBsec, I am guessing there are many complex reasons for the
difference, but one of the more obvious being the need for the CPU to
perform all the necessary R6 overheads.

If we look at a single RAID 6 configuration, then I am guessing if we
increase the speed of the CPU from eg 2.0GHz to 2.8GHz (quad core
xeon) then the RAID6 calculations would be faster?  Would other
overheads also be faster and if so is there any know relationship
between CPU Hz and MD performance (maybe even rough rule of thumb eg
double cpu Hz and increase R6 performance by 20% etc)

If however I start to think of multiple RAID6 configurations maybe via
iSCSI etc... then I wonder if MD would be better served with more CPUs
instead... for example 2 x Quad core 2.0GHz xeons instead of 1 x 2.8.
  This theory is dependent on linux / md effectively parallel
processing the overheads and I have no knowledge in this area... hence
the question.

Any thoughts anyone?
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[Sujit K M]

I think this will be deeply involved to the RAID(Hardware) than the CPU/Cores.

For example if you have an RAID0 at 1.5GBsec and an RAID6 At 700MBsec(as
you have said) on Dual Core and Quad Core. I think There will be no performance
gains with RAID0 or RAID6 on the software side. But if you use an RAID of Higher
Capacity it will be much more improved. I think it is very difficult
to think in terms of
RAID alone I think this is much more involved with Architectures like
Fibre Channel,
IP SAN etc.

On Wed, Nov 4, 2009 at 3:19 PM, mark delfman <markdelfman@googlemail.com> wrote:
> Hi... I am wondering if anyone can offer some advice on MD performance
> related to CPU (speed and or cores).  The basic question (probably too
> basic) is “for more MD performance are you better with more cpu’s or a
> faster single cpu”.  (in an ideal world we would have lots of very
> fast CPUs, but given we never have enough money....).
>
> Is there any grounding to the following logic:
>
> Presuming that a RAID0 will deliver 1.5GBsec and a RAID6 circa
> 700MBsec, I am guessing there are many complex reasons for the
> difference, but one of the more obvious being the need for the CPU to
> perform all the necessary R6 overheads.
>
> If we look at a single RAID 6 configuration, then I am guessing if we
> increase the speed of the CPU from eg 2.0GHz to 2.8GHz (quad core
> xeon) then the RAID6 calculations would be faster?  Would other
> overheads also be faster and if so is there any know relationship
> between CPU Hz and MD performance (maybe even rough rule of thumb eg
> double cpu Hz and increase R6 performance by 20% etc)
>
> If however I start to think of multiple RAID6 configurations maybe via
> iSCSI etc... then I wonder if MD would be better served with more CPUs
> instead... for example 2 x Quad core 2.0GHz xeons instead of 1 x 2.8.
>  This theory is dependent on linux / md effectively parallel
> processing the overheads and I have no knowledge in this area... hence
> the question.
>
> Any thoughts anyone?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
-- Sujit K M

blog(http://kmsujit.blogspot.com/)
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[mark delfman]

the performance is measured direct to the MD device and the drives are
connected direct via multi-channel SAS.  There is no intermediate
target layer (no fibre or iscsi etc).  When I mentioned iscsi is was
really talking about possible user configurations, sorry if this was
misleading

As we increase hardware performance the ratios of R0 and R6
performance stay much the same, so I was presuming that the drop in
performance is purely down to software overheads.  I am not saying it
is a bad drop, I am just wondering if we can reduce it by increasing
the software resources


On Wed, Nov 4, 2009 at 12:08 PM, Sujit K M <sjt.kar@gmail.com> wrote:
> I think this will be deeply involved to the RAID(Hardware) than the CPU/Cores.
>
> For example if you have an RAID0 at 1.5GBsec and an RAID6 At 700MBsec(as
> you have said) on Dual Core and Quad Core. I think There will be no performance
> gains with RAID0 or RAID6 on the software side. But if you use an RAID of Higher
> Capacity it will be much more improved. I think it is very difficult
> to think in terms of
> RAID alone I think this is much more involved with Architectures like
> Fibre Channel,
> IP SAN etc.
>
> On Wed, Nov 4, 2009 at 3:19 PM, mark delfman <markdelfman@googlemail.com> wrote:
>> Hi... I am wondering if anyone can offer some advice on MD performance
>> related to CPU (speed and or cores).  The basic question (probably too
>> basic) is “for more MD performance are you better with more cpu’s or a
>> faster single cpu”.  (in an ideal world we would have lots of very
>> fast CPUs, but given we never have enough money....).
>>
>> Is there any grounding to the following logic:
>>
>> Presuming that a RAID0 will deliver 1.5GBsec and a RAID6 circa
>> 700MBsec, I am guessing there are many complex reasons for the
>> difference, but one of the more obvious being the need for the CPU to
>> perform all the necessary R6 overheads.
>>
>> If we look at a single RAID 6 configuration, then I am guessing if we
>> increase the speed of the CPU from eg 2.0GHz to 2.8GHz (quad core
>> xeon) then the RAID6 calculations would be faster?  Would other
>> overheads also be faster and if so is there any know relationship
>> between CPU Hz and MD performance (maybe even rough rule of thumb eg
>> double cpu Hz and increase R6 performance by 20% etc)
>>
>> If however I start to think of multiple RAID6 configurations maybe via
>> iSCSI etc... then I wonder if MD would be better served with more CPUs
>> instead... for example 2 x Quad core 2.0GHz xeons instead of 1 x 2.8.
>>  This theory is dependent on linux / md effectively parallel
>> processing the overheads and I have no knowledge in this area... hence
>> the question.
>>
>> Any thoughts anyone?
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
>
> --
> -- Sujit K M
>
> blog(http://kmsujit.blogspot.com/)
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[Bill Davidsen]

mark delfman wrote:
> Hi... I am wondering if anyone can offer some advice on MD performance
> related to CPU (speed and or cores).  The basic question (probably too
> basic) is “for more MD performance are you better with more cpu’s or a
> faster single cpu”.  (in an ideal world we would have lots of very
> fast CPUs, but given we never have enough money....).
>
> Is there any grounding to the following logic:
>
> Presuming that a RAID0 will deliver 1.5GBsec and a RAID6 circa
> 700MBsec, I am guessing there are many complex reasons for the
> difference, but one of the more obvious being the need for the CPU to
> perform all the necessary R6 overheads.
>
> If we look at a single RAID 6 configuration, then I am guessing if we
> increase the speed of the CPU from eg 2.0GHz to 2.8GHz (quad core
> xeon) then the RAID6 calculations would be faster?  Would other
> overheads also be faster and if so is there any know relationship
> between CPU Hz and MD performance (maybe even rough rule of thumb eg
> double cpu Hz and increase R6 performance by 20% etc)
>
> If however I start to think of multiple RAID6 configurations maybe via
> iSCSI etc... then I wonder if MD would be better served with more CPUs
> instead... for example 2 x Quad core 2.0GHz xeons instead of 1 x 2.8.
>   This theory is dependent on linux / md effectively parallel
> processing the overheads and I have no knowledge in this area... hence
> the question.
>
> Any thoughts anyone?
>   

Your logic is correct, but it implies that you expect "faster 
calculation" to mean "faster write performance," and that is usually 
true only at very low or very high write loads.

Very low, because you get the io queued a few ns faster. Since the disk 
still has to do the write, this is essentially meaningless.

Very high, because with many drives and a huge write volume you could, 
in theory, start having CPU issues.

I suggest that before you worry over much on that, you look at CPU usage 
at idle and then at gradually increasing write load, and look at system 
time vs. GB/sec to see if you are actually getting anywhere near the 
limit, or even up enough to notice. In my look at this a few years ago I 
didn't see any issues, but that was with only eight drives in the array. 
Measurement is always good, but in general drive performance is the 
limiting factor rather than CPU.

You didn't ask: if you use ext[34] filesystems, there is a gain to be 
had from tuning the stripe and stride parameters, at least for large 
sequential io. My measurements were on 2.6.26, so are out of date, but 
less head motion is always better.

Others may have more experience, other than load testing the array has 
never been stressed, performance of backup servers is less important 
than reliability.

-- 
Bill Davidsen <davidsen@tmr.com>
  Unintended results are the well-earned reward for incompetence.


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[mark delfman]

Thank you Bill... maybe a silly question but doesn’t the fact we can
achieve 1.6GBsec on MD RAID0 mean we have drive bandwidth available to
MD to use?  To the block device (sas drives in this case) it does not
know if the data is been xor’d or just striped, its just blocks of
data.  So wouldn’t this mean that we know that MD RAID6 ‘could’
achieve up to 1.6GB if the software could send it out quicker?

Hence my logic of ‘deal with MD software aspects faster = faster writes’

Ok, I know that in many applications we would have to deal with read
modify write but I am not sure in this case that would be true given
we are just kicking out one large stream and that would I suspect
write a complete stripe... (or maybe I could be wrong here and that
may be the limitation?)


Mark


On Wed, Nov 4, 2009 at 2:05 PM, Bill Davidsen <davidsen@tmr.com> wrote:
> mark delfman wrote:
>>
>> Hi... I am wondering if anyone can offer some advice on MD performance
>> related to CPU (speed and or cores).  The basic question (probably too
>> basic) is “for more MD performance are you better with more cpu’s or a
>> faster single cpu”.  (in an ideal world we would have lots of very
>> fast CPUs, but given we never have enough money....).
>>
>> Is there any grounding to the following logic:
>>
>> Presuming that a RAID0 will deliver 1.5GBsec and a RAID6 circa
>> 700MBsec, I am guessing there are many complex reasons for the
>> difference, but one of the more obvious being the need for the CPU to
>> perform all the necessary R6 overheads.
>>
>> If we look at a single RAID 6 configuration, then I am guessing if we
>> increase the speed of the CPU from eg 2.0GHz to 2.8GHz (quad core
>> xeon) then the RAID6 calculations would be faster?  Would other
>> overheads also be faster and if so is there any know relationship
>> between CPU Hz and MD performance (maybe even rough rule of thumb eg
>> double cpu Hz and increase R6 performance by 20% etc)
>>
>> If however I start to think of multiple RAID6 configurations maybe via
>> iSCSI etc... then I wonder if MD would be better served with more CPUs
>> instead... for example 2 x Quad core 2.0GHz xeons instead of 1 x 2.8.
>>  This theory is dependent on linux / md effectively parallel
>> processing the overheads and I have no knowledge in this area... hence
>> the question.
>>
>> Any thoughts anyone?
>>
>
> Your logic is correct, but it implies that you expect "faster calculation"
> to mean "faster write performance," and that is usually true only at very
> low or very high write loads.
>
> Very low, because you get the io queued a few ns faster. Since the disk
> still has to do the write, this is essentially meaningless.
>
> Very high, because with many drives and a huge write volume you could, in
> theory, start having CPU issues.
>
> I suggest that before you worry over much on that, you look at CPU usage at
> idle and then at gradually increasing write load, and look at system time
> vs. GB/sec to see if you are actually getting anywhere near the limit, or
> even up enough to notice. In my look at this a few years ago I didn't see
> any issues, but that was with only eight drives in the array. Measurement is
> always good, but in general drive performance is the limiting factor rather
> than CPU.
>
> You didn't ask: if you use ext[34] filesystems, there is a gain to be had
> from tuning the stripe and stride parameters, at least for large sequential
> io. My measurements were on 2.6.26, so are out of date, but less head motion
> is always better.
>
> Others may have more experience, other than load testing the array has never
> been stressed, performance of backup servers is less important than
> reliability.
>
> --
> Bill Davidsen <davidsen@tmr.com>
>  Unintended results are the well-earned reward for incompetence.
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[John Hughes]

Bill Davidsen wrote:
> mark delfman wrote:
>> Hi... I am wondering if anyone can offer some advice on MD performance
>> related to CPU (speed and or cores).  
>
> You didn't ask: if you use ext[34] filesystems, there is a gain to be 
> had from tuning the stripe and stride parameters, at least for large 
> sequential io. My measurements were on 2.6.26, so are out of date, but 
> less head motion is always better. 
An external journal can be a huge win for ext3 if you are doing lots of 
file creates/deletes.  (Striped raid, i.e. raid0/raid5/raid6/raid10 
doesn't help the journal much as it is used sequentially.  In my tests 
putting the journal on a raid1 and the rest of the fs on a raid10 makes 
things go much faster.  If you have enough disks of course :-))

Re: MD performance options: More CPU’s or more Hz’s?

[Goswin von Brederlow]

mark delfman <markdelfman@googlemail.com> writes:

> Hi... I am wondering if anyone can offer some advice on MD performance
> related to CPU (speed and or cores).  The basic question (probably too
> basic) is “for more MD performance are you better with more cpu’s or a
> faster single cpu”.  (in an ideal world we would have lots of very
> fast CPUs, but given we never have enough money....).

In older kernels the raid6 code was single threaded. There a single
faster cpu would give you more speed as all other cores would remain
totaly idle.

But the raid code has been moving to use multiple cores and hardware
acceleration when available for a while now. And since raid operations
can be distributed among cores nicely (e.g. each core gets one stripe)
more cores will be better than a single faster core. The combined
speed of many cores will usualy be more than a faster single core has
at a smaller price.

> Is there any grounding to the following logic:
>
> Presuming that a RAID0 will deliver 1.5GBsec and a RAID6 circa
> 700MBsec, I am guessing there are many complex reasons for the
> difference, but one of the more obvious being the need for the CPU to
> perform all the necessary R6 overheads.

Back with ~2.6.18 I could only get 180MB/s out of a raid6 because the
cpu would hit 100% used then. Splitting the raid6 into 3 raid6 (same
config, 1/3 the size each) made it use 3 cores at trice the overall
speed. But the raid driver has improved since then.

> If we look at a single RAID 6 configuration, then I am guessing if we
> increase the speed of the CPU from eg 2.0GHz to 2.8GHz (quad core
> xeon) then the RAID6 calculations would be faster?  Would other
> overheads also be faster and if so is there any know relationship
> between CPU Hz and MD performance (maybe even rough rule of thumb eg
> double cpu Hz and increase R6 performance by 20% etc)

Unless you hit other bottlenecks double the cpu Hz shoud give double
R6 performance. But I doubt you ever get that because you always hit
other bottlenecks. For example if the 2.0GHz runs R6 at 100% of the
ram speed then 2.8GHz will give 0% improvement.

> If however I start to think of multiple RAID6 configurations maybe via
> iSCSI etc... then I wonder if MD would be better served with more CPUs
> instead... for example 2 x Quad core 2.0GHz xeons instead of 1 x 2.8.
>   This theory is dependent on linux / md effectively parallel
> processing the overheads and I have no knowledge in this area... hence
> the question.
>
> Any thoughts anyone?

Consider all factors, esspecially ram speed and bus/controler speed.
But generally the sum of all cores is more relevant than each core for
throughput.

MfG
        Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[jim owens]

mark delfman wrote:
> Thank you Bill... maybe a silly question but doesn’t the fact we can
> achieve 1.6GBsec on MD RAID0 mean we have drive bandwidth available to
> MD to use?  To the block device (sas drives in this case) it does not
> know if the data is been xor’d or just striped, its just blocks of
> data.  So wouldn’t this mean that we know that MD RAID6 ‘could’
> achieve up to 1.6GB if the software could send it out quicker?
> 
> Hence my logic of ‘deal with MD software aspects faster = faster writes’

Are your tests and assumptions valid... remember that raid6
will put 1.6GB of data on the drives for each 800MB of data
from the application in minimum 4 stripe mode (2 data stripes,
1 P stripe, 1 Q stripe).

So the proper comparison of raid6 software overhead is to
run the same application writing a fixed large amount of data
to 4 drives at raid0 and the same 4 drives at raid6.

   overhead = raid0_time * 2 - raid6_time;

Of course if you add more data stripes, the ratio of
user data to disk data gets better than 1/2.

jim
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[jim owens]

jim owens wrote:
> mark delfman wrote:
>> Thank you Bill... maybe a silly question but doesn’t the fact we can
>> achieve 1.6GBsec on MD RAID0 mean we have drive bandwidth available to
>> MD to use?  To the block device (sas drives in this case) it does not
>> know if the data is been xor’d or just striped, its just blocks of
>> data.  So wouldn’t this mean that we know that MD RAID6 ‘could’
>> achieve up to 1.6GB if the software could send it out quicker?
>>
>> Hence my logic of ‘deal with MD software aspects faster = faster writes’
> 
> Are your tests and assumptions valid... remember that raid6
> will put 1.6GB of data on the drives for each 800MB of data
> from the application in minimum 4 stripe mode (2 data stripes,
> 1 P stripe, 1 Q stripe).
> 
> So the proper comparison of raid6 software overhead is to
> run the same application writing a fixed large amount of data
> to 4 drives at raid0 and the same 4 drives at raid6.
> 
>   overhead = raid0_time * 2 - raid6_time;

UGH... wrote that backwards:

overhead = raid6_time - raid0_time * 2;

> Of course if you add more data stripes, the ratio of
> user data to disk data gets better than 1/2.
> 
> jim
> 
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[jim owens]

jim owens wrote:
>> So the proper comparison of raid6 software overhead is to
>> run the same application writing a fixed large amount of data
>> to 4 drives at raid0 and the same 4 drives at raid6.

Actually, it is probably better to run the application with

   2 X data to raid0 and 1 X data to raid6

so the amount of written disk data is the same.

That way the comparison will remove the bus/disk difference
caused by sending different amounts of data and:

overhead = raid6_time - raid0_time;

jim

Re: MD performance options: More CPU’s or more Hz’s?

[mark delfman]

Hi Jim,

I think this is a great point... i had not thought of the extra two
chunks of data being written... BUT, not sure if in this case it is
the limiter as we are using 12 drives.




the hardware does bottleneck at around 1.6GBs for writes (reaches this
with 8 / 9 drives).




On Wed, Nov 4, 2009 at 3:56 PM, jim owens <jowens@hp.com> wrote:
> jim owens wrote:
>>>
>>> So the proper comparison of raid6 software overhead is to
>>> run the same application writing a fixed large amount of data
>>> to 4 drives at raid0 and the same 4 drives at raid6.
>
> Actually, it is probably better to run the application with
>
>  2 X data to raid0 and 1 X data to raid6
>
> so the amount of written disk data is the same.
>
> That way the comparison will remove the bus/disk difference
> caused by sending different amounts of data and:
>
> overhead = raid6_time - raid0_time;
>
> jim
>
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD performance options: More CPU’s or more Hz’s?

[jim owens]

mark delfman wrote:
> I think this is a great point... i had not thought of the extra two
> chunks of data being written... BUT, not sure if in this case it is
> the limiter as we are using 12 drives.

Disclaimer... I'm a filesystem guy not a raid guy, so someone
who is may say I'm completely wrong.

IMO 12 drives actually makes raid6 performance much worst.

Think it through, raid0 writes are substripe granularity,
raid6 must either write all 12 (10 data, 2 check) stripes
at once or if you write 1 stripe, read 9 data stripes to
build and write the 2 check stripes.

The problem is even if you have a good application sending
writes in the 10-stripe-length-multiples of the set, the
kernel layers may chunk it and deliver it to md in smaller
random sizes.

Unless it is a single stream writing and md buffers the
whole stripe set, writes will cause md reads.

And you will never have a single stream from a filesystem
because metadata will be updated at some point.  You can
minimize that by only doing overwrites.  Allocating writes
are terrible in all filesystems because a lot of metadata
has to be modified.  Metadata writes are also a performance
killer because they are small (usually under 1 stripe) and
always cause seeks.

> the hardware does bottleneck at around 1.6GBs for writes (reaches this
> with 8 / 9 drives).

So compare at 8 drives using raw writes of 6-stripe-lengths
where raid0 uses a 4 transfers for each 3 raid6 transfers.

jim

Re: MD performance options: More CPUs or more Hzs?

[Goswin von Brederlow]

jim owens <jowens@hp.com> writes:

> mark delfman wrote:
>> I think this is a great point... i had not thought of the extra two
>> chunks of data being written... BUT, not sure if in this case it is
>> the limiter as we are using 12 drives.
>
> Disclaimer... I'm a filesystem guy not a raid guy, so someone
> who is may say I'm completely wrong.
>
> IMO 12 drives actually makes raid6 performance much worst.

Yes, but not for the reason you say for this test.

> Think it through, raid0 writes are substripe granularity,
> raid6 must either write all 12 (10 data, 2 check) stripes
> at once or if you write 1 stripe, read 9 data stripes to
> build and write the 2 check stripes.

I think theoretically you could update the p and q chunks but afaik
linux doesn't support that and always recomputes.

> The problem is even if you have a good application sending
> writes in the 10-stripe-length-multiples of the set, the
> kernel layers may chunk it and deliver it to md in smaller
> random sizes.

That should just fill the stripe cache till the stripe is complete or
the cache is full.

> Unless it is a single stream writing and md buffers the
> whole stripe set, writes will cause md reads.

Which I think he is testing exactly.

> And you will never have a single stream from a filesystem
> because metadata will be updated at some point.  You can
> minimize that by only doing overwrites.  Allocating writes
> are terrible in all filesystems because a lot of metadata
> has to be modified.  Metadata writes are also a performance
> killer because they are small (usually under 1 stripe) and
> always cause seeks.

Again the stripe cache should mitigate that. And ext4 (with external
journal?) should not write metadata so often anyway.

What about btrfs? The COW semantic could really improve thigs as it
will not overwrite a block from an old stripe but lineary fill new
stripes.

>> the hardware does bottleneck at around 1.6GBs for writes (reaches this
>> with 8 / 9 drives).

That would indicate that you reached the limit of your controler or
bus(es).

Lets assume we can transfere 1.6GB/s of data to the 12 drives (as
raid0 shows we can). So each drive can get 136MB/s. In a 12 disk raid6
there are 10 data chunks and 2 parity so ideal performance would be
1.36GB/s. Certainly more than the 700MB/s measured. So do look at top
and see where the cpu usage lies.

> So compare at 8 drives using raw writes of 6-stripe-lengths
> where raid0 uses a 4 transfers for each 3 raid6 transfers.

Well, do compare raid6 with 4,5,6,7,8,9,10,11 and 12 disks. The more
disks you have the more expensive the parity calculation becomes. Also
consider running 2 6 disk raid5. Not quite as secure but if speed is
more important...

MfG
        Goswin

Re: MD performance options: More CPU’s or more Hz’s?

[Bill Davidsen]

John Hughes wrote:
> Bill Davidsen wrote:
>> mark delfman wrote:
>>> Hi... I am wondering if anyone can offer some advice on MD performance
>>> related to CPU (speed and or cores).  
>>
>> You didn't ask: if you use ext[34] filesystems, there is a gain to be 
>> had from tuning the stripe and stride parameters, at least for large 
>> sequential io. My measurements were on 2.6.26, so are out of date, 
>> but less head motion is always better. 
> An external journal can be a huge win for ext3 if you are doing lots 
> of file creates/deletes.  (Striped raid, i.e. raid0/raid5/raid6/raid10 
> doesn't help the journal much as it is used sequentially.  In my tests 
> putting the journal on a raid1 and the rest of the fs on a raid10 
> makes things go much faster.  If you have enough disks of course :-))

You might try mounting with data=journal and see what performance you 
get with small reads and writes. Might surprise you. I hadn't thought of 
using raid1 (or raid0 if speed is more important than survival), but I 
have been playing with using SSD just for the journal. As you say, when 
you do lots of creates or deletes it makes a big difference.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein

Re: MD performance options: More CPU’s or more Hz’s?

[John Hughes]

Bill Davidsen wrote:
> John Hughes wrote:
>> An external journal can be a huge win for ext3 if you are doing lots 
>> of file creates/deletes.  (Striped raid, i.e. 
>> raid0/raid5/raid6/raid10 doesn't help the journal much as it is used 
>> sequentially.  In my tests putting the journal on a raid1 and the 
>> rest of the fs on a raid10 makes things go much faster.  If you have 
>> enough disks of course :-))
>
> You might try mounting with data=journal and see what performance you 
> get with small reads and writes. Might surprise you. I hadn't thought 
> of using raid1 (or raid0 if speed is more important than survival), 
> but I have been playing with using SSD just for the journal. As you 
> say, when you do lots of creates or deletes it makes a big difference.
Yup, data=journal will be in the next tests I do.