MD write performance issue

MD write performance issue

[mark delfman]

After further work we are sure that there is a significant write
performance issue with either the Kernel+MD or with
Me_Being_Stupid+MD.   I would be happy to find out it is the latter,
but given the possibility it may be Kernel+MD I thought I should raise
this again under a new thread with below info.

On previous threads I stated that on later kernels XFS > MD dropped up
to 50% performance... I now have tested several filesystems (it is not
XFS related) and several Kernel’s (it is related to Kernel)

Below is the basic summary of the various Kernel write performance
results... all on the same hardware and Open SUSE foundation.
1.1GBsec is the max this particular hardware will provide and in a
single stream XFS would normally match this.  You can see that
depending on Kernel the write performance can drop to 50%.....

(note: we have tried this with a hardware raid and there is no
performance drop between direct or via a FS.)

I really am stuck as to where to go from here, I would like to use .30
kernels and I am sure that there is a solution to this but do not have
enough linux experience to think of any further steps, so very much
appreciate any advice.



Linux linux-tlfp 2.6.27.7-vanilla #1 SMP Thu Oct 15 19:52:43 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAW: 1.1 GB/s
XFS: 1.1 GB/s
(tests with simple DD’s, but I have also  tried FIO etc)

2.6.30.8

RAW: 1.1 GB/s
XFS: 900 MB/s

Linux linux-tlfp 2.6.28.4-vanilla #1 SMP Thu Oct 15 21:36:51 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAW: 1.1 GB/s
XFS: 1.1 GB/s



Linux linux-tlfp 2.6.31.4-vanilla #1 SMP Thu Oct 15 22:11:16 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAW 1,1 GB/s
XFS: 920 MB/s



Linux linux-tlfp 2.6.28.10-vanilla #1 SMP Thu Oct 15 22:44:26 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAW: 1.1
XFS: 684 MB/s



Linux linux-tlfp 2.6.27.37-vanilla #1 SMP Thu Oct 15 23:32:53 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAW: 1.1 GB/s
XFS 520 MB/s


Linux linux-tlfp 2.6.27.20-vanilla #1 SMP Thu Oct 15 23:59:32 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAw 1.1 GB/s
XFS: 487 MB/s



Linux linux-tlfp 2.6.27.14-vanilla #1 SMP Fri Oct 16 00:56:25 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAW: 1.1
XFS 1.1
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: MD write performance issue

[Asdo]

mark delfman wrote:
> After further work we are sure that there is a significant write
> performance issue with either the Kernel+MD or...

Hm!
Pretty strange repeated ups and downs of the speed with increasing 
kernel versions.

Have you checked:
that compile options are the same (preferably by taking 2.6.31 compile 
options and porting them down)
disk schedulers are the same
the test was long enough to level jitters, like 2-3 minutes
Also: looking at "iostat -x 1" during the transfer could show something...

Apart from this, I confirm I noticed in my 2.6.31-rc? earlier tests, 
that performances on xfs writes were very inconsistent :
These were my benchmarks (I wrote them on file at that time):

Stripe_cache_size was 1024, 13 devices raid-5:

bs=1M -> 206MB/s
bs=256K -> 229MB/s

retrying soon after, identical settings:

bs=1M -> 129MB/s
bs=256K -> 140MB/s


Transfer speed was hence very unreliable, depending on something that is 
not clearly user visible... maybe dirty page cache? I thought that 
depending on the exact amount of data being pushed out by the pdflush at 
the first round, that would cause a sequence of read-modify-write stuff 
which would cause further read-modify-write and further instability 
later on. But I was doing that with raid-5 while you Mark are using 
raid-0 right? My theory doesn't hold on raid-0.

Re: MD write performance issue

[mark delfman]

Quick update on .30.x kernels... which are still showing MD reduced performance:


Linux linux-tlfp 2.6.30-vanilla #1 SMP Fri Oct 16 14:22:54 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAW: 1.1
XFS 870 MB/s



Linux linux-tlfp 2.6.31.3-vanilla #1 SMP Fri Oct 16 14:52:09 BST 2009
x86_64 x86_64 x86_64 GNU/Linux

RAW: 1.1
XFS 920 MB/s


linux-tlfp:/ # uname -a
Linux linux-tlfp 2.6.31.2-vanilla #1 SMP Fri Oct 16 15:44:44 BST 2009
x86_64 x86_64 x86_64 GNU/Linux


RAW: 1.1
XFS: 935 MB/s


On Fri, Oct 16, 2009 at 11:42 AM, Asdo <asdo@shiftmail.org> wrote:
> mark delfman wrote:
>>
>> After further work we are sure that there is a significant write
>> performance issue with either the Kernel+MD or...
>
> Hm!
> Pretty strange repeated ups and downs of the speed with increasing kernel
> versions.
>
> Have you checked:
> that compile options are the same (preferably by taking 2.6.31 compile
> options and porting them down)
> disk schedulers are the same
> the test was long enough to level jitters, like 2-3 minutes
> Also: looking at "iostat -x 1" during the transfer could show something...
>
> Apart from this, I confirm I noticed in my 2.6.31-rc? earlier tests, that
> performances on xfs writes were very inconsistent :
> These were my benchmarks (I wrote them on file at that time):
>
> Stripe_cache_size was 1024, 13 devices raid-5:
>
> bs=1M -> 206MB/s
> bs=256K -> 229MB/s
>
> retrying soon after, identical settings:
>
> bs=1M -> 129MB/s
> bs=256K -> 140MB/s
>
>
> Transfer speed was hence very unreliable, depending on something that is not
> clearly user visible... maybe dirty page cache? I thought that depending on
> the exact amount of data being pushed out by the pdflush at the first round,
> that would cause a sequence of read-modify-write stuff which would cause
> further read-modify-write and further instability later on. But I was doing
> that with raid-5 while you Mark are using raid-0 right? My theory doesn't
> hold on raid-0.
>
>
>