Re: [Gelato-technical] Serious performance degradation on a RAID with

Re: [Gelato-technical] Serious performance degradation on a RAID with

[David Mosberger]

>>>>> On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius <Andreas.Hirstius@cern.ch> said:

  Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers
  Andreas> and 24x WD740GD HDD in a software RAID0 configuration
  Andreas> (using md).  With kernel 2.6.11 the read performance on the
  Andreas> md is reduced by a factor of 20 (!!)  compared to previous
  Andreas> kernels.  The write rate to the md doesn't change!! (it
  Andreas> actually improves a bit).

Is there any reason to believe this change in behavior is
ia64-specific?  I doubt it.  The q-syscollect profiles on your
web-site shows that the CPU is basically completely idle.  You may
want to try 2.6.10 to see whether the bad behavior was introduced
before or after that.  Unfortunately, 2.6 is developing very rapidly
so you may have to do more binary searching between 2.6.9 and 2.6.10
or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity.
You may also want to look at the bitkeeper changelogs to see if you
can find anything suspicious (you can do that easily via the web by
browsing the source code [1]).  Lastly, you may want to ask on
linux-kernel whether anybody can think of a change that could have
this effect.

	--david

[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html

Re: [Gelato-technical] Serious performance degradation on a RAID

[Rob Fowler]

I've recently had discussions with several vendors who have mentioned
similar magnitudes of disk performance degradation due to the
coupling of vibration between cooling fans and disks.  This can 
dramatically increase seek time by keeping the arm from settling.
In one case, upgrading a chassis fan caused disk throughput to go down
by a factor of 16.  The solution is careful attention to vibration
damping in mountings.  Vibration caused by fan bearing wear could have
a similar effect.

Can you go back to your old kernel to verify that it is still
giving you the same performance?

-- Rob Fowler


David Mosberger wrote:
>>>>>>On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius <Andreas.Hirstius@cern.ch> said:
> 
> 
>   Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers
>   Andreas> and 24x WD740GD HDD in a software RAID0 configuration
>   Andreas> (using md).  With kernel 2.6.11 the read performance on the
>   Andreas> md is reduced by a factor of 20 (!!)  compared to previous
>   Andreas> kernels.  The write rate to the md doesn't change!! (it
>   Andreas> actually improves a bit).
> 
> Is there any reason to believe this change in behavior is
> ia64-specific?  I doubt it.  The q-syscollect profiles on your
> web-site shows that the CPU is basically completely idle.  You may
> want to try 2.6.10 to see whether the bad behavior was introduced
> before or after that.  Unfortunately, 2.6 is developing very rapidly
> so you may have to do more binary searching between 2.6.9 and 2.6.10
> or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity.
> You may also want to look at the bitkeeper changelogs to see if you
> can find anything suspicious (you can do that easily via the web by
> browsing the source code [1]).  Lastly, you may want to ask on
> linux-kernel whether anybody can think of a change that could have
> this effect.
> 
> 	--david
> 
> [1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html
> 
> _______________________________________________
> Gelato-technical mailing list
> Gelato-technical@gelato.unsw.edu.au
> https://www.gelato.unsw.edu.au/mailman/listinfo/gelato-technical

Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11

[Duraid Madina]

On Tue, Apr 05, 2005 at 01:25:40PM -0700, David Mosberger wrote:
> >>>>> On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius <Andreas.Hirstius@cern.ch> said:
> 
> Is there any reason to believe this change in behavior is
> ia64-specific?  I doubt it.

Probably not ia64-specifc again, but I wonder what you make of
http://bugme.osdl.org/show_bug.cgi?idB68 . The LSI MPT scsi hosts
are _particularly_ prevalent in the ia64 world.

	Duraid

Re: [Gelato-technical] Serious performance degradation on a RAID with

[David Mosberger]

>>>>> On Wed, 6 Apr 2005 11:13:30 +0900, Duraid Madina <duraid@kinoko.c.u-tokyo.ac.jp> said:

  Duraid> On Tue, Apr 05, 2005 at 01:25:40PM -0700, David Mosberger
  Duraid> wrote:
  >> >>>>> On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius
  >> <Andreas.Hirstius@cern.ch> said:

  >> Is there any reason to believe this change in behavior is
  >> ia64-specific?  I doubt it.

  Duraid> Probably not ia64-specifc again, but I wonder what you make
  Duraid> of http://bugme.osdl.org/show_bug.cgi?idB68 . The LSI MPT
  Duraid> scsi hosts are _particularly_ prevalent in the ia64 world.

Did you compile the driver as a module?  If so, this should be of
interest:

 http://linux.bkbits.net:8080/linux-2.5/cset@4241a3b18VBJETehoTH3y5vcZilSwA

	--david

Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11

[Duraid Madina]

On Tue, Apr 05, 2005 at 09:45:34PM -0700, David Mosberger wrote:
> >>>>> On Wed, 6 Apr 2005 11:13:30 +0900, Duraid Madina <duraid@kinoko.c.u-tokyo.ac.jp> said:
> 
> Did you compile the driver as a module?

Nope. You're not seeing the problem?

	Duraid

Re: [Gelato-technical] Serious performance degradation on a RAID with kernel 2.6.11

[David Mosberger]

>>>>> On Wed, 6 Apr 2005 13:53:23 +0900, Duraid Madina <duraid@kinoko.c.u-tokyo.ac.jp> said:

  Duraid> On Tue, Apr 05, 2005 at 09:45:34PM -0700, David Mosberger
  Duraid> wrote:
  >> >>>>> On Wed, 6 Apr 2005 11:13:30 +0900, Duraid Madina
  >> <duraid@kinoko.c.u-tokyo.ac.jp> said:

  >> Did you compile the driver as a module?

  Duraid> Nope. You're not seeing the problem?

I see it with 2.6.11 (38MB/s). 2.6.12-rc2 is definitely better
(53MB/s) but IIRC we used to be closer to 60MB/s, so perhaps something
still isn't quite optimal.

	--david

RE: [Gelato-technical] Serious performance degradation on a RAID withkernel 2.6.11

[Chen, Kenneth W]

On Tue, Apr 05, 2005 at 09:45:34PM -0700, David Mosberger wrote:
>
> Did you compile the driver as a module?


Duraid Madina wrote on Tuesday, April 05, 2005 9:53 PM
> Nope. You're not seeing the problem?

That is the reason for the performance problem.  And this is definitely
*not* ia64 specific.  See this discussion which started with performance
problem seen on opteron and reproduced on DELL machine as well.
http://marc.theaimsgroup.com/?t\x111028927300002&r=1&w=2

The changeset that david pointed out is the right fix.  Or you can just
take the small patch I posted in the above thread.

- Ken

Re: [Gelato-technical] Serious performance degradation on a RAID

[Andreas Hirstius]

David Mosberger wrote:

>>>>>>On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius <Andreas.Hirstius@cern.ch> said:
>>>>>>            
>>>>>>
>
>  Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers
>  Andreas> and 24x WD740GD HDD in a software RAID0 configuration
>  Andreas> (using md).  With kernel 2.6.11 the read performance on the
>  Andreas> md is reduced by a factor of 20 (!!)  compared to previous
>  Andreas> kernels.  The write rate to the md doesn't change!! (it
>  Andreas> actually improves a bit).
>
>Is there any reason to believe this change in behavior is
>ia64-specific?  I doubt it.  The q-syscollect profiles on your
>web-site shows that the CPU is basically completely idle.  You may
>want to try 2.6.10 to see whether the bad behavior was introduced
>before or after that.  Unfortunately, 2.6 is developing very rapidly
>so you may have to do more binary searching between 2.6.9 and 2.6.10
>or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity.
>You may also want to look at the bitkeeper changelogs to see if you
>can find anything suspicious (you can do that easily via the web by
>browsing the source code [1]).  Lastly, you may want to ask on
>linux-kernel whether anybody can think of a change that could have
>this effect.
>
>  
>
The problems started with 2.6.10-bk7...
(and I've tested 2.6.9, 2.6.10, 2.6.10-bk5 - -bk14,  2.6.11-rc[1-5], 
2.6.11, 2.6.11-mm[12], 2.6.12-rc[12])

The only other platform I have is a dual-xeon and I don't see the 
problem there (same controllers/disks)...

Because of this strange correlation between block size and page size and 
the fact, that I don't see a problem on x86 I suspected an ia64-specific 
problem. My "favourite" candidate is the conversion to 4 level 
pagetables between 2.6.10-bk6 and 2.6.10-bk7.


Andreas


>	--david
>
>[1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html
>-
>To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>  
>

Re: [Gelato-technical] Serious performance degradation on a RAID

[Andreas Hirstius]

Rob Fowler wrote:

> I've recently had discussions with several vendors who have mentioned
> similar magnitudes of disk performance degradation due to the
> coupling of vibration between cooling fans and disks.  This can 
> dramatically increase seek time by keeping the arm from settling.
> In one case, upgrading a chassis fan caused disk throughput to go down
> by a factor of 16.  The solution is careful attention to vibration
> damping in mountings.  Vibration caused by fan bearing wear could have
> a similar effect.
>
We've also seen similiar problems before... So I made sure it's not 
something "mechanical" ;-)

> Can you go back to your old kernel to verify that it is still
> giving you the same performance?
>
I'm constantly going back and forth between "working" and "not working" 
kernels.
The performance with a given kernel doesn't change...


Andreas

> -- Rob Fowler
>
>
> David Mosberger wrote:
>
>>>>>>> On Tue, 05 Apr 2005 20:11:51 +0200, Andreas Hirstius 
>>>>>>> <Andreas.Hirstius@cern.ch> said:
>>>>>>
>>
>>
>>   Andreas> Hi, We have a rx4640 with 3x 3Ware 9500 SATA controllers
>>   Andreas> and 24x WD740GD HDD in a software RAID0 configuration
>>   Andreas> (using md).  With kernel 2.6.11 the read performance on the
>>   Andreas> md is reduced by a factor of 20 (!!)  compared to previous
>>   Andreas> kernels.  The write rate to the md doesn't change!! (it
>>   Andreas> actually improves a bit).
>>
>> Is there any reason to believe this change in behavior is
>> ia64-specific?  I doubt it.  The q-syscollect profiles on your
>> web-site shows that the CPU is basically completely idle.  You may
>> want to try 2.6.10 to see whether the bad behavior was introduced
>> before or after that.  Unfortunately, 2.6 is developing very rapidly
>> so you may have to do more binary searching between 2.6.9 and 2.6.10
>> or 2.6.10 and 2.6.11 to pin the problem down to a useful granularity.
>> You may also want to look at the bitkeeper changelogs to see if you
>> can find anything suspicious (you can do that easily via the web by
>> browsing the source code [1]).  Lastly, you may want to ask on
>> linux-kernel whether anybody can think of a change that could have
>> this effect.
>>
>>     --david
>>
>> [1] http://linux.bkbits.net:8080/linux-2.5/src?nav=index.html
>>
>> _______________________________________________
>> Gelato-technical mailing list
>> Gelato-technical@gelato.unsw.edu.au
>> https://www.gelato.unsw.edu.au/mailman/listinfo/gelato-technical
>
>

Re: [Gelato-technical] Serious performance degradation on a RAID

[David Mosberger]

>>>>> On Wed, 06 Apr 2005 09:29:59 +0200, Andreas Hirstius <Andreas.Hirstius@cern.ch> said:

  Andreas> Because of this strange correlation between block size and
  Andreas> page size and the fact, that I don't see a problem on x86 I
  Andreas> suspected an ia64-specific problem. My "favourite"
  Andreas> candidate is the conversion to 4 level pagetables between
  Andreas> 2.6.10-bk6 and 2.6.10-bk7.

That's certainly a possibility.  But didn't you say the problem also
occurred when using 4KB page size on ia64?

	--david

Re: [Gelato-technical] Serious performance degradation on a RAID

[Andreas Hirstius]

David Mosberger wrote:

>>>>>>On Wed, 06 Apr 2005 09:29:59 +0200, Andreas Hirstius <Andreas.Hirstius@cern.ch> said:
>>>>>>            
>>>>>>
>
>  Andreas> Because of this strange correlation between block size and
>  Andreas> page size and the fact, that I don't see a problem on x86 I
>  Andreas> suspected an ia64-specific problem. My "favourite"
>  Andreas> candidate is the conversion to 4 level pagetables between
>  Andreas> 2.6.10-bk6 and 2.6.10-bk7.
>
>That's certainly a possibility.  But didn't you say the problem also
>occurred when using 4KB page size on ia64?
>
>  
>
Yes, the problem appears at any page size...


Andreas

>	--david
>
>  
>

Re: [Gelato-technical] Serious performance degradation on a RAID

[Andreas Hirstius]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1252", Size: 1909 bytes --]



Chen, Kenneth W wrote:

>On Tue, Apr 05, 2005 at 09:45:34PM -0700, David Mosberger wrote:
>  
>
>>Did you compile the driver as a module?
>>    
>>
>
>
>Duraid Madina wrote on Tuesday, April 05, 2005 9:53 PM
>  
>
>>Nope. You're not seeing the problem?
>>    
>>
>
>That is the reason for the performance problem.  And this is definitely
>*not* ia64 specific.  See this discussion which started with performance
>problem seen on opteron and reproduced on DELL machine as well.
>http://marc.theaimsgroup.com/?t\x111028927300002&r=1&w=2
>
>The changeset that david pointed out is the right fix.  Or you can just
>take the small patch I posted in the above thread.
>
>  
>

The problems mentioned in this discussion about the Fusion-MPT I can 
perfectly reproduce on my boxes with 2.6.11, but only with pages sizes 
4k and 8k;
With 16k and 64k pages I don't see any problem at all !!!
Actually I see block size <-> page size correlations which are similiar 
to the correlations I see in my setup...
i.e. 2.6.11 and 4k pages:
    dd ...bs@96: 40MB/s
    dd ...bs92: 74MB/s

i.e. 2.6.11 and 8k pages:
    dd ...bs92: 56MB/s
    dd ...bs\x16384: 74MB/s
   
with the driver as module I don't see any change (??).
And I also don't see any change in performance for 4k/8k pages, when 
applying either your small patch or the cset david pointed to...
Since the fusion-mpt driver in 2.6.12-rc2 has this cset applied and no 
other changes, I also don't see improvements with this kernel...
(Again with 16k or 64k pages no performance problem in 2.6.12-rc2 with 
the fusion-mpt driver...)

But the driver in questions is 3w-9xxx and there have been no chances at 
all in this driver between 2.6.10 and 2.6.12-rc1. The changed driver in  
-rc2 still has the problem.
And I've tried the driver built-in and as a module (default)...

Andreas

>- Ken
>
>
>
>  
>

Re: [Gelato-technical] Serious performance degradation on a RAID

[David Mosberger]

>>>>> On Wed, 06 Apr 2005 10:33:07 +0200, Andreas Hirstius <Andreas.Hirstius@cern.ch> said:

  Andreas> But the driver in questions is 3w-9xxx and there have been
  Andreas> no chances at all in this driver between 2.6.10 and
  Andreas> 2.6.12-rc1. The changed driver in -rc2 still has the
  Andreas> problem.  And I've tried the driver built-in and as a
  Andreas> module (default)...

Are you sure the problem is due to the driver and not RAID0?  What if
you dd directly from the underlying device?

	--david

Re: [Gelato-technical] Serious performance degradation on a RAID

[Andreas Hirstius]

David Mosberger wrote:

>>>>>>On Wed, 06 Apr 2005 10:33:07 +0200, Andreas Hirstius <Andreas.Hirstius@cern.ch> said:
>>>>>>            
>>>>>>
>
>  Andreas> But the driver in questions is 3w-9xxx and there have been
>  Andreas> no chances at all in this driver between 2.6.10 and
>  Andreas> 2.6.12-rc1. The changed driver in -rc2 still has the
>  Andreas> problem.  And I've tried the driver built-in and as a
>  Andreas> module (default)...
>
>Are you sure the problem is due to the driver and not RAID0?  What if
>you dd directly from the underlying device?
>
>  
>
To make sure it's not the md/raid I just took the driver/md directory 
from 2.6.9 and 2.6.10 and used it in 2.6.11 => no change in behaviour ...
(The changes in md.c are minor and there are no changes in raid0.c 
between .10 and .11 anyway)

As I say on the webpage, a dd on the underlying devices works fine...

It might still be a higher level interface to the md, that's wrong.

And I'm not saying that it's the 3w-9xx driver!! It hasn't changed 
between 2.6.10 and 2.6.11, so pretty much impossible to introduce bugs 
there...
(It was just mentioned wrt to the fusion-mpt driver.)

Andreas

>	--david
>
>  
>

RE: [Gelato-technical] Serious performance degradation on a RAID withkernel 2.6.11

[Chen, Kenneth W]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="windows-1252", Size: 1006 bytes --]

Andreas Hirstius wrote on Wednesday, April 06, 2005 1:33 AM
> The problems mentioned in this discussion about the Fusion-MPT I can
> perfectly reproduce on my boxes with 2.6.11, but only with pages sizes
> 4k and 8k;
> With 16k and 64k pages I don't see any problem at all !!!
> Actually I see block size <-> page size correlations which are similiar
> to the correlations I see in my setup...
> i.e. 2.6.11 and 4k pages:
>     dd ...bs@96: 40MB/s
>     dd ...bs92: 74MB/s
>
> i.e. 2.6.11 and 8k pages:
>     dd ...bs92: 56MB/s
>     dd ...bs\x16384: 74MB/s

It is very odd.  I think it has something to do with generic file read
ahead logic.  The read ahead window did not push far enough to keep the
disk busy.  Some of the size calculation are in unit of page, so smaller
page size will have smaller effective read ahead size.  Try look into
mm/readahead.c.  I bet you can find the answer to your performance problem
there.  If not, then I guess we have to keep on looking ;-)