How to improve performance of 2.6 kernel

How to improve performance of 2.6 kernel

[kernel coder]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

Hi,

I have just ported linux kernel 2.6.10 for MIPS4Kc-core board.Before
porting 2.6 kernel ,2.4.20 was running on this board. When I took
benchmarks for both the kernels for comparison, I found out that
linux-2.4.20 was giving much better results than linux-2.6.10. The
specs for the board are as follows:

--------------------------------------------------------------
133MHz MIPS4kc
32MB 133Mhz SDRAM
Cache Size: 16KB
Timer interrupt is set to 100 interrupts/sec
The hard disk attached to the board had ultraDMA 5 capability
--------------------------------------------------------------

I am also attaching the benchmarks I took for both the kernels. Is
there any way I can improve 2.6's performance? Thanks.

[-- Attachment #2: Comparison 2.4.20, 2.4.32-rc-1, 2.6.10(11th Oct-2005).xls --]
[-- Type: application/vnd.ms-excel, Size: 110592 bytes --]

Re: How to improve performance of 2.6 kernel

[Stuart Longland]

[-- Attachment #1: Type: text/plain, Size: 643 bytes --]

kernel coder wrote:
> I am also attaching the benchmarks I took for both the kernels. Is
> there any way I can improve 2.6's performance? Thanks.

Hi,
	Don't mean to nitpick... but do you think it wise to use a proprietry
format like Microsoft Excel on a prodominantly Linux-user list? :-)
Tab-delimited would probably be better for distribution on a mailing list.

	How much slower are we talking here?  And in what regards?

Regards,
-- 
Stuart Longland (aka Redhatter)              .'''.
Gentoo Linux/MIPS Cobalt and Docs Developer  '.'` :
. . . . . . . . . . . . . . . . . . . . . .   .'.'
http://dev.gentoo.org/~redhatter             :.'

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: How to improve performance of 2.6 kernel

[Kevin D. Kissell]

>kernel coder wrote:
>> I am also attaching the benchmarks I took for both the kernels. Is
>> there any way I can improve 2.6's performance? Thanks.
>
>Hi,
> Don't mean to nitpick... but do you think it wise to use a proprietry
>format like Microsoft Excel on a prodominantly Linux-user list? :-)
>Tab-delimited would probably be better for distribution on a mailing list.

There are 2D data plots that are hard to do as ASCII art. ;o)
It opened fine for me in Open Office - there was one set of graphs
that looked to be empty, but I couldn't tell if that was OO screwing
up or the way the spreadsheet went out.  But the raw data was
legible and troubling.

> How much slower are we talking here?  And in what regards?

Memory copy, network I/O, and disk I/O, by factors ranging from
about 10% to almost 50% - I'd say about 25% on the average.
If this can't be explained by a configuration error, we have a real
problem here, but if that's the case, I'm surprised no one has raised
a red flag earlier.

        Regards,

        Kevin K.

Re: How to improve performance of 2.6 kernel

[Kevin D. Kissell]

>kernel coder wrote:
>> I am also attaching the benchmarks I took for both the kernels. Is
>> there any way I can improve 2.6's performance? Thanks.
>
>Hi,
> Don't mean to nitpick... but do you think it wise to use a proprietry
>format like Microsoft Excel on a prodominantly Linux-user list? :-)
>Tab-delimited would probably be better for distribution on a mailing list.

There are 2D data plots that are hard to do as ASCII art. ;o)
It opened fine for me in Open Office - there was one set of graphs
that looked to be empty, but I couldn't tell if that was OO screwing
up or the way the spreadsheet went out.  But the raw data was
legible and troubling.

> How much slower are we talking here?  And in what regards?

Memory copy, network I/O, and disk I/O, by factors ranging from
about 10% to almost 50% - I'd say about 25% on the average.
If this can't be explained by a configuration error, we have a real
problem here, but if that's the case, I'm surprised no one has raised
a red flag earlier.

        Regards,

        Kevin K.

Re: How to improve performance of 2.6 kernel

[Dan Malek]

On Oct 17, 2005, at 11:19 AM, Kevin D. Kissell wrote:

> If this can't be explained by a configuration error, we have a real
> problem here, but if that's the case, I'm surprised no one has raised
> a red flag earlier.

It has been discussed on other processor architecture lists.
It's been hard to justify the move from 2.4 to 2.6 on resource
challenged embedded systems, which unfortunately make up the
majority of systems running Linux.  There are various processor
specific modifications (mostly around VM, MMU, and cache
management) being attempted to bring the performance level
back up to 2.4.  If these were back ported to 2.4, I suspect the
performance difference would be even greater.

Of course, the speed and resources of workstations masks
the problems, so most developers don't "feel" the system is
any different. There isn't interest in the detailed performance
measurements we have to do on embedded systems when
we do an upgrade and realize it doesn't meet the performance
goals.  This is usually just written off with the "....  well, you have
new features .." statement, but somehow it doesn't add up.

There isn't any magic configuration option or quick fix.  You
have to take the time to dig into the details of a specific
performance issue and then try to optimize anything you can
to improve the situation.

Thanks.

	-- Dan

Re: How to improve performance of 2.6 kernel

[Dan Malek]

On Oct 17, 2005, at 11:19 AM, Kevin D. Kissell wrote:

> If this can't be explained by a configuration error, we have a real
> problem here, but if that's the case, I'm surprised no one has raised
> a red flag earlier.

It has been discussed on other processor architecture lists.
It's been hard to justify the move from 2.4 to 2.6 on resource
challenged embedded systems, which unfortunately make up the
majority of systems running Linux.  There are various processor
specific modifications (mostly around VM, MMU, and cache
management) being attempted to bring the performance level
back up to 2.4.  If these were back ported to 2.4, I suspect the
performance difference would be even greater.

Of course, the speed and resources of workstations masks
the problems, so most developers don't "feel" the system is
any different. There isn't interest in the detailed performance
measurements we have to do on embedded systems when
we do an upgrade and realize it doesn't meet the performance
goals.  This is usually just written off with the "....  well, you have
new features .." statement, but somehow it doesn't add up.

There isn't any magic configuration option or quick fix.  You
have to take the time to dig into the details of a specific
performance issue and then try to optimize anything you can
to improve the situation.

Thanks.

	-- Dan

Fwd: How to improve performance of 2.6 kernel

[kernel coder]

[-- Attachment #1: Type: text/plain, Size: 2055 bytes --]

Hi,

I did lmbench benchmarks tests... and the results i got were pretty
weird.. I am attaching the jpegs :) of the graphs i made in MS Excel.
Btw, I have implemented NAPI in both 2.4.20 and 2.6.10. I ported the
code to linux-2.6 in order to increase the board's efficiency but I'm
quite dissapointed with the results so far :(.



---------- Forwarded message ----------
From: Dan Malek <dan@embeddedalley.com>
Date: Oct 17, 2005 9:38 PM
Subject: Re: How to improve performance of 2.6 kernel
To: "Kevin D. Kissell" <KevinK@mips.com>
Cc: kernel coder <lhrkernelcoder@gmail.com>,
linux-mips@linux-mips.org, Stuart Longland <redhatter@gentoo.org>



On Oct 17, 2005, at 11:19 AM, Kevin D. Kissell wrote:

> If this can't be explained by a configuration error, we have a real
> problem here, but if that's the case, I'm surprised no one has raised
> a red flag earlier.

It has been discussed on other processor architecture lists.
It's been hard to justify the move from 2.4 to 2.6 on resource
challenged embedded systems, which unfortunately make up the
majority of systems running Linux.  There are various processor
specific modifications (mostly around VM, MMU, and cache
management) being attempted to bring the performance level
back up to 2.4.  If these were back ported to 2.4, I suspect the
performance difference would be even greater.

Of course, the speed and resources of workstations masks
the problems, so most developers don't "feel" the system is
any different. There isn't interest in the detailed performance
measurements we have to do on embedded systems when
we do an upgrade and realize it doesn't meet the performance
goals.  This is usually just written off with the "....  well, you have
new features .." statement, but somehow it doesn't add up.

There isn't any magic configuration option or quick fix.  You
have to take the time to dig into the details of a specific
performance issue and then try to optimize anything you can
to improve the situation.

Thanks.

        -- Dan

[-- Attachment #2: diagram1.JPG --]
[-- Type: image/jpeg, Size: 91133 bytes --]

[-- Attachment #3: diagram2.JPG --]
[-- Type: image/jpeg, Size: 89134 bytes --]

[-- Attachment #4: diagram3.JPG --]
[-- Type: image/jpeg, Size: 83849 bytes --]

Re: Fwd: How to improve performance of 2.6 kernel

[Ralf Baechle]

On Wed, Oct 19, 2005 at 10:55:01AM +0500, kernel coder wrote:

> I did lmbench benchmarks tests... and the results i got were pretty
> weird.. I am attaching the jpegs :) of the graphs i made in MS Excel.

We're happy with cold, raw ASCII numbers :)

> Btw, I have implemented NAPI in both 2.4.20 and 2.6.10. I ported the
> code to linux-2.6 in order to increase the board's efficiency but I'm
> quite dissapointed with the results so far :(.

NAPI is doing it's job which is keeping a system responsive under extreme
loads very well.  The pre-NAPI behaviour was simply locking up thus making
systems easily DOS-able.  NAPI is not meant to improve latency; it isn't
meant but frequently mistaken to.

  Ralf

Re: How to improve performance of 2.6 kernel

[Ralf Baechle]

On Mon, Oct 17, 2005 at 04:29:44AM -0700, kernel coder wrote:

> I have just ported linux kernel 2.6.10 for MIPS4Kc-core board.Before
> porting 2.6 kernel ,2.4.20 was running on this board. When I took
> benchmarks for both the kernels for comparison, I found out that
> linux-2.4.20 was giving much better results than linux-2.6.10. The
> specs for the board are as follows:
> 
> --------------------------------------------------------------
> 133MHz MIPS4kc

How many TLB entries does your 4Kc have?  2.6 is hitting the TLB harder
and system that have small TLBs tend to suffer from that at the bottom
line even though all the other benefits of 2.6.

It would be interesting to see lmbench numbers for the system configurations
you've tested.  Lmbench is a well defined workload that's proven useful
in isolating such issues.

Thanks,

  Ralf

RE: Fwd: How to improve performance of 2.6 kernel

[Mitchell, Earl]

If you are looking at network performance ...
A while back I briefly looked at this and it
looked like the per packet processing time had
gone up significantly between 2.4.20 and
2.6.x. Problem was not NAPI which switches
from interrupt to polling mode when you get a burst 
of packets. I actually traced a single packet from Rx 
to Tx. Unfortunately I didn't save my data and did 
not isolate where increased time was being spent. 

-earlm

> -----Original Message-----
> From: linux-mips-bounce@linux-mips.org
> [mailto:linux-mips-bounce@linux-mips.org]On Behalf Of Ralf Baechle
> Sent: Wednesday, October 19, 2005 8:54 AM
> To: kernel coder
> Cc: linux-mips@linux-mips.org
> Subject: Re: Fwd: How to improve performance of 2.6 kernel
> 
> 
> On Wed, Oct 19, 2005 at 10:55:01AM +0500, kernel coder wrote:
> 
> > I did lmbench benchmarks tests... and the results i got were pretty
> > weird.. I am attaching the jpegs :) of the graphs i made in 
> MS Excel.
> 
> We're happy with cold, raw ASCII numbers :)
> 
> > Btw, I have implemented NAPI in both 2.4.20 and 2.6.10. I ported the
> > code to linux-2.6 in order to increase the board's 
> efficiency but I'm
> > quite dissapointed with the results so far :(.
> 
> NAPI is doing it's job which is keeping a system responsive 
> under extreme
> loads very well.  The pre-NAPI behaviour was simply locking 
> up thus making
> systems easily DOS-able.  NAPI is not meant to improve 
> latency; it isn't
> meant but frequently mistaken to.
> 
>   Ralf
> 
> 

RE: Fwd: How to improve performance of 2.6 kernel

[Mitchell, Earl]

If you are looking at network performance ...
A while back I briefly looked at this and it
looked like the per packet processing time had
gone up significantly between 2.4.20 and
2.6.x. Problem was not NAPI which switches
from interrupt to polling mode when you get a burst 
of packets. I actually traced a single packet from Rx 
to Tx. Unfortunately I didn't save my data and did 
not isolate where increased time was being spent. 

-earlm

> -----Original Message-----
> From: linux-mips-bounce@linux-mips.org
> [mailto:linux-mips-bounce@linux-mips.org]On Behalf Of Ralf Baechle
> Sent: Wednesday, October 19, 2005 8:54 AM
> To: kernel coder
> Cc: linux-mips@linux-mips.org
> Subject: Re: Fwd: How to improve performance of 2.6 kernel
> 
> 
> On Wed, Oct 19, 2005 at 10:55:01AM +0500, kernel coder wrote:
> 
> > I did lmbench benchmarks tests... and the results i got were pretty
> > weird.. I am attaching the jpegs :) of the graphs i made in 
> MS Excel.
> 
> We're happy with cold, raw ASCII numbers :)
> 
> > Btw, I have implemented NAPI in both 2.4.20 and 2.6.10. I ported the
> > code to linux-2.6 in order to increase the board's 
> efficiency but I'm
> > quite dissapointed with the results so far :(.
> 
> NAPI is doing it's job which is keeping a system responsive 
> under extreme
> loads very well.  The pre-NAPI behaviour was simply locking 
> up thus making
> systems easily DOS-able.  NAPI is not meant to improve 
> latency; it isn't
> meant but frequently mistaken to.
> 
>   Ralf
> 
> 

Re: Fwd: How to improve performance of 2.6 kernel

[Ralf Baechle]

On Wed, Oct 19, 2005 at 11:12:22AM -0700, Mitchell, Earl wrote:

> If you are looking at network performance ...
> A while back I briefly looked at this and it
> looked like the per packet processing time had
> gone up significantly between 2.4.20 and
> 2.6.x. Problem was not NAPI which switches
> from interrupt to polling mode when you get a burst 
> of packets. I actually traced a single packet from Rx 
> to Tx. Unfortunately I didn't save my data and did 
> not isolate where increased time was being spent. 

A common cause for low networking performance is connection tracking.  Just
loading the module will have serious impact.

  Ralf