Re: 2.6.12 Performance problems

Re: 2.6.12 Performance problems

[Andrew Morton]

Danial Thom <danial_thom@yahoo.com> wrote:
>
> I just started fiddling with 2.6.12, and there
> seems to be a big drop-off in performance from
> 2.4.x in terms of networking on a uniprocessor
> system. Just bridging packets through the
> machine, 2.6.12 starts dropping packets at
> ~100Kpps, whereas 2.4.x doesn't start dropping
> until over 350Kpps on the same hardware (2.0Ghz
> Opteron with e1000 driver). This is pitiful
> prformance for this hardware. I've 
> increased the rx ring in the e1000 driver to 512
> with little change (interrupt moderation is set
> to 8000 Ints/second). Has "tuning" for MP 
> destroyed UP performance altogether, or is there
> some tuning parameter that could make a 4-fold
> difference? All debugging is off and there are 
> no messages on the console or in the error logs.
> The kernel is the standard kernel.org dowload
> config with SMP turned off and the intel ethernet
> card drivers as modules without any other
> changes, which is exactly the config for my 2.4
> kernels.
> 

(added netdev to cc)

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:

> I think the concensus is that 2.6 has made trade
> offs that lower raw throughput, which is what a
> networking device needs. So as a router or
> network appliance, 2.6 seems less suitable. A raw
> bridging test on a 2.0Ghz operton system:
> 
> FreeBSD 4.9: Drops no packets at 900K pps
> Linux 2.4.24: Starts dropping packets at 350K pps
> Linux 2.6.12: Starts dropping packets at 100K pps

I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz
PCI-X bus.  NIC is dual Intel pro/1000.  Kernel is close to stock 2.6.11.

I used brctl to create a bridge with the two GigE adapters in it and
used pktgen to stream traffic through it (250kpps in one direction, 1kpps in
the other.)

I see a reasonable amount of drops at 250kpps (60 byte packets):
about 60,000,000 packets received, 20,700 dropped.

Interestingly, the system is about 60% idle according to top,
and still dropping pkts, so it would seem that the system could
be better utilized!

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> 
> > I think the concensus is that 2.6 has made
> trade
> > offs that lower raw throughput, which is what
> a
> > networking device needs. So as a router or
> > network appliance, 2.6 seems less suitable. A
> raw
> > bridging test on a 2.0Ghz operton system:
> > 
> > FreeBSD 4.9: Drops no packets at 900K pps
> > Linux 2.4.24: Starts dropping packets at 350K
> pps
> > Linux 2.6.12: Starts dropping packets at 100K
> pps
> 
> I ran some quick tests using kernel 2.6.11, 1ms
> tick (HZ=1000), SMP kernel.
> Hardware is P-IV 3.0Ghz + HT on a new
> SuperMicro motherboard with 64/133Mhz
> PCI-X bus.  NIC is dual Intel pro/1000.  Kernel
> is close to stock 2.6.11.
> 
> I used brctl to create a bridge with the two
> GigE adapters in it and
> used pktgen to stream traffic through it
> (250kpps in one direction, 1kpps in
> the other.)
> 
> I see a reasonable amount of drops at 250kpps
> (60 byte packets):
> about 60,000,000 packets received, 20,700
> dropped.
> 
> Interestingly, the system is about 60% idle
> according to top,
> and still dropping pkts, so it would seem that
> the system could
> be better utilized!
> 
> Ben
>

What GigE adapters did you use? Clearly every
driver is going to be different. My experience is
that a 3.4Ghz P4 is about the performance of a
2.0Ghz Opteron. I have to try your tuning script
tomorrow.

If your test is still set up, try compiling
something large while doing the test. The drops
go through the roof in my tests.

Danial


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:
> 
> --- Ben Greear <greearb@candelatech.com> wrote:
> 
> 
>>Danial Thom wrote:
>>
>>
>>>I think the concensus is that 2.6 has made
>>
>>trade
>>
>>>offs that lower raw throughput, which is what
>>
>>a
>>
>>>networking device needs. So as a router or
>>>network appliance, 2.6 seems less suitable. A
>>
>>raw
>>
>>>bridging test on a 2.0Ghz operton system:
>>>
>>>FreeBSD 4.9: Drops no packets at 900K pps
>>>Linux 2.4.24: Starts dropping packets at 350K
>>
>>pps
>>
>>>Linux 2.6.12: Starts dropping packets at 100K
>>
>>pps
>>
>>I ran some quick tests using kernel 2.6.11, 1ms
>>tick (HZ=1000), SMP kernel.
>>Hardware is P-IV 3.0Ghz + HT on a new
>>SuperMicro motherboard with 64/133Mhz
>>PCI-X bus.  NIC is dual Intel pro/1000.  Kernel
>>is close to stock 2.6.11.

> What GigE adapters did you use? Clearly every
> driver is going to be different. My experience is
> that a 3.4Ghz P4 is about the performance of a
> 2.0Ghz Opteron. I have to try your tuning script
> tomorrow.

Intel pro/1000, as I mentioned.  I haven't tried any other
NIC that comes close in performance to the e1000.

> If your test is still set up, try compiling
> something large while doing the test. The drops
> go through the roof in my tests.

Installing RH9 on the box now to try some tests...

Disk access always robs networking, in my experience, so
I am not supprised you see bad ntwk performance while
compiling.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:
> 
> --- Ben Greear <greearb@candelatech.com> wrote:
> 
> 
>>Danial Thom wrote:
>>
>>
>>>I think the concensus is that 2.6 has made
>>
>>trade
>>
>>>offs that lower raw throughput, which is what
>>
>>a
>>
>>>networking device needs. So as a router or
>>>network appliance, 2.6 seems less suitable. A
>>
>>raw
>>
>>>bridging test on a 2.0Ghz operton system:
>>>
>>>FreeBSD 4.9: Drops no packets at 900K pps
>>>Linux 2.4.24: Starts dropping packets at 350K
>>
>>pps
>>
>>>Linux 2.6.12: Starts dropping packets at 100K
>>
>>pps
>>
>>I ran some quick tests using kernel 2.6.11, 1ms
>>tick (HZ=1000), SMP kernel.
>>Hardware is P-IV 3.0Ghz + HT on a new
>>SuperMicro motherboard with 64/133Mhz
>>PCI-X bus.  NIC is dual Intel pro/1000.  Kernel
>>is close to stock 2.6.11.
>>
>>I used brctl to create a bridge with the two
>>GigE adapters in it and
>>used pktgen to stream traffic through it
>>(250kpps in one direction, 1kpps in
>>the other.)
>>
>>I see a reasonable amount of drops at 250kpps
>>(60 byte packets):
>>about 60,000,000 packets received, 20,700
>>dropped.

I get slightly worse performance on this system when running RH9
with kernel 2.4.29 (my hacks, HZ=1000, SMP).  Tried increasing
e1000 descriptors to 2048 tx and rx, but that didn't help, or at least
not much.

Will try some other tunings, but I doubt it will affect performance
enough to come close to the discrepency that you show between 2.4
and 2.6 kernels...

I tried copying a 500MB CDROM to HD on my RH9 system, and only 6kpps
of the 250kpps get through the bridge...btw.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> > 
> > --- Ben Greear <greearb@candelatech.com>
> wrote:
> > 
> > 
> >>Danial Thom wrote:
> >>
> >>
> >>>I think the concensus is that 2.6 has made
> >>
> >>trade
> >>
> >>>offs that lower raw throughput, which is
> what
> >>
> >>a
> >>
> >>>networking device needs. So as a router or
> >>>network appliance, 2.6 seems less suitable.
> A
> >>
> >>raw
> >>
> >>>bridging test on a 2.0Ghz operton system:
> >>>
> >>>FreeBSD 4.9: Drops no packets at 900K pps
> >>>Linux 2.4.24: Starts dropping packets at
> 350K
> >>
> >>pps
> >>
> >>>Linux 2.6.12: Starts dropping packets at
> 100K
> >>
> >>pps
> >>
> >>I ran some quick tests using kernel 2.6.11,
> 1ms
> >>tick (HZ=1000), SMP kernel.
> >>Hardware is P-IV 3.0Ghz + HT on a new
> >>SuperMicro motherboard with 64/133Mhz
> >>PCI-X bus.  NIC is dual Intel pro/1000. 
> Kernel
> >>is close to stock 2.6.11.
> >>
> >>I used brctl to create a bridge with the two
> >>GigE adapters in it and
> >>used pktgen to stream traffic through it
> >>(250kpps in one direction, 1kpps in
> >>the other.)
> >>
> >>I see a reasonable amount of drops at 250kpps
> >>(60 byte packets):
> >>about 60,000,000 packets received, 20,700
> >>dropped.
> 
> I get slightly worse performance on this system
> when running RH9
> with kernel 2.4.29 (my hacks, HZ=1000, SMP). 
> Tried increasing
> e1000 descriptors to 2048 tx and rx, but that
> didn't help, or at least
> not much.
> 
> Will try some other tunings, but I doubt it
> will affect performance
> enough to come close to the discrepency that
> you show between 2.4
> and 2.6 kernels...
> 
> I tried copying a 500MB CDROM to HD on my RH9
> system, and only 6kpps
> of the 250kpps get through the bridge...btw.

The tests I reported where on UP systems. Perhaps
the default settings are better for this in 2.4,
since that is what I used, and you used your
hacks for both.

Are you getting drops or overruns (or both)? I
would assume drops is a decision to drop rather
than an overrun which is a ring overrun. Overruns
would imply more about performance than tuning,
I'd think.

I wouldn't think that HT would be appropriate for
this sort of setup...?

You're using a dual PCI-X NIC rather than the
onboard ports? Supermicro runs their onboard
controllers at 32bit/33mhz for some mindless
reason.

Danial

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:

> The tests I reported where on UP systems. Perhaps
> the default settings are better for this in 2.4,
> since that is what I used, and you used your
> hacks for both.

My modifications to the kernel are unlikely to speed anything
up, and probably will slow things down ever so slightly.

I can try with a UP kernel, but my machine at least has a single
processor.  I'm using the SMP kernel to take advantage of HT.

> Are you getting drops or overruns (or both)? I
> would assume drops is a decision to drop rather
> than an overrun which is a ring overrun. Overruns
> would imply more about performance than tuning,
> I'd think.

I was seeing lots of NIC errors...in fact, it was showing a great many
more errors than packets sent to it, so I just ignored them.

I increased the TxDescriptors and RxDescriptors and that helped a little.

Increasing the transmit queue for the NIC to 2000 also helped a little.

> I wouldn't think that HT would be appropriate for
> this sort of setup...?

2.6.11 seems to be faster when running SMP kernel on this system.
> 
> You're using a dual PCI-X NIC rather than the
> onboard ports? Supermicro runs their onboard

Of course.  Never found a motherboard yet with decent built-in
NICs.  The built-ins on this board are tg3 and they must be on
a slow bus, because they cannot go faster than about 700Mbps
(using big pkts).

I'll benchmark things again when 2.6.13 comes out and try to
get some more detailed numbers...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> 
> > The tests I reported where on UP systems.
> Perhaps
> > the default settings are better for this in
> 2.4,
> > since that is what I used, and you used your
> > hacks for both.
> 
> My modifications to the kernel are unlikely to
> speed anything
> up, and probably will slow things down ever so
> slightly.
> 
> I can try with a UP kernel, but my machine at
> least has a single
> processor.  I'm using the SMP kernel to take
> advantage of HT.
> 
> > Are you getting drops or overruns (or both)?
> I
> > would assume drops is a decision to drop
> rather
> > than an overrun which is a ring overrun.
> Overruns
> > would imply more about performance than
> tuning,
> > I'd think.
> 
> I was seeing lots of NIC errors...in fact, it
> was showing a great many
> more errors than packets sent to it, so I just
> ignored them.
> 
> I increased the TxDescriptors and RxDescriptors
> and that helped a little.
> 
> Increasing the transmit queue for the NIC to
> 2000 also helped a little.
> 
> > I wouldn't think that HT would be appropriate
> for
> > this sort of setup...?
> 
> 2.6.11 seems to be faster when running SMP
> kernel on this system.

HT and SMP are not the same animal, are they? My
understanding is that an HT aware scheduler is
likely to make things worse most of the time,
particularly for systems not running a lot of
threads..


> > 
> > You're using a dual PCI-X NIC rather than the
> > onboard ports? Supermicro runs their onboard
> 
> Of course.  Never found a motherboard yet with
> decent built-in
> NICs.  The built-ins on this board are tg3 and
> they must be on
> a slow bus, because they cannot go faster than
> about 700Mbps
> (using big pkts).

If its the P8SCI or the same design they are on a
1X PCIE thats shared with the PCI-X. Pretty hokey
stuff. Its also a low-end controller amongst the
broadcom parts.

Danial


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> > 
> > --- Ben Greear <greearb@candelatech.com>
> wrote:
> > 
> > 
> >>Danial Thom wrote:
> >>
> >>
> >>>I think the concensus is that 2.6 has made
> >>
> >>trade
> >>
> >>>offs that lower raw throughput, which is
> what
> >>
> >>a
> >>
> >>>networking device needs. So as a router or
> >>>network appliance, 2.6 seems less suitable.
> A
> >>
> >>raw
> >>
> >>>bridging test on a 2.0Ghz operton system:
> >>>
> >>>FreeBSD 4.9: Drops no packets at 900K pps
> >>>Linux 2.4.24: Starts dropping packets at
> 350K
> >>
> >>pps
> >>
> >>>Linux 2.6.12: Starts dropping packets at
> 100K
> >>
> >>pps
> >>
> >>I ran some quick tests using kernel 2.6.11,
> 1ms
> >>tick (HZ=1000), SMP kernel.
> >>Hardware is P-IV 3.0Ghz + HT on a new
> >>SuperMicro motherboard with 64/133Mhz
> >>PCI-X bus.  NIC is dual Intel pro/1000. 
> Kernel
> >>is close to stock 2.6.11.
> 
> > What GigE adapters did you use? Clearly every
> > driver is going to be different. My
> experience is
> > that a 3.4Ghz P4 is about the performance of
> a
> > 2.0Ghz Opteron. I have to try your tuning
> script
> > tomorrow.
> 
> Intel pro/1000, as I mentioned.  I haven't
> tried any other
> NIC that comes close in performance to the
> e1000.
> 
> > If your test is still set up, try compiling
> > something large while doing the test. The
> drops
> > go through the roof in my tests.
> 
> Installing RH9 on the box now to try some
> tests...
> 
> Disk access always robs networking, in my
> experience, so
> I am not supprised you see bad ntwk performance
> while
> compiling.
> 
> Ben

It would be useful if there were some way to find
out "what" is getting "robbed". If networking has
priority, then what is keeping it from getting
back to processing the rx interrupts? 

Ah, the e1000 has built-in interrupt moderation.
I can't get into my lab until tomorrow afternoon,
but if you get a chance try setting ITR in
e1000_main.c to something larger, like 20K. and
see if it makes a difference. At 200K pps that
would cause an interrupt every 10 packets, which
may allow the routine to grab back the cpu more
often.


Danial

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Benjamin LaHaise]

On Thu, Aug 25, 2005 at 09:55:27AM -0700, Ben Greear wrote:
> Of course.  Never found a motherboard yet with decent built-in
> NICs.  The built-ins on this board are tg3 and they must be on
> a slow bus, because they cannot go faster than about 700Mbps
> (using big pkts).

There should be a number of decent boards out on the market these days.  
Try picking up one with a CSA gige adapter (a dedicated high bandwidth 
link to an e1000) or PCI Express (although that is harder to pick up on 
from the specifications of most motherboards).

		-ben

Re: 2.6.12 Performance problems

[Danial Thom]

--- Danial Thom <danial_thom@yahoo.com> wrote:

> 
> 
> --- Ben Greear <greearb@candelatech.com> wrote:
> 
> > Danial Thom wrote:
> > > 
> > > --- Ben Greear <greearb@candelatech.com>
> > wrote:
> > > 
> > > 
> > >>Danial Thom wrote:
> > >>
> > >>
> > >>>I think the concensus is that 2.6 has made
> > >>
> > >>trade
> > >>
> > >>>offs that lower raw throughput, which is
> > what
> > >>
> > >>a
> > >>
> > >>>networking device needs. So as a router or
> > >>>network appliance, 2.6 seems less
> suitable.
> > A
> > >>
> > >>raw
> > >>
> > >>>bridging test on a 2.0Ghz operton system:
> > >>>
> > >>>FreeBSD 4.9: Drops no packets at 900K pps
> > >>>Linux 2.4.24: Starts dropping packets at
> > 350K
> > >>
> > >>pps
> > >>
> > >>>Linux 2.6.12: Starts dropping packets at
> > 100K
> > >>
> > >>pps
> > >>
> > >>I ran some quick tests using kernel 2.6.11,
> > 1ms
> > >>tick (HZ=1000), SMP kernel.
> > >>Hardware is P-IV 3.0Ghz + HT on a new
> > >>SuperMicro motherboard with 64/133Mhz
> > >>PCI-X bus.  NIC is dual Intel pro/1000. 
> > Kernel
> > >>is close to stock 2.6.11.
> > 
> > > What GigE adapters did you use? Clearly
> every
> > > driver is going to be different. My
> > experience is
> > > that a 3.4Ghz P4 is about the performance
> of
> > a
> > > 2.0Ghz Opteron. I have to try your tuning
> > script
> > > tomorrow.
> > 
> > Intel pro/1000, as I mentioned.  I haven't
> > tried any other
> > NIC that comes close in performance to the
> > e1000.
> > 
> > > If your test is still set up, try compiling
> > > something large while doing the test. The
> > drops
> > > go through the roof in my tests.
> > 
> > Installing RH9 on the box now to try some
> > tests...
> > 
> > Disk access always robs networking, in my
> > experience, so
> > I am not supprised you see bad ntwk
> performance
> > while
> > compiling.
> > 
> > Ben
> 
> It would be useful if there were some way to
> find
> out "what" is getting "robbed". If networking
> has
> priority, then what is keeping it from getting
> back to processing the rx interrupts? 
> 
> Ah, the e1000 has built-in interrupt
> moderation.
> I can't get into my lab until tomorrow
> afternoon,
> but if you get a chance try setting ITR in
> e1000_main.c to something larger, like 20K. and
> see if it makes a difference. At 200K pps that
> would cause an interrupt every 10 packets,
> which
> may allow the routine to grab back the cpu more
> often.
> 
> 
> Danial
> 

Just FYI, setting interrupt moderation to 20,000
didn't make much difference. 


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Vladimir B. Savkin]

On Wed, Aug 24, 2005 at 11:08:43PM -0700, Danial Thom wrote:
> If your test is still set up, try compiling
> something large while doing the test. The drops
> go through the roof in my tests.
> 
Couldn't this happen because ksoftirqd by default 
has a nice value of 19?

~
:wq
                                        With best regards, 
                                           Vladimir Savkin. 

Re: 2.6.12 Performance problems

[Danial Thom]

--- "Vladimir B. Savkin" <master@sectorb.msk.ru>
wrote:

> On Wed, Aug 24, 2005 at 11:08:43PM -0700,
> Danial Thom wrote:
> > If your test is still set up, try compiling
> > something large while doing the test. The
> drops
> > go through the roof in my tests.
> > 
> Couldn't this happen because ksoftirqd by
> default 
> has a nice value of 19?
> 
> ~
> :wq
>                                         With
> best regards, 
>                                           
> Vladimir Savkin. 

renicing ksoftirqd improves it a bit, but
still tons of loss, even at only 120Kpps.


Danial


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs