Re: 2.6.12 Performance problems

Re: 2.6.12 Performance problems

[Danial Thom]

--- Jesper Juhl <jesper.juhl@gmail.com> wrote:

> On 8/21/05, Danial Thom <danial_thom@yahoo.com>
> wrote:
> > I just started fiddling with 2.6.12, and
> there
> > seems to be a big drop-off in performance
> from
> > 2.4.x in terms of networking on a
> uniprocessor
> > system. Just bridging packets through the
> > machine, 2.6.12 starts dropping packets at
> > ~100Kpps, whereas 2.4.x doesn't start
> dropping
> > until over 350Kpps on the same hardware
> (2.0Ghz
> > Opteron with e1000 driver). This is pitiful
> > prformance for this hardware. I've
> > increased the rx ring in the e1000 driver to
> 512
> > with little change (interrupt moderation is
> set
> > to 8000 Ints/second). Has "tuning" for MP
> > destroyed UP performance altogether, or is
> there
> > some tuning parameter that could make a
> 4-fold
> > difference? All debugging is off and there
> are
> > no messages on the console or in the error
> logs.
> > The kernel is the standard kernel.org dowload
> > config with SMP turned off and the intel
> ethernet
> > card drivers as modules without any other
> > changes, which is exactly the config for my
> 2.4
> > kernels.
> > 
> 
> If you have preemtion enabled you could disable
> it. Low latency comes
> at the cost of decreased throughput - can't
> have both. Also try using
> a HZ of 100 if you are currently using 1000,
> that should also improve
> throughput a little at the cost of slightly
> higher latencies.
> 
> I doubt that it'll do any huge difference, but
> if it does, then that
> would probably be valuable info.
> 
Ok, well you'll have to explain this one:

"Low latency comes at the cost of decreased
throughput - can't have both"

Seems to be a bit backwards. Threading the kernel
adds latency, so its the additional latency in
the kernel that causes the drop in throughput. Do
you mean that kernel performance has been
sacrificed in order to be able to service other
threads more quickly, even when there are no
other threads to be serviced?

Danial



		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Helge Hafting]

Danial Thom wrote:

>--- Jesper Juhl <jesper.juhl@gmail.com> wrote:
>
>  
>
>>On 8/21/05, Danial Thom <danial_thom@yahoo.com>
>>wrote:
>>    
>>
>>>I just started fiddling with 2.6.12, and
>>>      
>>>
>>there
>>    
>>
>>>seems to be a big drop-off in performance
>>>      
>>>
>>from
>>    
>>
>>>2.4.x in terms of networking on a
>>>      
>>>
>>uniprocessor
>>    
>>
>>>system. Just bridging packets through the
>>>machine, 2.6.12 starts dropping packets at
>>>~100Kpps, whereas 2.4.x doesn't start
>>>      
>>>
>>dropping
>>    
>>
>>>until over 350Kpps on the same hardware
>>>      
>>>
>>(2.0Ghz
>>    
>>
>>>Opteron with e1000 driver). This is pitiful
>>>prformance for this hardware. I've
>>>increased the rx ring in the e1000 driver to
>>>      
>>>
>>512
>>    
>>
>>>with little change (interrupt moderation is
>>>      
>>>
>>set
>>    
>>
>>>to 8000 Ints/second). Has "tuning" for MP
>>>destroyed UP performance altogether, or is
>>>      
>>>
>>there
>>    
>>
>>>some tuning parameter that could make a
>>>      
>>>
>>4-fold
>>    
>>
>>>difference? All debugging is off and there
>>>      
>>>
>>are
>>    
>>
>>>no messages on the console or in the error
>>>      
>>>
>>logs.
>>    
>>
>>>The kernel is the standard kernel.org dowload
>>>config with SMP turned off and the intel
>>>      
>>>
>>ethernet
>>    
>>
>>>card drivers as modules without any other
>>>changes, which is exactly the config for my
>>>      
>>>
>>2.4
>>    
>>
>>>kernels.
>>>
>>>      
>>>
>>If you have preemtion enabled you could disable
>>it. Low latency comes
>>at the cost of decreased throughput - can't
>>have both. Also try using
>>a HZ of 100 if you are currently using 1000,
>>that should also improve
>>throughput a little at the cost of slightly
>>higher latencies.
>>
>>I doubt that it'll do any huge difference, but
>>if it does, then that
>>would probably be valuable info.
>>
>>    
>>
>Ok, well you'll have to explain this one:
>
>"Low latency comes at the cost of decreased
>throughput - can't have both"
>  
>
Configuring "preempt" gives lower latency, because then
almost anything can be interrupted (preempted).  You can then
get very quick responses to some things, i.e. interrupts and such.

The cost comes, because _something_ was interrupted, something
that instead would run to completion first in a kernel made without 
"preempt".
So that other thing, whatever it is, got slower. 

And the problem is bigger than merely "things happens in a different order".
Switching the cpu from one job to another have a big overhead.  
Particularly,
the cpu caches have to be refilled more often, which takes time.  
Running one
big job to completion fills the cache with that job's data _once_.  If 
the job
is preempted a couple of times you have to bring it into cache three
times instead, and that will cost you, performance wise.

This is not _necessarily_ your problem, but trying a 2.6 kernel without 
preempt
and with hz=100 (both things configurable through normal kernel 
configuration)
will clearly show if this is the problem in your case.  If you're lucky, 
this is all
you need to get your performance back.  If not, then at least it is an
important datapoint for those trying to figure it out.  Nobody here want
2.6 to have 1/4 of the performance of 2.4!

Helge Hafting

Re: 2.6.12 Performance problems

[Danial Thom]

--- Helge Hafting <helge.hafting@aitel.hist.no>
wrote:

> Danial Thom wrote:
> 
> >--- Jesper Juhl <jesper.juhl@gmail.com> wrote:
> >
> >  
> >
> >>On 8/21/05, Danial Thom
> <danial_thom@yahoo.com>
> >>wrote:
> >>    
> >>
> >>>I just started fiddling with 2.6.12, and
> >>>      
> >>>
> >>there
> >>    
> >>
> >>>seems to be a big drop-off in performance
> >>>      
> >>>
> >>from
> >>    
> >>
> >>>2.4.x in terms of networking on a
> >>>      
> >>>
> >>uniprocessor
> >>    
> >>
> >>>system. Just bridging packets through the
> >>>machine, 2.6.12 starts dropping packets at
> >>>~100Kpps, whereas 2.4.x doesn't start
> >>>      
> >>>
> >>dropping
> >>    
> >>
> >>>until over 350Kpps on the same hardware
> >>>      
> >>>
> >>(2.0Ghz
> >>    
> >>
> >>>Opteron with e1000 driver). This is pitiful
> >>>prformance for this hardware. I've
> >>>increased the rx ring in the e1000 driver to
> >>>      
> >>>
> >>512
> >>    
> >>
> >>>with little change (interrupt moderation is
> >>>      
> >>>
> >>set
> >>    
> >>
> >>>to 8000 Ints/second). Has "tuning" for MP
> >>>destroyed UP performance altogether, or is
> >>>      
> >>>
> >>there
> >>    
> >>
> >>>some tuning parameter that could make a
> >>>      
> >>>
> >>4-fold
> >>    
> >>
> >>>difference? All debugging is off and there
> >>>      
> >>>
> >>are
> >>    
> >>
> >>>no messages on the console or in the error
> >>>      
> >>>
> >>logs.
> >>    
> >>
> >>>The kernel is the standard kernel.org
> dowload
> >>>config with SMP turned off and the intel
> >>>      
> >>>
> >>ethernet
> >>    
> >>
> >>>card drivers as modules without any other
> >>>changes, which is exactly the config for my
> >>>      
> >>>
> >>2.4
> >>    
> >>
> >>>kernels.
> >>>
> >>>      
> >>>
> >>If you have preemtion enabled you could
> disable
> >>it. Low latency comes
> >>at the cost of decreased throughput - can't
> >>have both. Also try using
> >>a HZ of 100 if you are currently using 1000,
> >>that should also improve
> >>throughput a little at the cost of slightly
> >>higher latencies.
> >>
> >>I doubt that it'll do any huge difference,
> but
> >>if it does, then that
> >>would probably be valuable info.
> >>
> >>    
> >>
> >Ok, well you'll have to explain this one:
> >
> >"Low latency comes at the cost of decreased
> >throughput - can't have both"
> >  
> >
> Configuring "preempt" gives lower latency,
> because then
> almost anything can be interrupted (preempted).
>  You can then
> get very quick responses to some things, i.e.
> interrupts and such.

I think part of the problem is the continued
misuse of the word "latency". Latency, in
language terms, means "unexplained delay". Its
wrong here because for one, its explainable. But
it also depends on your perspective. The
"latency" is increased for kernel tasks, while it
may be reduced for something that is getting the
benefit of preempting the kernel. So you really
can't say "the price of reduced latency is lower
throughput", because thats simply backwards.
You've increased the kernel tasks latency by
allowing it to be pre-empted. Reduced latency
implies higher efficiency. All you've done here
is shift the latency from one task to another, so
there is no reduction overall, in fact there is
probably a marginal increase due to the overhead
of pre-emption vs doing nothing.

DT



		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Patrick McHardy]

Danial Thom wrote:
> I think part of the problem is the continued
> misuse of the word "latency". Latency, in
> language terms, means "unexplained delay". Its
> wrong here because for one, its explainable. But
> it also depends on your perspective. The
> "latency" is increased for kernel tasks, while it
> may be reduced for something that is getting the
> benefit of preempting the kernel. So you really
> can't say "the price of reduced latency is lower
> throughput", because thats simply backwards.
> You've increased the kernel tasks latency by
> allowing it to be pre-empted. Reduced latency
> implies higher efficiency. All you've done here
> is shift the latency from one task to another, so
> there is no reduction overall, in fact there is
> probably a marginal increase due to the overhead
> of pre-emption vs doing nothing.

If instead of complaining you would provide the information
I've asked for two days ago someone might actually be able
to help you.

Re: 2.6.12 Performance problems

[Danial Thom]

--- Patrick McHardy <kaber@trash.net> wrote:

> Danial Thom wrote:
> > I think part of the problem is the continued
> > misuse of the word "latency". Latency, in
> > language terms, means "unexplained delay".
> Its
> > wrong here because for one, its explainable.
> But
> > it also depends on your perspective. The
> > "latency" is increased for kernel tasks,
> while it
> > may be reduced for something that is getting
> the
> > benefit of preempting the kernel. So you
> really
> > can't say "the price of reduced latency is
> lower
> > throughput", because thats simply backwards.
> > You've increased the kernel tasks latency by
> > allowing it to be pre-empted. Reduced latency
> > implies higher efficiency. All you've done
> here
> > is shift the latency from one task to
> another, so
> > there is no reduction overall, in fact there
> is
> > probably a marginal increase due to the
> overhead
> > of pre-emption vs doing nothing.
> 
> If instead of complaining you would provide the
> information
> I've asked for two days ago someone might
> actually be able
> to help you.

Because gaining an understanding of how the
settings work is better than having 30 guys
telling me to tune something that is only going
to make a marginal difference. I didn't ask you
to tell me what was wrong with my setup, only
whether its expected that 2.6 would be less
useful in a UP setup than 2.4, which I think
you've answered. 

D

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Jesper Juhl]

On 8/24/05, Danial Thom <danial_thom@yahoo.com> wrote:
> --- Patrick McHardy <kaber@trash.net> wrote:
> 
> > Danial Thom wrote:
> > > I think part of the problem is the continued
> > > misuse of the word "latency". Latency, in
> > > language terms, means "unexplained delay".
> > Its
> > > wrong here because for one, its explainable.
> > But
> > > it also depends on your perspective. The
> > > "latency" is increased for kernel tasks,
> > while it
> > > may be reduced for something that is getting
> > the
> > > benefit of preempting the kernel. So you
> > really
> > > can't say "the price of reduced latency is
> > lower
> > > throughput", because thats simply backwards.
> > > You've increased the kernel tasks latency by
> > > allowing it to be pre-empted. Reduced latency
> > > implies higher efficiency. All you've done
> > here
> > > is shift the latency from one task to
> > another, so
> > > there is no reduction overall, in fact there
> > is
> > > probably a marginal increase due to the
> > overhead
> > > of pre-emption vs doing nothing.
> >
> > If instead of complaining you would provide the
> > information
> > I've asked for two days ago someone might
> > actually be able
> > to help you.
> 
> Because gaining an understanding of how the
> settings work is better than having 30 guys
> telling me to tune something that is only going
> to make a marginal difference. I didn't ask you
> to tell me what was wrong with my setup, only
> whether its expected that 2.6 would be less
> useful in a UP setup than 2.4, which I think
> you've answered.
> 

I hope you're implying that the answer is; no, it's not expected that
2.6 is less useful in a UP setup than 2.4  :-)

-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: 2.6.12 Performance problems

[Danial Thom]

--- Jesper Juhl <jesper.juhl@gmail.com> wrote:

> On 8/24/05, Danial Thom <danial_thom@yahoo.com>
> wrote:
> > --- Patrick McHardy <kaber@trash.net> wrote:
> > 
> > > Danial Thom wrote:
> > > > I think part of the problem is the
> continued
> > > > misuse of the word "latency". Latency, in
> > > > language terms, means "unexplained
> delay".
> > > Its
> > > > wrong here because for one, its
> explainable.
> > > But
> > > > it also depends on your perspective. The
> > > > "latency" is increased for kernel tasks,
> > > while it
> > > > may be reduced for something that is
> getting
> > > the
> > > > benefit of preempting the kernel. So you
> > > really
> > > > can't say "the price of reduced latency
> is
> > > lower
> > > > throughput", because thats simply
> backwards.
> > > > You've increased the kernel tasks latency
> by
> > > > allowing it to be pre-empted. Reduced
> latency
> > > > implies higher efficiency. All you've
> done
> > > here
> > > > is shift the latency from one task to
> > > another, so
> > > > there is no reduction overall, in fact
> there
> > > is
> > > > probably a marginal increase due to the
> > > overhead
> > > > of pre-emption vs doing nothing.
> > >
> > > If instead of complaining you would provide
> the
> > > information
> > > I've asked for two days ago someone might
> > > actually be able
> > > to help you.
> > 
> > Because gaining an understanding of how the
> > settings work is better than having 30 guys
> > telling me to tune something that is only
> going
> > to make a marginal difference. I didn't ask
> you
> > to tell me what was wrong with my setup, only
> > whether its expected that 2.6 would be less
> > useful in a UP setup than 2.4, which I think
> > you've answered.
> > 
> 
> I hope you're implying that the answer is; no,
> it's not expected that
> 2.6 is less useful in a UP setup than 2.4  :-)

I think the concensus is that 2.6 has made trade
offs that lower raw throughput, which is what a
networking device needs. So as a router or
network appliance, 2.6 seems less suitable. A raw
bridging test on a 2.0Ghz operton system:

FreeBSD 4.9: Drops no packets at 900K pps
Linux 2.4.24: Starts dropping packets at 350K pps
Linux 2.6.12: Starts dropping packets at 100K pps

Now the 2.6.12 keyboard is always nice and
snappy, but thats not what I need. I can't have a
box drop traffic if some admin decides to
recompile some application. Linux is fine on
low-medium speed networks, but at a certain
capacity, depending on the specs of the machine
of course, linux drops packets. 

If I do a "make install" in BSD when on a busy
network, it takes a long time, but it doesn't
drop packets. Linux compiles a lot faster, but it
drops buckets of packets. Its just not the
priority thats needed for a networking device.

Danial



		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:

> I think the concensus is that 2.6 has made trade
> offs that lower raw throughput, which is what a
> networking device needs. So as a router or
> network appliance, 2.6 seems less suitable. A raw
> bridging test on a 2.0Ghz operton system:
> 
> FreeBSD 4.9: Drops no packets at 900K pps
> Linux 2.4.24: Starts dropping packets at 350K pps
> Linux 2.6.12: Starts dropping packets at 100K pps

I ran some quick tests using kernel 2.6.11, 1ms tick (HZ=1000), SMP kernel.
Hardware is P-IV 3.0Ghz + HT on a new SuperMicro motherboard with 64/133Mhz
PCI-X bus.  NIC is dual Intel pro/1000.  Kernel is close to stock 2.6.11.

I used brctl to create a bridge with the two GigE adapters in it and
used pktgen to stream traffic through it (250kpps in one direction, 1kpps in
the other.)

I see a reasonable amount of drops at 250kpps (60 byte packets):
about 60,000,000 packets received, 20,700 dropped.

Interestingly, the system is about 60% idle according to top,
and still dropping pkts, so it would seem that the system could
be better utilized!

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> 
> > I think the concensus is that 2.6 has made
> trade
> > offs that lower raw throughput, which is what
> a
> > networking device needs. So as a router or
> > network appliance, 2.6 seems less suitable. A
> raw
> > bridging test on a 2.0Ghz operton system:
> > 
> > FreeBSD 4.9: Drops no packets at 900K pps
> > Linux 2.4.24: Starts dropping packets at 350K
> pps
> > Linux 2.6.12: Starts dropping packets at 100K
> pps
> 
> I ran some quick tests using kernel 2.6.11, 1ms
> tick (HZ=1000), SMP kernel.
> Hardware is P-IV 3.0Ghz + HT on a new
> SuperMicro motherboard with 64/133Mhz
> PCI-X bus.  NIC is dual Intel pro/1000.  Kernel
> is close to stock 2.6.11.
> 
> I used brctl to create a bridge with the two
> GigE adapters in it and
> used pktgen to stream traffic through it
> (250kpps in one direction, 1kpps in
> the other.)
> 
> I see a reasonable amount of drops at 250kpps
> (60 byte packets):
> about 60,000,000 packets received, 20,700
> dropped.
> 
> Interestingly, the system is about 60% idle
> according to top,
> and still dropping pkts, so it would seem that
> the system could
> be better utilized!
> 
> Ben
>

What GigE adapters did you use? Clearly every
driver is going to be different. My experience is
that a 3.4Ghz P4 is about the performance of a
2.0Ghz Opteron. I have to try your tuning script
tomorrow.

If your test is still set up, try compiling
something large while doing the test. The drops
go through the roof in my tests.

Danial


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:
> 
> --- Ben Greear <greearb@candelatech.com> wrote:
> 
> 
>>Danial Thom wrote:
>>
>>
>>>I think the concensus is that 2.6 has made
>>
>>trade
>>
>>>offs that lower raw throughput, which is what
>>
>>a
>>
>>>networking device needs. So as a router or
>>>network appliance, 2.6 seems less suitable. A
>>
>>raw
>>
>>>bridging test on a 2.0Ghz operton system:
>>>
>>>FreeBSD 4.9: Drops no packets at 900K pps
>>>Linux 2.4.24: Starts dropping packets at 350K
>>
>>pps
>>
>>>Linux 2.6.12: Starts dropping packets at 100K
>>
>>pps
>>
>>I ran some quick tests using kernel 2.6.11, 1ms
>>tick (HZ=1000), SMP kernel.
>>Hardware is P-IV 3.0Ghz + HT on a new
>>SuperMicro motherboard with 64/133Mhz
>>PCI-X bus.  NIC is dual Intel pro/1000.  Kernel
>>is close to stock 2.6.11.

> What GigE adapters did you use? Clearly every
> driver is going to be different. My experience is
> that a 3.4Ghz P4 is about the performance of a
> 2.0Ghz Opteron. I have to try your tuning script
> tomorrow.

Intel pro/1000, as I mentioned.  I haven't tried any other
NIC that comes close in performance to the e1000.

> If your test is still set up, try compiling
> something large while doing the test. The drops
> go through the roof in my tests.

Installing RH9 on the box now to try some tests...

Disk access always robs networking, in my experience, so
I am not supprised you see bad ntwk performance while
compiling.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> > 
> > --- Ben Greear <greearb@candelatech.com>
> wrote:
> > 
> > 
> >>Danial Thom wrote:
> >>
> >>
> >>>I think the concensus is that 2.6 has made
> >>
> >>trade
> >>
> >>>offs that lower raw throughput, which is
> what
> >>
> >>a
> >>
> >>>networking device needs. So as a router or
> >>>network appliance, 2.6 seems less suitable.
> A
> >>
> >>raw
> >>
> >>>bridging test on a 2.0Ghz operton system:
> >>>
> >>>FreeBSD 4.9: Drops no packets at 900K pps
> >>>Linux 2.4.24: Starts dropping packets at
> 350K
> >>
> >>pps
> >>
> >>>Linux 2.6.12: Starts dropping packets at
> 100K
> >>
> >>pps
> >>
> >>I ran some quick tests using kernel 2.6.11,
> 1ms
> >>tick (HZ=1000), SMP kernel.
> >>Hardware is P-IV 3.0Ghz + HT on a new
> >>SuperMicro motherboard with 64/133Mhz
> >>PCI-X bus.  NIC is dual Intel pro/1000. 
> Kernel
> >>is close to stock 2.6.11.
> 
> > What GigE adapters did you use? Clearly every
> > driver is going to be different. My
> experience is
> > that a 3.4Ghz P4 is about the performance of
> a
> > 2.0Ghz Opteron. I have to try your tuning
> script
> > tomorrow.
> 
> Intel pro/1000, as I mentioned.  I haven't
> tried any other
> NIC that comes close in performance to the
> e1000.
> 
> > If your test is still set up, try compiling
> > something large while doing the test. The
> drops
> > go through the roof in my tests.
> 
> Installing RH9 on the box now to try some
> tests...
> 
> Disk access always robs networking, in my
> experience, so
> I am not supprised you see bad ntwk performance
> while
> compiling.
> 
> Ben

It would be useful if there were some way to find
out "what" is getting "robbed". If networking has
priority, then what is keeping it from getting
back to processing the rx interrupts? 

Ah, the e1000 has built-in interrupt moderation.
I can't get into my lab until tomorrow afternoon,
but if you get a chance try setting ITR in
e1000_main.c to something larger, like 20K. and
see if it makes a difference. At 200K pps that
would cause an interrupt every 10 packets, which
may allow the routine to grab back the cpu more
often.


Danial

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Danial Thom]

--- Danial Thom <danial_thom@yahoo.com> wrote:

> 
> 
> --- Ben Greear <greearb@candelatech.com> wrote:
> 
> > Danial Thom wrote:
> > > 
> > > --- Ben Greear <greearb@candelatech.com>
> > wrote:
> > > 
> > > 
> > >>Danial Thom wrote:
> > >>
> > >>
> > >>>I think the concensus is that 2.6 has made
> > >>
> > >>trade
> > >>
> > >>>offs that lower raw throughput, which is
> > what
> > >>
> > >>a
> > >>
> > >>>networking device needs. So as a router or
> > >>>network appliance, 2.6 seems less
> suitable.
> > A
> > >>
> > >>raw
> > >>
> > >>>bridging test on a 2.0Ghz operton system:
> > >>>
> > >>>FreeBSD 4.9: Drops no packets at 900K pps
> > >>>Linux 2.4.24: Starts dropping packets at
> > 350K
> > >>
> > >>pps
> > >>
> > >>>Linux 2.6.12: Starts dropping packets at
> > 100K
> > >>
> > >>pps
> > >>
> > >>I ran some quick tests using kernel 2.6.11,
> > 1ms
> > >>tick (HZ=1000), SMP kernel.
> > >>Hardware is P-IV 3.0Ghz + HT on a new
> > >>SuperMicro motherboard with 64/133Mhz
> > >>PCI-X bus.  NIC is dual Intel pro/1000. 
> > Kernel
> > >>is close to stock 2.6.11.
> > 
> > > What GigE adapters did you use? Clearly
> every
> > > driver is going to be different. My
> > experience is
> > > that a 3.4Ghz P4 is about the performance
> of
> > a
> > > 2.0Ghz Opteron. I have to try your tuning
> > script
> > > tomorrow.
> > 
> > Intel pro/1000, as I mentioned.  I haven't
> > tried any other
> > NIC that comes close in performance to the
> > e1000.
> > 
> > > If your test is still set up, try compiling
> > > something large while doing the test. The
> > drops
> > > go through the roof in my tests.
> > 
> > Installing RH9 on the box now to try some
> > tests...
> > 
> > Disk access always robs networking, in my
> > experience, so
> > I am not supprised you see bad ntwk
> performance
> > while
> > compiling.
> > 
> > Ben
> 
> It would be useful if there were some way to
> find
> out "what" is getting "robbed". If networking
> has
> priority, then what is keeping it from getting
> back to processing the rx interrupts? 
> 
> Ah, the e1000 has built-in interrupt
> moderation.
> I can't get into my lab until tomorrow
> afternoon,
> but if you get a chance try setting ITR in
> e1000_main.c to something larger, like 20K. and
> see if it makes a difference. At 200K pps that
> would cause an interrupt every 10 packets,
> which
> may allow the routine to grab back the cpu more
> often.
> 
> 
> Danial
> 

Just FYI, setting interrupt moderation to 20,000
didn't make much difference. 


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:
> 
> --- Ben Greear <greearb@candelatech.com> wrote:
> 
> 
>>Danial Thom wrote:
>>
>>
>>>I think the concensus is that 2.6 has made
>>
>>trade
>>
>>>offs that lower raw throughput, which is what
>>
>>a
>>
>>>networking device needs. So as a router or
>>>network appliance, 2.6 seems less suitable. A
>>
>>raw
>>
>>>bridging test on a 2.0Ghz operton system:
>>>
>>>FreeBSD 4.9: Drops no packets at 900K pps
>>>Linux 2.4.24: Starts dropping packets at 350K
>>
>>pps
>>
>>>Linux 2.6.12: Starts dropping packets at 100K
>>
>>pps
>>
>>I ran some quick tests using kernel 2.6.11, 1ms
>>tick (HZ=1000), SMP kernel.
>>Hardware is P-IV 3.0Ghz + HT on a new
>>SuperMicro motherboard with 64/133Mhz
>>PCI-X bus.  NIC is dual Intel pro/1000.  Kernel
>>is close to stock 2.6.11.
>>
>>I used brctl to create a bridge with the two
>>GigE adapters in it and
>>used pktgen to stream traffic through it
>>(250kpps in one direction, 1kpps in
>>the other.)
>>
>>I see a reasonable amount of drops at 250kpps
>>(60 byte packets):
>>about 60,000,000 packets received, 20,700
>>dropped.

I get slightly worse performance on this system when running RH9
with kernel 2.4.29 (my hacks, HZ=1000, SMP).  Tried increasing
e1000 descriptors to 2048 tx and rx, but that didn't help, or at least
not much.

Will try some other tunings, but I doubt it will affect performance
enough to come close to the discrepency that you show between 2.4
and 2.6 kernels...

I tried copying a 500MB CDROM to HD on my RH9 system, and only 6kpps
of the 250kpps get through the bridge...btw.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> > 
> > --- Ben Greear <greearb@candelatech.com>
> wrote:
> > 
> > 
> >>Danial Thom wrote:
> >>
> >>
> >>>I think the concensus is that 2.6 has made
> >>
> >>trade
> >>
> >>>offs that lower raw throughput, which is
> what
> >>
> >>a
> >>
> >>>networking device needs. So as a router or
> >>>network appliance, 2.6 seems less suitable.
> A
> >>
> >>raw
> >>
> >>>bridging test on a 2.0Ghz operton system:
> >>>
> >>>FreeBSD 4.9: Drops no packets at 900K pps
> >>>Linux 2.4.24: Starts dropping packets at
> 350K
> >>
> >>pps
> >>
> >>>Linux 2.6.12: Starts dropping packets at
> 100K
> >>
> >>pps
> >>
> >>I ran some quick tests using kernel 2.6.11,
> 1ms
> >>tick (HZ=1000), SMP kernel.
> >>Hardware is P-IV 3.0Ghz + HT on a new
> >>SuperMicro motherboard with 64/133Mhz
> >>PCI-X bus.  NIC is dual Intel pro/1000. 
> Kernel
> >>is close to stock 2.6.11.
> >>
> >>I used brctl to create a bridge with the two
> >>GigE adapters in it and
> >>used pktgen to stream traffic through it
> >>(250kpps in one direction, 1kpps in
> >>the other.)
> >>
> >>I see a reasonable amount of drops at 250kpps
> >>(60 byte packets):
> >>about 60,000,000 packets received, 20,700
> >>dropped.
> 
> I get slightly worse performance on this system
> when running RH9
> with kernel 2.4.29 (my hacks, HZ=1000, SMP). 
> Tried increasing
> e1000 descriptors to 2048 tx and rx, but that
> didn't help, or at least
> not much.
> 
> Will try some other tunings, but I doubt it
> will affect performance
> enough to come close to the discrepency that
> you show between 2.4
> and 2.6 kernels...
> 
> I tried copying a 500MB CDROM to HD on my RH9
> system, and only 6kpps
> of the 250kpps get through the bridge...btw.

The tests I reported where on UP systems. Perhaps
the default settings are better for this in 2.4,
since that is what I used, and you used your
hacks for both.

Are you getting drops or overruns (or both)? I
would assume drops is a decision to drop rather
than an overrun which is a ring overrun. Overruns
would imply more about performance than tuning,
I'd think.

I wouldn't think that HT would be appropriate for
this sort of setup...?

You're using a dual PCI-X NIC rather than the
onboard ports? Supermicro runs their onboard
controllers at 32bit/33mhz for some mindless
reason.

Danial

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:

> The tests I reported where on UP systems. Perhaps
> the default settings are better for this in 2.4,
> since that is what I used, and you used your
> hacks for both.

My modifications to the kernel are unlikely to speed anything
up, and probably will slow things down ever so slightly.

I can try with a UP kernel, but my machine at least has a single
processor.  I'm using the SMP kernel to take advantage of HT.

> Are you getting drops or overruns (or both)? I
> would assume drops is a decision to drop rather
> than an overrun which is a ring overrun. Overruns
> would imply more about performance than tuning,
> I'd think.

I was seeing lots of NIC errors...in fact, it was showing a great many
more errors than packets sent to it, so I just ignored them.

I increased the TxDescriptors and RxDescriptors and that helped a little.

Increasing the transmit queue for the NIC to 2000 also helped a little.

> I wouldn't think that HT would be appropriate for
> this sort of setup...?

2.6.11 seems to be faster when running SMP kernel on this system.
> 
> You're using a dual PCI-X NIC rather than the
> onboard ports? Supermicro runs their onboard

Of course.  Never found a motherboard yet with decent built-in
NICs.  The built-ins on this board are tg3 and they must be on
a slow bus, because they cannot go faster than about 700Mbps
(using big pkts).

I'll benchmark things again when 2.6.13 comes out and try to
get some more detailed numbers...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> 
> > The tests I reported where on UP systems.
> Perhaps
> > the default settings are better for this in
> 2.4,
> > since that is what I used, and you used your
> > hacks for both.
> 
> My modifications to the kernel are unlikely to
> speed anything
> up, and probably will slow things down ever so
> slightly.
> 
> I can try with a UP kernel, but my machine at
> least has a single
> processor.  I'm using the SMP kernel to take
> advantage of HT.
> 
> > Are you getting drops or overruns (or both)?
> I
> > would assume drops is a decision to drop
> rather
> > than an overrun which is a ring overrun.
> Overruns
> > would imply more about performance than
> tuning,
> > I'd think.
> 
> I was seeing lots of NIC errors...in fact, it
> was showing a great many
> more errors than packets sent to it, so I just
> ignored them.
> 
> I increased the TxDescriptors and RxDescriptors
> and that helped a little.
> 
> Increasing the transmit queue for the NIC to
> 2000 also helped a little.
> 
> > I wouldn't think that HT would be appropriate
> for
> > this sort of setup...?
> 
> 2.6.11 seems to be faster when running SMP
> kernel on this system.

HT and SMP are not the same animal, are they? My
understanding is that an HT aware scheduler is
likely to make things worse most of the time,
particularly for systems not running a lot of
threads..


> > 
> > You're using a dual PCI-X NIC rather than the
> > onboard ports? Supermicro runs their onboard
> 
> Of course.  Never found a motherboard yet with
> decent built-in
> NICs.  The built-ins on this board are tg3 and
> they must be on
> a slow bus, because they cannot go faster than
> about 700Mbps
> (using big pkts).

If its the P8SCI or the same design they are on a
1X PCIE thats shared with the PCI-X. Pretty hokey
stuff. Its also a low-end controller amongst the
broadcom parts.

Danial


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Benjamin LaHaise]

On Thu, Aug 25, 2005 at 09:55:27AM -0700, Ben Greear wrote:
> Of course.  Never found a motherboard yet with decent built-in
> NICs.  The built-ins on this board are tg3 and they must be on
> a slow bus, because they cannot go faster than about 700Mbps
> (using big pkts).

There should be a number of decent boards out on the market these days.  
Try picking up one with a CSA gige adapter (a dedicated high bandwidth 
link to an e1000) or PCI Express (although that is harder to pick up on 
from the specifications of most motherboards).

		-ben

Re: 2.6.12 Performance problems

[Vladimir B. Savkin]

On Wed, Aug 24, 2005 at 11:08:43PM -0700, Danial Thom wrote:
> If your test is still set up, try compiling
> something large while doing the test. The drops
> go through the roof in my tests.
> 
Couldn't this happen because ksoftirqd by default 
has a nice value of 19?

~
:wq
                                        With best regards, 
                                           Vladimir Savkin. 

Re: 2.6.12 Performance problems

[Danial Thom]

--- "Vladimir B. Savkin" <master@sectorb.msk.ru>
wrote:

> On Wed, Aug 24, 2005 at 11:08:43PM -0700,
> Danial Thom wrote:
> > If your test is still set up, try compiling
> > something large while doing the test. The
> drops
> > go through the roof in my tests.
> > 
> Couldn't this happen because ksoftirqd by
> default 
> has a nice value of 19?
> 
> ~
> :wq
>                                         With
> best regards, 
>                                           
> Vladimir Savkin. 

renicing ksoftirqd improves it a bit, but
still tons of loss, even at only 120Kpps.


Danial


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Sven-Thorsten Dietrich]

On Tue, 2005-08-23 at 10:10 -0700, Danial Thom wrote:
> 

> > >Ok, well you'll have to explain this one:
> > >
> > >"Low latency comes at the cost of decreased
> > >throughput - can't have both"
> > >  
> > >
> > Configuring "preempt" gives lower latency,
> > because then
> > almost anything can be interrupted (preempted).
> >  You can then
> > get very quick responses to some things, i.e.
> > interrupts and such.
> 
> I think part of the problem is the continued
> misuse of the word "latency". Latency, in
> language terms, means "unexplained delay".

latency

n 
1: (computer science) the time it takes for a specific block of data on
a data track to rotate around to the read/write head [syn: rotational
latency] 
2: the time that elapses between a stimulus and the response to it [syn:
reaction time, response time, latent period] 
3: the state of being not yet evident or active

No apparent references to "unexplained" in association with the word
latency.

Re: 2.6.12 Performance problems

[Danial Thom]

--- Sven-Thorsten Dietrich <sven@mvista.com>
wrote:

> On Tue, 2005-08-23 at 10:10 -0700, Danial Thom
> wrote:
> > 
> 
> > > >Ok, well you'll have to explain this one:
> > > >
> > > >"Low latency comes at the cost of
> decreased
> > > >throughput - can't have both"
> > > >  
> > > >
> > > Configuring "preempt" gives lower latency,
> > > because then
> > > almost anything can be interrupted
> (preempted).
> > >  You can then
> > > get very quick responses to some things,
> i.e.
> > > interrupts and such.
> > 
> > I think part of the problem is the continued
> > misuse of the word "latency". Latency, in
> > language terms, means "unexplained delay".
> 
> latency
> 
> n 
> 1: (computer science) the time it takes for a
> specific block of data on
> a data track to rotate around to the read/write
> head [syn: rotational
> latency] 
> 2: the time that elapses between a stimulus and
> the response to it [syn:
> reaction time, response time, latent period] 
> 3: the state of being not yet evident or active
> 
> No apparent references to "unexplained" in
> association with the word
> latency.

Teaching English is not my thing, but latent
means "dormant", which means doing nothing. So
the time it takes to perform a task is not
latency. Its the time it really takes compared to
the time it ought to take if there was no
overhead. If you have a perfect implementation
then you have no latency. Your definition implies
that there is no way to have zero latency, which
is just wrong.

I've seen computer dictionaries that define
latency as the time it takes to get a response.
That would mean a network switch's latency is
different with different sized packets, which is
just plain stupid. 

None of this is helpful, but since no one has
been able to tell me how to tune it to provide
absolute priority to the network stack I'll
assume it can't be done.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Sven-Thorsten Dietrich]

On Tue, 2005-08-23 at 13:10 -0700, Danial Thom wrote:
> 
> None of this is helpful, but since no one has
> been able to tell me how to tune it to provide
> absolute priority to the network stack I'll
> assume it can't be done.

History has proven that camp wrong almost 100% of the time.

You were told to turn off kernel preemption. 

A diligent comparison requires that, since 2.4 does not support kernel
preemption, and a fair comparison requires holding all other things
constant.

In addition, there were several IP-level features mentioned in emails,
that have been added to 2.6.

You need to make sure those are all off by default, to keep your
comparison relevant.

All the answers are before you, review those emails, turn all that stuff
off and retest.

Re: 2.6.12 Performance problems

[Danial Thom]

--- Sven-Thorsten Dietrich <sven@mvista.com>
wrote:

> On Tue, 2005-08-23 at 13:10 -0700, Danial Thom
> wrote:
> > 
> > None of this is helpful, but since no one has
> > been able to tell me how to tune it to
> provide
> > absolute priority to the network stack I'll
> > assume it can't be done.
> 
> History has proven that camp wrong almost 100%
> of the time.
> 
> You were told to turn off kernel preemption. 
> 
> A diligent comparison requires that, since 2.4
> does not support kernel
> preemption, and a fair comparison requires
> holding all other things
> constant.
> 
> In addition, there were several IP-level
> features mentioned in emails,
> that have been added to 2.6.
> 
> You need to make sure those are all off by
> default, to keep your
> comparison relevant.
> 
> All the answers are before you, review those
> emails, turn all that stuff
> off and retest.

I had tried turning off pre-emption, with little
difference. However linux had the same properties
before there was such a setting (of liking to
drop packets here and there for no apparent
reason under heavy load), so I didn't expect it
to make a huge difference. It seems typing on the
keyboard has the same effect with or without
pre-emption enabled. 

IP is not involved in this test, so no IP stack
issues should be relevent.

Danial



		
__________________________________ 
Yahoo! Mail for Mobile 
Take Yahoo! Mail with you! Check email on your mobile phone. 
http://mobile.yahoo.com/learn/mail 

Re: 2.6.12 Performance problems

[Patrick McHardy]

Danial Thom wrote:
> None of this is helpful, but since no one has
> been able to tell me how to tune it to provide
> absolute priority to the network stack I'll
> assume it can't be done.

The network stack already has priority over user processes,
except when executed in process context, so preemption has
no direct impact on briding or routing performance.

The reason why noone answered your question is because you
don't ask but claim or assume.

Re: 2.6.12 Performance problems

[Ben Greear]

Patrick McHardy wrote:
> Danial Thom wrote:
> 
>>None of this is helpful, but since no one has
>>been able to tell me how to tune it to provide
>>absolute priority to the network stack I'll
>>assume it can't be done.
> 
> 
> The network stack already has priority over user processes,
> except when executed in process context, so preemption has
> no direct impact on briding or routing performance.

Well, NAPI puts a lot more of the packet processing in
process context, including the code that would do routing
I believe.

To give networking more priority, you can re-nice the ksoftirqd
process(es) to a high-priority level like -18 or so.

You can also reserve more memory for the networking stack
and increase the size of the permitted buffers.

I am also interested to hear how your system responds to compiling
w/out PREEMPT.

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Patrick McHardy <kaber@trash.net> wrote:

> Danial Thom wrote:
> > None of this is helpful, but since no one has
> > been able to tell me how to tune it to
> provide
> > absolute priority to the network stack I'll
> > assume it can't be done.
> 
> The network stack already has priority over
> user processes,
> except when executed in process context, so
> preemption has
> no direct impact on briding or routing
> performance.
> 
> The reason why noone answered your question is
> because you
> don't ask but claim or assume.
> 

No, its because guys like you snip out content
when they reply, and/or only read the parts of
messages that you want to read, so when other
people enter a thread, they miss the questions
that were asked long ago. Quoting my post on Aug
22:

"All of this aside, I need to measure the raw 
capabilities of the kernel. With 'bsd OSes I can 
tell what the breaking point is by driving the 
machine to livelock. Linux seems to have a soft, 
floating capacity in that it will drop packets 
here and there for no isolatable reason. I'm 
having difficulty making a case for its use in a 
networking appliance, as dropped packets are not 
acceptable. How do I tune the "its ok to drop 
packets when x occurs" algorithm to be "its never

ok to drop packets unless x occurs" (such as a 
queue depth)? Is it possible?"

Danial





__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Jesper Juhl]

> > >>If you have preemtion enabled you could
> > disable
> > >>it. Low latency comes
> > >>at the cost of decreased throughput - can't
> > >>have both. Also try using
> > >>a HZ of 100 if you are currently using 1000,
> > >>that should also improve
> > >>throughput a little at the cost of slightly
> > >>higher latencies.
> > >>
> > >>I doubt that it'll do any huge difference,
> > but
> > >>if it does, then that
> > >>would probably be valuable info.
> > >>
> > >>
> > >>
> > >Ok, well you'll have to explain this one:
> > >
> > >"Low latency comes at the cost of decreased
> > >throughput - can't have both"
> > >
> > >
> > Configuring "preempt" gives lower latency,
> > because then
> > almost anything can be interrupted (preempted).
> >  You can then
> > get very quick responses to some things, i.e.
> > interrupts and such.
> 
> I think part of the problem is the continued
> misuse of the word "latency". Latency, in
> language terms, means "unexplained delay". Its
> wrong here because for one, its explainable. But
> it also depends on your perspective. The
> "latency" is increased for kernel tasks, while it
> may be reduced for something that is getting the
> benefit of preempting the kernel. So you really
> can't say "the price of reduced latency is lower
> throughput", because thats simply backwards.
> You've increased the kernel tasks latency by
> allowing it to be pre-empted. Reduced latency
> implies higher efficiency. All you've done here
> is shift the latency from one task to another, so
> there is no reduction overall, in fact there is
> probably a marginal increase due to the overhead
> of pre-emption vs doing nothing.
> 


No.  That's simply not correct.

1. Preempt adds overhead in tracking when it's safe to preempt and
when it is not and overhead for checking if something needs to preempt
something else at various points.
 This means more bookkeeping, which means more work done by the kernel
and less work done on behalf of user processes == lower overall
throughput / bandwith [1].

2. Every time a process is preempted work has to be done to switch to
the new process, caches will be flushed/cold, stuff may have to paged
in/out, etc.
 This means more copying of process related data and more cache misses
== lower overall throughput.

But, generally the overhead associated with preemption is low, which
is also why I said in a previous email that on many systems you won't
even notice it.


But, this whole thing has gotten sidetracked into a discussion about;
is preempt good or bad, what does the word "latency" means exactely
(and I don't agree with your definition) [2].

When I originally suggested to you to try a non-preempt kernel (if
your current one is even using preempt, which you haven't told us) I
was simply trying to gather a datapoint.
Since preemption is a major thing that has changed from 2.4 to 2.6 and
since in some cases it *does* impact performance I thought it would be
a valid thing to eliminate it from the list of things that could be
causing your problem. I also believe I said that I didn't think it
would make a big difference, but that it /might/ and we might as well
see what difference it does make (if any).

So, if instead of arguing the exact meaning of a word, making
statements about you /assuming/ that HZ or PREEMPT won't affect
things, you had instead just *tested* it then we would have saved 2
days of arguing and getting nowhere and could instead have gotten that
little detail out of the way and then moved on to the next thing to
check.

When I made the suggestion I had hoped for a reply along the lines of this : 

I just tried a HZ=1000 + PREEMPT vs a HZ=100 + NO-PREEMPT kernel, and
the NO-PREEMPT kernel achieves x% higher throughput. We seem to have a
problem with PREEMPT.

or

Sorry Jesper, you were wrong, booting a kernel with HZ=100 and no
PREEMPT makes absolutely no difference to my network throughput, so
the bug must lie elsewhere.


If you'd done that (and what would it have taken? all of 30min perhaps
to build two kernels and test them), then everyone would have had a
valid piece of information and could have gone off looking for a
preempt related bug or put preempt out of their minds and gone hunting
for the bug somewhere else.
That was my intention with the original suggestion to test a no
preempt kernel, not to start arguing the merrits of preemption in
general or the exact meaning of the word "latency".

Finally, here's a little article you might like to read :
http://www.linuxdevices.com/articles/AT5152980814.html



[1] http://en.wikipedia.org/wiki/Comparison_of_latency_and_bandwidth
[2] http://en.wikipedia.org/wiki/Latency_%28engineering%29


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: 2.6.12 Performance problems

[Danial Thom]

--- Jesper Juhl <jesper.juhl@gmail.com> wrote:

> > > >>If you have preemtion enabled you could
> > > disable
> > > >>it. Low latency comes
> > > >>at the cost of decreased throughput -
> can't
> > > >>have both. Also try using
> > > >>a HZ of 100 if you are currently using
> 1000,
> > > >>that should also improve
> > > >>throughput a little at the cost of
> slightly
> > > >>higher latencies.
> > > >>
> > > >>I doubt that it'll do any huge
> difference,
> > > but
> > > >>if it does, then that
> > > >>would probably be valuable info.
> > > >>
> > > >>
> > > >>
> > > >Ok, well you'll have to explain this one:
> > > >
> > > >"Low latency comes at the cost of
> decreased
> > > >throughput - can't have both"
> > > >
> > > >
> > > Configuring "preempt" gives lower latency,
> > > because then
> > > almost anything can be interrupted
> (preempted).
> > >  You can then
> > > get very quick responses to some things,
> i.e.
> > > interrupts and such.
> > 
> > I think part of the problem is the continued
> > misuse of the word "latency". Latency, in
> > language terms, means "unexplained delay".
> Its
> > wrong here because for one, its explainable.
> But
> > it also depends on your perspective. The
> > "latency" is increased for kernel tasks,
> while it
> > may be reduced for something that is getting
> the
> > benefit of preempting the kernel. So you
> really
> > can't say "the price of reduced latency is
> lower
> > throughput", because thats simply backwards.
> > You've increased the kernel tasks latency by
> > allowing it to be pre-empted. Reduced latency
> > implies higher efficiency. All you've done
> here
> > is shift the latency from one task to
> another, so
> > there is no reduction overall, in fact there
> is
> > probably a marginal increase due to the
> overhead
> > of pre-emption vs doing nothing.
> > 
> 
> 
> No.  That's simply not correct.
> 
> 1. Preempt adds overhead in tracking when it's
> safe to preempt and
> when it is not and overhead for checking if
> something needs to preempt
> something else at various points.
>  This means more bookkeeping, which means more
> work done by the kernel
> and less work done on behalf of user processes
> == lower overall
> throughput / bandwith [1].
> 
> 2. Every time a process is preempted work has
> to be done to switch to
> the new process, caches will be flushed/cold,
> stuff may have to paged
> in/out, etc.
>  This means more copying of process related
> data and more cache misses
> == lower overall throughput.
> 
> But, generally the overhead associated with
> preemption is low, which
> is also why I said in a previous email that on
> many systems you won't
> even notice it.
> 
> 
> But, this whole thing has gotten sidetracked
> into a discussion about;
> is preempt good or bad, what does the word
> "latency" means exactely
> (and I don't agree with your definition) [2].
> 
> When I originally suggested to you to try a
> non-preempt kernel (if
> your current one is even using preempt, which
> you haven't told us) I
> was simply trying to gather a datapoint.
> Since preemption is a major thing that has
> changed from 2.4 to 2.6 and
> since in some cases it *does* impact
> performance I thought it would be
> a valid thing to eliminate it from the list of
> things that could be
> causing your problem. I also believe I said
> that I didn't think it
> would make a big difference, but that it
> /might/ and we might as well
> see what difference it does make (if any).
> 
> So, if instead of arguing the exact meaning of
> a word, making
> statements about you /assuming/ that HZ or
> PREEMPT won't affect
> things, you had instead just *tested* it then
> we would have saved 2
> days of arguing and getting nowhere and could
> instead have gotten that
> little detail out of the way and then moved on
> to the next thing to
> check.
> 
> When I made the suggestion I had hoped for a
> reply along the lines of this : 
> 
> I just tried a HZ=1000 + PREEMPT vs a HZ=100 +
> NO-PREEMPT kernel, and
> the NO-PREEMPT kernel achieves x% higher
> throughput. We seem to have a
> problem with PREEMPT.
> 
> or
> 
> Sorry Jesper, you were wrong, booting a kernel
> with HZ=100 and no
> PREEMPT makes absolutely no difference to my
> network throughput, so
> the bug must lie elsewhere.
> 
> 
> If you'd done that (and what would it have
> taken? all of 30min perhaps
> to build two kernels and test them), then
> everyone would have had a
> valid piece of information and could have gone
> off looking for a
> preempt related bug or put preempt out of their
> minds and gone hunting
> for the bug somewhere else.
> That was my intention with the original
> suggestion to test a no
> preempt kernel, not to start arguing the
> merrits of preemption in
> general or the exact meaning of the word
> "latency".

I did do it, and there was no difference. It
seemed that no one thought it was going to make a
3-fold difference, so it wasn't worthy of
reporting. When someone suggests "try this,
although I don't think its going to make much of
a difference", I usually don't get excited about
spending time trying it out. But I did, and it
made no difference. So I'm trying to get a handle
on the philosophy of how things work, so I tune
my tests without having to ask you, so I can get
my boss off my case about trying the new linux
version because he read somewhere that its
really, really fast. 

Secondly, I think you said above what I said,
which is that pre-emption bookeeping adds
latency, because overhead = latency. The precise
meaning may not matter, but when the term is used
completely backwards, then it can be
misinterpreted or just plain disregarded. If
someone says "its slower because we reduced the
latency", I wonder if I'm talking to someone who
has credibility, because its like saying "only
the shortest guys could reach the top shelf". It
just makes no common sense. Pre-emption doesn't
reduce latency, it reduces CONTENTION for the
CPU. Using the word latency to describe anything
thats timing based is just wrong.

> 
> Finally, here's a little article you might like
> to read :
>
http://www.linuxdevices.com/articles/AT5152980814.html

Articles are nice, mainly because they tell the
sunny side of the story without mentioning the
trade offs. If dropping packets is the price you
pay for it, then its worthy of mention. I don't
need a real-time OS, I need an OS that can
process a million packets/second without dropping
any. So my question isn't how suitable linux is
for mythTV, its about turning off all of the crap
that makes songs play  nice and getting it to
stop dropping traffic, because dropped traffic
piss off my clients (who mostly use Windows to
listen to music).

Danial

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Danial Thom]

*confused by the top-posting..*

--- Luigi Genoni
<genoni@darkstar.linuxpratico.net> wrote:

> maybe it is possible to be more clear.
> 
> voluntary kernel preemption adds explicit
> preemption points into the
> kernel and  full kernel preemption makes all
> kernel code preemptible. This
> way even when a process is executing some
> syscall in kernel space, it can
> be volontary or involontary preempted.
> 
> For interactive users the systems seems to be
> smarter, but when the system
> is doing a lot of work in kernel space, then
> you of course have to lose
> something.
> 
> Also just to check id it's the case or not to
> preempt means you lose
> something. This something is usually troughput.
> In your case I would not
> use a preemptible kernel.
> 
> SOmething similar could be said about Timer
> frequency, but here the lost
> is connected to the number to interrupts you
> have to manage.
> 
> The point is that a desktop where the users
> simple need a smooth sysstem
> to be userd interactivelly, but not real CPU
> power, and a server where you
> need hourse power are different topics and need
> different kernel
> behaviour.
> 
> 
> 
> On Sun, August 21, 2005 19:07, Danial Thom
> wrote:
> 
> > Ok, well you'll have to explain this one:
> >
> >
> > "Low latency comes at the cost of decreased
> > throughput - can't have both"
> >
> > Seems to be a bit backwards. Threading the
> kernel
> > adds latency, so its the additional latency
> in the kernel that causes the
> > drop in throughput. Do you mean that kernel
> performance has been sacrificed
> > in order to be able to service other threads
> more quickly, even when there
> > are no other threads to be serviced?
> >
> > Danial

The issue I have with that logic is that you seem
to use "kernel" in a general sense without regard
to what its doing. Dropping packets is always
detrimental to the user regardless of what he's
using the computer for. An audio stream that
drops packets isn't going to be "smooth" at the
user level.

All of this aside, I need to measure the raw
capabilities of the kernel. With 'bsd OSes I can
tell what the breaking point is by driving the
machine to livelock. Linux seems to have a soft,
floating capacity in that it will drop packets
here and there for no isolatable reason. I'm
having difficulty making a case for its use in a
networking appliance, as dropped packets are not
acceptable. How do I tune the "its ok to drop
packets when x occurs" algorithm to be "its never
ok to drop packets unless x occurs" (such as a
queue depth)? Is it possible?

Danial


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Adrian Bunk]

On Mon, Aug 22, 2005 at 08:41:11AM -0700, Danial Thom wrote:
>...
> 
> The issue I have with that logic is that you seem
> to use "kernel" in a general sense without regard
> to what its doing. Dropping packets is always
> detrimental to the user regardless of what he's
> using the computer for. An audio stream that
> drops packets isn't going to be "smooth" at the
> user level.


That's not always true.

Imagine a slow computer with a GBit ethernet connection, where the user 
is downloading files from a server that can utilize the full 
network connection while listening to music from his local disk with 
XMMS.

In this case, the audio stream is not depending on the network 
connection. And the user might prefer dropped packages over a stuttering 
XMMS.


> All of this aside, I need to measure the raw
> capabilities of the kernel. With 'bsd OSes I can
> tell what the breaking point is by driving the
> machine to livelock. Linux seems to have a soft,
> floating capacity in that it will drop packets
> here and there for no isolatable reason. I'm
> having difficulty making a case for its use in a
> networking appliance, as dropped packets are not
> acceptable. How do I tune the "its ok to drop
> packets when x occurs" algorithm to be "its never
> ok to drop packets unless x occurs" (such as a
> queue depth)? Is it possible?


What do you want to achieve with this thread?
1. Try to proof that Linux is inferior to *BSD?
2. Get help with your problem?


If your goal is 1. feel free to do so, but please do so on more 
appropriate lists.


If your goal is 2., you must help the people who are trying to help you 
with _your_ problem.

E.g. Patrick McHardy asked you:

<--  snip  -->

In that case please send more information, like both 2.4
and 2.6 configs, dmesg output from booting, lspci output,
other hardware details, ..

<--  snip  -->

Crystal balls are rare, and if your goal is really to get problem solved 
you should send the information other people require for debugging your 
problem.


> Danial

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: 2.6.12 Performance problems

[Danial Thom]

--- Adrian Bunk <bunk@stusta.de> wrote:

> On Mon, Aug 22, 2005 at 08:41:11AM -0700,
> Danial Thom wrote:
> >...
> > 
> > The issue I have with that logic is that you
> seem
> > to use "kernel" in a general sense without
> regard
> > to what its doing. Dropping packets is always
> > detrimental to the user regardless of what
> he's
> > using the computer for. An audio stream that
> > drops packets isn't going to be "smooth" at
> the
> > user level.
> 
> 
> That's not always true.
> 
> Imagine a slow computer with a GBit ethernet
> connection, where the user 
> is downloading files from a server that can
> utilize the full 
> network connection while listening to music
> from his local disk with 
> XMMS.
> 
> In this case, the audio stream is not depending
> on the network 
> connection. And the user might prefer dropped
> packages over a stuttering 
> XMMS.
> 
Audio connections are going to be windowed/flowed
in some way (thats how the internet works) so
you're not dealing with real capacity issues in
that example. You're not going to overrun your
ethernet adapter with an application; the issue
is running applications on servers that are also
doing a lot of other things. You can't isolate a
particular application at the device level, so
you drop packets randomly; not just ones you
don't care about. If your application can't run
well on your machine without dropping real
traffic, then you need a faster machine, not an
OS that compensates in a negative way.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Adrian Bunk]

On Fri, Aug 26, 2005 at 08:34:14AM -0700, Danial Thom wrote:
> 
> --- Adrian Bunk <bunk@stusta.de> wrote:
> > 
> > That's not always true.
> > 
> > Imagine a slow computer with a GBit ethernet
> > connection, where the user 
> > is downloading files from a server that can
> > utilize the full 
> > network connection while listening to music
> > from his local disk with 
> > XMMS.
> > 
> > In this case, the audio stream is not depending
> > on the network 
> > connection. And the user might prefer dropped
> > packages over a stuttering 
> > XMMS.

> Audio connections are going to be windowed/flowed
> in some way (thats how the internet works) so
>...

I was talking about an audio stream coming from a file on the
"local disk", IOW something like an mp3 file.

But the most interesting thing about your email is not what you were 
answering to, but which part of my email you silently omitted. Since you 
are not answering questions that might help to debug the problem you 
claim to have, it seems your intention is not getting a Linux problem 
fixed...

> DT

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: 2.6.12 Performance problems

[Danial Thom]

--- Adrian Bunk <bunk@stusta.de> wrote:

> On Fri, Aug 26, 2005 at 08:34:14AM -0700,
> Danial Thom wrote:
> > 
> > --- Adrian Bunk <bunk@stusta.de> wrote:
> > > 
> > > That's not always true.
> > > 
> > > Imagine a slow computer with a GBit
> ethernet
> > > connection, where the user 
> > > is downloading files from a server that can
> > > utilize the full 
> > > network connection while listening to music
> > > from his local disk with 
> > > XMMS.
> > > 
> > > In this case, the audio stream is not
> depending
> > > on the network 
> > > connection. And the user might prefer
> dropped
> > > packages over a stuttering 
> > > XMMS.
> 
> > Audio connections are going to be
> windowed/flowed
> > in some way (thats how the internet works) so
> >...
> 
> I was talking about an audio stream coming from
> a file on the
> "local disk", IOW something like an mp3 file.
> 
> But the most interesting thing about your email
> is not what you were 
> answering to, but which part of my email you
> silently omitted. Since you 
> are not answering questions that might help to
> debug the problem you 
> claim to have, it seems your intention is not
> getting a Linux problem 
> fixed...

I don't think I'm obligated to answer every
single person who pipes into a thread. People who
say "show me your config and dmesg" are not
useful. Linux has long had a philisophical
problem of dropping packets as a "performance
feature", and we've already established I think
that you can't eliminate it altogether, if you
read the thread carefully.

Also, if you're on a slow machine you can't
download faster than your machine can handle it,
because the "server on a gig network" can't send
faster then you tell it to, not matter how fast
it is. You'll never overrun a machine with 1 or 2
or 5 connections. Having a clue how things work
is important. 


DT


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Adrian Bunk]

On Fri, Aug 26, 2005 at 10:06:51AM -0700, Danial Thom wrote:
>...
> I don't think I'm obligated to answer every
> single person who pipes into a thread. People who
> say "show me your config and dmesg" are not
> useful. Linux has long had a philisophical
> problem of dropping packets as a "performance
> feature", and we've already established I think
> that you can't eliminate it altogether, if you
> read the thread carefully.
>...

You say you have observed for a long time a problem.

The only person participating in this thread who is one of the 
networking maintainers is Patrick McHardy.

Did _he_ say this was a known and unfixable problem?

He wanted to help you and you pissed him off because you refuxed to give 
him dmesg, .config and other information that would have helped him to 
debug your problem. If you don't feel obliged to help the persons 
responsible for the part of the kernel you have a problem with to debug 
your problem your whole initial mail was pointless.

> DT

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: 2.6.12 Performance problems

[Danial Thom]

--- Adrian Bunk <bunk@stusta.de> wrote:

> On Fri, Aug 26, 2005 at 10:06:51AM -0700,
> Danial Thom wrote:
> >...
> > I don't think I'm obligated to answer every
> > single person who pipes into a thread. People
> who
> > say "show me your config and dmesg" are not
> > useful. Linux has long had a philisophical
> > problem of dropping packets as a "performance
> > feature", and we've already established I
> think
> > that you can't eliminate it altogether, if
> you
> > read the thread carefully.
> >...
> 
> You say you have observed for a long time a
> problem.
> 
> The only person participating in this thread
> who is one of the 
> networking maintainers is Patrick McHardy.
> 
> Did _he_ say this was a known and unfixable
> problem?

I don't know. He said that the network stack  has
absolute priority, so perhaps he can explain why
compiling a kernel causes an exponential increase
in packet loss? 

The FreeBSD "network maintainers" are largely
clueless about just about every problem in 5.x,
so I'm not sure the title qualifies someone as
knowing all there is to know. I haven't heard him
claim that he can do the things I'm doing without
any receive errors or drops. I'd like to hear
about what test he runs to test this sort of
problem.

Before you can solve a problem you have to have a
clear understanding of what the problem is, and
admit that you have a problem. 
> 
> He wanted to help you and you pissed him off
> because you refuxed to give 
> him dmesg, .config and other information that
> would have helped him to 
> debug your problem. If you don't feel obliged
> to help the persons 
> responsible for the part of the kernel you have
> a problem with to debug 
> your problem your whole initial mail was
> pointless.

I didn't refuse. I just chose to take help from
Ben, because Ben took the time to reproduce the
problem and to provide useful settings that made
sense to me. There's nothing wrong with my
machine. 

I don't know who is who; I judge people based on
the intelligence of their analysis. People
responsible for the network stack may not be the
right people, because this is more likely a
scheduler issue. There's probably nothing wrong
with the stack or the drivers; the machine just
doesn't react fast enough to avert ring overruns.

DT



		
__________________________________ 
Yahoo! Mail 
Stay connected, organized, and protected. Take the tour: 
http://tour.mail.yahoo.com/mailtour.html 

Re: 2.6.12 Performance problems

[Ben Greear]

Danial Thom wrote:

> I didn't refuse. I just chose to take help from
> Ben, because Ben took the time to reproduce the
> problem and to provide useful settings that made
> sense to me. There's nothing wrong with my
> machine. 

Well, I didn't see the slowdown on my system when I tried 2.6
v/s 2.4.  You reported a 3x slowdown.  In your original mail,
you didn't mention trying to do compiles at the same time
as your network test.  I *did* see a pathetic slowdown (from 250kpps
bridging to about 6kpps getting through the bridge) when I coppied
a 512MB CDROM to the HD, but let's get the pure network slowdown
on your system figured out before we start looking at the impact
of disk IO.

So, if you see a 3x slowdown on an unloaded machine when going
from 2.4 to 2.6, then there is something unique about your system
and we should figure out what it is.

Also, my numbers (about 250kpps with moderate amount of drops)
was a lot better than the 100kpps you reported for 2.6.  My
hardware is different (P-IV, HT, 3.0Ghz 1MB Cache, 1GB RAM,
dual Intel pro/1000 NIC in PCI-X 66/133 slot), but I would not
expect that to be 2x faster than your Opteron (or whatever you had).

You could mention what you are using to generate traffic.  (I was
using pktgen to generate a stream of 60 byte pkts.)

I think you should give us some better information on exactly
what you are doing on 2.4 v/s 2.6.  And for heaven's sake,
don't hesitate to send kernel configs and hardware listings
to folks that ask.  Ruling out problems is as useful as finding
problems in this sort of thing.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: 2.6.12 Performance problems

[Danial Thom]

--- Ben Greear <greearb@candelatech.com> wrote:

> Danial Thom wrote:
> 
> > I didn't refuse. I just chose to take help
> from
> > Ben, because Ben took the time to reproduce
> the
> > problem and to provide useful settings that
> made
> > sense to me. There's nothing wrong with my
> > machine. 
> 
> Well, I didn't see the slowdown on my system
> when I tried 2.6
> v/s 2.4.  You reported a 3x slowdown.  In your
> original mail,
> you didn't mention trying to do compiles at the
> same time
> as your network test.  I *did* see a pathetic
> slowdown (from 250kpps
> bridging to about 6kpps getting through the
> bridge) when I coppied
> a 512MB CDROM to the HD, but let's get the pure
> network slowdown
> on your system figured out before we start
> looking at the impact
> of disk IO.
> 
> So, if you see a 3x slowdown on an unloaded
> machine when going
> from 2.4 to 2.6, then there is something unique
> about your system
> and we should figure out what it is.
> 
> Also, my numbers (about 250kpps with moderate
> amount of drops)
> was a lot better than the 100kpps you reported
> for 2.6.  My
> hardware is different (P-IV, HT, 3.0Ghz 1MB
> Cache, 1GB RAM,
> dual Intel pro/1000 NIC in PCI-X 66/133 slot),
> but I would not
> expect that to be 2x faster than your Opteron
> (or whatever you had).
> 
> You could mention what you are using to
> generate traffic.  (I was
> using pktgen to generate a stream of 60 byte
> pkts.)
> 
> I think you should give us some better
> information on exactly
> what you are doing on 2.4 v/s 2.6.  And for
> heaven's sake,
> don't hesitate to send kernel configs and
> hardware listings
> to folks that ask.  Ruling out problems is as
> useful as finding
> problems in this sort of thing.

Ok, good point, but I think  the original issue
has evolved into "why does linux drop packets
even when its not close to capacity". Its not
really a "slow down" now that we've been
discussing it, its the point at which packet loss
begins to occur. And since it appears that packet
loss can occur at usage levels not even close to
capacity, I can't really determine the capacity.
Since CPU usage also doesn't seem to register
correctly, I can't get a handle on the load for
various activities. So it may not have anything
to do with "slowness". I feel like I'm trying to
weigh a big blob of Jell-o that keeps falling off
the scale. 

The bottom line is that if I can't get it to stop
dropping packets, the performance is moot,
because its simply not suitable for my needs.

DT


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Danial Thom]

--- Jesper Juhl <jesper.juhl@gmail.com> wrote:

> On 8/21/05, Jesper Juhl <jesper.juhl@gmail.com>
> wrote:
> > On 8/21/05, Danial Thom
> <danial_thom@yahoo.com> wrote:
> > > >
> > > Ok, well you'll have to explain this one:
> > >
> > > "Low latency comes at the cost of decreased
> > > throughput - can't have both"
> > >
> > > Seems to be a bit backwards. Threading the
> kernel
> > > adds latency, so its the additional latency
> in
> > > the kernel that causes the drop in
> throughput. Do
> > > you mean that kernel performance has been
> > > sacrificed in order to be able to service
> other
> > > threads more quickly, even when there are
> no
> > > other threads to be serviced?
> > >
> > 
> > Ok, let me start with the way HZ influences
> things.
> > 
> [snip]
> 
> A small followup.
> 
> I'm not saying that the value of HZ or your
> preempt setting (whatever
> it may be) is definately the cause of your
> problem. All I'm saying is
> that it might be a contributing factor, so it's
> probably something
> that's worth a little bit of time testing.
> 
> In many cases people running a server on
> resonably new hardware with
> HZ=1000 and full preempt won't even notice, but
> that's depending on
> the load on the server and what jobs it has.
> For some tasks it
> matters, for some the differences in
> performance is negligible.
> 
> You problem could very well be something else
> entirely, but try a
> kernel build with PREEMPT_NONE and HZ=100 and
> see if it makes a big
> difference (or if that's your current config,
> then try the opposite,
> HZ=1000 and PREEMPT). If it does make a
> difference, then that's a
> valuable piece of information to report on the
> list. If it turns out
> it makes next to no difference at all, then
> that as well is relevant
> information as then people will know that HZ &
> preempt is not the
> cause and can focus on finding the problem
> elsewhere.
> 
Yes. Hz isn't going to make much difference on a
2.0Ghz opteron, but I can see how premption can
cause packet loss. Shouldn't packet processing be
the highest priority process? It seems pointless
to "keep the audio buffers full" if you're
dropping packets as a result. 

Also some clown typing on the keyboard shouldn't
cause packet loss. Trading network integrity for
snappy responsiveness is a bad trade.

Danial


		
__________________________________ 
Yahoo! Mail for Mobile 
Take Yahoo! Mail with you! Check email on your mobile phone. 
http://mobile.yahoo.com/learn/mail 

Re: 2.6.12 Performance problems

[Jesper Juhl]

On 8/21/05, Danial Thom <danial_thom@yahoo.com> wrote:
> 
> 
> --- Jesper Juhl <jesper.juhl@gmail.com> wrote:
> 
> > On 8/21/05, Jesper Juhl <jesper.juhl@gmail.com>
> > wrote:
> > > On 8/21/05, Danial Thom
> > <danial_thom@yahoo.com> wrote:
> > > > >
> > > > Ok, well you'll have to explain this one:
> > > >
> > > > "Low latency comes at the cost of decreased
> > > > throughput - can't have both"
> > > >
> > > > Seems to be a bit backwards. Threading the
> > kernel
> > > > adds latency, so its the additional latency
> > in
> > > > the kernel that causes the drop in
> > throughput. Do
> > > > you mean that kernel performance has been
> > > > sacrificed in order to be able to service
> > other
> > > > threads more quickly, even when there are
> > no
> > > > other threads to be serviced?
> > > >
> > >
> > > Ok, let me start with the way HZ influences
> > things.
> > >
> > [snip]
> >
> > A small followup.
> >
> > I'm not saying that the value of HZ or your
> > preempt setting (whatever
> > it may be) is definately the cause of your
> > problem. All I'm saying is
> > that it might be a contributing factor, so it's
> > probably something
> > that's worth a little bit of time testing.
> >
> > In many cases people running a server on
> > resonably new hardware with
> > HZ=1000 and full preempt won't even notice, but
> > that's depending on
> > the load on the server and what jobs it has.
> > For some tasks it
> > matters, for some the differences in
> > performance is negligible.
> >
> > You problem could very well be something else
> > entirely, but try a
> > kernel build with PREEMPT_NONE and HZ=100 and
> > see if it makes a big
> > difference (or if that's your current config,
> > then try the opposite,
> > HZ=1000 and PREEMPT). If it does make a
> > difference, then that's a
> > valuable piece of information to report on the
> > list. If it turns out
> > it makes next to no difference at all, then
> > that as well is relevant
> > information as then people will know that HZ &
> > preempt is not the
> > cause and can focus on finding the problem
> > elsewhere.
> >
> Yes. Hz isn't going to make much difference on a
> 2.0Ghz opteron, but I can see how premption can
> cause packet loss. Shouldn't packet processing be
> the highest priority process? It seems pointless
> to "keep the audio buffers full" if you're
> dropping packets as a result.
> 
> Also some clown typing on the keyboard shouldn't
> cause packet loss. Trading network integrity for
> snappy responsiveness is a bad trade.
> 

Since you choose to reply to a public list on a private email I guess
I should post the bit I'd [snip]'d above, so everyone else can get the
whole picture  :

On 8/21/05, Jesper Juhl <jesper.juhl@gmail.com> wrote:
> On 8/21/05, Danial Thom <danial_thom@yahoo.com> wrote:
> > >
> > Ok, well you'll have to explain this one:
> >
> > "Low latency comes at the cost of decreased
> > throughput - can't have both"
> >
> > Seems to be a bit backwards. Threading the kernel
> > adds latency, so its the additional latency in
> > the kernel that causes the drop in throughput. Do
> > you mean that kernel performance has been
> > sacrificed in order to be able to service other
> > threads more quickly, even when there are no
> > other threads to be serviced?
> >
> 
> Ok, let me start with the way HZ influences things.
> 
> The value of HZ was increased from 100 to 1000 in 2.6.x (and later
> made a config option so people can choose at compile time between 100,
> 250 & 1000).
> 
> This means that in 2.6 you have ten times as many timer interrupts
> happening every second compared to 2.4.  This is great for things that
> need to be served on short notice, like audio buffers that need to be
> filled *now* before the music skips and can't wait 10ms bacause the
> user will notice. But, all those interrupts add overhead and thus the
> total amount of work that can be done every second will be slightly
> lower even though response times are good.   For a desktop HZ=1000 is
> usually a very nice thing, but for a server that doesn't need good
> response times (low latency) HZ=100 or HZ=250 will provide better
> overall throughput and is often preferable.
> 
> It's more or less the same thing with kernel preemption. When the
> kernel is fully preemtible it means that kernel functions can be
> interrupted at (almost) any time if there happens to be a higher
> priority process/thread that is runnable. When the kernel is not
> preemtible, system calls etc always run to completion before being
> interrupted.
> 
> Preemption is great for interactive applications where you want user
> interfaces to respond fast, want multimedia applications to be able to
> refill buffers at regular intervals (with great precision on the
> timing) etc.  With preempt, such high applications will be able to
> interrupt/preempt other lower priority processes (like a background
> run of updatedb running from cron for example) even when those
> processes are in the middle of a system call, whereas without
> preemption they are forced to wait several milliseconds for the other
> apps syscall to complete before they can get a chance to run - which
> the user notices as sluggish gui response, music or video skips etc.
> 
> There are a few prices you pay for preemption.  1) extra overhead for
> bookkeeping (is it safe to preempt now, more locks need to be taken
> and dropped to make it safe etc).  2) preempted processes don't get to
> run for the entire duration of their timeslice in one go.  3) You get
> more context switch overhead - when the kernel has to switch between
> apps more often it needs to save and restore process context more
> often.
> All of this hurts overall throughput but gives you nice interactive
> response times (low latency).
> 
> So, you can't have both at once. You can't get maximum throughput at
> the same time as you get very low response times. It's one or the
> other depending on your need.
> 
> With 2.4 you have HZ=100, no preemption. So there you have a kernel
> optimized for maximum throughput, but latencies are high which is a
> pain for interactive and multimedia apps.
> 
> With 2.6, the default is HZ=1000 and preempt = PREEMPT_NONE  which is
> a compromise.
> But luckily you can modify the default.  As I already mentioned above,
> you can pick between HZ=100, 250 & 1000 these days, and the preemtion
> model gives you 3 choices :
> 
> PREEMPT_NONE - No Forced Preemption (Server)
> PREEMPT_VOLUNTARY - Voluntary Kernel Preemption (Desktop)
> PREEMPT - Preemptible Kernel (Low-Latency Desktop)
> 
> So you are free to mix and match to get the kernel behaviour that
> suits you best for any given task.
> 
> Personally I run (most) my servers with HZ=100 and PREEMPT_NONE
> and my personal desktop at home has HZ=1000 and PREEMPT.
> 
> 
> Did that make things a little more clear?
> 
> 


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

Re: 2.6.12 Performance problems

[Denis Vlasenko]

On Sunday 21 August 2005 23:21, Danial Thom wrote:
> > You problem could very well be something else
> > entirely, but try a
> > kernel build with PREEMPT_NONE and HZ=100 and
> > see if it makes a big
> > difference (or if that's your current config,
> > then try the opposite,
> > HZ=1000 and PREEMPT). If it does make a
> > difference, then that's a
> > valuable piece of information to report on the
> > list. If it turns out
> > it makes next to no difference at all, then
> > that as well is relevant
> > information as then people will know that HZ &
> > preempt is not the
> > cause and can focus on finding the problem
> > elsewhere.
>
> Yes. Hz isn't going to make much difference on a
> 2.0Ghz opteron, but I can see how premption can
> cause packet loss. Shouldn't packet processing be
> the highest priority process? It seems pointless
> to "keep the audio buffers full" if you're
> dropping packets as a result. 
> 
> Also some clown typing on the keyboard shouldn't
> cause packet loss. Trading network integrity for
> snappy responsiveness is a bad trade.

You do not need to argue about usefulness of preempt
(or lack thereof). You need to try non-PREEMPT kernel
as suggested (if you really are interested in fixing
performance degradation you observe, that is).

http://www.catb.org/~esr/faqs/smart-questions.html
--
vda

Re: 2.6.12 Performance problems

[Danial Thom]

--- Jesper Juhl <jesper.juhl@gmail.com> wrote:

> On 8/21/05, Danial Thom <danial_thom@yahoo.com>
> wrote:
> > I just started fiddling with 2.6.12, and
> there
> > seems to be a big drop-off in performance
> from
> > 2.4.x in terms of networking on a
> uniprocessor
> > system. Just bridging packets through the
> > machine, 2.6.12 starts dropping packets at
> > ~100Kpps, whereas 2.4.x doesn't start
> dropping
> > until over 350Kpps on the same hardware
> (2.0Ghz
> > Opteron with e1000 driver). This is pitiful
> > prformance for this hardware. I've
> > increased the rx ring in the e1000 driver to
> 512
> > with little change (interrupt moderation is
> set
> > to 8000 Ints/second). Has "tuning" for MP
> > destroyed UP performance altogether, or is
> there
> > some tuning parameter that could make a
> 4-fold
> > difference? All debugging is off and there
> are
> > no messages on the console or in the error
> logs.
> > The kernel is the standard kernel.org dowload
> > config with SMP turned off and the intel
> ethernet
> > card drivers as modules without any other
> > changes, which is exactly the config for my
> 2.4
> > kernels.
> > 
> 
> If you have preemtion enabled you could disable
> it. Low latency comes
> at the cost of decreased throughput - can't
> have both. Also try using
> a HZ of 100 if you are currently using 1000,
> that should also improve
> throughput a little at the cost of slightly
> higher latencies.
> 
> I doubt that it'll do any huge difference, but
> if it does, then that
> would probably be valuable info.
> 
Ok, well you'll have to explain this one:

"Low latency comes at the cost of decreased
throughput - can't have both"

Seems to be a bit backwards. Threading the kernel
adds latency, so its the additional latency in
the kernel that causes the drop in throughput. Do
you mean that kernel performance has been
sacrificed in order to be able to service other
threads more quickly, even when there are no
other threads to be serviced?

Danial



		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

2.6.12 Performance problems

[Danial Thom]

I just started fiddling with 2.6.12, and there
seems to be a big drop-off in performance from
2.4.x in terms of networking on a uniprocessor
system. Just bridging packets through the
machine, 2.6.12 starts dropping packets at
~100Kpps, whereas 2.4.x doesn't start dropping
until over 350Kpps on the same hardware (2.0Ghz
Opteron with e1000 driver). This is pitiful
prformance for this hardware. I've 
increased the rx ring in the e1000 driver to 512
with little change (interrupt moderation is set
to 8000 Ints/second). Has "tuning" for MP 
destroyed UP performance altogether, or is there
some tuning parameter that could make a 4-fold
difference? All debugging is off and there are 
no messages on the console or in the error logs.
The kernel is the standard kernel.org dowload
config with SMP turned off and the intel ethernet
card drivers as modules without any other
changes, which is exactly the config for my 2.4
kernels.

Danial


		
____________________________________________________
Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: 2.6.12 Performance problems

[Patrick McHardy]

Danial Thom wrote:
> I just started fiddling with 2.6.12, and there
> seems to be a big drop-off in performance from
> 2.4.x in terms of networking on a uniprocessor
> system. Just bridging packets through the
> machine, 2.6.12 starts dropping packets at
> ~100Kpps, whereas 2.4.x doesn't start dropping
> until over 350Kpps on the same hardware (2.0Ghz
> Opteron with e1000 driver). This is pitiful
> prformance for this hardware. I've 
> increased the rx ring in the e1000 driver to 512
> with little change (interrupt moderation is set
> to 8000 Ints/second). Has "tuning" for MP 
> destroyed UP performance altogether, or is there
> some tuning parameter that could make a 4-fold
> difference? All debugging is off and there are 
> no messages on the console or in the error logs.
> The kernel is the standard kernel.org dowload
> config with SMP turned off and the intel ethernet
> card drivers as modules without any other
> changes, which is exactly the config for my 2.4
> kernels.

Do you have netfilter enabled? Briging netfilter was
added in 2.6, enabling it will influence performance
negatively. Otherwise, is this performance drop
visible in other setups besides bridging as well?

Re: 2.6.12 Performance problems

[Danial Thom]

--- Patrick McHardy <kaber@trash.net> wrote:

> Danial Thom wrote:
> > I just started fiddling with 2.6.12, and
> there
> > seems to be a big drop-off in performance
> from
> > 2.4.x in terms of networking on a
> uniprocessor
> > system. Just bridging packets through the
> > machine, 2.6.12 starts dropping packets at
> > ~100Kpps, whereas 2.4.x doesn't start
> dropping
> > until over 350Kpps on the same hardware
> (2.0Ghz
> > Opteron with e1000 driver). This is pitiful
> > prformance for this hardware. I've 
> > increased the rx ring in the e1000 driver to
> 512
> > with little change (interrupt moderation is
> set
> > to 8000 Ints/second). Has "tuning" for MP 
> > destroyed UP performance altogether, or is
> there
> > some tuning parameter that could make a
> 4-fold
> > difference? All debugging is off and there
> are 
> > no messages on the console or in the error
> logs.
> > The kernel is the standard kernel.org dowload
> > config with SMP turned off and the intel
> ethernet
> > card drivers as modules without any other
> > changes, which is exactly the config for my
> 2.4
> > kernels.
> 
> Do you have netfilter enabled? Briging
> netfilter was
> added in 2.6, enabling it will influence
> performance
> negatively. Otherwise, is this performance drop
> visible in other setups besides bridging as
> well?
> 

Yes, bridging is clean. I also routed with the
same performance drop.

Danial


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: 2.6.12 Performance problems

[Patrick McHardy]

Danial Thom wrote:
> --- Patrick McHardy <kaber@trash.net> wrote:
>
>>Do you have netfilter enabled? Briging
>>netfilter was
>>added in 2.6, enabling it will influence
>>performance
>>negatively. Otherwise, is this performance drop
>>visible in other setups besides bridging as
>>well? 
> 
> Yes, bridging is clean. I also routed with the
> same performance drop.

In that case please send more information, like both 2.4
and 2.6 configs, dmesg output from booting, lspci output,
other hardware details, ..

Re: 2.6.12 Performance problems

[Andrew Morton]

Danial Thom <danial_thom@yahoo.com> wrote:
>
> I just started fiddling with 2.6.12, and there
> seems to be a big drop-off in performance from
> 2.4.x in terms of networking on a uniprocessor
> system. Just bridging packets through the
> machine, 2.6.12 starts dropping packets at
> ~100Kpps, whereas 2.4.x doesn't start dropping
> until over 350Kpps on the same hardware (2.0Ghz
> Opteron with e1000 driver). This is pitiful
> prformance for this hardware. I've 
> increased the rx ring in the e1000 driver to 512
> with little change (interrupt moderation is set
> to 8000 Ints/second). Has "tuning" for MP 
> destroyed UP performance altogether, or is there
> some tuning parameter that could make a 4-fold
> difference? All debugging is off and there are 
> no messages on the console or in the error logs.
> The kernel is the standard kernel.org dowload
> config with SMP turned off and the intel ethernet
> card drivers as modules without any other
> changes, which is exactly the config for my 2.4
> kernels.
> 

(added netdev to cc)