Re: _High_ CPU usage while routing (mostly) small UDP packets

[Karsten Desler <kdesler@soohrt.org>]

* P@draigBrady.com wrote:
> Karsten Desler wrote:
> >* David S. Miller wrote:
> >
> >>It's spending nearly half of it's time in iptables.
> >>Try to consolidate your rules if possible.  This is the
> >>part of netfilter that really doesn't scale well at all.
> >>
> >
> >Removing the iptables rules helps reducing the load a little, but the
> >majority of time is still spent somewhere else.
> 
> Well with NAPI it can be hard to tell CPU usage.
> You may need to use something like cyclesoak to get
> a true idea of CPU left.

Thanks, I'll look into it.

> Also have a look at http://www.hipac.org/ as netfilter
> has silly scalability properties.

I did before, but I read on Harald Weltes' weblog that 2.4 gives
slightly worse pps results than 2.6, and since the cpu usage is as high
as it is, I didn't want to take any more performance hits.
I'll try to see what performance impact the netfilter rules have during
peak load.

> I also notice that a lot of time is spent allocating
> and freeing the packet buffers (and possible hidden
> time due to cache misses due to allocating on one
> CPU and freeing on another?).
> How many [RT]xDescriptors do you have configured by the way?

256. I increased them to 1024 shortly after the profiling run, but
didn't notice any change in the cpu usage (will try again with cyclesoak).

> Anyway attached is a small patch that I used to make the e1000
> "own" the packet buffers, and hence it does not alloc/free
> per packet at all. Now this has only been tested in one
> configuration where I was just sniffing the packets, so
> definitely YMMV.

Thanks, I'll give it a spin.

Cheers,
 Karsten