Re: Strange CPU load when flushing route cache (kernel 2.6.31.6)

[Jesper Dangaard Brouer <jdb@comx.dk>]

On Mon, 2009-11-23 at 11:29 +0100, Eric Dumazet wrote:
> Jesper Dangaard Brouer a écrit :
> > Hi Eric and netdev,
> > 
> > I have observed a strange route cache behaviour when I upgraded some
> > of my production Linux routers (1Gbit/s tg3) to kernel 2.6.31.6 (from
> > kernel 2.6.25.7).
> > 
> > Every time the route cache is flushed I get a CPU spike (in softirq)
> > with a tail.  I have attached some graphs that illustrate the issue
> > (hope vger.kernel.org will allow these attachments...)
> > 
> > 
> > I have done some tuning of the route cache:
> > 
> >  # From /etc/sysctl.conf
> >  #
> >  # Adjusting the route cache flush interval
> >  net/ipv4/route/secret_interval = 1200
> > 
> >  # Limiting the route cache size
> >  # ip_dst_cache slab objects is 256 bytes.
> >  # 2000000 * 256 bytes = 512 MB
> >  net/ipv4/route/max_size = 2000000
> > 
> > Boot parameters: "rhash_entries=262143 vmalloc=256M"
> > 
> > The rhash_entries is for the route cache hash size.  The vmalloc is
> > needed because I have _very_ large iptables rulesets (and is running
> > on a 32-bit kernel, due to old hardware).
> > 
> > Any thoughs on how to avoid these CPU spikes?
> > Or where the issue occurs in the code?
> > 
> 
> Sure, after a flush, we have to rebuild the cache, so extra work is expected.

But the old 2.6.25.7 do NOT show this behavior... That is the real
issue...

> (We receive a packet, notice the cached entry is obsolete, free it, allocate a new one
> and inert it into cache)
> 
> If you dont want these spikes, just dont flush cache :)

I did the cache flushing due to some historical issues, that I think you
did a fix for... Guess I can drop the flushing and see if the garbage
collection can keep up...

> Do you run a 2G/2G User/Kernel split kernel ?

Not sure, how do I check?

I do use a 32-bit kernel (due to the production machines runs an old
32-bit Slackware OS install and some of the machines cannot run 64-bit).

-- 
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network Kernel Developer
  Cand. Scient Datalog / MSc.CS
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer