Re: Route cache performance under stress

[Simon Kirby <sim@netnation.com>]

On Sun, Jun 08, 2003 at 07:55:58PM -0400, CIT/Paul wrote:

> A denial of service attack going on, OR you just have millions of hosts
> going through the router (if you were an ISP).  Anything with seeminly
> Random source ips (something like juno-z.101f.c will generate worst case
> scenario for forwarding packets) will cause the cache to constantly
> Add new entries at pretty much the rate of the attack.. This can stifle
> just about any linux router with a measly 10 megabits/second of traffic
> unless
> The router is tuned up to a large degree (NAPI, certain nics, route
> cache timings, etc.) and even then it can still be destroyed no matter
> what
> The cpu is with less than 100,000 packets per second and in mosts cases
> less than 30k.. That's why it's just no acceptable for companies using
> it as a replacement for say a cisco 7200 VXR series (npe300,400 nsf-1,
> etc.) which can do 300K+ packet per second of routing (and yes it can
> even route juno-z.101f.c at 300kpps, I have tested it).   Linux has no
> problem doing 300kpps from a single source to a single destination
> provided you have NAPI or ITR or something limiting the interrupts.. The
> overhead is the route cache and the related systems that use it and also
> netfilter is very slow :/  One of these days they will fix it..... If

Whoa, wait a second.

You got a 7200 VXR to do 300kpps?  I would have liked to see that.
We couldn't get our 7206 VXR routers to do anything more than about 12
Mbit/second of small packets, which I believe is about 40,000 packets
per second.  This is with CEF disabled, because it ended up duplicating
packets and doing some other strange things with CEF enabled.

Also, I remember trying with a bucketload of netfilter rules and finding
that the performance difference was hardly noticeable.

Linux can route small packets with random src/dst at much faster than 10
Mbit/sec.  It depeends on the hardware as you say, but it shouldn't ever
be that slow on reasonable hardware.

I remember back even in 1998 with the 2.0 kernel (before the route cache
existed) on a Celeron 300A with eepro100 cards (eepro100 driver, no
interrupt coalescing, definitely no NAPI) was cable of routing at least
20 Mbit/second of SYN packets from random sources.  In fact, I remember
it happily choking some old 3Com switches we had at the time.

I recently saw 90 Mbit/second of additional traffic (small packets with
random sources) going through our routers (now single Athlon 1800MP (MP
for APIC), tg3, NAPI, BGP routing tables), and they didn't seem to care. 
It's definitely not yet perfect, but it's not bad.  The hashing fixes for
large routing tables which Dave M. recently posted has made the situation
much better -- it was very broken before.  What did your routing table
look like when you were doing tests?

I have fiddled with the route cache garbage collection parameters a bit,
but I haven't really been able to reduce the CPU usage by much at all. 
Really, though, shouldn't the route cache overhead be fairly small in
comparison to everything else involved in forwarding?

Simon-