Re: miserable performance of 2.6.21 under network load

[Chris Snook <csnook@redhat.com>]

Aaron Porter wrote:
> 	I'm in the process up upgrading a pool of apache servers from
> 2.6.17.8 to 2.6.21.5, and we're seeing a pretty major change in behavior.
> Under identical network load, 2.6.21 has a load average more than 3 times
> higher, cpu 0 spends well over 90% of its time in interrupts (vs ~30%
> under 2.6.17). When we hit 3k apache sessions, ksoftirqd eats 100% of cpu0
> and our network traffic drops off rapidly. The end result is that 2.6.17
> performs twice as well under this load.

Is it always CPU 0, or does it move?  Are you running irqbalance?  If you're 
running irqbalance, you can run a script that alternates between 'cat 
/proc/interrupts' and 'mpstat -P ALL 5 10' and watch the offending interrupt 
jump around between processors.  It's not as informative as oprofile, as Andi 
suggested, but it's really easy to set up.

	-- Chris