Re: miserable performance of 2.6.21 under network load

[Chris Snook <csnook@redhat.com>]

Aaron Porter wrote:
> On Tue, Jul 24, 2007 at 08:48:00PM +0200, Andi Kleen wrote:
>> Aaron Porter <atporter@primate.net> writes:
>>
>>> 	I'm in the process up upgrading a pool of apache servers from
>>> 2.6.17.8 to 2.6.21.5, and we're seeing a pretty major change in behavior.
>>> Under identical network load, 2.6.21 has a load average more than 3 times
>>> higher, cpu 0 spends well over 90% of its time in interrupts (vs ~30%
>>> under 2.6.17). When we hit 3k apache sessions, ksoftirqd eats 100% of cpu0
>>> and our network traffic drops off rapidly. The end result is that 2.6.17
>>> performs twice as well under this load.
>> Can you oprofile it?
> 
> # opreport -l
> CPU: AMD64 processors, speed 1994.52 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
> samples  %        app name                 symbol name
> 914379   48.8404  vmlinux-2.6.21.5         check_poison_obj
> 341920   18.2632  vmlinux-2.6.21.5         poison_obj

I bet you have CONFIG_DEBUG_SLAB turned off in your 2.6.17 kernel, and turned on 
in your 2.6.21 kernel.

	-- Chris