Hi Jesse,

Thanks for the prompt reply.

> you could also try using the pktgen kernel module that does just what 
> you've said, but doesn't use the stack at all.  What packet size?
The payload size was set at 1470 bytes (to which I add the UDP, IP and 
ethernet header sizes when computing the throughput).

> myself and others will want to see your .config, and dmesg can you reply 
> with those attached?  I'm suspecting the scheduler myself.
Attached.

> are you sure some memory manager debug didn't get turned on?  A diff of 
> your previous config against the new config (or attach them both) might 
> be interesting.
Attached. The format is
CONFIG_VARIABLE: 2.6.25.10-value | 2.6.26-value
with '-' indicating the variable does not exist in that config file.

> This sounds a little bit like lock contention possibly on the skb_free 
> call in tx e1000_clean_tx_irq.
I still don't understand how this can result in soft IRQ time showing up 
on processors that do not handle the interrupts. It might be a 
HyperThreading glitch, though it shouldn't, as each logical processor 
has its own APIC, and the code does not distinguish between logical and 
physical processors (not at this level, anyway).

> have you tried both kernels without hyperthreading?  It usually doesn't 
> help with I/O workloads.
Actually it seems to help. My experiments evolve around hardware 
threads, so disabling HyperThreading is kind of counter-productive ;-)

> Jesse Brandeburg (1):
>       e1000: remove irq_sem
I'll see what happens when this code is re-enabled.

> you can check /proc/net/softnet_stat output to see how much you're NAPI 
> polling during these tests.  It looks like the interrupt rate is coming 
> down because you're polling in 26, but not in 25.
I'll give it a look.

> If you have the oprofile data, can you send me more details about the 
> hotspots with opannotate --assembly -p /path/to/e1000_source e1000
Attached.

Thanks again,
Elad