Re: >10% performance degradation since 2.6.18

Jeff Garzik
2009-07-04T09:22:34Z

Andi Kleen wrote:
>> for networking, especially for incoming data such as new connections,
>> that isn't the case.. that's more or less randomly (well hash based)
>> distributed.
>
> Ok. Still binding them all to a single CPU all is quite dumb. It
> makes MSI-X quite useless and probably even harmful.
>
> We don't default to socket power saving for normal scheduling either,
> but only when you specify a special knob. I don't see why interrupts
> should be different.

In the pre-MSI-X days, you'd have cachelines bouncing all over the place
if you distributed networking interrupts across CPUs, particularly given
that NAPI would run some things on a single CPU anyway.

Today, machines are faster, we have multiple interrupts per device, and
we have multiple RX/TX queues. I would be interested to see hard
numbers (as opposed to guesses) about various new ways to distributed
interrupts across CPUs.

What's the best setup for power usage?
What's the best setup for performance?
Are they the same?
Is it most optimal to have the interrupt for socket $X occur on the same
CPU as where the app is running?
If yes, how to best handle when the scheduler moves app to another CPU?
Should we reprogram the NIC hardware flow steering mechanism at that point?

Interesting questions, and I hope we'd see some hard number comparisons
before solutions start flowing into the kernel.

Jeff