Re: Low performance Intel 10GE NIC (3.2.10) on 2.6.38 Kernel

[Alexander Duyck <alexander.h.duyck@intel.com>]

On 4/14/2011 10:49 AM, Eric Dumazet wrote:
> Le jeudi 14 avril 2011 à 18:57 +0200, Eric Dumazet a écrit :
>> Le jeudi 14 avril 2011 à 18:56 +0200, Peter Zijlstra a écrit :
>>> On Thu, 2011-04-14 at 09:42 -0700, Alexander Duyck wrote:
>>>
>>>> I'm doing some more digging into this now.  One thought that occurred to
>>>> me is that if the patch you mention is having some sort of effect this
>>>> could be a sign of perhaps a kernel timer or scheduling problem.
>>>
>>> Right, so the removal of the NO_HZ throttle will allow the CPU to go
>>> into C states more often, this could result in longer wake-up times for
>>> IRQs.
>>>
>>> We reverted because:
>>>    - it caused significant battery drain due to not going into C states
>>>      often enough, and
>>>    - its a much better idea to implement these things in the idle
>>>      governor since it already has the job of guestimating the idle
>>>      duration.
>>>
>>> I really can't remember back far enough to even come up with a theory of
>>> why kernels prior to merging the NO_HZ throttle would not exhibit this
>>> problem.
>>>
>>>
>>>
>>
>> Normally, Wei Gu already asked to not use C states.
>>
>> http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01804533/c01804533.pdf
>>
>> How can we/he check this ?
>>
>>
>
> Anyway, this could explain a latency problem, not packet drops.
>
> With NAPI, we should get few hardware irqs under load.
>
> Once softirq started, scheduler is out of the equation.

The problem is on these newer systems it is becoming significantly 
harder to get locked into the polling only state.  In many cases we will 
just complete all of the RX work in a single poll and go back to 
interrupts.  This is especially true when traffic is spread out across 
multiple queues and CPUs.

I'm thinking that maybe powertop results for before that patch and after 
that patch should be pretty telling.  It should tell us if C states are 
active, and if so it will also tell us if we are being woken by 
interrupts or if we are staying in the polling state.

Thanks,

Alex