Re: RFC: possible NAPI improvements to reduce interrupt rates for low traffic rates

[jamal <hadi@cyberus.ca>]

On Wed, 2007-12-09 at 14:50 +0100, James Chapman wrote:

> By low traffic, I assume you mean a rate at which the NAPI driver 
> doesn't stay in polled mode. 

i.e:
"one interupt per packet per napi poll" which cause about 1-2 more IOs
in comparison to the case where you didnt do NAPI.

> The problem is that that rate is getting 
> higher all the time, as interface and CPU speeds increase. 

indeed;
While i dont want to throw more work at you, with some of the things
that improve the IO cost like PCI express, MSI, and some of the
intelligent things the tg3 does, is this problem still rampant etc? I
think if you can find (seems you have) one "modern" machine (with MSI
and a tg3 etc) that has this problem circa 2007 that will be a good
start.

> This results 
> in too many interrupts and NAPI thrashing in/out of polled mode very 
> quickly.

indeed.

> Yes please. We need an analysis of what happens to cpu usage, latency, 
> pps etc when various factors are changed, e.g. input pps, NAPI busy-idle 
> delay etc. The main purpose of my RFC wasn't to push a patch into the 
> kernel right now, it was to highlight the issue and to find out if 
> others were already working on it. The feedback has been good so far. I 
> just need to find some time to do some testing. :)

I love your message. From a blackbox perspective, yes we have some
challenges for NAPI below certain thresholds of traffic. 
My claim (in the paper) was the discrepancy between the cost of IO
access vs cost of RAM vs cost of caches vs CPU speeds has gotten too
high.
CPU Vendors have been paying close attention to most but IO. So avoiding
IO when you can is a good thing.

> Jamal, do you have more details? Are people saying NAPI gets too much of 
> the CPU pie because they profiled it?

In the old days Manfred Spraul actually did profile.
Most of the other folks were running benchmarks which account for cpu
use in addition to resources like bandwidth and latency. And so while
bandwidth and latency didnt affect them that much, they observed their
benchmarks didnt look good at low rates (even when they looked excellent
at high rates) because of CPU.

>  Are they complaining that system 
> behavior degrades too much under certain network traffic conditions?

yes - Under low traffic, high speed cpu youd notice a slightly higher
cpu use.

>  
> Mouse cursor movement jittery? Real-time apps such as music/video 
> players starved of CPU? Is it possible they blame NAPI because they see 
> tangible effects on their system, not because measured CPU usage is 
> high? 

If i recall correctly transactional type benchmarks is where this was
observed. Some IBM and Intel people bring it up every few months and
maybe Rick Jones once in a while.
Rick, care to comment on the benchmarks?

> I say this because my music/video player and mouse cursor behave 
> _much_ better with my NAPI changes during general use, despite the 
> increase in measured cpu load. Even ftp can make my system's mouse 
> cursor jitter...

Like i told you in my other email - i did notice something similar, i
just couldnt put my finger to it and at some points thought i was
imagining it.

cheers,
jamal