Re: e1000 full-duplex TCP performance well below wire speed

["Kok, Auke" <auke-jan.h.kok@intel.com>]

Bruce Allen wrote:
> Hi Jesse,
> 
>>> It's good to be talking directly to one of the e1000 developers and
>>> maintainers.  Although at this point I am starting to think that the
>>> issue may be TCP stack related and nothing to do with the NIC.  Am I
>>> correct that these are quite distinct parts of the kernel?
>>
>> Yes, quite.
> 
> OK.  I hope that there is also someone knowledgable about the TCP stack
> who is following this thread. (Perhaps you also know this part of the
> kernel, but I am assuming that your expertise is on the e1000/NIC bits.)
> 
>>> Important note: we ARE able to get full duplex wire speed (over 900
>>> Mb/s simulaneously in both directions) using UDP.  The problems occur
>>> only with TCP connections.
>>
>> That eliminates bus bandwidth issues, probably, but small packets take
>> up a lot of extra descriptors, bus bandwidth, CPU, and cache resources.
> 
> I see.  Your concern is the extra ACK packets associated with TCP.  Even
> those these represent a small volume of data (around 5% with MTU=1500,
> and less at larger MTU) they double the number of packets that must be
> handled by the system compared to UDP transmission at the same data
> rate. Is that correct?


A lot of people tend to forget that the pci-express bus has enough bandwidth on
first glance - 2.5gbit/sec for 1gbit of traffix, but apart from data going over it
there is significant overhead going on: each packet requires transmit, cleanup and
buffer transactions, and there are many irq register clears per second (slow
ioread/writes). The transactions double for TCP ack processing, and this all
accumulates and starts to introduce latency, higher cpu utilization etc...

Auke