Re: [RFC PATCH 1/2] net: Add new network device function to allow for MMIO batching

[Eric Dumazet <eric.dumazet@gmail.com>]

On Thu, 2012-07-12 at 08:39 -0700, Alexander Duyck wrote:

> The problem is in both of the cases where I have seen the issue the
> qdisc is actually empty.
> 

You mean a router workload, with links of same bandwidth.
(BQL doesnt trigger)

Frankly what percentage of linux powered machines act as high perf
routers ?

> In the case of pktgen it does not use the qdisc layer at all.  It just
> directly calls ndo_start_xmit.

pktgen is in kernel, adding a complete() call in it is certainly ok,
if we can avoid kernel bloat.

I mean, pktgen represents less than 0.000001 % of real workloads.

> 
> In the standard networking case we never fill the qdisc because the MMIO
> write stalls the entire CPU so the application never gets a chance to
> get ahead of the hardware.  From what I can tell the only case in which
> the qdisc_run solution would work is if the ndo_start_xmit was called on
> a different CPU from the application that is doing the transmitting.

Hey, I can tell that qdisc is not empty on many workloads.
But BQL and TSO mean we only send one or two packets per qdisc run.

I understand this MMIO batching helps routers workloads, or workloads
using many small packets.

But on other workloads, this adds a significant latency source
(NET_TX_SOFTIRQ)

It would be good to instrument the extra delay on a single UDP send.

(entering do_softirq() path is not a few instructions...)