Re: [RFC] Make TCP prequeue configurable

[David Miller <davem@davemloft.net>]

From: John Heffner <jheffner@psc.edu>
Date: Thu, 27 Sep 2007 22:26:02 -0400

> I think it really does help in case (4) with old NICs that don't do rx 
> checksumming.  I'm not sure how many people really care about this 
> anymore, but probably some...?
> 
> OTOH, it would be nice to get rid of sysctl_tcp_low_latency.

I know most high end apps use poll() so won't sleep in recvmsg()
directly, but occasisionally they will, and even those that have a
poll() triggered recvmsg() will run the backlog and do prequeue if
packets arrive while they are processing the existing receive packets
which is quite common.

So for any app that ends up doing a prequeue it's a win because there
is the issue of scheduling and cpu usage charging.

If the ACK's are coming out of the stack at the rate that the
application can pull data out of the receive queue, and no faster,
this will pace the sender to send precisely how fast the receiver can
get onto the cpu depending upon load.

Furthermore, prequeue puts the stack input processing work into user
context, which means that the users will be charged more fairly for
the work that is done for them.

When packets get fully processed in softirq context, that's bad
because this is cpu usage which doesn't get charged to the user, and
for TCP input processing this cpu usage is non-trivial and is
multiplied by packet count.