RFC: Nagle latency tuning

[Christopher Snook <csnook@redhat.com>]

Hey folks --

We frequently get requests from customers for a tunable to disable Nagle 
system-wide, to be bug-for-bug compatible with Solaris.  We routinely reject 
these requests, as letting naive TCP apps accidentally flood the network is 
considered harmful.  Still, it would be very nice if we could reduce 
Nagle-induced latencies system-wide, if we could do so without disabling Nagle 
completely.

If you write a multi-threaded app that sends lots of small messages across TCP 
sockets, and you do not use TCP_NODELAY, you'll often see 40 ms latencies as the 
network stack waits for more senders to fill an MTU-sized packet before 
transmitting.  Even worse, these apps may work fine across the LAN with a 1500 
MTU and then counterintuitively perform much worse over loopback with a 16436 MTU.

To combat this, many apps set TCP_NODELAY, often without the abundance of 
caution that option should entail.  Other apps leave it alone, and suffer 
accordingly.

If we could simply lower this latency, without changing the fundamental behavior 
of the TCP stack, it would be a great benefit to many latency-sensitive apps, 
and discourage the unnecessary use of TCP_NODELAY.

I'm afraid I don't know the TCP stack intimately enough to understand what side 
effects this might have.  Can someone more familiar with the nagle 
implementations please enlighten me on how this could be done, or why it 
shouldn't be?

-- Chris