Re: [PATCH] Add TCP_NO_DELAYED_ACK socket option

[Rick Jones <rick.jones2@hp.com>]

>> If the networks where this happens are indeed truly private, can they run a
>> private kernel?  Or use an LD_PRELOAD hack to wedge-in a
>> setsockopt(TCP_NODELAY) call into the application?  Or set something like
>> tcp_naglim_def on the application system(s)?  Or have the server application
>> make a setsockopt(TCP_MAXSEG) call before listen() to a value one byte below
>> that of what the application is sending?
>
> We control our server.  We don't control the server at the other end.
> We've tried to get them to do any of the above, but they seem
> unwilling or unable to do it.  I suspect that they're using various
> pieces from various third-party vendors that just don't care.

Making the setsockopt(TCP_MAXSEG) would be at your end :)  Presumably 
based on the minimum message size.  That would cause the connection to 
have an MSS == the request size so every request send should take the 
"is this send plus any queued unsent data >= MSS" path.

Another "at your end" possibility would be setting a rather small 
SO_RCVBUF size at your end before calling listen(), in hopes of 
triggering the window update.

>> Is the application actually "virtuous" in sending logically associated data
>> in one "send" call, and simply running afoul of Nagle+DelayedACK in having
>> multiple distinct requests outstanding at once, or is it actually quite evil
>> in that it is sending logically associated data in separate send calls?
>>
>
> The remote application generates messages meant for us, and they
> appear to send each message in its own segment.  I don't have the
> source, so I don't know whether they're really using one send call per
> message or whether they're using MSG_MORE, TCP_CORK, so some other
> mechanism.  Each message is time-sensitive and should be received as
> soon as possible afterq its sent (i.e. one-half rtt).  Unfortunately,
> when they send two messages and we don't ack the first one, the second
> gets delayed.  Turning off delayed acks helps but does not completely
> solve the problem.

If it is write,write,read  (multiple sends per logical message) in a 
packet trace you should see a partial request in the first segment, 
followed by the rest of the request  (and perhaps the second through 
Nth) in the second segment.  Or, I suppose your server application would 
have a receive complete with the first part of the first request, 
getting the second part of the request in a subsequent receive call.

If it is multiple requests at a time each sent in one send call, you 
should see a first segment arriving with a complete request within it, 
followed by a second segment with the next request(s).

rick jones