Re: Detecting TCP loss on the receiving side?

[Dan Noe <dnoe@limebrokerage.com>]

On 07/10/2008 05:20 PM, Rick Jones wrote:
> John Heffner wrote:
>> Looking for loss at the receiver is a bit tricky.  It doesn't look
>> like struct tcp_info has enough information to do this easily.  If you
>> are able to install a custom kernel on this machine, the Web100 patch
>> would be able to gather enough information to figure it out.  The
>> basic idea would be to look for a difference between RcvNxt and
>> RcvMax.
>
> And even then it depends on the connections having multiple segments
> in flight at one time.  Although I suppose that cuts both ways and
> affects the tracing too, but perhaps not to the same extent.
>
> Dan - seeing "brokerage" in your email and worries about latency makes
> me think that your app(s) are pushing around lots of small messages -
> are those spread-out across lots of connections, or are they
> consolidated into a rather smaller number of connections?  Also, what
> is the magnitude of the latency in these latency events?

Yeah, without going into too much detail:  We are getting data from a
market via a TCP connection.  Usually this is one or two connections. 
It is dominated by receiving traffic (from peer).  The message rates can
be very bursty, and the size is small.  The links are usually short and
fairly fat - metro fast Ethernet, for example, and the geographic
latency is low.  We believe loss is occurring but we're trying to
determine how often and where.  The data comes on fairly fast and a loss
event (even with fast retransmit) means things piling up in queues and a
sudden flood of data coming into userspace (which is a pretty serious
latency hit on this scale).

More and more places are moving to multicast (which has its own quirks)
but some are still using TCP :)

Cheers,
Dan