Re: Very low latency TCP for clusters

[Nivedita Singhvi <niv@us.ibm.com>]

Tom Herbert wrote:
> We have been looking at best case TCP latencies that might be achieved
> within a cluster (low loss fabric).  The goal is to have latency
> numbers roughly comparable to that which can be produced using RDMA/IB
> in a low latency configuration  (<5 usecs round trip on netperf TCP_RR
> test with one byte data for directly connected hosts as a starting
> point).  This would be without changing sockets API, fabric, and
> preferably not using TCP offload or a user space stack.

Over what media are you doing this? 10gbe? gbe? Whatever numbers
I've seen for latency have been superior on IB, and I'd be very
interested in any effort to get lower latencies over other transport.

> I think there are at least two techniques that will drive down TCP
> latency: per connection queues and polling queues.  Per connection
> queues (supported by device) should eliminate costs of connection
> look-up, hopefully some locking.  Polling becomes viable as core
> counts on systems increase, and burning a few CPUs for networking
> polling on behalf of very low-latency threads would be reasonable.

Have you got any profiling data that captures where your
particular latencies are? Also, have you tried a real-time
kernel?

thanks,
Nivedita