Very low latency TCP for clusters

Very low latency TCP for clusters

[Tom Herbert]

We have been looking at best case TCP latencies that might be achieved
within a cluster (low loss fabric).  The goal is to have latency
numbers roughly comparable to that which can be produced using RDMA/IB
in a low latency configuration  (<5 usecs round trip on netperf TCP_RR
test with one byte data for directly connected hosts as a starting
point).  This would be without changing sockets API, fabric, and
preferably not using TCP offload or a user space stack.

I think there are at least two techniques that will drive down TCP
latency: per connection queues and polling queues.  Per connection
queues (supported by device) should eliminate costs of connection
look-up, hopefully some locking.  Polling becomes viable as core
counts on systems increase, and burning a few CPUs for networking
polling on behalf of very low-latency threads would be reasonable.

Are there any efforts in progress to integrate per connection queues
in the stack or integrate polling of queues?

Thanks,
Tom

Re: Very low latency TCP for clusters

[David Miller]

From: Tom Herbert <therbert@google.com>
Date: Mon, 19 Jul 2010 10:05:19 -0700

> Per connection queues (supported by device) should eliminate costs
> of connection look-up, hopefully some locking.

What are these per-connection queues exactly?

Is it like GRO and just accumulates in-order packets for a flow?

Or it is something more like Jacobson's net channels?

If it's the former, we have it already.  If it's the
latter we've found it to be utterly impractical due to all
of the facilities we have which live between the device
and the socket layer (netfilter, packet scheduler, IPSEC,
etc.)

Re: Very low latency TCP for clusters

[Eric Dumazet]

Le lundi 19 juillet 2010 à 10:05 -0700, Tom Herbert a écrit :
> We have been looking at best case TCP latencies that might be achieved
> within a cluster (low loss fabric).  The goal is to have latency
> numbers roughly comparable to that which can be produced using RDMA/IB
> in a low latency configuration  (<5 usecs round trip on netperf TCP_RR
> test with one byte data for directly connected hosts as a starting
> point).  This would be without changing sockets API, fabric, and
> preferably not using TCP offload or a user space stack.
> 
> I think there are at least two techniques that will drive down TCP
> latency: per connection queues and polling queues.  Per connection
> queues (supported by device) should eliminate costs of connection
> look-up, hopefully some locking.  Polling becomes viable as core
> counts on systems increase, and burning a few CPUs for networking
> polling on behalf of very low-latency threads would be reasonable.
> 
> Are there any efforts in progress to integrate per connection queues
> in the stack or integrate polling of queues?

aka "net channel" ;)

What a nightmare...

Anyway, 5 us roundtrip TCP_RR (including user thread work), seems a bit
utopic right now.

Even on loopback

Re: Very low latency TCP for clusters

[Tom Herbert]

On Mon, Jul 19, 2010 at 10:41 AM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le lundi 19 juillet 2010 à 10:05 -0700, Tom Herbert a écrit :
>> We have been looking at best case TCP latencies that might be achieved
>> within a cluster (low loss fabric).  The goal is to have latency
>> numbers roughly comparable to that which can be produced using RDMA/IB
>> in a low latency configuration  (<5 usecs round trip on netperf TCP_RR
>> test with one byte data for directly connected hosts as a starting
>> point).  This would be without changing sockets API, fabric, and
>> preferably not using TCP offload or a user space stack.
>>
>> I think there are at least two techniques that will drive down TCP
>> latency: per connection queues and polling queues.  Per connection
>> queues (supported by device) should eliminate costs of connection
>> look-up, hopefully some locking.  Polling becomes viable as core
>> counts on systems increase, and burning a few CPUs for networking
>> polling on behalf of very low-latency threads would be reasonable.
>>
>> Are there any efforts in progress to integrate per connection queues
>> in the stack or integrate polling of queues?
>
> aka "net channel" ;)
>
I don't think this is the same.  I am thinking of a device that
supports multi-queue where individual queues can be programming to
accept an exact 4-tuple, from the device's point of view I don't think
there's much beyond that and it is otherwise treated as just another
packet queue.  However, kernel may be able to use it to shortcut some
processing.  I believe such functionality is already supported in
Intel's flow director and possibly by some other vendors.

> What a nightmare...
>
I prefer to think of it as challenge, needing to resort to stateful
offload to get low latency would be the nightmare ;-)

> Anyway, 5 us roundtrip TCP_RR (including user thread work), seems a bit
> utopic right now.
>
> Even on loopback
>

I see about 7 usecs as best number on loopback, so I believe this is
in the ballpark.  As I mentioned above, this about "best case" latency
of a single thread, so we assume any amount of pinning or other
customized configuration to that purpose.

>
>
>

Re: Very low latency TCP for clusters

[David Miller]

From: Tom Herbert <therbert@google.com>
Date: Mon, 19 Jul 2010 11:44:05 -0700

> I am thinking of a device that supports multi-queue where individual
> queues can be programming to accept an exact 4-tuple,

NIU can already do this by programming the TCAM and segregating the
queues the TCAM entries map to.

There has also been talk of just having "a lot" of queues.  Maybe, on
the order of thousands.

Re: Very low latency TCP for clusters

[Eric Dumazet]

Le lundi 19 juillet 2010 à 11:44 -0700, Tom Herbert a écrit :

> I see about 7 usecs as best number on loopback, so I believe this is
> in the ballpark.  As I mentioned above, this about "best case" latency
> of a single thread, so we assume any amount of pinning or other
> customized configuration to that purpose.

Well, given I get 29 us on a ping between two machines (Gb link, no
process involved on receiver, only softirq), I really doubt we can reach
5 us on a tcp test involving a user process on both side ;)

$ ping -f -c 10000 -q 192.168.0.1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.

--- 192.168.0.1 ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 395ms
rtt min/avg/max/mdev = 0.029/0.032/0.434/0.010 ms, ipg/ewma 0.039/0.033
ms


On loopback, its a bit better of course, but there is no cache miss nor
scheduler involved.

$ ping -f -c 10000 -q 127.0.0.1
PING 127.0.0.1 (127.0.0.1) 56(84) bytes of data.

--- 127.0.0.1 ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 55ms
rtt min/avg/max/mdev = 0.002/0.002/0.015/0.001 ms, ipg/ewma 0.005/0.002
ms

Re: Very low latency TCP for clusters

[Tom Herbert]

On Mon, Jul 19, 2010 at 3:03 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
> Le lundi 19 juillet 2010 à 11:44 -0700, Tom Herbert a écrit :
>
>> I see about 7 usecs as best number on loopback, so I believe this is
>> in the ballpark.  As I mentioned above, this about "best case" latency
>> of a single thread, so we assume any amount of pinning or other
>> customized configuration to that purpose.
>
> Well, given I get 29 us on a ping between two machines (Gb link, no
> process involved on receiver, only softirq), I really doubt we can reach
> 5 us on a tcp test involving a user process on both side ;)
>
That's pretty pokey ;-) I see numbers around 25 usecs between to
machines, this is with TCP_NBRR.  With TCP_RR it's more like 35 usecs,
so eliminating the scheduler is already a big reduction.  That leaves
18 usecs in device time, interrupt processing, network, and cache
misses; 7 usecs in TCP processing, user space.  While 5 usecs is an
aggressive goal, I am not ready to concede that there's an
architectural limit in either NICs, TCP, or sockets that can't be
overcome.

Re: Very low latency TCP for clusters

[Eric Dumazet]

Le lundi 19 juillet 2010 à 16:37 -0700, Tom Herbert a écrit :

> That's pretty pokey ;-) I see numbers around 25 usecs between to
> machines, this is with TCP_NBRR.  With TCP_RR it's more like 35 usecs,
> so eliminating the scheduler is already a big reduction.  That leaves
> 18 usecs in device time, interrupt processing, network, and cache
> misses; 7 usecs in TCP processing, user space.  While 5 usecs is an
> aggressive goal, I am not ready to concede that there's an
> architectural limit in either NICs, TCP, or sockets that can't be
> overcome.

Last time I tried TCP_NBRR, it was not working (not even compiled in), I
guess I should submit a bug report to Rick ;)

Re: Very low latency TCP for clusters

[Rick Jones]

Eric Dumazet wrote:
> Le lundi 19 juillet 2010 à 16:37 -0700, Tom Herbert a écrit :
> 
>>That's pretty pokey ;-) I see numbers around 25 usecs between to
>>machines, this is with TCP_NBRR.  With TCP_RR it's more like 35 usecs,
>>so eliminating the scheduler is already a big reduction.  That leaves
>>18 usecs in device time, interrupt processing, network, and cache
>>misses; 7 usecs in TCP processing, user space.  While 5 usecs is an
>>aggressive goal, I am not ready to concede that there's an
>>architectural limit in either NICs, TCP, or sockets that can't be
>>overcome.
> 
> Last time I tried TCP_NBRR, it was not working (not even compiled in), I
> guess I should submit a bug report to Rick ;)

Indeed!-)  Actually, my first thought upon reading what Tom wrote was "Wow, I'm 
amazed it still works" :)  That code probably hasn't been visited much since the 
heydays of T/TCP (transactional).

Getting it compiled-in probably required a hand-editing of the config.h file 
after the ./configure.  Patches to add a --enable-nbrr would be graciously accepted.

happy benchmarking,

rick jones

Re: Very low latency TCP for clusters

[Brian Bloniarz]

Tom Herbert wrote:
> On Mon, Jul 19, 2010 at 3:03 PM, Eric Dumazet <eric.dumazet@gmail.com> wrote:
>> Le lundi 19 juillet 2010 à 11:44 -0700, Tom Herbert a écrit :
>>
>>> I see about 7 usecs as best number on loopback, so I believe this is
>>> in the ballpark.  As I mentioned above, this about "best case" latency
>>> of a single thread, so we assume any amount of pinning or other
>>> customized configuration to that purpose.
>> Well, given I get 29 us on a ping between two machines (Gb link, no
>> process involved on receiver, only softirq), I really doubt we can reach
>> 5 us on a tcp test involving a user process on both side ;)
>>
> That's pretty pokey ;-) I see numbers around 25 usecs between to
> machines, this is with TCP_NBRR.  With TCP_RR it's more like 35 usecs,
> so eliminating the scheduler is already a big reduction.  That leaves
> 18 usecs in device time, interrupt processing, network, and cache
> misses; 7 usecs in TCP processing, user space.  While 5 usecs is an
> aggressive goal, I am not ready to concede that there's an
> architectural limit in either NICs, TCP, or sockets that can't be
> overcome.

Have you toyed with the NIC's interrupt coalescing yet?
I'm wondering if any part of the 25usecs is that.

Re: Very low latency TCP for clusters

[Rick Jones]

Tom Herbert wrote:
> We have been looking at best case TCP latencies that might be achieved
> within a cluster (low loss fabric).  The goal is to have latency
> numbers roughly comparable to that which can be produced using RDMA/IB
> in a low latency configuration  (<5 usecs round trip on netperf TCP_RR
> test with one byte data for directly connected hosts as a starting
> point).  This would be without changing sockets API, fabric, and
> preferably not using TCP offload or a user space stack.
> 
> I think there are at least two techniques that will drive down TCP
> latency: per connection queues and polling queues.  Per connection
> queues (supported by device) should eliminate costs of connection
> look-up, hopefully some locking.  Polling becomes viable as core
> counts on systems increase, and burning a few CPUs for networking
> polling on behalf of very low-latency threads would be reasonable.

Likely preaching to the choir - but "just so long as it doesn't give the 
system's coherence fits."  Every once and again there are things stuck into the 
idle loop of various OSes on the premis that it is only burning cycles on that 
idle core, but ends-up trashing cache lines and/or the memory subsystem and so 
drags-down other cores.

Just how close to even 5 usecs/tran is the service demand on a TCP_RR test now? 
  The best I've seen for a 10GbE NIC under SLES11 SP1 (sorry, not latest 
upstream) has been 10-12.6 usec/tran, but the range went as high as 20 or more - 
depended on where netperf/netserver were running relative to the interrupt CPU:

ftp://ftp.netperf.org/netperf/misc/dl380g6_X5560_sles11sp1_ad386a_cxgb3_1.1.3-ko_b2b_to_same_1500mtu_20100602.csv
ftp://ftp.netperf.org/netperf/misc/dl380g6_X5560_sles11sp1_nc550_be2net_2.102.147s_b2b_to_same_1500mtu_20100520.csv

Getting rid of connection lookup and some locking will no doubt be necessary, 
but I suspect there will be a lot more to it as well.  Quite a few sacred 
path-length cows may have to be slaughtered along the way to get the service 
demand << 5 microseconds to allow the < 5 usec RTT.

happy benchmarking,

rick jones

Re: Very low latency TCP for clusters

[Nivedita Singhvi]

Tom Herbert wrote:
> We have been looking at best case TCP latencies that might be achieved
> within a cluster (low loss fabric).  The goal is to have latency
> numbers roughly comparable to that which can be produced using RDMA/IB
> in a low latency configuration  (<5 usecs round trip on netperf TCP_RR
> test with one byte data for directly connected hosts as a starting
> point).  This would be without changing sockets API, fabric, and
> preferably not using TCP offload or a user space stack.

Over what media are you doing this? 10gbe? gbe? Whatever numbers
I've seen for latency have been superior on IB, and I'd be very
interested in any effort to get lower latencies over other transport.

> I think there are at least two techniques that will drive down TCP
> latency: per connection queues and polling queues.  Per connection
> queues (supported by device) should eliminate costs of connection
> look-up, hopefully some locking.  Polling becomes viable as core
> counts on systems increase, and burning a few CPUs for networking
> polling on behalf of very low-latency threads would be reasonable.

Have you got any profiling data that captures where your
particular latencies are? Also, have you tried a real-time
kernel?

thanks,
Nivedita

Re: Very low latency TCP for clusters

[Mitchell Erblich]

Tom, et al,

		To possibly remove 3-way handshake costs for
		small data transfers a number of years ago, their
		was an experimental RFC that was a Transactional
		TCP, where the SYN, data, and FIN were sent as
		a tuple in the first segmernt.

		If a company has control of the two end systems
		and the intermediate systems, then it should be
		possible for an app to declare in a setsockopt call
		this experimental flow.
		
		Mitchell Erblich
		============

On Jul 19, 2010, at 10:05 AM, Tom Herbert wrote:

> We have been looking at best case TCP latencies that might be achieved
> within a cluster (low loss fabric).  The goal is to have latency
> numbers roughly comparable to that which can be produced using RDMA/IB
> in a low latency configuration  (<5 usecs round trip on netperf TCP_RR
> test with one byte data for directly connected hosts as a starting
> point).  This would be without changing sockets API, fabric, and
> preferably not using TCP offload or a user space stack.
> 
> I think there are at least two techniques that will drive down TCP
> latency: per connection queues and polling queues.  Per connection
> queues (supported by device) should eliminate costs of connection
> look-up, hopefully some locking.  Polling becomes viable as core
> counts on systems increase, and burning a few CPUs for networking
> polling on behalf of very low-latency threads would be reasonable.
> 
> Are there any efforts in progress to integrate per connection queues
> in the stack or integrate polling of queues?
> 
> Thanks,
> Tom
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Very low latency TCP for clusters

[Tom Herbert]

On Mon, Jul 19, 2010 at 12:46 PM, Mitchell Erblich
<erblichs@earthlink.net> wrote:
> Tom, et al,
>
>                To possibly remove 3-way handshake costs for
>                small data transfers a number of years ago, their
>                was an experimental RFC that was a Transactional
>                TCP, where the SYN, data, and FIN were sent as
>                a tuple in the first segmernt.
>
In the context of this discussion, latency is about how fast packets
can be sent and received between applications over an established
connection.  The application would likely be doing some form of RPC
over long lived connections,  reducing latency here allows for more
tightly coupled computing models.  This sort of thing would be used on
a well connected, low-loss cluster within a data center.

Separately, there is interest in speeding up 3WHS primarily for
connections to the Internet.  We have already proposed to reduce
latency by raising the the initial congestion window (there's another
netdev thread on this if your interested).  T/TCP, which your
referring to, was abandoned due to unsolved security issues and
susceptibility to denial of service attacks.  I believe the current
IETF answer to this would be to use SCTP, but that represents a
deployment conundrum.  Address these issues to get data with the TCP
SYN is still compelling IMO.

Tom


>                If a company has control of the two end systems
>                and the intermediate systems, then it should be
>                possible for an app to declare in a setsockopt call
>                this experimental flow.
>
>                Mitchell Erblich
>                ============
>
> On Jul 19, 2010, at 10:05 AM, Tom Herbert wrote:
>
>> We have been looking at best case TCP latencies that might be achieved
>> within a cluster (low loss fabric).  The goal is to have latency
>> numbers roughly comparable to that which can be produced using RDMA/IB
>> in a low latency configuration  (<5 usecs round trip on netperf TCP_RR
>> test with one byte data for directly connected hosts as a starting
>> point).  This would be without changing sockets API, fabric, and
>> preferably not using TCP offload or a user space stack.
>>
>> I think there are at least two techniques that will drive down TCP
>> latency: per connection queues and polling queues.  Per connection
>> queues (supported by device) should eliminate costs of connection
>> look-up, hopefully some locking.  Polling becomes viable as core
>> counts on systems increase, and burning a few CPUs for networking
>> polling on behalf of very low-latency threads would be reasonable.
>>
>> Are there any efforts in progress to integrate per connection queues
>> in the stack or integrate polling of queues?
>>
>> Thanks,
>> Tom
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>