Minimizing TCP latency

Minimizing TCP latency

[Ben Hoyt]

Oops, very sorry, I replied to an existing message but forgot to edit
the subject line to start a new thread. Try again:

Hi netdev,

We're working on some TCP networking stuff where the key is low
latency -- in other words, we don't really care about data throughput,
but rather getting the end-to-end TCP packet transfer time down.

I know there's quite a few networking folks and kernel hackers on this
list, so I thought I'd see if anyone was interested (or knows someone
who might be interested) in doing a few hours of consultancy work to
give us some pointers on this. I'm a competent programmer, but
kernel-level network programming is not my area of expertise.

I'm happy to receive some pointers via this list -- and others might
benefit from that too -- but I'm also keen on some dedicated
consultancy hours if anyone's interested.

Thanks,
Ben.

Re: Minimizing TCP latency

[Ben Hoyt]

[In reply to David Miller:]
> First, please don't hijack an unrelated thread to start talking about
> something different.

I'm sorry, my mistake -- I apologized in the new thread.

> Second, the mailing lists at vger.kernel.org are not for making
> employment of consultancy offers.

Fair enough. Though another netdev reader advised me that this would
be a good place to ask.

I know brevity is the soul of wit, but it's not the most encouraging
for a list newbie to be told, "You're doing it wrong, and you're in
the wrong place." You never know, I may have something to contribute
after all is said and done. :-)

So, if this isn't the place to ask, do you have any pointers in the
right direction?

-Ben

Re: Minimizing TCP latency

[Ben Hoyt]

> I'm happy to receive some pointers via this list [...]

Sorry, I should also have posted more about what I've done in terms of
reducing latency so far:

* We've already got TCP_NODELAY activated.

* We've also turned off interrupt coalescing on the network card (for
example, see http://www.29west.com/docs/THPM/latency-interrupt-coalescing.html).

* We've modified our process's CPU affinity so it runs on a core by
itself, reducing interrupt-handling latency slightly further.

* Our code is written in C, and we've tried to keep it as close to the
kernel's socket calls as possible.

So we're currently looking further into ways to reduce latency, which
currently looks like it'll mean digging deeper into the kernel's
networking options and innards.

FYI, we're using RHE release 4 and kernel version 2.6.9 on x86_64 machines.

-Ben

Re: Minimizing TCP latency

[Eric Dumazet]

Le lundi 29 mars 2010 à 16:24 +1300, Ben Hoyt a écrit :
> > I'm happy to receive some pointers via this list [...]
> 
> Sorry, I should also have posted more about what I've done in terms of
> reducing latency so far:
> 
> * We've already got TCP_NODELAY activated.

One machine sends messages, and messages are not aknowledged by other
part ? Is it one way communication ?

> 
> * We've also turned off interrupt coalescing on the network card (for
> example, see http://www.29west.com/docs/THPM/latency-interrupt-coalescing.html).
> 
> * We've modified our process's CPU affinity so it runs on a core by
> itself, reducing interrupt-handling latency slightly further.
> 
> * Our code is written in C, and we've tried to keep it as close to the
> kernel's socket calls as possible.
> 



So we're currently looking further into ways to reduce latency, which
> currently looks like it'll mean digging deeper into the kernel's
> networking options and innards.
> 
> FYI, we're using RHE release 4 and kernel version 2.6.9 on x86_64 machines.

What are the targets ?  (time constraints)

Mono threaded application handling one tcp flow ?

Are you handling a very light load on few flows, or many flows ? 

Are machines all in the same LAN ?

Re: Minimizing TCP latency

[Neil Horman]

On Mon, Mar 29, 2010 at 04:24:04PM +1300, Ben Hoyt wrote:
> > I'm happy to receive some pointers via this list [...]
> 
> Sorry, I should also have posted more about what I've done in terms of
> reducing latency so far:
> 
> * We've already got TCP_NODELAY activated.
> 
> * We've also turned off interrupt coalescing on the network card (for
> example, see http://www.29west.com/docs/THPM/latency-interrupt-coalescing.html).
> 
Careful, this might wind up hurting you.  It decreases latency from the card,
but might result in flooding one or more cores with additional interrupts that
delay processing of packets further down the line.

> * We've modified our process's CPU affinity so it runs on a core by
> itself, reducing interrupt-handling latency slightly further.
> 
Whats your interrupt affinity look like?

> * Our code is written in C, and we've tried to keep it as close to the
> kernel's socket calls as possible.
> 
> So we're currently looking further into ways to reduce latency, which
> currently looks like it'll mean digging deeper into the kernel's
> networking options and innards.
> 
> FYI, we're using RHE release 4 and kernel version 2.6.9 on x86_64 machines.
> 
> -Ben
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: Minimizing TCP latency

[Ben Hutchings]

On Mon, 2010-03-29 at 16:24 +1300, Ben Hoyt wrote:
> > I'm happy to receive some pointers via this list [...]
> 
> Sorry, I should also have posted more about what I've done in terms of
> reducing latency so far:
> 
> * We've already got TCP_NODELAY activated.
> 
> * We've also turned off interrupt coalescing on the network card (for
> example, see http://www.29west.com/docs/THPM/latency-interrupt-coalescing.html).
> 
> * We've modified our process's CPU affinity so it runs on a core by
> itself, reducing interrupt-handling latency slightly further.
> 
> * Our code is written in C, and we've tried to keep it as close to the
> kernel's socket calls as possible.
> 
> So we're currently looking further into ways to reduce latency, which
> currently looks like it'll mean digging deeper into the kernel's
> networking options and innards.

Choose low-latency NICs.  Read the tuning guide, but run your own
benchmarks.

> FYI, we're using RHE release 4 and kernel version 2.6.9 on x86_64 machines.

You're not going to get much help here in working on that kernel.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Minimizing TCP latency

[Ben Hoyt]

> One machine sends messages, and messages are not aknowledged by other
> part ? Is it one way communication ?

No, it's two-way communication. It's really the round-trip
time/latency that's important.

> What are the targets ?  (time constraints)

We're currently seeing a round-trip time of about 40us between two
Linux servers on our network (so presumably about 20us one way).
Anything quicker than that would be nice, but we're hoping to get down
to about half that.

> Mono threaded application handling one tcp flow ?
> Are you handling a very light load on few flows, or many flows ?
> Are machines all in the same LAN ?

It's a single-threaded application handling one TCP session with light
load, maybe one or two 512-byte packets per minute. The machines are
on the same localized network. There's a (fairly high-speed) router in
between, but I think most of the delays are being introduced at the
NIC and Linux levels.

Thanks,
Ben.

Re: Minimizing TCP latency

[Ben Hoyt]

> Choose low-latency NICs.  Read the tuning guide, but run your own
> benchmarks.

Is this the tuning guide you're referring to?
http://fasterdata.es.net/TCP-tuning/linux.html

>> FYI, we're using RHE release 4 and kernel version 2.6.9 on x86_64 machines.
> You're not going to get much help here in working on that kernel.

Thanks for that. I didn't realise 2.6.9 was so old -- I'll upgrade.

-Ben

Re: Minimizing TCP latency

[Ben Hutchings]

On Tue, 2010-03-30 at 15:36 +1300, Ben Hoyt wrote:
> > Choose low-latency NICs.  Read the tuning guide, but run your own
> > benchmarks.
> 
> Is this the tuning guide you're referring to?
> http://fasterdata.es.net/TCP-tuning/linux.html

No, and that doesn't say anything about latency.

There are quite a few generic tuning knobs (buffer and ring sizes,
interrupt moderation and affinity).  However, many vendors of
high-performance NICs, including Solarflare, provide a performance
tuning guide as part of their documentation, which can include
hardware-specific advice.

Ben.

-- 
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Minimizing TCP latency

[Mikael Abrahamsson]

On Tue, 30 Mar 2010, Ben Hoyt wrote:

> We're currently seeing a round-trip time of about 40us between two
> Linux servers on our network (so presumably about 20us one way).
> Anything quicker than that would be nice, but we're hoping to get down
> to about half that.

Serialisation delay on GigE for a 512 byte packet is 4us if my 
calculations are correct, so if you have some equipment in between that 
means two serialisations either way, meaning on GigE just the 
serialisation delay is 16us RTT.

You'd probably be better off directly connection the two machines with 
10GE NIC, or at least buy equipment that'll do cut-through switching if 
you want to bring the delay down. I'm even surprised you got it down this 
low...