tcp wifi upload performance and lots of ACKs

tcp wifi upload performance and lots of ACKs

[Ben Greear]

I'm going some TCP performance testing on wifi -> LAN interface connections.  With
UDP, we can get around 250Mbps of payload throughput.  With TCP, max is about 80Mbps.

I think the problem is that there are way too many ACK packets, and bi-directional
traffic on wifi interfaces really slows things down.  (About 7000 pkts per second in
upload direction, 2000 pps download.  And the vast majority of the download pkts
are 66 byte ACK pkts from what I can tell.)

Kernel is 3.3.7+

Anyone know of any tuning parameters that would let the receiving socket wait a
bit longer and send more ACK data in fewer packets?

Packet traces and other info available if anyone wants to take a look.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: tcp wifi upload performance and lots of ACKs

[Daniel Baluta]

On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear <greearb@candelatech.com> wrote:
> I'm going some TCP performance testing on wifi -> LAN interface connections.
>  With
> UDP, we can get around 250Mbps of payload throughput.  With TCP, max is
> about 80Mbps.
>
> I think the problem is that there are way too many ACK packets, and
> bi-directional
> traffic on wifi interfaces really slows things down.  (About 7000 pkts per
> second in
> upload direction, 2000 pps download.  And the vast majority of the download
> pkts
> are 66 byte ACK pkts from what I can tell.)
>
> Kernel is 3.3.7+
>
> Anyone know of any tuning parameters that would let the receiving socket
> wait a
> bit longer and send more ACK data in fewer packets?

An ACK is generated after every second full sized segment or a timeout
expires.

Currently, there is no way to tune these parameters. Here is an experimental
patch [1]. If anyone, thinks that this patch has a chance to get accepted
I will be happily try to further improve it.

>
> Packet traces and other info available if anyone wants to take a look.

thanks,
Daniel.

[1] http://marc.info/?l=linux-netdev&m=131983649130350&w=2

Re: tcp wifi upload performance and lots of ACKs

[Ben Greear]

On 06/04/2012 12:22 PM, Daniel Baluta wrote:
> On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear<greearb@candelatech.com>  wrote:
>> I'm going some TCP performance testing on wifi ->  LAN interface connections.
>>   With
>> UDP, we can get around 250Mbps of payload throughput.  With TCP, max is
>> about 80Mbps.
>>
>> I think the problem is that there are way too many ACK packets, and
>> bi-directional
>> traffic on wifi interfaces really slows things down.  (About 7000 pkts per
>> second in
>> upload direction, 2000 pps download.  And the vast majority of the download
>> pkts
>> are 66 byte ACK pkts from what I can tell.)
>>
>> Kernel is 3.3.7+
>>
>> Anyone know of any tuning parameters that would let the receiving socket
>> wait a
>> bit longer and send more ACK data in fewer packets?
>
> An ACK is generated after every second full sized segment or a timeout
> expires.
>
> Currently, there is no way to tune these parameters. Here is an experimental
> patch [1]. If anyone, thinks that this patch has a chance to get accepted
> I will be happily try to further improve it.

It looks like it could be useful for my case, but I would also want
per-socket options to set the min/max ack delay so that the settings
are not just system-wide.

I can at least test this, and perhaps even hack on the code if
you are not interested in the per-socket settings...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: tcp wifi upload performance and lots of ACKs

[Daniel Baluta]

On Mon, Jun 4, 2012 at 11:09 PM, Ben Greear <greearb@candelatech.com> wrote:
> On 06/04/2012 12:22 PM, Daniel Baluta wrote:
>>
>> On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear<greearb@candelatech.com>
>>  wrote:
>>>
>>> I'm going some TCP performance testing on wifi ->  LAN interface
>>> connections.
>>>  With
>>> UDP, we can get around 250Mbps of payload throughput.  With TCP, max is
>>> about 80Mbps.
>>>
>>> I think the problem is that there are way too many ACK packets, and
>>> bi-directional
>>> traffic on wifi interfaces really slows things down.  (About 7000 pkts
>>> per
>>> second in
>>> upload direction, 2000 pps download.  And the vast majority of the
>>> download
>>> pkts
>>> are 66 byte ACK pkts from what I can tell.)
>>>
>>> Kernel is 3.3.7+
>>>
>>> Anyone know of any tuning parameters that would let the receiving socket
>>> wait a
>>> bit longer and send more ACK data in fewer packets?
>>
>>
>> An ACK is generated after every second full sized segment or a timeout
>> expires.
>>
>> Currently, there is no way to tune these parameters. Here is an
>> experimental
>> patch [1]. If anyone, thinks that this patch has a chance to get accepted
>> I will be happily try to further improve it.
>
>
> It looks like it could be useful for my case, but I would also want
> per-socket options to set the min/max ack delay so that the settings
> are not just system-wide.
>
> I can at least test this, and perhaps even hack on the code if
> you are not interested in the per-socket settings...

Yes, indeed this should be a per socket option. I will be
back with a patch in the next days.

Meanwhile, it would be great if you could test my previous patch.

thanks,
Daniel.

Re: tcp wifi upload performance and lots of ACKs

[Ben Greear]

On 06/04/2012 12:22 PM, Daniel Baluta wrote:

> Currently, there is no way to tune these parameters. Here is an experimental
> patch [1]. If anyone, thinks that this patch has a chance to get accepted
> I will be happily try to further improve it.
>
>>
>> Packet traces and other info available if anyone wants to take a look.
>
> thanks,
> Daniel.
>
> [1] http://marc.info/?l=linux-netdev&m=131983649130350&w=2

I tried your patch in 3.5.0-rc1.

Doesn't seem to help any...do you have suggested settings for
the proc values?  (I tried increasing tcp_delack_segs up to 10,
but no significant increase...)

I'll keep poking around.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: tcp wifi upload performance and lots of ACKs

[Ben Greear]

On 06/06/2012 05:26 PM, Ben Greear wrote:
> On 06/04/2012 12:22 PM, Daniel Baluta wrote:
>
>> Currently, there is no way to tune these parameters. Here is an experimental
>> patch [1]. If anyone, thinks that this patch has a chance to get accepted
>> I will be happily try to further improve it.
>>
>>>
>>> Packet traces and other info available if anyone wants to take a look.
>>
>> thanks,
>> Daniel.
>>
>> [1] http://marc.info/?l=linux-netdev&m=131983649130350&w=2
>
> I tried your patch in 3.5.0-rc1.
>
> Doesn't seem to help any...do you have suggested settings for
> the proc values? (I tried increasing tcp_delack_segs up to 10,
> but no significant increase...)
>
> I'll keep poking around.

Ahh, forcing send-buffer to be large helps significantly..wonder
if that was my original problem.

Will get some more useful results tomorrow...

Thanks,
Ben

>
> Thanks,
> Ben
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: tcp wifi upload performance and lots of ACKs

[Ben Greear]

On 06/04/2012 12:22 PM, Daniel Baluta wrote:
> On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear<greearb@candelatech.com>  wrote:
>> I'm going some TCP performance testing on wifi ->  LAN interface connections.
>>   With
>> UDP, we can get around 250Mbps of payload throughput.  With TCP, max is
>> about 80Mbps.
>>
>> I think the problem is that there are way too many ACK packets, and
>> bi-directional
>> traffic on wifi interfaces really slows things down.  (About 7000 pkts per
>> second in
>> upload direction, 2000 pps download.  And the vast majority of the download
>> pkts
>> are 66 byte ACK pkts from what I can tell.)

> [1] http://marc.info/?l=linux-netdev&m=131983649130350&w=2

After a bit more playing, I did notice a reliable 5% increase in
traffic (200Mbps -> 210Mbps) from changing the delack segments
to 20 from the default of 1.  That is enough to be useful to me,
and there may be more significant gains to be found...
I haven't done a full matrix of testing yet.

I read through the original thread, and to summarize:

* Need to make the values per-socket.
* No multiplication in hot path.
* Would be nice to make it automatic.

The first seems fairly trivial..just add a new set of socket-opts (or
maybe just one that can take all 3 values?) and store the settings in
the socket structs.

As for getting rid of the multiply..I think you cannot just use shifts.
That does not give the needed granularity.  An alternative is
to update a cached computation every time the mss or socket-opt changes.

As for a magic heuristic to figure this out, I think that would be
quite tricky to do right.  So, maybe add that in the future, but
for the present, just allowing applications to set the value seems
enough.

We could support configurable system-wide defaults so that users of
programs that do not know this new sockopt can still take advantage
of the feature.

Does this sound like a reasonable solution?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

RE: tcp wifi upload performance and lots of ACKs

[David Laight]

 

> -----Original Message-----
> From: netdev-owner@vger.kernel.org 
> [mailto:netdev-owner@vger.kernel.org] On Behalf Of Ben Greear
> Sent: 07 June 2012 05:16
> To: Daniel Baluta
> Cc: netdev
> Subject: Re: tcp wifi upload performance and lots of ACKs
> 
> On 06/04/2012 12:22 PM, Daniel Baluta wrote:
> > On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear<greearb@candelatech.com>
wrote:
> >> I'm going some TCP performance testing on wifi ->  LAN interface
connections.
> >>   With
> >> UDP, we can get around 250Mbps of payload throughput.  With TCP,
max is
> >> about 80Mbps.
> >>
> >> I think the problem is that there are way too many ACK packets, and
> >> bi-directional
> >> traffic on wifi interfaces really slows things down.  (About 7000
pkts per
> >> second in
> >> upload direction, 2000 pps download.  And the vast majority of the
download
> >> pkts
> >> are 66 byte ACK pkts from what I can tell.)
> 
> > [1] http://marc.info/?l=linux-netdev&m=131983649130350&w=2
> 
> After a bit more playing, I did notice a reliable 5% increase in
> traffic (200Mbps -> 210Mbps) from changing the delack segments
> to 20 from the default of 1.  That is enough to be useful to me,
> and there may be more significant gains to be found...
> I haven't done a full matrix of testing yet.

Does this delaying of acks have a detrimental effect on the
sending end?
I've seen very bad interactions between delayed acks and
(I believe) the 'slow start' code on connections with
one-directional data, Nagle disabled and very low RTT.

What I saw was the sender sending 4 data packets, then
sitting waiting for an ack - in spite of accumulating
several kB of data to send.

Delaying acks further will only make this worse.

	David

Re: tcp wifi upload performance and lots of ACKs

[Ben Greear]

On 06/07/2012 05:20 AM, David Laight wrote:
>
>
>> -----Original Message-----
>> From: netdev-owner@vger.kernel.org
>> [mailto:netdev-owner@vger.kernel.org] On Behalf Of Ben Greear
>> Sent: 07 June 2012 05:16
>> To: Daniel Baluta
>> Cc: netdev
>> Subject: Re: tcp wifi upload performance and lots of ACKs
>>
>> On 06/04/2012 12:22 PM, Daniel Baluta wrote:
>>> On Mon, Jun 4, 2012 at 9:29 PM, Ben Greear<greearb@candelatech.com>
> wrote:
>>>> I'm going some TCP performance testing on wifi ->   LAN interface
> connections.
>>>>    With
>>>> UDP, we can get around 250Mbps of payload throughput.  With TCP,
> max is
>>>> about 80Mbps.
>>>>
>>>> I think the problem is that there are way too many ACK packets, and
>>>> bi-directional
>>>> traffic on wifi interfaces really slows things down.  (About 7000
> pkts per
>>>> second in
>>>> upload direction, 2000 pps download.  And the vast majority of the
> download
>>>> pkts
>>>> are 66 byte ACK pkts from what I can tell.)
>>
>>> [1] http://marc.info/?l=linux-netdev&m=131983649130350&w=2
>>
>> After a bit more playing, I did notice a reliable 5% increase in
>> traffic (200Mbps ->  210Mbps) from changing the delack segments
>> to 20 from the default of 1.  That is enough to be useful to me,
>> and there may be more significant gains to be found...
>> I haven't done a full matrix of testing yet.
>
> Does this delaying of acks have a detrimental effect on the
> sending end?
> I've seen very bad interactions between delayed acks and
> (I believe) the 'slow start' code on connections with
> one-directional data, Nagle disabled and very low RTT.
>
> What I saw was the sender sending 4 data packets, then
> sitting waiting for an ack - in spite of accumulating
> several kB of data to send.
>
> Delaying acks further will only make this worse.

I'm sure it's not for everyone in all cases.  In my case, I'm
sending long-term bulk transfer, at high speeds, over wifi network
which has some latency.  Tested one-way traffic so far.

With the patch and delayed acks, I get more sender throughput than
without (200Mbps -> 210Mbps).

Thanks,
Ben

>
> 	David
>
>


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: tcp wifi upload performance and lots of ACKs

[Rick Jones]

> Does this delaying of acks have a detrimental effect on the
> sending end?
> I've seen very bad interactions between delayed acks and
> (I believe) the 'slow start' code on connections with
> one-directional data, Nagle disabled and very low RTT.
>
> What I saw was the sender sending 4 data packets, then
> sitting waiting for an ack - in spite of accumulating
> several kB of data to send.
>
> Delaying acks further will only make this worse.

At least two stacks have a reasonable ACK avoidance heuristic.  Those 
would be HP-UX and Solaris (Mac OS 9 had one as well, IIRC).  The 
heuristics are rather similar because the two TCP stacks share a common 
ancestor.  I used to interact with HP-UX's regularly, my statements will 
be based on that, and an assumption Solaris is similar.

Both attempt to divine what the senders' congestion window happens to be 
and be certain to send an ACK before that is exhausted.  So, at the 
start of a connection, there will be the usual, more rapid 
ACKnowledgement.  As things happen "normally" then the number of 
segments per ACK increases until it hits a configurable limit.  There 
are conditions which will cut the limit in half on a given connection - 
one is the sending of an ACK via the standalone ACK timer (this is from 
memory, so may be a bit off).  There are probably a few other conditions 
that drop the limit by half.  The heuristic attempts to learn in each 
connection what the reasonable limit on ACK avoidance might be so there 
isn't a per-connection control, just the global controls via ndd.  As 
conditions causing the limit to be cut in half arise, the connection 
naturally and irrevocably falls back to the usual "ack-every-other" 
behaviour.

When there is little to no packet loss, and a rather regular stream of 
data, this works rather well indeed.  For example in a LAN or Data 
Center.  You can run netperf TCP_STREAM with the limit at the default, 
and with the limit set to two and see the considerable difference in 
service demand on either side.

This may not work well when the sender has a congestion window growth 
heuristic different from what the ACK avoidance heuristic assumes.  If I 
recall correctly, the heuristic in HP-UX assumes the sender grows cwnd 
by the number of bytes/segments ACKnowledged. If the sender grows the 
cwnd by only one segment per ACK rather than by the bytes ACKnowldeged 
by the ACK the growth of the cwnd will be slowed.  In a LAN that may be 
papered-over a bit, but it will become quite noticable in a higher RTT 
environment.  Probably not as noticable for a sufficiently short 
connection, or a long one, but will be for ones in the middle.  The 
short connection doesn't need much cwnd in the first place, and the 
heuristic works its way up to avoiding ACKs, and the long one will be 
long enough to have the ACK avoidance heuristic gravitate down to 
ack-every-other.

rick jones

Re: tcp wifi upload performance and lots of ACKs

[David Miller]

From: Rick Jones <rick.jones2@hp.com>
Date: Thu, 07 Jun 2012 10:51:20 -0700

> This may not work well when the sender has a congestion window growth
> heuristic different from what the ACK avoidance heuristic assumes.  If
> I recall correctly, the heuristic in HP-UX assumes the sender grows
> cwnd by the number of bytes/segments ACKnowledged.

Linux violates this assumption.

Re: tcp wifi upload performance and lots of ACKs

[Eric Dumazet]

On Mon, 2012-06-04 at 11:29 -0700, Ben Greear wrote:
> I'm going some TCP performance testing on wifi -> LAN interface connections.  With
> UDP, we can get around 250Mbps of payload throughput.  With TCP, max is about 80Mbps.
> 
> I think the problem is that there are way too many ACK packets, and bi-directional
> traffic on wifi interfaces really slows things down.  (About 7000 pkts per second in
> upload direction, 2000 pps download.  And the vast majority of the download pkts
> are 66 byte ACK pkts from what I can tell.)
> 
> Kernel is 3.3.7+
> 
> Anyone know of any tuning parameters that would let the receiving socket wait a
> bit longer and send more ACK data in fewer packets?

Well, thats half duplex days...

There is the ACK every 2 packets rule of thumb, so that tcp sender can
increase its cwnd.

Then, some folks tried submitting patches to make '2' more like 10 or
15, but this went nowhere.

Other idea is to arm a timer and defer ACK sending, in the hope we
receive another packet very soon.

That could be done with a special qdisc, sort of netem...

Re: tcp wifi upload performance and lots of ACKs

[Ben Greear]

On 06/04/2012 12:32 PM, Eric Dumazet wrote:
> On Mon, 2012-06-04 at 11:29 -0700, Ben Greear wrote:
>> I'm going some TCP performance testing on wifi ->  LAN interface connections.  With
>> UDP, we can get around 250Mbps of payload throughput.  With TCP, max is about 80Mbps.
>>
>> I think the problem is that there are way too many ACK packets, and bi-directional
>> traffic on wifi interfaces really slows things down.  (About 7000 pkts per second in
>> upload direction, 2000 pps download.  And the vast majority of the download pkts
>> are 66 byte ACK pkts from what I can tell.)
>>
>> Kernel is 3.3.7+
>>
>> Anyone know of any tuning parameters that would let the receiving socket wait a
>> bit longer and send more ACK data in fewer packets?
>
> Well, thats half duplex days...

WiFi is still half-duplex, and isn't changing any time soon I think.

> There is the ACK every 2 packets rule of thumb, so that tcp sender can
> increase its cwnd.
>
> Then, some folks tried submitting patches to make '2' more like 10 or
> 15, but this went nowhere.
>
> Other idea is to arm a timer and defer ACK sending, in the hope we
> receive another packet very soon.

 From a quick read of Daniel's patch..it seems there is already a
timer that could be used for this?

> That could be done with a special qdisc, sort of netem...

That sounds like a horrible hack :)

Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: tcp wifi upload performance and lots of ACKs

[Eric Dumazet]

On Mon, 2012-06-04 at 13:12 -0700, Ben Greear wrote:

> That sounds like a horrible hack :)
> 

Yep, but could be usefull on a gateway/router, if you cant change the
TCP stack of the clients.

Re: tcp wifi upload performance and lots of ACKs

[Glen Turner]

On 05/06/2012, at 3:59 AM, Ben Greear wrote:
> 
> Anyone know of any tuning parameters that would let the receiving socket wait a
> bit longer and send more ACK data in fewer packets?

It's perhaps better to think of Acks as offering an opportunity for the transmitting TCP algorithm to sample the congestion of the path. That's particularly important in Slow Start (when the transmitting algorithm is trying to send data exponentially faster) and both traditional TCP and Linux's CUBIC+Hystart will not ramp up transmission as quickly as they otherwise would when starved of samples.

Of course you've got to weigh slower ramp up in transmission rate against the loss in throughput due to the  turn-around time on the half-duplex wireless media. But I don't see the proposed patch doing any weighing of considerations at all, but punting that to user space which has even less information about the condition of TCP (particularly the receiver's guess as to the slow-start/congestion-avoidance mode of the transmitter) and less information about the attributes of the media (particularly the turn-around time of half-duplex media on the path) to make a decision with.

There's a nice summary of the tradeoffs of delayed Acks in RFC2525 section 2.13, although the discussion is dated (pre-ABC, pre-HyStart).

Note the firm requirement in RFC2581 section 4.2 that an Ack be sent within 500ms. There are implementations in the field which have delayed Acks out to 2s, these are  notoriously painful and many modern algorithms drop back into a low performance mode when facing those receivers.

- Glen