2.6.19-rc1: Volanomark slowdown

2.6.19-rc1: Volanomark slowdown

[Tim Chen]

The patch

[TCP]: Send ACKs each 2nd received segment
commit: 1ef9696c909060ccdae3ade245ca88692b49285b
http://kernel.org/git/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1ef9696c909060ccdae3ade245ca88692b49285b

reduced Volanomark benchmark throughput by 10%.  
This is because Volanomark sends 
short message (<100 bytes) on its TCP
connections.  This patch increases the number of ACKs 
traffic by 3.5 times.  

By adopting this patch, we assume that with
small segment, having short delay is important 
enough that we are willing to reduce bandwidth 
with more ACKs.  

Is there any real application out there
that this new behavior could be a concern?

Tim 

Re: 2.6.19-rc1: Volanomark slowdown

[David Miller]

From: Tim Chen <tim.c.chen@linux.intel.com>
Date: Tue, 07 Nov 2006 10:32:34 -0800

[ Please bring up networking questions on "netdev@vger.kernel.org"
  as that is the place where networking developers read bug reports
  and questions, they by-in-large do not read linux-kernel at all. ]

> [TCP]: Send ACKs each 2nd received segment
> commit: 1ef9696c909060ccdae3ade245ca88692b49285b
> http://kernel.org/git/?
> p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1ef9696c909060ccdae3ade245ca88692b49285b
> 
> reduced Volanomark benchmark throughput by 10%.  
> This is because Volanomark sends 
> short message (<100 bytes) on its TCP
> connections.  This patch increases the number of ACKs 
> traffic by 3.5 times.  
> 
> By adopting this patch, we assume that with
> small segment, having short delay is important 
> enough that we are willing to reduce bandwidth 
> with more ACKs.  
> 
> Is there any real application out there
> that this new behavior could be a concern?

That's unfortunate, because without that patch connections can hang
which is more important to fix than your performance test.  :-)

If we don't ACK every two segments, stacks which grow the congestion
window based upon packet counting will not grow the congestion window
properly when they are sending smaller than MSS sized segments.

This topic has been discussed quite a bit, you may want to do some
searching in the netdev archives to read some of that.

Re: 2.6.19-rc1: Volanomark slowdown

[John Heffner]

David Miller wrote:
> If we don't ACK every two segments, stacks which grow the congestion
> window based upon packet counting will not grow the congestion window
> properly when they are sending smaller than MSS sized segments.

The only stack I know of that does this currently is linux, and in doing 
so does not conform to the spec. ;)  Sending to a BSD receiver will 
result in the same behavior, so the "right place" to fix this is on the 
sending side.  (I know the issue of packet vs. byte counting has come up 
many times over the last 10 years or so, and many arguments have been 
made on either side... I don't mean this to be flame bait but it's clear 
what will happen in this scenario.)

One way of viewing the current situation is that linux's packet counting 
plus ABC is more conservative than byte counting -- sometimes much more 
so.  Packet counting without ABC may be more or less conservative than 
byte counting, depending on segment sizes and receiver ACK strategy. 
Without ABC, linux is vulnerable to aggressive ACKing to inflate the 
cwnd.  This is a kind of ugly state of affairs.

Unfortunately I see no clear way to reconcile these issues short of 
switching to byte counting.  Obviously this would be a big change as 
packet counting is deeply ingrained in not only the congestion control 
but also the recovery code.

   -John

Re: 2.6.19-rc1: Volanomark slowdown

[David Miller]

From: John Heffner <jheffner@psc.edu>
Date: Tue, 07 Nov 2006 16:50:33 -0500

> The only stack I know of that does this currently is linux, and in doing 
> so does not conform to the spec. ;)  Sending to a BSD receiver will 
> result in the same behavior, so the "right place" to fix this is on the 
> sending side.  (I know the issue of packet vs. byte counting has come up 
> many times over the last 10 years or so, and many arguments have been 
> made on either side... I don't mean this to be flame bait but it's clear 
> what will happen in this scenario.)

John, you cannot change the N-million existing Linux systems
out there doing congestion control via byte counting.  You
cannot do this no matter how much you wish it so :-)

Re: 2.6.19-rc1: Volanomark slowdown

[John Heffner]

David Miller wrote:
> From: John Heffner <jheffner@psc.edu>
> Date: Tue, 07 Nov 2006 16:50:33 -0500
> 
>> The only stack I know of that does this currently is linux, and in doing 
>> so does not conform to the spec. ;)  Sending to a BSD receiver will 
>> result in the same behavior, so the "right place" to fix this is on the 
>> sending side.  (I know the issue of packet vs. byte counting has come up 
>> many times over the last 10 years or so, and many arguments have been 
>> made on either side... I don't mean this to be flame bait but it's clear 
>> what will happen in this scenario.)
> 
> John, you cannot change the N-million existing Linux systems
> out there doing congestion control via byte counting.  You
> cannot do this no matter how much you wish it so :-)

That would make our lives easier, wouldn't it? ;)  Clearly there are 
some combinations of TCP stacks out there that won't interoperate well 
under certain workloads.  Making new versions of the stack work well is 
the best we can hope for...

Fixing the sending side does not mean we have to back out the 
work-around on the receiving side.

   -John

Re: 2.6.19-rc1: Volanomark slowdown

[Alexey Kuznetsov]

Hell]!

> > reduced Volanomark benchmark throughput by 10%.  

The irony of it is that java vm used to be one of victims
of over-delayed acks.

I will look, there is a little chance that it is possible
to detect the situation and to stretch ACKs.

There is one little question though. If you see a visible
difference in performance, does it mean that you see Volaconomark
used to show much better numbers comparing to another OSes? :-)

Alexey

Re: 2.6.19-rc1: Volanomark slowdown

[Arjan van de Ven]

On Tue, 2006-11-07 at 10:32 -0800, Tim Chen wrote:
> The patch
> 
> [TCP]: Send ACKs each 2nd received segment
> commit: 1ef9696c909060ccdae3ade245ca88692b49285b
> http://kernel.org/git/?
> p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1ef9696c909060ccdae3ade245ca88692b49285b
> 
> reduced Volanomark benchmark throughput by 10%.  
> This is because Volanomark sends 
> short message (<100 bytes) on its TCP
> connections.  This patch increases the number of ACKs 
> traffic by 3.5 times.  
> 
> By adopting this patch, we assume that with
> small segment, having short delay is important 
> enough that we are willing to reduce bandwidth 
> with more ACKs.  

I wonder if it's an option to use low priority QoS fields for these acks
(heck I don't even know if ACKs have such fields in their packet) so
that they can get dropped if there are more packets then there is
bandwidth ....


-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: 2.6.19-rc1: Volanomark slowdown

[Olaf Kirch]

On Wed, Nov 08, 2006 at 04:55:18PM +0100, Arjan van de Ven wrote:
> I wonder if it's an option to use low priority QoS fields for these acks
> (heck I don't even know if ACKs have such fields in their packet) so
> that they can get dropped if there are more packets then there is
> bandwidth ....

Is it proven that the number of ACKs actually cause bandwidth problems?
I found Volanomark to exercise the scheduler more than anything else,
so maybe the slowdown, while triggered by an increased number of ACKs,
is caused by something else entirely.

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

Re: 2.6.19-rc1: Volanomark slowdown

[Tim Chen]

On Wed, 2006-11-08 at 17:29 +0100, Olaf Kirch wrote:

> Is it proven that the number of ACKs actually cause bandwidth problems?
> I found Volanomark to exercise the scheduler more than anything else,
> so maybe the slowdown, while triggered by an increased number of ACKs,
> is caused by something else entirely.
> 

The patch in question affects purely TCP and not the scheduler.  I don't
think the scheduler has anything to do with the slowdown seen after
the patch is applied.

The total number of messages being exchanged around the chatrooms in 
Volanomark remain unchanged.  But ACKS increase by 3.5 times and
segments received increase by 38% from netstat.  

ACK is comparable in size to the actual Volanomark messages as 
those are  pretty small (<100 byte).

So I think it is reasonable to conclude that the increase in TCP traffic
reduce the bandwidth and throughput in Volanomark.

However, Volanomark is just a benchmark to alert us to changes.  
If in real applications with small segment, this patch is 
needed to fix congestion window adjustment as Dave pointed 
out, and impact on bandwidth not as important, so be it.

Tim

Re: 2.6.19-rc1: Volanomark slowdown

[Arjan van de Ven]

> However, Volanomark is just a benchmark to alert us to changes.  
> If in real applications with small segment, this patch is 
> needed to fix congestion window adjustment as Dave pointed 
> out, and impact on bandwidth not as important, so be it.
> 
> Tim

if we can get the best of both worlds (by having the extra acks drop on
congestion) it's of course nicer ;)

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: 2.6.19-rc1: Volanomark slowdown

[Olaf Kirch]

On Wed, Nov 08, 2006 at 10:38:52AM -0800, Tim Chen wrote:
> The patch in question affects purely TCP and not the scheduler.  I don't

I know.

> think the scheduler has anything to do with the slowdown seen after
> the patch is applied.

In fixing performance issues, the most obvious explanation isn't always
the right one. It's quite possible you're right, sure.

What I'm saying though is that it doesn't rhyme with what I've seen of
Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
come close to saturating a Gigabit pipe before it maxed out on CPU load.

> The total number of messages being exchanged around the chatrooms in 
> Volanomark remain unchanged.  But ACKS increase by 3.5 times and
> segments received increase by 38% from netstat.  

> So I think it is reasonable to conclude that the increase in TCP traffic
> reduce the bandwidth and throughput in Volanomark.

You could count the number of outbound packets dropped on the server.

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

Re: 2.6.19-rc1: Volanomark slowdown

[Tim Chen]

On Wed, 2006-11-08 at 23:10 +0100, Olaf Kirch wrote:

> 
> In fixing performance issues, the most obvious explanation isn't always
> the right one. It's quite possible you're right, sure.
> 
> What I'm saying though is that it doesn't rhyme with what I've seen of
> Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
> come close to saturating a Gigabit pipe before it maxed out on CPU load.
> 

I am running Volanomark in a loopback mode on a 2P woodcrest box 
(4 cores).  So the configuration is a bit different.  

In my testing, the CPU utilization is at 100%.  So
increase in ACKs will cost CPU to devote more
time to process those ACKs and reduce throughput.

> 
> You could count the number of outbound packets dropped on the server.
> 

As I'm running in loopback mode, there are no dropped packets.

Thanks.

Tim

Re: 2.6.19-rc1: Volanomark slowdown

[Stephen Hemminger]

On Wed, 08 Nov 2006 14:07:32 -0800
Tim Chen <tim.c.chen@linux.intel.com> wrote:

> On Wed, 2006-11-08 at 23:10 +0100, Olaf Kirch wrote:
> 
> > 
> > In fixing performance issues, the most obvious explanation isn't always
> > the right one. It's quite possible you're right, sure.
> > 
> > What I'm saying though is that it doesn't rhyme with what I've seen of
> > Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
> > come close to saturating a Gigabit pipe before it maxed out on CPU load.
> > 
> 
> I am running Volanomark in a loopback mode on a 2P woodcrest box 
> (4 cores).  So the configuration is a bit different.  
> 
> In my testing, the CPU utilization is at 100%.  So
> increase in ACKs will cost CPU to devote more
> time to process those ACKs and reduce throughput.
> 
> > 
> > You could count the number of outbound packets dropped on the server.
> > 
> 
> As I'm running in loopback mode, there are no dropped packets.
> 

Optimizing for loopback is perversion; perversion can be fun but it gets
to be a obsession then it's sick.

-- 
Stephen Hemminger <shemminger@osdl.org>

Re: 2.6.19-rc1: Volanomark slowdown

[Tim Chen]

On Wed, 2006-11-08 at 15:00 -0800, Stephen Hemminger wrote:

> 
> Optimizing for loopback is perversion; perversion can be fun but it gets
> to be a obsession then it's sick.
> 

It is not my intention to optimize for this case, but rather to
detect change in kernel behavior.

That's why in my original mail I ask if increase in ACKs could
cause problem for any real application.  

Tim

Re: 2.6.19-rc1: Volanomark slowdown

[Olaf Kirch]

On Wed, Nov 08, 2006 at 02:07:32PM -0800, Tim Chen wrote:
> In my testing, the CPU utilization is at 100%.  So
> increase in ACKs will cost CPU to devote more
> time to process those ACKs and reduce throughput.

Oh, I see. I would test on a real network with real clients. I doubt
you would observe a noticeable effect there.

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

Re: 2.6.19-rc1: Volanomark slowdown

[Stephen Hemminger]

On Wed, 8 Nov 2006 23:10:28 +0100
Olaf Kirch <okir@suse.de> wrote:

> On Wed, Nov 08, 2006 at 10:38:52AM -0800, Tim Chen wrote:
> > The patch in question affects purely TCP and not the scheduler.  I don't
> 
> I know.
> 
> > think the scheduler has anything to do with the slowdown seen after
> > the patch is applied.
> 
> In fixing performance issues, the most obvious explanation isn't always
> the right one. It's quite possible you're right, sure.
> 
> What I'm saying though is that it doesn't rhyme with what I've seen of
> Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
> come close to saturating a Gigabit pipe before it maxed out on CPU load.
> 
> > The total number of messages being exchanged around the chatrooms in 
> > Volanomark remain unchanged.  But ACKS increase by 3.5 times and
> > segments received increase by 38% from netstat.  
> 
> > So I think it is reasonable to conclude that the increase in TCP traffic
> > reduce the bandwidth and throughput in Volanomark.
> 
> You could count the number of outbound packets dropped on the server.
> 
> Olaf

Also under benchmark stress, the load can get so high that timers go
off that normally don't. For example, I have seen delayed ack timer
cause extra ack's when at lower loads the response happened quick enough
that the ACK was piggybacked.


-- 
Stephen Hemminger <shemminger@osdl.org>

Re: 2.6.19-rc1: Volanomark slowdown

[Rick Jones]

On Wed, 2006-11-08 at 23:10 +0100, Olaf Kirch wrote:
> What I'm saying though is that it doesn't rhyme with what I've seen of
> Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
> come close to saturating a Gigabit pipe before it maxed out on CPU load.

That actually supports the hypothesis doesn't it?  The issue being the
increased number of ACKs causing additional CPU overhead not saturating
a NIC if any involved.

One of these days I may have to try to look more closely at what volano
does relative to netperf - I remember that someone tried very hard (was
it you alexy?) to show a perfomance effect with netperf and it didn't do
it :(

rick jones