2.6.19-rc1: Volanomark slowdown

2.6.19-rc1: Volanomark slowdown

[Tim Chen]

The patch

[TCP]: Send ACKs each 2nd received segment
commit: 1ef9696c909060ccdae3ade245ca88692b49285b
http://kernel.org/git/?
p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1ef9696c909060ccdae3ade245ca88692b49285b

reduced Volanomark benchmark throughput by 10%.  
This is because Volanomark sends 
short message (<100 bytes) on its TCP
connections.  This patch increases the number of ACKs 
traffic by 3.5 times.  

By adopting this patch, we assume that with
small segment, having short delay is important 
enough that we are willing to reduce bandwidth 
with more ACKs.  

Is there any real application out there
that this new behavior could be a concern?

Tim 

Re: 2.6.19-rc1: Volanomark slowdown

[Arjan van de Ven]

On Tue, 2006-11-07 at 10:32 -0800, Tim Chen wrote:
> The patch
> 
> [TCP]: Send ACKs each 2nd received segment
> commit: 1ef9696c909060ccdae3ade245ca88692b49285b
> http://kernel.org/git/?
> p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=1ef9696c909060ccdae3ade245ca88692b49285b
> 
> reduced Volanomark benchmark throughput by 10%.  
> This is because Volanomark sends 
> short message (<100 bytes) on its TCP
> connections.  This patch increases the number of ACKs 
> traffic by 3.5 times.  
> 
> By adopting this patch, we assume that with
> small segment, having short delay is important 
> enough that we are willing to reduce bandwidth 
> with more ACKs.  

I wonder if it's an option to use low priority QoS fields for these acks
(heck I don't even know if ACKs have such fields in their packet) so
that they can get dropped if there are more packets then there is
bandwidth ....


-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: 2.6.19-rc1: Volanomark slowdown

[Olaf Kirch]

On Wed, Nov 08, 2006 at 04:55:18PM +0100, Arjan van de Ven wrote:
> I wonder if it's an option to use low priority QoS fields for these acks
> (heck I don't even know if ACKs have such fields in their packet) so
> that they can get dropped if there are more packets then there is
> bandwidth ....

Is it proven that the number of ACKs actually cause bandwidth problems?
I found Volanomark to exercise the scheduler more than anything else,
so maybe the slowdown, while triggered by an increased number of ACKs,
is caused by something else entirely.

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

Re: 2.6.19-rc1: Volanomark slowdown

[Tim Chen]

On Wed, 2006-11-08 at 17:29 +0100, Olaf Kirch wrote:

> Is it proven that the number of ACKs actually cause bandwidth problems?
> I found Volanomark to exercise the scheduler more than anything else,
> so maybe the slowdown, while triggered by an increased number of ACKs,
> is caused by something else entirely.
> 

The patch in question affects purely TCP and not the scheduler.  I don't
think the scheduler has anything to do with the slowdown seen after
the patch is applied.

The total number of messages being exchanged around the chatrooms in 
Volanomark remain unchanged.  But ACKS increase by 3.5 times and
segments received increase by 38% from netstat.  

ACK is comparable in size to the actual Volanomark messages as 
those are  pretty small (<100 byte).

So I think it is reasonable to conclude that the increase in TCP traffic
reduce the bandwidth and throughput in Volanomark.

However, Volanomark is just a benchmark to alert us to changes.  
If in real applications with small segment, this patch is 
needed to fix congestion window adjustment as Dave pointed 
out, and impact on bandwidth not as important, so be it.

Tim

Re: 2.6.19-rc1: Volanomark slowdown

[Arjan van de Ven]

> However, Volanomark is just a benchmark to alert us to changes.  
> If in real applications with small segment, this patch is 
> needed to fix congestion window adjustment as Dave pointed 
> out, and impact on bandwidth not as important, so be it.
> 
> Tim

if we can get the best of both worlds (by having the extra acks drop on
congestion) it's of course nicer ;)

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: 2.6.19-rc1: Volanomark slowdown

[Olaf Kirch]

On Wed, Nov 08, 2006 at 10:38:52AM -0800, Tim Chen wrote:
> The patch in question affects purely TCP and not the scheduler.  I don't

I know.

> think the scheduler has anything to do with the slowdown seen after
> the patch is applied.

In fixing performance issues, the most obvious explanation isn't always
the right one. It's quite possible you're right, sure.

What I'm saying though is that it doesn't rhyme with what I've seen of
Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
come close to saturating a Gigabit pipe before it maxed out on CPU load.

> The total number of messages being exchanged around the chatrooms in 
> Volanomark remain unchanged.  But ACKS increase by 3.5 times and
> segments received increase by 38% from netstat.  

> So I think it is reasonable to conclude that the increase in TCP traffic
> reduce the bandwidth and throughput in Volanomark.

You could count the number of outbound packets dropped on the server.

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

Re: 2.6.19-rc1: Volanomark slowdown

[Tim Chen]

On Wed, 2006-11-08 at 23:10 +0100, Olaf Kirch wrote:

> 
> In fixing performance issues, the most obvious explanation isn't always
> the right one. It's quite possible you're right, sure.
> 
> What I'm saying though is that it doesn't rhyme with what I've seen of
> Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
> come close to saturating a Gigabit pipe before it maxed out on CPU load.
> 

I am running Volanomark in a loopback mode on a 2P woodcrest box 
(4 cores).  So the configuration is a bit different.  

In my testing, the CPU utilization is at 100%.  So
increase in ACKs will cost CPU to devote more
time to process those ACKs and reduce throughput.

> 
> You could count the number of outbound packets dropped on the server.
> 

As I'm running in loopback mode, there are no dropped packets.

Thanks.

Tim

Re: 2.6.19-rc1: Volanomark slowdown

[Stephen Hemminger]

On Wed, 08 Nov 2006 14:07:32 -0800
Tim Chen <tim.c.chen@linux.intel.com> wrote:

> On Wed, 2006-11-08 at 23:10 +0100, Olaf Kirch wrote:
> 
> > 
> > In fixing performance issues, the most obvious explanation isn't always
> > the right one. It's quite possible you're right, sure.
> > 
> > What I'm saying though is that it doesn't rhyme with what I've seen of
> > Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
> > come close to saturating a Gigabit pipe before it maxed out on CPU load.
> > 
> 
> I am running Volanomark in a loopback mode on a 2P woodcrest box 
> (4 cores).  So the configuration is a bit different.  
> 
> In my testing, the CPU utilization is at 100%.  So
> increase in ACKs will cost CPU to devote more
> time to process those ACKs and reduce throughput.
> 
> > 
> > You could count the number of outbound packets dropped on the server.
> > 
> 
> As I'm running in loopback mode, there are no dropped packets.
> 

Optimizing for loopback is perversion; perversion can be fun but it gets
to be a obsession then it's sick.

-- 
Stephen Hemminger <shemminger@osdl.org>

Re: 2.6.19-rc1: Volanomark slowdown

[Tim Chen]

On Wed, 2006-11-08 at 15:00 -0800, Stephen Hemminger wrote:

> 
> Optimizing for loopback is perversion; perversion can be fun but it gets
> to be a obsession then it's sick.
> 

It is not my intention to optimize for this case, but rather to
detect change in kernel behavior.

That's why in my original mail I ask if increase in ACKs could
cause problem for any real application.  

Tim

Re: 2.6.19-rc1: Volanomark slowdown

[Olaf Kirch]

On Wed, Nov 08, 2006 at 02:07:32PM -0800, Tim Chen wrote:
> In my testing, the CPU utilization is at 100%.  So
> increase in ACKs will cost CPU to devote more
> time to process those ACKs and reduce throughput.

Oh, I see. I would test on a real network with real clients. I doubt
you would observe a noticeable effect there.

Olaf
-- 
Walks like a duck. Quacks like a duck. Must be a chicken.

Re: 2.6.19-rc1: Volanomark slowdown

[Stephen Hemminger]

On Wed, 8 Nov 2006 23:10:28 +0100
Olaf Kirch <okir@suse.de> wrote:

> On Wed, Nov 08, 2006 at 10:38:52AM -0800, Tim Chen wrote:
> > The patch in question affects purely TCP and not the scheduler.  I don't
> 
> I know.
> 
> > think the scheduler has anything to do with the slowdown seen after
> > the patch is applied.
> 
> In fixing performance issues, the most obvious explanation isn't always
> the right one. It's quite possible you're right, sure.
> 
> What I'm saying though is that it doesn't rhyme with what I've seen of
> Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
> come close to saturating a Gigabit pipe before it maxed out on CPU load.
> 
> > The total number of messages being exchanged around the chatrooms in 
> > Volanomark remain unchanged.  But ACKS increase by 3.5 times and
> > segments received increase by 38% from netstat.  
> 
> > So I think it is reasonable to conclude that the increase in TCP traffic
> > reduce the bandwidth and throughput in Volanomark.
> 
> You could count the number of outbound packets dropped on the server.
> 
> Olaf

Also under benchmark stress, the load can get so high that timers go
off that normally don't. For example, I have seen delayed ack timer
cause extra ack's when at lower loads the response happened quick enough
that the ACK was piggybacked.


-- 
Stephen Hemminger <shemminger@osdl.org>

Re: 2.6.19-rc1: Volanomark slowdown

[Rick Jones]

On Wed, 2006-11-08 at 23:10 +0100, Olaf Kirch wrote:
> What I'm saying though is that it doesn't rhyme with what I've seen of
> Volanomark - we ran 2.6.16 on a 4p Intel box for instance and it didn't
> come close to saturating a Gigabit pipe before it maxed out on CPU load.

That actually supports the hypothesis doesn't it?  The issue being the
increased number of ACKs causing additional CPU overhead not saturating
a NIC if any involved.

One of these days I may have to try to look more closely at what volano
does relative to netperf - I remember that someone tried very hard (was
it you alexy?) to show a perfomance effect with netperf and it didn't do
it :(

rick jones