Throughput Bug?

Throughput Bug?

[Matthew Faulkner]

Hey all

I'm using netperf to perform TCP throughput tests via the localhost
interface. This is being done on a SMP machine. I'm forcing the
netperf server and client to run on the same core. However, for any
packet sizes below 523 the throughput is much lower compared to the
throughput when the packet sizes are greater than 524.

Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB
 65536  65536    523    30.01        81.49   50.00    50.00    11.984  11.984
 65536  65536    524    30.01       460.61   49.99    49.99    2.120   2.120

The chances are i'm being stupid and there is an obvious reason for
this, but when i put  the server and client on different cores i don't
see this effect.

Any help explaining this will be greatly appreciated.

Machine details:

Linux 2.6.22-2-amd64 #1 SMP Thu Aug 30 23:43:59 UTC 2007 x86_64 GNU/Linux

sched_affinity is used by netperf internally to set the core affinity.

I tried this on 2.6.18 and i got the same problem!

Re: Throughput Bug?

[Rick Jones]

Matthew Faulkner wrote:
> Hey all
> 
> I'm using netperf to perform TCP throughput tests via the localhost
> interface. This is being done on a SMP machine. I'm forcing the
> netperf server and client to run on the same core. However, for any
> packet sizes below 523 the throughput is much lower compared to the
> throughput when the packet sizes are greater than 524.
> 
> Recv   Send    Send                          Utilization       Service Demand
> Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
> Size   Size    Size     Time     Throughput  local    remote   local   remote
> bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB
>  65536  65536    523    30.01        81.49   50.00    50.00    11.984  11.984
>  65536  65536    524    30.01       460.61   49.99    49.99    2.120   2.120
> 
> The chances are i'm being stupid and there is an obvious reason for
> this, but when i put  the server and client on different cores i don't
> see this effect.
> 
> Any help explaining this will be greatly appreciated.

One minor nit, but perhaps one that may help in the diagnosis - unless you set 
-D (lack of the full test banner, or a copy of the command line precludes 
knowing), and perhaps even then, all the -m option _really_ does for a 
TCP_STREAM test is set the size of the buffer passed to the transport on each 
send() call.  It is then entirely up to TCP as to how that gets 
merged/sliced/diced into TCP segments.

I forget what the MTU is of loopback, but you can get netperf to report the MSS 
for the connection by setting verbosity to 2 or more with the global -v option.

A packet trace might be interesting.  Seems that is possible under Linux with 
tcpdump.  If it were not possible, another netperf-level thing I might do is 
configure with --enable-histogram and recompile netperf (netserver does not need 
to be recompiled, although it doesn't take much longer once netperf is 
recompiled) and use the -v 2 again.  That will give you a histogram of the time 
spent in the send() call, which might be interesting if it ever blocks.


> Machine details:
> 
> Linux 2.6.22-2-amd64 #1 SMP Thu Aug 30 23:43:59 UTC 2007 x86_64 GNU/Linux

FWIW, with an "earlier" kernel I am not sure I can name since I'm not sure it is 
shipping (sorry, it was just what was on my system at the moment) don't see that 
_big_ difference between 523 and 524 regardless of TCP_NODELAY:

[root@hpcpc105 netperf2_trunk]# netperf -T 0 -c -C -- -m 524
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain 
(127.0.0.1) port 0 AF_INET : cpu bind
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

  87380  87380    524    10.00      2264.18   25.00    25.00    3.618   3.618
[root@hpcpc105 netperf2_trunk]# netperf -T 0 -c -C -- -m 523
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain 
(127.0.0.1) port 0 AF_INET : cpu bind
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

  87380  87380    523    10.00      3356.05   25.01    25.01    2.442   2.442


[root@hpcpc105 netperf2_trunk]# netperf -T 0 -c -C -- -m 523 -D
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain 
(127.0.0.1) port 0 AF_INET : nodelay : cpu bind
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

  87380  87380    523    10.00       398.87   25.00    25.00    20.539  20.537
[root@hpcpc105 netperf2_trunk]# netperf -T 0 -c -C -- -m 524 -D
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain 
(127.0.0.1) port 0 AF_INET : nodelay : cpu bind
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

  87380  87380    524    10.00       439.33   25.00    25.00    18.646  18.644

Although, if I do constrain the socket buffers to 64KB I _do_ see the behaviour 
on the older kernel as well:

[root@hpcpc105 netperf2_trunk]# netperf -T 0 -c -C -- -m 523 -s 64K -S 64K
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain 
(127.0.0.1) port 0 AF_INET : cpu bind
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

131072 131072    523    10.00       406.61   25.00    25.00    20.146  20.145
[root@hpcpc105 netperf2_trunk]# netperf -T 0 -c -C -- -m 524 -s 64K -S 64K
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost.localdomain 
(127.0.0.1) port 0 AF_INET : cpu bind
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

131072 131072    524    10.00      2017.12   25.02    25.03    4.065   4.066


(yes, this is a four-core system, hence 25% CPU util reported by netperf).

> sched_affinity is used by netperf internally to set the core affinity.
> 
> I tried this on 2.6.18 and i got the same problem!

I can say that the kernel I tried was based on 2.6.18...  So, due dilligence and 
no good deed going unpunished suggests that Matthew and I are now in a race to 
take some tcpdump traces :)

rick jones

> -
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Throughput Bug?

[Bill Fink]

On Thu, 18 Oct 2007, Matthew Faulkner wrote:

> Hey all
> 
> I'm using netperf to perform TCP throughput tests via the localhost
> interface. This is being done on a SMP machine. I'm forcing the
> netperf server and client to run on the same core. However, for any
> packet sizes below 523 the throughput is much lower compared to the
> throughput when the packet sizes are greater than 524.
> 
> Recv   Send    Send                          Utilization       Service Demand
> Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
> Size   Size    Size     Time     Throughput  local    remote   local   remote
> bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB
>  65536  65536    523    30.01        81.49   50.00    50.00    11.984  11.984
>  65536  65536    524    30.01       460.61   49.99    49.99    2.120   2.120
> 
> The chances are i'm being stupid and there is an obvious reason for
> this, but when i put  the server and client on different cores i don't
> see this effect.
> 
> Any help explaining this will be greatly appreciated.
> 
> Machine details:
> 
> Linux 2.6.22-2-amd64 #1 SMP Thu Aug 30 23:43:59 UTC 2007 x86_64 GNU/Linux
> 
> sched_affinity is used by netperf internally to set the core affinity.

I don't know if it's relevant, but note that 524 bytes + 52 bytes
of IP(20)/TCP(20)/TimeStamp(12) overhead gives a 576 byte packet,
which is the specified size that all IP routers must handle (and
the smallest value possible during PMTU discovery I believe).  A
message size of 523 bytes would be 1 less than that.  Could this
possibly have to do with ABC (possibly try disabling it if set)?

						-Bill

Re: Throughput Bug?

[Matthew Faulkner]

I removed the socket sizes in an attempt to reproduce your results
Rick and i managed to do so, but only when i launch netperf by typing
in the follow cmd in to the bash shell.

/home/cheka/netperf-2.4.4/src/netperf -T 0,0 -l 10 -t TCP_STREAM -c
100 -C 100 -f M -P 0 -- -m 523

As soon as i try to launch netperf (with the same line as i do manual)
from within a script of any form (be it php or bash) the difference
between 523 and 524 appears again.

The php script i'm using is pasted below (it's the same as the bash
script that comes with netperf to provide the tcp_stream)

<?php
        $START=522;
        $END=524;
        $MULT=1;
        $ADD=1;
        $MAXPROC=1; // This is the maximum number of CPU's you have so
we can assign the client to different CPUs to show the same problem
between 523 and 524 does not occur unless it's on CPU 0 and CPU 0

        $DURATION = 10; // Length of test
        $LOC_CPU = "-c 100"; // Report the local CPU info
        $REM_CPU = "-C 100"; // Report the remove CPU info

       $NETSERVER = "netserver"; //path to netserver
       $NETPERF = "netperf"; // path to netperf

        for($i=0; $i<=$MAXPROC; $i++) {
                echo "0,$i\n";
                $MESSAGE = $START;

                while($MESSAGE <= $END) {
                        passthru('killall netserver > /dev/null'); //
tried it with and without the following restarts of netserver
                        passthru('sleep 5');
                        passthru("$NETSERVER");
                        passthru('sleep 5');
                        echo "$NETPERF -T 0,$i -l $DURATION -t
TCP_STREAM $LOC_CPU $REM_CPU -f M -P 0 -- -m $MESSAGE\n"; // let's see
what we try to exec
                        passthru("$NETPERF -T 0,$i -l $DURATION -t
TCP_STREAM $LOC_CPU $REM_CPU -f M -P 0 -- -m $MESSAGE"); // exec it -
this will also print to screen
                        passthru('sleep 5'); // sleep
                        $MESSAGE += $ADD;
                        $MESSAGE *= $MULT;
                }
        }
?>


On 19/10/2007, Bill Fink <billfink@mindspring.com> wrote:
> On Thu, 18 Oct 2007, Matthew Faulkner wrote:
>
> > Hey all
> >
> > I'm using netperf to perform TCP throughput tests via the localhost
> > interface. This is being done on a SMP machine. I'm forcing the
> > netperf server and client to run on the same core. However, for any
> > packet sizes below 523 the throughput is much lower compared to the
> > throughput when the packet sizes are greater than 524.
> >
> > Recv   Send    Send                          Utilization       Service Demand
> > Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
> > Size   Size    Size     Time     Throughput  local    remote   local   remote
> > bytes  bytes   bytes    secs.    MBytes  /s  % S      % S      us/KB   us/KB
> >  65536  65536    523    30.01        81.49   50.00    50.00    11.984  11.984
> >  65536  65536    524    30.01       460.61   49.99    49.99    2.120   2.120
> >
> > The chances are i'm being stupid and there is an obvious reason for
> > this, but when i put  the server and client on different cores i don't
> > see this effect.
> >
> > Any help explaining this will be greatly appreciated.
> >
> > Machine details:
> >
> > Linux 2.6.22-2-amd64 #1 SMP Thu Aug 30 23:43:59 UTC 2007 x86_64 GNU/Linux
> >
> > sched_affinity is used by netperf internally to set the core affinity.
>
> I don't know if it's relevant, but note that 524 bytes + 52 bytes
> of IP(20)/TCP(20)/TimeStamp(12) overhead gives a 576 byte packet,
> which is the specified size that all IP routers must handle (and
> the smallest value possible during PMTU discovery I believe).  A
> message size of 523 bytes would be 1 less than that.  Could this
> possibly have to do with ABC (possibly try disabling it if set)?
>
>                                                 -Bill
>

Re: Throughput Bug?

[Rick Jones]

Matthew Faulkner wrote:
> I removed the socket sizes in an attempt to reproduce your results
> Rick and i managed to do so, but only when i launch netperf by typing
> in the follow cmd in to the bash shell.
> 
> /home/cheka/netperf-2.4.4/src/netperf -T 0,0 -l 10 -t TCP_STREAM -c
> 100 -C 100 -f M -P 0 -- -m 523
> 
> As soon as i try to launch netperf (with the same line as i do manual)
> from within a script of any form (be it php or bash) the difference
> between 523 and 524 appears again.
> 
> The php script i'm using is pasted below (it's the same as the bash
> script that comes with netperf to provide the tcp_stream)

well, bash on some platforms I guess - when those netperf scripts were first 
written, i'm not even sure bash was a gleam in its author's eye :)

> <?php
>         $START=522;
>         $END=524;
>         $MULT=1;
>         $ADD=1;
>         $MAXPROC=1; // This is the maximum number of CPU's you have so
> we can assign the client to different CPUs to show the same problem
> between 523 and 524 does not occur unless it's on CPU 0 and CPU 0
> 
>         $DURATION = 10; // Length of test
>         $LOC_CPU = "-c 100"; // Report the local CPU info
>         $REM_CPU = "-C 100"; // Report the remove CPU info
> 
>        $NETSERVER = "netserver"; //path to netserver
>        $NETPERF = "netperf"; // path to netperf
> 
>         for($i=0; $i<=$MAXPROC; $i++) {
>                 echo "0,$i\n";
>                 $MESSAGE = $START;
> 
>                 while($MESSAGE <= $END) {
>                         passthru('killall netserver > /dev/null'); //
> tried it with and without the following restarts of netserver
>                         passthru('sleep 5');
>                         passthru("$NETSERVER");
>                         passthru('sleep 5');
>                         echo "$NETPERF -T 0,$i -l $DURATION -t
> TCP_STREAM $LOC_CPU $REM_CPU -f M -P 0 -- -m $MESSAGE\n"; // let's see
> what we try to exec
>                         passthru("$NETPERF -T 0,$i -l $DURATION -t
> TCP_STREAM $LOC_CPU $REM_CPU -f M -P 0 -- -m $MESSAGE"); // exec it -
> this will also print to screen
>                         passthru('sleep 5'); // sleep
>                         $MESSAGE += $ADD;
>                         $MESSAGE *= $MULT;
>                 }
>         }
> ?>

While I wouldn't know broken php if it reared up and bit me on the backside, the 
above looks like a fairly straightforward trnaslation of some of the old netperf 
scripts with the add and mult  bits.  I don't see anything amis there.

While it is often a case of famous last words, I've dropped netperf-talk from 
this as I don't think there is a netperf issue, just an issue demonstrated with 
netperf.  Besides, netperf-talk, being a closed list (my simplistic attempts to 
deal with spam) would cause problems for most readers of netdev when/if they 
were to contribute to the thread...

> 
> 
> On 19/10/2007, Bill Fink <billfink@mindspring.com> wrote:
>>I don't know if it's relevant, but note that 524 bytes + 52 bytes
>>of IP(20)/TCP(20)/TimeStamp(12) overhead gives a 576 byte packet,
>>which is the specified size that all IP routers must handle (and
>>the smallest value possible during PMTU discovery I believe).  A
>>message size of 523 bytes would be 1 less than that.  Could this
>>possibly have to do with ABC (possibly try disabling it if set)?

ABC might be good to check.  It might also be worthwhile to try setting the 
lowlatency sysctl - both processes being on the same CPU might interact poorly 
with the attempts to run things on the receiver's stack.

rick jones

I guess I've not managed to lose the race to a packet trace... :)