[RFC] gianfar: low gigabit throughput

[RFC] gianfar: low gigabit throughput

[Anton Vorontsov]

Hi all,

Down here few question regarding networking throughput, I would
appreciate any thoughts or ideas.

On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing
relatively low TCP throughput using gianfar driver...

The maximum value I've seen with the current kernels is 142 Mb/s of TCP
and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):

  root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
  TCP STREAM TEST to 10.0.1.1
  #Cpu utilization 0.10
  Recv   Send    Send
  Socket Socket  Message  Elapsed
  Size   Size    Size     Time     Throughput
  bytes  bytes   bytes    secs.    10^6bits/sec

  206848 212992  32768    10.00     142.40

  root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344
  UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1
  #Cpu utilization 100.00
  Socket  Message  Elapsed      Messages
  Size    Size     Time         Okay Errors   Throughput
  bytes   bytes    secs            #      #   10^6bits/sec

  212992   32768   10.00       13539      0     354.84
  206848           10.00       13539            354.84


Is this normal?

netperf running in loopback gives me 329 Mb/s of TCP throughput:

  root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
  TCP STREAM TEST to 127.0.0.1
  #Cpu utilization 100.00
  #Cpu utilization 100.00
  Recv   Send    Send
  Socket Socket  Message  Elapsed
  Size   Size    Size     Time     Throughput
  bytes  bytes   bytes    secs.    10^6bits/sec

  212992 212992  32768    10.00     329.60


May I consider this as a something that is close to the Linux'
theoretical maximum for this setup? Or this isn't reliable test?


I can compare with teh MPC8377E-RDB (very similar board - exactly the same
ethernet phy, the same drivers are used, i.e. everything is the same from
the ethernet stand point), but running at 666 MHz, CSB at 333MHz:

         |CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
  ------------------------------------------
  MPC8377|    666|    333|     646|     264|
  MPC8315|    400|    133|     354|     142|
  ------------------------------------------
  RATIO  |    1.6|    2.5|     1.8|     1.8|

It seems that things are really dependant on the CPU/CSB speed.

I've tried to tune gianfar driver in various ways... and it gave
some positive results with this patch:

diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
index fd487be..b5943f9 100644
--- a/drivers/net/gianfar.h
+++ b/drivers/net/gianfar.h
@@ -123,8 +123,8 @@ extern const char gfar_driver_version[];
 #define GFAR_10_TIME    25600
 
 #define DEFAULT_TX_COALESCE 1
-#define DEFAULT_TXCOUNT	16
-#define DEFAULT_TXTIME	21
+#define DEFAULT_TXCOUNT	80
+#define DEFAULT_TXTIME	105
 
 #define DEFAULT_RXTIME	21


Basically this raises the tx interrupts coalescing threshold (raising
it more didn't help, as well as didn't help raising rx thresholds).
Now:

root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
TCP STREAM TEST to 10.0.1.1
#Cpu utilization 100.00
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

206848 212992  32768    3.00      163.04


That is +21 Mb/s (14% up). Not fantastic, but good anyway.

As expected, the latency increased too:

Before the patch:

--- 10.0.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 18997ms
rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 ms

After:

--- 10.0.1.1 ping statistics ---
22 packets transmitted, 22 received, 0% packet loss, time 20997ms
rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms


34% up... heh. Should we sacrifice the latency in favour of throughput?
Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of
latency? ;-)


Thanks in advance,

p.s. Btw, the patch above helps even better on the on the -rt kernels,
since on the -rt kernels the throughput is near 100 Mb/s, with the
patch the throughput is close to 140 Mb/s.

-- 
Anton Vorontsov
email: cbouatmailru@gmail.com
irc://irc.freenode.net/bd2

Re: [RFC] gianfar: low gigabit throughput

[Rick Jones]

Anton Vorontsov wrote:
> Hi all,
> 
> Down here few question regarding networking throughput, I would
> appreciate any thoughts or ideas.
> 
> On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing
> relatively low TCP throughput using gianfar driver...

What is the "target" of the test - is it another of those boards, or 
something else?

> 
> The maximum value I've seen with the current kernels is 142 Mb/s of TCP
> and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
> 
>   root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
>   TCP STREAM TEST to 10.0.1.1
>   #Cpu utilization 0.10
>   Recv   Send    Send
>   Socket Socket  Message  Elapsed
>   Size   Size    Size     Time     Throughput
>   bytes  bytes   bytes    secs.    10^6bits/sec
> 
>   206848 212992  32768    10.00     142.40
> 
>   root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344
>   UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1
>   #Cpu utilization 100.00
>   Socket  Message  Elapsed      Messages
>   Size    Size     Time         Okay Errors   Throughput
>   bytes   bytes    secs            #      #   10^6bits/sec
> 
>   212992   32768   10.00       13539      0     354.84
>   206848           10.00       13539            354.84
> 

I have _got_ to make CPU utilization enabled by default one of these 
days :)  At least for mechanisms which don't require calibration.

> Is this normal?

Does gianfar do TSO?  If not, what happens when you tell UDP_STREAM to 
send 1472 byte messages to bypass IP fragmentation?

While stock netperf won't report what the socket buffer size becomes 
when you allow autotuning to rear its head, you can take the top of 
trunk and enable the "omni" tests (./configure --enable-omni) and those 
versions of *_STREAM etc can report what the socket buffer size was at 
the beginning and at the end of the test. You can let the stack autotune 
and see if anything changes there.  You can do the same with stock 
netperf, just it will only report the initial socket buffer sizes...

> netperf running in loopback gives me 329 Mb/s of TCP throughput:
> 
>   root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
>   TCP STREAM TEST to 127.0.0.1
>   #Cpu utilization 100.00
>   #Cpu utilization 100.00
>   Recv   Send    Send
>   Socket Socket  Message  Elapsed
>   Size   Size    Size     Time     Throughput
>   bytes  bytes   bytes    secs.    10^6bits/sec
> 
>   212992 212992  32768    10.00     329.60
> 
> 
> May I consider this as a something that is close to the Linux'
> theoretical maximum for this setup? Or this isn't reliable test?

I'm always leery of using a loopback number.  It excercises both send 
and receive at the same time, but without the driver.  Also, lo tends to 
have a much larger MTU than a "standard" NIC and if that NIC doesn't to 
TSO and LRO that can be a big difference in the number of times up and 
down the stack per KB transferred.

> I can compare with teh MPC8377E-RDB (very similar board - exactly the same
> ethernet phy, the same drivers are used, i.e. everything is the same from
> the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
> 
>          |CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
>   ------------------------------------------
>   MPC8377|    666|    333|     646|     264|
>   MPC8315|    400|    133|     354|     142|
>   ------------------------------------------
>   RATIO  |    1.6|    2.5|     1.8|     1.8|
> 
> It seems that things are really dependant on the CPU/CSB speed.

What is the nature of the DMA stream between the two tests?  I find it 
interesting that the TCP Mb/s went up by more than the CPU MHz and 
wonder how much the Bus MHz came into play there - perhaps there were 
more DMA's to setup or across a broader memory footprint for TCP than 
for UDP.

> 
> I've tried to tune gianfar driver in various ways... and it gave
> some positive results with this patch:
> 
> diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
> index fd487be..b5943f9 100644
> --- a/drivers/net/gianfar.h
> +++ b/drivers/net/gianfar.h
> @@ -123,8 +123,8 @@ extern const char gfar_driver_version[];
>  #define GFAR_10_TIME    25600
>  
>  #define DEFAULT_TX_COALESCE 1
> -#define DEFAULT_TXCOUNT	16
> -#define DEFAULT_TXTIME	21
> +#define DEFAULT_TXCOUNT	80
> +#define DEFAULT_TXTIME	105
>  
>  #define DEFAULT_RXTIME	21

No ethtool coalescing tuning support for gianfar?-)

> Basically this raises the tx interrupts coalescing threshold (raising
> it more didn't help, as well as didn't help raising rx thresholds).
> Now:
> 
> root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
> TCP STREAM TEST to 10.0.1.1
> #Cpu utilization 100.00
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
> 
> 206848 212992  32768    3.00      163.04
> 
> 
> That is +21 Mb/s (14% up). Not fantastic, but good anyway.
> 
> As expected, the latency increased too:
> 
> Before the patch:
> 
> --- 10.0.1.1 ping statistics ---
> 20 packets transmitted, 20 received, 0% packet loss, time 18997ms
> rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 ms
> 
> After:
> 
> --- 10.0.1.1 ping statistics ---
> 22 packets transmitted, 22 received, 0% packet loss, time 20997ms
> rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms
> 
> 
> 34% up... heh. Should we sacrifice the latency in favour of throughput?
> Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of
> latency? ;-)

Well, I'm not always fond of that sort of trade-off:

ftp://ftp.cup.hp.com/dist/networking/briefs/

there should be a nic latency vs tput writeup there.

> 
> 
> Thanks in advance,
> 
> p.s. Btw, the patch above helps even better on the on the -rt kernels,
> since on the -rt kernels the throughput is near 100 Mb/s, with the
> patch the throughput is close to 140 Mb/s.
> 

Re: [RFC] gianfar: low gigabit throughput

[Andy Fleming]

>>
>> I've tried to tune gianfar driver in various ways... and it gave
>> some positive results with this patch:
>> diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
>> index fd487be..b5943f9 100644
>> --- a/drivers/net/gianfar.h
>> +++ b/drivers/net/gianfar.h
>> @@ -123,8 +123,8 @@ extern const char gfar_driver_version[];
>> #define GFAR_10_TIME    25600
>>  #define DEFAULT_TX_COALESCE 1
>> -#define DEFAULT_TXCOUNT	16
>> -#define DEFAULT_TXTIME	21
>> +#define DEFAULT_TXCOUNT	80
>> +#define DEFAULT_TXTIME	105
>>  #define DEFAULT_RXTIME	21
>
> No ethtool coalescing tuning support for gianfar?-)

Yeah, there's coalescing tuning in gianfar.

Anton, those numbers aren't too surprising on a 400 MHz machine, I  
think.  But I'd be happy to see any analysis on performance  
bottlenecks in the driver.  And patches to fix those bottlenecks are  
even better!  :)

Andy

Re: [RFC] gianfar: low gigabit throughput

[Anton Vorontsov]

On Tue, May 06, 2008 at 01:07:14PM -0700, Rick Jones wrote:
> Anton Vorontsov wrote:
>> Hi all,
>>
>> Down here few question regarding networking throughput, I would
>> appreciate any thoughts or ideas.
>>
>> On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing
>> relatively low TCP throughput using gianfar driver...
>
> What is the "target" of the test - is it another of those boards, or  
> something else?

I've tried in various ways (except for another of these boards, I don't
have any), for example really fast machine with gbit ethernet, but most
of the time I'm testing with MPC8315 and MPC8377 interconnected.

The other interesting thing is that when netserver is running on the
MPC8315 (slow) target, and netperf on the MPC8377 (fast), the TCP
and UDP throughput increases.

root@mpc837x_rdb:~# netperf -l 3 -H 10.0.1.2 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
TCP STREAM TEST to 10.0.1.2
#Cpu utilization 40.66
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

212992 206848  32768    3.00      309.87   

So the slow target can receive TCP packets with 300 Mb/s, but TCP
packet generation (or transmitting) is slower.

>>
>> The maximum value I've seen with the current kernels is 142 Mb/s of TCP
>> and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
>>
>>   root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
>>   TCP STREAM TEST to 10.0.1.1
>>   #Cpu utilization 0.10
>>   Recv   Send    Send
>>   Socket Socket  Message  Elapsed
>>   Size   Size    Size     Time     Throughput
>>   bytes  bytes   bytes    secs.    10^6bits/sec
>>
>>   206848 212992  32768    10.00     142.40
>>
>>   root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344
>>   UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1
>>   #Cpu utilization 100.00
>>   Socket  Message  Elapsed      Messages
>>   Size    Size     Time         Okay Errors   Throughput
>>   bytes   bytes    secs            #      #   10^6bits/sec
>>
>>   212992   32768   10.00       13539      0     354.84
>>   206848           10.00       13539            354.84
>>
>
> I have _got_ to make CPU utilization enabled by default one of these  
> days :)  At least for mechanisms which don't require calibration.

Heh, I've skipped the calibration chapter in the netperf manual. :-D
Should revert to it.

>> Is this normal?
>
> Does gianfar do TSO?

Afaik, it doesn't. The hardware can do header recognition/verification
including checksums, and also could generate checksums for the TCP/IP.

But IP fragmentation and re-assembly on the software's shoulders.

> If not, what happens when you tell UDP_STREAM to  
> send 1472 byte messages to bypass IP fragmentation?

A bit worse throughput:

root@b1:~# netperf -l 3 -H 10.0.1.1 -t UDP_STREAM -- -m 1472 -s 157344 -S 157344
UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1
#Cpu utilization 100.00
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992    1472   3.00        73377      0     287.86
206848           3.00        73377            287.86

And 32 * 1472:

root@b1:~# netperf -l 3 -H 10.0.1.1 -t UDP_STREAM -- -m 47104 -s 157344 -S 157344
UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1
#Cpu utilization 100.00
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

212992   47104   3.00         3124      0     392.13
206848           3.00         3124            392.13

So things becomes much better when the message size increases
(I think the netperf then eating less cpu, and gives some precessing
time to the kernel?).

The same for TCP packets with mlen of 1448.

> While stock netperf won't report what the socket buffer size becomes  
> when you allow autotuning to rear its head, you can take the top of  
> trunk and enable the "omni" tests (./configure --enable-omni) and those  
> versions of *_STREAM etc can report what the socket buffer size was at  
> the beginning and at the end of the test. You can let the stack autotune  
> and see if anything changes there.  You can do the same with stock  
> netperf, just it will only report the initial socket buffer sizes...

Thanks, will try.

>> netperf running in loopback gives me 329 Mb/s of TCP throughput:
>>
>>   root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
>>   TCP STREAM TEST to 127.0.0.1
>>   #Cpu utilization 100.00
>>   #Cpu utilization 100.00
>>   Recv   Send    Send
>>   Socket Socket  Message  Elapsed
>>   Size   Size    Size     Time     Throughput
>>   bytes  bytes   bytes    secs.    10^6bits/sec
>>
>>   212992 212992  32768    10.00     329.60
>>
>>
>> May I consider this as a something that is close to the Linux'
>> theoretical maximum for this setup? Or this isn't reliable test?
>
> I'm always leery of using a loopback number.  It excercises both send  
> and receive at the same time, but without the driver.  Also, lo tends to  
> have a much larger MTU than a "standard" NIC and if that NIC doesn't to  
> TSO and LRO that can be a big difference in the number of times up and  
> down the stack per KB transferred.

I see. Will be cautious with it, too. ;-)

>> I can compare with teh MPC8377E-RDB (very similar board - exactly the same
>> ethernet phy, the same drivers are used, i.e. everything is the same from
>> the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
>>
>>          |CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
>>   ------------------------------------------
>>   MPC8377|    666|    333|     646|     264|
>>   MPC8315|    400|    133|     354|     142|
>>   ------------------------------------------
>>   RATIO  |    1.6|    2.5|     1.8|     1.8|
>>
>> It seems that things are really dependant on the CPU/CSB speed.
>
> What is the nature of the DMA stream between the two tests?  I find it  
> interesting that the TCP Mb/s went up by more than the CPU MHz and  
> wonder how much the Bus MHz came into play there - perhaps there were  
> more DMA's to setup or across a broader memory footprint for TCP than  
> for UDP.

The gianfar indeed does a lot of dma on the "buffer descriptors", so
probably the bus speed matters a lot. And combination of CPU and Bus
gives the final result.

>>
>> I've tried to tune gianfar driver in various ways... and it gave
>> some positive results with this patch:
>>
>> diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
>> index fd487be..b5943f9 100644
>> --- a/drivers/net/gianfar.h
>> +++ b/drivers/net/gianfar.h
>> @@ -123,8 +123,8 @@ extern const char gfar_driver_version[];
>>  #define GFAR_10_TIME    25600
>>   #define DEFAULT_TX_COALESCE 1
>> -#define DEFAULT_TXCOUNT	16
>> -#define DEFAULT_TXTIME	21
>> +#define DEFAULT_TXCOUNT	80
>> +#define DEFAULT_TXTIME	105
>>   #define DEFAULT_RXTIME	21
>
> No ethtool coalescing tuning support for gianfar?-)

Heh. :-) I should have looked into gianfar_ethtool.c before editing
anything. Yes, there is.

>> Basically this raises the tx interrupts coalescing threshold (raising
>> it more didn't help, as well as didn't help raising rx thresholds).
>> Now:
>>
>> root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
>> TCP STREAM TEST to 10.0.1.1
>> #Cpu utilization 100.00
>> Recv   Send    Send
>> Socket Socket  Message  Elapsed
>> Size   Size    Size     Time     Throughput
>> bytes  bytes   bytes    secs.    10^6bits/sec
>>
>> 206848 212992  32768    3.00      163.04
>>
>>
>> That is +21 Mb/s (14% up). Not fantastic, but good anyway.
>>
>> As expected, the latency increased too:
>>
>> Before the patch:
>>
>> --- 10.0.1.1 ping statistics ---
>> 20 packets transmitted, 20 received, 0% packet loss, time 18997ms
>> rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 ms
>>
>> After:
>>
>> --- 10.0.1.1 ping statistics ---
>> 22 packets transmitted, 22 received, 0% packet loss, time 20997ms
>> rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms
>>
>>
>> 34% up... heh. Should we sacrifice the latency in favour of throughput?
>> Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of
>> latency? ;-)
>
> Well, I'm not always fond of that sort of trade-off:
>
> ftp://ftp.cup.hp.com/dist/networking/briefs/
>
> there should be a nic latency vs tput writeup there.

Thanks!

-- 
Anton Vorontsov
email: cbouatmailru@gmail.com
irc://irc.freenode.net/bd2

Re: [RFC] gianfar: low gigabit throughput

[Anton Vorontsov]

On Tue, May 06, 2008 at 03:29:06PM -0500, Andy Fleming wrote:
>>>
>>> I've tried to tune gianfar driver in various ways... and it gave
>>> some positive results with this patch:
>>> diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
>>> index fd487be..b5943f9 100644
>>> --- a/drivers/net/gianfar.h
>>> +++ b/drivers/net/gianfar.h
>>> @@ -123,8 +123,8 @@ extern const char gfar_driver_version[];
>>> #define GFAR_10_TIME    25600
>>>  #define DEFAULT_TX_COALESCE 1
>>> -#define DEFAULT_TXCOUNT	16
>>> -#define DEFAULT_TXTIME	21
>>> +#define DEFAULT_TXCOUNT	80
>>> +#define DEFAULT_TXTIME	105
>>>  #define DEFAULT_RXTIME	21
>>
>> No ethtool coalescing tuning support for gianfar?-)
>
> Yeah, there's coalescing tuning in gianfar.
>
> Anton, those numbers aren't too surprising on a 400 MHz machine, I  
> think.

This is my thinking too, actually. I've seen the MPC885 board with CPU
running at 133 MHz and CSB at 66MHz. It was using the completely
different driver (fs_enet's FEC), and it failed to produce even 60 Mb/s
(with a 100 Mb/s PHY). 50-55 Mb/s was quite normal for that board.

Which means, I think, that ~160 Mb/s is quite good for the 400/133
MHz board.

> But I'd be happy to see any analysis on performance bottlenecks 
> in the driver.  And patches to fix those bottlenecks are even better!  :)

The thing is that I don't see obvious bottlenecks, so I'm afraid we'll
not able to do x2 boost or something... Just fine tuning the driver here
and there... Interrupts coalescing was very obvious step to boost the
troughtput, but there are its drawbacks...

Thanks,

-- 
Anton Vorontsov
email: cbouatmailru@gmail.com
irc://irc.freenode.net/bd2

Re: [RFC] gianfar: low gigabit throughput

[André Schwarz]

Anton,

we've just built a digital GigEVision camera based on a MPC8343 running=20
at 266/400 csb/core speed.

Transmission is done from a kernel module that allocates skb into which=20
the image data is DMA'd by an external PCI master.
As soon as the image data is complete all buffers are sent out via=20
dev->hard_start_xmit ...

Bandwidth is currently 1.3MPixel @ 50Hz which give 65MBytes/sec=20
(~520MBit/s).
Of course it's UDP _without_ checksumming ....

Actually I have no sensor available that gives higher bandwidth ... but=20
having a look at transmission time I'm sure the MPC8343 can easily go up=20
to +800MBit.


Obviously your cpu time is consumed on a higher level.

Cheers,
Andr=E9


Anton Vorontsov wrote:
> Hi all,
>
> Down here few question regarding networking throughput, I would
> appreciate any thoughts or ideas.
>
> On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing
> relatively low TCP throughput using gianfar driver...
>
> The maximum value I've seen with the current kernels is 142 Mb/s of TCP
> and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
>
>   root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157=
344 -S 157344
>   TCP STREAM TEST to 10.0.1.1
>   #Cpu utilization 0.10
>   Recv   Send    Send
>   Socket Socket  Message  Elapsed
>   Size   Size    Size     Time     Throughput
>   bytes  bytes   bytes    secs.    10^6bits/sec
>
>   206848 212992  32768    10.00     142.40
>
>   root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157=
344 -S 157344
>   UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1
>   #Cpu utilization 100.00
>   Socket  Message  Elapsed      Messages
>   Size    Size     Time         Okay Errors   Throughput
>   bytes   bytes    secs            #      #   10^6bits/sec
>
>   212992   32768   10.00       13539      0     354.84
>   206848           10.00       13539            354.84
>
>
> Is this normal?
>
> netperf running in loopback gives me 329 Mb/s of TCP throughput:
>
>   root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 15=
7344 -S 157344
>   TCP STREAM TEST to 127.0.0.1
>   #Cpu utilization 100.00
>   #Cpu utilization 100.00
>   Recv   Send    Send
>   Socket Socket  Message  Elapsed
>   Size   Size    Size     Time     Throughput
>   bytes  bytes   bytes    secs.    10^6bits/sec
>
>   212992 212992  32768    10.00     329.60
>
>
> May I consider this as a something that is close to the Linux'
> theoretical maximum for this setup? Or this isn't reliable test?
>
>
> I can compare with teh MPC8377E-RDB (very similar board - exactly the s=
ame
> ethernet phy, the same drivers are used, i.e. everything is the same fr=
om
> the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
>
>          |CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
>   ------------------------------------------
>   MPC8377|    666|    333|     646|     264|
>   MPC8315|    400|    133|     354|     142|
>   ------------------------------------------
>   RATIO  |    1.6|    2.5|     1.8|     1.8|
>
> It seems that things are really dependant on the CPU/CSB speed.
>
> I've tried to tune gianfar driver in various ways... and it gave
> some positive results with this patch:
>
> diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
> index fd487be..b5943f9 100644
> --- a/drivers/net/gianfar.h
> +++ b/drivers/net/gianfar.h
> @@ -123,8 +123,8 @@ extern const char gfar_driver_version[];
>  #define GFAR_10_TIME    25600
> =20
>  #define DEFAULT_TX_COALESCE 1
> -#define DEFAULT_TXCOUNT	16
> -#define DEFAULT_TXTIME	21
> +#define DEFAULT_TXCOUNT	80
> +#define DEFAULT_TXTIME	105
> =20
>  #define DEFAULT_RXTIME	21
>
>
> Basically this raises the tx interrupts coalescing threshold (raising
> it more didn't help, as well as didn't help raising rx thresholds).
> Now:
>
> root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344=
 -S 157344
> TCP STREAM TEST to 10.0.1.1
> #Cpu utilization 100.00
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
>
> 206848 212992  32768    3.00      163.04
>
>
> That is +21 Mb/s (14% up). Not fantastic, but good anyway.
>
> As expected, the latency increased too:
>
> Before the patch:
>
> --- 10.0.1.1 ping statistics ---
> 20 packets transmitted, 20 received, 0% packet loss, time 18997ms
> rtt min/avg/max/mdev =3D 0.108/0.124/0.173/0.022 ms
>
> After:
>
> --- 10.0.1.1 ping statistics ---
> 22 packets transmitted, 22 received, 0% packet loss, time 20997ms
> rtt min/avg/max/mdev =3D 0.158/0.167/0.182/0.004 ms
>
>
> 34% up... heh. Should we sacrifice the latency in favour of throughput?
> Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of
> latency? ;-)
>
>
> Thanks in advance,
>
> p.s. Btw, the patch above helps even better on the on the -rt kernels,
> since on the -rt kernels the throughput is near 100 Mb/s, with the
> patch the throughput is close to 140 Mb/s.
>
>  =20


MATRIX VISION GmbH, Talstra=DFe 16, DE-71570 Oppenweiler  - Registergeric=
ht: Amtsgericht Stuttgart, HRB 271090
Gesch=E4ftsf=FChrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner

Re: [RFC] gianfar: low gigabit throughput

[Anton Vorontsov]

On Wed, May 07, 2008 at 05:52:57PM +0200, André Schwarz wrote:
> Anton,
>

Much thanks for the information!

> we've just built a digital GigEVision camera based on a MPC8343 running  
> at 266/400 csb/core speed.
>
> Transmission is done from a kernel module that allocates skb into which  
> the image data is DMA'd by an external PCI master.
> As soon as the image data is complete all buffers are sent out via  
> dev->hard_start_xmit ...

Ah. So no userspace and packet generation expenses.. I see. This should
be definitely faster than netperf. But generally this fits into the
picture: MPC8315 running at 266 CSB would probably give better UDP
througput (300 Mb/s currenty).

> Bandwidth is currently 1.3MPixel @ 50Hz which give 65MBytes/sec  
> (~520MBit/s).
> Of course it's UDP _without_ checksumming ....
>
> Actually I have no sensor available that gives higher bandwidth ... but  
> having a look at transmission time I'm sure the MPC8343 can easily go up  
> to +800MBit.
>
>
> Obviously your cpu time is consumed on a higher level.
>
> Cheers,
> André
>
>
> Anton Vorontsov wrote:
>> Hi all,
>>
>> Down here few question regarding networking throughput, I would
>> appreciate any thoughts or ideas.
>>
>> On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing
>> relatively low TCP throughput using gianfar driver...
>>
>> The maximum value I've seen with the current kernels is 142 Mb/s of TCP
>> and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
>>
>>   root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
>>   TCP STREAM TEST to 10.0.1.1
>>   #Cpu utilization 0.10
>>   Recv   Send    Send
>>   Socket Socket  Message  Elapsed
>>   Size   Size    Size     Time     Throughput
>>   bytes  bytes   bytes    secs.    10^6bits/sec
>>
>>   206848 212992  32768    10.00     142.40
>>
>>   root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344
>>   UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1
>>   #Cpu utilization 100.00
>>   Socket  Message  Elapsed      Messages
>>   Size    Size     Time         Okay Errors   Throughput
>>   bytes   bytes    secs            #      #   10^6bits/sec
>>
>>   212992   32768   10.00       13539      0     354.84
>>   206848           10.00       13539            354.84
>>
>>
>> Is this normal?
>>
>> netperf running in loopback gives me 329 Mb/s of TCP throughput:
>>
>>   root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
>>   TCP STREAM TEST to 127.0.0.1
>>   #Cpu utilization 100.00
>>   #Cpu utilization 100.00
>>   Recv   Send    Send
>>   Socket Socket  Message  Elapsed
>>   Size   Size    Size     Time     Throughput
>>   bytes  bytes   bytes    secs.    10^6bits/sec
>>
>>   212992 212992  32768    10.00     329.60
>>
>>
>> May I consider this as a something that is close to the Linux'
>> theoretical maximum for this setup? Or this isn't reliable test?
>>
>>
>> I can compare with teh MPC8377E-RDB (very similar board - exactly the same
>> ethernet phy, the same drivers are used, i.e. everything is the same from
>> the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
>>
>>          |CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
>>   ------------------------------------------
>>   MPC8377|    666|    333|     646|     264|
>>   MPC8315|    400|    133|     354|     142|
>>   ------------------------------------------
>>   RATIO  |    1.6|    2.5|     1.8|     1.8|
>>
>> It seems that things are really dependant on the CPU/CSB speed.
>>
>> I've tried to tune gianfar driver in various ways... and it gave
>> some positive results with this patch:
>>
>> diff --git a/drivers/net/gianfar.h b/drivers/net/gianfar.h
>> index fd487be..b5943f9 100644
>> --- a/drivers/net/gianfar.h
>> +++ b/drivers/net/gianfar.h
>> @@ -123,8 +123,8 @@ extern const char gfar_driver_version[];
>>  #define GFAR_10_TIME    25600
>>   #define DEFAULT_TX_COALESCE 1
>> -#define DEFAULT_TXCOUNT	16
>> -#define DEFAULT_TXTIME	21
>> +#define DEFAULT_TXCOUNT	80
>> +#define DEFAULT_TXTIME	105
>>   #define DEFAULT_RXTIME	21
>>
>>
>> Basically this raises the tx interrupts coalescing threshold (raising
>> it more didn't help, as well as didn't help raising rx thresholds).
>> Now:
>>
>> root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344
>> TCP STREAM TEST to 10.0.1.1
>> #Cpu utilization 100.00
>> Recv   Send    Send
>> Socket Socket  Message  Elapsed
>> Size   Size    Size     Time     Throughput
>> bytes  bytes   bytes    secs.    10^6bits/sec
>>
>> 206848 212992  32768    3.00      163.04
>>
>>
>> That is +21 Mb/s (14% up). Not fantastic, but good anyway.
>>
>> As expected, the latency increased too:
>>
>> Before the patch:
>>
>> --- 10.0.1.1 ping statistics ---
>> 20 packets transmitted, 20 received, 0% packet loss, time 18997ms
>> rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 ms
>>
>> After:
>>
>> --- 10.0.1.1 ping statistics ---
>> 22 packets transmitted, 22 received, 0% packet loss, time 20997ms
>> rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms
>>
>>
>> 34% up... heh. Should we sacrifice the latency in favour of throughput?
>> Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of
>> latency? ;-)
>>
>>
>> Thanks in advance,
>>
>> p.s. Btw, the patch above helps even better on the on the -rt kernels,
>> since on the -rt kernels the throughput is near 100 Mb/s, with the
>> patch the throughput is close to 140 Mb/s.
>>
>>   
>
>
> MATRIX VISION GmbH, Talstraße 16, DE-71570 Oppenweiler  - Registergericht: Amtsgericht Stuttgart, HRB 271090
> Geschäftsführer: Gerhard Thullner, Werner Armingeon, Uwe Furtner

-- 
Anton Vorontsov
email: cbouatmailru@gmail.com
irc://irc.freenode.net/bd2

Re: [RFC] gianfar: low gigabit throughput

[Rick Jones]

>>I have _got_ to make CPU utilization enabled by default one of these  
>>days :)  At least for mechanisms which don't require calibration.
> 
> 
> Heh, I've skipped the calibration chapter in the netperf manual. :-D
> Should revert to it.

Under linux, the CPU utilization mechanism in netperf does not require 
calibration, so you can add a -c (and -C if the remote is also linux) to 
the global command line options.  Netperf will the report CPU util and 
calculate the "service demand" which will be the quantity of CPU 
consumed per unit of work.

> So things becomes much better when the message size increases
> (I think the netperf then eating less cpu, and gives some precessing
> time to the kernel?).

Unless the compiler isn't doing a very good job, or perhaps if you've 
enabled histograms (./configure --enable-histogram) and set verbosity to 
2 or more (not the case here), netperf itself shouldn't be consuming 
very much CPU at all up in user space.  Now, as the size of the buffers 
passed to the transport increases, it does mean fewer system calls per 
KB transferred, which should indeed be more efficient.

>>What is the nature of the DMA stream between the two tests?  I find it  
>>interesting that the TCP Mb/s went up by more than the CPU MHz and  
>>wonder how much the Bus MHz came into play there - perhaps there were  
>>more DMA's to setup or across a broader memory footprint for TCP than  
>>for UDP.
> 
> 
> The gianfar indeed does a lot of dma on the "buffer descriptors", so
> probably the bus speed matters a lot. And combination of CPU and Bus
> gives the final result.

Do you have any way to calculate bus utilization - logic analyzer, or 
perhaps some performance counters in the hardware?

If you have an HP-UX or Solaris system handy to act as a receiver, you 
might give that a try - it would be interesting to see the effect of 
their ACK avoidance heuristics on TCP throughput.  One non-trivial 
difference I keep forgetting to mention is that TCP will have that 
returning ACK stream that UDP will not.

rick jones