MVEBU and MVNETA driver

MVEBU and MVNETA driver

[Greg]

Hello All,

This is my first post there, let me introduce myself: I am an embedded 
systems designer working in a French ISP. My goal is to design devices 
with a minimalistic software package installed that system/software 
engineers can use as a base for their work. Said in a few words : I'm 
not the kernel expert but I have some knowledge of it.

That being said, I'm currently working with a Marvell ArmadaXP SoC and 
I'm performing the transition from Marvell "Linux Support Package" to 
MVEBU in the 3.8.8 Vanilla.
Having the system up and running was actually quite easy, Thomas 
Petazzoni and his team did a very good job!

I'm facing a problem with ethernet transmission (reception -seems- to be 
fine), at best is it slow (~90Mbps), at worst the kernel OOPSes at 
various locations dealing with memory (tcp_v4_destroy_sock, free_block 
and skb_put seen so far) and ends up crashing right after.
I'm simply using iperf to generate traffic and the system has been 
stable for a while with Marvell's LSP so I can say with confidence the 
hardware is pretty stable.

My question is fairly simple : before complaining about the network 
driver (which might not even be involved in this), what is the starting 
point to gather more information about this problem and what data should 
I provide you to help you advising me ?

Thank you very much!

MVEBU and MVNETA driver

[Willy Tarreau]

Hi Greg,

On Fri, Apr 19, 2013 at 03:07:40PM +0200, Greg wrote:
> Hello All,
> 
> This is my first post there, let me introduce myself: I am an embedded 
> systems designer working in a French ISP. My goal is to design devices 
> with a minimalistic software package installed that system/software 
> engineers can use as a base for their work. Said in a few words : I'm 
> not the kernel expert but I have some knowledge of it.
> 
> That being said, I'm currently working with a Marvell ArmadaXP SoC and 
> I'm performing the transition from Marvell "Linux Support Package" to 
> MVEBU in the 3.8.8 Vanilla.
> Having the system up and running was actually quite easy, Thomas 
> Petazzoni and his team did a very good job!
> 
> I'm facing a problem with ethernet transmission (reception -seems- to be 
> fine), at best is it slow (~90Mbps), at worst the kernel OOPSes at 
> various locations dealing with memory (tcp_v4_destroy_sock, free_block 
> and skb_put seen so far) and ends up crashing right after.
> I'm simply using iperf to generate traffic and the system has been 
> stable for a while with Marvell's LSP so I can say with confidence the 
> hardware is pretty stable.
> 
> My question is fairly simple : before complaining about the network 
> driver (which might not even be involved in this), what is the starting 
> point to gather more information about this problem and what data should 
> I provide you to help you advising me ?

You need to get the following commit from mainline to fix this issue :

   ee40a116ebf139f900c3d2e6febb8388738e96d0

Cheers,
Willy

PS: is it indiscrete to ask what board this is ?

MVEBU and MVNETA driver

[Greg]

Le 19/04/2013 15:12, Willy Tarreau a ?crit :
> Hi Greg,
>
> On Fri, Apr 19, 2013 at 03:07:40PM +0200, Greg wrote:
>> I'm facing a problem with ethernet transmission (reception -seems- to be
>> fine), at best is it slow (~90Mbps), at worst the kernel OOPSes at
>> various locations dealing with memory (tcp_v4_destroy_sock, free_block
>> and skb_put seen so far) and ends up crashing right after.
>> I'm simply using iperf to generate traffic and the system has been
>> stable for a while with Marvell's LSP so I can say with confidence the
>> hardware is pretty stable.
>>
> You need to get the following commit from mainline to fix this issue :
>
>     ee40a116ebf139f900c3d2e6febb8388738e96d0
>
Willy, thanks for your fast an accurate answer, this seems to solve the 
stability issue but not the performance issue.

Please look at this strange behavior : 1 iperf will only transmit at 
~100Mbps, 100 parallel iperf will transmit at ~950Mb. Reception is not 
an issue.
I'm thinking of a socket <-> txQ relationship that would limit 
transmission rate, but I have no idea on how this is implemented.

> PS: is it indiscrete to ask what board this is ?
I designed the board myself, if you already heard about it I'd be glad 
to know you told you about it ;-)

Cheers,

MVEBU and MVNETA driver

[Willy Tarreau]

On Fri, Apr 19, 2013 at 04:13:19PM +0200, Greg wrote:
> Willy, thanks for your fast an accurate answer, this seems to solve the 
> stability issue but not the performance issue.

OK.

> Please look at this strange behavior : 1 iperf will only transmit at 
> ~100Mbps, 100 parallel iperf will transmit at ~950Mb. Reception is not 
> an issue.
> I'm thinking of a socket <-> txQ relationship that would limit 
> transmission rate, but I have no idea on how this is implemented.

Strange, I have not experienced this. Yes a socket size limit could be
the reason. Have you tried with netcat or any simple TCP server instead ?

> >PS: is it indiscrete to ask what board this is ?
> I designed the board myself, if you already heard about it I'd be glad 
> to know you told you about it ;-)

Wow, impressed !

Willy

MVEBU and MVNETA driver

[Gregory CLEMENT]

On 04/19/2013 04:43 PM, Willy Tarreau wrote:
> On Fri, Apr 19, 2013 at 04:13:19PM +0200, Greg wrote:
>> Willy, thanks for your fast an accurate answer, this seems to solve the 
>> stability issue but not the performance issue.
> 
> OK.
> 
>> Please look at this strange behavior : 1 iperf will only transmit at 
>> ~100Mbps, 100 parallel iperf will transmit at ~950Mb. Reception is not 
>> an issue.
>> I'm thinking of a socket <-> txQ relationship that would limit 
>> transmission rate, but I have no idea on how this is implemented.
> 
> Strange, I have not experienced this. Yes a socket size limit could be
> the reason. Have you tried with netcat or any simple TCP server instead ?
> 

I also have this behavior:
iperf -c 192.168.0.19 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.19, TCP port 5001
TCP window size: 20.7 KByte (default)
------------------------------------------------------------
[  5] local 192.168.0.37 port 48473 connected with 192.168.0.19 port 5001
[  4] local 192.168.0.37 port 5001 connected with 192.168.0.19 port 36987
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   128 MBytes   107 Mbits/sec
[  4]  0.0-10.0 sec  1.07 GBytes   919 Mbits/sec

but I only need 10 thread in iperf to be close to the Gb
------------------------------------------------------------
Client connecting to 192.168.0.19, TCP port 5001
TCP window size: 20.7 KByte (default)
------------------------------------------------------------
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   117 MBytes  98.2 Mbits/sec
[ 12]  0.0-10.0 sec  42.5 MBytes  35.6 Mbits/sec
[  8]  0.0-10.0 sec   131 MBytes   110 Mbits/sec
[  5]  0.0-10.0 sec   127 MBytes   107 Mbits/sec
[  3]  0.0-10.0 sec   118 MBytes  98.4 Mbits/sec
[ 10]  0.0-10.0 sec   130 MBytes   109 Mbits/sec
[  7]  0.0-10.0 sec   128 MBytes   107 Mbits/sec
[  9]  0.0-10.0 sec  83.2 MBytes  69.7 Mbits/sec
[  6]  0.0-10.0 sec   110 MBytes  92.5 Mbits/sec
[ 11]  0.0-10.0 sec   132 MBytes   110 Mbits/sec
[SUM]  0.0-10.0 sec  1.09 GBytes   936 Mbits/sec

I am not a network expert and until now we didn't try to
have the best performance the hardware can offer. We mainly
focused on features.

Regards,
-- 
Gregory Clement, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

MVEBU and MVNETA driver

[Greg]

Le 19/04/2013 16:59, Gregory CLEMENT a ?crit :
> On 04/19/2013 04:43 PM, Willy Tarreau wrote:
>> On Fri, Apr 19, 2013 at 04:13:19PM +0200, Greg wrote:
>>> Willy, thanks for your fast an accurate answer, this seems to solve the
>>> stability issue but not the performance issue.
>> OK.
>>
>>> Please look at this strange behavior : 1 iperf will only transmit at
>>> ~100Mbps, 100 parallel iperf will transmit at ~950Mb. Reception is not
>>> an issue.
>>> I'm thinking of a socket <-> txQ relationship that would limit
>>> transmission rate, but I have no idea on how this is implemented.
>> Strange, I have not experienced this. Yes a socket size limit could be
>> the reason. Have you tried with netcat or any simple TCP server instead ?
>>
> I also have this behavior:
> iperf -c 192.168.0.19 -d
> ------------------------------------------------------------
> Server listening on TCP port 5001
> TCP window size: 85.3 KByte (default)
> ------------------------------------------------------------
> ------------------------------------------------------------
> Client connecting to 192.168.0.19, TCP port 5001
> TCP window size: 20.7 KByte (default)
> ------------------------------------------------------------
> [  5] local 192.168.0.37 port 48473 connected with 192.168.0.19 port 5001
> [  4] local 192.168.0.37 port 5001 connected with 192.168.0.19 port 36987
> [ ID] Interval       Transfer     Bandwidth
> [  5]  0.0-10.0 sec   128 MBytes   107 Mbits/sec
> [  4]  0.0-10.0 sec  1.07 GBytes   919 Mbits/sec
>
> but I only need 10 thread in iperf to be close to the Gb
> ------------------------------------------------------------
> Client connecting to 192.168.0.19, TCP port 5001
> TCP window size: 20.7 KByte (default)
> ------------------------------------------------------------
> [ ID] Interval       Transfer     Bandwidth
> [  4]  0.0-10.0 sec   117 MBytes  98.2 Mbits/sec
> [ 12]  0.0-10.0 sec  42.5 MBytes  35.6 Mbits/sec
> [  8]  0.0-10.0 sec   131 MBytes   110 Mbits/sec
> [  5]  0.0-10.0 sec   127 MBytes   107 Mbits/sec
> [  3]  0.0-10.0 sec   118 MBytes  98.4 Mbits/sec
> [ 10]  0.0-10.0 sec   130 MBytes   109 Mbits/sec
> [  7]  0.0-10.0 sec   128 MBytes   107 Mbits/sec
> [  9]  0.0-10.0 sec  83.2 MBytes  69.7 Mbits/sec
> [  6]  0.0-10.0 sec   110 MBytes  92.5 Mbits/sec
> [ 11]  0.0-10.0 sec   132 MBytes   110 Mbits/sec
> [SUM]  0.0-10.0 sec  1.09 GBytes   936 Mbits/sec
>
> I am not a network expert and until now we didn't try to
> have the best performance the hardware can offer. We mainly
> focused on features.
>
I've got the same numbers here.
A simple tool like netcat shows ~470Mbps but this limited by the CPU.
Iperf isn't CPU limited

The same test with Marvell LSP shows ~940Mbps in both directions.

Regards,

MVEBU and MVNETA driver

[Willy Tarreau]

On Fri, Apr 19, 2013 at 06:02:19PM +0200, Greg wrote:
> A simple tool like netcat shows ~470Mbps but this limited by the CPU.
> Iperf isn't CPU limited
> 
> The same test with Marvell LSP shows ~940Mbps in both directions.

I've retested here on my mirabox using httpterm as the tcp server.
Using the normal send() syscall, it emits 839 Mbps on a single TCP
connection. Using splice(), it emits 969 Mbps in the same conditions.

Here's what ethtool is saying about the NIC's tuning :
# ethtool -k eth0
Offload parameters for eth0:
rx-checksumming: off
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

I've tried changing other settings such as the interrupt delays
(rx-usecs, rx-frames, tx-frames) but this has no effect. So in
short, I cannot reproduce this behaviour. Are you using the SLOB
allocator ? I'm asking because I remember experiencing very poor
performance with it in the past.

Regards,
Willy

MVEBU and MVNETA driver

[Greg]

Le 22/04/2013 08:20, Willy Tarreau a ?crit :
> On Fri, Apr 19, 2013 at 06:02:19PM +0200, Greg wrote:
>> A simple tool like netcat shows ~470Mbps but this limited by the CPU.
>> Iperf isn't CPU limited
>>
>> The same test with Marvell LSP shows ~940Mbps in both directions.
> I've retested here on my mirabox using httpterm as the tcp server.
> Using the normal send() syscall, it emits 839 Mbps on a single TCP
> connection. Using splice(), it emits 969 Mbps in the same conditions.
>
> Here's what ethtool is saying about the NIC's tuning :
> # ethtool -k eth0
> Offload parameters for eth0:
> rx-checksumming: off
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off
> rx-vlan-offload: off
> tx-vlan-offload: off
> ntuple-filters: off
> receive-hashing: off
>
> I've tried changing other settings such as the interrupt delays
> (rx-usecs, rx-frames, tx-frames) but this has no effect. So in
> short, I cannot reproduce this behaviour. Are you using the SLOB
> allocator ? I'm asking because I remember experiencing very poor
> performance with it in the past.
>
Willy,

I'm using SLAB allocator.
As for offloading, I can't seem to be able to turn it on :
> $ ethtool -K eth0 gro on
> Cannot set device GRO settings: Operation not supported
> $ ethtool -K eth0 gso on
> Cannot set device generic segmentation offload settings: Operation not 
> supported
Default values :
> $ ethtool -k eth0
> Offload parameters for eth0:
> rx-checksumming: off
> tx-checksumming: on
> scatter-gather: on
> tcp-segmentation-offload: off
> udp-fragmentation-offload: off
> generic-segmentation-offload: off
> generic-receive-offload: off
> large-receive-offload: off
> rx-vlan-offload: off
> tx-vlan-offload: off
> ntuple-filters: off
> receive-hashing: off

I can see 1 difference between our setups, Mirabox has an Armada370, I 
have an ArmadaXP, it would be interesting to know what Gregory is using.

Another difference on my setup, and something I wanted to report you : 
there is no PHY on my system. I had to instantiate a dummy PHY in the 
device tree blob and bind it to the mvneta driver because it would fail 
at init without PHY.

Regards,

MVEBU and MVNETA driver

[Willy Tarreau]

On Mon, Apr 22, 2013 at 10:19:41AM +0200, Greg wrote:
> I'm using SLAB allocator.
> As for offloading, I can't seem to be able to turn it on :
> >$ ethtool -K eth0 gro on
> >Cannot set device GRO settings: Operation not supported
> >$ ethtool -K eth0 gso on
> >Cannot set device generic segmentation offload settings: Operation not 
> >supported
> Default values :
> >$ ethtool -k eth0
> >Offload parameters for eth0:
> >rx-checksumming: off
> >tx-checksumming: on
> >scatter-gather: on
> >tcp-segmentation-offload: off
> >udp-fragmentation-offload: off
> >generic-segmentation-offload: off
> >generic-receive-offload: off
> >large-receive-offload: off
> >rx-vlan-offload: off
> >tx-vlan-offload: off
> >ntuple-filters: off
> >receive-hashing: off
> 
> I can see 1 difference between our setups, Mirabox has an Armada370, I 
> have an ArmadaXP, it would be interesting to know what Gregory is using.

OK so I guess you have a kernel version prior to 3.9-rc7. There is a bug
in the driver which causes it to report tx cksum and sg but they're not
enabled. You need to disable then enable them :

  ethtool -K eth0 tx off
  ethtool -K eth0 sg off
  ethtool -K eth0 tx on
  ethtool -K eth0 sg on

Then you can enable gso/gro. And that clearly explains a significant
performance difference.

> Another difference on my setup, and something I wanted to report you : 
> there is no PHY on my system. I had to instantiate a dummy PHY in the 
> device tree blob and bind it to the mvneta driver because it would fail 
> at init without PHY.

I think this is more something for Thomas then.

Regards,
Willy

MVEBU and MVNETA driver

[Greg]

Le 22/04/2013 11:08, Willy Tarreau a ?crit :
> On Mon, Apr 22, 2013 at 10:19:41AM +0200, Greg wrote:
>> I'm using SLAB allocator.
>> As for offloading, I can't seem to be able to turn it on :
>>> $ ethtool -K eth0 gro on
>>> Cannot set device GRO settings: Operation not supported
>>> $ ethtool -K eth0 gso on
>>> Cannot set device generic segmentation offload settings: Operation not
>>> supported
>> Default values :
>>> $ ethtool -k eth0
>>> Offload parameters for eth0:
>>> rx-checksumming: off
>>> tx-checksumming: on
>>> scatter-gather: on
>>> tcp-segmentation-offload: off
>>> udp-fragmentation-offload: off
>>> generic-segmentation-offload: off
>>> generic-receive-offload: off
>>> large-receive-offload: off
>>> rx-vlan-offload: off
>>> tx-vlan-offload: off
>>> ntuple-filters: off
>>> receive-hashing: off
>> I can see 1 difference between our setups, Mirabox has an Armada370, I
>> have an ArmadaXP, it would be interesting to know what Gregory is using.
> OK so I guess you have a kernel version prior to 3.9-rc7. There is a bug
> in the driver which causes it to report tx cksum and sg but they're not
> enabled. You need to disable then enable them :
>
>    ethtool -K eth0 tx off
>    ethtool -K eth0 sg off
>    ethtool -K eth0 tx on
>    ethtool -K eth0 sg on
>
> Then you can enable gso/gro. And that clearly explains a significant
> performance difference.
Yes, as explain in my original mail, I'm using 3.8.8
I followed your recommendations and performance is now ~930-940Mbits in 
both directions which is fine.
Thanks !
>> Another difference on my setup, and something I wanted to report you :
>> there is no PHY on my system. I had to instantiate a dummy PHY in the
>> device tree blob and bind it to the mvneta driver because it would fail
>> at init without PHY.
> I think this is more something for Thomas then.
>
The "failure" is located in mvneta_mdio_probe but I guess this is more 
like an architectural problem.
The MVNETA driver assumes a PHY is present, but this is not a 
requirement to establish link between 2 MACs.
I guess using a dummy PHY is a not-so-bad idea for my situation at this 
time.

Regards,