[B.A.T.M.A.N.] wired performance

[B.A.T.M.A.N.] wired performance

[dan]

Anyone done any performance testing over wired links?  ie, if two
ethernet ports are gigabit and are added to bat0, what kind of
throughput is expected across these interfaces and how much CPU is
needed to get up near wire-speed?

Re: [B.A.T.M.A.N.] wired performance

[dan]

I'm only able to test in virtualbox at the moment.

I have a 3.1Ghz i5 and I'm able to do iperf across a batman-adv
network (nodes A-B-C with A and C not directly connected).

iperf between a<>b is ~1Gbps, A<>C is ~560Mbps and B has 1 core hit
100% by ksoftirqd.

So it looks like I'm stuck in a single thread.

I'm going to add a 4th node and do a double hop A<>D iperf and see
what happens.  full duplex interfaces here so I don't think I'll see a
big drop in the next test.

On Wed, Aug 16, 2017 at 9:39 AM, dan <dandenson@gmail.com> wrote:
> Anyone done any performance testing over wired links?  ie, if two
> ethernet ports are gigabit and are added to bat0, what kind of
> throughput is expected across these interfaces and how much CPU is
> needed to get up near wire-speed?

Re: [B.A.T.M.A.N.] wired performance

[dan]

2 hops on full duplex links I'm getting 464Mbps.  There is some loss
at each hop and I'm not sure why.  Could be virtualbox and resource
contension that isn't showing up in task manager and/or top on the
vm's.

is this to be expected?

On Sat, Aug 19, 2017 at 12:25 PM, dan <dandenson@gmail.com> wrote:
> I'm only able to test in virtualbox at the moment.
>
> I have a 3.1Ghz i5 and I'm able to do iperf across a batman-adv
> network (nodes A-B-C with A and C not directly connected).
>
> iperf between a<>b is ~1Gbps, A<>C is ~560Mbps and B has 1 core hit
> 100% by ksoftirqd.
>
> So it looks like I'm stuck in a single thread.
>
> I'm going to add a 4th node and do a double hop A<>D iperf and see
> what happens.  full duplex interfaces here so I don't think I'll see a
> big drop in the next test.
>
> On Wed, Aug 16, 2017 at 9:39 AM, dan <dandenson@gmail.com> wrote:
>> Anyone done any performance testing over wired links?  ie, if two
>> ethernet ports are gigabit and are added to bat0, what kind of
>> throughput is expected across these interfaces and how much CPU is
>> needed to get up near wire-speed?

Re: [B.A.T.M.A.N.] wired performance

[Sven Eckelmann]

[-- Attachment #1: Type: text/plain, Size: 1649 bytes --]

On Mittwoch, 16. August 2017 09:39:00 CEST dan wrote:
> Anyone done any performance testing over wired links?  ie, if two
> ethernet ports are gigabit and are added to bat0, what kind of
> throughput is expected across these interfaces and how much CPU is
> needed to get up near wire-speed?

I can easily reach gigabit here between an i5-3230M (receiver; r8169) and an 
i7-6700K (sender; igb). I was able to reach 926 Mbits/sec in a single stream 
iperf 2.0.9+dfsg1-1 30 second test. Both ethernet devices were configured with 
an MTU of 1560. Both systems were idling most of the time (> 90% of time while
also running things like xapian, akonadi, ...).

There are some ethernet chips which have offloading features which work for IP 
or similar layer 3 protocols. This layer 3 specific offloading will not work 
when batman-adv is added as a layer between the ethernet header and the IP 
header. I have even seen chips which then went completely slow and got only 
fast again after the offloading features were disabled in hardware (either 
using ethtool or by patching the driver).

And there is currently no special batman-adv support for the flow 
dissector [1] in the kernel. This could be also a reason why multiple flows 
are not distributed well to different cores when you enable RPS/XPS. It is not 
yet know whether this will actually be helpful but at least someone interested 
could do same research and implement a proof-of-concept patches for further 
testing.

Kind regards,
	Sven

[1] https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tree/net/core/flow_dissector.c?id=d6e1e46f69fbe956e877cdd00dbfb002baddf577#n576

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [B.A.T.M.A.N.] wired performance

[Andrew Lunn]

> And there is currently no special batman-adv support for the flow 
> dissector [1] in the kernel. This could be also a reason why multiple flows 
> are not distributed well to different cores when you enable RPS/XPS. It is not 
> yet know whether this will actually be helpful but at least someone interested 
> could do same research and implement a proof-of-concept patches for further 
> testing.

Hi Sven

DSA just added a patch to help with flow dissectors. Maybe it can be
generalized and made to work for BATMAN as well?

commit 43e665287f931a167cd2eea3387efda901bff0ce
Author: John Crispin <john@phrozen.org>
Date:   Wed Aug 9 14:41:19 2017 +0200

    net-next: dsa: fix flow dissection
    
    RPS and probably other kernel features are currently broken on some if not
    all DSA devices. The root cause of this is that skb_hash will call the
    flow_dissector. At this point the skb still contains the magic switch
    header and the skb->protocol field is not set up to the correct 802.3
    value yet. By the time the tag specific code is called, removing the header
    and properly setting the protocol an invalid hash is already set. In the
    case of the mt7530 this will result in all flows always having the same
    hash.

Re: [B.A.T.M.A.N.] wired performance

[dan]

On Sun, Aug 20, 2017 at 8:05 AM, Andrew Lunn <andrew@lunn.ch> wrote:
>> And there is currently no special batman-adv support for the flow
>> dissector [1] in the kernel. This could be also a reason why multiple flows
>> are not distributed well to different cores when you enable RPS/XPS. It is not
>> yet know whether this will actually be helpful but at least someone interested
>> could do same research and implement a proof-of-concept patches for further
>> testing.
>
> Hi Sven
>
> DSA just added a patch to help with flow dissectors. Maybe it can be
> generalized and made to work for BATMAN as well?
>
> commit 43e665287f931a167cd2eea3387efda901bff0ce
> Author: John Crispin <john@phrozen.org>
> Date:   Wed Aug 9 14:41:19 2017 +0200
>
>     net-next: dsa: fix flow dissection
>
>     RPS and probably other kernel features are currently broken on some if not
>     all DSA devices. The root cause of this is that skb_hash will call the
>     flow_dissector. At this point the skb still contains the magic switch
>     header and the skb->protocol field is not set up to the correct 802.3
>     value yet. By the time the tag specific code is called, removing the header
>     and properly setting the protocol an invalid hash is already set. In the
>     case of the mt7530 this will result in all flows always having the same
>     hash.

Ok, maybe virtualbox specific slowness then.  I'm running batman-adv
2015.2 on ubuntu kernel 4.4.0, so a little out of date.

I'm actually wanting to target something embedded, like a pc-engines
APU2c4.  It has 3 onboard ethernets and I can add up to 8 more via
pci-e.  But that has a 1Ghz CPU (quad core) and being stuck on 1 core
1/4ers the potential.

The test network I'm designing is handling wireless PtP backhauls and
having batman-adv make all tower sites look like a single switch,
almost list a replacement for shortest path bridging.  1Gbps would be
the real base line performance here.

I have another question on the list about running on top of vlans,
specifically do to something like router-on-a-stick with a wisp
specific PoE switch and running a vlan across backhaul links and
putting those vlans in batman-adv.  That way I don't have issue
accessing the ptp link interfaces themselves.