[B.A.T.M.A.N.] BW degradation on p2p links?

[B.A.T.M.A.N.] BW degradation on p2p links?

[Guido Iribarren]

Hi,
i'm participating in the development of a wireless mesh network in
Delta de Tigre, near Buenos Aires, Argentina.
Landscape is absolutely flat, with dense forest 15-20mts tall. Nodes
are being installed along the river, 50 - 150mts apart, with clear LOS
at ground level.
I collaborated with Nico Echaniz last month, bulding another community
network in Cordoba, and given the success, I decided to start one more
here. I'm also really thankful with Elektra, for recommending the
tl-m3220 to Nico!
We chose batman over olsr, in part because of avahi stuff, but also to
have a real-world implementation and thus collaborate in testing /
debugging :) So far, there are  only 4 working nodes, but luckily I'm
already facing unexpected behaviour.

I'm sorry for the long email, it's mostly terminal logs.
In short, iperf between two neighbouring nodes reports >20mbps, yet
running the same iperf between nodes separated by one hop reports
inexplicably low <5mbps.

Links are point to point, with different radios on different channels,
no freq overlap.

A=[ath9k]--(50m)--[ath9k]=B=[ath9k_htc]--(150m)--[ath9k_htc]=C=[ath9k]--(50m)--[ath9k]=D

A is a tplink mr3420 + wn722n
B and C are tplink mr3220 + wn722n
D is a tplink mr3220 (single interface)

All of them are running openwrt trunk r30919.
the internal iface of the mr3x20 uses ath9k, and the wn722n runs with ath9k_htc

the link between B and C is done in infrastructure mode, B is ap and C
is managed. Radios are set to channel 9, HT40-.
C <-> D is made with ath9k , in channel 1 , HT20.

(all hostnames have been obscured for readability :P )

nodeB -> nodeC = 31mbps
nodeC -> nodeD = 21mbps
nodeB -> nodeD = 3mbps

nodeD -> nodeC = 21mbps
nodeC -> nodeB = 21mbps
nodeD -> nodeB = 10mbps

nodeD# batctl tr B_mesh
traceroute to B_mesh (56:e6:fc:be:29:d3), 50 hops max, 20 byte packets
 1: C_mesh (56:e6:fc:b9:b6:47)  1.155 ms  0.834 ms  0.796 ms
 2: B_mesh (56:e6:fc:be:29:d3)  6.523 ms  2.719 ms  1.917 ms

nodeB# batctl tr D_mesh
traceroute to D_mesh (56:e6:fc:b9:b7:01), 50 hops max, 20 byte packets
 1: C_mesh (56:e6:fc:b9:b6:47)  1.323 ms  1.117 ms  0.974 ms
 2: D_mesh (56:e6:fc:b9:b7:01)  2.278 ms  1.839 ms  2.164 ms


### iperf between nodeB -> nodeC
root@nodeB:~# iperf -c nodeC -w 320k  -i 1
------------------------------------------------------------
Client connecting to nodeC, TCP port 5001
TCP window size:  320 KByte
------------------------------------------------------------
[  3] local 10.6.0.1 port 35759 connected with 10.6.0.32 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  3.63 MBytes  30.4 Mbits/sec
[  3]  1.0- 2.0 sec  3.63 MBytes  30.4 Mbits/sec
[  3]  2.0- 3.0 sec  4.00 MBytes  33.6 Mbits/sec
[  3]  3.0- 4.0 sec  3.63 MBytes  30.4 Mbits/sec
[  3]  4.0- 5.0 sec  3.88 MBytes  32.5 Mbits/sec
[  3]  5.0- 6.0 sec  3.88 MBytes  32.5 Mbits/sec
[  3]  6.0- 7.0 sec  4.00 MBytes  33.6 Mbits/sec
[  3]  7.0- 8.0 sec  3.75 MBytes  31.5 Mbits/sec
[  3]  8.0- 9.0 sec  3.75 MBytes  31.5 Mbits/sec
[  3]  9.0-10.0 sec  3.75 MBytes  31.5 Mbits/sec
[  3]  0.0-10.1 sec  38.0 MBytes  31.7 Mbits/sec

### iperf between nodeC -> nodeD
nodeC# iperf -c nodeD -w 320k -i 1
------------------------------------------------------------
Client connecting to nodeD, TCP port 5001
TCP window size:  320 KByte
------------------------------------------------------------
[  3] local 10.6.0.32 port 43655 connected with 10.6.0.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec  2.50 MBytes  21.0 Mbits/sec
[  3]  1.0- 2.0 sec  2.63 MBytes  22.0 Mbits/sec
[  3]  2.0- 3.0 sec  2.50 MBytes  21.0 Mbits/sec
[  3]  3.0- 4.0 sec  2.88 MBytes  24.1 Mbits/sec
[  3]  4.0- 5.0 sec  2.63 MBytes  22.0 Mbits/sec
[  3]  5.0- 6.0 sec  2.63 MBytes  22.0 Mbits/sec
[  3]  6.0- 7.0 sec  2.50 MBytes  21.0 Mbits/sec
[  3]  7.0- 8.0 sec  2.25 MBytes  18.9 Mbits/sec
[  3]  8.0- 9.0 sec  2.63 MBytes  22.0 Mbits/sec
[  3]  9.0-10.0 sec  2.63 MBytes  22.0 Mbits/sec
[  3]  0.0-10.1 sec  25.9 MBytes  21.5 Mbits/sec

### Now, enter the mistery..
### Best iperf run after several attemps yielding 1 - 2 mbps
# iperf -c charly-muelle -w 320k -i 1 -t 20
------------------------------------------------------------
Client connecting to charly-muelle, TCP port 5001
TCP window size:  320 KByte
------------------------------------------------------------
[  3] local 10.6.0.1 port 55093 connected with 10.6.0.16 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   896 KBytes  7.34 Mbits/sec
[  3]  1.0- 2.0 sec   256 KBytes  2.10 Mbits/sec
[  3]  2.0- 3.0 sec   128 KBytes  1.05 Mbits/sec
[  3]  3.0- 4.0 sec   128 KBytes  1.05 Mbits/sec
[  3]  4.0- 5.0 sec   256 KBytes  2.10 Mbits/sec
[  3]  5.0- 6.0 sec   384 KBytes  3.15 Mbits/sec
[  3]  6.0- 7.0 sec   128 KBytes  1.05 Mbits/sec
[  3]  7.0- 8.0 sec   384 KBytes  3.15 Mbits/sec
[  3]  8.0- 9.0 sec   256 KBytes  2.10 Mbits/sec
[  3]  9.0-10.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 10.0-11.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 11.0-12.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 12.0-13.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 13.0-14.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 14.0-15.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 15.0-16.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 16.0-17.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 17.0-18.0 sec   512 KBytes  4.19 Mbits/sec
[  3] 18.0-19.0 sec   640 KBytes  5.24 Mbits/sec
[  3] 19.0-20.0 sec   512 KBytes  4.19 Mbits/sec
[  3]  0.0-20.3 sec  8.88 MBytes  3.66 Mbits/sec

### and the way back... (i'll snip the logs generously)
nodeD# iperf -c nodeC -w320k
[  3]  0.0-10.1 sec  25.5 MBytes  21.2 Mbits/sec

nodeC# iperf -c nodeB -w 320k
[  3]  0.0-10.0 sec  25.5 MBytes  21.3 Mbits/sec

nodeD# iperf -c nodeB -w320k
[  3]  0.0-10.2 sec  12.3 MBytes  10.0 Mbits/sec

This can be reproduced at will, and i don't understand what's happening.

Looks like node C is getting .. overwhelmed..(?) when forwarding the
packets between B and D
Even more intriguing is that the problem is much worse when going B ->
D (tenfold slower) than from D -> B (twice slower)

##### more screenlogs follow

### similar in node B and C (in D, there's no wlan1)
# batctl -v
batctl 2012.0.0 [batman-adv: 2012.0.0]
# batctl if
wlan0-1: active # ath9k, internal, adhoc mode
wlan1: active # ath9k_htc, usb, infrastructure mode
# brctl show
bridge name	bridge id		STP enabled	interfaces
br-lan		8000.228082e1fd06	no		wlan0
							eth0
							bat0

### originators seen from nodeB
nodeB# batctl o | cut -b -100
[B.A.T.M.A.N. adv 2012.0.0, MainIF/MAC: wlan0-1/56:e6:fc:be:29:d3 (bat0)]
  Originator      last-seen (#/255)           Nexthop [outgoingIF]:
Potential nexthops ...
          A_wlan1    0.830s   (250)           A_wlan1 [     wlan1]:
        C_mesh (  0)
           D_mesh    0.320s   (230)           C_wlan1 [     wlan1]:
       C_wlan1 (230)
           A_mesh    0.890s   (250)           A_wlan1 [     wlan1]:
       A_wlan1 (250)
           C_mesh    0.800s   (241)           C_wlan1 [     wlan1]:
       C_wlan1 (241)
          C_wlan1    0.720s   (241)           C_wlan1 [     wlan1]:
       A_wlan1 (226)

### originators seen from nodeC
nodeC# batctl o |cut -b -100
[B.A.T.M.A.N. adv 2012.0.0, MainIF/MAC: wlan0-1/56:e6:fc:b9:b6:47 (bat0)]
  Originator      last-seen (#/255)           Nexthop [outgoingIF]:
Potential nexthops ...
           B_mesh    0.830s   (255)           B_wlan1 [     wlan1]:
        D_mesh (  0)
          A_wlan1    0.180s   (242)           A_wlan1 [     wlan1]:
       B_wlan1 (229)
          B_wlan1    0.280s   (255)           B_wlan1 [     wlan1]:
       A_wlan1 (233)
           D_mesh    0.030s   (235)            D_mesh [   wlan0-1]:
       A_wlan1 (194)
           A_mesh    0.430s   (242)           A_wlan1 [     wlan1]:
       B_wlan1 (230)


Any ideas, thoughts, pointers? Any additional info needed?

I've RTFW, RTFM, and even RTF-PDF concerning bandwidth degradation on
single-interface-node mesh networks, and that's the reason why i'm (so
far unsuccessfully) trying to overcome that problem with the
additional USB interface.

Thanks a lot for the attention!

Guido

Re: [B.A.T.M.A.N.] BW degradation on p2p links?

[Marek Lindner]

Hi,

> i'm participating in the development of a wireless mesh network in
> Delta de Tigre, near Buenos Aires, Argentina.
> Landscape is absolutely flat, with dense forest 15-20mts tall. Nodes
> are being installed along the river, 50 - 150mts apart, with clear LOS
> at ground level.
> I collaborated with Nico Echaniz last month, bulding another community
> network in Cordoba, and given the success, I decided to start one more
> here. I'm also really thankful with Elektra, for recommending the
> tl-m3220 to Nico!
> We chose batman over olsr, in part because of avahi stuff, but also to
> have a real-world implementation and thus collaborate in testing /
> debugging :) So far, there are  only 4 working nodes, but luckily I'm
> already facing unexpected behaviour.
> 
> I'm sorry for the long email, it's mostly terminal logs.
> In short, iperf between two neighbouring nodes reports >20mbps, yet
> running the same iperf between nodes separated by one hop reports
> inexplicably low <5mbps.
>
> [..]
>
> Any ideas, thoughts, pointers? Any additional info needed?

welcome on the list!
I admit not having dived into all the details of your mail - the bigger part 
of the batman team is sitting in Athens at the moment (see battlemesh.org). 
The best next step is to find out whether this performance issue is a batman-
adv or a wifi driver problem. May I suggest you remove batman-adv and repeat 
the same test using static routing ? If you still see the problem it is the 
wifi that is not working otherwise we have to fix batman-adv.

Regards,
Marek

Re: [B.A.T.M.A.N.] BW degradation on p2p links?

[Guido Iribarren]

Hey thanks marek! I'm sorry i didn't think of trying that in the first
place. Ath9k_htc has indeed a few issues so i wouldn't be suprised
it's the culprit.

I was aware of the battlemesh v5 and last week i even piperlurked the
mail archives, but didn't realize it would happen this week. Best luck
in athens then!
I will continue my little mesh-battle here.
Will try static routes tomorrow and report anything interesting.

Cheers,
guido

On 3/26/12, Marek Lindner <lindner_marek@yahoo.de> wrote:
>
> Hi,
>
>> i'm participating in the development of a wireless mesh network in
>> Delta de Tigre, near Buenos Aires, Argentina.
>> Landscape is absolutely flat, with dense forest 15-20mts tall. Nodes
>> are being installed along the river, 50 - 150mts apart, with clear LOS
>> at ground level.
>> I collaborated with Nico Echaniz last month, bulding another community
>> network in Cordoba, and given the success, I decided to start one more
>> here. I'm also really thankful with Elektra, for recommending the
>> tl-m3220 to Nico!
>> We chose batman over olsr, in part because of avahi stuff, but also to
>> have a real-world implementation and thus collaborate in testing /
>> debugging :) So far, there are  only 4 working nodes, but luckily I'm
>> already facing unexpected behaviour.
>>
>> I'm sorry for the long email, it's mostly terminal logs.
>> In short, iperf between two neighbouring nodes reports >20mbps, yet
>> running the same iperf between nodes separated by one hop reports
>> inexplicably low <5mbps.
>>
>> [..]
>>
>> Any ideas, thoughts, pointers? Any additional info needed?
>
> welcome on the list!
> I admit not having dived into all the details of your mail - the bigger part
> of the batman team is sitting in Athens at the moment (see battlemesh.org).
> The best next step is to find out whether this performance issue is a
> batman-
> adv or a wifi driver problem. May I suggest you remove batman-adv and repeat
> the same test using static routing ? If you still see the problem it is the
> wifi that is not working otherwise we have to fix batman-adv.
>
> Regards,
> Marek
>
>

Re: [B.A.T.M.A.N.] BW degradation on p2p links?

[Guido Iribarren]

[-- Attachment #1: Type: text/plain, Size: 2050 bytes --]

(this time i'll keep the mail short, insane details are attached as text :) )

Well, luckily for batman-adv devs, but unfortunately for me, the
problem is in fact related to relaying between wlan interfaces in
mr3x20 hardware (at least)

I understand this is not anymore batman related, but maybe someone has
experience on this, or can point me into the right direction /
maillist ?
Given you have a good deal of equipments at the BattleMesh, maybe
someone has already tried adding a wifi dongle and can confirm seeing
this behaviour?
(in a line of 2-radio nodes with static routing, packet throughput is
halved on each hop)

========

I've repeated the iperf tests in a controlled environment, this time
using different subnets on each interface and static routes built with
"route add". iptables flushed with -F and policy -P FORWARD ACCEPT
no batman at all.

Packets coming in through wlan1 and out through wlan0 (or viceversa)
experience (unexpected) "bandwidth degradation". Throughput drops from
30mbps to 15mbps (rough avg)
The speed is closely comparable to using only 1 radio (packet comes in
through wlan0, goes out through wlan0), so adding the dongle makes no
net difference. (besides alternating the channel for transmission,
throughput is halved over one hop anyway)
2 consecutive hops, alternating wifi radios, yield 1/3 throughput.
(like it happens in a 1-radio mesh)

this does not happen if the packet comes in through eth0 and goes out
through wlanX (or viceversa) . Throughput is maintained high at
30mbps.

=========

So the hardware (or kernel??) is uncapable of keeping up with the
transfer speed *when two radios are involved*.

Has anyone bumped into a similar case, or has experience with these
equipments? Maybe (again!) i am doing something wrong?

If this is confirmed, my only alternative to maintain good throughput
along the mesh will be to install 2 x mr3220 on each node, connected
back-to-back with ethernet cable. This will definitely work but the
cost for each node owner evidently rises ;(

Thanks a lot!

Guido

[-- Attachment #2: iperfs-forwarding --]
[-- Type: application/octet-stream, Size: 1552 bytes --]

.. mr3420 ...... mr3220+722 ...... mr3220 ...... TX / RX
[cpu  eth0] -> [eth0    cpu] ................... 94 / 92
[cpu wlan0] -> [wlan1   cpu] ................... 35 / 33
...............[cpu   wlan0] -> [wlan0   cpu] .. 32 / 33
...............[cpu   wlan1] -> [wlan0   cpu] .. 32 / 35
................................[cpu   wlan0] .. 45 / 32
[cpu  eth0] -> [eth0  wlan0] -> [wlan0   cpu] .. 30 / 34
[cpu  eth0] -> [eth0  wlan1] -> [wlan0   cpu] .. 32 / 30
...............[cpu   wlan0] -> [wlan0 wlan0] .. 21 / 16
[cpu wlan0] -> [wlan1 wlan0] -> [wlan0   cpu] .. 13 / 13
[cpu wlan0] -> [wlan1 wlan1] -> [wlan0   cpu] .. 15 / 15
[cpu wlan0] -> [wlan0 wlan0] -> [wlan0   cpu] .. 15 / 15
[cpu wlan0] -> [wlan0 wlan0] -> [wlan0  eth0] .. 19 / 15
[cpu wlan0] -> [wlan0 wlan0] -> [wlan0 wlan0] .. 13 / 10
[cpu wlan0] -> [wlan1 wlan0] -> [wlan0 wlan0] .. 12 / 10

"mr3220+722" has a tplink tl-wn722n connected to usb port
"cpu" means iperf traffic started or ended there (as a client or server)

in all cases wlan0 represents internal wifi (ath9k)
and wlan1 represents tl-wn722n (ath9k_htc)
the 4 lines that end in "wlan0" means that instead of iperf traffic ending in the last mr3220, it was forwarded to a laptop (running iperf) connected through the wifi interface as client.
the line that ends in "eth0" is analog, but the laptop was connected to the switch with a cable.

cmdline used was "iperf -c x.x.x.x -w320k -i1 -L5002 -r"
static routes were created as needed with "route add"
all speeds noted are in megabits per second (mbps) as reported by iperf.

Re: [B.A.T.M.A.N.] BW degradation on p2p links?

[Simon Wunderlich]

[-- Attachment #1: Type: text/plain, Size: 3088 bytes --]

Hey Guido,

the problem using one radio is explained further here (from another
company, but this applies to any WiFi mesh technology)

http://revolutionwifi.blogspot.com/2012/02/mesh-network-performance-impact.html

Also, if you use two-radio nodes, make sure that you use one module
on 2.4 GHz and one 5 GHz - if you use two modules in the same 
frequency band with omni antennas, even if you use separate channels
(like 1 and 11), they may interfere with each other if the antennas
are near to each other. There are quite a few ppl who can confirm this,
e.g. check this:

https://hackerspace.be/Wbm2009v2/TestInterferenceResults

Possible Solutions:
 * use different bands
 * use directional or sector antennas
 * have a reasonable distance between your omni antennas

Cheers,
	Simon

On Wed, Mar 28, 2012 at 07:41:05PM -0300, Guido Iribarren wrote:
> (this time i'll keep the mail short, insane details are attached as text :) )
> 
> Well, luckily for batman-adv devs, but unfortunately for me, the
> problem is in fact related to relaying between wlan interfaces in
> mr3x20 hardware (at least)
> 
> I understand this is not anymore batman related, but maybe someone has
> experience on this, or can point me into the right direction /
> maillist ?
> Given you have a good deal of equipments at the BattleMesh, maybe
> someone has already tried adding a wifi dongle and can confirm seeing
> this behaviour?
> (in a line of 2-radio nodes with static routing, packet throughput is
> halved on each hop)
> 
> ========
> 
> I've repeated the iperf tests in a controlled environment, this time
> using different subnets on each interface and static routes built with
> "route add". iptables flushed with -F and policy -P FORWARD ACCEPT
> no batman at all.
> 
> Packets coming in through wlan1 and out through wlan0 (or viceversa)
> experience (unexpected) "bandwidth degradation". Throughput drops from
> 30mbps to 15mbps (rough avg)
> The speed is closely comparable to using only 1 radio (packet comes in
> through wlan0, goes out through wlan0), so adding the dongle makes no
> net difference. (besides alternating the channel for transmission,
> throughput is halved over one hop anyway)
> 2 consecutive hops, alternating wifi radios, yield 1/3 throughput.
> (like it happens in a 1-radio mesh)
> 
> this does not happen if the packet comes in through eth0 and goes out
> through wlanX (or viceversa) . Throughput is maintained high at
> 30mbps.
> 
> =========
> 
> So the hardware (or kernel??) is uncapable of keeping up with the
> transfer speed *when two radios are involved*.
> 
> Has anyone bumped into a similar case, or has experience with these
> equipments? Maybe (again!) i am doing something wrong?
> 
> If this is confirmed, my only alternative to maintain good throughput
> along the mesh will be to install 2 x mr3220 on each node, connected
> back-to-back with ethernet cable. This will definitely work but the
> cost for each node owner evidently rises ;(
> 
> Thanks a lot!
> 
> Guido



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [B.A.T.M.A.N.] BW degradation on p2p links?

[Guido Iribarren]

Simon, you saved my day,

On Thu, Mar 29, 2012 at 9:51 AM, Simon Wunderlich
<simon.wunderlich@s2003.tu-chemnitz.de> wrote:
> the problem using one radio is explained further here (from another
> company, but this applies to any WiFi mesh technology)
>
> http://revolutionwifi.blogspot.com/2012/02/mesh-network-performance-impact.html

I was fully aware of this problem with single-radio nodes, and that's
why I was trying to build cheap two-radio nodes using mr3220 + usb
dongles.

>
> Also, if you use two-radio nodes,
> they may interfere with each other if the antennas
> are near to each other. There are quite a few ppl who can confirm this,
> e.g. check this:
>
> https://hackerspace.be/Wbm2009v2/TestInterferenceResults

Ah! you hit the nail right in the head!
I walked around the battlemesh wiki a couple of weeks ago, interested
into past events results, but only came across an rsync daemon serving
a tree with videos, photos and results, mostly in raw data form,
collected during the last (before athens) WBM edition.

I think i was a bit short-sighted, since the Wbm2009v2 link (and the
rest of that wbm wiki) was a wonderful read!

I did a couple of tests, and indeed increasing the distance between
the dongle and the router makes the whole difference.

I googled more on the subject and found a paper with thorough testing results
http://edoc.hu-berlin.de/oa/conferences/reSNAVwf8bA/PDF/28TjAjjEtU7ts.pdf

and even finer-grained testing here
http://userver.ftw.at/~valerio/files/wons.pdf

None of them use 802.11n equipment, so i set out to make my own experiment.
In my case, i needed about 1.5 meter between omni antennas to achieve
full throughput, placing both omni antennas perpendicular to the
ground and on the same plane.
On the other hand, if i place them one over the other (that is, align
their vertical axis, while holding them at different altitudes) a
distance as little as 20cm was enough to overcome interference.
(Sounds reasonable given the omnis gain spatial pattern)

I was able to get 20mbps from A to C (and viceversa), hopping on B.
[A ch=11] -- [ ch=11 B(2 radio) ch=1] -- [ch=1 C]
This was done using either BATMAN or static routing, giving almost
equal results (~1mbps less using batman)

If i put the two radios in B really close to each other (<5 cm)
throughput fell down to 8 mbps.
It's a relief it was such a simple issue!

Thanks a lot Simon and Marek for getting me into the right path,

So far we haven't heard of anyone implementing OpenWRT+batman-adv on
mr3220+usb to build two-radio-nodes mesh network, which means we're
mostly in a "solo adventure"
so your help and attention was even more appreciated!

Guido