Re: [B.A.T.M.A.N.] [Babel-users] A peer-reviewed assessment of OLSR, BATMAN and Babel

Re: [B.A.T.M.A.N.] [Babel-users] A peer-reviewed assessment of OLSR, BATMAN and Babel

[Chun-Ping Wang]

Hi all, 

Many thanks for all your feedbacks.  It will definately help us to improve our work.
However, I think there is quite a bit of confusions at very beginning.  So I would 
like to make some clarifications for this paper.

As it has been mentioned earlier, this paper was to study all the protocols in
their default setting.  In term of "default setting" for OLSR, we used original 
config file shipped in olsr v0.5.5.  As far as I know, this config file is based on 
the LQ mode of OLSR  (i.e. olsrd.conf.default.lq ).  So, correct me if I am wrong, 
we have been using OLSR with ETX to begin with. We only modified the HELLO and
TC interval to improve static node performance. The results shown in our paper is 
actually the performance of OLSR with ETX.  

You may also aware the protocols that we used are already out-dated.  This is because 
this work was done back in 2008.  We are more than happy to undergo another study 
with more recent protocols.

Regards, 
Jerry









________________________________________
From: Henning Rogge [hrogge@googlemail.com]
Sent: Monday, November 09, 2009 8:07 PM
To: Mehran Abolhasan
Cc: Juliusz Chroboczek; babel-users@lists.alioth.debian.org; olsr-users@lists.olsr.org; b.a.t.m.a.n@lists.open-mesh.net; Brett Hagelstein; justin.lipman@gmail.com; Brett Hegelstein; Chun-Ping Wang
Subject: Re: [Babel-users] A peer-reviewed assessment of OLSR, BATMAN and       Babel

On Mon, Nov 9, 2009 at 05:46, Dr. Mehran Abolhasan <mehrana@uow.edu.au> wrote:
> Hi All,
>
> Many Thanks for all your comments. Just a couple of point of clarification.
>
> 1. The aim of this paper was to study all the protocols in their default
> settings. We did not
> switch off ETX with olsr (note we used the olsr version from olsr.org). In
> fact the link quality metric was left to 2 by default. We are well aware
> that ETX provide more stable routes than hop count.
The problem is that the "RFC" mode of olsr.org is not well tested, the
MPR algorithm is broken in 0.5.6 and RFC conform OLSR networks do not
work in practice for non-trivial networks. I'm surprised that you got
results this good.

> 2. In terms of looking at performance using different parameters, we will be
> doing this in our future studies.
> Also, note that the conference papers were limited to 4 pages only.
If you plan to start your tests, feel free to contact the olsr mailing
lists for some suggestions for parameters. ;)

> 3. In terms of overheads, given that this was a small scale indoor test-bed,
> we believed the amount of overhead introduced into the network is not
> signficant enough to adversely affect the network. So we did not look into
> overheads for this paper, however we would do this for larger test-beds.
>
> 4. We previously ran OLSR using various different outdoor and indoor
> test-beds and at the time we were doing the experimentations BATMAN and
> BABEL were more stable.
Noone surprised (without ETX).

Henning Rogge

--
"Wo kämen wir hin, wenn alle sagten, wo kämem wir hin, und niemand
ginge, um einmal zu schauen, wohin man käme, wenn man ginge." (Kurt
Marti)

Re: [B.A.T.M.A.N.] [Babel-users] A peer-reviewed assessment of OLSR, BATMAN and Babel

[Henning Rogge]

[-- Attachment #1: Type: Text/Plain, Size: 1529 bytes --]

Am Montag 09 November 2009 14:00:24 schrieb Chun-Ping Wang:
> Hi all,
> 
> Many thanks for all your feedbacks.  It will definately help us to improve
>  our work. However, I think there is quite a bit of confusions at very
>  beginning.  So I would like to make some clarifications for this paper.
> 
> As it has been mentioned earlier, this paper was to study all the protocols
>  in their default setting.  In term of "default setting" for OLSR, we used
>  original config file shipped in olsr v0.5.5.  As far as I know, this
>  config file is based on the LQ mode of OLSR  (i.e. olsrd.conf.default.lq
>  ).  So, correct me if I am wrong, we have been using OLSR with ETX to
>  begin with. We only modified the HELLO and TC interval to improve static
>  node performance. The results shown in our paper is actually the
>  performance of OLSR with ETX.
Hmm... olsrd.conf.default.lq should be with ETX...

> You may also aware the protocols that we used are already out-dated.  This
>  is because this work was done back in 2008.  We are more than happy to
>  undergo another study with more recent protocols.
The implementation was outdated, not the protocol... but the protocol does 
only mention hopcount, so most people just use "RFC compatible" OLSR.

I would be interested what config file you used for the test. If you plan to 
redo the test we (the olsr.org team) would be very interested in getting 
feedback what kind of problems you have so that we can improve our software.

Henning Rogge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[B.A.T.M.A.N.] A peer-reviewed assessment of OLSR, BATMAN and Babel

[Juliusz Chroboczek]

I've just come upon an interesting paper that experimentally compares
the performance of OLSR, BATMAN and Babel.

    Real-world Performance of Current Proactive Multi-hop Mesh
    Protocols.  M. Abolhasan, B. Hagelstein, J. C.-P. Wang.

    http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1747&context=infopapers

Short summary: see Table II on the last page.


A few comments on the paper:

1. Section II (the informal description of the protocols) doesn't make
   much sense.  Ignore it.

2. They evaluated original OLSR, not OLSR-ETX as used by our friends in
   Vienna and Berlin.

3. The results in Figure 3 would appear to imply that there's a bug in
   Babel -- it loses a packet every time it switches routes.  I think
   I understand why.

4. They ran the routing daemons with the default parameters.  This means
   that BATMAN ran with an OGM interval of 1 second, while Babel used
   a Hello interval of 4 seconds.  It would have been interesting to see
   the results with similar parameters.

5. They didn't measure the amount of routing protocol traffic.

                                        Juliusz

Re: [B.A.T.M.A.N.] [Babel-users] A peer-reviewed assessment of OLSR, BATMAN and Babel

[Henning Rogge]

[-- Attachment #1: Type: Text/Plain, Size: 1079 bytes --]

Am Freitag 06 November 2009 20:05:31 schrieb Juliusz Chroboczek:
> I've just come upon an interesting paper that experimentally compares
> the performance of OLSR, BATMAN and Babel.
> 
>     Real-world Performance of Current Proactive Multi-hop Mesh
>     Protocols.  M. Abolhasan, B. Hagelstein, J. C.-P. Wang.
>    
>  http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1747&context=infopapers
Interesting link Juliusz, thank you.
 
> Short summary: see Table II on the last page.
> 
> 
> A few comments on the paper:
> 
> 1. Section II (the informal description of the protocols) doesn't make
>    much sense.  Ignore it.
> 
> 2. They evaluated original OLSR, not OLSR-ETX as used by our friends in
>    Vienna and Berlin.
This is the bane of the OLSR protocol. Most people not doing mesh-research 
just "use the RFC compatible OLSR" to compare it with anything else... and 
discover (as we all know) that hopcount metric does not work. I'm fighting for 
including a simple ETX implementation into the coming OLSRv2 RFC at the 
moment.

Henning Rogge

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [B.A.T.M.A.N.] A peer-reviewed assessment of OLSR, BATMAN and Babel

[Dr. Mehran Abolhasan]

Hi All,

Many Thanks for all your comments. Just a couple of point of clarification.

1. The aim of this paper was to study all the protocols in their default 
settings. We did not
switch off ETX with olsr (note we used the olsr version from olsr.org). In 
fact the link quality metric was left to 2 by default. We are well aware 
that ETX provide more stable routes than hop count.

2. In terms of looking at performance using different parameters, we will be 
doing this in our future studies.
Also, note that the conference papers were limited to 4 pages only.

3. In terms of overheads, given that this was a small scale indoor test-bed, 
we believed the amount of overhead introduced into the network is not 
signficant enough to adversely affect the network. So we did not look into 
overheads for this paper, however we would do this for larger test-beds.

4. We previously ran OLSR using various different outdoor and indoor 
test-beds and at the time we were doing the experimentations BATMAN and 
BABEL were more stable.

We would be interested to hear what other aspects of the protocols (such as 
different parameters) you would like to see studied.


Kind regards,
Mehran




----- Original Message ----- 
From: "Juliusz Chroboczek" <jch@pps.jussieu.fr>
To: <babel-users@lists.alioth.debian.org>; <olsr-users@lists.olsr.org>; 
<b.a.t.m.a.n@lists.open-mesh.net>
Cc: "Mehran Abolhasan" <mehrana@uow.edu.au>; "Brett Hagelstein" 
<bretth@uow.edu.au>; "Chun-Ping Wang" <jerryw@uow.edu.au>
Sent: Saturday, November 07, 2009 6:05 AM
Subject: A peer-reviewed assessment of OLSR, BATMAN and Babel


I've just come upon an interesting paper that experimentally compares
the performance of OLSR, BATMAN and Babel.

    Real-world Performance of Current Proactive Multi-hop Mesh
    Protocols.  M. Abolhasan, B. Hagelstein, J. C.-P. Wang.

    http://ro.uow.edu.au/cgi/viewcontent.cgi?article=1747&context=infopapers

Short summary: see Table II on the last page.


A few comments on the paper:

1. Section II (the informal description of the protocols) doesn't make
   much sense.  Ignore it.

2. They evaluated original OLSR, not OLSR-ETX as used by our friends in
   Vienna and Berlin.

3. The results in Figure 3 would appear to imply that there's a bug in
   Babel -- it loses a packet every time it switches routes.  I think
   I understand why.

4. They ran the routing daemons with the default parameters.  This means
   that BATMAN ran with an OGM interval of 1 second, while Babel used
   a Hello interval of 4 seconds.  It would have been interesting to see
   the results with similar parameters.

5. They didn't measure the amount of routing protocol traffic.

                                        Juliusz

Re: [B.A.T.M.A.N.] [Babel-users] A peer-reviewed assessment of OLSR, BATMAN and Babel

[Henning Rogge]

On Mon, Nov 9, 2009 at 05:46, Dr. Mehran Abolhasan <mehrana@uow.edu.au> wrote:
> Hi All,
>
> Many Thanks for all your comments. Just a couple of point of clarification.
>
> 1. The aim of this paper was to study all the protocols in their default
> settings. We did not
> switch off ETX with olsr (note we used the olsr version from olsr.org). In
> fact the link quality metric was left to 2 by default. We are well aware
> that ETX provide more stable routes than hop count.
The problem is that the "RFC" mode of olsr.org is not well tested, the
MPR algorithm is broken in 0.5.6 and RFC conform OLSR networks do not
work in practice for non-trivial networks. I'm surprised that you got
results this good.

> 2. In terms of looking at performance using different parameters, we will be
> doing this in our future studies.
> Also, note that the conference papers were limited to 4 pages only.
If you plan to start your tests, feel free to contact the olsr mailing
lists for some suggestions for parameters. ;)

> 3. In terms of overheads, given that this was a small scale indoor test-bed,
> we believed the amount of overhead introduced into the network is not
> signficant enough to adversely affect the network. So we did not look into
> overheads for this paper, however we would do this for larger test-beds.
>
> 4. We previously ran OLSR using various different outdoor and indoor
> test-beds and at the time we were doing the experimentations BATMAN and
> BABEL were more stable.
Noone surprised (without ETX).

Henning Rogge

-- 
"Wo kämen wir hin, wenn alle sagten, wo kämem wir hin, und niemand
ginge, um einmal zu schauen, wohin man käme, wenn man ginge." (Kurt
Marti)