[B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Freifunk Dresden]

Hi,

I'm currently try using batmand-experimental Rev.972.

I have encountered a strange behaviour. I'm running two
WRT54 (GL and GS).
Routing is working and there is enough memory. I have setup
a cron job to call "batmand -c -d [2,7,8,9]" every minutes to
update the status files on ramdisk. The webinterface then
access the content of the status file which reduces the cpu load.

The WRT54GS is working almost, but the WRT54GL hangs after a while
as described below:

The call to "batmand -c -d [2,7,8,9]" blocks batmand completely.
batman does not do any routing and OGM processing. As result the
router leaves the network.
I can still call "batmand -c -r 3" and verify with "batmand -c" that the
options were set. But OGMs are not processed.
Any call to access the debug-information is blocked.

After killall batmand and restart the call to "batmand -c -d x" is possible
serveral times until batman hangs. The process list shows this
"batmand -c -d x" - process.

I have compiled batmand for whiterussian_rc6 and with the following options:
(The email server has a problem with the assignment character so I have
removed it in this email)
CFLAGS         -Wall -Os
LDFLAGS        -lpthread
CFLAGS_MIPS    -Wall -Os -DREVISION_VERSION $(REVISION_VERSION)
LDFLAGS_MIPS   -lpthread

I had to remove the -pg option because it was not possible to compile.
Also in whiterussian_rc the CFLAGS_MIPS/LDFLAGS_MIPS are not used (I think).

Any Idea
/Stephan

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[elektra]

Hi -
>
> Routing is working and there is enough memory. I have setup
> a cron job to call "batmand -c -d [2,7,8,9]" every minutes to
> update the status files on ramdisk. The webinterface then
> access the content of the status file which reduces the cpu load.
just a stupid question to verify things - you have not forgotten to add 
-b for batch mode to the command?

cu elektra

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Freifunk Dresden]

Zitat von elektra <onelektra@gmx.net>:

[Zitattext verstecken]
Hi -

Routing is working and there is enough memory. I have setup
a cron job to call "batmand -c -d [2,7,8,9]" every minutes to
update the status files on ramdisk. The webinterface then
access the content of the status file which reduces the cpu load.
just a stupid question to verify things - you have not forgotten to  
add -b for batch mode to the command?
yes, you are right. I just forgot this option in the previous post.
The firmware uses the -b option.
When I create the firmware I'm using the whiterussian kit 1.4.5. It generates
the same firmware for different routers. So when I flash the WRT54GS and the
WRT54GL with the same firmware, the WRT54GL batmand hangs. There are
still 2Mbyte Ram unused. restarting batman does almost not change the memory
consumtion. I use the "top" command for this.

I have made a log when the batmand stopps. Perhaps it helps you a little.
http://www.ddmesh.de/batmand-hanglog.txt

/Stephan

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Axel Neumann]

Hi Stephan,

I have several WRT54GL here and I can execute something like
"batmand -cbd8"
as often as I want. It never hangs. 
Can you attach the cron file that is executed and that causes the problem?

ciao,
axel

On Sonntag 03 Februar 2008, Freifunk Dresden wrote:
> Hi,
>
> I'm currently try using batmand-experimental Rev.972.
>
> I have encountered a strange behaviour. I'm running two
> WRT54 (GL and GS).
> Routing is working and there is enough memory. I have setup
> a cron job to call "batmand -c -d [2,7,8,9]" every minutes to
> update the status files on ramdisk. The webinterface then
> access the content of the status file which reduces the cpu load.
>
> The WRT54GS is working almost, but the WRT54GL hangs after a while
> as described below:
>
> The call to "batmand -c -d [2,7,8,9]" blocks batmand completely.
> batman does not do any routing and OGM processing. As result the
> router leaves the network.
> I can still call "batmand -c -r 3" and verify with "batmand -c" that the
> options were set. But OGMs are not processed.
> Any call to access the debug-information is blocked.
>
> After killall batmand and restart the call to "batmand -c -d x" is possible
> serveral times until batman hangs. The process list shows this
> "batmand -c -d x" - process.
>
> I have compiled batmand for whiterussian_rc6 and with the following
> options: (The email server has a problem with the assignment character so I
> have removed it in this email)
> CFLAGS         -Wall -Os
> LDFLAGS        -lpthread
> CFLAGS_MIPS    -Wall -Os -DREVISION_VERSION $(REVISION_VERSION)
> LDFLAGS_MIPS   -lpthread
>
> I had to remove the -pg option because it was not possible to compile.
> Also in whiterussian_rc the CFLAGS_MIPS/LDFLAGS_MIPS are not used (I
> think).
>
> Any Idea
> /Stephan
>
> _______________________________________________
> B.A.T.M.A.N mailing list
> B.A.T.M.A.N@open-mesh.net
> https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Freifunk Dresden]

Hi Axel,

> I have several WRT54GL here and I can execute something like
> "batmand -cbd8"
> as often as I want. It never hangs.
> Can you attach the cron file that is executed and that causes the problem?
The crontab contains an entry such like:
0-59/1 * * * * /etc/init.d/S53batmand check

this calles the following commands after another:
batmand -cb -d2 >/tmp/batmand_gateway
batmand -cb -d7 >/tmp/batmand_gateway
batmand -cb -d8 >/tmp/batmand_gateway
batmand -cb -d9 >/tmp/batmand_gateway

But I also did put these commands into an loop:
while true; do batmand -cb -d2;....;done

I have also disabled this "logging" completely and let only run batmand to
build up the net. I can not say if the access to the debug output leads to
blocking the batmand faster.I also have seen that batmand blocks after  
awhile if
it is only running for building the network.

I have put a logfile on my webpage.
http://www.ddmesh.de/batmand-hanglog.txt

In one of the previous threads someone had a problem with "batmand  
going crazy".
I'm not sure to remember right. But I think that it had to do with  
sequence number that's wrapping around.
The logfile ends at the time batmand stopps. At the end of this log you
will find something like "prevRxSeqno: 0, currRxSeqno-prevRxSeqno 0," perhabs
it is the same reason.

batmand is currently started with two interfaces eth1 and tbb. eth1 is  
the wireless interface and tbb is a tun/tap device that is used by vpn  
tincd.
tincd has got invalid hostnames, so it never creates a connection.
Perhabs batmand has a problem with this kind of "dead" interfaces.
I have tried to remove this tbb interface when starting batmand.  
batmand was running at least for two days. But the "dead" interface  
may also have no influence to this problem.
Currently batmand is running since 10 hours with eth1 and tbb (dead  
interface).

I never have seen this problem with the WRT54GS, only with GL.

/Stephan

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Axel Neumann]

Hello,

On Dienstag 12 Februar 2008, Freifunk Dresden wrote:
> I have also disabled this "logging" completely and let only run batmand to
> build up the net. I can not say if the access to the debug output leads to
> blocking the batmand faster.I also have seen that batmand blocks after
> awhile if it is only running for building the network.
Ok, then it does not strictly depend on the logging.

>
> I have put a logfile on my webpage.
> http://www.ddmesh.de/batmand-hanglog.txt
>
> In one of the previous threads someone had a problem with "batmand
> going crazy".
> I'm not sure to remember right. But I think that it had to do with
> sequence number that's wrapping around.
> The logfile ends at the time batmand stopps. At the end of this log you
> will find something like "prevRxSeqno: 0, currRxSeqno-prevRxSeqno 0,"
> perhabs it is the same reason.
I checked the log file.
the "prevRxSeqno: 0..." line is no problem. The "0" comes from a bad debug 
statement. If you search your debug log you'll see many of these lines.
The "going crazy..." thing was related to overlapping uptime - thats also 
another story. 

>
> batmand is currently started with two interfaces eth1 and tbb. eth1 is
> the wireless interface and tbb is a tun/tap device that is used by vpn
> tincd.
> tincd has got invalid hostnames, so it never creates a connection.
> Perhabs batmand has a problem with this kind of "dead" interfaces.
> I have tried to remove this tbb interface when starting batmand.
> batmand was running at least for two days. But the "dead" interface
> may also have no influence to this problem.
> Currently batmand is running since 10 hours with eth1 and tbb (dead
> interface).
Can you verify if the problem also occures if batmand is started without any 
tap devices? 
Can you check for other syslog messages that might be related to the stopping 
batmand? What does logread say ?

The strange thing is that the debug-level-4 output stops in the middle of an 
action. Can you also check for the number of batmand processes before and 
after the stopped batmand process?

Have you ever tried what happens if you connect the tap interface to a bridge 
and bind batmand to the bridge device instead?

Last but not least: have you observed (or explicitly not observed) this 
phenomenon also with previous revisions in the same scenario ? 

>
> I never have seen this problem with the WRT54GS, only with GL.
Is the batmand on the WRT54GS also bound to a tinc interface ?

ciao,
axel


>
> /Stephan
>
>
>
>
> _______________________________________________
> B.A.T.M.A.N mailing list
> B.A.T.M.A.N@open-mesh.net
> https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Freifunk Dresden]

Hello,

> Can you verify if the problem also occures if batmand is started without any
> tap devices?
my last 10h log (with tap dev) was also crashing. I'm currently let run
without the tap dev since few hours. I like to run it longer.

> Can you check for other syslog messages that might be related to the stopping
> batmand? What does logread say ?
I have looked at that, but did not find any strange log


> The strange thing is that the debug-level-4 output stops in the middle of an
> action. Can you also check for the number of batmand processes before and
> after the stopped batmand process?
The number of task are the same.
But I have seen, that when the -d4 output stopps and I keep this  
batmand running
when accessing an different log level from another terminal, I see the
socket-connection logs in -d4 output.

Also I still can just call "batmand -c" to see the parameters and  
current gateway settings. I also can change the gateway settings.

The batmand seems to stop processing any OGMs.

> Have you ever tried what happens if you connect the tap interface to a bridge
> and bind batmand to the bridge device instead?
I haven't tried it, yet. but this also came in my mind. I will this  
check after
finishing the "no-tap-dev-test"

> Last but not least: have you observed (or explicitly not observed) this
> phenomenon also with previous revisions in the same scenario ?
I can not say, because implementing tinc and updateing the batmand  
version was at same time.

>> I never have seen this problem with the WRT54GS, only with GL.
> Is the batmand on the WRT54GS also bound to a tinc interface ?
Yes, the GL is running standalone with stubid tincsetup and also the  
GS was running with same parameters and standalone (no network cable).

Perhaps it is more random and is depending on speed of the router when the
event occurs.

bye Stephan

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Axel Neumann]

Hi,

>
> > The strange thing is that the debug-level-4 output stops in the middle of
> > an action. Can you also check for the number of batmand processes before
> > and after the stopped batmand process?
>
> The number of task are the same.
> But I have seen, that when the -d4 output stopps and I keep this
> batmand running
> when accessing an different log level from another terminal, I see the
> socket-connection logs in -d4 output.
>
> Also I still can just call "batmand -c" to see the parameters and
> current gateway settings. I also can change the gateway settings.
>
> The batmand seems to stop processing any OGMs.

The messages you see are logged from another thread (not the thread which is 
doing the OGM processing). Thats also the reason why some of the dynamically 
changeable parameters _seem_ to be processed. I guess for example a
"batmand -c -a 1.2.3.4/32" wont be processed completely.
In this case a simultaneous running "batmand -cd3" _should_ report:
[    162940] Unix socket: got connection
[    162946] got request: 10
[    162947] Unix socket: Requesting adding of HNA 1.2.3.4/32 - put this on 
todo list...
[    162951] got request: 10
[    162952] Unix client closed connection ...
[    163157] found todo item, adding HNA 1.2.3.4/32 atype 1

I guess everything except the last line will be shown. The last line is 
generated from the OGM-processing thread which seems to be blocked.

Perhaps, if you can find a way to reliable reproduce this kind of problem then 
it would be much easier to fix it. Just an idea, what happens with batmand 
(bound to the tap interface) when stopping the running tincd like this:  
kill -STOP $(pidof tincd)
and later on:
kill -CONT $(pidof tincd)

ciao,
axel
>
>
> > Have you ever tried what happens if you connect the tap interface to a
> > bridge and bind batmand to the bridge device instead?
>
> I haven't tried it, yet. but this also came in my mind. I will this
> check after
> finishing the "no-tap-dev-test"
>
> > Last but not least: have you observed (or explicitly not observed) this
> > phenomenon also with previous revisions in the same scenario ?
>
> I can not say, because implementing tinc and updateing the batmand
> version was at same time.
>
> >> I never have seen this problem with the WRT54GS, only with GL.
> >
> > Is the batmand on the WRT54GS also bound to a tinc interface ?
>
> Yes, the GL is running standalone with stubid tincsetup and also the
> GS was running with same parameters and standalone (no network cable).
>
> Perhaps it is more random and is depending on speed of the router when the
> event occurs.
>
> bye Stephan
>
>
> _______________________________________________
> B.A.T.M.A.N mailing list
> B.A.T.M.A.N@open-mesh.net
> https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Freifunk Dresden]

Hi Axel,

I have done some checks. First I have added the tincd tap interface to
a bridge and used the bridge as parameter for batmand. This is currently
running since 10h.

The second what I have tried is to call "kill -STOP $(pidof tincd)".
On the WRT54GL batmand stopped quite fast, but it stopped also fast without
the "kill". On the WRT54SL where I never have seen batmand stopping, the
"kill" has stopped batmand after 5-10 minutes. I could reproduce this serveral
times with the GS.

When calling batmand -c -a 120.56.56.56/32 I get the following output on
batmand -cd3:

[    275531] Unix socket: got connection
[    275539] got request: 10
[    275540] Unix socket: Requesting adding of HNA 120.56.56.34/32 -  
put this on todo list...
[    275544] got request: 10
[    275545] Unix client closed connection ...

Three batmand processes are running and batmand -c returns:
WARNING: You are using BatMan-eXp 0.3-alpha (compatibility version 10) !
/sbin/batmand [not-all-options-displayed] -r 2 -a 10.12.10.16/28 eth1 tbb

After calling "killall batmand" only one of three batmand processes is killed.
I have to killall -9 to get rid of it.

Regards
/Stephan

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Axel Neumann]

Hi Stephan,

can you check if rv980 solves the problem ?

ciao,
axel

On Donnerstag 14 Februar 2008, Freifunk Dresden wrote:
> Hi Axel,
>
> I have done some checks. First I have added the tincd tap interface to
> a bridge and used the bridge as parameter for batmand. This is currently
> running since 10h.
>
> The second what I have tried is to call "kill -STOP $(pidof tincd)".
> On the WRT54GL batmand stopped quite fast, but it stopped also fast without
> the "kill". On the WRT54SL where I never have seen batmand stopping, the
> "kill" has stopped batmand after 5-10 minutes. I could reproduce this
> serveral times with the GS.
>
> When calling batmand -c -a 120.56.56.56/32 I get the following output on
> batmand -cd3:
>
> [    275531] Unix socket: got connection
> [    275539] got request: 10
> [    275540] Unix socket: Requesting adding of HNA 120.56.56.34/32 -
> put this on todo list...
> [    275544] got request: 10
> [    275545] Unix client closed connection ...
>
> Three batmand processes are running and batmand -c returns:
> WARNING: You are using BatMan-eXp 0.3-alpha (compatibility version 10) !
> /sbin/batmand [not-all-options-displayed] -r 2 -a 10.12.10.16/28 eth1 tbb
>
> After calling "killall batmand" only one of three batmand processes is
> killed. I have to killall -9 to get rid of it.
>
> Regards
> /Stephan
>
>
>
>
>
>
> _______________________________________________
> B.A.T.M.A.N mailing list
> B.A.T.M.A.N@open-mesh.net
> https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Freifunk Dresden]

Hi Axel,

when I use rv980 and call "kill -STOP $(pidof tincd)" then batman works
for a while (1-2 minutes) and exists with the following error:

[    411413] Error - can't send udp packet: Resource temporarily unavailable.
Connection terminated by remote host

When I restart batmand when tincd is still stopped, the OGM are processed and
I see the neighbour nodes until I get the same error after about 1-2 minutes.
good news, it does not block.

Because of the error message I have inserted at posix/posix.c: 705 "return 0"
(send_udp_packet()).

I have tested two wrt54gl that had the problems before with the rev980 and
the "return 0" statement. Both routers has be running for 18 hours when
tincd was running without creating a connection.
After that I called "kill -STOP $(pidof tincd)" on both routers. Both routers
running now since 13 hours without problems.


Please check and insert the "return 0" at posix/posix:705 if it is the right
solution.

Have a nice week

/Stephan


Zitat von Axel Neumann <axel@open-mesh.net>:

> Hi Stephan,
>
> can you check if rv980 solves the problem ?
>
> ciao,
> axel
>
> On Donnerstag 14 Februar 2008, Freifunk Dresden wrote:
>> Hi Axel,
>>
>> I have done some checks. First I have added the tincd tap interface to
>> a bridge and used the bridge as parameter for batmand. This is currently
>> running since 10h.
>>
>> The second what I have tried is to call "kill -STOP $(pidof tincd)".
>> On the WRT54GL batmand stopped quite fast, but it stopped also fast without
>> the "kill". On the WRT54SL where I never have seen batmand stopping, the
>> "kill" has stopped batmand after 5-10 minutes. I could reproduce this
>> serveral times with the GS.
>>
>> When calling batmand -c -a 120.56.56.56/32 I get the following output on
>> batmand -cd3:
>>
>> [    275531] Unix socket: got connection
>> [    275539] got request: 10
>> [    275540] Unix socket: Requesting adding of HNA 120.56.56.34/32 -
>> put this on todo list...
>> [    275544] got request: 10
>> [    275545] Unix client closed connection ...
>>
>> Three batmand processes are running and batmand -c returns:
>> WARNING: You are using BatMan-eXp 0.3-alpha (compatibility version 10) !
>> /sbin/batmand [not-all-options-displayed] -r 2 -a 10.12.10.16/28 eth1 tbb
>>
>> After calling "killall batmand" only one of three batmand processes is
>> killed. I have to killall -9 to get rid of it.
>>
>> Regards
>> /Stephan
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> B.A.T.M.A.N mailing list
>> B.A.T.M.A.N@open-mesh.net
>> https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n
>
>
> _______________________________________________
> B.A.T.M.A.N mailing list
> B.A.T.M.A.N@open-mesh.net
> https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n
>

Re: [B.A.T.M.A.N.] Rev 972: batmand -c -d n hangs sometimes

[Axel Neumann]

Hi Stephan,

thanks for that hint. I changed that lines with rv982 and also made 
the --resist-blocked-send switch the default behavior now. I also believe 
that terminating the daemon due to any temporary unavailable device or 
blocked port is too paranoid. 
This should also help with other previously reported problems like
"[B.A.T.M.A.N.] strange batman aborting" indicated by Jan

ciao,
axel

On Sonntag 17 Februar 2008, Freifunk Dresden wrote:
> Hi Axel,
>
> when I use rv980 and call "kill -STOP $(pidof tincd)" then batman works
> for a while (1-2 minutes) and exists with the following error:
>
> [    411413] Error - can't send udp packet: Resource temporarily
> unavailable. Connection terminated by remote host
>
> When I restart batmand when tincd is still stopped, the OGM are processed
> and I see the neighbour nodes until I get the same error after about 1-2
> minutes. good news, it does not block.
>
> Because of the error message I have inserted at posix/posix.c: 705 "return
> 0" (send_udp_packet()).
>
> I have tested two wrt54gl that had the problems before with the rev980 and
> the "return 0" statement. Both routers has be running for 18 hours when
> tincd was running without creating a connection.
> After that I called "kill -STOP $(pidof tincd)" on both routers. Both
> routers running now since 13 hours without problems.
>
>
> Please check and insert the "return 0" at posix/posix:705 if it is the
> right solution.
>
> Have a nice week
>
> /Stephan
>
> Zitat von Axel Neumann <axel@open-mesh.net>:
> > Hi Stephan,
> >
> > can you check if rv980 solves the problem ?
> >
> > ciao,
> > axel
> >
> > On Donnerstag 14 Februar 2008, Freifunk Dresden wrote:
> >> Hi Axel,
> >>
> >> I have done some checks. First I have added the tincd tap interface to
> >> a bridge and used the bridge as parameter for batmand. This is currently
> >> running since 10h.
> >>
> >> The second what I have tried is to call "kill -STOP $(pidof tincd)".
> >> On the WRT54GL batmand stopped quite fast, but it stopped also fast
> >> without the "kill". On the WRT54SL where I never have seen batmand
> >> stopping, the "kill" has stopped batmand after 5-10 minutes. I could
> >> reproduce this serveral times with the GS.
> >>
> >> When calling batmand -c -a 120.56.56.56/32 I get the following output on
> >> batmand -cd3:
> >>
> >> [    275531] Unix socket: got connection
> >> [    275539] got request: 10
> >> [    275540] Unix socket: Requesting adding of HNA 120.56.56.34/32 -
> >> put this on todo list...
> >> [    275544] got request: 10
> >> [    275545] Unix client closed connection ...
> >>
> >> Three batmand processes are running and batmand -c returns:
> >> WARNING: You are using BatMan-eXp 0.3-alpha (compatibility version 10) !
> >> /sbin/batmand [not-all-options-displayed] -r 2 -a 10.12.10.16/28 eth1
> >> tbb
> >>
> >> After calling "killall batmand" only one of three batmand processes is
> >> killed. I have to killall -9 to get rid of it.
> >>
> >> Regards
> >> /Stephan
> >>
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> B.A.T.M.A.N mailing list
> >> B.A.T.M.A.N@open-mesh.net
> >> https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n
> >
> > _______________________________________________
> > B.A.T.M.A.N mailing list
> > B.A.T.M.A.N@open-mesh.net
> > https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n
>
> _______________________________________________
> B.A.T.M.A.N mailing list
> B.A.T.M.A.N@open-mesh.net
> https://list.open-mesh.net/mm/listinfo/b.a.t.m.a.n