[2.6.20-rt8] "Neighbour table overflow."

[2.6.20-rt8] "Neighbour table overflow."

[Guennadi Liakhovetski]

Hi

After fixing the irda spinlock init bug, here's the next one I've run 
into: after a long irnet/ppp test with a periodic disturbance, whereby the 
ppp connection is broken and restored every ~20 secnds, I've got this from 
the kernel:

user.warn kernel: Neighbour table overflow.

and the system cannot build any new outgoing network connections - ppp or 
ethernet. I can connect from outside, but from the system produces:

# ping 192.168.1.1
connect: No buffer space available

where "new" connections is to new hosts. To known hosts connections can be 
further built. And, here's another one:

# ping 192.168.1.5
PING 192.168.1.5 (192.168.1.5) 56(84) bytes of data.
64 bytes from 192.168.1.5: icmp_seq=1 ttl=64 time=3.45 ms
64 bytes from 192.168.1.5: icmp_seq=1 ttl=64 time=8.34 ms (DUP!)
64 bytes from 192.168.1.5: icmp_seq=1 ttl=64 time=9.29 ms (DUP!)
64 bytes from 192.168.1.5: icmp_seq=1 ttl=64 time=11.5 ms (DUP!)
64 bytes from 192.168.1.5: icmp_seq=2 ttl=64 time=4.79 ms
64 bytes from 192.168.1.5: icmp_seq=2 ttl=64 time=7.86 ms (DUP!)
64 bytes from 192.168.1.5: icmp_seq=2 ttl=64 time=10.8 ms (DUP!)
64 bytes from 192.168.1.5: icmp_seq=2 ttl=64 time=12.0 ms (DUP!)
64 bytes from 192.168.1.5: icmp_seq=3 ttl=64 time=4.04 ms
64 bytes from 192.168.1.5: icmp_seq=3 ttl=64 time=8.17 ms (DUP!)
64 bytes from 192.168.1.5: icmp_seq=3 ttl=64 time=11.1 ms (DUP!)
64 bytes from 192.168.1.5: icmp_seq=3 ttl=64 time=13.2 ms (DUP!)

any ideas?...

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: [2.6.20-rt8] "Neighbour table overflow."

[Ingo Molnar]

* Guennadi Liakhovetski <gl@dsa-ac.de> wrote:

> Hi
> 
> After fixing the irda spinlock init bug, [...]

could you send the patch for that? Is it upstream already?

> user.warn kernel: Neighbour table overflow.

hm, i havent seen this one yet.

	Ingo

Re: [irda-users] [2.6.20-rt8] "Neighbour table overflow."

[Samuel Ortiz]

Hi Ingo,

On 3/16/2007, "Ingo Molnar" <mingo@elte.hu> wrote:

>
>* Guennadi Liakhovetski <gl@dsa-ac.de> wrote:
>
>> Hi
>>
>> After fixing the irda spinlock init bug, [...]
>
>could you send the patch for that? Is it upstream already?
Guennadi sent it to me earlier this afternoon. I will forward it upstream
later today.

Cheers,
Samuel.


>> user.warn kernel: Neighbour table overflow.
>
>hm, i havent seen this one yet.
>
>	Ingo
>
>-------------------------------------------------------------------------
>Take Surveys. Earn Cash. Influence the Future of IT
>Join SourceForge.net's Techsay panel and you'll get the chance to share your
>opinions on IT & business topics through brief surveys-and earn cash
>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>_______________________________________________
>irda-users mailing list
>irda-users@lists.sourceforge.net
>http://lists.sourceforge.net/lists/listinfo/irda-users

Re: [2.6.20-rt8] "Neighbour table overflow."

[Guennadi Liakhovetski]

On Fri, 16 Mar 2007, Ingo Molnar wrote:

> * Guennadi Liakhovetski <gl@dsa-ac.de> wrote:
>>
>> After fixing the irda spinlock init bug, [...]
>
> could you send the patch for that? Is it upstream already?

It's indeed an IrDA bug (though mainly visible under rt-preempt) and the 
patch has been sent to the maintainer and to stable.

>> user.warn kernel: Neighbour table overflow.
>
> hm, i havent seen this one yet.

it comes from net/ipv4/route.c::rt_intern_hash()... That's all I can say 
so far. Any debug ideas? Could you try on any your rt-test system just a 
loop like

while true; do
 	ifdown eth0
 	ifup eth0
 	ping -c 1 host
done

or similar. Or, better yet with starting / stopping pppd?

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: [2.6.20-rt8] "Neighbour table overflow."

[Ingo Molnar]

* Guennadi Liakhovetski <gl@dsa-ac.de> wrote:

> > could you send the patch for that? Is it upstream already?
> 
> It's indeed an IrDA bug (though mainly visible under rt-preempt) and 
> the patch has been sent to the maintainer and to stable.

could you please bounce it to me too? I usually put such patches into 
-rt straight away and i drop them when they show up upstream.

	Ingo

Re: [irda-users] [2.6.20-rt8] "Neighbour table overflow."

[Samuel Ortiz]

Reply from Ingo, got mistakenly discarded by the irda-users mailing list:

* Guennadi Liakhovetski <gl@dsa-ac.de> wrote:

> > could you send the patch for that? Is it upstream already?
>
> It's indeed an IrDA bug (though mainly visible under rt-preempt) and
> the patch has been sent to the maintainer and to stable.

could you please bounce it to me too? I usually put such patches into
-rt straight away and i drop them when they show up upstream.

Ingo

Re: [irda-users] [2.6.20-rt8] "Neighbour table overflow."

[Samuel Ortiz]

Here you go, Ingo:

 Index: net/irda/irttp.c
===================================================================
RCS file: /home/dsa/cvs/linux-2_6/net/irda/irttp.c,v
retrieving revision 1.1.1.11
diff -u -r1.1.1.11 irttp.c
--- a/net/irda/irttp.c 5 Mar 2007 13:18:59 -0000 1.1.1.11
+++ b/net/irda/irttp.c 16 Mar 2007 11:04:31 -0000
@@ -1455,6 +1468,7 @@

/* Not everything should be copied */
new->notify.instance = instance;
+ spin_lock_init(&new->lock);
init_timer(&new->todo_timer);

skb_queue_head_init(&new->rx_queue);


On 3/16/2007, "Samuel Ortiz" <samuel@sortiz.org> wrote:

>
>Reply from Ingo, got mistakenly discarded by the irda-users mailing list:
>
>* Guennadi Liakhovetski <gl@dsa-ac.de> wrote:
>
>> > could you send the patch for that? Is it upstream already?
>>
>> It's indeed an IrDA bug (though mainly visible under rt-preempt) and
>> the patch has been sent to the maintainer and to stable.
>
>could you please bounce it to me too? I usually put such patches into
>-rt straight away and i drop them when they show up upstream.
>
>Ingo
>
>-------------------------------------------------------------------------
>Take Surveys. Earn Cash. Influence the Future of IT
>Join SourceForge.net's Techsay panel and you'll get the chance to share your
>opinions on IT & business topics through brief surveys-and earn cash
>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>_______________________________________________
>irda-users mailing list
>irda-users@lists.sourceforge.net
>http://lists.sourceforge.net/lists/listinfo/irda-users

Re: [2.6.20-rt8] "Neighbour table overflow."

[David Miller]

From: Guennadi Liakhovetski <gl@dsa-ac.de>
Date: Fri, 16 Mar 2007 17:26:57 +0100 (CET)

> it comes from net/ipv4/route.c::rt_intern_hash()... That's all I can say 
> so far. Any debug ideas? Could you try on any your rt-test system just a 
> loop like

There is an SKB leak somewhere, and that leaked SKB has references to
a route, that's what that message usually means.

After you accumulate enough leaked packets with routed attached them,
the system refuses to create any more of them.

Probably if you look at /proc/slabinfo you'll see a lot of allocated
sk_buff and rtinfo objects.

Re: [2.6.20-rt8] "Neighbour table overflow."

[Guennadi Liakhovetski]

On Fri, 16 Mar 2007, David Miller wrote:

> From: Guennadi Liakhovetski <gl-o/hVf8ie6tKzQB+pC5nmwQ@public.gmane.org>
> Date: Fri, 16 Mar 2007 17:26:57 +0100 (CET)
>
>> it comes from net/ipv4/route.c::rt_intern_hash()... That's all I can say
>> so far. Any debug ideas? Could you try on any your rt-test system just a
>> loop like
>
> There is an SKB leak somewhere, and that leaked SKB has references to
> a route, that's what that message usually means.

Do you have an idea how to debug this? I tried just replacing all 
(dev_)alloc_skb and (dev_)kfree_skb in net/irda and drivers/net/irda, and 
I get for each skb address more free than alloc calls...

> After you accumulate enough leaked packets with routed attached them,
> the system refuses to create any more of them.

What does this mean - packets with routing attached to them? Are these 
only some specific packets and the rest don't have this? can I detect 
them?

> Probably if you look at /proc/slabinfo you'll see a lot of allocated
> sk_buff and rtinfo objects.

skbuff_head_cache grows

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

2.6.20-rt8 APIC error

[joel silvestre]

Hello, 

found this for the first and only time this morning:


Mar 20 11:42:32 localhost powersaved: Loading acpi_cpufreq
Mar 20 11:42:32 localhost kernel:  [<c0104f7a>] dump_trace+0x63/0x1e5
Mar 20 11:42:32 localhost kernel:  [<c0105116>] show_trace_log_lvl
+0x1a/0x2f
Mar 20 11:42:32 localhost kernel:  [<c0105731>] show_trace+0x12/0x14
Mar 20 11:42:32 localhost kernel:  [<c01057c4>] dump_stack+0x16/0x18
Mar 20 11:42:32 localhost kernel:  [<c010ef6e>] smp_error_interrupt
+0x55/0x5c
Mar 20 11:42:32 localhost kernel:  [<c01049d0>] error_interrupt
+0x28/0x30
Mar 20 11:42:32 localhost kernel:  [<c01751cf>] shrink_dcache_parent
+0x3c/0xdc
Mar 20 11:42:32 localhost kernel:  [<c016b997>] dentry_unhash+0x1e/0x72
Mar 20 11:42:32 localhost kernel:  [<c016cb2e>] vfs_rmdir+0x79/0xf4
Mar 20 11:42:32 localhost kernel:  [<c016e634>] do_rmdir+0x89/0xbb
Mar 20 11:42:32 localhost kernel:  [<c016e6a5>] sys_rmdir+0x10/0x12
Mar 20 11:42:32 localhost kernel:  [<c0103f40>] syscall_call+0x7/0xb
Mar 20 11:42:32 localhost kernel:  [<b7f322ed>] 0xb7f322ed
Mar 20 11:42:32 localhost kernel:  =======================
Mar 20 11:42:32 localhost powersaved: enter 'acpi_cpufreq' into
CPUFREQD_MODULE in /etc/powersave/cpufreq.

joel

Re: [2.6.20-rt8] "Neighbour table overflow."

[Guennadi Liakhovetski]

On Mon, 19 Mar 2007, Guennadi Liakhovetski wrote:

> On Fri, 16 Mar 2007, David Miller wrote:
>>
>> There is an SKB leak somewhere, and that leaked SKB has references to
>> a route, that's what that message usually means.
>
> Do you have an idea how to debug this? I tried just replacing all 
> (dev_)alloc_skb and (dev_)kfree_skb in net/irda and drivers/net/irda, and I 
> get for each skb address more free than alloc calls...

I wrote a small hashing debugger - from __aloc_skb I store the new skb on 
a hash, from __kfree_skb I remove it from it. The first strange thing that 
occurred to me is that __kfree_skb is very often called on already freed 
skbs. Is it normal? It seems strange as trying to free memory that you no 
longer own can free a newly allocated object at that address.

As for the original problem, looks like the packets that get leaked are 
from af_irda.c::irda_sendmsg(). Any ideas where and when these skbs should 
be freed?

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [irda-users] [2.6.20-rt8] "Neighbour table overflow."

[Guennadi Liakhovetski]

On Tue, 20 Mar 2007, Guennadi Liakhovetski wrote:

> As for the original problem, looks like the packets that get leaked are
> from af_irda.c::irda_sendmsg(). Any ideas where and when these skbs should
> be freed?

No, I actually only see that they are created in sock_alloc_send_skb(), 
so, I assumed it was the call in af_irda.c::irda_sendmsg(), which, 
however, seems to be not the case now.

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany

Re: [irda-users] [2.6.20-rt8] "Neighbour table overflow."

[Samuel Ortiz]

On 3/20/2007, "Guennadi Liakhovetski" <gl@dsa-ac.de> wrote:

>On Tue, 20 Mar 2007, Guennadi Liakhovetski wrote:
>
>> As for the original problem, looks like the packets that get leaked are
>> from af_irda.c::irda_sendmsg(). Any ideas where and when these skbs should
>> be freed?
>
>No, I actually only see that they are created in sock_alloc_send_skb(),
>so, I assumed it was the call in af_irda.c::irda_sendmsg(), which,
>however, seems to be not the case now.
Well, if you're using IrNET, then those sock_alloc_send_skb() calls
should come from the IP code (net/ipv4/af_inet.c). Do you have a way to
verify that ?
The skb should eventually be freed by the IrDA hardware driver, when the
frame is actually transmitted. Are you using the pxaficp_ir driver ?

Cheers,
Samuel.

P.S.: I propose we remove the -rt people and Mingo from this discussion
until we know for sure that it's an RT bug. If no one objects, I'll
remove them from the CC list.


>Thanks
>Guennadi
>---------------------------------
>Guennadi Liakhovetski, Ph.D.
>DSA Daten- und Systemtechnik GmbH
>Pascalstr. 28
>D-52076 Aachen
>Germany
>
>-------------------------------------------------------------------------
>Take Surveys. Earn Cash. Influence the Future of IT
>Join SourceForge.net's Techsay panel and you'll get the chance to share your
>opinions on IT & business topics through brief surveys-and earn cash
>http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
>_______________________________________________
>irda-users mailing list
>irda-users@lists.sourceforge.net
>http://lists.sourceforge.net/lists/listinfo/irda-users

Re: [irda-users] [2.6.20-rt8] "Neighbour table overflow."

[Guennadi Liakhovetski]

On Tue, 20 Mar 2007, Samuel Ortiz wrote:

> Well, if you're using IrNET, then those sock_alloc_send_skb() calls
> should come from the IP code (net/ipv4/af_inet.c). Do you have a way to
> verify that ?

The test is running atm. If I've done everything right, I'll have the 
result tomorrow.

> The skb should eventually be freed by the IrDA hardware driver, when the
> frame is actually transmitted. Are you using the pxaficp_ir driver ?

Hardware driver?... hm, interesting. Yes, it's pxaficp_ir.

> P.S.: I propose we remove the -rt people and Mingo from this discussion
> until we know for sure that it's an RT bug. If no one objects, I'll
> remove them from the CC list.

No objections.

Thanks
Guennadi
---------------------------------
Guennadi Liakhovetski, Ph.D.
DSA Daten- und Systemtechnik GmbH
Pascalstr. 28
D-52076 Aachen
Germany