What does "Neighbour table overflow" message indicate?

What does "Neighbour table overflow" message indicate?

[Steve Snyder]

I just got this sequence of messages in my system log:

Jul 28 19:47:44 sunburn kernel: Neighbour table overflow.
Jul 28 19:47:44 sunburn last message repeated 9 times
Jul 28 19:47:49 sunburn kernel: NET: 53 messages suppressed.
Jul 28 19:47:49 sunburn kernel: Neighbour table overflow.
Jul 28 19:48:07 sunburn kernel: NET: 21 messages suppressed.
Jul 28 19:48:07 sunburn kernel: Neighbour table overflow.
Jul 28 19:48:09 sunburn last message repeated 3 times
Jul 28 19:48:14 sunburn kernel: NET: 4 messages suppressed.
Jul 28 19:48:14 sunburn kernel: Neighbour table overflow.

This is on a RedHat v7.1 + SMP kernel v2.4.7 system.  What is the kernel 
trying to tell me here?

Please cc me as I am not a subscriber to this list.

Thanks.

Re: What does "Neighbour table overflow" message indicate?

[Steve Snyder]

No, and no errors are shown for it either:

# ifconfig lo
lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:196907 errors:0 dropped:0 overruns:0 frame:0
          TX packets:196907 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0

All *seems* well.  Just that 30-second period of messages and then silence.

Thanks for the response.


On Saturday 28 July 2001 08:38 pm, you wrote:
> is lo down?
>
>   --cw
>
> On Sat, Jul 28, 2001 at 08:23:14PM -0500, Steve Snyder wrote:
>     I just got this sequence of messages in my system log:
>
>     Jul 28 19:47:44 sunburn kernel: Neighbour table overflow.
>     Jul 28 19:47:44 sunburn last message repeated 9 times
>     Jul 28 19:47:49 sunburn kernel: NET: 53 messages suppressed.
>     Jul 28 19:47:49 sunburn kernel: Neighbour table overflow.
>     Jul 28 19:48:07 sunburn kernel: NET: 21 messages suppressed.
>     Jul 28 19:48:07 sunburn kernel: Neighbour table overflow.
>     Jul 28 19:48:09 sunburn last message repeated 3 times
>     Jul 28 19:48:14 sunburn kernel: NET: 4 messages suppressed.
>     Jul 28 19:48:14 sunburn kernel: Neighbour table overflow.
>
>     This is on a RedHat v7.1 + SMP kernel v2.4.7 system.  What is the
> kernel trying to tell me here?
>
>     Please cc me as I am not a subscriber to this list.
>
>     Thanks.
>     -
>     To unsubscribe from this list: send the line "unsubscribe
> linux-kernel" in the body of a message to majordomo@vger.kernel.org
>     More majordomo info at  http://vger.kernel.org/majordomo-info.html
>     Please read the FAQ at  http://www.tux.org/lkml/

Re: What does "Neighbour table overflow" message indicate?

[Chris Wedgwood]

On Sat, Jul 28, 2001 at 08:53:48PM -0500, Steve Snyder wrote:

    No, and no errors are shown for it either:

    # ifconfig lo
    lo        Link encap:Local Loopback
              inet addr:127.0.0.1  Mask:255.0.0.0
              UP LOOPBACK RUNNING  MTU:16436  Metric:1
              RX packets:196907 errors:0 dropped:0 overruns:0 frame:0
              TX packets:196907 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0

    All *seems* well.  Just that 30-second period of messages and then
    silence.


What is the machine doing?  What kind of network is it attached to and
with how many hosts on it?



  --cw

Re: What does "Neighbour table overflow" message indicate?

[Steve Snyder]

On Saturday 28 July 2001 08:57 pm, Chris Wedgwood wrote:
> On Sat, Jul 28, 2001 at 08:53:48PM -0500, Steve Snyder wrote:
>
>     No, and no errors are shown for it either:
>
>     # ifconfig lo
>     lo        Link encap:Local Loopback
>               inet addr:127.0.0.1  Mask:255.0.0.0
>               UP LOOPBACK RUNNING  MTU:16436  Metric:1
>               RX packets:196907 errors:0 dropped:0 overruns:0 frame:0
>               TX packets:196907 errors:0 dropped:0 overruns:0 carrier:0
>               collisions:0 txqueuelen:0
>
>     All *seems* well.  Just that 30-second period of messages and then
>     silence.
>
>
> What is the machine doing?  What kind of network is it attached to and
> with how many hosts on it?

It is a server for a small LAN.  Interfaces: eth0=LAN, eth1=cable modem.  I 
believe that I was playing Quake3 (multi-player across internet) on one of 
the LAN's client machines when the message were logged.  No corresponding 
messages are seen in the client's (another RHL v7.1 box) system log, but 
then, it's not running iptables.

Further snooping shows the error msg text in file inux/net/ipv4/route.c:

    if (net_ratelimit())
        printk("Neighbour table overflow.\n");

The reference to "net_ratelimit" make me wonder if it is related to 
iptables.  I am using iptable, and have since kernel 2.4.1, but I've seen 
these messages before.  Hmmm.

Re: What does "Neighbour table overflow" message indicate?

[Riley Williams]

Hi Steve.

 > I just got this sequence of messages in my system log:
 >
 > Jul 28 19:47:44 sunburn kernel: Neighbour table overflow.
 > Jul 28 19:47:44 sunburn last message repeated 9 times
 > Jul 28 19:47:49 sunburn kernel: NET: 53 messages suppressed.
 > Jul 28 19:47:49 sunburn kernel: Neighbour table overflow.
 > Jul 28 19:48:07 sunburn kernel: NET: 21 messages suppressed.
 > Jul 28 19:48:07 sunburn kernel: Neighbour table overflow.
 > Jul 28 19:48:09 sunburn last message repeated 3 times
 > Jul 28 19:48:14 sunburn kernel: NET: 4 messages suppressed.
 > Jul 28 19:48:14 sunburn kernel: Neighbour table overflow.
 >
 > This is on a RedHat v7.1 + SMP kernel v2.4.7 system.  What is
 > the kernel trying to tell me here?
 >
 > Please cc me as I am not a subscriber to this list.

This could be on completely the wrong track, but here's one of the
entries from the 2.4.5 kernel's Configure.help file (I don't yet have
2.4.7 on my system):

 Q> ARP daemon support (EXPERIMENTAL)
 Q> CONFIG_ARPD
 Q>   Normally, the kernel maintains an internal cache which maps IP
 Q>   addresses to hardware addresses on the local network, so that
 Q>   Ethernet/Token Ring/ etc. frames are sent to the proper address
 Q>   on the physical networking layer. For small networks having a
 Q>   few hundred directly connected hosts or less, keeping this
 Q>   address resolution (ARP) cache inside the kernel works well.
 Q>
 Q>   However, maintaining an internal ARP cache does not work well
 Q>   for very large switched networks, and will use a lot of kernel
 Q>   memory if TCP/IP connections are made to many machines on the
 Q>   network.
 Q>
 Q>   If you say Y here, the kernel's internal ARP cache will never
 Q>   grow to more than 256 entries (the oldest entries are expired
 Q>   in a LIFO manner) and communication will be attempted with the
 Q>   user space ARP daemon arpd. Arpd then answers the address
 Q>   resolution request either from its own cache or by asking the
 Q>   net.
 Q>
 Q>   This code is experimental and also obsolete. If you want to
 Q>   use it, you need to find a version of the daemon arpd on the
 Q>   net somewhere, and you should also say Y to "Kernel/User
 Q>   network link driver", below. If unsure, say N.

The text in there looks suspiciously related to your problem to me.

Best wishes from Riley.

Re: What does "Neighbour table overflow" message indicate?

[Eric W. Biederman]

> Further snooping shows the error msg text in file inux/net/ipv4/route.c:
> 
>     if (net_ratelimit())
>         printk("Neighbour table overflow.\n");

> 
> The reference to "net_ratelimit" make me wonder if it is related to 
> iptables.  I am using iptable, and have since kernel 2.4.1, but I've seen 
> these messages before.  Hmmm.

My experience with this is the message occurs when you a machine starts
arping for a non-existent ip address.  I suspect net_ratelimit triggers
when there are too many arps.

Run tcpdump -n -i eth0 (assuming your network is on eth0) and see if you
see an arp request that never gets answered.

Eric

Re: What does "Neighbour table overflow" message indicate?

[Kurt Roeckx]

On Sat, Jul 28, 2001 at 09:15:11PM -0500, Steve Snyder wrote:
> 
> Further snooping shows the error msg text in file inux/net/ipv4/route.c:
> 
>     if (net_ratelimit())
>         printk("Neighbour table overflow.\n");
> 
> The reference to "net_ratelimit" make me wonder if it is related to 
> iptables.  I am using iptable, and have since kernel 2.4.1, but I've seen 
> these messages before.  Hmmm.

net_ratelimit() is there to only log something every 5 seconds,
so your logs don't get flooded.  It should be used for every
printk that has to do with net.

See core/utils.c


Kurt

Re: What does "Neighbour table overflow" message indicate?

[jeff millar]

We used to get this from an embedded PowerPC processor under 2.2.x when the
hardware to device driver interface got screwed up.

jeff

----- Original Message -----
From: "Riley Williams" <rhw@MemAlpha.CX>
To: "Steve Snyder" <swsnyder@home.com>
Cc: "Linux Kernel" <linux-kernel@vger.kernel.org>
Sent: Sunday, July 29, 2001 1:41 AM
Subject: Re: What does "Neighbour table overflow" message indicate?


> Hi Steve.
>
>  > I just got this sequence of messages in my system log:
>  >
>  > Jul 28 19:47:44 sunburn kernel: Neighbour table overflow.
>  > Jul 28 19:47:44 sunburn last message repeated 9 times
>  > Jul 28 19:47:49 sunburn kernel: NET: 53 messages suppressed.
>  > Jul 28 19:47:49 sunburn kernel: Neighbour table overflow.
>  > Jul 28 19:48:07 sunburn kernel: NET: 21 messages suppressed.
>  > Jul 28 19:48:07 sunburn kernel: Neighbour table overflow.
>  > Jul 28 19:48:09 sunburn last message repeated 3 times
>  > Jul 28 19:48:14 sunburn kernel: NET: 4 messages suppressed.
>  > Jul 28 19:48:14 sunburn kernel: Neighbour table overflow.
>  >
>  > This is on a RedHat v7.1 + SMP kernel v2.4.7 system.  What is
>  > the kernel trying to tell me here?
>  >
>  > Please cc me as I am not a subscriber to this list.
>
> This could be on completely the wrong track, but here's one of the
> entries from the 2.4.5 kernel's Configure.help file (I don't yet have
> 2.4.7 on my system):
>
>  Q> ARP daemon support (EXPERIMENTAL)
>  Q> CONFIG_ARPD
>  Q>   Normally, the kernel maintains an internal cache which maps IP
>  Q>   addresses to hardware addresses on the local network, so that
>  Q>   Ethernet/Token Ring/ etc. frames are sent to the proper address
>  Q>   on the physical networking layer. For small networks having a
>  Q>   few hundred directly connected hosts or less, keeping this
>  Q>   address resolution (ARP) cache inside the kernel works well.
>  Q>
>  Q>   However, maintaining an internal ARP cache does not work well
>  Q>   for very large switched networks, and will use a lot of kernel
>  Q>   memory if TCP/IP connections are made to many machines on the
>  Q>   network.
>  Q>
>  Q>   If you say Y here, the kernel's internal ARP cache will never
>  Q>   grow to more than 256 entries (the oldest entries are expired
>  Q>   in a LIFO manner) and communication will be attempted with the
>  Q>   user space ARP daemon arpd. Arpd then answers the address
>  Q>   resolution request either from its own cache or by asking the
>  Q>   net.
>  Q>
>  Q>   This code is experimental and also obsolete. If you want to
>  Q>   use it, you need to find a version of the daemon arpd on the
>  Q>   net somewhere, and you should also say Y to "Kernel/User
>  Q>   network link driver", below. If unsure, say N.
>
> The text in there looks suspiciously related to your problem to me.
>
> Best wishes from Riley.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: What does "Neighbour table overflow" message indicate?

[Bernd Eckenfels]

In article <01072821151103.01125@mercury.snydernet.lan> you wrote:
>    if (net_ratelimit())
>        printk("Neighbour table overflow.\n");

> The reference to "net_ratelimit" make me wonder if it is related to 
> iptables.  I am using iptable, and have since kernel 2.4.1, but I've seen 
> these messages before.  Hmmm.

Net ratelimit is used to limit the rate of messages or actions done by the
network module. In this case it only ensures, that the printk message is not
printed too often. The actual condition why the message is printed is above
this if.

Greetings
Bernd

Re: What does "Neighbour table overflow" message indicate?

[Carlos O'Donell Jr.]

> network module. In this case it only ensures, that the printk message is not
> printed too often. The actual condition why the message is printed is above
> this if.
> 
> Greetings
> Bernd
> -

Snyder,

Just by looking at your email, @home, I can guess that your
cable modem is connected to an HFC Cable network segment.

In general these segments are extremely large and due to the
nature of the users, can cause large amounts of arp broadcast
traffic during peak times.

The message you are seeing is directly related to your arp cache
overflowing.

I've seen this message during high traffic hours on my 2.2.x 
firewall.

Things to check:

- Is your netmask set correctly?
- Check to see how many hosts are on your segment?

======================================================
Why the kernel spat what it spat : blow by blow
======================================================

N.B. Using 2.4.7 Kernel Source.

I think the critical point is:

In route.c:

639: int err = arp_bind_neighbour(&rt->u.dst);
640:                if (err) {
...			[snip]

Which means that if the binding of an arp neighbour fails, then
we trod down the path closer towards that printk, that has
caused us so much distress.

In arp.c, we look for "arp_bind_neighbour" and find it on line 429:

Right off the bat, we hope that:

434:        if (dev == NULL)
435:                return -EINVAL;

Isn't the case :)

Unless, it's alredy bound, then the next line is the case...

436:        if (n == NULL) {

And the only return that is non-zero is from:

440:                n = __neigh_lookup_errno(
441:#ifdef CONFIG_ATM_CLIP
442:                    dev->type == ARPHRD_ATM ? &clip_tbl :
443:#endif
444:                    &arp_tbl, &nexthop, dev);
445:                if (IS_ERR(n)) 
446:                        return PTR_ERR(n);

So __neigh_lookup_errno is the culprit...

In ./include/net/neighbour.h we have the function defined:

266:static inline struct neighbour *
267:__neigh_lookup_errno(struct neigh_table *tbl, const void *pkey,
268:struct net_device *dev)
...
275:       return neigh_create(tbl, pkey, dev)

Is the interesting point.. since our table is overflowing, we
need to find the point where the entry is created :)

Off we go to line 288 in ./net/core/neighbour.c:
(I love to trace source!)

296:        n = neigh_alloc(tbl);
297:        if (n == NULL)
298:                return ERR_PTR(-ENOBUFS);

Hrm... -ENOBUFS :)

In neigh_alloc, same file:

235:         if (tbl->entries > tbl->gc_thresh3 ||
236:            (tbl->entries > tbl->gc_thresh2 &&
237:             now - tbl->last_flush > 5*HZ)) {
238:                if (neigh_forced_gc(tbl) == 0 &&
239:                    tbl->entries > tbl->gc_thresh3)
240:                        return NULL;
241:        }

Which leads us to note that if the cache is growing faster than
the garbage collecting (ref counting code) is being done, and we
begin to exceed our allocations, we will trigger a table 
overflow.

Can you make the tables bigger?
What type of inpact does this have?
Should we be asking @Home to make segments smaller? 
(Probably not possible)

In ./net/ipv4/arp.c you could change the GC collection parameters...
I'm not sure how they were tuned?

Line 187:
        gc_interval:    30 * HZ,
        gc_thresh1:     128,
        gc_thresh2:     512,
        gc_thresh3:     1024,

Hrm... just pondering.


=================================================================

Cheers,
Carlos O'Donell Jr.

Re: What does "Neighbour table overflow" message indicate?

[Rob Landley]

On Monday 30 July 2001 08:38, Carlos O'Donell Jr. wrote:
> > network module. In this case it only ensures, that the printk message is
> > not printed too often. The actual condition why the message is printed is
> > above this if.
> >
> > Greetings
> > Bernd
> > -
>
> Snyder,
>
> Just by looking at your email, @home, I can guess that your
> cable modem is connected to an HFC Cable network segment.

Random datapoint: it does this on Road Runner cable modems in Austin, too 
(2.2.17 or so did, anyway.  Haven't had a monitor physically hooked up to my 
old 486 gateway since I moved it to 2.4).  Basically any cable modem that 
acts like a hub instead of a router (letting all packets through instead of 
just the one for the mac address it's connected to).

I made it go away somehow (~6 months back).  Possibly deselecting all the ARP 
stuff when I recompiled, I honestly don't remember at this point.  2.4 never 
did it (that I saw), but I compiled that sucker myself and don't log into my 
gateway much (it's a dumb ip masquerader running no daemons) so...

> In general these segments are extremely large and due to the
> nature of the users, can cause large amounts of arp broadcast
> traffic during peak times.

Forget peak times.  It did this 24/7 for me.

Bloated the logs something fierce.  I vaguely remember looking in the source 
and finding the message and going "so why doesn't it just delete something 
out of the neighbor table then?"  Never did figure that one out.  (It 
happened when the machine wasn't actually in USE, just connected to the net 
but otherwise idle.  No connections masquerading through it, nothing...)

Rob