Re: [LARTC] neighbor table overflow

From: "Marco C. Coelho" <maillist1@argontech.net>
To: lartc@vger.kernel.org
Subject: Re: [LARTC] neighbor table overflow
Date: Mon, 19 Nov 2007 22:36:31 +0000	[thread overview]
Message-ID: <47420FEF.9070303@argontech.net> (raw)
In-Reply-To: <200710230146.27081.peet@altlinux.org>

[-- Attachment #1.1: Type: text/plain, Size: 10951 bytes --]

Still beating the same bush!

I've done all the possible suggestions so far.  I still was getting a 
neighbor table overflow.
Looking at the MAN 7 ARP pages, I see:

       gc_thresh1
              The minimum number of entries to keep in the ARP cache.  
The garbage collector will not run if there are
              fewer than this number of entries in the cache.  Defaults 
to 128.

       gc_thresh2
              The soft maximum number of entries to keep in the ARP 
cache.  The garbage collector will allow the  num-
              ber of entries to exceed this for 5 seconds before 
collection will be performed.  Defaults to 512.

       gc_thresh3
              The  hard  maximum number of entries to keep in the ARP 
cache.  The garbage collector will always run if
              there are more than this number of entries in the cache.  
Defaults to 1024.

Since this box never gets less than 500 pppoe connections, this Sat I 
changed
                          WAS     NOW  
gc_thresh1      512         1024
gc_thresh2     2048        2048
gc_thresh3     4096        4096

what's strange is when I do an 'arp -an' I only get three entries back. 
(ips changed to protect the guilty).  Shouldn't this show the arp entries

? (x.202.x.3) at 00:03:47:2D:8B:F9 [ether] on eth0
? (x.202.x.1) at 00:03:E3:88:EC:C2 [ether] on eth0
? (x.202.x.2) at 00:18:8B:76:EC:D8 [ether] on eth0
? (x.202.x.9) at 00:90:27:43:C2:CF [ether] on eth0

ip route | grep link provides:

snip (lots of pppoe connects)
x.202.x.237 dev ppp53  proto kernel  scope link  src 10.20.1.1
x.202.x.235 dev ppp339  proto kernel  scope link  src 10.20.1.1
x.202.x.232 dev ppp185  proto kernel  scope link  src 10.20.1.1
x.202.x.231 dev ppp313  proto kernel  scope link  src 10.20.1.1
x.202.x.230 dev ppp67  proto kernel  scope link  src 10.20.1.1
x.202.x.226 dev ppp74  proto kernel  scope link  src 10.20.1.1
x.202.x.224 dev ppp150  proto kernel  scope link  src 10.20.1.1
x.202.x.0/24 dev eth0  proto kernel  scope link  src x.202.224.8
192.168.1.0/24 dev eth3  proto kernel  scope link  src 192.168.1.8

I don't think we are doing anything too special with this box that we 
would see a kernel issue no one else is seeing.  Can arp poisoning cause 
this?

a dmesg after a clean reboot only gives:

Shorewall:all2all:REJECT:IN=ppp413 OUT= MAC= SRC=x.202.x.165 
DST=10.20.1.1 LEN=60 TOS=0x00 PREC=0x00 TTL=254 ID=39752 PROTO=ICMP 
TYPE=8 CODE=0 ID=25040 SEQ=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=72 TOS=0x00 PREC=0x00 TTL=126 ID=48363 PROTO=UDP 
SPT=427 DPT=427 LEN=52
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48492 DF PROTO=TCP 
SPT=36005 DPT=9220 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48493 DF PROTO=TCP 
SPT=36005 DPT=9220 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48517 DF PROTO=TCP 
SPT=36005 DPT=9220 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48518 DF PROTO=TCP 
SPT=33969 DPT=16398 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=72 TOS=0x00 PREC=0x00 TTL=126 ID=48519 PROTO=UDP 
SPT=427 DPT=427 LEN=52
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48522 DF PROTO=TCP 
SPT=33969 DPT=16398 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48526 DF PROTO=TCP 
SPT=33969 DPT=16398 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48614 DF PROTO=TCP 
SPT=35790 DPT=9220 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48630 DF PROTO=TCP 
SPT=35790 DPT=9220 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48x6 DF PROTO=TCP 
SPT=35790 DPT=9220 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48x8 DF PROTO=TCP 
SPT=34718 DPT=16398 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48663 DF PROTO=TCP 
SPT=34718 DPT=16398 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.202.x.110 
DST=192.168.1.7 LEN=48 TOS=0x00 PREC=0x00 TTL=126 ID=48679 DF PROTO=TCP 
SPT=34718 DPT=16398 WINDOW=16384 RES=0x00 SYN URGP=0
Shorewall:all2all:REJECT:IN=ppp160 OUT=eth3 SRC=x.y.x.110 
DST=192.168.1.7 LEN=72 TOS=0x00 PREC=0x00 TTL=126 ID=48724 PROTO=UDP 
SPT=427 DPT=427 LEN=52

Kernel Version 2.6.18-8.1.6

Looking for any suggestions.

Marco

Andrei Kovacs wrote:
> On 10/25/07, Marco C. Coelho <maillist1@argontech.net> wrote:
>   
>>  Looking into it further an ip route shows:
>>
>>  x.0.0.0/8 via x.y.224.1 dev eth0  proto zebra  metric 20 equalize
>>
>>  So the x.0.0.0 announce is coming into this box through OSPF  (zebra)
>>
>>  The 169.254.0.0/16 is being automajically added through the sysconfig
>> network scripts.  I'm looking into why.
>>
>>     
>
> Add "NOZEROCONF=yes" in /etc/sysconfig/network and the 169.254.0.0/16
> network won't be created anymore.
>
>   
>>  In either case I still don't see why these entries would make the neighbor
>> table overflow.  Could it have been the previous fix to the hosts file?
>>
>>  mc
>>
>>  Alexandru Dragoi wrote:
>>  Marco C. Coelho wrote:
>>
>>
>>  the ip route with a grep for link returns:
>>
>> snip** too long
>> x.y.x.198 dev ppp436 proto kernel scope link src 10.20.1.1
>> x.y.x.196 dev ppp421 proto kernel scope link src 10.20.1.1
>> x.y.x.197 dev ppp211 proto kernel scope link src 10.20.0.1
>> x.y.x.194 dev ppp13 proto kernel scope link src 10.20.1.1
>> x.y.x.192 dev ppp404 proto kernel scope link src 10.20.1.1
>> x.y.x.254 dev ppp194 proto kernel scope link src 10.20.1.1
>> x.y.x.253 dev ppp130 proto kernel scope link src 10.20.1.1
>> x.y.x.252 dev ppp243 proto kernel scope link src 10.20.1.1
>> x.y.x.249 dev ppp195 proto kernel scope link src 10.20.1.1
>> x.y.x.248 dev ppp254 proto kernel scope link src 10.20.1.1
>> x.y.x.247 dev ppp235 proto kernel scope link src 10.20.1.1
>> x.y.x.242 dev ppp78 proto kernel scope link src 10.20.1.1
>> x.y.x.240 dev ppp328 proto kernel scope link src 10.20.1.1
>> x.y.x.237 dev ppp44 proto kernel scope link src 10.20.1.1
>> x.y.x.236 dev ppp122 proto kernel scope link src 10.20.1.1
>> x.y.x.234 dev ppp316 proto kernel scope link src 10.20.1.1
>> x.y.x.232 dev ppp132 proto kernel scope link src 10.20.1.1
>> x.y.x.231 dev ppp104 proto kernel scope link src 10.20.0.1
>> x.y.x.226 dev ppp179 proto kernel scope link src 10.20.0.1
>> x.y.224.0/24 dev eth0 proto kernel scope link src x.y.224.8
>> 192.168.1.0/24 dev eth3 proto kernel scope link src 192.168.1.8
>> 169.254.0.0/16 dev eth3 scope link
>>
>>  The one above must be deleted, many redhat-like distros attach
>> 169.254.0.0/16.
>>
>>
>>  All the pppoe terminations (pppd) are shown, as well as the last three
>> subnets. I'll have to see where the 169.254.0.0/16 is coming from?
>>
>> mc
>>
>>
>>
>>
>> Alexandru Dragoi wrote:
>>
>>
>>  Marco C. Coelho wrote:
>>
>>
>>
>>  This box is doing a lot. It terminates 1000 PPPoE connections,
>> provides traffic shaping using TC/HTB, authenticates all users via
>> Radius. It also runs OSPF routing for the internal network. Looking
>> at a simple route output I see all the PPP connections coming through
>> the box, and due to the OSPF I also see the rest of my network
>> announcements. The only strange things are:
>>
>> 1. The last man working on this box had mistakenly edited the hosts
>> file and added the machine name and complete domain name to the local
>> host 127.0.0.1 name. It should only be pointed to the eth0
>> interface. I have changed this.
>>
>> 2. The route output is making an announcement
>>
>>  x.0.0.0 argontech.net 255.0.0.0 UG 20
>> 0 0 eth0
>>
>>
>>  This doesn't look dangerous for your problem, I was only talking about
>> directly connected networks:
>>
>> # ip route |grep link
>>
>>
>>
>>
>>  My public IP space is a /20 within that space, not the whole Class A.
>> I have not found which box is announcing this within my network yet.
>>
>>
>>
>>
>>
>> Jeff Welling wrote:
>>
>>
>>
>>
>>  On 10/23/07 06:56, Alexandru Dragoi wrote:
>>
>>
>>
>>  What about checking your routing table? you may have link routes
>> for massive subnets (like 85.0.0.0/8 or 140.20.0.0/16). Some
>> programs prefer to use "standard" netmask of classes A and B.
>>
>>
>>  I'm betting that the OP has other things going on seeing has how
>> s/he mentioned PPPoE, which to my knowledge is a layer 2 protocol,
>> and thus not subject to typical routing scenarios. In essence the
>> OP could have thousands of PPPoE connections terminating on one
>> system with the ARP cache having to deal with where to send traffic
>> to which MAC address. There is not a lot of room for routing in such
>> a scenario.
>>
>>
>>
>>  I agree with Peter's suggestion, arpd. I ran into the neighbor table
>> overflow problem recently, at the hands of our ISP. I was in the
>> process of recompiling the kernel and mucking with arpd (I couldn't
>> get it to run/start properly) when the problem disappeared as quickly
>> as it showed up. Lucky for me, this was some kind of ISP problem, I
>> was able to determine that much through `tcpdump -i X -n arpd`.
>>
>> My 'two cents' is that you try arpd, I did a bit of looking when I
>> came across that problem and it seemed to be the last ditch effort
>> when changing the gc threshold had no effect. Wasn't able to confirm
>> that it worked for sure though.
>>
>> Cheers.
>> _______________________________________________
>> LARTC mailing list
>> LARTC@mailman.ds9a.nl
>> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>>
>>
>>
>>  _______________________________________________
>> LARTC mailing list
>> LARTC@mailman.ds9a.nl
>> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>>
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> LARTC mailing list
>> LARTC@mailman.ds9a.nl
>> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>>
>>
>>
>>
>>
>> _______________________________________________
>> LARTC mailing list
>> LARTC@mailman.ds9a.nl
>> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>>
>>
>>     
>
>   

[-- Attachment #1.2: Type: text/html, Size: 13040 bytes --]

[-- Attachment #2: Type: text/plain, Size: 143 bytes --]

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc