sendto: No buffer space available

sendto: No buffer space available

[andre.correa]

Hi list,

I  have  a  Linux  2.4.19  box  doing NAT, PPPoE, Traffic Shapping and
Firewalling.  It  is  a  2xPIII  733MHz with 512Mb RAM. Everything was
working  just  fine  until 5 or 6 days ago we started having some strange
behavior.

Under  moderate  traffic,  15  to  20  NAT users, we find that traffic
suddenly  stops  for 10 or 15 seconds and then comes back. During this
periods  I've  figured  out  that  if I ping my interfaces or Internet
addresses I get:

sendto: No buffer space available
ping: sent 64 octets to xxx.xxx.xxx.xxx, ret=-1

I've  made  lots  of  searchs  in  mailling lists, Internet and in the
kernel  source  but couldn't work on it.

Can you guys help me to solve this problem?

tks in advance for your help and attention.

Andre
andre.correa@pobox.com

Re: sendto: No buffer space available

[Bob Keyes]

What does netstat -n show you?
I have seen this error when there are thousands of TCP connections open at
the same time (this was using the 'naptha' DoS demonstration tool I
wrote).

On Mon, 2 Dec 2002 andre.correa@pobox.com wrote:

>
> Hi list,
>
> I  have  a  Linux  2.4.19  box  doing NAT, PPPoE, Traffic Shapping and
> Firewalling.  It  is  a  2xPIII  733MHz with 512Mb RAM. Everything was
> working  just  fine  until 5 or 6 days ago we started having some strange
> behavior.
>
> Under  moderate  traffic,  15  to  20  NAT users, we find that traffic
> suddenly  stops  for 10 or 15 seconds and then comes back. During this
> periods  I've  figured  out  that  if I ping my interfaces or Internet
> addresses I get:
>
> sendto: No buffer space available
> ping: sent 64 octets to xxx.xxx.xxx.xxx, ret=-1
>
> I've  made  lots  of  searchs  in  mailling lists, Internet and in the
> kernel  source  but couldn't work on it.
>
> Can you guys help me to solve this problem?
>
> tks in advance for your help and attention.
>
> Andre
> andre.correa@pobox.com
>
>
>

Re[2]: sendto: No buffer space available

[andre.correa]

Hi list and Bob, my netstat -n shows nothing unusual:

root@mybox:~# netstat -n
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0     20 xxx.xxx.xxx.xx:22       yy.yy.yy.yy:1105        ESTABLISHED 
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags       Type       State         I-Node Path
unix  24     [ ]         DGRAM                    57     /dev/log
unix  2      [ ]         DGRAM                    7051   
unix  2      [ ]         DGRAM                    4997   
unix  2      [ ]         DGRAM                    4928   
unix  2      [ ]         DGRAM                    2820   
unix  2      [ ]         DGRAM                    2730   
unix  2      [ ]         DGRAM                    2690   
unix  2      [ ]         DGRAM                    2650   
unix  2      [ ]         DGRAM                    2550   
unix  2      [ ]         DGRAM                    2500   
unix  2      [ ]         DGRAM                    2460   
unix  2      [ ]         DGRAM                    2330   
unix  2      [ ]         DGRAM                    2240   
unix  2      [ ]         DGRAM                    2200   
unix  2      [ ]         DGRAM                    2134   
unix  2      [ ]         DGRAM                    2081   
unix  2      [ ]         DGRAM                    2041   
unix  2      [ ]         DGRAM                    2001   
unix  2      [ ]         DGRAM                    1921   
unix  2      [ ]         DGRAM                    1909   
unix  2      [ ]         DGRAM                    1765   
unix  2      [ ]         DGRAM                    1757   
unix  2      [ ]         DGRAM                    60     

that is it...

tks

Andre



On 02/12/02, Bob Keyes wrote:
BK> What does netstat -n show you?
BK> I have seen this error when there are thousands of TCP connections open at
BK> the same time (this was using the 'naptha' DoS demonstration tool I
BK> wrote).

BK> On Mon, 2 Dec 2002 andre.correa@pobox.com wrote:

>>
>> Hi list,
>>
>> I  have  a  Linux  2.4.19  box  doing NAT, PPPoE, Traffic Shapping and
>> Firewalling.  It  is  a  2xPIII  733MHz with 512Mb RAM. Everything was
>> working  just  fine  until 5 or 6 days ago we started having some strange
>> behavior.
>>
>> Under  moderate  traffic,  15  to  20  NAT users, we find that traffic
>> suddenly  stops  for 10 or 15 seconds and then comes back. During this
>> periods  I've  figured  out  that  if I ping my interfaces or Internet
>> addresses I get:
>>
>> sendto: No buffer space available
>> ping: sent 64 octets to xxx.xxx.xxx.xxx, ret=-1
>>
>> I've  made  lots  of  searchs  in  mailling lists, Internet and in the
>> kernel  source  but couldn't work on it.
>>
>> Can you guys help me to solve this problem?
>>
>> tks in advance for your help and attention.
>>
>> Andre
>> andre.correa@pobox.com
>>
>>
>>



Andre Correa
andre.docena@pobox.com

Too many ARP entries and Re: sendto: No buffer space available

[andre.correa]

Hi, I am writting to answer myself and send a new question to the list.
I've had problems in my NAT/PPPoE box, with traffic stopping suddenly
and then coming back in a few seconds and I've found that my neighbour
table was getting full. When it is full, no new ARP entries can be
created and no new traffic can happen. Now I encreased this values:

echo 512 > /proc/sys/net/ipv4/neigh/default/gc_thresh1
echo 2048 > /proc/sys/net/ipv4/neigh/default/gc_thresh2
echo 4096 > /proc/sys/net/ipv4/neigh/default/gc_thresh3

But  there  is  still a question for me. Looking at my arp table, I
see that there are =~ 150 entries, seconds passing and more entries
coming, 20 seconds after I can have =~1100, it goes on until it reachs
=~2200  entries,  then it goes back to the =~100 and starts over again.

I  have  less  then  50  NAT users. Is it normal to have some many ARP
entries with this variation? Looking the ARP table I see my "Internet"
interface with lots of entries, with internet host IP addresses and my
gateway's NIC MAC address.

Isn't ARP supposed to keep entries just to local network systems?

Is it all normal? And if so, how big can gc_threash[1,2,3] be?

tks in advance.

Andre
andre.correa@pobox.com



On 02/12/02, andre.correa@pobox.com wrote:

acpc> Hi list,

acpc> I  have  a  Linux  2.4.19  box  doing NAT, PPPoE, Traffic Shapping and
acpc> Firewalling.  It  is  a  2xPIII  733MHz with 512Mb RAM. Everything was
acpc> working  just  fine  until 5 or 6 days ago we started having some strange
acpc> behavior.

acpc> Under  moderate  traffic,  15  to  20  NAT users, we find that traffic
acpc> suddenly  stops  for 10 or 15 seconds and then comes back. During this
acpc> periods  I've  figured  out  that  if I ping my interfaces or Internet
acpc> addresses I get:

acpc> sendto: No buffer space available
acpc> ping: sent 64 octets to xxx.xxx.xxx.xxx, ret=-1

acpc> I've  made  lots  of  searchs  in  mailling lists, Internet and in the
acpc> kernel  source  but couldn't work on it.

acpc> Can you guys help me to solve this problem?

acpc> tks in advance for your help and attention.

acpc> Andre
acpc> andre.correa@pobox.com




Andre Correa
andre.docena@pobox.com

Re: Too many ARP entries and Re: sendto: No buffer space available

[Cedric Blancher]

Le lun 02/12/2002 à 21:28, andre.correa@pobox.com a écrit :
> But  there  is  still a question for me. Looking at my arp table, I
> see that there are =~ 150 entries, seconds passing and more entries
> coming, 20 seconds after I can have =~1100, it goes on until it reachs
> =~2200  entries,  then it goes back to the =~100 and starts over again.

Wierd...

> I  have  less  then  50  NAT users. Is it normal to have some many ARP
> entries with this variation? Looking the ARP table I see my "Internet"
> interface with lots of entries, with internet host IP addresses and my
> gateway's NIC MAC address.
> Isn't ARP supposed to keep entries just to local network systems?

Yes it is.
ARP is supposed to keep track of IP/MAC associations for network
directly routed to interface, i.e. directly connected, aka local LANs.

> Is it all normal? And if so, how big can gc_threash[1,2,3] be?

It is not normal. You should monitor ARP traffic on your network using
arpwatch (see Freshmeat, available as .deb, .rpm too) to see if someone
would be playing ARP cache poisoning (see http://www.arp-sk.org/).
-- 
Cédric Blancher  <blancher@cartel-securite.fr>
IT systems and networks security expert  - Cartel Sécurité
Phone : +33 (0)1 44 06 97 87 - Fax: +33 (0)1 44 06 97 99
PGP KeyID:157E98EE  FingerPrint:FA62226DA9E72FA8AECAA240008B480E157E98EE

Re: Too many ARP entries and Re: sendto: No buffer space available

[Nick Drage]

On Tue, Dec 03, 2002 at 02:08:54PM +0100, Cedric Blancher wrote:
> Le lun 02/12/2002 à 21:28, andre.correa@pobox.com a écrit :
> > But  there  is  still a question for me. Looking at my arp table, I
> > see that there are =~ 150 entries, seconds passing and more entries
> > coming, 20 seconds after I can have =~1100, it goes on until it reachs
> > =~2200  entries,  then it goes back to the =~100 and starts over again.
> 
> Wierd...

Weird, certainly... haven't seen anything like this before.

<snip>

> It is not normal. You should monitor ARP traffic on your network using
> arpwatch (see Freshmeat, available as .deb, .rpm too) to see if someone
> would be playing ARP cache poisoning (see http://www.arp-sk.org/).

I haven't looked at arpwatch recetly, but presumably that will just scream
blue bloody murder.

What does

tcpdump -npevvvi <<interface>> arp 

look like?

The original paragraph of:

> > I  have  less  then  50  NAT users. Is it normal to have some many ARP
> > entries with this variation? Looking the ARP table I see my "Internet"
> > interface with lots of entries, with internet host IP addresses and my
> > gateway's NIC MAC address.

Isn't quite as clear as required.  Andre, any chance you could cut and paste
a few examples, so we can try to understand the symptoms a bit better?

-- 
FunkyJesus System Administration Team

Re[2]: Too many ARP entries and Re: sendto: No buffer space available

[andre.correa]

Hi again, looking at TCPDump I see this wierd traffic:

root@linuxbox:~# tcpdump -i eth1 | grep arp
tcpdump: listening on eth1
Dec  3 11:16:52 linuxbox kernel: device eth1 entered promiscuous mode
11:17:03.059629 arp who-has 64.12.163.212 tell linuxbox
11:17:03.060569 arp reply 64.12.163.212 is-at 0:2:b9:1d:db:41
11:17:07.669629 arp who-has 172.18.1.218 tell linuxbox
11:17:07.670610 arp reply 172.18.1.218 is-at 0:2:b9:1d:db:41
11:17:07.839630 arp who-has 64.12.27.135 tell linuxbox
11:17:07.840544 arp reply 64.12.27.135 is-at 0:2:b9:1d:db:41
11:17:07.850840 arp who-has baym-cs17.msgr.hotmail.com tell linuxbox
11:17:07.852219 arp reply baym-cs17.msgr.hotmail.com is-at 0:2:b9:1d:db:41
11:17:09.888162 arp who-has 207.46.106.80 tell linuxbox
11:17:09.889078 arp reply 207.46.106.80 is-at 0:2:b9:1d:db:41
11:17:10.389189 arp who-has 204.152.184.64 tell linuxbox
11:17:10.390134 arp reply 204.152.184.64 is-at 0:2:b9:1d:db:41
11:17:10.640043 arp who-has 200.225.157.104 tell linuxbox
11:17:10.640967 arp reply 200.225.157.104 is-at 0:2:b9:1d:db:41
11:17:10.689240 arp who-has 200.225.157.165 tell linuxbox
11:17:10.690768 arp reply 200.225.157.165 is-at 0:2:b9:1d:db:41
11:17:10.893170 arp who-has 200.225.157.163 tell linuxbox
11:17:10.894088 arp reply 200.225.157.163 is-at 0:2:b9:1d:db:41
11:17:10.980746 arp who-has 200.225.157.167 tell linuxbox
11:17:10.981714 arp reply 200.225.157.167 is-at 0:2:b9:1d:db:41
11:17:11.504255 arp who-has a.gtld-servers.net tell linuxbox
11:17:11.505926 arp reply a.gtld-servers.net is-at 0:2:b9:1d:db:41

2183 packets received by filter
0 packets dropped by kernel

We   see   my   linux  box  asking  for MAC addresses of hosts outside
its "local" network and my gateway, a Cisco 2621 answering those
broadcasts with its own MAC address.

For  what  I know, both are doing wrong. My box is not supposed to ask
for those MACs and the Cisco is not supposed to answer.

Does anybody have seen these before or have any ideas what would cause
it?

tks in advance.

Andre



On 03/12/02, Cedric Blancher wrote:
CB> Le lun 02/12/2002 à 21:28, andre.correa@pobox.com a écrit :
>> But  there  is  still a question for me. Looking at my arp table, I
>> see that there are =~ 150 entries, seconds passing and more entries
>> coming, 20 seconds after I can have =~1100, it goes on until it reachs
>> =~2200  entries,  then it goes back to the =~100 and starts over again.

CB> Wierd...

>> I  have  less  then  50  NAT users. Is it normal to have some many ARP
>> entries with this variation? Looking the ARP table I see my "Internet"
>> interface with lots of entries, with internet host IP addresses and my
>> gateway's NIC MAC address.
>> Isn't ARP supposed to keep entries just to local network systems?

CB> Yes it is.
CB> ARP is supposed to keep track of IP/MAC associations for network
CB> directly routed to interface, i.e. directly connected, aka local LANs.

>> Is it all normal? And if so, how big can gc_threash[1,2,3] be?

CB> It is not normal. You should monitor ARP traffic on your network using
CB> arpwatch (see Freshmeat, available as .deb, .rpm too) to see if someone
CB> would be playing ARP cache poisoning (see http://www.arp-sk.org/).

Re: Too many ARP entries and Re: sendto: No buffer space available

[Nick Drage]

On Tue, Dec 03, 2002 at 12:27:24PM -0200, andre.correa@pobox.com wrote:

> root@linuxbox:~# tcpdump -i eth1 | grep arp
> tcpdump: listening on eth1
> Dec  3 11:16:52 linuxbox kernel: device eth1 entered promiscuous mode

<snip>

> 11:17:10.390134 arp reply 204.152.184.64 is-at 0:2:b9:1d:db:41
> 11:17:10.640043 arp who-has 200.225.157.104 tell linuxbox
> 11:17:10.640967 arp reply 200.225.157.104 is-at 0:2:b9:1d:db:41
> 11:17:10.689240 arp who-has 200.225.157.165 tell linuxbox
> 11:17:10.690768 arp reply 200.225.157.165 is-at 0:2:b9:1d:db:41
> 11:17:10.893170 arp who-has 200.225.157.163 tell linuxbox
> 11:17:10.894088 arp reply 200.225.157.163 is-at 0:2:b9:1d:db:41
> 11:17:10.980746 arp who-has 200.225.157.167 tell linuxbox
> 11:17:10.981714 arp reply 200.225.157.167 is-at 0:2:b9:1d:db:41
> 11:17:11.504255 arp who-has a.gtld-servers.net tell linuxbox
> 11:17:11.505926 arp reply a.gtld-servers.net is-at 0:2:b9:1d:db:41
> 
> 2183 packets received by filter
> 0 packets dropped by kernel
> 
> We   see   my   linux  box  asking  for MAC addresses of hosts outside
> its "local" network and my gateway, a Cisco 2621 answering those
> broadcasts with its own MAC address.

Yes, very peculiar.  Your linuxbox appears to think the Internet is one big
switched network :)

What does 

netstat -rn give you?

> For  what  I know, both are doing wrong. My box is not supposed to ask
> for those MACs and the Cisco is not supposed to answer.

Yes.  Weren't you using PPPoE or similar?  Not familiar with that at all but
that might be related.

> Does anybody have seen these before or have any ideas what would cause
> it?

Out of interest, where have you looked for answered to this problem? 
Looking for overflowing arp tables via www.google.com or similar might give
you the answers you need.

-- 
FunkyJesus System Administration Team

Re: Too many ARP entries and Re: sendto: No buffer space available

[Paul Frieden]

Is your default gateway configured?  It sounds like your router is 
running proxy arp.  If you have the default route set to an interface, 
but without a gateway IP, it will arp to find the IP.  Since the Cisco 
by default has proxy-arp enabled, it will reply that the IP address is 
accessable via its own MAC address.  If you set your default gateway 
correctly, it should resolve the issue.

Paul


Nick Drage wrote:
> On Tue, Dec 03, 2002 at 12:27:24PM -0200, andre.correa@pobox.com wrote:
> 
> 
>>root@linuxbox:~# tcpdump -i eth1 | grep arp
>>tcpdump: listening on eth1
>>Dec  3 11:16:52 linuxbox kernel: device eth1 entered promiscuous mode
> 
> 
> <snip>
> 
>>11:17:10.390134 arp reply 204.152.184.64 is-at 0:2:b9:1d:db:41
>>11:17:10.640043 arp who-has 200.225.157.104 tell linuxbox
>>11:17:10.640967 arp reply 200.225.157.104 is-at 0:2:b9:1d:db:41
>>11:17:10.689240 arp who-has 200.225.157.165 tell linuxbox
>>11:17:10.690768 arp reply 200.225.157.165 is-at 0:2:b9:1d:db:41
>>11:17:10.893170 arp who-has 200.225.157.163 tell linuxbox
>>11:17:10.894088 arp reply 200.225.157.163 is-at 0:2:b9:1d:db:41
>>11:17:10.980746 arp who-has 200.225.157.167 tell linuxbox
>>11:17:10.981714 arp reply 200.225.157.167 is-at 0:2:b9:1d:db:41
>>11:17:11.504255 arp who-has a.gtld-servers.net tell linuxbox
>>11:17:11.505926 arp reply a.gtld-servers.net is-at 0:2:b9:1d:db:41
>>
>>2183 packets received by filter
>>0 packets dropped by kernel
>>
>>We   see   my   linux  box  asking  for MAC addresses of hosts outside
>>its "local" network and my gateway, a Cisco 2621 answering those
>>broadcasts with its own MAC address.
> 
> 
> Yes, very peculiar.  Your linuxbox appears to think the Internet is one big
> switched network :)
> 
> What does 
> 
> netstat -rn give you?
> 
> 
>>For  what  I know, both are doing wrong. My box is not supposed to ask
>>for those MACs and the Cisco is not supposed to answer.
> 
> 
> Yes.  Weren't you using PPPoE or similar?  Not familiar with that at all but
> that might be related.
> 
> 
>>Does anybody have seen these before or have any ideas what would cause
>>it?
> 
> 
> Out of interest, where have you looked for answered to this problem? 
> Looking for overflowing arp tables via www.google.com or similar might give
> you the answers you need.
> 

Re: Too many ARP entries and Re: sendto: No buffer space available

[Ard van Breemen]

On Tue, Dec 03, 2002 at 12:27:24PM -0200, andre.correa@pobox.com wrote:
> We   see   my   linux  box  asking  for MAC addresses of hosts outside
> its "local" network and my gateway, a Cisco 2621 answering those
> broadcasts with its own MAC address.
> 
> For  what  I know, both are doing wrong. My box is not supposed to ask
> for those MACs and the Cisco is not supposed to answer.
Your cisco is configured to do proxy-arp. This might be a policy
decision. I have proxy-arp on, because I have very small subnets,
but pretend to be a /24 to the customers. Nobody notices it.

The only fault in your setup is that you probably have:
ip route add default dev <internetdev>

Be aware that ip route show might not show you the details.
If you do for example this:
ip route add 172.16.0.1/32 dev eth0
ip route add default via 172.16.0.1
ip route del 172.16.0.1/32 dev eth0
you would see a natural "gatewayed" route, but with something
peculiar: 172.16.0.1 was local at the time of addition, so it
will send everything to the interface as local traffic, not
gatewayed!

route -n will tell you the real routing. Eh, but only for the
main routing table... :-).
Anyway: fix your default gateway.
-- 
procedure signature;
begin  { telegraaf.com
} writeln('<ard@telegraafnet.nl> SMA-IS | Geeks don't get viruses');
end