neighbour table overflow errors. leaking neighbours I think

neighbour table overflow errors. leaking neighbours I think

[Bruce Jones]

I am seeing neighbour table overflow messages so I began exploring
neighbour structure reference counts to try and understand the issue
and see why they are not being reclaimed by the garbage collector.
I'm pretty new to the kernel so have just been sprinkling printks
around to try to understand how this all works.

The box is running 2.6.32.50.  I know that is old, but the current
code for 3.2.1 looks like it is doing basically the same thing.

I am running a test that is blasting the box with lots of bogus tcp
packets.  This results in calls to tcp_v4_send_reset().  That then
calls ip_send_reply() which calls ip_route_output_key() which causes a
new neighbour entry to be allocated.  Then back in ip_send_reply(),
ip_append_data() is called.  I find that this call is failing and
returning ENOBUFS.   The return value from ip_append_data() is not
checked in ip_send_reply() so nothing is cleaned up.  I see that other
calls to ip_append_data() do check the return value and call some
cleanup function.  Those cleanup functions do things like release skb,
dsts, etc.  I image I might be leaking those things too, though I have
only looked at nieghbours.  My question is should there be a similar
function to clean up if ip_append_data fails() in ip_send_reply() or
is there a reason that that particular call doesn't check for errors
when the others do?

Thanks,
Bruce

Re: neighbour table overflow errors. leaking neighbours I think

[David Miller]

From: Bruce Jones <brucerjones@gmail.com>
Date: Fri, 13 Jan 2012 17:29:59 -0700

> I am seeing neighbour table overflow messages so I began exploring
> neighbour structure reference counts to try and understand the issue
> and see why they are not being reclaimed by the garbage collector.

Every route holds a reference to a neighbour entry, so until the
routes are purged or flushed, the neighbours can't be reclaimed.

Re: neighbour table overflow errors. leaking neighbours I think

[Eric Dumazet]

Le vendredi 13 janvier 2012 à 17:29 -0700, Bruce Jones a écrit :
> I am seeing neighbour table overflow messages so I began exploring
> neighbour structure reference counts to try and understand the issue
> and see why they are not being reclaimed by the garbage collector.
> I'm pretty new to the kernel so have just been sprinkling printks
> around to try to understand how this all works.
> 
> The box is running 2.6.32.50.  I know that is old, but the current
> code for 3.2.1 looks like it is doing basically the same thing.
> 
> I am running a test that is blasting the box with lots of bogus tcp
> packets.  This results in calls to tcp_v4_send_reset().  That then
> calls ip_send_reply() which calls ip_route_output_key() which causes a
> new neighbour entry to be allocated.  Then back in ip_send_reply(),
> ip_append_data() is called.  I find that this call is failing and
> returning ENOBUFS.   The return value from ip_append_data() is not
> checked in ip_send_reply() so nothing is cleaned up.  I see that other
> calls to ip_append_data() do check the return value and call some
> cleanup function.  Those cleanup functions do things like release skb,
> dsts, etc.  I image I might be leaking those things too, though I have
> only looked at nieghbours.  My question is should there be a similar
> function to clean up if ip_append_data fails() in ip_send_reply() or
> is there a reason that that particular call doesn't check for errors
> when the others do?
> 

ip_send_reply() does the necessary route cleanup.

I see no leak there.


void ip_send_reply(struct sock *sk, struct sk_buff *skb, __be32 daddr,
                   const struct ip_reply_arg *arg, unsigned int len)
{
...
        rt = ip_route_output_key(sock_net(sk), &fl4);
        if (IS_ERR(rt))
                return;
	bh_lock_sock(sk);

	...
	ip_append_data(sk, &fl4, ip_reply_glue_bits, arg->iov->iov_base,
			len, 0,
                       &ipc, &rt, MSG_DONTWAIT);
	...

	bh_unlock_sock(sk);

	ip_rt_put(rt); // HERE
}


What is your network configuration ?  (ip ro)

I suspect your are attached to a /16 network and you forgot to increase
neigh limits.

# grep . /proc/sys/net/ipv4/neigh/default/gc_thresh*
/proc/sys/net/ipv4/neigh/default/gc_thresh1:128
/proc/sys/net/ipv4/neigh/default/gc_thresh2:512
/proc/sys/net/ipv4/neigh/default/gc_thresh3:1024