IPV6 ndisc:: Bad NIC causing IPV6 NDP to stop working

IPV6 ndisc:: Bad NIC causing IPV6 NDP to stop working

[Menny_Hamburger]

Hi,

Our machines runs EL5.8 x86_64.
We have witnessed several cases where we suspect that a bad NIC on the machine caused IPV6 neighbour discovery to stop working on all the other NICs - when this happens ping6 fails on every NIC we try it.
>From looking into the code I see that there is only a single socket assigned for NDP; Does it sound logical to allocate a socket per interface instead of a single global socket.
I have found the following thread in LKML: https://lkml.org/lkml/2006/11/29/335, and it seems that this allocation issue still exists in EL5 based kernels - could this cause the above problem?

Thanks,
Menny

Re: IPV6 ndisc:: Bad NIC causing IPV6 NDP to stop working

[Eric Dumazet]

On Thu, 2012-06-21 at 08:59 +0100, Menny_Hamburger@Dell.com wrote:
> Hi,
> 
> Our machines runs EL5.8 x86_64.
> We have witnessed several cases where we suspect that a bad NIC on the machine caused IPV6 neighbour discovery to stop working on all the other NICs - when this happens ping6 fails on every NIC we try it.
> From looking into the code I see that there is only a single socket assigned for NDP; Does it sound logical to allocate a socket per interface instead of a single global socket.
> I have found the following thread in LKML: https://lkml.org/lkml/2006/11/29/335, and it seems that this allocation issue still exists in EL5 based kernels - could this cause the above problem?
> 

What is a bad NIC, and why not fixing it ?

RE: IPV6 ndisc:: Bad NIC causing IPV6 NDP to stop working

[Menny_Hamburger]

For high availability reasons, the machines discussed run with a number of NICs per subnet, where our own proprietary service fixes up routing when a NIC goes wild.
We schedule a fix in the field but our goal is to eliminate as many single points of failure as we can, so that our systems will still run properly when something goes wrong.
We encountered this issue on some proprietary NICs but also with bnx2, where we get "chip not in correct endian mode" errors (This is another problem that may require a separate discussion).

-----Original Message-----
From: Eric Dumazet [mailto:eric.dumazet@gmail.com] 
Sent: 21 June, 2012 11:23
To: Hamburger, Menny
Cc: netdev@vger.kernel.org
Subject: Re: IPV6 ndisc:: Bad NIC causing IPV6 NDP to stop working

On Thu, 2012-06-21 at 08:59 +0100, Menny_Hamburger@Dell.com wrote:
> Hi,
> 
> Our machines runs EL5.8 x86_64.
> We have witnessed several cases where we suspect that a bad NIC on the machine caused IPV6 neighbour discovery to stop working on all the other NICs - when this happens ping6 fails on every NIC we try it.
> From looking into the code I see that there is only a single socket assigned for NDP; Does it sound logical to allocate a socket per interface instead of a single global socket.
> I have found the following thread in LKML: https://lkml.org/lkml/2006/11/29/335, and it seems that this allocation issue still exists in EL5 based kernels - could this cause the above problem?
> 

What is a bad NIC, and why not fixing it ?

RE: IPV6 ndisc:: Bad NIC causing IPV6 NDP to stop working

[Eric Dumazet]

Please don't top post on this list.

On Thu, 2012-06-21 at 09:43 +0100, Menny_Hamburger@Dell.com wrote:
> For high availability reasons, the machines discussed run with a
> number of NICs per subnet, where our own proprietary service fixes up
> routing when a NIC goes wild.
> We schedule a fix in the field but our goal is to eliminate as many
> single points of failure as we can, so that our systems will still run
> properly when something goes wrong.

Even if a NIC does memory corruption or some nasty bug ?
That sounds great :)

> We encountered this issue on some proprietary NICs but also with bnx2,
> where we get "chip not in correct endian mode" errors (This is another
> problem that may require a separate discussion).

Until very recently, we used to orphan skb before giving them to device
transmit. So you probably use a very old kernel.

I guess we could just do a regular alloc_skb(), it makes no sense to
limit in-flight ND skbs, we have Qdisc/device limits anyway.

BTW, I have no idea why ndisc_build_skb() is EXPORTed

 net/ipv6/ndisc.c |   24 ++++++------------------
 1 file changed, 6 insertions(+), 18 deletions(-)


diff --git a/net/ipv6/ndisc.c b/net/ipv6/ndisc.c
index 69a6330..f149d85 100644
--- a/net/ipv6/ndisc.c
+++ b/net/ipv6/ndisc.c
@@ -429,7 +429,6 @@ struct sk_buff *ndisc_build_skb(struct net_device *dev,
 	int hlen = LL_RESERVED_SPACE(dev);
 	int tlen = dev->needed_tailroom;
 	int len;
-	int err;
 	u8 *opt;
 
 	if (!dev->addr_len)
@@ -439,15 +438,10 @@ struct sk_buff *ndisc_build_skb(struct net_device *dev,
 	if (llinfo)
 		len += ndisc_opt_addr_space(dev);
 
-	skb = sock_alloc_send_skb(sk,
-				  (MAX_HEADER + sizeof(struct ipv6hdr) +
-				   len + hlen + tlen),
-				  1, &err);
-	if (!skb) {
-		ND_PRINTK(0, err, "ND: %s failed to allocate an skb, err=%d\n",
-			  __func__, err);
+	skb = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + len + hlen + tlen,
+			GFP_ATOMIC);
+	if (!skb)
 		return NULL;
-	}
 
 	skb_reserve(skb, hlen);
 	ip6_nd_hdr(sk, skb, dev, saddr, daddr, IPPROTO_ICMPV6, len);
@@ -1550,16 +1544,10 @@ void ndisc_send_redirect(struct sk_buff *skb, const struct in6_addr *target)
 
 	hlen = LL_RESERVED_SPACE(dev);
 	tlen = dev->needed_tailroom;
-	buff = sock_alloc_send_skb(sk,
-				   (MAX_HEADER + sizeof(struct ipv6hdr) +
-				    len + hlen + tlen),
-				   1, &err);
-	if (buff == NULL) {
-		ND_PRINTK(0, err,
-			  "Redirect: %s failed to allocate an skb, err=%d\n",
-			  __func__, err);
+	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + len + hlen + tlen,
+			 GFP_ATOMIC);
+	if (!buff)
 		goto release;
-	}
 
 	skb_reserve(buff, hlen);
 	ip6_nd_hdr(sk, buff, dev, &saddr_buf, &ipv6_hdr(skb)->saddr,

RE: IPV6 ndisc:: Bad NIC causing IPV6 NDP to stop working

[Menny_Hamburger]

After some debugging we found that our network driver sometimes fails to release the skb after calls to tx_start_xmit, which caused the ndisc socket send buffer to become full.
(sock_alloc_send_skb would fail on (atomic_read(&sk->sk_wmwm_alloc) < sk->sk_sndbuf) ). 
We also witnessed allocation failures in cases where we have a number of different NICS (1GB, 10GB, ...) - the slowest NIC becomes a bottleneck.
Using alloc_skb instead of sock_alloc_skb fixes this problem; my only concern is that in the extreme cases of a stray driver or some memory corruption, the absence of an upper limit to the skb 
allocation may consume the skbuff cache and cause all networking to behave badly.

Thanks,
Menny

net/ipv6/ndisc.c |   24 ++++++------------------
 1 file changed, 6 insertions(+), 18 deletions(-)


diff --git a/net/ipv6/ndisc.c b/net/ipv6/ndisc.c
index 69a6330..f149d85 100644
--- a/net/ipv6/ndisc.c
+++ b/net/ipv6/ndisc.c
@@ -429,7 +429,6 @@ struct sk_buff *ndisc_build_skb(struct net_device *dev,
 	int hlen = LL_RESERVED_SPACE(dev);
 	int tlen = dev->needed_tailroom;
 	int len;
-	int err;
 	u8 *opt;
 
 	if (!dev->addr_len)
@@ -439,15 +438,10 @@ struct sk_buff *ndisc_build_skb(struct net_device *dev,
 	if (llinfo)
 		len += ndisc_opt_addr_space(dev);
 
-	skb = sock_alloc_send_skb(sk,
-				  (MAX_HEADER + sizeof(struct ipv6hdr) +
-				   len + hlen + tlen),
-				  1, &err);
-	if (!skb) {
-		ND_PRINTK(0, err, "ND: %s failed to allocate an skb, err=%d\n",
-			  __func__, err);
+	skb = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + len + hlen + tlen,
+			GFP_ATOMIC);
+	if (!skb)
 		return NULL;
-	}
 
 	skb_reserve(skb, hlen);
 	ip6_nd_hdr(sk, skb, dev, saddr, daddr, IPPROTO_ICMPV6, len);
@@ -1550,16 +1544,10 @@ void ndisc_send_redirect(struct sk_buff *skb, const struct in6_addr *target)
 
 	hlen = LL_RESERVED_SPACE(dev);
 	tlen = dev->needed_tailroom;
-	buff = sock_alloc_send_skb(sk,
-				   (MAX_HEADER + sizeof(struct ipv6hdr) +
-				    len + hlen + tlen),
-				   1, &err);
-	if (buff == NULL) {
-		ND_PRINTK(0, err,
-			  "Redirect: %s failed to allocate an skb, err=%d\n",
-			  __func__, err);
+	buff = alloc_skb(MAX_HEADER + sizeof(struct ipv6hdr) + len + hlen + tlen,
+			 GFP_ATOMIC);
+	if (!buff)
 		goto release;
-	}
 
 	skb_reserve(buff, hlen);
 	ip6_nd_hdr(sk, buff, dev, &saddr_buf, &ipv6_hdr(skb)->saddr,