3.3-rc snmp6 panic

3.3-rc snmp6 panic

[Dave Jones]

I've been seeing an occasional panic when I shut down my router
since I put 3.3 on there. It happens about once a week, always during
shutdown.  It wedges before I can get a good capture of the trace.
This is the best I've captured so far.. https://twitpic.com/8wh5l5
(apologies in advance for blurriness)

>From comparing the Code: line, and the objdump output, the code it's
choking on in mld_sendpack seems to be a skb_dst macro in the NF_HOOK..


        err = NF_HOOK(NFPROTO_IPV6, NF_INET_LOCAL_OUT, skb, NULL, skb->dev,
    2dd9:       4c 8b 43 20             mov    0x20(%rbx),%r8
    2ddd:       e9 00 00 00 00          jmpq   2de2 <mld_sendpack+0x1b2>
static inline struct dst_entry *skb_dst(const struct sk_buff *skb)
{
        /* If refdst was not refcounted, check we still are in a 
         * rcu_read_lock section
         */
        WARN_ON((skb->_skb_refdst & SKB_DST_NOREF) &&
    2de2:       48 8b 43 58             mov    0x58(%rbx),%rax
    2de6:       a8 01                   test   $0x1,%al
    2de8:       0f 85 d2 01 00 00       jne    2fc0 <mld_sendpack+0x390>
                !rcu_read_lock_held() &&
                !rcu_read_lock_bh_held());
        return (struct dst_entry *)(skb->_skb_refdst & SKB_DST_PTRMASK);
    2dee:       48 83 e0 fe             and    $0xfffffffffffffffe,%rax
    2df2:       48 89 df                mov    %rbx,%rdi
    2df5:       ff 50 58                callq  *0x58(%rax)           <-----  BOOM


This machine is running an snmpd, for my mrtg setup, so the teardown of that
service is probably what's triggering it. But I can start/stop it in a loop
as much as I want without it happening, so maybe the kernel needs to accumulate
some state from it for a while first ?

Anyone have any ideas what's happening here ?

	Dave

Re: 3.3-rc snmp6 panic

[Eric Dumazet]

Le jeudi 15 mars 2012 à 01:25 -0400, Dave Jones a écrit :
> I've been seeing an occasional panic when I shut down my router
> since I put 3.3 on there. It happens about once a week, always during
> shutdown.  It wedges before I can get a good capture of the trace.
> This is the best I've captured so far.. https://twitpic.com/8wh5l5
> (apologies in advance for blurriness)
> 
> From comparing the Code: line, and the objdump output, the code it's
> choking on in mld_sendpack seems to be a skb_dst macro in the NF_HOOK..
> 
> 
>         err = NF_HOOK(NFPROTO_IPV6, NF_INET_LOCAL_OUT, skb, NULL, skb->dev,
>     2dd9:       4c 8b 43 20             mov    0x20(%rbx),%r8
>     2ddd:       e9 00 00 00 00          jmpq   2de2 <mld_sendpack+0x1b2>
> static inline struct dst_entry *skb_dst(const struct sk_buff *skb)
> {
>         /* If refdst was not refcounted, check we still are in a 
>          * rcu_read_lock section
>          */
>         WARN_ON((skb->_skb_refdst & SKB_DST_NOREF) &&
>     2de2:       48 8b 43 58             mov    0x58(%rbx),%rax
>     2de6:       a8 01                   test   $0x1,%al
>     2de8:       0f 85 d2 01 00 00       jne    2fc0 <mld_sendpack+0x390>
>                 !rcu_read_lock_held() &&
>                 !rcu_read_lock_bh_held());
>         return (struct dst_entry *)(skb->_skb_refdst & SKB_DST_PTRMASK);
>     2dee:       48 83 e0 fe             and    $0xfffffffffffffffe,%rax
>     2df2:       48 89 df                mov    %rbx,%rdi
>     2df5:       ff 50 58                callq  *0x58(%rax)           <-----  BOOM
> 
> 
> This machine is running an snmpd, for my mrtg setup, so the teardown of that
> service is probably what's triggering it. But I can start/stop it in a loop
> as much as I want without it happening, so maybe the kernel needs to accumulate
> some state from it for a while first ?
> 
> Anyone have any ideas what's happening here ?
> 

Hmm, please try :

diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 8c2e3ab..22b7664 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1077,7 +1077,7 @@ struct dst_entry *icmp6_dst_alloc(struct net_device *dev,
 	struct net *net = dev_net(dev);
 
 	if (unlikely(!idev))
-		return NULL;
+		return ERR_PTR(-ENODEV);
 
 	rt = ip6_dst_alloc(&net->ipv6.ip6_dst_ops, dev, 0);
 	if (unlikely(!rt)) {

Re: 3.3-rc snmp6 panic

[Eric Dumazet]

Le mercredi 14 mars 2012 à 23:56 -0700, Eric Dumazet a écrit :
> Le jeudi 15 mars 2012 à 01:25 -0400, Dave Jones a écrit :
> > I've been seeing an occasional panic when I shut down my router
> > since I put 3.3 on there. It happens about once a week, always during
> > shutdown.  It wedges before I can get a good capture of the trace.
> > This is the best I've captured so far.. https://twitpic.com/8wh5l5
> > (apologies in advance for blurriness)
> > 
> > From comparing the Code: line, and the objdump output, the code it's
> > choking on in mld_sendpack seems to be a skb_dst macro in the NF_HOOK..
> > 
> > 
> >         err = NF_HOOK(NFPROTO_IPV6, NF_INET_LOCAL_OUT, skb, NULL, skb->dev,
> >     2dd9:       4c 8b 43 20             mov    0x20(%rbx),%r8
> >     2ddd:       e9 00 00 00 00          jmpq   2de2 <mld_sendpack+0x1b2>
> > static inline struct dst_entry *skb_dst(const struct sk_buff *skb)
> > {
> >         /* If refdst was not refcounted, check we still are in a 
> >          * rcu_read_lock section
> >          */
> >         WARN_ON((skb->_skb_refdst & SKB_DST_NOREF) &&
> >     2de2:       48 8b 43 58             mov    0x58(%rbx),%rax
> >     2de6:       a8 01                   test   $0x1,%al
> >     2de8:       0f 85 d2 01 00 00       jne    2fc0 <mld_sendpack+0x390>
> >                 !rcu_read_lock_held() &&
> >                 !rcu_read_lock_bh_held());
> >         return (struct dst_entry *)(skb->_skb_refdst & SKB_DST_PTRMASK);
> >     2dee:       48 83 e0 fe             and    $0xfffffffffffffffe,%rax
> >     2df2:       48 89 df                mov    %rbx,%rdi
> >     2df5:       ff 50 58                callq  *0x58(%rax)           <-----  BOOM
> > 
> > 
> > This machine is running an snmpd, for my mrtg setup, so the teardown of that
> > service is probably what's triggering it. But I can start/stop it in a loop
> > as much as I want without it happening, so maybe the kernel needs to accumulate
> > some state from it for a while first ?
> > 
> > Anyone have any ideas what's happening here ?
> > 
> 
> Hmm, please try :

Since patch is trivial, here is the official submission

Thanks !

[PATCH] ipv6: fix icmp6_dst_alloc()

commit 87a115783 ( ipv6: Move xfrm_lookup() call down into
icmp6_dst_alloc().) forgot to convert one error path, leading
to crashes in mld_sendpack()

Many thanks to Dave Jones for providing a very complete bug report.

Reported-by: Dave Jones <davej@redhat.com>
Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>
---
 net/ipv6/route.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 8c2e3ab..22b7664 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1077,7 +1077,7 @@ struct dst_entry *icmp6_dst_alloc(struct net_device *dev,
 	struct net *net = dev_net(dev);
 
 	if (unlikely(!idev))
-		return NULL;
+		return ERR_PTR(-ENODEV);
 
 	rt = ip6_dst_alloc(&net->ipv6.ip6_dst_ops, dev, 0);
 	if (unlikely(!rt)) {

Re: 3.3-rc snmp6 panic

[David Miller]

From: Eric Dumazet <eric.dumazet@gmail.com>
Date: Thu, 15 Mar 2012 00:13:11 -0700

> [PATCH] ipv6: fix icmp6_dst_alloc()
> 
> commit 87a115783 ( ipv6: Move xfrm_lookup() call down into
> icmp6_dst_alloc().) forgot to convert one error path, leading
> to crashes in mld_sendpack()
> 
> Many thanks to Dave Jones for providing a very complete bug report.
> 
> Reported-by: Dave Jones <davej@redhat.com>
> Signed-off-by: Eric Dumazet <eric.dumazet@gmail.com>

Applied and queued up for -stable, thanks.

Re: 3.3-rc snmp6 panic

[Dave Jones]

On Wed, Mar 14, 2012 at 11:56:15PM -0700, Eric Dumazet wrote:
 
 > > This machine is running an snmpd, for my mrtg setup, so the teardown of that
 > > service is probably what's triggering it. But I can start/stop it in a loop
 > > as much as I want without it happening, so maybe the kernel needs to accumulate
 > > some state from it for a while first ?
 > > 
 > > Anyone have any ideas what's happening here ?
 > 
 > Hmm, please try :

Will do. Given I can't reproduce it quickly though, it might be a while
before I can provide feedback on whether it does the trick :)

thanks,

	Dave