All of lore.kernel.org
 help / color / mirror / Atom feed
From: Ben Greear <greearb@candelatech.com>
To: Eric Dumazet <eric.dumazet@gmail.com>
Cc: netdev <netdev@vger.kernel.org>,
	"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: 3.7.3+:  Bad paging request in ip_rcv_finish while running NFS traffic.
Date: Wed, 23 Jan 2013 16:13:28 -0800	[thread overview]
Message-ID: <51007CA8.2050105@candelatech.com> (raw)
In-Reply-To: <1358985688.12374.1247.camel@edumazet-glaptop>

On 01/23/2013 04:01 PM, Eric Dumazet wrote:
> On Wed, 2013-01-23 at 15:55 -0800, Ben Greear wrote:
>> On 01/22/2013 06:32 PM, Ben Greear wrote:
>>
>> So, I'm slowly making some progress.  I've verified that the skb
>> has bogus dst (0xdeadbeef) at the top of the ip_rcv_finish
>> method.  I'm trying to track it backwards and figure out which
>> device it belongs to, etc....takes a while to reproduce though.
>>
>> One thing about this stack trace below...the dev_seq_stop() does
>> a rcu read-unlock.  Now, I can't figure out exactly how ip_rcv()
>> can cause dev_seq_stop() to run, but if this stack is legit,
>> then maybe by the time we enter the ip_rcv_finish() code we are
>> running without rcu_readlock() held?
>>
>> If so, that would probably explain the bug.
>>
>
> The whole thing is run under rcu_read_lock() done in
> __netif_receive_skb()

I was worried that the dev_seq_stop might be called
incorrectly causing an asymetric unlock.  I have no
idea how that might happened, but several crashes
have that dev_seq_stop method listed, so it got me suspicious.

>
> My suspicion was that we called netif_rx() from macvlan leaving a
> not refcounted skb dst.
>
> But the patch I sent to you didnt solve the bug, so its something else.
>
> You could trace at which point the dst was released. (where you set
> dst->input/output to deadbeef)

My current code is in some garbage collector timer code, but I can
work on saving the call-site that first pokes the dst into the
garbage collection list...

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com


WARNING: multiple messages have this Message-ID (diff)
From: Ben Greear <greearb-my8/4N5VtI7c+919tysfdA@public.gmane.org>
To: Eric Dumazet <eric.dumazet-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
Cc: netdev <netdev-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>,
	"linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org"
	<linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org>
Subject: Re: 3.7.3+:  Bad paging request in ip_rcv_finish while running NFS traffic.
Date: Wed, 23 Jan 2013 16:13:28 -0800	[thread overview]
Message-ID: <51007CA8.2050105@candelatech.com> (raw)
In-Reply-To: <1358985688.12374.1247.camel@edumazet-glaptop>

On 01/23/2013 04:01 PM, Eric Dumazet wrote:
> On Wed, 2013-01-23 at 15:55 -0800, Ben Greear wrote:
>> On 01/22/2013 06:32 PM, Ben Greear wrote:
>>
>> So, I'm slowly making some progress.  I've verified that the skb
>> has bogus dst (0xdeadbeef) at the top of the ip_rcv_finish
>> method.  I'm trying to track it backwards and figure out which
>> device it belongs to, etc....takes a while to reproduce though.
>>
>> One thing about this stack trace below...the dev_seq_stop() does
>> a rcu read-unlock.  Now, I can't figure out exactly how ip_rcv()
>> can cause dev_seq_stop() to run, but if this stack is legit,
>> then maybe by the time we enter the ip_rcv_finish() code we are
>> running without rcu_readlock() held?
>>
>> If so, that would probably explain the bug.
>>
>
> The whole thing is run under rcu_read_lock() done in
> __netif_receive_skb()

I was worried that the dev_seq_stop might be called
incorrectly causing an asymetric unlock.  I have no
idea how that might happened, but several crashes
have that dev_seq_stop method listed, so it got me suspicious.

>
> My suspicion was that we called netif_rx() from macvlan leaving a
> not refcounted skb dst.
>
> But the patch I sent to you didnt solve the bug, so its something else.
>
> You could trace at which point the dst was released. (where you set
> dst->input/output to deadbeef)

My current code is in some garbage collector timer code, but I can
work on saving the call-site that first pokes the dst into the
garbage collection list...

Thanks,
Ben

-- 
Ben Greear <greearb-my8/4N5VtI7c+919tysfdA@public.gmane.org>
Candela Technologies Inc  http://www.candelatech.com

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

  reply	other threads:[~2013-01-24  0:13 UTC|newest]

Thread overview: 53+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-01-21 21:07 3.7.3+: Bad paging request in ip_rcv_finish while running NFS traffic Ben Greear
2013-01-21 21:07 ` Ben Greear
2013-01-22  0:32 ` Ben Greear
2013-01-22  4:40   ` Eric Dumazet
2013-01-22  5:57     ` Ben Greear
2013-01-22  5:57       ` Ben Greear
2013-01-22 17:08       ` Ben Greear
2013-01-22 17:08         ` Ben Greear
2013-01-22 17:17         ` Eric Dumazet
2013-01-22 17:17           ` Eric Dumazet
2013-01-22 17:26           ` Ben Greear
2013-01-22 17:26             ` Ben Greear
2013-01-22 17:26           ` Eric Dumazet
2013-01-22 22:18             ` Ben Greear
2013-01-22 22:18               ` Ben Greear
2013-01-23  2:32               ` Ben Greear
2013-01-23  2:32                 ` Ben Greear
2013-01-23  6:11                 ` Eric Dumazet
2013-01-23  7:14                   ` Ben Greear
2013-01-23  7:14                     ` Ben Greear
2013-01-23 13:35                     ` Eric Dumazet
2013-01-23 13:35                       ` Eric Dumazet
2013-01-23 18:15                       ` Ben Greear
2013-01-23 18:15                         ` Ben Greear
2013-01-23 21:43                         ` Eric Dumazet
2013-01-23 14:42                     ` Eric Dumazet
2013-01-23 14:42                       ` Eric Dumazet
2013-01-23 21:53                       ` Ben Greear
2013-01-23 21:53                         ` Ben Greear
2013-01-23 23:55                 ` Ben Greear
2013-01-23 23:55                   ` Ben Greear
2013-01-24  0:01                   ` Eric Dumazet
2013-01-24  0:01                     ` Eric Dumazet
2013-01-24  0:13                     ` Ben Greear [this message]
2013-01-24  0:13                       ` Ben Greear
2013-01-24  0:23                       ` Eric Dumazet
2013-01-24  0:23                         ` Eric Dumazet
2013-01-24  0:38                         ` Ben Greear
2013-01-24  0:45                           ` Eric Dumazet
2013-01-24  0:51                             ` Ben Greear
2013-01-24  1:00                               ` Eric Dumazet
2013-01-24  1:06                                 ` Ben Greear
2013-01-24  1:10                                   ` Eric Dumazet
2013-01-24  1:45                                     ` Eric Dumazet
2013-01-24  4:26                                       ` Ben Greear
2013-01-24  5:39                                         ` Eric Dumazet
2013-01-24 20:03                                       ` Ben Greear
2013-01-24 20:59                                         ` Eric Dumazet
2013-01-24 21:01                                           ` Ben Greear
2013-01-25 17:44                                       ` [PATCH] net: loopback: fix a dst refcounting issue Eric Dumazet
2013-01-27  6:32                                         ` David Miller
2013-01-27 17:25                                           ` Eric Dumazet
2013-01-28  0:26                                             ` David Miller

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=51007CA8.2050105@candelatech.com \
    --to=greearb@candelatech.com \
    --cc=eric.dumazet@gmail.com \
    --cc=linux-nfs@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.