Netdev List
 help / color / mirror / Atom feed
From: Nikolay Aleksandrov <nikolay@cumulusnetworks.com>
To: Linus Torvalds <torvalds@linux-foundation.org>,
	Andrey Konovalov <andreyknvl@google.com>,
	Eric Dumazet <edumazet@google.com>
Cc: "David S. Miller" <davem@davemloft.net>,
	Alexey Kuznetsov <kuznet@ms2.inr.ac.ru>,
	James Morris <jmorris@namei.org>,
	Hideaki YOSHIFUJI <yoshfuji@linux-ipv6.org>,
	Patrick McHardy <kaber@trash.net>,
	netdev <netdev@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Alexander Duyck <alexander.h.duyck@intel.com>,
	David Ahern <dsa@cumulusnetworks.com>,
	Daniel Borkmann <daniel@iogearbox.net>, tcharding <me@tobin.cc>,
	Jiri Pirko <jiri@mellanox.com>,
	stephen hemminger <stephen@networkplumber.org>,
	Dmitry Vyukov <dvyukov@google.com>,
	Kostya Serebryany <kcc@google.com>,
	syzkaller <syzkaller@googlegroups.com>
Subject: Re: net/core: BUG in unregister_netdevice_many
Date: Fri, 21 Apr 2017 21:30:30 +0300	[thread overview]
Message-ID: <2e044c75-c0e5-f5d5-1d94-7609d7af86fa@cumulusnetworks.com> (raw)
In-Reply-To: <CA+55aFyTFzML+_8E_kDjX=YFkaDi_ejGo15UgGtAT29244UwgA@mail.gmail.com>

On 21/04/17 20:42, Linus Torvalds wrote:
> On Fri, Apr 21, 2017 at 10:25 AM, Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>>
>> I'm assuming that the real cause is simply that "dev->reg_state" ends
>> up being NETREG_UNREGISTERING or something. Maybe the BUG_ON() could
>> be just removed, and replaced by the previous warning about
>> NETREG_UNINITIALIZED.
>>
>> Something like the attached (TOTALLY UNTESTED) patch.
> 
> .. might as well test it.
> 
> That patch doesn't fix the problem, but it does show that yes, it was
> NETREG_UNREGISTERING:
> 
>   unregister_netdevice: device pim6reg/ffff962dc4606000 was not registered (2)
> 
> but then immediately afterwards we get
> 
>   general protection fault: 0000 [#1] SMP
>   Workqueue: netns cleanup_net
>   RIP: 0010:dev_shutdown+0xe/0xc0
>   Call Trace:
>      rollback_registered_many+0x2a5/0x440
>      unregister_netdevice_many+0x1e/0xb0
>      default_device_exit_batch+0x145/0x170
> 
> which is due to a
> 
>         mov    0x388(%rdi),%eax
> 
> where %rdi is 0xdead000000000090. That is at the very beginning of
> dev_shutdown, it's "dev" itself that has that value, so it comes from
> (_another_) invocation of rollback_registered_many(), when it does
> that
> 
>         list_for_each_entry(dev, head, unreg_list) {
> 
> so it seems to be a case of another "list_del() leaves list in bad
> state", and it was the added test for "dev->reg_state !=
> NETREG_REGISTERED" that did that
> 
>         list_del(&dev->unreg_list);
> 
> and left random contents in the unreg_list.
> 
> So that "handle error case" was almost certainly just buggy too.
> 
> And the bug seems to be that we're trying to unregister a netdevice
> that has already been unregistered.
> 
> Over to Eric and networking people. This oops is user-triggerable, and
> leaves the machine in a bad state (the original BUG_ON() and the new
> GP fault both happen while holding the RTNL, so networking is not
> healthy afterwards.
> 
>                       Linus
> 

Right, I've already posted a patch for ip6mr that should fix the issue.
CCed you and LKML just now.

Thanks,
 Nik

  reply	other threads:[~2017-04-21 18:30 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-04-21 12:48 net/core: BUG in unregister_netdevice_many Andrey Konovalov
2017-04-21 17:25 ` Linus Torvalds
2017-04-21 17:42   ` Linus Torvalds
2017-04-21 18:30     ` Nikolay Aleksandrov [this message]
2017-04-21 19:37     ` David Miller
2017-04-21 18:55   ` Cong Wang
2017-04-21 19:38     ` David Miller
2017-04-21 17:42 ` [PATCH net] ip6mr: fix notification device destruction Nikolay Aleksandrov
2017-04-21 18:30   ` Nikolay Aleksandrov
2017-04-21 18:46     ` Andrey Konovalov
2017-04-21 19:36     ` David Miller
2017-04-21 19:50       ` Nikolay Aleksandrov
2017-04-21 19:55         ` David Miller
2017-04-21 19:56         ` Nikolay Aleksandrov
2017-04-21 19:58           ` David Miller

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2e044c75-c0e5-f5d5-1d94-7609d7af86fa@cumulusnetworks.com \
    --to=nikolay@cumulusnetworks.com \
    --cc=alexander.h.duyck@intel.com \
    --cc=andreyknvl@google.com \
    --cc=daniel@iogearbox.net \
    --cc=davem@davemloft.net \
    --cc=dsa@cumulusnetworks.com \
    --cc=dvyukov@google.com \
    --cc=edumazet@google.com \
    --cc=jiri@mellanox.com \
    --cc=jmorris@namei.org \
    --cc=kaber@trash.net \
    --cc=kcc@google.com \
    --cc=kuznet@ms2.inr.ac.ru \
    --cc=linux-kernel@vger.kernel.org \
    --cc=me@tobin.cc \
    --cc=netdev@vger.kernel.org \
    --cc=stephen@networkplumber.org \
    --cc=syzkaller@googlegroups.com \
    --cc=torvalds@linux-foundation.org \
    --cc=yoshfuji@linux-ipv6.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox