All of lore.kernel.org
 help / color / mirror / Atom feed
From: w@1wt.eu (Willy Tarreau)
To: linux-arm-kernel@lists.infradead.org
Subject: mvneta: oops in __rcu_read_lock on mirabox
Date: Mon, 16 Sep 2013 08:50:47 +0200	[thread overview]
Message-ID: <20130916065047.GH27487@1wt.eu> (raw)
In-Reply-To: <20130915205701.5c61a444@skate>

Hi Thomas,

On Sun, Sep 15, 2013 at 08:57:01PM +0200, Thomas Petazzoni wrote:
> Hello Ethan,
> 
> On Sat, 14 Sep 2013 18:05:32 -0700, Ethan Tuttle wrote:
> > When I upgraded my mirabox from 3.11-rc4 to 3.11, I started seeing
> > oopses while receiving network traffic (see below).  Sending a flood
> > ping will trigger the oops within a few minutes.
> > 
> > The stack looks similar, but not identical to, the one reported
> > earlier by Jochen De Smet[1].  In my case the PC is always
> > __rcu_read_lock.
> > 
> > A git bisect found a878764 "Merge
> > git://git.kernel.org/pub/scm/linux/kernel/git/davem/net" to be the
> > first bad commit... interesting, because neither of the merge parents
> > produce the oops.  I rebased the net changes onto the other merge
> > parent and bisected that series, which identified 702821f "net: revert
> > 8728c544a9c ("net: dev_pick_tx() fix")" as the first bad commit.
> > Indeed, reverting 702821f from 3.11 produces a kernel which stands up
> > to a ping flood for hours.
> > 
> > Each of the times I reproduced this, it was identified as "Unhandled
> > prefetch abort: unknown 25 (0x409) at 0xc0036ea0", except once when I
> > got "unknown 16 (0x400)".
> > 
> > I'm assuming this is an mvneta bug that was exposed by 702821f.
> > That's just a guess, and I don't have the skills to debug this any
> > further.  In any case, I figured the maintainers would want to know
> > about it.
> 
> Thanks a lot for the report and the detailed investigation.
> Unfortunately, I don't have Armada 370 hardware with me this week, so
> I'm unable to test and reproduce the issue.
> 
> However, I've added a bunch of Armada 370 people/maintainers in Cc,
> hopefully they can at least try to reproduce and confirm that reverting
> this patch makes the problem go away, which would confirm that we
> should look for a bug in the mvneta driver around this problem.

I'm currently testing on 3.11.1 (which I had here) and am not getting
any issue after 50M packets. My kernel is running in thumb mode and
without SMP.

Ethan, we'll need your config I guess.

Thanks,
Willy

WARNING: multiple messages have this Message-ID (diff)
From: Willy Tarreau <w@1wt.eu>
To: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: "Ethan Tuttle" <ethan@ethantuttle.com>,
	"Andrew Lunn" <andrew@lunn.ch>,
	"Jason Cooper" <jason@lakedaemon.net>,
	netdev@vger.kernel.org,
	"Ezequiel Garcia" <ezequiel.garcia@free-electrons.com>,
	"Gregory Clément" <gregory.clement@free-electrons.com>,
	linux-arm-kernel@lists.infradead.org
Subject: Re: mvneta: oops in __rcu_read_lock on mirabox
Date: Mon, 16 Sep 2013 08:50:47 +0200	[thread overview]
Message-ID: <20130916065047.GH27487@1wt.eu> (raw)
In-Reply-To: <20130915205701.5c61a444@skate>

Hi Thomas,

On Sun, Sep 15, 2013 at 08:57:01PM +0200, Thomas Petazzoni wrote:
> Hello Ethan,
> 
> On Sat, 14 Sep 2013 18:05:32 -0700, Ethan Tuttle wrote:
> > When I upgraded my mirabox from 3.11-rc4 to 3.11, I started seeing
> > oopses while receiving network traffic (see below).  Sending a flood
> > ping will trigger the oops within a few minutes.
> > 
> > The stack looks similar, but not identical to, the one reported
> > earlier by Jochen De Smet[1].  In my case the PC is always
> > __rcu_read_lock.
> > 
> > A git bisect found a878764 "Merge
> > git://git.kernel.org/pub/scm/linux/kernel/git/davem/net" to be the
> > first bad commit... interesting, because neither of the merge parents
> > produce the oops.  I rebased the net changes onto the other merge
> > parent and bisected that series, which identified 702821f "net: revert
> > 8728c544a9c ("net: dev_pick_tx() fix")" as the first bad commit.
> > Indeed, reverting 702821f from 3.11 produces a kernel which stands up
> > to a ping flood for hours.
> > 
> > Each of the times I reproduced this, it was identified as "Unhandled
> > prefetch abort: unknown 25 (0x409) at 0xc0036ea0", except once when I
> > got "unknown 16 (0x400)".
> > 
> > I'm assuming this is an mvneta bug that was exposed by 702821f.
> > That's just a guess, and I don't have the skills to debug this any
> > further.  In any case, I figured the maintainers would want to know
> > about it.
> 
> Thanks a lot for the report and the detailed investigation.
> Unfortunately, I don't have Armada 370 hardware with me this week, so
> I'm unable to test and reproduce the issue.
> 
> However, I've added a bunch of Armada 370 people/maintainers in Cc,
> hopefully they can at least try to reproduce and confirm that reverting
> this patch makes the problem go away, which would confirm that we
> should look for a bug in the mvneta driver around this problem.

I'm currently testing on 3.11.1 (which I had here) and am not getting
any issue after 50M packets. My kernel is running in thumb mode and
without SMP.

Ethan, we'll need your config I guess.

Thanks,
Willy

  reply	other threads:[~2013-09-16  6:50 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2013-09-15  1:05 mvneta: oops in __rcu_read_lock on mirabox Ethan Tuttle
2013-09-15  1:05 ` Ethan Tuttle
2013-09-15 18:57 ` Thomas Petazzoni
2013-09-15 18:57   ` Thomas Petazzoni
2013-09-16  6:50   ` Willy Tarreau [this message]
2013-09-16  6:50     ` Willy Tarreau
2013-09-16  8:56     ` Ethan Tuttle
2013-09-16  8:56       ` Ethan Tuttle
2013-09-16 15:51     ` Thomas Petazzoni
2013-09-16 15:51       ` Thomas Petazzoni
2013-09-16 16:22       ` Russell King - ARM Linux
2013-09-16 16:22         ` Russell King - ARM Linux
2013-09-16 16:24         ` Thomas Petazzoni
2013-09-16 16:24           ` Thomas Petazzoni
2013-09-16 17:14           ` Russell King - ARM Linux
2013-09-16 17:14             ` Russell King - ARM Linux
2013-09-16 17:45             ` Willy Tarreau
2013-09-16 17:45               ` Willy Tarreau
2013-09-16 18:25               ` Russell King - ARM Linux
2013-09-16 18:25                 ` Russell King - ARM Linux
2013-09-16 16:35       ` Ethan Tuttle
2013-09-16 16:35         ` Ethan Tuttle
2013-09-16 16:39         ` Willy Tarreau
2013-09-16 16:39           ` Willy Tarreau
2013-09-16 16:44           ` Willy Tarreau
2013-09-16 16:44             ` Willy Tarreau
2013-09-16 17:24             ` Ethan Tuttle
2013-09-16 17:24               ` Ethan Tuttle
2013-09-16 17:47               ` Willy Tarreau
2013-09-16 17:47                 ` Willy Tarreau
2013-09-16 18:28                 ` Russell King - ARM Linux
2013-09-16 18:28                   ` Russell King - ARM Linux
2013-09-17  3:43                   ` Ethan Tuttle
2013-09-17  3:43                     ` Ethan Tuttle
2013-09-17  6:01                     ` Willy Tarreau
2013-09-17  6:01                       ` Willy Tarreau
2013-09-18  6:30                       ` Ethan Tuttle
2013-09-18  6:30                         ` Ethan Tuttle
2013-09-18 16:35                         ` Thomas Petazzoni
2013-09-18 16:35                           ` Thomas Petazzoni
2013-09-18 16:49                           ` Willy Tarreau
2013-09-18 16:49                             ` Willy Tarreau

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20130916065047.GH27487@1wt.eu \
    --to=w@1wt.eu \
    --cc=linux-arm-kernel@lists.infradead.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.