From: Dave Jones <davej@redhat.com>
To: Steven Rostedt <rostedt@goodmis.org>
Cc: Linux Kernel <linux-kernel@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
Frederic Weisbecker <fweisbec@gmail.com>,
Ingo Molnar <mingo@kernel.org>
Subject: Re: trinity finds ftrace/perf bug. Film at 11.
Date: Thu, 12 Sep 2013 14:38:49 -0400 [thread overview]
Message-ID: <20130912183849.GA16674@redhat.com> (raw)
In-Reply-To: <20130912141913.2ffc7077@gandalf.local.home>
On Thu, Sep 12, 2013 at 02:19:13PM -0400, Steven Rostedt wrote:
>
> This is something that either Peter or Frederic need to look at.
I spent yesterday trying to bisect this, but kept running into other perf/ftrace traces,
which makes it pretty much impossible.
The current one I'm staring at is this from LIST_DEBUG..
list_del corruption. next->prev should be prev (ffff88000fb812b0), but was ffff88000fb812b0. (next=ffff88009df8b7b0).
The sharp eyed will notice that those first two addresses are actually the same.
So we matched the next->prev != prev test, but at the time we got to the printk that follows
in the WARN, the inverse was true. Smells like a race of some kind.
[ 2481.228418] Call Trace:
[ 2481.232705] [<ffffffff8171f5dd>] dump_stack+0x54/0x74
[ 2481.237025] [<ffffffff810540bd>] warn_slowpath_common+0x7d/0xa0
[ 2481.241357] [<ffffffff8105412c>] warn_slowpath_fmt+0x4c/0x50
[ 2481.245637] [<ffffffff810540e5>] ? warn_slowpath_fmt+0x5/0x50
[ 2481.249890] [<ffffffff8132c222>] __list_del_entry+0x62/0xf0
[ 2481.254109] [<ffffffff81143a88>] list_del_event+0xc8/0xe0
[ 2481.258310] [<ffffffff81143b2b>] perf_remove_from_context+0x8b/0xf0
[ 2481.262497] [<ffffffff81145444>] perf_event_release_kernel+0x54/0x90
[ 2481.266688] [<ffffffff811464d3>] put_event+0x133/0x1c0
[ 2481.270866] [<ffffffff811463d0>] ? put_event+0x30/0x1c0
[ 2481.275066] [<ffffffff81146570>] perf_release+0x10/0x20
[ 2481.279277] [<ffffffff811bf2ca>] __fput+0xfa/0x2e0
[ 2481.283488] [<ffffffff811bf4fe>] ____fput+0xe/0x10
[ 2481.287713] [<ffffffff8107c11c>] task_work_run+0xac/0xe0
[ 2481.291958] [<ffffffff810559bc>] do_exit+0x2cc/0xcc0
[ 2481.296194] [<ffffffff81732880>] ? ftrace_call+0x5/0x2f
[ 2481.300435] [<ffffffff810577cc>] do_group_exit+0x4c/0xc0
[ 2481.304654] [<ffffffff81057854>] SyS_exit_group+0x14/0x20
[ 2481.308857] [<ffffffff81732e54>] tracesys+0xdd/0xe2
[ 2481.313051] ---[ end trace fa3e551eb20e0c0f ]---
Dave
next prev parent reply other threads:[~2013-09-12 18:39 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-09-11 13:54 trinity finds ftrace/perf bug. Film at 11 Dave Jones
2013-09-12 18:19 ` Steven Rostedt
2013-09-12 18:38 ` Dave Jones [this message]
2013-10-03 15:44 ` Peter Zijlstra
2013-10-03 16:57 ` Ingo Molnar
2013-09-13 10:56 ` Peter Zijlstra
2013-09-13 14:16 ` Dave Jones
2013-09-16 9:37 ` Peter Zijlstra
2013-09-16 10:25 ` Ingo Molnar
2013-09-16 10:43 ` Peter Zijlstra
2013-09-16 16:45 ` Dave Jones
2013-09-16 16:44 ` Dave Jones
2013-09-18 11:34 ` Peter Zijlstra
2013-09-18 13:22 ` Dave Jones
2013-09-13 14:32 ` Steven Rostedt
2013-09-13 14:58 ` Peter Zijlstra
2013-09-13 11:14 ` Peter Zijlstra
2013-09-13 14:28 ` Dave Jones
2013-09-13 14:58 ` Peter Zijlstra
2013-11-19 19:18 ` [tip:perf/urgent] perf: Remove fragile swevent hlist optimization tip-bot for Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130912183849.GA16674@redhat.com \
--to=davej@redhat.com \
--cc=fweisbec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).