From: Dave Jones <davej@redhat.com>
To: Oleg Nesterov <oleg@redhat.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Linux Kernel <linux-kernel@vger.kernel.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
"Eric W. Biederman" <ebiederm@xmission.com>,
Andrey Vagin <avagin@openvz.org>,
Steven Rostedt <rostedt@goodmis.org>
Subject: Re: frequent softlockups with 3.10rc6.
Date: Wed, 26 Jun 2013 20:22:55 -0400 [thread overview]
Message-ID: <20130627002255.GA16553@redhat.com> (raw)
In-Reply-To: <20130626191853.GA29049@redhat.com>
On Wed, Jun 26, 2013 at 09:18:53PM +0200, Oleg Nesterov wrote:
> On 06/25, Dave Jones wrote:
> >
> > Took a lot longer to trigger this time. (13 hours of runtime).
>
> And _perhaps_ this means that 3.10-rc7 without 8aac6270 needs more
> time to hit the same bug ;)
Ok, that didn't take long. 4 hours in, and I hit it on rc7 with 8aac6270 reverted.
So that's the 2nd commit I've mistakenly blamed for this bug.
Crap. I'm going to have to redo the bisecting, and give it a whole day
at each step to be sure. That's going to take a while.
Anyone got any ideas better than a week of non-stop bisecting ?
What I've gathered so far:
- Only affects two machines I have (both Intel Quad core Haswell, one with SSD, one with hybrid SSD)
- One machine is XFS, the other EXT4.
- When the lockup occurs, it happens on all cores.
- It's nearly always a sync() call that triggers it looking like this..
irq event stamp: 8465043
hardirqs last enabled at (8465042): [<ffffffff816ebc60>] restore_args+0x0/0x30
hardirqs last disabled at (8465043): [<ffffffff816f476a>] apic_timer_interrupt+0x6a/0x80
softirqs last enabled at (8464292): [<ffffffff81054204>] __do_softirq+0x194/0x440
softirqs last disabled at (8464295): [<ffffffff8105466d>] irq_exit+0xcd/0xe0
RIP: 0010:[<ffffffff81054121>] [<ffffffff81054121>] __do_softirq+0xb1/0x440
Call Trace:
<IRQ>
[<ffffffff8105466d>] irq_exit+0xcd/0xe0
[<ffffffff816f560b>] smp_apic_timer_interrupt+0x6b/0x9b
[<ffffffff816f476f>] apic_timer_interrupt+0x6f/0x80
<EOI>
[<ffffffff816ebc60>] ? retint_restore_args+0xe/0xe
[<ffffffff810b9c56>] ? lock_acquire+0xa6/0x1f0
[<ffffffff811da892>] ? sync_inodes_sb+0x1c2/0x2a0
[<ffffffff816eaba0>] _raw_spin_lock+0x40/0x80
[<ffffffff811da892>] ? sync_inodes_sb+0x1c2/0x2a0
[<ffffffff811da892>] sync_inodes_sb+0x1c2/0x2a0
[<ffffffff816e8206>] ? wait_for_completion+0x36/0x110
[<ffffffff811e04f0>] ? generic_write_sync+0x70/0x70
[<ffffffff811e0509>] sync_inodes_one_sb+0x19/0x20
[<ffffffff811b0e62>] iterate_supers+0xb2/0x110
[<ffffffff811e0775>] sys_sync+0x35/0x90
[<ffffffff816f3d14>] tracesys+0xdd/0xe2
I'll work on trying to narrow down what trinity is doing. That might at least
make it easier to reproduce it in a shorter timeframe.
Dave
next prev parent reply other threads:[~2013-06-27 0:23 UTC|newest]
Thread overview: 93+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-06-19 16:45 frequent softlockups with 3.10rc6 Dave Jones
2013-06-19 17:53 ` Dave Jones
2013-06-19 18:13 ` Paul E. McKenney
2013-06-19 18:42 ` Dave Jones
2013-06-20 0:12 ` Dave Jones
2013-06-20 16:16 ` Paul E. McKenney
2013-06-20 16:27 ` Dave Jones
2013-06-21 15:11 ` Dave Jones
2013-06-21 19:59 ` Oleg Nesterov
2013-06-22 1:37 ` Dave Jones
2013-06-22 17:31 ` Oleg Nesterov
2013-06-22 21:59 ` Dave Jones
2013-06-23 5:00 ` Andrew Vagin
2013-06-23 14:36 ` Oleg Nesterov
2013-06-23 15:06 ` Dave Jones
2013-06-23 16:04 ` Oleg Nesterov
2013-06-24 0:21 ` Dave Jones
2013-06-24 2:00 ` Dave Jones
2013-06-24 14:39 ` Oleg Nesterov
2013-06-24 14:52 ` Steven Rostedt
2013-06-24 16:00 ` Dave Jones
2013-06-24 16:24 ` Steven Rostedt
2013-06-24 16:51 ` Dave Jones
2013-06-24 17:04 ` Steven Rostedt
2013-06-25 16:55 ` Dave Jones
2013-06-25 17:21 ` Steven Rostedt
2013-06-25 17:23 ` Steven Rostedt
2013-06-25 17:26 ` Dave Jones
2013-06-25 17:31 ` Steven Rostedt
2013-06-25 17:32 ` Steven Rostedt
2013-06-25 17:29 ` Steven Rostedt
2013-06-25 17:34 ` Dave Jones
2013-06-24 16:37 ` Oleg Nesterov
2013-06-24 16:49 ` Dave Jones
2013-06-24 15:57 ` Dave Jones
2013-06-24 17:35 ` Oleg Nesterov
2013-06-24 17:44 ` Dave Jones
2013-06-24 17:53 ` Steven Rostedt
2013-06-24 18:00 ` Dave Jones
2013-06-25 15:35 ` Dave Jones
2013-06-25 16:23 ` Steven Rostedt
2013-06-26 5:23 ` Dave Jones
2013-06-26 19:52 ` Steven Rostedt
2013-06-26 20:00 ` Dave Jones
2013-06-27 3:01 ` Steven Rostedt
2013-06-26 5:48 ` Dave Jones
2013-06-26 19:18 ` Oleg Nesterov
2013-06-26 19:40 ` Dave Jones
2013-06-27 0:22 ` Dave Jones [this message]
2013-06-27 1:06 ` Eric W. Biederman
2013-06-27 2:32 ` Tejun Heo
2013-06-27 7:55 ` Dave Chinner
2013-06-27 10:06 ` Dave Chinner
2013-06-27 12:52 ` Dave Chinner
2013-06-27 15:21 ` Dave Jones
2013-06-28 1:13 ` Dave Chinner
2013-06-28 3:58 ` Dave Chinner
2013-06-28 10:28 ` Jan Kara
2013-06-29 3:39 ` Dave Chinner
2013-07-01 12:00 ` Jan Kara
2013-07-02 6:29 ` Dave Chinner
2013-07-02 8:19 ` Jan Kara
2013-07-02 12:38 ` Dave Chinner
2013-07-02 14:05 ` Jan Kara
2013-07-02 16:13 ` Linus Torvalds
2013-07-02 16:57 ` Jan Kara
2013-07-02 17:38 ` Linus Torvalds
2013-07-03 3:07 ` Dave Chinner
2013-07-03 3:28 ` Linus Torvalds
2013-07-03 4:49 ` Dave Chinner
2013-07-04 7:19 ` Andrew Morton
2013-06-29 20:13 ` Dave Jones
2013-06-29 22:23 ` Linus Torvalds
2013-06-29 23:44 ` Dave Jones
2013-06-30 0:21 ` Steven Rostedt
2013-07-01 12:49 ` Pavel Machek
2013-06-30 0:17 ` Steven Rostedt
2013-06-30 2:05 ` Dave Chinner
2013-06-30 2:34 ` Dave Chinner
2013-06-27 14:30 ` Dave Jones
2013-06-28 1:18 ` Dave Chinner
2013-06-28 2:54 ` Linus Torvalds
2013-06-28 3:54 ` Dave Chinner
2013-06-28 5:59 ` Linus Torvalds
2013-06-28 7:21 ` Dave Chinner
2013-06-28 8:22 ` Linus Torvalds
2013-06-28 8:32 ` Al Viro
2013-06-28 8:22 ` Al Viro
2013-06-28 9:49 ` Jan Kara
2013-07-01 17:57 ` block layer softlockup Dave Jones
2013-07-02 2:07 ` Dave Chinner
2013-07-02 6:01 ` Dave Jones
2013-07-02 7:30 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130627002255.GA16553@redhat.com \
--to=davej@redhat.com \
--cc=avagin@openvz.org \
--cc=ebiederm@xmission.com \
--cc=linux-kernel@vger.kernel.org \
--cc=oleg@redhat.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=rostedt@goodmis.org \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).