From: Dave Jones <davej@redhat.com>
To: Oleg Nesterov <oleg@redhat.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Linux Kernel <linux-kernel@vger.kernel.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
"Eric W. Biederman" <ebiederm@xmission.com>,
Andrey Vagin <avagin@openvz.org>
Subject: Re: frequent softlockups with 3.10rc6.
Date: Sun, 23 Jun 2013 11:06:03 -0400 [thread overview]
Message-ID: <20130623150603.GA32313@redhat.com> (raw)
In-Reply-To: <20130623143634.GA2000@redhat.com>
On Sun, Jun 23, 2013 at 04:36:34PM +0200, Oleg Nesterov wrote:
> > > Dave, I am sorry but all I can do is to ask you to do more testing.
> > > Could you please reproduce the lockup again on the clean Linus's
> > > current ? (and _without_ reverting 8aac6270, of course).
> >
> > I'll give it a shot. Just rebuilt clean tree, and restarted the tests.
>
> Thanks a lot.
ok, hit it on rc7 without the revert
[11018.927809] [sched_delayed] sched: RT throttling activated
[11054.897670] BUG: soft lockup - CPU#2 stuck for 22s! [trinity-child2:14482]
[11054.898503] Modules linked in: bridge stp snd_seq_dummy tun fuse hidp bnep rfcomm can_raw ipt_ULOG can_bcm nfnetlink af_rxrpc llc2 rose caif_socket caif can netrom appletalk af_802154 scsi_transport_iscsi nfc pppoe pppox ppp_generic slhc ipx p8023 psnap p8022 llc ax25 irda crc_ccitt af_key bluetooth rfkill x25 rds atm phonet coretemp hwmon kvm_intel kvm snd_hda_codec_realtek crc32c_intel ghash_clmulni_intel snd_hda_codec_hdmi microcode snd_hda_intel snd_hda_codec pcspkr snd_hwdep snd_seq snd_seq_device snd_pcm e1000e snd_page_alloc ptp snd_timer pps_core snd soundcore xfs libcrc32c
[11054.905490] irq event stamp: 3857095
[11054.905926] hardirqs last enabled at (3857094): [<ffffffff816ed9a0>] restore_args+0x0/0x30
[11054.906945] hardirqs last disabled at (3857095): [<ffffffff816f64aa>] apic_timer_interrupt+0x6a/0x80
[11054.908054] softirqs last enabled at (3856322): [<ffffffff810542e4>] __do_softirq+0x194/0x440
[11054.909102] softirqs last disabled at (3856325): [<ffffffff8105474d>] irq_exit+0xcd/0xe0
[11054.910088] CPU: 2 PID: 14482 Comm: trinity-child2 Not tainted 3.10.0-rc7+ #31
[11054.912900] task: ffff8801ae44ca40 ti: ffff88021fe60000 task.ti: ffff88021fe60000
[11054.913800] RIP: 0010:[<ffffffff81054201>] [<ffffffff81054201>] __do_softirq+0xb1/0x440
[11054.914786] RSP: 0018:ffff880244c03f08 EFLAGS: 00000206
[11054.915428] RAX: ffff8801ae44ca40 RBX: ffffffff816ed9a0 RCX: 0000000000000000
[11054.916286] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8801ae44ca40
[11054.917143] RBP: ffff880244c03f70 R08: 0000000000000000 R09: 0000000000000000
[11054.918002] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880244c03e78
[11054.919799] R13: ffffffff816f64af R14: ffff880244c03f70 R15: 0000000000000000
[11054.921601] FS: 00007f36e952f740(0000) GS:ffff880244c00000(0000) knlGS:0000000000000000
[11054.923529] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[11054.925182] CR2: 0000003850a74cf0 CR3: 00000002081b9000 CR4: 00000000001407e0
[11054.927009] DR0: 0000000001550000 DR1: 0000000000000000 DR2: 0000000000000000
[11054.928830] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000600
[11054.930643] Stack:
[11054.931862] 0000000a00406040 0000000100106b66 ffff88021fe61fd8 ffff88021fe61fd8
[11054.933777] ffff88021fe61fd8 ffff8801ae44ce38 ffff88021fe61fd8 ffffffff00000002
[11054.935694] ffff8801ae44ca40 0000000000000000 ffffea0008f49280 ffff8802434cc240
[11054.937617] Call Trace:
[11054.938916] <IRQ>
[11054.940347] [<ffffffff8105474d>] irq_exit+0xcd/0xe0
[11054.941790] [<ffffffff816f734b>] smp_apic_timer_interrupt+0x6b/0x9b
[11054.943550] [<ffffffff816f64af>] apic_timer_interrupt+0x6f/0x80
[11054.945270] <EOI>
[11054.946705] [<ffffffff816ed9a0>] ? retint_restore_args+0xe/0xe
[11054.948267] [<ffffffff816ecd67>] ? _raw_spin_unlock_irqrestore+0x67/0x80
[11054.950090] [<ffffffff816e28ec>] __slab_free+0x5f/0x39f
[11054.951737] [<ffffffff816ecd75>] ? _raw_spin_unlock_irqrestore+0x75/0x80
[11054.953556] [<ffffffff8131506e>] ? debug_check_no_obj_freed+0x14e/0x250
[11054.955362] [<ffffffff81199335>] ? kmem_cache_free+0x95/0x300
[11054.957065] [<ffffffff8119958c>] kmem_cache_free+0x2ec/0x300
[11054.958759] [<ffffffff81047104>] ? __put_task_struct+0x64/0x140
[11054.960485] [<ffffffff81047104>] __put_task_struct+0x64/0x140
[11054.962184] [<ffffffff81086d4f>] finish_task_switch+0x11f/0x130
[11054.963899] [<ffffffff81086c77>] ? finish_task_switch+0x47/0x130
[11054.965632] [<ffffffff816eae24>] __schedule+0x444/0xa40
[11054.967271] [<ffffffff816eba83>] preempt_schedule_irq+0x53/0x90
[11054.968994] [<ffffffff816edab6>] retint_kernel+0x26/0x30
[11054.970656] [<ffffffff81145877>] ? user_enter+0x87/0xd0
[11054.972305] [<ffffffff8100f6a8>] syscall_trace_leave+0x78/0x140
[11054.974029] [<ffffffff816f5b2f>] int_check_syscall_exit_work+0x34/0x3d
[11054.975819] Code: 48 89 45 b8 48 89 45 b0 48 89 45 a8 66 0f 1f 44 00 00 65 c7 04 25 80 0f 1d 00 00 00 00 00 e8 a7 35 06 00 fb 49 c7 c6 00 41 c0 81 <eb> 0e 0f 1f 44 00 00 49 83 c6 08 41 d1 ef 74 6c 41 f6 c7 01 74
next prev parent reply other threads:[~2013-06-23 15:06 UTC|newest]
Thread overview: 93+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-06-19 16:45 frequent softlockups with 3.10rc6 Dave Jones
2013-06-19 17:53 ` Dave Jones
2013-06-19 18:13 ` Paul E. McKenney
2013-06-19 18:42 ` Dave Jones
2013-06-20 0:12 ` Dave Jones
2013-06-20 16:16 ` Paul E. McKenney
2013-06-20 16:27 ` Dave Jones
2013-06-21 15:11 ` Dave Jones
2013-06-21 19:59 ` Oleg Nesterov
2013-06-22 1:37 ` Dave Jones
2013-06-22 17:31 ` Oleg Nesterov
2013-06-22 21:59 ` Dave Jones
2013-06-23 5:00 ` Andrew Vagin
2013-06-23 14:36 ` Oleg Nesterov
2013-06-23 15:06 ` Dave Jones [this message]
2013-06-23 16:04 ` Oleg Nesterov
2013-06-24 0:21 ` Dave Jones
2013-06-24 2:00 ` Dave Jones
2013-06-24 14:39 ` Oleg Nesterov
2013-06-24 14:52 ` Steven Rostedt
2013-06-24 16:00 ` Dave Jones
2013-06-24 16:24 ` Steven Rostedt
2013-06-24 16:51 ` Dave Jones
2013-06-24 17:04 ` Steven Rostedt
2013-06-25 16:55 ` Dave Jones
2013-06-25 17:21 ` Steven Rostedt
2013-06-25 17:23 ` Steven Rostedt
2013-06-25 17:26 ` Dave Jones
2013-06-25 17:31 ` Steven Rostedt
2013-06-25 17:32 ` Steven Rostedt
2013-06-25 17:29 ` Steven Rostedt
2013-06-25 17:34 ` Dave Jones
2013-06-24 16:37 ` Oleg Nesterov
2013-06-24 16:49 ` Dave Jones
2013-06-24 15:57 ` Dave Jones
2013-06-24 17:35 ` Oleg Nesterov
2013-06-24 17:44 ` Dave Jones
2013-06-24 17:53 ` Steven Rostedt
2013-06-24 18:00 ` Dave Jones
2013-06-25 15:35 ` Dave Jones
2013-06-25 16:23 ` Steven Rostedt
2013-06-26 5:23 ` Dave Jones
2013-06-26 19:52 ` Steven Rostedt
2013-06-26 20:00 ` Dave Jones
2013-06-27 3:01 ` Steven Rostedt
2013-06-26 5:48 ` Dave Jones
2013-06-26 19:18 ` Oleg Nesterov
2013-06-26 19:40 ` Dave Jones
2013-06-27 0:22 ` Dave Jones
2013-06-27 1:06 ` Eric W. Biederman
2013-06-27 2:32 ` Tejun Heo
2013-06-27 7:55 ` Dave Chinner
2013-06-27 10:06 ` Dave Chinner
2013-06-27 12:52 ` Dave Chinner
2013-06-27 15:21 ` Dave Jones
2013-06-28 1:13 ` Dave Chinner
2013-06-28 3:58 ` Dave Chinner
2013-06-28 10:28 ` Jan Kara
2013-06-29 3:39 ` Dave Chinner
2013-07-01 12:00 ` Jan Kara
2013-07-02 6:29 ` Dave Chinner
2013-07-02 8:19 ` Jan Kara
2013-07-02 12:38 ` Dave Chinner
2013-07-02 14:05 ` Jan Kara
2013-07-02 16:13 ` Linus Torvalds
2013-07-02 16:57 ` Jan Kara
2013-07-02 17:38 ` Linus Torvalds
2013-07-03 3:07 ` Dave Chinner
2013-07-03 3:28 ` Linus Torvalds
2013-07-03 4:49 ` Dave Chinner
2013-07-04 7:19 ` Andrew Morton
2013-06-29 20:13 ` Dave Jones
2013-06-29 22:23 ` Linus Torvalds
2013-06-29 23:44 ` Dave Jones
2013-06-30 0:21 ` Steven Rostedt
2013-07-01 12:49 ` Pavel Machek
2013-06-30 0:17 ` Steven Rostedt
2013-06-30 2:05 ` Dave Chinner
2013-06-30 2:34 ` Dave Chinner
2013-06-27 14:30 ` Dave Jones
2013-06-28 1:18 ` Dave Chinner
2013-06-28 2:54 ` Linus Torvalds
2013-06-28 3:54 ` Dave Chinner
2013-06-28 5:59 ` Linus Torvalds
2013-06-28 7:21 ` Dave Chinner
2013-06-28 8:22 ` Linus Torvalds
2013-06-28 8:32 ` Al Viro
2013-06-28 8:22 ` Al Viro
2013-06-28 9:49 ` Jan Kara
2013-07-01 17:57 ` block layer softlockup Dave Jones
2013-07-02 2:07 ` Dave Chinner
2013-07-02 6:01 ` Dave Jones
2013-07-02 7:30 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20130623150603.GA32313@redhat.com \
--to=davej@redhat.com \
--cc=avagin@openvz.org \
--cc=ebiederm@xmission.com \
--cc=linux-kernel@vger.kernel.org \
--cc=oleg@redhat.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).