From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753202Ab1JXTCN (ORCPT ); Mon, 24 Oct 2011 15:02:13 -0400 Received: from peace.netnation.com ([204.174.223.2]:46568 "EHLO peace.netnation.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752119Ab1JXTCL (ORCPT ); Mon, 24 Oct 2011 15:02:11 -0400 Date: Mon, 24 Oct 2011 12:02:03 -0700 From: Simon Kirby To: Thomas Gleixner , David Miller Cc: Peter Zijlstra , Linus Torvalds , Linux Kernel Mailing List , Dave Jones , Martin Schwidefsky , Ingo Molnar Subject: Re: Linux 3.1-rc9 Message-ID: <20111024190203.GA24410@hostway.ca> References: <1318847658.6594.40.camel@twins> <1318874090.4172.84.camel@twins> <1318879396.4172.92.camel@twins> <1318928713.21167.4.camel@twins> <20111018182046.GF1309@hostway.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Oct 18, 2011 at 09:48:51PM +0200, Thomas Gleixner wrote: > On Tue, 18 Oct 2011, Simon Kirby wrote: > > Looks good running on three boxes since this morning (unpatched kernel > > hangs in ~15 minutes). > > > > While I have your eyes, does this hang trace make any sense (which > > happened a couple of times with your previous patch applied)? > > > > http://0x.ca/sim/ref/3.1-rc9/3.1-rc9-tcp-lockup.log > > > > I don't see how all CPUs could be spinning on the same lock without > > reentry, and I don't see the any in the backtraces. > > Weird. > > Which version of Peters patches was this, the extra lock or the > atomic64 thingy? The first one with the extra lock. I never tried the atomic64 one. Anyway, that's fixed now. > It does not look related. Could you try to reproduce that problem with > lockdep enabled? lockdep might make it go away, but it's definitely > worth a try. Trying now... ...Whoops, never sent this email. Ok, hit the hang about 4 more times, but only this morning on a box with a serial cable attached. Yay! Simon- [216695.579770] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216695.589435] [216695.589437] ======================================================= [216695.593380] [ INFO: possible circular locking dependency detected ] [216695.593380] 3.1.0-rc10-hw-lockdep+ #51 [216695.593380] ------------------------------------------------------- [216695.593380] kworker/0:1/0 is trying to acquire lock: [216695.593380] (&icsk->icsk_retransmit_timer){+.-.-.}, at: [] run_timer_softirq+0x198/0x410 [216695.593380] [216695.593380] but task is already holding lock: [216695.593380] (slock-AF_INET){+.-.-.}, at: [] sk_clone+0x120/0x420 [216695.593380] [216695.593380] which lock already depends on the new lock. [216695.593380] [216695.593380] [216695.593380] the existing dependency chain (in reverse order) is: [216695.593380] [216695.593380] -> #1 (slock-AF_INET){+.-.-.}: [216695.593380] [] lock_acquire+0x109/0x140 [216695.593380] [] _raw_spin_lock+0x3c/0x50 [216695.593380] [] tcp_write_timer+0x23/0x230 [216695.682901] [] run_timer_softirq+0x26d/0x410 [216695.682901] [] __do_softirq+0x138/0x250 [216695.682901] [] call_softirq+0x1c/0x30 [216695.682901] [] do_softirq+0x95/0xd0 [216695.682901] [] irq_exit+0xdd/0x110 [216695.682901] [] smp_apic_timer_interrupt+0x69/0xa0 [216695.682901] [] apic_timer_interrupt+0x73/0x80 [216695.682901] [] cpu_idle+0x96/0xf0 [216695.682901] [] start_secondary+0x1ca/0x1ff [216695.682901] [216695.682901] -> #0 (&icsk->icsk_retransmit_timer){+.-.-.}: [216695.682901] [] __lock_acquire+0x2040/0x2180 [216695.682901] [] lock_acquire+0x109/0x140 [216695.682901] [] run_timer_softirq+0x219/0x410 [216695.682901] [] __do_softirq+0x138/0x250 [216695.682901] [] call_softirq+0x1c/0x30 [216695.682901] [] do_softirq+0x95/0xd0 [216695.682901] [] irq_exit+0xdd/0x110 [216695.682901] [] do_IRQ+0x64/0xe0 [216695.682901] [] ret_from_intr+0x0/0x1a [216695.682901] [] cpu_idle+0x96/0xf0 [216695.682901] [] start_secondary+0x1ca/0x1ff [216695.682901] [216695.682901] other info that might help us debug this: [216695.682901] [216695.682901] Possible unsafe locking scenario: [216695.682901] [216695.682901] CPU0 CPU1 [216695.682901] ---- ---- [216695.682901] lock(slock-AF_INET); [216695.682901] lock(&icsk->icsk_retransmit_timer); [216695.682901] lock(slock-AF_INET); [216695.682901] lock(&icsk->icsk_retransmit_timer); [216695.682901] [216695.682901] *** DEADLOCK *** [216695.682901] [216695.682901] 1 lock held by kworker/0:1/0: [216695.682901] #0: (slock-AF_INET){+.-.-.}, at: [] sk_clone+0x120/0x420 [216695.682901] [216695.682901] stack backtrace: [216695.682901] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 [216695.682901] Call Trace: [216695.682901] [] print_circular_bug+0x21b/0x330 [216695.682901] [] __lock_acquire+0x2040/0x2180 [216695.682901] [] ? nf_conntrack_free+0x42/0x50 [nf_conntrack] [216695.682901] [] ? nf_conntrack_free+0x42/0x50 [nf_conntrack] [216695.682901] [] lock_acquire+0x109/0x140 [216695.682901] [] ? run_timer_softirq+0x198/0x410 [216695.682901] [] ? trace_hardirqs_on_caller+0x7c/0x1c0 [216695.682901] [] run_timer_softirq+0x219/0x410 [216695.682901] [] ? run_timer_softirq+0x198/0x410 [216695.682901] [] ? printk+0x67/0x69 [216695.682901] [] ? tcp_delack_timer+0x230/0x230 [216695.682901] [] __do_softirq+0x138/0x250 [216695.682901] [] call_softirq+0x1c/0x30 [216695.682901] [] do_softirq+0x95/0xd0 [216695.682901] [] irq_exit+0xdd/0x110 [216695.682901] [] do_IRQ+0x64/0xe0 [216695.682901] [] common_interrupt+0x73/0x73 [216695.682901] [] ? mwait_idle+0x14e/0x170 [216695.682901] [] ? mwait_idle+0x145/0x170 [216695.682901] [] cpu_idle+0x96/0xf0 [216695.682901] [] start_secondary+0x1ca/0x1ff [216696.019296] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000105? [216697.762956] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216698.597297] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216701.489681] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216701.667999] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216704.580592] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [216709.468971] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [216712.845904] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216716.588502] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [216725.072958] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [216725.603879] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216725.828374] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216727.588978] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216735.513864] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216740.581530] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [216756.278571] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [218855.312903] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [218855.323133] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [218858.293355] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [218864.301938] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [218876.333821] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [218885.332651] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [218900.313590] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [220821.012017] TCP: Peer 32.176.160.153:49226/80 unexpectedly shrunk window 665256753:665268993 (repaired) [221075.224300] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221075.234579] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221075.277593] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221075.780515] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221075.780713] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221077.349279] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221077.905587] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [221077.915567] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [221081.498430] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221081.703277] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [221082.088513] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221082.167985] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221089.772578] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221090.487927] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [221090.686394] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221094.587131] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221105.255699] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000101, exited with 00000102? [221105.280699] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221105.291634] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221106.325794] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221107.286029] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221107.622736] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221107.734471] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [221120.381643] huh, entered softirq 3 NET_RX ffffffff81613740 preempt_count 00000102, exited with 00000103? [223936.264020] BUG: soft lockup - CPU#0 stuck for 22s! [swapper:0] [223936.268002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223936.268002] irq event stamp: 2595159887 [223936.268002] hardirqs last enabled at (2595159887): [] mwait_idle+0x145/0x170 [223936.268002] hardirqs last disabled at (2595159886): [] cpu_idle+0x79/0xf0 [223936.268002] softirqs last enabled at (2595159878): [] __do_softirq+0x1f6/0x250 [223936.268002] softirqs last disabled at (2595159873): [] call_softirq+0x1c/0x30 [223936.268002] CPU 0 [223936.268002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223936.268002] [223936.268002] Pid: 0, comm: swapper Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223936.268002] RIP: 0010:[] [] delay_tsc+0x73/0xd0 [223936.268002] RSP: 0018:ffff88022fc03cf0 EFLAGS: 00000202 [223936.268002] RAX: 00017b5d5932dd02 RBX: ffffffff816f6334 RCX: 000000005932dd02 [223936.372028] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/0:0:0] [223936.372031] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223936.372042] irq event stamp: 2598787699 [223936.372044] hardirqs last enabled at (2598787699): [] mwait_idle+0x145/0x170 [223936.372054] hardirqs last disabled at (2598787698): [] cpu_idle+0x79/0xf0 [223936.372058] softirqs last enabled at (2598787696): [] __do_softirq+0x1f6/0x250 [223936.372063] softirqs last disabled at (2598787681): [] call_softirq+0x1c/0x30 [223936.372069] CPU 1 [223936.372070] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223936.372079] [223936.372081] Pid: 0, comm: kworker/0:0 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223936.372086] RIP: 0010:[] [] native_read_tsc+0xb/0x20 [223936.372091] RSP: 0018:ffff88022fc43ce0 EFLAGS: 00000202 [223936.372093] RAX: 0000000000017b5d RBX: ffffffff816f6334 RCX: 00000000652f810e [223936.372096] RDX: 0000000000017b5d RSI: 0000000000000000 RDI: 0000000000000001 [223936.372098] RBP: ffff88022fc43ce0 R08: 00000000652f80c8 R09: 0000000000000000 [223936.372101] R10: ffff88022690dd60 R11: 0000000000000000 R12: ffff88022fc43c58 [223936.372103] R13: ffffffff816feb33 R14: ffff88022fc43ce0 R15: 00000000180bbeb8 [223936.372106] FS: 0000000000000000(0000) GS:ffff88022fc40000(0000) knlGS:0000000000000000 [223936.372108] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223936.372111] CR2: 00007f472ba6a6fc CR3: 0000000126bb7000 CR4: 00000000000006e0 [223936.372113] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223936.372116] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223936.372119] Process kworker/0:0 (pid: 0, threadinfo ffff880226928000, task ffff88022690dd60) [223936.372121] Stack: [223936.372123] ffff88022fc43d30 ffffffff813a4eaf ffff880226928000 00000000652f8090 [223936.372128] 000000012fc43d18 ffff88002e90e348 00000000180bbeb8 000000006efcdc62 [223936.372132] 0000000000000001 ffff88022690dd60 ffff88022fc43d40 ffffffff813a4f6a [223936.372136] Call Trace: [223936.372139] [223936.372144] [] delay_tsc+0x3f/0xd0 [223936.372148] [] __delay+0xa/0x10 [223936.372153] [] do_raw_spin_lock+0x13b/0x180 [223936.372159] [] _raw_spin_lock+0x44/0x50 [223936.372164] [] ? tcp_keepalive_timer+0x23/0x270 [223936.372168] [] tcp_keepalive_timer+0x23/0x270 [223936.372174] [] run_timer_softirq+0x26d/0x410 [223936.372178] [] ? run_timer_softirq+0x198/0x410 [223936.372182] [] ? tcp_init_xmit_timers+0x20/0x20 [223936.372186] [] __do_softirq+0x138/0x250 [223936.372190] [] call_softirq+0x1c/0x30 [223936.372194] [] do_softirq+0x95/0xd0 [223936.372198] [] irq_exit+0xdd/0x110 [223936.372203] [] smp_apic_timer_interrupt+0x69/0xa0 [223936.372208] [] apic_timer_interrupt+0x73/0x80 [223936.372210] [223936.372214] [] ? mwait_idle+0x145/0x170 [223936.372218] [] ? mwait_idle+0x14e/0x170 [223936.372222] [] ? mwait_idle+0x145/0x170 [223936.372226] [] cpu_idle+0x96/0xf0 [223936.372230] [] start_secondary+0x1ca/0x1ff [223936.372233] Code: a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 48 89 d0 [223936.372253] c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 [223936.372262] Call Trace: [223936.372264] [] delay_tsc+0x3f/0xd0 [223936.372269] [] __delay+0xa/0x10 [223936.372272] [] do_raw_spin_lock+0x13b/0x180 [223936.372276] [] _raw_spin_lock+0x44/0x50 [223936.372280] [] ? tcp_keepalive_timer+0x23/0x270 [223936.372283] [] tcp_keepalive_timer+0x23/0x270 [223936.372286] [] run_timer_softirq+0x26d/0x410 [223936.372289] [] ? run_timer_softirq+0x198/0x410 [223936.372293] [] ? tcp_init_xmit_timers+0x20/0x20 [223936.372297] [] __do_softirq+0x138/0x250 [223936.372300] [] call_softirq+0x1c/0x30 [223936.372303] [] do_softirq+0x95/0xd0 [223936.372307] [] irq_exit+0xdd/0x110 [223936.372310] [] smp_apic_timer_interrupt+0x69/0xa0 [223936.372313] [] apic_timer_interrupt+0x73/0x80 [223936.372315] [] ? mwait_idle+0x145/0x170 [223936.372321] [] ? mwait_idle+0x14e/0x170 [223936.372324] [] ? mwait_idle+0x145/0x170 [223936.372327] [] cpu_idle+0x96/0xf0 [223936.372331] [] start_secondary+0x1ca/0x1ff [223936.476032] BUG: soft lockup - CPU#2 stuck for 23s! [kworker/0:1:0] [223936.476034] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223936.476043] irq event stamp: 2613824057 [223936.476045] hardirqs last enabled at (2613824057): [] mwait_idle+0x145/0x170 [223936.476050] hardirqs last disabled at (2613824056): [] cpu_idle+0x79/0xf0 [223936.476054] softirqs last enabled at (2613824048): [] __do_softirq+0x1f6/0x250 [223936.476058] softirqs last disabled at (2613824031): [] call_softirq+0x1c/0x30 [223936.476062] CPU 2 [223936.476063] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223936.476071] [223936.476073] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223936.476077] RIP: 0010:[] [] native_read_tsc+0x6/0x20 [223936.476082] RSP: 0018:ffff88022fc83ce0 EFLAGS: 00000202 [223936.476084] RAX: 0000000070ba7dfc RBX: ffffffff813a60ae RCX: 0000000070ba7dc4 [223936.476086] RDX: 0000000000017b5d RSI: 0000000000000000 RDI: 0000000000000001 [223936.476089] RBP: ffff88022fc83ce0 R08: 0000000070ba7d7e R09: 0000000000000000 [223936.476091] R10: ffff880226948000 R11: 0000000000000000 R12: ffff88022fc83c58 [223936.476093] R13: ffffffff816feb33 R14: ffff88022fc83ce0 R15: 00000000182285f9 [223936.476096] FS: 0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000 [223936.476099] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223936.476101] CR2: 00007f25e7874d7f CR3: 0000000124c0d000 CR4: 00000000000006e0 [223936.476104] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223936.476106] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223936.476109] Process kworker/0:1 (pid: 0, threadinfo ffff88022693e000, task ffff880226948000) [223936.476111] Stack: [223936.476113] ffff88022fc83d30 ffffffff813a4ee8 ffff88022693e000 0000000070ba7dc4 [223936.476117] 00000002ffffff10 ffff88006afd8948 00000000182285f9 000000006efcdc62 [223936.476121] 0000000000000001 ffff880226948000 ffff88022fc83d40 ffffffff813a4f6a [223936.476126] Call Trace: [223936.476128] [223936.476132] [] delay_tsc+0x78/0xd0 [223936.476136] [] __delay+0xa/0x10 [223936.476141] [] do_raw_spin_lock+0x13b/0x180 [223936.476147] [] _raw_spin_lock+0x44/0x50 [223936.476153] [] ? tcp_keepalive_timer+0x23/0x270 [223936.476157] [] tcp_keepalive_timer+0x23/0x270 [223936.476163] [] run_timer_softirq+0x26d/0x410 [223936.476167] [] ? run_timer_softirq+0x198/0x410 [223936.476171] [] ? tcp_init_xmit_timers+0x20/0x20 [223936.476176] [] __do_softirq+0x138/0x250 [223936.476180] [] call_softirq+0x1c/0x30 [223936.476184] [] do_softirq+0x95/0xd0 [223936.476187] [] irq_exit+0xdd/0x110 [223936.476193] [] smp_apic_timer_interrupt+0x69/0xa0 [223936.476197] [] apic_timer_interrupt+0x73/0x80 [223936.476199] [223936.476203] [] ? mwait_idle+0x145/0x170 [223936.476207] [] ? mwait_idle+0x14e/0x170 [223936.476211] [] ? mwait_idle+0x145/0x170 [223936.476215] [] cpu_idle+0x96/0xf0 [223936.476219] [] start_secondary+0x1ca/0x1ff [223936.476222] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 [223936.476241] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 [223936.476251] Call Trace: [223936.476252] [] delay_tsc+0x78/0xd0 [223936.476257] [] __delay+0xa/0x10 [223936.476261] [] do_raw_spin_lock+0x13b/0x180 [223936.476265] [] _raw_spin_lock+0x44/0x50 [223936.476268] [] ? tcp_keepalive_timer+0x23/0x270 [223936.476272] [] tcp_keepalive_timer+0x23/0x270 [223936.476275] [] run_timer_softirq+0x26d/0x410 [223936.476278] [] ? run_timer_softirq+0x198/0x410 [223936.476282] [] ? tcp_init_xmit_timers+0x20/0x20 [223936.476286] [] __do_softirq+0x138/0x250 [223936.476289] [] call_softirq+0x1c/0x30 [223936.476292] [] do_softirq+0x95/0xd0 [223936.476295] [] irq_exit+0xdd/0x110 [223936.476299] [] smp_apic_timer_interrupt+0x69/0xa0 [223936.476302] [] apic_timer_interrupt+0x73/0x80 [223936.476304] [] ? mwait_idle+0x145/0x170 [223936.476310] [] ? mwait_idle+0x14e/0x170 [223936.476313] [] ? mwait_idle+0x145/0x170 [223936.476316] [] cpu_idle+0x96/0xf0 [223936.476320] [] start_secondary+0x1ca/0x1ff [223936.580039] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/0:1:0] [223936.580041] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223936.580050] irq event stamp: 2615464042 [223936.580052] hardirqs last enabled at (2615464042): [] _raw_spin_unlock_irq+0x2b/0x50 [223936.580057] hardirqs last disabled at (2615464041): [] _raw_spin_lock_irq+0x18/0x60 [223936.580061] softirqs last enabled at (2615463964): [] _local_bh_enable+0xe/0x10 [223936.580065] softirqs last disabled at (2615463965): [] call_softirq+0x1c/0x30 [223936.580069] CPU 3 [223936.580070] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223936.580078] [223936.580080] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223936.580085] RIP: 0010:[] [] native_read_tsc+0x6/0x20 [223936.580090] RSP: 0018:ffff88022fcc3ce0 EFLAGS: 00000202 [223936.580092] RAX: 000000007c457b06 RBX: ffffffff816f6334 RCX: 000000007c457ad5 [223936.580094] RDX: 0000000000017b5d RSI: ffffffff818f9896 RDI: 0000000000000001 [223936.580097] RBP: ffff88022fcc3ce0 R08: 000000007c457a88 R09: 0000000000000000 [223936.580099] R10: ffff880226981f20 R11: 0000000000000000 R12: ffff88022fcc3c58 [223936.580101] R13: ffffffff816feb33 R14: ffff88022fcc3ce0 R15: 00000000183a1380 [223936.580104] FS: 0000000000000000(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000 [223936.580107] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223936.580109] CR2: 0000000000f38820 CR3: 0000000104b52000 CR4: 00000000000006e0 [223936.580112] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223936.580114] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223936.580117] Process kworker/0:1 (pid: 0, threadinfo ffff880226988000, task ffff880226981f20) [223936.580119] Stack: [223936.580120] ffff88022fcc3d30 ffffffff813a4ee8 ffff880226988000 000000007c457ad5 [223936.580125] 00000003ffffff10 ffff880031438948 00000000183a1380 000000006efcdc62 [223936.580129] 0000000000000001 ffff880226981f20 ffff88022fcc3d40 ffffffff813a4f6a [223936.580133] Call Trace: [223936.580135] [223936.580138] [] delay_tsc+0x78/0xd0 [223936.580142] [] __delay+0xa/0x10 [223936.580147] [] do_raw_spin_lock+0x13b/0x180 [223936.580151] [] _raw_spin_lock+0x44/0x50 [223936.580156] [] ? tcp_keepalive_timer+0x23/0x270 [223936.580160] [] tcp_keepalive_timer+0x23/0x270 [223936.580164] [] run_timer_softirq+0x26d/0x410 [223936.580168] [] ? run_timer_softirq+0x198/0x410 [223936.580172] [] ? tcp_init_xmit_timers+0x20/0x20 [223936.580176] [] __do_softirq+0x138/0x250 [223936.580181] [] call_softirq+0x1c/0x30 [223936.580185] [] do_softirq+0x95/0xd0 [223936.580188] [] irq_exit+0xdd/0x110 [223936.580192] [] smp_apic_timer_interrupt+0x69/0xa0 [223936.580196] [] apic_timer_interrupt+0x73/0x80 [223936.580199] [223936.580202] [] ? mwait_idle+0x145/0x170 [223936.580206] [] ? mwait_idle+0x14e/0x170 [223936.580211] [] ? mwait_idle+0x145/0x170 [223936.580214] [] cpu_idle+0x96/0xf0 [223936.580218] [] start_secondary+0x1ca/0x1ff [223936.580221] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 [223936.580240] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 [223936.580250] Call Trace: [223936.580251] [] delay_tsc+0x78/0xd0 [223936.580256] [] __delay+0xa/0x10 [223936.580260] [] do_raw_spin_lock+0x13b/0x180 [223936.580264] [] _raw_spin_lock+0x44/0x50 [223936.580267] [] ? tcp_keepalive_timer+0x23/0x270 [223936.580270] [] tcp_keepalive_timer+0x23/0x270 [223936.580274] [] run_timer_softirq+0x26d/0x410 [223936.580277] [] ? run_timer_softirq+0x198/0x410 [223936.580280] [] ? tcp_init_xmit_timers+0x20/0x20 [223936.580284] [] __do_softirq+0x138/0x250 [223936.580288] [] call_softirq+0x1c/0x30 [223936.580291] [] do_softirq+0x95/0xd0 [223936.580294] [] irq_exit+0xdd/0x110 [223936.580297] [] smp_apic_timer_interrupt+0x69/0xa0 [223936.580300] [] apic_timer_interrupt+0x73/0x80 [223936.580302] [] ? mwait_idle+0x145/0x170 [223936.580308] [] ? mwait_idle+0x14e/0x170 [223936.580312] [] ? mwait_idle+0x145/0x170 [223936.580315] [] cpu_idle+0x96/0xf0 [223936.580318] [] start_secondary+0x1ca/0x1ff [223936.268002] RDX: 000000005932dd02 RSI: ffffffff818f9896 RDI: 0000000000000001 [223936.268002] RBP: ffff88022fc03d30 R08: 000000005932dcb5 R09: 0000000000000000 [223936.268002] R10: ffffffff81a2b020 R11: 0000000000000000 R12: ffff88022fc03c68 [223936.268002] R13: ffffffff816feb33 R14: ffff88022fc03d30 R15: 0000000017f328cd [223936.268002] FS: 0000000000000000(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000 [223936.268002] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223936.268002] CR2: 00007f25e7bc13a0 CR3: 00000001426fc000 CR4: 00000000000006f0 [223936.268002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223936.268002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223936.268002] Process swapper (pid: 0, threadinfo ffffffff819a6000, task ffffffff81a2b020) [223936.268002] Stack: [223936.268002] ffffffff819a6000 000000005932dd02 000000002fc03d18 ffff8801f6c22448 [223936.268002] 0000000017f328cd 000000006efcdc62 0000000000000001 ffffffff81a2b020 [223936.268002] ffff88022fc03d40 ffffffff813a4f6a ffff88022fc03d80 ffffffff813ac2ab [223936.268002] Call Trace: [223936.268002] [223936.268002] [] __delay+0xa/0x10 [223936.268002] [] do_raw_spin_lock+0x13b/0x180 [223936.268002] [] _raw_spin_lock+0x44/0x50 [223936.268002] [] ? tcp_keepalive_timer+0x23/0x270 [223936.268002] [] tcp_keepalive_timer+0x23/0x270 [223936.268002] [] run_timer_softirq+0x26d/0x410 [223936.268002] [] ? run_timer_softirq+0x198/0x410 [223936.268002] [] ? tcp_init_xmit_timers+0x20/0x20 [223936.268002] [] __do_softirq+0x138/0x250 [223936.268002] [] call_softirq+0x1c/0x30 [223936.268002] [] do_softirq+0x95/0xd0 [223936.268002] [] irq_exit+0xdd/0x110 [223936.268002] [] smp_apic_timer_interrupt+0x69/0xa0 [223936.268002] [] apic_timer_interrupt+0x73/0x80 [223936.268002] [223936.268002] [] ? mwait_idle+0x145/0x170 [223936.268002] [] ? mwait_idle+0x14e/0x170 [223936.268002] [] ? mwait_idle+0x145/0x170 [223936.268002] [] cpu_idle+0x96/0xf0 [223936.268002] [] rest_init+0xd1/0xe0 [223936.268002] [] ? csum_partial_copy_generic+0x170/0x170 [223936.268002] [] start_kernel+0x360/0x3ac [223936.268002] [] x86_64_start_reservations+0x82/0x89 [223936.268002] [] x86_64_start_kernel+0x10f/0x12a [223936.268002] [] ? early_idt_handlers+0x140/0x140 [223936.268002] Code: 4c 89 7d c8 eb 1f 66 90 48 8b 45 c0 83 68 1c 01 f3 90 83 40 1c 01 65 44 8b 3c 25 50 d3 00 00 44 3b 7d d4 75 3b 66 66 90 0f ae e8 b8 60 c7 ff 66 90 4c 63 c0 4c 89 c0 48 2b 45 c8 48 39 d8 72 [223936.268002] Call Trace: [223936.268002] [] __delay+0xa/0x10 [223936.268002] [] do_raw_spin_lock+0x13b/0x180 [223936.268002] [] _raw_spin_lock+0x44/0x50 [223936.268002] [] ? tcp_keepalive_timer+0x23/0x270 [223936.268002] [] tcp_keepalive_timer+0x23/0x270 [223936.268002] [] run_timer_softirq+0x26d/0x410 [223936.268002] [] ? run_timer_softirq+0x198/0x410 [223936.268002] [] ? tcp_init_xmit_timers+0x20/0x20 [223936.268002] [] __do_softirq+0x138/0x250 [223936.268002] [] call_softirq+0x1c/0x30 [223936.268002] [] do_softirq+0x95/0xd0 [223936.268002] [] irq_exit+0xdd/0x110 [223936.268002] [] smp_apic_timer_interrupt+0x69/0xa0 [223936.268002] [] apic_timer_interrupt+0x73/0x80 [223936.268002] [] ? mwait_idle+0x145/0x170 [223936.268002] [] ? mwait_idle+0x14e/0x170 [223936.268002] [] ? mwait_idle+0x145/0x170 [223936.268002] [] cpu_idle+0x96/0xf0 [223936.268002] [] rest_init+0xd1/0xe0 [223936.268002] [] ? csum_partial_copy_generic+0x170/0x170 [223936.268002] [] start_kernel+0x360/0x3ac [223936.268002] [] x86_64_start_reservations+0x82/0x89 [223936.268002] [] x86_64_start_kernel+0x10f/0x12a [223936.268002] [] ? early_idt_handlers+0x140/0x140 [223964.264018] BUG: soft lockup - CPU#0 stuck for 22s! [swapper:0] [223964.268002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223964.268002] irq event stamp: 2595159887 [223964.268002] hardirqs last enabled at (2595159887): [] mwait_idle+0x145/0x170 [223964.268002] hardirqs last disabled at (2595159886): [] cpu_idle+0x79/0xf0 [223964.268002] softirqs last enabled at (2595159878): [] __do_softirq+0x1f6/0x250 [223964.268002] softirqs last disabled at (2595159873): [] call_softirq+0x1c/0x30 [223964.268002] CPU 0 [223964.268002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223964.268002] [223964.268002] Pid: 0, comm: swapper Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223964.268002] RIP: 0010:[] [] native_read_tsc+0x6/0x20 [223964.268002] RSP: 0018:ffff88022fc03ce0 EFLAGS: 00000202 [223964.268002] RAX: 000000007cb6c61b RBX: ffffffff816f6334 RCX: 000000007cb6c5e3 [223964.372025] BUG: soft lockup - CPU#1 stuck for 23s! [kworker/0:0:0] [223964.372027] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223964.372036] irq event stamp: 2598787699 [223964.372037] hardirqs last enabled at (2598787699): [] mwait_idle+0x145/0x170 [223964.372042] hardirqs last disabled at (2598787698): [] cpu_idle+0x79/0xf0 [223964.372045] softirqs last enabled at (2598787696): [] __do_softirq+0x1f6/0x250 [223964.372049] softirqs last disabled at (2598787681): [] call_softirq+0x1c/0x30 [223964.372052] CPU 1 [223964.372053] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223964.372061] [223964.372063] Pid: 0, comm: kworker/0:0 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223964.372067] RIP: 0010:[] [] read_persistent_clock+0x30/0x30 [223964.372072] RSP: 0018:ffff88022fc43ce8 EFLAGS: 00000202 [223964.372074] RAX: 0000000000000001 RBX: ffff88022fc43c68 RCX: 0000000088b369fd [223964.372076] RDX: 0000000000000200 RSI: 0000000000000000 RDI: 0000000000000001 [223964.372078] RBP: ffff88022fc43d30 R08: ffffffff88b369fd R09: 0000000000000000 [223964.372081] R10: ffff88022690dd60 R11: 0000000000000000 R12: ffff88022fc43c58 [223964.372083] R13: ffffffff816feb33 R14: ffff88022fc43d30 R15: 00000000307e58b4 [223964.372086] FS: 0000000000000000(0000) GS:ffff88022fc40000(0000) knlGS:0000000000000000 [223964.372089] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223964.372091] CR2: 00007f472ba6a6fc CR3: 0000000126bb7000 CR4: 00000000000006e0 [223964.372093] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223964.372096] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223964.372098] Process kworker/0:0 (pid: 0, threadinfo ffff880226928000, task ffff88022690dd60) [223964.372100] Stack: [223964.372102] ffffffff813a4eaf ffff880226928000 ffffffff88b369c5 000000012fc43d18 [223964.372106] ffff88002e90e348 00000000307e58b4 000000006efcdc62 0000000000000001 [223964.372111] ffff88022690dd60 ffff88022fc43d40 ffffffff813a4f6a ffff88022fc43d80 [223964.372115] Call Trace: [223964.372116] [223964.372119] [] ? delay_tsc+0x3f/0xd0 [223964.372123] [] __delay+0xa/0x10 [223964.372127] [] do_raw_spin_lock+0x13b/0x180 [223964.372132] [] _raw_spin_lock+0x44/0x50 [223964.372136] [] ? tcp_keepalive_timer+0x23/0x270 [223964.372140] [] tcp_keepalive_timer+0x23/0x270 [223964.372144] [] run_timer_softirq+0x26d/0x410 [223964.372148] [] ? run_timer_softirq+0x198/0x410 [223964.372153] [] ? tcp_init_xmit_timers+0x20/0x20 [223964.372158] [] __do_softirq+0x138/0x250 [223964.372162] [] call_softirq+0x1c/0x30 [223964.372166] [] do_softirq+0x95/0xd0 [223964.372170] [] irq_exit+0xdd/0x110 [223964.372174] [] smp_apic_timer_interrupt+0x69/0xa0 [223964.372178] [] apic_timer_interrupt+0x73/0x80 [223964.372180] [223964.372184] [] ? mwait_idle+0x145/0x170 [223964.372188] [] ? mwait_idle+0x14e/0x170 [223964.372192] [] ? mwait_idle+0x145/0x170 [223964.372196] [] cpu_idle+0x96/0xf0 [223964.372200] [] start_secondary+0x1ca/0x1ff [223964.372203] Code: 48 89 fb 48 83 ec 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 [223964.372221] 48 89 e5 0f 31 89 c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 [223964.372231] Call Trace: [223964.372232] [] ? delay_tsc+0x3f/0xd0 [223964.372237] [] __delay+0xa/0x10 [223964.372241] [] do_raw_spin_lock+0x13b/0x180 [223964.372245] [] _raw_spin_lock+0x44/0x50 [223964.372248] [] ? tcp_keepalive_timer+0x23/0x270 [223964.372251] [] tcp_keepalive_timer+0x23/0x270 [223964.372255] [] run_timer_softirq+0x26d/0x410 [223964.372258] [] ? run_timer_softirq+0x198/0x410 [223964.372261] [] ? tcp_init_xmit_timers+0x20/0x20 [223964.372265] [] __do_softirq+0x138/0x250 [223964.372268] [] call_softirq+0x1c/0x30 [223964.372271] [] do_softirq+0x95/0xd0 [223964.372275] [] irq_exit+0xdd/0x110 [223964.372278] [] smp_apic_timer_interrupt+0x69/0xa0 [223964.372281] [] apic_timer_interrupt+0x73/0x80 [223964.372282] [] ? mwait_idle+0x145/0x170 [223964.372288] [] ? mwait_idle+0x14e/0x170 [223964.372292] [] ? mwait_idle+0x145/0x170 [223964.372295] [] cpu_idle+0x96/0xf0 [223964.372298] [] start_secondary+0x1ca/0x1ff [223964.476031] BUG: soft lockup - CPU#2 stuck for 23s! [kworker/0:1:0] [223964.476033] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223964.476042] irq event stamp: 2613824057 [223964.476043] hardirqs last enabled at (2613824057): [] mwait_idle+0x145/0x170 [223964.476048] hardirqs last disabled at (2613824056): [] cpu_idle+0x79/0xf0 [223964.476051] softirqs last enabled at (2613824048): [] __do_softirq+0x1f6/0x250 [223964.476055] softirqs last disabled at (2613824031): [] call_softirq+0x1c/0x30 [223964.476059] CPU 2 [223964.476060] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223964.476067] [223964.476070] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223964.476074] RIP: 0010:[] [] native_read_tsc+0x6/0x20 [223964.476078] RSP: 0018:ffff88022fc83ce0 EFLAGS: 00000206 [223964.476080] RAX: 00000000943e6715 RBX: ffffffff816f6334 RCX: 00000000943e66dd [223964.476083] RDX: 0000000000017b69 RSI: 0000000000000000 RDI: 0000000000000001 [223964.476085] RBP: ffff88022fc83ce0 R08: ffffffff943e6697 R09: 0000000000000000 [223964.476087] R10: ffff880226948000 R11: 0000000000000000 R12: ffff88022fc83c58 [223964.476090] R13: ffffffff816feb33 R14: ffff88022fc83ce0 R15: 000000003094ad30 [223964.476092] FS: 0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000 [223964.476095] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223964.476097] CR2: 00007f25e7874d7f CR3: 0000000124c0d000 CR4: 00000000000006e0 [223964.476100] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223964.476102] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223964.476105] Process kworker/0:1 (pid: 0, threadinfo ffff88022693e000, task ffff880226948000) [223964.476107] Stack: [223964.476108] ffff88022fc83d30 ffffffff813a4ee8 ffff88022693e000 ffffffff943e66dd [223964.476113] 00000002ffffff10 ffff88006afd8948 000000003094ad30 000000006efcdc62 [223964.476117] 0000000000000001 ffff880226948000 ffff88022fc83d40 ffffffff813a4f6a [223964.476121] Call Trace: [223964.476123] [223964.476126] [] delay_tsc+0x78/0xd0 [223964.476130] [] __delay+0xa/0x10 [223964.476134] [] do_raw_spin_lock+0x13b/0x180 [223964.476139] [] _raw_spin_lock+0x44/0x50 [223964.476143] [] ? tcp_keepalive_timer+0x23/0x270 [223964.476147] [] tcp_keepalive_timer+0x23/0x270 [223964.476151] [] run_timer_softirq+0x26d/0x410 [223964.476155] [] ? run_timer_softirq+0x198/0x410 [223964.476159] [] ? tcp_init_xmit_timers+0x20/0x20 [223964.476164] [] __do_softirq+0x138/0x250 [223964.476168] [] call_softirq+0x1c/0x30 [223964.476172] [] do_softirq+0x95/0xd0 [223964.476176] [] irq_exit+0xdd/0x110 [223964.476180] [] smp_apic_timer_interrupt+0x69/0xa0 [223964.476184] [] apic_timer_interrupt+0x73/0x80 [223964.476186] [223964.476190] [] ? mwait_idle+0x145/0x170 [223964.476194] [] ? mwait_idle+0x14e/0x170 [223964.476198] [] ? mwait_idle+0x145/0x170 [223964.476202] [] cpu_idle+0x96/0xf0 [223964.476206] [] start_secondary+0x1ca/0x1ff [223964.476208] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 [223964.476227] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 [223964.476236] Call Trace: [223964.476238] [] delay_tsc+0x78/0xd0 [223964.476243] [] __delay+0xa/0x10 [223964.476246] [] do_raw_spin_lock+0x13b/0x180 [223964.476250] [] _raw_spin_lock+0x44/0x50 [223964.476254] [] ? tcp_keepalive_timer+0x23/0x270 [223964.476257] [] tcp_keepalive_timer+0x23/0x270 [223964.476260] [] run_timer_softirq+0x26d/0x410 [223964.476264] [] ? run_timer_softirq+0x198/0x410 [223964.476267] [] ? tcp_init_xmit_timers+0x20/0x20 [223964.476271] [] __do_softirq+0x138/0x250 [223964.476274] [] call_softirq+0x1c/0x30 [223964.476277] [] do_softirq+0x95/0xd0 [223964.476281] [] irq_exit+0xdd/0x110 [223964.476284] [] smp_apic_timer_interrupt+0x69/0xa0 [223964.476287] [] apic_timer_interrupt+0x73/0x80 [223964.476289] [] ? mwait_idle+0x145/0x170 [223964.476295] [] ? mwait_idle+0x14e/0x170 [223964.476298] [] ? mwait_idle+0x145/0x170 [223964.476301] [] cpu_idle+0x96/0xf0 [223964.476304] [] start_secondary+0x1ca/0x1ff [223964.580038] BUG: soft lockup - CPU#3 stuck for 23s! [kworker/0:1:0] [223964.580040] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223964.580049] irq event stamp: 2615464042 [223964.580050] hardirqs last enabled at (2615464042): [] _raw_spin_unlock_irq+0x2b/0x50 [223964.580054] hardirqs last disabled at (2615464041): [] _raw_spin_lock_irq+0x18/0x60 [223964.580058] softirqs last enabled at (2615463964): [] _local_bh_enable+0xe/0x10 [223964.580062] softirqs last disabled at (2615463965): [] call_softirq+0x1c/0x30 [223964.580066] CPU 3 [223964.580067] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223964.580075] [223964.580077] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223964.580081] RIP: 0010:[] [] native_read_tsc+0x6/0x20 [223964.580086] RSP: 0018:ffff88022fcc3ce0 EFLAGS: 00000206 [223964.580088] RAX: 000000009fc963af RBX: ffffffff816f6334 RCX: 000000009fc96377 [223964.580090] RDX: 0000000000017b69 RSI: ffffffff818f9896 RDI: 0000000000000001 [223964.580093] RBP: ffff88022fcc3ce0 R08: ffffffff9fc96331 R09: 0000000000000000 [223964.580095] R10: ffff880226981f20 R11: 0000000000000000 R12: ffff88022fcc3c58 [223964.580097] R13: ffffffff816feb33 R14: ffff88022fcc3ce0 R15: 0000000030ac88b0 [223964.580100] FS: 0000000000000000(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000 [223964.580103] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223964.580105] CR2: 0000000000f38820 CR3: 0000000104b52000 CR4: 00000000000006e0 [223964.580107] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223964.580110] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223964.580112] Process kworker/0:1 (pid: 0, threadinfo ffff880226988000, task ffff880226981f20) [223964.580114] Stack: [223964.580116] ffff88022fcc3d30 ffffffff813a4ee8 ffff880226988000 ffffffff9fc96377 [223964.580120] 000000039c3b34d8 ffff880031438948 0000000030ac88b0 000000006efcdc62 [223964.580124] 0000000000000001 ffff880226981f20 ffff88022fcc3d40 ffffffff813a4f6a [223964.580128] Call Trace: [223964.580130] [223964.580133] [] delay_tsc+0x78/0xd0 [223964.580137] [] __delay+0xa/0x10 [223964.580141] [] do_raw_spin_lock+0x13b/0x180 [223964.580146] [] _raw_spin_lock+0x44/0x50 [223964.580150] [] ? tcp_keepalive_timer+0x23/0x270 [223964.580154] [] tcp_keepalive_timer+0x23/0x270 [223964.580158] [] run_timer_softirq+0x26d/0x410 [223964.580162] [] ? run_timer_softirq+0x198/0x410 [223964.580167] [] ? tcp_init_xmit_timers+0x20/0x20 [223964.580171] [] __do_softirq+0x138/0x250 [223964.580176] [] call_softirq+0x1c/0x30 [223964.580180] [] do_softirq+0x95/0xd0 [223964.580184] [] irq_exit+0xdd/0x110 [223964.580188] [] smp_apic_timer_interrupt+0x69/0xa0 [223964.580192] [] apic_timer_interrupt+0x73/0x80 [223964.580194] [223964.580198] [] ? mwait_idle+0x145/0x170 [223964.580202] [] ? mwait_idle+0x14e/0x170 [223964.580206] [] ? mwait_idle+0x145/0x170 [223964.580210] [] cpu_idle+0x96/0xf0 [223964.580214] [] start_secondary+0x1ca/0x1ff [223964.580217] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 [223964.580235] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 [223964.580245] Call Trace: [223964.580246] [] delay_tsc+0x78/0xd0 [223964.580252] [] __delay+0xa/0x10 [223964.580255] [] do_raw_spin_lock+0x13b/0x180 [223964.580259] [] _raw_spin_lock+0x44/0x50 [223964.580262] [] ? tcp_keepalive_timer+0x23/0x270 [223964.580265] [] tcp_keepalive_timer+0x23/0x270 [223964.580269] [] run_timer_softirq+0x26d/0x410 [223964.580272] [] ? run_timer_softirq+0x198/0x410 [223964.580276] [] ? tcp_init_xmit_timers+0x20/0x20 [223964.580279] [] __do_softirq+0x138/0x250 [223964.580283] [] call_softirq+0x1c/0x30 [223964.580286] [] do_softirq+0x95/0xd0 [223964.580289] [] irq_exit+0xdd/0x110 [223964.580292] [] smp_apic_timer_interrupt+0x69/0xa0 [223964.580295] [] apic_timer_interrupt+0x73/0x80 [223964.580297] [] ? mwait_idle+0x145/0x170 [223964.580303] [] ? mwait_idle+0x14e/0x170 [223964.580307] [] ? mwait_idle+0x145/0x170 [223964.580310] [] cpu_idle+0x96/0xf0 [223964.580313] [] start_secondary+0x1ca/0x1ff [223964.268002] RDX: 0000000000017b69 RSI: ffffffff818f9896 RDI: 0000000000000001 [223964.268002] RBP: ffff88022fc03ce0 R08: 000000007cb6c596 R09: 0000000000000000 [223964.268002] R10: ffffffff81a2b020 R11: 0000000000000000 R12: ffff88022fc03c58 [223964.268002] R13: ffffffff816feb33 R14: ffff88022fc03ce0 R15: 000000002eb85d38 [223964.268002] FS: 0000000000000000(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000 [223964.268002] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223964.268002] CR2: 00007f25e7bc13a0 CR3: 00000001426fc000 CR4: 00000000000006f0 [223964.268002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223964.268002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223964.268002] Process swapper (pid: 0, threadinfo ffffffff819a6000, task ffffffff81a2b020) [223964.268002] Stack: [223964.268002] ffff88022fc03d30 ffffffff813a4ee8 ffffffff819a6000 000000007cb6c5e3 [223964.268002] 000000007c44ac9c ffff8801f6c22448 000000002eb85d38 000000006efcdc62 [223964.268002] 0000000000000001 ffffffff81a2b020 ffff88022fc03d40 ffffffff813a4f6a [223964.268002] Call Trace: [223964.268002] [223964.268002] [] delay_tsc+0x78/0xd0 [223964.268002] [] __delay+0xa/0x10 [223964.268002] [] do_raw_spin_lock+0x13b/0x180 [223964.268002] [] _raw_spin_lock+0x44/0x50 [223964.268002] [] ? tcp_keepalive_timer+0x23/0x270 [223964.268002] [] tcp_keepalive_timer+0x23/0x270 [223964.268002] [] run_timer_softirq+0x26d/0x410 [223964.268002] [] ? run_timer_softirq+0x198/0x410 [223964.268002] [] ? tcp_init_xmit_timers+0x20/0x20 [223964.268002] [] __do_softirq+0x138/0x250 [223964.268002] [] call_softirq+0x1c/0x30 [223964.268002] [] do_softirq+0x95/0xd0 [223964.268002] [] irq_exit+0xdd/0x110 [223964.268002] [] smp_apic_timer_interrupt+0x69/0xa0 [223964.268002] [] apic_timer_interrupt+0x73/0x80 [223964.268002] [223964.268002] [] ? mwait_idle+0x145/0x170 [223964.268002] [] ? mwait_idle+0x14e/0x170 [223964.268002] [] ? mwait_idle+0x145/0x170 [223964.268002] [] cpu_idle+0x96/0xf0 [223964.268002] [] rest_init+0xd1/0xe0 [223964.268002] [] ? csum_partial_copy_generic+0x170/0x170 [223964.268002] [] start_kernel+0x360/0x3ac [223964.268002] [] x86_64_start_reservations+0x82/0x89 [223964.268002] [] x86_64_start_kernel+0x10f/0x12a [223964.268002] [] ? early_idt_handlers+0x140/0x140 [223964.268002] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 [223964.268002] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 [223964.268002] Call Trace: [223964.268002] [] delay_tsc+0x78/0xd0 [223964.268002] [] __delay+0xa/0x10 [223964.268002] [] do_raw_spin_lock+0x13b/0x180 [223964.268002] [] _raw_spin_lock+0x44/0x50 [223964.268002] [] ? tcp_keepalive_timer+0x23/0x270 [223964.268002] [] tcp_keepalive_timer+0x23/0x270 [223964.268002] [] run_timer_softirq+0x26d/0x410 [223964.268002] [] ? run_timer_softirq+0x198/0x410 [223964.268002] [] ? tcp_init_xmit_timers+0x20/0x20 [223964.268002] [] __do_softirq+0x138/0x250 [223964.268002] [] call_softirq+0x1c/0x30 [223964.268002] [] do_softirq+0x95/0xd0 [223964.268002] [] irq_exit+0xdd/0x110 [223964.268002] [] smp_apic_timer_interrupt+0x69/0xa0 [223964.268002] [] apic_timer_interrupt+0x73/0x80 [223964.268002] [] ? mwait_idle+0x145/0x170 [223964.268002] [] ? mwait_idle+0x14e/0x170 [223964.268002] [] ? mwait_idle+0x145/0x170 [223964.268002] [] cpu_idle+0x96/0xf0 [223964.268002] [] rest_init+0xd1/0xe0 [223964.268002] [] ? csum_partial_copy_generic+0x170/0x170 [223964.268002] [] start_kernel+0x360/0x3ac [223964.268002] [] x86_64_start_reservations+0x82/0x89 [223964.268002] [] x86_64_start_kernel+0x10f/0x12a [223964.268002] [] ? early_idt_handlers+0x140/0x140 [223968.815995] INFO: rcu_sched_state detected stall on CPU 1 (t=15000 jiffies) [223968.819995] INFO: rcu_sched_state detected stalls on CPUs/tasks: { 1} (detected by 3, t=15002 jiffies) [223968.820000] sending NMI to all CPUs: [223968.820002] NMI backtrace for cpu 3 [223968.820002] CPU 3 [223968.820002] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223968.820002] [223968.820002] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223968.820002] RIP: 0010:[] [] __const_udelay+0x16/0x40 [223968.820002] RSP: 0018:ffff88022fcc3a90 EFLAGS: 00000002 [223968.820002] RAX: 0000000000e34d8a RBX: 0000000000000001 RCX: 0000000001062560 [223968.820002] RDX: 000000000071a6c5 RSI: 0000000000000002 RDI: 0000000000418958 [223968.820002] RBP: ffff88022fcc3ab0 R08: 0000000000000002 R09: 0000000000000000 [223968.820002] R10: 0000000000000006 R11: 000000000000000a R12: ffffffff81a40d80 [223968.820002] R13: 0000000000000010 R14: ffffffff81a40e40 R15: ffffffff81a40fc0 [223968.820002] FS: 0000000000000000(0000) GS:ffff88022fcc0000(0000) knlGS:0000000000000000 [223968.820002] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223968.820002] CR2: 0000000000f38820 CR3: 0000000104b52000 CR4: 00000000000006e0 [223968.820002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223968.820002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223968.820002] Process kworker/0:1 (pid: 0, threadinfo ffff880226988000, task ffff880226981f20) [223968.820002] Stack: [223968.820002] ffff88022fcc3ab0 ffffffff81031695 ffff88022fccdfa0 ffff88022fccdfa0 [223968.820002] ffff88022fcc3af0 ffffffff810bb9d2 ffffffff81a40fc0 0000000000000003 [223968.820002] 0000000000000003 ffff880226981f20 ffffffff810921f0 ffff88022fcc3be0 [223968.820002] Call Trace: [223968.820002] [223968.820002] [] ? arch_trigger_all_cpu_backtrace+0x65/0x90 [223968.820002] [] __rcu_pending+0x382/0x3b0 [223968.820002] [] ? tick_nohz_handler+0x100/0x100 [223968.820002] [] rcu_check_callbacks+0x67/0x130 [223968.820002] [] update_process_times+0x41/0x80 [223968.820002] [] tick_sched_timer+0x66/0xc0 [223968.820002] [] __run_hrtimer+0xfe/0x1e0 [223968.820002] [] hrtimer_interrupt+0xcd/0x1f0 [223968.820002] [] smp_apic_timer_interrupt+0x64/0xa0 [223968.820002] [] apic_timer_interrupt+0x73/0x80 [223968.820002] [] ? retint_restore_args+0x13/0x13 [223968.820002] [] ? native_read_tsc+0x6/0x20 [223968.820002] [] delay_tsc+0x3f/0xd0 [223968.820002] [] __delay+0xa/0x10 [223968.820002] [] do_raw_spin_lock+0x13b/0x180 [223968.820002] [] _raw_spin_lock+0x44/0x50 [223968.820002] [] ? tcp_keepalive_timer+0x23/0x270 [223968.820002] [] tcp_keepalive_timer+0x23/0x270 [223968.820002] [] run_timer_softirq+0x26d/0x410 [223968.820002] [] ? run_timer_softirq+0x198/0x410 [223968.820002] [] ? tcp_init_xmit_timers+0x20/0x20 [223968.820002] [] __do_softirq+0x138/0x250 [223968.820002] [] call_softirq+0x1c/0x30 [223968.820002] [] do_softirq+0x95/0xd0 [223968.820002] [] irq_exit+0xdd/0x110 [223968.820002] [] smp_apic_timer_interrupt+0x69/0xa0 [223968.820002] [] apic_timer_interrupt+0x73/0x80 [223968.820002] [223968.820002] [] ? mwait_idle+0x145/0x170 [223968.820002] [] ? mwait_idle+0x14e/0x170 [223968.820002] [] ? mwait_idle+0x145/0x170 [223968.820002] [] cpu_idle+0x96/0xf0 [223968.820002] [] start_secondary+0x1ca/0x1ff [223968.820002] Code: 00 00 00 00 00 55 48 89 e5 ff 15 8e a5 6c 00 c9 c3 0f 1f 40 00 55 48 8d 0c bd 00 00 00 00 65 48 8b 14 25 58 2d 01 00 48 8d 04 12 [223968.820002] c1 e2 06 48 89 e5 48 29 c2 48 89 c8 f7 e2 48 8d 7a 01 ff 15 [223968.820002] Call Trace: [223968.820002] [] ? arch_trigger_all_cpu_backtrace+0x65/0x90 [223968.820002] [] __rcu_pending+0x382/0x3b0 [223968.820002] [] ? tick_nohz_handler+0x100/0x100 [223968.820002] [] rcu_check_callbacks+0x67/0x130 [223968.820002] [] update_process_times+0x41/0x80 [223968.820002] [] tick_sched_timer+0x66/0xc0 [223968.820002] [] __run_hrtimer+0xfe/0x1e0 [223968.820002] [] hrtimer_interrupt+0xcd/0x1f0 [223968.820002] [] smp_apic_timer_interrupt+0x64/0xa0 [223968.820002] [] apic_timer_interrupt+0x73/0x80 [223968.820002] [] ? retint_restore_args+0x13/0x13 [223968.820002] [] ? native_read_tsc+0x6/0x20 [223968.820002] [] delay_tsc+0x3f/0xd0 [223968.820002] [] __delay+0xa/0x10 [223968.820002] [] do_raw_spin_lock+0x13b/0x180 [223968.820002] [] _raw_spin_lock+0x44/0x50 [223968.820002] [] ? tcp_keepalive_timer+0x23/0x270 [223968.820002] [] tcp_keepalive_timer+0x23/0x270 [223968.820002] [] run_timer_softirq+0x26d/0x410 [223968.820002] [] ? run_timer_softirq+0x198/0x410 [223968.820002] [] ? tcp_init_xmit_timers+0x20/0x20 [223968.820002] [] __do_softirq+0x138/0x250 [223968.820002] [] call_softirq+0x1c/0x30 [223968.820002] [] do_softirq+0x95/0xd0 [223968.820002] [] irq_exit+0xdd/0x110 [223968.820002] [] smp_apic_timer_interrupt+0x69/0xa0 [223968.820002] [] apic_timer_interrupt+0x73/0x80 [223968.820002] [] ? mwait_idle+0x145/0x170 [223968.820002] [] ? mwait_idle+0x14e/0x170 [223968.820002] [] ? mwait_idle+0x145/0x170 [223968.820002] [] cpu_idle+0x96/0xf0 [223968.820002] [] start_secondary+0x1ca/0x1ff [223968.820335] NMI backtrace for cpu 0 [223968.820337] CPU 0 [223968.820338] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223968.820347] [223968.820349] Pid: 0, comm: swapper Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223968.820353] RIP: 0010:[] [] delay_tsc+0x80/0xd0 [223968.820358] RSP: 0018:ffff88022fc03cf0 EFLAGS: 00000206 [223968.820360] RAX: 000000007659b10f RBX: 0000000000000001 RCX: 000000007659b10f [223968.820363] RDX: 000000007659b10f RSI: ffffffff818f9896 RDI: 0000000000000001 [223968.820365] RBP: ffff88022fc03d30 R08: 000000007659b10f R09: 0000000000000000 [223968.820367] R10: ffffffff81a2b020 R11: 0000000000000000 R12: 0000000031026962 [223968.820370] R13: 000000006efcdc62 R14: ffffffff819a6000 R15: 000000007659b0de [223968.820373] FS: 0000000000000000(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000 [223968.820375] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223968.820377] CR2: 00007f25e7bc13a0 CR3: 00000001426fc000 CR4: 00000000000006f0 [223968.820380] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223968.820382] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223968.820385] Process swapper (pid: 0, threadinfo ffffffff819a6000, task ffffffff81a2b020) [223968.820387] Stack: [223968.820388] ffffffff819a6000 000000007659b0de 00000000818f9896 ffff8801f6c22448 [223968.820393] 0000000031026962 000000006efcdc62 0000000000000001 ffffffff81a2b020 [223968.820397] ffff88022fc03d40 ffffffff813a4f6a ffff88022fc03d80 ffffffff813ac2ab [223968.820401] Call Trace: [223968.820402] [223968.820406] [] __delay+0xa/0x10 [223968.820410] [] do_raw_spin_lock+0x13b/0x180 [223968.820414] [] _raw_spin_lock+0x44/0x50 [223968.820417] [] ? tcp_keepalive_timer+0x23/0x270 [223968.820420] [] tcp_keepalive_timer+0x23/0x270 [223968.820424] [] run_timer_softirq+0x26d/0x410 [223968.820427] [] ? run_timer_softirq+0x198/0x410 [223968.820430] [] ? tcp_init_xmit_timers+0x20/0x20 [223968.820434] [] __do_softirq+0x138/0x250 [223968.820437] [] call_softirq+0x1c/0x30 [223968.820441] [] do_softirq+0x95/0xd0 [223968.820444] [] irq_exit+0xdd/0x110 [223968.820447] [] smp_apic_timer_interrupt+0x69/0xa0 [223968.820450] [] apic_timer_interrupt+0x73/0x80 [223968.820452] [223968.820455] [] ? mwait_idle+0x145/0x170 [223968.820459] [] ? mwait_idle+0x14e/0x170 [223968.820462] [] ? mwait_idle+0x145/0x170 [223968.820465] [] cpu_idle+0x96/0xf0 [223968.820468] [] rest_init+0xd1/0xe0 [223968.820471] [] ? csum_partial_copy_generic+0x170/0x170 [223968.820475] [] start_kernel+0x360/0x3ac [223968.820478] [] x86_64_start_reservations+0x82/0x89 [223968.820481] [] x86_64_start_kernel+0x10f/0x12a [223968.820484] [] ? early_idt_handlers+0x140/0x140 [223968.820486] Code: 68 1c 01 f3 90 83 40 1c 01 65 44 8b 3c 25 50 d3 00 00 44 3b 7d d4 75 3b 66 66 90 0f ae e8 e8 b8 60 c7 ff 66 90 4c 63 c0 4c 89 c0 [223968.820504] 2b 45 c8 48 39 d8 72 c7 65 48 8b 04 25 08 c4 00 00 83 a8 44 [223968.820514] Call Trace: [223968.820515] [] __delay+0xa/0x10 [223968.820521] [] do_raw_spin_lock+0x13b/0x180 [223968.820525] [] _raw_spin_lock+0x44/0x50 [223968.820528] [] ? tcp_keepalive_timer+0x23/0x270 [223968.820532] [] tcp_keepalive_timer+0x23/0x270 [223968.820535] [] run_timer_softirq+0x26d/0x410 [223968.820538] [] ? run_timer_softirq+0x198/0x410 [223968.820542] [] ? tcp_init_xmit_timers+0x20/0x20 [223968.820546] [] __do_softirq+0x138/0x250 [223968.820549] [] call_softirq+0x1c/0x30 [223968.820552] [] do_softirq+0x95/0xd0 [223968.820556] [] irq_exit+0xdd/0x110 [223968.820559] [] smp_apic_timer_interrupt+0x69/0xa0 [223968.820562] [] apic_timer_interrupt+0x73/0x80 [223968.820564] [] ? mwait_idle+0x145/0x170 [223968.820570] [] ? mwait_idle+0x14e/0x170 [223968.820573] [] ? mwait_idle+0x145/0x170 [223968.820576] [] cpu_idle+0x96/0xf0 [223968.820579] [] rest_init+0xd1/0xe0 [223968.820583] [] ? csum_partial_copy_generic+0x170/0x170 [223968.820586] [] start_kernel+0x360/0x3ac [223968.820589] [] x86_64_start_reservations+0x82/0x89 [223968.820593] [] x86_64_start_kernel+0x10f/0x12a [223968.820596] [] ? early_idt_handlers+0x140/0x140 [223968.820599] NMI backtrace for cpu 2 [223968.820600] CPU 2 [223968.820602] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223968.820610] [223968.820612] Pid: 0, comm: kworker/0:1 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223968.820616] RIP: 0010:[] [] native_read_tsc+0x6/0x20 [223968.820621] RSP: 0018:ffff88022fc83ce0 EFLAGS: 00000202 [223968.820623] RAX: 000000007659b116 RBX: 0000000000000001 RCX: 000000007659b0e5 [223968.820625] RDX: 0000000000017b6b RSI: 0000000000000000 RDI: 0000000000000001 [223968.820628] RBP: ffff88022fc83ce0 R08: 000000007659b098 R09: 0000000000000000 [223968.820630] R10: ffff880226948000 R11: 0000000000000000 R12: 00000000345f87d7 [223968.820632] R13: 000000006efcdc62 R14: ffff88022693e000 R15: 000000007659b0e5 [223968.820635] FS: 0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000 [223968.820638] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223968.820640] CR2: 00007f25e7874d7f CR3: 0000000124c0d000 CR4: 00000000000006e0 [223968.820642] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223968.820645] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223968.820647] Process kworker/0:1 (pid: 0, threadinfo ffff88022693e000, task ffff880226948000) [223968.820649] Stack: [223968.820651] ffff88022fc83d30 ffffffff813a4ee8 ffff88022693e000 000000007659b0e5 [223968.820655] 000000026b4044c5 ffff88006afd8948 00000000345f87d7 000000006efcdc62 [223968.820659] 0000000000000001 ffff880226948000 ffff88022fc83d40 ffffffff813a4f6a [223968.820663] Call Trace: [223968.820665] [223968.820668] [] delay_tsc+0x78/0xd0 [223968.820671] [] __delay+0xa/0x10 [223968.820674] [] do_raw_spin_lock+0x13b/0x180 [223968.820678] [] _raw_spin_lock+0x44/0x50 [223968.820682] [] ? tcp_keepalive_timer+0x23/0x270 [223968.820685] [] tcp_keepalive_timer+0x23/0x270 [223968.820688] [] run_timer_softirq+0x26d/0x410 [223968.820691] [] ? run_timer_softirq+0x198/0x410 [223968.820695] [] ? tcp_init_xmit_timers+0x20/0x20 [223968.820699] [] __do_softirq+0x138/0x250 [223968.820702] [] call_softirq+0x1c/0x30 [223968.820705] [] do_softirq+0x95/0xd0 [223968.820708] [] irq_exit+0xdd/0x110 [223968.820712] [] smp_apic_timer_interrupt+0x69/0xa0 [223968.820715] [] apic_timer_interrupt+0x73/0x80 [223968.820717] [223968.820720] [] ? mwait_idle+0x145/0x170 [223968.820723] [] ? mwait_idle+0x14e/0x170 [223968.820727] [] ? mwait_idle+0x145/0x170 [223968.820730] [] cpu_idle+0x96/0xf0 [223968.820733] [] start_secondary+0x1ca/0x1ff [223968.820735] Code: 08 ff 15 46 5c a1 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 [223968.820753] c1 48 89 d0 48 c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 [223968.820763] Call Trace: [223968.820764] [] delay_tsc+0x78/0xd0 [223968.820769] [] __delay+0xa/0x10 [223968.820773] [] do_raw_spin_lock+0x13b/0x180 [223968.820777] [] _raw_spin_lock+0x44/0x50 [223968.820780] [] ? tcp_keepalive_timer+0x23/0x270 [223968.820783] [] tcp_keepalive_timer+0x23/0x270 [223968.820787] [] run_timer_softirq+0x26d/0x410 [223968.820790] [] ? run_timer_softirq+0x198/0x410 [223968.820793] [] ? tcp_init_xmit_timers+0x20/0x20 [223968.820797] [] __do_softirq+0x138/0x250 [223968.820801] [] call_softirq+0x1c/0x30 [223968.820804] [] do_softirq+0x95/0xd0 [223968.820807] [] irq_exit+0xdd/0x110 [223968.820810] [] smp_apic_timer_interrupt+0x69/0xa0 [223968.820813] [] apic_timer_interrupt+0x73/0x80 [223968.820815] [] ? mwait_idle+0x145/0x170 [223968.820821] [] ? mwait_idle+0x14e/0x170 [223968.820824] [] ? mwait_idle+0x145/0x170 [223968.820827] [] cpu_idle+0x96/0xf0 [223968.820831] [] start_secondary+0x1ca/0x1ff [223968.816001] NMI backtrace for cpu 1 [223968.816001] CPU 1 [223968.816001] Modules linked in: ipmi_devintf ipmi_si ipmi_msghandler xt_recent nf_conntrack_ftp xt_state xt_owner nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 bnx2 [223968.816001] [223968.816001] Pid: 0, comm: kworker/0:0 Not tainted 3.1.0-rc10-hw-lockdep+ #51 Dell Inc. PowerEdge 1950/0M788G [223968.816001] RIP: 0010:[] [] io_serial_out+0x15/0x20 [223968.816001] RSP: 0018:ffff88022fc437f0 EFLAGS: 00000002 [223968.816001] RAX: 0000000000000073 RBX: ffffffff8243eec0 RCX: 0000000000000000 [223968.816001] RDX: 00000000000003f8 RSI: 00000000000003f8 RDI: ffffffff8243eec0 [223968.816001] RBP: ffff88022fc437f0 R08: 000000007659a435 R09: 0000000000000000 [223968.816001] R10: 0000000000000001 R11: 000000000000000a R12: 0000000000000073 [223968.816001] R13: ffffffff81bc648d R14: 0000000000000050 R15: ffffffff8243eec0 [223968.816001] FS: 0000000000000000(0000) GS:ffff88022fc40000(0000) knlGS:0000000000000000 [223968.816001] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [223968.816001] CR2: 00007f472ba6a6fc CR3: 0000000126bb7000 CR4: 00000000000006e0 [223968.816001] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [223968.816001] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [223968.816001] Process kworker/0:0 (pid: 0, threadinfo ffff880226928000, task ffff88022690dd60) [223968.816001] Stack: [223968.816001] ffff88022fc43810 ffffffff814410dc 0000000000000030 ffffffff814410b0 [223968.816001] ffff88022fc43850 ffffffff8143cdb5 0000000000000087 0000000000000000 [223968.816001] ffffffff8243eec0 0000000000000001 0000000000000087 000000000000000d [223968.816001] Call Trace: [223968.816001] [223968.816001] [] serial8250_console_putchar+0x2c/0x40 [223968.816001] [] ? wait_for_xmitr+0xa0/0xa0 [223968.816001] [] uart_console_write+0x35/0x70 [223968.816001] [] serial8250_console_write+0xbe/0x1a0 [223968.816001] [] __call_console_drivers+0x8e/0xb0 [223968.816001] [] _call_console_drivers+0x45/0x70 [223968.816001] [] console_unlock+0x17f/0x2b0 [223968.816001] [] vprintk+0x1fd/0x520 [223968.816001] [] printk+0x67/0x69 [223968.816001] [] ? _raw_spin_unlock+0x26/0x40 [223968.816001] [] ? account_system_time+0xab/0x190 [223968.816001] [] __rcu_pending+0x194/0x3b0 [223968.816001] [] ? tick_nohz_handler+0x100/0x100 [223968.816001] [] rcu_check_callbacks+0x67/0x130 [223968.816001] [] update_process_times+0x41/0x80 [223968.816001] [] tick_sched_timer+0x66/0xc0 [223968.816001] [] __run_hrtimer+0xfe/0x1e0 [223968.816001] [] hrtimer_interrupt+0xcd/0x1f0 [223968.816001] [] smp_apic_timer_interrupt+0x64/0xa0 [223968.816001] [] apic_timer_interrupt+0x73/0x80 [223968.816001] [] ? retint_restore_args+0x13/0x13 [223968.816001] [] ? native_read_tsc+0x6/0x20 [223968.816001] [] delay_tsc+0x3f/0xd0 [223968.816001] [] __delay+0xa/0x10 [223968.816001] [] do_raw_spin_lock+0x13b/0x180 [223968.816001] [] _raw_spin_lock+0x44/0x50 [223968.816001] [] ? tcp_keepalive_timer+0x23/0x270 [223968.816001] [] tcp_keepalive_timer+0x23/0x270 [223968.816001] [] run_timer_softirq+0x26d/0x410 [223968.816001] [] ? run_timer_softirq+0x198/0x410 [223968.816001] [] ? tcp_init_xmit_timers+0x20/0x20 [223968.816001] [] __do_softirq+0x138/0x250 [223968.816001] [] call_softirq+0x1c/0x30 [223968.816001] [] do_softirq+0x95/0xd0 [223968.816001] [] irq_exit+0xdd/0x110 [223968.816001] [] smp_apic_timer_interrupt+0x69/0xa0 [223968.816001] [] apic_timer_interrupt+0x73/0x80 [223968.816001] [223968.816001] [] ? mwait_idle+0x145/0x170 [223968.816001] [] ? mwait_idle+0x14e/0x170 [223968.816001] [] ? mwait_idle+0x145/0x170 [223968.816001] [] cpu_idle+0x96/0xf0 [223968.816001] [] start_secondary+0x1ca/0x1ff [223968.816001] Code: 48 89 e5 d3 e2 03 57 38 ec 0f b6 c0 c9 c3 0f 1f 84 00 00 00 00 00 0f b6 8f 81 00 00 00 55 89 d0 48 89 e5 d3 e6 03 77 38 89 f2 ee c3 66 0f 1f 84 00 00 00 00 00 55 80 bf 82 00 00 00 08 48 89 [223968.816001] Call Trace: [223968.816001] [] serial8250_console_putchar+0x2c/0x40 [223968.816001] [] ? wait_for_xmitr+0xa0/0xa0 [223968.816001] [] uart_console_write+0x35/0x70 [223968.816001] [] serial8250_console_write+0xbe/0x1a0 [223968.816001] [] __call_console_drivers+0x8e/0xb0 [223968.816001] [] _call_console_drivers+0x45/0x70 [223968.816001] [] console_unlock+0x17f/0x2b0 [223968.816001] [] vprintk+0x1fd/0x520 [223968.816001] [] printk+0x67/0x69 [223968.816001] [] ? _raw_spin_unlock+0x26/0x40 [223968.816001] [] ? account_system_time+0xab/0x190 [223968.816001] [] __rcu_pending+0x194/0x3b0 [223968.816001] [] ? tick_nohz_handler+0x100/0x100 [223968.816001] [] rcu_check_callbacks+0x67/0x130 [223968.816001] [] update_process_times+0x41/0x80 [223968.816001] [] tick_sched_timer+0x66/0xc0 [223968.816001] [] __run_hrtimer+0xfe/0x1e0 [223968.816001] [] hrtimer_interrupt+0xcd/0x1f0 [223968.816001] [] smp_apic_timer_interrupt+0x64/0xa0 [223968.816001] [] apic_timer_interrupt+0x73/0x80 [223968.816001] [] ? retint_restore_args+0x13/0x13 [223968.816001] [] ? native_read_tsc+0x6/0x20 [223968.816001] [] delay_tsc+0x3f/0xd0 [223968.816001] [] __delay+0xa/0x10 [223968.816001] [] do_raw_spin_lock+0x13b/0x180 [223968.816001] [] _raw_spin_lock+0x44/0x50 [223968.816001] [] ? tcp_keepalive_timer+0x23/0x270 [223968.816001] [] tcp_keepalive_timer+0x23/0x270 [223968.816001] [] run_timer_softirq+0x26d/0x410 [223968.816001] [] ? run_timer_softirq+0x198/0x410 [223968.816001] [] ? tcp_init_xmit_timers+0x20/0x20 [223968.816001] [] __do_softirq+0x138/0x250 [223968.816001] [] call_softirq+0x1c/0x30 [223968.816001] [] do_softirq+0x95/0xd0 [223968.816001] [] irq_exit+0xdd/0x110 [223968.816001] [] smp_apic_timer_interrupt+0x69/0xa0 [223968.816001] [] apic_timer_interrupt+0x73/0x80 [223968.816001] [] ? mwait_idle+0x145/0x170 [223968.816001] [] ? mwait_idle+0x14e/0x170 [223968.816001] [] ? mwait_idle+0x145/0x170 [223968.816001] [] cpu_idle+0x96/0xf0 [223968.816001] [] start_secondary+0x1ca/0x1ff [ goes on for another ~300kB, trimmed ]