From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stephen Hemminger Subject: [BUG] CPU lockup detected Date: Wed, 17 Oct 2007 08:35:54 -0700 Message-ID: <20071017083554.5a4e1f16@freepuppy.rosehill> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: netdev@vger.kernel.org To: Patrick McHardy Return-path: Received: from smtp2.linux-foundation.org ([207.189.120.14]:54727 "EHLO smtp2.linux-foundation.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934850AbXJQPgO (ORCPT ); Wed, 17 Oct 2007 11:36:14 -0400 Sender: netdev-owner@vger.kernel.org List-Id: netdev.vger.kernel.org Possible suspects TCP MD5 and/or netfilter conntrack. Begin forwarded message: Date: Wed, 17 Oct 2007 07:34:50 -0700 (PDT) From: bugme-daemon@bugzilla.kernel.org To: shemminger@linux-foundation.org Subject: [Bug 9173] New: BUG: soft lockup detected on CPU#0 - maybe related to TCP_MD5SIG http://bugzilla.kernel.org/show_bug.cgi?id=9173 Summary: BUG: soft lockup detected on CPU#0 - maybe related to TCP_MD5SIG Product: Networking Version: 2.5 KernelVersion: 2.6.20-9-server-lp2 Platform: All OS/Version: Linux Tree: Mainline Status: NEW Severity: normal Priority: P1 Component: IPV4 AssignedTo: shemminger@linux-foundation.org ReportedBy: tore@linpro.no CC: yoshfuji@linux-ipv6.org Most recent kernel where this bug did not occur: 2.6.12 (with TCP_MD5SIG-implementation from http://hasso.linux.ee/doku.php/english:network:rfc2385) Distribution: Ubuntu 6.06.1 LTS Hardware Environment: Sun X4100 (x86_64, SMP) Software Environment: Ubuntu 2.6.20-9-server-lp2 (-lp2 because it's recompiled with TCP_MD5SIG enabled), 64-bits userspace Problem Description: The server is a border router running Quagga for BGP and OSPF, and usually forwards 4-500Mbps worth of traffic between around 80 VLAN interfaces. Four network interfaces, bonded pairwise. It has three BGP sessions with MD5 signatures enabled. The server has an identical twin (for failover) which has also locked up like this, although it happens much more frequently on the active one (no matter which one is active, unfortunately). We've got lots of these servers, but only the border routers have had these lockups. Once in a while (say, once every four to six weeks) it will flood the console with BUG: soft lockup detected on CPU#0! and shortly after fail completely.This time I had increased the default prink level one notch and got the back traces too. I'm not used to reading those, but the md5sig stuff seems to stand out... I'll try to attach the trace somehow (got an error message about the bug being to large when attempting to include it here). Steps to reproduce: It happens completely out of the blue, so I don't know how. Tore --- Serial console started. To stop, type ESC ( [616333.267002] BUG: soft lockup detected on CPU#0! [616333.276201] [616333.276202] Call Trace: [616333.284407] [] softlockup_tick+0xfb/0x120 [616333.297135] [] update_process_times+0x57/0x90 [616333.309315] [] smp_local_timer_interrupt+0x34/0x60 [616333.322359] [] smp_apic_timer_interrupt+0x59/0x80 [616333.335230] [] apic_timer_interrupt+0x66/0x70 [616333.347410] [] __lock_text_start+0x7/0x10 [616333.358896] [] __tcp_get_md5sig_pool+0xf/0x50 [616333.371073] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616333.383954] [] :nf_conntrack:tcp_error+0x159/0x200 [616333.397006] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616333.411647] [] _read_lock_bh+0x9/0x20 [616333.422441] [] :ip_tables:ipt_do_table+0x305/0x370 [616333.435488] [] tcp_v4_do_rcv+0x10b/0x520 [616333.446805] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616333.462656] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616333.476742] [] nf_iterate+0x5c/0xa0 [616333.487190] [] tcp_v4_rcv+0xa0e/0xa80 [616333.497985] [] ip_local_deliver+0x1b2/0x280 [616333.509817] [] ip_rcv+0x4d7/0x540 [616333.519918] [] netif_receive_skb+0x2fa/0x330 [616333.531932] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616333.545332] [] :e1000:e1000_clean+0x7f/0x250 [616333.557336] [] run_rebalance_domains+0xe6/0x3c0 [616333.569861] [] net_rx_action+0xbe/0x1f0 [616333.581003] [] __do_softirq+0x63/0xd0 [616333.591796] [] call_softirq+0x1c/0x28 [616333.602587] [] do_softirq+0x2c/0x90 [616333.613036] [] do_IRQ+0xc6/0xf0 [616333.622789] [] ret_from_intr+0x0/0xa [616333.633405] [] tcp_free_md5sig_pool+0x17/0x60 [616333.646813] [] tcp_free_md5sig_pool+0x10/0x60 [616333.658991] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616333.672036] [] __pollwait+0x0/0x130 [616333.682484] [] default_wake_function+0x0/0x10 [616333.694662] [] tcp_transmit_skb+0x55c/0x790 [616333.706498] [] __tcp_push_pending_frames+0x80d/0x950 [616333.719887] [] tcp_sendmsg+0x0/0xb00 [616333.730508] [] tcp_sendmsg+0x9a9/0xb00 [616333.741478] [] sock_aio_write+0x13f/0x180 [616333.752970] [] do_sync_write+0xcf/0x120 [616333.764111] [] autoremove_wake_function+0x0/0x30 [616333.776809] [] lock_task_sighand+0x3a/0x80 [616333.788469] [] getrusage+0x23e/0x260 [616333.799090] [] vfs_write+0xf8/0x1b0 [616333.809541] [] sys_write+0x53/0x90 [616333.819815] [] system_call+0x7e/0x83 [616333.830435] [616343.252556] BUG: soft lockup detected on CPU#0! [616343.261784] [616343.261785] Call Trace: [616343.270020] [] softlockup_tick+0xfb/0x120 [616343.282731] [] update_process_times+0x57/0x90 [616343.294909] [] smp_local_timer_interrupt+0x34/0x60 [616343.307955] [] smp_apic_timer_interrupt+0x59/0x80 [616343.320825] [] apic_timer_interrupt+0x66/0x70 [616343.333007] [] __lock_text_start+0x7/0x10 [616343.344497] [] __tcp_get_md5sig_pool+0xf/0x50 [616343.356676] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616343.369554] [] :nf_conntrack:tcp_error+0x159/0x200 [616343.382603] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616343.397238] [] _read_lock_bh+0x9/0x20 [616343.408031] [] :ip_tables:ipt_do_table+0x305/0x370 [616343.421077] [] tcp_v4_do_rcv+0x10b/0x520 [616343.432394] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616343.448245] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616343.462330] [] nf_iterate+0x5c/0xa0 [616343.472778] [] tcp_v4_rcv+0xa0e/0xa80 [616343.483573] [] ip_local_deliver+0x1b2/0x280 [616343.495405] [] ip_rcv+0x4d7/0x540 [616343.505505] [] netif_receive_skb+0x2fa/0x330 [616343.517515] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616343.530915] [] :e1000:e1000_clean+0x7f/0x250 [616343.542920] [] run_rebalance_domains+0xe6/0x3c0 [616343.555445] [] net_rx_action+0xbe/0x1f0 [616343.566585] [] __do_softirq+0x63/0xd0 [616343.577378] [] call_softirq+0x1c/0x28 [616343.588169] [] do_softirq+0x2c/0x90 [616343.598614] [] do_IRQ+0xc6/0xf0 [616343.608366] [] ret_from_intr+0x0/0xa [616343.618982] [] tcp_free_md5sig_pool+0x17/0x60 [616343.632390] [] tcp_free_md5sig_pool+0x10/0x60 [616343.644570] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616343.657615] [] __pollwait+0x0/0x130 [616343.668063] [] default_wake_function+0x0/0x10 [616343.680242] [] tcp_transmit_skb+0x55c/0x790 [616343.692078] [] __tcp_push_pending_frames+0x80d/0x950 [616343.705467] [] tcp_sendmsg+0x0/0xb00 [616343.716090] [] tcp_sendmsg+0x9a9/0xb00 [616343.727059] [] sock_aio_write+0x13f/0x180 [616343.738551] [] do_sync_write+0xcf/0x120 [616343.749692] [] autoremove_wake_function+0x0/0x30 [616343.762393] [] lock_task_sighand+0x3a/0x80 [616343.774051] [] getrusage+0x23e/0x260 [616343.784673] [] vfs_write+0xf8/0x1b0 [616343.795119] [] sys_write+0x53/0x90 [616343.805392] [] system_call+0x7e/0x83 [616343.816012] [616353.238112] BUG: soft lockup detected on CPU#0! [616353.247343] [616353.247343] Call Trace: [616353.255579] [] softlockup_tick+0xfb/0x120 [616353.268290] [] update_process_times+0x57/0x90 [616353.280467] [] smp_local_timer_interrupt+0x34/0x60 [616353.293511] [] smp_apic_timer_interrupt+0x59/0x80 [616353.306387] [] apic_timer_interrupt+0x66/0x70 [616353.318567] [] __lock_text_start+0x7/0x10 [616353.330053] [] __tcp_get_md5sig_pool+0xf/0x50 [616353.342230] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616353.355108] [] :nf_conntrack:tcp_error+0x159/0x200 [616353.368157] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616353.382792] [] _read_lock_bh+0x9/0x20 [616353.393585] [] :ip_tables:ipt_do_table+0x305/0x370 [616353.406631] [] tcp_v4_do_rcv+0x10b/0x520 [616353.417948] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616353.433798] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616353.447883] [] nf_iterate+0x5c/0xa0 [616353.458331] [] tcp_v4_rcv+0xa0e/0xa80 [616353.469127] [] ip_local_deliver+0x1b2/0x280 [616353.480958] [] ip_rcv+0x4d7/0x540 [616353.491058] [] netif_receive_skb+0x2fa/0x330 [616353.503068] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616353.516468] [] :e1000:e1000_clean+0x7f/0x250 [616353.528472] [] run_rebalance_domains+0xe6/0x3c0 [616353.540997] [] net_rx_action+0xbe/0x1f0 [616353.552138] [] __do_softirq+0x63/0xd0 [616353.562931] [] call_softirq+0x1c/0x28 [616353.573724] [] do_softirq+0x2c/0x90 [616353.584169] [] do_IRQ+0xc6/0xf0 [616353.593921] [] ret_from_intr+0x0/0xa [616353.604536] [] tcp_free_md5sig_pool+0x17/0x60 [616353.617944] [] tcp_free_md5sig_pool+0x10/0x60 [616353.630124] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616353.643168] [] __pollwait+0x0/0x130 [616353.653616] [] default_wake_function+0x0/0x10 [616353.665797] [] tcp_transmit_skb+0x55c/0x790 [616353.677632] [] __tcp_push_pending_frames+0x80d/0x950 [616353.691021] [] tcp_sendmsg+0x0/0xb00 [616353.701643] [] tcp_sendmsg+0x9a9/0xb00 [616353.712612] [] sock_aio_write+0x13f/0x180 [616353.724104] [] do_sync_write+0xcf/0x120 [616353.735248] [] autoremove_wake_function+0x0/0x30 [616353.747948] [] lock_task_sighand+0x3a/0x80 [616353.759606] [] getrusage+0x23e/0x260 [616353.770226] [] vfs_write+0xf8/0x1b0 [616353.780670] [] sys_write+0x53/0x90 [616353.790943] [] system_call+0x7e/0x83 [616353.801563] [616363.223667] BUG: soft lockup detected on CPU#0! [616363.232899] [616363.232899] Call Trace: [616363.241133] [] softlockup_tick+0xfb/0x120 [616363.253845] [] update_process_times+0x57/0x90 [616363.266023] [] smp_local_timer_interrupt+0x34/0x60 [616363.279066] [] smp_apic_timer_interrupt+0x59/0x80 [616363.291937] [] apic_timer_interrupt+0x66/0x70 [616363.304120] [] __lock_text_start+0x5/0x10 [616363.315606] [] __tcp_get_md5sig_pool+0xf/0x50 [616363.327785] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616363.340662] [] :nf_conntrack:tcp_error+0x159/0x200 [616363.353711] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616363.368346] [] _read_lock_bh+0x9/0x20 [616363.379137] [] :ip_tables:ipt_do_table+0x305/0x370 [616363.392183] [] tcp_v4_do_rcv+0x10b/0x520 [616363.403501] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616363.419350] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616363.433432] [] nf_iterate+0x5c/0xa0 [616363.443878] [] tcp_v4_rcv+0xa0e/0xa80 [616363.454672] [] ip_local_deliver+0x1b2/0x280 [616363.466504] [] ip_rcv+0x4d7/0x540 [616363.476605] [] netif_receive_skb+0x2fa/0x330 [616363.488620] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616363.503705] [] :e1000:e1000_clean+0x7f/0x250 [616363.515709] [] run_rebalance_domains+0xe6/0x3c0 [616363.528234] [] net_rx_action+0xbe/0x1f0 [616363.539375] [] __do_softirq+0x63/0xd0 [616363.550168] [] call_softirq+0x1c/0x28 [616363.560962] [] do_softirq+0x2c/0x90 [616363.571410] [] do_IRQ+0xc6/0xf0 [616363.581162] [] ret_from_intr+0x0/0xa [616363.591778] [] tcp_free_md5sig_pool+0x17/0x60 [616363.605186] [] tcp_free_md5sig_pool+0x10/0x60 [616363.617363] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616363.630407] [] __pollwait+0x0/0x130 [616363.640855] [] default_wake_function+0x0/0x10 [616363.653035] [] tcp_transmit_skb+0x55c/0x790 [616363.664871] [] __tcp_push_pending_frames+0x80d/0x950 [616363.678260] [] tcp_sendmsg+0x0/0xb00 [616363.688882] [] tcp_sendmsg+0x9a9/0xb00 [616363.699851] [] sock_aio_write+0x13f/0x180 [616363.711342] [] do_sync_write+0xcf/0x120 [616363.722483] [] autoremove_wake_function+0x0/0x30 [616363.735181] [] lock_task_sighand+0x3a/0x80 [616363.746838] [] getrusage+0x23e/0x260 [616363.757461] [] vfs_write+0xf8/0x1b0 [616363.767911] [] sys_write+0x53/0x90 [616363.778185] [] system_call+0x7e/0x83 [616363.788805] [616373.209223] BUG: soft lockup detected on CPU#0! [616373.218454] [616373.218455] Call Trace: [616373.226689] [] softlockup_tick+0xfb/0x120 [616373.239400] [] update_process_times+0x57/0x90 [616373.251580] [] smp_local_timer_interrupt+0x34/0x60 [616373.264623] [] smp_apic_timer_interrupt+0x59/0x80 [616373.277493] [] apic_timer_interrupt+0x66/0x70 [616373.289673] [] __lock_text_start+0x7/0x10 [616373.301158] [] __tcp_get_md5sig_pool+0xf/0x50 [616373.313335] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616373.326212] [] :nf_conntrack:tcp_error+0x159/0x200 [616373.339261] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616373.353899] [] _read_lock_bh+0x9/0x20 [616373.364696] [] :ip_tables:ipt_do_table+0x305/0x370 [616373.377741] [] tcp_v4_do_rcv+0x10b/0x520 [616373.389057] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616373.404907] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616373.418992] [] nf_iterate+0x5c/0xa0 [616373.429440] [] tcp_v4_rcv+0xa0e/0xa80 [616373.440235] [] ip_local_deliver+0x1b2/0x280 [616373.452068] [] ip_rcv+0x4d7/0x540 [616373.462168] [] netif_receive_skb+0x2fa/0x330 [616373.474178] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616373.487578] [] :e1000:e1000_clean+0x7f/0x250 [616373.499582] [] run_rebalance_domains+0xe6/0x3c0 [616373.512107] [] net_rx_action+0xbe/0x1f0 [616373.523248] [] __do_softirq+0x63/0xd0 [616373.534039] [] call_softirq+0x1c/0x28 [616373.544830] [] do_softirq+0x2c/0x90 [616373.555275] [] do_IRQ+0xc6/0xf0 [616373.565027] [] ret_from_intr+0x0/0xa [616373.575643] [] tcp_free_md5sig_pool+0x17/0x60 [616373.589051] [] tcp_free_md5sig_pool+0x10/0x60 [616373.601229] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616373.614273] [] __pollwait+0x0/0x130 [616373.624720] [] default_wake_function+0x0/0x10 [616373.636901] [] tcp_transmit_skb+0x55c/0x790 [616373.648735] [] __tcp_push_pending_frames+0x80d/0x950 [616373.662124] [] tcp_sendmsg+0x0/0xb00 [616373.672746] [] tcp_sendmsg+0x9a9/0xb00 [616373.683715] [] sock_aio_write+0x13f/0x180 [616373.695206] [] do_sync_write+0xcf/0x120 [616373.706346] [] autoremove_wake_function+0x0/0x30 [616373.719043] [] lock_task_sighand+0x3a/0x80 [616373.730703] [] getrusage+0x23e/0x260 [616373.741325] [] vfs_write+0xf8/0x1b0 [616373.751771] [] sys_write+0x53/0x90 [616373.762044] [] system_call+0x7e/0x83 [616373.772664] [616383.194779] BUG: soft lockup detected on CPU#0! [616383.204008] [616383.204008] Call Trace: [616383.212240] [] softlockup_tick+0xfb/0x120 [616383.224949] [] update_process_times+0x57/0x90 [616383.237128] [] smp_local_timer_interrupt+0x34/0x60 [616383.250172] [] smp_apic_timer_interrupt+0x59/0x80 [616383.263043] [] apic_timer_interrupt+0x66/0x70 [616383.275225] [] __lock_text_start+0x5/0x10 [616383.286712] [] __tcp_get_md5sig_pool+0xf/0x50 [616383.298888] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616383.311764] [] :nf_conntrack:tcp_error+0x159/0x200 [616383.324813] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616383.339450] [] _read_lock_bh+0x9/0x20 [616383.350244] [] :ip_tables:ipt_do_table+0x305/0x370 [616383.363289] [] tcp_v4_do_rcv+0x10b/0x520 [616383.374606] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616383.390457] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616383.404540] [] nf_iterate+0x5c/0xa0 [616383.414986] [] tcp_v4_rcv+0xa0e/0xa80 [616383.425782] [] ip_local_deliver+0x1b2/0x280 [616383.437613] [] ip_rcv+0x4d7/0x540 [616383.447713] [] netif_receive_skb+0x2fa/0x330 [616383.459725] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616383.473124] [] :e1000:e1000_clean+0x7f/0x250 [616383.485143] [] run_rebalance_domains+0xe6/0x3c0 [616383.497670] [] net_rx_action+0xbe/0x1f0 [616383.508810] [] __do_softirq+0x63/0xd0 [616383.519603] [] call_softirq+0x1c/0x28 [616383.530396] [] do_softirq+0x2c/0x90 [616383.540841] [] do_IRQ+0xc6/0xf0 [616383.550593] [] ret_from_intr+0x0/0xa [616383.561208] [] tcp_free_md5sig_pool+0x17/0x60 [616383.574616] [] tcp_free_md5sig_pool+0x10/0x60 [616383.586793] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616383.599842] [] __pollwait+0x0/0x130 [616383.610290] [] default_wake_function+0x0/0x10 [616383.622468] [] tcp_transmit_skb+0x55c/0x790 [616383.634303] [] __tcp_push_pending_frames+0x80d/0x950 [616383.647696] [] tcp_sendmsg+0x0/0xb00 [616383.658319] [] tcp_sendmsg+0x9a9/0xb00 [616383.669288] [] sock_aio_write+0x13f/0x180 [616383.680780] [] do_sync_write+0xcf/0x120 [616383.691922] [] autoremove_wake_function+0x0/0x30 [616383.704621] [] lock_task_sighand+0x3a/0x80 [616383.716281] [] getrusage+0x23e/0x260 [616383.726903] [] vfs_write+0xf8/0x1b0 [616383.737349] [] sys_write+0x53/0x90 [616383.747622] [] system_call+0x7e/0x83 [616383.758242] [616393.180335] BUG: soft lockup detected on CPU#0! [616393.189563] [616393.189564] Call Trace: [616393.197798] [] softlockup_tick+0xfb/0x120 [616393.210509] [] update_process_times+0x57/0x90 [616393.222687] [] smp_local_timer_interrupt+0x34/0x60 [616393.235730] [] smp_apic_timer_interrupt+0x59/0x80 [616393.248603] [] apic_timer_interrupt+0x66/0x70 [616393.260785] [] __lock_text_start+0x7/0x10 [616393.272272] [] __tcp_get_md5sig_pool+0xf/0x50 [616393.284451] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616393.297328] [] :nf_conntrack:tcp_error+0x159/0x200 [616393.310381] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616393.325019] [] _read_lock_bh+0x9/0x20 [616393.335812] [] :ip_tables:ipt_do_table+0x305/0x370 [616393.348859] [] tcp_v4_do_rcv+0x10b/0x520 [616393.360176] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616393.376025] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616393.390110] [] nf_iterate+0x5c/0xa0 [616393.400557] [] tcp_v4_rcv+0xa0e/0xa80 [616393.411349] [] ip_local_deliver+0x1b2/0x280 [616393.423182] [] ip_rcv+0x4d7/0x540 [616393.433282] [] netif_receive_skb+0x2fa/0x330 [616393.445292] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616393.458691] [] :e1000:e1000_clean+0x7f/0x250 [616393.470696] [] run_rebalance_domains+0xe6/0x3c0 [616393.483223] [] net_rx_action+0xbe/0x1f0 [616393.494363] [] __do_softirq+0x63/0xd0 [616393.505156] [] call_softirq+0x1c/0x28 [616393.515949] [] do_softirq+0x2c/0x90 [616393.526394] [] do_IRQ+0xc6/0xf0 [616393.536147] [] ret_from_intr+0x0/0xa [616393.546765] [] tcp_free_md5sig_pool+0x17/0x60 [616393.560174] [] tcp_free_md5sig_pool+0x10/0x60 [616393.572352] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616393.585399] [] __pollwait+0x0/0x130 [616393.595848] [] default_wake_function+0x0/0x10 [616393.608038] [] tcp_transmit_skb+0x55c/0x790 [616393.619872] [] __tcp_push_pending_frames+0x80d/0x950 [616393.633261] [] tcp_sendmsg+0x0/0xb00 [616393.643883] [] tcp_sendmsg+0x9a9/0xb00 [616393.654852] [] sock_aio_write+0x13f/0x180 [616393.666344] [] do_sync_write+0xcf/0x120 [616393.677484] [] autoremove_wake_function+0x0/0x30 [616393.690183] [] lock_task_sighand+0x3a/0x80 [616393.701843] [] getrusage+0x23e/0x260 [616393.712466] [] vfs_write+0xf8/0x1b0 [616393.722912] [] sys_write+0x53/0x90 [616393.734870] [] system_call+0x7e/0x83 [616393.745489] [616403.165890] BUG: soft lockup detected on CPU#0! [616403.175119] [616403.175120] Call Trace: [616403.183353] [] softlockup_tick+0xfb/0x120 [616403.196062] [] update_process_times+0x57/0x90 [616403.208240] [] smp_local_timer_interrupt+0x34/0x60 [616403.221283] [] smp_apic_timer_interrupt+0x59/0x80 [616403.234156] [] apic_timer_interrupt+0x66/0x70 [616403.246335] [] __lock_text_start+0x5/0x10 [616403.257821] [] __tcp_get_md5sig_pool+0xf/0x50 [616403.270000] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616403.282877] [] :nf_conntrack:tcp_error+0x159/0x200 [616403.295926] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616403.310561] [] _read_lock_bh+0x9/0x20 [616403.321353] [] :ip_tables:ipt_do_table+0x305/0x370 [616403.334399] [] tcp_v4_do_rcv+0x10b/0x520 [616403.345716] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616403.361564] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616403.375647] [] nf_iterate+0x5c/0xa0 [616403.386094] [] tcp_v4_rcv+0xa0e/0xa80 [616403.396892] [] ip_local_deliver+0x1b2/0x280 [616403.408726] [] ip_rcv+0x4d7/0x540 [616403.418830] [] netif_receive_skb+0x2fa/0x330 [616403.430840] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616403.444239] [] :e1000:e1000_clean+0x7f/0x250 [616403.456242] [] run_rebalance_domains+0xe6/0x3c0 [616403.468768] [] net_rx_action+0xbe/0x1f0 [616403.479907] [] __do_softirq+0x63/0xd0 [616403.490700] [] call_softirq+0x1c/0x28 [616403.501493] [] do_softirq+0x2c/0x90 [616403.511938] [] do_IRQ+0xc6/0xf0 [616403.521692] [] ret_from_intr+0x0/0xa [616403.532306] [] tcp_free_md5sig_pool+0x17/0x60 [616403.545715] [] tcp_free_md5sig_pool+0x10/0x60 [616403.557894] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616403.570938] [] __pollwait+0x0/0x130 [616403.581390] [] default_wake_function+0x0/0x10 [616403.593568] [] tcp_transmit_skb+0x55c/0x790 [616403.605403] [] __tcp_push_pending_frames+0x80d/0x950 [616403.618792] [] tcp_sendmsg+0x0/0xb00 [616403.629415] [] tcp_sendmsg+0x9a9/0xb00 [616403.640384] [] sock_aio_write+0x13f/0x180 [616403.651875] [] do_sync_write+0xcf/0x120 [616403.663014] [] autoremove_wake_function+0x0/0x30 [616403.675713] [] lock_task_sighand+0x3a/0x80 [616403.687373] [] getrusage+0x23e/0x260 [616403.697994] [] vfs_write+0xf8/0x1b0 [616403.708440] [] sys_write+0x53/0x90 [616403.718714] [] system_call+0x7e/0x83 [616403.729334] [616413.151446] BUG: soft lockup detected on CPU#0! [616413.160677] [616413.160678] Call Trace: [616413.168911] [] softlockup_tick+0xfb/0x120 [616413.181620] [] update_process_times+0x57/0x90 [616413.193798] [] smp_local_timer_interrupt+0x34/0x60 [616413.206841] [] smp_apic_timer_interrupt+0x59/0x80 [616413.219712] [] apic_timer_interrupt+0x66/0x70 [616413.231893] [] __lock_text_start+0x5/0x10 [616413.243378] [] __tcp_get_md5sig_pool+0xf/0x50 [616413.255557] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616413.268433] [] :nf_conntrack:tcp_error+0x159/0x200 [616413.281482] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616413.296118] [] _read_lock_bh+0x9/0x20 [616413.306911] [] :ip_tables:ipt_do_table+0x305/0x370 [616413.319957] [] tcp_v4_do_rcv+0x10b/0x520 [616413.331274] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616413.347124] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616413.361210] [] nf_iterate+0x5c/0xa0 [616413.371657] [] tcp_v4_rcv+0xa0e/0xa80 [616413.382449] [] ip_local_deliver+0x1b2/0x280 [616413.394282] [] ip_rcv+0x4d7/0x540 [616413.404382] [] netif_receive_skb+0x2fa/0x330 [616413.416392] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616413.429791] [] :e1000:e1000_clean+0x7f/0x250 [616413.441796] [] run_rebalance_domains+0xe6/0x3c0 [616413.454323] [] net_rx_action+0xbe/0x1f0 [616413.465469] [] __do_softirq+0x63/0xd0 [616413.476262] [] call_softirq+0x1c/0x28 [616413.487056] [] do_softirq+0x2c/0x90 [616413.497501] [] do_IRQ+0xc6/0xf0 [616413.507252] [] ret_from_intr+0x0/0xa [616413.517867] [] tcp_free_md5sig_pool+0x17/0x60 [616413.531279] [] tcp_free_md5sig_pool+0x10/0x60 [616413.543457] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616413.556501] [] __pollwait+0x0/0x130 [616413.566950] [] default_wake_function+0x0/0x10 [616413.579135] [] tcp_transmit_skb+0x55c/0x790 [616413.590970] [] __tcp_push_pending_frames+0x80d/0x950 [616413.604358] [] tcp_sendmsg+0x0/0xb00 [616413.614980] [] tcp_sendmsg+0x9a9/0xb00 [616413.625948] [] sock_aio_write+0x13f/0x180 [616413.637441] [] do_sync_write+0xcf/0x120 [616413.648581] [] autoremove_wake_function+0x0/0x30 [616413.661278] [] lock_task_sighand+0x3a/0x80 [616413.672936] [] getrusage+0x23e/0x260 [616413.683557] [] vfs_write+0xf8/0x1b0 [616413.694003] [] sys_write+0x53/0x90 [616413.704276] [] system_call+0x7e/0x83 [616413.714896] [616423.137002] BUG: soft lockup detected on CPU#0! [616423.146231] [616423.146231] Call Trace: [616423.154466] [] softlockup_tick+0xfb/0x120 [616423.167178] [] update_process_times+0x57/0x90 [616423.179357] [] smp_local_timer_interrupt+0x34/0x60 [616423.192400] [] smp_apic_timer_interrupt+0x59/0x80 [616423.205270] [] apic_timer_interrupt+0x66/0x70 [616423.217452] [] __lock_text_start+0x5/0x10 [616423.228937] [] __tcp_get_md5sig_pool+0xf/0x50 [616423.241116] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616423.253993] [] :nf_conntrack:tcp_error+0x159/0x200 [616423.267044] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616423.281684] [] _read_lock_bh+0x9/0x20 [616423.292477] [] :ip_tables:ipt_do_table+0x305/0x370 [616423.305523] [] tcp_v4_do_rcv+0x10b/0x520 [616423.316840] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616423.332689] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616423.346774] [] nf_iterate+0x5c/0xa0 [616423.357222] [] tcp_v4_rcv+0xa0e/0xa80 [616423.368016] [] ip_local_deliver+0x1b2/0x280 [616423.379850] [] ip_rcv+0x4d7/0x540 [616423.389950] [] netif_receive_skb+0x2fa/0x330 [616423.401961] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616423.415365] [] :e1000:e1000_clean+0x7f/0x250 [616423.427370] [] run_rebalance_domains+0xe6/0x3c0 [616423.439895] [] net_rx_action+0xbe/0x1f0 [616423.451034] [] __do_softirq+0x63/0xd0 [616423.461825] [] call_softirq+0x1c/0x28 [616423.472616] [] do_softirq+0x2c/0x90 [616423.483061] [] do_IRQ+0xc6/0xf0 [616423.492813] [] ret_from_intr+0x0/0xa [616423.503428] [] tcp_free_md5sig_pool+0x17/0x60 [616423.516836] [] tcp_free_md5sig_pool+0x10/0x60 [616423.529020] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616423.542064] [] __pollwait+0x0/0x130 [616423.552513] [] default_wake_function+0x0/0x10 [616423.564693] [] tcp_transmit_skb+0x55c/0x790 [616423.576529] [] __tcp_push_pending_frames+0x80d/0x950 [616423.589918] [] tcp_sendmsg+0x0/0xb00 [616423.600540] [] tcp_sendmsg+0x9a9/0xb00 [616423.611509] [] sock_aio_write+0x13f/0x180 [616423.623001] [] do_sync_write+0xcf/0x120 [616423.634141] [] autoremove_wake_function+0x0/0x30 [616423.646840] [] lock_task_sighand+0x3a/0x80 [616423.658497] [] getrusage+0x23e/0x260 [616423.669119] [] vfs_write+0xf8/0x1b0 [616423.679565] [] sys_write+0x53/0x90 [616423.689840] [] system_call+0x7e/0x83 [616423.700460] [616433.122558] BUG: soft lockup detected on CPU#0! [616433.131787] [616433.131787] Call Trace: [616433.140022] [] softlockup_tick+0xfb/0x120 [616433.152733] [] update_process_times+0x57/0x90 [616433.164913] [] smp_local_timer_interrupt+0x34/0x60 [616433.177956] [] smp_apic_timer_interrupt+0x59/0x80 [616433.190829] [] apic_timer_interrupt+0x66/0x70 [616433.203008] [] __lock_text_start+0x7/0x10 [616433.214494] [] __tcp_get_md5sig_pool+0xf/0x50 [616433.226672] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616433.239554] [] :nf_conntrack:tcp_error+0x159/0x200 [616433.252605] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616433.267241] [] _read_lock_bh+0x9/0x20 [616433.278035] [] :ip_tables:ipt_do_table+0x305/0x370 [616433.291079] [] tcp_v4_do_rcv+0x10b/0x520 [616433.302400] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616433.318251] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616433.332334] [] nf_iterate+0x5c/0xa0 [616433.342780] [] tcp_v4_rcv+0xa0e/0xa80 [616433.355262] [] ip_local_deliver+0x1b2/0x280 [616433.367095] [] ip_rcv+0x4d7/0x540 [616433.377199] [] netif_receive_skb+0x2fa/0x330 [616433.389209] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616433.402609] [] :e1000:e1000_clean+0x7f/0x250 [616433.414614] [] run_rebalance_domains+0xe6/0x3c0 [616433.427139] [] net_rx_action+0xbe/0x1f0 [616433.438279] [] __do_softirq+0x63/0xd0 [616433.449073] [] call_softirq+0x1c/0x28 [616433.459866] [] do_softirq+0x2c/0x90 [616433.470310] [] do_IRQ+0xc6/0xf0 [616433.480063] [] ret_from_intr+0x0/0xa [616433.490678] [] tcp_free_md5sig_pool+0x17/0x60 [616433.504085] [] tcp_free_md5sig_pool+0x10/0x60 [616433.516264] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616433.529309] [] __pollwait+0x0/0x130 [616433.539761] [] default_wake_function+0x0/0x10 [616433.551939] [] tcp_transmit_skb+0x55c/0x790 [616433.563775] [] __tcp_push_pending_frames+0x80d/0x950 [616433.577164] [] tcp_sendmsg+0x0/0xb00 [616433.587786] [] tcp_sendmsg+0x9a9/0xb00 [616433.598755] [] sock_aio_write+0x13f/0x180 [616433.610247] [] do_sync_write+0xcf/0x120 [616433.621390] [] autoremove_wake_function+0x0/0x30 [616433.634089] [] lock_task_sighand+0x3a/0x80 [616433.645749] [] getrusage+0x23e/0x260 [616433.656369] [] vfs_write+0xf8/0x1b0 [616433.666815] [] sys_write+0x53/0x90 [616433.677088] [] system_call+0x7e/0x83 [616433.687710] [616443.108113] BUG: soft lockup detected on CPU#0! [616443.117342] [616443.117343] Call Trace: [616443.125582] [] softlockup_tick+0xfb/0x120 [616443.138293] [] update_process_times+0x57/0x90 [616443.150473] [] smp_local_timer_interrupt+0x34/0x60 [616443.163520] [] smp_apic_timer_interrupt+0x59/0x80 [616443.176391] [] apic_timer_interrupt+0x66/0x70 [616443.188571] [] __lock_text_start+0x5/0x10 [616443.200057] [] __tcp_get_md5sig_pool+0xf/0x50 [616443.212234] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616443.225110] [] :nf_conntrack:tcp_error+0x159/0x200 [616443.238159] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616443.252795] [] _read_lock_bh+0x9/0x20 [616443.263588] [] :ip_tables:ipt_do_table+0x305/0x370 [616443.276633] [] tcp_v4_do_rcv+0x10b/0x520 [616443.287947] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616443.303798] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616443.317881] [] nf_iterate+0x5c/0xa0 [616443.328330] [] tcp_v4_rcv+0xa0e/0xa80 [616443.339124] [] ip_local_deliver+0x1b2/0x280 [616443.350957] [] ip_rcv+0x4d7/0x540 [616443.361057] [] netif_receive_skb+0x2fa/0x330 [616443.373067] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616443.386466] [] :e1000:e1000_clean+0x7f/0x250 [616443.398470] [] run_rebalance_domains+0xe6/0x3c0 [616443.410995] [] net_rx_action+0xbe/0x1f0 [616443.422135] [] __do_softirq+0x63/0xd0 [616443.432928] [] call_softirq+0x1c/0x28 [616443.443721] [] do_softirq+0x2c/0x90 [616443.454167] [] do_IRQ+0xc6/0xf0 [616443.463918] [] ret_from_intr+0x0/0xa [616443.474534] [] tcp_free_md5sig_pool+0x17/0x60 [616443.487942] [] tcp_free_md5sig_pool+0x10/0x60 [616443.500119] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616443.513164] [] __pollwait+0x0/0x130 [616443.523613] [] default_wake_function+0x0/0x10 [616443.535792] [] tcp_transmit_skb+0x55c/0x790 [616443.547627] [] __tcp_push_pending_frames+0x80d/0x950 [616443.561016] [] tcp_sendmsg+0x0/0xb00 [616443.571638] [] tcp_sendmsg+0x9a9/0xb00 [616443.582606] [] sock_aio_write+0x13f/0x180 [616443.594099] [] do_sync_write+0xcf/0x120 [616443.605239] [] autoremove_wake_function+0x0/0x30 [616443.617938] [] lock_task_sighand+0x3a/0x80 [616443.629598] [] getrusage+0x23e/0x260 [616443.640226] [] vfs_write+0xf8/0x1b0 [616443.650672] [] sys_write+0x53/0x90 [616443.660944] [] system_call+0x7e/0x83 [616443.671564] [616453.093669] BUG: soft lockup detected on CPU#0! [616453.102898] [616453.102899] Call Trace: [616453.111131] [] softlockup_tick+0xfb/0x120 [616453.123842] [] update_process_times+0x57/0x90 [616453.136020] [] smp_local_timer_interrupt+0x34/0x60 [616453.149063] [] smp_apic_timer_interrupt+0x59/0x80 [616453.161936] [] apic_timer_interrupt+0x66/0x70 [616453.174115] [] __lock_text_start+0x7/0x10 [616453.185600] [] __tcp_get_md5sig_pool+0xf/0x50 [616453.197778] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616453.210655] [] :nf_conntrack:tcp_error+0x159/0x200 [616453.223704] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616453.238340] [] _read_lock_bh+0x9/0x20 [616453.249134] [] :ip_tables:ipt_do_table+0x305/0x370 [616453.262180] [] tcp_v4_do_rcv+0x10b/0x520 [616453.273497] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616453.289345] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616453.303428] [] nf_iterate+0x5c/0xa0 [616453.313876] [] tcp_v4_rcv+0xa0e/0xa80 [616453.324671] [] ip_local_deliver+0x1b2/0x280 [616453.336504] [] ip_rcv+0x4d7/0x540 [616453.346605] [] netif_receive_skb+0x2fa/0x330 [616453.358616] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616453.372016] [] :e1000:e1000_clean+0x7f/0x250 [616453.384021] [] run_rebalance_domains+0xe6/0x3c0 [616453.396546] [] net_rx_action+0xbe/0x1f0 [616453.407688] [] __do_softirq+0x63/0xd0 [616453.418481] [] call_softirq+0x1c/0x28 [616453.429272] [] do_softirq+0x2c/0x90 [616453.439717] [] do_IRQ+0xc6/0xf0 [616453.449473] [] ret_from_intr+0x0/0xa [616453.460089] [] tcp_free_md5sig_pool+0x17/0x60 [616453.473497] [] tcp_free_md5sig_pool+0x10/0x60 [616453.485676] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616453.498720] [] __pollwait+0x0/0x130 [616453.509169] [] default_wake_function+0x0/0x10 [616453.521349] [] tcp_transmit_skb+0x55c/0x790 [616453.533186] [] __tcp_push_pending_frames+0x80d/0x950 [616453.546575] [] tcp_sendmsg+0x0/0xb00 [616453.557211] [] tcp_sendmsg+0x9a9/0xb00 [616453.568184] [] sock_aio_write+0x13f/0x180 [616453.579675] [] do_sync_write+0xcf/0x120 [616453.590815] [] autoremove_wake_function+0x0/0x30 [616453.603513] [] lock_task_sighand+0x3a/0x80 [616453.615170] [] getrusage+0x23e/0x260 [616453.625792] [] vfs_write+0xf8/0x1b0 [616453.636238] [] sys_write+0x53/0x90 [616453.646511] [] system_call+0x7e/0x83 [616453.657130] [616463.079225] BUG: soft lockup detected on CPU#0! [616463.088454] [616463.088455] Call Trace: [616463.096689] [] softlockup_tick+0xfb/0x120 [616463.109400] [] update_process_times+0x57/0x90 [616463.121578] [] smp_local_timer_interrupt+0x34/0x60 [616463.134621] [] smp_apic_timer_interrupt+0x59/0x80 [616463.147493] [] apic_timer_interrupt+0x66/0x70 [616463.159672] [] __lock_text_start+0x5/0x10 [616463.171157] [] __tcp_get_md5sig_pool+0xf/0x50 [616463.183333] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616463.196210] [] :nf_conntrack:tcp_error+0x159/0x200 [616463.209259] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616463.223898] [] _read_lock_bh+0x9/0x20 [616463.234691] [] :ip_tables:ipt_do_table+0x305/0x370 [616463.247738] [] tcp_v4_do_rcv+0x10b/0x520 [616463.259054] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616463.274904] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616463.288989] [] nf_iterate+0x5c/0xa0 [616463.299437] [] tcp_v4_rcv+0xa0e/0xa80 [616463.310231] [] ip_local_deliver+0x1b2/0x280 [616463.322064] [] ip_rcv+0x4d7/0x540 [616463.332164] [] netif_receive_skb+0x2fa/0x330 [616463.344175] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616463.357574] [] :e1000:e1000_clean+0x7f/0x250 [616463.369579] [] run_rebalance_domains+0xe6/0x3c0 [616463.382104] [] net_rx_action+0xbe/0x1f0 [616463.393243] [] __do_softirq+0x63/0xd0 [616463.404037] [] call_softirq+0x1c/0x28 [616463.414829] [] do_softirq+0x2c/0x90 [616463.425273] [] do_IRQ+0xc6/0xf0 [616463.435026] [] ret_from_intr+0x0/0xa [616463.445642] [] tcp_free_md5sig_pool+0x17/0x60 [616463.459050] [] tcp_free_md5sig_pool+0x10/0x60 [616463.471228] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616463.484272] [] __pollwait+0x0/0x130 [616463.494719] [] default_wake_function+0x0/0x10 [616463.506900] [] tcp_transmit_skb+0x55c/0x790 [616463.518733] [] __tcp_push_pending_frames+0x80d/0x950 [616463.532123] [] tcp_sendmsg+0x0/0xb00 [616463.542745] [] tcp_sendmsg+0x9a9/0xb00 [616463.553714] [] sock_aio_write+0x13f/0x180 [616463.565205] [] do_sync_write+0xcf/0x120 [616463.576346] [] autoremove_wake_function+0x0/0x30 [616463.590729] [] lock_task_sighand+0x3a/0x80 [616463.602390] [] getrusage+0x23e/0x260 [616463.613011] [] vfs_write+0xf8/0x1b0 [616463.623456] [] sys_write+0x53/0x90 [616463.633730] [] system_call+0x7e/0x83 [616463.644350] [616473.064781] BUG: soft lockup detected on CPU#0! [616473.074010] [616473.074011] Call Trace: [616473.082245] [] softlockup_tick+0xfb/0x120 [616473.094956] [] update_process_times+0x57/0x90 [616473.107135] [] smp_local_timer_interrupt+0x34/0x60 [616473.120180] [] smp_apic_timer_interrupt+0x59/0x80 [616473.133050] [] apic_timer_interrupt+0x66/0x70 [616473.145230] [] __lock_text_start+0x7/0x10 [616473.156715] [] __tcp_get_md5sig_pool+0xf/0x50 [616473.168894] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616473.181770] [] :nf_conntrack:tcp_error+0x159/0x200 [616473.194821] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616473.209457] [] _read_lock_bh+0x9/0x20 [616473.220250] [] :ip_tables:ipt_do_table+0x305/0x370 [616473.233295] [] tcp_v4_do_rcv+0x10b/0x520 [616473.244612] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616473.260465] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616473.274548] [] nf_iterate+0x5c/0xa0 [616473.284994] [] tcp_v4_rcv+0xa0e/0xa80 [616473.295789] [] ip_local_deliver+0x1b2/0x280 [616473.307621] [] ip_rcv+0x4d7/0x540 [616473.317723] [] netif_receive_skb+0x2fa/0x330 [616473.329735] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616473.343134] [] :e1000:e1000_clean+0x7f/0x250 [616473.355140] [] run_rebalance_domains+0xe6/0x3c0 [616473.367665] [] net_rx_action+0xbe/0x1f0 [616473.378806] [] __do_softirq+0x63/0xd0 [616473.389599] [] call_softirq+0x1c/0x28 [616473.400392] [] do_softirq+0x2c/0x90 [616473.410837] [] do_IRQ+0xc6/0xf0 [616473.420588] [] ret_from_intr+0x0/0xa [616473.431203] [] tcp_free_md5sig_pool+0x17/0x60 [616473.444611] [] tcp_free_md5sig_pool+0x10/0x60 [616473.456787] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616473.469832] [] __pollwait+0x0/0x130 [616473.480280] [] default_wake_function+0x0/0x10 [616473.492457] [] tcp_transmit_skb+0x55c/0x790 [616473.504292] [] __tcp_push_pending_frames+0x80d/0x950 [616473.517680] [] tcp_sendmsg+0x0/0xb00 [616473.528301] [] tcp_sendmsg+0x9a9/0xb00 [616473.539270] [] sock_aio_write+0x13f/0x180 [616473.550764] [] do_sync_write+0xcf/0x120 [616473.561904] [] autoremove_wake_function+0x0/0x30 [616473.574603] [] lock_task_sighand+0x3a/0x80 [616473.586261] [] getrusage+0x23e/0x260 [616473.596884] [] vfs_write+0xf8/0x1b0 [616473.607330] [] sys_write+0x53/0x90 [616473.617604] [] system_call+0x7e/0x83 [616473.628224] [616483.050336] BUG: soft lockup detected on CPU#0! [616483.059565] [616483.059566] Call Trace: [616483.067801] [] softlockup_tick+0xfb/0x120 [616483.080511] [] update_process_times+0x57/0x90 [616483.092692] [] smp_local_timer_interrupt+0x34/0x60 [616483.105735] [] smp_apic_timer_interrupt+0x59/0x80 [616483.118612] [] apic_timer_interrupt+0x66/0x70 [616483.130791] [] __lock_text_start+0x7/0x10 [616483.142278] [] __tcp_get_md5sig_pool+0xf/0x50 [616483.154454] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616483.167332] [] :nf_conntrack:tcp_error+0x159/0x200 [616483.180381] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616483.195016] [] _read_lock_bh+0x9/0x20 [616483.205807] [] :ip_tables:ipt_do_table+0x305/0x370 [616483.218853] [] tcp_v4_do_rcv+0x10b/0x520 [616483.230170] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616483.246020] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616483.260106] [] nf_iterate+0x5c/0xa0 [616483.270552] [] tcp_v4_rcv+0xa0e/0xa80 [616483.281347] [] ip_local_deliver+0x1b2/0x280 [616483.293178] [] ip_rcv+0x4d7/0x540 [616483.303279] [] netif_receive_skb+0x2fa/0x330 [616483.315290] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616483.328689] [] :e1000:e1000_clean+0x7f/0x250 [616483.340694] [] run_rebalance_domains+0xe6/0x3c0 [616483.353220] [] net_rx_action+0xbe/0x1f0 [616483.364359] [] __do_softirq+0x63/0xd0 [616483.375153] [] call_softirq+0x1c/0x28 [616483.385944] [] do_softirq+0x2c/0x90 [616483.396388] [] do_IRQ+0xc6/0xf0 [616483.406141] [] ret_from_intr+0x0/0xa [616483.416756] [] tcp_free_md5sig_pool+0x17/0x60 [616483.430165] [] tcp_free_md5sig_pool+0x10/0x60 [616483.442343] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616483.455388] [] __pollwait+0x0/0x130 [616483.465837] [] default_wake_function+0x0/0x10 [616483.478017] [] tcp_transmit_skb+0x55c/0x790 [616483.489853] [] __tcp_push_pending_frames+0x80d/0x950 [616483.503242] [] tcp_sendmsg+0x0/0xb00 [616483.513864] [] tcp_sendmsg+0x9a9/0xb00 [616483.524833] [] sock_aio_write+0x13f/0x180 [616483.536324] [] do_sync_write+0xcf/0x120 [616483.547469] [] autoremove_wake_function+0x0/0x30 [616483.560168] [] lock_task_sighand+0x3a/0x80 [616483.571828] [] getrusage+0x23e/0x260 [616483.582450] [] vfs_write+0xf8/0x1b0 [616483.592897] [] sys_write+0x53/0x90 [616483.603169] [] system_call+0x7e/0x83 [616483.613788] [616493.035892] BUG: soft lockup detected on CPU#0! [616493.045121] [616493.045122] Call Trace: [616493.053357] [] softlockup_tick+0xfb/0x120 [616493.066068] [] update_process_times+0x57/0x90 [616493.078247] [] smp_local_timer_interrupt+0x34/0x60 [616493.091289] [] smp_apic_timer_interrupt+0x59/0x80 [616493.104160] [] apic_timer_interrupt+0x66/0x70 [616493.116342] [] __lock_text_start+0x7/0x10 [616493.127827] [] __tcp_get_md5sig_pool+0xf/0x50 [616493.140004] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616493.152882] [] :nf_conntrack:tcp_error+0x159/0x200 [616493.165931] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616493.180568] [] _read_lock_bh+0x9/0x20 [616493.191361] [] :ip_tables:ipt_do_table+0x305/0x370 [616493.204408] [] tcp_v4_do_rcv+0x10b/0x520 [616493.215726] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616493.231575] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616493.245660] [] nf_iterate+0x5c/0xa0 [616493.256109] [] tcp_v4_rcv+0xa0e/0xa80 [616493.266903] [] ip_local_deliver+0x1b2/0x280 [616493.278736] [] ip_rcv+0x4d7/0x540 [616493.288837] [] netif_receive_skb+0x2fa/0x330 [616493.300847] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616493.314246] [] :e1000:e1000_clean+0x7f/0x250 [616493.326249] [] run_rebalance_domains+0xe6/0x3c0 [616493.338778] [] net_rx_action+0xbe/0x1f0 [616493.349919] [] __do_softirq+0x63/0xd0 [616493.360717] [] call_softirq+0x1c/0x28 [616493.371509] [] do_softirq+0x2c/0x90 [616493.381954] [] do_IRQ+0xc6/0xf0 [616493.391707] [] ret_from_intr+0x0/0xa [616493.402325] [] tcp_free_md5sig_pool+0x17/0x60 [616493.415730] [] tcp_free_md5sig_pool+0x10/0x60 [616493.427910] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616493.440957] [] __pollwait+0x0/0x130 [616493.451404] [] default_wake_function+0x0/0x10 [616493.463582] [] tcp_transmit_skb+0x55c/0x790 [616493.475422] [] __tcp_push_pending_frames+0x80d/0x950 [616493.488811] [] tcp_sendmsg+0x0/0xb00 [616493.499434] [] tcp_sendmsg+0x9a9/0xb00 [616493.510402] [] sock_aio_write+0x13f/0x180 [616493.521894] [] do_sync_write+0xcf/0x120 [616493.533034] [] autoremove_wake_function+0x0/0x30 [616493.545733] [] lock_task_sighand+0x3a/0x80 [616493.557390] [] getrusage+0x23e/0x260 [616493.568012] [] vfs_write+0xf8/0x1b0 [616493.578459] [] sys_write+0x53/0x90 [616493.588732] [] system_call+0x7e/0x83 [616493.599351] [616503.021448] BUG: soft lockup detected on CPU#0! [616503.030677] [616503.030677] Call Trace: [616503.038912] [] softlockup_tick+0xfb/0x120 [616503.051624] [] update_process_times+0x57/0x90 [616503.063803] [] smp_local_timer_interrupt+0x34/0x60 [616503.076847] [] smp_apic_timer_interrupt+0x59/0x80 [616503.089719] [] apic_timer_interrupt+0x66/0x70 [616503.101900] [] __lock_text_start+0x5/0x10 [616503.113385] [] __tcp_get_md5sig_pool+0xf/0x50 [616503.125561] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616503.138439] [] :nf_conntrack:tcp_error+0x159/0x200 [616503.151488] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616503.166123] [] _read_lock_bh+0x9/0x20 [616503.176917] [] :ip_tables:ipt_do_table+0x305/0x370 [616503.189963] [] tcp_v4_do_rcv+0x10b/0x520 [616503.201280] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616503.218817] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616503.232902] [] nf_iterate+0x5c/0xa0 [616503.243348] [] tcp_v4_rcv+0xa0e/0xa80 [616503.254142] [] ip_local_deliver+0x1b2/0x280 [616503.265974] [] ip_rcv+0x4d7/0x540 [616503.276074] [] netif_receive_skb+0x2fa/0x330 [616503.288084] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616503.301483] [] :e1000:e1000_clean+0x7f/0x250 [616503.313487] [] run_rebalance_domains+0xe6/0x3c0 [616503.326012] [] net_rx_action+0xbe/0x1f0 [616503.337152] [] __do_softirq+0x63/0xd0 [616503.347945] [] call_softirq+0x1c/0x28 [616503.358737] [] do_softirq+0x2c/0x90 [616503.369181] [] do_IRQ+0xc6/0xf0 [616503.378934] [] ret_from_intr+0x0/0xa [616503.389549] [] tcp_free_md5sig_pool+0x17/0x60 [616503.402958] [] tcp_free_md5sig_pool+0x10/0x60 [616503.415135] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616503.428180] [] __pollwait+0x0/0x130 [616503.438627] [] default_wake_function+0x0/0x10 [616503.450808] [] tcp_transmit_skb+0x55c/0x790 [616503.462642] [] __tcp_push_pending_frames+0x80d/0x950 [616503.476030] [] tcp_sendmsg+0x0/0xb00 [616503.486652] [] tcp_sendmsg+0x9a9/0xb00 [616503.497621] [] sock_aio_write+0x13f/0x180 [616503.509117] [] do_sync_write+0xcf/0x120 [616503.520259] [] autoremove_wake_function+0x0/0x30 [616503.532959] [] lock_task_sighand+0x3a/0x80 [616503.544616] [] getrusage+0x23e/0x260 [616503.555237] [] vfs_write+0xf8/0x1b0 [616503.565683] [] sys_write+0x53/0x90 [616503.575957] [] system_call+0x7e/0x83 [616503.586577] [616513.007004] BUG: soft lockup detected on CPU#0! [616513.016235] [616513.016236] Call Trace: [616513.024470] [] softlockup_tick+0xfb/0x120 [616513.037179] [] update_process_times+0x57/0x90 [616513.049357] [] smp_local_timer_interrupt+0x34/0x60 [616513.062400] [] smp_apic_timer_interrupt+0x59/0x80 [616513.075274] [] apic_timer_interrupt+0x66/0x70 [616513.087453] [] __lock_text_start+0x7/0x10 [616513.098941] [] __tcp_get_md5sig_pool+0xf/0x50 [616513.111120] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616513.123995] [] :nf_conntrack:tcp_error+0x159/0x200 [616513.137044] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616513.151679] [] _read_lock_bh+0x9/0x20 [616513.162473] [] :ip_tables:ipt_do_table+0x305/0x370 [616513.175519] [] tcp_v4_do_rcv+0x10b/0x520 [616513.186836] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616513.202688] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616513.216771] [] nf_iterate+0x5c/0xa0 [616513.227218] [] tcp_v4_rcv+0xa0e/0xa80 [616513.238012] [] ip_local_deliver+0x1b2/0x280 [616513.249845] [] ip_rcv+0x4d7/0x540 [616513.259945] [] netif_receive_skb+0x2fa/0x330 [616513.271954] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616513.285354] [] :e1000:e1000_clean+0x7f/0x250 [616513.297357] [] run_rebalance_domains+0xe6/0x3c0 [616513.309882] [] net_rx_action+0xbe/0x1f0 [616513.321023] [] __do_softirq+0x63/0xd0 [616513.331817] [] call_softirq+0x1c/0x28 [616513.342608] [] do_softirq+0x2c/0x90 [616513.353053] [] do_IRQ+0xc6/0xf0 [616513.362806] [] ret_from_intr+0x0/0xa [616513.373422] [] tcp_free_md5sig_pool+0x17/0x60 [616513.386829] [] tcp_free_md5sig_pool+0x10/0x60 [616513.399007] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616513.412052] [] __pollwait+0x0/0x130 [616513.422501] [] default_wake_function+0x0/0x10 [616513.434681] [] tcp_transmit_skb+0x55c/0x790 [616513.446516] [] __tcp_push_pending_frames+0x80d/0x950 [616513.459905] [] tcp_sendmsg+0x0/0xb00 [616513.470528] [] tcp_sendmsg+0x9a9/0xb00 [616513.481496] [] sock_aio_write+0x13f/0x180 [616513.492989] [] do_sync_write+0xcf/0x120 [616513.504129] [] autoremove_wake_function+0x0/0x30 [616513.516831] [] lock_task_sighand+0x3a/0x80 [616513.528490] [] getrusage+0x23e/0x260 [616513.539111] [] vfs_write+0xf8/0x1b0 [616513.549557] [] sys_write+0x53/0x90 [616513.559829] [] system_call+0x7e/0x83 [616513.570449] [616522.992560] BUG: soft lockup detected on CPU#0! [616523.001791] [616523.001791] Call Trace: [616523.010023] [] softlockup_tick+0xfb/0x120 [616523.022736] [] update_process_times+0x57/0x90 [616523.034915] [] smp_local_timer_interrupt+0x34/0x60 [616523.047958] [] smp_apic_timer_interrupt+0x59/0x80 [616523.060828] [] apic_timer_interrupt+0x66/0x70 [616523.073007] [] __lock_text_start+0x5/0x10 [616523.084492] [] __tcp_get_md5sig_pool+0xf/0x50 [616523.096672] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616523.109549] [] :nf_conntrack:tcp_error+0x159/0x200 [616523.122605] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616523.137242] [] _read_lock_bh+0x9/0x20 [616523.148035] [] :ip_tables:ipt_do_table+0x305/0x370 [616523.161080] [] tcp_v4_do_rcv+0x10b/0x520 [616523.172397] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616523.188247] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616523.202332] [] nf_iterate+0x5c/0xa0 [616523.212779] [] tcp_v4_rcv+0xa0e/0xa80 [616523.223573] [] ip_local_deliver+0x1b2/0x280 [616523.235406] [] ip_rcv+0x4d7/0x540 [616523.245507] [] netif_receive_skb+0x2fa/0x330 [616523.257518] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616523.270917] [] :e1000:e1000_clean+0x7f/0x250 [616523.282921] [] run_rebalance_domains+0xe6/0x3c0 [616523.295446] [] net_rx_action+0xbe/0x1f0 [616523.306586] [] __do_softirq+0x63/0xd0 [616523.317379] [] call_softirq+0x1c/0x28 [616523.328172] [] do_softirq+0x2c/0x90 [616523.338617] [] do_IRQ+0xc6/0xf0 [616523.348370] [] ret_from_intr+0x0/0xa [616523.358990] [] tcp_free_md5sig_pool+0x17/0x60 [616523.372398] [] tcp_free_md5sig_pool+0x10/0x60 [616523.384576] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616523.397621] [] __pollwait+0x0/0x130 [616523.408069] [] default_wake_function+0x0/0x10 [616523.420247] [] tcp_transmit_skb+0x55c/0x790 [616523.432082] [] __tcp_push_pending_frames+0x80d/0x950 [616523.445472] [] tcp_sendmsg+0x0/0xb00 [616523.456094] [] tcp_sendmsg+0x9a9/0xb00 [616523.467063] [] sock_aio_write+0x13f/0x180 [616523.478554] [] do_sync_write+0xcf/0x120 [616523.489694] [] autoremove_wake_function+0x0/0x30 [616523.502393] [] lock_task_sighand+0x3a/0x80 [616523.514053] [] getrusage+0x23e/0x260 [616523.524674] [] vfs_write+0xf8/0x1b0 [616523.535120] [] sys_write+0x53/0x90 [616523.545393] [] system_call+0x7e/0x83 [616523.556013] [616532.978115] BUG: soft lockup detected on CPU#0! [616532.987344] [616532.987345] Call Trace: [616532.995576] [] softlockup_tick+0xfb/0x120 [616533.008288] [] update_process_times+0x57/0x90 [616533.020465] [] smp_local_timer_interrupt+0x34/0x60 [616533.033508] [] smp_apic_timer_interrupt+0x59/0x80 [616533.046378] [] apic_timer_interrupt+0x66/0x70 [616533.058560] [] __lock_text_start+0x7/0x10 [616533.070045] [] __tcp_get_md5sig_pool+0xf/0x50 [616533.082223] [] tcp_v4_do_calc_md5_hash+0x59/0x2e0 [616533.095099] [] :nf_conntrack:tcp_error+0x159/0x200 [616533.108149] [] :nf_conntrack:__nf_conntrack_find+0x19/0x120 [616533.122786] [] _read_lock_bh+0x9/0x20 [616533.133576] [] :ip_tables:ipt_do_table+0x305/0x370 [616533.146622] [] tcp_v4_do_rcv+0x10b/0x520 [616533.157939] [] :nf_conntrack:nf_ct_deliver_cached_events+0x61/0xb0 [616533.173788] [] :nf_conntrack_ipv4:ipv4_confirm+0x48/0x60 [616533.187873] [] nf_iterate+0x5c/0xa0 [616533.198320] [] tcp_v4_rcv+0xa0e/0xa80 [616533.209117] [] ip_local_deliver+0x1b2/0x280 [616533.220949] [] ip_rcv+0x4d7/0x540 [616533.231053] [] netif_receive_skb+0x2fa/0x330 [616533.243066] [] :e1000:e1000_clean_rx_irq+0x451/0x520 [616533.256466] [] :e1000:e1000_clean+0x7f/0x250 [616533.268471] [] run_rebalance_domains+0xe6/0x3c0 [616533.280996] [] net_rx_action+0xbe/0x1f0 [616533.292137] [] __do_softirq+0x63/0xd0 [616533.302930] [] call_softirq+0x1c/0x28 [616533.313723] [] do_softirq+0x2c/0x90 [616533.324168] [] do_IRQ+0xc6/0xf0 [616533.333919] [] ret_from_intr+0x0/0xa [616533.344535] [] tcp_free_md5sig_pool+0x17/0x60 [616533.357942] [] tcp_free_md5sig_pool+0x10/0x60 [616533.370119] [] tcp_v4_do_calc_md5_hash+0x271/0x2e0 [616533.383166] [] __pollwait+0x0/0x130 [616533.393614] [] default_wake_function+0x0/0x10 [616533.405792] [] tcp_transmit_skb+0x55c/0x790 [616533.417628] [] __tcp_push_pending_frames+0x80d/0x950 [616533.431017] [] tcp_sendmsg+0x0/0xb00 [616533.443328] [] tcp_sendmsg+0x9a9/0xb00 [616533.454296] [] sock_aio_write+0x13f/0x180 [616533.465787] [] do_sync_write+0xcf/0x120 [616533.476928] [] autoremove_wake_function+0x0/0x30 [616533.489625] [] lock_task_sighand+0x3a/0x80 [616533.501285] [] getrusage+0x23e/0x260 [616533.511906] [] vfs_write+0xf8/0x1b0 [616533.522353] [] sys_write+0x53/0x90 [616533.532626] [] system_call+0x7e/0x83 [616533.543253]