From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ingo Molnar Subject: Re: unsafe locks seen with netperf on net-2.6.29 tree Date: Mon, 29 Dec 2008 12:28:58 +0100 Message-ID: <20081229112858.GA16385@elte.hu> References: <9929d2390812250225w19bcd2f7n11357ff26de78c52@mail.gmail.com> <20081225112658.GA7260@gondor.apana.org.au> <1230300535.9487.292.camel@twins> <1230410308.9487.295.camel@twins> <1230544927.16718.12.camel@twins> <20081229103154.GA9691@gondor.apana.org.au> <20081229103735.GA9763@gondor.apana.org.au> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Peter Zijlstra , "Tantilov, Emil S" , "Kirsher, Jeffrey T" , netdev , David Miller , "Waskiewicz Jr, Peter P" , "Duyck, Alexander H" , Eric Dumazet To: Herbert Xu Return-path: Received: from mx2.mail.elte.hu ([157.181.151.9]:56516 "EHLO mx2.mail.elte.hu" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751308AbYL2L3f (ORCPT ); Mon, 29 Dec 2008 06:29:35 -0500 Content-Disposition: inline In-Reply-To: <20081229103735.GA9763@gondor.apana.org.au> Sender: netdev-owner@vger.kernel.org List-ID: * Herbert Xu wrote: > On Mon, Dec 29, 2008 at 09:31:54PM +1100, Herbert Xu wrote: > > > > You also need to patch the other places where we use that percpu > > counter from process context, e.g., for /proc/net/tcp. > > In fact, it looks like just about every spot in the original > changeset (dd24c00191d5e4a1ae896aafe33c6b8095ab4bd1) may be run > from process context. So you might be better off making BH-disabling > variants of the percpu counter ops. i got the splat further below with Peter's workaround applied. Ingo [ 65.163041] [ 65.163041] ====================================================== [ 65.164028] [ INFO: soft-safe -> soft-unsafe lock order detected ] [ 65.164028] 2.6.28-tip-03865-gb8c8c2c-dirty #13131 [ 65.164028] ------------------------------------------------------ [ 65.164028] ssh/2856 [HC0[0]:SC0[1]:HE1:SE0] is trying to acquire: [ 65.164028] (&fbc->lock){--..}, at: [] __percpu_counter_add+0x52/0x7a [ 65.164028] [ 65.164028] and this task is already holding: [ 65.164028] (slock-AF_INET){-+..}, at: [] tcp_close+0x184/0x335 [ 65.164028] which would create a new lock dependency: [ 65.164028] (slock-AF_INET){-+..} -> (&fbc->lock){--..} [ 65.164028] [ 65.164028] but this new dependency connects a soft-irq-safe lock: [ 65.164028] (slock-AF_INET){-+..} [ 65.164028] ... which became soft-irq-safe at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] tcp_delack_timer+0x17/0x1b1 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] to a soft-irq-unsafe lock: [ 65.164028] (&fbc->lock){--..} [ 65.164028] ... which became soft-irq-unsafe at: [ 65.164028] ... [] __lock_acquire+0x58f/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __percpu_counter_add+0x52/0x7a [ 65.164028] [] get_empty_filp+0x9f/0x1b0 [ 65.164028] [] path_lookup_open+0x23/0x7a [ 65.164028] [] do_filp_open+0xa7/0x67e [ 65.164028] [] do_sys_open+0x47/0xbc [ 65.164028] [] sys_open+0x23/0x2b [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] other info that might help us debug this: [ 65.164028] [ 65.164028] 1 lock held by ssh/2856: [ 65.164028] #0: (slock-AF_INET){-+..}, at: [] tcp_close+0x184/0x335 [ 65.164028] [ 65.164028] the soft-irq-safe lock's dependencies: [ 65.164028] -> (slock-AF_INET){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] lock_sock_nested+0x2c/0xc3 [ 65.164028] [] tcp_close+0x1a/0x335 [ 65.164028] [] inet_release+0x47/0x4d [ 65.164028] [] sock_release+0x15/0x58 [ 65.164028] [] sock_close+0x21/0x25 [ 65.164028] [] __fput+0xcf/0x1a2 [ 65.164028] [] fput+0x1c/0x1e [ 65.164028] [] filp_close+0x55/0x5f [ 65.164028] [] sys_close+0x6d/0xa6 [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] tcp_delack_timer+0x17/0x1b1 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] lock_sock_nested+0x2c/0xc3 [ 65.164028] [] tcp_close+0x1a/0x335 [ 65.164028] [] inet_release+0x47/0x4d [ 65.164028] [] sock_release+0x15/0x58 [ 65.164028] [] sock_close+0x21/0x25 [ 65.164028] [] __fput+0xcf/0x1a2 [ 65.164028] [] fput+0x1c/0x1e [ 65.164028] [] filp_close+0x55/0x5f [ 65.164028] [] sys_close+0x6d/0xa6 [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] af_family_slock_keys+0x10/0x120 [ 65.164028] -> (pool_lock){.+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __debug_object_init+0xb5/0x29e [ 65.164028] [] debug_object_init+0x13/0x16 [ 65.164028] [] hrtimer_init+0x1b/0x2b [ 65.164028] [] sched_init+0x99/0x333 [ 65.164028] [] start_kernel+0x13e/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __debug_object_init+0x47/0x29e [ 65.164028] [] debug_object_init+0x13/0x16 [ 65.164028] [] timer_fixup_activate+0x2d/0x7c [ 65.164028] [] debug_object_fixup+0x17/0x22 [ 65.164028] [] debug_object_activate+0xb6/0xbd [ 65.164028] [] __mod_timer+0x6d/0xc8 [ 65.164028] [] mod_timer+0x38/0x3c [ 65.164028] [] inet_twsk_schedule+0x13f/0x158 [ 65.164028] [] tcp_time_wait+0x169/0x1a7 [ 65.164028] [] tcp_fin+0x80/0x16a [ 65.164028] [] tcp_data_queue+0x293/0xa6d [ 65.164028] [] tcp_rcv_state_process+0x852/0x8b7 [ 65.164028] [] tcp_v4_do_rcv+0x25c/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] pool_lock+0x10/0x24 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __debug_object_init+0x47/0x29e [ 65.164028] [] debug_object_init+0x13/0x16 [ 65.164028] [] init_timer+0x15/0x1f [ 65.164028] [] inet_csk_init_xmit_timers+0x25/0x6a [ 65.164028] [] tcp_init_xmit_timers+0x1c/0x1f [ 65.164028] [] tcp_create_openreq_child+0x190/0x3c4 [ 65.164028] [] tcp_v4_syn_recv_sock+0x53/0x1c0 [ 65.164028] [] tcp_check_req+0x1e0/0x36d [ 65.164028] [] tcp_v4_do_rcv+0x1b2/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&obj_hash[i].lock){++..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __debug_object_init+0x95/0x29e [ 65.164028] [] debug_object_init+0x13/0x16 [ 65.164028] [] hrtimer_init+0x1b/0x2b [ 65.164028] [] sched_init+0x99/0x333 [ 65.164028] [] start_kernel+0x13e/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] debug_object_deactivate+0x29/0x8f [ 65.164028] [] __run_hrtimer+0x47/0xae [ 65.164028] [] hrtimer_run_queues+0xf6/0x124 [ 65.164028] [] run_local_timers+0xd/0x1e [ 65.164028] [] update_process_times+0x29/0x53 [ 65.164028] [] tick_periodic+0x6b/0x6d [ 65.164028] [] tick_handle_periodic+0x1e/0x60 [ 65.164028] [] __irqentry_text_start+0x74/0x87 [ 65.164028] [] apic_timer_interrupt+0x2d/0x34 [ 65.164028] [] printk+0x1a/0x1c [ 65.164028] [] do_one_initcall+0x4a/0x17f [ 65.164028] [] kernel_init+0xfc/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] debug_object_deactivate+0x29/0x8f [ 65.164028] [] run_timer_softirq+0x116/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.19955+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __debug_object_init+0xb5/0x29e [ 65.164028] [] debug_object_init+0x13/0x16 [ 65.164028] [] hrtimer_init+0x1b/0x2b [ 65.164028] [] sched_init+0x99/0x333 [ 65.164028] [] start_kernel+0x13e/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __debug_object_init+0x95/0x29e [ 65.164028] [] debug_object_init+0x13/0x16 [ 65.164028] [] init_timer+0x15/0x1f [ 65.164028] [] inet_csk_init_xmit_timers+0x25/0x6a [ 65.164028] [] tcp_init_xmit_timers+0x1c/0x1f [ 65.164028] [] tcp_create_openreq_child+0x190/0x3c4 [ 65.164028] [] tcp_v4_syn_recv_sock+0x53/0x1c0 [ 65.164028] [] tcp_check_req+0x1e0/0x36d [ 65.164028] [] tcp_v4_do_rcv+0x1b2/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&hashinfo->ehash_locks[i]){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __inet_hash_nolisten+0xea/0x11f [ 65.164028] [] __inet_hash_connect+0x17a/0x213 [ 65.164028] [] inet_hash_connect+0x3b/0x42 [ 65.164028] [] tcp_v4_connect+0x33f/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __inet_twsk_hashdance+0xa0/0x103 [ 65.164028] [] tcp_time_wait+0x131/0x1a7 [ 65.164028] [] tcp_fin+0x80/0x16a [ 65.164028] [] tcp_data_queue+0x293/0xa6d [ 65.164028] [] tcp_rcv_state_process+0x852/0x8b7 [ 65.164028] [] tcp_v4_do_rcv+0x25c/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __inet_hash_nolisten+0xea/0x11f [ 65.164028] [] __inet_hash_connect+0x17a/0x213 [ 65.164028] [] inet_hash_connect+0x3b/0x42 [ 65.164028] [] tcp_v4_connect+0x33f/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.38748+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __inet_hash_nolisten+0xea/0x11f [ 65.164028] [] tcp_v4_syn_recv_sock+0x179/0x1c0 [ 65.164028] [] tcp_check_req+0x1e0/0x36d [ 65.164028] [] tcp_v4_do_rcv+0x1b2/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&tcp_hashinfo.bhash[i].lock){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] inet_csk_get_port+0xc4/0x1d2 [ 65.164028] [] inet_bind+0x102/0x1a4 [ 65.164028] [] sys_bind+0x53/0x72 [ 65.164028] [] sys_socketcall+0x7f/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __inet_twsk_hashdance+0x4f/0x103 [ 65.164028] [] tcp_time_wait+0x131/0x1a7 [ 65.164028] [] tcp_fin+0x80/0x16a [ 65.164028] [] tcp_data_queue+0x293/0xa6d [ 65.164028] [] tcp_rcv_state_process+0x852/0x8b7 [ 65.164028] [] tcp_v4_do_rcv+0x25c/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] inet_csk_get_port+0xc4/0x1d2 [ 65.164028] [] inet_bind+0x102/0x1a4 [ 65.164028] [] sys_bind+0x53/0x72 [ 65.164028] [] sys_socketcall+0x7f/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.45901+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __inet_hash_nolisten+0xea/0x11f [ 65.164028] [] __inet_hash_connect+0x17a/0x213 [ 65.164028] [] inet_hash_connect+0x3b/0x42 [ 65.164028] [] tcp_v4_connect+0x33f/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&n->list_lock){.+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] deactivate_slab+0x71/0xd6 [ 65.164028] [] __slab_alloc+0x91/0x441 [ 65.164028] [] __kmalloc+0x99/0x10c [ 65.164028] [] kmem_cache_create+0x1bd/0x1d4 [ 65.164028] [] vfs_caches_init+0x63/0x107 [ 65.164028] [] start_kernel+0x2cc/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __slab_free+0x1b5/0x243 [ 65.164028] [] kfree+0xc1/0xf7 [ 65.164028] [] rcu_free_old_probes+0xd/0xf [ 65.164028] [] __rcu_process_callbacks+0x162/0x1f7 [ 65.164028] [] rcu_process_callbacks+0x26/0x46 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.25975+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __slab_alloc+0xa8/0x441 [ 65.164028] [] kmem_cache_alloc+0x63/0xd1 [ 65.164028] [] inet_bind_bucket_create+0x19/0x51 [ 65.164028] [] __inet_hash_connect+0x101/0x213 [ 65.164028] [] inet_hash_connect+0x3b/0x42 [ 65.164028] [] tcp_v4_connect+0x33f/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __inet_inherit_port+0x32/0x66 [ 65.164028] [] tcp_v4_syn_recv_sock+0x183/0x1c0 [ 65.164028] [] tcp_check_req+0x1e0/0x36d [ 65.164028] [] tcp_v4_do_rcv+0x1b2/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&queue->syn_wait_lock){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _write_lock_bh+0x28/0x58 [ 65.164028] [] reqsk_queue_alloc+0xba/0xcf [ 65.164028] [] inet_csk_listen_start+0x1e/0xc2 [ 65.164028] [] inet_listen+0x43/0x67 [ 65.164028] [] sys_listen+0x3f/0x59 [ 65.164028] [] sys_socketcall+0x9a/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _write_lock+0x23/0x53 [ 65.164028] [] inet_csk_reqsk_queue_hash_add+0xca/0x101 [ 65.164028] [] tcp_v4_conn_request+0x3fe/0x435 [ 65.164028] [] tcp_rcv_state_process+0x62/0x8b7 [ 65.164028] [] tcp_v4_do_rcv+0x25c/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _write_lock_bh+0x28/0x58 [ 65.164028] [] reqsk_queue_alloc+0xba/0xcf [ 65.164028] [] inet_csk_listen_start+0x1e/0xc2 [ 65.164028] [] inet_listen+0x43/0x67 [ 65.164028] [] sys_listen+0x3f/0x59 [ 65.164028] [] sys_socketcall+0x9a/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.34733+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _write_lock+0x23/0x53 [ 65.164028] [] tcp_check_req+0x24f/0x36d [ 65.164028] [] tcp_v4_do_rcv+0x1b2/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&base->lock){++..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] lock_timer_base+0x24/0x43 [ 65.164028] [] __mod_timer+0x2f/0xc8 [ 65.164028] [] mod_timer+0x38/0x3c [ 65.164028] [] con_init+0xa1/0x215 [ 65.164028] [] console_init+0x12/0x20 [ 65.164028] [] start_kernel+0x22a/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] lock_timer_base+0x24/0x43 [ 65.164028] [] __mod_timer+0x2f/0xc8 [ 65.164028] [] mod_timer+0x38/0x3c [ 65.164028] [] __netdev_watchdog_up+0x47/0x55 [ 65.164028] [] netif_carrier_on+0x34/0x37 [ 65.164028] [] nv_linkchange+0x21/0x5b [ 65.164028] [] nv_link_irq+0x30/0x33 [ 65.164028] [] nv_nic_irq_optimized+0xfe/0x22d [ 65.164028] [] handle_IRQ_event+0x1f/0x54 [ 65.164028] [] handle_fasteoi_irq+0xaa/0xb7 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irq+0x32/0x62 [ 65.164028] [] run_timer_softirq+0x37/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.23710+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] debug_object_activate+0x2c/0xbd [ 65.164028] [] __mod_timer+0x6d/0xc8 [ 65.164028] [] mod_timer+0x38/0x3c [ 65.164028] [] con_init+0xa1/0x215 [ 65.164028] [] console_init+0x12/0x20 [ 65.164028] [] start_kernel+0x22a/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __debug_object_init+0x47/0x29e [ 65.164028] [] debug_object_init+0x13/0x16 [ 65.164028] [] timer_fixup_activate+0x2d/0x7c [ 65.164028] [] debug_object_fixup+0x17/0x22 [ 65.164028] [] debug_object_activate+0xb6/0xbd [ 65.164028] [] __mod_timer+0x6d/0xc8 [ 65.164028] [] __reschedule_timeout+0x6f/0xaf [ 65.164028] [] reschedule_timeout+0x2d/0x3f [ 65.164028] [] floppy_init+0x172/0xdb3 [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0xfc/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&base->lock/1){....} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_nested+0x21/0x51 [ 65.164028] [] timer_cpu_notify+0x1a2/0x22b [ 65.164028] [] notifier_call_chain+0x49/0x71 [ 65.164028] [] __raw_notifier_call_chain+0x13/0x15 [ 65.164028] [] raw_notifier_call_chain+0x11/0x13 [ 65.164028] [] _cpu_down+0x168/0x221 [ 65.164028] [] cpu_down+0x35/0x57 [ 65.164028] [] store_online+0x2a/0x5e [ 65.164028] [] sysdev_store+0x20/0x28 [ 65.164028] [] sysfs_write_file+0xbd/0xe8 [ 65.164028] [] vfs_write+0x91/0x138 [ 65.164028] [] sys_write+0x40/0x65 [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.23710+0x1/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] debug_object_deactivate+0x29/0x8f [ 65.164028] [] migrate_timer_list+0x1d/0x4d [ 65.164028] [] timer_cpu_notify+0x1b8/0x22b [ 65.164028] [] notifier_call_chain+0x49/0x71 [ 65.164028] [] __raw_notifier_call_chain+0x13/0x15 [ 65.164028] [] raw_notifier_call_chain+0x11/0x13 [ 65.164028] [] _cpu_down+0x168/0x221 [ 65.164028] [] cpu_down+0x35/0x57 [ 65.164028] [] store_online+0x2a/0x5e [ 65.164028] [] sysdev_store+0x20/0x28 [ 65.164028] [] sysfs_write_file+0xbd/0xe8 [ 65.164028] [] vfs_write+0x91/0x138 [ 65.164028] [] sys_write+0x40/0x65 [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_nested+0x21/0x51 [ 65.164028] [] timer_cpu_notify+0x1a2/0x22b [ 65.164028] [] notifier_call_chain+0x49/0x71 [ 65.164028] [] __raw_notifier_call_chain+0x13/0x15 [ 65.164028] [] raw_notifier_call_chain+0x11/0x13 [ 65.164028] [] _cpu_down+0x168/0x221 [ 65.164028] [] cpu_down+0x35/0x57 [ 65.164028] [] store_online+0x2a/0x5e [ 65.164028] [] sysdev_store+0x20/0x28 [ 65.164028] [] sysfs_write_file+0xbd/0xe8 [ 65.164028] [] vfs_write+0x91/0x138 [ 65.164028] [] sys_write+0x40/0x65 [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] lock_timer_base+0x24/0x43 [ 65.164028] [] del_timer+0x26/0x66 [ 65.164028] [] sk_stop_timer+0x17/0x22 [ 65.164028] [] inet_csk_delete_keepalive_timer+0x13/0x15 [ 65.164028] [] tcp_check_req+0x282/0x36d [ 65.164028] [] tcp_v4_do_rcv+0x1b2/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&q->lock){++..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irq+0x32/0x62 [ 65.164028] [] wait_for_common+0x2f/0x110 [ 65.164028] [] wait_for_completion+0x17/0x19 [ 65.164028] [] kthread_create+0x75/0xa0 [ 65.164028] [] migration_call+0x39/0x4d6 [ 65.164028] [] migration_init+0x1d/0x4b [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0x53/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __wake_up+0x1a/0x40 [ 65.164028] [] insert_work+0x48/0x50 [ 65.164028] [] __queue_work+0x22/0x30 [ 65.164028] [] queue_work_on+0x3a/0x46 [ 65.164028] [] queue_work+0x1a/0x1d [ 65.164028] [] schedule_work+0x14/0x16 [ 65.164028] [] schedule_bh+0x17/0x19 [ 65.164028] [] floppy_interrupt+0x15a/0x171 [ 65.164028] [] floppy_hardint+0x22/0xe1 [ 65.164028] [] handle_IRQ_event+0x1f/0x54 [ 65.164028] [] handle_edge_irq+0xa6/0x10d [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] complete+0x17/0x43 [ 65.164028] [] wakeme_after_rcu+0x10/0x12 [ 65.164028] [] __rcu_process_callbacks+0x162/0x1f7 [ 65.164028] [] rcu_process_callbacks+0x26/0x46 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.18770+0x0/0x8 [ 65.164028] -> (&rq->lock){++..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] rq_attach_root+0x19/0x191 [ 65.164028] [] sched_init+0x214/0x333 [ 65.164028] [] start_kernel+0x13e/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] scheduler_tick+0x3a/0x211 [ 65.164028] [] update_process_times+0x47/0x53 [ 65.164028] [] tick_periodic+0x6b/0x6d [ 65.164028] [] tick_handle_periodic+0x1e/0x60 [ 65.164028] [] timer_interrupt+0x46/0x4d [ 65.164028] [] handle_IRQ_event+0x1f/0x54 [ 65.164028] [] handle_level_irq+0xbe/0xcb [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] scheduler_tick+0x3a/0x211 [ 65.164028] [] update_process_times+0x47/0x53 [ 65.164028] [] tick_periodic+0x6b/0x6d [ 65.164028] [] tick_handle_periodic+0x1e/0x60 [ 65.164028] [] __irqentry_text_start+0x74/0x87 [ 65.164028] [] apic_timer_interrupt+0x2d/0x34 [ 65.164028] [] run_timer_softirq+0x19c/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.45602+0x0/0x8 [ 65.164028] -> (&vec->lock){.+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] cpupri_set+0xc1/0x12f [ 65.164028] [] rq_online_rt+0x92/0x9a [ 65.164028] [] set_rq_online+0x72/0x80 [ 65.164028] [] rq_attach_root+0x17e/0x191 [ 65.164028] [] sched_init+0x214/0x333 [ 65.164028] [] start_kernel+0x13e/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] cpupri_set+0x51/0x12f [ 65.164028] [] __enqueue_rt_entity+0xa3/0x14a [ 65.164028] [] enqueue_task_rt+0x35/0x47 [ 65.164028] [] enqueue_task+0x51/0x5d [ 65.164028] [] activate_task+0x1e/0x24 [ 65.164028] [] try_to_wake_up+0x23e/0x2bf [ 65.164028] [] wake_up_process+0x14/0x16 [ 65.164028] [] rebalance_domains+0x3ea/0x4d5 [ 65.164028] [] run_rebalance_domains+0x33/0xf2 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.15525+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] cpupri_set+0xc1/0x12f [ 65.164028] [] rq_online_rt+0x92/0x9a [ 65.164028] [] set_rq_online+0x72/0x80 [ 65.164028] [] rq_attach_root+0x17e/0x191 [ 65.164028] [] sched_init+0x214/0x333 [ 65.164028] [] start_kernel+0x13e/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&rt_b->rt_runtime_lock){.+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __enqueue_rt_entity+0xe7/0x14a [ 65.164028] [] enqueue_task_rt+0x35/0x47 [ 65.164028] [] enqueue_task+0x51/0x5d [ 65.164028] [] activate_task+0x1e/0x24 [ 65.164028] [] try_to_wake_up+0x23e/0x2bf [ 65.164028] [] wake_up_process+0x14/0x16 [ 65.164028] [] migration_call+0xb9/0x4d6 [ 65.164028] [] migration_init+0x3b/0x4b [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0x53/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __enqueue_rt_entity+0xe7/0x14a [ 65.164028] [] enqueue_task_rt+0x35/0x47 [ 65.164028] [] enqueue_task+0x51/0x5d [ 65.164028] [] activate_task+0x1e/0x24 [ 65.164028] [] try_to_wake_up+0x23e/0x2bf [ 65.164028] [] wake_up_process+0x14/0x16 [ 65.164028] [] rebalance_domains+0x3ea/0x4d5 [ 65.164028] [] run_rebalance_domains+0x33/0xf2 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.37294+0x0/0x8 [ 65.164028] -> (&cpu_base->lock){++..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] lock_hrtimer_base+0x1d/0x38 [ 65.164028] [] hrtimer_start_range_ns+0x1e/0x142 [ 65.164028] [] __enqueue_rt_entity+0x133/0x14a [ 65.164028] [] enqueue_task_rt+0x35/0x47 [ 65.164028] [] enqueue_task+0x51/0x5d [ 65.164028] [] activate_task+0x1e/0x24 [ 65.164028] [] try_to_wake_up+0x23e/0x2bf [ 65.164028] [] wake_up_process+0x14/0x16 [ 65.164028] [] migration_call+0xb9/0x4d6 [ 65.164028] [] migration_init+0x3b/0x4b [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0x53/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] hrtimer_run_queues+0xd5/0x124 [ 65.164028] [] run_local_timers+0xd/0x1e [ 65.164028] [] update_process_times+0x29/0x53 [ 65.164028] [] tick_periodic+0x6b/0x6d [ 65.164028] [] tick_handle_periodic+0x1e/0x60 [ 65.164028] [] __irqentry_text_start+0x74/0x87 [ 65.164028] [] apic_timer_interrupt+0x2d/0x34 [ 65.164028] [] printk+0x1a/0x1c [ 65.164028] [] do_one_initcall+0xc4/0x17f [ 65.164028] [] kernel_init+0x53/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] hrtimer_run_queues+0xd5/0x124 [ 65.164028] [] run_local_timers+0xd/0x1e [ 65.164028] [] update_process_times+0x29/0x53 [ 65.164028] [] tick_periodic+0x6b/0x6d [ 65.164028] [] tick_handle_periodic+0x1e/0x60 [ 65.164028] [] __irqentry_text_start+0x74/0x87 [ 65.164028] [] apic_timer_interrupt+0x2d/0x34 [ 65.164028] [] run_timer_softirq+0x19c/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.21090+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] debug_object_activate+0x2c/0xbd [ 65.164028] [] enqueue_hrtimer+0x2b/0x15d [ 65.164028] [] hrtimer_start_range_ns+0x12a/0x142 [ 65.164028] [] __enqueue_rt_entity+0x133/0x14a [ 65.164028] [] enqueue_task_rt+0x35/0x47 [ 65.164028] [] enqueue_task+0x51/0x5d [ 65.164028] [] activate_task+0x1e/0x24 [ 65.164028] [] try_to_wake_up+0x23e/0x2bf [ 65.164028] [] wake_up_process+0x14/0x16 [ 65.164028] [] migration_call+0xb9/0x4d6 [ 65.164028] [] migration_init+0x3b/0x4b [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0x53/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&cpu_base->lock/1){....} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_nested+0x21/0x51 [ 65.164028] [] hrtimer_cpu_notify+0xd2/0x179 [ 65.164028] [] notifier_call_chain+0x49/0x71 [ 65.164028] [] __raw_notifier_call_chain+0x13/0x15 [ 65.164028] [] raw_notifier_call_chain+0x11/0x13 [ 65.164028] [] _cpu_down+0x168/0x221 [ 65.164028] [] cpu_down+0x35/0x57 [ 65.164028] [] store_online+0x2a/0x5e [ 65.164028] [] sysdev_store+0x20/0x28 [ 65.164028] [] sysfs_write_file+0xbd/0xe8 [ 65.164028] [] vfs_write+0x91/0x138 [ 65.164028] [] sys_write+0x40/0x65 [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.21090+0x1/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] debug_object_deactivate+0x29/0x8f [ 65.164028] [] hrtimer_cpu_notify+0x10e/0x179 [ 65.164028] [] notifier_call_chain+0x49/0x71 [ 65.164028] [] __raw_notifier_call_chain+0x13/0x15 [ 65.164028] [] raw_notifier_call_chain+0x11/0x13 [ 65.164028] [] _cpu_down+0x168/0x221 [ 65.164028] [] cpu_down+0x35/0x57 [ 65.164028] [] store_online+0x2a/0x5e [ 65.164028] [] sysdev_store+0x20/0x28 [ 65.164028] [] sysfs_write_file+0xbd/0xe8 [ 65.164028] [] vfs_write+0x91/0x138 [ 65.164028] [] sys_write+0x40/0x65 [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_nested+0x21/0x51 [ 65.164028] [] hrtimer_cpu_notify+0xd2/0x179 [ 65.164028] [] notifier_call_chain+0x49/0x71 [ 65.164028] [] __raw_notifier_call_chain+0x13/0x15 [ 65.164028] [] raw_notifier_call_chain+0x11/0x13 [ 65.164028] [] _cpu_down+0x168/0x221 [ 65.164028] [] cpu_down+0x35/0x57 [ 65.164028] [] store_online+0x2a/0x5e [ 65.164028] [] sysdev_store+0x20/0x28 [ 65.164028] [] sysfs_write_file+0xbd/0xe8 [ 65.164028] [] vfs_write+0x91/0x138 [ 65.164028] [] sys_write+0x40/0x65 [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] lock_hrtimer_base+0x1d/0x38 [ 65.164028] [] hrtimer_start_range_ns+0x1e/0x142 [ 65.164028] [] __enqueue_rt_entity+0x133/0x14a [ 65.164028] [] enqueue_task_rt+0x35/0x47 [ 65.164028] [] enqueue_task+0x51/0x5d [ 65.164028] [] activate_task+0x1e/0x24 [ 65.164028] [] try_to_wake_up+0x23e/0x2bf [ 65.164028] [] wake_up_process+0x14/0x16 [ 65.164028] [] migration_call+0xb9/0x4d6 [ 65.164028] [] migration_init+0x3b/0x4b [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0x53/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&rt_rq->rt_runtime_lock){++..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] update_curr_rt+0xdf/0x198 [ 65.164028] [] dequeue_task_rt+0x13/0x2f [ 65.164028] [] dequeue_task+0xfb/0x10a [ 65.164028] [] deactivate_task+0x1e/0x24 [ 65.164028] [] schedule+0x168/0x98a [ 65.164028] [] migration_thread+0x1b3/0x24a [ 65.164028] [] kthread+0x40/0x69 [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] sched_rt_period_timer+0xc0/0x239 [ 65.164028] [] __run_hrtimer+0x7c/0xae [ 65.164028] [] hrtimer_run_queues+0xf6/0x124 [ 65.164028] [] run_local_timers+0xd/0x1e [ 65.164028] [] update_process_times+0x29/0x53 [ 65.164028] [] tick_periodic+0x6b/0x6d [ 65.164028] [] tick_handle_periodic+0x1e/0x60 [ 65.164028] [] __irqentry_text_start+0x74/0x87 [ 65.164028] [] apic_timer_interrupt+0x2d/0x34 [ 65.164028] [] default_idle+0x5d/0x97 [ 65.164028] [] cpu_idle+0x84/0xa5 [ 65.164028] [] rest_init+0x53/0x55 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] sched_rt_period_timer+0xc0/0x239 [ 65.164028] [] __run_hrtimer+0x7c/0xae [ 65.164028] [] hrtimer_run_queues+0xf6/0x124 [ 65.164028] [] run_local_timers+0xd/0x1e [ 65.164028] [] update_process_times+0x29/0x53 [ 65.164028] [] tick_periodic+0x6b/0x6d [ 65.164028] [] tick_handle_periodic+0x1e/0x60 [ 65.164028] [] __irqentry_text_start+0x74/0x87 [ 65.164028] [] apic_timer_interrupt+0x2d/0x34 [ 65.164028] [] mempool_free_slab+0x13/0x15 [ 65.164028] [] mempool_free+0x6c/0x74 [ 65.164028] [] bio_free+0x2d/0x49 [ 65.164028] [] bio_fs_destructor+0x13/0x15 [ 65.164028] [] bio_put+0x2b/0x2d [ 65.164028] [] end_bio_bh_io_sync+0x2f/0x32 [ 65.164028] [] bio_endio+0x2d/0x30 [ 65.164028] [] req_bio_endio+0x81/0x9e [ 65.164028] [] __end_that_request_first+0x16c/0x265 [ 65.164028] [] end_that_request_data+0x19/0x47 [ 65.164028] [] blk_end_io+0x21/0x6f [ 65.164028] [] blk_end_request+0x11/0x13 [ 65.164028] [] scsi_end_request+0x24/0x80 [ 65.164028] [] scsi_io_completion+0x1a4/0x376 [ 65.164028] [] scsi_finish_command+0xd1/0xd9 [ 65.164028] [] scsi_softirq_done+0xf3/0xfb [ 65.164028] [] blk_done_softirq+0x72/0x81 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.45584+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __enable_runtime+0x33/0x7d [ 65.164028] [] rq_online_rt+0x78/0x9a [ 65.164028] [] set_rq_online+0x72/0x80 [ 65.164028] [] migration_call+0x127/0x4d6 [ 65.164028] [] notifier_call_chain+0x49/0x71 [ 65.164028] [] __raw_notifier_call_chain+0x13/0x15 [ 65.164028] [] raw_notifier_call_chain+0x11/0x13 [ 65.164028] [] _cpu_up+0xbd/0xef [ 65.164028] [] cpu_up+0x49/0x59 [ 65.164028] [] kernel_init+0x9b/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __enqueue_rt_entity+0xe7/0x14a [ 65.164028] [] enqueue_task_rt+0x35/0x47 [ 65.164028] [] enqueue_task+0x51/0x5d [ 65.164028] [] activate_task+0x1e/0x24 [ 65.164028] [] try_to_wake_up+0x23e/0x2bf [ 65.164028] [] wake_up_process+0x14/0x16 [ 65.164028] [] migration_call+0xb9/0x4d6 [ 65.164028] [] migration_init+0x3b/0x4b [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0x53/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] update_curr_rt+0xdf/0x198 [ 65.164028] [] dequeue_task_rt+0x13/0x2f [ 65.164028] [] dequeue_task+0xfb/0x10a [ 65.164028] [] deactivate_task+0x1e/0x24 [ 65.164028] [] schedule+0x168/0x98a [ 65.164028] [] migration_thread+0x1b3/0x24a [ 65.164028] [] kthread+0x40/0x69 [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&zone->lock){.+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] free_pages_bulk+0x21/0x1f2 [ 65.164028] [] free_hot_cold_page+0x1d8/0x233 [ 65.164028] [] free_hot_page+0xf/0x11 [ 65.164028] [] __free_pages+0x2a/0x35 [ 65.164028] [] __free_pages_bootmem+0x7e/0x82 [ 65.164028] [] free_all_bootmem_core+0xcb/0x180 [ 65.164028] [] free_all_bootmem+0xd/0xf [ 65.164028] [] mem_init+0x26/0x261 [ 65.164028] [] start_kernel+0x25d/0x30a [ 65.164028] [] i386_start_kernel+0x56/0x5e [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __free_pages_ok+0x1a9/0x386 [ 65.164028] [] __free_pages+0x33/0x35 [ 65.164028] [] __free_slab+0xad/0xb5 [ 65.164028] [] discard_slab+0x32/0x34 [ 65.164028] [] __slab_free+0xc6/0x243 [ 65.164028] [] kmem_cache_free+0x9c/0xd2 [ 65.164028] [] free_task+0x38/0x3b [ 65.164028] [] __put_task_struct+0xd0/0xd5 [ 65.164028] [] delayed_put_task_struct+0x41/0x45 [ 65.164028] [] __rcu_process_callbacks+0x162/0x1f7 [ 65.164028] [] rcu_process_callbacks+0x26/0x46 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.31326+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] rmqueue_bulk+0x2b/0x70 [ 65.164028] [] get_page_from_freelist+0x167/0x42a [ 65.164028] [] __alloc_pages_internal+0xb8/0x370 [ 65.164028] [] __slab_alloc+0x14a/0x441 [ 65.164028] [] kmem_cache_alloc+0x63/0xd1 [ 65.164028] [] alloc_cpumask_var+0x23/0x6f [ 65.164028] [] schedule+0x1a9/0x98a [ 65.164028] [] watchdog+0x3f/0x1d5 [ 65.164028] [] kthread+0x40/0x69 [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] deactivate_slab+0x71/0xd6 [ 65.164028] [] __slab_alloc+0x91/0x441 [ 65.164028] [] kmem_cache_alloc+0x63/0xd1 [ 65.164028] [] alloc_cpumask_var+0x23/0x6f [ 65.164028] [] schedule+0x1a9/0x98a [ 65.164028] [] schedule_timeout+0x1b/0x97 [ 65.164028] [] wait_for_common+0xc1/0x110 [ 65.164028] [] wait_for_completion+0x17/0x19 [ 65.164028] [] call_usermodehelper_exec+0x90/0xd0 [ 65.164028] [] kobject_uevent_env+0x489/0x4c5 [ 65.164028] [] kobject_uevent+0xa/0xc [ 65.164028] [] kset_register+0x2e/0x34 [ 65.164028] [] sysdev_class_register+0x96/0x9e [ 65.164028] [] cpu_dev_init+0xf/0x49 [ 65.164028] [] driver_init+0x26/0x28 [ 65.164028] [] kernel_init+0xe9/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (trace_wait.lock){.+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __wake_up+0x1a/0x40 [ 65.164028] [] trace_wake_up+0x2b/0x2e [ 65.164028] [] trace_boot_call+0x74/0x7b [ 65.164028] [] do_one_initcall+0x60/0x17f [ 65.164028] [] kernel_init+0xfc/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __wake_up+0x1a/0x40 [ 65.164028] [] trace_wake_up+0x2b/0x2e [ 65.164028] [] tracing_sched_wakeup_trace+0xa9/0xb4 [ 65.164028] [] probe_sched_wakeup+0x71/0xa5 [ 65.164028] [] try_to_wake_up+0x25e/0x2bf [ 65.164028] [] wake_up_process+0x14/0x16 [ 65.164028] [] pdflush_operation+0x76/0x8c [ 65.164028] [] wb_timer_fn+0x14/0x30 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] trace_wait+0x10/0x2c [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __wake_up+0x1a/0x40 [ 65.164028] [] trace_wake_up+0x2b/0x2e [ 65.164028] [] tracing_sched_wakeup_trace+0xa9/0xb4 [ 65.164028] [] probe_sched_wakeup+0x71/0xa5 [ 65.164028] [] wake_up_new_task+0x8f/0xc9 [ 65.164028] [] do_fork+0x205/0x2a2 [ 65.164028] [] kernel_thread+0x7c/0x84 [ 65.164028] [] kthreadd+0xb1/0x14d [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&rq->lock/1){.+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_nested+0x21/0x51 [ 65.164028] [] double_rq_lock+0x53/0x85 [ 65.164028] [] rebalance_domains+0x224/0x4d5 [ 65.164028] [] run_rebalance_domains+0x33/0xf2 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_nested+0x21/0x51 [ 65.164028] [] double_rq_lock+0x53/0x85 [ 65.164028] [] rebalance_domains+0x224/0x4d5 [ 65.164028] [] run_rebalance_domains+0x33/0xf2 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.45602+0x1/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_nested+0x21/0x51 [ 65.164028] [] double_rq_lock+0x53/0x85 [ 65.164028] [] rebalance_domains+0x224/0x4d5 [ 65.164028] [] run_rebalance_domains+0x33/0xf2 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] task_rq_lock+0x4e/0x78 [ 65.164028] [] try_to_wake_up+0xa1/0x2bf [ 65.164028] [] default_wake_function+0x10/0x12 [ 65.164028] [] __wake_up_common+0x3a/0x60 [ 65.164028] [] complete+0x30/0x43 [ 65.164028] [] kthread+0x24/0x69 [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __wake_up_sync+0x1e/0x49 [ 65.164028] [] sock_def_readable+0x3d/0x68 [ 65.164028] [] tcp_child_process+0x4c/0x9a [ 65.164028] [] tcp_v4_do_rcv+0x241/0x2a5 [ 65.164028] [] tcp_v4_rcv+0x67d/0x6d2 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (tcp_lock){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _write_lock_bh+0x28/0x58 [ 65.164028] [] tcp_packet+0x7a/0xea6 [ 65.164028] [] nf_conntrack_in+0x6e9/0x7d4 [ 65.164028] [] ipv4_conntrack_local+0x55/0x5c [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] __ip_local_out+0x82/0x89 [ 65.164028] [] ip_local_out+0x10/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_connect+0x2ad/0x388 [ 65.164028] [] tcp_v4_connect+0x447/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _write_lock_bh+0x28/0x58 [ 65.164028] [] tcp_packet+0x7a/0xea6 [ 65.164028] [] nf_conntrack_in+0x6e9/0x7d4 [ 65.164028] [] ipv4_conntrack_in+0x1a/0x1c [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] ip_rcv+0x1e2/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _write_lock_bh+0x28/0x58 [ 65.164028] [] tcp_packet+0x7a/0xea6 [ 65.164028] [] nf_conntrack_in+0x6e9/0x7d4 [ 65.164028] [] ipv4_conntrack_local+0x55/0x5c [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] __ip_local_out+0x82/0x89 [ 65.164028] [] ip_local_out+0x10/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_connect+0x2ad/0x388 [ 65.164028] [] tcp_v4_connect+0x447/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] tcp_lock+0x10/0x24 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _write_lock_bh+0x28/0x58 [ 65.164028] [] tcp_packet+0x7a/0xea6 [ 65.164028] [] nf_conntrack_in+0x6e9/0x7d4 [ 65.164028] [] ipv4_conntrack_local+0x55/0x5c [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] __ip_local_out+0x82/0x89 [ 65.164028] [] ip_local_out+0x10/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_send_ack+0xb3/0xbb [ 65.164028] [] tcp_delack_timer+0x15a/0x1b1 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (nf_conntrack_lock){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] nf_conntrack_in+0x3bf/0x7d4 [ 65.164028] [] ipv4_conntrack_local+0x55/0x5c [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] __ip_local_out+0x82/0x89 [ 65.164028] [] ip_local_out+0x10/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_connect+0x2ad/0x388 [ 65.164028] [] tcp_v4_connect+0x447/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] __nf_ct_refresh_acct+0x57/0x167 [ 65.164028] [] tcp_packet+0xe88/0xea6 [ 65.164028] [] nf_conntrack_in+0x6e9/0x7d4 [ 65.164028] [] ipv4_conntrack_in+0x1a/0x1c [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] ip_rcv+0x1e2/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] nf_conntrack_in+0x3bf/0x7d4 [ 65.164028] [] ipv4_conntrack_local+0x55/0x5c [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] __ip_local_out+0x82/0x89 [ 65.164028] [] ip_local_out+0x10/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_connect+0x2ad/0x388 [ 65.164028] [] tcp_v4_connect+0x447/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] nf_conntrack_lock+0x10/0x24 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] lock_timer_base+0x24/0x43 [ 65.164028] [] __mod_timer+0x2f/0xc8 [ 65.164028] [] __nf_conntrack_confirm+0x23c/0x30f [ 65.164028] [] ipv4_confirm+0xa7/0xc5 [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] ip_output+0x68/0x7c [ 65.164028] [] ip_local_out+0x1d/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_connect+0x2ad/0x388 [ 65.164028] [] tcp_v4_connect+0x447/0x49e [ 65.164028] [] inet_stream_connect+0x8f/0x204 [ 65.164028] [] sys_connect+0x59/0x76 [ 65.164028] [] sys_socketcall+0x91/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] __nf_ct_refresh_acct+0x57/0x167 [ 65.164028] [] tcp_packet+0xe88/0xea6 [ 65.164028] [] nf_conntrack_in+0x6e9/0x7d4 [ 65.164028] [] ipv4_conntrack_local+0x55/0x5c [ 65.164028] [] nf_iterate+0x34/0x80 [ 65.164028] [] nf_hook_slow+0x5e/0xca [ 65.164028] [] __ip_local_out+0x82/0x89 [ 65.164028] [] ip_local_out+0x10/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_send_ack+0xb3/0xbb [ 65.164028] [] tcp_delack_timer+0x15a/0x1b1 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&list->lock#4){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] dev_deactivate+0x11d/0x15a [ 65.164028] [] __linkwatch_run_queue+0x112/0x141 [ 65.164028] [] linkwatch_event+0x25/0x2c [ 65.164028] [] run_workqueue+0xc3/0x193 [ 65.164028] [] worker_thread+0xbb/0xc7 [ 65.164028] [] kthread+0x40/0x69 [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] dev_queue_xmit+0x304/0x482 [ 65.164028] [] arp_xmit+0x38/0x3d [ 65.164028] [] arp_send+0x37/0x3c [ 65.164028] [] arp_solicit+0x16e/0x185 [ 65.164028] [] neigh_timer_handler+0x24f/0x298 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] dev_deactivate+0x11d/0x15a [ 65.164028] [] __linkwatch_run_queue+0x112/0x141 [ 65.164028] [] linkwatch_event+0x25/0x2c [ 65.164028] [] run_workqueue+0xc3/0x193 [ 65.164028] [] worker_thread+0xbb/0xc7 [ 65.164028] [] kthread+0x40/0x69 [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.22497+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] dev_queue_xmit+0x304/0x482 [ 65.164028] [] ip_finish_output+0x1e0/0x218 [ 65.164028] [] ip_output+0x77/0x7c [ 65.164028] [] ip_local_out+0x1d/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_send_ack+0xb3/0xbb [ 65.164028] [] tcp_delack_timer+0x15a/0x1b1 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (_xmit_ETHER#2){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] dev_deactivate+0x7e/0x15a [ 65.164028] [] __linkwatch_run_queue+0x112/0x141 [ 65.164028] [] linkwatch_event+0x25/0x2c [ 65.164028] [] run_workqueue+0xc3/0x193 [ 65.164028] [] worker_thread+0xbb/0xc7 [ 65.164028] [] kthread+0x40/0x69 [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] dev_watchdog+0x43/0x193 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] dev_deactivate+0x7e/0x15a [ 65.164028] [] __linkwatch_run_queue+0x112/0x141 [ 65.164028] [] linkwatch_event+0x25/0x2c [ 65.164028] [] run_workqueue+0xc3/0x193 [ 65.164028] [] worker_thread+0xbb/0xc7 [ 65.164028] [] kthread+0x40/0x69 [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] netdev_xmit_lock_key+0x8/0x1c8 [ 65.164028] -> (&np->lock){++..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irq+0x32/0x62 [ 65.164028] [] nv_probe+0xb87/0x1b85 [ 65.164028] [] pci_device_probe+0x3e/0x5e [ 65.164028] [] driver_probe_device+0x158/0x235 [ 65.164028] [] __driver_attach+0x54/0x76 [ 65.164028] [] bus_for_each_dev+0x43/0x65 [ 65.164028] [] driver_attach+0x19/0x1b [ 65.164028] [] bus_add_driver+0xef/0x202 [ 65.164028] [] driver_register+0x76/0xd2 [ 65.164028] [] __pci_register_driver+0x58/0x84 [ 65.164028] [] init_nic+0x14/0x16 [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0xfc/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] nv_nic_irq_optimized+0x82/0x22d [ 65.164028] [] handle_IRQ_event+0x1f/0x54 [ 65.164028] [] handle_fasteoi_irq+0xaa/0xb7 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W<3>INFO: RCU detected CPU 0 stall (t=4294911085/2500 jiffies) [ 75.156028] Pid: 2701, comm: distccd Not tainted 2.6.28-tip-03865-gb8c8c2c-dirty #13131 [ 75.156028] Call Trace: [ 75.156028] [] __rcu_pending+0x50/0x182 [ 75.156028] [] rcu_pending+0x21/0x4c [ 75.156028] [] update_process_times+0x30/0x53 [ 75.156028] [] tick_periodic+0x6b/0x6d [ 75.156028] [] tick_handle_periodic+0x1e/0x60 [ 75.156028] [] __irqentry_text_start+0x74/0x87 [ 75.156028] [] apic_timer_interrupt+0x2d/0x34 [ 75.156028] [] ? delay_tsc+0xe/0xb5 [ 75.156028] [] __delay+0xe/0x10 [ 75.156028] [] _raw_spin_lock+0xa0/0xf9 [ 75.156028] [] _spin_lock_nested+0x40/0x51 [ 75.156028] [] tcp_v4_rcv+0x2da/0x6d2 [ 75.156028] [] ? ip_local_deliver_finish+0x32/0x1c5 [ 75.156028] [] ip_local_deliver_finish+0x112/0x1c5 [ 75.156028] [] ip_local_deliver+0x59/0x63 [ 75.156028] [] ip_rcv_finish+0x285/0x29b [ 75.156028] [] ip_rcv+0x1f1/0x21b [ 75.156028] [] netif_receive_skb+0x2f2/0x325 [ 75.156028] [] nv_napi_poll+0x3b9/0x4d2 [ 75.156028] [] ? net_rx_action+0x67/0x1f7 [ 75.156028] [] net_rx_action+0xd6/0x1f7 [ 75.156028] [] __do_softirq+0x9d/0x154 [ 75.156028] [] ? __do_softirq+0x0/0x154 [ 75.156028] [] ? handle_fasteoi_irq+0x0/0xb7 [ 75.156028] [] ? irq_exit+0x49/0x77 [ 75.156028] [] ? do_IRQ+0xb5/0xcb [ 75.156028] [] ? common_interrupt+0x2c/0x34 [ 75.156028] [] ? __ticket_spin_is_locked+0xb/0x16 [ 75.156028] [] ? _raw_spin_unlock+0x2d/0x92 [ 75.156028] [] ? _spin_unlock+0x22/0x25 [ 75.156028] [] ? inode_add_bytes+0x5b/0x61 [ 75.156028] [] ? ext3_new_blocks+0x7f/0x5b5 [ 75.156028] [] ? mutex_lock_nested+0x2b9/0x2d5 [ 75.156028] [] ? mutex_lock_nested+0x2cd/0x2d5 [ 75.156028] [] ? ext3_get_blocks_handle+0x136/0x816 [ 75.156028] [] ? ext3_get_blocks_handle+0x340/0x816 [ 75.156028] [] ? ext3_get_block+0x90/0xc5 [ 75.156028] [] ? __block_prepare_write+0x152/0x32b [ 75.156028] [] ? block_write_begin+0x7e/0xdb [ 75.156028] [] ? ext3_get_block+0x0/0xc5 [ 75.156028] [] ? ext3_write_begin+0xcd/0x192 [ 75.156028] [] ? ext3_get_block+0x0/0xc5 [ 75.156028] [] ? generic_file_buffered_write+0xce/0x204 [ 75.156028] [] ? __generic_file_aio_write_nolock+0x3f9/0x433 [ 75.156028] [] ? mark_held_locks+0x45/0x5c [ 75.156028] [] ? mutex_lock_nested+0x2cd/0x2d5 [ 75.156028] [] ? generic_file_aio_write+0x69/0xbd [ 75.156028] [] ? ext3_file_write+0x1f/0x90 [ 75.156028] [] ? do_sync_write+0xc0/0xfe [ 75.156028] [] ? autoremove_wake_function+0x0/0x35 [ 75.156028] [] ? lock_release_holdtime+0x8a/0x92 [ 75.156028] [] ? dnotify_parent+0x5d/0x62 [ 75.156028] [] ? _raw_spin_unlock+0x8e/0x92 [ 75.156028] [] ? security_file_permission+0x14/0x16 [ 75.156028] [] ? do_sync_write+0x0/0xfe [ 75.156028] [] ? vfs_write+0x91/0x138 [ 75.156028] [] ? sys_write+0x40/0x65 [ 75.156028] [] ? sysenter_do_call+0x12/0x3f [ 65.164028] at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] nv_nic_irq_optimized+0x82/0x22d [ 65.164028] [] handle_IRQ_event+0x1f/0x54 [ 65.164028] [] handle_fasteoi_irq+0xaa/0xb7 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.35695+0x0/0x8 [ 65.164028] -> (lweventlist_lock){+...} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] linkwatch_add_event+0x15/0x36 [ 65.164028] [] linkwatch_fire_event+0x2d/0x43 [ 65.164028] [] netif_carrier_off+0x26/0x28 [ 65.164028] [] bond_create+0x297/0x2fd [ 65.164028] [] bonding_init+0x720/0x7a6 [ 65.164028] [] do_one_initcall+0x6a/0x17f [ 65.164028] [] kernel_init+0xfc/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] linkwatch_add_event+0x15/0x36 [ 65.164028] [] linkwatch_fire_event+0x2d/0x43 [ 65.164028] [] netif_carrier_on+0x27/0x37 [ 65.164028] [] nv_linkchange+0x21/0x5b [ 65.164028] [] nv_link_irq+0x30/0x33 [ 65.164028] [] nv_nic_irq_optimized+0xfe/0x22d [ 65.164028] [] handle_IRQ_event+0x1f/0x54 [ 65.164028] [] handle_fasteoi_irq+0xaa/0xb7 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] lweventlist_lock+0x10/0x24 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] linkwatch_add_event+0x15/0x36 [ 65.164028] [] linkwatch_fire_event+0x2d/0x43 [ 65.164028] [] netif_carrier_off+0x26/0x28 [ 65.164028] [] nv_open+0x4b4/0x518 [ 65.164028] [] dev_open+0x6c/0xa1 [ 65.164028] [] dev_change_flags+0x9b/0x149 [ 65.164028] [] devinet_ioctl+0x215/0x4e3 [ 65.164028] [] inet_ioctl+0x93/0xac [ 65.164028] [] sock_ioctl+0x1cc/0x1f0 [ 65.164028] [] vfs_ioctl+0x27/0x6c [ 65.164028] [] do_vfs_ioctl+0x3c3/0x3f5 [ 65.164028] [] sys_ioctl+0x45/0x5f [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (&cwq->lock){++..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __queue_work+0x14/0x30 [ 65.164028] [] queue_work_on+0x3a/0x46 [ 65.164028] [] queue_work+0x1a/0x1d [ 65.164028] [] call_usermodehelper_exec+0x83/0xd0 [ 65.164028] [] kobject_uevent_env+0x489/0x4c5 [ 65.164028] [] kobject_uevent+0xa/0xc [ 65.164028] [] kset_register+0x2e/0x34 [ 65.164028] [] bus_register+0x93/0x231 [ 65.164028] [] platform_bus_init+0x1e/0x33 [ 65.164028] [] driver_init+0x1c/0x28 [ 65.164028] [] kernel_init+0xe9/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] in-hardirq-W at: [ 65.164028] [] __lock_acquire+0x4f9/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __queue_work+0x14/0x30 [ 65.164028] [] queue_work_on+0x3a/0x46 [ 65.164028] [] queue_work+0x1a/0x1d [ 65.164028] [] schedule_work+0x14/0x16 [ 65.164028] [] schedule_bh+0x17/0x19 [ 65.164028] [] floppy_interrupt+0x15a/0x171 [ 65.164028] [] floppy_hardint+0x22/0xe1 [ 65.164028] [] handle_IRQ_event+0x1f/0x54 [ 65.164028] [] handle_edge_irq+0xa6/0x10d [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __queue_work+0x14/0x30 [ 65.164028] [] delayed_work_timer_fn+0x26/0x29 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.24394+0x0/0x8 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __wake_up+0x1a/0x40 [ 65.164028] [] insert_work+0x48/0x50 [ 65.164028] [] __queue_work+0x22/0x30 [ 65.164028] [] queue_work_on+0x3a/0x46 [ 65.164028] [] queue_work+0x1a/0x1d [ 65.164028] [] call_usermodehelper_exec+0x83/0xd0 [ 65.164028] [] kobject_uevent_env+0x489/0x4c5 [ 65.164028] [] kobject_uevent+0xa/0xc [ 65.164028] [] kset_register+0x2e/0x34 [ 65.164028] [] bus_register+0x93/0x231 [ 65.164028] [] platform_bus_init+0x1e/0x33 [ 65.164028] [] driver_init+0x1c/0x28 [ 65.164028] [] kernel_init+0xe9/0x14a [ 65.164028] [] kernel_thread_helper+0x7/0x10 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] __queue_work+0x14/0x30 [ 65.164028] [] queue_work_on+0x3a/0x46 [ 65.164028] [] queue_work+0x1a/0x1d [ 65.164028] [] queue_delayed_work+0x11/0x23 [ 65.164028] [] schedule_delayed_work+0x16/0x18 [ 65.164028] [] linkwatch_schedule_work+0x4e/0x87 [ 65.164028] [] linkwatch_fire_event+0x3f/0x43 [ 65.164028] [] netif_carrier_off+0x26/0x28 [ 65.164028] [] nv_open+0x4b4/0x518 [ 65.164028] [] dev_open+0x6c/0xa1 [ 65.164028] [] dev_change_flags+0x9b/0x149 [ 65.164028] [] devinet_ioctl+0x215/0x4e3 [ 65.164028] [] inet_ioctl+0x93/0xac [ 65.164028] [] sock_ioctl+0x1cc/0x1f0 [ 65.164028] [] vfs_ioctl+0x27/0x6c [ 65.164028] [] do_vfs_ioctl+0x3c3/0x3f5 [ 65.164028] [] sys_ioctl+0x45/0x5f [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] lock_timer_base+0x24/0x43 [ 65.164028] [] __mod_timer+0x2f/0xc8 [ 65.164028] [] mod_timer+0x38/0x3c [ 65.164028] [] nv_open+0x4f8/0x518 [ 65.164028] [] dev_open+0x6c/0xa1 [ 65.164028] [] dev_change_flags+0x9b/0x149 [ 65.164028] [] devinet_ioctl+0x215/0x4e3 [ 65.164028] [] inet_ioctl+0x93/0xac [ 65.164028] [] sock_ioctl+0x1cc/0x1f0 [ 65.164028] [] vfs_ioctl+0x27/0x6c [ 65.164028] [] do_vfs_ioctl+0x3c3/0x3f5 [ 65.164028] [] sys_ioctl+0x45/0x5f [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_irqsave+0x3f/0x72 [ 65.164028] [] nv_start_xmit_optimized+0x36a/0x411 [ 65.164028] [] dev_hard_start_xmit+0x235/0x299 [ 65.164028] [] __qdisc_run+0xcc/0x1ad [ 65.164028] [] dev_queue_xmit+0x341/0x482 [ 65.164028] [] packet_sendmsg+0x1c1/0x20a [ 65.164028] [] sock_sendmsg+0xcf/0xe6 [ 65.164028] [] sys_sendto+0xa9/0xc8 [ 65.164028] [] sys_socketcall+0xf5/0x18a [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __qdisc_run+0xb1/0x1ad [ 65.164028] [] dev_queue_xmit+0x341/0x482 [ 65.164028] [] ip_finish_output+0x1e0/0x218 [ 65.164028] [] ip_output+0x77/0x7c [ 65.164028] [] ip_local_out+0x1d/0x20 [ 65.164028] [] ip_queue_xmit+0x2b6/0x2f7 [ 65.164028] [] tcp_transmit_skb+0x5ca/0x604 [ 65.164028] [] tcp_send_ack+0xb3/0xbb [ 65.164028] [] tcp_delack_timer+0x15a/0x1b1 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (rt_peer_lock){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] rt_bind_peer+0x23/0x4d [ 65.164028] [] __ip_select_ident+0x2e/0xb9 [ 65.164028] [] ip_push_pending_frames+0x214/0x2d2 [ 65.164028] [] icmp_push_reply+0xc9/0xd4 [ 65.164028] [] icmp_reply+0x141/0x15d [ 65.164028] [] icmp_echo+0x4c/0x4e [ 65.164028] [] icmp_rcv+0x1d5/0x209 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] rt_bind_peer+0x23/0x4d [ 65.164028] [] __ip_select_ident+0x2e/0xb9 [ 65.164028] [] ip_push_pending_frames+0x214/0x2d2 [ 65.164028] [] icmp_push_reply+0xc9/0xd4 [ 65.164028] [] icmp_reply+0x141/0x15d [ 65.164028] [] icmp_echo+0x4c/0x4e [ 65.164028] [] icmp_rcv+0x1d5/0x209 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] rt_bind_peer+0x23/0x4d [ 65.164028] [] __ip_select_ident+0x2e/0xb9 [ 65.164028] [] ip_push_pending_frames+0x214/0x2d2 [ 65.164028] [] icmp_push_reply+0xc9/0xd4 [ 65.164028] [] icmp_reply+0x141/0x15d [ 65.164028] [] icmp_echo+0x4c/0x4e [ 65.164028] [] icmp_rcv+0x1d5/0x209 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] rt_peer_lock.45038+0x10/0x24 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] rt_bind_peer+0x23/0x4d [ 65.164028] [] __ip_select_ident+0x2e/0xb9 [ 65.164028] [] ip_push_pending_frames+0x214/0x2d2 [ 65.164028] [] icmp_push_reply+0xc9/0xd4 [ 65.164028] [] icmp_reply+0x141/0x15d [ 65.164028] [] icmp_echo+0x4c/0x4e [ 65.164028] [] icmp_rcv+0x1d5/0x209 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] -> (inet_peer_idlock){-+..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] __ip_select_ident+0x42/0xb9 [ 65.164028] [] ip_push_pending_frames+0x214/0x2d2 [ 65.164028] [] icmp_push_reply+0xc9/0xd4 [ 65.164028] [] icmp_reply+0x141/0x15d [ 65.164028] [] icmp_echo+0x4c/0x4e [ 65.164028] [] icmp_rcv+0x1d5/0x209 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] in-softirq-W at: [ 65.164028] [] __lock_acquire+0x51a/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] __ip_select_ident+0x42/0xb9 [ 65.164028] [] ip_push_pending_frames+0x214/0x2d2 [ 65.164028] [] icmp_push_reply+0xc9/0xd4 [ 65.164028] [] icmp_reply+0x141/0x15d [ 65.164028] [] icmp_echo+0x4c/0x4e [ 65.164028] [] icmp_rcv+0x1d5/0x209 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] __ip_select_ident+0x42/0xb9 [ 65.164028] [] ip_push_pending_frames+0x214/0x2d2 [ 65.164028] [] icmp_push_reply+0xc9/0xd4 [ 65.164028] [] icmp_reply+0x141/0x15d [ 65.164028] [] icmp_echo+0x4c/0x4e [ 65.164028] [] icmp_rcv+0x1d5/0x209 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] inet_peer_idlock+0x10/0x24 [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock_bh+0x28/0x58 [ 65.164028] [] __ip_select_ident+0x42/0xb9 [ 65.164028] [] ip_push_pending_frames+0x214/0x2d2 [ 65.164028] [] icmp_push_reply+0xc9/0xd4 [ 65.164028] [] icmp_reply+0x141/0x15d [ 65.164028] [] icmp_echo+0x4c/0x4e [ 65.164028] [] icmp_rcv+0x1d5/0x209 [ 65.164028] [] ip_local_deliver_finish+0x112/0x1c5 [ 65.164028] [] ip_local_deliver+0x59/0x63 [ 65.164028] [] ip_rcv_finish+0x285/0x29b [ 65.164028] [] ip_rcv+0x1f1/0x21b [ 65.164028] [] netif_receive_skb+0x2f2/0x325 [ 65.164028] [] nv_napi_poll+0x3b9/0x4d2 [ 65.164028] [] net_rx_action+0xd6/0x1f7 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] ... acquired at: [ 65.164028] [] validate_chain+0x8f9/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] deactivate_slab+0x71/0xd6 [ 65.164028] [] __slab_alloc+0x91/0x441 [ 65.164028] [] kmem_cache_alloc+0x63/0xd1 [ 65.164028] [] __alloc_skb+0x2e/0x10c [ 65.164028] [] tcp_send_ack+0x2a/0xbb [ 65.164028] [] tcp_delack_timer+0x15a/0x1b1 [ 65.164028] [] run_timer_softirq+0x144/0x1a4 [ 65.164028] [] __do_softirq+0x9d/0x154 [ 65.164028] [] 0xffffffff [ 65.164028] [ 65.164028] [ 65.164028] the soft-irq-unsafe lock's dependencies: [ 65.164028] -> (&fbc->lock){--..} ops: 0 { [ 65.164028] initial-use at: [ 65.164028] [] __lock_acquire+0x5a4/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __percpu_counter_add+0x52/0x7a [ 65.164028] [] get_empty_filp+0x9f/0x1b0 [ 65.164028] [] path_lookup_open+0x23/0x7a [ 65.164028] [] do_filp_open+0xa7/0x67e [ 65.164028] [] do_sys_open+0x47/0xbc [ 65.164028] [] sys_open+0x23/0x2b [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] softirq-on-W at: [ 65.164028] [] __lock_acquire+0x58f/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __percpu_counter_add+0x52/0x7a [ 65.164028] [] get_empty_filp+0x9f/0x1b0 [ 65.164028] [] path_lookup_open+0x23/0x7a [ 65.164028] [] do_filp_open+0xa7/0x67e [ 65.164028] [] do_sys_open+0x47/0xbc [ 65.164028] [] sys_open+0x23/0x2b [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] hardirq-on-W at: [ 65.164028] [] __lock_acquire+0x56e/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] __percpu_counter_add+0x52/0x7a [ 65.164028] [] get_empty_filp+0x9f/0x1b0 [ 65.164028] [] path_lookup_open+0x23/0x7a [ 65.164028] [] do_filp_open+0xa7/0x67e [ 65.164028] [] do_sys_open+0x47/0xbc [ 65.164028] [] sys_open+0x23/0x2b [ 65.164028] [] sysenter_do_call+0x12/0x3f [ 65.164028] [] 0xffffffff [ 65.164028] } [ 65.164028] ... key at: [] __key.12530+0x0/0x8 [ 65.164028] [ 65.164028] stack backtrace: [ 65.164028] Pid: 2856, comm: ssh Not tainted 2.6.28-tip-03865-gb8c8c2c-dirty #13131 [ 65.164028] Call Trace: [ 65.164028] [] check_usage+0x345/0x356 [ 65.164028] [] validate_chain+0x67f/0xbce [ 65.164028] [] __lock_acquire+0x6e0/0x746 [ 65.164028] [] lock_acquire+0x5b/0x81 [ 65.164028] [] ? __percpu_counter_add+0x52/0x7a [ 65.164028] [] _spin_lock+0x23/0x53 [ 65.164028] [] ? __percpu_counter_add+0x52/0x7a [ 65.164028] [] __percpu_counter_add+0x52/0x7a [ 65.164028] [] tcp_close+0x1b2/0x335 [ 65.164028] [] inet_release+0x47/0x4d [ 65.164028] [] sock_release+0x15/0x58 [ 65.164028] [] sock_close+0x21/0x25 [ 65.164028] [] __fput+0xcf/0x1a2 [ 65.164028] [] fput+0x1c/0x1e [ 65.164028] [] filp_close+0x55/0x5f [ 65.164028] [] sys_close+0x6d/0xa6 [ 65.164028] [] sysenter_do_call+0x12/0x3f