From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ben Greear Subject: network related lockdep splat in 3.0-rc6+ Date: Fri, 08 Jul 2011 15:38:17 -0700 Message-ID: <4E1786D9.3020209@candelatech.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit To: netdev Return-path: Received: from mail.candelatech.com ([208.74.158.172]:34001 "EHLO ns3.lanforge.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751872Ab1GHWiS (ORCPT ); Fri, 8 Jul 2011 18:38:18 -0400 Received: from [192.168.100.195] (firewall.candelatech.com [70.89.124.249]) (authenticated bits=0) by ns3.lanforge.com (8.14.2/8.14.2) with ESMTP id p68McH9f013262 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 8 Jul 2011 15:38:17 -0700 Sender: netdev-owner@vger.kernel.org List-ID: This has some additional NFS patches as well, but is otherwise un-tainted. ======================================================= [ INFO: possible circular locking dependency detected ] 3.0.0-rc6+ #13 ------------------------------------------------------- gnuserver/2266 is trying to acquire lock: (rcu_node_level_0){..-...}, at: [] rcu_report_unblock_qs_rnp+0x52/0x72 but task is already holding lock: (&rq->lock){-.-.-.}, at: [] sched_ttwu_pending+0x34/0x58 which lock already depends on the new lock. the existing dependency chain (in reverse order) is: -> #3 (&rq->lock){-.-.-.}: [] lock_acquire+0xf4/0x14b [] _raw_spin_lock+0x36/0x45 [] __task_rq_lock+0x5b/0x89 [] wake_up_new_task+0x41/0x116 [] do_fork+0x207/0x2f1 [] kernel_thread+0x70/0x72 [] rest_init+0x21/0xd7 [] start_kernel+0x3bd/0x3c8 [] x86_64_start_reservations+0xb8/0xbc [] x86_64_start_kernel+0x101/0x110 -> #2 (&p->pi_lock){-.-.-.}: [] lock_acquire+0xf4/0x14b [] _raw_spin_lock_irqsave+0x4e/0x60 [] try_to_wake_up+0x29/0x1a0 [] default_wake_function+0xd/0xf [] autoremove_wake_function+0x13/0x38 [] __wake_up_common+0x49/0x7f [] __wake_up+0x34/0x48 [] rcu_report_exp_rnp+0x50/0x89 [] __rcu_read_unlock+0x1e9/0x24e [] rcu_read_unlock+0x21/0x23 [] fget_light+0xa2/0xac [] do_sys_poll+0x1ff/0x3e5 [] sys_poll+0x50/0xba [] system_call_fastpath+0x16/0x1b -> #1 (sync_rcu_preempt_exp_wq.lock){......}: [] lock_acquire+0xf4/0x14b [] _raw_spin_lock_irqsave+0x4e/0x60 [] __wake_up+0x1d/0x48 [] rcu_report_exp_rnp+0x50/0x89 [] sync_rcu_preempt_exp_init.clone.0+0x3e/0x53 [] synchronize_rcu_expedited+0xdb/0x1c3 [] synchronize_net+0x25/0x2e [] rollback_registered_many+0xee/0x1e1 [] unregister_netdevice_many+0x14/0x55 [] default_device_exit_batch+0x98/0xb4 [] ops_exit_list+0x46/0x4e [] cleanup_net+0xeb/0x17d [] process_one_work+0x230/0x41d [] worker_thread+0x133/0x217 [] kthread+0x7d/0x85 [] kernel_thread_helper+0x4/0x10 -> #0 (rcu_node_level_0){..-...}: [] __lock_acquire+0xae6/0xdd5 [] lock_acquire+0xf4/0x14b [] _raw_spin_lock+0x36/0x45 [] rcu_report_unblock_qs_rnp+0x52/0x72 [] __rcu_read_unlock+0x1a7/0x24e [] rcu_read_unlock+0x21/0x23 [] cpuacct_charge+0x53/0x5b [] update_curr+0x11f/0x15a [] enqueue_task_fair+0x46/0x22a [] enqueue_task+0x61/0x68 [] activate_task+0x28/0x30 [] ttwu_activate+0x12/0x34 [] ttwu_do_activate.clone.4+0x2d/0x3f [] sched_ttwu_pending+0x43/0x58 [] scheduler_ipi+0x9/0xb [] smp_reschedule_interrupt+0x25/0x27 [] reschedule_interrupt+0x13/0x20 [] rcu_read_unlock+0x21/0x23 [] ip_queue_xmit+0x35e/0x3b1 [] tcp_transmit_skb+0x785/0x7c3 [] tcp_write_xmit+0x806/0x8f5 [] __tcp_push_pending_frames+0x20/0x4d [] tcp_push+0x84/0x86 [] tcp_sendmsg+0x674/0x775 [] inet_sendmsg+0x61/0x6a [] __sock_sendmsg_nosec+0x58/0x61 [] __sock_sendmsg+0x3d/0x48 [] sock_sendmsg+0xa3/0xbc [] sys_sendto+0xfa/0x11f [] system_call_fastpath+0x16/0x1b other info that might help us debug this: Chain exists of: rcu_node_level_0 --> &p->pi_lock --> &rq->lock Possible unsafe locking scenario: CPU0 CPU1 ---- ---- lock(&rq->lock); lock(&p->pi_lock); lock(&rq->lock); lock(rcu_node_level_0); *** DEADLOCK *** 2 locks held by gnuserver/2266: #0: (sk_lock-AF_INET){+.+.+.}, at: [] tcp_sendmsg+0x21/0x775 #1: (&rq->lock){-.-.-.}, at: [] sched_ttwu_pending+0x34/0x58 stack backtrace: Pid: 2266, comm: gnuserver Not tainted 3.0.0-rc6+ #13 Call Trace: [] print_circular_bug+0x1fe/0x20f [] __lock_acquire+0xae6/0xdd5 [] ? rcu_report_unblock_qs_rnp+0x52/0x72 [] lock_acquire+0xf4/0x14b [] ? rcu_report_unblock_qs_rnp+0x52/0x72 [] _raw_spin_lock+0x36/0x45 [] ? rcu_report_unblock_qs_rnp+0x52/0x72 [] ? _raw_spin_unlock+0x45/0x52 [] rcu_report_unblock_qs_rnp+0x52/0x72 [] ? __rcu_read_unlock+0xdc/0x24e [] __rcu_read_unlock+0x1a7/0x24e [] rcu_read_unlock+0x21/0x23 [] cpuacct_charge+0x53/0x5b [] update_curr+0x11f/0x15a [] enqueue_task_fair+0x46/0x22a [] enqueue_task+0x61/0x68 [] activate_task+0x28/0x30 [] ttwu_activate+0x12/0x34 [] ttwu_do_activate.clone.4+0x2d/0x3f [] sched_ttwu_pending+0x43/0x58 [] scheduler_ipi+0x9/0xb [] smp_reschedule_interrupt+0x25/0x27 [] reschedule_interrupt+0x13/0x20 [] ? __rcu_read_unlock+0x52/0x24e [] rcu_read_unlock+0x21/0x23 [] ip_queue_xmit+0x35e/0x3b1 [] ? ip_send_reply+0x247/0x247 [] tcp_transmit_skb+0x785/0x7c3 [] tcp_write_xmit+0x806/0x8f5 [] ? might_fault+0x4e/0x9e [] ? copy_from_user+0x2a/0x2c [] __tcp_push_pending_frames+0x20/0x4d [] tcp_push+0x84/0x86 [] tcp_sendmsg+0x674/0x775 [] inet_sendmsg+0x61/0x6a [] __sock_sendmsg_nosec+0x58/0x61 [] __sock_sendmsg+0x3d/0x48 [] sock_sendmsg+0xa3/0xbc [] ? handle_pte_fault+0x7fc/0x84d [] ? fget_light+0x35/0xac [] ? sockfd_lookup_light+0x1b/0x53 [] sys_sendto+0xfa/0x11f [] ? mntput_no_expire+0x52/0x109 [] ? mntput+0x21/0x23 [] ? fput+0x1a3/0x1b2 [] ? audit_syscall_entry+0x119/0x145 [] system_call_fastpath+0x16/0x1b -- Ben Greear Candela Technologies Inc http://www.candelatech.com