From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ying Xue Subject: Re: [PATCH 1/2] tipc: avoid possible deadlock while remove link_timeout() Date: Fri, 9 Aug 2013 10:39:58 +0800 Message-ID: <5204567E.2050805@windriver.com> References: <520376BA.5040509@huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Cc: Jon Maloy , Allan Stephens , "David S. Miller" , Netdev , To: Ding Tianhong Return-path: Received: from mail.windriver.com ([147.11.1.11]:48027 "EHLO mail.windriver.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S966749Ab3HICkZ (ORCPT ); Thu, 8 Aug 2013 22:40:25 -0400 In-Reply-To: <520376BA.5040509@huawei.com> Sender: netdev-owner@vger.kernel.org List-ID: On 08/08/2013 06:45 PM, Ding Tianhong wrote: > We met lockdep warning when enable and disable the bearer for commands such as: > > tipc-config -netid=1234 -addr=1.1.3 -be=eth:eth0 > tipc-config -netid=1234 -addr=1.1.3 -bd=eth:eth0 > > [ 3001.445459] tipc: Established link <1.1.3:eth0-1.1.2:br0> on network plane A > [ 3029.457875] tipc: Disabling bearer > [ 3029.458066] > [ 3029.458071] ====================================================== > [ 3029.458075] [ INFO: possible circular locking dependency detected ] > [ 3029.458080] 3.11.0-rc3-wwd-default #4 Not tainted > [ 3029.458084] ------------------------------------------------------- > [ 3029.458088] rmmod/7092 is trying to acquire lock: > [ 3029.458092] (((timer))#3){+.-...}, at: [] del_timer_sync+0x0/0xd0 > [ 3029.458107] > [ 3029.458107] but task is already holding lock: > [ 3029.458112] (&(&b_ptr->lock)->rlock){+.-...}, at: [] bearer_disable+0x33/0xd0 [tipc] > [ 3029.458126] > [ 3029.458126] which lock already depends on the new lock. > [ 3029.458126] > [ 3029.458132] > [ 3029.458132] the existing dependency chain (in reverse order) is: > [ 3029.458137] > [ 3029.458137] -> #2 (&(&b_ptr->lock)->rlock){+.-...}: > [ 3029.458143] [] validate_chain+0x6dd/0x870 > [ 3029.458151] [] __lock_acquire+0x3db/0x670 > [ 3029.458156] [] lock_acquire+0x103/0x130 > [ 3029.458161] [] _raw_spin_lock_bh+0x41/0x80 > [ 3029.458169] [] tipc_bearer_blocked+0x20/0x40 [tipc] > [ 3029.458176] [] tipc_link_send_proto_msg+0x35b/0x520 tipc] > [ 3029.458184] [] link_state_event+0x33a/0x590 [tipc] > [ 3029.458191] [] link_start+0x29/0x40 [tipc] > [ 3029.458198] [] process_signal_queue+0x7f/0xc0 [tipc] > [ 3029.458206] [] tasklet_action+0x6d/0xf0 > [ 3029.458214] [] __do_softirq+0x16a/0x2e0 > [ 3029.458219] [] run_ksoftirqd+0x35/0x50 > [ 3029.458224] [] smpboot_thread_fn+0x1e2/0x2f0 > [ 3029.458235] [] kthread+0xde/0xf0 > [ 3029.458242] [] ret_from_fork+0x7c/0xb0 > [ 3029.458250] > [ 3029.458250] -> #1 (&(&n_ptr->lock)->rlock){+.-...}: > [ 3029.458257] [] validate_chain+0x6dd/0x870 > [ 3029.458262] [] __lock_acquire+0x3db/0x670 > [ 3029.458268] [] lock_acquire+0x103/0x130 > [ 3029.458273] [] _raw_spin_lock_bh+0x41/0x80 > [ 3029.458279] [] link_timeout+0x1c/0x170 [tipc] > [ 3029.458287] [] call_timer_fn+0xda/0x1e0 > [ 3029.458292] [] run_timer_softirq+0x2a7/0x2d0 > [ 3029.458298] [] __do_softirq+0x16a/0x2e0 > [ 3029.458304] [] irq_exit+0xd5/0xe0 > [ 3029.458309] [] smp_apic_timer_interrupt+0x45/0x60 > [ 3029.458319] [] apic_timer_interrupt+0x6f/0x80 > [ 3029.458325] [] arch_cpu_idle+0x1e/0x30 > [ 3029.458332] [] cpu_idle_loop+0x1fd/0x280 > [ 3029.458338] [] cpu_startup_entry+0x1e/0x20 > [ 3029.458343] [] rest_init+0xc1/0xd0 > [ 3029.458349] [] start_kernel+0x3a3/0x451 > [ 3029.458356] [] x86_64_start_reservations+0x1b/0x32 > [ 3029.458362] [] x86_64_start_kernel+0x13a/0x141 > [ 3029.458368] > [ 3029.458368] -> #0 (((timer))#3){+.-...}: > [ 3029.458375] [] check_prev_add+0x43e/0x4b0 > [ 3029.458380] [] validate_chain+0x6dd/0x870 > [ 3029.458386] [] __lock_acquire+0x3db/0x670 > [ 3029.458391] [] lock_acquire+0x103/0x130 > [ 3029.458397] [] del_timer_sync+0x3d/0xd0 > [ 3029.458402] [] tipc_link_delete+0x1e/0xb0 [tipc] > [ 3029.458410] [] bearer_disable+0x78/0xd0 [tipc] > [ 3029.458417] [] tipc_bearer_stop+0x34/0x60 [tipc] > [ 3029.458423] [] tipc_net_stop+0x2b/0x90 [tipc] > [ 3029.458432] [] tipc_exit+0x9/0xc0 [tipc] > [ 3029.458439] [] SyS_delete_module+0x198/0x290 > [ 3029.458445] [] system_call_fastpath+0x16/0x1b > [ 3029.458451] > [ 3029.458451] other info that might help us debug this: > [ 3029.458451] > [ 3029.458458] Chain exists of: > [ 3029.458458] ((timer))#3 --> &(&n_ptr->lock)->rlock --> &(&b_ptr->lock)->rlock > [ 3029.458458] > [ 3029.458469] Possible unsafe locking scenario: > [ 3029.458469] > [ 3029.458474] CPU0 CPU1 > [ 3029.458478] ---- ---- > [ 3029.458481] lock(&(&b_ptr->lock)->rlock); > [ 3029.458486] lock(&(&n_ptr->lock)->rlock); > [ 3029.458492] lock(&(&b_ptr->lock)->rlock); > [ 3029.458497] lock(((timer))#3); > [ 3029.458502] > [ 3029.458502] *** DEADLOCK *** > [ 3029.458502] > [ 3029.458508] 2 locks held by rmmod/7092: > [ 3029.458511] #0: (tipc_net_lock){++.-..}, at: [] tipc_net_stop+0x26/0x90 [tipc] > [ 3029.458523] #1: (&(&b_ptr->lock)->rlock){+.-...}, at: []bearer_disable+0x33/0xd0 [tipc] > [ 3029.458535] > [ 3029.458535] stack backtrace: > [ 3029.458541] CPU: 3 PID: 7092 Comm: rmmod Not tainted 3.11.0-rc3-wwd-default #4 > [ 3029.458546] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 > [ 3029.458550] 00000000ffffffff ffff88010fd09c08 ffffffff814d03dd 0000000000000000 > [ 3029.458559] ffffffff8205fca0 ffff88010fd09c48 ffffffff810b1c4f 000000000fd09c48 > [ 3029.458566] ffff88010fd09c68 ffff88010e4d4fc0 0000000000000000 ffff88010e4d56f0 > [ 3029.458574] Call Trace: > [ 3029.458579] [] dump_stack+0x4d/0xa0 > [ 3029.458585] [] print_circular_bug+0x10f/0x120 > [ 3029.458591] [] check_prev_add+0x43e/0x4b0 > [ 3029.458598] [] ? native_sched_clock+0x26/0x90 > [ 3029.458604] [] validate_chain+0x6dd/0x870 > [ 3029.458612] [] ? sched_clock_cpu+0xd8/0x110 > [ 3029.458618] [] __lock_acquire+0x3db/0x670 > [ 3029.458624] [] lock_acquire+0x103/0x130 > [ 3029.458629] [] ? try_to_del_timer_sync+0x70/0x70 > [ 3029.458635] [] del_timer_sync+0x3d/0xd0 > [ 3029.458641] [] ? try_to_del_timer_sync+0x70/0x70 > [ 3029.458649] [] tipc_link_delete+0x1e/0xb0 [tipc] > [ 3029.458656] [] bearer_disable+0x78/0xd0 [tipc] > [ 3029.458663] [] tipc_bearer_stop+0x34/0x60 [tipc] > [ 3029.458671] [] tipc_net_stop+0x2b/0x90 [tipc] > [ 3029.458679] [] tipc_exit+0x9/0xc0 [tipc] > [ 3029.458685] [] SyS_delete_module+0x198/0x290 > [ 3029.458691] [] system_call_fastpath+0x16/0x1b > > ---------------------------------------------------------------------- > > The problem is that the tipc_link_delete() will cancel the timer l_ptr->timer when > the b_ptr->lock is hold, but the l_ptr->timer still call b_ptr->lock to finish the > work, so the dead lock occurs. > I guess the issue would disappear if you apply below patch: http://article.gmane.org/gmane.network.tipc.general/4380 And the patch will be submitted into net-next tree soon. Regards, Ying > We should unlock the b_ptr->lock when del the l_ptr->timer. > > Reported-by: Wang Weidong > Signed-off-by: Ding Tianhong > --- > net/tipc/bearer.c | 8 +++++++- > net/tipc/link.c | 2 ++ > 2 files changed, 9 insertions(+), 1 deletion(-) > > diff --git a/net/tipc/bearer.c b/net/tipc/bearer.c > index cb29ef7..7687211 100644 > --- a/net/tipc/bearer.c > +++ b/net/tipc/bearer.c > @@ -460,14 +460,20 @@ static void bearer_disable(struct tipc_bearer *b_ptr) > { > struct tipc_link *l_ptr; > struct tipc_link *temp_l_ptr; > + struct list_head list; > > pr_info("Disabling bearer <%s>\n", b_ptr->name); > spin_lock_bh(&b_ptr->lock); > b_ptr->blocked = 1; > b_ptr->media->disable_bearer(b_ptr); > - list_for_each_entry_safe(l_ptr, temp_l_ptr, &b_ptr->links, link_list) { > + list_replace_init(&b_ptr->links, &list); > + spin_unlock_bh(&b_ptr->lock); > + > + list_for_each_entry_safe(l_ptr, temp_l_ptr, &list, link_list) { > tipc_link_delete(l_ptr); > } > + > + spin_lock_bh(&b_ptr->lock); > if (b_ptr->link_req) > tipc_disc_delete(b_ptr->link_req); > spin_unlock_bh(&b_ptr->lock); > diff --git a/net/tipc/link.c b/net/tipc/link.c > index 0cc3d90..a145718 100644 > --- a/net/tipc/link.c > +++ b/net/tipc/link.c > @@ -384,10 +384,12 @@ void tipc_link_delete(struct tipc_link *l_ptr) > k_cancel_timer(&l_ptr->timer); > > tipc_node_lock(l_ptr->owner); > + spin_lock_bh(&l_ptr->b_ptr->lock); > tipc_link_reset(l_ptr); > tipc_node_detach_link(l_ptr->owner, l_ptr); > tipc_link_stop(l_ptr); > list_del_init(&l_ptr->link_list); > + spin_unlock_bh(&l_ptr->b_ptr->lock); > tipc_node_unlock(l_ptr->owner); > k_term_timer(&l_ptr->timer); > kfree(l_ptr); >