From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andy Furniss Date: Mon, 25 Jun 2007 01:26:12 +0000 Subject: Re: [LARTC] HTB deadlock Message-Id: <467F19B4.1080307@andyfurniss.entadsl.com> List-Id: References: <1181752830.9399.66.camel@ranko-fc2.spidernet.net> In-Reply-To: <1181752830.9399.66.camel@ranko-fc2.spidernet.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: lartc@vger.kernel.org Ranko Zivojnovic wrote: > Greetings, > > I've been experiencing problems with HTB where the whole machine locks > up. This usually happens when the whole qdisc is being removed and > occasionally when a leaf is being removed. > > Common is that it always happens when some sort of removal is in > progress. > > Console output I have captured is at the end of this message. The same > behavior exists from vanilla 2.6.19.7 and above. It is possible that the > problem also exist in the earlier versions however I did not go further > back. > > I also believe I have found where the actual problem is: > > qdisc_destroy() function is always called with dev->queue_lock locked. > htb_destroy() function up the stack is using del_timer_sync() call to > deactivate HTB qdisc timers. > >>>From the comments in the source where del_timer_sync() is defined: > > ---copy/paste--- > /** > * del_timer_sync - deactivate a timer and wait for the handler to finish. > * @timer: the timer to be deactivated > * > * This function only differs from del_timer() on SMP: besides deactivating > * the timer it also makes sure the handler has finished executing on other > * CPUs. > * > * Synchronization rules: Callers must prevent restarting of the timer, > * otherwise this function is meaningless. It must not be called from > * interrupt contexts. The caller must not hold locks which would prevent > * completion of the timer's handler. The timer's handler must not call > * add_timer_on(). Upon exit the timer is not queued and the handler is > * not running on any CPU. > * > * The function returns whether it has deactivated a pending timer or not. > */ > ---copy/paste--- > > Now, htb_rate_timer() does exactly what appears to be the source of the > problem - it tries obtain dev->queue_lock - and given the right moment > (timer fired handler while qdisc_destroy was holding the lock) - system > locks up - del_timer_sync is waiting for handler to finish while the > handler is waiting for the dev->queue_lock. > > Of course I could also be completely wrong here and missing something > not so obvious. > > I could also attempt to fix this but I haven't dealt with this code in > the past so I was hoping someone with better insight might just have an > elegant solution up his sleeve. > > Best regards, > > Ranko > > PS: If this list is not the right place for this report - please let me > know. You should send bug reports to netdev@vger.kernel.org > > -----------CONSOLE (2.6.19.7)----------- > BUG: soft lockup detected on CPU#3! > [] softlockup_tick+0x93/0xc2 > [] update_process_times+0x26/0x5c > [] smp_apic_timer_interrupt+0x97/0xb2 > [] apic_timer_interrupt+0x1f/0x24 > [] klist_next+0x4/0x8a > [] _spin_unlock_irqrestore+0xa/0xc > [] try_to_del_timer_sync+0x47/0x4f > [] del_timer_sync+0xe/0x14 > [] htb_destroy+0x20/0x7b [sch_htb] > [] qdisc_destroy+0x44/0x8d > [] htb_destroy_class+0xd0/0x12d [sch_htb] > [] htb_destroy_class+0x52/0x12d [sch_htb] > [] htb_destroy+0x3f/0x7b [sch_htb] > [] qdisc_destroy+0x44/0x8d > [] htb_destroy_class+0xd0/0x12d [sch_htb] > [] htb_destroy_class+0x52/0x12d [sch_htb] > [] htb_destroy+0x3f/0x7b [sch_htb] > [] qdisc_destroy+0x44/0x8d > [] tc_get_qdisc+0x1a3/0x1ef > [] tc_get_qdisc+0x0/0x1ef > [] rtnetlink_rcv_msg+0x158/0x215 > [] rtnetlink_rcv_msg+0x0/0x215 > [] netlink_run_queue+0x88/0x11d > [] rtnetlink_rcv+0x26/0x42 > [] netlink_data_ready+0x12/0x54 > [] netlink_sendskb+0x1c/0x33 > [] netlink_sendmsg+0x1ee/0x2d7 > [] sock_sendmsg+0xe5/0x100 > [] autoremove_wake_function+0x0/0x37 > [] autoremove_wake_function+0x0/0x37 > [] sock_sendmsg+0xe5/0x100 > [] copy_from_user+0x33/0x69 > [] sys_sendmsg+0x12d/0x243 > [] _read_unlock_irq+0x5/0x7 > [] find_get_page+0x37/0x42 > [] filemap_nopage+0x30c/0x3a3 > [] __handle_mm_fault+0x21c/0x943 > [] _spin_unlock_bh+0x5/0xd > [] sock_setsockopt+0x63/0x59d > [] anon_vma_prepare+0x1b/0xcb > [] sys_socketcall+0x24f/0x271 > [] do_page_fault+0x0/0x600 > [] sysenter_past_esp+0x56/0x79 > ===========> BUG: soft lockup detected on CPU#1! > [] softlockup_tick+0x93/0xc2 > [] update_process_times+0x26/0x5c > [] smp_apic_timer_interrupt+0x97/0xb2 > [] apic_timer_interrupt+0x1f/0x24 > [] blk_do_ordered+0x70/0x27e > [] _raw_spin_lock+0xaa/0x13e > [] htb_rate_timer+0x18/0xc4 [sch_htb] > [] run_timer_softirq+0x163/0x189 > [] htb_rate_timer+0x0/0xc4 [sch_htb] > [] __do_softirq+0x70/0xdb > [] do_softirq+0x3b/0x42 > [] smp_apic_timer_interrupt+0x9c/0xb2 > [] apic_timer_interrupt+0x1f/0x24 > [] mwait_idle_with_hints+0x3b/0x3f > [] mwait_idle+0xc/0x1b > [] cpu_idle+0x63/0x79 > ===========> BUG: soft lockup detected on CPU#2! > [] softlockup_tick+0x93/0xc2 > [] update_process_times+0x26/0x5c > [] smp_apic_timer_interrupt+0x97/0xb2 > [] apic_timer_interrupt+0x1f/0x24 > [] blk_do_ordered+0x70/0x27e > [] _raw_spin_lock+0xaa/0x13e > [] dev_queue_xmit+0x53/0x2e4 > [] neigh_connected_output+0x80/0xa0 > [] ip_output+0x1b5/0x24b > [] ip_finish_output+0x0/0x192 > [] ip_forward+0x1c8/0x2b9 > [] ip_forward_finish+0x0/0x37 > [] ip_rcv+0x2a5/0x538 > [] ip_rcv_finish+0x0/0x2aa > [] __netdev_alloc_skb+0x12/0x2a > [] ip_rcv+0x0/0x538 > [] netif_receive_skb+0x218/0x318 > [] bitmap_get_counter+0x41/0x1e6 > [] e1000_clean_rx_irq+0x12c/0x4ef [e1000] > [] e1000_clean_rx_irq+0x0/0x4ef [e1000] > [] e1000_clean+0xe5/0x130 [e1000] > [] net_rx_action+0xbc/0x1d5 > [] __do_softirq+0x70/0xdb > [] do_softirq+0x3b/0x42 > [] do_IRQ+0x6c/0xda > [] common_interrupt+0x1a/0x20 > [] mwait_idle_with_hints+0x3b/0x3f > [] mwait_idle+0xc/0x1b > [] cpu_idle+0x63/0x79 > ===========> BUG: soft lockup detected on CPU#0! > [] softlockup_tick+0x93/0xc2 > [] update_process_times+0x26/0x5c > [] smp_apic_timer_interrupt+0x97/0xb2 > [] apic_timer_interrupt+0x1f/0x24 > [] delay_tsc+0x7/0x13 > [] __delay+0x6/0x7 > [] _raw_spin_lock+0xb8/0x13e > [] dev_queue_xmit+0x53/0x2e4 > [] neigh_connected_output+0x80/0xa0 > [] ip_output+0x1b5/0x24b > [] ip_finish_output+0x0/0x192 > [] ip_forward+0x1c8/0x2b9 > [] ip_forward_finish+0x0/0x37 > [] ip_rcv+0x2a5/0x538 > [] ip_rcv_finish+0x0/0x2aa > [] __alloc_skb+0x47/0xf3 > [] ip_rcv+0x0/0x538 > [] netif_receive_skb+0x218/0x318 > [] bitmap_get_counter+0x41/0x1e6 > [] tg3_poll+0x6d3/0x906 [tg3] > [] net_rx_action+0xbc/0x1d5 > [] __do_softirq+0x70/0xdb > [] do_softirq+0x3b/0x42 > [] do_IRQ+0x6c/0xda > [] common_interrupt+0x1a/0x20 > [] mwait_idle_with_hints+0x3b/0x3f > [] mwait_idle+0xc/0x1b > [] cpu_idle+0x63/0x79 > [] start_kernel+0x353/0x423 > [] unknown_bootoption+0x0/0x260 > ===========> -----------CONSOLE----------- > > > > > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc