From mboxrd@z Thu Jan 1 00:00:00 1970 From: Remy Bohmer Subject: System deadlock on 2.6.24-rt20 Date: Fri, 3 Apr 2009 11:48:49 +0200 Message-ID: <3efb10970904030248t39a13176led25b9dc7ff9383c@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: LKML , linux-rt-users To: Steven Rostedt , Ingo Molnar , Thomas Gleixner Return-path: Received: from mail-fx0-f158.google.com ([209.85.220.158]:64852 "EHLO mail-fx0-f158.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755626AbZDCJsw (ORCPT ); Fri, 3 Apr 2009 05:48:52 -0400 Sender: linux-rt-users-owner@vger.kernel.org List-ID: Hello All, We encountered a nasty system lockup on an X86 system running 2.6.24.7-rt20. (Intel Core2Duo system) What happens is that the system freezes completely after some time. After a lot of testing we are now able to reproduce this problem within 5 minutes of testing. The backtrace was not printed to the serial terminal, but when we accidentally used the VGA terminal the backtrace was there. So we used/hacked the earlyprintk mechanism to get the logging out. Does anybody recognise these backtraces? And if so, is there a fix for it already? There are several independent processes involved, all with RT-priorities. Kind regards, Remy ============= Trace 1 ======================== <0>BUG: spinlock lockup on CPU#0, A_EngineNode/2487, f78c5004 <0>BUG: spinlock lockup on CPU#1, A_EngineNode/2296, c3e13400 <4>Pid: 2487, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4 <4> [] Pid: 2296, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4 <4> [] show_trace_log_lvl+0x1a/0x30 <4> [] show_trace_log_lvl+0x1a/0x30 <4> [] show_trace+0x12/0x20 <4> [] show_trace+0x12/0x20 <4> [] dump_stack+0x6c/0x80 <4> [] dump_stack+0x6c/0x80 <4>_raw_spin_lock+0x141/0x180 <4> [] [] _raw_spin_lock+0x141/0x180 <4> [] __spin_lock_irqsave+0x4a/0x60 <4> [] __spin_lock+0x3d/0x50 <4> [] cpupri_set+0x4a/0xd0 <4> [] task_rq_lock+0x3c/0x70 <4> [] dequeue_task_rt+0xd8/0x1f0 <4> [] try_to_wake_up+0x2c/0x110 <4> [] dequeue_task+0x15/0x30 <4> [] default_wake_function+0x18/0x20 <4> [] deactivate_task+0x28/0x40 <4> [] __wake_up_common+0x48/0x70 <4> [] __schedule+0x2ff/0x780 <4> [] __wake_up+0x33/0x60 <4> [] schedule+0x33/0x120 <4> [] wake_futex+0x3e/0x50 <4>futex_wait+0x26f/0x3a0 <4> [] [] futex_wake+0x98/0x100 <4>do_futex+0x8c/0xb40 <4> [] [] do_futex+0x3f4/0xb40 <4>sys_futex+0x85/0x100 <4> [] [] sys_futex+0x85/0x100 <4>sysenter_past_esp+0x5f/0x99 <4> ======================= <4> [] sysenter_past_esp+0x5f/0x99 <4> ======================= ============= Trace 2 ======================== <0>BUG: spinlock lockup on CPU#0, A_EngineNode/3069, f78c5004 <4>Pid: 3069, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4 <4> [] show_trace_log_lvl+0x1a/0x30 <4> [] show_trace+0x12/0x20 <4> [] dump_stack+0x6c/0x80 <4> [] _raw_spin_lock+0x141/0x180 <0>BUG: spinlock lockup on CPU#1, usbpui/2208, c3e13400 <4>Pid: 2208, comm: usbpui Not tainted 2.6.24.7-rt20 #4 <4> [] [] __spin_lock_irqsave+0x4a/0x60 <4> [] show_trace_log_lvl+0x1a/0x30 <4> [] cpupri_set+0x4a/0xd0 <4>show_trace+0x12/0x20 <4> [] [] dump_stack+0x6c/0x80 <4> [] dequeue_task_rt+0xd8/0x1f0 <4> [] _raw_spin_lock+0x141/0x180 <4>dequeue_task+0x15/0x30 <4> [] [] __spin_lock+0x3d/0x50 <4>deactivate_task+0x28/0x40 <4> [] [] task_rq_lock+0x3c/0x70 <4> [] __schedule+0x2ff/0x780 <4>try_to_wake_up+0x2c/0x110 <4> [] [] schedule+0x33/0x120 <4> [] default_wake_function+0x18/0x20 <4> [] futex_wait+0x26f/0x3a0 <4> [] __wake_up_common+0x48/0x70 <4>do_futex+0x8c/0xb40 <4> [] [] __wake_up+0x33/0x60 <4>sys_futex+0x85/0x100 <4> [] [] sysenter_past_esp+0x5f/0x99 <4> ======================= <4>wake_futex+0x3e/0x50 <4> [] futex_wake+0x98/0x100 <4> [] do_futex+0x3f4/0xb40 <4> [] sys_futex+0x85/0x100 <4> [] sysenter_past_esp+0x5f/0x99 <4> ======================= ============= Trace 3 ======================== <0>BUG: spinlock lockup on CPU#0, A_EngineNode/2525, f78c5004 <0>BUG: spinlock lockup on CPU#1, A_EngineNode/2498, c3e13400 <4>Pid: 2498, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4 <4>Pid: 2525, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4 <4> [] [] show_trace_log_lvl+0x1a/0x30 <4>show_trace_log_lvl+0x1a/0x30 <4> [] [] show_trace+0x12/0x20 <4>show_trace+0x12/0x20 <4> [] [] dump_stack+0x6c/0x80 <4>dump_stack+0x6c/0x80 <4> [] [] _raw_spin_lock+0x141/0x180 <4> [] _raw_spin_lock+0x141/0x180 <4> [] __spin_lock+0x3d/0x50 <4>__spin_lock_irqsave+0x4a/0x60 <4> [] [] task_rq_lock+0x3c/0x70 <4>cpupri_set+0x4a/0xd0 <4> [] [] dequeue_task_rt+0xd8/0x1f0 <4>try_to_wake_up+0x2c/0x110 <4> [] [] dequeue_task+0x15/0x30 <4>default_wake_function+0x18/0x20 <4> [] [] deactivate_task+0x28/0x40 <4>__wake_up_common+0x48/0x70 <4> [] [] __wake_up+0x33/0x60 <4>__schedule+0x2ff/0x780 <4> [] [] schedule+0x33/0x120 <4> [] wake_futex+0x3e/0x50 <4> [] futex_wait+0x26f/0x3a0 <4> [] futex_wake+0x98/0x100 <4>do_futex+0x8c/0xb40 <4> [] [] sys_futex+0x85/0x100 <4>do_futex+0x3f4/0xb40 <4> [] [] sysenter_past_esp+0x5f/0x99 <4>sys_futex+0x85/0x100 <4> [] ======================= <4>sysenter_past_esp+0x5f/0x99 <4> ======================= ============= Trace 4 ======================== <0>BUG: spinlock lockup on CPU#0, A_EngineNode/3044, f78c5004 <0>BUG: spinlock lockup on CPU#1, A_EngineNode/3001, c3e13400 <4>Pid: 3044, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4 <4>Pid: 3001, comm: A_EngineNode Not tainted 2.6.24.7-rt20 #4 <4> [] [] show_trace_log_lvl+0x1a/0x30 <4>show_trace_log_lvl+0x1a/0x30 <4> [] [] show_trace+0x12/0x20 <4>show_trace+0x12/0x20 <4> [] [] dump_stack+0x6c/0x80 <4>dump_stack+0x6c/0x80 <4> [] [] _raw_spin_lock+0x141/0x180 <4>_raw_spin_lock+0x141/0x180 <4> [] [] __spin_lock+0x3d/0x50 <4>__spin_lock_irqsave+0x4a/0x60 <4> [] [] task_rq_lock+0x3c/0x70 <4>cpupri_set+0x4a/0xd0 <4> [] [] try_to_wake_up+0x2c/0x110 <4>dequeue_task_rt+0xd8/0x1f0 <4> [] [] wake_up_process_mutex+0x19/0x20 <4> [] dequeue_task+0x15/0x30 <4> [] wakeup_next_waiter+0x11c/0x1c0 <4> [] deactivate_task+0x28/0x40 <4>rt_spin_lock_slowunlock+0x4c/0x70 <4> [] [] rt_spin_unlock+0x52/0x60 <4>__schedule+0x2ff/0x780 <4> [] [] schedule+0x33/0x120 <4> [] futex_wake+0xab/0x100 <4> [] rt_spin_lock_slowlock+0x13c/0x1f4 <4> [] do_futex+0x3f4/0xb40 <4>__lock_text_start+0x2a/0x50 <4> [] [] sys_futex+0x85/0x100 <4> [] rt_spin_lock+0x32/0x40 <4> [] sysenter_past_esp+0x5f/0x99 <4> ======================= <4>unqueue_me+0x18/0xa0 <4> [] futex_wait+0x2aa/0x3a0 <4> [] do_futex+0x8c/0xb40 <4> [] sys_futex+0x85/0x100 <4> [] sysenter_past_esp+0x5f/0x99 <4> =======================