Hi John, The below lockup warning pops up very occasionally in kvm guest kernels and it's bisected down to commit 2a8c0883c3cfffcc148ea606e2a4e7453cd75e73 Author: John Stultz Date: Fri Jul 13 01:21:56 2012 -0400 time: Move xtime_nsec adjustment underflow handling timekeeping_adjust [ 8.340327] Freeing unused kernel memory: 460k freed [ 1107.515496] [sched_delayed] sched: RT throttling activated [ 1107.516439] BUG: soft lockup - CPU#1 stuck for 1024s! [swapper/1:0] [ 1107.516439] irq event stamp: 13644 [ 1107.516439] hardirqs last enabled at (13641): [] restore_all_notrace+0x0/0x18 [ 1107.516439] hardirqs last disabled at (13642): [] apic_timer_interrupt+0x2f/0x40 [ 1107.516439] softirqs last enabled at (13644): [] _local_bh_enable+0xd/0xf [ 1107.516439] softirqs last disabled at (13643): [] irq_enter+0x2c/0x4d [ 1107.516439] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-00023-g8a13a4c #66 [ 1107.516439] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1107.516439] EIP is at native_safe_halt+0x5/0x7 [ 1107.516439] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1107.516439] ESI: cfff9e80 EDI: cfff9e70 EBP: ce071f6c ESP: ce071f6c [ 1107.516439] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1107.516439] CR0: 8005003b CR2: b766c130 CR3: 0e83b000 CR4: 00000690 [ 1107.516439] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1107.516439] DR6: ffff0ff0 DR7: 00000400 [ 1107.516439] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) [ 1107.516439] Stack: [ 1107.516439] ce071f74 c1008a89 ce071f88 c10092c6 cde0a192 dc4426ba 168f476e ce071fb4 [ 1107.516439] c166d17b 00000000 00000000 00000000 00000000 86be5003 168f476e 01000800 [ 1107.516439] 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 1107.516439] Call Trace: [ 1107.516439] [] default_idle+0x2f/0x55 [ 1107.516439] [] cpu_idle+0xc7/0xf9 [ 1107.516439] [] start_secondary+0x296/0x29e [ 1107.516439] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c7 89 55 f0 89 Below are all the oops messages during the bisect. [ 4.459980] Freeing unused kernel memory: 460k freed [ 1103.514811] BUG: soft lockup - CPU#1 stuck for 1024s! [swapper/1:0] [ 1103.514811] irq event stamp: 7390 [ 1103.514811] hardirqs last enabled at (7387): [] restore_all_notrace+0x0/0x18 [ 1103.514811] hardirqs last disabled at (7388): [] apic_timer_interrupt+0x2f/0x40 [ 1103.514811] softirqs last enabled at (7390): [] _local_bh_enable+0xd/0xf [ 1103.514811] softirqs last disabled at (7389): [] irq_enter+0x2c/0x4d [ 1103.514811] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.514811] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1103.514811] EIP is at native_safe_halt+0x5/0x7 [ 1103.523837] [sched_delayed] sched: RT throttling activated [ 1103.514811] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1103.514811] ESI: c19716f0 EDI: ffff8b9d EBP: ce071f68 ESP: ce071f68 [ 1103.514811] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.514811] CR0: 8005003b CR2: b767dfb8 CR3: 0ee9c000 CR4: 00000690 [ 1103.514811] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.514811] DR6: ffff0ff0 DR7: 00000400 [ 1103.514811] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) [ 1103.514811] Stack: [ 1103.514811] ce071f70 c1008a09 ce071f88 c1009250 a987df14 b7eb6434 f2365b03 cfff9fa0 [ 1103.514811] ce071fb4 c166ed10 00000000 00000000 00000000 00000000 626563a0 f2365b03 [ 1103.514811] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1103.514811] Call Trace: [ 1103.514811] [] default_idle+0x2f/0x55 [ 1103.514811] [] cpu_idle+0xc9/0xfb [ 1103.514811] [] start_secondary+0x292/0x29a [ 1103.514811] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1103.514811] Kernel panic - not syncing: softlockup: hung tasks [ 1103.514811] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.514811] Call Trace: [ 1103.514811] [] panic+0x97/0x1aa [ 1103.514811] [] watchdog_timer_fn+0x115/0x134 [ 1103.514811] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.514811] [] ? __touch_watchdog+0x1d/0x1d [ 1103.514811] [] run_local_timers+0x8/0x14 [ 1103.514811] [] update_process_times+0x24/0x58 [ 1103.514811] [] tick_periodic+0x66/0x72 [ 1103.514811] [] tick_handle_periodic+0x19/0x61 [ 1103.514811] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.514811] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.514811] [] apic_timer_interrupt+0x36/0x40 [ 1103.514811] [] ? native_safe_halt+0x5/0x7 [ 1103.514811] [] default_idle+0x2f/0x55 [ 1103.514811] [] cpu_idle+0xc9/0xfb [ 1103.514811] [] start_secondary+0x292/0x29a [ 1103.514811] Rebooting in 10 seconds.. -- [ 5.274691] Freeing unused kernel memory: 460k freed [ 1104.516417] BUG: soft lockup - CPU#1 stuck for 1024s! [umount:132] [ 1104.516417] irq event stamp: 3118 [ 1104.516417] hardirqs last enabled at (3117): [] _raw_spin_unlock_irqrestore+0x42/0x63 [ 1104.516417] hardirqs last disabled at (3118): [] apic_timer_interrupt+0x2f/0x40 [ 1104.516417] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1104.516417] softirqs last disabled at (0): [< (null)>] (null) [ 1104.516417] Pid: 132, comm: umount Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.516417] EIP: 0060:[] EFLAGS: 00000202 CPU: 1 [ 1104.522300] [sched_delayed] sched: RT throttling activated [ 1104.516417] EIP is at unmap_single_vma+0x206/0x473 [ 1104.516417] EAX: ce8afec0 EBX: ce68bde0 ECX: 0a66f025 EDX: b76d7000 [ 1104.516417] ESI: ce89eb5c EDI: 00000000 EBP: ce8afe98 ESP: ce8afe34 [ 1104.516417] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.516417] CR0: 80050033 CR2: b75bbfb8 CR3: 0e897000 CR4: 00000690 [ 1104.516417] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.516417] DR6: ffff0ff0 DR7: 00000400 [ 1104.516417] Process umount (pid: 132, ti=ce8ae000 task=cf8a9130 task.ti=ce8ae000) [ 1104.516417] Stack: [ 1104.516417] 00000002 00000203 006f0203 00000000 cf8a95a4 b76fbfff ce148f34 ce89eb58 [ 1104.516417] 0a66f025 ce897b74 ce897b74 ce148f00 00000000 ce8afec0 b76fc000 b76fc000 [ 1104.516417] ce8ad1b8 b76d7000 ce8afe80 ffffffff 00000000 00000000 ce8ad1b8 ce8afec0 [ 1104.516417] Call Trace: [ 1104.516417] [] unmap_vmas+0x2a/0x40 [ 1104.516417] [] exit_mmap+0x76/0xe4 [ 1104.516417] [] mmput+0x42/0xa7 [ 1104.516417] [] exit_mm+0xde/0xe6 [ 1104.516417] [] do_exit+0x1d0/0x72c [ 1104.516417] [] ? vfs_write+0x105/0x10f [ 1104.516417] [] do_group_exit+0x5e/0x81 [ 1104.516417] [] sys_exit_group+0x13/0x13 [ 1104.516417] [] sysenter_do_call+0x12/0x38 [ 1104.516417] Code: 89 d0 8d b6 00 00 00 00 89 45 bc 83 3d b8 1f 91 c1 00 75 02 0f 0b 8b 45 c8 8b 55 e0 89 f1 8d b6 00 00 00 00 8b 45 d0 80 48 0c 01 <85> db 0f 84 8e 01 00 00 80 7d cc 00 74 5a 8b 0f 85 c9 74 54 8b [ 1104.516417] Kernel panic - not syncing: softlockup: hung tasks [ 1104.516417] Pid: 132, comm: umount Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.516417] Call Trace: [ 1104.516417] [] panic+0x97/0x1aa [ 1104.516417] [] watchdog_timer_fn+0x115/0x134 [ 1104.516417] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.516417] [] ? __touch_watchdog+0x1d/0x1d [ 1104.516417] [] run_local_timers+0x8/0x14 [ 1104.516417] [] update_process_times+0x24/0x58 [ 1104.516417] [] tick_periodic+0x66/0x72 [ 1104.516417] [] tick_handle_periodic+0x19/0x61 [ 1104.516417] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.516417] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.516417] [] apic_timer_interrupt+0x36/0x40 [ 1104.516417] [] ? munlock_vma_pages_range+0x34/0xa1 [ 1104.516417] [] ? unmap_single_vma+0x206/0x473 [ 1104.516417] [] unmap_vmas+0x2a/0x40 [ 1104.516417] [] exit_mmap+0x76/0xe4 [ 1104.516417] [] mmput+0x42/0xa7 [ 1104.516417] [] exit_mm+0xde/0xe6 [ 1104.516417] [] do_exit+0x1d0/0x72c [ 1104.516417] [] ? vfs_write+0x105/0x10f [ 1104.516417] [] do_group_exit+0x5e/0x81 [ 1104.516417] [] sys_exit_group+0x13/0x13 [ 1104.516417] [] sysenter_do_call+0x12/0x38 [ 1104.516417] Rebooting in 10 seconds.. -- [ 5.814040] Freeing unused kernel memory: 460k freed [ 1104.524796] BUG: soft lockup - CPU#1 stuck for 1023s! [sh:130] [ 1104.524796] irq event stamp: 1712 [ 1104.524796] hardirqs last enabled at (1711): [] do_page_fault+0x98/0x36e [ 1104.524796] hardirqs last disabled at (1712): [] apic_timer_interrupt+0x2f/0x40 [ 1104.524796] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1104.524796] softirqs last disabled at (0): [< (null)>] (null) [ 1104.524796] Pid: 130, comm: sh Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.524796] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.524796] EIP is at lock_acquire+0xb4/0xc0 [ 1104.524796] EAX: 00000282 EBX: cf89d5f0 ECX: 006f4012 EDX: 006f4012 [ 1104.524796] ESI: 00000000 EDI: c19136b8 EBP: ce8bdedc ESP: ce8bdeb0 [ 1104.524796] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.524796] CR0: 8005003b CR2: b75df2c0 CR3: 0e872000 CR4: 00000690 [ 1104.524796] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.524796] DR6: ffff0ff0 DR7: 00000400 [ 1104.524796] Process sh (pid: 130, ti=ce8bc000 task=cf89d5f0 task.ti=ce8bc000) [ 1104.524796] Stack: [ 1104.524796] 00000002 00000001 00000000 00000000 c10c2db3 00000000 00000000 00000282 [ 1104.524796] ce85a780 0000000b ce85a780 ce8bdefc c10c2de0 00000002 00000001 00000000 [ 1104.524796] c10c2db3 ce8909f8 b75df2c0 ce8bdf20 c10aef10 ce8bdf20 c10432c5 00000001 [ 1104.524796] Call Trace: [ 1104.524796] [] ? mem_cgroup_count_vm_event+0x1e/0x163 [ 1104.524796] [] mem_cgroup_count_vm_event+0x4b/0x163 [ 1104.531682] [sched_delayed] sched: RT throttling activated [ 1104.524796] [] ? mem_cgroup_count_vm_event+0x1e/0x163 [ 1104.524796] [] handle_mm_fault+0x2f/0xb0 [ 1104.524796] [] ? down_read_trylock+0x51/0x59 [ 1104.524796] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.524796] [] do_page_fault+0x336/0x36e [ 1104.524796] [] ? fget_light+0xf7/0x334 [ 1104.524796] [] ? copy_to_user+0x3f/0x46 [ 1104.524796] [] ? restore_all+0xf/0xf [ 1104.524796] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.524796] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.524796] [] do_async_page_fault+0x23/0x5f [ 1104.524796] [] error_code+0x6c/0x80 [ 1104.524796] Code: 45 08 89 04 24 8b 4d ec 89 f2 89 f8 e8 57 e7 ff ff c7 83 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 83 c4 20 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 1c [ 1104.524796] Kernel panic - not syncing: softlockup: hung tasks [ 1104.524796] Pid: 130, comm: sh Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.524796] Call Trace: [ 1104.524796] [] panic+0x97/0x1aa [ 1104.524796] [] watchdog_timer_fn+0x115/0x134 [ 1104.524796] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.524796] [] ? __touch_watchdog+0x1d/0x1d [ 1104.524796] [] run_local_timers+0x8/0x14 [ 1104.524796] [] update_process_times+0x24/0x58 [ 1104.524796] [] tick_periodic+0x66/0x72 [ 1104.524796] [] tick_handle_periodic+0x19/0x61 [ 1104.524796] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.524796] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.524796] [] apic_timer_interrupt+0x36/0x40 [ 1104.524796] [] ? lock_acquire+0xb4/0xc0 [ 1104.524796] [] ? mem_cgroup_count_vm_event+0x1e/0x163 [ 1104.524796] [] mem_cgroup_count_vm_event+0x4b/0x163 [ 1104.524796] [] ? mem_cgroup_count_vm_event+0x1e/0x163 [ 1104.524796] [] handle_mm_fault+0x2f/0xb0 [ 1104.524796] [] ? down_read_trylock+0x51/0x59 [ 1104.524796] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.524796] [] do_page_fault+0x336/0x36e [ 1104.524796] [] ? fget_light+0xf7/0x334 [ 1104.524796] [] ? copy_to_user+0x3f/0x46 [ 1104.524796] [] ? restore_all+0xf/0xf [ 1104.524796] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.524796] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.524796] [] do_async_page_fault+0x23/0x5f [ 1104.524796] [] error_code+0x6c/0x80 [ 1104.524796] Rebooting in 10 seconds.. -- [ 1103.513737] [sched_delayed] sched: RT throttling activated [ 1103.513714] BUG: soft lockup - CPU#1 stuck for 1024s! [hwclock:126] [ 1103.513714] irq event stamp: 6902 [ 1103.513714] hardirqs last enabled at (6901): [] _raw_spin_unlock_irq+0x22/0x40 [ 1103.513714] hardirqs last disabled at (6902): [] apic_timer_interrupt+0x2f/0x40 [ 1103.513714] softirqs last enabled at (2618): [] unix_sock_destructor+0xae/0xb3 [ 1103.513714] softirqs last disabled at (2616): [] unix_sock_destructor+0x9b/0xb3 [ 1103.513714] Pid: 126, comm: hwclock Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.513714] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1103.513714] EIP is at _raw_spin_unlock_irq+0x2f/0x40 [ 1103.513714] EAX: cf8aa260 EBX: ce832084 ECX: 00000000 EDX: 00000007 [ 1103.513714] ESI: 00000001 EDI: 00000000 EBP: ce895f28 ESP: ce895f24 [ 1103.513714] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.513714] CR0: 8005003b CR2: b7642fb8 CR3: 019f0000 CR4: 00000690 [ 1103.513714] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.513714] DR6: ffff0ff0 DR7: 00000400 [ 1103.513714] Process hwclock (pid: 126, ti=ce894000 task=cf8aa260 task.ti=ce894000) [ 1103.513714] Stack: [ 1103.513714] 00000000 ce895f40 c1320af4 cf8aa260 cf8aa260 00000000 0000000f ce895f8c [ 1103.513714] c102ce62 b7642fb8 00000000 00000282 afb408d9 00001043 ce895f94 ce895f9c [ 1103.513714] 00000000 00000001 c102dd6d c12b3946 00000007 00000006 00000000 cf883000 [ 1103.513714] Call Trace: [ 1103.513714] [] disassociate_ctty+0xce/0x1cb [ 1103.513714] [] do_exit+0x21f/0x72c [ 1103.513714] [] ? do_sys_settimeofday+0x88/0x93 [ 1103.513714] [] ? _copy_from_user+0x37/0x4b [ 1103.513714] [] do_group_exit+0x5e/0x81 [ 1103.513714] [] sys_exit_group+0x13/0x13 [ 1103.513714] [] sysenter_do_call+0x12/0x38 [ 1103.513714] Code: 89 c3 8d 40 10 8b 4d 04 ba 01 00 00 00 e8 1e 8b 9e ff 89 d8 e8 38 40 c4 ff e8 69 6a 9e ff 83 3d 94 20 91 c1 00 75 02 0f 0b fb 90 <8d> 74 26 00 89 e0 25 00 e0 ff ff ff 48 14 5b 5d c3 55 89 e5 53 [ 1103.513714] Kernel panic - not syncing: softlockup: hung tasks [ 1103.513714] Pid: 126, comm: hwclock Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.513714] Call Trace: [ 1103.513714] [] panic+0x97/0x1aa [ 1103.513714] [] watchdog_timer_fn+0x115/0x134 [ 1103.513714] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.513714] [] ? __touch_watchdog+0x1d/0x1d [ 1103.513714] [] run_local_timers+0x8/0x14 [ 1103.513714] [] update_process_times+0x24/0x58 [ 1103.513714] [] tick_periodic+0x66/0x72 [ 1103.513714] [] tick_handle_periodic+0x19/0x61 [ 1103.513714] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.513714] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.513714] [] apic_timer_interrupt+0x36/0x40 [ 1103.513714] [] ? _raw_spin_unlock_irq+0x2f/0x40 [ 1103.513714] [] disassociate_ctty+0xce/0x1cb [ 1103.513714] [] do_exit+0x21f/0x72c [ 1103.513714] [] ? do_sys_settimeofday+0x88/0x93 [ 1103.513714] [] ? _copy_from_user+0x37/0x4b [ 1103.513714] [] do_group_exit+0x5e/0x81 [ 1103.513714] [] sys_exit_group+0x13/0x13 [ 1103.513714] [] sysenter_do_call+0x12/0x38 [ 1103.513714] Rebooting in 10 seconds.. -- [ 1103.518186] [sched_delayed] sched: RT throttling activated [ 1103.517941] BUG: soft lockup - CPU#1 stuck for 1023s! [chroot:133] [ 1103.517941] irq event stamp: 1984 [ 1103.517941] hardirqs last enabled at (1983): [] do_page_fault+0x98/0x36e [ 1103.517941] hardirqs last disabled at (1984): [] apic_timer_interrupt+0x2f/0x40 [ 1103.517941] softirqs last enabled at (1934): [] unix_sock_destructor+0xae/0xb3 [ 1103.517941] softirqs last disabled at (1932): [] unix_sock_destructor+0x9b/0xb3 [ 1103.517941] Pid: 133, comm: chroot Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.517941] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.517941] EIP is at lock_is_held+0x9f/0xb2 [ 1103.517941] EAX: 00000282 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 1103.517941] ESI: ce88e720 EDI: c19136b8 EBP: ce861ef8 ESP: ce861ee8 [ 1103.517941] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.517941] CR0: 80050033 CR2: b763c720 CR3: 0e872000 CR4: 00000690 [ 1103.517941] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.517941] DR6: ffff0ff0 DR7: 00000400 [ 1103.517941] Process chroot (pid: 133, ti=ce860000 task=ce88e720 task.ti=ce860000) [ 1103.517941] Stack: [ 1103.517941] 00000282 0000046c c181eefe 00000000 ce861f20 c104ba8b c101d888 ce861f20 [ 1103.517941] c10432c5 00000001 00000002 ce861fb4 ce861fb4 c101d888 ce861f9c c101e6cd [ 1103.517941] 00000028 ce861f38 ce88e720 ce83ca54 ce83ca00 ce88e720 00000028 00000004 [ 1103.517941] Call Trace: [ 1103.517941] [] __might_sleep+0x2b/0x1cd [ 1103.517941] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517941] [] ? down_read_trylock+0x51/0x59 [ 1103.517941] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517941] [] do_page_fault+0x193/0x36e [ 1103.517941] [] ? _raw_spin_unlock_irq+0x22/0x40 [ 1103.517941] [] ? restore_all+0xf/0xf [ 1103.517941] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517941] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517941] [] do_async_page_fault+0x23/0x5f [ 1103.517941] [] error_code+0x6c/0x80 [ 1103.517941] Code: 7c e2 31 c9 eb 05 b9 01 00 00 00 64 a1 28 06 9e c1 c7 80 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 eb 05 b9 01 00 00 00 89 c8 5b 5b 5e 5f 5d c3 55 89 [ 1103.517941] Kernel panic - not syncing: softlockup: hung tasks [ 1103.517941] Pid: 133, comm: chroot Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.517941] Call Trace: [ 1103.517941] [] panic+0x97/0x1aa [ 1103.517941] [] watchdog_timer_fn+0x115/0x134 [ 1103.517941] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.517941] [] ? __touch_watchdog+0x1d/0x1d [ 1103.517941] [] run_local_timers+0x8/0x14 [ 1103.517941] [] update_process_times+0x24/0x58 [ 1103.517941] [] tick_periodic+0x66/0x72 [ 1103.517941] [] tick_handle_periodic+0x19/0x61 [ 1103.517941] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.517941] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.517941] [] apic_timer_interrupt+0x36/0x40 [ 1103.517941] [] ? lock_is_held+0x9f/0xb2 [ 1103.517941] [] __might_sleep+0x2b/0x1cd [ 1103.517941] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517941] [] ? down_read_trylock+0x51/0x59 [ 1103.517941] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517941] [] do_page_fault+0x193/0x36e [ 1103.517941] [] ? _raw_spin_unlock_irq+0x22/0x40 [ 1103.517941] [] ? restore_all+0xf/0xf [ 1103.517941] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517941] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517941] [] do_async_page_fault+0x23/0x5f [ 1103.517941] [] error_code+0x6c/0x80 [ 1103.517941] Rebooting in 10 seconds.. -- [ 5.212332] Freeing unused kernel memory: 460k freed [ 1104.518431] BUG: soft lockup - CPU#1 stuck for 1024s! [swapper/1:0] [ 1104.518431] irq event stamp: 5734 [ 1104.518431] hardirqs last enabled at (5731): [] restore_all_notrace+0x0/0x18 [ 1104.518431] hardirqs last disabled at (5732): [] apic_timer_interrupt+0x2f/0x40 [ 1104.518431] softirqs last enabled at (5734): [] _local_bh_enable+0xd/0xf [ 1104.518431] softirqs last disabled at (5733): [] irq_enter+0x2c/0x4d [ 1104.518431] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.518431] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.518431] EIP is at native_safe_halt+0x5/0x7 [ 1104.518431] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1104.518431] ESI: c19716f0 EDI: ffff8b9f EBP: ce071f68 ESP: ce071f68 [ 1104.518431] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.518431] CR0: 8005003b CR2: b768ffb8 CR3: 0f8a4000 CR4: 00000690 [ 1104.518431] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.518431] DR6: ffff0ff0 DR7: 00000400 [ 1104.518431] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) [ 1104.518431] Stack: [ 1104.518431] ce071f70 c1008a09 ce071f88 c1009250 fcc4d513 0b285a35 45735797 cfff9fa0 [ 1104.518431] ce071fb4 c166ed10 00000000 00000000 00000000 00000000 b5a26032 45735797 [ 1104.518431] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.518431] Call Trace: [ 1104.518431] [] default_idle+0x2f/0x55 [ 1104.518431] [] cpu_idle+0xc9/0xfb [ 1104.518431] [] start_secondary+0x292/0x29a [ 1104.518431] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1104.518431] Kernel panic - not syncing: softlockup: hung tasks [ 1104.518431] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.518431] Call Trace: [ 1104.518431] [] panic+0x97/0x1aa [ 1104.518431] [] watchdog_timer_fn+0x115/0x134 [ 1104.518431] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.518431] [] ? __touch_watchdog+0x1d/0x1d [ 1104.518431] [] run_local_timers+0x8/0x14 [ 1104.518431] [] update_process_times+0x24/0x58 [ 1104.518431] [] tick_periodic+0x66/0x72 [ 1104.518431] [] tick_handle_periodic+0x19/0x61 [ 1104.518431] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.518431] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.518431] [] apic_timer_interrupt+0x36/0x40 [ 1104.518431] [] ? native_safe_halt+0x5/0x7 [ 1104.518431] [] default_idle+0x2f/0x55 [ 1104.518431] [] cpu_idle+0xc9/0xfb [ 1104.518431] [] start_secondary+0x292/0x29a [ 1104.518431] Rebooting in 10 seconds.. -- [ 1104.513600] [sched_delayed] sched: RT throttling activated [ 1104.513573] BUG: soft lockup - CPU#1 stuck for 1025s! [run-parts:123] [ 1104.513573] irq event stamp: 1114 [ 1104.513573] hardirqs last enabled at (1113): [] restore_all_notrace+0x0/0x18 [ 1104.513573] hardirqs last disabled at (1114): [] apic_timer_interrupt+0x2f/0x40 [ 1104.513573] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1104.513573] softirqs last disabled at (0): [< (null)>] (null) [ 1104.513573] Pid: 123, comm: run-parts Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.513573] EIP: 0060:[] EFLAGS: 00000202 CPU: 1 -- [ 5.941592] Freeing unused kernel memory: 460k freed [ 1104.521335] BUG: soft lockup - CPU#1 stuck for 1023s! [umount:132] [ 1104.521335] irq event stamp: 2688 [ 1104.521335] hardirqs last enabled at (2687): [] sysenter_exit+0xf/0x1e [ 1104.521335] hardirqs last disabled at (2688): [] apic_timer_interrupt+0x2f/0x40 [ 1104.521335] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1104.521335] softirqs last disabled at (0): [< (null)>] (null) [ 1104.521335] Pid: 132, comm: umount Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.521335] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.521335] EIP is at lock_acquire+0xb4/0xc0 [ 1104.521335] EAX: 00000282 EBX: cf899130 ECX: 000001e4 EDX: 000001e4 [ 1104.521335] ESI: 00000000 EDI: cb48f9bc EBP: ce8bfeec ESP: ce8bfec0 [ 1104.521335] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.521335] CR0: 80050033 CR2: b773f8d0 CR3: 0e8a1000 CR4: 00000690 [ 1104.521335] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.521335] DR6: ffff0ff0 DR7: 00000400 [ 1104.521335] Process umount (pid: 132, ti=ce8be000 task=cf899130 task.ti=ce8be000) [ 1104.521335] Stack: [ 1104.521335] 00000000 00000002 00000000 00000000 c10d99bd 00000000 00000000 00000282 [ 1104.521335] cb48f9ac ffffff9c cb48f9ac ce8bff08 c167e6a2 00000000 00000002 00000000 [ 1104.521335] c10d99bd cb48f960 ce8bff20 c10d99bd ffffff9c ce8bff38 ffffff9c 00000000 [ 1104.521335] Call Trace: [ 1104.521335] [] ? dput+0x37/0x143 [ 1104.521335] [] _raw_spin_lock+0x3d/0x4a [ 1104.521335] [] ? dput+0x37/0x143 [ 1104.521335] [] dput+0x37/0x143 [ 1104.521335] [] path_put+0xe/0x18 [ 1104.521335] [] vfs_fstatat+0x56/0x60 [ 1104.521335] [] vfs_stat+0x19/0x1b [ 1104.521335] [] sys_stat64+0x11/0x22 [ 1104.521335] [] sysenter_do_call+0x12/0x38 [ 1104.521335] Code: 45 08 89 04 24 8b 4d ec 89 f2 89 f8 e8 57 e7 ff ff c7 83 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 83 c4 20 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 1c [ 1104.521335] Kernel panic - not syncing: softlockup: hung tasks [ 1104.521335] Pid: 132, comm: umount Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.521335] Call Trace: [ 1104.521335] [] panic+0x97/0x1aa [ 1104.521335] [] watchdog_timer_fn+0x115/0x134 [ 1104.521335] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.521335] [] ? __touch_watchdog+0x1d/0x1d [ 1104.521335] [] run_local_timers+0x8/0x14 [ 1104.521335] [] update_process_times+0x24/0x58 [ 1104.521335] [] tick_periodic+0x66/0x72 [ 1104.521335] [] tick_handle_periodic+0x19/0x61 [ 1104.521335] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.521335] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.521335] [] apic_timer_interrupt+0x36/0x40 [ 1104.521335] [] ? pipe_fcntl+0xbc/0x140 [ 1104.521335] [] ? lock_acquire+0xb4/0xc0 [ 1104.521335] [] ? dput+0x37/0x143 [ 1104.521335] [] _raw_spin_lock+0x3d/0x4a [ 1104.521335] [] ? dput+0x37/0x143 [ 1104.521335] [] dput+0x37/0x143 [ 1104.521335] [] path_put+0xe/0x18 [ 1104.521335] [] vfs_fstatat+0x56/0x60 [ 1104.521335] [] vfs_stat+0x19/0x1b [ 1104.521335] [] sys_stat64+0x11/0x22 [ 1104.521335] [] sysenter_do_call+0x12/0x38 [ 1104.521335] Rebooting in 10 seconds.. -- [ 5.028838] Freeing unused kernel memory: 460k freed [ 1104.517771] BUG: soft lockup - CPU#1 stuck for 1025s! [mountall:135] [ 1104.517771] irq event stamp: 9160 [ 1104.517771] hardirqs last enabled at (9159): [] restore_all_notrace+0x0/0x18 [ 1104.517771] hardirqs last disabled at (9160): [] apic_timer_interrupt+0x2f/0x40 [ 1104.517771] softirqs last enabled at (3012): [] unix_sock_destructor+0xae/0xb3 [ 1104.517771] softirqs last disabled at (3010): [] unix_sock_destructor+0x9b/0xb3 [ 1104.517771] Pid: 135, comm: mountall Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.517771] EIP: 0073:[] EFLAGS: 00000202 CPU: 1 [ 1104.517771] EIP is at 0xb7722c34 [ 1104.517771] EAX: ff8760ae EBX: b7739ff4 ECX: 000000ae EDX: 00020415 [ 1104.517771] ESI: b76f1590 EDI: 00000005 EBP: 007fc3b0 ESP: bfc17470 [ 1104.517771] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b [ 1104.517771] Process mountall (pid: 135, ti=ce8c6000 task=ce842260 task.ti=ce8c6000) [ 1104.517771] [ 1104.517771] Kernel panic - not syncing: softlockup: hung tasks [ 1104.517771] Pid: 135, comm: mountall Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.517771] Call Trace: [ 1104.526641] [sched_delayed] sched: RT throttling activated [ 1104.517771] [] panic+0x97/0x1aa [ 1104.517771] [] watchdog_timer_fn+0x115/0x134 [ 1104.517771] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.517771] [] ? __touch_watchdog+0x1d/0x1d [ 1104.517771] [] run_local_timers+0x8/0x14 [ 1104.517771] [] update_process_times+0x24/0x58 [ 1104.517771] [] tick_periodic+0x66/0x72 [ 1104.517771] [] tick_handle_periodic+0x19/0x61 [ 1104.517771] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.517771] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.517771] [] apic_timer_interrupt+0x36/0x40 [ 1104.517771] Rebooting in 10 seconds.. -- [ 5.786476] Freeing unused kernel memory: 460k freed [ 1104.518564] BUG: soft lockup - CPU#1 stuck for 1023s! [grep:129] [ 1104.518564] irq event stamp: 2294 [ 1104.518564] hardirqs last enabled at (2293): [] _raw_spin_unlock_irqrestore+0x42/0x63 [ 1104.518564] hardirqs last disabled at (2294): [] apic_timer_interrupt+0x2f/0x40 [ 1104.518564] softirqs last enabled at (240): [] __do_softirq+0x131/0x153 [ 1104.523873] [sched_delayed] sched: RT throttling activated [ 1104.518564] softirqs last disabled at (205): [] do_softirq+0x74/0xdf [ 1104.518564] Pid: 129, comm: grep Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.518564] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.518564] EIP is at _raw_spin_unlock_irqrestore+0x51/0x63 [ 1104.518564] EAX: 00000282 EBX: 00000282 ECX: 00000000 EDX: 00000007 [ 1104.518564] ESI: c196c730 EDI: 00000002 EBP: ce89fe2c ESP: ce89fe24 [ 1104.518564] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.518564] CR0: 8005003b CR2: b7627fb8 CR3: 019f0000 CR4: 00000690 [ 1104.518564] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.518564] DR6: ffff0ff0 DR7: 00000400 [ 1104.518564] Process grep (pid: 129, ti=ce89e000 task=ce849130 task.ti=ce89e000) [ 1104.518564] Stack: [ 1104.518564] ce538018 c196c490 ce89fe54 c10a0736 00000000 c109fe68 c196c490 ce68f3c0 [ 1104.518564] 00000282 00000001 ce538098 00000005 ce89fe68 c10a0a6d ce89fec0 ce89fee4 [ 1104.518564] ce89fed8 ce89fe70 c10a0a8a ce89fe8c c10b9614 bff4f000 00000008 ce89fec0 [ 1104.518564] Call Trace: [ 1104.518564] [] pagevec_lru_move_fn+0x8d/0xa8 [ 1104.518564] [] ? pagevec_move_tail_fn+0x54/0x54 [ 1104.518564] [] lru_add_drain_cpu+0xe0/0xe5 [ 1104.518564] [] lru_add_drain+0x18/0x24 [ 1104.518564] [] free_pages_and_swap_cache+0x13/0x7b [ 1104.518564] [] tlb_flush_mmu+0x56/0x6c [ 1104.518564] [] tlb_finish_mmu+0x12/0x31 [ 1104.518564] [] exit_mmap+0x96/0xe4 [ 1104.518564] [] mmput+0x42/0xa7 [ 1104.518564] [] exit_mm+0xde/0xe6 [ 1104.518564] [] do_exit+0x1d0/0x72c [ 1104.518564] [] do_group_exit+0x5e/0x81 [ 1104.518564] [] sys_exit_group+0x13/0x13 [ 1104.518564] [] sysenter_do_call+0x12/0x38 [ 1104.518564] Code: 3d 8c 20 91 c1 00 74 1d 89 d8 50 9d 8d 74 26 00 e8 dd 6b 9e ff eb 18 e8 ac 6a 9e ff 83 3d 8c 20 91 c1 00 75 02 0f 0b 89 d8 50 9d <8d> 74 26 00 89 e0 25 00 e0 ff ff ff 48 14 5b 5e 5d c3 55 89 e5 [ 1104.518564] Kernel panic - not syncing: softlockup: hung tasks [ 1104.518564] Pid: 129, comm: grep Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.518564] Call Trace: [ 1104.518564] [] panic+0x97/0x1aa [ 1104.518564] [] watchdog_timer_fn+0x115/0x134 [ 1104.518564] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.518564] [] ? __touch_watchdog+0x1d/0x1d [ 1104.518564] [] run_local_timers+0x8/0x14 [ 1104.518564] [] update_process_times+0x24/0x58 [ 1104.518564] [] tick_periodic+0x66/0x72 [ 1104.518564] [] tick_handle_periodic+0x19/0x61 [ 1104.518564] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.518564] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.518564] [] apic_timer_interrupt+0x36/0x40 [ 1104.518564] [] ? _raw_spin_unlock_irqrestore+0x51/0x63 [ 1104.518564] [] pagevec_lru_move_fn+0x8d/0xa8 [ 1104.518564] [] ? pagevec_move_tail_fn+0x54/0x54 [ 1104.518564] [] lru_add_drain_cpu+0xe0/0xe5 [ 1104.518564] [] lru_add_drain+0x18/0x24 [ 1104.518564] [] free_pages_and_swap_cache+0x13/0x7b [ 1104.518564] [] tlb_flush_mmu+0x56/0x6c [ 1104.518564] [] tlb_finish_mmu+0x12/0x31 [ 1104.518564] [] exit_mmap+0x96/0xe4 [ 1104.518564] [] mmput+0x42/0xa7 [ 1104.518564] [] exit_mm+0xde/0xe6 [ 1104.518564] [] do_exit+0x1d0/0x72c [ 1104.518564] [] do_group_exit+0x5e/0x81 [ 1104.518564] [] sys_exit_group+0x13/0x13 [ 1104.518564] [] sysenter_do_call+0x12/0x38 [ 1104.518564] Rebooting in 10 seconds.. -- [ 5.159006] Freeing unused kernel memory: 460k freed [ 1104.513734] BUG: soft lockup - CPU#1 stuck for 1025s! [swapper/1:0] [ 1104.522610] [sched_delayed] sched: RT throttling activated [ 1104.513734] irq event stamp: 12942 [ 1104.513734] hardirqs last enabled at (12939): [] restore_all_notrace+0x0/0x18 [ 1104.513734] hardirqs last disabled at (12940): [] apic_timer_interrupt+0x2f/0x40 [ 1104.513734] softirqs last enabled at (12942): [] _local_bh_enable+0xd/0xf [ 1104.513734] softirqs last disabled at (12941): [] irq_enter+0x2c/0x4d [ 1104.513734] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.513734] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.513734] EIP is at native_safe_halt+0x5/0x7 [ 1104.513734] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1104.513734] ESI: c19716f0 EDI: ffff8b9f EBP: ce071f68 ESP: ce071f68 [ 1104.513734] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.513734] CR0: 8005003b CR2: b7710000 CR3: 0ee89000 CR4: 00000690 [ 1104.513734] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.513734] DR6: ffff0ff0 DR7: 00000400 [ 1104.513734] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) [ 1104.513734] Stack: [ 1104.513734] ce071f70 c1008a09 ce071f88 c1009250 302764a8 3e8ae9cb 78d59122 cfff9fa0 [ 1104.552695] ce071fb4 c166ed10 00000000 00000000 00000000 00000000 e90499bd 78d59122 [ 1104.552695] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.552695] Call Trace: [ 1104.552695] [] default_idle+0x2f/0x55 [ 1104.552695] [] cpu_idle+0xc9/0xfb [ 1104.552695] [] start_secondary+0x292/0x29a [ 1104.552695] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1104.552695] Kernel panic - not syncing: softlockup: hung tasks [ 1104.552695] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.552695] Call Trace: [ 1104.552695] [] panic+0x97/0x1aa [ 1104.552695] [] watchdog_timer_fn+0x115/0x134 [ 1104.552695] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.552695] [] ? __touch_watchdog+0x1d/0x1d [ 1104.552695] [] run_local_timers+0x8/0x14 [ 1104.552695] [] update_process_times+0x24/0x58 [ 1104.552695] [] tick_periodic+0x66/0x72 [ 1104.552695] [] tick_handle_periodic+0x19/0x61 [ 1104.552695] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.552695] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.552695] [] apic_timer_interrupt+0x36/0x40 [ 1104.552695] [] ? native_safe_halt+0x5/0x7 [ 1104.552695] [] default_idle+0x2f/0x55 [ 1104.552695] [] cpu_idle+0xc9/0xfb [ 1104.552695] [] start_secondary+0x292/0x29a [ 1104.552695] Rebooting in 10 seconds.. -- [ 5.258753] Freeing unused kernel memory: 460k freed [ 1104.517703] BUG: soft lockup - CPU#1 stuck for 1025s! [plymouthd:125] [ 1104.517703] irq event stamp: 6312 [ 1104.517703] hardirqs last enabled at (6311): [] restore_all_notrace+0x0/0x18 [ 1104.517703] hardirqs last disabled at (6312): [] apic_timer_interrupt+0x2f/0x40 [ 1104.517703] softirqs last enabled at (4300): [] __do_softirq+0x131/0x153 [ 1104.517703] softirqs last disabled at (4287): [] do_softirq+0x74/0xdf [ 1104.517703] Pid: 125, comm: plymouthd Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.517703] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.517703] EIP is at unmap_single_vma+0xf/0x473 [ 1104.517703] EAX: ce86fe60 EBX: ce846318 ECX: b76db000 EDX: ce846318 [ 1104.526960] [sched_delayed] sched: RT throttling activated [ 1104.517703] ESI: ce86fe60 EDI: b76db000 EBP: ce86fe34 ESP: ce86fdd0 [ 1104.517703] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.517703] CR0: 8005003b CR2: b76daedc CR3: 0e86a000 CR4: 00000690 [ 1104.517703] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.517703] DR6: ffff0ff0 DR7: 00000400 [ 1104.517703] Process plymouthd (pid: 125, ti=ce86e000 task=cf86e720 task.ti=ce86e000) [ 1104.517703] Stack: [ 1104.517703] ce86fe04 c10a06a1 00000000 00000001 ce538120 00000282 c196c730 00000001 [ 1104.517703] ce86fdf0 ce86fdf0 ce538118 c196c490 00000001 ce86fe60 c10a0743 00000001 [ 1104.517703] ce846318 c196c490 ce73e940 00000282 00000001 ce538118 ce846318 ce86fe60 [ 1104.517703] Call Trace: [ 1104.517703] [] ? release_pages+0x171/0x179 [ 1104.517703] [] ? pagevec_lru_move_fn+0x9a/0xa8 [ 1104.517703] [] unmap_vmas+0x2a/0x40 [ 1104.517703] [] unmap_region+0x5c/0x92 [ 1104.517703] [] do_munmap+0x1d3/0x247 [ 1104.517703] [] mmap_region+0x68/0x38a [ 1104.517703] [] do_mmap_pgoff+0x1ef/0x22d [ 1104.517703] [] vm_mmap_pgoff+0x47/0x59 [ 1104.517703] [] sys_mmap_pgoff+0x80/0xa2 [ 1104.517703] [] syscall_call+0x7/0xb [ 1104.517703] Code: 00 00 00 85 db 75 04 eb 09 89 c6 89 d8 e8 34 31 ff ff 89 f0 83 c4 2c 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 58 89 45 d0 89 55 dc <8b> 42 04 89 ce 39 c1 73 02 89 c6 8b 4d dc 8b 51 08 39 d6 0f 83 [ 1104.517703] Kernel panic - not syncing: softlockup: hung tasks [ 1104.517703] Pid: 125, comm: plymouthd Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.517703] Call Trace: [ 1104.517703] [] panic+0x97/0x1aa [ 1104.517703] [] watchdog_timer_fn+0x115/0x134 [ 1104.517703] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.517703] [] ? __touch_watchdog+0x1d/0x1d [ 1104.517703] [] run_local_timers+0x8/0x14 [ 1104.517703] [] update_process_times+0x24/0x58 [ 1104.517703] [] tick_periodic+0x66/0x72 [ 1104.517703] [] tick_handle_periodic+0x19/0x61 [ 1104.517703] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.517703] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.517703] [] apic_timer_interrupt+0x36/0x40 [ 1104.517703] [] ? timecounter_init+0x13/0x21 [ 1104.517703] [] ? unmap_single_vma+0xf/0x473 [ 1104.517703] [] ? release_pages+0x171/0x179 [ 1104.517703] [] ? pagevec_lru_move_fn+0x9a/0xa8 [ 1104.517703] [] unmap_vmas+0x2a/0x40 [ 1104.517703] [] unmap_region+0x5c/0x92 [ 1104.517703] [] do_munmap+0x1d3/0x247 [ 1104.517703] [] mmap_region+0x68/0x38a [ 1104.517703] [] do_mmap_pgoff+0x1ef/0x22d [ 1104.517703] [] vm_mmap_pgoff+0x47/0x59 [ 1104.517703] [] sys_mmap_pgoff+0x80/0xa2 [ 1104.517703] [] syscall_call+0x7/0xb [ 1104.517703] Rebooting in 10 seconds.. -- [ 5.088389] Freeing unused kernel memory: 460k freed [ 1104.515816] BUG: soft lockup - CPU#1 stuck for 1025s! [99-trinity:126] -- [ 4.371778] Freeing unused kernel memory: 460k freed [ 1103.516949] BUG: soft lockup - CPU#1 stuck for 1024s! [99-trinity:135] [ 1103.516949] irq event stamp: 214 [ 1103.516949] hardirqs last enabled at (213): [] do_page_fault+0x98/0x36e [ 1103.516949] hardirqs last disabled at (214): [] apic_timer_interrupt+0x2f/0x40 [ 1103.516949] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1103.516949] softirqs last disabled at (0): [< (null)>] (null) [ 1103.516949] Pid: 135, comm: 99-trinity Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.516949] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.516949] EIP is at lock_is_held+0x9f/0xb2 [ 1103.516949] EAX: 00000282 EBX: 00000001 ECX: 00000001 EDX: 00000003 [ 1103.516949] ESI: ce8b3390 EDI: c19136b8 EBP: ce8cdedc ESP: ce8cdecc [ 1103.516949] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.526882] [sched_delayed] sched: RT throttling activated [ 1103.516949] CR0: 80050033 CR2: 080aaed0 CR3: 0e898000 CR4: 00000690 [ 1103.516949] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.516949] DR6: ffff0ff0 DR7: 00000400 mountall: ply-ev[ 1103.516949] Process 99-trinity (pid: 135, ti=ce8cc000 task=ce8b3390 task.ti=ce8cc000) ent-loop.c:497: [ 1103.516949] Stack: ply_event_loop_n[ 1103.516949] 00000282ew: Assertion `l ce8b3390oop->epoll_fd >= 0000000b 0' failed. ce130c80 ce8cdefc c10c2e53 00000002 00000001 [ 1103.516949] 00000000 c10c2db3 ce89a6e0 080aaed0 ce8cdf20 c10aef10 ce8cdf20 c10432c5 [ 1103.516949] 00000001 ce89a6e0 ce89a6e0 ce8cdfb4 c101d888 ce8cdf9c c101e870 00000028 -- [ 4.924030] Freeing unused kernel memory: 460k freed [ 1103.513755] BUG: soft lockup - CPU#1 stuck for 1024s! [init:1] [ 1103.513755] irq event stamp: 744676 [ 1103.513755] hardirqs last enabled at (744675): [] _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.513755] hardirqs last disabled at (744676): [] apic_timer_interrupt+0x2f/0x40 [ 1103.513755] softirqs last enabled at (741152): [] __do_softirq+0x131/0x153 [ 1103.513755] softirqs last disabled at (741139): [] do_softirq+0x74/0xdf [ 1103.513755] Pid: 1, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.513755] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.513755] EIP is at _raw_spin_unlock_irqrestore+0x51/0x63 [ 1103.513755] EAX: 00000282 EBX: 00000282 ECX: 00000000 EDX: 00000007 [ 1103.513755] ESI: cf89b778 EDI: 00000001 EBP: ce043dfc ESP: ce043df4 [ 1103.513755] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.513755] CR0: 80050033 CR2: b76aa920 CR3: 0ee86000 CR4: 00000690 [ 1103.523273] [sched_delayed] sched: RT throttling activated [ 1103.513755] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.513755] DR6: ffff0ff0 DR7: 00000400 [ 1103.513755] Process init (pid: 1, ti=ce042000 task=ce048000 task.ti=ce042000) [ 1103.513755] Stack: [ 1103.513755] 00000000 cf89b390 ce043e20 c104a23a 00000282 cf89b778 0cb4c000 00000000 [ 1103.513755] cf89b390 00000000 00000007 ce043e28 c104a280 ce043e38 c10306b6 cf89b390 [ 1103.513755] 00000000 ce043ef4 c1031596 003f8011 00000000 ce048474 00000000 ce048000 [ 1103.513755] Call Trace: [ 1103.513755] [] try_to_wake_up+0x248/0x266 [ 1103.513755] [] wake_up_state+0xa/0xc [ 1103.513755] [] ptrace_resume+0x76/0x7c [ 1103.513755] [] ptrace_request+0x368/0x4f8 [ 1103.513755] [] ? __lock_acquire+0x1444/0x1455 [ 1103.513755] [] ? __lock_acquire+0x1444/0x1455 [ 1103.513755] [] ? lock_release_non_nested+0x7e/0x21b [ 1103.513755] [] ? __lock_acquire+0x1444/0x1455 [ 1103.513755] [] arch_ptrace+0x474/0x48a [ 1103.513755] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.513755] [] ? trace_hardirqs_on+0xb/0xd [ 1103.513755] [] ? wait_task_inactive+0x6f/0xc1 [ 1103.513755] [] ? ptrace_check_attach+0xb4/0xc4 [ 1103.513755] [] sys_ptrace+0x34f/0x379 [ 1103.513755] [] ? sys_ptrace+0x6e/0x379 [ 1103.513755] [] sysenter_do_call+0x12/0x38 [ 1103.513755] Code: 3d 8c 20 91 c1 00 74 1d 89 d8 50 9d 8d 74 26 00 e8 dd 6b 9e ff eb 18 e8 ac 6a 9e ff 83 3d 8c 20 91 c1 00 75 02 0f 0b 89 d8 50 9d <8d> 74 26 00 89 e0 25 00 e0 ff ff ff 48 14 5b 5e 5d c3 55 89 e5 [ 1103.513755] Kernel panic - not syncing: softlockup: hung tasks [ 1103.513755] Pid: 1, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.513755] Call Trace: [ 1103.513755] [] panic+0x97/0x1aa [ 1103.513755] [] watchdog_timer_fn+0x115/0x134 [ 1103.513755] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.513755] [] ? __touch_watchdog+0x1d/0x1d [ 1103.513755] [] run_local_timers+0x8/0x14 [ 1103.513755] [] update_process_times+0x24/0x58 [ 1103.513755] [] tick_periodic+0x66/0x72 [ 1103.513755] [] tick_handle_periodic+0x19/0x61 [ 1103.513755] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.513755] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.513755] [] apic_timer_interrupt+0x36/0x40 [ 1103.513755] [] ? _raw_spin_unlock_irqrestore+0x51/0x63 [ 1103.513755] [] try_to_wake_up+0x248/0x266 [ 1103.513755] [] wake_up_state+0xa/0xc [ 1103.513755] [] ptrace_resume+0x76/0x7c [ 1103.513755] [] ptrace_request+0x368/0x4f8 [ 1103.513755] [] ? __lock_acquire+0x1444/0x1455 [ 1103.513755] [] ? __lock_acquire+0x1444/0x1455 [ 1103.513755] [] ? lock_release_non_nested+0x7e/0x21b [ 1103.513755] [] ? __lock_acquire+0x1444/0x1455 [ 1103.513755] [] arch_ptrace+0x474/0x48a [ 1103.513755] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.513755] [] ? trace_hardirqs_on+0xb/0xd [ 1103.513755] [] ? wait_task_inactive+0x6f/0xc1 [ 1103.513755] [] ? ptrace_check_attach+0xb4/0xc4 [ 1103.513755] [] sys_ptrace+0x34f/0x379 [ 1103.513755] [] ? sys_ptrace+0x6e/0x379 [ 1103.513755] [] sysenter_do_call+0x12/0x38 [ 1103.513755] Rebooting in 10 seconds.. -- [ 1104.514595] [sched_delayed] sched: RT throttling activated [ 1104.514574] BUG: soft lockup - CPU#1 stuck for 1023s! [hwclock:126] [ 1104.514574] irq event stamp: 6886 [ 1104.514574] hardirqs last enabled at (6885): [] get_page_from_freelist+0x2ca/0x456 [ 1104.514574] hardirqs last disabled at (6886): [] apic_timer_interrupt+0x2f/0x40 [ 1104.514574] softirqs last enabled at (2696): [] unix_sock_destructor+0xae/0xb3 [ 1104.514574] softirqs last disabled at (2694): [] unix_sock_destructor+0x9b/0xb3 [ 1104.514574] Pid: 126, comm: hwclock Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.514574] EIP: 0060:[] EFLAGS: 00000246 CPU: 1 [ 1104.514574] EIP is at unmap_single_vma+0x180/0x473 [ 1104.514574] EAX: ce73eb80 EBX: ce73eb80 ECX: 0ffdc067 EDX: bf8e4000 [ 1104.514574] ESI: ce870390 EDI: 00000000 EBP: ce87de98 ESP: ce87de34 [ 1104.514574] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.514574] CR0: 8005003b CR2: b7666fb8 CR3: 0e877000 CR4: 00000690 [ 1104.514574] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.514574] DR6: ffff0ff0 DR7: 00000400 [ 1104.514574] Process hwclock (pid: 126, ti=ce87c000 task=cf8844c0 task.ti=ce87c000) [ 1104.514574] Stack: [ 1104.514574] 00000002 00000203 006f8203 00000000 cf884934 bf8e4fff ce852a34 ce87030c [ 1104.514574] 0ffc9067 ce877bf8 ce877bf8 ce852a00 00000000 ce87dec0 bf8e5000 bf8e5000 [ 1104.514574] ce839318 bf8e4000 ce87de80 00000000 ffffffff 00000000 ce839318 ce87dec0 [ 1104.514574] Call Trace: [ 1104.514574] [] unmap_vmas+0x2a/0x40 [ 1104.514574] [] exit_mmap+0x76/0xe4 [ 1104.514574] [] mmput+0x42/0xa7 [ 1104.514574] [] exit_mm+0xde/0xe6 [ 1104.514574] [] do_exit+0x1d0/0x72c [ 1104.514574] [] ? do_sys_settimeofday+0x88/0x93 [ 1104.514574] [] ? _copy_from_user+0x37/0x4b [ 1104.514574] [] do_group_exit+0x5e/0x81 [ 1104.514574] [] sys_exit_group+0x13/0x13 [ 1104.514574] [] sysenter_do_call+0x12/0x38 [ 1104.514574] Code: 7d 0c 8b 0e 85 c9 0f 84 3b 02 00 00 f7 c1 01 01 00 00 0f 84 73 01 00 00 8b 55 e0 8b 45 dc e8 d2 f7 ff ff 89 c3 85 ff 0f 95 45 cc <85> c0 74 30 80 7d cc 00 74 2a 8b 47 04 85 c0 74 09 3b 43 04 0f [ 1104.514574] Kernel panic - not syncing: softlockup: hung tasks [ 1104.514574] Pid: 126, comm: hwclock Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.514574] Call Trace: [ 1104.514574] [] panic+0x97/0x1aa [ 1104.514574] [] watchdog_timer_fn+0x115/0x134 [ 1104.514574] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.514574] [] ? __touch_watchdog+0x1d/0x1d [ 1104.514574] [] run_local_timers+0x8/0x14 [ 1104.514574] [] update_process_times+0x24/0x58 [ 1104.514574] [] tick_periodic+0x66/0x72 [ 1104.514574] [] tick_handle_periodic+0x19/0x61 [ 1104.514574] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.514574] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.514574] [] apic_timer_interrupt+0x36/0x40 [ 1104.514574] [] ? kvm_guest_cpu_init+0x10b/0x214 [ 1104.514574] [] ? unmap_single_vma+0x180/0x473 [ 1104.514574] [] unmap_vmas+0x2a/0x40 [ 1104.514574] [] exit_mmap+0x76/0xe4 [ 1104.514574] [] mmput+0x42/0xa7 [ 1104.514574] [] exit_mm+0xde/0xe6 [ 1104.514574] [] do_exit+0x1d0/0x72c [ 1104.514574] [] ? do_sys_settimeofday+0x88/0x93 [ 1104.514574] [] ? _copy_from_user+0x37/0x4b [ 1104.514574] [] do_group_exit+0x5e/0x81 [ 1104.514574] [] sys_exit_group+0x13/0x13 [ 1104.514574] [] sysenter_do_call+0x12/0x38 [ 1104.514574] Rebooting in 10 seconds.. -- [ 4.775380] Freeing unused kernel memory: 460k freed [ 1103.517476] BUG: soft lockup - CPU#1 stuck for 1024s! [sleep:135] [ 1103.517476] irq event stamp: 1030 [ 1103.517476] hardirqs last enabled at (1029): [] do_page_fault+0x98/0x36e [ 1103.517476] hardirqs last disabled at (1030): [] apic_timer_interrupt+0x2f/0x40 [ 1103.521211] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1103.521211] softirqs last disabled at (0): [< (null)>] (null) [ 1103.521211] Pid: 135, comm: sleep Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.521211] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.526940] [sched_delayed] sched: RT throttling activated [ 1103.521211] EIP is at lock_acquire+0xb4/0xc0 [ 1103.521211] EAX: 00000282 EBX: ce89e720 ECX: 006fc012 EDX: 006fc012 [ 1103.521211] ESI: 00000000 EDI: c19136b8 EBP: ce8cfedc ESP: ce8cfeb0 [ 1103.521211] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.521211] CR0: 8005003b CR2: b77222c0 CR3: 0e8c4000 CR4: 00000690 [ 1103.521211] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.521211] DR6: ffff0ff0 DR7: 00000400 [ 1103.521211] Process sleep (pid: 135, ti=ce8ce000 task=ce89e720 task.ti=ce8ce000) [ 1103.521211] Stack: [ 1103.521211] 00000002 00000001 00000000 00000000 c10c2db3 00000000 00000000 00000282 [ 1103.521211] ce84aa00 0000000b ce84aa00 ce8cfefc c10c2de0 00000002 00000001 00000000 [ 1103.521211] c10c2db3 ce843948 b77222c0 ce8cff20 c10aef10 ce8cff20 c10432c5 00000001 [ 1103.521211] Call Trace: [ 1103.521211] [] ? mem_cgroup_count_vm_event+0x1e/0x163 [ 1103.521211] [] mem_cgroup_count_vm_event+0x4b/0x163 [ 1103.521211] [] ? mem_cgroup_count_vm_event+0x1e/0x163 [ 1103.521211] [] handle_mm_fault+0x2f/0xb0 [ 1103.521211] [] ? down_read_trylock+0x51/0x59 [ 1103.521211] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.521211] [] do_page_fault+0x336/0x36e [ 1103.521211] [] ? vfs_read+0x103/0x10d [ 1103.521211] [] ? restore_all+0xf/0xf [ 1103.521211] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.521211] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.521211] [] do_async_page_fault+0x23/0x5f [ 1103.521211] [] error_code+0x6c/0x80 [ 1103.521211] Code: 45 08 89 04 24 8b 4d ec 89 f2 89 f8 e8 57 e7 ff ff c7 83 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 83 c4 20 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 1c [ 1103.521211] Kernel panic - not syncing: softlockup: hung tasks [ 1103.521211] Pid: 135, comm: sleep Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.521211] Call Trace: [ 1103.521211] [] panic+0x97/0x1aa [ 1103.521211] [] watchdog_timer_fn+0x115/0x134 [ 1103.521211] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.521211] [] ? __touch_watchdog+0x1d/0x1d [ 1103.521211] [] run_local_timers+0x8/0x14 [ 1103.521211] [] update_process_times+0x24/0x58 [ 1103.521211] [] tick_periodic+0x66/0x72 [ 1103.521211] [] tick_handle_periodic+0x19/0x61 [ 1103.521211] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.521211] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.521211] [] apic_timer_interrupt+0x36/0x40 [ 1103.521211] [] ? lock_acquire+0xb4/0xc0 [ 1103.521211] [] ? mem_cgroup_count_vm_event+0x1e/0x163 [ 1103.521211] [] mem_cgroup_count_vm_event+0x4b/0x163 [ 1103.521211] [] ? mem_cgroup_count_vm_event+0x1e/0x163 [ 1103.521211] [] handle_mm_fault+0x2f/0xb0 [ 1103.521211] [] ? down_read_trylock+0x51/0x59 [ 1103.521211] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.521211] [] do_page_fault+0x336/0x36e [ 1103.521211] [] ? vfs_read+0x103/0x10d [ 1103.521211] [] ? restore_all+0xf/0xf [ 1103.521211] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.521211] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.521211] [] do_async_page_fault+0x23/0x5f [ 1103.521211] [] error_code+0x6c/0x80 [ 1103.521211] Rebooting in 10 seconds.. -- [ 4.999346] Freeing unused kernel memory: 460k freed [ 1104.515803] BUG: soft lockup - CPU#1 stuck for 1025s! [swapper/1:0] [ 1104.515803] irq event stamp: 12116 [ 1104.515803] hardirqs last enabled at (12113): [] restore_all_notrace+0x0/0x18 [ 1104.515803] hardirqs last disabled at (12114): [] apic_timer_interrupt+0x2f/0x40 [ 1104.515803] softirqs last enabled at (12116): [] _local_bh_enable+0xd/0xf [ 1104.515803] softirqs last disabled at (12115): [] irq_enter+0x2c/0x4d [ 1104.515803] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.515803] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.515803] EIP is at native_safe_halt+0x5/0x7 [ 1104.515803] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1104.515803] ESI: c19716f0 EDI: ffff8b9d EBP: ce071f68 ESP: ce071f68 [ 1104.515803] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.515803] CR0: 8005003b CR2: b76c7fb8 CR3: 0ee89000 CR4: 00000690 [ 1104.515803] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.522791] [sched_delayed] sched: RT throttling activated [ 1104.527860] DR6: ffff0ff0 DR7: 00000400 [ 1104.527860] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) [ 1104.527860] Stack: [ 1104.527860] ce071f70 c1008a09 ce071f88 c1009250 d2ce2da2 e131b2c1 1b7c62b3 cfff9fa0 [ 1104.527860] ce071fb4 c166ed10 00000000 00000000 00000000 00000000 8bab6b51 1b7c62b3 [ 1104.527860] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.527860] Call Trace: [ 1104.527860] [] default_idle+0x2f/0x55 [ 1104.527860] [] cpu_idle+0xc9/0xfb [ 1104.527860] [] start_secondary+0x292/0x29a [ 1104.527860] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1104.527860] Kernel panic - not syncing: softlockup: hung tasks [ 1104.527860] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.527860] Call Trace: [ 1104.527860] [] panic+0x97/0x1aa [ 1104.527860] [] watchdog_timer_fn+0x115/0x134 [ 1104.527860] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.527860] [] ? __touch_watchdog+0x1d/0x1d [ 1104.527860] [] run_local_timers+0x8/0x14 [ 1104.527860] [] update_process_times+0x24/0x58 [ 1104.527860] [] tick_periodic+0x66/0x72 [ 1104.527860] [] tick_handle_periodic+0x19/0x61 [ 1104.527860] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.527860] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.527860] [] apic_timer_interrupt+0x36/0x40 [ 1104.527860] [] ? native_safe_halt+0x5/0x7 [ 1104.527860] [] default_idle+0x2f/0x55 [ 1104.527860] [] cpu_idle+0xc9/0xfb [ 1104.527860] [] start_secondary+0x292/0x29a [ 1104.527860] Rebooting in 10 seconds.. -- [ 8.316068] Freeing unused kernel memory: 460k freed [ 76.723703] BUG: soft lockup - CPU#1 stuck for 64s! [sh:133] [ 76.723703] irq event stamp: 1490 [ 76.723703] hardirqs last enabled at (1489): [] do_page_fault+0x98/0x36e [ 76.723703] hardirqs last disabled at (1490): [] apic_timer_interrupt+0x2f/0x40 [ 76.723703] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 76.723703] softirqs last disabled at (0): [< (null)>] (null) [ 76.723703] Pid: 133, comm: sh Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 76.723703] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 76.723703] EIP is at lock_acquire+0xb4/0xc0 [ 76.723703] EAX: 00000282 EBX: ce8ab390 ECX: f46fc012 EDX: f46fc012 [ 76.730213] [sched_delayed] sched: RT throttling activated [ 76.723703] ESI: 00000000 EDI: c19136b8 EBP: ce893e48 ESP: ce893e1c [ 76.723703] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 76.723703] CR0: 8005003b CR2: b752d2ad CR3: 0e889000 CR4: 00000690 [ 76.723703] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 76.723703] DR6: ffff0ff0 DR7: 00000400 [ 76.723703] Process sh (pid: 133, ti=ce892000 task=ce8ab390 task.ti=ce892000) [ 76.723703] Stack: [ 76.723703] 00000002 00000001 00000000 00000000 c10b402b 00000000 00000000 00000282 [ 76.723703] ce69cae0 00000000 0aed7025 ce893e6c c10b4058 00000002 00000001 00000000 [ 76.723703] c10b402b 00000000 c10ac260 ce69cae0 ce893eb8 c10ac2ee 00000000 ce870a34 [ 76.723703] Call Trace: [ 76.723703] [] ? page_add_file_rmap+0xff/0x153 [ 76.723703] [] page_add_file_rmap+0x12c/0x153 [ 76.723703] [] ? page_add_file_rmap+0xff/0x153 [ 76.723703] [] ? __do_fault+0x222/0x3c1 [ 76.723703] [] __do_fault+0x2b0/0x3c1 [ 76.723703] [] handle_pte_fault+0x2e6/0x7a7 [ 76.723703] [] handle_mm_fault+0xa1/0xb0 [ 76.723703] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 76.723703] [] do_page_fault+0x336/0x36e [ 76.723703] [] ? might_fault+0x7b/0x99 [ 76.723703] [] ? set_tls_desc+0x14a/0x15d [ 76.723703] [] ? restore_all+0xf/0xf [ 76.723703] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 76.723703] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 76.723703] [] do_async_page_fault+0x23/0x5f [ 76.723703] [] error_code+0x6c/0x80 [ 76.723703] Code: 45 08 89 04 24 8b 4d ec 89 f2 89 f8 e8 57 e7 ff ff c7 83 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 83 c4 20 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 1c [ 76.723703] Kernel panic - not syncing: softlockup: hung tasks [ 76.723703] Pid: 133, comm: sh Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 76.723703] Call Trace: [ 76.723703] [] panic+0x97/0x1aa [ 76.723703] [] watchdog_timer_fn+0x115/0x134 [ 76.723703] [] hrtimer_run_queues+0x14c/0x1c6 [ 76.723703] [] ? __touch_watchdog+0x1d/0x1d [ 76.723703] [] run_local_timers+0x8/0x14 [ 76.723703] [] update_process_times+0x24/0x58 [ 76.723703] [] tick_periodic+0x66/0x72 [ 76.723703] [] tick_handle_periodic+0x19/0x61 [ 76.723703] [] smp_apic_timer_interrupt+0x6d/0x7f [ 76.723703] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 76.723703] [] apic_timer_interrupt+0x36/0x40 [ 76.723703] [] ? lock_acquire+0xb4/0xc0 [ 76.723703] [] ? page_add_file_rmap+0xff/0x153 [ 76.723703] [] page_add_file_rmap+0x12c/0x153 [ 76.838708] [] ? page_add_file_rmap+0xff/0x153 [ 76.838708] [] ? __do_fault+0x222/0x3c1 [ 76.838708] [] __do_fault+0x2b0/0x3c1 [ 76.838708] [] handle_pte_fault+0x2e6/0x7a7 [ 76.838708] [] handle_mm_fault+0xa1/0xb0 [ 76.838708] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 76.838708] [] do_page_fault+0x336/0x36e [ 76.838708] [] ? might_fault+0x7b/0x99 [ 76.838708] [] ? set_tls_desc+0x14a/0x15d [ 76.838708] [] ? restore_all+0xf/0xf [ 76.838708] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 76.838708] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 76.838708] [] do_async_page_fault+0x23/0x5f [ 76.838708] [] error_code+0x6c/0x80 [ 76.838708] Rebooting in 10 seconds.. -- [ 4.225859] Freeing unused kernel memory: 460k freed [ 1103.516157] BUG: soft lockup - CPU#1 stuck for 1024s! [sh:134] [ 1103.516157] irq event stamp: 3580 [ 1103.516157] hardirqs last enabled at (3579): [] __mutex_unlock_slowpath+0x114/0x12c [ 1103.516157] hardirqs last disabled at (3580): [] apic_timer_interrupt+0x2f/0x40 [ 1103.516157] softirqs last enabled at (2952): [] unix_sock_destructor+0xae/0xb3 [ 1103.516157] softirqs last disabled at (2950): [] unix_sock_destructor+0x9b/0xb3 [ 1103.516157] Pid: 134, comm: sh Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.516157] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.516157] EIP is at lock_acquire+0xb4/0xc0 [ 1103.516157] EAX: 00000282 EBX: cf891130 ECX: 006f437b EDX: 006f437b [ 1103.524872] [sched_delayed] sched: RT throttling activated -- [ 5.072722] Freeing unused kernel memory: 460k freed [ 1104.521453] BUG: soft lockup - CPU#1 stuck for 1024s! [trinity:133] [ 1104.521453] irq event stamp: 5428 [ 1104.521453] hardirqs last enabled at (5427): [] get_page_from_freelist+0x2ca/0x456 [ 1104.521453] hardirqs last disabled at (5428): [] apic_timer_interrupt+0x2f/0x40 [ 1104.521453] softirqs last enabled at (3010): [] unix_sock_destructor+0xae/0xb3 [ 1104.521453] softirqs last disabled at (3008): [] unix_sock_destructor+0x9b/0xb3 [ 1104.521453] Pid: 133, comm: trinity Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.521453] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.530421] [sched_delayed] sched: RT throttling activated [ 1104.521453] EIP is at lock_acquire+0xb4/0xc0 [ 1104.521453] EAX: 00000282 EBX: cf869130 ECX: 006f8012 EDX: 006f8012 [ 1104.521453] ESI: 00000000 EDI: c19136b8 EBP: ce8cfe04 ESP: ce8cfdd8 [ 1104.521453] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.521453] CR0: 8005003b CR2: 086e4e68 CR3: 0e883000 CR4: 00000690 [ 1104.521453] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.521453] DR6: ffff0ff0 DR7: 00000400 [ 1104.521453] Process trinity (pid: 133, ti=ce8ce000 task=cf869130 task.ti=ce8ce000) [ 1104.521453] Stack: [ 1104.521453] 00000002 00000001 00000000 00000000 c1096b46 00000000 00000000 00000282 [ 1104.521453] c70b97c8 00000a5c 0000069c ce8cfe2c c1096b73 00000002 00000001 00000000 [ 1104.521453] c1096b46 00000000 c6d5a500 00000a5c 00000000 ce8cfe6c c1096e6d 00000000 [ 1104.521453] Call Trace: [ 1104.521453] [] ? find_get_page+0x17/0x1a6 [ 1104.521453] [] find_get_page+0x44/0x1a6 [ 1104.521453] [] ? find_get_page+0x17/0x1a6 [ 1104.521453] [] filemap_fault+0x76/0x369 [ 1104.521453] [] __do_fault+0xa6/0x3c1 [ 1104.521453] [] handle_pte_fault+0x2e6/0x7a7 [ 1104.521453] [] handle_mm_fault+0xa1/0xb0 [ 1104.521453] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.521453] [] do_page_fault+0x336/0x36e [ 1104.521453] [] ? up_write+0x16/0x2b [ 1104.521453] [] ? vm_mmap_pgoff+0x50/0x59 [ 1104.521453] [] ? restore_all+0xf/0xf [ 1104.521453] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.521453] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.521453] [] do_async_page_fault+0x23/0x5f [ 1104.521453] [] error_code+0x6c/0x80 [ 1104.521453] Code: 45 08 89 04 24 8b 4d ec 89 f2 89 f8 e8 57 e7 ff ff c7 83 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 83 c4 20 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 1c [ 1104.521453] Kernel panic - not syncing: softlockup: hung tasks [ 1104.521453] Pid: 133, comm: trinity Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.521453] Call Trace: [ 1104.521453] [] panic+0x97/0x1aa [ 1104.521453] [] watchdog_timer_fn+0x115/0x134 [ 1104.521453] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.521453] [] ? __touch_watchdog+0x1d/0x1d [ 1104.521453] [] run_local_timers+0x8/0x14 [ 1104.521453] [] update_process_times+0x24/0x58 [ 1104.521453] [] tick_periodic+0x66/0x72 [ 1104.521453] [] tick_handle_periodic+0x19/0x61 [ 1104.521453] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.521453] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.521453] [] apic_timer_interrupt+0x36/0x40 [ 1104.521453] [] ? lock_acquire+0xb4/0xc0 [ 1104.521453] [] ? find_get_page+0x17/0x1a6 [ 1104.521453] [] find_get_page+0x44/0x1a6 [ 1104.521453] [] ? find_get_page+0x17/0x1a6 [ 1104.521453] [] filemap_fault+0x76/0x369 [ 1104.521453] [] __do_fault+0xa6/0x3c1 [ 1104.521453] [] handle_pte_fault+0x2e6/0x7a7 [ 1104.521453] [] handle_mm_fault+0xa1/0xb0 [ 1104.521453] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.521453] [] do_page_fault+0x336/0x36e [ 1104.521453] [] ? up_write+0x16/0x2b [ 1104.521453] [] ? vm_mmap_pgoff+0x50/0x59 [ 1104.521453] [] ? restore_all+0xf/0xf [ 1104.521453] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.521453] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.521453] [] do_async_page_fault+0x23/0x5f [ 1104.521453] [] error_code+0x6c/0x80 [ 1104.521453] Rebooting in 10 seconds.. -- [ 6.167052] Freeing unused kernel memory: 460k freed [ 1105.525217] BUG: soft lockup - CPU#1 stuck for 1024s! [umount:132] [ 1105.525217] irq event stamp: 1012 [ 1105.525217] hardirqs last enabled at (1011): [] do_page_fault+0x98/0x36e [ 1105.525217] hardirqs last disabled at (1012): [] apic_timer_interrupt+0x2f/0x40 [ 1105.525217] softirqs last enabled at (604): [ 1105.528841] [sched_delayed] sched: RT throttling activated [] __do_softirq+0x131/0x153 [ 1105.525217] softirqs last disabled at (583): [] do_softirq+0x74/0xdf [ 1105.525217] Pid: 132, comm: umount Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1105.525217] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1105.525217] EIP is at lock_acquire+0xb4/0xc0 [ 1105.525217] EAX: 00000282 EBX: cf893390 ECX: 006f637c EDX: 006f637c [ 1105.525217] ESI: 00000000 EDI: ce8aa544 EBP: ce8afe50 ESP: ce8afe24 [ 1105.525217] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1105.525217] CR0: 8005003b CR2: b7749073 CR3: 0e89a000 CR4: 00000690 [ 1105.525217] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1105.525217] DR6: ffff0ff0 DR7: 00000400 [ 1105.525217] Process umount (pid: 132, ti=ce8ae000 task=cf893390 task.ti=ce8ae000) [ 1105.525217] Stack: [ 1105.525217] 00000000 00000002 00000000 00000000 c10ac260 00000000 00000000 00000282 [ 1105.525217] ce8aa534 00000000 00000009 ce8afe6c c167e6a2 00000000 00000002 00000000 [ 1105.525217] c10ac260 ce689ca0 ce8afeb8 c10ac260 00000000 ce8aa534 ce89dd24 00000000 [ 1105.525217] Call Trace: [ 1105.525217] [] ? __do_fault+0x222/0x3c1 [ 1105.525217] [] _raw_spin_lock+0x3d/0x4a [ 1105.525217] [] ? __do_fault+0x222/0x3c1 [ 1105.525217] [] __do_fault+0x222/0x3c1 [ 1105.525217] [] handle_pte_fault+0x2e6/0x7a7 [ 1105.525217] [] handle_mm_fault+0xa1/0xb0 [ 1105.525217] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1105.525217] [] do_page_fault+0x336/0x36e [ 1105.525217] [] ? call_rcu_sched+0x15/0x17 [ 1105.525217] [] ? __put_cred+0x35/0x37 [ 1105.525217] [] ? restore_all+0xf/0xf [ 1105.525217] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1105.525217] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1105.525217] [] do_async_page_fault+0x23/0x5f [ 1105.525217] [] error_code+0x6c/0x80 [ 1105.525217] Code: 45 08 89 04 24 8b 4d ec 89 f2 89 f8 e8 57 e7 ff ff c7 83 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 83 c4 20 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 1c [ 1105.525217] Kernel panic - not syncing: softlockup: hung tasks [ 1105.525217] Pid: 132, comm: umount Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1105.525217] Call Trace: [ 1105.525217] [] panic+0x97/0x1aa [ 1105.525217] [] watchdog_timer_fn+0x115/0x134 [ 1105.525217] [] hrtimer_run_queues+0x14c/0x1c6 [ 1105.525217] [] ? __touch_watchdog+0x1d/0x1d [ 1105.525217] [] run_local_timers+0x8/0x14 [ 1105.525217] [] update_process_times+0x24/0x58 [ 1105.525217] [] tick_periodic+0x66/0x72 [ 1105.525217] [] tick_handle_periodic+0x19/0x61 [ 1105.525217] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1105.525217] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1105.525217] [] apic_timer_interrupt+0x36/0x40 [ 1105.525217] [] ? lock_acquire+0xb4/0xc0 [ 1105.525217] [] ? __do_fault+0x222/0x3c1 [ 1105.525217] [] _raw_spin_lock+0x3d/0x4a [ 1105.525217] [] ? __do_fault+0x222/0x3c1 [ 1105.525217] [] __do_fault+0x222/0x3c1 [ 1105.525217] [] handle_pte_fault+0x2e6/0x7a7 [ 1105.525217] [] handle_mm_fault+0xa1/0xb0 [ 1105.525217] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1105.525217] [] do_page_fault+0x336/0x36e [ 1105.525217] [] ? call_rcu_sched+0x15/0x17 [ 1105.525217] [] ? __put_cred+0x35/0x37 [ 1105.525217] [] ? restore_all+0xf/0xf [ 1105.525217] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1105.525217] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1105.525217] [] do_async_page_fault+0x23/0x5f [ 1105.525217] [] error_code+0x6c/0x80 [ 1105.525217] Rebooting in 10 seconds.. -- [ 4.943616] Freeing unused kernel memory: 460k freed [ 1103.516727] BUG: soft lockup - CPU#1 stuck for 1023s! [99-trinity:129] [ 1103.516727] irq event stamp: 632 [ 1103.516727] hardirqs last enabled at (631): [] __mutex_unlock_slowpath+0x114/0x12c [ 1103.516727] hardirqs last disabled at (632): [] apic_timer_interrupt+0x2f/0x40 [ 1103.516727] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1103.516727] softirqs last disabled at (0): [ 1103.522351] [sched_delayed] sched: RT throttling activated [< (null)>] (null) [ 1103.516727] Pid: 129, comm: 99-trinity Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.516727] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.516727] EIP is at lock_is_held+0x9f/0xb2 [ 1103.516727] EAX: 00000282 EBX: 00000002 ECX: 00000000 EDX: 00000000 [ 1103.516727] ESI: cf899130 EDI: c19136a8 EBP: ce8bbddc ESP: ce8bbdcc [ 1103.516727] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.516727] CR0: 8005003b CR2: 0807cbf0 CR3: 0e8b0000 CR4: 00000690 [ 1103.516727] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.516727] DR6: ffff0ff0 DR7: 00000400 [ 1103.516727] Process 99-trinity (pid: 129, ti=ce8ba000 task=cf899130 task.ti=ce8ba000) [ 1103.516727] Stack: [ 1103.516727] 00000282 000003c8 c182c540 00000000 ce8bbe04 c104bac6 00000001 00000282 [ 1103.516727] ce828a34 00000000 ce828a34 023d8045 00000000 ce8b254c ce8bbe64 c10ada87 [ 1103.516727] 00000040 00000006 c1bcc504 ce829210 ce828a34 ce8291b4 ce8b2548 ce8acb74 [ 1103.516727] Call Trace: [ 1103.516727] [] __might_sleep+0x66/0x1cd [ 1103.516727] [] copy_pte_range+0x382/0x3bf [ 1103.516727] [] copy_page_range+0xc8/0xdc [ 1103.516727] [] dup_mm+0x285/0x3a5 [ 1103.516727] [] copy_process+0x881/0xf8a [ 1103.516727] [] do_fork+0xef/0x257 [ 1103.516727] [] ? might_fault+0x90/0x99 [ 1103.516727] [] ? copy_to_user+0x3f/0x46 [ 1103.516727] [] sys_clone+0x2b/0x32 [ 1103.516727] [] ptregs_clone+0x15/0x20 [ 1103.516727] [] ? sysenter_do_call+0x12/0x38 [ 1103.516727] Code: 7c e2 31 c9 eb 05 b9 01 00 00 00 64 a1 28 06 9e c1 c7 80 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 eb 05 b9 01 00 00 00 89 c8 5b 5b 5e 5f 5d c3 55 89 [ 1103.516727] Kernel panic - not syncing: softlockup: hung tasks [ 1103.516727] Pid: 129, comm: 99-trinity Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.516727] Call Trace: [ 1103.516727] [] panic+0x97/0x1aa [ 1103.516727] [] watchdog_timer_fn+0x115/0x134 [ 1103.516727] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.516727] [] ? __touch_watchdog+0x1d/0x1d [ 1103.516727] [] run_local_timers+0x8/0x14 [ 1103.516727] [] update_process_times+0x24/0x58 [ 1103.516727] [] tick_periodic+0x66/0x72 [ 1103.516727] [] tick_handle_periodic+0x19/0x61 [ 1103.516727] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.516727] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.516727] [] apic_timer_interrupt+0x36/0x40 [ 1103.516727] [] ? lock_is_held+0x9f/0xb2 [ 1103.516727] [] __might_sleep+0x66/0x1cd [ 1103.516727] [] copy_pte_range+0x382/0x3bf [ 1103.516727] [] copy_page_range+0xc8/0xdc [ 1103.516727] [] dup_mm+0x285/0x3a5 [ 1103.516727] [] copy_process+0x881/0xf8a [ 1103.516727] [] do_fork+0xef/0x257 [ 1103.516727] [] ? might_fault+0x90/0x99 [ 1103.516727] [] ? copy_to_user+0x3f/0x46 [ 1103.516727] [] sys_clone+0x2b/0x32 [ 1103.516727] [] ptregs_clone+0x15/0x20 [ 1103.516727] [] ? sysenter_do_call+0x12/0x38 [ 1103.516727] Rebooting in 10 seconds.. -- [ 1103.522466] [sched_delayed] sched: RT throttling activated [ 1103.514897] BUG: soft lockup - CPU#1 stuck for 1023s! [init:1] [ 1103.514897] irq event stamp: 746164 [ 1103.514897] hardirqs last enabled at (746163): [] _raw_spin_unlock_irq+0x22/0x40 [ 1103.514897] hardirqs last disabled at (746164): [] apic_timer_interrupt+0x2f/0x40 [ 1103.514897] softirqs last enabled at (745922): [] __do_softirq+0x131/0x153 [ 1103.514897] softirqs last disabled at (745919): [] do_softirq+0x74/0xdf [ 1103.514897] Pid: 1, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.514897] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1103.514897] EIP is at _raw_spin_unlock_irq+0x2f/0x40 [ 1103.514897] EAX: ce048000 EBX: ce848504 ECX: 00000000 EDX: 00000007 [ 1103.514897] ESI: ce848158 EDI: ce043f94 EBP: ce043e8c ESP: ce043e88 [ 1103.514897] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.514897] CR0: 8005003b CR2: b75f1fb8 CR3: 0eecd000 CR4: 00000690 [ 1103.514897] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.514897] DR6: ffff0ff0 DR7: 00000400 [ 1103.514897] Process init (pid: 1, ti=ce042000 task=ce048000 task.ti=ce042000) [ 1103.514897] Stack: [ 1103.514897] 00000011 ce043ecc c1035a94 0000037b 0000037b 00000000 ce04845c ce048374 [ 1103.514897] ce043f80 ce848000 ce044000 ce043fb4 ce848504 00000001 ce043fb4 bfbc73a0 [ 1103.514897] 00000000 ce043fa0 c1001cf3 00000000 00000000 ce844080 ce04845c ce048000 [ 1103.514897] Call Trace: [ 1103.514897] [] get_signal_to_deliver+0x670/0x67a [ 1103.514897] [] do_signal+0x25/0x6a5 [ 1103.514897] [] ? lock_release_non_nested+0x7e/0x21b [ 1103.514897] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.514897] [] ? might_fault+0x7b/0x99 [ 1103.514897] [] do_notify_resume+0x22/0x56 [ 1103.514897] [] work_notifysig+0x2b/0x3b [ 1103.514897] Code: 89 c3 8d 40 10 8b 4d 04 ba 01 00 00 00 e8 1e 8b 9e ff 89 d8 e8 38 40 c4 ff e8 69 6a 9e ff 83 3d 94 20 91 c1 00 75 02 0f 0b fb 90 <8d> 74 26 00 89 e0 25 00 e0 ff ff ff 48 14 5b 5d c3 55 89 e5 53 [ 1103.514897] Kernel panic - not syncing: softlockup: hung tasks [ 1103.514897] Pid: 1, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.514897] Call Trace: [ 1103.514897] [] panic+0x97/0x1aa [ 1103.514897] [] watchdog_timer_fn+0x115/0x134 [ 1103.514897] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.514897] [] ? __touch_watchdog+0x1d/0x1d [ 1103.514897] [] run_local_timers+0x8/0x14 [ 1103.514897] [] update_process_times+0x24/0x58 [ 1103.514897] [] tick_periodic+0x66/0x72 [ 1103.514897] [] tick_handle_periodic+0x19/0x61 [ 1103.514897] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.514897] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.514897] [] apic_timer_interrupt+0x36/0x40 [ 1103.514897] [] ? _raw_spin_unlock_irq+0x2f/0x40 [ 1103.514897] [] get_signal_to_deliver+0x670/0x67a [ 1103.514897] [] do_signal+0x25/0x6a5 [ 1103.514897] [] ? lock_release_non_nested+0x7e/0x21b [ 1103.514897] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.514897] [] ? might_fault+0x7b/0x99 [ 1103.514897] [] do_notify_resume+0x22/0x56 [ 1103.514897] [] work_notifysig+0x2b/0x3b [ 1103.514897] Rebooting in 10 seconds.. -- [ 5.094780] Freeing unused kernel memory: 460k freed [ 1104.514155] BUG: soft lockup - CPU#1 stuck for 1024s! [sh:133] [ 1104.514155] irq event stamp: 3594 [ 1104.514155] hardirqs last enabled at (3593): [] debug_check_no_locks_freed+0x14b/0x167 [ 1104.514155] hardirqs last disabled at (3594): [] apic_timer_interrupt+0x2f/0x40 [ 1104.514155] softirqs last enabled at (3024): [] unix_sock_destructor+0xae/0xb3 [ 1104.514155] softirqs last disabled at (3022): [] unix_sock_destructor+0x9b/0xb3 [ 1104.514155] Pid: 133, comm: sh Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.522513] [sched_delayed] sched: RT throttling activated [ 1104.514155] EIP: 0060:[] EFLAGS: 00000246 CPU: 1 [ 1104.514155] EIP is at generic_permission+0x2d/0x22b [ 1104.514155] EAX: ce881f80 EBX: 00000000 ECX: 00000000 EDX: 00000001 [ 1104.514155] ESI: cf891130 EDI: cb4b78c0 EBP: ce8bbe24 ESP: ce8bbe10 [ 1104.514155] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.514155] CR0: 80050033 CR2: b74f3267 CR3: 0e88e000 CR4: 00000690 [ 1104.514155] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.514155] DR6: ffff0ff0 DR7: 00000400 [ 1104.514155] Process sh (pid: 133, ti=ce8ba000 task=cf891130 task.ti=ce8ba000) [ 1104.514155] Stack: [ 1104.514155] 000041ed 00000001 cb4b78c0 cf891130 ce8bbef8 ce8bbe38 c10d07c3 00000000 [ 1104.514155] ce028800 cf891130 ce8bbe44 c10d0808 ce8bbef8 ce8bbe98 c10d18cb cf891130 [ 1104.514155] ce8bbe64 c1098035 cf891130 cf891130 ce8bbe90 cf891130 cae37000 c1021b4a [ 1104.514155] Call Trace: [ 1104.514155] [] __inode_permission+0x51/0x56 [ 1104.514155] [] inode_permission+0x40/0x42 [ 1104.514155] [] link_path_walk+0x68/0x64b [ 1104.514155] [] ? read_cache_page_async+0x22/0x27 [ 1104.514155] [] ? kmap+0x5a/0x5c [ 1104.514155] [] ? page_getlink.isra.27+0x34/0x4f [ 1104.514155] [] path_openat+0x256/0x352 [ 1104.514155] [] do_filp_open+0x26/0x67 [ 1104.514155] [] ? alloc_fd+0x175/0x180 [ 1104.514155] [] do_sys_open+0xe6/0x171 [ 1104.514155] [] ? call_rcu_sched+0x15/0x17 [ 1104.514155] [] ? __put_cred+0x35/0x37 [ 1104.514155] [] sys_open+0x24/0x2a [ 1104.514155] [] syscall_call+0x7/0xb [ 1104.514155] Code: e5 57 56 53 83 ec 08 89 c7 89 55 f0 0f b7 00 89 45 ec 8b 5f 04 e8 81 d6 f6 ff 64 a1 28 06 9e c1 8b 80 c0 02 00 00 39 58 1c 75 09 6d ec 06 e9 66 01 00 00 8b 47 1c f6 40 32 01 0f 84 49 01 00 [ 1104.514155] Kernel panic - not syncing: softlockup: hung tasks [ 1104.514155] Pid: 133, comm: sh Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.514155] Call Trace: [ 1104.514155] [] panic+0x97/0x1aa [ 1104.514155] [] watchdog_timer_fn+0x115/0x134 [ 1104.514155] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.552525] [] ? __touch_watchdog+0x1d/0x1d [ 1104.552525] [] run_local_timers+0x8/0x14 [ 1104.552525] [] update_process_times+0x24/0x58 [ 1104.552525] [] tick_periodic+0x66/0x72 [ 1104.552525] [] tick_handle_periodic+0x19/0x61 [ 1104.552525] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.552525] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.552525] [] apic_timer_interrupt+0x36/0x40 [ 1104.552525] [] ? generic_permission+0x2d/0x22b [ 1104.552525] [] __inode_permission+0x51/0x56 [ 1104.552525] [] inode_permission+0x40/0x42 [ 1104.552525] [] link_path_walk+0x68/0x64b [ 1104.552525] [] ? read_cache_page_async+0x22/0x27 [ 1104.552525] [] ? kmap+0x5a/0x5c [ 1104.552525] [] ? page_getlink.isra.27+0x34/0x4f [ 1104.552525] [] path_openat+0x256/0x352 [ 1104.552525] [] do_filp_open+0x26/0x67 [ 1104.552525] [] ? alloc_fd+0x175/0x180 [ 1104.552525] [] do_sys_open+0xe6/0x171 [ 1104.552525] [] ? call_rcu_sched+0x15/0x17 [ 1104.552525] [] ? __put_cred+0x35/0x37 [ 1104.552525] [] sys_open+0x24/0x2a [ 1104.552525] [] syscall_call+0x7/0xb [ 1104.552525] Rebooting in 10 seconds.. -- [ 4.485480] Freeing unused kernel memory: 460k freed [ 1103.513144] BUG: soft lockup - CPU#1 stuck for 1024s! [init:132] [ 1103.513144] irq event stamp: 544 [ 1103.513144] hardirqs last enabled at (543): [] debug_check_no_locks_freed+0x14b/0x167 [ 1103.513144] hardirqs last disabled at (544): [] apic_timer_interrupt+0x2f/0x40 [ 1103.513144] softirqs last enabled at (122): [] __do_softirq+0x131/0x153 [ 1103.513144] softirqs last disabled at (95): [] do_softirq+0x74/0xdf [ 1103.513144] Pid: 132, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.513144] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.513144] EIP is at lock_is_held+0x9f/0xb2 [ 1103.513144] EAX: 00000282 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 1103.513144] ESI: cf8c9130 EDI: c1913698 EBP: ce8dbcf4 ESP: ce8dbce4 [ 1103.513144] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.522293] [sched_delayed] sched: RT throttling activated [ 1103.513144] CR0: 8005003b CR2: b7776000 CR3: 0e892000 CR4: 00000690 [ 1103.513144] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.513144] DR6: ffff0ff0 DR7: 00000400 [ 1103.513144] Process init (pid: 132, ti=ce8da000 task=cf8c9130 task.ti=ce8da000) [ 1103.513144] Stack: [ 1103.513144] 00000282 00000452 c182a2ca 00000000 ce8dbd1c c104bb01 ce8dbe34 00000001 [ 1103.513144] ce8da000 c784a100 00000080 00000000 ffffffff 00000000 ce8dbdac c1097c2a [ 1103.513144] 00000001 c1918cd4 ce8dbd60 00000282 00000000 00000000 00000001 ce891c84 [ 1103.513144] Call Trace: [ 1103.513144] [] __might_sleep+0xa1/0x1cd [ 1103.513144] [] generic_file_aio_read+0x268/0x5f1 [ 1103.513144] [] ? lg_local_unlock+0x16/0x2f [ 1103.513144] [] ? mntput_no_expire+0x2b/0xfb [ 1103.513144] [] do_sync_read+0x75/0xb0 [ 1103.513144] [] vfs_read+0x8b/0x10d [ 1103.513144] [] ? wait_on_retry_sync_kiocb+0x3f/0x3f [ 1103.513144] [] kernel_read+0x30/0x41 [ 1103.513144] [] load_elf_binary+0x1ea/0xfbd [ 1103.513144] [] search_binary_handler+0x14d/0x22b [ 1103.513144] [] ? search_binary_handler+0x3a/0x22b [ 1103.513144] [] ? load_elf_library+0x1bb/0x1bb [ 1103.513144] [] do_execve+0x38b/0x456 [ 1103.513144] [] ? do_execve+0xeb/0x456 [ 1103.513144] [] sys_execve+0x31/0x53 [ 1103.513144] [] ptregs_execve+0x12/0x20 [ 1103.513144] [] ? sysenter_do_call+0x12/0x38 [ 1103.513144] Code: 7c e2 31 c9 eb 05 b9 01 00 00 00 64 a1 28 06 9e c1 c7 80 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 eb 05 b9 01 00 00 00 89 c8 5b 5b 5e 5f 5d c3 55 89 [ 1103.513144] Kernel panic - not syncing: softlockup: hung tasks [ 1103.513144] Pid: 132, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.513144] Call Trace: [ 1103.513144] [] panic+0x97/0x1aa [ 1103.513144] [] watchdog_timer_fn+0x115/0x134 [ 1103.513144] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.513144] [] ? __lock_acquire+0x1444/0x1455 [ 1103.513144] [] ? __touch_watchdog+0x1d/0x1d [ 1103.513144] [] run_local_timers+0x8/0x14 [ 1103.513144] [] update_process_times+0x24/0x58 [ 1103.513144] [] tick_periodic+0x66/0x72 [ 1103.513144] [] tick_handle_periodic+0x19/0x61 [ 1103.513144] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.513144] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.513144] [] apic_timer_interrupt+0x36/0x40 [ 1103.513144] [] ? lock_is_held+0x9f/0xb2 [ 1103.513144] [] __might_sleep+0xa1/0x1cd [ 1103.513144] [] generic_file_aio_read+0x268/0x5f1 [ 1103.513144] [] ? lg_local_unlock+0x16/0x2f [ 1103.513144] [] ? mntput_no_expire+0x2b/0xfb [ 1103.513144] [] do_sync_read+0x75/0xb0 [ 1103.513144] [] vfs_read+0x8b/0x10d [ 1103.513144] [] ? wait_on_retry_sync_kiocb+0x3f/0x3f [ 1103.513144] [] kernel_read+0x30/0x41 [ 1103.513144] [] load_elf_binary+0x1ea/0xfbd [ 1103.513144] [] search_binary_handler+0x14d/0x22b [ 1103.513144] [] ? search_binary_handler+0x3a/0x22b [ 1103.513144] [] ? load_elf_library+0x1bb/0x1bb [ 1103.513144] [] do_execve+0x38b/0x456 [ 1103.513144] [] ? do_execve+0xeb/0x456 [ 1103.513144] [] sys_execve+0x31/0x53 [ 1103.513144] [] ptregs_execve+0x12/0x20 [ 1103.513144] [] ? sysenter_do_call+0x12/0x38 [ 1103.513144] Rebooting in 10 seconds.. -- [ 5.718013] Freeing unused kernel memory: 460k freed [ 1104.517767] BUG: soft lockup - CPU#1 stuck for 1023s! [swapper/1:0] [ 1104.517767] irq event stamp: 8968 [ 1104.517767] hardirqs last enabled at (8965): [] restore_all_notrace+0x0/0x18 [ 1104.517767] hardirqs last disabled at (8966): [] apic_timer_interrupt+0x2f/0x40 [ 1104.517767] softirqs last enabled at (8968): [] _local_bh_enable+0xd/0xf [ 1104.517767] softirqs last disabled at (8967): [] irq_enter+0x2c/0x4d [ 1104.517767] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.517767] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.517767] EIP is at native_safe_halt+0x5/0x7 [ 1104.517767] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1104.517767] ESI: c19716f0 EDI: ffff8b9f EBP: ce071f68 ESP: ce071f68 [ 1104.517767] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.517767] CR0: 8005003b CR2: b765ebd0 CR3: 0e89b000 CR4: 00000690 [ 1104.517767] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.517767] DR6: ffff0ff0 DR7: 00000400 [ 1104.517767] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) [ 1104.517767] Stack: [ 1104.517767] ce071f70 c1008a09 ce071f88 c1009250 43ed9c2d 52512150 8c9c3063 cfff9fa0 [ 1104.517767] ce071fb4 c166ed10 00000000 00000000 00000000 00000000 fccb38fe 8c9c3063 [ 1104.517767] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.517767] Call Trace: [ 1104.517767] [] default_idle+0x2f/0x55 [ 1104.517767] [] cpu_idle+0xc9/0xfb [ 1104.517767] [] start_secondary+0x292/0x29a [ 1104.517767] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1104.517767] Kernel panic - not syncing: softlockup: hung tasks [ 1104.517767] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.517767] Call Trace: [ 1104.517767] [] panic+0x97/0x1aa [ 1104.517767] [] watchdog_timer_fn+0x115/0x134 [ 1104.517767] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.517767] [] ? __touch_watchdog+0x1d/0x1d [ 1104.517767] [] run_local_timers+0x8/0x14 [ 1104.517767] [] update_process_times+0x24/0x58 [ 1104.517767] [] tick_periodic+0x66/0x72 [ 1104.517767] [] tick_handle_periodic+0x19/0x61 [ 1104.517767] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.517767] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.517767] [] apic_timer_interrupt+0x36/0x40 [ 1104.517767] [] ? native_safe_halt+0x5/0x7 [ 1104.517767] [] default_idle+0x2f/0x55 [ 1104.517767] [] cpu_idle+0xc9/0xfb [ 1104.517767] [] start_secondary+0x292/0x29a mountall: ply-event-loop.c:497: ply_event_loop_new: Assertion `loop->epoll_fd >= 0' failed. [ 1104.517767] Rebooting in 10 seconds.. -- [ 1104.513744] [sched_delayed] sched: RT throttling activated [ 1104.514576] BUG: soft lockup - CPU#1 stuck for 1024s! [init:1] [ 1104.514576] irq event stamp: 747944 [ 1104.514576] hardirqs last enabled at (747943): [] _raw_spin_unlock_irqrestore+0x42/0x63 [ 1104.514576] hardirqs last disabled at (747944): [] apic_timer_interrupt+0x2f/0x40 [ 1104.514576] softirqs last enabled at (744332): [] __do_softirq+0x131/0x153 [ 1104.514576] softirqs last disabled at (744329): [] do_softirq+0x74/0xdf [ 1104.514576] Pid: 1, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.514576] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.514576] EIP is at _raw_spin_unlock_irqrestore+0x51/0x63 [ 1104.514576] EAX: 00000282 EBX: 00000282 ECX: 00000000 EDX: 00000007 [ 1104.514576] ESI: cf8859d8 EDI: 00000001 EBP: ce043dfc ESP: ce043df4 [ 1104.514576] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.514576] CR0: 80050033 CR2: b76a9e60 CR3: 0eebd000 CR4: 00000690 [ 1104.514576] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.514576] DR6: ffff0ff0 DR7: 00000400 [ 1104.514576] Process init (pid: 1, ti=ce042000 task=ce048000 task.ti=ce042000) [ 1104.514576] Stack: [ 1104.514576] 00000000 cf8855f0 ce043e20 c104a23a 00000282 cf8859d8 0cb4c000 00000000 [ 1104.514576] cf8855f0 00000000 00000007 ce043e28 c104a280 ce043e38 c10306b6 cf8855f0 [ 1104.514576] 00000000 ce043ef4 c1031596 003f8011 00000000 ce048474 00000000 ce048000 [ 1104.514576] Call Trace: [ 1104.514576] [] try_to_wake_up+0x248/0x266 [ 1104.514576] [] wake_up_state+0xa/0xc [ 1104.514576] [] ptrace_resume+0x76/0x7c [ 1104.514576] [] ptrace_request+0x368/0x4f8 [ 1104.514576] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514576] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514576] [] ? lock_release_non_nested+0x7e/0x21b [ 1104.514576] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514576] [] arch_ptrace+0x474/0x48a [ 1104.514576] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1104.514576] [] ? trace_hardirqs_on+0xb/0xd [ 1104.514576] [] ? wait_task_inactive+0x6f/0xc1 [ 1104.514576] [] ? ptrace_check_attach+0xb4/0xc4 [ 1104.514576] [] sys_ptrace+0x34f/0x379 [ 1104.514576] [] ? sys_ptrace+0x6e/0x379 [ 1104.514576] [] sysenter_do_call+0x12/0x38 [ 1104.514576] Code: 3d 8c 20 91 c1 00 74 1d 89 d8 50 9d 8d 74 26 00 e8 dd 6b 9e ff eb 18 e8 ac 6a 9e ff 83 3d 8c 20 91 c1 00 75 02 0f 0b 89 d8 50 9d <8d> 74 26 00 89 e0 25 00 e0 ff ff ff 48 14 5b 5e 5d c3 55 89 e5 [ 1104.514576] Kernel panic - not syncing: softlockup: hung tasks [ 1104.514576] Pid: 1, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.514576] Call Trace: [ 1104.514576] [] panic+0x97/0x1aa [ 1104.514576] [] watchdog_timer_fn+0x115/0x134 [ 1104.514576] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.514576] [] ? __touch_watchdog+0x1d/0x1d [ 1104.514576] [] run_local_timers+0x8/0x14 [ 1104.514576] [] update_process_times+0x24/0x58 [ 1104.514576] [] tick_periodic+0x66/0x72 [ 1104.514576] [] tick_handle_periodic+0x19/0x61 [ 1104.514576] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.514576] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.514576] [] apic_timer_interrupt+0x36/0x40 [ 1104.514576] [] ? _raw_spin_unlock_irqrestore+0x51/0x63 [ 1104.514576] [] try_to_wake_up+0x248/0x266 [ 1104.514576] [] wake_up_state+0xa/0xc [ 1104.514576] [] ptrace_resume+0x76/0x7c [ 1104.514576] [] ptrace_request+0x368/0x4f8 [ 1104.514576] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514576] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514576] [] ? lock_release_non_nested+0x7e/0x21b [ 1104.514576] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514576] [] arch_ptrace+0x474/0x48a [ 1104.514576] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1104.514576] [] ? trace_hardirqs_on+0xb/0xd [ 1104.514576] [] ? wait_task_inactive+0x6f/0xc1 [ 1104.514576] [] ? ptrace_check_attach+0xb4/0xc4 [ 1104.514576] [] sys_ptrace+0x34f/0x379 [ 1104.514576] [] ? sys_ptrace+0x6e/0x379 [ 1104.514576] [] sysenter_do_call+0x12/0x38 [ 1104.514576] Rebooting in 10 seconds.. -- [ 5.876251] Freeing unused kernel memory: 460k freed [ 1104.522650] BUG: soft lockup - CPU#1 stuck for 1023s! [swapper/1:0] [ 1104.522650] irq event stamp: 14128 [ 1104.522650] hardirqs last enabled at (14125): [] restore_all_notrace+0x0/0x18 [ 1104.522650] hardirqs last disabled at (14126): [] apic_timer_interrupt+0x2f/0x40 [ 1104.528994] [sched_delayed] sched: RT throttling activated [ 1104.532753] softirqs last enabled at (14128): [] _local_bh_enable+0xd/0xf [ 1104.532753] softirqs last disabled at (14127): [] irq_enter+0x2c/0x4d [ 1104.532753] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.540165] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.540165] EIP is at native_safe_halt+0x5/0x7 [ 1104.540165] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1104.540165] ESI: c19716f0 EDI: ffff8ba4 EBP: ce071f68 ESP: ce071f68 [ 1104.540165] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.540165] CR0: 8005003b CR2: b77b152c CR3: 019f0000 CR4: 00000690 [ 1104.540165] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 -- [ 6.225744] Freeing unused kernel memory: 460k freed [ 1105.529503] BUG: soft lockup - CPU#1 stuck for 1024s! [swapper/1:0] [ 1105.529503] irq event stamp: 14064 [ 1105.529503] hardirqs last enabled at (14061): [] restore_all_notrace+0x0/0x18 [ 1105.529503] hardirqs last disabled at (14062): [] apic_timer_interrupt+0x2f/0x40 [ 1105.529503] softirqs last enabled at (14064): [] _local_bh_enable+0xd/0xf [ 1105.529503] softirqs last disabled at (14063): [] irq_enter+0x2c/0x4d [ 1105.529503] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1105.529503] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1105.529503] EIP is at native_safe_halt+0x5/0x7 [ 1105.529503] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1105.529503] ESI: c19716f0 EDI: ffff8b9e EBP: ce071f68 ESP: ce071f68 [ 1105.529503] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1105.529503] CR0: 8005003b CR2: b7667fb8 CR3: 019f0000 CR4: 00000690 [ 1105.529503] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1105.529503] DR6: ffff0ff0 DR7: 00000400 [ 1105.529503] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) [ 1105.529503] Stack: [ 1105.529503] ce071f70 c1008a09 ce071f88 c1009250 b6ce783b c531fd5c ff7d2fe6 cfff9fa0 [ 1105.529503] ce071fb4 c166ed10 00000000 00000000 00000000 00000000 6fac3882 ff7d2fe6 [ 1105.529503] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1105.529503] Call Trace: [ 1105.529503] [] default_idle+0x2f/0x55 [ 1105.529503] [] cpu_idle+0xc9/0xfb [ 1105.529503] [] start_secondary+0x292/0x29a [ 1105.529503] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1105.529503] Kernel panic - not syncing: softlockup: hung tasks [ 1105.529503] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1105.529503] Call Trace: [ 1105.529503] [] panic+0x97/0x1aa [ 1105.529503] [] watchdog_timer_fn+0x115/0x134 [ 1105.529503] [] hrtimer_run_queues+0x14c/0x1c6 [ 1105.529503] [] ? __touch_watchdog+0x1d/0x1d [ 1105.529503] [] run_local_timers+0x8/0x14 [ 1105.529503] [] update_process_times+0x24/0x58 [ 1105.529503] [] tick_periodic+0x66/0x72 [ 1105.529503] [] tick_handle_periodic+0x19/0x61 [ 1105.529503] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1105.529503] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1105.529503] [] apic_timer_interrupt+0x36/0x40 [ 1105.529503] [] ? native_safe_halt+0x5/0x7 [ 1105.529503] [] default_idle+0x2f/0x55 [ 1105.529503] [] cpu_idle+0xc9/0xfb [ 1105.529503] [] start_secondary+0x292/0x29a mountall: ply-event-loop.c:497: ply_event_loop_new: Assertion `loop->epoll_fd >= 0' failed. [ 1105.529503] Rebooting in 10 seconds.. -- [ 5.084082] Freeing unused kernel memory: 460k freed [ 1104.521392] BUG: soft lockup - CPU#1 stuck for 1025s! [swapper/1:0] [ 1104.521414] irq event stamp: 14272 [ 1104.521414] hardirqs last enabled at (14269): [] default_idle+0x1f/0x55 [ 1104.523434] [sched_delayed] sched: RT throttling activated [ 1104.521414] hardirqs last disabled at (14270): [] apic_timer_interrupt+0x2f/0x40 [ 1104.521414] softirqs last enabled at (14272): [] _local_bh_enable+0xd/0xf [ 1104.521414] softirqs last disabled at (14271): [] irq_enter+0x2c/0x4d [ 1104.521414] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.521414] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.521414] EIP is at native_safe_halt+0x5/0x7 [ 1104.521414] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1104.521414] ESI: c19716f0 EDI: ffff8b9f EBP: ce071f68 ESP: ce071f68 [ 1104.521414] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.521414] CR0: 8005003b CR2: b7611bd0 CR3: 0e866000 CR4: 00000690 [ 1104.521414] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 mountall: ply-ev[ 1104.521414] DR6: ffff0ff0 DR7: 00000400 ent-loop.c:497: [ 1104.521414] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) ply_event_loop_n[ 1104.521414] Stack: ew: Assertion `l[ 1104.521414] ce071f70oop->epoll_fd >= c1008a09 0' failed. ce071f88 c1009250 22af12a3 311297c7 6b5d3c03 cfff9fa0 [ 1104.521414] ce071fb4 c166ed10 00000000 00000000 00000000 00000000 db8c449d 6b5d3c03 -- [ 5.014137] Freeing unused kernel memory: 460k freed [ 1104.514173] BUG: soft lockup - CPU#1 stuck for 1024s! [init:1] [ 1104.514173] irq event stamp: 745706 [ 1104.514173] hardirqs last enabled at (745705): [] restore_all_notrace+0x0/0x18 [ 1104.514173] hardirqs last disabled at (745706): [] apic_timer_interrupt+0x2f/0x40 [ 1104.514173] softirqs last enabled at (741514): [] __do_softirq+0x131/0x153 [ 1104.514173] softirqs last disabled at (741501): [] do_softirq+0x74/0xdf [ 1104.514173] Pid: 1, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.514173] EIP: 0060:[] EFLAGS: 00000202 CPU: 1 [ 1104.522897] [sched_delayed] sched: RT throttling activated [ 1104.514173] EIP is at current_kernel_time+0x10/0x48 [ 1104.514173] EAX: 000003f3 EBX: ce02d800 ECX: 00020001 EDX: 0000001d [ 1104.514173] ESI: c797d300 EDI: 00000000 EBP: ce043e50 ESP: ce043e48 [ 1104.514173] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.514173] CR0: 8005003b CR2: b7762000 CR3: 0eec0000 CR4: 00000690 [ 1104.514173] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.514173] DR6: ffff0ff0 DR7: 00000400 [ 1104.514173] Process init (pid: 1, ti=ce042000 task=ce048000 task.ti=ce042000) [ 1104.514173] Stack: [ 1104.514173] c797d300 00000000 ce043e5c c102de8e c70ca210 ce043e7c c10dd8d0 00000001 [ 1104.514173] 00000041 0000001d cf83a3c0 00000001 00000001 ce043ecc c10cf835 00000041 [ 1104.514173] cf83a40c ce048000 cf869c00 00000008 00000000 c70ca298 c797d300 00000010 [ 1104.514173] Call Trace: [ 1104.514173] [] current_fs_time+0xb/0x19 [ 1104.514173] [] file_update_time+0x21/0x91 [ 1104.514173] [] pipe_write+0x3bf/0x3cf [ 1104.514173] [] do_sync_write+0x75/0xb0 [ 1104.514173] [] vfs_write+0x8d/0x10f [ 1104.514173] [] ? vfs_read+0x10d/0x10d [ 1104.514173] [] sys_write+0x3d/0x61 [ 1104.514173] [] sysenter_do_call+0x12/0x38 [ 1104.514173] Code: a1 c1 0f ad fe d3 ef f6 c1 20 74 04 89 fe 31 ff a1 70 d6 a1 c1 89 f2 5e 5f 5d c3 55 89 e5 57 56 a1 b4 d6 a1 c1 a8 01 74 04 f3 90 f3 8b 15 70 d6 a1 c1 8b 35 78 d6 a1 c1 8b 3d 7c d6 a1 c1 8b [ 1104.514173] Kernel panic - not syncing: softlockup: hung tasks [ 1104.514173] Pid: 1, comm: init Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1104.514173] Call Trace: [ 1104.514173] [] panic+0x97/0x1aa [ 1104.514173] [] watchdog_timer_fn+0x115/0x134 [ 1104.514173] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.514173] [] ? __touch_watchdog+0x1d/0x1d [ 1104.514173] [] run_local_timers+0x8/0x14 [ 1104.514173] [] update_process_times+0x24/0x58 [ 1104.514173] [] tick_periodic+0x66/0x72 [ 1104.514173] [] tick_handle_periodic+0x19/0x61 [ 1104.514173] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.514173] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.514173] [] apic_timer_interrupt+0x36/0x40 [ 1104.514173] [] ? kvm_guest_cpu_init+0x10b/0x214 [ 1104.514173] [] ? current_kernel_time+0x10/0x48 [ 1104.514173] [] current_fs_time+0xb/0x19 [ 1104.514173] [] file_update_time+0x21/0x91 [ 1104.514173] [] pipe_write+0x3bf/0x3cf [ 1104.514173] [] do_sync_write+0x75/0xb0 [ 1104.514173] [] vfs_write+0x8d/0x10f [ 1104.514173] [] ? vfs_read+0x10d/0x10d [ 1104.514173] [] sys_write+0x3d/0x61 [ 1104.514173] [] sysenter_do_call+0x12/0x38 [ 1104.514173] Rebooting in 10 seconds.. -- [ 4.537105] Freeing unused kernel memory: 460k freed [ 1103.514239] BUG: soft lockup - CPU#1 stuck for 1024s! [swapper/1:0] [ 1103.514239] irq event stamp: 7850 [ 1103.514239] hardirqs last enabled at (7847): [] restore_all_notrace+0x0/0x18 [ 1103.514239] hardirqs last disabled at (7848): [] apic_timer_interrupt+0x2f/0x40 [ 1103.514239] softirqs last enabled at (7850): [] _local_bh_enable+0xd/0xf [ 1103.514239] softirqs last disabled at (7849): [] irq_enter+0x2c/0x4d [ 1103.514239] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.514239] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1103.514239] EIP is at native_safe_halt+0x5/0x7 [ 1103.514239] EAX: ce06a260 EBX: ffffffff ECX: ce06a260 EDX: 00000007 [ 1103.514239] ESI: c19716f0 EDI: ffff8b9d EBP: ce071f68 ESP: ce071f68 [ 1103.523641] [sched_delayed] sched: RT throttling activated [ 1103.514239] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.514239] CR0: 8005003b CR2: b7677fb8 CR3: 0ee96000 CR4: 00000690 [ 1103.514239] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.514239] DR6: ffff0ff0 DR7: 00000400 [ 1103.514239] Process swapper/1 (pid: 0, ti=ce070000 task=ce06a260 task.ti=ce070000) [ 1103.514239] Stack: [ 1103.514239] ce071f70 c1008a09 ce071f88 c1009250 d5b8c18f e41c46af 1e665633 cfff9fa0 [ 1103.514239] ce071fb4 c166ed10 00000000 00000000 00000000 00000000 8e955ed0 1e665633 [ 1103.514239] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1103.514239] Call Trace: [ 1103.514239] [] default_idle+0x2f/0x55 [ 1103.514239] [] cpu_idle+0xc9/0xfb [ 1103.514239] [] start_secondary+0x292/0x29a [ 1103.514239] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1103.514239] Kernel panic - not syncing: softlockup: hung tasks [ 1103.514239] Pid: 0, comm: swapper/1 Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.514239] Call Trace: [ 1103.514239] [] panic+0x97/0x1aa [ 1103.514239] [] watchdog_timer_fn+0x115/0x134 [ 1103.514239] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.514239] [] ? __touch_watchdog+0x1d/0x1d [ 1103.514239] [] run_local_timers+0x8/0x14 [ 1103.514239] [] update_process_times+0x24/0x58 [ 1103.514239] [] tick_periodic+0x66/0x72 [ 1103.514239] [] tick_handle_periodic+0x19/0x61 [ 1103.514239] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.514239] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.514239] [] apic_timer_interrupt+0x36/0x40 [ 1103.514239] [] ? native_safe_halt+0x5/0x7 [ 1103.514239] [] default_idle+0x2f/0x55 [ 1103.514239] [] cpu_idle+0xc9/0xfb [ 1103.514239] [] start_secondary+0x292/0x29a [ 1103.514239] Rebooting in 10 seconds.. -- [ 6.525425] Freeing unused kernel memory: 464k freed [ 1105.523006] BUG: soft lockup - CPU#1 stuck for 1023s! [99-trinity:133] [ 1105.523006] irq event stamp: 632 [ 1105.523006] hardirqs last enabled at (631): [] __mutex_unlock_slowpath+0x114/0x12c [ 1105.523006] hardirqs last disabled at (632): [] apic_timer_interrupt+0x2f/0x40 [ 1105.523006] softirqs last enabled at (0): [] copy_process+0x471/0xf82 [ 1105.523006] softirqs last disabled at (0): [< (null)>] (null) [ 1105.523006] Pid: 133, comm: 99-trinity Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1105.523006] EIP: 0060:[] EFLAGS: 00000202 CPU: 1 [ 1105.523006] EIP is at lockdep_trace_alloc+0x3/0xe0 [ 1105.523006] EAX: 000280da EBX: 000280da ECX: c1958810 EDX: 00000000 [ 1105.523006] ESI: 0000000a EDI: ce807680 EBP: ce8a3d5c ESP: ce8a3d5c [ 1105.523006] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1105.523006] CR0: 80050033 CR2: b75e5baf CR3: 0e868000 CR4: 00000690 [ 1105.523006] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1105.523006] DR6: ffff0ff0 DR7: 00000400 [ 1105.523006] Process 99-trinity (pid: 133, ti=ce8a2000 task=ce8a8000 task.ti=ce8a2000) [ 1105.523006] Stack: [ 1105.523006] ce8a3dfc c109a6d2 ce8a8000 00000040 00000006 c1bc8eb4 ce8a8458 ce8a8000 [ 1105.523006] ce8a3da0 c1063a52 00000378 00000007 00000000 00000000 ce8a8000 c1900ad8 [ 1105.523006] ce8a3db4 00000286 00000286 000000aa c181b224 00000000 ce8a3ddc 00000286 [ 1105.523006] Call Trace: [ 1105.523006] [] __alloc_pages_nodemask+0x78/0x5db [ 1105.523006] [] ? mark_held_locks+0xa7/0xce [ 1105.523006] [] ? anon_vma_prepare+0x24/0x131 [ 1105.523006] [] handle_pte_fault+0x16b/0x74c [ 1105.523006] [] ? kmap_atomic_prot+0x11f/0x133 [ 1105.523006] [] handle_mm_fault+0x95/0xa4 [ 1105.523006] [] __get_user_pages+0x264/0x39e [ 1105.523006] [] get_user_pages+0x4b/0x53 [ 1105.523006] [] get_arg_page+0x4a/0xa9 [ 1105.523006] [] copy_strings+0xfe/0x1c6 [ 1105.523006] [] copy_strings_kernel+0x1c/0x2b [ 1105.523006] [] do_execve+0x343/0x456 [ 1105.523006] [] ? do_execve+0xeb/0x456 [ 1105.523006] [] sys_execve+0x31/0x53 [ 1105.523006] [] ptregs_execve+0x12/0x20 [ 1105.523006] [] ? sysenter_do_call+0x12/0x38 [ 1105.523006] Code: 3d 1c b7 95 c1 01 74 16 ba 9a 0a 00 00 b8 8b 35 81 c1 e8 7e 26 fc ff c6 05 1c b7 95 c1 01 64 ff 05 bc dc 9c c1 5b 5d c3 55 89 e5 <57> 56 53 89 c6 64 8b 1d 28 c6 9c c1 83 bb 54 04 00 00 00 0f 85 [ 1105.523006] Kernel panic - not syncing: softlockup: hung tasks [ 1105.523006] Pid: 133, comm: 99-trinity Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1105.523006] Call Trace: [ 1105.523006] [] panic+0x81/0x194 [ 1105.523006] [] watchdog_timer_fn+0x115/0x134 [ 1105.523006] [] hrtimer_run_queues+0x14c/0x1c6 [ 1105.523006] [] ? __touch_watchdog+0x1d/0x1d [ 1105.523006] [] run_local_timers+0x8/0x14 [ 1105.523006] [] update_process_times+0x24/0x58 [ 1105.523006] [] tick_periodic+0x66/0x72 [ 1105.523006] [] tick_handle_periodic+0x19/0x61 [ 1105.523006] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1105.523006] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1105.523006] [] apic_timer_interrupt+0x36/0x40 [ 1105.523006] [] ? lockdep_trace_alloc+0x3/0xe0 [ 1105.523006] [] __alloc_pages_nodemask+0x78/0x5db [ 1105.523006] [] ? mark_held_locks+0xa7/0xce [ 1105.523006] [] ? anon_vma_prepare+0x24/0x131 [ 1105.581916] [] handle_pte_fault+0x16b/0x74c [ 1105.581916] [] ? kmap_atomic_prot+0x11f/0x133 [ 1105.581916] [] handle_mm_fault+0x95/0xa4 [ 1105.581916] [] __get_user_pages+0x264/0x39e [ 1105.581916] [] get_user_pages+0x4b/0x53 [ 1105.581916] [] get_arg_page+0x4a/0xa9 [ 1105.581916] [] copy_strings+0xfe/0x1c6 [ 1105.581916] [] copy_strings_kernel+0x1c/0x2b [ 1105.581916] [] do_execve+0x343/0x456 [ 1105.581916] [] ? do_execve+0xeb/0x456 [ 1105.581916] [] sys_execve+0x31/0x53 [ 1105.581916] [] ptregs_execve+0x12/0x20 [ 1105.581916] [] ? sysenter_do_call+0x12/0x38 [ 1105.581916] Rebooting in 10 seconds.. -- [ 4.563976] Freeing unused kernel memory: 464k freed [ 1103.520656] BUG: soft lockup - CPU#1 stuck for 1023s! [shutdown:135] [ 1103.520656] irq event stamp: 2580 [ 1103.520656] hardirqs last enabled at (2579): [] mutex_lock_nested+0x2da/0x308 [ 1103.520656] hardirqs last disabled at (2580): [] apic_timer_interrupt+0x2f/0x40 [ 1103.520656] softirqs last enabled at (0): [] copy_process+0x471/0xf82 mountall: ply-ev[ 1103.520656] softirqs last disabled at (0): ent-loop.c:497: [< (null)>] (null) ply_event_loop_n[ 1103.520656] Pid: 135, comm: shutdown Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 ew: Assertion `l[ 1103.520656] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1103.520656] EIP is at mutex_lock_nested+0x2ea/0x308 [ 1103.520656] EAX: 00000286 EBX: cb4e699c ECX: 00000006 EDX: cf8826c8 [ 1103.520656] ESI: ce82dbb0 EDI: ce873d7c EBP: ce873d98 ESP: ce873d5c [ 1103.520656] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.520656] CR0: 8005003b CR2: b77911b8 CR3: 0e88b000 CR4: 00000690 [ 1103.520656] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 oop->epoll_fd >=[ 1103.520656] DR6: ffff0ff0 DR7: 00000400 [ 1103.520656] Process shutdown (pid: 135, ti=ce872000 task=cf882240 task.ti=ce872000) 0' failed. [ 1103.520656] Stack: [ 1103.520656] 00000000 00000002 00000000 c10addf7 cb4e69c0 cf882240 cb4e69a0 00000286 [ 1103.520656] ce873d7c ce873d7c 00000000 ce873d7c ce82d790 ce82dbb0 00000000 ce873dec [ 1103.520656] c10addf7 00000000 00000000 00000000 00000000 b775e000 ce0f6400 00000000 [ 1103.520656] Call Trace: [ 1103.520656] [] ? vma_adjust+0x134/0x30e [ 1103.520656] [] vma_adjust+0x134/0x30e [ 1103.520656] [] __split_vma+0xea/0x13e [ 1103.520656] [] do_munmap+0xab/0x24b [ 1103.520656] [] vm_munmap+0x30/0x41 [ 1103.520656] [] elf_map+0x80/0xa6 [ 1103.520656] [] load_elf_binary+0x7b7/0xfbd [ 1103.520656] [] search_binary_handler+0x14d/0x22b [ 1103.520656] [] ? search_binary_handler+0x3a/0x22b [ 1103.520656] [] ? load_elf_library+0x1bb/0x1bb [ 1103.520656] [] do_execve+0x38b/0x456 [ 1103.520656] [] ? do_execve+0xeb/0x456 [ 1103.520656] [] sys_execve+0x31/0x53 [ 1103.520656] [] ptregs_execve+0x12/0x20 [ 1103.520656] [] ? sysenter_do_call+0x12/0x38 [ 1103.520656] Code: d4 8f c1 00 74 1e 8b 45 e0 50 9d 8d 74 26 00 e8 5b 81 9f ff eb 19 e8 2a 80 9f ff 83 3d 6c d4 8f c1 00 75 02 0f 0b 8b 45 e0 50 9d <8d> 74 26 00 8d 45 e4 e8 c0 5f 9f ff 89 e0 25 00 e0 ff ff ff 48 [ 1103.520656] Kernel panic - not syncing: softlockup: hung tasks [ 1103.520656] Pid: 135, comm: shutdown Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 [ 1103.520656] Call Trace: [ 1103.520656] [] panic+0x81/0x194 [ 1103.520656] [] watchdog_timer_fn+0x115/0x134 [ 1103.520656] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.520656] [] ? __touch_watchdog+0x1d/0x1d [ 1103.520656] [] run_local_timers+0x8/0x14 [ 1103.520656] [] update_process_times+0x24/0x58 [ 1103.520656] [] tick_periodic+0x66/0x72 [ 1103.520656] [] tick_handle_periodic+0x19/0x61 [ 1103.520656] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.520656] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.520656] [] apic_timer_interrupt+0x36/0x40 [ 1103.520656] [] ? mutex_lock_nested+0x2ea/0x308 [ 1103.520656] [] ? vma_adjust+0x134/0x30e [ 1103.520656] [] vma_adjust+0x134/0x30e [ 1103.520656] [] __split_vma+0xea/0x13e [ 1103.520656] [] do_munmap+0xab/0x24b [ 1103.520656] [] vm_munmap+0x30/0x41 [ 1103.520656] [] elf_map+0x80/0xa6 [ 1103.520656] [] load_elf_binary+0x7b7/0xfbd [ 1103.520656] [] search_binary_handler+0x14d/0x22b [ 1103.520656] [] ? search_binary_handler+0x3a/0x22b [ 1103.520656] [] ? load_elf_library+0x1bb/0x1bb [ 1103.520656] [] do_execve+0x38b/0x456 [ 1103.520656] [] ? do_execve+0xeb/0x456 [ 1103.520656] [] sys_execve+0x31/0x53 [ 1103.520656] [] ptregs_execve+0x12/0x20 [ 1103.520656] [] ? sysenter_do_call+0x12/0x38 [ 1103.520656] Rebooting in 10 seconds.. -- [ 1103.521634] [sched_delayed] sched: RT throttling activated [ 1103.523066] BUG: soft lockup - CPU#1 stuck for 1023s! [hwclock:126] [ 1103.523066] irq event stamp: 7436 [ 1103.523066] hardirqs last enabled at (7435): [] _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.523066] hardirqs last disabled at (7436): [] apic_timer_interrupt+0x2f/0x40 [ 1103.523066] softirqs last enabled at (3224): [] unix_sock_destructor+0xae/0xb3 [ 1103.523066] softirqs last disabled at (3222): [] unix_sock_destructor+0x9b/0xb3 [ 1103.523066] Pid: 126, comm: hwclock Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.523066] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.523066] EIP is at _raw_spin_unlock_irqrestore+0x51/0x63 [ 1103.523066] EAX: 00000282 EBX: 00000282 ECX: 00000000 EDX: 00000007 [ 1103.523066] ESI: c1fa259c EDI: 00000000 EBP: ce891da4 ESP: ce891d9c [ 1103.523066] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.523066] CR0: 8005003b CR2: b76b7fb8 CR3: 019f0000 CR4: 00000690 [ 1103.523066] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.523066] DR6: ffff0ff0 DR7: 00000400 [ 1103.523066] Process hwclock (pid: 126, ti=ce890000 task=cf89a260 task.ti=ce890000) [ 1103.523066] Stack: [ 1103.523066] 00000001 00000001 ce891dd8 c12c3bde ce891de0 00000282 c1fa2598 ce44d000 [ 1103.523066] c1fa259c ce44c000 ce44c000 00000000 ce707980 00001000 00000000 ce891de0 [ 1103.523066] c12c4324 ce891e04 c109bf2e 00000000 00000001 00000000 00000001 ce707a00 [ 1103.523066] Call Trace: [ 1103.523066] [] __debug_check_no_obj_freed+0xe4/0x137 [ 1103.523066] [] debug_check_no_obj_freed+0x11/0x15 [ 1103.523066] [] free_pages_prepare+0x10a/0x10e [ 1103.523066] [] free_hot_cold_page+0x20/0x155 [ 1103.523066] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.523066] [] free_hot_cold_page_list+0x1d/0x31 [ 1103.523066] [] release_pages+0x171/0x179 [ 1103.523066] [] free_pages_and_swap_cache+0x68/0x7b [ 1103.523066] [] tlb_flush_mmu+0x56/0x6c [ 1103.523066] [] tlb_finish_mmu+0x12/0x31 [ 1103.523066] [] exit_mmap+0x96/0xe4 [ 1103.523066] [] mmput+0x42/0xa7 [ 1103.523066] [] exit_mm+0xde/0xe6 [ 1103.523066] [] do_exit+0x1d0/0x72c [ 1103.523066] [] ? do_sys_settimeofday+0x88/0x93 [ 1103.523066] [] ? _copy_from_user+0x37/0x4b [ 1103.523066] [] do_group_exit+0x5e/0x81 [ 1103.523066] [] sys_exit_group+0x13/0x13 [ 1103.523066] [] sysenter_do_call+0x12/0x38 [ 1103.523066] Code: 3d 8c 20 91 c1 00 74 1d 89 d8 50 9d 8d 74 26 00 e8 dd 6b 9e ff eb 18 e8 ac 6a 9e ff 83 3d 8c 20 91 c1 00 75 02 0f 0b 89 d8 50 9d <8d> 74 26 00 89 e0 25 00 e0 ff ff ff 48 14 5b 5e 5d c3 55 89 e5 [ 1103.523066] Kernel panic - not syncing: softlockup: hung tasks [ 1103.523066] Pid: 126, comm: hwclock Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1103.523066] Call Trace: [ 1103.523066] [] panic+0x97/0x1aa [ 1103.523066] [] watchdog_timer_fn+0x115/0x134 [ 1103.523066] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.523066] [] ? __touch_watchdog+0x1d/0x1d [ 1103.523066] [] run_local_timers+0x8/0x14 [ 1103.523066] [] update_process_times+0x24/0x58 [ 1103.523066] [] tick_periodic+0x66/0x72 [ 1103.523066] [] tick_handle_periodic+0x19/0x61 [ 1103.523066] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.523066] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.523066] [] apic_timer_interrupt+0x36/0x40 [ 1103.523066] [] ? _raw_spin_unlock_irqrestore+0x51/0x63 [ 1103.523066] [] __debug_check_no_obj_freed+0xe4/0x137 [ 1103.523066] [] debug_check_no_obj_freed+0x11/0x15 [ 1103.523066] [] free_pages_prepare+0x10a/0x10e [ 1103.523066] [] free_hot_cold_page+0x20/0x155 [ 1103.523066] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.523066] [] free_hot_cold_page_list+0x1d/0x31 [ 1103.523066] [] release_pages+0x171/0x179 [ 1103.523066] [] free_pages_and_swap_cache+0x68/0x7b [ 1103.523066] [] tlb_flush_mmu+0x56/0x6c [ 1103.523066] [] tlb_finish_mmu+0x12/0x31 [ 1103.523066] [] exit_mmap+0x96/0xe4 [ 1103.523066] [] mmput+0x42/0xa7 [ 1103.523066] [] exit_mm+0xde/0xe6 [ 1103.523066] [] do_exit+0x1d0/0x72c [ 1103.523066] [] ? do_sys_settimeofday+0x88/0x93 [ 1103.523066] [] ? _copy_from_user+0x37/0x4b [ 1103.523066] [] do_group_exit+0x5e/0x81 [ 1103.523066] [] sys_exit_group+0x13/0x13 [ 1103.523066] [] sysenter_do_call+0x12/0x38 [ 1103.523066] Rebooting in 10 seconds.. -- [ 4.801716] Freeing unused kernel memory: 460k freed [ 1103.512941] BUG: soft lockup - CPU#1 stuck for 1023s! [sleep:134] [ 1103.512941] irq event stamp: 780 [ 1103.512941] hardirqs last enabled at (779): [] do_page_fault+0x98/0x36e [ 1103.512941] hardirqs last disabled at (780): [] apic_timer_interrupt+0x2f/0x40 [ 1103.517007] softirqs last enabled at (22): [] __do_softirq+0x131/0x153 -- [ 1105.513232] [sched_delayed] sched: RT throttling activated [ 1105.513206] BUG: soft lockup - CPU#1 stuck for 1023s! [rc.local:123] [ 1105.513206] irq event stamp: 326 [ 1105.513206] hardirqs last enabled at (325): [] debug_check_no_locks_freed+0x14b/0x167 [ 1105.513206] hardirqs last disabled at (326): [] apic_timer_interrupt+0x2f/0x40 [ 1105.513206] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1105.513206] softirqs last disabled at (0): [< (null)>] (null) [ 1105.513206] Pid: 123, comm: rc.local Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1105.513206] EIP: 0060:[] EFLAGS: 00000246 CPU: 1 [ 1105.513206] EIP is at __mnt_drop_write+0x1/0x1a [ 1105.513206] EAX: ce014110 EBX: cb438000 ECX: 00000000 EDX: 00000001 [ 1105.513206] ESI: ce014110 EDI: 00000000 EBP: ce879dbc ESP: ce879da4 [ 1105.513206] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1105.513206] CR0: 80050033 CR2: b7553baf CR3: 0e85a000 CR4: 00000690 [ 1105.513206] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1105.513206] DR6: ffff0ff0 DR7: 00000400 [ 1105.513206] Process rc.local (pid: 123, ti=ce878000 task=cf8ad5f0 task.ti=ce878000) [ 1105.513206] Stack: [ 1105.513206] ce879dbc c10dd7bf 504bf223 ffffff88 ce889800 ce6a2f80 ce879e4c c1097f77 [ 1105.513206] 00001000 006f41e4 00000000 00000080 00000000 00000000 00000001 ce889884 [ 1105.513206] ce879e90 00000000 00001000 00000080 00000000 cb438000 ce879ed4 00000000 [ 1105.513206] Call Trace: [ 1105.513206] [] ? touch_atime+0xdc/0x104 [ 1105.513206] [] generic_file_aio_read+0x5b5/0x5f1 [ 1105.513206] [] do_sync_read+0x75/0xb0 [ 1105.513206] [] vfs_read+0x8b/0x10d [ 1105.513206] [] ? might_fault+0x26/0x99 [ 1105.513206] [] ? wait_on_retry_sync_kiocb+0x3f/0x3f [ 1105.513206] [] kernel_read+0x30/0x41 [ 1105.513206] [] prepare_binprm+0xfb/0x103 [ 1105.513206] [] do_execve+0x327/0x456 [ 1105.513206] [] ? do_execve+0xeb/0x456 [ 1105.546433] [] sys_execve+0x31/0x53 [ 1105.546433] [] ptregs_execve+0x12/0x20 [ 1105.546433] [] ? sysenter_do_call+0x12/0x38 [ 1105.546433] Code: 00 00 e8 6c ba fe ff 89 d8 e8 91 ff ff ff 89 c6 85 c0 74 10 8b 43 08 8b 40 04 ba 01 00 00 00 e8 dd b9 fe ff 89 f0 5b 5e 5d c3 55 <89> e5 89 e2 81 e2 00 e0 ff ff ff 42 14 8b 40 0c 64 ff 48 04 ff [ 1105.546433] Kernel panic - not syncing: softlockup: hung tasks [ 1105.546433] Pid: 123, comm: rc.local Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1105.546433] Call Trace: [ 1105.546433] [] panic+0x97/0x1aa [ 1105.546433] [] watchdog_timer_fn+0x115/0x134 [ 1105.546433] [] hrtimer_run_queues+0x14c/0x1c6 [ 1105.546433] [] ? __touch_watchdog+0x1d/0x1d [ 1105.546433] [] run_local_timers+0x8/0x14 [ 1105.546433] [] update_process_times+0x24/0x58 [ 1105.546433] [] tick_periodic+0x66/0x72 [ 1105.546433] [] tick_handle_periodic+0x19/0x61 [ 1105.546433] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1105.546433] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1105.546433] [] apic_timer_interrupt+0x36/0x40 [ 1105.546433] [] ? pipe_fcntl+0xbc/0x140 [ 1105.546433] [] ? __mnt_drop_write+0x1/0x1a [ 1105.546433] [] ? touch_atime+0xdc/0x104 [ 1105.546433] [] generic_file_aio_read+0x5b5/0x5f1 [ 1105.546433] [] do_sync_read+0x75/0xb0 [ 1105.546433] [] vfs_read+0x8b/0x10d [ 1105.546433] [] ? might_fault+0x26/0x99 [ 1105.546433] [] ? wait_on_retry_sync_kiocb+0x3f/0x3f [ 1105.546433] [] kernel_read+0x30/0x41 [ 1105.546433] [] prepare_binprm+0xfb/0x103 [ 1105.546433] [] do_execve+0x327/0x456 [ 1105.546433] [] ? do_execve+0xeb/0x456 [ 1105.546433] [] sys_execve+0x31/0x53 [ 1105.546433] [] ptregs_execve+0x12/0x20 [ 1105.546433] [] ? sysenter_do_call+0x12/0x38 [ 1105.546433] Rebooting in 10 seconds.. -- [ 4.563365] Freeing unused kernel memory: 464k freed [ 1103.514494] BUG: soft lockup - CPU#1 stuck for 1024s! [rc.local:122] [ 1103.514494] irq event stamp: 8 [ 1103.514494] hardirqs last enabled at (7): [] do_page_fault+0x98/0x36e [ 1103.514494] hardirqs last disabled at (8): [] apic_timer_interrupt+0x2f/0x40 [ 1103.514494] softirqs last enabled at (0): [] copy_process+0x471/0xf82 [ 1103.514494] softirqs last disabled at (0): [< (null)>] (null) [ 1103.514494] Pid: 122, comm: rc.local Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 [ 1103.520829] [sched_delayed] sched: RT throttling activated [ 1103.514494] EIP: 0060:[] EFLAGS: 00000212 CPU: 1 [ 1103.514494] EIP is at find_vma+0x48/0x68 [ 1103.514494] EAX: ce816780 EBX: ceea0d68 ECX: ceea0ddc EDX: b7654d0c [ 1103.514494] ESI: ce849fb4 EDI: c101c2e6 EBP: ce849f20 ESP: ce849f1c [ 1103.514494] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.514494] CR0: 8005003b CR2: b7654d0c CR3: 0e842000 CR4: 00000690 [ 1103.514494] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.514494] DR6: ffff0ff0 DR7: 00000400 [ 1103.514494] Process rc.local (pid: 122, ti=ce848000 task=cf1fd5a0 task.ti=ce848000) [ 1103.514494] Stack: [ 1103.514494] ce849fb4 ce849f9c c101d0e5 00000000 00000000 cf1fd5a0 ce8167d4 ce816780 [ 1103.514494] cf1fd5a0 00000028 00000004 b7654d0c c12aa250 00000001 00000003 00000000 [ 1103.514494] c166e9b5 bffffffd b75a0728 00000000 00000001 ce816780 00000007 00000006 [ 1103.514494] Call Trace: [ 1103.514494] [] do_page_fault+0x1a3/0x36e [ 1103.514494] [] ? trace_hardirqs_on_thunk+0xc/0x10 [ 1103.514494] [] ? restore_all+0xf/0xf [ 1103.514494] [] ? restore_all+0xf/0xf [ 1103.514494] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.514494] [] do_async_page_fault+0x23/0x5f [ 1103.514494] [] error_code+0x6c/0x80 [ 1103.514494] Code: e8 2e 81 f7 ff c6 05 05 94 95 c1 01 eb 38 8b 58 08 85 db 74 0a 39 53 08 76 05 39 53 04 76 27 8b 48 04 31 db eb 15 39 51 ec 76 0d <8d> 59 e4 39 51 e8 76 0c 8b 49 08 eb 03 8b 49 04 85 c9 75 e7 85 [ 1103.514494] Kernel panic - not syncing: softlockup: hung tasks [ 1103.514494] Pid: 122, comm: rc.local Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 [ 1103.514494] Call Trace: [ 1103.514494] [] panic+0x81/0x194 [ 1103.514494] [] watchdog_timer_fn+0x115/0x134 [ 1103.514494] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.514494] [] ? __touch_watchdog+0x1d/0x1d [ 1103.514494] [] run_local_timers+0x8/0x14 [ 1103.514494] [] update_process_times+0x24/0x58 [ 1103.514494] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.514494] [] tick_periodic+0x66/0x72 [ 1103.514494] [] tick_handle_periodic+0x19/0x61 [ 1103.514494] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.514494] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.514494] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.514494] [] apic_timer_interrupt+0x36/0x40 [ 1103.514494] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.514494] [] ? intel_pmu_lbr_init_core+0x34/0x3a [ 1103.514494] [] ? posix_cpu_timer_set+0x29e/0x361 [ 1103.514494] [] ? find_vma+0x48/0x68 [ 1103.514494] [] do_page_fault+0x1a3/0x36e [ 1103.514494] [] ? trace_hardirqs_on_thunk+0xc/0x10 [ 1103.514494] [] ? restore_all+0xf/0xf [ 1103.514494] [] ? restore_all+0xf/0xf [ 1103.514494] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.514494] [] do_async_page_fault+0x23/0x5f [ 1103.514494] [] error_code+0x6c/0x80 [ 1103.514494] Rebooting in 10 seconds.. -- [ 4.379731] Freeing unused kernel memory: 464k freed [ 1103.518739] BUG: soft lockup - CPU#1 stuck for 1024s! [init:1] [ 1103.518739] irq event stamp: 761948 [ 1103.518739] hardirqs last enabled at (761947): [] do_page_fault+0x98/0x36e [ 1103.518739] hardirqs last disabled at (761948): [] apic_timer_interrupt+0x2f/0x40 [ 1103.518739] softirqs last enabled at (761912): [] unix_accept+0xd2/0xe8 [ 1103.518739] softirqs last disabled at (761910): mountall: ply-ev[] _raw_write_lock_bh+0xe/0x4f ent-loop.c:497: [ 1103.518739] Pid: 1, comm: init Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 ply_event_loop_n[ 1103.518739] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 ew: Assertion `l[ 1103.518739] EIP is at lock_release+0x194/0x19f oop->epoll_fd >=[ 1103.518739] EAX: 00000286 EBX: ce040000 ECX: 0000036f EDX: 0000036f 0' failed. [ 1103.518739] ESI: 0000036f EDI: 00000000 EBP: ce03de60 ESP: ce03de4c [ 1103.528082] [sched_delayed] sched: RT throttling activated [ 1103.518739] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.518739] CR0: 80050033 CR2: b7665b90 CR3: 0f1ea000 CR4: 00000690 [ 1103.518739] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.518739] DR6: ffff0ff0 DR7: 00000400 [ 1103.518739] Process init (pid: 1, ti=ce03c000 task=ce040000 task.ti=ce03c000) [ 1103.518739] Stack: [ 1103.518739] 00000001 00000286 ce116534 00000000 00000000 ce03de6c c166e11b ce68afe0 [ 1103.518739] ce03deb8 c10a95e0 c10acb05 ce116534 cf1eb994 00000000 b7665b90 ce116500 [ 1103.518739] 00000000 00000200 cf81cf20 0a5ff025 00000028 0000000a b7665000 ce68afe0 [ 1103.518739] Call Trace: [ 1103.518739] [] _raw_spin_unlock+0x16/0x2a [ 1103.518739] [] __do_fault+0x2fa/0x38a [ 1103.518739] [] ? might_fault+0x90/0x97 [ 1103.518739] [] handle_pte_fault+0x2c3/0x74c [ 1103.518739] [] ? kmap_atomic_prot+0x11f/0x133 [ 1103.518739] [] handle_mm_fault+0x95/0xa4 [ 1103.518739] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.518739] [] do_page_fault+0x336/0x36e [ 1103.518739] [] ? _copy_from_user+0x37/0x4b [ 1103.518739] [] ? sys_socketcall+0xf1/0x2b2 [ 1103.518739] [] ? restore_all+0xf/0xf [ 1103.518739] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.518739] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.518739] [] do_async_page_fault+0x23/0x5f [ 1103.518739] [] error_code+0x6c/0x80 [ 1103.518739] Code: 85 c0 74 07 89 d8 e8 0a c5 ff ff 64 a1 28 a6 9c c1 c7 80 54 04 00 00 00 00 00 00 83 3d 6c d4 8f c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 5f 58 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 20 89 [ 1103.518739] Kernel panic - not syncing: softlockup: hung tasks [ 1103.518739] Pid: 1, comm: init Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 [ 1103.518739] Call Trace: [ 1103.518739] [] panic+0x81/0x194 [ 1103.518739] [] watchdog_timer_fn+0x115/0x134 [ 1103.518739] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.518739] [] ? __touch_watchdog+0x1d/0x1d [ 1103.518739] [] run_local_timers+0x8/0x14 [ 1103.518739] [] update_process_times+0x24/0x58 [ 1103.518739] [] tick_periodic+0x66/0x72 [ 1103.518739] [] tick_handle_periodic+0x19/0x61 [ 1103.518739] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.518739] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.518739] [] apic_timer_interrupt+0x36/0x40 [ 1103.518739] [] ? lock_release+0x194/0x19f [ 1103.518739] [] _raw_spin_unlock+0x16/0x2a [ 1103.518739] [] __do_fault+0x2fa/0x38a [ 1103.518739] [] ? might_fault+0x90/0x97 [ 1103.518739] [] handle_pte_fault+0x2c3/0x74c [ 1103.518739] [] ? kmap_atomic_prot+0x11f/0x133 [ 1103.518739] [] handle_mm_fault+0x95/0xa4 [ 1103.518739] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.518739] [] do_page_fault+0x336/0x36e [ 1103.518739] [] ? _copy_from_user+0x37/0x4b [ 1103.518739] [] ? sys_socketcall+0xf1/0x2b2 [ 1103.518739] [] ? restore_all+0xf/0xf [ 1103.518739] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.518739] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.518739] [] do_async_page_fault+0x23/0x5f [ 1103.518739] [] error_code+0x6c/0x80 [ 1103.518739] Rebooting in 10 seconds.. -- [ 5.795207] Freeing unused kernel memory: 464k freed [ 1104.515634] BUG: soft lockup - CPU#1 stuck for 1023s! [99-trinity:126] [ 1104.515634] irq event stamp: 732 [ 1104.515634] hardirqs last enabled at (731): [] do_page_fault+0x98/0x36e [ 1104.515634] hardirqs last disabled at (732): [] apic_timer_interrupt+0x2f/0x40 [ 1104.515634] softirqs last enabled at (0): [] copy_process+0x471/0xf82 [ 1104.515634] softirqs last disabled at (0): [< (null)>] (null) [ 1104.515634] Pid: 126, comm: 99-trinity Not tainted 3.5.0-bisect2-01297-ga17f29a #84 [ 1104.515634] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.515634] EIP is at lock_is_held+0x9f/0xb2 [ 1104.515634] EAX: 00000286 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 1104.515634] ESI: ce859120 EDI: c18feae8 EBP: ce869ef8 ESP: ce869ee8 [ 1104.515634] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.515634] CR0: 8005003b CR2: b7779ee0 CR3: 0e849000 CR4: 00000690 [ 1104.515634] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.515634] DR6: ffff0ff0 DR7: 00000400 [ 1104.515634] Process 99-trinity (pid: 126, ti=ce868000 task=ce859120 task.ti=ce868000) [ 1104.515634] Stack: [ 1104.515634] 00000286 0000046c c180ae2c 00000000 ce869f20 c1049cb6 c101c2da ce869f20 [ 1104.515634] c10415f1 00000001 00000002 ce869fb4 ce869fb4 c101c2da ce869f9c c101d0c9 [ 1104.515634] 00000028 00000000 00000000 ce834f54 ce834f00 ce859120 00000029 00000007 [ 1104.515634] Call Trace: [ 1104.515634] [] __might_sleep+0x66/0x1cd [ 1104.515634] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515634] [] ? down_read_trylock+0x51/0x59 [ 1104.515634] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515634] [] do_page_fault+0x193/0x36e [ 1104.515634] [] ? up_write+0x16/0x2b [ 1104.515634] [] ? vm_mmap_pgoff+0x50/0x59 [ 1104.515634] [] ? restore_all+0xf/0xf [ 1104.515634] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515634] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515634] [] do_async_page_fault+0x23/0x5f [ 1104.515634] [] error_code+0x6c/0x80 [ 1104.515634] Code: 7c e2 31 c9 eb 05 b9 01 00 00 00 64 a1 28 a6 9c c1 c7 80 54 04 00 00 00 00 00 00 83 3d 6c d4 8f c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 eb 05 b9 01 00 00 00 89 c8 5b 5b 5e 5f 5d c3 55 89 [ 1104.515634] Kernel panic - not syncing: softlockup: hung tasks [ 1104.515634] Pid: 126, comm: 99-trinity Not tainted 3.5.0-bisect2-01297-ga17f29a #84 [ 1104.515634] Call Trace: [ 1104.515634] [] panic+0x81/0x194 [ 1104.515634] [] watchdog_timer_fn+0x115/0x134 [ 1104.515634] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.515634] [] ? __touch_watchdog+0x1d/0x1d [ 1104.515634] [] run_local_timers+0x8/0x14 [ 1104.515634] [] update_process_times+0x24/0x58 [ 1104.515634] [] tick_periodic+0x66/0x72 [ 1104.515634] [] tick_handle_periodic+0x19/0x61 [ 1104.515634] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.515634] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.515634] [] apic_timer_interrupt+0x36/0x40 [ 1104.515634] [] ? lock_is_held+0x9f/0xb2 [ 1104.515634] [] __might_sleep+0x66/0x1cd [ 1104.515634] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515634] [] ? down_read_trylock+0x51/0x59 [ 1104.515634] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515634] [] do_page_fault+0x193/0x36e [ 1104.515634] [] ? up_write+0x16/0x2b [ 1104.515634] [] ? vm_mmap_pgoff+0x50/0x59 [ 1104.515634] [] ? restore_all+0xf/0xf [ 1104.515634] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515634] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515634] [] do_async_page_fault+0x23/0x5f [ 1104.515634] [] error_code+0x6c/0x80 [ 1104.515634] Rebooting in 10 seconds.. -- [ 4.912862] Freeing unused kernel memory: 460k freed [ 1103.516433] BUG: soft lockup - CPU#1 stuck for 1023s! [99-trinity:128] [ 1103.516433] irq event stamp: 32 [ 1103.516433] hardirqs last enabled at (31): [] do_page_fault+0x98/0x36e [ 1103.516433] hardirqs last disabled at (32): [] apic_timer_interrupt+0x2f/0x40 [ 1103.516433] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1103.516433] softirqs last disabled at (0): [< (null)>] (null) [ 1103.516433] Pid: 128, comm: 99-trinity Not tainted 3.6.0-rc1-bisect2 #80 [ 1103.516433] EIP: 0060:[] EFLAGS: 00000202 CPU: 1 [ 1103.516433] EIP is at get_page_from_freelist+0x1/0x456 [ 1103.516433] EAX: 000200da EBX: 000200da ECX: 00000000 EDX: 00000000 [ 1103.516433] ESI: 00000003 EDI: c196cfa0 EBP: ce857e78 ESP: ce857de0 [ 1103.516433] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.516433] CR0: 8005003b CR2: b7779460 CR3: 0e878000 CR4: 00000690 [ 1103.516433] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.516433] DR6: ffff0ff0 DR7: 00000400 [ 1103.516433] Process 99-trinity (pid: 128, ti=ce856000 task=cf8b2260 task.ti=ce856000) [ 1103.516433] Stack: [ 1103.516433] ce857e78 c109c83a c196cfa0 00000003 00000041 c196c490 00000002 c1823e91 [ 1103.516433] 00000000 ce857e2c 00000000 00000000 cf8b2260 c1913698 ce857e30 00000282 [ 1103.516433] 00000282 000000aa c182c5ad 000200da 00000010 00000282 0000037b 0000037b [ 1103.516433] Call Trace: [ 1103.516433] [] ? __alloc_pages_nodemask+0x11e/0x5f3 [ 1103.516433] [] ? anon_vma_prepare+0x24/0x131 [ 1103.516433] [] do_wp_page+0x40d/0x63c [ 1103.516433] [] ? _raw_spin_lock+0x44/0x4a [ 1103.516433] [] handle_pte_fault+0x758/0x7a7 [ 1103.516433] [] handle_mm_fault+0xa1/0xb0 [ 1103.516433] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.516433] [] do_page_fault+0x336/0x36e [ 1103.516433] [] ? trace_hardirqs_on_thunk+0xc/0x10 [ 1103.516433] [] ? restore_all+0xf/0xf [ 1103.516433] [] ? restore_all+0xf/0xf [ 1103.516433] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.516433] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.516433] [] do_async_page_fault+0x23/0x5f [ 1103.516433] [] error_code+0x6c/0x80 [ 1103.516433] Code: be 00 10 00 00 8a 4d e4 d3 e6 89 d8 e8 e6 f4 00 00 89 f2 e8 b2 c2 fc ff 89 d8 e8 d8 f4 00 00 89 f2 e8 c5 83 22 00 b0 01 eb ce 55 <89> e5 57 56 53 83 ec 5c 89 45 dc 89 55 c0 89 4d e4 8b 45 14 2b [ 1103.516433] Kernel panic - not syncing: softlockup: hung tasks [ 1103.516433] Pid: 128, comm: 99-trinity Not tainted 3.6.0-rc1-bisect2 #80 [ 1103.516433] Call Trace: [ 1103.516433] [] panic+0x97/0x1aa [ 1103.516433] [] watchdog_timer_fn+0x115/0x134 [ 1103.516433] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.516433] [] ? __touch_watchdog+0x1d/0x1d [ 1103.516433] [] run_local_timers+0x8/0x14 [ 1103.516433] [] update_process_times+0x24/0x58 [ 1103.516433] [] tick_periodic+0x66/0x72 [ 1103.516433] [] tick_handle_periodic+0x19/0x61 [ 1103.516433] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.516433] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.516433] [] apic_timer_interrupt+0x36/0x40 [ 1103.516433] [] ? get_page_from_freelist+0x1/0x456 [ 1103.516433] [] ? __alloc_pages_nodemask+0x11e/0x5f3 [ 1103.516433] [] ? anon_vma_prepare+0x24/0x131 [ 1103.516433] [] do_wp_page+0x40d/0x63c [ 1103.516433] [] ? _raw_spin_lock+0x44/0x4a [ 1103.516433] [] handle_pte_fault+0x758/0x7a7 [ 1103.516433] [] handle_mm_fault+0xa1/0xb0 [ 1103.516433] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.516433] [] do_page_fault+0x336/0x36e [ 1103.516433] [] ? trace_hardirqs_on_thunk+0xc/0x10 [ 1103.516433] [] ? restore_all+0xf/0xf [ 1103.516433] [] ? restore_all+0xf/0xf [ 1103.516433] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.516433] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.516433] [] do_async_page_fault+0x23/0x5f [ 1103.516433] [] error_code+0x6c/0x80 [ 1103.516433] Rebooting in 10 seconds.. -- [ 5.018929] Freeing unused kernel memory: 464k freed [ 1104.515153] BUG: soft lockup - CPU#1 stuck for 1025s! [sh:135] [ 1104.515153] irq event stamp: 3314 [ 1104.515153] hardirqs last enabled at (3313): [] restore_all_notrace+0x0/0x18 [ 1104.515153] hardirqs last disabled at (3314): [] apic_timer_interrupt+0x2f/0x40 [ 1104.515153] softirqs last enabled at (3026): [] unix_sock_destructor+0xae/0xb3 [ 1104.515153] softirqs last disabled at (3024): [] unix_sock_destructor+0x9b/0xb3 [ 1104.515153] Pid: 135, comm: sh Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.515153] EIP: 0060:[] EFLAGS: 00000296 CPU: 1 [ 1104.515153] EIP is at generic_file_aio_read+0x27f/0x5f1 [ 1104.524187] [sched_delayed] sched: RT throttling activated [ 1104.515153] EAX: ce688fe0 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 1104.515153] ESI: ffffffff EDI: 00000000 EBP: ce893ecc ESP: ce893e44 [ 1104.515153] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.515153] CR0: 80050033 CR2: b75d4198 CR3: 0e8a5000 CR4: 00000690 [ 1104.515153] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.515153] DR6: ffff0ff0 DR7: 00000400 [ 1104.515153] Process sh (pid: 135, ti=ce892000 task=cf8b3360 task.ti=ce892000) [ 1104.515153] Stack: [ 1104.515153] 00000001 00000000 ce893e68 00000000 00000000 00000000 00000001 ce8a4c84 [ 1104.515153] ce893f10 00000000 00000001 00000fff ffffffff cb4d6210 ce893f54 00000000 [ 1104.515153] ce8a4c00 00000000 cb4d6328 00000001 c10c8890 00000001 00000000 c1bb0d04 [ 1104.515153] Call Trace: [ 1104.515153] [] ? path_put+0x15/0x18 [ 1104.515153] [] do_sync_read+0x9e/0xd6 [ 1104.515153] [] vfs_read+0x8b/0x10d [ 1104.515153] [] ? wait_on_retry_sync_kiocb+0x3f/0x3f [ 1104.515153] [] sys_read+0x3d/0x61 [ 1104.515153] [] syscall_call+0x7/0xb [ 1104.515153] Code: a8 89 df 31 c9 ba 52 04 00 00 b8 3a 8f 81 c1 e8 04 40 fb ff e8 ed 92 5d 00 8b 5d 90 29 fb 89 5d a0 89 fa 8b 45 c0 e8 fc ee ff ff <89> c6 85 c0 75 2c 8b 5d a0 89 5c 24 04 89 3c 24 8b 4d b8 8b 55 [ 1104.515153] Kernel panic - not syncing: softlockup: hung tasks [ 1104.515153] Pid: 135, comm: sh Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.515153] Call Trace: [ 1104.515153] [] panic+0x81/0x194 [ 1104.515153] [] watchdog_timer_fn+0x115/0x134 [ 1104.515153] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.515153] [] ? __lock_acquire+0x1444/0x1455 [ 1104.515153] [] ? __touch_watchdog+0x1d/0x1d [ 1104.515153] [] run_local_timers+0x8/0x14 [ 1104.515153] [] update_process_times+0x24/0x58 [ 1104.515153] [] tick_periodic+0x66/0x72 [ 1104.515153] [] tick_handle_periodic+0x19/0x61 [ 1104.515153] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.515153] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.515153] [] apic_timer_interrupt+0x36/0x40 [ 1104.515153] [] ? perf_event_comm_output+0x31/0xe9 [ 1104.515153] [] ? generic_file_aio_read+0x27f/0x5f1 [ 1104.515153] [] ? path_put+0x15/0x18 [ 1104.515153] [] do_sync_read+0x9e/0xd6 [ 1104.515153] [] vfs_read+0x8b/0x10d [ 1104.515153] [] ? wait_on_retry_sync_kiocb+0x3f/0x3f [ 1104.515153] [] sys_read+0x3d/0x61 [ 1104.515153] [] syscall_call+0x7/0xb [ 1104.515153] Rebooting in 10 seconds.. -- [ 5.793227] Freeing unused kernel memory: 464k freed [ 1104.521267] BUG: soft lockup - CPU#1 stuck for 1023s! [swapper/1:0] [ 1104.521267] irq event stamp: 13836 [ 1104.521267] hardirqs last enabled at (13833): [] restore_all_notrace+0x0/0x18 [ 1104.521267] hardirqs last disabled at (13834): [] apic_timer_interrupt+0x2f/0x40 [ 1104.521267] softirqs last enabled at (13836): [ 1104.527910] [sched_delayed] sched: RT throttling activated [] _local_bh_enable+0xd/0xf [ 1104.528470] softirqs last disabled at (13835): [] irq_enter+0x2c/0x4d [ 1104.528470] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-01297-ga17f29a #84 [ 1104.528470] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.528470] EIP is at native_safe_halt+0x5/0x7 [ 1104.528470] EAX: ce060000 EBX: ffffffff ECX: ce060000 EDX: 00000007 [ 1104.528470] ESI: c195a818 EDI: ffff8b9e EBP: ce05ff68 ESP: ce05ff68 [ 1104.528470] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.528470] CR0: 8005003b CR2: b75f5fb8 CR3: 0ee64000 CR4: 00000690 [ 1104.528470] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.528470] DR6: ffff0ff0 DR7: 00000400 [ 1104.528470] Process swapper/1 (pid: 0, ti=ce05e000 task=ce060000 task.ti=ce05e000) [ 1104.528470] Stack: [ 1104.528470] ce05ff70 c10089d9 ce05ff88 c1009220 3c8ef9e1 4af27f04 853dc1b9 cfff9fa0 [ 1104.528470] ce05ffb4 c165d2ba 00000000 00000000 00000000 00000000 f56cca54 853dc1b9 [ 1104.528470] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.528470] Call Trace: [ 1104.528470] [] default_idle+0x2f/0x55 [ 1104.528470] [] cpu_idle+0xc9/0xfb [ 1104.528470] [] start_secondary+0x292/0x29a [ 1104.528470] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1104.528470] Kernel panic - not syncing: softlockup: hung tasks [ 1104.528470] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-01297-ga17f29a #84 [ 1104.528470] Call Trace: [ 1104.528470] [] panic+0x81/0x194 [ 1104.528470] [] watchdog_timer_fn+0x115/0x134 [ 1104.528470] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.528470] [] ? __touch_watchdog+0x1d/0x1d [ 1104.528470] [] run_local_timers+0x8/0x14 [ 1104.528470] [] update_process_times+0x24/0x58 [ 1104.528470] [] tick_periodic+0x66/0x72 [ 1104.528470] [] tick_handle_periodic+0x19/0x61 [ 1104.528470] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.528470] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.528470] [] apic_timer_interrupt+0x36/0x40 [ 1104.528470] [] ? native_safe_halt+0x5/0x7 [ 1104.528470] [] default_idle+0x2f/0x55 [ 1104.528470] [] cpu_idle+0xc9/0xfb [ 1104.528470] [] start_secondary+0x292/0x29a [ 1104.528470] Rebooting in 10 seconds.. -- [ 1104.519836] [sched_delayed] sched: RT throttling activated [ 1104.520679] BUG: soft lockup - CPU#1 stuck for 1023s! [swapper/1:0] [ 1104.520679] irq event stamp: 6818 [ 1104.520679] hardirqs last enabled at (6815): [] default_idle+0x1f/0x55 [ 1104.520679] hardirqs last disabled at (6816): [] apic_timer_interrupt+0x2f/0x40 [ 1104.520679] softirqs last enabled at (6818): [] _local_bh_enable+0xd/0xf [ 1104.520679] softirqs last disabled at (6817): [] irq_enter+0x2c/0x4d [ 1104.520679] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.520679] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.520679] EIP is at native_safe_halt+0x5/0x7 [ 1104.520679] EAX: ce062240 EBX: ffffffff ECX: ce062240 EDX: 00000007 [ 1104.520679] ESI: c195cbf0 EDI: ffff8ba3 EBP: ce06bf68 ESP: ce06bf68 [ 1104.533562] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.533562] CR0: 8005003b CR2: b754c460 CR3: 0e88f000 CR4: 00000690 [ 1104.533562] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.533562] DR6: ffff0ff0 DR7: 00000400 [ 1104.533562] Process swapper/1 (pid: 0, ti=ce06a000 task=ce062240 task.ti=ce06a000) [ 1104.533562] Stack: [ 1104.533562] ce06bf70 c10089d9 ce06bf88 c1009220 a752cb21 b5b65043 f001826a cfff9fa0 [ 1104.533562] ce06bfb4 c166027a 00000000 00000000 00000000 00000000 60308b05 f001826a [ 1104.533562] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.533562] Call Trace: [ 1104.533562] [] default_idle+0x2f/0x55 [ 1104.533562] [] cpu_idle+0xc9/0xfb [ 1104.533562] [] start_secondary+0x292/0x29a [ 1104.533562] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1104.533562] Kernel panic - not syncing: softlockup: hung tasks [ 1104.533562] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.533562] Call Trace: [ 1104.533562] [] panic+0x81/0x194 [ 1104.533562] [] watchdog_timer_fn+0x115/0x134 [ 1104.533562] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.533562] [] ? __touch_watchdog+0x1d/0x1d [ 1104.533562] [] run_local_timers+0x8/0x14 [ 1104.533562] [] update_process_times+0x24/0x58 [ 1104.533562] [] tick_periodic+0x66/0x72 [ 1104.533562] [] tick_handle_periodic+0x19/0x61 [ 1104.533562] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.533562] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.533562] [] apic_timer_interrupt+0x36/0x40 [ 1104.533562] [] ? native_safe_halt+0x5/0x7 [ 1104.533562] [] default_idle+0x2f/0x55 [ 1104.533562] [] cpu_idle+0xc9/0xfb [ 1104.533562] [] start_secondary+0x292/0x29a [ 1104.533562] Rebooting in 10 seconds.. -- [ 1104.522067] [sched_delayed] sched: RT throttling activated [ 1104.522043] BUG: soft lockup - CPU#1 stuck for 1024s! [99-trinity:129] [ 1104.522043] irq event stamp: 2948 [ 1104.522043] hardirqs last enabled at (2947): [] restore_all_notrace+0x0/0x18 [ 1104.522043] hardirqs last disabled at (2948): [] apic_timer_interrupt+0x2f/0x40 [ 1104.522043] softirqs last enabled at (2348): [] unix_sock_destructor+0xae/0xb3 [ 1104.522043] softirqs last disabled at (2346): [] unix_sock_destructor+0x9b/0xb3 [ 1104.522043] Pid: 129, comm: 99-trinity Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.522043] EIP: 0060:[] EFLAGS: 00000246 CPU: 1 [ 1104.522043] EIP is at system_call+0x0/0x45 [ 1104.522043] EAX: 000000c0 EBX: 00000000 ECX: 0000c404 EDX: 00000005 [ 1104.522043] ESI: 00000802 EDI: 00000003 EBP: 00000000 ESP: ce881fe4 [ 1104.522043] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 [ 1104.522043] CR0: 80050033 CR2: b752b267 CR3: 0e867000 CR4: 00000690 [ 1104.522043] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.522043] DR6: ffff0ff0 DR7: 00000400 [ 1104.522043] Process 99-trinity (pid: 129, ti=ce880000 task=cf8b3360 task.ti=ce880000) [ 1104.522043] Stack: [ 1104.522043] b773fd63 00000073 00000246 bfc9319c 0000007b cccccccc cccccccc [ 1104.522043] Call Trace: [ 1104.522043] Code: 67 9f ff 5a 59 58 fa 8d b6 00 00 00 00 e8 3d a2 c3 ff 8b 4d 08 81 e1 2e fe 00 00 0f 85 0a 01 00 00 eb 7e 90 8d b4 26 00 00 00 00 <50> fc 0f a8 0f a0 06 1e 50 55 57 56 52 51 53 ba 7b 00 00 00 8e [ 1104.522043] Kernel panic - not syncing: softlockup: hung tasks [ 1104.522043] Pid: 129, comm: 99-trinity Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.522043] Call Trace: [ 1104.522043] [] panic+0x81/0x194 [ 1104.522043] [] watchdog_timer_fn+0x115/0x134 [ 1104.522043] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.522043] [] ? __touch_watchdog+0x1d/0x1d [ 1104.522043] [] run_local_timers+0x8/0x14 [ 1104.522043] [] update_process_times+0x24/0x58 [ 1104.522043] [] tick_periodic+0x66/0x72 [ 1104.522043] [] tick_handle_periodic+0x19/0x61 [ 1104.522043] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.522043] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.522043] [] apic_timer_interrupt+0x36/0x40 [ 1104.522043] [] ? resume_userspace+0x30/0x30 [ 1104.522043] Rebooting in 10 seconds.. -- [ 5.773899] Freeing unused kernel memory: 464k freed [ 1104.518996] BUG: soft lockup - CPU#1 stuck for 1023s! [sh:134] [ 1104.518996] irq event stamp: 1020 [ 1104.518996] hardirqs last enabled at (1019): [] __mutex_unlock_slowpath+0x114/0x12c [ 1104.518996] hardirqs last disabled at (1020): [] apic_timer_interrupt+0x2f/0x40 [ 1104.518996] softirqs last enabled at (0): [] copy_process+0x471/0xf82 [ 1104.518996] softirqs last disabled at (0): [< (null)>] (null) [ 1104.518996] Pid: 134, comm: sh Not tainted 3.5.0-bisect2-00646-g5b160bd #85 [ 1104.518996] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.526690] [sched_delayed] sched: RT throttling activated [ 1104.518996] EIP is at lock_acquire+0xb4/0xc0 [ 1104.518996] EAX: 00000286 EBX: cf8966c0 ECX: 000001e3 EDX: 000001e3 [ 1104.518996] ESI: 00000000 EDI: cb732cdc EBP: ce869e3c ESP: ce869e10 [ 1104.518996] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.518996] CR0: 8005003b CR2: 08124f6c CR3: 0e86c000 CR4: 00000690 [ 1104.518996] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.518996] DR6: ffff0ff0 DR7: 00000400 [ 1104.518996] Process sh (pid: 134, ti=ce868000 task=cf8966c0 task.ti=ce868000) [ 1104.518996] Stack: [ 1104.518996] 00000000 00000002 00000000 00000000 c10d0b7e 00000000 00000000 00000286 [ 1104.518996] cb732ccc fffffffe cb732ccc ce869e58 c166d29a 00000000 00000002 00000000 [ 1104.518996] c10d0b7e cb732c80 ce869e70 c10d0b7e fffffffe ce869ef4 fffffffe 00000041 [ 1104.518996] Call Trace: [ 1104.518996] [] ? dput+0x37/0x143 [ 1104.518996] [] _raw_spin_lock+0x3d/0x4a [ 1104.518996] [] ? dput+0x37/0x143 [ 1104.518996] [] dput+0x37/0x143 [ 1104.518996] [] path_put+0xe/0x18 [ 1104.518996] [] terminate_walk+0x10/0x86 [ 1104.518996] [] path_lookupat+0x1cd/0x535 [ 1104.518996] [] do_path_lookup+0x1f/0x8e [ 1104.518996] [] user_path_at_empty+0x41/0x66 [ 1104.518996] [] ? do_page_fault+0x312/0x36e [ 1104.518996] [] ? kmem_cache_alloc+0xab/0xcb [ 1104.518996] [] user_path_at+0x1a/0x1f [ 1104.518996] [] sys_faccessat+0x90/0x150 [ 1104.518996] [] ? restore_all+0xf/0xf [ 1104.518996] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.518996] [] sys_access+0x20/0x22 [ 1104.518996] [] syscall_call+0x7/0xb [ 1104.518996] Code: 45 08 89 04 24 8b 4d ec 89 f2 89 f8 e8 57 e7 ff ff c7 83 54 04 00 00 00 00 00 00 83 3d 6c b4 8f c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 83 c4 20 5b 5e 5f 5d c3 55 89 e5 57 56 53 83 ec 1c [ 1104.518996] Kernel panic - not syncing: softlockup: hung tasks [ 1104.518996] Pid: 134, comm: sh Not tainted 3.5.0-bisect2-00646-g5b160bd #85 [ 1104.518996] Call Trace: [ 1104.518996] [] panic+0x81/0x194 [ 1104.518996] [] watchdog_timer_fn+0x115/0x134 [ 1104.518996] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.518996] [] ? __touch_watchdog+0x1d/0x1d [ 1104.518996] [] run_local_timers+0x8/0x14 [ 1104.518996] [] update_process_times+0x24/0x58 [ 1104.518996] [] tick_periodic+0x66/0x72 [ 1104.518996] [] tick_handle_periodic+0x19/0x61 [ 1104.518996] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.518996] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.518996] [] apic_timer_interrupt+0x36/0x40 [ 1104.518996] [] ? lock_acquire+0xb4/0xc0 [ 1104.518996] [] ? dput+0x37/0x143 [ 1104.518996] [] _raw_spin_lock+0x3d/0x4a [ 1104.518996] [] ? dput+0x37/0x143 [ 1104.518996] [] dput+0x37/0x143 [ 1104.518996] [] path_put+0xe/0x18 [ 1104.518996] [] terminate_walk+0x10/0x86 [ 1104.518996] [] path_lookupat+0x1cd/0x535 [ 1104.518996] [] do_path_lookup+0x1f/0x8e [ 1104.518996] [] user_path_at_empty+0x41/0x66 [ 1104.518996] [] ? do_page_fault+0x312/0x36e [ 1104.518996] [] ? kmem_cache_alloc+0xab/0xcb [ 1104.518996] [] user_path_at+0x1a/0x1f [ 1104.518996] [] sys_faccessat+0x90/0x150 [ 1104.518996] [] ? restore_all+0xf/0xf [ 1104.518996] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.518996] [] sys_access+0x20/0x22 [ 1104.518996] [] syscall_call+0x7/0xb [ 1104.518996] Rebooting in 10 seconds.. -- [ 1104.515042] [sched_delayed] sched: RT throttling activated [ 1104.515017] BUG: soft lockup - CPU#1 stuck for 1024s! [99-trinity:129] [ 1104.515017] irq event stamp: 1404 [ 1104.515017] hardirqs last enabled at (1403): [] restore_all_notrace+0x0/0x18 [ 1104.515017] hardirqs last disabled at (1404): [] apic_timer_interrupt+0x2f/0x40 [ 1104.515017] softirqs last enabled at (0): [] copy_process+0x471/0xf82 [ 1104.515017] softirqs last disabled at (0): [< (null)>] (null) [ 1104.515017] Pid: 129, comm: 99-trinity Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.515017] EIP: 0073:[] EFLAGS: 00000202 CPU: 1 [ 1104.515017] EIP is at 0xb77231e7 [ 1104.515017] EAX: 08124f88 EBX: b7735ff4 ECX: b75466c0 EDX: 0805db8c [ 1104.515017] ESI: b7736918 EDI: 00000000 EBP: 08124f50 ESP: bfba3268 [ 1104.515017] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b [ 1104.515017] Process 99-trinity (pid: 129, ti=ce890000 task=cf8ad5a0 task.ti=ce890000) [ 1104.515017] [ 1104.515017] Kernel panic - not syncing: softlockup: hung tasks [ 1104.515017] Pid: 129, comm: 99-trinity Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.515017] Call Trace: [ 1104.515017] [] panic+0x81/0x194 [ 1104.515017] [] watchdog_timer_fn+0x115/0x134 [ 1104.515017] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.515017] [] ? __touch_watchdog+0x1d/0x1d [ 1104.515017] [] run_local_timers+0x8/0x14 [ 1104.515017] [] update_process_times+0x24/0x58 [ 1104.515017] [] tick_periodic+0x66/0x72 [ 1104.515017] [] tick_handle_periodic+0x19/0x61 [ 1104.515017] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.515017] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.515017] [] apic_timer_interrupt+0x36/0x40 [ 1104.515017] Rebooting in 10 seconds.. -- [ 4.455784] Freeing unused kernel memory: 464k freed [ 1103.514772] BUG: soft lockup - CPU#1 stuck for 1024s! [init:1] [ 1103.524384] [sched_delayed] sched: RT throttling activated [ 1103.514772] irq event stamp: 759084 [ 1103.514772] hardirqs last enabled at (759083): [] _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.514772] hardirqs last disabled at (759084): [] apic_timer_interrupt+0x2f/0x40 [ 1103.514772] softirqs last enabled at (754272): [] unix_sock_destructor+0xae/0xb3 [ 1103.514772] softirqs last disabled at (754270): [] unix_sock_destructor+0x9b/0xb3 [ 1103.514772] [ 1103.514772] Pid: 1, comm: init Not tainted 3.5.0-bisect2-00242-g3992c03 #88 [ 1103.514772] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.514772] EIP is at _raw_spin_unlock_irqrestore+0x51/0x63 [ 1103.514772] EAX: 00000282 EBX: 00000282 ECX: 00000000 EDX: 00000007 [ 1103.514772] ESI: ce8583e4 EDI: 00000001 EBP: ce03ddfc ESP: ce03ddf4 [ 1103.514772] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.514772] CR0: 80050033 CR2: 080b1bc0 CR3: 0e80b000 CR4: 00000690 [ 1103.514772] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.514772] DR6: ffff0ff0 DR7: 00000400 [ 1103.514772] Process init (pid: 1, ti=ce03c000 task=ce040000 task.ti=ce03c000) [ 1103.514772] Stack: [ 1103.514772] 00000000 ce858000 ce03de20 c104842a 00000282 ce8583e4 0cb64000 00000000 [ 1103.514772] ce858000 00000000 00000007 ce03de28 c1048470 ce03de38 c102ec42 ce858000 [ 1103.514772] 00000000 ce03def4 c102fb22 003f0011 00000000 ce040470 00000000 ce040000 [ 1103.514772] Call Trace: [ 1103.514772] [] try_to_wake_up+0x248/0x266 [ 1103.514772] [] wake_up_state+0xa/0xc [ 1103.514772] [] ptrace_resume+0x76/0x7c [ 1103.514772] [] ptrace_request+0x368/0x4f8 [ 1103.514772] [] ? __lock_acquire+0x1444/0x1455 [ 1103.514772] [] ? __lock_acquire+0x1444/0x1455 [ 1103.514772] [] ? lock_release_non_nested+0x7e/0x21b [ 1103.514772] [] ? __lock_acquire+0x1444/0x1455 [ 1103.514772] [] arch_ptrace+0x474/0x48a [ 1103.514772] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.514772] [] ? trace_hardirqs_on+0xb/0xd [ 1103.514772] [] ? wait_task_inactive+0x6f/0xc1 [ 1103.514772] [] ? ptrace_check_attach+0xb4/0xc4 [ 1103.514772] [] sys_ptrace+0x34f/0x379 [ 1103.514772] [] ? sys_ptrace+0x6e/0x379 [ 1103.514772] [] sysenter_do_call+0x12/0x38 [ 1103.514772] Code: 3d 6c b4 8f c1 00 74 1d 89 d8 50 9d 8d 74 26 00 e8 7d 5d 9f ff eb 18 e8 4c 5c 9f ff 83 3d 6c b4 8f c1 00 75 02 0f 0b 89 d8 50 9d <8d> 74 26 00 89 e0 25 00 e0 ff ff ff 48 14 5b 5e 5d c3 55 89 e5 [ 1103.514772] Kernel panic - not syncing: softlockup: hung tasks [ 1103.514772] Pid: 1, comm: init Not tainted 3.5.0-bisect2-00242-g3992c03 #88 [ 1103.514772] Call Trace: [ 1103.514772] [] panic+0x81/0x194 [ 1103.514772] [] watchdog_timer_fn+0x115/0x134 [ 1103.514772] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.514772] [] ? __touch_watchdog+0x1d/0x1d [ 1103.514772] [] run_local_timers+0x8/0x14 [ 1103.514772] [] update_process_times+0x24/0x58 [ 1103.514772] [] tick_periodic+0x66/0x72 [ 1103.514772] [] tick_handle_periodic+0x19/0x61 [ 1103.514772] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.514772] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.514772] [] apic_timer_interrupt+0x36/0x40 [ 1103.514772] [] ? _raw_spin_unlock_irqrestore+0x51/0x63 [ 1103.514772] [] try_to_wake_up+0x248/0x266 [ 1103.514772] [] wake_up_state+0xa/0xc [ 1103.514772] [] ptrace_resume+0x76/0x7c [ 1103.514772] [] ptrace_request+0x368/0x4f8 [ 1103.514772] [] ? __lock_acquire+0x1444/0x1455 [ 1103.514772] [] ? __lock_acquire+0x1444/0x1455 [ 1103.514772] [] ? lock_release_non_nested+0x7e/0x21b [ 1103.514772] [] ? __lock_acquire+0x1444/0x1455 [ 1103.514772] [] arch_ptrace+0x474/0x48a [ 1103.514772] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1103.514772] [] ? trace_hardirqs_on+0xb/0xd [ 1103.514772] [] ? wait_task_inactive+0x6f/0xc1 [ 1103.514772] [] ? ptrace_check_attach+0xb4/0xc4 [ 1103.514772] [] sys_ptrace+0x34f/0x379 [ 1103.514772] [] ? sys_ptrace+0x6e/0x379 [ 1103.514772] [] sysenter_do_call+0x12/0x38 [ 1103.514772] Rebooting in 10 seconds.. -- [ 5.165476] Freeing unused kernel memory: 464k freed [ 1104.519906] BUG: soft lockup - CPU#1 stuck for 1024s! [swapper/1:0] [ 1104.519906] irq event stamp: 6640 [ 1104.519906] hardirqs last enabled at (6637): [] restore_all_notrace+0x0/0x18 [ 1104.519906] hardirqs last disabled at (6638): [] apic_timer_interrupt+0x2f/0x40 [ 1104.519906] softirqs last enabled at (6640): [] _local_bh_enable+0xd/0xf [ 1104.519906] softirqs last disabled at (6639): [] irq_enter+0x2c/0x4d [ 1104.519906] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.519906] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.519906] EIP is at native_safe_halt+0x5/0x7 [ 1104.519906] EAX: ce062240 EBX: ffffffff ECX: ce062240 EDX: 00000007 [ 1104.519906] ESI: c195cbf0 EDI: ffff8b9e EBP: ce06bf68 ESP: ce06bf68 [ 1104.519906] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.519906] CR0: 8005003b CR2: b76acc50 CR3: 019dc000 CR4: 00000690 [ 1104.519906] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.519906] DR6: ffff0ff0 DR7: 00000400 [ 1104.519906] Process swapper/1 (pid: 0, ti=ce06a000 task=ce062240 task.ti=ce06a000) [ 1104.519906] Stack: [ 1104.519906] ce06bf70 c10089d9 ce06bf88 c1009220 7b380d04 899b9224 c3e68548 cfff9fa0 [ 1104.519906] ce06bfb4 c166027a 00000000 00000000 00000000 00000000 34158de5 c3e68548 [ 1104.519906] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.519906] Call Trace: [ 1104.519906] [] default_idle+0x2f/0x55 [ 1104.519906] [] cpu_idle+0xc9/0xfb [ 1104.519906] [] start_secondary+0x292/0x29a [ 1104.519906] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1104.519906] Kernel panic - not syncing: softlockup: hung tasks [ 1104.519906] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1104.519906] Call Trace: [ 1104.519906] [] panic+0x81/0x194 [ 1104.519906] [] watchdog_timer_fn+0x115/0x134 [ 1104.519906] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.519906] [] ? __touch_watchdog+0x1d/0x1d [ 1104.519906] [] run_local_timers+0x8/0x14 [ 1104.519906] [] update_process_times+0x24/0x58 [ 1104.519906] [] tick_periodic+0x66/0x72 [ 1104.519906] [] tick_handle_periodic+0x19/0x61 [ 1104.519906] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.519906] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.519906] [] apic_timer_interrupt+0x36/0x40 [ 1104.519906] [] ? native_safe_halt+0x5/0x7 [ 1104.519906] [] default_idle+0x2f/0x55 [ 1104.519906] [] cpu_idle+0xc9/0xfb [ 1104.519906] [] start_secondary+0x292/0x29a [ 1104.519906] Rebooting in 10 seconds.. -- [ 4.683031] Freeing unused kernel memory: 460k freed [ 1103.515937] BUG: soft lockup - CPU#1 stuck for 1024s! [chroot:134] [ 1103.515937] irq event stamp: 3200 [ 1103.515937] hardirqs last enabled at (3199): [] debug_check_no_locks_freed+0x14b/0x167 [ 1103.515937] hardirqs last disabled at (3200): [] apic_timer_interrupt+0x2f/0x40 [ 1103.515937] softirqs last enabled at (3056): [] unix_sock_destructor+0xae/0xb3 [ 1103.515937] softirqs last disabled at (3054): [] unix_sock_destructor+0x9b/0xb3 [ 1103.515937] Pid: 134, comm: chroot Not tainted 3.6.0-rc1-bisect2 #80 [ 1103.515937] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1103.515937] EIP is at lock_is_held+0x9f/0xb2 [ 1103.515937] EAX: 00000282 EBX: 00000001 ECX: 00000000 EDX: 00000000 [ 1103.515937] ESI: ce8b2260 EDI: c1913698 EBP: ce8c7f0c ESP: ce8c7efc [ 1103.515937] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.515937] CR0: 8005003b CR2: 080b6730 CR3: 0e8be000 CR4: 00000690 [ 1103.525323] [sched_delayed] sched: RT throttling activated [ 1103.515937] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.515937] DR6: ffff0ff0 DR7: 00000400 [ 1103.515937] Process chroot (pid: 134, ti=ce8c6000 task=ce8b2260 task.ti=ce8c6000) [ 1103.515937] Stack: [ 1103.515937] 00000282 000001c5 c182dba3 00000000 ce8c7f34 c104bb01 bfc5c5fc ce8c7f34 [ 1103.515937] c10af78b 00000001 00000002 00000008 ce8b2260 bfc5c5fc ce8c7f48 c10cc958 [ 1103.515937] c7b3a300 00000006 00000001 ce8c7f88 c10cde5f 00000002 00000001 00000000 [ 1103.515937] Call Trace: [ 1103.515937] [] __might_sleep+0xa1/0x1cd [ 1103.515937] [] ? might_fault+0x90/0x99 [ 1103.515937] [] count.constprop.35+0x78/0x82 [ 1103.515937] [] do_execve+0x310/0x456 [ 1103.515937] [] ? do_execve+0xeb/0x456 [ 1103.515937] [] sys_execve+0x31/0x53 [ 1103.515937] [] ptregs_execve+0x12/0x20 [ 1103.515937] [] ? sysenter_do_call+0x12/0x38 [ 1103.515937] Code: 7c e2 31 c9 eb 05 b9 01 00 00 00 64 a1 28 06 9e c1 c7 80 58 04 00 00 00 00 00 00 83 3d 8c 20 91 c1 00 75 02 0f 0b 8b 45 f0 50 9d <8d> 74 26 00 eb 05 b9 01 00 00 00 89 c8 5b 5b 5e 5f 5d c3 55 89 [ 1103.515937] Kernel panic - not syncing: softlockup: hung tasks [ 1103.515937] Pid: 134, comm: chroot Not tainted 3.6.0-rc1-bisect2 #80 [ 1103.515937] Call Trace: [ 1103.515937] [] panic+0x97/0x1aa [ 1103.515937] [] watchdog_timer_fn+0x115/0x134 [ 1103.515937] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.515937] [] ? __lock_acquire+0x1444/0x1455 [ 1103.515937] [] ? __touch_watchdog+0x1d/0x1d [ 1103.515937] [] run_local_timers+0x8/0x14 [ 1103.515937] [] update_process_times+0x24/0x58 [ 1103.515937] [] tick_periodic+0x66/0x72 [ 1103.515937] [] tick_handle_periodic+0x19/0x61 [ 1103.515937] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.515937] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.515937] [] apic_timer_interrupt+0x36/0x40 [ 1103.515937] [] ? lock_is_held+0x9f/0xb2 [ 1103.515937] [] __might_sleep+0xa1/0x1cd [ 1103.515937] [] ? might_fault+0x90/0x99 [ 1103.515937] [] count.constprop.35+0x78/0x82 [ 1103.515937] [] do_execve+0x310/0x456 [ 1103.515937] [] ? do_execve+0xeb/0x456 [ 1103.515937] [] sys_execve+0x31/0x53 [ 1103.515937] [] ptregs_execve+0x12/0x20 [ 1103.515937] [] ? sysenter_do_call+0x12/0x38 [ 1103.515937] Rebooting in 10 seconds.. -- [ 5.214556] Freeing unused kernel memory: 464k freed [ 1104.514802] BUG: soft lockup - CPU#1 stuck for 1025s! [init:1] [ 1104.514802] irq event stamp: 756026 [ 1104.514802] hardirqs last enabled at (756025): [] _raw_spin_unlock_irqrestore+0x42/0x63 [ 1104.514802] hardirqs last disabled at (756026): [] apic_timer_interrupt+0x2f/0x40 [ 1104.514802] softirqs last enabled at (751184): [] unix_sock_destructor+0xae/0xb3 [ 1104.514802] softirqs last disabled at (751182): [] unix_sock_destructor+0x9b/0xb3 [ 1104.514802] Pid: 1, comm: init Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 [ 1104.514802] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.514802] EIP is at _raw_spin_unlock_irqrestore+0x51/0x63 [ 1104.514802] EAX: 00000286 EBX: 00000286 ECX: 00000000 EDX: 00000007 [ 1104.514802] ESI: ce861504 EDI: 00000001 EBP: ce03ddfc ESP: ce03ddf4 [ 1104.514802] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.524377] [sched_delayed] sched: RT throttling activated [ 1104.514802] CR0: 80050033 CR2: b760b600 CR3: 0ee6a000 CR4: 00000690 [ 1104.514802] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.514802] DR6: ffff0ff0 DR7: 00000400 [ 1104.514802] Process init (pid: 1, ti=ce03c000 task=ce040000 task.ti=ce03c000) [ 1104.514802] Stack: [ 1104.514802] 00000000 ce861120 ce03de20 c104840a 00000286 ce861504 0cb62000 00000000 [ 1104.514802] ce861120 00000000 00000007 ce03de28 c1048450 ce03de38 c102ec22 ce861120 [ 1104.514802] 00000000 ce03def4 c102fb02 003f0011 00000000 ce040470 00000000 ce040000 [ 1104.514802] Call Trace: [ 1104.514802] [] try_to_wake_up+0x248/0x266 [ 1104.514802] [] wake_up_state+0xa/0xc [ 1104.514802] [] ptrace_resume+0x76/0x7c [ 1104.514802] [] ptrace_request+0x368/0x4f8 [ 1104.514802] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514802] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514802] [] ? lock_release_non_nested+0x7e/0x21b [ 1104.514802] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514802] [] arch_ptrace+0x474/0x48a [ 1104.514802] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1104.514802] [] ? trace_hardirqs_on+0xb/0xd [ 1104.514802] [] ? wait_task_inactive+0x6f/0xc1 [ 1104.514802] [] ? ptrace_check_attach+0xb4/0xc4 [ 1104.514802] [] sys_ptrace+0x34f/0x379 [ 1104.514802] [] ? sys_ptrace+0x6e/0x379 [ 1104.514802] [] sysenter_do_call+0x12/0x38 [ 1104.514802] Code: 3d 6c d4 8f c1 00 74 1d 89 d8 50 9d 8d 74 26 00 e8 6d 58 9f ff eb 18 e8 3c 57 9f ff 83 3d 6c d4 8f c1 00 75 02 0f 0b 89 d8 50 9d <8d> 74 26 00 89 e0 25 00 e0 ff ff ff 48 14 5b 5e 5d c3 55 89 e5 [ 1104.514802] Kernel panic - not syncing: softlockup: hung tasks [ 1104.514802] Pid: 1, comm: init Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 [ 1104.514802] Call Trace: [ 1104.514802] [] panic+0x81/0x194 [ 1104.514802] [] watchdog_timer_fn+0x115/0x134 [ 1104.514802] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.514802] [] ? __touch_watchdog+0x1d/0x1d [ 1104.514802] [] run_local_timers+0x8/0x14 [ 1104.514802] [] update_process_times+0x24/0x58 [ 1104.514802] [] tick_periodic+0x66/0x72 [ 1104.514802] [] tick_handle_periodic+0x19/0x61 [ 1104.514802] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.514802] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.514802] [] apic_timer_interrupt+0x36/0x40 [ 1104.514802] [] ? _raw_spin_unlock_irqrestore+0x51/0x63 [ 1104.514802] [] try_to_wake_up+0x248/0x266 [ 1104.514802] [] wake_up_state+0xa/0xc [ 1104.514802] [] ptrace_resume+0x76/0x7c [ 1104.514802] [] ptrace_request+0x368/0x4f8 [ 1104.514802] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514802] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514802] [] ? lock_release_non_nested+0x7e/0x21b [ 1104.514802] [] ? __lock_acquire+0x1444/0x1455 [ 1104.514802] [] arch_ptrace+0x474/0x48a [ 1104.514802] [] ? _raw_spin_unlock_irqrestore+0x42/0x63 [ 1104.514802] [] ? trace_hardirqs_on+0xb/0xd [ 1104.514802] [] ? wait_task_inactive+0x6f/0xc1 [ 1104.514802] [] ? ptrace_check_attach+0xb4/0xc4 [ 1104.514802] [] sys_ptrace+0x34f/0x379 [ 1104.514802] [] ? sys_ptrace+0x6e/0x379 [ 1104.514802] [] sysenter_do_call+0x12/0x38 [ 1104.514802] Rebooting in 10 seconds.. -- [ 5.704992] Freeing unused kernel memory: 464k freed [ 1104.527176] BUG: soft lockup - CPU#1 stuck for 1023s! [swapper/1:0] [ 1104.527176] irq event stamp: 14280 [ 1104.527176] hardirqs last enabled at (14277): [] restore_all_notrace+0x0/0x18 [ 1104.527176] hardirqs last disabled at (14278): [] apic_timer_interrupt+0x2f/0x40 [ 1104.527176] softirqs last enabled at (14280): [] _local_bh_enable+0xd/0xf [ 1104.527176] softirqs last disabled at (14279): [] irq_enter+0x2c/0x4d [ 1104.527176] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-01297-ga17f29a #84 [ 1104.527176] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1104.527176] EIP is at native_safe_halt+0x5/0x7 [ 1104.527176] EAX: ce060000 EBX: ffffffff ECX: ce060000 EDX: 00000007 [ 1104.527176] ESI: c195a818 EDI: ffff8b9e EBP: ce05ff68 ESP: ce05ff68 [ 1104.527176] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.527176] CR0: 8005003b CR2: b76c8000 CR3: 0ee58000 CR4: 00000690 [ 1104.527176] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.527176] DR6: ffff0ff0 DR7: 00000400 [ 1104.527176] Process swapper/1 (pid: 0, ti=ce05e000 task=ce060000 task.ti=ce05e000) [ 1104.527176] Stack: [ 1104.527176] ce05ff70 c10089d9 ce05ff88 c1009220 a6cdc943 b5314e64 ef7c5be0 cfff9fa0 [ 1104.527176] ce05ffb4 c165d2ba 00000000 00000000 00000000 00000000 5fab647c ef7c5be0 [ 1104.527176] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.527176] Call Trace: [ 1104.527176] [] default_idle+0x2f/0x55 [ 1104.527176] [] cpu_idle+0xc9/0xfb [ 1104.527176] [] start_secondary+0x292/0x29a [ 1104.527176] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1104.527176] Kernel panic - not syncing: softlockup: hung tasks [ 1104.527176] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-01297-ga17f29a #84 [ 1104.527176] Call Trace: [ 1104.527176] [] panic+0x81/0x194 [ 1104.527176] [] watchdog_timer_fn+0x115/0x134 [ 1104.527176] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.527176] [] ? __touch_watchdog+0x1d/0x1d [ 1104.527176] [] run_local_timers+0x8/0x14 [ 1104.527176] [] update_process_times+0x24/0x58 [ 1104.527176] [] tick_periodic+0x66/0x72 [ 1104.527176] [] tick_handle_periodic+0x19/0x61 [ 1104.527176] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.527176] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.527176] [] apic_timer_interrupt+0x36/0x40 [ 1104.527176] [] ? native_safe_halt+0x5/0x7 [ 1104.527176] [] default_idle+0x2f/0x55 [ 1104.527176] [] cpu_idle+0xc9/0xfb [ 1104.527176] [] start_secondary+0x292/0x29a [ 1104.527176] Rebooting in 10 seconds.. -- [ 6.525425] Freeing unused kernel memory: 464k freed [ 1105.523006] BUG: soft lockup - CPU#1 stuck for 1023s! [99-trinity:133] [ 1105.523006] irq event stamp: 632 [ 1105.523006] hardirqs last enabled at (631): [] __mutex_unlock_slowpath+0x114/0x12c [ 1105.523006] hardirqs last disabled at (632): [] apic_timer_interrupt+0x2f/0x40 [ 1105.523006] softirqs last enabled at (0): [] copy_process+0x471/0xf82 [ 1105.523006] softirqs last disabled at (0): [< (null)>] (null) [ 1105.523006] Pid: 133, comm: 99-trinity Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1105.523006] EIP: 0060:[] EFLAGS: 00000202 CPU: 1 [ 1105.523006] EIP is at lockdep_trace_alloc+0x3/0xe0 [ 1105.523006] EAX: 000280da EBX: 000280da ECX: c1958810 EDX: 00000000 [ 1105.523006] ESI: 0000000a EDI: ce807680 EBP: ce8a3d5c ESP: ce8a3d5c [ 1105.523006] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1105.523006] CR0: 80050033 CR2: b75e5baf CR3: 0e868000 CR4: 00000690 [ 1105.523006] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1105.523006] DR6: ffff0ff0 DR7: 00000400 [ 1105.523006] Process 99-trinity (pid: 133, ti=ce8a2000 task=ce8a8000 task.ti=ce8a2000) [ 1105.523006] Stack: [ 1105.523006] ce8a3dfc c109a6d2 ce8a8000 00000040 00000006 c1bc8eb4 ce8a8458 ce8a8000 [ 1105.523006] ce8a3da0 c1063a52 00000378 00000007 00000000 00000000 ce8a8000 c1900ad8 [ 1105.523006] ce8a3db4 00000286 00000286 000000aa c181b224 00000000 ce8a3ddc 00000286 [ 1105.523006] Call Trace: [ 1105.523006] [] __alloc_pages_nodemask+0x78/0x5db [ 1105.523006] [] ? mark_held_locks+0xa7/0xce [ 1105.523006] [] ? anon_vma_prepare+0x24/0x131 [ 1105.523006] [] handle_pte_fault+0x16b/0x74c [ 1105.523006] [] ? kmap_atomic_prot+0x11f/0x133 [ 1105.523006] [] handle_mm_fault+0x95/0xa4 [ 1105.523006] [] __get_user_pages+0x264/0x39e [ 1105.523006] [] get_user_pages+0x4b/0x53 [ 1105.523006] [] get_arg_page+0x4a/0xa9 [ 1105.523006] [] copy_strings+0xfe/0x1c6 [ 1105.523006] [] copy_strings_kernel+0x1c/0x2b [ 1105.523006] [] do_execve+0x343/0x456 [ 1105.523006] [] ? do_execve+0xeb/0x456 [ 1105.523006] [] sys_execve+0x31/0x53 [ 1105.523006] [] ptregs_execve+0x12/0x20 [ 1105.523006] [] ? sysenter_do_call+0x12/0x38 [ 1105.523006] Code: 3d 1c b7 95 c1 01 74 16 ba 9a 0a 00 00 b8 8b 35 81 c1 e8 7e 26 fc ff c6 05 1c b7 95 c1 01 64 ff 05 bc dc 9c c1 5b 5d c3 55 89 e5 <57> 56 53 89 c6 64 8b 1d 28 c6 9c c1 83 bb 54 04 00 00 00 0f 85 [ 1105.523006] Kernel panic - not syncing: softlockup: hung tasks [ 1105.523006] Pid: 133, comm: 99-trinity Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1105.523006] Call Trace: [ 1105.523006] [] panic+0x81/0x194 [ 1105.523006] [] watchdog_timer_fn+0x115/0x134 [ 1105.523006] [] hrtimer_run_queues+0x14c/0x1c6 [ 1105.523006] [] ? __touch_watchdog+0x1d/0x1d [ 1105.523006] [] run_local_timers+0x8/0x14 [ 1105.523006] [] update_process_times+0x24/0x58 [ 1105.523006] [] tick_periodic+0x66/0x72 [ 1105.523006] [] tick_handle_periodic+0x19/0x61 [ 1105.523006] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1105.523006] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1105.523006] [] apic_timer_interrupt+0x36/0x40 [ 1105.523006] [] ? lockdep_trace_alloc+0x3/0xe0 [ 1105.523006] [] __alloc_pages_nodemask+0x78/0x5db [ 1105.523006] [] ? mark_held_locks+0xa7/0xce [ 1105.523006] [] ? anon_vma_prepare+0x24/0x131 [ 1105.581916] [] handle_pte_fault+0x16b/0x74c [ 1105.581916] [] ? kmap_atomic_prot+0x11f/0x133 [ 1105.581916] [] handle_mm_fault+0x95/0xa4 [ 1105.581916] [] __get_user_pages+0x264/0x39e [ 1105.581916] [] get_user_pages+0x4b/0x53 [ 1105.581916] [] get_arg_page+0x4a/0xa9 [ 1105.581916] [] copy_strings+0xfe/0x1c6 [ 1105.581916] [] copy_strings_kernel+0x1c/0x2b [ 1105.581916] [] do_execve+0x343/0x456 [ 1105.581916] [] ? do_execve+0xeb/0x456 [ 1105.581916] [] sys_execve+0x31/0x53 [ 1105.581916] [] ptregs_execve+0x12/0x20 [ 1105.581916] [] ? sysenter_do_call+0x12/0x38 [ 1105.581916] Rebooting in 10 seconds.. -- [ 4.563976] Freeing unused kernel memory: 464k freed [ 1103.520656] BUG: soft lockup - CPU#1 stuck for 1023s! [shutdown:135] [ 1103.520656] irq event stamp: 2580 [ 1103.520656] hardirqs last enabled at (2579): [] mutex_lock_nested+0x2da/0x308 [ 1103.520656] hardirqs last disabled at (2580): [] apic_timer_interrupt+0x2f/0x40 [ 1103.520656] softirqs last enabled at (0): [] copy_process+0x471/0xf82 mountall: ply-ev[ 1103.520656] softirqs last disabled at (0): ent-loop.c:497: [< (null)>] (null) ply_event_loop_n[ 1103.520656] Pid: 135, comm: shutdown Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 ew: Assertion `l[ 1103.520656] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1103.520656] EIP is at mutex_lock_nested+0x2ea/0x308 [ 1103.520656] EAX: 00000286 EBX: cb4e699c ECX: 00000006 EDX: cf8826c8 [ 1103.520656] ESI: ce82dbb0 EDI: ce873d7c EBP: ce873d98 ESP: ce873d5c [ 1103.520656] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.520656] CR0: 8005003b CR2: b77911b8 CR3: 0e88b000 CR4: 00000690 [ 1103.520656] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 oop->epoll_fd >=[ 1103.520656] DR6: ffff0ff0 DR7: 00000400 [ 1103.520656] Process shutdown (pid: 135, ti=ce872000 task=cf882240 task.ti=ce872000) 0' failed. [ 1103.520656] Stack: [ 1103.520656] 00000000 00000002 00000000 c10addf7 cb4e69c0 cf882240 cb4e69a0 00000286 [ 1103.520656] ce873d7c ce873d7c 00000000 ce873d7c ce82d790 ce82dbb0 00000000 ce873dec [ 1103.520656] c10addf7 00000000 00000000 00000000 00000000 b775e000 ce0f6400 00000000 [ 1103.520656] Call Trace: [ 1103.520656] [] ? vma_adjust+0x134/0x30e [ 1103.520656] [] vma_adjust+0x134/0x30e [ 1103.520656] [] __split_vma+0xea/0x13e [ 1103.520656] [] do_munmap+0xab/0x24b [ 1103.520656] [] vm_munmap+0x30/0x41 [ 1103.520656] [] elf_map+0x80/0xa6 [ 1103.520656] [] load_elf_binary+0x7b7/0xfbd [ 1103.520656] [] search_binary_handler+0x14d/0x22b [ 1103.520656] [] ? search_binary_handler+0x3a/0x22b [ 1103.520656] [] ? load_elf_library+0x1bb/0x1bb [ 1103.520656] [] do_execve+0x38b/0x456 [ 1103.520656] [] ? do_execve+0xeb/0x456 [ 1103.520656] [] sys_execve+0x31/0x53 [ 1103.520656] [] ptregs_execve+0x12/0x20 [ 1103.520656] [] ? sysenter_do_call+0x12/0x38 [ 1103.520656] Code: d4 8f c1 00 74 1e 8b 45 e0 50 9d 8d 74 26 00 e8 5b 81 9f ff eb 19 e8 2a 80 9f ff 83 3d 6c d4 8f c1 00 75 02 0f 0b 8b 45 e0 50 9d <8d> 74 26 00 8d 45 e4 e8 c0 5f 9f ff 89 e0 25 00 e0 ff ff ff 48 [ 1103.520656] Kernel panic - not syncing: softlockup: hung tasks [ 1103.520656] Pid: 135, comm: shutdown Not tainted 3.5.0-bisect2-00313-g7cd58b0 #86 [ 1103.520656] Call Trace: [ 1103.520656] [] panic+0x81/0x194 [ 1103.520656] [] watchdog_timer_fn+0x115/0x134 [ 1103.520656] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.520656] [] ? __touch_watchdog+0x1d/0x1d [ 1103.520656] [] run_local_timers+0x8/0x14 [ 1103.520656] [] update_process_times+0x24/0x58 [ 1103.520656] [] tick_periodic+0x66/0x72 [ 1103.520656] [] tick_handle_periodic+0x19/0x61 [ 1103.520656] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.520656] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.520656] [] apic_timer_interrupt+0x36/0x40 [ 1103.520656] [] ? mutex_lock_nested+0x2ea/0x308 [ 1103.520656] [] ? vma_adjust+0x134/0x30e [ 1103.520656] [] vma_adjust+0x134/0x30e [ 1103.520656] [] __split_vma+0xea/0x13e [ 1103.520656] [] do_munmap+0xab/0x24b [ 1103.520656] [] vm_munmap+0x30/0x41 [ 1103.520656] [] elf_map+0x80/0xa6 [ 1103.520656] [] load_elf_binary+0x7b7/0xfbd [ 1103.520656] [] search_binary_handler+0x14d/0x22b [ 1103.520656] [] ? search_binary_handler+0x3a/0x22b [ 1103.520656] [] ? load_elf_library+0x1bb/0x1bb [ 1103.520656] [] do_execve+0x38b/0x456 [ 1103.520656] [] ? do_execve+0xeb/0x456 [ 1103.520656] [] sys_execve+0x31/0x53 [ 1103.520656] [] ptregs_execve+0x12/0x20 [ 1103.520656] [] ? sysenter_do_call+0x12/0x38 [ 1103.520656] Rebooting in 10 seconds.. -- [ 4.801716] Freeing unused kernel memory: 460k freed [ 1103.512941] BUG: soft lockup - CPU#1 stuck for 1023s! [sleep:134] [ 1103.512941] irq event stamp: 780 [ 1103.512941] hardirqs last enabled at (779): [] do_page_fault+0x98/0x36e [ 1103.512941] hardirqs last disabled at (780): [] apic_timer_interrupt+0x2f/0x40 [ 1103.517007] softirqs last enabled at (22): [] __do_softirq+0x131/0x153 -- [ 5.230581] Freeing unused kernel memory: 464k freed [ 1104.515302] BUG: soft lockup - CPU#1 stuck for 1024s! [run-parts:122] [ 1104.515302] irq event stamp: 1238 [ 1104.515302] hardirqs last enabled at (1237): [] do_page_fault+0x98/0x36e [ 1104.515302] hardirqs last disabled at (1238): [] apic_timer_interrupt+0x2f/0x40 [ 1104.515302] softirqs last enabled at (0): [] copy_process+0x471/0xf82 [ 1104.515302] softirqs last disabled at (0): [< (null)>] (null) [ 1104.515302] Pid: 122, comm: run-parts Not tainted 3.5.0-bisect2-00646-g5b160bd #85 [ 1104.515302] EIP: 0060:[] EFLAGS: 00000246 CPU: 1 [ 1104.515302] EIP is at rcu_lockdep_current_cpu_online+0x0/0x51 [ 1104.515302] EAX: 00000000 EBX: cb4e695c ECX: 00000001 EDX: ce83c000 [ 1104.522494] [sched_delayed] sched: RT throttling activated [ 1104.515302] ESI: 0000000c EDI: cb4bf851 EBP: ce83ddfc ESP: ce83dde0 [ 1104.515302] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.515302] CR0: 8005003b CR2: b774b696 CR3: 0e863000 CR4: 00000690 [ 1104.515302] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.515302] DR6: ffff0ff0 DR7: 00000400 [ 1104.515302] Process run-parts (pid: 122, ti=ce83c000 task=cf8a0000 task.ti=ce83c000) [ 1104.515302] Stack: [ 1104.515302] c12a4768 00000000 00000001 00000000 cb4e695c 00000021 0000000c ce83de04 [ 1104.515302] c12a4a88 ce83de2c c1094dcd 00000002 00000001 00000000 c1094d5a cb4e695c [ 1104.515302] ce861200 00000021 00000000 ce83de6c c1095082 00000000 00000001 0000036f [ 1104.515302] Call Trace: [ 1104.515302] [] ? radix_tree_lookup_element+0x3c/0x107 [ 1104.515302] [] radix_tree_lookup_slot+0xd/0xf [ 1104.515302] [] find_get_page+0x8a/0x1a6 [ 1104.515302] [] ? find_get_page+0x17/0x1a6 [ 1104.515302] [] filemap_fault+0x77/0x35b [ 1104.515302] [] __do_fault+0x88/0x38a [ 1104.515302] [] handle_pte_fault+0x2c3/0x74c [ 1104.515302] [] ? kmap_atomic_prot+0x11f/0x133 [ 1104.515302] [] handle_mm_fault+0x95/0xa4 [ 1104.515302] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515302] [] do_page_fault+0x336/0x36e [ 1104.515302] [] ? might_fault+0x7b/0x97 [ 1104.515302] [] ? set_tls_desc+0x14a/0x15d [ 1104.515302] [] ? restore_all+0xf/0xf [ 1104.515302] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515302] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515302] [] do_async_page_fault+0x23/0x5f [ 1104.515302] [] error_code+0x6c/0x80 [ 1104.515302] Code: 51 c7 04 24 00 00 00 00 b9 00 f4 8f c1 e8 b9 fd ff ff c9 c3 55 89 e5 51 c7 04 24 00 00 00 00 b9 90 f6 8f c1 e8 a2 fd ff ff c9 c3 <55> 89 e5 89 e2 81 e2 00 e0 ff ff 8b 4a 14 b0 01 f7 c1 00 00 00 [ 1104.515302] Kernel panic - not syncing: softlockup: hung tasks [ 1104.515302] Pid: 122, comm: run-parts Not tainted 3.5.0-bisect2-00646-g5b160bd #85 [ 1104.515302] Call Trace: [ 1104.515302] [] panic+0x81/0x194 [ 1104.515302] [] watchdog_timer_fn+0x115/0x134 [ 1104.515302] [] hrtimer_run_queues+0x14c/0x1c6 [ 1104.515302] [] ? __touch_watchdog+0x1d/0x1d [ 1104.515302] [] run_local_timers+0x8/0x14 [ 1104.515302] [] update_process_times+0x24/0x58 [ 1104.515302] [] tick_periodic+0x66/0x72 [ 1104.515302] [] tick_handle_periodic+0x19/0x61 [ 1104.515302] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1104.515302] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1104.515302] [] apic_timer_interrupt+0x36/0x40 [ 1104.515302] [] ? call_rcu_sched+0x17/0x17 [ 1104.515302] [] ? radix_tree_lookup_element+0x3c/0x107 [ 1104.515302] [] radix_tree_lookup_slot+0xd/0xf [ 1104.515302] [] find_get_page+0x8a/0x1a6 [ 1104.515302] [] ? find_get_page+0x17/0x1a6 [ 1104.515302] [] filemap_fault+0x77/0x35b [ 1104.515302] [] __do_fault+0x88/0x38a [ 1104.515302] [] handle_pte_fault+0x2c3/0x74c [ 1104.515302] [] ? kmap_atomic_prot+0x11f/0x133 [ 1104.515302] [] handle_mm_fault+0x95/0xa4 [ 1104.515302] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515302] [] do_page_fault+0x336/0x36e [ 1104.515302] [] ? might_fault+0x7b/0x97 [ 1104.515302] [] ? set_tls_desc+0x14a/0x15d [ 1104.515302] [] ? restore_all+0xf/0xf [ 1104.515302] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515302] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1104.515302] [] do_async_page_fault+0x23/0x5f [ 1104.515302] [] error_code+0x6c/0x80 [ 1104.515302] Rebooting in 10 seconds.. -- [ 1105.513232] [sched_delayed] sched: RT throttling activated [ 1105.513206] BUG: soft lockup - CPU#1 stuck for 1023s! [rc.local:123] [ 1105.513206] irq event stamp: 326 [ 1105.513206] hardirqs last enabled at (325): [] debug_check_no_locks_freed+0x14b/0x167 [ 1105.513206] hardirqs last disabled at (326): [] apic_timer_interrupt+0x2f/0x40 [ 1105.513206] softirqs last enabled at (0): [] copy_process+0x465/0xf8a [ 1105.513206] softirqs last disabled at (0): [< (null)>] (null) [ 1105.513206] Pid: 123, comm: rc.local Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1105.513206] EIP: 0060:[] EFLAGS: 00000246 CPU: 1 [ 1105.513206] EIP is at __mnt_drop_write+0x1/0x1a [ 1105.513206] EAX: ce014110 EBX: cb438000 ECX: 00000000 EDX: 00000001 [ 1105.513206] ESI: ce014110 EDI: 00000000 EBP: ce879dbc ESP: ce879da4 [ 1105.513206] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1105.513206] CR0: 80050033 CR2: b7553baf CR3: 0e85a000 CR4: 00000690 [ 1105.513206] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1105.513206] DR6: ffff0ff0 DR7: 00000400 [ 1105.513206] Process rc.local (pid: 123, ti=ce878000 task=cf8ad5f0 task.ti=ce878000) [ 1105.513206] Stack: [ 1105.513206] ce879dbc c10dd7bf 504bf223 ffffff88 ce889800 ce6a2f80 ce879e4c c1097f77 [ 1105.513206] 00001000 006f41e4 00000000 00000080 00000000 00000000 00000001 ce889884 [ 1105.513206] ce879e90 00000000 00001000 00000080 00000000 cb438000 ce879ed4 00000000 [ 1105.513206] Call Trace: [ 1105.513206] [] ? touch_atime+0xdc/0x104 [ 1105.513206] [] generic_file_aio_read+0x5b5/0x5f1 [ 1105.513206] [] do_sync_read+0x75/0xb0 [ 1105.513206] [] vfs_read+0x8b/0x10d [ 1105.513206] [] ? might_fault+0x26/0x99 [ 1105.513206] [] ? wait_on_retry_sync_kiocb+0x3f/0x3f [ 1105.513206] [] kernel_read+0x30/0x41 [ 1105.513206] [] prepare_binprm+0xfb/0x103 [ 1105.513206] [] do_execve+0x327/0x456 [ 1105.513206] [] ? do_execve+0xeb/0x456 [ 1105.546433] [] sys_execve+0x31/0x53 [ 1105.546433] [] ptregs_execve+0x12/0x20 [ 1105.546433] [] ? sysenter_do_call+0x12/0x38 [ 1105.546433] Code: 00 00 e8 6c ba fe ff 89 d8 e8 91 ff ff ff 89 c6 85 c0 74 10 8b 43 08 8b 40 04 ba 01 00 00 00 e8 dd b9 fe ff 89 f0 5b 5e 5d c3 55 <89> e5 89 e2 81 e2 00 e0 ff ff ff 42 14 8b 40 0c 64 ff 48 04 ff [ 1105.546433] Kernel panic - not syncing: softlockup: hung tasks [ 1105.546433] Pid: 123, comm: rc.local Not tainted 3.6.0-rc1-bisect2-00023-g8a13a4c #79 [ 1105.546433] Call Trace: [ 1105.546433] [] panic+0x97/0x1aa [ 1105.546433] [] watchdog_timer_fn+0x115/0x134 [ 1105.546433] [] hrtimer_run_queues+0x14c/0x1c6 [ 1105.546433] [] ? __touch_watchdog+0x1d/0x1d [ 1105.546433] [] run_local_timers+0x8/0x14 [ 1105.546433] [] update_process_times+0x24/0x58 [ 1105.546433] [] tick_periodic+0x66/0x72 [ 1105.546433] [] tick_handle_periodic+0x19/0x61 [ 1105.546433] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1105.546433] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1105.546433] [] apic_timer_interrupt+0x36/0x40 [ 1105.546433] [] ? pipe_fcntl+0xbc/0x140 [ 1105.546433] [] ? __mnt_drop_write+0x1/0x1a [ 1105.546433] [] ? touch_atime+0xdc/0x104 [ 1105.546433] [] generic_file_aio_read+0x5b5/0x5f1 [ 1105.546433] [] do_sync_read+0x75/0xb0 [ 1105.546433] [] vfs_read+0x8b/0x10d [ 1105.546433] [] ? might_fault+0x26/0x99 [ 1105.546433] [] ? wait_on_retry_sync_kiocb+0x3f/0x3f [ 1105.546433] [] kernel_read+0x30/0x41 [ 1105.546433] [] prepare_binprm+0xfb/0x103 [ 1105.546433] [] do_execve+0x327/0x456 [ 1105.546433] [] ? do_execve+0xeb/0x456 [ 1105.546433] [] sys_execve+0x31/0x53 [ 1105.546433] [] ptregs_execve+0x12/0x20 [ 1105.546433] [] ? sysenter_do_call+0x12/0x38 [ 1105.546433] Rebooting in 10 seconds.. -- [ 6.292044] Freeing unused kernel memory: 464k freed [ 1105.523875] BUG: soft lockup - CPU#1 stuck for 1024s! [swapper/1:0] [ 1105.523875] irq event stamp: 13590 [ 1105.523875] hardirqs last enabled at (13587): [] restore_all_notrace+0x0/0x18 [ 1105.523875] hardirqs last disabled at (13588): [] apic_timer_interrupt+0x2f/0x40 [ 1105.523875] softirqs last enabled at (13590): [] _local_bh_enable+0xd/0xf [ 1105.523875] softirqs last disabled at (13589): [] irq_enter+0x2c/0x4d [ 1105.523875] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1105.523875] EIP: 0060:[] EFLAGS: 00000282 CPU: 1 [ 1105.523875] EIP is at native_safe_halt+0x5/0x7 [ 1105.523875] EAX: ce062240 EBX: ffffffff ECX: ce062240 EDX: 00000007 [ 1105.523875] ESI: c195cbf0 EDI: ffff8b9f EBP: ce06bf68 ESP: ce06bf68 [ 1105.523875] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1105.523875] CR0: 8005003b CR2: b76e4fb8 CR3: 0e888000 CR4: 00000690 [ 1105.523875] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1105.523875] DR6: ffff0ff0 DR7: 00000400 [ 1105.523875] Process swapper/1 (pid: 0, ti=ce06a000 task=ce062240 task.ti=ce06a000) [ 1105.523875] Stack: [ 1105.523875] ce06bf70 c10089d9 ce06bf88 c1009220 d1bf6876 e022ed99 1a6ddff1 cfff9fa0 [ 1105.523875] ce06bfb4 c166027a 00000000 00000000 00000000 00000000 8a9ce88b 1a6ddff1 [ 1105.523875] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1105.523875] Call Trace: [ 1105.523875] [] default_idle+0x2f/0x55 [ 1105.523875] [] cpu_idle+0xc9/0xfb [ 1105.523875] [] start_secondary+0x292/0x29a [ 1105.523875] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1105.523875] Kernel panic - not syncing: softlockup: hung tasks [ 1105.523875] Pid: 0, comm: swapper/1 Not tainted 3.5.0-bisect2-05018-g614a6d43 #82 [ 1105.523875] Call Trace: [ 1105.523875] [] panic+0x81/0x194 [ 1105.523875] [] watchdog_timer_fn+0x115/0x134 [ 1105.523875] [] hrtimer_run_queues+0x14c/0x1c6 [ 1105.523875] [] ? __touch_watchdog+0x1d/0x1d [ 1105.523875] [] run_local_timers+0x8/0x14 [ 1105.523875] [] update_process_times+0x24/0x58 [ 1105.523875] [] tick_periodic+0x66/0x72 [ 1105.523875] [] tick_handle_periodic+0x19/0x61 [ 1105.523875] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1105.523875] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1105.523875] [] apic_timer_interrupt+0x36/0x40 [ 1105.523875] [] ? native_safe_halt+0x5/0x7 [ 1105.523875] [] default_idle+0x2f/0x55 [ 1105.523875] [] cpu_idle+0xc9/0xfb [ 1105.523875] [] start_secondary+0x292/0x29a mountall: ply-event-loop.c:497: ply_event_loop_new: Assertion `loop->epoll_fd >= 0' failed. [ 1105.523875] Rebooting in 10 seconds.. -- [ 1103.519588] [sched_delayed] sched: RT throttling activated [ 1103.517937] BUG: soft lockup - CPU#1 stuck for 1024s! [trinity:131] [ 1103.517937] irq event stamp: 5400 [ 1103.517937] hardirqs last disabled at (5400): [] apic_timer_interrupt+0x2f/0x40 [ 1103.517937] softirqs last enabled at (2968): [] unix_sock_destructor+0xae/0xb3 [ 1103.517937] softirqs last disabled at (2966): [] unix_sock_destructor+0x9b/0xb3 [ 1103.517937] [ 1103.517937] Pid: 131, comm: trinity Not tainted 3.5.0-rc7-bisect2-00018-g2a8c088 #93 [ 1103.517937] EIP: 0060:[] EFLAGS: 00010206 CPU: 1 [ 1103.517937] EIP is at __do_fault+0x100/0x38a [ 1103.517937] EAX: c23b8000 EBX: c63b9000 ECX: 00000400 EDX: 00000163 [ 1103.517937] ESI: c63b9000 EDI: c23b8000 EBP: ce8b1eb8 ESP: ce8b1e74 [ 1103.517937] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.517937] CR0: 8005003b CR2: 0824cc48 CR3: 0e86d000 CR4: 00000690 [ 1103.517937] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.517937] DR6: ffff0ff0 DR7: 00000400 [ 1103.517937] Process trinity (pid: 131, ti=ce8b0000 task=ce8a9120 task.ti=ce8b0000) [ 1103.517937] Stack: [ 1103.517937] 00000000 ce826a34 00000001 00000001 0824cc48 ce826a00 00000370 00000200 [ 1103.517937] ce850370 ce586700 00000029 00000204 0824c000 ce606720 ce850370 ce878930 [ 1103.517937] 00000204 ce8b1efc c10aa215 ce86d080 00000204 00000029 00000000 ce70ff00 [ 1103.517937] Call Trace: [ 1103.517937] [] handle_pte_fault+0x2c3/0x74c [ 1103.517937] [] ? kmap_atomic_prot+0x11f/0x133 [ 1103.517937] [] handle_mm_fault+0x95/0xa4 [ 1103.517937] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517937] [] do_page_fault+0x336/0x36e [ 1103.517937] [] ? up_write+0x16/0x2b [ 1103.517937] [] ? vm_mmap_pgoff+0x50/0x59 [ 1103.517937] [] ? restore_all+0xf/0xf [ 1103.517937] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517937] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517937] [] do_async_page_fault+0x23/0x5f [ 1103.517937] [] error_code+0x6c/0x80 [ 1103.517937] Code: c8 00 0f 84 cd 00 00 00 8b 55 dc f6 42 18 08 75 42 89 d8 e8 30 76 f7 ff 89 c3 8b 45 e0 e8 26 76 f7 ff b9 00 04 00 00 89 c7 89 de a5 e8 36 76 f7 ff 89 d8 e8 2f 76 f7 ff 8b 45 e0 0f ba 28 03 [ 1103.517937] Kernel panic - not syncing: softlockup: hung tasks [ 1103.517937] Pid: 131, comm: trinity Not tainted 3.5.0-rc7-bisect2-00018-g2a8c088 #93 [ 1103.517937] Call Trace: [ 1103.517937] [] panic+0x81/0x194 [ 1103.517937] [] watchdog_timer_fn+0x115/0x134 [ 1103.517937] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.517937] [] ? __touch_watchdog+0x1d/0x1d [ 1103.517937] [] run_local_timers+0x8/0x14 [ 1103.517937] [] update_process_times+0x24/0x58 [ 1103.517937] [] tick_periodic+0x66/0x72 [ 1103.517937] [] tick_handle_periodic+0x19/0x61 [ 1103.517937] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.517937] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.517937] [] apic_timer_interrupt+0x36/0x40 [ 1103.517937] [] ? __do_fault+0x100/0x38a [ 1103.517937] [] handle_pte_fault+0x2c3/0x74c [ 1103.517937] [] ? kmap_atomic_prot+0x11f/0x133 [ 1103.517937] [] handle_mm_fault+0x95/0xa4 [ 1103.517937] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517937] [] do_page_fault+0x336/0x36e [ 1103.517937] [] ? up_write+0x16/0x2b [ 1103.517937] [] ? vm_mmap_pgoff+0x50/0x59 [ 1103.517937] [] ? restore_all+0xf/0xf [ 1103.517937] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517937] [] ? kvm_read_and_reset_pf_reason+0x33/0x33 [ 1103.517937] [] do_async_page_fault+0x23/0x5f [ 1103.517937] [] error_code+0x6c/0x80 [ 1103.517937] hardirqs last enabled at (5399): [] get_page_from_freelist+0x2ca/0x446 [ 1103.517937] Rebooting in 10 seconds.. -- [ 1104.518348] [sched_delayed] sched: RT throttling activated [ 1104.519792] BUG: soft lockup - CPU#1 stuck for 1025s! [swapper/1:0] [ 1104.519792] irq event stamp: 12060 [ 1104.519792] hardirqs last enabled at (12057): [] restore_all_notrace+0x0/0x18 [ 1104.519792] hardirqs last disabled at (12058): [] apic_timer_interrupt+0x2f/0x40 [ 1104.519792] softirqs last enabled at (12060): [] _local_bh_enable+0xd/0xf [ 1104.519792] softirqs last disabled at (12059): [] irq_enter+0x2c/0x4d [ 1104.519792] [ 1104.519792] Pid: 0, comm: swapper/1 Not tainted 3.5.0-rc7-bisect2-00018-g2a8c088 #93 [ 1104.519792] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1104.519792] EIP is at native_safe_halt+0x5/0x7 [ 1104.519792] EAX: ce060000 EBX: ffffffff ECX: ce060000 EDX: 00000007 [ 1104.519792] ESI: c1954c80 EDI: ffff8b9e EBP: ce05ff68 ESP: ce05ff68 [ 1104.519792] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1104.519792] CR0: 8005003b CR2: b77e052c CR3: 0f878000 CR4: 00000690 [ 1104.519792] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1104.519792] DR6: ffff0ff0 DR7: 00000400 [ 1104.519792] Process swapper/1 (pid: 0, ti=ce05e000 task=ce060000 task.ti=ce05e000) [ 1104.519792] Stack: [ 1104.519792] ce05ff70 c10089b1 ce05ff88 c100920c 157956ca 23dcdbea 5e27d951 cfff9fa0 [ 1104.519792] ce05ffb4 c165b4be 00000000 00000000 00000000 00000000 ce56e1ef 5e27d951 [ 1104.519792] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1104.519792] Call Trace: -- [ 4.512630] Freeing unused kernel memory: 456k freed [ 1103.517164] BUG: soft lockup - CPU#1 stuck for 1023s! [swapper/1:0] [ 1103.517164] irq event stamp: 5018 [ 1103.517164] hardirqs last enabled at (5015): [] default_idle+0x1f/0x55 mountall: ply-ev[ 1103.517164] hardirqs last disabled at (5016): ent-loop.c:497: [] apic_timer_interrupt+0x2f/0x40 ply_event_loop_n[ 1103.517164] softirqs last enabled at (5018): ew: Assertion `l[] _local_bh_enable+0xd/0xf oop->epoll_fd >=[ 1103.517164] softirqs last disabled at (5017): 0' failed.[] irq_enter+0x2c/0x4d [ 1103.526001] [sched_delayed] sched: RT throttling activated [ 1103.517164] [ 1103.517164] Pid: 0, comm: swapper/1 Not tainted 3.5.0-rc7-bisect2-00018-g2a8c088 #93 [ 1103.517164] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1103.517164] EIP is at native_safe_halt+0x5/0x7 [ 1103.517164] EAX: ce060000 EBX: ffffffff ECX: ce060000 EDX: 00000007 [ 1103.517164] ESI: c1954c80 EDI: ffff8b9e EBP: ce05ff68 ESP: ce05ff68 [ 1103.517164] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.517164] CR0: 8005003b CR2: b7665bd0 CR3: 0e852000 CR4: 00000690 [ 1103.517164] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.517164] DR6: ffff0ff0 DR7: 00000400 [ 1103.517164] Process swapper/1 (pid: 0, ti=ce05e000 task=ce060000 task.ti=ce05e000) [ 1103.517164] Stack: [ 1103.517164] ce05ff70 c10089b1 ce05ff88 c100920c -- [ 4.696250] Freeing unused kernel memory: 456k freed [ 1103.520327] BUG: soft lockup - CPU#1 stuck for 1023s! [swapper/1:0] [ 1103.520327] irq event stamp: 13124 [ 1103.520327] hardirqs last enabled at (13121): [] restore_all_notrace+0x0/0x18 [ 1103.520327] hardirqs last disabled at (13122): [] apic_timer_interrupt+0x2f/0x40 [ 1103.520327] softirqs last enabled at (13124): [] _local_bh_enable+0xd/0xf [ 1103.520327] softirqs last disabled at (13123): [] irq_enter+0x2c/0x4d [ 1103.520327] [ 1103.520327] Pid: 0, comm: swapper/1 Not tainted 3.5.0-rc7-bisect2-00018-g2a8c088 #93 [ 1103.520327] EIP: 0060:[] EFLAGS: 00000286 CPU: 1 [ 1103.520327] EIP is at native_safe_halt+0x5/0x7 [ 1103.520327] EAX: ce060000 EBX: ffffffff ECX: ce060000 EDX: 00000007 [ 1103.520327] ESI: c1954c80 EDI: ffff8b9f EBP: ce05ff68 ESP: ce05ff68 [ 1103.520327] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 [ 1103.520327] CR0: 8005003b CR2: 08051f68 CR3: 0e876000 CR4: 00000690 [ 1103.520327] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [ 1103.520327] DR6: ffff0ff0 DR7: 00000400 [ 1103.520327] Process swapper/1 (pid: 0, ti=ce05e000 task=ce060000 task.ti=ce05e000) [ 1103.520327] Stack: [ 1103.520327] ce05ff70 c10089b1 ce05ff88 c100920c 95c987b1 a42d0cd0 de77b93b cfff9fa0 [ 1103.520327] ce05ffb4 c165b4be 00000000 00000000 00000000 00000000 4ea6c1d9 de77b93b [ 1103.520327] 01000800 0000d248 00000000 00000000 00000000 00000000 00000000 00000000 [ 1103.520327] Call Trace: [ 1103.520327] [] default_idle+0x2f/0x55 [ 1103.520327] [] cpu_idle+0xc9/0xfb [ 1103.520327] [] start_secondary+0x29c/0x2a4 [ 1103.520327] Code: 0f 22 e0 5d c3 55 89 e5 0f 09 5d c3 55 89 e5 9c 58 5d c3 55 89 e5 50 9d 5d c3 55 89 e5 fa 5d c3 55 89 e5 fb 5d c3 55 89 e5 fb f4 <5d> c3 55 89 e5 f4 5d c3 55 89 e5 57 56 53 53 89 c6 89 55 f0 89 [ 1103.520327] Kernel panic - not syncing: softlockup: hung tasks [ 1103.520327] Pid: 0, comm: swapper/1 Not tainted 3.5.0-rc7-bisect2-00018-g2a8c088 #93 [ 1103.520327] Call Trace: [ 1103.520327] [] panic+0x81/0x194 [ 1103.520327] [] watchdog_timer_fn+0x115/0x134 [ 1103.520327] [] hrtimer_run_queues+0x14c/0x1c6 [ 1103.520327] [] ? __touch_watchdog+0x1d/0x1d [ 1103.520327] [] run_local_timers+0x8/0x14 [ 1103.520327] [] update_process_times+0x24/0x58 [ 1103.520327] [] tick_periodic+0x66/0x72 [ 1103.520327] [] tick_handle_periodic+0x19/0x61 [ 1103.520327] [] smp_apic_timer_interrupt+0x6d/0x7f [ 1103.520327] [] ? trace_hardirqs_off_thunk+0xc/0x10 [ 1103.520327] [] apic_timer_interrupt+0x36/0x40 [ 1103.520327] [] ? native_safe_halt+0x5/0x7 [ 1103.520327] [] default_idle+0x2f/0x55 [ 1103.520327] [] cpu_idle+0xc9/0xfb [ 1103.520327] [] start_secondary+0x29c/0x2a4 [ 1103.520327] Rebooting in 10 seconds.. Thanks, Fengguang