Re: [Patch 0/7] Per cpu thread hotplug infrastructure - V3

["Srivatsa S. Bhat" <srivatsa.bhat@linux.vnet.ibm.com>]

On 07/16/2012 04:12 PM, Thomas Gleixner wrote:
> The following series implements the infrastructure for parking and
> unparking kernel threads to avoid the full teardown and fork on cpu
> hotplug operations along with management infrastructure for hotplug
> and users.
> 
> Changes vs. V2:
> 
>  Use callbacks for all functionality. Thanks to Rusty for pointing
>  that out. It makes the use sites nice and simple and keeps all the
>  code which would be duplicated otherwise on the core.
> 

Hi Thomas,

I am observing stalls detected by RCU and the subsequent NMI backtraces
while running CPU hotplug stress tests on this patchset. Will check whether
I hit this without your patches too... (But I think the baseline is a little
bit old.. 3.5-rc1..)


----------------------------

Booting Node 1 Processor 7 APIC 0x16
NMI watchdog: enabled, takes one hw-pmu counter.
lockdep: fixing up alternatives.
Booting Node 0 Processor 8 APIC 0x1
NMI watchdog: enabled, takes one hw-pmu counter.
INFO: task stress_cpu_hotp:6903 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
stress_cpu_hotp D 0000000000000000     0  6903   6850 0x00000000
 ffff8808cda09ad8 0000000000000046 0000000000000001 ffff8808cda08010
 ffff8808cda08000 ffff8808cda08000 ffff8808cda08000 ffff8808cda09fd8
 ffff8808cda08000 ffff8808cda09fd8 ffff8808d04f45c0 ffff8808d2d8a200
Call Trace:
 [<ffffffff81465164>] ? cpufreq_notify_transition+0x174/0x2f0
 [<ffffffffa00b1200>] ? acpi_cpufreq_cpu_exit+0xb0/0xb0 [acpi_cpufreq]
 [<ffffffff810c21f0>] ? mark_held_locks+0x80/0x120
 [<ffffffff81591ab7>] ? __mutex_unlock_slowpath+0xf7/0x190
 [<ffffffff815939c9>] schedule+0x29/0x70
 [<ffffffff815915e5>] schedule_timeout+0x2d5/0x3f0
 [<ffffffff810c21f0>] ? mark_held_locks+0x80/0x120
 [<ffffffff81594eb0>] ? _raw_spin_unlock_irq+0x30/0x50
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff81593845>] wait_for_common+0x115/0x190
 [<ffffffff810983d0>] ? try_to_wake_up+0x2f0/0x2f0
 [<ffffffff81078bd2>] ? queue_delayed_work_on+0x132/0x140
 [<ffffffff8159399d>] wait_for_completion+0x1d/0x20
 [<ffffffff81043839>] hpet_cpuhp_notify+0x119/0x160
 [<ffffffff81043a40>] ? init_one_hpet_msi_clockevent+0x1c0/0x1c0
 [<ffffffff810784b0>] ? __queue_work+0x500/0x500
 [<ffffffff81599512>] notifier_call_chain+0x72/0x130
 [<ffffffff810890fe>] __raw_notifier_call_chain+0xe/0x10
 [<ffffffff8158af88>] _cpu_up+0x15b/0x164
 [<ffffffff8158b04d>] cpu_up+0xbc/0x11f
 [<ffffffff8157a87d>] store_online+0x9d/0xe0
 [<ffffffff81396be0>] dev_attr_store+0x20/0x30
 [<ffffffff81222119>] sysfs_write_file+0xd9/0x150
 [<ffffffff811a6a5b>] vfs_write+0xcb/0x130
 [<ffffffff811a6bb5>] sys_write+0x55/0x90
 [<ffffffff8159e629>] system_call_fastpath+0x16/0x1b
5 locks held by stress_cpu_hotp/6903:
 #0:  (&buffer->mutex){+.+.+.}, at: [<ffffffff81222089>] sysfs_write_file+0x49/0x150
 #1:  (s_active#111){.+.+.+}, at: [<ffffffff81222102>] sysfs_write_file+0xc2/0x150
 #2:  (x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [<ffffffff81039df7>] cpu_hotplug_driver_lock+0x17/0x20
 #3:  (cpu_add_remove_lock){+.+.+.}, at: [<ffffffff8158b009>] cpu_up+0x78/0x11f
 #4:  (cpu_hotplug.lock){+.+.+.}, at: [<ffffffff81059cef>] cpu_hotplug_begin+0x2f/0x60
lockdep: fixing up alternatives.
Booting Node 0 Processor 9 APIC 0x3
NMI watchdog: enabled, takes one hw-pmu counter.
lockdep: fixing up alternatives.
Booting Node 0 Processor 10 APIC 0x5
CPU 2 is now offline
CPU 10 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 3 is now offline
CPU 11 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 4 is now offline
CPU 5 MCA banks CMCI:6 CMCI:8
CPU 12 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 5 is now offline
CPU 6 MCA banks CMCI:6 CMCI:8
CPU 13 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 6 is now offline
CPU 7 MCA banks CMCI:6 CMCI:8
CPU 14 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 7 is now offline
CPU 12 MCA banks CMCI:6 CMCI:8
CPU 15 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 8 is now offline
------------[ cut here ]------------
WARNING: at kernel/rcutree.c:1545 rcu_do_batch+0x35d/0x370()
Hardware name: IBM System x -[7870C4Q]-
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas
Pid: 11788, comm: migration/9 Not tainted 3.5.0-rc1-threadhotplugv3-paulfix #1
Call Trace:
 <IRQ>  [<ffffffff810fe58d>] ? rcu_do_batch+0x35d/0x370
 [<ffffffff810fe58d>] ? rcu_do_batch+0x35d/0x370
 [<ffffffff8105672f>] warn_slowpath_common+0x7f/0xc0
 [<ffffffff8105678a>] warn_slowpath_null+0x1a/0x20
 [<ffffffff810fe58d>] rcu_do_batch+0x35d/0x370
 [<ffffffff810c24e0>] ? trace_hardirqs_on_caller+0x70/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff811000a3>] __rcu_process_callbacks+0x193/0x1f0
 [<ffffffff8110018d>] rcu_process_callbacks+0x8d/0x260
 [<ffffffff81060e71>] __do_softirq+0x151/0x3e0
 [<ffffffff8159fb6c>] call_softirq+0x1c/0x30
 [<ffffffff810194fd>] do_softirq+0x9d/0xd0
 [<ffffffff81060555>] irq_exit+0xe5/0x100
 [<ffffffff815a04ee>] smp_apic_timer_interrupt+0x6e/0x99
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 <EOI>  [<ffffffff810e6232>] ? stop_machine_cpu_stop+0xe2/0x140
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
---[ end trace e18e7d52508d4205 ]---
CPU 9 is now offline
CPU 10 is now offline
Booting Node 1 Processor 15 APIC 0x17
NMI watchdog: enabled, takes one hw-pmu counter.
CPU 1 is now offline
CPU 9 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 2 is now offline
CPU 10 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 3 is now offline
CPU 11 MCA banks CMCI:2 CMCI:3 CMCI:5
NOHZ: local_softirq_pending 100
CPU 4 is now offline
CPU 5 MCA banks CMCI:6 CMCI:8
CPU 12 MCA banks CMCI:2 CMCI:3 CMCI:5
Broke affinity for irq 81
CPU 5 is now offline
CPU 6 MCA banks CMCI:6 CMCI:8
CPU 13 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 6 is now offline
CPU 7 MCA banks CMCI:6 CMCI:8
CPU 14 MCA banks CMCI:2 CMCI:3 CMCI:5
NOHZ: local_softirq_pending 200
CPU 7 is now offline
CPU 12 MCA banks CMCI:6 CMCI:8
CPU 15 MCA banks CMCI:2 CMCI:3 CMCI:5
CPU 8 is now offline
CPU 9 is now offline
CPU 10 is now offline
CPU 11 is now offline
CPU 12 is now offline
CPU 13 MCA banks CMCI:6 CMCI:8
CPU 13 is now offline
CPU 14 MCA banks CMCI:6 CMCI:8
CPU 14 is now offline
CPU 15 MCA banks CMCI:6 CMCI:8
CPU 15 is now offline
Booting Node 0 Processor 10 APIC 0x5
NMI watchdog: enabled, takes one hw-pmu counter.
lockdep: fixing up alternatives.
Booting Node 0 Processor 11 APIC 0x7
NMI watchdog: enabled, takes one hw-pmu counter.
INFO: rcu_sched self-detected stall on CPU { 3}  (t=91070 jiffies)
INFO: rcu_sched self-detected stall on CPU { 4}  (t=91070 jiffies)
sending NMI to all CPUs:
NMI backtrace for cpu 0
CPU 0 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

INFO: rcu_sched self-detected stall on CPU
Pid: 0, comm: swapper/0 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-
 {
/68Y8033     
 5
RIP: 0010:[<ffffffff8103b741>] 
 [<ffffffff8103b741>] default_send_IPI_mask_sequence_phys+0xa1/0x130
RSP: 0018:ffff8808ffc03d98  EFLAGS: 00000002
RAX: 00000000000010ef RBX: 0000000000000006 RCX: 0000000000000010
}  (t=91070 jiffies)
RDX: 0000000000000fc0 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8808ffc03dd8 R08: 0000000000000010 R09: ffffffff81da9d60
R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000014
R13: 00000000000000ef R14: 000000000000dd40 R15: 00000000000000ef
FS:  0000000000000000(0000) GS:ffff8808ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/0 (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a14420)
Stack:
 0000000000000000 ffffffff81da9d60 0000000000000092 000000000000ea20
 ffffffff81da9d60 000001732e239870 ffffffff81a1ef00 7fffffffffffffff
 ffff8808ffc03de8 ffffffff8103e80e ffff8808ffc03df8 ffffffff8103a43b
Call Trace:
 <IRQ> 
 [<ffffffff8103e80e>] physflat_send_IPI_mask+0xe/0x10
 [<ffffffff8103a43b>] lapic_timer_broadcast+0x1b/0x20
 [<ffffffff810bb126>] tick_do_broadcast+0x66/0xa0
 [<ffffffff810bb254>] tick_handle_oneshot_broadcast+0xf4/0x130
 [<ffffffff81019de5>] timer_interrupt+0x15/0x20
 [<ffffffff810f7cbc>] handle_irq_event_percpu+0xcc/0x3e0
 [<ffffffff810f8018>] handle_irq_event+0x48/0x70
 [<ffffffff810fb755>] handle_edge_irq+0x65/0x130
 [<ffffffff8101958e>] handle_irq+0x5e/0x150
 [<ffffffff81599736>] ? atomic_notifier_call_chain+0x16/0x20
 [<ffffffff815a0400>] do_IRQ+0x60/0xe0
 [<ffffffff815951ef>] common_interrupt+0x6f/0x6f
 <EOI> 
 [<ffffffff81306a18>] ? intel_idle+0xc8/0x170
 [<ffffffff81306a14>] ? intel_idle+0xc4/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff815939c9>] ? schedule+0x29/0x70
 [<ffffffff815769af>] rest_init+0xff/0x180
 [<ffffffff815768b0>] ? csum_partial_copy_generic+0x170/0x170
 [<ffffffff81b17f2a>] start_kernel+0x3be/0x48c
 [<ffffffff81b17965>] ? kernel_init+0x290/0x290
 [<ffffffff8158f9be>] ? memblock_reserve+0x2e/0x4e
 [<ffffffff81b172d5>] x86_64_start_reservations+0xa5/0xc2
 [<ffffffff81b17401>] x86_64_start_kernel+0x10f/0x12a
 [<ffffffff81b17120>] ? early_idt_handlers+0x120/0x120
Code: a0 e3 ae 81 41 83 fd 02 45 0f b7 24 06 74 4b 8b 04 25 00 a3 5f ff f6 c4 10 74 1b 0f 1f 80 00 00 00 00 f3 90 8b 04 25 00 a3 5f ff <f6> c4 10 75 f2 8b 0d 5c ea ab 00 44 89 e0 c1 e0 18 89 04 25 10 
NMI backtrace for cpu 1
CPU 1 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/1 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff8101fc7b>]  [<ffffffff8101fc7b>] native_read_tsc+0xb/0x20
RSP: 0018:ffff8808d2d8dc70  EFLAGS: 00000002
RAX: 00000000000004d1 RBX: 0000000000000001 RCX: 00000000f9c4c38a
RDX: 00000000000004d1 RSI: 01a8000000000000 RDI: 0000000000000001
RBP: ffff8808d2d8dc70 R08: 00000000f9c4c356 R09: ffff8808d2d8aab8
R10: ffff8808d2d8aab8 R11: 0000000000000000 R12: 00000000000011dc
R13: 00000000aeda8ed8 R14: ffff8808d2d8c000 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff8808ffc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/1 (pid: 0, threadinfo ffff8808d2d8c000, task ffff8808d2d8a200)
Stack:
 ffff8808d2d8dcb8 ffffffff812aff41 ffff8808d2d8c000 00000001f9c4c32e
 ffffffff81a34580 00000000000011dc 00000000aeda8ed8 0000000000000001
 ffff8808d2d8de64 ffff8808d2d8dcc8 ffffffff812affef ffff8808d2d8dcf8
Call Trace:
 [<ffffffff812aff41>] delay_tsc+0x41/0xc0
 [<ffffffff812affef>] __delay+0xf/0x20
 [<ffffffff812b769b>] do_raw_spin_lock+0xab/0x150
 [<ffffffff815946b1>] _raw_spin_lock_irqsave+0x61/0x70
 [<ffffffff810baf03>] ? tick_broadcast_oneshot_control+0x83/0x130
 [<ffffffff810baf03>] tick_broadcast_oneshot_control+0x83/0x130
 [<ffffffff810bab68>] tick_notify+0xf8/0x200
 [<ffffffff81599512>] notifier_call_chain+0x72/0x130
 [<ffffffff81089116>] raw_notifier_call_chain+0x16/0x20
 [<ffffffff810ba48d>] clockevents_notify+0x3d/0x150
 [<ffffffff81306a32>] intel_idle+0xe2/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: 9f 00 48 c7 43 08 00 00 00 00 48 89 03 48 83 c4 08 5b c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 0f 31 89 c1 48 89 d0 <48> c1 e0 20 89 ca 48 09 d0 c9 c3 66 2e 0f 1f 84 00 00 00 00 00 
NMI backtrace for cpu 2
CPU 2 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/2 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff812aff3c>]  [<ffffffff812aff3c>] delay_tsc+0x3c/0xc0
RSP: 0018:ffff8808d2db1d40  EFLAGS: 00000006
RAX: 0000000000005f7d RBX: 0000000000000001 RCX: 000000005f7d5f7c
RDX: ffff8808d2db1fd8 RSI: 0018000000000000 RDI: 0000000000000001
RBP: ffff8808d2db1d78 R08: 00000000f9c93f00 R09: ffff8808d2daead0
R10: ffff8808d2daead0 R11: 0000000000000001 R12: 000000000000117f
R13: 00000000aeda8ed8 R14: ffff8808d2db0000 R15: 0000000000000002
FS:  0000000000000000(0000) GS:ffff8808ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/2 (pid: 0, threadinfo ffff8808d2db0000, task ffff8808d2dae240)
Stack:
 ffff8808d2db0000 00000002f9c93ecc ffffffff81a344c0 000000000000117f
 00000000aeda8ed8 0000000000000001 ffff8808ffc98970 ffff8808d2db1d88
 ffffffff812affef ffff8808d2db1db8 ffffffff812b769b ffffffff81a344c0
Call Trace:
 [<ffffffff812affef>] __delay+0xf/0x20
 [<ffffffff812b769b>] do_raw_spin_lock+0xab/0x150
 [<ffffffff815946b1>] _raw_spin_lock_irqsave+0x61/0x70
 [<ffffffff810ba478>] ? clockevents_notify+0x28/0x150
 [<ffffffff810ba478>] clockevents_notify+0x28/0x150
 [<ffffffff81306a32>] intel_idle+0xe2/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: 66 66 66 66 90 65 48 8b 14 25 c8 c7 00 00 4c 8d b2 28 e0 ff ff 89 fb 41 83 46 1c 01 65 44 8b 3c 25 08 dd 00 00 66 66 90 0f ae e8 <e8> 2f fd d6 ff 66 90 89 45 d0 44 89 7d d4 4c 89 75 c8 eb 1d 48 
NMI backtrace for cpu 3
CPU 3 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/3 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff8137c0f8>]  [<ffffffff8137c0f8>] io_serial_in+0x18/0x20
RSP: 0018:ffff8808ffcc3be8  EFLAGS: 00000002
RAX: ffff8808d2db7f00 RBX: ffffffff826876b0 RCX: 0000000000000000
RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff826876b0
RBP: ffff8808ffcc3be8 R08: 00000000f9cdb3aa R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000f32 R12: 000000000000270b
R13: 0000000000000020 R14: 0000000000000043 R15: ffffffff826876b0
FS:  0000000000000000(0000) GS:ffff8808ffcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/3 (pid: 0, threadinfo ffff8808d2db6000, task ffff8808d2db4280)
Stack:
 ffff8808ffcc3c18 ffffffff8137c96b 000000000000ffff ffffffff826876b0
 0000000000000020 ffffffff81d1f740 ffff8808ffcc3c38 ffffffff8137c9f6
 0000000000000006 ffffffff8137c9d0 ffff8808ffcc3c78 ffffffff81377fbd
Call Trace:
 <IRQ> 
 [<ffffffff8137c96b>] wait_for_xmitr+0x3b/0xa0
 [<ffffffff8137c9f6>] serial8250_console_putchar+0x26/0x40
 [<ffffffff8137c9d0>] ? wait_for_xmitr+0xa0/0xa0
 [<ffffffff81377fbd>] uart_console_write+0x3d/0x70
 [<ffffffff8137f423>] serial8250_console_write+0xc3/0x1a0
 [<ffffffff810577b7>] call_console_drivers+0xa7/0x1b0
 [<ffffffff81057ced>] console_unlock+0x10d/0x220
 [<ffffffff81058436>] vprintk_emit+0x1a6/0x610
 [<ffffffff810c36c1>] ? __lock_acquire+0x301/0x520
 [<ffffffff8159128c>] printk+0x4f/0x51
 [<ffffffff810fe8a4>] print_cpu_stall+0x74/0x130
 [<ffffffff8110064e>] __rcu_pending+0xee/0x210
 [<ffffffff81100878>] rcu_check_callbacks+0x108/0x310
 [<ffffffff81068c66>] update_process_times+0x46/0x90
 [<ffffffff810bbf66>] tick_sched_timer+0x66/0xc0
 [<ffffffff810bbf00>] ? tick_nohz_handler+0x100/0x100
 [<ffffffff81086a4e>] __run_hrtimer+0x12e/0x330
 [<ffffffff81086e9f>] hrtimer_interrupt+0xcf/0x200
 [<ffffffff815a04e9>] smp_apic_timer_interrupt+0x69/0x99
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 <EOI> 
 [<ffffffff81306a18>] ? intel_idle+0xc8/0x170
 [<ffffffff81306a14>] ? intel_idle+0xc4/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: 00 00 00 d3 e6 48 63 f6 48 03 77 40 8b 06 c9 c3 0f 1f 00 55 48 89 e5 66 66 66 66 90 0f b6 8f 91 00 00 00 89 f2 d3 e2 03 57 38 ec <0f> b6 c0 c9 c3 0f 1f 00 55 48 89 e5 66 66 66 66 90 0f b6 8f 91 
NMI backtrace for cpu 4
CPU 4 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/4 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff812aff5e>]  [<ffffffff812aff5e>] delay_tsc+0x5e/0xc0
RSP: 0018:ffff88117fc03d10  EFLAGS: 00000002
RAX: ffff8808d2dba000 RBX: 00000000000479ad RCX: 00000000f9d24aaa
RDX: 00000000f9d24aaa RSI: 0000000000000005 RDI: 00000000000479ad
RBP: ffff88117fc03d48 R08: 00000000f9d24aaa R09: ffffffff81af8a60
R10: 0000000000000001 R11: 0000000000000f34 R12: 0000000000001000
R13: 0000000000000002 R14: ffff8808d2dba000 R15: 0000000000000004
FS:  0000000000000000(0000) GS:ffff88117fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f9f8b3ba000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/4 (pid: 0, threadinfo ffff8808d2dba000, task ffff8808d2db82c0)
Stack:
 ffff8808d2dba000 00000004f9d24a08 0000000000000000 0000000000001000
 0000000000000002 000000000000dd40 0000000000000002 ffff88117fc03d58
 ffffffff812b0028 ffff88117fc03d78 ffffffff8103b212 0000000000000005
Call Trace:
 <IRQ> 
 [<ffffffff812b0028>] __const_udelay+0x28/0x30
 [<ffffffff8103b212>] native_safe_apic_wait_icr_idle+0x22/0x60
 [<ffffffff8103b77d>] default_send_IPI_mask_sequence_phys+0xdd/0x130
 [<ffffffff8103e829>] physflat_send_IPI_all+0x19/0x20
 [<ffffffff8103b85c>] arch_trigger_all_cpu_backtrace+0x6c/0xb0
 [<ffffffff810fe8a9>] print_cpu_stall+0x79/0x130
 [<ffffffff8110064e>] __rcu_pending+0xee/0x210
 [<ffffffff81100878>] rcu_check_callbacks+0x108/0x310
 [<ffffffff81068c66>] update_process_times+0x46/0x90
 [<ffffffff810bbf66>] tick_sched_timer+0x66/0xc0
 [<ffffffff810bbf00>] ? tick_nohz_handler+0x100/0x100
 [<ffffffff81086a4e>] __run_hrtimer+0x12e/0x330
 [<ffffffff81086e9f>] hrtimer_interrupt+0xcf/0x200
 [<ffffffff815a04e9>] smp_apic_timer_interrupt+0x69/0x99
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 <EOI> 
 [<ffffffff81306a18>] ? intel_idle+0xc8/0x170
 [<ffffffff81306a14>] ? intel_idle+0xc4/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: dd 00 00 66 66 90 0f ae e8 e8 2f fd d6 ff 66 90 89 45 d0 44 89 7d d4 4c 89 75 c8 eb 1d 48 8b 45 c8 83 68 1c 01 f3 90 83 40 1c 01 <65> 44 8b 3c 25 08 dd 00 00 44 3b 7d d4 75 36 66 66 90 0f ae e8 
NMI backtrace for cpu 5
CPU 5 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/5 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810699e7>]  [<ffffffff810699e7>] run_timer_softirq+0xb7/0x2c0
RSP: 0000:ffff88117fc43e30  EFLAGS: 00000002
RAX: 0000000100139276 RBX: 0000000000000075 RCX: ffff881152e08790
RDX: 0000000100139275 RSI: ffff88117fc43df0 RDI: ffff881152e08000
RBP: ffff88117fc43eb0 R08: ffff8808d0893d70 R09: ffff8808d2dbcb90
R10: ffff8808d2dbcb90 R11: 0000000000000001 R12: ffff881152e08000
R13: 0000000000000141 R14: ffff88117fc43e70 R15: ffffffff810784b0
FS:  0000000000000000(0000) GS:ffff88117fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/5 (pid: 0, threadinfo ffff8808d2dbe000, task ffff8808d2dbc300)
Stack:
 0000000000000002 ffff8808d2dbe000 ffff8808d2dbe000 ffff88117fc50ac0
 ffff881152e09c50 ffff881152e09850 ffff881152e09450 ffff881152e09050
 ffff88117fc43e70 ffff88117fc43e70 ffff88117fc43ea0 0000000000000282
Call Trace:
 <IRQ> 
 [<ffffffff81060e71>] __do_softirq+0x151/0x3e0
 [<ffffffff8159fb6c>] call_softirq+0x1c/0x30
 [<ffffffff810194fd>] do_softirq+0x9d/0xd0
 [<ffffffff81060555>] irq_exit+0xe5/0x100
 [<ffffffff815a04ee>] smp_apic_timer_interrupt+0x6e/0x99
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 <EOI> 
 [<ffffffff81306a18>] ? intel_idle+0xc8/0x170
 [<ffffffff81306a14>] ? intel_idle+0xc4/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: 00 00 00 00 48 8b 05 39 76 a9 00 49 8b 54 24 40 48 39 d0 0f 88 6b 01 00 00 89 d3 81 e3 ff 00 00 00 0f 84 35 01 00 00 48 8d 42 01 <49> 89 44 24 40 48 63 c3 48 83 c0 04 48 c1 e0 04 4a 8b 54 20 10 
NMI backtrace for cpu 6
CPU 6 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/6 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff81069a09>]  [<ffffffff81069a09>] run_timer_softirq+0xd9/0x2c0
RSP: 0000:ffff88117fc83e30  EFLAGS: 00000006
RAX: 0000000000000550 RBX: 0000000000000051 RCX: ffff881152e58560
RDX: ffff881152e58560 RSI: ffff88117fc83df0 RDI: ffff881152e58000
RBP: ffff88117fc83eb0 R08: ffffffffff640027 R09: ffff8808d2dc0bd0
R10: ffff8808d2dc0bd0 R11: 0000000000000001 R12: ffff881152e58000
R13: 0000000000000141 R14: ffff88117fc83e70 R15: ffffffff810689f0
FS:  0000000000000000(0000) GS:ffff88117fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/6 (pid: 0, threadinfo ffff8808d2dc2000, task ffff8808d2dc0340)
Stack:
 0000000000000002 ffff8808d2dc2000 ffff8808d2dc2000 ffff8808d28d4740
 ffff881152e59c50 ffff881152e59850 ffff881152e59450 ffff881152e59050
 ffff881152e58560 ffff88117fc83e70 ffff88117fc83ea0 0000000000000282
Call Trace:
 <IRQ> 
 [<ffffffff81060e71>] __do_softirq+0x151/0x3e0
 [<ffffffff8159fb6c>] call_softirq+0x1c/0x30
 [<ffffffff810194fd>] do_softirq+0x9d/0xd0
 [<ffffffff81060555>] irq_exit+0xe5/0x100
 [<ffffffff815a04ee>] smp_apic_timer_interrupt+0x6e/0x99
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 <EOI> 
 [<ffffffff81306a18>] ? intel_idle+0xc8/0x170
 [<ffffffff81306a14>] ? intel_idle+0xc4/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: 84 35 01 00 00 48 8d 42 01 49 89 44 24 40 48 63 c3 48 83 c0 04 48 c1 e0 04 4a 8b 54 20 10 49 8d 4c 04 10 48 89 55 c0 4c 89 72 08 <4a> 8b 54 20 18 48 89 55 c8 4c 89 32 4a 89 4c 20 10 4a 89 4c 20 
NMI backtrace for cpu 7
CPU 7 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 6687, comm: pickup Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff81043018>]  [<ffffffff81043018>] read_hpet+0x18/0x20
RSP: 0018:ffff88117fcc3e98  EFLAGS: 00000086
INFO: task stress_cpu_hotp:6903 blocked for more than 120 seconds.
RAX: 00000000516a74c9 RBX: ffffffff81a1efc0 RCX: 7fffffffffffffff
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
RDX: 0000000000000000 RSI: 000001732e339800 RDI: ffffffff81a1efc0
RBP: ffff88117fcc3e98 R08: fffffffffe04007f R09: ffff8808cdb43238
R10: ffff8808cdb43238 R11: 0000000000000000 R12: 00000000238c1d76
stress_cpu_hotp D 0000000000000000     0  6903   6850 0x00000000
R13: 00000000245c0de7 R14: 0000000000000639 R15: 000000000001dcbc
FS:  0000000000000000(0000) GS:ffff88117fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f63164d8dd8 CR3: 00000008c9745000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 ffff8808cda09ad8 0000000000000046 0000000000000001 ffff8808cda08010
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process pickup (pid: 6687, threadinfo ffff8808cc59a000, task ffff8808cdb42980)
Stack:
 ffff8808cda08000 ffff8808cda08000 ffff8808cda08000
 ffff88117fcc3ed8
 ffff8808cda09fd8

 ffffffff810b446c ffff88117fcce818
 ffff8808cda08000
 000001732e339800

 ffff8808cda09fd8 ffff8808d04f45c0
 ffff88117fccdd80
 ffff8808d2dae240

Call Trace:
 0000000000000000 000001732e339800 000001732e339800
 [<ffffffff81465164>] ? cpufreq_notify_transition+0x174/0x2f0
 ffff88117fcc3f08 ffffffff810ba353 000001732e339800
 [<ffffffffa00b1200>] ? acpi_cpufreq_cpu_exit+0xb0/0xb0 [acpi_cpufreq]
 ffff88117fcce8c0
Call Trace:
 [<ffffffff810c21f0>] ? mark_held_locks+0x80/0x120
 <IRQ> 
 [<ffffffff81591ab7>] ? __mutex_unlock_slowpath+0xf7/0x190
 [<ffffffff810b446c>] ktime_get+0x6c/0xe0
 [<ffffffff815939c9>] schedule+0x29/0x70
 [<ffffffff810ba353>] clockevents_program_event+0x43/0x100
 [<ffffffff815915e5>] schedule_timeout+0x2d5/0x3f0
 [<ffffffff810bbae4>] tick_program_event+0x24/0x30
 [<ffffffff810c21f0>] ? mark_held_locks+0x80/0x120
 [<ffffffff81086ee6>] hrtimer_interrupt+0x116/0x200
 [<ffffffff81594eb0>] ? _raw_spin_unlock_irq+0x30/0x50
 [<ffffffff815a04e9>] smp_apic_timer_interrupt+0x69/0x99
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 [<ffffffff81593845>] wait_for_common+0x115/0x190
 <EOI> 
 [<ffffffff8118cd93>] ? kmem_cache_free+0x183/0x280
 [<ffffffff810983d0>] ? try_to_wake_up+0x2f0/0x2f0
 [<ffffffff81171e8e>] unlink_anon_vmas+0xae/0x180
 [<ffffffff81078bd2>] ? queue_delayed_work_on+0x132/0x140
 [<ffffffff81167039>] free_pgtables+0xa9/0x110
 [<ffffffff8159399d>] wait_for_completion+0x1d/0x20
 [<ffffffff8116c766>] exit_mmap+0x96/0x140
 [<ffffffff81043839>] hpet_cpuhp_notify+0x119/0x160
 [<ffffffff810545e7>] mmput+0x87/0x110
 [<ffffffff8105affa>] exit_mm+0x10a/0x140
 [<ffffffff81043a40>] ? init_one_hpet_msi_clockevent+0x1c0/0x1c0
 [<ffffffff81594eb0>] ? _raw_spin_unlock_irq+0x30/0x50
 [<ffffffff810784b0>] ? __queue_work+0x500/0x500
 [<ffffffff81365598>] ? tty_audit_exit+0x28/0xa0
 [<ffffffff81599512>] notifier_call_chain+0x72/0x130
 [<ffffffff8105daa8>] do_exit+0x168/0x4b0
 [<ffffffff810890fe>] __raw_notifier_call_chain+0xe/0x10
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff8158af88>] _cpu_up+0x15b/0x164
 [<ffffffff8105de3c>] do_group_exit+0x4c/0xc0
 [<ffffffff8158b04d>] cpu_up+0xbc/0x11f
 [<ffffffff8105dec7>] sys_exit_group+0x17/0x20
 [<ffffffff8157a87d>] store_online+0x9d/0xe0
 [<ffffffff8159e629>] system_call_fastpath+0x16/0x1b
 [<ffffffff81396be0>] dev_attr_store+0x20/0x30
Code: 
 [<ffffffff81222119>] sysfs_write_file+0xd9/0x150
c1 80 c9 
 [<ffffffff811a6a5b>] vfs_write+0xcb/0x130
0f 4c c2 
 [<ffffffff811a6bb5>] sys_write+0x55/0x90
c3 66 66 
 [<ffffffff8159e629>] system_call_fastpath+0x16/0x1b
5 locks held by stress_cpu_hotp/6903:
66 2e 0f 1f 
 #0: 
84 
 (
00 
&buffer->mutex
00 
){+.+.+.}
00 
, at: 
[<ffffffff81222089>] sysfs_write_file+0x49/0x150
00 00 
 #1: 
55 
 (
48 
s_active
89 
#111
e5 
){.+.+.+}
66 
, at: 
[<ffffffff81222102>] sysfs_write_file+0xc2/0x150
66 66 
 #2: 
66 
 (
90 
x86_cpu_hotplug_driver_mutex
48 
){+.+.+.}
8b 
, at: 
[<ffffffff81039df7>] cpu_hotplug_driver_lock+0x17/0x20
05 a8 
 #3: 
36 
 (
cc 
cpu_add_remove_lock
00 
){+.+.+.}
48 
, at: 
[<ffffffff8158b009>] cpu_up+0x78/0x11f
05 f0 
 #4: 
00 
 (
00 
cpu_hotplug.lock
00 
){+.+.+.}
8b 
, at: 
[<ffffffff81059cef>] cpu_hotplug_begin+0x2f/0x60
00 <89> c0 c9 c3 0f 1f 40 00 55 48 89 e5 66 66 66 66 90 8b 35 91 36 
NMI backtrace for cpu 8
CPU 8 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/8 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff81069a15>]  [<ffffffff81069a15>] run_timer_softirq+0xe5/0x2c0
RSP: 0018:ffff8808ffd03e30  EFLAGS: 00000002
RAX: 0000000000000ce0 RBX: 00000000000000ca RCX: ffff8808d2f98cf0
RDX: ffff8808ffd03e70 RSI: ffff8808ffd03df0 RDI: ffff8808d2f98000
RBP: ffff8808ffd03eb0 R08: ffffffffff640027 R09: ffff8808d2dcac50
R10: ffff8808d2dcac50 R11: 0000000000000001 R12: ffff8808d2f98000
R13: 0000000000000141 R14: ffff8808ffd03e70 R15: ffffffff810784b0
FS:  0000000000000000(0000) GS:ffff8808ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/8 (pid: 0, threadinfo ffff8808d2dcc000, task ffff8808d2dca3c0)
Stack:
 0000000000000002 ffff8808d2dcc000 ffff8808d2dcc000 ffff8808ffd109a0
 ffff8808d2f99c50 ffff8808d2f99850 ffff8808d2f99450 ffff8808d2f99050
 ffff8808ffd03e70 ffff8808ffd03e70 ffff8808ffd03ea0 0000000000000282
Call Trace:
 <IRQ> 
 [<ffffffff81060e71>] __do_softirq+0x151/0x3e0
 [<ffffffff8159fb6c>] call_softirq+0x1c/0x30
 [<ffffffff810194fd>] do_softirq+0x9d/0xd0
 [<ffffffff81060555>] irq_exit+0xe5/0x100
 [<ffffffff815a04ee>] smp_apic_timer_interrupt+0x6e/0x99
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 <EOI> 
 [<ffffffff81306a18>] ? intel_idle+0xc8/0x170
 [<ffffffff81306a14>] ? intel_idle+0xc4/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: 24 40 48 63 c3 48 83 c0 04 48 c1 e0 04 4a 8b 54 20 10 49 8d 4c 04 10 48 89 55 c0 4c 89 72 08 4a 8b 54 20 18 48 89 55 c8 4c 89 32 <4a> 89 4c 20 10 4a 89 4c 20 18 48 8b 5d c0 4c 39 f3 74 98 48 8b 
NMI backtrace for cpu 9
CPU 9 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 5, comm: kworker/u:0 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff8107a6d2>]  [<ffffffff8107a6d2>] process_one_work+0x312/0x480
RSP: 0018:ffff8808d2d3dd40  EFLAGS: 00000046
RAX: 0000000000000000 RBX: ffff8808cfb34030 RCX: 000000002dfe2dfe
RDX: 0000000000000000 RSI: 000000008d0b56c0 RDI: ffffffff81da2d00
RBP: ffff8808d2d3ddd0 R08: fffffffffe340073 R09: ffff8808d2d3a9d0
R10: ffff8808d2d3a9d0 R11: 0000000000000001 R12: ffff8808cfb34028
R13: ffff881152ca8980 R14: ffffffff81da2d00 R15: ffff8808d0b56c00
FS:  0000000000000000(0000) GS:ffff8808ffd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f28bade13e0 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/u:0 (pid: 5, threadinfo ffff8808d2d3c000, task ffff8808d2d3a140)
Stack:
 ffffffff8107a4ff 0000000000000046 0000000000000004 ffffffff8107b2d8
 ffffffff00000000 0000000000000000 ffffffffa0b23b00 0ffff8808d0b56c0
 ffffffffa0b26f98 ffffffff820155d0 0000000000000000 ffffffffa0b25353
Call Trace:
 [<ffffffff8107a4ff>] ? process_one_work+0x13f/0x480
 [<ffffffff8107b2d8>] ? worker_thread+0x68/0x3b0
 [<ffffffffa0b23b00>] ? edac_pci_reset_delay_period+0x60/0x60 [edac_core]
 [<ffffffff8107b3f6>] worker_thread+0x186/0x3b0
 [<ffffffff8107b270>] ? manage_workers+0x120/0x120
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 48 89 02 74 04 48 89 50 08 49 c7 45 00 00 00 00 00 49 c7 45 08 00 00 00 00 48 c1 6d a8 04 49 c7 45 10 00 00 00 00 31 d2 8b 75 a8 <49> c7 45 18 00 00 00 00 4c 89 ff 83 e6 0f e8 2b fc ff ff 48 83 
NMI backtrace for cpu 10
CPU 10 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16471, comm: kworker/10:1 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff8108fe29>]  [<ffffffff8108fe29>] __might_sleep+0x19/0x240
RSP: 0018:ffff8808c9657cc0  EFLAGS: 00000286
RAX: 000000010013bf80 RBX: ffff8808d253d900 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 000000000000109d RDI: ffffffff817c61e4
RBP: ffff8808c9657ce0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000003 R11: 0000000000000001 R12: ffff8808d253d840
R13: ffff8808d25309c0 R14: 0000000000000001 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8808ffd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fee87eccb60 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/10:1 (pid: 16471, threadinfo ffff8808c9656000, task ffff8808cf8e4380)
Stack:
 ffff8808d253d840 ffff8808d253d900 ffff8808d253d840 ffff8808d25309c0
 ffff8808c9657d30 ffffffff8118d7c4 ffff8808ffd90ac0 00000000c9657d80
 0000000000000000 ffff8808ffd90ac8 ffff8808ffd90ac0 ffff8808cf888140
Call Trace:
 [<ffffffff8118d7c4>] cache_reap+0xf4/0x270
 [<ffffffff8107a5c1>] process_one_work+0x201/0x480
 [<ffffffff8107a4ff>] ? process_one_work+0x13f/0x480
 [<ffffffff8107b2d8>] ? worker_thread+0x68/0x3b0
 [<ffffffff8118d6d0>] ? drain_freelist+0xb0/0xb0
 [<ffffffff8107b3f6>] worker_thread+0x186/0x3b0
 [<ffffffff8107b270>] ? manage_workers+0x120/0x120
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 55 55 48 83 c4 08 5b c9 c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 4c 89 6d f8 66 66 66 66 90 <49> 89 fd 41 89 f4 89 d3 e8 2a d6 fe ff 85 c0 74 0d 80 3d d8 e6 
NMI backtrace for cpu 11
CPU 11 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/11 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810699f7>]  [<ffffffff810699f7>] run_timer_softirq+0xc7/0x2c0
RSP: 0018:ffff8808ffdc3e30  EFLAGS: 00000002
RAX: 0000000000000e70 RBX: 00000000000000e3 RCX: ffff8808d2898e70
RDX: 0000000100131fe3 RSI: ffff8808ffdc3df0 RDI: ffff8808d2898000
RBP: ffff8808ffdc3eb0 R08: ffffffffff640027 R09: ffff8808d2dd8d10
R10: ffff8808d2dd8d10 R11: 0000000000000001 R12: ffff8808d2898000
R13: 0000000000000141 R14: ffff8808ffdc3e70 R15: ffffffff810784b0
FS:  0000000000000000(0000) GS:ffff8808ffdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/11 (pid: 0, threadinfo ffff8808d2dda000, task ffff8808d2dd8480)
Stack:
 0000000000000002 ffff8808d2dda000 ffff8808d2dda000 ffff8808ffdd09a0
 ffff8808d2899c50 ffff8808d2899850 ffff8808d2899450 ffff8808d2899050
 ffff8808ffdc3e70 ffff8808ffdc3e70 ffff8808ffdc3ea0 0000000000000282
Call Trace:
 <IRQ> 
 [<ffffffff81060e71>] __do_softirq+0x151/0x3e0
 [<ffffffff8159fb6c>] call_softirq+0x1c/0x30
 [<ffffffff810194fd>] do_softirq+0x9d/0xd0
 [<ffffffff81060555>] irq_exit+0xe5/0x100
 [<ffffffff815a04ee>] smp_apic_timer_interrupt+0x6e/0x99
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 <EOI> 
 [<ffffffff81306a18>] ? intel_idle+0xc8/0x170
 [<ffffffff81306a14>] ? intel_idle+0xc4/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: 48 39 d0 0f 88 6b 01 00 00 89 d3 81 e3 ff 00 00 00 0f 84 35 01 00 00 48 8d 42 01 49 89 44 24 40 48 63 c3 48 83 c0 04 48 c1 e0 04 <4a> 8b 54 20 10 49 8d 4c 04 10 48 89 55 c0 4c 89 72 08 4a 8b 54 
lockdep: fixing up alternatives.
Booting Node 1 Processor 12 APIC 0x11
sending NMI to all CPUs:
NMI backtrace for cpu 9
CPU 9 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16456, comm: migration/9 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d5>]  [<ffffffff810e61d5>] stop_machine_cpu_stop+0x85/0x140
RSP: 0018:ffff8808d2f93d20  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808d2da0400
RDX: ffff8808d2da0400 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff8808d2f93d50 R08: ffff8808ffd4ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff8808ffd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/9 (pid: 16456, threadinfo ffff8808d2f92000, task ffff8808d2da0400)
Stack:
 00000000d2f93d30 ffff8808d2ddfe28 ffff8808d2da0400 ffff8808ffd4eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808d2f93e50 ffffffff810e5e53
 ffff8808d2f92000 ffff8808d2f93fd8 ffff8808d2f92000 ffff8808ffd4ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 03 74 4b f0 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 <89> d8 f3 90 41 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 
NMI backtrace for cpu 10
CPU 10 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16466, comm: migration/10 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d9>]  [<ffffffff810e61d9>] stop_machine_cpu_stop+0x89/0x140
RSP: 0018:ffff8808cfa63d20  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808cbcc81c0
RDX: ffff8808cbcc81c0 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff8808cfa63d50 R08: ffff8808ffd8ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff8808ffd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/10 (pid: 16466, threadinfo ffff8808cfa62000, task ffff8808cbcc81c0)
Stack:
 00000000cfa63d30 ffff8808d2ddfe28 ffff8808cbcc81c0 ffff8808ffd8eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808cfa63e50 ffffffff810e5e53
 ffff8808cfa62000 ffff8808cfa63fd8 ffff8808cfa62000 ffff8808ffd8ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 89 d8 f3 90 <41> 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 90 66 66 90 
NMI backtrace for cpu 11
CPU 11 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16475, comm: migration/11 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d0>]  [<ffffffff810e61d0>] stop_machine_cpu_stop+0x80/0x140
RSP: 0018:ffff8808ce103d20  EFLAGS: 00000246
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808ce0a4640
RDX: ffff8808ce0a4640 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff8808ce103d50 R08: ffff8808ffdcebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff8808ffdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/11 (pid: 16475, threadinfo ffff8808ce102000, task ffff8808ce0a4640)
Stack:
 00000000ce103d30 ffff8808d2ddfe28 ffff8808ce0a4640 ffff8808ffdceb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808ce103e50 ffffffff810e5e53
 ffff8808ce102000 ffff8808ce103fd8 ffff8808ce102000 ffff8808ffdcebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 44 00 00 83 fb 03 74 4b f0 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 <83> fb 04 74 43 89 d8 f3 90 41 8b 5c 24 20 39 c3 74 ee 83 fb 02 
NMI backtrace for cpu 2
CPU 2 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16393, comm: migration/2 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d9>]  [<ffffffff810e61d9>] stop_machine_cpu_stop+0x89/0x140
RSP: 0018:ffff8808cdaf1d20  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808cb53a400
RDX: ffff8808cb53a400 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff8808cdaf1d50 R08: ffff8808ffc8ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff8808ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f97183ca000 CR3: 00000008c965d000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/2 (pid: 16393, threadinfo ffff8808cdaf0000, task ffff8808cb53a400)
Stack:
 00000000cdaf1d30 ffff8808d2ddfe28 ffff8808cb53a400 ffff8808ffc8eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808cdaf1e50 ffffffff810e5e53
 ffff8808cdaf0000 ffff8808cdaf1fd8 ffff8808cdaf0000 ffff8808ffc8ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 89 d8 f3 90 <41> 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 90 66 66 90 
NMI backtrace for cpu 0
CPU 0 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 6, comm: migration/0 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d9>]  [<ffffffff810e61d9>] stop_machine_cpu_stop+0x89/0x140
RSP: 0018:ffff8808d2d83d20  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808d2d80180
RDX: ffff8808d2d80180 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff8808d2d83d50 R08: ffff8808ffc0ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff8808ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/0 (pid: 6, threadinfo ffff8808d2d82000, task ffff8808d2d80180)
Stack:
 00000000d2d83d30 ffff8808d2ddfe28 ffff8808d2d80180 ffff8808ffc0eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808d2d83e50 ffffffff810e5e53
 ffff8808d2d82000 ffff8808d2d83fd8 ffff8808d2d82000 ffff8808ffc0ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 89 d8 f3 90 <41> 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 90 66 66 90 
NMI backtrace for cpu 3
CPU 3 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 0, comm: swapper/3 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff8137c0f8>]  [<ffffffff8137c0f8>] io_serial_in+0x18/0x20
RSP: 0018:ffff8808ffcc3bc8  EFLAGS: 00000002
RAX: ffff8808d2db7f00 RBX: ffffffff826876b0 RCX: 0000000000000000
RDX: 00000000000002fd RSI: 0000000000000005 RDI: ffffffff826876b0
RBP: ffff8808ffcc3bc8 R08: 0000000067ca504e R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000001191 R12: 0000000000002709
R13: 0000000000000020 R14: 00000000000000c9 R15: ffffffff826876b0
FS:  0000000000000000(0000) GS:ffff8808ffcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffffff600400 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process swapper/3 (pid: 0, threadinfo ffff8808d2db6000, task ffff8808d2db4280)
Stack:
 ffff8808ffcc3bf8 ffffffff8137c96b ffffffff812aff78 ffffffff826876b0
 0000000000000030 ffffffff81d1f740 ffff8808ffcc3c18 ffffffff8137c9f6
 000000000000000a ffffffff8137c9d0 ffff8808ffcc3c58 ffffffff81377fbd
Call Trace:
 <IRQ> 
 [<ffffffff8137c96b>] wait_for_xmitr+0x3b/0xa0
 [<ffffffff812aff78>] ? delay_tsc+0x78/0xc0
 [<ffffffff8137c9f6>] serial8250_console_putchar+0x26/0x40
 [<ffffffff8137c9d0>] ? wait_for_xmitr+0xa0/0xa0
 [<ffffffff81377fbd>] uart_console_write+0x3d/0x70
 [<ffffffff8137f423>] serial8250_console_write+0xc3/0x1a0
 [<ffffffff810577b7>] call_console_drivers+0xa7/0x1b0
 [<ffffffff81057ced>] console_unlock+0x10d/0x220
 [<ffffffff81058436>] vprintk_emit+0x1a6/0x610
 [<ffffffff8159128c>] printk+0x4f/0x51
 [<ffffffff8159128c>] ? printk+0x4f/0x51
 [<ffffffff8103b84a>] arch_trigger_all_cpu_backtrace+0x5a/0xb0
 [<ffffffff810fe8a9>] print_cpu_stall+0x79/0x130
 [<ffffffff8110064e>] __rcu_pending+0xee/0x210
 [<ffffffff81100878>] rcu_check_callbacks+0x108/0x310
 [<ffffffff81068c66>] update_process_times+0x46/0x90
 [<ffffffff810bbf66>] tick_sched_timer+0x66/0xc0
 [<ffffffff810bbf00>] ? tick_nohz_handler+0x100/0x100
 [<ffffffff81086a4e>] __run_hrtimer+0x12e/0x330
 [<ffffffff81086e9f>] hrtimer_interrupt+0xcf/0x200
 [<ffffffff815a04e9>] smp_apic_timer_interrupt+0x69/0x99
 [<ffffffff8159f16f>] apic_timer_interrupt+0x6f/0x80
 <EOI> 
 [<ffffffff81306a18>] ? intel_idle+0xc8/0x170
 [<ffffffff81306a14>] ? intel_idle+0xc4/0x170
 [<ffffffff81467107>] cpuidle_enter+0x17/0x20
 [<ffffffff81467ab3>] cpuidle_idle_call+0x143/0x290
 [<ffffffff810205c5>] cpu_idle+0xe5/0x140
 [<ffffffff8158991d>] start_secondary+0xd1/0xd4
Code: 00 00 00 d3 e6 48 63 f6 48 03 77 40 8b 06 c9 c3 0f 1f 00 55 48 89 e5 66 66 66 66 90 0f b6 8f 91 00 00 00 89 f2 d3 e2 03 57 38 ec <0f> b6 c0 c9 c3 0f 1f 00 55 48 89 e5 66 66 66 66 90 0f b6 8f 91 
NMI backtrace for cpu 1
CPU 1 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16384, comm: migration/1 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d9>]  [<ffffffff810e61d9>] stop_machine_cpu_stop+0x89/0x140
RSP: 0018:ffff8808cda55d20  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808d0db23c0
RDX: ffff8808d0db23c0 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff8808cda55d50 R08: ffff8808ffc4ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff8808ffc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f8d3cfa8570 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/1 (pid: 16384, threadinfo ffff8808cda54000, task ffff8808d0db23c0)
Stack:
 00000000cda55d30 ffff8808d2ddfe28 ffff8808d0db23c0 ffff8808ffc4eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808cda55e50 ffffffff810e5e53
 ffff8808cda54000 ffff8808cda55fd8 ffff8808cda54000 ffff8808ffc4ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 89 d8 f3 90 <41> 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 90 66 66 90 
NMI backtrace for cpu 8
CPU 8 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16447, comm: migration/8 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d9>]  [<ffffffff810e61d9>] stop_machine_cpu_stop+0x89/0x140
RSP: 0018:ffff8808d0819d20  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff8808cf92e040
RDX: ffff8808cf92e040 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff8808d0819d50 R08: ffff8808ffd0ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff8808ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fee875cf0a0 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/8 (pid: 16447, threadinfo ffff8808d0818000, task ffff8808cf92e040)
Stack:
 00000000d0819d30 ffff8808d2ddfe28 ffff8808cf92e040 ffff8808ffd0eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff8808d0819e50 ffffffff810e5e53
 ffff8808d0818000 ffff8808d0819fd8 ffff8808d0818000 ffff8808ffd0ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 89 d8 f3 90 <41> 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 90 66 66 90 
NMI backtrace for cpu 4
CPU 4 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16411, comm: migration/4 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d9>]  [<ffffffff810e61d9>] stop_machine_cpu_stop+0x89/0x140
RSP: 0018:ffff88114d66dd20  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff881152f0aa00
RDX: ffff881152f0aa00 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff88114d66dd50 R08: ffff88117fc0ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff88117fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f9f8b3ba000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/4 (pid: 16411, threadinfo ffff88114d66c000, task ffff881152f0aa00)
Stack:
 000000004d66dd30 ffff8808d2ddfe28 ffff881152f0aa00 ffff88117fc0eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff88114d66de50 ffffffff810e5e53
 ffff88114d66c000 ffff88114d66dfd8 ffff88114d66c000 ffff88117fc0ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 89 d8 f3 90 <41> 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 90 66 66 90 
NMI backtrace for cpu 5
CPU 5 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16420, comm: migration/5 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d0>]  [<ffffffff810e61d0>] stop_machine_cpu_stop+0x80/0x140
RSP: 0018:ffff881150bfdd20  EFLAGS: 00000246
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff881152cc2780
RDX: ffff881152cc2780 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff881150bfdd50 R08: ffff88117fc4ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff88117fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f9f8b3ba000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/5 (pid: 16420, threadinfo ffff881150bfc000, task ffff881152cc2780)
Stack:
 0000000050bfdd30 ffff8808d2ddfe28 ffff881152cc2780 ffff88117fc4eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff881150bfde50 ffffffff810e5e53
 ffff881150bfc000 ffff881150bfdfd8 ffff881150bfc000 ffff88117fc4ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 44 00 00 83 fb 03 74 4b f0 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 <83> fb 04 74 43 89 d8 f3 90 41 8b 5c 24 20 39 c3 74 ee 83 fb 02 
NMI backtrace for cpu 7
CPU 7 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16438, comm: migration/7 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d0>]  [<ffffffff810e61d0>] stop_machine_cpu_stop+0x80/0x140
RSP: 0018:ffff88114d437d20  EFLAGS: 00000246
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff881152fac900
RDX: ffff881152fac900 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff88114d437d50 R08: ffff88117fccebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff88117fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f63164d8dd8 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/7 (pid: 16438, threadinfo ffff88114d436000, task ffff881152fac900)
Stack:
 000000004d437d30 ffff8808d2ddfe28 ffff881152fac900 ffff88117fcceb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff88114d437e50 ffffffff810e5e53
 ffff88114d436000 ffff88114d437fd8 ffff88114d436000 ffff88117fccebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 44 00 00 83 fb 03 74 4b f0 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 <83> fb 04 74 43 89 d8 f3 90 41 8b 5c 24 20 39 c3 74 ee 83 fb 02 
NMI backtrace for cpu 6
CPU 6 
Modules linked in: ipv6 cpufreq_ondemand cpufreq_powersave acpi_cpufreq freq_table mperf dm_mod ioatdma cdc_ether usbnet mii bnx2 i7core_edac kvm_intel i2c_i801 dca shpchp edac_core i2c_core sg kvm lpc_ich mfd_core microcode serio_raw sd_mod crc_t10dif ext3 mbcache jbd mptsas mptscsih mptbase scsi_transport_sas

Pid: 16429, comm: migration/6 Tainted: G        W    3.5.0-rc1-threadhotplugv3-paulfix #1 IBM IBM System x -[7870C4Q]-/68Y8033     
RIP: 0010:[<ffffffff810e61d5>]  [<ffffffff810e61d5>] stop_machine_cpu_stop+0x85/0x140
RSP: 0000:ffff881152e93d20  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff881152664840
RDX: ffff881152664840 RSI: 0000000000000001 RDI: ffff8808d2ddfdf8
RBP: ffff881152e93d50 R08: ffff88117fc8ebc0 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8808d2ddfdf8
R13: ffff8808d2ddfe1c R14: 0000000000000282 R15: ffff8808d2ddfd01
FS:  0000000000000000(0000) GS:ffff88117fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000001a0c000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process migration/6 (pid: 16429, threadinfo ffff881152e92000, task ffff881152664840)
Stack:
 0000000052e93d30 ffff8808d2ddfe28 ffff881152664840 ffff88117fc8eb80
 ffffffff810e6150 ffff8808d2ddfdf8 ffff881152e93e50 ffffffff810e5e53
 ffff881152e92000 ffff881152e93fd8 ffff881152e92000 ffff88117fc8ebc0
Call Trace:
 [<ffffffff810e6150>] ? stop_one_cpu_nowait+0x50/0x50
 [<ffffffff810e5e53>] cpu_stopper_thread+0xf3/0x1e0
 [<ffffffff81594f14>] ? _raw_spin_unlock_irqrestore+0x44/0x90
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff810c262d>] ? trace_hardirqs_on+0xd/0x10
 [<ffffffff810e5d60>] ? res_counter_init+0x60/0x60
 [<ffffffff810811ee>] kthread+0xee/0x100
 [<ffffffff8159fa74>] kernel_thread_helper+0x4/0x10
 [<ffffffff815952b0>] ? retint_restore_args+0x13/0x13
 [<ffffffff81081100>] ? __init_kthread_worker+0x70/0x70
 [<ffffffff8159fa70>] ? gs_change+0x13/0x13
Code: 03 74 4b f0 41 ff 4d 00 0f 94 c0 84 c0 74 17 41 8b 54 24 20 41 8b 44 24 10 83 c2 01 41 89 44 24 24 41 89 54 24 20 83 fb 04 74 43 <89> d8 f3 90 41 8b 5c 24 20 39 c3 74 ee 83 fb 02 75 c1 fa 66 66 
Uhhuh. NMI received for unknown reason 3d on CPU 2.
Uhhuh. NMI received for unknown reason 2d on CPU 3.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 3d on CPU 4.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 3d on CPU 5.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 2d on CPU 6.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 2d on CPU 7.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 3d on CPU 8.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 2d on CPU 9.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 2d on CPU 10.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Uhhuh. NMI received for unknown reason 3d on CPU 11.
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
Do you have a strange power saving mode enabled?
Dazed and confused, but trying to continue
NMI watchdog: enabled, takes one hw-pmu counter.
Booting Node 1 Processor 12 APIC 0x11
NMI watchdog: enabled, takes one hw-pmu counter.
INFO: task stress_cpu_hotp:6903 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
stress_cpu_hotp D 0000000000000000     0  6903   6850 0x00000000
 ffff8808cda09ad8 0000000000000046 0000000000000001 ffff8808cda08010
 ffff8808cda08000 ffff8808cda08000 ffff8808cda08000 ffff8808cda09fd8
 ffff8808cda08000 ffff8808cda09fd8 ffff8808d04f45c0 ffff8808d2dbc300
Call Trace:
 [<ffffffff81465164>] ? cpufreq_notify_transition+0x174/0x2f0
 [<ffffffffa00b1200>] ? acpi_cpufreq_cpu_exit+0xb0/0xb0 [acpi_cpufreq]
 [<ffffffff810c21f0>] ? mark_held_locks+0x80/0x120
 [<ffffffff81591ab7>] ? __mutex_unlock_slowpath+0xf7/0x190
 [<ffffffff815939c9>] schedule+0x29/0x70
 [<ffffffff815915e5>] schedule_timeout+0x2d5/0x3f0
 [<ffffffff810c21f0>] ? mark_held_locks+0x80/0x120
 [<ffffffff81594eb0>] ? _raw_spin_unlock_irq+0x30/0x50
 [<ffffffff810c259d>] ? trace_hardirqs_on_caller+0x12d/0x1b0
 [<ffffffff81593845>] wait_for_common+0x115/0x190
 [<ffffffff810983d0>] ? try_to_wake_up+0x2f0/0x2f0
 [<ffffffff81078bd2>] ? queue_delayed_work_on+0x132/0x140
 [<ffffffff8159399d>] wait_for_completion+0x1d/0x20
 [<ffffffff81043839>] hpet_cpuhp_notify+0x119/0x160
 [<ffffffff81043a40>] ? init_one_hpet_msi_clockevent+0x1c0/0x1c0
 [<ffffffff810784b0>] ? __queue_work+0x500/0x500
 [<ffffffff81599512>] notifier_call_chain+0x72/0x130
 [<ffffffff810890fe>] __raw_notifier_call_chain+0xe/0x10
 [<ffffffff8158af88>] _cpu_up+0x15b/0x164
 [<ffffffff8158b04d>] cpu_up+0xbc/0x11f
 [<ffffffff8157a87d>] store_online+0x9d/0xe0
 [<ffffffff81396be0>] dev_attr_store+0x20/0x30
 [<ffffffff81222119>] sysfs_write_file+0xd9/0x150
 [<ffffffff811a6a5b>] vfs_write+0xcb/0x130
 [<ffffffff811a6bb5>] sys_write+0x55/0x90
 [<ffffffff8159e629>] system_call_fastpath+0x16/0x1b
5 locks held by stress_cpu_hotp/6903:
 #0:  (&buffer->mutex){+.+.+.}, at: [<ffffffff81222089>] sysfs_write_file+0x49/0x150
 #1:  (s_active#111){.+.+.+}, at: [<ffffffff81222102>] sysfs_write_file+0xc2/0x150
 #2:  (x86_cpu_hotplug_driver_mutex){+.+.+.}, at: [<ffffffff81039df7>] cpu_hotplug_driver_lock+0x17/0x20
 #3:  (cpu_add_remove_lock){+.+.+.}, at: [<ffffffff8158b009>] cpu_up+0x78/0x11f
 #4:  (cpu_hotplug.lock){+.+.+.}, at: [<ffffffff81059cef>] cpu_hotplug_begin+0x2f/0x60
lockdep: fixing up alternatives.
Booting Node 1 Processor 13 APIC 0x13
CPU 1 is now offline
CPU 9 MCA banks CMCI:2 CMCI:3 CMCI:5
NOHZ: local_softirq_pending 200
NOHZ: local_softirq_pending 200