cpu stall and hyperthread

[Dong Liu <dliu.cn@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 380 bytes --]

Hi,

I'm running 3.4.3-rt11 and 3.2.18-rt29 on a system with Intel I7 860, I 
noticed that with hyperthread enabled, I'm getting kernel error

  rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 5, 
t=60002 jiffies)

But with hyperthread disabled, no such error.

Is preempt_rt not supposed to be used with hyperthread?

Thanks,

Dong

P.S. Full dmesg output attached.

[-- Attachment #2: rt-cpu-stall-with-ht.txt --]
[-- Type: text/plain, Size: 31828 bytes --]

[ 7218.536199] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 5, t=60002 jiffies)
[ 7218.536210] sending NMI to all CPUs:
[ 7218.536218] NMI backtrace for cpu 7
[ 7218.536222] CPU 7 
[ 7218.536225] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle 8021q garp sunrpc bridge stp llc binfmt_misc vhost_net macvtap macvlan tun kvm_intel kvm uinput joydev microcode serio_raw pcspkr i7core_edac edac_core i2c_i801 iTCO_wdt iTCO_vendor_support r8169 mii snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video [last unloaded: scsi_wait_scan]
[ 7218.536293] 
[ 7218.536298] Pid: 0, comm: swapper/7 Not tainted 3.2.18-1.el6.alu_preempt_rt.x86_64-rt29 #1 Hewlett-Packard HP Elite 7100 Microtower PC/2A9Ch
[ 7218.536306] RIP: 0010:[<ffffffff8103dbb6>]  [<ffffffff8103dbb6>] hpet_legacy_next_event+0x16/0x50
[ 7218.536318] RSP: 0018:ffff880122943ce8  EFLAGS: 00000002
[ 7218.536322] RAX: 000000001ac98327 RBX: ffffffff81c1a500 RCX: 0000000000000020
[ 7218.536327] RDX: 0000000225c17d03 RSI: ffffffff81c1a500 RDI: 0000000000002e45
[ 7218.536331] RBP: ffff880122943ce8 R08: 0000000000000000 R09: 0000000000000000
[ 7218.536335] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 7218.536339] R13: 00000000000c9f9a R14: 0000000000000004 R15: 0000000000000007
[ 7218.536344] FS:  0000000000000000(0000) GS:ffff88012fdc0000(0000) knlGS:0000000000000000
[ 7218.536349] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7218.536354] CR2: 0000000001173200 CR3: 0000000001c08000 CR4: 00000000000026e0
[ 7218.536358] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7218.536363] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7218.536368] Process swapper/7 (pid: 0, threadinfo ffff880122942000, task ffff880122940500)
[ 7218.536371] Stack:
[ 7218.536374]  ffff880122943d18 ffffffff810a9034 ffff880122943d18 0000000000000002
[ 7218.536382]  ffff88012fdcba40 ffffffff81c1a500 ffff880122943d58 ffffffff810a9dd8
[ 7218.536389]  ffff880122943d38 ffff880122943e84 ffff880122943e84 00000000ffffffff
[ 7218.536396] Call Trace:
[ 7218.536405]  [<ffffffff810a9034>] clockevents_program_event+0x74/0x120
[ 7218.536413]  [<ffffffff810a9dd8>] tick_broadcast_oneshot_control+0x128/0x130
[ 7218.536420]  [<ffffffff810a94f5>] tick_notify+0x325/0x410
[ 7218.536427]  [<ffffffff8101e542>] ? __switch_to_xtra+0x172/0x1a0
[ 7218.536435]  [<ffffffff816005c5>] notifier_call_chain+0x55/0x80
[ 7218.536443]  [<ffffffff8109f036>] raw_notifier_call_chain+0x16/0x20
[ 7218.536450]  [<ffffffff810a89ad>] clockevents_notify+0x3d/0x140
[ 7218.536458]  [<ffffffff81326006>] intel_idle+0x136/0x150
[ 7218.536466]  [<ffffffff814b0919>] ? menu_select+0xf9/0x3b0
[ 7218.536473]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.536481]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.536488]  [<ffffffff815f4958>] start_secondary+0x230/0x237
[ 7218.536493] Code: 0f 1f 44 00 00 31 c0 c9 c3 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 48 8b 05 50 e2 ea 00 8b 80 f0 00 00 00 <8d> 3c 38 48 8b 05 40 e2 ea 00 89 b8 08 01 00 00 48 8b 05 33 e2 
[ 7218.536543] Call Trace:
[ 7218.536548]  [<ffffffff810a9034>] clockevents_program_event+0x74/0x120
[ 7218.536555]  [<ffffffff810a9dd8>] tick_broadcast_oneshot_control+0x128/0x130
[ 7218.536562]  [<ffffffff810a94f5>] tick_notify+0x325/0x410
[ 7218.536567]  [<ffffffff8101e542>] ? __switch_to_xtra+0x172/0x1a0
[ 7218.536574]  [<ffffffff816005c5>] notifier_call_chain+0x55/0x80
[ 7218.536581]  [<ffffffff8109f036>] raw_notifier_call_chain+0x16/0x20
[ 7218.536587]  [<ffffffff810a89ad>] clockevents_notify+0x3d/0x140
[ 7218.536593]  [<ffffffff81326006>] intel_idle+0x136/0x150
[ 7218.536599]  [<ffffffff814b0919>] ? menu_select+0xf9/0x3b0
[ 7218.536606]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.536612]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.536617]  [<ffffffff815f4958>] start_secondary+0x230/0x237
[ 7218.536625] NMI backtrace for cpu 2
[ 7218.536629] CPU 2 
[ 7218.536631] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle 8021q garp sunrpc bridge stp llc binfmt_misc vhost_net macvtap macvlan tun kvm_intel kvm uinput joydev microcode serio_raw pcspkr i7core_edac edac_core i2c_i801 iTCO_wdt iTCO_vendor_support r8169 mii snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video [last unloaded: scsi_wait_scan]
[ 7218.536694] 
[ 7218.536698] Pid: 18, comm: ksoftirqd/2 Not tainted 3.2.18-1.el6.alu_preempt_rt.x86_64-rt29 #1 Hewlett-Packard HP Elite 7100 Microtower PC/2A9Ch
[ 7218.536706] RIP: 0010:[<ffffffff812cb7a6>]  [<ffffffff812cb7a6>] cpumask_next_and+0x36/0x40
[ 7218.536717] RSP: 0018:ffff88012281ba70  EFLAGS: 00000012
[ 7218.536722] RAX: 0000000000000100 RBX: ffff88011df53198 RCX: 0000000000000008
[ 7218.536726] RDX: 0000000000000008 RSI: 0000000000000100 RDI: 0000000000000000
[ 7218.536730] RBP: ffff88012281ba80 R08: ffff88011df531b8 R09: 0000000000000100
[ 7218.536735] R10: 0000000000000000 R11: 0000000000000002 R12: ffff88012fc8c260
[ 7218.536739] R13: 0000000000000000 R14: 0000000000011e00 R15: 0000000000000000
[ 7218.536744] FS:  0000000000000000(0000) GS:ffff88012fc80000(0000) knlGS:0000000000000000
[ 7218.536749] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7218.536753] CR2: 000000000114d098 CR3: 0000000001c08000 CR4: 00000000000026e0
[ 7218.536758] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7218.536762] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7218.536767] Process ksoftirqd/2 (pid: 18, threadinfo ffff88012281a000, task ffff8801228184c0)
[ 7218.536771] Stack:
[ 7218.536773]  0000000000011e00 ffff88012fc8c260 ffff88012281bc50 ffffffff8105d2f9
[ 7218.536781]  ffff88012281bbe0 ffff88012281bb30 ffff88012281bad0 ffff880045dca2d0
[ 7218.536788]  ffff88012281bd40 ffff88012281bdec 0000000200dca950 ffff88012fc8c260
[ 7218.536795] Call Trace:
[ 7218.536804]  [<ffffffff8105d2f9>] find_busiest_group+0x239/0xb10
[ 7218.536817]  [<ffffffff8106e2c2>] load_balance+0xf2/0x990
[ 7218.536824]  [<ffffffff81068921>] ? __dequeue_rt_entity+0x201/0x2c0
[ 7218.536831]  [<ffffffff81069c69>] ? update_shares+0x109/0x120
[ 7218.536837]  [<ffffffff81068e49>] ? dequeue_task_rt+0x39/0x50
[ 7218.536845]  [<ffffffff815fa7bd>] __schedule+0x7cd/0x880
[ 7218.536853]  [<ffffffff815facde>] schedule+0x2e/0xa0
[ 7218.536859]  [<ffffffff815fafb8>] schedule_preempt_disabled+0x18/0x30
[ 7218.536867]  [<ffffffff8107bee5>] run_ksoftirqd+0x1b5/0x1e0
[ 7218.536874]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.536880]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.536889]  [<ffffffff81098626>] kthread+0x96/0xa0
[ 7218.536895]  [<ffffffff8105f199>] ? finish_task_switch+0x49/0xf0
[ 7218.536904]  [<ffffffff816062b4>] kernel_thread_helper+0x4/0x10
[ 7218.536911]  [<ffffffff81098590>] ? kthreadd+0x1f0/0x1f0
[ 7218.536918]  [<ffffffff816062b0>] ? gs_change+0x13/0x13
[ 7218.536921] Code: 53 48 89 f3 90 83 c0 01 be 00 01 00 00 48 89 df 48 63 d0 e8 6d 16 01 00 3b 05 ef c6 a0 00 7d 0b 41 0f a3 04 24 19 d2 85 d2 74 da <5b> 41 5c c9 c3 0f 1f 44 00 00 55 b8 ff ff ff ff 48 89 e5 41 54 
[ 7218.536971] Call Trace:
[ 7218.536977]  [<ffffffff8105d2f9>] find_busiest_group+0x239/0xb10
[ 7218.536988]  [<ffffffff8106e2c2>] load_balance+0xf2/0x990
[ 7218.536995]  [<ffffffff81068921>] ? __dequeue_rt_entity+0x201/0x2c0
[ 7218.537001]  [<ffffffff81069c69>] ? update_shares+0x109/0x120
[ 7218.537007]  [<ffffffff81068e49>] ? dequeue_task_rt+0x39/0x50
[ 7218.537014]  [<ffffffff815fa7bd>] __schedule+0x7cd/0x880
[ 7218.537021]  [<ffffffff815facde>] schedule+0x2e/0xa0
[ 7218.537027]  [<ffffffff815fafb8>] schedule_preempt_disabled+0x18/0x30
[ 7218.537033]  [<ffffffff8107bee5>] run_ksoftirqd+0x1b5/0x1e0
[ 7218.537040]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537046]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537053]  [<ffffffff81098626>] kthread+0x96/0xa0
[ 7218.537059]  [<ffffffff8105f199>] ? finish_task_switch+0x49/0xf0
[ 7218.537066]  [<ffffffff816062b4>] kernel_thread_helper+0x4/0x10
[ 7218.537073]  [<ffffffff81098590>] ? kthreadd+0x1f0/0x1f0
[ 7218.537080]  [<ffffffff816062b0>] ? gs_change+0x13/0x13
[ 7218.537084] NMI backtrace for cpu 6
[ 7218.537087] CPU 6 
[ 7218.537089] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle 8021q garp sunrpc bridge stp llc binfmt_misc vhost_net macvtap macvlan tun kvm_intel kvm uinput joydev microcode serio_raw pcspkr i7core_edac edac_core i2c_i801 iTCO_wdt iTCO_vendor_support r8169 mii snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video [last unloaded: scsi_wait_scan]
[ 7218.537141] 
[ 7218.537144] Pid: 38, comm: ksoftirqd/6 Not tainted 3.2.18-1.el6.alu_preempt_rt.x86_64-rt29 #1 Hewlett-Packard HP Elite 7100 Microtower PC/2A9Ch
[ 7218.537150] RIP: 0010:[<ffffffff812dce68>]  [<ffffffff812dce68>] find_next_bit+0x68/0xb0
[ 7218.537158] RSP: 0018:ffff8801228fda60  EFLAGS: 00000006
[ 7218.537161] RAX: 0000000000000000 RBX: ffff88011df53198 RCX: 0000000000000008
[ 7218.537164] RDX: 0000000000000008 RSI: 0000000000000100 RDI: 0000000000000040
[ 7218.537167] RBP: ffff8801228fda60 R08: ffff88011df531b8 R09: 0000000000000100
[ 7218.537170] R10: 0000000000000000 R11: 0000000000000002 R12: ffff88012fd8c260
[ 7218.537173] R13: 0000000000000000 R14: 0000000000011e00 R15: 0000000000000000
[ 7218.537177] FS:  0000000000000000(0000) GS:ffff88012fd80000(0000) knlGS:0000000000000000
[ 7218.537181] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7218.537184] CR2: 0000003642817230 CR3: 0000000001c08000 CR4: 00000000000026e0
[ 7218.537187] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7218.537191] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7218.537194] Process ksoftirqd/6 (pid: 38, threadinfo ffff8801228fc000, task ffff8801228f6340)
[ 7218.537197] Stack:
[ 7218.537199]  ffff8801228fda80 ffffffff812cb793 0000000000011e00 ffff88012fd8c260
[ 7218.537204]  ffff8801228fdc50 ffffffff8105d2f9 ffff8801228fdbe0 ffff8801228fdb30
[ 7218.537209]  ffff8801228fdad0 ffff880045c6a550 ffff8801228fdd40 ffff8801228fddec
[ 7218.537214] Call Trace:
[ 7218.537218]  [<ffffffff812cb793>] cpumask_next_and+0x23/0x40
[ 7218.537224]  [<ffffffff8105d2f9>] find_busiest_group+0x239/0xb10
[ 7218.537232]  [<ffffffff8106e2c2>] load_balance+0xf2/0x990
[ 7218.537237]  [<ffffffff81068921>] ? __dequeue_rt_entity+0x201/0x2c0
[ 7218.537243]  [<ffffffff81069c69>] ? update_shares+0x109/0x120
[ 7218.537247]  [<ffffffff81068e49>] ? dequeue_task_rt+0x39/0x50
[ 7218.537252]  [<ffffffff815fa7bd>] __schedule+0x7cd/0x880
[ 7218.537258]  [<ffffffff815facde>] schedule+0x2e/0xa0
[ 7218.537262]  [<ffffffff815fafb8>] schedule_preempt_disabled+0x18/0x30
[ 7218.537267]  [<ffffffff8107bee5>] run_ksoftirqd+0x1b5/0x1e0
[ 7218.537272]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537277]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537282]  [<ffffffff81098626>] kthread+0x96/0xa0
[ 7218.537287]  [<ffffffff8105f199>] ? finish_task_switch+0x49/0xf0
[ 7218.537293]  [<ffffffff816062b4>] kernel_thread_helper+0x4/0x10
[ 7218.537298]  [<ffffffff81098590>] ? kthreadd+0x1f0/0x1f0
[ 7218.537303]  [<ffffffff816062b0>] ? gs_change+0x13/0x13
[ 7218.537306] Code: 83 ff 3f 76 3e 48 85 c0 75 4f 49 83 c0 08 48 83 ef 40 49 83 c1 40 eb 17 0f 1f 00 49 8b 00 48 85 c0 75 36 49 83 c0 08 49 83 c1 40 <48> 83 ef 40 48 f7 c7 c0 ff ff ff 75 e3 48 85 ff 4c 89 c8 74 20 
[ 7218.537339] Call Trace:
[ 7218.537343]  [<ffffffff812cb793>] cpumask_next_and+0x23/0x40
[ 7218.537348]  [<ffffffff8105d2f9>] find_busiest_group+0x239/0xb10
[ 7218.537356]  [<ffffffff8106e2c2>] load_balance+0xf2/0x990
[ 7218.537361]  [<ffffffff81068921>] ? __dequeue_rt_entity+0x201/0x2c0
[ 7218.537366]  [<ffffffff81069c69>] ? update_shares+0x109/0x120
[ 7218.537371]  [<ffffffff81068e49>] ? dequeue_task_rt+0x39/0x50
[ 7218.537376]  [<ffffffff815fa7bd>] __schedule+0x7cd/0x880
[ 7218.537381]  [<ffffffff815facde>] schedule+0x2e/0xa0
[ 7218.537386]  [<ffffffff815fafb8>] schedule_preempt_disabled+0x18/0x30
[ 7218.537391]  [<ffffffff8107bee5>] run_ksoftirqd+0x1b5/0x1e0
[ 7218.537395]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537400]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537405]  [<ffffffff81098626>] kthread+0x96/0xa0
[ 7218.537410]  [<ffffffff8105f199>] ? finish_task_switch+0x49/0xf0
[ 7218.537415]  [<ffffffff816062b4>] kernel_thread_helper+0x4/0x10
[ 7218.537421]  [<ffffffff81098590>] ? kthreadd+0x1f0/0x1f0
[ 7218.537426]  [<ffffffff816062b0>] ? gs_change+0x13/0x13
[ 7218.537430] NMI backtrace for cpu 4
[ 7218.537434] CPU 4 
[ 7218.537437] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle 8021q garp sunrpc bridge stp llc binfmt_misc vhost_net macvtap macvlan tun kvm_intel kvm uinput joydev microcode serio_raw pcspkr i7core_edac edac_core i2c_i801 iTCO_wdt iTCO_vendor_support r8169 mii snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video [last unloaded: scsi_wait_scan]
[ 7218.537498] 
[ 7218.537502] Pid: 28, comm: ksoftirqd/4 Not tainted 3.2.18-1.el6.alu_preempt_rt.x86_64-rt29 #1 Hewlett-Packard HP Elite 7100 Microtower PC/2A9Ch
[ 7218.537510] RIP: 0010:[<ffffffff8106e9b8>]  [<ffffffff8106e9b8>] load_balance+0x7e8/0x990
[ 7218.537521] RSP: 0018:ffff88012289dc60  EFLAGS: 00000012
[ 7218.537525] RAX: 0000000000000200 RBX: 0000000000000000 RCX: 0000000000000002
[ 7218.537529] RDX: 0000000000000200 RSI: ffff88012289dd40 RDI: ffff880120684000
[ 7218.537534] RBP: ffff88012289dd80 R08: ffff88012289ddec R09: 0000000000000001
[ 7218.537538] R10: 0000000000000000 R11: 0000000000000002 R12: ffff88012289a430
[ 7218.537542] R13: 0000000000000004 R14: ffff880120684000 R15: 000000010069c3a7
[ 7218.537548] FS:  0000000000000000(0000) GS:ffff88012fd00000(0000) knlGS:0000000000000000
[ 7218.537552] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7218.537557] CR2: 00007fb277790000 CR3: 0000000001c08000 CR4: 00000000000026e0
[ 7218.537561] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7218.537566] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7218.537571] Process ksoftirqd/4 (pid: 28, threadinfo ffff88012289c000, task ffff88012289a040)
[ 7218.537574] Stack:
[ 7218.537576]  ffff88011cab2400 0000000300000001 0000000000011e00 ffff880122878038
[ 7218.537584]  ffff88012289ddec 0000000000011e00 0000000000000002 0000000000000000
[ 7218.537591]  00001266601b9ae6 ffff88012fd0c260 ffff88012289dce0 ffff88012289a040
[ 7218.537598] Call Trace:
[ 7218.537606]  [<ffffffff81068921>] ? __dequeue_rt_entity+0x201/0x2c0
[ 7218.537613]  [<ffffffff81069c69>] ? update_shares+0x109/0x120
[ 7218.537619]  [<ffffffff81068e49>] ? dequeue_task_rt+0x39/0x50
[ 7218.537627]  [<ffffffff815fa7bd>] __schedule+0x7cd/0x880
[ 7218.537633]  [<ffffffff8107ba93>] ? __do_softirq_common+0x153/0x2a0
[ 7218.537640]  [<ffffffff815facde>] schedule+0x2e/0xa0
[ 7218.537647]  [<ffffffff815fafb8>] schedule_preempt_disabled+0x18/0x30
[ 7218.537653]  [<ffffffff8107bee5>] run_ksoftirqd+0x1b5/0x1e0
[ 7218.537660]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537666]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537673]  [<ffffffff81098626>] kthread+0x96/0xa0
[ 7218.537680]  [<ffffffff8105f199>] ? finish_task_switch+0x49/0xf0
[ 7218.537687]  [<ffffffff816062b4>] kernel_thread_helper+0x4/0x10
[ 7218.537695]  [<ffffffff81098590>] ? kthreadd+0x1f0/0x1f0
[ 7218.537701]  [<ffffffff816062b0>] ? gs_change+0x13/0x13
[ 7218.537704] Code: ff ff ff 4d 89 d7 e9 45 fd ff ff 48 89 45 98 e9 f3 fd ff ff 48 8b 7d 90 c7 47 64 00 00 00 00 4c 8b 45 90 49 8b 40 18 41 89 40 60 <48> 81 c4 f8 00 00 00 89 d8 5b 41 5c 41 5d 41 5e 41 5f c9 c3 4c 
[ 7218.537754] Call Trace:
[ 7218.537761]  [<ffffffff81068921>] ? __dequeue_rt_entity+0x201/0x2c0
[ 7218.537768]  [<ffffffff81069c69>] ? update_shares+0x109/0x120
[ 7218.537774]  [<ffffffff81068e49>] ? dequeue_task_rt+0x39/0x50
[ 7218.537780]  [<ffffffff815fa7bd>] __schedule+0x7cd/0x880
[ 7218.537786]  [<ffffffff8107ba93>] ? __do_softirq_common+0x153/0x2a0
[ 7218.537794]  [<ffffffff815facde>] schedule+0x2e/0xa0
[ 7218.537800]  [<ffffffff815fafb8>] schedule_preempt_disabled+0x18/0x30
[ 7218.537806]  [<ffffffff8107bee5>] run_ksoftirqd+0x1b5/0x1e0
[ 7218.537812]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537819]  [<ffffffff8107bd30>] ? __thread_do_softirq+0x150/0x150
[ 7218.537825]  [<ffffffff81098626>] kthread+0x96/0xa0
[ 7218.537831]  [<ffffffff8105f199>] ? finish_task_switch+0x49/0xf0
[ 7218.537838]  [<ffffffff816062b4>] kernel_thread_helper+0x4/0x10
[ 7218.537846]  [<ffffffff81098590>] ? kthreadd+0x1f0/0x1f0
[ 7218.537852]  [<ffffffff816062b0>] ? gs_change+0x13/0x13
[ 7218.537855] NMI backtrace for cpu 5
[ 7218.537858] CPU 5 
[ 7218.537860] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle 8021q garp sunrpc bridge stp llc binfmt_misc vhost_net macvtap macvlan tun kvm_intel kvm uinput joydev microcode serio_raw pcspkr i7core_edac edac_core i2c_i801 iTCO_wdt iTCO_vendor_support r8169 mii snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video [last unloaded: scsi_wait_scan]
[ 7218.537921] 
[ 7218.537926] Pid: 0, comm: swapper/5 Not tainted 3.2.18-1.el6.alu_preempt_rt.x86_64-rt29 #1 Hewlett-Packard HP Elite 7100 Microtower PC/2A9Ch
[ 7218.537934] RIP: 0010:[<ffffffff812d7a32>]  [<ffffffff812d7a32>] __const_udelay+0x12/0x30
[ 7218.537943] RSP: 0018:ffff88012fd43d98  EFLAGS: 00000002
[ 7218.537947] RAX: 0000000000000000 RBX: 0000000000000001 RCX: 0000000000000094
[ 7218.537952] RDX: 00000000002a9c44 RSI: 0000000000000100 RDI: 0000000000418958
[ 7218.537956] RBP: ffff88012fd43d98 R08: 0000000000000004 R09: 000000000000000a
[ 7218.537960] R10: 0000000000000006 R11: 0000000000000000 R12: ffffffff81c37500
[ 7218.537965] R13: ffffffff81c37600 R14: 0000000000000001 R15: ffffffff81c38600
[ 7218.537970] FS:  0000000000000000(0000) GS:ffff88012fd40000(0000) knlGS:0000000000000000
[ 7218.537975] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7218.537979] CR2: 0000003642817230 CR3: 0000000001c08000 CR4: 00000000000026e0
[ 7218.537984] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7218.537988] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7218.537993] Process swapper/5 (pid: 0, threadinfo ffff8801228e0000, task ffff8801228de200)
[ 7218.537996] Stack:
[ 7218.537999]  ffff88012fd43db8 ffffffff81035a2d 000000000000fffb ffff88012fd4cf40
[ 7218.538007]  ffff88012fd43e28 ffffffff810f06df ffff880100000000 0000000000000000
[ 7218.538013]  00000000000f41a8 0000000000000003 000000000000ea62 0000000000000087
[ 7218.538021] Call Trace:
[ 7218.538023]  <IRQ> 
[ 7218.538031]  [<ffffffff81035a2d>] arch_trigger_all_cpu_backtrace+0x6d/0xa0
[ 7218.538040]  [<ffffffff810f06df>] __rcu_pending+0x40f/0x510
[ 7218.538047]  [<ffffffff810f0a8a>] rcu_check_callbacks+0x2aa/0x2c0
[ 7218.538055]  [<ffffffff81083ebd>] update_process_times+0x4d/0x70
[ 7218.538063]  [<ffffffff810aab73>] tick_sched_timer+0x63/0xc0
[ 7218.538069]  [<ffffffff8109d686>] __run_hrtimer+0x76/0x270
[ 7218.538075]  [<ffffffff810aab10>] ? tick_nohz_handler+0x110/0x110
[ 7218.538082]  [<ffffffff8109db60>] hrtimer_interrupt+0xf0/0x230
[ 7218.538089]  [<ffffffff81606d16>] smp_apic_timer_interrupt+0x66/0x98
[ 7218.538096]  [<ffffffff81604c1e>] apic_timer_interrupt+0x6e/0x80
[ 7218.538100]  <EOI> 
[ 7218.538107]  [<ffffffff81325fb0>] ? intel_idle+0xe0/0x150
[ 7218.538113]  [<ffffffff81325f96>] ? intel_idle+0xc6/0x150
[ 7218.538121]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.538128]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.538134]  [<ffffffff815f4958>] start_secondary+0x230/0x237
[ 7218.538139] Code: 66 90 ff 15 d1 c4 99 00 c9 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 65 48 8b 14 25 58 1d 01 00 <48> 8d 04 bd 00 00 00 00 48 69 d2 fa 00 00 00 f7 e2 48 8d 7a 01 
[ 7218.538188] Call Trace:
[ 7218.538191]  <IRQ>  [<ffffffff81035a2d>] arch_trigger_all_cpu_backtrace+0x6d/0xa0
[ 7218.538202]  [<ffffffff810f06df>] __rcu_pending+0x40f/0x510
[ 7218.538210]  [<ffffffff810f0a8a>] rcu_check_callbacks+0x2aa/0x2c0
[ 7218.538216]  [<ffffffff81083ebd>] update_process_times+0x4d/0x70
[ 7218.538222]  [<ffffffff810aab73>] tick_sched_timer+0x63/0xc0
[ 7218.538229]  [<ffffffff8109d686>] __run_hrtimer+0x76/0x270
[ 7218.538235]  [<ffffffff810aab10>] ? tick_nohz_handler+0x110/0x110
[ 7218.538241]  [<ffffffff8109db60>] hrtimer_interrupt+0xf0/0x230
[ 7218.538248]  [<ffffffff81606d16>] smp_apic_timer_interrupt+0x66/0x98
[ 7218.538254]  [<ffffffff81604c1e>] apic_timer_interrupt+0x6e/0x80
[ 7218.538258]  <EOI>  [<ffffffff81325fb0>] ? intel_idle+0xe0/0x150
[ 7218.538268]  [<ffffffff81325f96>] ? intel_idle+0xc6/0x150
[ 7218.538275]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.538281]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.538287]  [<ffffffff815f4958>] start_secondary+0x230/0x237
[ 7218.538293] NMI backtrace for cpu 1
[ 7218.538297] CPU 1 
[ 7218.538299] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle 8021q garp sunrpc bridge stp llc binfmt_misc vhost_net macvtap macvlan tun kvm_intel kvm uinput joydev microcode serio_raw pcspkr i7core_edac edac_core i2c_i801 iTCO_wdt iTCO_vendor_support r8169 mii snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video [last unloaded: scsi_wait_scan]
[ 7218.538365] 
[ 7218.538369] Pid: 0, comm: swapper/1 Not tainted 3.2.18-1.el6.alu_preempt_rt.x86_64-rt29 #1 Hewlett-Packard HP Elite 7100 Microtower PC/2A9Ch
[ 7218.538378] RIP: 0010:[<ffffffff81325f86>]  [<ffffffff81325f86>] intel_idle+0xb6/0x150
[ 7218.538387] RSP: 0018:ffff880122ffde58  EFLAGS: 00000046
[ 7218.538392] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 7218.538397] RDX: 0000000000000000 RSI: ffff880122ffdfd8 RDI: ffffffff81c16100
[ 7218.538401] RBP: ffff880122ffdeb8 R08: 0000000000000000 R09: 0000000000000000
[ 7218.538406] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[ 7218.538411] R13: ffff88012fc58a70 R14: 0000000000000020 R15: 129b8774d980e5ec
[ 7218.538417] FS:  0000000000000000(0000) GS:ffff88012fc40000(0000) knlGS:0000000000000000
[ 7218.538423] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7218.538427] CR2: 00000000011e5990 CR3: 0000000001c08000 CR4: 00000000000026e0
[ 7218.538432] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7218.538437] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7218.538443] Process swapper/1 (pid: 0, threadinfo ffff880122ffc000, task ffff880122ffa380)
[ 7218.538447] Stack:
[ 7218.538449]  ffff880122ffdeb8 ffffffff814b0919 0000000000000000 00000000000cb768
[ 7218.538457]  0000000000000000 00000001000cb768 00000000ffffffff ffff88012fc58a70
[ 7218.538465]  ffffffff81c76ca0 0000000000000000 0000000000000003 0000000000000001
[ 7218.538473] Call Trace:
[ 7218.538480]  [<ffffffff814b0919>] ? menu_select+0xf9/0x3b0
[ 7218.538488]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.538495]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.538502]  [<ffffffff815f4958>] start_secondary+0x230/0x237
[ 7218.538507] Code: c7 48 8d 86 38 e0 ff ff 83 e2 08 75 1e 31 d2 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 05 cf d7 ff 4c 29 f8 48 89 c7 e8 0a 46 d5 ff 49 89 c6 49 89 
[ 7218.538564] Call Trace:
[ 7218.538570]  [<ffffffff814b0919>] ? menu_select+0xf9/0x3b0
[ 7218.538578]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.538585]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.538591]  [<ffffffff815f4958>] start_secondary+0x230/0x237
[ 7218.538598] NMI backtrace for cpu 0
[ 7218.538601] CPU 0 
[ 7218.538602] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle 8021q garp sunrpc bridge stp llc binfmt_misc vhost_net macvtap macvlan tun kvm_intel kvm uinput joydev microcode serio_raw pcspkr i7core_edac edac_core i2c_i801 iTCO_wdt iTCO_vendor_support r8169 mii snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video [last unloaded: scsi_wait_scan]
[ 7218.538644] 
[ 7218.538648] Pid: 0, comm: swapper/0 Not tainted 3.2.18-1.el6.alu_preempt_rt.x86_64-rt29 #1 Hewlett-Packard HP Elite 7100 Microtower PC/2A9Ch
[ 7218.538653] RIP: 0010:[<ffffffff81325f86>]  [<ffffffff81325f86>] intel_idle+0xb6/0x150
[ 7218.538660] RSP: 0018:ffffffff81c01e48  EFLAGS: 00000046
[ 7218.538663] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 7218.538667] RDX: 0000000000000000 RSI: ffffffff81c01fd8 RDI: ffffffff81c16100
[ 7218.538670] RBP: ffffffff81c01ea8 R08: 0000000000000000 R09: 0000000000000000
[ 7218.538673] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[ 7218.538676] R13: ffff88012fc18a70 R14: 0000000000000020 R15: 129b8774d980e757
[ 7218.538680] FS:  0000000000000000(0000) GS:ffff88012fc00000(0000) knlGS:0000000000000000
[ 7218.538683] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7218.538686] CR2: 000000000125a040 CR3: 0000000001c08000 CR4: 00000000000026e0
[ 7218.538690] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7218.538693] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7218.538697] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c10020)
[ 7218.538699] Stack:
[ 7218.538701]  ffffffff81c01ea8 ffffffff814b0919 0000000000000000 00000000000cb63c
[ 7218.538706]  0000000000000000 00000000000cb63c 00000000ffffffff ffff88012fc18a70
[ 7218.538711]  ffffffff81c76ca0 0000000000000000 0000000000000003 0000000000000000
[ 7218.538716] Call Trace:
[ 7218.538721]  [<ffffffff814b0919>] ? menu_select+0xf9/0x3b0
[ 7218.538727]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.538732]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.538737]  [<ffffffff815e2565>] rest_init+0x85/0x90
[ 7218.538743]  [<ffffffff81cf3d94>] start_kernel+0x3d3/0x3de
[ 7218.538748]  [<ffffffff81cf3346>] x86_64_start_reservations+0x131/0x135
[ 7218.538753]  [<ffffffff81cf344d>] x86_64_start_kernel+0x103/0x112
[ 7218.538756] Code: c7 48 8d 86 38 e0 ff ff 83 e2 08 75 1e 31 d2 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 05 cf d7 ff 4c 29 f8 48 89 c7 e8 0a 46 d5 ff 49 89 c6 49 89 
[ 7218.538790] Call Trace:
[ 7218.538794]  [<ffffffff814b0919>] ? menu_select+0xf9/0x3b0
[ 7218.538799]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.538804]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.538808]  [<ffffffff815e2565>] rest_init+0x85/0x90
[ 7218.538812]  [<ffffffff81cf3d94>] start_kernel+0x3d3/0x3de
[ 7218.538817]  [<ffffffff81cf3346>] x86_64_start_reservations+0x131/0x135
[ 7218.538822]  [<ffffffff81cf344d>] x86_64_start_kernel+0x103/0x112
[ 7218.538826] NMI backtrace for cpu 3
[ 7218.538828] CPU 3 
[ 7218.538830] Modules linked in: fuse ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_CHECKSUM iptable_mangle 8021q garp sunrpc bridge stp llc binfmt_misc vhost_net macvtap macvlan tun kvm_intel kvm uinput joydev microcode serio_raw pcspkr i7core_edac edac_core i2c_i801 iTCO_wdt iTCO_vendor_support r8169 mii snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc firewire_ohci firewire_core crc_itu_t nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video [last unloaded: scsi_wait_scan]
[ 7218.538872] 
[ 7218.538875] Pid: 0, comm: swapper/3 Not tainted 3.2.18-1.el6.alu_preempt_rt.x86_64-rt29 #1 Hewlett-Packard HP Elite 7100 Microtower PC/2A9Ch
[ 7218.538881] RIP: 0010:[<ffffffff81325f86>]  [<ffffffff81325f86>] intel_idle+0xb6/0x150
[ 7218.538887] RSP: 0018:ffff88012285fe58  EFLAGS: 00000046
[ 7218.538890] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
[ 7218.538893] RDX: 0000000000000000 RSI: ffff88012285ffd8 RDI: ffffffff81c16100
[ 7218.538896] RBP: ffff88012285feb8 R08: 0000000000000000 R09: 0000000000000000
[ 7218.538899] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000003
[ 7218.538902] R13: ffff88012fcd8a70 R14: 0000000000000020 R15: 129b8774d9809ca4
[ 7218.538906] FS:  0000000000000000(0000) GS:ffff88012fcc0000(0000) knlGS:0000000000000000
[ 7218.538910] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 7218.538913] CR2: 00000000011f2c60 CR3: 0000000001c08000 CR4: 00000000000026e0
[ 7218.538916] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 7218.538919] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 7218.538923] Process swapper/3 (pid: 0, threadinfo ffff88012285e000, task ffff88012285c680)
[ 7218.538925] Stack:
[ 7218.538927]  ffff88012285feb8 ffffffff814b0919 ffffffffffffffc3 000000003b8a4bca
[ 7218.538932]  ffffffffffffffc3 000000033b8a4bca 00000000ffffffff ffff88012fcd8a70
[ 7218.538937]  ffffffff81c76ca0 0000000000000000 0000000000000003 0000000000000003
[ 7218.538942] Call Trace:
[ 7218.538947]  [<ffffffff814b0919>] ? menu_select+0xf9/0x3b0
[ 7218.538953]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.538958]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.538962]  [<ffffffff815f4958>] start_secondary+0x230/0x237
[ 7218.538965] Code: c7 48 8d 86 38 e0 ff ff 83 e2 08 75 1e 31 d2 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> 05 cf d7 ff 4c 29 f8 48 89 c7 e8 0a 46 d5 ff 49 89 c6 49 89 
[ 7218.538999] Call Trace:
[ 7218.539003]  [<ffffffff814b0919>] ? menu_select+0xf9/0x3b0
[ 7218.539009]  [<ffffffff814af6b1>] cpuidle_idle_call+0xd1/0x1d0
[ 7218.539013]  [<ffffffff81014a5a>] cpu_idle+0xba/0x110
[ 7218.539017]  [<ffffffff815f4958>] start_secondary+0x230/0x237