* [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps
@ 2015-06-15 7:02 ` Huang Ying
0 siblings, 0 replies; 12+ messages in thread
From: Huang Ying @ 2015-06-15 7:02 UTC (permalink / raw)
To: lkp
[-- Attachment #1: Type: text/plain, Size: 150781 bytes --]
FYI, we noticed the below changes on
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts")
testcase/path_params/tbox_group: netperf/performance-300s-200%-TCP_RR/brickland3
3cb96f0c97330834 52f518a3a7c2f80551a38d38be
---------------- --------------------------
%stddev %change %stddev
\ | \
1444 ± 1% -30.5% 1004 ± 1% netperf.Throughput_tps
899.43 ± 3% -91.6% 75.48 ± 3% netperf.time.user_time
10202 ± 0% -92.4% 778 ± 1% netperf.time.system_time
3565 ± 4% -92.5% 268 ± 3% netperf.time.percent_of_cpu_this_job_got
98265 ± 3% -31.1% 67662 ± 4% netperf.time.minor_page_faults
25540 ± 11% -88.3% 2991 ± 16% netperf.time.involuntary_context_switches
69135723 ± 1% -30.0% 48364754 ± 1% netperf.time.voluntary_context_switches
37347 ± 4% +13.1% 42256 ± 3% uptime.idle
44 ± 5% -93.2% 3 ± 0% vmstat.procs.r
407528 ± 3% -26.3% 300511 ± 4% vmstat.system.cs
446620 ± 4% -92.6% 33083 ± 6% vmstat.system.in
178635 ± 1% -14.6% 152519 ± 0% meminfo.Active
82997 ± 2% -29.5% 58518 ± 1% meminfo.Active(anon)
36056 ± 3% -63.2% 13281 ± 0% meminfo.Shmem
496435 ± 3% +19.4% 592640 ± 2% numa-meminfo.node0.MemUsed
480933 ± 3% -7.3% 445927 ± 2% numa-meminfo.node2.MemUsed
3997 ± 1% +12.1% 4482 ± 3% slabinfo.kmalloc-2048.active_objs
4183 ± 1% +12.2% 4694 ± 3% slabinfo.kmalloc-2048.num_objs
313 ± 15% -78.0% 69 ± 30% slabinfo.tw_sock_TCP.active_objs
313 ± 15% -78.0% 69 ± 30% slabinfo.tw_sock_TCP.num_objs
1016370 ± 1% -90.2% 99283 ± 1% softirqs.HRTIMER
1556902 ± 0% -84.0% 248959 ± 2% softirqs.RCU
3471142 ± 1% -92.3% 267496 ± 3% softirqs.SCHED
4270170 ± 1% -82.2% 758706 ± 2% softirqs.TIMER
2.347e+09 ± 6% -63.5% 8.569e+08 ± 39% cpuidle.C1-IVT-4S.time
22319046 ± 3% +62.6% 36299214 ± 3% cpuidle.C1-IVT-4S.usage
1.591e+10 ± 9% +128.3% 3.632e+10 ± 3% cpuidle.C6-IVT-4S.time
7 ± 21% +514.0% 44 ± 31% cpuidle.POLL.usage
25540 ± 11% -88.3% 2991 ± 16% time.involuntary_context_switches
98265 ± 3% -31.1% 67662 ± 4% time.minor_page_faults
3565 ± 4% -92.5% 268 ± 3% time.percent_of_cpu_this_job_got
10202 ± 0% -92.4% 778 ± 1% time.system_time
899.43 ± 3% -91.6% 75.48 ± 3% time.user_time
69135723 ± 1% -30.0% 48364754 ± 1% time.voluntary_context_switches
268764 ± 14% +408.8% 1367464 ± 3% numa-numastat.node0.numa_hit
268758 ± 14% +408.8% 1367424 ± 3% numa-numastat.node0.local_node
438574 ± 26% -45.0% 241074 ± 2% numa-numastat.node1.local_node
438574 ± 26% -45.0% 241076 ± 2% numa-numastat.node1.numa_hit
333989 ± 23% -27.9% 240841 ± 3% numa-numastat.node2.local_node
333989 ± 23% -27.9% 240844 ± 3% numa-numastat.node2.numa_hit
381233 ± 26% -35.6% 245556 ± 3% numa-numastat.node3.local_node
381238 ± 26% -35.6% 245561 ± 3% numa-numastat.node3.numa_hit
482 ± 4% +11.9% 540 ± 1% numa-vmstat.node0.nr_alloc_batch
251510 ± 9% +212.0% 784815 ± 8% numa-vmstat.node0.numa_local
289771 ± 8% +184.0% 823025 ± 7% numa-vmstat.node0.numa_hit
454081 ± 12% -25.4% 338719 ± 4% numa-vmstat.node1.numa_hit
521 ± 1% +13.5% 592 ± 3% numa-vmstat.node1.nr_alloc_batch
395201 ± 14% -28.9% 281018 ± 4% numa-vmstat.node1.numa_local
529 ± 3% +10.1% 583 ± 1% numa-vmstat.node2.nr_alloc_batch
525 ± 6% +12.8% 593 ± 1% numa-vmstat.node3.nr_alloc_batch
51.25 ± 3% -92.9% 3.65 ± 3% turbostat.%Busy
1635 ± 3% -93.0% 114 ± 3% turbostat.Avg_MHz
45.20 ± 4% -70.1% 13.52 ± 21% turbostat.CPU%c1
357.80 ± 3% -71.5% 101.83 ± 6% turbostat.CorWatt
53 ± 2% -21.2% 42 ± 6% turbostat.CoreTmp
53 ± 1% -20.7% 42 ± 6% turbostat.PkgTmp
424 ± 2% -61.0% 165 ± 3% turbostat.PkgWatt
62.78 ± 1% -8.9% 57.17 ± 3% turbostat.RAMWatt
20746 ± 2% -29.5% 14628 ± 1% proc-vmstat.nr_active_anon
2024 ± 1% +13.4% 2295 ± 0% proc-vmstat.nr_alloc_batch
9013 ± 3% -63.2% 3319 ± 0% proc-vmstat.nr_shmem
49576 ± 5% -60.3% 19667 ± 13% proc-vmstat.numa_hint_faults
44013 ± 5% -64.6% 15598 ± 16% proc-vmstat.numa_hint_faults_local
1420966 ± 2% +47.3% 2092933 ± 1% proc-vmstat.numa_hit
1420955 ± 2% +47.3% 2092883 ± 1% proc-vmstat.numa_local
53703 ± 5% -57.9% 22601 ± 13% proc-vmstat.numa_pte_updates
8470 ± 1% -81.8% 1543 ± 3% proc-vmstat.pgactivate
162499 ± 5% +90.3% 309245 ± 5% proc-vmstat.pgalloc_dma32
5918865 ± 2% +85.7% 10992818 ± 3% proc-vmstat.pgalloc_normal
6085763 ± 2% +86.0% 11317294 ± 3% proc-vmstat.pgfree
2579 ± 5% -79.4% 532 ± 2% latency_stats.avg.do_wait.SyS_wait4.system_call_fastpath
858 ± 5% +60.1% 1375 ± 4% latency_stats.avg.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
2337 ± 4% -79.5% 479 ± 2% latency_stats.avg.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath
875 ± 7% +74.6% 1528 ± 10% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
455 ± 1% +108.8% 951 ± 1% latency_stats.avg.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
541 ± 14% +137.6% 1287 ± 4% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_revalidate_dentry_attr.[cifs].cifs_revalidate_dentry.[cifs].cifs_d_revalidate.[cifs].lookup_fast.do_last.path_openat.do_filp_open.do_sys_open
655 ± 20% +96.9% 1291 ± 9% latency_stats.avg.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
69119790 ± 1% -30.0% 48351473 ± 1% latency_stats.hits.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
443 ± 3% -10.0% 399 ± 2% latency_stats.hits.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
190 ± 6% -95.8% 8 ± 38% latency_stats.hits.stop_one_cpu.sched_exec.do_execveat_common.SyS_execve.return_from_execve
2157 ± 23% +66.0% 3581 ± 22% latency_stats.max.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
4912 ± 0% +1.6% 4992 ± 0% latency_stats.max.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
3.151e+10 ± 0% +46.1% 4.603e+10 ± 0% latency_stats.sum.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath
16780629 ± 2% -78.9% 3539898 ± 4% latency_stats.sum.do_wait.SyS_wait4.system_call_fastpath
25402986 ± 2% -78.5% 5473439 ± 4% latency_stats.sum.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath
12072 ± 31% +112.0% 25588 ± 10% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_revalidate_dentry_attr.[cifs].cifs_revalidate_dentry.[cifs].cifs_d_revalidate.[cifs].lookup_fast.do_last.path_openat.do_filp_open.do_sys_open
290547 ± 19% +77.5% 515581 ± 9% latency_stats.sum.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath
24407 ± 9% +74.1% 42503 ± 9% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat
245753 ± 23% +44.5% 355113 ± 9% latency_stats.sum.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath
17.85 ± 4% -16.4% 14.92 ± 2% perf-profile.cpu-cycles.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive
0.97 ± 4% -85.4% 0.14 ± 32% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle
2.19 ± 8% -93.6% 0.14 ± 10% perf-profile.cpu-cycles._raw_spin_lock.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
3.88 ± 9% -94.1% 0.23 ± 21% perf-profile.cpu-cycles.load_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout
6.29 ± 7% -96.7% 0.21 ± 40% perf-profile.cpu-cycles._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task
0.94 ± 5% -90.7% 0.09 ± 13% perf-profile.cpu-cycles.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__schedule.schedule
0.94 ± 5% -90.6% 0.09 ± 13% perf-profile.cpu-cycles.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__schedule
4.19 ± 2% +61.7% 6.77 ± 3% perf-profile.cpu-cycles.ktime_get.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
15.10 ± 3% -27.5% 10.94 ± 1% perf-profile.cpu-cycles.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
15.19 ± 3% -27.2% 11.07 ± 1% perf-profile.cpu-cycles.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
15.35 ± 3% -27.0% 11.20 ± 1% perf-profile.cpu-cycles.SYSC_recvfrom.sys_recvfrom.system_call_fastpath
15.44 ± 3% -27.0% 11.27 ± 1% perf-profile.cpu-cycles.sys_recvfrom.system_call_fastpath
14.72 ± 3% -26.5% 10.82 ± 1% perf-profile.cpu-cycles.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom
1.57 ± 3% +33.4% 2.10 ± 4% perf-profile.cpu-cycles.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
15.03 ± 4% -25.0% 11.27 ± 2% perf-profile.cpu-cycles.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
23.41 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ret_from_intr.cpu_startup_entry.start_secondary
1.19 ± 12% -80.5% 0.23 ± 19% perf-profile.cpu-cycles.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__run_hrtimer
4.24 ± 2% +58.3% 6.72 ± 3% perf-profile.cpu-cycles.read_hpet.ktime_get.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry
15.05 ± 4% -24.9% 11.30 ± 2% perf-profile.cpu-cycles.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue
2.81 ± 9% -90.6% 0.26 ± 20% perf-profile.cpu-cycles.find_busiest_group.load_balance.pick_next_task_fair.__schedule.schedule
9.12 ± 4% -52.1% 4.36 ± 4% perf-profile.cpu-cycles.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
0.46 ± 2% +127.4% 1.04 ± 6% perf-profile.cpu-cycles.pick_next_task_fair.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry
2.52 ± 2% -62.9% 0.94 ± 31% perf-profile.cpu-cycles.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt
1.58 ± 8% -81.7% 0.29 ± 21% perf-profile.cpu-cycles.tick_sched_handle.isra.18.tick_sched_timer.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt
2.49 ± 1% -64.0% 0.90 ± 33% perf-profile.cpu-cycles.tick_sched_timer.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt
15.06 ± 4% -24.9% 11.32 ± 2% perf-profile.cpu-cycles.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv
1.35 ± 3% +32.3% 1.79 ± 8% perf-profile.cpu-cycles.deactivate_task.__schedule.schedule.schedule_timeout.sk_wait_data
23.23 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
5.90 ± 2% -23.8% 4.50 ± 6% perf-profile.cpu-cycles.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle
1.72 ± 8% -83.2% 0.29 ± 21% perf-profile.cpu-cycles.update_process_times.tick_sched_handle.tick_sched_timer.__run_hrtimer.hrtimer_interrupt
1.55 ± 6% +48.4% 2.31 ± 5% perf-profile.cpu-cycles.dev_queue_xmit_sk.ip_finish_output.ip_output.ip_local_out_sk.ip_queue_xmit
1.35 ± 3% +30.7% 1.76 ± 7% perf-profile.cpu-cycles.dequeue_task.deactivate_task.__schedule.schedule.schedule_timeout
15.75 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
1.54 ± 5% +48.3% 2.28 ± 5% perf-profile.cpu-cycles.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output.ip_output.ip_local_out_sk
3.70 ± 1% +49.0% 5.51 ± 1% perf-profile.cpu-cycles.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg
72.13 ± 0% -24.2% 54.66 ± 0% perf-profile.cpu-cycles.cpu_startup_entry.start_secondary
8.28 ± 5% -61.0% 3.23 ± 5% perf-profile.cpu-cycles.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg
3.44 ± 3% +51.2% 5.19 ± 1% perf-profile.cpu-cycles.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg
2.75 ± 9% -91.1% 0.24 ± 20% perf-profile.cpu-cycles.update_sd_lb_stats.find_busiest_group.load_balance.pick_next_task_fair.__schedule
8.36 ± 5% -61.0% 3.26 ± 5% perf-profile.cpu-cycles.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg
0.33 ± 29% +218.5% 1.06 ± 10% perf-profile.cpu-cycles.sk_stream_alloc_skb.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
15.13 ± 4% -24.2% 11.46 ± 2% perf-profile.cpu-cycles.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish
72.31 ± 0% -24.2% 54.83 ± 0% perf-profile.cpu-cycles.start_secondary
14.20 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__do_softirq.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry
1.30 ± 3% +24.4% 1.62 ± 8% perf-profile.cpu-cycles.dequeue_task_fair.dequeue_task.deactivate_task.__schedule.schedule
1.34 ± 5% +30.0% 1.74 ± 4% perf-profile.cpu-cycles.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output.ip_output
8.93 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ret_from_intr.cpuidle_enter.cpu_startup_entry.start_secondary
15.18 ± 1% +26.1% 19.14 ± 3% perf-profile.cpu-cycles.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
4.09 ± 0% +9.8% 4.49 ± 4% perf-profile.cpu-cycles.ktime_get.clockevents_program_event.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns
8.08 ± 5% -62.1% 3.06 ± 6% perf-profile.cpu-cycles.__schedule.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg
3.87 ± 2% +12.2% 4.35 ± 7% perf-profile.cpu-cycles.read_hpet.ktime_get.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry
5.80 ± 1% -12.5% 5.07 ± 5% perf-profile.cpu-cycles.ktime_get_with_offset.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll
0.36 ± 4% +188.3% 1.02 ± 5% perf-profile.cpu-cycles.dev_hard_start_xmit.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output
4.72 ± 3% +186.8% 13.52 ± 6% perf-profile.cpu-cycles.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
10.92 ± 4% -11.1% 9.71 ± 4% perf-profile.cpu-cycles.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate
2.95 ± 3% -18.6% 2.40 ± 3% perf-profile.cpu-cycles.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary
3.83 ± 2% +14.0% 4.36 ± 7% perf-profile.cpu-cycles.ktime_get.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
8.92 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry.start_secondary
7.38 ± 1% +13.6% 8.38 ± 2% perf-profile.cpu-cycles.sock_sendmsg.SYSC_sendto.sys_sendto.system_call_fastpath
11.19 ± 4% -11.1% 9.95 ± 2% perf-profile.cpu-cycles.ttwu_do_activate.constprop.88.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common
15.76 ± 4% -20.9% 12.46 ± 3% perf-profile.cpu-cycles.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver
7.73 ± 2% +14.4% 8.85 ± 2% perf-profile.cpu-cycles.SYSC_sendto.sys_sendto.system_call_fastpath
13.61 ± 1% +12.6% 15.31 ± 3% perf-profile.cpu-cycles.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
2.97 ± 2% +14.2% 3.40 ± 4% perf-profile.cpu-cycles.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
7.79 ± 2% +14.7% 8.93 ± 2% perf-profile.cpu-cycles.sys_sendto.system_call_fastpath
4.63 ± 1% +24.6% 5.77 ± 1% perf-profile.cpu-cycles.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom
2.87 ± 2% +13.8% 3.26 ± 3% perf-profile.cpu-cycles.ip_output.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit
6.59 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry
3.04 ± 3% -17.9% 2.49 ± 3% perf-profile.cpu-cycles.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary
11.17 ± 4% -18.6% 9.09 ± 2% perf-profile.cpu-cycles.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate.try_to_wake_up
23.39 ± 2% -12.3% 20.51 ± 1% perf-profile.cpu-cycles.system_call_fastpath
7.03 ± 1% +16.0% 8.15 ± 2% perf-profile.cpu-cycles.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto.system_call_fastpath
4.16 ± 0% +14.6% 4.76 ± 4% perf-profile.cpu-cycles.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick
4.15 ± 0% +14.7% 4.76 ± 4% perf-profile.cpu-cycles.clockevents_program_event.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start
3.05 ± 3% -16.3% 2.55 ± 3% perf-profile.cpu-cycles.schedule_preempt_disabled.cpu_startup_entry.start_secondary
6.33 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__do_softirq.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter
6.78 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr.cpu_startup_entry
6.75 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_enter.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary
4.73 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter.irq_enter
10.95 ± 5% -13.5% 9.47 ± 2% perf-profile.cpu-cycles.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function
4.62 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter
4.21 ± 1% +21.2% 5.10 ± 4% perf-profile.cpu-cycles.ktime_get.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
1.29 ± 4% +21.1% 1.57 ± 3% perf-profile.cpu-cycles.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
9.15 ± 1% +16.1% 10.63 ± 2% perf-profile.cpu-cycles.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
10.94 ± 5% -13.5% 9.46 ± 2% perf-profile.cpu-cycles.enqueue_task.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function
0.17 ± 35% +479.8% 0.96 ± 10% perf-profile.cpu-cycles.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg.inet_sendmsg.sock_sendmsg
4.32 ± 9% -88.6% 0.49 ± 13% perf-profile.cpu-cycles.pick_next_task_fair.__schedule.schedule.schedule_timeout.sk_wait_data
4.54 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_nohz_stop_idle.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr
16.82 ± 4% -15.0% 14.30 ± 3% perf-profile.cpu-cycles.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb
16.81 ± 4% -15.0% 14.28 ± 3% perf-profile.cpu-cycles.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core
4.28 ± 2% -14.4% 3.66 ± 6% perf-profile.cpu-cycles.ktime_get.clockevents_program_event.tick_program_event.__remove_hrtimer.hrtimer_try_to_cancel
24.19 ± 3% -15.8% 20.36 ± 3% perf-profile.cpu-cycles.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action.__do_softirq
23.93 ± 3% -15.8% 20.15 ± 3% perf-profile.cpu-cycles.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action
4.03 ± 1% +15.2% 4.64 ± 3% perf-profile.cpu-cycles.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg
9.82 ± 4% -14.5% 8.40 ± 3% perf-profile.cpu-cycles.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task
13.12 ± 1% +15.9% 15.21 ± 3% perf-profile.cpu-cycles.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary
6.71 ± 1% +18.5% 7.95 ± 2% perf-profile.cpu-cycles.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto
8.57 ± 1% +17.2% 10.05 ± 3% perf-profile.cpu-cycles.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry
16.71 ± 4% -16.1% 14.02 ± 3% perf-profile.cpu-cycles.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv
0.14 ± 15% +573.8% 0.94 ± 9% perf-profile.cpu-cycles.tcp_send_delayed_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process
0.10 ± 23% +830.0% 0.93 ± 5% perf-profile.cpu-cycles.ixgbe_xmit_frame.dev_hard_start_xmit.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk
0.08 ± 6% +1213.0% 1.01 ± 4% perf-profile.cpu-cycles.is_module_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency
4.48 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter.irq_enter.do_IRQ
4.57 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr
8.57 ± 1% +16.9% 10.02 ± 3% perf-profile.cpu-cycles.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter
17.72 ± 4% -16.3% 14.84 ± 2% perf-profile.cpu-cycles.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal
4.57 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.read_hpet.ktime_get.tick_irq_enter.irq_enter.do_IRQ
0.96 ± 4% -85.4% 0.14 ± 35% perf-profile.cpu-cycles.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times
18.03 ± 4% -16.6% 15.03 ± 2% perf-profile.cpu-cycles.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll
4.99 ± 1% +17.0% 5.84 ± 2% perf-profile.cpu-cycles.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto
18.00 ± 4% -16.7% 15.00 ± 2% perf-profile.cpu-cycles.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq
3.16 ± 1% +20.8% 3.82 ± 3% perf-profile.cpu-cycles.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push
4.87 ± 2% +18.7% 5.78 ± 2% perf-profile.cpu-cycles.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg
2.36 ± 1% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_enter.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry
0.23 ± 6% +443.4% 1.23 ± 3% perf-profile.cpu-cycles.is_module_text_address.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk
0.47 ± 4% +684.2% 3.65 ± 3% perf-profile.cpu-cycles.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency
1.37 ± 4% +239.8% 4.65 ± 7% perf-profile.cpu-cycles.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
1.34 ± 4% +243.2% 4.61 ± 7% perf-profile.cpu-cycles.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry
0.43 ± 7% +1444.2% 6.64 ± 15% perf-profile.cpu-cycles.intel_idle.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary
0.54 ± 7% +3863.4% 21.27 ± 3% perf-profile.cpu-cycles.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action.__do_softirq.run_ksoftirqd
0.54 ± 7% +4049.4% 22.41 ± 2% perf-profile.cpu-cycles.ixgbe_poll.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn
0.55 ± 6% +3975.2% 22.48 ± 2% perf-profile.cpu-cycles.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread
1.28 ± 3% +246.7% 4.43 ± 7% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit
4.21 ± 0% +18.0% 4.97 ± 3% perf-profile.cpu-cycles.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter
5.58 ± 1% -15.9% 4.69 ± 3% perf-profile.cpu-cycles.read_hpet.ktime_get_with_offset.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq
4.77 ± 1% +19.6% 5.70 ± 2% perf-profile.cpu-cycles.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg
1.33 ± 4% +248.1% 4.62 ± 7% perf-profile.cpu-cycles.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
1.92 ± 2% +283.1% 7.36 ± 2% perf-profile.cpu-cycles.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair
1.93 ± 2% +285.6% 7.46 ± 2% perf-profile.cpu-cycles.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task
2.20 ± 3% +19.0% 2.61 ± 4% perf-profile.cpu-cycles.ip_finish_output.ip_output.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb
16.33 ± 4% -17.9% 13.40 ± 3% perf-profile.cpu-cycles.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish
0.99 ± 4% +2179.8% 22.61 ± 2% perf-profile.cpu-cycles.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
0.97 ± 4% +2217.9% 22.60 ± 2% perf-profile.cpu-cycles.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.13 ± 3% +1906.2% 22.67 ± 2% perf-profile.cpu-cycles.smpboot_thread_fn.kthread.ret_from_fork
1.15 ± 2% +1878.6% 22.82 ± 2% perf-profile.cpu-cycles.kthread.ret_from_fork
1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.x86_64_start_reservations.x86_64_start_kernel
1.15 ± 2% +1879.0% 22.82 ± 2% perf-profile.cpu-cycles.ret_from_fork
2.34 ± 1% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr.cpuidle_enter
1.62 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.do_IRQ.ret_from_intr
1.61 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.read_hpet.ktime_get.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit
1.56 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_nohz_irq_exit.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry
1.48 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.do_IRQ
0.21 ± 12% +351.6% 0.96 ± 8% perf-profile.cpu-cycles.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg
1.72 ± 2% +301.4% 6.90 ± 2% perf-profile.cpu-cycles.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity
1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.x86_64_start_kernel
1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.start_kernel.x86_64_start_reservations.x86_64_start_kernel
4.21 ± 1% +20.6% 5.08 ± 4% perf-profile.cpu-cycles.read_hpet.ktime_get.tick_nohz_idle_exit.cpu_startup_entry.start_secondary
528 ± 6% +61.7% 854 ± 1% sched_debug.cfs_rq[0]:/.tg_runnable_contrib
24162 ± 6% +62.3% 39214 ± 1% sched_debug.cfs_rq[0]:/.avg->runnable_avg_sum
4882658 ± 9% -96.6% 167048 ± 8% sched_debug.cfs_rq[0]:/.min_vruntime
37303 ± 1% -89.7% 3826 ± 4% sched_debug.cfs_rq[0]:/.tg->runnable_avg
77353 ± 9% +81.9% 140694 ± 7% sched_debug.cfs_rq[0]:/.exec_clock
38349 ± 2% -89.4% 4070 ± 3% sched_debug.cfs_rq[100]:/.tg->runnable_avg
38365 ± 2% -89.4% 4071 ± 3% sched_debug.cfs_rq[101]:/.tg->runnable_avg
38369 ± 2% -89.4% 4074 ± 3% sched_debug.cfs_rq[102]:/.tg->runnable_avg
38382 ± 2% -89.4% 4075 ± 3% sched_debug.cfs_rq[103]:/.tg->runnable_avg
38383 ± 2% -89.4% 4077 ± 3% sched_debug.cfs_rq[104]:/.tg->runnable_avg
38391 ± 2% -89.4% 4079 ± 3% sched_debug.cfs_rq[105]:/.tg->runnable_avg
38388 ± 2% -89.4% 4083 ± 3% sched_debug.cfs_rq[106]:/.tg->runnable_avg
38402 ± 2% -89.4% 4084 ± 3% sched_debug.cfs_rq[107]:/.tg->runnable_avg
38401 ± 2% -89.4% 4087 ± 3% sched_debug.cfs_rq[108]:/.tg->runnable_avg
38413 ± 2% -89.4% 4088 ± 3% sched_debug.cfs_rq[109]:/.tg->runnable_avg
548 ± 3% -93.4% 36 ± 22% sched_debug.cfs_rq[10]:/.tg_runnable_contrib
37450 ± 1% -89.7% 3851 ± 4% sched_debug.cfs_rq[10]:/.tg->runnable_avg
5772463 ± 8% -98.4% 93465 ± 22% sched_debug.cfs_rq[10]:/.min_vruntime
76594 ± 8% -91.7% 6366 ± 15% sched_debug.cfs_rq[10]:/.exec_clock
25036 ± 3% -93.3% 1688 ± 20% sched_debug.cfs_rq[10]:/.avg->runnable_avg_sum
38410 ± 2% -89.4% 4089 ± 3% sched_debug.cfs_rq[110]:/.tg->runnable_avg
38417 ± 2% -89.4% 4088 ± 3% sched_debug.cfs_rq[111]:/.tg->runnable_avg
38435 ± 2% -89.4% 4091 ± 3% sched_debug.cfs_rq[112]:/.tg->runnable_avg
38435 ± 2% -89.4% 4092 ± 3% sched_debug.cfs_rq[113]:/.tg->runnable_avg
38428 ± 2% -89.3% 4095 ± 3% sched_debug.cfs_rq[114]:/.tg->runnable_avg
38439 ± 1% -89.3% 4096 ± 3% sched_debug.cfs_rq[115]:/.tg->runnable_avg
38445 ± 2% -89.3% 4095 ± 3% sched_debug.cfs_rq[116]:/.tg->runnable_avg
38449 ± 2% -89.3% 4097 ± 3% sched_debug.cfs_rq[117]:/.tg->runnable_avg
38461 ± 2% -89.3% 4100 ± 3% sched_debug.cfs_rq[118]:/.tg->runnable_avg
38460 ± 1% -89.3% 4102 ± 3% sched_debug.cfs_rq[119]:/.tg->runnable_avg
76608 ± 7% -91.7% 6356 ± 16% sched_debug.cfs_rq[11]:/.exec_clock
546 ± 2% -93.5% 35 ± 25% sched_debug.cfs_rq[11]:/.tg_runnable_contrib
37465 ± 1% -89.7% 3852 ± 4% sched_debug.cfs_rq[11]:/.tg->runnable_avg
24985 ± 2% -93.3% 1664 ± 24% sched_debug.cfs_rq[11]:/.avg->runnable_avg_sum
5775629 ± 7% -98.4% 91917 ± 12% sched_debug.cfs_rq[11]:/.min_vruntime
76166 ± 8% -92.2% 5923 ± 22% sched_debug.cfs_rq[12]:/.exec_clock
5756491 ± 8% -98.5% 86026 ± 20% sched_debug.cfs_rq[12]:/.min_vruntime
5 ± 31% -100.0% 0 ± 0% sched_debug.cfs_rq[12]:/.runnable_load_avg
37492 ± 1% -89.7% 3853 ± 4% sched_debug.cfs_rq[12]:/.tg->runnable_avg
75892 ± 8% -91.8% 6211 ± 19% sched_debug.cfs_rq[13]:/.exec_clock
37506 ± 1% -89.7% 3856 ± 4% sched_debug.cfs_rq[13]:/.tg->runnable_avg
5727275 ± 7% -98.4% 93734 ± 18% sched_debug.cfs_rq[13]:/.min_vruntime
37528 ± 1% -89.7% 3857 ± 4% sched_debug.cfs_rq[14]:/.tg->runnable_avg
25080 ± 5% -94.4% 1394 ± 43% sched_debug.cfs_rq[14]:/.avg->runnable_avg_sum
75791 ± 8% -92.2% 5907 ± 24% sched_debug.cfs_rq[14]:/.exec_clock
550 ± 4% -94.6% 29 ± 43% sched_debug.cfs_rq[14]:/.tg_runnable_contrib
6 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[14]:/.runnable_load_avg
5717919 ± 7% -98.4% 90360 ± 22% sched_debug.cfs_rq[14]:/.min_vruntime
37554 ± 1% -89.7% 3860 ± 4% sched_debug.cfs_rq[15]:/.tg->runnable_avg
71299 ± 11% -96.4% 2537 ± 18% sched_debug.cfs_rq[15]:/.exec_clock
5471648 ± 11% -99.1% 50259 ± 22% sched_debug.cfs_rq[15]:/.min_vruntime
71273 ± 12% -96.7% 2353 ± 10% sched_debug.cfs_rq[16]:/.exec_clock
5469742 ± 11% -99.0% 53306 ± 24% sched_debug.cfs_rq[16]:/.min_vruntime
37563 ± 1% -89.7% 3864 ± 4% sched_debug.cfs_rq[16]:/.tg->runnable_avg
2807093.70 ± 36% -100.0% 0.00 ± 0% sched_debug.cfs_rq[17]:/.MIN_vruntime
6 ± 31% -100.0% 0 ± 0% sched_debug.cfs_rq[17]:/.runnable_load_avg
37568 ± 1% -89.7% 3865 ± 4% sched_debug.cfs_rq[17]:/.tg->runnable_avg
71654 ± 12% -96.9% 2229 ± 14% sched_debug.cfs_rq[17]:/.exec_clock
5474236 ± 11% -99.1% 50364 ± 25% sched_debug.cfs_rq[17]:/.min_vruntime
2807093.70 ± 36% -100.0% 0.00 ± 0% sched_debug.cfs_rq[17]:/.max_vruntime
37574 ± 1% -89.7% 3866 ± 4% sched_debug.cfs_rq[18]:/.tg->runnable_avg
2611368.18 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[18]:/.max_vruntime
2611367.85 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[18]:/.MIN_vruntime
5469964 ± 11% -99.2% 42263 ± 36% sched_debug.cfs_rq[18]:/.min_vruntime
4 ± 28% -100.0% 0 ± 0% sched_debug.cfs_rq[19]:/.runnable_load_avg
5460827 ± 11% -99.2% 46250 ± 33% sched_debug.cfs_rq[19]:/.min_vruntime
37603 ± 1% -89.7% 3871 ± 4% sched_debug.cfs_rq[19]:/.tg->runnable_avg
5931126 ± 8% -86.0% 829332 ± 28% sched_debug.cfs_rq[1]:/.min_vruntime
37333 ± 1% -89.7% 3834 ± 4% sched_debug.cfs_rq[1]:/.tg->runnable_avg
1048433 ± 16% -36.8% 662283 ± 34% sched_debug.cfs_rq[1]:/.spread0
5501605 ± 11% -99.1% 46820 ± 29% sched_debug.cfs_rq[20]:/.min_vruntime
71374 ± 11% -97.4% 1825 ± 22% sched_debug.cfs_rq[20]:/.exec_clock
37630 ± 1% -89.7% 3876 ± 4% sched_debug.cfs_rq[20]:/.tg->runnable_avg
4 ± 49% -100.0% 0 ± 0% sched_debug.cfs_rq[20]:/.runnable_load_avg
5435674 ± 11% -99.1% 48697 ± 31% sched_debug.cfs_rq[21]:/.min_vruntime
37645 ± 1% -89.7% 3875 ± 4% sched_debug.cfs_rq[21]:/.tg->runnable_avg
7 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[21]:/.load
122 ± 48% -100.0% 0 ± 0% sched_debug.cfs_rq[21]:/.utilization_load_avg
70205 ± 12% -97.3% 1867 ± 36% sched_debug.cfs_rq[21]:/.exec_clock
7 ± 44% -100.0% 0 ± 0% sched_debug.cfs_rq[22]:/.load
37661 ± 1% -89.7% 3876 ± 4% sched_debug.cfs_rq[22]:/.tg->runnable_avg
5491828 ± 11% -99.2% 45143 ± 26% sched_debug.cfs_rq[22]:/.min_vruntime
6 ± 32% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.load
4 ± 35% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.runnable_load_avg
90 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.utilization_load_avg
37652 ± 1% -89.7% 3879 ± 4% sched_debug.cfs_rq[23]:/.tg->runnable_avg
5499468 ± 11% -99.2% 44829 ± 30% sched_debug.cfs_rq[23]:/.min_vruntime
37651 ± 1% -89.7% 3882 ± 4% sched_debug.cfs_rq[24]:/.tg->runnable_avg
7 ± 47% -100.0% 0 ± 0% sched_debug.cfs_rq[24]:/.load
2002338.86 ± 45% -100.0% 0.00 ± 0% sched_debug.cfs_rq[24]:/.MIN_vruntime
5458480 ± 11% -99.2% 45851 ± 22% sched_debug.cfs_rq[24]:/.min_vruntime
2002338.86 ± 45% -100.0% 0.00 ± 0% sched_debug.cfs_rq[24]:/.max_vruntime
499 ± 11% -98.4% 7 ± 49% sched_debug.cfs_rq[24]:/.tg_runnable_contrib
22791 ± 11% -98.3% 393 ± 47% sched_debug.cfs_rq[24]:/.avg->runnable_avg_sum
5501548 ± 11% -99.2% 42017 ± 39% sched_debug.cfs_rq[25]:/.min_vruntime
37675 ± 1% -89.7% 3883 ± 4% sched_debug.cfs_rq[25]:/.tg->runnable_avg
37701 ± 1% -89.7% 3887 ± 4% sched_debug.cfs_rq[26]:/.tg->runnable_avg
5460570 ± 11% -99.2% 42120 ± 20% sched_debug.cfs_rq[26]:/.min_vruntime
72021 ± 8% -97.9% 1500 ± 39% sched_debug.cfs_rq[26]:/.exec_clock
5443461 ± 11% -99.3% 39281 ± 30% sched_debug.cfs_rq[27]:/.min_vruntime
37720 ± 1% -89.7% 3890 ± 4% sched_debug.cfs_rq[27]:/.tg->runnable_avg
70407 ± 11% -97.6% 1657 ± 41% sched_debug.cfs_rq[27]:/.exec_clock
37737 ± 1% -89.7% 3890 ± 4% sched_debug.cfs_rq[28]:/.tg->runnable_avg
71813 ± 12% -98.0% 1424 ± 44% sched_debug.cfs_rq[28]:/.exec_clock
5482457 ± 11% -99.2% 42096 ± 24% sched_debug.cfs_rq[28]:/.min_vruntime
5478035 ± 11% -99.2% 41368 ± 31% sched_debug.cfs_rq[29]:/.min_vruntime
37748 ± 1% -89.7% 3893 ± 4% sched_debug.cfs_rq[29]:/.tg->runnable_avg
5924938 ± 8% -86.5% 799517 ± 29% sched_debug.cfs_rq[2]:/.min_vruntime
37344 ± 1% -89.7% 3839 ± 4% sched_debug.cfs_rq[2]:/.tg->runnable_avg
1042225 ± 20% -39.3% 632468 ± 35% sched_debug.cfs_rq[2]:/.spread0
37765 ± 1% -89.7% 3895 ± 4% sched_debug.cfs_rq[30]:/.tg->runnable_avg
2514014.52 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[30]:/.max_vruntime
5791091 ± 11% -99.1% 49961 ± 26% sched_debug.cfs_rq[30]:/.min_vruntime
2514014.52 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[30]:/.MIN_vruntime
76335 ± 12% -97.0% 2285 ± 16% sched_debug.cfs_rq[30]:/.exec_clock
76769 ± 11% -97.4% 2026 ± 16% sched_debug.cfs_rq[31]:/.exec_clock
2370182.85 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[31]:/.MIN_vruntime
5806067 ± 11% -99.2% 46410 ± 22% sched_debug.cfs_rq[31]:/.min_vruntime
2370183.19 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[31]:/.max_vruntime
37783 ± 1% -89.7% 3903 ± 4% sched_debug.cfs_rq[31]:/.tg->runnable_avg
77107 ± 12% -97.4% 1980 ± 30% sched_debug.cfs_rq[32]:/.exec_clock
5865705 ± 12% -99.2% 45792 ± 32% sched_debug.cfs_rq[32]:/.min_vruntime
37808 ± 1% -89.7% 3909 ± 4% sched_debug.cfs_rq[32]:/.tg->runnable_avg
24950 ± 9% -96.2% 943 ± 23% sched_debug.cfs_rq[33]:/.avg->runnable_avg_sum
37805 ± 1% -89.6% 3914 ± 4% sched_debug.cfs_rq[33]:/.tg->runnable_avg
5827248 ± 12% -99.2% 44413 ± 27% sched_debug.cfs_rq[33]:/.min_vruntime
76627 ± 13% -97.6% 1812 ± 27% sched_debug.cfs_rq[33]:/.exec_clock
546 ± 8% -96.4% 19 ± 22% sched_debug.cfs_rq[33]:/.tg_runnable_contrib
37804 ± 1% -89.6% 3915 ± 4% sched_debug.cfs_rq[34]:/.tg->runnable_avg
76396 ± 12% -97.7% 1738 ± 29% sched_debug.cfs_rq[34]:/.exec_clock
2052042.52 ± 30% -100.0% 0.00 ± 0% sched_debug.cfs_rq[34]:/.MIN_vruntime
5789564 ± 11% -99.2% 43627 ± 23% sched_debug.cfs_rq[34]:/.min_vruntime
2052042.52 ± 30% -100.0% 0.00 ± 0% sched_debug.cfs_rq[34]:/.max_vruntime
37810 ± 1% -89.6% 3915 ± 4% sched_debug.cfs_rq[35]:/.tg->runnable_avg
5793549 ± 11% -99.3% 40722 ± 15% sched_debug.cfs_rq[35]:/.min_vruntime
76584 ± 12% -97.8% 1682 ± 27% sched_debug.cfs_rq[35]:/.exec_clock
553 ± 8% -98.9% 6 ± 36% sched_debug.cfs_rq[36]:/.tg_runnable_contrib
37800 ± 1% -89.6% 3917 ± 4% sched_debug.cfs_rq[36]:/.tg->runnable_avg
25249 ± 8% -98.7% 330 ± 28% sched_debug.cfs_rq[36]:/.avg->runnable_avg_sum
76722 ± 13% -97.9% 1576 ± 41% sched_debug.cfs_rq[36]:/.exec_clock
5843915 ± 12% -99.3% 40252 ± 43% sched_debug.cfs_rq[36]:/.min_vruntime
75923 ± 12% -97.7% 1718 ± 38% sched_debug.cfs_rq[37]:/.exec_clock
5768738 ± 11% -99.3% 42192 ± 39% sched_debug.cfs_rq[37]:/.min_vruntime
535 ± 9% -98.8% 6 ± 28% sched_debug.cfs_rq[37]:/.tg_runnable_contrib
37811 ± 1% -89.6% 3924 ± 4% sched_debug.cfs_rq[37]:/.tg->runnable_avg
24479 ± 9% -98.7% 316 ± 26% sched_debug.cfs_rq[37]:/.avg->runnable_avg_sum
7 ± 32% -100.0% 0 ± 0% sched_debug.cfs_rq[38]:/.load
37833 ± 1% -89.6% 3925 ± 4% sched_debug.cfs_rq[38]:/.tg->runnable_avg
5 ± 30% -100.0% 0 ± 0% sched_debug.cfs_rq[38]:/.runnable_load_avg
5826004 ± 12% -99.3% 43456 ± 44% sched_debug.cfs_rq[38]:/.min_vruntime
544 ± 10% -98.4% 8 ± 27% sched_debug.cfs_rq[39]:/.tg_runnable_contrib
6 ± 21% -100.0% 0 ± 0% sched_debug.cfs_rq[39]:/.runnable_load_avg
76688 ± 13% -98.2% 1415 ± 35% sched_debug.cfs_rq[39]:/.exec_clock
37837 ± 1% -89.6% 3925 ± 4% sched_debug.cfs_rq[39]:/.tg->runnable_avg
24864 ± 9% -98.2% 454 ± 23% sched_debug.cfs_rq[39]:/.avg->runnable_avg_sum
5835422 ± 12% -99.4% 36160 ± 31% sched_debug.cfs_rq[39]:/.min_vruntime
26232 ± 5% -22.4% 20344 ± 8% sched_debug.cfs_rq[3]:/.avg->runnable_avg_sum
574 ± 6% -22.5% 445 ± 8% sched_debug.cfs_rq[3]:/.tg_runnable_contrib
37364 ± 1% -89.7% 3840 ± 4% sched_debug.cfs_rq[3]:/.tg->runnable_avg
9 ± 49% +335.2% 39 ± 44% sched_debug.cfs_rq[3]:/.load
5891140 ± 8% -87.4% 740621 ± 29% sched_debug.cfs_rq[3]:/.min_vruntime
1008386 ± 18% -43.1% 573573 ± 35% sched_debug.cfs_rq[3]:/.spread0
76125 ± 12% -98.0% 1494 ± 47% sched_debug.cfs_rq[40]:/.exec_clock
37838 ± 1% -89.6% 3926 ± 4% sched_debug.cfs_rq[40]:/.tg->runnable_avg
5784431 ± 11% -99.3% 38831 ± 36% sched_debug.cfs_rq[40]:/.min_vruntime
76346 ± 13% -98.1% 1455 ± 41% sched_debug.cfs_rq[41]:/.exec_clock
2279884.43 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[41]:/.MIN_vruntime
5802087 ± 12% -99.3% 40195 ± 34% sched_debug.cfs_rq[41]:/.min_vruntime
5 ± 29% -100.0% 0 ± 0% sched_debug.cfs_rq[41]:/.runnable_load_avg
2279884.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[41]:/.max_vruntime
37855 ± 1% -89.6% 3929 ± 4% sched_debug.cfs_rq[41]:/.tg->runnable_avg
134 ± 42% -100.0% 0 ± 0% sched_debug.cfs_rq[41]:/.utilization_load_avg
37858 ± 1% -89.6% 3932 ± 4% sched_debug.cfs_rq[42]:/.tg->runnable_avg
5839210 ± 12% -99.3% 39152 ± 47% sched_debug.cfs_rq[42]:/.min_vruntime
76766 ± 13% -98.2% 1387 ± 46% sched_debug.cfs_rq[42]:/.exec_clock
5 ± 20% -100.0% 0 ± 0% sched_debug.cfs_rq[42]:/.runnable_load_avg
76865 ± 13% -98.1% 1491 ± 35% sched_debug.cfs_rq[43]:/.exec_clock
5 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[43]:/.runnable_load_avg
7 ± 44% -100.0% 0 ± 0% sched_debug.cfs_rq[43]:/.load
3274498.62 ± 42% -100.0% 0.00 ± 0% sched_debug.cfs_rq[43]:/.max_vruntime
5845980 ± 12% -99.3% 42925 ± 28% sched_debug.cfs_rq[43]:/.min_vruntime
3274498.62 ± 42% -100.0% 0.00 ± 0% sched_debug.cfs_rq[43]:/.MIN_vruntime
37884 ± 1% -89.6% 3934 ± 4% sched_debug.cfs_rq[43]:/.tg->runnable_avg
76461 ± 13% -97.9% 1582 ± 36% sched_debug.cfs_rq[44]:/.exec_clock
5801262 ± 12% -99.3% 39028 ± 29% sched_debug.cfs_rq[44]:/.min_vruntime
37904 ± 1% -89.6% 3939 ± 4% sched_debug.cfs_rq[44]:/.tg->runnable_avg
37919 ± 1% -89.6% 3945 ± 4% sched_debug.cfs_rq[45]:/.tg->runnable_avg
75128 ± 8% -96.8% 2372 ± 25% sched_debug.cfs_rq[45]:/.exec_clock
5733040 ± 8% -99.2% 47682 ± 38% sched_debug.cfs_rq[45]:/.min_vruntime
74823 ± 8% -97.4% 1957 ± 15% sched_debug.cfs_rq[46]:/.exec_clock
2181820.38 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[46]:/.MIN_vruntime
5706201 ± 7% -99.2% 45069 ± 29% sched_debug.cfs_rq[46]:/.min_vruntime
37931 ± 1% -89.6% 3949 ± 4% sched_debug.cfs_rq[46]:/.tg->runnable_avg
2181820.38 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[46]:/.max_vruntime
37937 ± 1% -89.6% 3952 ± 4% sched_debug.cfs_rq[47]:/.tg->runnable_avg
75941 ± 7% -97.7% 1740 ± 26% sched_debug.cfs_rq[47]:/.exec_clock
5795769 ± 7% -99.3% 43188 ± 29% sched_debug.cfs_rq[47]:/.min_vruntime
74784 ± 7% -97.9% 1582 ± 27% sched_debug.cfs_rq[48]:/.exec_clock
5703468 ± 7% -99.3% 39942 ± 31% sched_debug.cfs_rq[48]:/.min_vruntime
37947 ± 1% -89.6% 3954 ± 4% sched_debug.cfs_rq[48]:/.tg->runnable_avg
5 ± 33% -100.0% 0 ± 0% sched_debug.cfs_rq[49]:/.runnable_load_avg
75045 ± 7% -97.9% 1592 ± 23% sched_debug.cfs_rq[49]:/.exec_clock
2349144.49 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[49]:/.MIN_vruntime
5717654 ± 7% -99.3% 41062 ± 37% sched_debug.cfs_rq[49]:/.min_vruntime
2349144.49 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[49]:/.max_vruntime
37976 ± 1% -89.6% 3956 ± 4% sched_debug.cfs_rq[49]:/.tg->runnable_avg
26117 ± 3% -50.7% 12870 ± 7% sched_debug.cfs_rq[4]:/.avg->runnable_avg_sum
37364 ± 1% -89.7% 3843 ± 4% sched_debug.cfs_rq[4]:/.tg->runnable_avg
78451 ± 8% -47.9% 40905 ± 20% sched_debug.cfs_rq[4]:/.exec_clock
571 ± 4% -50.5% 282 ± 7% sched_debug.cfs_rq[4]:/.tg_runnable_contrib
5828333 ± 8% -91.7% 486365 ± 28% sched_debug.cfs_rq[4]:/.min_vruntime
945560 ± 28% -66.2% 319316 ± 41% sched_debug.cfs_rq[4]:/.spread0
92 ± 24% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.utilization_load_avg
75109 ± 7% -97.7% 1699 ± 37% sched_debug.cfs_rq[50]:/.exec_clock
6 ± 49% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.load
5726100 ± 6% -99.2% 45365 ± 30% sched_debug.cfs_rq[50]:/.min_vruntime
543 ± 11% -98.9% 6 ± 49% sched_debug.cfs_rq[50]:/.tg_runnable_contrib
24764 ± 11% -98.7% 328 ± 39% sched_debug.cfs_rq[50]:/.avg->runnable_avg_sum
37997 ± 1% -89.6% 3959 ± 4% sched_debug.cfs_rq[50]:/.tg->runnable_avg
4 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.runnable_load_avg
5701479 ± 6% -99.3% 37520 ± 36% sched_debug.cfs_rq[51]:/.min_vruntime
38008 ± 1% -89.6% 3959 ± 4% sched_debug.cfs_rq[51]:/.tg->runnable_avg
74668 ± 7% -98.1% 1411 ± 41% sched_debug.cfs_rq[51]:/.exec_clock
25473 ± 12% -98.4% 408 ± 48% sched_debug.cfs_rq[52]:/.avg->runnable_avg_sum
75465 ± 8% -98.1% 1450 ± 23% sched_debug.cfs_rq[52]:/.exec_clock
3567331.44 ± 29% -100.0% 0.00 ± 0% sched_debug.cfs_rq[52]:/.MIN_vruntime
5767898 ± 7% -99.3% 38845 ± 30% sched_debug.cfs_rq[52]:/.min_vruntime
8 ± 41% -100.0% 0 ± 0% sched_debug.cfs_rq[52]:/.runnable_load_avg
3567331.77 ± 29% -100.0% 0.00 ± 0% sched_debug.cfs_rq[52]:/.max_vruntime
38019 ± 1% -89.6% 3962 ± 4% sched_debug.cfs_rq[52]:/.tg->runnable_avg
38036 ± 2% -89.6% 3966 ± 4% sched_debug.cfs_rq[53]:/.tg->runnable_avg
75493 ± 8% -98.0% 1520 ± 30% sched_debug.cfs_rq[53]:/.exec_clock
2564844.76 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[53]:/.max_vruntime
5768160 ± 7% -99.2% 44245 ± 35% sched_debug.cfs_rq[53]:/.min_vruntime
2564844.76 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[53]:/.MIN_vruntime
75523 ± 9% -98.1% 1443 ± 40% sched_debug.cfs_rq[54]:/.exec_clock
38055 ± 2% -89.6% 3969 ± 4% sched_debug.cfs_rq[54]:/.tg->runnable_avg
5757595 ± 8% -99.3% 41312 ± 28% sched_debug.cfs_rq[54]:/.min_vruntime
5731615 ± 7% -99.3% 38668 ± 27% sched_debug.cfs_rq[55]:/.min_vruntime
75160 ± 8% -98.0% 1473 ± 39% sched_debug.cfs_rq[55]:/.exec_clock
38085 ± 2% -89.6% 3971 ± 4% sched_debug.cfs_rq[55]:/.tg->runnable_avg
552 ± 13% -99.0% 5 ± 49% sched_debug.cfs_rq[56]:/.tg_runnable_contrib
5759032 ± 7% -99.3% 37467 ± 32% sched_debug.cfs_rq[56]:/.min_vruntime
38101 ± 2% -89.6% 3980 ± 4% sched_debug.cfs_rq[56]:/.tg->runnable_avg
25129 ± 13% -98.8% 291 ± 44% sched_debug.cfs_rq[56]:/.avg->runnable_avg_sum
75483 ± 8% -98.1% 1421 ± 48% sched_debug.cfs_rq[56]:/.exec_clock
5 ± 29% -100.0% 0 ± 0% sched_debug.cfs_rq[57]:/.runnable_load_avg
74873 ± 8% -97.9% 1582 ± 49% sched_debug.cfs_rq[57]:/.exec_clock
5707036 ± 7% -99.2% 43462 ± 45% sched_debug.cfs_rq[57]:/.min_vruntime
38113 ± 2% -89.6% 3981 ± 4% sched_debug.cfs_rq[57]:/.tg->runnable_avg
2056049.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[58]:/.MIN_vruntime
2056049.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[58]:/.max_vruntime
5638172 ± 8% -99.3% 40579 ± 41% sched_debug.cfs_rq[58]:/.min_vruntime
38110 ± 1% -89.6% 3979 ± 4% sched_debug.cfs_rq[58]:/.tg->runnable_avg
74655 ± 8% -98.3% 1291 ± 34% sched_debug.cfs_rq[59]:/.exec_clock
9 ± 48% -100.0% 0 ± 0% sched_debug.cfs_rq[59]:/.load
38125 ± 1% -89.6% 3980 ± 4% sched_debug.cfs_rq[59]:/.tg->runnable_avg
5687789 ± 7% -99.3% 37622 ± 41% sched_debug.cfs_rq[59]:/.min_vruntime
569 ± 5% -89.4% 60 ± 18% sched_debug.cfs_rq[5]:/.tg_runnable_contrib
78491 ± 9% -87.5% 9836 ± 6% sched_debug.cfs_rq[5]:/.exec_clock
37386 ± 1% -89.7% 3845 ± 4% sched_debug.cfs_rq[5]:/.tg->runnable_avg
26026 ± 6% -89.3% 2782 ± 17% sched_debug.cfs_rq[5]:/.avg->runnable_avg_sum
5868941 ± 8% -97.8% 132022 ± 13% sched_debug.cfs_rq[5]:/.min_vruntime
50960 ± 9% -99.1% 438 ± 44% sched_debug.cfs_rq[60]:/.exec_clock
38133 ± 1% -89.6% 3981 ± 4% sched_debug.cfs_rq[60]:/.tg->runnable_avg
51720 ± 14% -97.9% 1072 ± 40% sched_debug.cfs_rq[61]:/.exec_clock
4337125 ± 13% -99.2% 35916 ± 38% sched_debug.cfs_rq[61]:/.min_vruntime
38160 ± 1% -89.6% 3984 ± 4% sched_debug.cfs_rq[61]:/.tg->runnable_avg
3 ± 39% -100.0% 0 ± 0% sched_debug.cfs_rq[62]:/.load
52202 ± 13% -98.8% 626 ± 34% sched_debug.cfs_rq[62]:/.exec_clock
38148 ± 1% -89.5% 3986 ± 4% sched_debug.cfs_rq[62]:/.tg->runnable_avg
14412 ± 18% -98.6% 194 ± 48% sched_debug.cfs_rq[62]:/.avg->runnable_avg_sum
2 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[62]:/.runnable_load_avg
38154 ± 2% -89.6% 3986 ± 4% sched_debug.cfs_rq[63]:/.tg->runnable_avg
1268924 ± 47% -97.9% 26358 ± 34% sched_debug.cfs_rq[64]:/.min_vruntime
64 ± 42% -95.8% 2 ± 41% sched_debug.cfs_rq[64]:/.tg_runnable_contrib
38173 ± 2% -89.6% 3988 ± 4% sched_debug.cfs_rq[64]:/.tg->runnable_avg
2993 ± 41% -95.1% 148 ± 36% sched_debug.cfs_rq[64]:/.avg->runnable_avg_sum
38152 ± 2% -89.6% 3985 ± 4% sched_debug.cfs_rq[65]:/.tg->runnable_avg
7 ± 47% -100.0% 0 ± 0% sched_debug.cfs_rq[65]:/.nr_spread_over
38162 ± 2% -89.5% 3989 ± 3% sched_debug.cfs_rq[66]:/.tg->runnable_avg
10 ± 30% -100.0% 0 ± 0% sched_debug.cfs_rq[66]:/.nr_spread_over
15875 ± 49% -95.7% 680 ± 31% sched_debug.cfs_rq[66]:/.exec_clock
38175 ± 2% -89.5% 3990 ± 3% sched_debug.cfs_rq[67]:/.tg->runnable_avg
38206 ± 2% -89.6% 3991 ± 3% sched_debug.cfs_rq[68]:/.tg->runnable_avg
38221 ± 2% -89.6% 3992 ± 3% sched_debug.cfs_rq[69]:/.tg->runnable_avg
15369 ± 47% -94.7% 812 ± 43% sched_debug.cfs_rq[69]:/.exec_clock
25441 ± 4% -91.8% 2076 ± 13% sched_debug.cfs_rq[6]:/.avg->runnable_avg_sum
557 ± 4% -92.0% 44 ± 13% sched_debug.cfs_rq[6]:/.tg_runnable_contrib
6 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[6]:/.load
5815571 ± 7% -98.1% 110120 ± 10% sched_debug.cfs_rq[6]:/.min_vruntime
77709 ± 8% -89.8% 7891 ± 9% sched_debug.cfs_rq[6]:/.exec_clock
5 ± 33% -100.0% 0 ± 0% sched_debug.cfs_rq[6]:/.runnable_load_avg
37407 ± 1% -89.7% 3849 ± 4% sched_debug.cfs_rq[6]:/.tg->runnable_avg
38225 ± 2% -89.5% 4001 ± 4% sched_debug.cfs_rq[70]:/.tg->runnable_avg
38235 ± 2% -89.5% 4005 ± 4% sched_debug.cfs_rq[71]:/.tg->runnable_avg
38238 ± 2% -89.5% 4007 ± 4% sched_debug.cfs_rq[72]:/.tg->runnable_avg
38253 ± 2% -89.5% 4007 ± 4% sched_debug.cfs_rq[73]:/.tg->runnable_avg
9 ± 27% -100.0% 0 ± 0% sched_debug.cfs_rq[73]:/.nr_spread_over
7 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[74]:/.nr_spread_over
38240 ± 2% -89.5% 4012 ± 4% sched_debug.cfs_rq[74]:/.tg->runnable_avg
38235 ± 2% -89.5% 4010 ± 4% sched_debug.cfs_rq[75]:/.tg->runnable_avg
38233 ± 2% -89.5% 4010 ± 4% sched_debug.cfs_rq[76]:/.tg->runnable_avg
38239 ± 2% -89.5% 4012 ± 4% sched_debug.cfs_rq[77]:/.tg->runnable_avg
38260 ± 2% -89.5% 4015 ± 4% sched_debug.cfs_rq[78]:/.tg->runnable_avg
38284 ± 2% -89.5% 4019 ± 3% sched_debug.cfs_rq[79]:/.tg->runnable_avg
25894 ± 3% -92.4% 1956 ± 12% sched_debug.cfs_rq[7]:/.avg->runnable_avg_sum
78030 ± 8% -90.8% 7193 ± 8% sched_debug.cfs_rq[7]:/.exec_clock
5863768 ± 7% -98.2% 105821 ± 12% sched_debug.cfs_rq[7]:/.min_vruntime
566 ± 3% -92.6% 42 ± 12% sched_debug.cfs_rq[7]:/.tg_runnable_contrib
37430 ± 1% -89.7% 3850 ± 4% sched_debug.cfs_rq[7]:/.tg->runnable_avg
38292 ± 2% -89.5% 4021 ± 3% sched_debug.cfs_rq[80]:/.tg->runnable_avg
38300 ± 2% -89.5% 4022 ± 3% sched_debug.cfs_rq[81]:/.tg->runnable_avg
38303 ± 2% -89.5% 4025 ± 3% sched_debug.cfs_rq[82]:/.tg->runnable_avg
38286 ± 2% -89.5% 4028 ± 3% sched_debug.cfs_rq[83]:/.tg->runnable_avg
38294 ± 2% -89.5% 4032 ± 3% sched_debug.cfs_rq[84]:/.tg->runnable_avg
38304 ± 2% -89.5% 4034 ± 3% sched_debug.cfs_rq[85]:/.tg->runnable_avg
38321 ± 2% -89.5% 4035 ± 3% sched_debug.cfs_rq[86]:/.tg->runnable_avg
38326 ± 2% -89.5% 4039 ± 3% sched_debug.cfs_rq[87]:/.tg->runnable_avg
38322 ± 2% -89.5% 4040 ± 3% sched_debug.cfs_rq[88]:/.tg->runnable_avg
38313 ± 2% -89.4% 4044 ± 3% sched_debug.cfs_rq[89]:/.tg->runnable_avg
37445 ± 1% -89.7% 3853 ± 4% sched_debug.cfs_rq[8]:/.tg->runnable_avg
553 ± 4% -92.0% 44 ± 5% sched_debug.cfs_rq[8]:/.tg_runnable_contrib
25270 ± 4% -91.8% 2076 ± 5% sched_debug.cfs_rq[8]:/.avg->runnable_avg_sum
77680 ± 8% -91.2% 6799 ± 6% sched_debug.cfs_rq[8]:/.exec_clock
5833584 ± 8% -98.3% 99293 ± 11% sched_debug.cfs_rq[8]:/.min_vruntime
38307 ± 2% -89.4% 4044 ± 3% sched_debug.cfs_rq[90]:/.tg->runnable_avg
38316 ± 2% -89.4% 4047 ± 3% sched_debug.cfs_rq[91]:/.tg->runnable_avg
4788 ± 48% -95.1% 235 ± 35% sched_debug.cfs_rq[91]:/.avg->runnable_avg_sum
103 ± 49% -95.8% 4 ± 39% sched_debug.cfs_rq[91]:/.tg_runnable_contrib
38320 ± 2% -89.4% 4051 ± 3% sched_debug.cfs_rq[92]:/.tg->runnable_avg
38332 ± 2% -89.4% 4052 ± 3% sched_debug.cfs_rq[93]:/.tg->runnable_avg
38341 ± 2% -89.4% 4053 ± 3% sched_debug.cfs_rq[94]:/.tg->runnable_avg
38329 ± 2% -89.4% 4056 ± 3% sched_debug.cfs_rq[95]:/.tg->runnable_avg
38315 ± 2% -89.4% 4060 ± 3% sched_debug.cfs_rq[96]:/.tg->runnable_avg
38318 ± 2% -89.4% 4061 ± 3% sched_debug.cfs_rq[97]:/.tg->runnable_avg
38321 ± 2% -89.4% 4064 ± 3% sched_debug.cfs_rq[98]:/.tg->runnable_avg
38326 ± 2% -89.4% 4066 ± 3% sched_debug.cfs_rq[99]:/.tg->runnable_avg
2599395.82 ± 28% -100.0% 0.00 ± 0% sched_debug.cfs_rq[9]:/.MIN_vruntime
37435 ± 1% -89.7% 3849 ± 4% sched_debug.cfs_rq[9]:/.tg->runnable_avg
7 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[9]:/.runnable_load_avg
25086 ± 4% -93.1% 1741 ± 14% sched_debug.cfs_rq[9]:/.avg->runnable_avg_sum
76760 ± 8% -91.3% 6708 ± 10% sched_debug.cfs_rq[9]:/.exec_clock
2599396.15 ± 28% -100.0% 0.00 ± 0% sched_debug.cfs_rq[9]:/.max_vruntime
550 ± 4% -93.2% 37 ± 15% sched_debug.cfs_rq[9]:/.tg_runnable_contrib
5775897 ± 7% -98.3% 99243 ± 18% sched_debug.cfs_rq[9]:/.min_vruntime
7 ± 39% +11559.1% 855 ± 0% sched_debug.cpu#0.cpu_load[3]
8 ± 35% +9767.3% 855 ± 0% sched_debug.cpu#0.cpu_load[2]
10 ± 25% +8443.3% 854 ± 0% sched_debug.cpu#0.cpu_load[1]
6 ± 39% +14133.3% 854 ± 0% sched_debug.cpu#0.cpu_load[4]
866410 ± 20% -42.2% 500354 ± 0% sched_debug.cpu#0.max_idle_balance_cost
401816 ± 6% +5302.7% 21709084 ± 9% sched_debug.cpu#0.ttwu_count
312394 ± 14% +1631.2% 5408220 ± 18% sched_debug.cpu#1.sched_goidle
177603 ± 24% -87.5% 22288 ± 17% sched_debug.cpu#1.ttwu_local
303227 ± 7% -46.2% 163110 ± 12% sched_debug.cpu#1.avg_idle
695685 ± 12% +1457.9% 10837841 ± 18% sched_debug.cpu#1.nr_switches
699186 ± 12% +1450.1% 10837966 ± 18% sched_debug.cpu#1.sched_count
461875 ± 8% -94.9% 23602 ± 15% sched_debug.cpu#1.ttwu_count
262569 ± 5% +240.9% 895031 ± 14% sched_debug.cpu#10.avg_idle
145636 ± 7% -30.0% 101980 ± 10% sched_debug.cpu#10.nr_load_updates
4 ± 36% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[1]
3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[3]
3 ± 34% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[2]
534226 ± 10% -99.7% 1612 ± 29% sched_debug.cpu#10.ttwu_count
182313 ± 10% -99.3% 1351 ± 29% sched_debug.cpu#10.ttwu_local
912692 ± 29% -45.2% 500000 ± 0% sched_debug.cpu#100.max_idle_balance_cost
890003 ± 25% -43.5% 502419 ± 0% sched_debug.cpu#101.max_idle_balance_cost
895162 ± 26% -42.0% 518876 ± 8% sched_debug.cpu#102.max_idle_balance_cost
1279055 ± 16% -18.8% 1038141 ± 8% sched_debug.cpu#102.avg_idle
1210646 ± 23% -19.2% 978478 ± 1% sched_debug.cpu#103.avg_idle
873261 ± 25% -42.7% 500000 ± 0% sched_debug.cpu#103.max_idle_balance_cost
864952 ± 24% -42.2% 500000 ± 0% sched_debug.cpu#104.max_idle_balance_cost
887413 ± 31% -43.7% 500000 ± 0% sched_debug.cpu#105.max_idle_balance_cost
854103 ± 27% -40.9% 504735 ± 2% sched_debug.cpu#106.max_idle_balance_cost
864081 ± 28% -42.0% 501124 ± 0% sched_debug.cpu#107.max_idle_balance_cost
898657 ± 28% -42.5% 516711 ± 6% sched_debug.cpu#108.max_idle_balance_cost
3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#11.cpu_load[2]
543726 ± 4% -99.7% 1760 ± 32% sched_debug.cpu#11.ttwu_count
145709 ± 7% -30.3% 101564 ± 9% sched_debug.cpu#11.nr_load_updates
270996 ± 1% +229.9% 894099 ± 14% sched_debug.cpu#11.avg_idle
2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#11.cpu_load[3]
185681 ± 10% -99.2% 1455 ± 30% sched_debug.cpu#11.ttwu_local
873654 ± 27% -42.5% 502080 ± 0% sched_debug.cpu#110.max_idle_balance_cost
858843 ± 30% -41.1% 505597 ± 1% sched_debug.cpu#111.max_idle_balance_cost
872547 ± 28% -42.7% 500133 ± 0% sched_debug.cpu#112.max_idle_balance_cost
850260 ± 28% -40.6% 505082 ± 2% sched_debug.cpu#113.max_idle_balance_cost
960366 ± 23% -47.8% 500984 ± 0% sched_debug.cpu#114.max_idle_balance_cost
869532 ± 27% -42.2% 502156 ± 0% sched_debug.cpu#115.max_idle_balance_cost
884039 ± 27% -42.9% 505044 ± 1% sched_debug.cpu#116.max_idle_balance_cost
855531 ± 30% -41.2% 502901 ± 1% sched_debug.cpu#117.max_idle_balance_cost
844778 ± 29% -40.2% 505544 ± 1% sched_debug.cpu#118.max_idle_balance_cost
867345 ± 27% -41.1% 510969 ± 4% sched_debug.cpu#119.max_idle_balance_cost
145505 ± 8% -34.3% 95650 ± 15% sched_debug.cpu#12.nr_load_updates
5 ± 44% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[1]
186685 ± 11% -99.3% 1223 ± 33% sched_debug.cpu#12.ttwu_local
4 ± 49% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[2]
280891 ± 3% +222.5% 905991 ± 14% sched_debug.cpu#12.avg_idle
3 ± 39% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[3]
187388 ± 11% -99.3% 1223 ± 30% sched_debug.cpu#13.ttwu_local
145475 ± 8% -31.9% 99013 ± 11% sched_debug.cpu#13.nr_load_updates
546320 ± 8% -99.7% 1508 ± 21% sched_debug.cpu#13.ttwu_count
3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#13.cpu_load[1]
282818 ± 5% +212.5% 883684 ± 13% sched_debug.cpu#13.avg_idle
145427 ± 7% -34.5% 95288 ± 17% sched_debug.cpu#14.nr_load_updates
4 ± 41% -100.0% 0 ± 0% sched_debug.cpu#14.cpu_load[3]
283614 ± 6% +204.3% 862952 ± 17% sched_debug.cpu#14.avg_idle
548763 ± 10% -99.7% 1508 ± 26% sched_debug.cpu#14.ttwu_count
187787 ± 10% -99.3% 1258 ± 24% sched_debug.cpu#14.ttwu_local
4 ± 46% -100.0% 0 ± 0% sched_debug.cpu#14.cpu_load[2]
910931 ± 10% -90.9% 83254 ± 28% sched_debug.cpu#15.sched_count
401751 ± 10% -89.7% 41393 ± 28% sched_debug.cpu#15.sched_goidle
532305 ± 9% -99.3% 3824 ± 15% sched_debug.cpu#15.ttwu_count
227026 ± 15% -98.7% 2840 ± 12% sched_debug.cpu#15.ttwu_local
2 ± 46% -100.0% 0 ± 0% sched_debug.cpu#15.cpu_load[2]
907583 ± 10% -90.8% 83244 ± 28% sched_debug.cpu#15.nr_switches
147307 ± 8% -64.1% 52925 ± 18% sched_debug.cpu#15.nr_load_updates
271896 ± 28% +268.2% 1001194 ± 0% sched_debug.cpu#15.avg_idle
228547 ± 14% -99.6% 1006 ± 46% sched_debug.cpu#16.ttwu_local
146688 ± 7% -64.8% 51642 ± 18% sched_debug.cpu#16.nr_load_updates
538118 ± 10% -99.7% 1836 ± 36% sched_debug.cpu#16.ttwu_count
401589 ± 9% -89.9% 40532 ± 27% sched_debug.cpu#16.sched_goidle
912282 ± 10% -91.1% 81551 ± 27% sched_debug.cpu#16.sched_count
274129 ± 10% +258.6% 983056 ± 2% sched_debug.cpu#16.avg_idle
909937 ± 10% -91.0% 81542 ± 27% sched_debug.cpu#16.nr_switches
399818 ± 10% -90.8% 36949 ± 41% sched_debug.cpu#17.sched_goidle
251250 ± 33% +290.1% 980211 ± 2% sched_debug.cpu#17.avg_idle
146193 ± 8% -67.5% 47493 ± 30% sched_debug.cpu#17.nr_load_updates
907652 ± 10% -91.8% 74306 ± 41% sched_debug.cpu#17.sched_count
905857 ± 10% -91.8% 74297 ± 41% sched_debug.cpu#17.nr_switches
535902 ± 9% -99.7% 1645 ± 28% sched_debug.cpu#17.ttwu_count
535263 ± 9% -99.6% 2125 ± 33% sched_debug.cpu#18.ttwu_count
904084 ± 11% -93.0% 62907 ± 44% sched_debug.cpu#18.nr_switches
274939 ± 9% +266.2% 1006769 ± 2% sched_debug.cpu#18.avg_idle
226681 ± 14% -99.5% 1123 ± 40% sched_debug.cpu#18.ttwu_local
905555 ± 11% -93.1% 62916 ± 43% sched_debug.cpu#18.sched_count
146165 ± 7% -71.0% 42321 ± 30% sched_debug.cpu#18.nr_load_updates
399028 ± 10% -92.2% 31022 ± 44% sched_debug.cpu#18.sched_goidle
146308 ± 8% -71.5% 41686 ± 35% sched_debug.cpu#19.nr_load_updates
529715 ± 9% -99.6% 2212 ± 43% sched_debug.cpu#19.ttwu_count
296225 ± 10% +232.9% 986018 ± 1% sched_debug.cpu#19.avg_idle
3 ± 38% +605.3% 22 ± 29% sched_debug.cpu#2.cpu_load[2]
683586 ± 9% +1403.7% 10278920 ± 19% sched_debug.cpu#2.sched_count
3 ± 41% +460.9% 21 ± 46% sched_debug.cpu#2.cpu_load[1]
420236 ± 7% -95.2% 20376 ± 17% sched_debug.cpu#2.ttwu_count
154717 ± 17% -87.2% 19810 ± 18% sched_debug.cpu#2.ttwu_local
680385 ± 9% +1410.7% 10278585 ± 19% sched_debug.cpu#2.nr_switches
300978 ± 12% -50.5% 149090 ± 40% sched_debug.cpu#2.avg_idle
300267 ± 10% +1608.3% 5129477 ± 19% sched_debug.cpu#2.sched_goidle
226744 ± 15% -99.7% 791 ± 12% sched_debug.cpu#20.ttwu_local
2 ± 41% -100.0% 0 ± 0% sched_debug.cpu#20.cpu_load[3]
539626 ± 11% -99.7% 1714 ± 13% sched_debug.cpu#20.ttwu_count
3 ± 48% -100.0% 0 ± 0% sched_debug.cpu#20.cpu_load[1]
264020 ± 16% +277.1% 995578 ± 0% sched_debug.cpu#20.avg_idle
145629 ± 7% -72.8% 39540 ± 38% sched_debug.cpu#20.nr_load_updates
525878 ± 11% -99.6% 1905 ± 16% sched_debug.cpu#21.ttwu_count
145597 ± 8% -72.4% 40198 ± 40% sched_debug.cpu#21.nr_load_updates
6 ± 40% -100.0% 0 ± 0% sched_debug.cpu#21.load
226311 ± 17% -99.6% 954 ± 17% sched_debug.cpu#21.ttwu_local
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#21.cpu_load[3]
282162 ± 10% +268.8% 1040716 ± 4% sched_debug.cpu#21.avg_idle
2 ± 40% -100.0% 0 ± 0% sched_debug.cpu#21.cpu_load[2]
4 ± 40% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[1]
3 ± 34% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[2]
145443 ± 7% -73.8% 38078 ± 39% sched_debug.cpu#22.nr_load_updates
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[3]
547461 ± 10% -99.6% 2080 ± 41% sched_debug.cpu#22.ttwu_count
293095 ± 9% +242.6% 1004209 ± 2% sched_debug.cpu#22.avg_idle
538395 ± 10% -99.7% 1761 ± 30% sched_debug.cpu#23.ttwu_count
145290 ± 8% -77.3% 33007 ± 45% sched_debug.cpu#23.nr_load_updates
312631 ± 11% +217.5% 992648 ± 1% sched_debug.cpu#23.avg_idle
5 ± 42% -100.0% 0 ± 0% sched_debug.cpu#23.load
274102 ± 9% +261.8% 991658 ± 1% sched_debug.cpu#24.avg_idle
145062 ± 8% -75.6% 35376 ± 39% sched_debug.cpu#24.nr_load_updates
305640 ± 8% +215.8% 965318 ± 7% sched_debug.cpu#25.avg_idle
554385 ± 4% -99.6% 2368 ± 47% sched_debug.cpu#25.ttwu_count
146404 ± 6% -77.2% 33321 ± 47% sched_debug.cpu#25.nr_load_updates
562613 ± 11% -99.6% 2233 ± 32% sched_debug.cpu#26.ttwu_count
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#26.cpu_load[2]
291842 ± 8% +239.5% 990850 ± 4% sched_debug.cpu#26.avg_idle
3 ± 37% -100.0% 0 ± 0% sched_debug.cpu#27.cpu_load[2]
4 ± 44% -100.0% 0 ± 0% sched_debug.cpu#27.cpu_load[1]
296953 ± 8% +234.6% 993660 ± 3% sched_debug.cpu#27.avg_idle
227387 ± 14% -99.6% 875 ± 46% sched_debug.cpu#28.ttwu_local
541565 ± 9% -99.7% 1729 ± 35% sched_debug.cpu#28.ttwu_count
287000 ± 9% +247.4% 996956 ± 1% sched_debug.cpu#28.avg_idle
292577 ± 5% +240.4% 995812 ± 9% sched_debug.cpu#29.avg_idle
2 ± 49% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[0]
2 ± 31% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[1]
3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[2]
538059 ± 11% -99.6% 1942 ± 46% sched_debug.cpu#29.ttwu_count
2 ± 40% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[3]
858348 ± 11% +988.5% 9343383 ± 19% sched_debug.cpu#3.nr_switches
583644 ± 14% -96.9% 18374 ± 19% sched_debug.cpu#3.ttwu_count
188558 ± 11% -90.6% 17658 ± 20% sched_debug.cpu#3.ttwu_local
861080 ± 11% +985.1% 9343434 ± 19% sched_debug.cpu#3.sched_count
3 ± 28% +325.0% 14 ± 46% sched_debug.cpu#3.cpu_load[2]
283363 ± 7% -36.9% 178917 ± 5% sched_debug.cpu#3.avg_idle
3 ± 27% +366.7% 14 ± 35% sched_debug.cpu#3.cpu_load[3]
365993 ± 11% +1174.1% 4662957 ± 19% sched_debug.cpu#3.sched_goidle
272296 ± 5% +263.0% 988390 ± 3% sched_debug.cpu#30.avg_idle
886491 ± 7% -90.7% 82339 ± 23% sched_debug.cpu#30.nr_switches
193322 ± 6% -99.6% 752 ± 14% sched_debug.cpu#30.ttwu_local
890539 ± 7% -90.7% 82392 ± 23% sched_debug.cpu#30.sched_count
145061 ± 7% -66.4% 48735 ± 20% sched_debug.cpu#30.nr_load_updates
385340 ± 7% -89.4% 40937 ± 23% sched_debug.cpu#30.sched_goidle
558855 ± 11% -99.7% 1486 ± 14% sched_debug.cpu#30.ttwu_count
607204 ± 10% -99.7% 1669 ± 28% sched_debug.cpu#31.ttwu_count
948582 ± 13% -92.4% 72412 ± 23% sched_debug.cpu#31.sched_count
147499 ± 6% -70.2% 44027 ± 20% sched_debug.cpu#31.nr_load_updates
231115 ± 34% -99.7% 707 ± 17% sched_debug.cpu#31.ttwu_local
414152 ± 16% -91.3% 35947 ± 23% sched_debug.cpu#31.sched_goidle
945517 ± 13% -92.3% 72360 ± 23% sched_debug.cpu#31.nr_switches
269881 ± 11% +260.8% 973666 ± 4% sched_debug.cpu#31.avg_idle
146258 ± 6% -72.3% 40536 ± 31% sched_debug.cpu#32.nr_load_updates
212916 ± 19% -99.7% 682 ± 15% sched_debug.cpu#32.ttwu_local
616801 ± 13% -99.8% 1439 ± 11% sched_debug.cpu#32.ttwu_count
886655 ± 10% -92.6% 65901 ± 36% sched_debug.cpu#32.sched_count
884019 ± 10% -92.6% 65851 ± 36% sched_debug.cpu#32.nr_switches
287282 ± 4% +221.9% 924747 ± 3% sched_debug.cpu#32.avg_idle
386416 ± 11% -91.5% 32707 ± 36% sched_debug.cpu#32.sched_goidle
381805 ± 6% -92.2% 29619 ± 35% sched_debug.cpu#33.sched_goidle
882037 ± 6% -93.2% 59641 ± 35% sched_debug.cpu#33.sched_count
264381 ± 4% +251.9% 930329 ± 7% sched_debug.cpu#33.avg_idle
580711 ± 13% -99.8% 1425 ± 8% sched_debug.cpu#33.ttwu_count
2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#33.cpu_load[2]
198746 ± 7% -99.7% 677 ± 13% sched_debug.cpu#33.ttwu_local
879687 ± 6% -93.2% 59612 ± 35% sched_debug.cpu#33.nr_switches
145237 ± 7% -73.9% 37912 ± 30% sched_debug.cpu#33.nr_load_updates
558432 ± 12% -99.7% 1937 ± 40% sched_debug.cpu#34.ttwu_count
271737 ± 6% +267.3% 998140 ± 8% sched_debug.cpu#34.avg_idle
144986 ± 7% -74.4% 37118 ± 42% sched_debug.cpu#34.nr_load_updates
192853 ± 6% -99.6% 781 ± 15% sched_debug.cpu#34.ttwu_local
255816 ± 24% +287.7% 991833 ± 1% sched_debug.cpu#35.avg_idle
144906 ± 7% -74.4% 37168 ± 35% sched_debug.cpu#35.nr_load_updates
554138 ± 12% -99.7% 1813 ± 9% sched_debug.cpu#35.ttwu_count
379077 ± 6% -92.2% 29594 ± 42% sched_debug.cpu#35.sched_goidle
879048 ± 7% -93.2% 59608 ± 42% sched_debug.cpu#35.sched_count
876729 ± 7% -93.2% 59559 ± 42% sched_debug.cpu#35.nr_switches
192396 ± 6% -99.6% 796 ± 19% sched_debug.cpu#35.ttwu_local
144811 ± 7% -76.8% 33645 ± 44% sched_debug.cpu#36.nr_load_updates
198260 ± 3% -99.7% 681 ± 15% sched_debug.cpu#36.ttwu_local
3 ± 38% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[2]
608120 ± 19% -99.7% 1607 ± 10% sched_debug.cpu#36.ttwu_count
272324 ± 10% +271.7% 1012309 ± 4% sched_debug.cpu#36.avg_idle
4 ± 47% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[0]
3 ± 48% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[1]
144880 ± 8% -75.5% 35491 ± 42% sched_debug.cpu#37.nr_load_updates
551377 ± 10% -99.7% 1487 ± 10% sched_debug.cpu#37.ttwu_count
193906 ± 7% -99.6% 680 ± 18% sched_debug.cpu#37.ttwu_local
277677 ± 3% +256.2% 989065 ± 1% sched_debug.cpu#37.avg_idle
4 ± 49% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[1]
271911 ± 6% +259.7% 978143 ± 1% sched_debug.cpu#38.avg_idle
4 ± 37% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[2]
588979 ± 20% -99.7% 1876 ± 49% sched_debug.cpu#38.ttwu_count
195832 ± 6% -99.7% 662 ± 13% sched_debug.cpu#38.ttwu_local
3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[3]
859499 ± 7% -94.2% 49501 ± 48% sched_debug.cpu#39.nr_switches
197080 ± 5% -99.7% 616 ± 14% sched_debug.cpu#39.ttwu_local
144875 ± 8% -78.5% 31180 ± 38% sched_debug.cpu#39.nr_load_updates
861583 ± 7% -94.2% 50175 ± 46% sched_debug.cpu#39.sched_count
372809 ± 7% -93.4% 24588 ± 48% sched_debug.cpu#39.sched_goidle
585682 ± 15% -99.8% 1422 ± 11% sched_debug.cpu#39.ttwu_count
254598 ± 28% +291.0% 995522 ± 0% sched_debug.cpu#39.avg_idle
384496 ± 7% +624.1% 2784133 ± 20% sched_debug.cpu#4.sched_goidle
903009 ± 6% +517.8% 5579241 ± 20% sched_debug.cpu#4.sched_count
900237 ± 6% +519.7% 5579190 ± 20% sched_debug.cpu#4.nr_switches
556589 ± 9% -97.9% 11720 ± 21% sched_debug.cpu#4.ttwu_count
193650 ± 19% -94.2% 11155 ± 20% sched_debug.cpu#4.ttwu_local
279229 ± 6% +261.3% 1008863 ± 1% sched_debug.cpu#40.avg_idle
564514 ± 13% -99.8% 1408 ± 18% sched_debug.cpu#40.ttwu_count
2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#40.cpu_load[0]
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#40.cpu_load[4]
196449 ± 8% -99.7% 599 ± 25% sched_debug.cpu#40.ttwu_local
195020 ± 4% -99.7% 665 ± 32% sched_debug.cpu#41.ttwu_local
569391 ± 15% -99.7% 1443 ± 22% sched_debug.cpu#41.ttwu_count
250174 ± 24% +299.0% 998144 ± 0% sched_debug.cpu#41.avg_idle
7 ± 46% -100.0% 0 ± 0% sched_debug.cpu#41.load
271444 ± 4% +267.2% 996765 ± 3% sched_debug.cpu#42.avg_idle
3 ± 35% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[2]
4 ± 43% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[1]
3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[3]
594265 ± 15% -99.8% 1417 ± 26% sched_debug.cpu#42.ttwu_count
198649 ± 7% -99.7% 578 ± 29% sched_debug.cpu#42.ttwu_local
270822 ± 9% +262.7% 982391 ± 2% sched_debug.cpu#43.avg_idle
3 ± 14% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[2]
595801 ± 15% -99.8% 1474 ± 35% sched_debug.cpu#43.ttwu_count
2 ± 24% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[3]
199411 ± 4% -99.7% 613 ± 22% sched_debug.cpu#43.ttwu_local
4 ± 25% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[1]
5 ± 23% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[0]
144742 ± 8% -79.2% 30152 ± 47% sched_debug.cpu#43.nr_load_updates
276311 ± 3% +258.0% 989070 ± 2% sched_debug.cpu#44.avg_idle
582806 ± 15% -99.8% 1362 ± 30% sched_debug.cpu#44.ttwu_count
144761 ± 8% -80.1% 28756 ± 45% sched_debug.cpu#44.nr_load_updates
144710 ± 7% -67.6% 46849 ± 27% sched_debug.cpu#45.nr_load_updates
851364 ± 13% -90.0% 85464 ± 31% sched_debug.cpu#45.sched_count
847235 ± 13% -89.9% 85423 ± 31% sched_debug.cpu#45.nr_switches
372059 ± 14% -88.6% 42469 ± 31% sched_debug.cpu#45.sched_goidle
203310 ± 19% -99.6% 843 ± 17% sched_debug.cpu#45.ttwu_local
281126 ± 7% +253.4% 993587 ± 5% sched_debug.cpu#45.avg_idle
573414 ± 12% -99.7% 1515 ± 21% sched_debug.cpu#45.ttwu_count
861853 ± 12% -92.4% 65635 ± 43% sched_debug.cpu#46.nr_switches
273192 ± 7% +268.5% 1006815 ± 2% sched_debug.cpu#46.avg_idle
864903 ± 12% -92.4% 65678 ± 43% sched_debug.cpu#46.sched_count
376696 ± 13% -91.3% 32592 ± 43% sched_debug.cpu#46.sched_goidle
552783 ± 9% -99.7% 1450 ± 14% sched_debug.cpu#46.ttwu_count
201314 ± 20% -99.6% 706 ± 19% sched_debug.cpu#46.ttwu_local
144703 ± 7% -74.6% 36762 ± 34% sched_debug.cpu#46.nr_load_updates
589 ± 34% -100.0% 0 ± 0% sched_debug.cpu#46.curr->pid
584555 ± 7% -99.7% 1599 ± 16% sched_debug.cpu#47.ttwu_count
203884 ± 19% -99.6% 723 ± 23% sched_debug.cpu#47.ttwu_local
144799 ± 7% -76.4% 34218 ± 46% sched_debug.cpu#47.nr_load_updates
274538 ± 6% +260.1% 988726 ± 1% sched_debug.cpu#47.avg_idle
3 ± 35% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[1]
144673 ± 8% -78.2% 31538 ± 48% sched_debug.cpu#48.nr_load_updates
543724 ± 4% -99.7% 1672 ± 18% sched_debug.cpu#48.ttwu_count
199514 ± 22% -99.7% 684 ± 34% sched_debug.cpu#48.ttwu_local
2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[2]
3 ± 44% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[0]
280359 ± 8% +263.5% 1019216 ± 4% sched_debug.cpu#48.avg_idle
564757 ± 6% -99.7% 1635 ± 18% sched_debug.cpu#49.ttwu_count
203456 ± 21% -99.7% 710 ± 29% sched_debug.cpu#49.ttwu_local
144637 ± 7% -78.9% 30527 ± 47% sched_debug.cpu#49.nr_load_updates
302435 ± 19% +233.8% 1009483 ± 3% sched_debug.cpu#49.avg_idle
7 ± 35% -100.0% 0 ± 0% sched_debug.cpu#5.load
576004 ± 11% -99.5% 2984 ± 18% sched_debug.cpu#5.ttwu_count
184101 ± 10% -98.6% 2497 ± 17% sched_debug.cpu#5.ttwu_local
256732 ± 5% +131.1% 593404 ± 27% sched_debug.cpu#5.avg_idle
2 ± 49% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[3]
200344 ± 20% -99.7% 688 ± 28% sched_debug.cpu#50.ttwu_local
2 ± 31% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[2]
563146 ± 5% -99.7% 1908 ± 44% sched_debug.cpu#50.ttwu_count
2 ± 41% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[1]
244904 ± 22% +302.4% 985599 ± 1% sched_debug.cpu#50.avg_idle
144490 ± 7% -79.9% 29035 ± 47% sched_debug.cpu#51.nr_load_updates
273633 ± 7% +265.2% 999259 ± 2% sched_debug.cpu#51.avg_idle
555030 ± 5% -99.7% 1398 ± 23% sched_debug.cpu#51.ttwu_count
202800 ± 21% -99.7% 617 ± 28% sched_debug.cpu#51.ttwu_local
204402 ± 20% -99.7% 672 ± 35% sched_debug.cpu#52.ttwu_local
6 ± 37% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[2]
266105 ± 8% +268.6% 980753 ± 3% sched_debug.cpu#52.avg_idle
8 ± 40% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[1]
4 ± 33% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[3]
590313 ± 10% -99.8% 1394 ± 15% sched_debug.cpu#52.ttwu_count
246340 ± 20% +293.7% 969729 ± 5% sched_debug.cpu#53.avg_idle
580316 ± 10% -99.7% 1479 ± 17% sched_debug.cpu#53.ttwu_count
202868 ± 20% -99.7% 559 ± 24% sched_debug.cpu#53.ttwu_local
275898 ± 8% +251.8% 970687 ± 6% sched_debug.cpu#54.avg_idle
580835 ± 12% -99.8% 1420 ± 27% sched_debug.cpu#54.ttwu_count
203860 ± 21% -99.7% 669 ± 25% sched_debug.cpu#54.ttwu_local
561511 ± 12% -99.8% 1320 ± 32% sched_debug.cpu#55.ttwu_count
199859 ± 21% -99.7% 615 ± 44% sched_debug.cpu#55.ttwu_local
269166 ± 3% +267.9% 990163 ± 1% sched_debug.cpu#55.avg_idle
592219 ± 10% -99.8% 1394 ± 20% sched_debug.cpu#56.ttwu_count
203190 ± 20% -99.7% 622 ± 21% sched_debug.cpu#56.ttwu_local
285148 ± 8% +244.9% 983467 ± 2% sched_debug.cpu#56.avg_idle
289395 ± 8% +243.8% 994802 ± 1% sched_debug.cpu#57.avg_idle
201174 ± 22% -99.7% 588 ± 32% sched_debug.cpu#57.ttwu_local
561875 ± 9% -99.8% 1333 ± 32% sched_debug.cpu#57.ttwu_count
532410 ± 13% -99.8% 1321 ± 14% sched_debug.cpu#58.ttwu_count
199150 ± 23% -99.7% 623 ± 30% sched_debug.cpu#58.ttwu_local
281144 ± 9% +248.6% 979942 ± 4% sched_debug.cpu#58.avg_idle
206191 ± 20% -99.7% 576 ± 34% sched_debug.cpu#59.ttwu_local
290618 ± 4% +237.8% 981621 ± 2% sched_debug.cpu#59.avg_idle
568974 ± 8% -99.8% 1378 ± 21% sched_debug.cpu#59.ttwu_count
145823 ± 7% -17.5% 120332 ± 3% sched_debug.cpu#6.nr_load_updates
179817 ± 10% -99.1% 1693 ± 19% sched_debug.cpu#6.ttwu_local
546345 ± 8% -99.6% 2115 ± 21% sched_debug.cpu#6.ttwu_count
278607 ± 3% +199.1% 833241 ± 18% sched_debug.cpu#6.avg_idle
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#6.cpu_load[3]
851921 ± 8% -40.4% 508151 ± 46% sched_debug.cpu#6.nr_switches
854405 ± 8% -40.5% 508206 ± 46% sched_debug.cpu#6.sched_count
1 ± 35% -100.0% 0 ± 0% sched_debug.cpu#6.cpu_load[4]
137215 ± 8% -92.7% 9982 ± 37% sched_debug.cpu#60.nr_load_updates
916934 ± 7% -39.4% 555767 ± 14% sched_debug.cpu#60.max_idle_balance_cost
496648 ± 21% +124.0% 1112535 ± 14% sched_debug.cpu#60.avg_idle
523174 ± 17% +90.8% 997981 ± 1% sched_debug.cpu#61.avg_idle
207882 ± 14% -99.9% 283 ± 41% sched_debug.cpu#61.ttwu_local
925690 ± 4% -45.3% 506045 ± 2% sched_debug.cpu#61.max_idle_balance_cost
303435 ± 20% -99.8% 571 ± 21% sched_debug.cpu#61.ttwu_count
203648 ± 13% -99.9% 158 ± 29% sched_debug.cpu#62.ttwu_local
904843 ± 5% -44.4% 502645 ± 1% sched_debug.cpu#62.max_idle_balance_cost
305661 ± 24% -99.9% 261 ± 30% sched_debug.cpu#62.ttwu_count
529 ± 32% -100.0% 0 ± 0% sched_debug.cpu#62.curr->pid
1080713 ± 10% -53.7% 500459 ± 0% sched_debug.cpu#63.max_idle_balance_cost
1699969 ± 10% -41.0% 1003536 ± 2% sched_debug.cpu#64.avg_idle
1096675 ± 9% -54.0% 504808 ± 2% sched_debug.cpu#64.max_idle_balance_cost
2374 ± 48% -91.1% 211 ± 27% sched_debug.cpu#64.ttwu_count
1073352 ± 9% -51.1% 524554 ± 7% sched_debug.cpu#65.max_idle_balance_cost
1599502 ± 9% -34.4% 1049932 ± 7% sched_debug.cpu#65.avg_idle
39322 ± 42% -72.4% 10848 ± 40% sched_debug.cpu#66.nr_load_updates
1409179 ± 19% -27.9% 1015680 ± 3% sched_debug.cpu#66.avg_idle
1076703 ± 7% -52.6% 510769 ± 4% sched_debug.cpu#66.max_idle_balance_cost
1086785 ± 9% -53.0% 510290 ± 4% sched_debug.cpu#67.max_idle_balance_cost
1411451 ± 16% -29.9% 989837 ± 2% sched_debug.cpu#68.avg_idle
38684 ± 43% -66.9% 12790 ± 34% sched_debug.cpu#68.nr_load_updates
1108217 ± 7% -54.9% 500229 ± 0% sched_debug.cpu#68.max_idle_balance_cost
1077891 ± 7% -47.7% 563733 ± 22% sched_debug.cpu#69.max_idle_balance_cost
566207 ± 7% -99.7% 1748 ± 21% sched_debug.cpu#7.ttwu_count
262075 ± 9% +236.5% 881956 ± 13% sched_debug.cpu#7.avg_idle
838568 ± 9% -53.9% 386340 ± 44% sched_debug.cpu#7.sched_count
357789 ± 9% -46.2% 192547 ± 44% sched_debug.cpu#7.sched_goidle
836222 ± 9% -53.8% 386289 ± 44% sched_debug.cpu#7.nr_switches
145736 ± 7% -22.7% 112707 ± 3% sched_debug.cpu#7.nr_load_updates
182859 ± 10% -99.2% 1443 ± 23% sched_debug.cpu#7.ttwu_local
1073432 ± 10% -51.3% 522486 ± 8% sched_debug.cpu#70.max_idle_balance_cost
1530831 ± 15% -35.2% 992013 ± 1% sched_debug.cpu#71.avg_idle
1054705 ± 6% -52.6% 500000 ± 0% sched_debug.cpu#71.max_idle_balance_cost
1069098 ± 9% -50.7% 527311 ± 9% sched_debug.cpu#72.max_idle_balance_cost
38445 ± 45% -61.5% 14811 ± 40% sched_debug.cpu#72.nr_load_updates
1356913 ± 13% -22.4% 1053465 ± 9% sched_debug.cpu#72.avg_idle
1332336 ± 10% -26.8% 975029 ± 6% sched_debug.cpu#73.avg_idle
1058495 ± 8% -51.8% 510063 ± 3% sched_debug.cpu#73.max_idle_balance_cost
2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#73.cpu_load[1]
1067890 ± 8% -52.7% 504615 ± 1% sched_debug.cpu#74.max_idle_balance_cost
682497 ± 27% -26.7% 500478 ± 0% sched_debug.cpu#76.max_idle_balance_cost
693411 ± 28% -27.4% 503241 ± 1% sched_debug.cpu#77.max_idle_balance_cost
734302 ± 27% -30.9% 507513 ± 2% sched_debug.cpu#78.max_idle_balance_cost
681179 ± 28% -21.9% 531938 ± 10% sched_debug.cpu#79.max_idle_balance_cost
540200 ± 8% -99.7% 1700 ± 13% sched_debug.cpu#8.ttwu_count
178504 ± 11% -99.2% 1373 ± 19% sched_debug.cpu#8.ttwu_local
145742 ± 7% -25.5% 108570 ± 3% sched_debug.cpu#8.nr_load_updates
361282 ± 8% -51.3% 176052 ± 43% sched_debug.cpu#8.sched_goidle
276970 ± 8% +230.5% 915429 ± 10% sched_debug.cpu#8.avg_idle
846491 ± 7% -58.3% 353331 ± 43% sched_debug.cpu#8.sched_count
843911 ± 7% -58.1% 353246 ± 43% sched_debug.cpu#8.nr_switches
1359 ± 48% -59.4% 552 ± 9% sched_debug.cpu#81.ttwu_count
682840 ± 27% -26.2% 503819 ± 1% sched_debug.cpu#81.max_idle_balance_cost
680252 ± 27% -24.4% 513984 ± 4% sched_debug.cpu#82.max_idle_balance_cost
688877 ± 33% -26.0% 509483 ± 3% sched_debug.cpu#83.max_idle_balance_cost
670276 ± 22% -24.9% 503665 ± 1% sched_debug.cpu#85.max_idle_balance_cost
680305 ± 22% -25.8% 504652 ± 1% sched_debug.cpu#86.max_idle_balance_cost
1390 ± 44% -65.2% 483 ± 6% sched_debug.cpu#87.ttwu_count
711200 ± 26% -29.2% 503212 ± 1% sched_debug.cpu#88.max_idle_balance_cost
649693 ± 26% -22.7% 502075 ± 0% sched_debug.cpu#89.max_idle_balance_cost
531933 ± 7% -99.7% 1700 ± 30% sched_debug.cpu#9.ttwu_count
180813 ± 10% -99.2% 1424 ± 27% sched_debug.cpu#9.ttwu_local
855358 ± 8% -58.0% 359635 ± 48% sched_debug.cpu#9.nr_switches
857752 ± 8% -58.1% 359679 ± 48% sched_debug.cpu#9.sched_count
366428 ± 8% -51.1% 179269 ± 48% sched_debug.cpu#9.sched_goidle
255284 ± 21% +256.0% 908885 ± 12% sched_debug.cpu#9.avg_idle
145661 ± 7% -28.0% 104899 ± 5% sched_debug.cpu#9.nr_load_updates
3 ± 33% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[2]
2 ± 38% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[3]
3 ± 40% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[1]
880077 ± 25% -43.2% 500000 ± 0% sched_debug.cpu#90.max_idle_balance_cost
1270327 ± 11% -21.3% 1000000 ± 0% sched_debug.cpu#91.avg_idle
886286 ± 22% -43.6% 500000 ± 0% sched_debug.cpu#91.max_idle_balance_cost
861842 ± 25% -42.0% 500000 ± 0% sched_debug.cpu#92.max_idle_balance_cost
877600 ± 26% -37.3% 550542 ± 20% sched_debug.cpu#93.max_idle_balance_cost
872812 ± 25% -42.7% 500149 ± 0% sched_debug.cpu#94.max_idle_balance_cost
880573 ± 26% -43.0% 501897 ± 0% sched_debug.cpu#95.max_idle_balance_cost
898390 ± 24% -43.6% 506730 ± 2% sched_debug.cpu#96.max_idle_balance_cost
1233750 ± 15% -17.9% 1013461 ± 2% sched_debug.cpu#96.avg_idle
1279764 ± 17% -22.4% 993620 ± 1% sched_debug.cpu#97.avg_idle
887849 ± 23% -43.7% 500000 ± 0% sched_debug.cpu#97.max_idle_balance_cost
881541 ± 22% -43.0% 502208 ± 0% sched_debug.cpu#98.max_idle_balance_cost
883063 ± 25% -43.1% 502857 ± 1% sched_debug.cpu#99.max_idle_balance_cost
brickland3: Brickland Ivy Bridge-EX
Memory: 512G
stderr.netperf:send_omni:recv_data_failed:No_route_to_host
1 ++----------------*-----*--------*------*-----------------------------*
| : : : : |
| : : :: : |
0.8 ++ : : : : : : : : :|
| : : : : : : : : :|
| : : : : : : : : :|
0.6 ++ : : : : : : : : :|
| : : : : : : : : :|
0.4 ++ : : : : : : : : : |
| : : : : : : : : : |
| : : : : : : : : : |
0.2 ++ : : : : : : : : : |
| : : : : :: : : |
| : : : : : : : |
0 *+-*--*--*--*--*-----*-----*--*------*-----*--*--*--*--*--*--*--*--*--+
last_state.booting
1 ++------------------------------------------------------*-------------*
| : |
| :: |
0.8 ++ : : :|
| : : :|
| : : :|
0.6 ++ : : :|
| : : :|
0.4 ++ : : : |
| : : : |
| : : : |
0.2 ++ : : : |
| : : : |
| : : : |
0 *+--*--*---*--*---*--*---*--*---*--*---*--*---*--*---*------*--*---*--+
softirqs.TIMER
6e+06 ++------------------------------------------------------------------+
| |
5e+06 ++ * |
| .*. +: |
|.* *.*.*. .* * * *.* *.* *.* : .*.*.*.*.*.*.*
4e+06 *+ * : : : : : : : * : : * |
| : : :: : : : : : : : : |
3e+06 ++ : : : : : : : : : :: : : : |
| : : : : : : : : : : : : :: |
2e+06 ++ : : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
| *: : * : : : : * :: |
1e+06 O+ O O O O O O O O O : O O : O O O OO O O O O |
| O O O* O * : : O : : O * |
0 ++----------O------------O---O-*-O-----*-O--------------------------+
softirqs.NET_RX
7e+07 ++------------------------------------------------------------------+
| |
6e+07 *+*.*.*.*.* * * * *.* *.* * *.*.* *.*.*.*.*.*.*.*
| : :: : : : : : : : : : : |
5e+07 ++ : :: : : : : : : : : : : |
| * : :: :: : : : : :: : : : |
4e+07 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : |
3e+07 ++ : : : : : : : : : : : : :: |
| : : : : : : : : : : : :: |
2e+07 ++ : : : : : : : : : : :: |
| : : :: * : : : : : :: |
1e+07 ++ *: : : : : : * : |
| : : : : : : : |
0 O+O-O-O-O-O-O-O-OO-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-OO-O-O-O-O-------+
softirqs.SCHED
4.5e+06 ++-----------------------------------------------*----------------+
| :: |
4e+06 ++ .*. * * :: |
3.5e+06 ++* *.* : * : * *.* *.* * *.* : *.**.*.*.*.*
* : +: : : : : : : : : : + |
3e+06 ++ * : :: :: : : : : : : : * |
2.5e+06 ++ : : : : : : : : : :: : : : |
| : : : : : : : : : :: : : : |
2e+06 ++ : : : : :: : : : :: : : : |
1.5e+06 ++ : : : : : : : : : : : : : |
| : : : : : : : : : : : : |
1e+06 ++ : : :: * : : : : : :: |
500000 ++ * : : : : : : * : |
O O OO O +O O O O O : O O :O O O O O O O OO |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
softirqs.RCU
1.8e+06 ++-----------------------------------------------*----------------+
| .** * .* .* +: .*. |
1.6e+06 *+ .*.* + .* : * * : * : * *.* : * **.*.*.*.*
1.4e+06 ++* * : : : : : : : : : : : |
| : :: :: : : : : : : : : |
1.2e+06 ++ : : : : : : : : : :: : : : |
1e+06 ++ : : : : : : : : : :: : : : |
| : : : : :: : : : :: : : : |
800000 ++ : : : : : : : : :: : : : |
600000 ++ : : : : : : : : : : : : |
| : : :: * : : : : : :: |
400000 ++ * : : : : : : * : |
200000 O+ O OO O +O O O O O : O O :O O O O O O OO |
| O O * * : : O : : O O * |
0 ++----O----O-------O-----O--O-*-O-----*-O-------------------------+
cpuidle.C1-IVT-4S.usage
4e+07 ++----------------------------------------------------------------+
O O OO O O O O O O O O O O O O O O |
3.5e+07 ++ O O O |
3e+07 ++ |
| |
2.5e+07 ++ |
*.*.*.*.* * * * *.* *.* * *.*. *. .**.*. .*.*
2e+07 ++ *. +: : : : : : : : : * : * * |
| * : :: :: : : : : :: : : : |
1.5e+07 ++ : : : : : : : : : :: : : : |
1e+07 ++ : : : : :: : : : :: : : : |
| : : : : : : : : : : : : : |
5e+06 ++ : : :: * : : : : : : : |
| *.: : : : : : * : |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
cpuidle.C6-IVT-4S.time
1.2e+11 ++----------------------------------------------------------------+
| |
1e+11 ++ * * |
| : : |
| : : * |
8e+10 ++ : : :: : |
| : : :: :: |
6e+10 ++ : : * : : : : |
| : : : : : : : |
4e+10 ++ : : : : : : : : |
O O OO O: :O O O O O O O O: O O O O O O OO |
| : : : : : : : |
2e+10 *+*.*.*.** * :.*.*.* *.* *.* * *.*.* *.*.**.*.*.*.*
| O * O + + + : O + + |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
turbostat.Avg_MHz
1800 ++-------------------------------------------------*-----*-----------+
|.*. .*.*.* * * * .* .* * *. +: + *.*. .*.*.|
1600 *+ * : : : : * : * : : : * : * * *
1400 ++ : : : : : : : : : : : : |
| : : : : :: : : : : :: : : : |
1200 ++ : : : : : : : : : : : : : : : : |
1000 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
800 ++ : : : : : : : : : : : : : : : : |
600 ++ : : : : : : : : : : : : : : : |
| : : : : : :: : : : : :: :: |
400 ++ * : : : : : : : : : : |
200 ++ : : : : : : : : : : |
O O O O O *.O O O * O O :O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-----------------------+
turbostat.%Busy
60 ++---------------------------------------------------------------------+
| .*.*.* * * .*. |
50 *+*.* : * : * .* .* * *. +: * *.*. .*.*.|
| : : : : * : * : : : * : : * *
| : : : : : : : : : : : : |
40 ++ : : : : : :: : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
30 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
20 ++ : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : :: |
| * : : :: : : : : : : : |
10 ++ : : : : : : : : : : |
O O O O O *.O O O * O O : O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O----O-*-O---O-----------------------+
turbostat.CPU%c1
50 *+-------------------------------------------------------*-------------+
45 ++* *.*.* * * * .* .* * *. : *.*.*. .*.*.|
| + + : : : : * : * : : : *.* : * *
40 ++ * : : : : : : : : : : : : |
35 ++ : : : : :: : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
30 ++ : : : : : : : : : : : : : : : : |
25 ++ : : : : : : : : : : : : : : : : |
20 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : O : : : : : O |
15 O+ O O O : O : O: :O:O :: : : : : :: O O |
10 ++ * : : : : O O : O : O : O O O * O O |
| : : : : : : : : : |
5 ++ *.* : * : : O : : O |
0 ++O---O-----O---O---O-*---O---O-*-O------*-O---------------------------+
turbostat.CPU%c6
100 ++O---O---------O---O-*-----------------------O-----------------------+
90 ++ .* : * O * |
O O O O O * O O O : O O O O : O O O * O O |
80 ++ * : : : : O : O O O |
70 ++ : : : : : :: : : |
| : : : : : : : : : :: |
60 ++ : : : : : : : : : : : : |
50 ++ : : : : : : : : : : : : |
40 ++ : : : : : : : : : : : : |
| : : : : : : : : : : : : |
30 ++ : : : : : : : : : : : : |
20 ++ : : : : : : : : : : |
| : : : : : : : : : |
10 ++ .*. : : : : *. .*. : :.*.: : .*. .*
0 *+*---*-*-*-O-*-----*---*-O---O-*-O---*-*-O-*---*---*---*-*-*-*---*-*-+
turbostat.CoreTmp
60 ++---------------------------------------------------------------------+
*.*.*.*.*.* * * * .* .* *.*.* .*.*. |
50 ++ : : : : * : * : * : : *.* *.*.*.*
O O O : :O: O: :O: : : : O : : :O : : |
| O: : : : : :O: :O : O : : O O: : O O:O O O O |
40 ++ : : : : : : : : : :: : * |
| O O * O.* O * : : : O: : O O |
30 ++ * * : : : : * |
| : : : : |
20 ++ : : : : |
| : : : : |
| : : : : |
10 ++ : : : : |
| : : : : |
0 ++----------O-------------O---O-*-O------*-O---------------------------+
turbostat.PkgTmp
60 ++---------------------------------------------------------------------+
*.*.*.*.*.* * * * .* .* *.*.* .*.*. .*
50 ++ : : : : * : * : * : : * *.*.*.* |
| O O : : : O: :O: : : : O : : :O : :O |
O O: :O: : : :O: :O : O : : O O: : O O: O O O |
40 ++ : : : : : : : : : :: : * |
| O O * O.* O * : : : O: : O O |
30 ++ * * : : : : * |
| : : : : |
20 ++ : : : : |
| : : : : |
| : : : : |
10 ++ : : : : |
| : : : : |
0 ++----------O-------------O---O-*-O------*-O---------------------------+
turbostat.Pkg%pc2
70 ++---------------------------------------------------------------------+
| O |
60 ++ O O O O |
O O O O O O O |
50 ++ O O O O O O O |
| O |
40 ++ O |
| O O O O * O |
30 ++ .* : O * |
| * * : : : * * : |
20 ++ : : : : : :: : : : |
| : : : : : : : : : : : : |
10 ++ : : : : : : : : : : : : |
| .*. : : : : *. .*. : :.*.: : .*. |
0 *+*---*-*-*-O-*-----*---*-O---O-*-O----*-*-O-*---*---*---*-*-*-*---*-*-*
turbostat.PkgWatt
450 *+*---*-*-*---------*---*-----*-------*-------------------*-------*---+
| * : * : : * : * : * *.*.* * *.*.* *.*
400 ++ : : : : : : : : : : : : |
350 ++ : : :: :: : : : : : : : : |
| : : : : : : : : : : : :: : : : |
300 ++ : : : : : : : : : : : : : : : : |
250 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
200 ++ : : : : : : : : : : : : : : |
150 O+ O O O : O : O: O:O ::O :O :O :O :O: :O O O O O O O O |
| * : : : : : : : : : * |
100 ++ *.: : : : : O : : : |
50 ++O O O * O * * : : : : O |
| : : : : |
0 ++----------O-------------O---O-*-O-----*-O---------------------------+
turbostat.CorWatt
400 ++--------------------------------------------------------------------+
*.*. .*.*.* * * * .* .* * *. .* *.*.*.*. .*.*.|
350 ++ * : : : : * : * : : : * : : * *
300 ++ : : : : : : : : : : : : |
| : : : :: :: : : : : :: : : : |
250 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
200 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
150 ++ : : : : : : : : : : : : : : |
100 O+ O O : O : O: O: : : :O : :O : O O O |
| O * : : : O : O O : O : : O O O * O O |
50 ++ *.: : * : : O : : * |
| O O O * O * : : : : O |
0 ++----------O-------------O---O-*-O-----*-O---------------------------+
netperf.Throughput_tps
1600 ++-------------------------------------------------------------------+
*.*.*.*.*.* * * * *.* *.* *.*.* *.*.*. .*.*. .*
1400 ++ : : : : : : : : * : : : * * |
1200 ++ : : : : : : : : : : : : |
| : : : : : : : : : : : : |
1000 O+ O O O: :O: O: :O:O: :O :O O :O O: :O O O:O:O O O O |
| : : : : : : : : : : : : : : : : |
800 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
600 ++ : : : : : : : : : : : : : : : : |
400 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : |
200 ++ : : : : : : : : : : : |
| : : : : : : : : : : : |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.user_time
1000 ++*-*----------------------------------------------------------------+
900 ++ : * *.*.*. .*.*.|
* : .*.*.* *. *. : : *.* *
800 ++ :.* : * * : * : * : : |
700 ++ * : : : : : : : :: : |
| : : : : : : : : : *.*.* : |
600 ++ : : : : : : : : : : : : : : |
500 ++ : : : : : : : : : : : : : : |
400 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : |
300 ++ : : : : : : : : : :: : : |
200 ++ : : : : : : : : : : :: |
| * : : * : : : : * : |
100 O+ O O O O +O O O O O :O O : O O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.system_time
12000 ++------------------------------------------------------------------+
| .*. * .* |
10000 ++*.* *.* * : * *.* *.* *.* : .*.*.*.*.*.*.*
* + +: : : : : : : * : : * |
| * : : : : : : : : : : : |
8000 ++ : : : :: : : : : : : : : |
| : : : : : : : : : : : : : : |
6000 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : :: |
4000 ++ : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
| :: :: * : : : : : :: |
2000 ++ *: : : : : : * : |
O O O O O O O O O O : O O : O O O OO O O O O |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
netperf.time.percent_of_cpu_this_job_got
4000 ++-------------------------------------------------*-----------------+
|.*. .*.*.* * * * .* .* *. +: .*.*.*. .*.*.|
3500 *+ * : : : : * : * : * : * : * * *
3000 ++ : : : : : : : : : : : : |
| : : : : :: : : : : : : : : |
2500 ++ : : : : : : : : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
2000 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
1500 ++ : : : : : : : : : : : : : : : |
1000 ++ : : : : : :: : : : : : : : : |
| * : : : : : : : : : :: |
500 ++ : : : : : : : : : : |
O O O O O *.O O O * O O :O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
netperf.time.voluntary_context_switches
7e+07 *+*-*-*-*-*---*----*---*---*-*-----*-*---------*-*-*--*-*-*-*-*-*-*-*
| : :: : : : : : : * : : : |
6e+07 ++ : :: : : : : : : : : : : |
| * : : :: : : : : : : : : |
5e+07 O+ O O O O: O: :O:O: :O :O :O :O O: :O O OO: O O O |
| : : : : : : : : : : : : : :O |
4e+07 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : :: |
3e+07 ++ : : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
2e+07 ++ : : : : : : : : : : :: |
| :: : * : : : : * : |
1e+07 ++ *: : : : : : : |
| : : : : : : : |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
netperf.time.involuntary_context_switches
35000 ++------------------------------------------------------------------+
| *. |
30000 ++ * + * .* |
| + : * * * : * : * *
25000 ++ * : .* * : : *.* *.* : : : : + + +|
| : * + +: : :: : : : : * : : : * *.* |
20000 ++ : * : : : : : : : : : : : : : |
*.* : : : : : : : : : :: : : * |
15000 ++ : : : : : : : : : : : : : : |
| : : : : :: : : : : : : :: |
10000 ++ : : : : : : : : : :: :: |
| : : : : * : : : : : :: |
5000 ++ *: : : : : : * ::O |
O O O O O O O O O O : O O : O O O OO O O O |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.user_time
1000 ++*-*----------------------------------------------------------------+
900 ++ : * *.*.*. .*.*.|
* : .*.*.* *. *. : : *.* *
800 ++ :.* : * * : * : * : : |
700 ++ * : : : : : : : :: : |
| : : : : : : : : : *.*.* : |
600 ++ : : : : : : : : : : : : : : |
500 ++ : : : : : : : : : : : : : : |
400 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : |
300 ++ : : : : : : : : : :: : : |
200 ++ : : : : : : : : : : :: |
| * : : * : : : : * : |
100 O+ O O O O +O O O O O :O O : O O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
time.system_time
12000 ++------------------------------------------------------------------+
| .*. * .* |
10000 ++*.* *.* * : * *.* *.* *.* : .*.*.*.*.*.*.*
* + +: : : : : : : * : : * |
| * : : : : : : : : : : : |
8000 ++ : : : :: : : : : : : : : |
| : : : : : : : : : : : : : : |
6000 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : :: |
4000 ++ : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
| :: :: * : : : : : :: |
2000 ++ *: : : : : : * : |
O O O O O O O O O O : O O : O O O OO O O O O |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.percent_of_cpu_this_job_got
4000 ++-------------------------------------------------*-----------------+
|.*. .*.*.* * * * .* .* *. +: .*.*.*. .*.*.|
3500 *+ * : : : : * : * : * : * : * * *
3000 ++ : : : : : : : : : : : : |
| : : : : :: : : : : : : : : |
2500 ++ : : : : : : : : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
2000 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
1500 ++ : : : : : : : : : : : : : : : |
1000 ++ : : : : : :: : : : : : : : : |
| * : : : : : : : : : :: |
500 ++ : : : : : : : : : : |
O O O O O *.O O O * O O :O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+
time.voluntary_context_switches
7e+07 *+*-*-*-*-*---*----*---*---*-*-----*-*---------*-*-*--*-*-*-*-*-*-*-*
| : :: : : : : : : * : : : |
6e+07 ++ : :: : : : : : : : : : : |
| * : : :: : : : : : : : : |
5e+07 O+ O O O O: O: :O:O: :O :O :O :O O: :O O OO: O O O |
| : : : : : : : : : : : : : :O |
4e+07 ++ : : : : : : : : : : : : : : |
| : : : : : : : : : : : : :: |
3e+07 ++ : : : : : : : : : : : : :: |
| : : : : : : : : : : :: |
2e+07 ++ : : : : : : : : : : :: |
| :: : * : : : : * : |
1e+07 ++ *: : : : : : : |
| : : : : : : : |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
time.involuntary_context_switches
35000 ++------------------------------------------------------------------+
| *. |
30000 ++ * + * .* |
| + : * * * : * : * *
25000 ++ * : .* * : : *.* *.* : : : : + + +|
| : * + +: : :: : : : : * : : : * *.* |
20000 ++ : * : : : : : : : : : : : : : |
*.* : : : : : : : : : :: : : * |
15000 ++ : : : : : : : : : : : : : : |
| : : : : :: : : : : : : :: |
10000 ++ : : : : : : : : : :: :: |
| : : : : * : : : : : :: |
5000 ++ *: : : : : : * ::O |
O O O O O O O O O O : O O : O O O OO O O O |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
vmstat.procs.r
50 ++----*---*------------------------------------------------------------+
| + * : * * * * * *. .* .*.*.* *. |
*.*.* : : : : +: +: * : * : * + + *.*
40 ++ : : : : * : * : : : : : * |
| : : : : :: : : : : : : : : |
| : : : : : : : : : : : :: : : : |
30 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : : |
20 ++ : : : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : |
| : : : : : :: : : : : : : :: |
10 ++ * : : : : : : : : : : |
| : : : : : : : : : : |
O O O O O *.O O O * O O : O O : O * O O O O O O O O |
0 ++O---O-----O---O-*-O-*---O---O-*-O----O-*-O---O-----------------------+
vmstat.system.in
500000 ++-----------------------------------------------------------------+
450000 *+*. .*.*.* * * * .* .* * *. .* *.**.*. .*.*.|
| * : : : : * : * : : : * : : * *
400000 ++ : : : : : : : : : : : : |
350000 ++ : : : : :: : : : : :: : : : |
| : : : : : : : : : : : : : : : : |
300000 ++ :: : : : : : : : : : : : : : : |
250000 ++ :: : : : : : : : : : : : : : : |
200000 ++ :: : : : : : : : : : : : : : : |
| : : : : : : : : : : : : : : : |
150000 ++ : : : : : :: : : : : :: : : |
100000 ++ * : : : : : : : : : : |
| : : : : : : : : : : |
50000 O+ O O O O *.O O O * O O :O O : O * O O O O O OO O |
0 ++O---O----O---O-*-O-*---O---O-*-O--O-*-O---O-------*--------------+
numa-numastat.node0.numa_hit
1.6e+06 ++----------------------------------------------------------------+
| O O O O O |
1.4e+06 O+ O O O O O O O O O O O O OO |
| O |
1.2e+06 ++ |
1e+06 ++ |
| |
800000 ++ |
| * * |
600000 ++ :: : |
400000 ++ : : : : |
| .*. : :.* : *. *. .*. .*.*. |
200000 *+*.* ** * : *. .* * : * * *.*.* *.*.** *.*
| : + * + : + : + + |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
numa-numastat.node0.local_node
1.6e+06 ++----------------------------------------------------------------+
| O O O O O |
1.4e+06 O+ O O O O O O O O O O O O OO |
| O |
1.2e+06 ++ |
1e+06 ++ |
| |
800000 ++ |
| * * |
600000 ++ :: : |
400000 ++ : : : : |
| .*. : :.* : *. *. .*. .*.*. |
200000 *+*.* ** * : *. .* * : * * *.*.* *.*.** *.*
| : + * + : + : + + |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
numa-vmstat.node0.numa_hit
1e+06 ++-----------------------------------------------------------------+
900000 ++ O |
| O O O O OO |
800000 O+ O O O O O O O O O O O O |
700000 ++ O |
| |
600000 ++ |
500000 ++ * * |
400000 ++ :: : : |
| .*. : : : : .*. .*. |
300000 *+*.* *.* *.*. .*. .* *.* *.* * *.*.* *.**.*.* *.*
200000 ++ O * O * : : : : + + |
| O O : : O: : O * |
100000 ++ : : : : |
0 ++---------O-------------O---O-*-O----*-O--------------------------+
numa-vmstat.node0.numa_local
900000 ++------------------------------------------------O----------------+
| O O O O OO |
800000 O+ O O O O O O O O O |
700000 ++ O O O O |
| |
600000 ++ |
500000 ++ |
| * * |
400000 ++ :: : : |
300000 ++ : : : : .* .*. |
|.*.*.*.*.* *.* *. .* *.* *.* * + .*.* *.**.*.* *.*
200000 *+ + + * : : : : * + + |
100000 ++O O O * O : : O: : O * |
| : : : : |
0 ++---------O-------------O---O-*-O----*-O--------------------------+
proc-vmstat.numa_hit
3e+06 ++----------------------------------------------------------------+
| * |
2.5e+06 ++ : |
| : : |
O O OO: :O O O O *O O O O O O O O O OO |
2e+06 ++ : : :: O O |
| : : : : *. |
1.5e+06 ++ .*.*.** *. * * *.* *.* + *.*.* .*.* .*.*. .*
*.* * : : : : : * : * * * |
1e+06 ++ : : : : : : : : : : |
| : : : : : : : : : : |
| : : : : : : : : : |
500000 ++ : * : : : : : : |
| O * O : : : : O : |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
proc-vmstat.numa_local
3e+06 ++----------------------------------------------------------------+
| * |
2.5e+06 ++ : |
| : : |
O O OO: :O O O O *O O O O O O O O O OO |
2e+06 ++ : : :: O O |
| : : : : *. |
1.5e+06 ++ .*.*.** *. * * *.* *.* + *.*.* .*.* .*.*. .*
*.* * : : : : : * : * * * |
1e+06 ++ : : : : : : : : : : |
| : : : : : : : : : : |
| : : : : : : : : : |
500000 ++ : * : : : : : : |
| O * O : : : : O : |
0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+
proc-vmstat.pgalloc_dma32
350000 ++-----------------------------------------------------------------+
O O O O O O O O O O O O |
300000 ++ O O O O O O O O |
| O |
250000 ++ |
| |
200000 ++ * * |
*. + + + *.* * * .* .* * *.*.* .**. .*.*
150000 ++*.* * : : :: * : * : : : : * *.*.* |
| : : : : : + : : : : : : : : |
100000 ++ : : : : * : : : : : : : : |
| : : : : : : : : : : : |
50000 ++ * : : : : : : : * : : |
| +: : : : : : : |
0 ++O---O----O---O-*-O-*---O---O-*-O--O-*-O---O-------*--------------+
proc-vmstat.pgalloc_normal
1.2e+07 ++----------------------------------------------------------------+
O O O O O O O O O O O O O O O O |
1e+07 ++ O O O O O |
| |
| |
8e+06 ++ |
| .*. |
6e+06 *+*.*.*.** * * * *.* *.* *.*.* *.*.**.*.*.*.*
| : : :: : : : : * : : : |
4e+06 ++ : : : : :: : : : :: : : : |
| : : : : * : : : :: : : : |
| : : : : : : : : * : : |
2e+06 ++ * : : : : : : : : : |
| +: : : : : : : |
0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+
proc-vmstat.pgfree
1.2e+07 ++------O------------O----------------O--O-----O-O-----O-O--------+
O O O O O O O O O O O O |
1e+07 ++ O |
| |
| |
8e+06 ++ |
| .*. .*. |
6e+06 *+*.* ** * * * *.* *.* * *.*.* *.*.**.*.*.*.*
| : : :: : : : : :: : : : |
4e+06 ++ : : : : :: : : : :: : : : |
| : : : : * : : : : :: : : |
| * : : : : : : : * : : |
2e+06 ++ : : : : : : : : : |
| :: * : : : : : |
0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+
proc-vmstat.numa_hint_faults
70000 ++------------------------------------------------------------------+
| |
60000 ++ * * * |
| +: +: .*.* :+ .*. |
50000 ++ .* *. * : * : * : : *.* *.*.*
|.* + + *.*. * * : : : : : : : |
40000 *+ * * : :: : : : : * : : * |
| : : :: : : : : :: : : : |
30000 ++ : : : : : : : : :: : : : |
| : : : : * : : : : : ::O |
20000 ++ O O O *O: :O:O : :O : : * O O OO: O O |
O O : : : : O : O :O :: O |
10000 ++ : : O : : : : :: |
| : : : : : : : |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
proc-vmstat.numa_hint_faults_local
60000 ++------------------------------------------------------------------+
| * * * |
50000 ++ +: +: :: |
| .* * : * : .*.* : : .*. |
| * + .*. * * : : : : * : : *.* *.*.*
40000 ++ * *.*.* : : : : : : : : : |
* : : :: : : : : * : : * |
30000 ++ : : : : : : : : : : : : : |
| : : : : : : : : : :: : : : |
20000 ++ : : : : : : : : : : : :: |
| O O O *O: :O:O * : :O : : : O O:O O O |
O O : : : : O : : O :O * O O:: O |
10000 ++ : : O : : : :: |
| : : : : : : : |
0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+
[*] bisect-good sample
[O] bisect-bad sample
To reproduce:
apt-get install ruby
git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
cd lkp-tests
bin/setup-local job.yaml # the job file attached in this email
bin/run-local job.yaml
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
Thanks,
Ying Huang
[-- Attachment #2: job.yaml --]
[-- Type: text/plain, Size: 3379 bytes --]
---
LKP_SERVER: inn
LKP_CGI_PORT: 80
testcase: netperf
default-monitors:
wait: pre-test
uptime:
iostat:
vmstat:
numa-numastat:
numa-vmstat:
numa-meminfo:
proc-vmstat:
proc-stat:
interval: 10
meminfo:
slabinfo:
interrupts:
lock_stat:
latency_stats:
softirqs:
bdi_dev_mapping:
diskstats:
nfsstat:
cpuidle:
cpufreq-stats:
turbostat:
pmeter:
sched_debug:
interval: 60
default-watchdogs:
watch-oom:
watchdog:
cpufreq_governor: performance
model: Westmere-EX
nr_cpu: 80
memory: 128G
hdd_partitions:
swap_partitions:
pmeter_server: lkp-st01
pmeter_device: yokogawa-wt310
commit: 0eab523874ab9850ec14f68751758dfb275361b9
nr_hdd_partitions: 0
rootfs_partition:
runtime: 300s
nr_threads: 200%
perf-profile:
freq: 800
cluster: cs-lkp-wsx02
if role server:
netserver:
if role client:
netperf:
test: TCP_RR
testbox: brickland3
tbox_group: brickland3
kconfig: x86_64-rhel
enqueue_time: 2015-06-13 01:32:11.981026644 +08:00
user: lkp
queue: cs
compiler: gcc-4.9
head_commit: 54896f27dd5c7e83c37c85eb1093f14543af2bfa
base_commit: d4a4f75cd8f29cd9464a5a32e9224a91571d6649
branch: next/master
rootfs: debian-x86_64-2015-02-07.cgz
result_root: "/result/netperf/performance-300s-200%-TCP_RR/brickland3/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/0"
direct_macs: a0:36:9f:18:d6:aa
direct_ips: 10.167.0.2
job_file: "/lkp/scheduled/brickland3/cs_netperf-performance-300s-200%-TCP_RR-debian-x86_64-2015-02-07.cgz-x86_64-rhel-0eab523874ab9850ec14f68751758dfb275361b9-0-20150613-5146-1kuhia6.yaml"
max_uptime: 1500
initrd: "/osimage/debian/debian-x86_64-2015-02-07.cgz"
bootloader_append:
- root=/dev/ram0
- user=lkp
- job=/lkp/scheduled/brickland3/cs_netperf-performance-300s-200%-TCP_RR-debian-x86_64-2015-02-07.cgz-x86_64-rhel-0eab523874ab9850ec14f68751758dfb275361b9-0-20150613-5146-1kuhia6.yaml
- ARCH=x86_64
- kconfig=x86_64-rhel
- branch=next/master
- commit=0eab523874ab9850ec14f68751758dfb275361b9
- BOOT_IMAGE=/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/vmlinuz-4.1.0-rc7-07770-g0eab523
- max_uptime=1500
- RESULT_ROOT=/result/netperf/performance-300s-200%-TCP_RR/brickland3/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/0
- LKP_SERVER=inn
- |2-
earlyprintk=ttyS0,115200 systemd.log_level=err
debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100
panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0
console=ttyS0,115200 console=tty0 vga=normal
rw
lkp_initrd: "/lkp/lkp/lkp-x86_64.cgz"
modules_initrd: "/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/modules.cgz"
bm_initrd: "/osimage/deps/debian-x86_64-2015-02-07.cgz/lkp.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/run-ipconfig.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/turbostat.cgz,/lkp/benchmarks/turbostat.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/netperf.cgz,/lkp/benchmarks/netperf.cgz"
node_roles: client
kernel: "/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/vmlinuz-4.1.0-rc7-07770-g0eab523"
dequeue_time: 2015-06-13 01:32:30.522650807 +08:00
job_state: finished
loadavg: 2.33 6.80 4.32 1/1045 12657
start_time: '1434130591'
end_time: '1434130922'
version: "/lkp/lkp/.src-20150612-214517"
[-- Attachment #3: reproduce.ksh --]
[-- Type: text/plain, Size: 9476 bytes --]
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu10/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu100/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu101/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu102/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu103/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu104/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu105/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu106/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu107/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu108/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu109/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu11/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu110/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu111/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu112/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu113/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu114/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu115/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu116/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu117/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu118/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu119/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu12/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu13/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu14/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu15/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu16/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu17/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu18/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu19/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu20/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu21/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu22/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu23/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu24/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu25/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu26/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu27/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu28/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu29/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu30/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu31/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu32/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu33/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu34/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu35/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu36/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu37/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu38/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu39/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu40/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu41/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu42/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu43/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu44/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu45/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu46/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu47/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu48/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu49/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu50/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu51/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu52/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu53/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu54/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu55/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu56/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu57/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu58/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu59/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu60/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu61/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu62/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu63/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu64/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu65/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu66/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu67/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu68/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu69/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu70/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu71/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu72/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu73/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu74/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu75/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu76/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu77/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu78/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu79/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu8/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu80/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu81/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu82/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu83/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu84/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu85/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu86/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu87/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu88/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu89/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu9/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu90/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu91/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu92/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu93/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu94/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu95/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu96/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu97/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu98/cpufreq/scaling_governor
echo performance > /sys/devices/system/cpu/cpu99/cpufreq/scaling_governor
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1
^ permalink raw reply [flat|nested] 12+ messages in thread* [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps @ 2015-06-15 7:02 ` Huang Ying 0 siblings, 0 replies; 12+ messages in thread From: Huang Ying @ 2015-06-15 7:02 UTC (permalink / raw) To: Jiang Liu; +Cc: Thomas Gleixner, LKML, LKP ML [-- Attachment #1: Type: text/plain, Size: 144987 bytes --] FYI, we noticed the below changes on git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") testcase/path_params/tbox_group: netperf/performance-300s-200%-TCP_RR/brickland3 3cb96f0c97330834 52f518a3a7c2f80551a38d38be ---------------- -------------------------- %stddev %change %stddev \ | \ 1444 ± 1% -30.5% 1004 ± 1% netperf.Throughput_tps 899.43 ± 3% -91.6% 75.48 ± 3% netperf.time.user_time 10202 ± 0% -92.4% 778 ± 1% netperf.time.system_time 3565 ± 4% -92.5% 268 ± 3% netperf.time.percent_of_cpu_this_job_got 98265 ± 3% -31.1% 67662 ± 4% netperf.time.minor_page_faults 25540 ± 11% -88.3% 2991 ± 16% netperf.time.involuntary_context_switches 69135723 ± 1% -30.0% 48364754 ± 1% netperf.time.voluntary_context_switches 37347 ± 4% +13.1% 42256 ± 3% uptime.idle 44 ± 5% -93.2% 3 ± 0% vmstat.procs.r 407528 ± 3% -26.3% 300511 ± 4% vmstat.system.cs 446620 ± 4% -92.6% 33083 ± 6% vmstat.system.in 178635 ± 1% -14.6% 152519 ± 0% meminfo.Active 82997 ± 2% -29.5% 58518 ± 1% meminfo.Active(anon) 36056 ± 3% -63.2% 13281 ± 0% meminfo.Shmem 496435 ± 3% +19.4% 592640 ± 2% numa-meminfo.node0.MemUsed 480933 ± 3% -7.3% 445927 ± 2% numa-meminfo.node2.MemUsed 3997 ± 1% +12.1% 4482 ± 3% slabinfo.kmalloc-2048.active_objs 4183 ± 1% +12.2% 4694 ± 3% slabinfo.kmalloc-2048.num_objs 313 ± 15% -78.0% 69 ± 30% slabinfo.tw_sock_TCP.active_objs 313 ± 15% -78.0% 69 ± 30% slabinfo.tw_sock_TCP.num_objs 1016370 ± 1% -90.2% 99283 ± 1% softirqs.HRTIMER 1556902 ± 0% -84.0% 248959 ± 2% softirqs.RCU 3471142 ± 1% -92.3% 267496 ± 3% softirqs.SCHED 4270170 ± 1% -82.2% 758706 ± 2% softirqs.TIMER 2.347e+09 ± 6% -63.5% 8.569e+08 ± 39% cpuidle.C1-IVT-4S.time 22319046 ± 3% +62.6% 36299214 ± 3% cpuidle.C1-IVT-4S.usage 1.591e+10 ± 9% +128.3% 3.632e+10 ± 3% cpuidle.C6-IVT-4S.time 7 ± 21% +514.0% 44 ± 31% cpuidle.POLL.usage 25540 ± 11% -88.3% 2991 ± 16% time.involuntary_context_switches 98265 ± 3% -31.1% 67662 ± 4% time.minor_page_faults 3565 ± 4% -92.5% 268 ± 3% time.percent_of_cpu_this_job_got 10202 ± 0% -92.4% 778 ± 1% time.system_time 899.43 ± 3% -91.6% 75.48 ± 3% time.user_time 69135723 ± 1% -30.0% 48364754 ± 1% time.voluntary_context_switches 268764 ± 14% +408.8% 1367464 ± 3% numa-numastat.node0.numa_hit 268758 ± 14% +408.8% 1367424 ± 3% numa-numastat.node0.local_node 438574 ± 26% -45.0% 241074 ± 2% numa-numastat.node1.local_node 438574 ± 26% -45.0% 241076 ± 2% numa-numastat.node1.numa_hit 333989 ± 23% -27.9% 240841 ± 3% numa-numastat.node2.local_node 333989 ± 23% -27.9% 240844 ± 3% numa-numastat.node2.numa_hit 381233 ± 26% -35.6% 245556 ± 3% numa-numastat.node3.local_node 381238 ± 26% -35.6% 245561 ± 3% numa-numastat.node3.numa_hit 482 ± 4% +11.9% 540 ± 1% numa-vmstat.node0.nr_alloc_batch 251510 ± 9% +212.0% 784815 ± 8% numa-vmstat.node0.numa_local 289771 ± 8% +184.0% 823025 ± 7% numa-vmstat.node0.numa_hit 454081 ± 12% -25.4% 338719 ± 4% numa-vmstat.node1.numa_hit 521 ± 1% +13.5% 592 ± 3% numa-vmstat.node1.nr_alloc_batch 395201 ± 14% -28.9% 281018 ± 4% numa-vmstat.node1.numa_local 529 ± 3% +10.1% 583 ± 1% numa-vmstat.node2.nr_alloc_batch 525 ± 6% +12.8% 593 ± 1% numa-vmstat.node3.nr_alloc_batch 51.25 ± 3% -92.9% 3.65 ± 3% turbostat.%Busy 1635 ± 3% -93.0% 114 ± 3% turbostat.Avg_MHz 45.20 ± 4% -70.1% 13.52 ± 21% turbostat.CPU%c1 357.80 ± 3% -71.5% 101.83 ± 6% turbostat.CorWatt 53 ± 2% -21.2% 42 ± 6% turbostat.CoreTmp 53 ± 1% -20.7% 42 ± 6% turbostat.PkgTmp 424 ± 2% -61.0% 165 ± 3% turbostat.PkgWatt 62.78 ± 1% -8.9% 57.17 ± 3% turbostat.RAMWatt 20746 ± 2% -29.5% 14628 ± 1% proc-vmstat.nr_active_anon 2024 ± 1% +13.4% 2295 ± 0% proc-vmstat.nr_alloc_batch 9013 ± 3% -63.2% 3319 ± 0% proc-vmstat.nr_shmem 49576 ± 5% -60.3% 19667 ± 13% proc-vmstat.numa_hint_faults 44013 ± 5% -64.6% 15598 ± 16% proc-vmstat.numa_hint_faults_local 1420966 ± 2% +47.3% 2092933 ± 1% proc-vmstat.numa_hit 1420955 ± 2% +47.3% 2092883 ± 1% proc-vmstat.numa_local 53703 ± 5% -57.9% 22601 ± 13% proc-vmstat.numa_pte_updates 8470 ± 1% -81.8% 1543 ± 3% proc-vmstat.pgactivate 162499 ± 5% +90.3% 309245 ± 5% proc-vmstat.pgalloc_dma32 5918865 ± 2% +85.7% 10992818 ± 3% proc-vmstat.pgalloc_normal 6085763 ± 2% +86.0% 11317294 ± 3% proc-vmstat.pgfree 2579 ± 5% -79.4% 532 ± 2% latency_stats.avg.do_wait.SyS_wait4.system_call_fastpath 858 ± 5% +60.1% 1375 ± 4% latency_stats.avg.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath 2337 ± 4% -79.5% 479 ± 2% latency_stats.avg.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath 875 ± 7% +74.6% 1528 ± 10% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat 455 ± 1% +108.8% 951 ± 1% latency_stats.avg.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath 541 ± 14% +137.6% 1287 ± 4% latency_stats.avg.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_revalidate_dentry_attr.[cifs].cifs_revalidate_dentry.[cifs].cifs_d_revalidate.[cifs].lookup_fast.do_last.path_openat.do_filp_open.do_sys_open 655 ± 20% +96.9% 1291 ± 9% latency_stats.avg.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath 69119790 ± 1% -30.0% 48351473 ± 1% latency_stats.hits.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath 443 ± 3% -10.0% 399 ± 2% latency_stats.hits.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath 190 ± 6% -95.8% 8 ± 38% latency_stats.hits.stop_one_cpu.sched_exec.do_execveat_common.SyS_execve.return_from_execve 2157 ± 23% +66.0% 3581 ± 22% latency_stats.max.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath 4912 ± 0% +1.6% 4992 ± 0% latency_stats.max.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath 3.151e+10 ± 0% +46.1% 4.603e+10 ± 0% latency_stats.sum.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.SyS_recvfrom.system_call_fastpath 16780629 ± 2% -78.9% 3539898 ± 4% latency_stats.sum.do_wait.SyS_wait4.system_call_fastpath 25402986 ± 2% -78.5% 5473439 ± 4% latency_stats.sum.pipe_wait.pipe_read.__vfs_read.vfs_read.SyS_read.system_call_fastpath 12072 ± 31% +112.0% 25588 ± 10% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSSMBUnixQPathInfo.[cifs].cifs_get_inode_info_unix.[cifs].cifs_revalidate_dentry_attr.[cifs].cifs_revalidate_dentry.[cifs].cifs_d_revalidate.[cifs].lookup_fast.do_last.path_openat.do_filp_open.do_sys_open 290547 ± 19% +77.5% 515581 ± 9% latency_stats.sum.poll_schedule_timeout.do_select.core_sys_select.SyS_select.system_call_fastpath 24407 ± 9% +74.1% 42503 ± 9% latency_stats.sum.wait_for_response.[cifs].SendReceive.[cifs].CIFSPOSIXCreate.[cifs].cifs_posix_open.[cifs].cifs_open.[cifs].do_dentry_open.vfs_open.do_last.path_openat.do_filp_open.do_sys_open.SyS_openat 245753 ± 23% +44.5% 355113 ± 9% latency_stats.sum.cifs_user_readv.[cifs].cifs_strict_readv.[cifs].__vfs_read.vfs_read.SyS_read.system_call_fastpath 17.85 ± 4% -16.4% 14.92 ± 2% perf-profile.cpu-cycles.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive 0.97 ± 4% -85.4% 0.14 ± 32% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.scheduler_tick.update_process_times.tick_sched_handle 2.19 ± 8% -93.6% 0.14 ± 10% perf-profile.cpu-cycles._raw_spin_lock.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common 3.88 ± 9% -94.1% 0.23 ± 21% perf-profile.cpu-cycles.load_balance.pick_next_task_fair.__schedule.schedule.schedule_timeout 6.29 ± 7% -96.7% 0.21 ± 40% perf-profile.cpu-cycles._raw_spin_lock_irqsave.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task 0.94 ± 5% -90.7% 0.09 ± 13% perf-profile.cpu-cycles.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__schedule.schedule 0.94 ± 5% -90.6% 0.09 ± 13% perf-profile.cpu-cycles.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.__schedule 4.19 ± 2% +61.7% 6.77 ± 3% perf-profile.cpu-cycles.ktime_get.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary 15.10 ± 3% -27.5% 10.94 ± 1% perf-profile.cpu-cycles.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.system_call_fastpath 15.19 ± 3% -27.2% 11.07 ± 1% perf-profile.cpu-cycles.sock_recvmsg.SYSC_recvfrom.sys_recvfrom.system_call_fastpath 15.35 ± 3% -27.0% 11.20 ± 1% perf-profile.cpu-cycles.SYSC_recvfrom.sys_recvfrom.system_call_fastpath 15.44 ± 3% -27.0% 11.27 ± 1% perf-profile.cpu-cycles.sys_recvfrom.system_call_fastpath 14.72 ± 3% -26.5% 10.82 ± 1% perf-profile.cpu-cycles.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom.sys_recvfrom 1.57 ± 3% +33.4% 2.10 ± 4% perf-profile.cpu-cycles.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg 15.03 ± 4% -25.0% 11.27 ± 2% perf-profile.cpu-cycles.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key 23.41 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ret_from_intr.cpu_startup_entry.start_secondary 1.19 ± 12% -80.5% 0.23 ± 19% perf-profile.cpu-cycles.scheduler_tick.update_process_times.tick_sched_handle.tick_sched_timer.__run_hrtimer 4.24 ± 2% +58.3% 6.72 ± 3% perf-profile.cpu-cycles.read_hpet.ktime_get.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry 15.05 ± 4% -24.9% 11.30 ± 2% perf-profile.cpu-cycles.default_wake_function.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue 2.81 ± 9% -90.6% 0.26 ± 20% perf-profile.cpu-cycles.find_busiest_group.load_balance.pick_next_task_fair.__schedule.schedule 9.12 ± 4% -52.1% 4.36 ± 4% perf-profile.cpu-cycles.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom 0.46 ± 2% +127.4% 1.04 ± 6% perf-profile.cpu-cycles.pick_next_task_fair.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry 2.52 ± 2% -62.9% 0.94 ± 31% perf-profile.cpu-cycles.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.58 ± 8% -81.7% 0.29 ± 21% perf-profile.cpu-cycles.tick_sched_handle.isra.18.tick_sched_timer.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt 2.49 ± 1% -64.0% 0.90 ± 33% perf-profile.cpu-cycles.tick_sched_timer.__run_hrtimer.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt 15.06 ± 4% -24.9% 11.32 ± 2% perf-profile.cpu-cycles.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv 1.35 ± 3% +32.3% 1.79 ± 8% perf-profile.cpu-cycles.deactivate_task.__schedule.schedule.schedule_timeout.sk_wait_data 23.23 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary 5.90 ± 2% -23.8% 4.50 ± 6% perf-profile.cpu-cycles.read_hpet.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle 1.72 ± 8% -83.2% 0.29 ± 21% perf-profile.cpu-cycles.update_process_times.tick_sched_handle.tick_sched_timer.__run_hrtimer.hrtimer_interrupt 1.55 ± 6% +48.4% 2.31 ± 5% perf-profile.cpu-cycles.dev_queue_xmit_sk.ip_finish_output.ip_output.ip_local_out_sk.ip_queue_xmit 1.35 ± 3% +30.7% 1.76 ± 7% perf-profile.cpu-cycles.dequeue_task.deactivate_task.__schedule.schedule.schedule_timeout 15.75 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary 1.54 ± 5% +48.3% 2.28 ± 5% perf-profile.cpu-cycles.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output.ip_output.ip_local_out_sk 3.70 ± 1% +49.0% 5.51 ± 1% perf-profile.cpu-cycles.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg 72.13 ± 0% -24.2% 54.66 ± 0% perf-profile.cpu-cycles.cpu_startup_entry.start_secondary 8.28 ± 5% -61.0% 3.23 ± 5% perf-profile.cpu-cycles.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg 3.44 ± 3% +51.2% 5.19 ± 1% perf-profile.cpu-cycles.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg 2.75 ± 9% -91.1% 0.24 ± 20% perf-profile.cpu-cycles.update_sd_lb_stats.find_busiest_group.load_balance.pick_next_task_fair.__schedule 8.36 ± 5% -61.0% 3.26 ± 5% perf-profile.cpu-cycles.schedule_timeout.sk_wait_data.tcp_recvmsg.inet_recvmsg.sock_recvmsg 0.33 ± 29% +218.5% 1.06 ± 10% perf-profile.cpu-cycles.sk_stream_alloc_skb.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto 15.13 ± 4% -24.2% 11.46 ± 2% perf-profile.cpu-cycles.__wake_up_common.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish 72.31 ± 0% -24.2% 54.83 ± 0% perf-profile.cpu-cycles.start_secondary 14.20 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__do_softirq.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry 1.30 ± 3% +24.4% 1.62 ± 8% perf-profile.cpu-cycles.dequeue_task_fair.dequeue_task.deactivate_task.__schedule.schedule 1.34 ± 5% +30.0% 1.74 ± 4% perf-profile.cpu-cycles.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output.ip_output 8.93 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ret_from_intr.cpuidle_enter.cpu_startup_entry.start_secondary 15.18 ± 1% +26.1% 19.14 ± 3% perf-profile.cpu-cycles.tick_nohz_idle_exit.cpu_startup_entry.start_secondary 4.09 ± 0% +9.8% 4.49 ± 4% perf-profile.cpu-cycles.ktime_get.clockevents_program_event.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns 8.08 ± 5% -62.1% 3.06 ± 6% perf-profile.cpu-cycles.__schedule.schedule.schedule_timeout.sk_wait_data.tcp_recvmsg 3.87 ± 2% +12.2% 4.35 ± 7% perf-profile.cpu-cycles.read_hpet.ktime_get.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry 5.80 ± 1% -12.5% 5.07 ± 5% perf-profile.cpu-cycles.ktime_get_with_offset.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll 0.36 ± 4% +188.3% 1.02 ± 5% perf-profile.cpu-cycles.dev_hard_start_xmit.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk.ip_finish_output 4.72 ± 3% +186.8% 13.52 ± 6% perf-profile.cpu-cycles.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary 10.92 ± 4% -11.1% 9.71 ± 4% perf-profile.cpu-cycles.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate 2.95 ± 3% -18.6% 2.40 ± 3% perf-profile.cpu-cycles.__schedule.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary 3.83 ± 2% +14.0% 4.36 ± 7% perf-profile.cpu-cycles.ktime_get.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary 8.92 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry.start_secondary 7.38 ± 1% +13.6% 8.38 ± 2% perf-profile.cpu-cycles.sock_sendmsg.SYSC_sendto.sys_sendto.system_call_fastpath 11.19 ± 4% -11.1% 9.95 ± 2% perf-profile.cpu-cycles.ttwu_do_activate.constprop.88.try_to_wake_up.default_wake_function.autoremove_wake_function.__wake_up_common 15.76 ± 4% -20.9% 12.46 ± 3% perf-profile.cpu-cycles.__wake_up_sync_key.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver 7.73 ± 2% +14.4% 8.85 ± 2% perf-profile.cpu-cycles.SYSC_sendto.sys_sendto.system_call_fastpath 13.61 ± 1% +12.6% 15.31 ± 3% perf-profile.cpu-cycles.tick_nohz_idle_enter.cpu_startup_entry.start_secondary 2.97 ± 2% +14.2% 3.40 ± 4% perf-profile.cpu-cycles.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames 7.79 ± 2% +14.7% 8.93 ± 2% perf-profile.cpu-cycles.sys_sendto.system_call_fastpath 4.63 ± 1% +24.6% 5.77 ± 1% perf-profile.cpu-cycles.tcp_prequeue_process.tcp_recvmsg.inet_recvmsg.sock_recvmsg.SYSC_recvfrom 2.87 ± 2% +13.8% 3.26 ± 3% perf-profile.cpu-cycles.ip_output.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit 6.59 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry 3.04 ± 3% -17.9% 2.49 ± 3% perf-profile.cpu-cycles.schedule.schedule_preempt_disabled.cpu_startup_entry.start_secondary 11.17 ± 4% -18.6% 9.09 ± 2% perf-profile.cpu-cycles.enqueue_task_fair.enqueue_task.activate_task.ttwu_do_activate.try_to_wake_up 23.39 ± 2% -12.3% 20.51 ± 1% perf-profile.cpu-cycles.system_call_fastpath 7.03 ± 1% +16.0% 8.15 ± 2% perf-profile.cpu-cycles.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto.system_call_fastpath 4.16 ± 0% +14.6% 4.76 ± 4% perf-profile.cpu-cycles.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick 4.15 ± 0% +14.7% 4.76 ± 4% perf-profile.cpu-cycles.clockevents_program_event.tick_program_event.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start 3.05 ± 3% -16.3% 2.55 ± 3% perf-profile.cpu-cycles.schedule_preempt_disabled.cpu_startup_entry.start_secondary 6.33 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__do_softirq.irq_exit.do_IRQ.ret_from_intr.cpuidle_enter 6.78 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr.cpu_startup_entry 6.75 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_enter.do_IRQ.ret_from_intr.cpu_startup_entry.start_secondary 4.73 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter.irq_enter 10.95 ± 5% -13.5% 9.47 ± 2% perf-profile.cpu-cycles.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function.autoremove_wake_function 4.62 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter 4.21 ± 1% +21.2% 5.10 ± 4% perf-profile.cpu-cycles.ktime_get.tick_nohz_idle_exit.cpu_startup_entry.start_secondary 1.29 ± 4% +21.1% 1.57 ± 3% perf-profile.cpu-cycles.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process 9.15 ± 1% +16.1% 10.63 ± 2% perf-profile.cpu-cycles.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary 10.94 ± 5% -13.5% 9.46 ± 2% perf-profile.cpu-cycles.enqueue_task.activate_task.ttwu_do_activate.try_to_wake_up.default_wake_function 0.17 ± 35% +479.8% 0.96 ± 10% perf-profile.cpu-cycles.__alloc_skb.sk_stream_alloc_skb.tcp_sendmsg.inet_sendmsg.sock_sendmsg 4.32 ± 9% -88.6% 0.49 ± 13% perf-profile.cpu-cycles.pick_next_task_fair.__schedule.schedule.schedule_timeout.sk_wait_data 4.54 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_nohz_stop_idle.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr 16.82 ± 4% -15.0% 14.30 ± 3% perf-profile.cpu-cycles.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb 16.81 ± 4% -15.0% 14.28 ± 3% perf-profile.cpu-cycles.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv.__netif_receive_skb_core 4.28 ± 2% -14.4% 3.66 ± 6% perf-profile.cpu-cycles.ktime_get.clockevents_program_event.tick_program_event.__remove_hrtimer.hrtimer_try_to_cancel 24.19 ± 3% -15.8% 20.36 ± 3% perf-profile.cpu-cycles.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action.__do_softirq 23.93 ± 3% -15.8% 20.15 ± 3% perf-profile.cpu-cycles.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action 4.03 ± 1% +15.2% 4.64 ± 3% perf-profile.cpu-cycles.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg 9.82 ± 4% -14.5% 8.40 ± 3% perf-profile.cpu-cycles.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task.activate_task 13.12 ± 1% +15.9% 15.21 ± 3% perf-profile.cpu-cycles.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry.start_secondary 6.71 ± 1% +18.5% 7.95 ± 2% perf-profile.cpu-cycles.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto.sys_sendto 8.57 ± 1% +17.2% 10.05 ± 3% perf-profile.cpu-cycles.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter.cpu_startup_entry 16.71 ± 4% -16.1% 14.02 ± 3% perf-profile.cpu-cycles.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish.ip_rcv 0.14 ± 15% +573.8% 0.94 ± 9% perf-profile.cpu-cycles.tcp_send_delayed_ack.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process 0.10 ± 23% +830.0% 0.93 ± 5% perf-profile.cpu-cycles.ixgbe_xmit_frame.dev_hard_start_xmit.sch_direct_xmit.__dev_queue_xmit.dev_queue_xmit_sk 0.08 ± 6% +1213.0% 1.01 ± 4% perf-profile.cpu-cycles.is_module_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency 4.48 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_irq_enter.irq_enter.do_IRQ 4.57 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr 8.57 ± 1% +16.9% 10.02 ± 3% perf-profile.cpu-cycles.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_idle_enter 17.72 ± 4% -16.3% 14.84 ± 2% perf-profile.cpu-cycles.ip_rcv_finish.ip_rcv.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal 4.57 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.read_hpet.ktime_get.tick_irq_enter.irq_enter.do_IRQ 0.96 ± 4% -85.4% 0.14 ± 35% perf-profile.cpu-cycles.read_hpet.ktime_get.sched_clock_tick.scheduler_tick.update_process_times 18.03 ± 4% -16.6% 15.03 ± 2% perf-profile.cpu-cycles.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq.ixgbe_poll 4.99 ± 1% +17.0% 5.84 ± 2% perf-profile.cpu-cycles.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg.SYSC_sendto 18.00 ± 4% -16.7% 15.00 ± 2% perf-profile.cpu-cycles.__netif_receive_skb_core.__netif_receive_skb.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq 3.16 ± 1% +20.8% 3.82 ± 3% perf-profile.cpu-cycles.ip_queue_xmit.tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_push 4.87 ± 2% +18.7% 5.78 ± 2% perf-profile.cpu-cycles.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg.sock_sendmsg 2.36 ± 1% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.irq_enter.do_IRQ.ret_from_intr.cpuidle_enter.cpu_startup_entry 0.23 ± 6% +443.4% 1.23 ± 3% perf-profile.cpu-cycles.is_module_text_address.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk 0.47 ± 4% +684.2% 3.65 ± 3% perf-profile.cpu-cycles.__kernel_text_address.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency 1.37 ± 4% +239.8% 4.65 ± 7% perf-profile.cpu-cycles.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry.start_secondary 1.34 ± 4% +243.2% 4.61 ± 7% perf-profile.cpu-cycles.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry 0.43 ± 7% +1444.2% 6.64 ± 15% perf-profile.cpu-cycles.intel_idle.cpuidle_enter_state.cpuidle_enter.cpu_startup_entry.start_secondary 0.54 ± 7% +3863.4% 21.27 ± 3% perf-profile.cpu-cycles.ixgbe_clean_rx_irq.ixgbe_poll.net_rx_action.__do_softirq.run_ksoftirqd 0.54 ± 7% +4049.4% 22.41 ± 2% perf-profile.cpu-cycles.ixgbe_poll.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn 0.55 ± 6% +3975.2% 22.48 ± 2% perf-profile.cpu-cycles.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread 1.28 ± 3% +246.7% 4.43 ± 7% perf-profile.cpu-cycles.ktime_get.sched_clock_tick.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit 4.21 ± 0% +18.0% 4.97 ± 3% perf-profile.cpu-cycles.__remove_hrtimer.__hrtimer_start_range_ns.hrtimer_start.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter 5.58 ± 1% -15.9% 4.69 ± 3% perf-profile.cpu-cycles.read_hpet.ktime_get_with_offset.netif_receive_skb_internal.napi_gro_receive.ixgbe_clean_rx_irq 4.77 ± 1% +19.6% 5.70 ± 2% perf-profile.cpu-cycles.tcp_write_xmit.__tcp_push_pending_frames.tcp_push.tcp_sendmsg.inet_sendmsg 1.33 ± 4% +248.1% 4.62 ± 7% perf-profile.cpu-cycles.sched_clock_idle_wakeup_event.tick_nohz_stop_idle.tick_nohz_idle_exit.cpu_startup_entry.start_secondary 1.92 ± 2% +283.1% 7.36 ± 2% perf-profile.cpu-cycles.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair 1.93 ± 2% +285.6% 7.46 ± 2% perf-profile.cpu-cycles.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.enqueue_task 2.20 ± 3% +19.0% 2.61 ± 4% perf-profile.cpu-cycles.ip_finish_output.ip_output.ip_local_out_sk.ip_queue_xmit.tcp_transmit_skb 16.33 ± 4% -17.9% 13.40 ± 3% perf-profile.cpu-cycles.tcp_prequeue.tcp_v4_rcv.ip_local_deliver_finish.ip_local_deliver.ip_rcv_finish 0.99 ± 4% +2179.8% 22.61 ± 2% perf-profile.cpu-cycles.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork 0.97 ± 4% +2217.9% 22.60 ± 2% perf-profile.cpu-cycles.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork 1.13 ± 3% +1906.2% 22.67 ± 2% perf-profile.cpu-cycles.smpboot_thread_fn.kthread.ret_from_fork 1.15 ± 2% +1878.6% 22.82 ± 2% perf-profile.cpu-cycles.kthread.ret_from_fork 1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.x86_64_start_reservations.x86_64_start_kernel 1.15 ± 2% +1879.0% 22.82 ± 2% perf-profile.cpu-cycles.ret_from_fork 2.34 ± 1% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_irq_enter.irq_enter.do_IRQ.ret_from_intr.cpuidle_enter 1.62 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.do_IRQ.ret_from_intr 1.61 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.read_hpet.ktime_get.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit 1.56 ± 4% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.tick_nohz_irq_exit.irq_exit.do_IRQ.ret_from_intr.cpu_startup_entry 1.48 ± 3% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.ktime_get.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.do_IRQ 0.21 ± 12% +351.6% 0.96 ± 8% perf-profile.cpu-cycles.__tcp_ack_snd_check.tcp_rcv_established.tcp_v4_do_rcv.tcp_prequeue_process.tcp_recvmsg 1.72 ± 2% +301.4% 6.90 ± 2% perf-profile.cpu-cycles.print_context_stack.dump_trace.save_stack_trace_tsk.__account_scheduler_latency.enqueue_entity 1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.x86_64_start_kernel 1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel 1.02 ± 2% -100.0% 0.00 ± 0% perf-profile.cpu-cycles.start_kernel.x86_64_start_reservations.x86_64_start_kernel 4.21 ± 1% +20.6% 5.08 ± 4% perf-profile.cpu-cycles.read_hpet.ktime_get.tick_nohz_idle_exit.cpu_startup_entry.start_secondary 528 ± 6% +61.7% 854 ± 1% sched_debug.cfs_rq[0]:/.tg_runnable_contrib 24162 ± 6% +62.3% 39214 ± 1% sched_debug.cfs_rq[0]:/.avg->runnable_avg_sum 4882658 ± 9% -96.6% 167048 ± 8% sched_debug.cfs_rq[0]:/.min_vruntime 37303 ± 1% -89.7% 3826 ± 4% sched_debug.cfs_rq[0]:/.tg->runnable_avg 77353 ± 9% +81.9% 140694 ± 7% sched_debug.cfs_rq[0]:/.exec_clock 38349 ± 2% -89.4% 4070 ± 3% sched_debug.cfs_rq[100]:/.tg->runnable_avg 38365 ± 2% -89.4% 4071 ± 3% sched_debug.cfs_rq[101]:/.tg->runnable_avg 38369 ± 2% -89.4% 4074 ± 3% sched_debug.cfs_rq[102]:/.tg->runnable_avg 38382 ± 2% -89.4% 4075 ± 3% sched_debug.cfs_rq[103]:/.tg->runnable_avg 38383 ± 2% -89.4% 4077 ± 3% sched_debug.cfs_rq[104]:/.tg->runnable_avg 38391 ± 2% -89.4% 4079 ± 3% sched_debug.cfs_rq[105]:/.tg->runnable_avg 38388 ± 2% -89.4% 4083 ± 3% sched_debug.cfs_rq[106]:/.tg->runnable_avg 38402 ± 2% -89.4% 4084 ± 3% sched_debug.cfs_rq[107]:/.tg->runnable_avg 38401 ± 2% -89.4% 4087 ± 3% sched_debug.cfs_rq[108]:/.tg->runnable_avg 38413 ± 2% -89.4% 4088 ± 3% sched_debug.cfs_rq[109]:/.tg->runnable_avg 548 ± 3% -93.4% 36 ± 22% sched_debug.cfs_rq[10]:/.tg_runnable_contrib 37450 ± 1% -89.7% 3851 ± 4% sched_debug.cfs_rq[10]:/.tg->runnable_avg 5772463 ± 8% -98.4% 93465 ± 22% sched_debug.cfs_rq[10]:/.min_vruntime 76594 ± 8% -91.7% 6366 ± 15% sched_debug.cfs_rq[10]:/.exec_clock 25036 ± 3% -93.3% 1688 ± 20% sched_debug.cfs_rq[10]:/.avg->runnable_avg_sum 38410 ± 2% -89.4% 4089 ± 3% sched_debug.cfs_rq[110]:/.tg->runnable_avg 38417 ± 2% -89.4% 4088 ± 3% sched_debug.cfs_rq[111]:/.tg->runnable_avg 38435 ± 2% -89.4% 4091 ± 3% sched_debug.cfs_rq[112]:/.tg->runnable_avg 38435 ± 2% -89.4% 4092 ± 3% sched_debug.cfs_rq[113]:/.tg->runnable_avg 38428 ± 2% -89.3% 4095 ± 3% sched_debug.cfs_rq[114]:/.tg->runnable_avg 38439 ± 1% -89.3% 4096 ± 3% sched_debug.cfs_rq[115]:/.tg->runnable_avg 38445 ± 2% -89.3% 4095 ± 3% sched_debug.cfs_rq[116]:/.tg->runnable_avg 38449 ± 2% -89.3% 4097 ± 3% sched_debug.cfs_rq[117]:/.tg->runnable_avg 38461 ± 2% -89.3% 4100 ± 3% sched_debug.cfs_rq[118]:/.tg->runnable_avg 38460 ± 1% -89.3% 4102 ± 3% sched_debug.cfs_rq[119]:/.tg->runnable_avg 76608 ± 7% -91.7% 6356 ± 16% sched_debug.cfs_rq[11]:/.exec_clock 546 ± 2% -93.5% 35 ± 25% sched_debug.cfs_rq[11]:/.tg_runnable_contrib 37465 ± 1% -89.7% 3852 ± 4% sched_debug.cfs_rq[11]:/.tg->runnable_avg 24985 ± 2% -93.3% 1664 ± 24% sched_debug.cfs_rq[11]:/.avg->runnable_avg_sum 5775629 ± 7% -98.4% 91917 ± 12% sched_debug.cfs_rq[11]:/.min_vruntime 76166 ± 8% -92.2% 5923 ± 22% sched_debug.cfs_rq[12]:/.exec_clock 5756491 ± 8% -98.5% 86026 ± 20% sched_debug.cfs_rq[12]:/.min_vruntime 5 ± 31% -100.0% 0 ± 0% sched_debug.cfs_rq[12]:/.runnable_load_avg 37492 ± 1% -89.7% 3853 ± 4% sched_debug.cfs_rq[12]:/.tg->runnable_avg 75892 ± 8% -91.8% 6211 ± 19% sched_debug.cfs_rq[13]:/.exec_clock 37506 ± 1% -89.7% 3856 ± 4% sched_debug.cfs_rq[13]:/.tg->runnable_avg 5727275 ± 7% -98.4% 93734 ± 18% sched_debug.cfs_rq[13]:/.min_vruntime 37528 ± 1% -89.7% 3857 ± 4% sched_debug.cfs_rq[14]:/.tg->runnable_avg 25080 ± 5% -94.4% 1394 ± 43% sched_debug.cfs_rq[14]:/.avg->runnable_avg_sum 75791 ± 8% -92.2% 5907 ± 24% sched_debug.cfs_rq[14]:/.exec_clock 550 ± 4% -94.6% 29 ± 43% sched_debug.cfs_rq[14]:/.tg_runnable_contrib 6 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[14]:/.runnable_load_avg 5717919 ± 7% -98.4% 90360 ± 22% sched_debug.cfs_rq[14]:/.min_vruntime 37554 ± 1% -89.7% 3860 ± 4% sched_debug.cfs_rq[15]:/.tg->runnable_avg 71299 ± 11% -96.4% 2537 ± 18% sched_debug.cfs_rq[15]:/.exec_clock 5471648 ± 11% -99.1% 50259 ± 22% sched_debug.cfs_rq[15]:/.min_vruntime 71273 ± 12% -96.7% 2353 ± 10% sched_debug.cfs_rq[16]:/.exec_clock 5469742 ± 11% -99.0% 53306 ± 24% sched_debug.cfs_rq[16]:/.min_vruntime 37563 ± 1% -89.7% 3864 ± 4% sched_debug.cfs_rq[16]:/.tg->runnable_avg 2807093.70 ± 36% -100.0% 0.00 ± 0% sched_debug.cfs_rq[17]:/.MIN_vruntime 6 ± 31% -100.0% 0 ± 0% sched_debug.cfs_rq[17]:/.runnable_load_avg 37568 ± 1% -89.7% 3865 ± 4% sched_debug.cfs_rq[17]:/.tg->runnable_avg 71654 ± 12% -96.9% 2229 ± 14% sched_debug.cfs_rq[17]:/.exec_clock 5474236 ± 11% -99.1% 50364 ± 25% sched_debug.cfs_rq[17]:/.min_vruntime 2807093.70 ± 36% -100.0% 0.00 ± 0% sched_debug.cfs_rq[17]:/.max_vruntime 37574 ± 1% -89.7% 3866 ± 4% sched_debug.cfs_rq[18]:/.tg->runnable_avg 2611368.18 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[18]:/.max_vruntime 2611367.85 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[18]:/.MIN_vruntime 5469964 ± 11% -99.2% 42263 ± 36% sched_debug.cfs_rq[18]:/.min_vruntime 4 ± 28% -100.0% 0 ± 0% sched_debug.cfs_rq[19]:/.runnable_load_avg 5460827 ± 11% -99.2% 46250 ± 33% sched_debug.cfs_rq[19]:/.min_vruntime 37603 ± 1% -89.7% 3871 ± 4% sched_debug.cfs_rq[19]:/.tg->runnable_avg 5931126 ± 8% -86.0% 829332 ± 28% sched_debug.cfs_rq[1]:/.min_vruntime 37333 ± 1% -89.7% 3834 ± 4% sched_debug.cfs_rq[1]:/.tg->runnable_avg 1048433 ± 16% -36.8% 662283 ± 34% sched_debug.cfs_rq[1]:/.spread0 5501605 ± 11% -99.1% 46820 ± 29% sched_debug.cfs_rq[20]:/.min_vruntime 71374 ± 11% -97.4% 1825 ± 22% sched_debug.cfs_rq[20]:/.exec_clock 37630 ± 1% -89.7% 3876 ± 4% sched_debug.cfs_rq[20]:/.tg->runnable_avg 4 ± 49% -100.0% 0 ± 0% sched_debug.cfs_rq[20]:/.runnable_load_avg 5435674 ± 11% -99.1% 48697 ± 31% sched_debug.cfs_rq[21]:/.min_vruntime 37645 ± 1% -89.7% 3875 ± 4% sched_debug.cfs_rq[21]:/.tg->runnable_avg 7 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[21]:/.load 122 ± 48% -100.0% 0 ± 0% sched_debug.cfs_rq[21]:/.utilization_load_avg 70205 ± 12% -97.3% 1867 ± 36% sched_debug.cfs_rq[21]:/.exec_clock 7 ± 44% -100.0% 0 ± 0% sched_debug.cfs_rq[22]:/.load 37661 ± 1% -89.7% 3876 ± 4% sched_debug.cfs_rq[22]:/.tg->runnable_avg 5491828 ± 11% -99.2% 45143 ± 26% sched_debug.cfs_rq[22]:/.min_vruntime 6 ± 32% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.load 4 ± 35% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.runnable_load_avg 90 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[23]:/.utilization_load_avg 37652 ± 1% -89.7% 3879 ± 4% sched_debug.cfs_rq[23]:/.tg->runnable_avg 5499468 ± 11% -99.2% 44829 ± 30% sched_debug.cfs_rq[23]:/.min_vruntime 37651 ± 1% -89.7% 3882 ± 4% sched_debug.cfs_rq[24]:/.tg->runnable_avg 7 ± 47% -100.0% 0 ± 0% sched_debug.cfs_rq[24]:/.load 2002338.86 ± 45% -100.0% 0.00 ± 0% sched_debug.cfs_rq[24]:/.MIN_vruntime 5458480 ± 11% -99.2% 45851 ± 22% sched_debug.cfs_rq[24]:/.min_vruntime 2002338.86 ± 45% -100.0% 0.00 ± 0% sched_debug.cfs_rq[24]:/.max_vruntime 499 ± 11% -98.4% 7 ± 49% sched_debug.cfs_rq[24]:/.tg_runnable_contrib 22791 ± 11% -98.3% 393 ± 47% sched_debug.cfs_rq[24]:/.avg->runnable_avg_sum 5501548 ± 11% -99.2% 42017 ± 39% sched_debug.cfs_rq[25]:/.min_vruntime 37675 ± 1% -89.7% 3883 ± 4% sched_debug.cfs_rq[25]:/.tg->runnable_avg 37701 ± 1% -89.7% 3887 ± 4% sched_debug.cfs_rq[26]:/.tg->runnable_avg 5460570 ± 11% -99.2% 42120 ± 20% sched_debug.cfs_rq[26]:/.min_vruntime 72021 ± 8% -97.9% 1500 ± 39% sched_debug.cfs_rq[26]:/.exec_clock 5443461 ± 11% -99.3% 39281 ± 30% sched_debug.cfs_rq[27]:/.min_vruntime 37720 ± 1% -89.7% 3890 ± 4% sched_debug.cfs_rq[27]:/.tg->runnable_avg 70407 ± 11% -97.6% 1657 ± 41% sched_debug.cfs_rq[27]:/.exec_clock 37737 ± 1% -89.7% 3890 ± 4% sched_debug.cfs_rq[28]:/.tg->runnable_avg 71813 ± 12% -98.0% 1424 ± 44% sched_debug.cfs_rq[28]:/.exec_clock 5482457 ± 11% -99.2% 42096 ± 24% sched_debug.cfs_rq[28]:/.min_vruntime 5478035 ± 11% -99.2% 41368 ± 31% sched_debug.cfs_rq[29]:/.min_vruntime 37748 ± 1% -89.7% 3893 ± 4% sched_debug.cfs_rq[29]:/.tg->runnable_avg 5924938 ± 8% -86.5% 799517 ± 29% sched_debug.cfs_rq[2]:/.min_vruntime 37344 ± 1% -89.7% 3839 ± 4% sched_debug.cfs_rq[2]:/.tg->runnable_avg 1042225 ± 20% -39.3% 632468 ± 35% sched_debug.cfs_rq[2]:/.spread0 37765 ± 1% -89.7% 3895 ± 4% sched_debug.cfs_rq[30]:/.tg->runnable_avg 2514014.52 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[30]:/.max_vruntime 5791091 ± 11% -99.1% 49961 ± 26% sched_debug.cfs_rq[30]:/.min_vruntime 2514014.52 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[30]:/.MIN_vruntime 76335 ± 12% -97.0% 2285 ± 16% sched_debug.cfs_rq[30]:/.exec_clock 76769 ± 11% -97.4% 2026 ± 16% sched_debug.cfs_rq[31]:/.exec_clock 2370182.85 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[31]:/.MIN_vruntime 5806067 ± 11% -99.2% 46410 ± 22% sched_debug.cfs_rq[31]:/.min_vruntime 2370183.19 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[31]:/.max_vruntime 37783 ± 1% -89.7% 3903 ± 4% sched_debug.cfs_rq[31]:/.tg->runnable_avg 77107 ± 12% -97.4% 1980 ± 30% sched_debug.cfs_rq[32]:/.exec_clock 5865705 ± 12% -99.2% 45792 ± 32% sched_debug.cfs_rq[32]:/.min_vruntime 37808 ± 1% -89.7% 3909 ± 4% sched_debug.cfs_rq[32]:/.tg->runnable_avg 24950 ± 9% -96.2% 943 ± 23% sched_debug.cfs_rq[33]:/.avg->runnable_avg_sum 37805 ± 1% -89.6% 3914 ± 4% sched_debug.cfs_rq[33]:/.tg->runnable_avg 5827248 ± 12% -99.2% 44413 ± 27% sched_debug.cfs_rq[33]:/.min_vruntime 76627 ± 13% -97.6% 1812 ± 27% sched_debug.cfs_rq[33]:/.exec_clock 546 ± 8% -96.4% 19 ± 22% sched_debug.cfs_rq[33]:/.tg_runnable_contrib 37804 ± 1% -89.6% 3915 ± 4% sched_debug.cfs_rq[34]:/.tg->runnable_avg 76396 ± 12% -97.7% 1738 ± 29% sched_debug.cfs_rq[34]:/.exec_clock 2052042.52 ± 30% -100.0% 0.00 ± 0% sched_debug.cfs_rq[34]:/.MIN_vruntime 5789564 ± 11% -99.2% 43627 ± 23% sched_debug.cfs_rq[34]:/.min_vruntime 2052042.52 ± 30% -100.0% 0.00 ± 0% sched_debug.cfs_rq[34]:/.max_vruntime 37810 ± 1% -89.6% 3915 ± 4% sched_debug.cfs_rq[35]:/.tg->runnable_avg 5793549 ± 11% -99.3% 40722 ± 15% sched_debug.cfs_rq[35]:/.min_vruntime 76584 ± 12% -97.8% 1682 ± 27% sched_debug.cfs_rq[35]:/.exec_clock 553 ± 8% -98.9% 6 ± 36% sched_debug.cfs_rq[36]:/.tg_runnable_contrib 37800 ± 1% -89.6% 3917 ± 4% sched_debug.cfs_rq[36]:/.tg->runnable_avg 25249 ± 8% -98.7% 330 ± 28% sched_debug.cfs_rq[36]:/.avg->runnable_avg_sum 76722 ± 13% -97.9% 1576 ± 41% sched_debug.cfs_rq[36]:/.exec_clock 5843915 ± 12% -99.3% 40252 ± 43% sched_debug.cfs_rq[36]:/.min_vruntime 75923 ± 12% -97.7% 1718 ± 38% sched_debug.cfs_rq[37]:/.exec_clock 5768738 ± 11% -99.3% 42192 ± 39% sched_debug.cfs_rq[37]:/.min_vruntime 535 ± 9% -98.8% 6 ± 28% sched_debug.cfs_rq[37]:/.tg_runnable_contrib 37811 ± 1% -89.6% 3924 ± 4% sched_debug.cfs_rq[37]:/.tg->runnable_avg 24479 ± 9% -98.7% 316 ± 26% sched_debug.cfs_rq[37]:/.avg->runnable_avg_sum 7 ± 32% -100.0% 0 ± 0% sched_debug.cfs_rq[38]:/.load 37833 ± 1% -89.6% 3925 ± 4% sched_debug.cfs_rq[38]:/.tg->runnable_avg 5 ± 30% -100.0% 0 ± 0% sched_debug.cfs_rq[38]:/.runnable_load_avg 5826004 ± 12% -99.3% 43456 ± 44% sched_debug.cfs_rq[38]:/.min_vruntime 544 ± 10% -98.4% 8 ± 27% sched_debug.cfs_rq[39]:/.tg_runnable_contrib 6 ± 21% -100.0% 0 ± 0% sched_debug.cfs_rq[39]:/.runnable_load_avg 76688 ± 13% -98.2% 1415 ± 35% sched_debug.cfs_rq[39]:/.exec_clock 37837 ± 1% -89.6% 3925 ± 4% sched_debug.cfs_rq[39]:/.tg->runnable_avg 24864 ± 9% -98.2% 454 ± 23% sched_debug.cfs_rq[39]:/.avg->runnable_avg_sum 5835422 ± 12% -99.4% 36160 ± 31% sched_debug.cfs_rq[39]:/.min_vruntime 26232 ± 5% -22.4% 20344 ± 8% sched_debug.cfs_rq[3]:/.avg->runnable_avg_sum 574 ± 6% -22.5% 445 ± 8% sched_debug.cfs_rq[3]:/.tg_runnable_contrib 37364 ± 1% -89.7% 3840 ± 4% sched_debug.cfs_rq[3]:/.tg->runnable_avg 9 ± 49% +335.2% 39 ± 44% sched_debug.cfs_rq[3]:/.load 5891140 ± 8% -87.4% 740621 ± 29% sched_debug.cfs_rq[3]:/.min_vruntime 1008386 ± 18% -43.1% 573573 ± 35% sched_debug.cfs_rq[3]:/.spread0 76125 ± 12% -98.0% 1494 ± 47% sched_debug.cfs_rq[40]:/.exec_clock 37838 ± 1% -89.6% 3926 ± 4% sched_debug.cfs_rq[40]:/.tg->runnable_avg 5784431 ± 11% -99.3% 38831 ± 36% sched_debug.cfs_rq[40]:/.min_vruntime 76346 ± 13% -98.1% 1455 ± 41% sched_debug.cfs_rq[41]:/.exec_clock 2279884.43 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[41]:/.MIN_vruntime 5802087 ± 12% -99.3% 40195 ± 34% sched_debug.cfs_rq[41]:/.min_vruntime 5 ± 29% -100.0% 0 ± 0% sched_debug.cfs_rq[41]:/.runnable_load_avg 2279884.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[41]:/.max_vruntime 37855 ± 1% -89.6% 3929 ± 4% sched_debug.cfs_rq[41]:/.tg->runnable_avg 134 ± 42% -100.0% 0 ± 0% sched_debug.cfs_rq[41]:/.utilization_load_avg 37858 ± 1% -89.6% 3932 ± 4% sched_debug.cfs_rq[42]:/.tg->runnable_avg 5839210 ± 12% -99.3% 39152 ± 47% sched_debug.cfs_rq[42]:/.min_vruntime 76766 ± 13% -98.2% 1387 ± 46% sched_debug.cfs_rq[42]:/.exec_clock 5 ± 20% -100.0% 0 ± 0% sched_debug.cfs_rq[42]:/.runnable_load_avg 76865 ± 13% -98.1% 1491 ± 35% sched_debug.cfs_rq[43]:/.exec_clock 5 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[43]:/.runnable_load_avg 7 ± 44% -100.0% 0 ± 0% sched_debug.cfs_rq[43]:/.load 3274498.62 ± 42% -100.0% 0.00 ± 0% sched_debug.cfs_rq[43]:/.max_vruntime 5845980 ± 12% -99.3% 42925 ± 28% sched_debug.cfs_rq[43]:/.min_vruntime 3274498.62 ± 42% -100.0% 0.00 ± 0% sched_debug.cfs_rq[43]:/.MIN_vruntime 37884 ± 1% -89.6% 3934 ± 4% sched_debug.cfs_rq[43]:/.tg->runnable_avg 76461 ± 13% -97.9% 1582 ± 36% sched_debug.cfs_rq[44]:/.exec_clock 5801262 ± 12% -99.3% 39028 ± 29% sched_debug.cfs_rq[44]:/.min_vruntime 37904 ± 1% -89.6% 3939 ± 4% sched_debug.cfs_rq[44]:/.tg->runnable_avg 37919 ± 1% -89.6% 3945 ± 4% sched_debug.cfs_rq[45]:/.tg->runnable_avg 75128 ± 8% -96.8% 2372 ± 25% sched_debug.cfs_rq[45]:/.exec_clock 5733040 ± 8% -99.2% 47682 ± 38% sched_debug.cfs_rq[45]:/.min_vruntime 74823 ± 8% -97.4% 1957 ± 15% sched_debug.cfs_rq[46]:/.exec_clock 2181820.38 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[46]:/.MIN_vruntime 5706201 ± 7% -99.2% 45069 ± 29% sched_debug.cfs_rq[46]:/.min_vruntime 37931 ± 1% -89.6% 3949 ± 4% sched_debug.cfs_rq[46]:/.tg->runnable_avg 2181820.38 ± 44% -100.0% 0.00 ± 0% sched_debug.cfs_rq[46]:/.max_vruntime 37937 ± 1% -89.6% 3952 ± 4% sched_debug.cfs_rq[47]:/.tg->runnable_avg 75941 ± 7% -97.7% 1740 ± 26% sched_debug.cfs_rq[47]:/.exec_clock 5795769 ± 7% -99.3% 43188 ± 29% sched_debug.cfs_rq[47]:/.min_vruntime 74784 ± 7% -97.9% 1582 ± 27% sched_debug.cfs_rq[48]:/.exec_clock 5703468 ± 7% -99.3% 39942 ± 31% sched_debug.cfs_rq[48]:/.min_vruntime 37947 ± 1% -89.6% 3954 ± 4% sched_debug.cfs_rq[48]:/.tg->runnable_avg 5 ± 33% -100.0% 0 ± 0% sched_debug.cfs_rq[49]:/.runnable_load_avg 75045 ± 7% -97.9% 1592 ± 23% sched_debug.cfs_rq[49]:/.exec_clock 2349144.49 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[49]:/.MIN_vruntime 5717654 ± 7% -99.3% 41062 ± 37% sched_debug.cfs_rq[49]:/.min_vruntime 2349144.49 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[49]:/.max_vruntime 37976 ± 1% -89.6% 3956 ± 4% sched_debug.cfs_rq[49]:/.tg->runnable_avg 26117 ± 3% -50.7% 12870 ± 7% sched_debug.cfs_rq[4]:/.avg->runnable_avg_sum 37364 ± 1% -89.7% 3843 ± 4% sched_debug.cfs_rq[4]:/.tg->runnable_avg 78451 ± 8% -47.9% 40905 ± 20% sched_debug.cfs_rq[4]:/.exec_clock 571 ± 4% -50.5% 282 ± 7% sched_debug.cfs_rq[4]:/.tg_runnable_contrib 5828333 ± 8% -91.7% 486365 ± 28% sched_debug.cfs_rq[4]:/.min_vruntime 945560 ± 28% -66.2% 319316 ± 41% sched_debug.cfs_rq[4]:/.spread0 92 ± 24% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.utilization_load_avg 75109 ± 7% -97.7% 1699 ± 37% sched_debug.cfs_rq[50]:/.exec_clock 6 ± 49% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.load 5726100 ± 6% -99.2% 45365 ± 30% sched_debug.cfs_rq[50]:/.min_vruntime 543 ± 11% -98.9% 6 ± 49% sched_debug.cfs_rq[50]:/.tg_runnable_contrib 24764 ± 11% -98.7% 328 ± 39% sched_debug.cfs_rq[50]:/.avg->runnable_avg_sum 37997 ± 1% -89.6% 3959 ± 4% sched_debug.cfs_rq[50]:/.tg->runnable_avg 4 ± 34% -100.0% 0 ± 0% sched_debug.cfs_rq[50]:/.runnable_load_avg 5701479 ± 6% -99.3% 37520 ± 36% sched_debug.cfs_rq[51]:/.min_vruntime 38008 ± 1% -89.6% 3959 ± 4% sched_debug.cfs_rq[51]:/.tg->runnable_avg 74668 ± 7% -98.1% 1411 ± 41% sched_debug.cfs_rq[51]:/.exec_clock 25473 ± 12% -98.4% 408 ± 48% sched_debug.cfs_rq[52]:/.avg->runnable_avg_sum 75465 ± 8% -98.1% 1450 ± 23% sched_debug.cfs_rq[52]:/.exec_clock 3567331.44 ± 29% -100.0% 0.00 ± 0% sched_debug.cfs_rq[52]:/.MIN_vruntime 5767898 ± 7% -99.3% 38845 ± 30% sched_debug.cfs_rq[52]:/.min_vruntime 8 ± 41% -100.0% 0 ± 0% sched_debug.cfs_rq[52]:/.runnable_load_avg 3567331.77 ± 29% -100.0% 0.00 ± 0% sched_debug.cfs_rq[52]:/.max_vruntime 38019 ± 1% -89.6% 3962 ± 4% sched_debug.cfs_rq[52]:/.tg->runnable_avg 38036 ± 2% -89.6% 3966 ± 4% sched_debug.cfs_rq[53]:/.tg->runnable_avg 75493 ± 8% -98.0% 1520 ± 30% sched_debug.cfs_rq[53]:/.exec_clock 2564844.76 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[53]:/.max_vruntime 5768160 ± 7% -99.2% 44245 ± 35% sched_debug.cfs_rq[53]:/.min_vruntime 2564844.76 ± 34% -100.0% 0.00 ± 0% sched_debug.cfs_rq[53]:/.MIN_vruntime 75523 ± 9% -98.1% 1443 ± 40% sched_debug.cfs_rq[54]:/.exec_clock 38055 ± 2% -89.6% 3969 ± 4% sched_debug.cfs_rq[54]:/.tg->runnable_avg 5757595 ± 8% -99.3% 41312 ± 28% sched_debug.cfs_rq[54]:/.min_vruntime 5731615 ± 7% -99.3% 38668 ± 27% sched_debug.cfs_rq[55]:/.min_vruntime 75160 ± 8% -98.0% 1473 ± 39% sched_debug.cfs_rq[55]:/.exec_clock 38085 ± 2% -89.6% 3971 ± 4% sched_debug.cfs_rq[55]:/.tg->runnable_avg 552 ± 13% -99.0% 5 ± 49% sched_debug.cfs_rq[56]:/.tg_runnable_contrib 5759032 ± 7% -99.3% 37467 ± 32% sched_debug.cfs_rq[56]:/.min_vruntime 38101 ± 2% -89.6% 3980 ± 4% sched_debug.cfs_rq[56]:/.tg->runnable_avg 25129 ± 13% -98.8% 291 ± 44% sched_debug.cfs_rq[56]:/.avg->runnable_avg_sum 75483 ± 8% -98.1% 1421 ± 48% sched_debug.cfs_rq[56]:/.exec_clock 5 ± 29% -100.0% 0 ± 0% sched_debug.cfs_rq[57]:/.runnable_load_avg 74873 ± 8% -97.9% 1582 ± 49% sched_debug.cfs_rq[57]:/.exec_clock 5707036 ± 7% -99.2% 43462 ± 45% sched_debug.cfs_rq[57]:/.min_vruntime 38113 ± 2% -89.6% 3981 ± 4% sched_debug.cfs_rq[57]:/.tg->runnable_avg 2056049.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[58]:/.MIN_vruntime 2056049.76 ± 47% -100.0% 0.00 ± 0% sched_debug.cfs_rq[58]:/.max_vruntime 5638172 ± 8% -99.3% 40579 ± 41% sched_debug.cfs_rq[58]:/.min_vruntime 38110 ± 1% -89.6% 3979 ± 4% sched_debug.cfs_rq[58]:/.tg->runnable_avg 74655 ± 8% -98.3% 1291 ± 34% sched_debug.cfs_rq[59]:/.exec_clock 9 ± 48% -100.0% 0 ± 0% sched_debug.cfs_rq[59]:/.load 38125 ± 1% -89.6% 3980 ± 4% sched_debug.cfs_rq[59]:/.tg->runnable_avg 5687789 ± 7% -99.3% 37622 ± 41% sched_debug.cfs_rq[59]:/.min_vruntime 569 ± 5% -89.4% 60 ± 18% sched_debug.cfs_rq[5]:/.tg_runnable_contrib 78491 ± 9% -87.5% 9836 ± 6% sched_debug.cfs_rq[5]:/.exec_clock 37386 ± 1% -89.7% 3845 ± 4% sched_debug.cfs_rq[5]:/.tg->runnable_avg 26026 ± 6% -89.3% 2782 ± 17% sched_debug.cfs_rq[5]:/.avg->runnable_avg_sum 5868941 ± 8% -97.8% 132022 ± 13% sched_debug.cfs_rq[5]:/.min_vruntime 50960 ± 9% -99.1% 438 ± 44% sched_debug.cfs_rq[60]:/.exec_clock 38133 ± 1% -89.6% 3981 ± 4% sched_debug.cfs_rq[60]:/.tg->runnable_avg 51720 ± 14% -97.9% 1072 ± 40% sched_debug.cfs_rq[61]:/.exec_clock 4337125 ± 13% -99.2% 35916 ± 38% sched_debug.cfs_rq[61]:/.min_vruntime 38160 ± 1% -89.6% 3984 ± 4% sched_debug.cfs_rq[61]:/.tg->runnable_avg 3 ± 39% -100.0% 0 ± 0% sched_debug.cfs_rq[62]:/.load 52202 ± 13% -98.8% 626 ± 34% sched_debug.cfs_rq[62]:/.exec_clock 38148 ± 1% -89.5% 3986 ± 4% sched_debug.cfs_rq[62]:/.tg->runnable_avg 14412 ± 18% -98.6% 194 ± 48% sched_debug.cfs_rq[62]:/.avg->runnable_avg_sum 2 ± 40% -100.0% 0 ± 0% sched_debug.cfs_rq[62]:/.runnable_load_avg 38154 ± 2% -89.6% 3986 ± 4% sched_debug.cfs_rq[63]:/.tg->runnable_avg 1268924 ± 47% -97.9% 26358 ± 34% sched_debug.cfs_rq[64]:/.min_vruntime 64 ± 42% -95.8% 2 ± 41% sched_debug.cfs_rq[64]:/.tg_runnable_contrib 38173 ± 2% -89.6% 3988 ± 4% sched_debug.cfs_rq[64]:/.tg->runnable_avg 2993 ± 41% -95.1% 148 ± 36% sched_debug.cfs_rq[64]:/.avg->runnable_avg_sum 38152 ± 2% -89.6% 3985 ± 4% sched_debug.cfs_rq[65]:/.tg->runnable_avg 7 ± 47% -100.0% 0 ± 0% sched_debug.cfs_rq[65]:/.nr_spread_over 38162 ± 2% -89.5% 3989 ± 3% sched_debug.cfs_rq[66]:/.tg->runnable_avg 10 ± 30% -100.0% 0 ± 0% sched_debug.cfs_rq[66]:/.nr_spread_over 15875 ± 49% -95.7% 680 ± 31% sched_debug.cfs_rq[66]:/.exec_clock 38175 ± 2% -89.5% 3990 ± 3% sched_debug.cfs_rq[67]:/.tg->runnable_avg 38206 ± 2% -89.6% 3991 ± 3% sched_debug.cfs_rq[68]:/.tg->runnable_avg 38221 ± 2% -89.6% 3992 ± 3% sched_debug.cfs_rq[69]:/.tg->runnable_avg 15369 ± 47% -94.7% 812 ± 43% sched_debug.cfs_rq[69]:/.exec_clock 25441 ± 4% -91.8% 2076 ± 13% sched_debug.cfs_rq[6]:/.avg->runnable_avg_sum 557 ± 4% -92.0% 44 ± 13% sched_debug.cfs_rq[6]:/.tg_runnable_contrib 6 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[6]:/.load 5815571 ± 7% -98.1% 110120 ± 10% sched_debug.cfs_rq[6]:/.min_vruntime 77709 ± 8% -89.8% 7891 ± 9% sched_debug.cfs_rq[6]:/.exec_clock 5 ± 33% -100.0% 0 ± 0% sched_debug.cfs_rq[6]:/.runnable_load_avg 37407 ± 1% -89.7% 3849 ± 4% sched_debug.cfs_rq[6]:/.tg->runnable_avg 38225 ± 2% -89.5% 4001 ± 4% sched_debug.cfs_rq[70]:/.tg->runnable_avg 38235 ± 2% -89.5% 4005 ± 4% sched_debug.cfs_rq[71]:/.tg->runnable_avg 38238 ± 2% -89.5% 4007 ± 4% sched_debug.cfs_rq[72]:/.tg->runnable_avg 38253 ± 2% -89.5% 4007 ± 4% sched_debug.cfs_rq[73]:/.tg->runnable_avg 9 ± 27% -100.0% 0 ± 0% sched_debug.cfs_rq[73]:/.nr_spread_over 7 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[74]:/.nr_spread_over 38240 ± 2% -89.5% 4012 ± 4% sched_debug.cfs_rq[74]:/.tg->runnable_avg 38235 ± 2% -89.5% 4010 ± 4% sched_debug.cfs_rq[75]:/.tg->runnable_avg 38233 ± 2% -89.5% 4010 ± 4% sched_debug.cfs_rq[76]:/.tg->runnable_avg 38239 ± 2% -89.5% 4012 ± 4% sched_debug.cfs_rq[77]:/.tg->runnable_avg 38260 ± 2% -89.5% 4015 ± 4% sched_debug.cfs_rq[78]:/.tg->runnable_avg 38284 ± 2% -89.5% 4019 ± 3% sched_debug.cfs_rq[79]:/.tg->runnable_avg 25894 ± 3% -92.4% 1956 ± 12% sched_debug.cfs_rq[7]:/.avg->runnable_avg_sum 78030 ± 8% -90.8% 7193 ± 8% sched_debug.cfs_rq[7]:/.exec_clock 5863768 ± 7% -98.2% 105821 ± 12% sched_debug.cfs_rq[7]:/.min_vruntime 566 ± 3% -92.6% 42 ± 12% sched_debug.cfs_rq[7]:/.tg_runnable_contrib 37430 ± 1% -89.7% 3850 ± 4% sched_debug.cfs_rq[7]:/.tg->runnable_avg 38292 ± 2% -89.5% 4021 ± 3% sched_debug.cfs_rq[80]:/.tg->runnable_avg 38300 ± 2% -89.5% 4022 ± 3% sched_debug.cfs_rq[81]:/.tg->runnable_avg 38303 ± 2% -89.5% 4025 ± 3% sched_debug.cfs_rq[82]:/.tg->runnable_avg 38286 ± 2% -89.5% 4028 ± 3% sched_debug.cfs_rq[83]:/.tg->runnable_avg 38294 ± 2% -89.5% 4032 ± 3% sched_debug.cfs_rq[84]:/.tg->runnable_avg 38304 ± 2% -89.5% 4034 ± 3% sched_debug.cfs_rq[85]:/.tg->runnable_avg 38321 ± 2% -89.5% 4035 ± 3% sched_debug.cfs_rq[86]:/.tg->runnable_avg 38326 ± 2% -89.5% 4039 ± 3% sched_debug.cfs_rq[87]:/.tg->runnable_avg 38322 ± 2% -89.5% 4040 ± 3% sched_debug.cfs_rq[88]:/.tg->runnable_avg 38313 ± 2% -89.4% 4044 ± 3% sched_debug.cfs_rq[89]:/.tg->runnable_avg 37445 ± 1% -89.7% 3853 ± 4% sched_debug.cfs_rq[8]:/.tg->runnable_avg 553 ± 4% -92.0% 44 ± 5% sched_debug.cfs_rq[8]:/.tg_runnable_contrib 25270 ± 4% -91.8% 2076 ± 5% sched_debug.cfs_rq[8]:/.avg->runnable_avg_sum 77680 ± 8% -91.2% 6799 ± 6% sched_debug.cfs_rq[8]:/.exec_clock 5833584 ± 8% -98.3% 99293 ± 11% sched_debug.cfs_rq[8]:/.min_vruntime 38307 ± 2% -89.4% 4044 ± 3% sched_debug.cfs_rq[90]:/.tg->runnable_avg 38316 ± 2% -89.4% 4047 ± 3% sched_debug.cfs_rq[91]:/.tg->runnable_avg 4788 ± 48% -95.1% 235 ± 35% sched_debug.cfs_rq[91]:/.avg->runnable_avg_sum 103 ± 49% -95.8% 4 ± 39% sched_debug.cfs_rq[91]:/.tg_runnable_contrib 38320 ± 2% -89.4% 4051 ± 3% sched_debug.cfs_rq[92]:/.tg->runnable_avg 38332 ± 2% -89.4% 4052 ± 3% sched_debug.cfs_rq[93]:/.tg->runnable_avg 38341 ± 2% -89.4% 4053 ± 3% sched_debug.cfs_rq[94]:/.tg->runnable_avg 38329 ± 2% -89.4% 4056 ± 3% sched_debug.cfs_rq[95]:/.tg->runnable_avg 38315 ± 2% -89.4% 4060 ± 3% sched_debug.cfs_rq[96]:/.tg->runnable_avg 38318 ± 2% -89.4% 4061 ± 3% sched_debug.cfs_rq[97]:/.tg->runnable_avg 38321 ± 2% -89.4% 4064 ± 3% sched_debug.cfs_rq[98]:/.tg->runnable_avg 38326 ± 2% -89.4% 4066 ± 3% sched_debug.cfs_rq[99]:/.tg->runnable_avg 2599395.82 ± 28% -100.0% 0.00 ± 0% sched_debug.cfs_rq[9]:/.MIN_vruntime 37435 ± 1% -89.7% 3849 ± 4% sched_debug.cfs_rq[9]:/.tg->runnable_avg 7 ± 45% -100.0% 0 ± 0% sched_debug.cfs_rq[9]:/.runnable_load_avg 25086 ± 4% -93.1% 1741 ± 14% sched_debug.cfs_rq[9]:/.avg->runnable_avg_sum 76760 ± 8% -91.3% 6708 ± 10% sched_debug.cfs_rq[9]:/.exec_clock 2599396.15 ± 28% -100.0% 0.00 ± 0% sched_debug.cfs_rq[9]:/.max_vruntime 550 ± 4% -93.2% 37 ± 15% sched_debug.cfs_rq[9]:/.tg_runnable_contrib 5775897 ± 7% -98.3% 99243 ± 18% sched_debug.cfs_rq[9]:/.min_vruntime 7 ± 39% +11559.1% 855 ± 0% sched_debug.cpu#0.cpu_load[3] 8 ± 35% +9767.3% 855 ± 0% sched_debug.cpu#0.cpu_load[2] 10 ± 25% +8443.3% 854 ± 0% sched_debug.cpu#0.cpu_load[1] 6 ± 39% +14133.3% 854 ± 0% sched_debug.cpu#0.cpu_load[4] 866410 ± 20% -42.2% 500354 ± 0% sched_debug.cpu#0.max_idle_balance_cost 401816 ± 6% +5302.7% 21709084 ± 9% sched_debug.cpu#0.ttwu_count 312394 ± 14% +1631.2% 5408220 ± 18% sched_debug.cpu#1.sched_goidle 177603 ± 24% -87.5% 22288 ± 17% sched_debug.cpu#1.ttwu_local 303227 ± 7% -46.2% 163110 ± 12% sched_debug.cpu#1.avg_idle 695685 ± 12% +1457.9% 10837841 ± 18% sched_debug.cpu#1.nr_switches 699186 ± 12% +1450.1% 10837966 ± 18% sched_debug.cpu#1.sched_count 461875 ± 8% -94.9% 23602 ± 15% sched_debug.cpu#1.ttwu_count 262569 ± 5% +240.9% 895031 ± 14% sched_debug.cpu#10.avg_idle 145636 ± 7% -30.0% 101980 ± 10% sched_debug.cpu#10.nr_load_updates 4 ± 36% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[1] 3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[3] 3 ± 34% -100.0% 0 ± 0% sched_debug.cpu#10.cpu_load[2] 534226 ± 10% -99.7% 1612 ± 29% sched_debug.cpu#10.ttwu_count 182313 ± 10% -99.3% 1351 ± 29% sched_debug.cpu#10.ttwu_local 912692 ± 29% -45.2% 500000 ± 0% sched_debug.cpu#100.max_idle_balance_cost 890003 ± 25% -43.5% 502419 ± 0% sched_debug.cpu#101.max_idle_balance_cost 895162 ± 26% -42.0% 518876 ± 8% sched_debug.cpu#102.max_idle_balance_cost 1279055 ± 16% -18.8% 1038141 ± 8% sched_debug.cpu#102.avg_idle 1210646 ± 23% -19.2% 978478 ± 1% sched_debug.cpu#103.avg_idle 873261 ± 25% -42.7% 500000 ± 0% sched_debug.cpu#103.max_idle_balance_cost 864952 ± 24% -42.2% 500000 ± 0% sched_debug.cpu#104.max_idle_balance_cost 887413 ± 31% -43.7% 500000 ± 0% sched_debug.cpu#105.max_idle_balance_cost 854103 ± 27% -40.9% 504735 ± 2% sched_debug.cpu#106.max_idle_balance_cost 864081 ± 28% -42.0% 501124 ± 0% sched_debug.cpu#107.max_idle_balance_cost 898657 ± 28% -42.5% 516711 ± 6% sched_debug.cpu#108.max_idle_balance_cost 3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#11.cpu_load[2] 543726 ± 4% -99.7% 1760 ± 32% sched_debug.cpu#11.ttwu_count 145709 ± 7% -30.3% 101564 ± 9% sched_debug.cpu#11.nr_load_updates 270996 ± 1% +229.9% 894099 ± 14% sched_debug.cpu#11.avg_idle 2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#11.cpu_load[3] 185681 ± 10% -99.2% 1455 ± 30% sched_debug.cpu#11.ttwu_local 873654 ± 27% -42.5% 502080 ± 0% sched_debug.cpu#110.max_idle_balance_cost 858843 ± 30% -41.1% 505597 ± 1% sched_debug.cpu#111.max_idle_balance_cost 872547 ± 28% -42.7% 500133 ± 0% sched_debug.cpu#112.max_idle_balance_cost 850260 ± 28% -40.6% 505082 ± 2% sched_debug.cpu#113.max_idle_balance_cost 960366 ± 23% -47.8% 500984 ± 0% sched_debug.cpu#114.max_idle_balance_cost 869532 ± 27% -42.2% 502156 ± 0% sched_debug.cpu#115.max_idle_balance_cost 884039 ± 27% -42.9% 505044 ± 1% sched_debug.cpu#116.max_idle_balance_cost 855531 ± 30% -41.2% 502901 ± 1% sched_debug.cpu#117.max_idle_balance_cost 844778 ± 29% -40.2% 505544 ± 1% sched_debug.cpu#118.max_idle_balance_cost 867345 ± 27% -41.1% 510969 ± 4% sched_debug.cpu#119.max_idle_balance_cost 145505 ± 8% -34.3% 95650 ± 15% sched_debug.cpu#12.nr_load_updates 5 ± 44% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[1] 186685 ± 11% -99.3% 1223 ± 33% sched_debug.cpu#12.ttwu_local 4 ± 49% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[2] 280891 ± 3% +222.5% 905991 ± 14% sched_debug.cpu#12.avg_idle 3 ± 39% -100.0% 0 ± 0% sched_debug.cpu#12.cpu_load[3] 187388 ± 11% -99.3% 1223 ± 30% sched_debug.cpu#13.ttwu_local 145475 ± 8% -31.9% 99013 ± 11% sched_debug.cpu#13.nr_load_updates 546320 ± 8% -99.7% 1508 ± 21% sched_debug.cpu#13.ttwu_count 3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#13.cpu_load[1] 282818 ± 5% +212.5% 883684 ± 13% sched_debug.cpu#13.avg_idle 145427 ± 7% -34.5% 95288 ± 17% sched_debug.cpu#14.nr_load_updates 4 ± 41% -100.0% 0 ± 0% sched_debug.cpu#14.cpu_load[3] 283614 ± 6% +204.3% 862952 ± 17% sched_debug.cpu#14.avg_idle 548763 ± 10% -99.7% 1508 ± 26% sched_debug.cpu#14.ttwu_count 187787 ± 10% -99.3% 1258 ± 24% sched_debug.cpu#14.ttwu_local 4 ± 46% -100.0% 0 ± 0% sched_debug.cpu#14.cpu_load[2] 910931 ± 10% -90.9% 83254 ± 28% sched_debug.cpu#15.sched_count 401751 ± 10% -89.7% 41393 ± 28% sched_debug.cpu#15.sched_goidle 532305 ± 9% -99.3% 3824 ± 15% sched_debug.cpu#15.ttwu_count 227026 ± 15% -98.7% 2840 ± 12% sched_debug.cpu#15.ttwu_local 2 ± 46% -100.0% 0 ± 0% sched_debug.cpu#15.cpu_load[2] 907583 ± 10% -90.8% 83244 ± 28% sched_debug.cpu#15.nr_switches 147307 ± 8% -64.1% 52925 ± 18% sched_debug.cpu#15.nr_load_updates 271896 ± 28% +268.2% 1001194 ± 0% sched_debug.cpu#15.avg_idle 228547 ± 14% -99.6% 1006 ± 46% sched_debug.cpu#16.ttwu_local 146688 ± 7% -64.8% 51642 ± 18% sched_debug.cpu#16.nr_load_updates 538118 ± 10% -99.7% 1836 ± 36% sched_debug.cpu#16.ttwu_count 401589 ± 9% -89.9% 40532 ± 27% sched_debug.cpu#16.sched_goidle 912282 ± 10% -91.1% 81551 ± 27% sched_debug.cpu#16.sched_count 274129 ± 10% +258.6% 983056 ± 2% sched_debug.cpu#16.avg_idle 909937 ± 10% -91.0% 81542 ± 27% sched_debug.cpu#16.nr_switches 399818 ± 10% -90.8% 36949 ± 41% sched_debug.cpu#17.sched_goidle 251250 ± 33% +290.1% 980211 ± 2% sched_debug.cpu#17.avg_idle 146193 ± 8% -67.5% 47493 ± 30% sched_debug.cpu#17.nr_load_updates 907652 ± 10% -91.8% 74306 ± 41% sched_debug.cpu#17.sched_count 905857 ± 10% -91.8% 74297 ± 41% sched_debug.cpu#17.nr_switches 535902 ± 9% -99.7% 1645 ± 28% sched_debug.cpu#17.ttwu_count 535263 ± 9% -99.6% 2125 ± 33% sched_debug.cpu#18.ttwu_count 904084 ± 11% -93.0% 62907 ± 44% sched_debug.cpu#18.nr_switches 274939 ± 9% +266.2% 1006769 ± 2% sched_debug.cpu#18.avg_idle 226681 ± 14% -99.5% 1123 ± 40% sched_debug.cpu#18.ttwu_local 905555 ± 11% -93.1% 62916 ± 43% sched_debug.cpu#18.sched_count 146165 ± 7% -71.0% 42321 ± 30% sched_debug.cpu#18.nr_load_updates 399028 ± 10% -92.2% 31022 ± 44% sched_debug.cpu#18.sched_goidle 146308 ± 8% -71.5% 41686 ± 35% sched_debug.cpu#19.nr_load_updates 529715 ± 9% -99.6% 2212 ± 43% sched_debug.cpu#19.ttwu_count 296225 ± 10% +232.9% 986018 ± 1% sched_debug.cpu#19.avg_idle 3 ± 38% +605.3% 22 ± 29% sched_debug.cpu#2.cpu_load[2] 683586 ± 9% +1403.7% 10278920 ± 19% sched_debug.cpu#2.sched_count 3 ± 41% +460.9% 21 ± 46% sched_debug.cpu#2.cpu_load[1] 420236 ± 7% -95.2% 20376 ± 17% sched_debug.cpu#2.ttwu_count 154717 ± 17% -87.2% 19810 ± 18% sched_debug.cpu#2.ttwu_local 680385 ± 9% +1410.7% 10278585 ± 19% sched_debug.cpu#2.nr_switches 300978 ± 12% -50.5% 149090 ± 40% sched_debug.cpu#2.avg_idle 300267 ± 10% +1608.3% 5129477 ± 19% sched_debug.cpu#2.sched_goidle 226744 ± 15% -99.7% 791 ± 12% sched_debug.cpu#20.ttwu_local 2 ± 41% -100.0% 0 ± 0% sched_debug.cpu#20.cpu_load[3] 539626 ± 11% -99.7% 1714 ± 13% sched_debug.cpu#20.ttwu_count 3 ± 48% -100.0% 0 ± 0% sched_debug.cpu#20.cpu_load[1] 264020 ± 16% +277.1% 995578 ± 0% sched_debug.cpu#20.avg_idle 145629 ± 7% -72.8% 39540 ± 38% sched_debug.cpu#20.nr_load_updates 525878 ± 11% -99.6% 1905 ± 16% sched_debug.cpu#21.ttwu_count 145597 ± 8% -72.4% 40198 ± 40% sched_debug.cpu#21.nr_load_updates 6 ± 40% -100.0% 0 ± 0% sched_debug.cpu#21.load 226311 ± 17% -99.6% 954 ± 17% sched_debug.cpu#21.ttwu_local 2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#21.cpu_load[3] 282162 ± 10% +268.8% 1040716 ± 4% sched_debug.cpu#21.avg_idle 2 ± 40% -100.0% 0 ± 0% sched_debug.cpu#21.cpu_load[2] 4 ± 40% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[1] 3 ± 34% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[2] 145443 ± 7% -73.8% 38078 ± 39% sched_debug.cpu#22.nr_load_updates 2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#22.cpu_load[3] 547461 ± 10% -99.6% 2080 ± 41% sched_debug.cpu#22.ttwu_count 293095 ± 9% +242.6% 1004209 ± 2% sched_debug.cpu#22.avg_idle 538395 ± 10% -99.7% 1761 ± 30% sched_debug.cpu#23.ttwu_count 145290 ± 8% -77.3% 33007 ± 45% sched_debug.cpu#23.nr_load_updates 312631 ± 11% +217.5% 992648 ± 1% sched_debug.cpu#23.avg_idle 5 ± 42% -100.0% 0 ± 0% sched_debug.cpu#23.load 274102 ± 9% +261.8% 991658 ± 1% sched_debug.cpu#24.avg_idle 145062 ± 8% -75.6% 35376 ± 39% sched_debug.cpu#24.nr_load_updates 305640 ± 8% +215.8% 965318 ± 7% sched_debug.cpu#25.avg_idle 554385 ± 4% -99.6% 2368 ± 47% sched_debug.cpu#25.ttwu_count 146404 ± 6% -77.2% 33321 ± 47% sched_debug.cpu#25.nr_load_updates 562613 ± 11% -99.6% 2233 ± 32% sched_debug.cpu#26.ttwu_count 2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#26.cpu_load[2] 291842 ± 8% +239.5% 990850 ± 4% sched_debug.cpu#26.avg_idle 3 ± 37% -100.0% 0 ± 0% sched_debug.cpu#27.cpu_load[2] 4 ± 44% -100.0% 0 ± 0% sched_debug.cpu#27.cpu_load[1] 296953 ± 8% +234.6% 993660 ± 3% sched_debug.cpu#27.avg_idle 227387 ± 14% -99.6% 875 ± 46% sched_debug.cpu#28.ttwu_local 541565 ± 9% -99.7% 1729 ± 35% sched_debug.cpu#28.ttwu_count 287000 ± 9% +247.4% 996956 ± 1% sched_debug.cpu#28.avg_idle 292577 ± 5% +240.4% 995812 ± 9% sched_debug.cpu#29.avg_idle 2 ± 49% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[0] 2 ± 31% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[1] 3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[2] 538059 ± 11% -99.6% 1942 ± 46% sched_debug.cpu#29.ttwu_count 2 ± 40% -100.0% 0 ± 0% sched_debug.cpu#29.cpu_load[3] 858348 ± 11% +988.5% 9343383 ± 19% sched_debug.cpu#3.nr_switches 583644 ± 14% -96.9% 18374 ± 19% sched_debug.cpu#3.ttwu_count 188558 ± 11% -90.6% 17658 ± 20% sched_debug.cpu#3.ttwu_local 861080 ± 11% +985.1% 9343434 ± 19% sched_debug.cpu#3.sched_count 3 ± 28% +325.0% 14 ± 46% sched_debug.cpu#3.cpu_load[2] 283363 ± 7% -36.9% 178917 ± 5% sched_debug.cpu#3.avg_idle 3 ± 27% +366.7% 14 ± 35% sched_debug.cpu#3.cpu_load[3] 365993 ± 11% +1174.1% 4662957 ± 19% sched_debug.cpu#3.sched_goidle 272296 ± 5% +263.0% 988390 ± 3% sched_debug.cpu#30.avg_idle 886491 ± 7% -90.7% 82339 ± 23% sched_debug.cpu#30.nr_switches 193322 ± 6% -99.6% 752 ± 14% sched_debug.cpu#30.ttwu_local 890539 ± 7% -90.7% 82392 ± 23% sched_debug.cpu#30.sched_count 145061 ± 7% -66.4% 48735 ± 20% sched_debug.cpu#30.nr_load_updates 385340 ± 7% -89.4% 40937 ± 23% sched_debug.cpu#30.sched_goidle 558855 ± 11% -99.7% 1486 ± 14% sched_debug.cpu#30.ttwu_count 607204 ± 10% -99.7% 1669 ± 28% sched_debug.cpu#31.ttwu_count 948582 ± 13% -92.4% 72412 ± 23% sched_debug.cpu#31.sched_count 147499 ± 6% -70.2% 44027 ± 20% sched_debug.cpu#31.nr_load_updates 231115 ± 34% -99.7% 707 ± 17% sched_debug.cpu#31.ttwu_local 414152 ± 16% -91.3% 35947 ± 23% sched_debug.cpu#31.sched_goidle 945517 ± 13% -92.3% 72360 ± 23% sched_debug.cpu#31.nr_switches 269881 ± 11% +260.8% 973666 ± 4% sched_debug.cpu#31.avg_idle 146258 ± 6% -72.3% 40536 ± 31% sched_debug.cpu#32.nr_load_updates 212916 ± 19% -99.7% 682 ± 15% sched_debug.cpu#32.ttwu_local 616801 ± 13% -99.8% 1439 ± 11% sched_debug.cpu#32.ttwu_count 886655 ± 10% -92.6% 65901 ± 36% sched_debug.cpu#32.sched_count 884019 ± 10% -92.6% 65851 ± 36% sched_debug.cpu#32.nr_switches 287282 ± 4% +221.9% 924747 ± 3% sched_debug.cpu#32.avg_idle 386416 ± 11% -91.5% 32707 ± 36% sched_debug.cpu#32.sched_goidle 381805 ± 6% -92.2% 29619 ± 35% sched_debug.cpu#33.sched_goidle 882037 ± 6% -93.2% 59641 ± 35% sched_debug.cpu#33.sched_count 264381 ± 4% +251.9% 930329 ± 7% sched_debug.cpu#33.avg_idle 580711 ± 13% -99.8% 1425 ± 8% sched_debug.cpu#33.ttwu_count 2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#33.cpu_load[2] 198746 ± 7% -99.7% 677 ± 13% sched_debug.cpu#33.ttwu_local 879687 ± 6% -93.2% 59612 ± 35% sched_debug.cpu#33.nr_switches 145237 ± 7% -73.9% 37912 ± 30% sched_debug.cpu#33.nr_load_updates 558432 ± 12% -99.7% 1937 ± 40% sched_debug.cpu#34.ttwu_count 271737 ± 6% +267.3% 998140 ± 8% sched_debug.cpu#34.avg_idle 144986 ± 7% -74.4% 37118 ± 42% sched_debug.cpu#34.nr_load_updates 192853 ± 6% -99.6% 781 ± 15% sched_debug.cpu#34.ttwu_local 255816 ± 24% +287.7% 991833 ± 1% sched_debug.cpu#35.avg_idle 144906 ± 7% -74.4% 37168 ± 35% sched_debug.cpu#35.nr_load_updates 554138 ± 12% -99.7% 1813 ± 9% sched_debug.cpu#35.ttwu_count 379077 ± 6% -92.2% 29594 ± 42% sched_debug.cpu#35.sched_goidle 879048 ± 7% -93.2% 59608 ± 42% sched_debug.cpu#35.sched_count 876729 ± 7% -93.2% 59559 ± 42% sched_debug.cpu#35.nr_switches 192396 ± 6% -99.6% 796 ± 19% sched_debug.cpu#35.ttwu_local 144811 ± 7% -76.8% 33645 ± 44% sched_debug.cpu#36.nr_load_updates 198260 ± 3% -99.7% 681 ± 15% sched_debug.cpu#36.ttwu_local 3 ± 38% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[2] 608120 ± 19% -99.7% 1607 ± 10% sched_debug.cpu#36.ttwu_count 272324 ± 10% +271.7% 1012309 ± 4% sched_debug.cpu#36.avg_idle 4 ± 47% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[0] 3 ± 48% -100.0% 0 ± 0% sched_debug.cpu#36.cpu_load[1] 144880 ± 8% -75.5% 35491 ± 42% sched_debug.cpu#37.nr_load_updates 551377 ± 10% -99.7% 1487 ± 10% sched_debug.cpu#37.ttwu_count 193906 ± 7% -99.6% 680 ± 18% sched_debug.cpu#37.ttwu_local 277677 ± 3% +256.2% 989065 ± 1% sched_debug.cpu#37.avg_idle 4 ± 49% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[1] 271911 ± 6% +259.7% 978143 ± 1% sched_debug.cpu#38.avg_idle 4 ± 37% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[2] 588979 ± 20% -99.7% 1876 ± 49% sched_debug.cpu#38.ttwu_count 195832 ± 6% -99.7% 662 ± 13% sched_debug.cpu#38.ttwu_local 3 ± 42% -100.0% 0 ± 0% sched_debug.cpu#38.cpu_load[3] 859499 ± 7% -94.2% 49501 ± 48% sched_debug.cpu#39.nr_switches 197080 ± 5% -99.7% 616 ± 14% sched_debug.cpu#39.ttwu_local 144875 ± 8% -78.5% 31180 ± 38% sched_debug.cpu#39.nr_load_updates 861583 ± 7% -94.2% 50175 ± 46% sched_debug.cpu#39.sched_count 372809 ± 7% -93.4% 24588 ± 48% sched_debug.cpu#39.sched_goidle 585682 ± 15% -99.8% 1422 ± 11% sched_debug.cpu#39.ttwu_count 254598 ± 28% +291.0% 995522 ± 0% sched_debug.cpu#39.avg_idle 384496 ± 7% +624.1% 2784133 ± 20% sched_debug.cpu#4.sched_goidle 903009 ± 6% +517.8% 5579241 ± 20% sched_debug.cpu#4.sched_count 900237 ± 6% +519.7% 5579190 ± 20% sched_debug.cpu#4.nr_switches 556589 ± 9% -97.9% 11720 ± 21% sched_debug.cpu#4.ttwu_count 193650 ± 19% -94.2% 11155 ± 20% sched_debug.cpu#4.ttwu_local 279229 ± 6% +261.3% 1008863 ± 1% sched_debug.cpu#40.avg_idle 564514 ± 13% -99.8% 1408 ± 18% sched_debug.cpu#40.ttwu_count 2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#40.cpu_load[0] 2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#40.cpu_load[4] 196449 ± 8% -99.7% 599 ± 25% sched_debug.cpu#40.ttwu_local 195020 ± 4% -99.7% 665 ± 32% sched_debug.cpu#41.ttwu_local 569391 ± 15% -99.7% 1443 ± 22% sched_debug.cpu#41.ttwu_count 250174 ± 24% +299.0% 998144 ± 0% sched_debug.cpu#41.avg_idle 7 ± 46% -100.0% 0 ± 0% sched_debug.cpu#41.load 271444 ± 4% +267.2% 996765 ± 3% sched_debug.cpu#42.avg_idle 3 ± 35% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[2] 4 ± 43% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[1] 3 ± 27% -100.0% 0 ± 0% sched_debug.cpu#42.cpu_load[3] 594265 ± 15% -99.8% 1417 ± 26% sched_debug.cpu#42.ttwu_count 198649 ± 7% -99.7% 578 ± 29% sched_debug.cpu#42.ttwu_local 270822 ± 9% +262.7% 982391 ± 2% sched_debug.cpu#43.avg_idle 3 ± 14% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[2] 595801 ± 15% -99.8% 1474 ± 35% sched_debug.cpu#43.ttwu_count 2 ± 24% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[3] 199411 ± 4% -99.7% 613 ± 22% sched_debug.cpu#43.ttwu_local 4 ± 25% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[1] 5 ± 23% -100.0% 0 ± 0% sched_debug.cpu#43.cpu_load[0] 144742 ± 8% -79.2% 30152 ± 47% sched_debug.cpu#43.nr_load_updates 276311 ± 3% +258.0% 989070 ± 2% sched_debug.cpu#44.avg_idle 582806 ± 15% -99.8% 1362 ± 30% sched_debug.cpu#44.ttwu_count 144761 ± 8% -80.1% 28756 ± 45% sched_debug.cpu#44.nr_load_updates 144710 ± 7% -67.6% 46849 ± 27% sched_debug.cpu#45.nr_load_updates 851364 ± 13% -90.0% 85464 ± 31% sched_debug.cpu#45.sched_count 847235 ± 13% -89.9% 85423 ± 31% sched_debug.cpu#45.nr_switches 372059 ± 14% -88.6% 42469 ± 31% sched_debug.cpu#45.sched_goidle 203310 ± 19% -99.6% 843 ± 17% sched_debug.cpu#45.ttwu_local 281126 ± 7% +253.4% 993587 ± 5% sched_debug.cpu#45.avg_idle 573414 ± 12% -99.7% 1515 ± 21% sched_debug.cpu#45.ttwu_count 861853 ± 12% -92.4% 65635 ± 43% sched_debug.cpu#46.nr_switches 273192 ± 7% +268.5% 1006815 ± 2% sched_debug.cpu#46.avg_idle 864903 ± 12% -92.4% 65678 ± 43% sched_debug.cpu#46.sched_count 376696 ± 13% -91.3% 32592 ± 43% sched_debug.cpu#46.sched_goidle 552783 ± 9% -99.7% 1450 ± 14% sched_debug.cpu#46.ttwu_count 201314 ± 20% -99.6% 706 ± 19% sched_debug.cpu#46.ttwu_local 144703 ± 7% -74.6% 36762 ± 34% sched_debug.cpu#46.nr_load_updates 589 ± 34% -100.0% 0 ± 0% sched_debug.cpu#46.curr->pid 584555 ± 7% -99.7% 1599 ± 16% sched_debug.cpu#47.ttwu_count 203884 ± 19% -99.6% 723 ± 23% sched_debug.cpu#47.ttwu_local 144799 ± 7% -76.4% 34218 ± 46% sched_debug.cpu#47.nr_load_updates 274538 ± 6% +260.1% 988726 ± 1% sched_debug.cpu#47.avg_idle 3 ± 35% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[1] 144673 ± 8% -78.2% 31538 ± 48% sched_debug.cpu#48.nr_load_updates 543724 ± 4% -99.7% 1672 ± 18% sched_debug.cpu#48.ttwu_count 199514 ± 22% -99.7% 684 ± 34% sched_debug.cpu#48.ttwu_local 2 ± 37% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[2] 3 ± 44% -100.0% 0 ± 0% sched_debug.cpu#48.cpu_load[0] 280359 ± 8% +263.5% 1019216 ± 4% sched_debug.cpu#48.avg_idle 564757 ± 6% -99.7% 1635 ± 18% sched_debug.cpu#49.ttwu_count 203456 ± 21% -99.7% 710 ± 29% sched_debug.cpu#49.ttwu_local 144637 ± 7% -78.9% 30527 ± 47% sched_debug.cpu#49.nr_load_updates 302435 ± 19% +233.8% 1009483 ± 3% sched_debug.cpu#49.avg_idle 7 ± 35% -100.0% 0 ± 0% sched_debug.cpu#5.load 576004 ± 11% -99.5% 2984 ± 18% sched_debug.cpu#5.ttwu_count 184101 ± 10% -98.6% 2497 ± 17% sched_debug.cpu#5.ttwu_local 256732 ± 5% +131.1% 593404 ± 27% sched_debug.cpu#5.avg_idle 2 ± 49% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[3] 200344 ± 20% -99.7% 688 ± 28% sched_debug.cpu#50.ttwu_local 2 ± 31% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[2] 563146 ± 5% -99.7% 1908 ± 44% sched_debug.cpu#50.ttwu_count 2 ± 41% -100.0% 0 ± 0% sched_debug.cpu#50.cpu_load[1] 244904 ± 22% +302.4% 985599 ± 1% sched_debug.cpu#50.avg_idle 144490 ± 7% -79.9% 29035 ± 47% sched_debug.cpu#51.nr_load_updates 273633 ± 7% +265.2% 999259 ± 2% sched_debug.cpu#51.avg_idle 555030 ± 5% -99.7% 1398 ± 23% sched_debug.cpu#51.ttwu_count 202800 ± 21% -99.7% 617 ± 28% sched_debug.cpu#51.ttwu_local 204402 ± 20% -99.7% 672 ± 35% sched_debug.cpu#52.ttwu_local 6 ± 37% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[2] 266105 ± 8% +268.6% 980753 ± 3% sched_debug.cpu#52.avg_idle 8 ± 40% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[1] 4 ± 33% -100.0% 0 ± 0% sched_debug.cpu#52.cpu_load[3] 590313 ± 10% -99.8% 1394 ± 15% sched_debug.cpu#52.ttwu_count 246340 ± 20% +293.7% 969729 ± 5% sched_debug.cpu#53.avg_idle 580316 ± 10% -99.7% 1479 ± 17% sched_debug.cpu#53.ttwu_count 202868 ± 20% -99.7% 559 ± 24% sched_debug.cpu#53.ttwu_local 275898 ± 8% +251.8% 970687 ± 6% sched_debug.cpu#54.avg_idle 580835 ± 12% -99.8% 1420 ± 27% sched_debug.cpu#54.ttwu_count 203860 ± 21% -99.7% 669 ± 25% sched_debug.cpu#54.ttwu_local 561511 ± 12% -99.8% 1320 ± 32% sched_debug.cpu#55.ttwu_count 199859 ± 21% -99.7% 615 ± 44% sched_debug.cpu#55.ttwu_local 269166 ± 3% +267.9% 990163 ± 1% sched_debug.cpu#55.avg_idle 592219 ± 10% -99.8% 1394 ± 20% sched_debug.cpu#56.ttwu_count 203190 ± 20% -99.7% 622 ± 21% sched_debug.cpu#56.ttwu_local 285148 ± 8% +244.9% 983467 ± 2% sched_debug.cpu#56.avg_idle 289395 ± 8% +243.8% 994802 ± 1% sched_debug.cpu#57.avg_idle 201174 ± 22% -99.7% 588 ± 32% sched_debug.cpu#57.ttwu_local 561875 ± 9% -99.8% 1333 ± 32% sched_debug.cpu#57.ttwu_count 532410 ± 13% -99.8% 1321 ± 14% sched_debug.cpu#58.ttwu_count 199150 ± 23% -99.7% 623 ± 30% sched_debug.cpu#58.ttwu_local 281144 ± 9% +248.6% 979942 ± 4% sched_debug.cpu#58.avg_idle 206191 ± 20% -99.7% 576 ± 34% sched_debug.cpu#59.ttwu_local 290618 ± 4% +237.8% 981621 ± 2% sched_debug.cpu#59.avg_idle 568974 ± 8% -99.8% 1378 ± 21% sched_debug.cpu#59.ttwu_count 145823 ± 7% -17.5% 120332 ± 3% sched_debug.cpu#6.nr_load_updates 179817 ± 10% -99.1% 1693 ± 19% sched_debug.cpu#6.ttwu_local 546345 ± 8% -99.6% 2115 ± 21% sched_debug.cpu#6.ttwu_count 278607 ± 3% +199.1% 833241 ± 18% sched_debug.cpu#6.avg_idle 2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#6.cpu_load[3] 851921 ± 8% -40.4% 508151 ± 46% sched_debug.cpu#6.nr_switches 854405 ± 8% -40.5% 508206 ± 46% sched_debug.cpu#6.sched_count 1 ± 35% -100.0% 0 ± 0% sched_debug.cpu#6.cpu_load[4] 137215 ± 8% -92.7% 9982 ± 37% sched_debug.cpu#60.nr_load_updates 916934 ± 7% -39.4% 555767 ± 14% sched_debug.cpu#60.max_idle_balance_cost 496648 ± 21% +124.0% 1112535 ± 14% sched_debug.cpu#60.avg_idle 523174 ± 17% +90.8% 997981 ± 1% sched_debug.cpu#61.avg_idle 207882 ± 14% -99.9% 283 ± 41% sched_debug.cpu#61.ttwu_local 925690 ± 4% -45.3% 506045 ± 2% sched_debug.cpu#61.max_idle_balance_cost 303435 ± 20% -99.8% 571 ± 21% sched_debug.cpu#61.ttwu_count 203648 ± 13% -99.9% 158 ± 29% sched_debug.cpu#62.ttwu_local 904843 ± 5% -44.4% 502645 ± 1% sched_debug.cpu#62.max_idle_balance_cost 305661 ± 24% -99.9% 261 ± 30% sched_debug.cpu#62.ttwu_count 529 ± 32% -100.0% 0 ± 0% sched_debug.cpu#62.curr->pid 1080713 ± 10% -53.7% 500459 ± 0% sched_debug.cpu#63.max_idle_balance_cost 1699969 ± 10% -41.0% 1003536 ± 2% sched_debug.cpu#64.avg_idle 1096675 ± 9% -54.0% 504808 ± 2% sched_debug.cpu#64.max_idle_balance_cost 2374 ± 48% -91.1% 211 ± 27% sched_debug.cpu#64.ttwu_count 1073352 ± 9% -51.1% 524554 ± 7% sched_debug.cpu#65.max_idle_balance_cost 1599502 ± 9% -34.4% 1049932 ± 7% sched_debug.cpu#65.avg_idle 39322 ± 42% -72.4% 10848 ± 40% sched_debug.cpu#66.nr_load_updates 1409179 ± 19% -27.9% 1015680 ± 3% sched_debug.cpu#66.avg_idle 1076703 ± 7% -52.6% 510769 ± 4% sched_debug.cpu#66.max_idle_balance_cost 1086785 ± 9% -53.0% 510290 ± 4% sched_debug.cpu#67.max_idle_balance_cost 1411451 ± 16% -29.9% 989837 ± 2% sched_debug.cpu#68.avg_idle 38684 ± 43% -66.9% 12790 ± 34% sched_debug.cpu#68.nr_load_updates 1108217 ± 7% -54.9% 500229 ± 0% sched_debug.cpu#68.max_idle_balance_cost 1077891 ± 7% -47.7% 563733 ± 22% sched_debug.cpu#69.max_idle_balance_cost 566207 ± 7% -99.7% 1748 ± 21% sched_debug.cpu#7.ttwu_count 262075 ± 9% +236.5% 881956 ± 13% sched_debug.cpu#7.avg_idle 838568 ± 9% -53.9% 386340 ± 44% sched_debug.cpu#7.sched_count 357789 ± 9% -46.2% 192547 ± 44% sched_debug.cpu#7.sched_goidle 836222 ± 9% -53.8% 386289 ± 44% sched_debug.cpu#7.nr_switches 145736 ± 7% -22.7% 112707 ± 3% sched_debug.cpu#7.nr_load_updates 182859 ± 10% -99.2% 1443 ± 23% sched_debug.cpu#7.ttwu_local 1073432 ± 10% -51.3% 522486 ± 8% sched_debug.cpu#70.max_idle_balance_cost 1530831 ± 15% -35.2% 992013 ± 1% sched_debug.cpu#71.avg_idle 1054705 ± 6% -52.6% 500000 ± 0% sched_debug.cpu#71.max_idle_balance_cost 1069098 ± 9% -50.7% 527311 ± 9% sched_debug.cpu#72.max_idle_balance_cost 38445 ± 45% -61.5% 14811 ± 40% sched_debug.cpu#72.nr_load_updates 1356913 ± 13% -22.4% 1053465 ± 9% sched_debug.cpu#72.avg_idle 1332336 ± 10% -26.8% 975029 ± 6% sched_debug.cpu#73.avg_idle 1058495 ± 8% -51.8% 510063 ± 3% sched_debug.cpu#73.max_idle_balance_cost 2 ± 47% -100.0% 0 ± 0% sched_debug.cpu#73.cpu_load[1] 1067890 ± 8% -52.7% 504615 ± 1% sched_debug.cpu#74.max_idle_balance_cost 682497 ± 27% -26.7% 500478 ± 0% sched_debug.cpu#76.max_idle_balance_cost 693411 ± 28% -27.4% 503241 ± 1% sched_debug.cpu#77.max_idle_balance_cost 734302 ± 27% -30.9% 507513 ± 2% sched_debug.cpu#78.max_idle_balance_cost 681179 ± 28% -21.9% 531938 ± 10% sched_debug.cpu#79.max_idle_balance_cost 540200 ± 8% -99.7% 1700 ± 13% sched_debug.cpu#8.ttwu_count 178504 ± 11% -99.2% 1373 ± 19% sched_debug.cpu#8.ttwu_local 145742 ± 7% -25.5% 108570 ± 3% sched_debug.cpu#8.nr_load_updates 361282 ± 8% -51.3% 176052 ± 43% sched_debug.cpu#8.sched_goidle 276970 ± 8% +230.5% 915429 ± 10% sched_debug.cpu#8.avg_idle 846491 ± 7% -58.3% 353331 ± 43% sched_debug.cpu#8.sched_count 843911 ± 7% -58.1% 353246 ± 43% sched_debug.cpu#8.nr_switches 1359 ± 48% -59.4% 552 ± 9% sched_debug.cpu#81.ttwu_count 682840 ± 27% -26.2% 503819 ± 1% sched_debug.cpu#81.max_idle_balance_cost 680252 ± 27% -24.4% 513984 ± 4% sched_debug.cpu#82.max_idle_balance_cost 688877 ± 33% -26.0% 509483 ± 3% sched_debug.cpu#83.max_idle_balance_cost 670276 ± 22% -24.9% 503665 ± 1% sched_debug.cpu#85.max_idle_balance_cost 680305 ± 22% -25.8% 504652 ± 1% sched_debug.cpu#86.max_idle_balance_cost 1390 ± 44% -65.2% 483 ± 6% sched_debug.cpu#87.ttwu_count 711200 ± 26% -29.2% 503212 ± 1% sched_debug.cpu#88.max_idle_balance_cost 649693 ± 26% -22.7% 502075 ± 0% sched_debug.cpu#89.max_idle_balance_cost 531933 ± 7% -99.7% 1700 ± 30% sched_debug.cpu#9.ttwu_count 180813 ± 10% -99.2% 1424 ± 27% sched_debug.cpu#9.ttwu_local 855358 ± 8% -58.0% 359635 ± 48% sched_debug.cpu#9.nr_switches 857752 ± 8% -58.1% 359679 ± 48% sched_debug.cpu#9.sched_count 366428 ± 8% -51.1% 179269 ± 48% sched_debug.cpu#9.sched_goidle 255284 ± 21% +256.0% 908885 ± 12% sched_debug.cpu#9.avg_idle 145661 ± 7% -28.0% 104899 ± 5% sched_debug.cpu#9.nr_load_updates 3 ± 33% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[2] 2 ± 38% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[3] 3 ± 40% -100.0% 0 ± 0% sched_debug.cpu#9.cpu_load[1] 880077 ± 25% -43.2% 500000 ± 0% sched_debug.cpu#90.max_idle_balance_cost 1270327 ± 11% -21.3% 1000000 ± 0% sched_debug.cpu#91.avg_idle 886286 ± 22% -43.6% 500000 ± 0% sched_debug.cpu#91.max_idle_balance_cost 861842 ± 25% -42.0% 500000 ± 0% sched_debug.cpu#92.max_idle_balance_cost 877600 ± 26% -37.3% 550542 ± 20% sched_debug.cpu#93.max_idle_balance_cost 872812 ± 25% -42.7% 500149 ± 0% sched_debug.cpu#94.max_idle_balance_cost 880573 ± 26% -43.0% 501897 ± 0% sched_debug.cpu#95.max_idle_balance_cost 898390 ± 24% -43.6% 506730 ± 2% sched_debug.cpu#96.max_idle_balance_cost 1233750 ± 15% -17.9% 1013461 ± 2% sched_debug.cpu#96.avg_idle 1279764 ± 17% -22.4% 993620 ± 1% sched_debug.cpu#97.avg_idle 887849 ± 23% -43.7% 500000 ± 0% sched_debug.cpu#97.max_idle_balance_cost 881541 ± 22% -43.0% 502208 ± 0% sched_debug.cpu#98.max_idle_balance_cost 883063 ± 25% -43.1% 502857 ± 1% sched_debug.cpu#99.max_idle_balance_cost brickland3: Brickland Ivy Bridge-EX Memory: 512G stderr.netperf:send_omni:recv_data_failed:No_route_to_host 1 ++----------------*-----*--------*------*-----------------------------* | : : : : | | : : :: : | 0.8 ++ : : : : : : : : :| | : : : : : : : : :| | : : : : : : : : :| 0.6 ++ : : : : : : : : :| | : : : : : : : : :| 0.4 ++ : : : : : : : : : | | : : : : : : : : : | | : : : : : : : : : | 0.2 ++ : : : : : : : : : | | : : : : :: : : | | : : : : : : : | 0 *+-*--*--*--*--*-----*-----*--*------*-----*--*--*--*--*--*--*--*--*--+ last_state.booting 1 ++------------------------------------------------------*-------------* | : | | :: | 0.8 ++ : : :| | : : :| | : : :| 0.6 ++ : : :| | : : :| 0.4 ++ : : : | | : : : | | : : : | 0.2 ++ : : : | | : : : | | : : : | 0 *+--*--*---*--*---*--*---*--*---*--*---*--*---*--*---*------*--*---*--+ softirqs.TIMER 6e+06 ++------------------------------------------------------------------+ | | 5e+06 ++ * | | .*. +: | |.* *.*.*. .* * * *.* *.* *.* : .*.*.*.*.*.*.* 4e+06 *+ * : : : : : : : * : : * | | : : :: : : : : : : : : | 3e+06 ++ : : : : : : : : : :: : : : | | : : : : : : : : : : : : :: | 2e+06 ++ : : : : : : : : : : : : :: | | : : : : : : : : : : :: | | *: : * : : : : * :: | 1e+06 O+ O O O O O O O O O : O O : O O O OO O O O O | | O O O* O * : : O : : O * | 0 ++----------O------------O---O-*-O-----*-O--------------------------+ softirqs.NET_RX 7e+07 ++------------------------------------------------------------------+ | | 6e+07 *+*.*.*.*.* * * * *.* *.* * *.*.* *.*.*.*.*.*.*.* | : :: : : : : : : : : : : | 5e+07 ++ : :: : : : : : : : : : : | | * : :: :: : : : : :: : : : | 4e+07 ++ : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : | 3e+07 ++ : : : : : : : : : : : : :: | | : : : : : : : : : : : :: | 2e+07 ++ : : : : : : : : : : :: | | : : :: * : : : : : :: | 1e+07 ++ *: : : : : : * : | | : : : : : : : | 0 O+O-O-O-O-O-O-O-OO-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-O-OO-O-O-O-O-------+ softirqs.SCHED 4.5e+06 ++-----------------------------------------------*----------------+ | :: | 4e+06 ++ .*. * * :: | 3.5e+06 ++* *.* : * : * *.* *.* * *.* : *.**.*.*.*.* * : +: : : : : : : : : : + | 3e+06 ++ * : :: :: : : : : : : : * | 2.5e+06 ++ : : : : : : : : : :: : : : | | : : : : : : : : : :: : : : | 2e+06 ++ : : : : :: : : : :: : : : | 1.5e+06 ++ : : : : : : : : : : : : : | | : : : : : : : : : : : : | 1e+06 ++ : : :: * : : : : : :: | 500000 ++ * : : : : : : * : | O O OO O +O O O O O : O O :O O O O O O O OO | 0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+ softirqs.RCU 1.8e+06 ++-----------------------------------------------*----------------+ | .** * .* .* +: .*. | 1.6e+06 *+ .*.* + .* : * * : * : * *.* : * **.*.*.*.* 1.4e+06 ++* * : : : : : : : : : : : | | : :: :: : : : : : : : : | 1.2e+06 ++ : : : : : : : : : :: : : : | 1e+06 ++ : : : : : : : : : :: : : : | | : : : : :: : : : :: : : : | 800000 ++ : : : : : : : : :: : : : | 600000 ++ : : : : : : : : : : : : | | : : :: * : : : : : :: | 400000 ++ * : : : : : : * : | 200000 O+ O OO O +O O O O O : O O :O O O O O O OO | | O O * * : : O : : O O * | 0 ++----O----O-------O-----O--O-*-O-----*-O-------------------------+ cpuidle.C1-IVT-4S.usage 4e+07 ++----------------------------------------------------------------+ O O OO O O O O O O O O O O O O O O | 3.5e+07 ++ O O O | 3e+07 ++ | | | 2.5e+07 ++ | *.*.*.*.* * * * *.* *.* * *.*. *. .**.*. .*.* 2e+07 ++ *. +: : : : : : : : : * : * * | | * : :: :: : : : : :: : : : | 1.5e+07 ++ : : : : : : : : : :: : : : | 1e+07 ++ : : : : :: : : : :: : : : | | : : : : : : : : : : : : : | 5e+06 ++ : : :: * : : : : : : : | | *.: : : : : : * : | 0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+ cpuidle.C6-IVT-4S.time 1.2e+11 ++----------------------------------------------------------------+ | | 1e+11 ++ * * | | : : | | : : * | 8e+10 ++ : : :: : | | : : :: :: | 6e+10 ++ : : * : : : : | | : : : : : : : | 4e+10 ++ : : : : : : : : | O O OO O: :O O O O O O O O: O O O O O O OO | | : : : : : : : | 2e+10 *+*.*.*.** * :.*.*.* *.* *.* * *.*.* *.*.**.*.*.*.* | O * O + + + : O + + | 0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+ turbostat.Avg_MHz 1800 ++-------------------------------------------------*-----*-----------+ |.*. .*.*.* * * * .* .* * *. +: + *.*. .*.*.| 1600 *+ * : : : : * : * : : : * : * * * 1400 ++ : : : : : : : : : : : : | | : : : : :: : : : : :: : : : | 1200 ++ : : : : : : : : : : : : : : : : | 1000 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 800 ++ : : : : : : : : : : : : : : : : | 600 ++ : : : : : : : : : : : : : : : | | : : : : : :: : : : : :: :: | 400 ++ * : : : : : : : : : : | 200 ++ : : : : : : : : : : | O O O O O *.O O O * O O :O O : O * O O O O O O O O | 0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-----------------------+ turbostat.%Busy 60 ++---------------------------------------------------------------------+ | .*.*.* * * .*. | 50 *+*.* : * : * .* .* * *. +: * *.*. .*.*.| | : : : : * : * : : : * : : * * | : : : : : : : : : : : : | 40 ++ : : : : : :: : : : : :: : : : | | : : : : : : : : : : : : : : : : | 30 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 20 ++ : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : :: | | * : : :: : : : : : : : | 10 ++ : : : : : : : : : : | O O O O O *.O O O * O O : O O : O * O O O O O O O O | 0 ++O---O-----O---O-*-O-*---O---O-*-O----O-*-O---O-----------------------+ turbostat.CPU%c1 50 *+-------------------------------------------------------*-------------+ 45 ++* *.*.* * * * .* .* * *. : *.*.*. .*.*.| | + + : : : : * : * : : : *.* : * * 40 ++ * : : : : : : : : : : : : | 35 ++ : : : : :: : : : : :: : : : | | : : : : : : : : : : : : : : : : | 30 ++ : : : : : : : : : : : : : : : : | 25 ++ : : : : : : : : : : : : : : : : | 20 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : O : : : : : O | 15 O+ O O O : O : O: :O:O :: : : : : :: O O | 10 ++ * : : : : O O : O : O : O O O * O O | | : : : : : : : : : | 5 ++ *.* : * : : O : : O | 0 ++O---O-----O---O---O-*---O---O-*-O------*-O---------------------------+ turbostat.CPU%c6 100 ++O---O---------O---O-*-----------------------O-----------------------+ 90 ++ .* : * O * | O O O O O * O O O : O O O O : O O O * O O | 80 ++ * : : : : O : O O O | 70 ++ : : : : : :: : : | | : : : : : : : : : :: | 60 ++ : : : : : : : : : : : : | 50 ++ : : : : : : : : : : : : | 40 ++ : : : : : : : : : : : : | | : : : : : : : : : : : : | 30 ++ : : : : : : : : : : : : | 20 ++ : : : : : : : : : : | | : : : : : : : : : | 10 ++ .*. : : : : *. .*. : :.*.: : .*. .* 0 *+*---*-*-*-O-*-----*---*-O---O-*-O---*-*-O-*---*---*---*-*-*-*---*-*-+ turbostat.CoreTmp 60 ++---------------------------------------------------------------------+ *.*.*.*.*.* * * * .* .* *.*.* .*.*. | 50 ++ : : : : * : * : * : : *.* *.*.*.* O O O : :O: O: :O: : : : O : : :O : : | | O: : : : : :O: :O : O : : O O: : O O:O O O O | 40 ++ : : : : : : : : : :: : * | | O O * O.* O * : : : O: : O O | 30 ++ * * : : : : * | | : : : : | 20 ++ : : : : | | : : : : | | : : : : | 10 ++ : : : : | | : : : : | 0 ++----------O-------------O---O-*-O------*-O---------------------------+ turbostat.PkgTmp 60 ++---------------------------------------------------------------------+ *.*.*.*.*.* * * * .* .* *.*.* .*.*. .* 50 ++ : : : : * : * : * : : * *.*.*.* | | O O : : : O: :O: : : : O : : :O : :O | O O: :O: : : :O: :O : O : : O O: : O O: O O O | 40 ++ : : : : : : : : : :: : * | | O O * O.* O * : : : O: : O O | 30 ++ * * : : : : * | | : : : : | 20 ++ : : : : | | : : : : | | : : : : | 10 ++ : : : : | | : : : : | 0 ++----------O-------------O---O-*-O------*-O---------------------------+ turbostat.Pkg%pc2 70 ++---------------------------------------------------------------------+ | O | 60 ++ O O O O | O O O O O O O | 50 ++ O O O O O O O | | O | 40 ++ O | | O O O O * O | 30 ++ .* : O * | | * * : : : * * : | 20 ++ : : : : : :: : : : | | : : : : : : : : : : : : | 10 ++ : : : : : : : : : : : : | | .*. : : : : *. .*. : :.*.: : .*. | 0 *+*---*-*-*-O-*-----*---*-O---O-*-O----*-*-O-*---*---*---*-*-*-*---*-*-* turbostat.PkgWatt 450 *+*---*-*-*---------*---*-----*-------*-------------------*-------*---+ | * : * : : * : * : * *.*.* * *.*.* *.* 400 ++ : : : : : : : : : : : : | 350 ++ : : :: :: : : : : : : : : | | : : : : : : : : : : : :: : : : | 300 ++ : : : : : : : : : : : : : : : : | 250 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 200 ++ : : : : : : : : : : : : : : | 150 O+ O O O : O : O: O:O ::O :O :O :O :O: :O O O O O O O O | | * : : : : : : : : : * | 100 ++ *.: : : : : O : : : | 50 ++O O O * O * * : : : : O | | : : : : | 0 ++----------O-------------O---O-*-O-----*-O---------------------------+ turbostat.CorWatt 400 ++--------------------------------------------------------------------+ *.*. .*.*.* * * * .* .* * *. .* *.*.*.*. .*.*.| 350 ++ * : : : : * : * : : : * : : * * 300 ++ : : : : : : : : : : : : | | : : : :: :: : : : : :: : : : | 250 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 200 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 150 ++ : : : : : : : : : : : : : : | 100 O+ O O : O : O: O: : : :O : :O : O O O | | O * : : : O : O O : O : : O O O * O O | 50 ++ *.: : * : : O : : * | | O O O * O * : : : : O | 0 ++----------O-------------O---O-*-O-----*-O---------------------------+ netperf.Throughput_tps 1600 ++-------------------------------------------------------------------+ *.*.*.*.*.* * * * *.* *.* *.*.* *.*.*. .*.*. .* 1400 ++ : : : : : : : : * : : : * * | 1200 ++ : : : : : : : : : : : : | | : : : : : : : : : : : : | 1000 O+ O O O: :O: O: :O:O: :O :O O :O O: :O O O:O:O O O O | | : : : : : : : : : : : : : : : : | 800 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 600 ++ : : : : : : : : : : : : : : : : | 400 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : | 200 ++ : : : : : : : : : : : | | : : : : : : : : : : : | 0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+ netperf.time.user_time 1000 ++*-*----------------------------------------------------------------+ 900 ++ : * *.*.*. .*.*.| * : .*.*.* *. *. : : *.* * 800 ++ :.* : * * : * : * : : | 700 ++ * : : : : : : : :: : | | : : : : : : : : : *.*.* : | 600 ++ : : : : : : : : : : : : : : | 500 ++ : : : : : : : : : : : : : : | 400 ++ : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : | 300 ++ : : : : : : : : : :: : : | 200 ++ : : : : : : : : : : :: | | * : : * : : : : * : | 100 O+ O O O O +O O O O O :O O : O O O O O O O O O | 0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+ netperf.time.system_time 12000 ++------------------------------------------------------------------+ | .*. * .* | 10000 ++*.* *.* * : * *.* *.* *.* : .*.*.*.*.*.*.* * + +: : : : : : : * : : * | | * : : : : : : : : : : : | 8000 ++ : : : :: : : : : : : : : | | : : : : : : : : : : : : : : | 6000 ++ : : : : : : : : : : : : : : | | : : : : : : : : : : : : :: | 4000 ++ : : : : : : : : : : : :: | | : : : : : : : : : : :: | | :: :: * : : : : : :: | 2000 ++ *: : : : : : * : | O O O O O O O O O O : O O : O O O OO O O O O | 0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+ netperf.time.percent_of_cpu_this_job_got 4000 ++-------------------------------------------------*-----------------+ |.*. .*.*.* * * * .* .* *. +: .*.*.*. .*.*.| 3500 *+ * : : : : * : * : * : * : * * * 3000 ++ : : : : : : : : : : : : | | : : : : :: : : : : : : : : | 2500 ++ : : : : : : : : : : : :: : : : | | : : : : : : : : : : : : : : : : | 2000 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 1500 ++ : : : : : : : : : : : : : : : | 1000 ++ : : : : : :: : : : : : : : : | | * : : : : : : : : : :: | 500 ++ : : : : : : : : : : | O O O O O *.O O O * O O :O O : O * O O O O O O O O | 0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+ netperf.time.voluntary_context_switches 7e+07 *+*-*-*-*-*---*----*---*---*-*-----*-*---------*-*-*--*-*-*-*-*-*-*-* | : :: : : : : : : * : : : | 6e+07 ++ : :: : : : : : : : : : : | | * : : :: : : : : : : : : | 5e+07 O+ O O O O: O: :O:O: :O :O :O :O O: :O O OO: O O O | | : : : : : : : : : : : : : :O | 4e+07 ++ : : : : : : : : : : : : : : | | : : : : : : : : : : : : :: | 3e+07 ++ : : : : : : : : : : : : :: | | : : : : : : : : : : :: | 2e+07 ++ : : : : : : : : : : :: | | :: : * : : : : * : | 1e+07 ++ *: : : : : : : | | : : : : : : : | 0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+ netperf.time.involuntary_context_switches 35000 ++------------------------------------------------------------------+ | *. | 30000 ++ * + * .* | | + : * * * : * : * * 25000 ++ * : .* * : : *.* *.* : : : : + + +| | : * + +: : :: : : : : * : : : * *.* | 20000 ++ : * : : : : : : : : : : : : : | *.* : : : : : : : : : :: : : * | 15000 ++ : : : : : : : : : : : : : : | | : : : : :: : : : : : : :: | 10000 ++ : : : : : : : : : :: :: | | : : : : * : : : : : :: | 5000 ++ *: : : : : : * ::O | O O O O O O O O O O : O O : O O O OO O O O | 0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+ time.user_time 1000 ++*-*----------------------------------------------------------------+ 900 ++ : * *.*.*. .*.*.| * : .*.*.* *. *. : : *.* * 800 ++ :.* : * * : * : * : : | 700 ++ * : : : : : : : :: : | | : : : : : : : : : *.*.* : | 600 ++ : : : : : : : : : : : : : : | 500 ++ : : : : : : : : : : : : : : | 400 ++ : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : | 300 ++ : : : : : : : : : :: : : | 200 ++ : : : : : : : : : : :: | | * : : * : : : : * : | 100 O+ O O O O +O O O O O :O O : O O O O O O O O O | 0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+ time.system_time 12000 ++------------------------------------------------------------------+ | .*. * .* | 10000 ++*.* *.* * : * *.* *.* *.* : .*.*.*.*.*.*.* * + +: : : : : : : * : : * | | * : : : : : : : : : : : | 8000 ++ : : : :: : : : : : : : : | | : : : : : : : : : : : : : : | 6000 ++ : : : : : : : : : : : : : : | | : : : : : : : : : : : : :: | 4000 ++ : : : : : : : : : : : :: | | : : : : : : : : : : :: | | :: :: * : : : : : :: | 2000 ++ *: : : : : : * : | O O O O O O O O O O : O O : O O O OO O O O O | 0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+ time.percent_of_cpu_this_job_got 4000 ++-------------------------------------------------*-----------------+ |.*. .*.*.* * * * .* .* *. +: .*.*.*. .*.*.| 3500 *+ * : : : : * : * : * : * : * * * 3000 ++ : : : : : : : : : : : : | | : : : : :: : : : : : : : : | 2500 ++ : : : : : : : : : : : :: : : : | | : : : : : : : : : : : : : : : : | 2000 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 1500 ++ : : : : : : : : : : : : : : : | 1000 ++ : : : : : :: : : : : : : : : | | * : : : : : : : : : :: | 500 ++ : : : : : : : : : : | O O O O O *.O O O * O O :O O : O * O O O O O O O O | 0 ++O---O-----O---O-*-O-*---O---O-*-O--O-*-O---O-------*---------------+ time.voluntary_context_switches 7e+07 *+*-*-*-*-*---*----*---*---*-*-----*-*---------*-*-*--*-*-*-*-*-*-*-* | : :: : : : : : : * : : : | 6e+07 ++ : :: : : : : : : : : : : | | * : : :: : : : : : : : : | 5e+07 O+ O O O O: O: :O:O: :O :O :O :O O: :O O OO: O O O | | : : : : : : : : : : : : : :O | 4e+07 ++ : : : : : : : : : : : : : : | | : : : : : : : : : : : : :: | 3e+07 ++ : : : : : : : : : : : : :: | | : : : : : : : : : : :: | 2e+07 ++ : : : : : : : : : : :: | | :: : * : : : : * : | 1e+07 ++ *: : : : : : : | | : : : : : : : | 0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+ time.involuntary_context_switches 35000 ++------------------------------------------------------------------+ | *. | 30000 ++ * + * .* | | + : * * * : * : * * 25000 ++ * : .* * : : *.* *.* : : : : + + +| | : * + +: : :: : : : : * : : : * *.* | 20000 ++ : * : : : : : : : : : : : : : | *.* : : : : : : : : : :: : : * | 15000 ++ : : : : : : : : : : : : : : | | : : : : :: : : : : : : :: | 10000 ++ : : : : : : : : : :: :: | | : : : : * : : : : : :: | 5000 ++ *: : : : : : * ::O | O O O O O O O O O O : O O : O O O OO O O O | 0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+ vmstat.procs.r 50 ++----*---*------------------------------------------------------------+ | + * : * * * * * *. .* .*.*.* *. | *.*.* : : : : +: +: * : * : * + + *.* 40 ++ : : : : * : * : : : : : * | | : : : : :: : : : : : : : : | | : : : : : : : : : : : :: : : : | 30 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : : | 20 ++ : : : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : | | : : : : : :: : : : : : : :: | 10 ++ * : : : : : : : : : : | | : : : : : : : : : : | O O O O O *.O O O * O O : O O : O * O O O O O O O O | 0 ++O---O-----O---O-*-O-*---O---O-*-O----O-*-O---O-----------------------+ vmstat.system.in 500000 ++-----------------------------------------------------------------+ 450000 *+*. .*.*.* * * * .* .* * *. .* *.**.*. .*.*.| | * : : : : * : * : : : * : : * * 400000 ++ : : : : : : : : : : : : | 350000 ++ : : : : :: : : : : :: : : : | | : : : : : : : : : : : : : : : : | 300000 ++ :: : : : : : : : : : : : : : : | 250000 ++ :: : : : : : : : : : : : : : : | 200000 ++ :: : : : : : : : : : : : : : : | | : : : : : : : : : : : : : : : | 150000 ++ : : : : : :: : : : : :: : : | 100000 ++ * : : : : : : : : : : | | : : : : : : : : : : | 50000 O+ O O O O *.O O O * O O :O O : O * O O O O O OO O | 0 ++O---O----O---O-*-O-*---O---O-*-O--O-*-O---O-------*--------------+ numa-numastat.node0.numa_hit 1.6e+06 ++----------------------------------------------------------------+ | O O O O O | 1.4e+06 O+ O O O O O O O O O O O O OO | | O | 1.2e+06 ++ | 1e+06 ++ | | | 800000 ++ | | * * | 600000 ++ :: : | 400000 ++ : : : : | | .*. : :.* : *. *. .*. .*.*. | 200000 *+*.* ** * : *. .* * : * * *.*.* *.*.** *.* | : + * + : + : + + | 0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+ numa-numastat.node0.local_node 1.6e+06 ++----------------------------------------------------------------+ | O O O O O | 1.4e+06 O+ O O O O O O O O O O O O OO | | O | 1.2e+06 ++ | 1e+06 ++ | | | 800000 ++ | | * * | 600000 ++ :: : | 400000 ++ : : : : | | .*. : :.* : *. *. .*. .*.*. | 200000 *+*.* ** * : *. .* * : * * *.*.* *.*.** *.* | : + * + : + : + + | 0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+ numa-vmstat.node0.numa_hit 1e+06 ++-----------------------------------------------------------------+ 900000 ++ O | | O O O O OO | 800000 O+ O O O O O O O O O O O O | 700000 ++ O | | | 600000 ++ | 500000 ++ * * | 400000 ++ :: : : | | .*. : : : : .*. .*. | 300000 *+*.* *.* *.*. .*. .* *.* *.* * *.*.* *.**.*.* *.* 200000 ++ O * O * : : : : + + | | O O : : O: : O * | 100000 ++ : : : : | 0 ++---------O-------------O---O-*-O----*-O--------------------------+ numa-vmstat.node0.numa_local 900000 ++------------------------------------------------O----------------+ | O O O O OO | 800000 O+ O O O O O O O O O | 700000 ++ O O O O | | | 600000 ++ | 500000 ++ | | * * | 400000 ++ :: : : | 300000 ++ : : : : .* .*. | |.*.*.*.*.* *.* *. .* *.* *.* * + .*.* *.**.*.* *.* 200000 *+ + + * : : : : * + + | 100000 ++O O O * O : : O: : O * | | : : : : | 0 ++---------O-------------O---O-*-O----*-O--------------------------+ proc-vmstat.numa_hit 3e+06 ++----------------------------------------------------------------+ | * | 2.5e+06 ++ : | | : : | O O OO: :O O O O *O O O O O O O O O OO | 2e+06 ++ : : :: O O | | : : : : *. | 1.5e+06 ++ .*.*.** *. * * *.* *.* + *.*.* .*.* .*.*. .* *.* * : : : : : * : * * * | 1e+06 ++ : : : : : : : : : : | | : : : : : : : : : : | | : : : : : : : : : | 500000 ++ : * : : : : : : | | O * O : : : : O : | 0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+ proc-vmstat.numa_local 3e+06 ++----------------------------------------------------------------+ | * | 2.5e+06 ++ : | | : : | O O OO: :O O O O *O O O O O O O O O OO | 2e+06 ++ : : :: O O | | : : : : *. | 1.5e+06 ++ .*.*.** *. * * *.* *.* + *.*.* .*.* .*.*. .* *.* * : : : : : * : * * * | 1e+06 ++ : : : : : : : : : : | | : : : : : : : : : : | | : : : : : : : : : | 500000 ++ : * : : : : : : | | O * O : : : : O : | 0 ++O--------O---O---------O--O-*-O---O-*-O----------*--------------+ proc-vmstat.pgalloc_dma32 350000 ++-----------------------------------------------------------------+ O O O O O O O O O O O O | 300000 ++ O O O O O O O O | | O | 250000 ++ | | | 200000 ++ * * | *. + + + *.* * * .* .* * *.*.* .**. .*.* 150000 ++*.* * : : :: * : * : : : : * *.*.* | | : : : : : + : : : : : : : : | 100000 ++ : : : : * : : : : : : : : | | : : : : : : : : : : : | 50000 ++ * : : : : : : : * : : | | +: : : : : : : | 0 ++O---O----O---O-*-O-*---O---O-*-O--O-*-O---O-------*--------------+ proc-vmstat.pgalloc_normal 1.2e+07 ++----------------------------------------------------------------+ O O O O O O O O O O O O O O O O | 1e+07 ++ O O O O O | | | | | 8e+06 ++ | | .*. | 6e+06 *+*.*.*.** * * * *.* *.* *.*.* *.*.**.*.*.*.* | : : :: : : : : * : : : | 4e+06 ++ : : : : :: : : : :: : : : | | : : : : * : : : :: : : : | | : : : : : : : : * : : | 2e+06 ++ * : : : : : : : : : | | +: : : : : : : | 0 ++O---O----O---O-*-O-*---O--O-*-O---O-*-O--O-------*--------------+ proc-vmstat.pgfree 1.2e+07 ++------O------------O----------------O--O-----O-O-----O-O--------+ O O O O O O O O O O O O | 1e+07 ++ O | | | | | 8e+06 ++ | | .*. .*. | 6e+06 *+*.* ** * * * *.* *.* * *.*.* *.*.**.*.*.*.* | : : :: : : : : :: : : : | 4e+06 ++ : : : : :: : : : :: : : : | | : : : : * : : : : :: : : | | * : : : : : : : * : : | 2e+06 ++ : : : : : : : : : | | :: * : : : : : | 0 ++O---O----O---O-*-O-----O--O-*-O---O-*-O--O-------*--------------+ proc-vmstat.numa_hint_faults 70000 ++------------------------------------------------------------------+ | | 60000 ++ * * * | | +: +: .*.* :+ .*. | 50000 ++ .* *. * : * : * : : *.* *.*.* |.* + + *.*. * * : : : : : : : | 40000 *+ * * : :: : : : : * : : * | | : : :: : : : : :: : : : | 30000 ++ : : : : : : : : :: : : : | | : : : : * : : : : : ::O | 20000 ++ O O O *O: :O:O : :O : : * O O OO: O O | O O : : : : O : O :O :: O | 10000 ++ : : O : : : : :: | | : : : : : : : | 0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+ proc-vmstat.numa_hint_faults_local 60000 ++------------------------------------------------------------------+ | * * * | 50000 ++ +: +: :: | | .* * : * : .*.* : : .*. | | * + .*. * * : : : : * : : *.* *.*.* 40000 ++ * *.*.* : : : : : : : : : | * : : :: : : : : * : : * | 30000 ++ : : : : : : : : : : : : : | | : : : : : : : : : :: : : : | 20000 ++ : : : : : : : : : : : :: | | O O O *O: :O:O * : :O : : : O O:O O O | O O : : : : O : : O :O * O O:: O | 10000 ++ : : O : : : :: | | : : : : : : : | 0 ++O---O-----O---O*-O-*---O---O-*-O---O-*-O---O------*---------------+ [*] bisect-good sample [O] bisect-bad sample To reproduce: apt-get install ruby git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/setup-local job.yaml # the job file attached in this email bin/run-local job.yaml Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Thanks, Ying Huang [-- Attachment #2: job.yaml --] [-- Type: text/plain, Size: 3379 bytes --] --- LKP_SERVER: inn LKP_CGI_PORT: 80 testcase: netperf default-monitors: wait: pre-test uptime: iostat: vmstat: numa-numastat: numa-vmstat: numa-meminfo: proc-vmstat: proc-stat: interval: 10 meminfo: slabinfo: interrupts: lock_stat: latency_stats: softirqs: bdi_dev_mapping: diskstats: nfsstat: cpuidle: cpufreq-stats: turbostat: pmeter: sched_debug: interval: 60 default-watchdogs: watch-oom: watchdog: cpufreq_governor: performance model: Westmere-EX nr_cpu: 80 memory: 128G hdd_partitions: swap_partitions: pmeter_server: lkp-st01 pmeter_device: yokogawa-wt310 commit: 0eab523874ab9850ec14f68751758dfb275361b9 nr_hdd_partitions: 0 rootfs_partition: runtime: 300s nr_threads: 200% perf-profile: freq: 800 cluster: cs-lkp-wsx02 if role server: netserver: if role client: netperf: test: TCP_RR testbox: brickland3 tbox_group: brickland3 kconfig: x86_64-rhel enqueue_time: 2015-06-13 01:32:11.981026644 +08:00 user: lkp queue: cs compiler: gcc-4.9 head_commit: 54896f27dd5c7e83c37c85eb1093f14543af2bfa base_commit: d4a4f75cd8f29cd9464a5a32e9224a91571d6649 branch: next/master rootfs: debian-x86_64-2015-02-07.cgz result_root: "/result/netperf/performance-300s-200%-TCP_RR/brickland3/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/0" direct_macs: a0:36:9f:18:d6:aa direct_ips: 10.167.0.2 job_file: "/lkp/scheduled/brickland3/cs_netperf-performance-300s-200%-TCP_RR-debian-x86_64-2015-02-07.cgz-x86_64-rhel-0eab523874ab9850ec14f68751758dfb275361b9-0-20150613-5146-1kuhia6.yaml" max_uptime: 1500 initrd: "/osimage/debian/debian-x86_64-2015-02-07.cgz" bootloader_append: - root=/dev/ram0 - user=lkp - job=/lkp/scheduled/brickland3/cs_netperf-performance-300s-200%-TCP_RR-debian-x86_64-2015-02-07.cgz-x86_64-rhel-0eab523874ab9850ec14f68751758dfb275361b9-0-20150613-5146-1kuhia6.yaml - ARCH=x86_64 - kconfig=x86_64-rhel - branch=next/master - commit=0eab523874ab9850ec14f68751758dfb275361b9 - BOOT_IMAGE=/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/vmlinuz-4.1.0-rc7-07770-g0eab523 - max_uptime=1500 - RESULT_ROOT=/result/netperf/performance-300s-200%-TCP_RR/brickland3/debian-x86_64-2015-02-07.cgz/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/0 - LKP_SERVER=inn - |2- earlyprintk=ttyS0,115200 systemd.log_level=err debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal rw lkp_initrd: "/lkp/lkp/lkp-x86_64.cgz" modules_initrd: "/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/modules.cgz" bm_initrd: "/osimage/deps/debian-x86_64-2015-02-07.cgz/lkp.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/run-ipconfig.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/turbostat.cgz,/lkp/benchmarks/turbostat.cgz,/osimage/deps/debian-x86_64-2015-02-07.cgz/netperf.cgz,/lkp/benchmarks/netperf.cgz" node_roles: client kernel: "/pkg/linux/x86_64-rhel/gcc-4.9/0eab523874ab9850ec14f68751758dfb275361b9/vmlinuz-4.1.0-rc7-07770-g0eab523" dequeue_time: 2015-06-13 01:32:30.522650807 +08:00 job_state: finished loadavg: 2.33 6.80 4.32 1/1045 12657 start_time: '1434130591' end_time: '1434130922' version: "/lkp/lkp/.src-20150612-214517" [-- Attachment #3: reproduce --] [-- Type: text/plain, Size: 9476 bytes --] echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu10/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu100/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu101/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu102/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu103/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu104/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu105/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu106/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu107/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu108/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu109/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu11/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu110/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu111/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu112/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu113/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu114/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu115/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu116/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu117/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu118/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu119/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu12/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu13/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu14/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu15/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu16/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu17/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu18/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu19/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu20/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu21/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu22/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu23/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu24/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu25/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu26/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu27/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu28/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu29/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu30/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu31/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu32/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu33/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu34/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu35/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu36/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu37/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu38/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu39/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu4/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu40/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu41/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu42/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu43/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu44/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu45/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu46/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu47/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu48/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu49/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu5/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu50/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu51/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu52/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu53/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu54/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu55/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu56/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu57/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu58/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu59/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu6/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu60/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu61/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu62/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu63/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu64/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu65/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu66/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu67/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu68/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu69/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu7/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu70/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu71/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu72/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu73/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu74/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu75/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu76/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu77/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu78/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu79/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu8/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu80/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu81/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu82/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu83/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu84/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu85/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu86/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu87/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu88/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu89/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu9/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu90/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu91/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu92/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu93/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu94/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu95/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu96/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu97/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu98/cpufreq/scaling_governor echo performance > /sys/devices/system/cpu/cpu99/cpufreq/scaling_governor netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 netperf -t TCP_RR -c -C -l 300 -H 10.167.0.1 ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps 2015-06-15 7:02 ` [lkp] " Huang Ying @ 2015-06-15 17:52 ` Thomas Gleixner -1 siblings, 0 replies; 12+ messages in thread From: Thomas Gleixner @ 2015-06-15 17:52 UTC (permalink / raw) To: lkp [-- Attachment #1: Type: text/plain, Size: 681 bytes --] On Mon, 15 Jun 2015, Huang Ying wrote: > FYI, we noticed the below changes on > > git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master > commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") > I really appreciate this testing effort, but the information provided is not really helpful. I asked this before. Can you pretty please, upload ALL relevant information (.config, full dmesg, below stats, /proc/interrupts ...) to some place where everyone interested can download them? Then themail contains a useful link instead of 200k waste of network bandwidth. Thanks, tglx ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps @ 2015-06-15 17:52 ` Thomas Gleixner 0 siblings, 0 replies; 12+ messages in thread From: Thomas Gleixner @ 2015-06-15 17:52 UTC (permalink / raw) To: Huang Ying; +Cc: Jiang Liu, LKML, LKP ML On Mon, 15 Jun 2015, Huang Ying wrote: > FYI, we noticed the below changes on > > git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master > commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") > I really appreciate this testing effort, but the information provided is not really helpful. I asked this before. Can you pretty please, upload ALL relevant information (.config, full dmesg, below stats, /proc/interrupts ...) to some place where everyone interested can download them? Then themail contains a useful link instead of 200k waste of network bandwidth. Thanks, tglx ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps 2015-06-15 17:52 ` [lkp] " Thomas Gleixner @ 2015-06-16 2:09 ` Jiang Liu -1 siblings, 0 replies; 12+ messages in thread From: Jiang Liu @ 2015-06-16 2:09 UTC (permalink / raw) To: lkp [-- Attachment #1: Type: text/plain, Size: 1469 bytes --] On 2015/6/16 1:52, Thomas Gleixner wrote: > On Mon, 15 Jun 2015, Huang Ying wrote: > >> FYI, we noticed the below changes on >> >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") >> > > I really appreciate this testing effort, but the information provided > is not really helpful. > > I asked this before. Can you pretty please, upload ALL relevant > information (.config, full dmesg, below stats, /proc/interrupts ...) > to some place where everyone interested can download them? > > Then themail contains a useful link instead of 200k waste of network bandwidth. Hi Ying and Thomas, I guess this report discloses a regression in hierarchy irqdomain, and which should have been fixed by the patch posted at: lkml.org/lkml/2015/6/1/80 The root cause is that, with hierarchy irqdomain enabled, there are multiple irq_data associated with one irq. And function irq_move_irq() on x86 uses a wrong copy of irq_data to check whether there's pending irq migration operation. So all irq migration /set_affinity operations will get pending for ever. This may affect network performance due to interrupt load balance issue. And the patch set posted at www.gossamer-threads.com/lists/linux/kernel/2185533 should have solved all such regressions. Thanks! Gerry > > Thanks, > > tglx > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps @ 2015-06-16 2:09 ` Jiang Liu 0 siblings, 0 replies; 12+ messages in thread From: Jiang Liu @ 2015-06-16 2:09 UTC (permalink / raw) To: Thomas Gleixner, Huang Ying; +Cc: LKML, LKP ML On 2015/6/16 1:52, Thomas Gleixner wrote: > On Mon, 15 Jun 2015, Huang Ying wrote: > >> FYI, we noticed the below changes on >> >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") >> > > I really appreciate this testing effort, but the information provided > is not really helpful. > > I asked this before. Can you pretty please, upload ALL relevant > information (.config, full dmesg, below stats, /proc/interrupts ...) > to some place where everyone interested can download them? > > Then themail contains a useful link instead of 200k waste of network bandwidth. Hi Ying and Thomas, I guess this report discloses a regression in hierarchy irqdomain, and which should have been fixed by the patch posted at: lkml.org/lkml/2015/6/1/80 The root cause is that, with hierarchy irqdomain enabled, there are multiple irq_data associated with one irq. And function irq_move_irq() on x86 uses a wrong copy of irq_data to check whether there's pending irq migration operation. So all irq migration /set_affinity operations will get pending for ever. This may affect network performance due to interrupt load balance issue. And the patch set posted at www.gossamer-threads.com/lists/linux/kernel/2185533 should have solved all such regressions. Thanks! Gerry > > Thanks, > > tglx > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps 2015-06-16 2:09 ` [lkp] " Jiang Liu @ 2015-06-16 5:41 ` Huang Ying -1 siblings, 0 replies; 12+ messages in thread From: Huang Ying @ 2015-06-16 5:41 UTC (permalink / raw) To: lkp [-- Attachment #1: Type: text/plain, Size: 1654 bytes --] Hi, Gerry, On Tue, 2015-06-16 at 10:09 +0800, Jiang Liu wrote: > On 2015/6/16 1:52, Thomas Gleixner wrote: > > On Mon, 15 Jun 2015, Huang Ying wrote: > > > >> FYI, we noticed the below changes on > >> > >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master > >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") > >> > > > > I really appreciate this testing effort, but the information provided > > is not really helpful. > > > > I asked this before. Can you pretty please, upload ALL relevant > > information (.config, full dmesg, below stats, /proc/interrupts ...) > > to some place where everyone interested can download them? > > > > Then themail contains a useful link instead of 200k waste of network bandwidth. > > Hi Ying and Thomas, > I guess this report discloses a regression in hierarchy > irqdomain, and which should have been fixed by the patch posted at: > lkml.org/lkml/2015/6/1/80 > The root cause is that, with hierarchy irqdomain enabled, > there are multiple irq_data associated with one irq. And function > irq_move_irq() on x86 uses a wrong copy of irq_data to check > whether there's pending irq migration operation. So all irq migration > /set_affinity operations will get pending for ever. This may > affect network performance due to interrupt load balance issue. > And the patch set posted at > www.gossamer-threads.com/lists/linux/kernel/2185533 > should have solved all such regressions. Thanks for your information. Do you have a git tree for this patchset? Best Regards, Huang, Ying ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps @ 2015-06-16 5:41 ` Huang Ying 0 siblings, 0 replies; 12+ messages in thread From: Huang Ying @ 2015-06-16 5:41 UTC (permalink / raw) To: Jiang Liu; +Cc: Thomas Gleixner, LKML, LKP ML Hi, Gerry, On Tue, 2015-06-16 at 10:09 +0800, Jiang Liu wrote: > On 2015/6/16 1:52, Thomas Gleixner wrote: > > On Mon, 15 Jun 2015, Huang Ying wrote: > > > >> FYI, we noticed the below changes on > >> > >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master > >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") > >> > > > > I really appreciate this testing effort, but the information provided > > is not really helpful. > > > > I asked this before. Can you pretty please, upload ALL relevant > > information (.config, full dmesg, below stats, /proc/interrupts ...) > > to some place where everyone interested can download them? > > > > Then themail contains a useful link instead of 200k waste of network bandwidth. > > Hi Ying and Thomas, > I guess this report discloses a regression in hierarchy > irqdomain, and which should have been fixed by the patch posted at: > lkml.org/lkml/2015/6/1/80 > The root cause is that, with hierarchy irqdomain enabled, > there are multiple irq_data associated with one irq. And function > irq_move_irq() on x86 uses a wrong copy of irq_data to check > whether there's pending irq migration operation. So all irq migration > /set_affinity operations will get pending for ever. This may > affect network performance due to interrupt load balance issue. > And the patch set posted at > www.gossamer-threads.com/lists/linux/kernel/2185533 > should have solved all such regressions. Thanks for your information. Do you have a git tree for this patchset? Best Regards, Huang, Ying ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps 2015-06-16 5:41 ` [lkp] " Huang Ying @ 2015-06-16 5:51 ` Jiang Liu -1 siblings, 0 replies; 12+ messages in thread From: Jiang Liu @ 2015-06-16 5:51 UTC (permalink / raw) To: lkp [-- Attachment #1: Type: text/plain, Size: 1081 bytes --] On 2015/6/16 13:41, Huang Ying wrote: > Hi, Gerry, > > On Tue, 2015-06-16 at 10:09 +0800, Jiang Liu wrote: >> On 2015/6/16 1:52, Thomas Gleixner wrote: >>> On Mon, 15 Jun 2015, Huang Ying wrote: >> lkml.org/lkml/2015/6/1/80 >> The root cause is that, with hierarchy irqdomain enabled, >> there are multiple irq_data associated with one irq. And function >> irq_move_irq() on x86 uses a wrong copy of irq_data to check >> whether there's pending irq migration operation. So all irq migration >> /set_affinity operations will get pending for ever. This may >> affect network performance due to interrupt load balance issue. >> And the patch set posted at >> www.gossamer-threads.com/lists/linux/kernel/2185533 >> should have solved all such regressions. > > Thanks for your information. Do you have a git tree for this patchset? Hi Ying, You may access the branch at: https://github.com/jiangliu/linux test/irq_data_v3 or view at url: https://github.com/jiangliu/linux/tree/test/irq_data_v3 Thanks! Gerry > > Best Regards, > Huang, Ying > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps @ 2015-06-16 5:51 ` Jiang Liu 0 siblings, 0 replies; 12+ messages in thread From: Jiang Liu @ 2015-06-16 5:51 UTC (permalink / raw) To: Huang Ying; +Cc: Thomas Gleixner, LKML, LKP ML On 2015/6/16 13:41, Huang Ying wrote: > Hi, Gerry, > > On Tue, 2015-06-16 at 10:09 +0800, Jiang Liu wrote: >> On 2015/6/16 1:52, Thomas Gleixner wrote: >>> On Mon, 15 Jun 2015, Huang Ying wrote: >> lkml.org/lkml/2015/6/1/80 >> The root cause is that, with hierarchy irqdomain enabled, >> there are multiple irq_data associated with one irq. And function >> irq_move_irq() on x86 uses a wrong copy of irq_data to check >> whether there's pending irq migration operation. So all irq migration >> /set_affinity operations will get pending for ever. This may >> affect network performance due to interrupt load balance issue. >> And the patch set posted at >> www.gossamer-threads.com/lists/linux/kernel/2185533 >> should have solved all such regressions. > > Thanks for your information. Do you have a git tree for this patchset? Hi Ying, You may access the branch at: https://github.com/jiangliu/linux test/irq_data_v3 or view at url: https://github.com/jiangliu/linux/tree/test/irq_data_v3 Thanks! Gerry > > Best Regards, > Huang, Ying > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps 2015-06-16 2:09 ` [lkp] " Jiang Liu @ 2015-06-16 6:13 ` Thomas Gleixner -1 siblings, 0 replies; 12+ messages in thread From: Thomas Gleixner @ 2015-06-16 6:13 UTC (permalink / raw) To: lkp [-- Attachment #1: Type: text/plain, Size: 1672 bytes --] On Tue, 16 Jun 2015, Jiang Liu wrote: > On 2015/6/16 1:52, Thomas Gleixner wrote: > > On Mon, 15 Jun 2015, Huang Ying wrote: > > > >> FYI, we noticed the below changes on > >> > >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master > >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") > >> > > > > I really appreciate this testing effort, but the information provided > > is not really helpful. > > > > I asked this before. Can you pretty please, upload ALL relevant > > information (.config, full dmesg, below stats, /proc/interrupts ...) > > to some place where everyone interested can download them? > > > > Then themail contains a useful link instead of 200k waste of network bandwidth. > > Hi Ying and Thomas, > I guess this report discloses a regression in hierarchy > irqdomain, and which should have been fixed by the patch posted at: > lkml.org/lkml/2015/6/1/80 > The root cause is that, with hierarchy irqdomain enabled, > there are multiple irq_data associated with one irq. And function > irq_move_irq() on x86 uses a wrong copy of irq_data to check > whether there's pending irq migration operation. So all irq migration > /set_affinity operations will get pending for ever. This may > affect network performance due to interrupt load balance issue. > And the patch set posted at > www.gossamer-threads.com/lists/linux/kernel/2185533 > should have solved all such regressions. Next time, please send fixes which need to be applied right away seperate from patch series which touch the world and some more. Thanks, tglx ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [lkp] [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps @ 2015-06-16 6:13 ` Thomas Gleixner 0 siblings, 0 replies; 12+ messages in thread From: Thomas Gleixner @ 2015-06-16 6:13 UTC (permalink / raw) To: Jiang Liu; +Cc: Huang Ying, LKML, LKP ML On Tue, 16 Jun 2015, Jiang Liu wrote: > On 2015/6/16 1:52, Thomas Gleixner wrote: > > On Mon, 15 Jun 2015, Huang Ying wrote: > > > >> FYI, we noticed the below changes on > >> > >> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master > >> commit 52f518a3a7c2f80551a38d38be28bc9f335e713c ("x86/MSI: Use hierarchical irqdomains to manage MSI interrupts") > >> > > > > I really appreciate this testing effort, but the information provided > > is not really helpful. > > > > I asked this before. Can you pretty please, upload ALL relevant > > information (.config, full dmesg, below stats, /proc/interrupts ...) > > to some place where everyone interested can download them? > > > > Then themail contains a useful link instead of 200k waste of network bandwidth. > > Hi Ying and Thomas, > I guess this report discloses a regression in hierarchy > irqdomain, and which should have been fixed by the patch posted at: > lkml.org/lkml/2015/6/1/80 > The root cause is that, with hierarchy irqdomain enabled, > there are multiple irq_data associated with one irq. And function > irq_move_irq() on x86 uses a wrong copy of irq_data to check > whether there's pending irq migration operation. So all irq migration > /set_affinity operations will get pending for ever. This may > affect network performance due to interrupt load balance issue. > And the patch set posted at > www.gossamer-threads.com/lists/linux/kernel/2185533 > should have solved all such regressions. Next time, please send fixes which need to be applied right away seperate from patch series which touch the world and some more. Thanks, tglx ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2015-06-16 6:13 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-06-15 7:02 [x86/MSI] 52f518a3a7c: -30.5% netperf.Throughput_tps Huang Ying 2015-06-15 7:02 ` [lkp] " Huang Ying 2015-06-15 17:52 ` Thomas Gleixner 2015-06-15 17:52 ` [lkp] " Thomas Gleixner 2015-06-16 2:09 ` Jiang Liu 2015-06-16 2:09 ` [lkp] " Jiang Liu 2015-06-16 5:41 ` Huang Ying 2015-06-16 5:41 ` [lkp] " Huang Ying 2015-06-16 5:51 ` Jiang Liu 2015-06-16 5:51 ` [lkp] " Jiang Liu 2015-06-16 6:13 ` Thomas Gleixner 2015-06-16 6:13 ` [lkp] " Thomas Gleixner
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.