FYI, we noticed a -1.7% regression of aim9.shell_rtns_1.ops_per_sec due to commit: commit a8d1bdd62b8f242a6e1eb6bbe9894187ba61eb7d ("sched: use load_avg for selecting idlest group") git://internal_merge_and_test_tree devel-catchup-201610200334 in testcase: aim9 on test machine: 80 threads Intel(R) Xeon(R) CPU E7- 8870 @ 2.40GHz with 128G memory with following parameters: testtime: 300s test: shell_rtns_1 cpufreq_governor: performance Suite IX is the "AIM Independent Resource Benchmark:" the famous synthetic benchmark. Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Details are as below: --------------------------------------------------------------------------------------------------> To reproduce: git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git cd lkp-tests bin/lkp install job.yaml # job file is attached in this email bin/lkp run job.yaml ========================================================================================= compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/testtime: gcc-6/performance/x86_64-rhel-7.2/debian-x86_64-2016-08-31.cgz/lkp-wsx02/shell_rtns_1/aim9/300s commit: b5a9b34078 ("sched/fair: Fix incorrect task group ->load_avg") a8d1bdd62b ("sched: use load_avg for selecting idlest group") b5a9b340789b2b24 a8d1bdd62b8f242a6e1eb6bbe9 ---------------- -------------------------- %stddev %change %stddev \ | \ 234.05 ± 0% -1.7% 229.96 ± 0% aim9.shell_rtns_1.ops_per_sec 221148 ± 0% -6.6% 206658 ± 0% aim9.time.involuntary_context_switches 21558690 ± 0% -1.5% 21232190 ± 0% aim9.time.minor_page_faults 82.75 ± 0% +1.5% 84.00 ± 0% aim9.time.percent_of_cpu_this_job_got 149.67 ± 0% +3.4% 154.80 ± 0% aim9.time.system_time 561354 ± 0% -1.8% 551395 ± 0% aim9.time.voluntary_context_switches 10879 ± 0% -1.1% 10759 ± 0% vmstat.system.cs 5269843 ± 1% -15.1% 4472309 ± 6% numa-numastat.node0.local_node 5269854 ± 1% -15.1% 4472311 ± 6% numa-numastat.node0.numa_hit 286016 ± 1% +14.8% 328243 ± 0% meminfo.Committed_AS 4584 ± 4% -10.5% 4102 ± 1% meminfo.PageTables 25594 ± 0% -9.1% 23277 ± 0% meminfo.Shmem 1111 ± 22% -43.3% 630.00 ± 23% numa-meminfo.node0.PageTables 29384 ± 9% -13.5% 25403 ± 9% numa-meminfo.node1.SUnreclaim 6413 ± 53% -71.5% 1825 ±121% numa-meminfo.node1.Shmem 89450 ± 5% -8.7% 81642 ± 2% cpuidle.C1-NHM.usage 1.169e+10 ± 0% +16.5% 1.362e+10 ± 0% cpuidle.C3-NHM.time 13400190 ± 0% +16.3% 15584170 ± 0% cpuidle.C3-NHM.usage 1.435e+09 ± 2% -85.4% 2.099e+08 ± 3% cpuidle.POLL.time 1143 ± 5% -11.2% 1016 ± 1% proc-vmstat.nr_page_table_pages 6395 ± 0% -9.0% 5819 ± 0% proc-vmstat.nr_shmem 6366 ± 0% +32.9% 8462 ± 0% proc-vmstat.numa_pte_updates 5762 ± 2% -16.3% 4822 ± 2% proc-vmstat.pgactivate 270.75 ± 21% -42.9% 154.50 ± 24% numa-vmstat.node0.nr_page_table_pages 2657027 ± 1% -14.6% 2270033 ± 7% numa-vmstat.node0.numa_hit 2657017 ± 1% -14.6% 2270032 ± 7% numa-vmstat.node0.numa_local 1601 ± 53% -71.5% 455.75 ±121% numa-vmstat.node1.nr_shmem 7344 ± 9% -13.5% 6350 ± 9% numa-vmstat.node1.nr_slab_unreclaimable 12.25 ± 1% -31.4% 8.40 ± 0% turbostat.%Busy 263.00 ± 1% -48.4% 135.75 ± 0% turbostat.Avg_MHz 2147 ± 0% -24.7% 1617 ± 0% turbostat.Bzy_MHz 17.28 ± 0% -17.8% 14.21 ± 0% turbostat.CPU%c1 46.47 ± 0% +17.3% 54.51 ± 0% turbostat.CPU%c3 34527 ± 1% -54.2% 15818 ± 1% slabinfo.cred_jar.active_objs 832.75 ± 1% -54.8% 376.00 ± 1% slabinfo.cred_jar.active_slabs 34991 ± 1% -54.8% 15823 ± 1% slabinfo.cred_jar.num_objs 832.75 ± 1% -54.8% 376.00 ± 1% slabinfo.cred_jar.num_slabs 39311 ± 5% -37.1% 24708 ± 5% slabinfo.kmalloc-32.active_objs 306.50 ± 5% -36.3% 195.25 ± 4% slabinfo.kmalloc-32.active_slabs 39335 ± 5% -36.3% 25044 ± 4% slabinfo.kmalloc-32.num_objs 306.50 ± 5% -36.3% 195.25 ± 4% slabinfo.kmalloc-32.num_slabs 29062 ± 2% +31.6% 38253 ± 2% slabinfo.vm_area_struct.active_objs 660.25 ± 2% +32.0% 871.75 ± 2% slabinfo.vm_area_struct.active_slabs 29069 ± 2% +32.0% 38373 ± 2% slabinfo.vm_area_struct.num_objs 660.25 ± 2% +32.0% 871.75 ± 2% slabinfo.vm_area_struct.num_slabs 1175 ± 8% +19.1% 1400 ± 6% sched_debug.cfs_rq:/.exec_clock.min 42315 ± 0% +27.4% 53900 ± 1% sched_debug.cfs_rq:/.min_vruntime.avg 27096 ± 7% +48.2% 40146 ± 4% sched_debug.cfs_rq:/.min_vruntime.min -14299 ±-24% -80.0% -2861 ±-163% sched_debug.cfs_rq:/.spread0.avg -29522 ±-17% -43.7% -16615 ±-22% sched_debug.cfs_rq:/.spread0.min 76.25 ± 7% -14.8% 65.00 ± 4% sched_debug.cfs_rq:/.util_avg.min 14287 ± 4% +17.1% 16732 ± 1% sched_debug.cpu.nr_load_updates.min 2556 ± 9% -25.9% 1893 ± 4% sched_debug.cpu.nr_load_updates.stddev 38039 ± 13% -17.1% 31526 ± 7% sched_debug.cpu.nr_switches.max 15489 ± 7% +17.9% 18267 ± 1% sched_debug.cpu.nr_switches.min 3857 ± 11% -38.7% 2365 ± 5% sched_debug.cpu.nr_switches.stddev 20.79 ± 26% -55.9% 9.17 ± 8% sched_debug.cpu.nr_uninterruptible.max 13847 ± 8% +20.7% 16712 ± 1% sched_debug.cpu.sched_count.min 15353 ± 11% -15.1% 13028 ± 3% sched_debug.cpu.sched_goidle.max 6198 ± 9% +22.8% 7612 ± 2% sched_debug.cpu.sched_goidle.min 1622 ± 8% -37.5% 1014 ± 2% sched_debug.cpu.sched_goidle.stddev 14707 ± 12% -17.5% 12140 ± 3% sched_debug.cpu.ttwu_count.max 1821 ± 6% -25.6% 1355 ± 6% sched_debug.cpu.ttwu_count.stddev 554.90 ± 51% -43.5% 313.25 ± 11% sched_debug.cpu.ttwu_local.stddev 0.03 ± 13% +85.9% 0.06 ± 15% sched_debug.rt_rq:/.rt_time.avg 0.25 ± 13% +41.0% 0.35 ± 13% sched_debug.rt_rq:/.rt_time.stddev 3.708e+11 ± 1% -63.4% 1.358e+11 ± 1% perf-stat.branch-instructions 1.28 ± 1% +174.6% 3.50 ± 1% perf-stat.branch-miss-rate% 6.66 ± 0% -18.8% 5.41 ± 1% perf-stat.cache-miss-rate% 1.249e+09 ± 0% -18.6% 1.016e+09 ± 0% perf-stat.cache-misses 3266933 ± 0% -0.9% 3237980 ± 0% perf-stat.context-switches 6.186e+12 ± 1% -48.7% 3.172e+12 ± 0% perf-stat.cpu-cycles 379880 ± 0% -1.5% 374126 ± 0% perf-stat.cpu-migrations 0.34 ± 2% +129.3% 0.78 ± 7% perf-stat.dTLB-load-miss-rate% 4.073e+11 ± 1% -57.8% 1.719e+11 ± 1% perf-stat.dTLB-loads 0.43 ± 0% -6.7% 0.40 ± 1% perf-stat.dTLB-store-miss-rate% 3.168e+08 ± 0% -8.1% 2.911e+08 ± 1% perf-stat.dTLB-store-misses 7.311e+10 ± 0% -1.4% 7.205e+10 ± 0% perf-stat.dTLB-stores 0.02 ± 1% +144.3% 0.05 ± 0% perf-stat.iTLB-load-miss-rate% 1.596e+12 ± 1% -59.0% 6.539e+11 ± 0% perf-stat.iTLB-loads 1.617e+12 ± 1% -59.4% 6.57e+11 ± 1% perf-stat.instructions 5214 ± 1% -59.4% 2115 ± 1% perf-stat.instructions-per-iTLB-miss 0.26 ± 0% -20.8% 0.21 ± 0% perf-stat.ipc 21862976 ± 0% -1.5% 21540685 ± 0% perf-stat.minor-faults 20126826 ± 2% -6.9% 18743631 ± 1% perf-stat.node-load-misses 1.134e+08 ± 1% -3.6% 1.092e+08 ± 0% perf-stat.node-loads 11.40 ± 2% -12.7% 9.95 ± 0% perf-stat.node-store-miss-rate% 57910486 ± 2% -22.1% 45114675 ± 2% perf-stat.node-store-misses 4.501e+08 ± 1% -9.3% 4.082e+08 ± 1% perf-stat.node-stores 21862976 ± 0% -1.5% 21540246 ± 0% perf-stat.page-faults 0.83 ± 2% +71.6% 1.42 ± 4% perf-profile.calltrace.cycles-pp.__do_page_fault.do_page_fault.page_fault 0.73 ± 4% +52.4% 1.12 ± 5% perf-profile.calltrace.cycles-pp.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt 2.71 ± 2% +68.5% 4.56 ± 3% perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary 0.77 ± 3% +68.2% 1.30 ± 3% perf-profile.calltrace.cycles-pp.clockevents_program_event.tick_program_event.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt 0.84 ± 2% +71.6% 1.44 ± 4% perf-profile.calltrace.cycles-pp.do_page_fault.page_fault 1.39 ± 3% +70.4% 2.36 ± 1% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_fastpath 0.70 ± 2% +72.8% 1.20 ± 3% perf-profile.calltrace.cycles-pp.handle_mm_fault.__do_page_fault.do_page_fault.page_fault 1.33 ± 2% +72.6% 2.30 ± 2% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter 55.01 ± 1% +31.7% 72.46 ± 1% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry 0.87 ± 4% +53.8% 1.33 ± 4% perf-profile.calltrace.cycles-pp.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle 0.73 ± 3% +70.3% 1.25 ± 2% perf-profile.calltrace.cycles-pp.lapic_next_event.clockevents_program_event.tick_program_event.hrtimer_interrupt.local_apic_timer_interrupt 1.38 ± 3% +73.1% 2.38 ± 3% perf-profile.calltrace.cycles-pp.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle 0.73 ± 3% +70.2% 1.24 ± 2% perf-profile.calltrace.cycles-pp.native_write_msr.lapic_next_event.clockevents_program_event.tick_program_event.hrtimer_interrupt 0.84 ± 2% +71.1% 1.44 ± 4% perf-profile.calltrace.cycles-pp.page_fault 34.29 ± 3% -71.7% 9.70 ± 12% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry 2.63 ± 3% +67.9% 4.42 ± 3% perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter.call_cpuidle.cpu_startup_entry 0.77 ± 4% +50.5% 1.16 ± 5% perf-profile.calltrace.cycles-pp.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter 0.66 ± 4% +44.5% 0.96 ± 4% perf-profile.calltrace.cycles-pp.tick_nohz_stop_sched_tick.__tick_nohz_idle_enter.tick_nohz_irq_exit.irq_exit.smp_apic_timer_interrupt 0.80 ± 3% +67.5% 1.34 ± 2% perf-profile.calltrace.cycles-pp.tick_program_event.hrtimer_interrupt.local_apic_timer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt 1.18 ± 3% +69.6% 1.99 ± 2% perf-profile.children.cycles-pp.__do_page_fault 0.81 ± 4% +53.1% 1.24 ± 6% perf-profile.children.cycles-pp.__tick_nohz_idle_enter 2.90 ± 3% +66.0% 4.82 ± 3% perf-profile.children.cycles-pp.apic_timer_interrupt 0.80 ± 3% +67.8% 1.34 ± 2% perf-profile.children.cycles-pp.clockevents_program_event 0.88 ± 4% +51.7% 1.33 ± 1% perf-profile.children.cycles-pp.do_execveat_common 1.20 ± 3% +69.7% 2.03 ± 2% perf-profile.children.cycles-pp.do_page_fault 1.42 ± 3% +49.4% 2.12 ± 1% perf-profile.children.cycles-pp.do_syscall_64 2.32 ± 4% +70.2% 3.95 ± 1% perf-profile.children.cycles-pp.entry_SYSCALL_64_fastpath 0.74 ± 4% +52.0% 1.12 ± 1% perf-profile.children.cycles-pp.exit_mmap 1.04 ± 3% +68.7% 1.76 ± 2% perf-profile.children.cycles-pp.handle_mm_fault 1.43 ± 2% +72.2% 2.46 ± 2% perf-profile.children.cycles-pp.hrtimer_interrupt 55.02 ± 1% +31.7% 72.47 ± 1% perf-profile.children.cycles-pp.intel_idle 0.94 ± 4% +56.7% 1.46 ± 5% perf-profile.children.cycles-pp.irq_exit 0.77 ± 3% +69.5% 1.31 ± 2% perf-profile.children.cycles-pp.lapic_next_event 1.46 ± 3% +73.1% 2.53 ± 3% perf-profile.children.cycles-pp.local_apic_timer_interrupt 0.74 ± 4% +52.0% 1.13 ± 1% perf-profile.children.cycles-pp.mmput 0.85 ± 3% +71.3% 1.45 ± 2% perf-profile.children.cycles-pp.native_write_msr 1.19 ± 3% +69.7% 2.02 ± 2% perf-profile.children.cycles-pp.page_fault 34.51 ± 3% -71.6% 9.81 ± 12% perf-profile.children.cycles-pp.poll_idle 1.42 ± 3% +49.7% 2.12 ± 1% perf-profile.children.cycles-pp.return_from_SYSCALL_64 2.83 ± 3% +65.4% 4.68 ± 3% perf-profile.children.cycles-pp.smp_apic_timer_interrupt 0.89 ± 3% +51.5% 1.35 ± 1% perf-profile.children.cycles-pp.sys_execve 0.80 ± 3% +50.2% 1.20 ± 6% perf-profile.children.cycles-pp.tick_nohz_irq_exit 0.73 ± 3% +46.4% 1.06 ± 4% perf-profile.children.cycles-pp.tick_nohz_stop_sched_tick 0.84 ± 3% +66.3% 1.39 ± 2% perf-profile.children.cycles-pp.tick_program_event 55.02 ± 1% +31.7% 72.47 ± 1% perf-profile.self.cycles-pp.intel_idle 0.85 ± 3% +71.3% 1.45 ± 2% perf-profile.self.cycles-pp.native_write_msr 34.51 ± 3% -71.6% 9.81 ± 12% perf-profile.self.cycles-pp.poll_idle Thanks, Xiaolong