Re: [PATCH] perf: Remove setting of page->index and ->mapping

kernel test robot
2024-08-16T06:36:03Z

Hello,

from commit message, it seems just some useless setting have been removed.
but quite some performance regression bisect by our bot finally point to this
change (already removed some not so significant changes to avoid this report
mail is too long).

we don't have enough knowledge to explain these, so just report FYI.
if you have any suggestion for our tests or need us run more tests, please
just let us know. thanks a lot!

kernel test robot noticed a -28.3% regression of stress-ng.dirmany.files_removed_per_sec on:

commit: a9ac22fb7ac3486a5bd89c25a2f784330ee92dd7 ("[PATCH] perf: Remove setting of page->index and ->mapping")
url: https://github.com/intel-lab-lkp/linux/commits/Matthew-Wilcox-Oracle/perf-Remove-setting-of-page-index-and-mapping/20240810-111222
base: https://git.kernel.org/cgit/linux/kernel/git/perf/perf-tools-next.git perf-tools-next
patch link: https://lore.kernel.org/all/20240809201038.2648058-1-willy@infradead.org/
patch subject: [PATCH] perf: Remove setting of page->index and ->mapping

testcase: stress-ng
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:

nr_threads: 100%
disk: 1HDD
testtime: 60s
fs: btrfs
test: dirmany
cpufreq_governor: performance

In addition to that, the commit also has significant impact on the following tests:

+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | aim9: aim9.exec_test.ops_per_sec -12.3% regression |
| test machine | 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory |
| test parameters | cpufreq_governor=performance |
| | test=exec_test |
| | testtime=300s |
+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | stress-ng: stress-ng.sem.sem_wait_calls_per_sec -56.8% regression |
| test machine | 224 threads 2 sockets Intel(R) Xeon(R) Platinum 8480CTDX (Sapphire Rapids) with 256G memory |
| test parameters | cpufreq_governor=performance |
| | nr_threads=100% |
| | test=sem |
| | testtime=60s |
+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | tbench: tbench.throughput_MB/s -42.5% regression |
| test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory |
| test parameters | cluster=cs-localhost |
| | cpufreq_governor=performance |
| | iterations=1 |
| | nr_threads=100% |
| | runtime=600 |
+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | uperf: uperf.Runstat_master_Mbps -25.4% regression |
| test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory |
| test parameters | cluster=cs-localhost |
| | cpufreq_governor=performance |
| | datasize=64 |
| | nr_threads=1 |
| | protocol=UDP |
| | runtime=30s |
| | winsize=32k |
+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | netperf: netperf.Throughput_tps -31.4% regression |
| test machine | 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory |
| test parameters | cluster=cs-localhost |
| | cpufreq_governor=performance |
| | ip=ipv4 |
| | nr_threads=50% |
| | runtime=300s |
| | test=SCTP_RR |
+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -61.9% regression |
| test machine | 104 threads 2 sockets (Skylake) with 192G memory |
| test parameters | cpufreq_governor=performance |
| | mode=process |
| | nr_task=100% |
| | test=context_switch1 |
+------------------+--------------------------------------------------------------------------------------------------------+
| testcase: change | fsmark: fsmark.files_per_sec -29.8% regression |
| test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory |
| test parameters | cpufreq_governor=performance |
| | disk=1BRD_32G |
| | filesize=4K |
| | fs2=nfsv4 |
| | fs=ext4 |
| | iterations=1x |
| | nr_files_per_directory=1fpd |
| | nr_threads=1t |
| | sync_method=fsyncBeforeClose |
| | test_size=4G |
+------------------+--------------------------------------------------------------------------------------------------------+

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202408161305.a3a5158a-oliver.sang@intel.com

Details are as below:
-------------------------------------------------------------------------------------------------->

The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240816/202408161305.a3a5158a-oliver.sang@intel.com

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/1HDD/btrfs/x86_64-rhel-8.3/100%/debian-12-x86_64-20240206.cgz/lkp-icl-2sp8/dirmany/stress-ng/60s

commit:
cb1898f58e ("perf annotate-data: Support --skip-empty option")
a9ac22fb7a ("perf: Remove setting of page->index and ->mapping")

cb1898f58e0f175d a9ac22fb7ac3486a5bd89c25a2f
---------------- ---------------------------
%stddev %change %stddev
\ | \
3.832e+09 ± 2% +21.3% 4.647e+09 ± 12% cpuidle..time
0.71 ± 4% +0.2 0.88 ± 3% mpstat.cpu.all.usr%
37575 ± 10% -86.8% 4959 ± 2% meminfo.Active(anon)
121149 ± 3% -28.6% 86511 meminfo.Shmem
103.55 +13.9% 117.98 ± 8% uptime.boot
5993 +13.0% 6775 ± 8% uptime.idle
238142 ± 6% -23.9% 181123 ± 11% vmstat.system.cs
157186 ± 7% +59.3% 250390 ± 10% vmstat.system.in
16677 ± 10% -86.6% 2238 ± 49% numa-meminfo.node0.Active(anon)
20916 ± 15% -87.0% 2724 ± 39% numa-meminfo.node1.Active(anon)
1379973 ± 75% +117.7% 3004793 ± 40% numa-meminfo.node1.FilePages
666634 ±158% +224.1% 2160724 ± 53% numa-meminfo.node1.Unevictable
1586 ± 12% -100.0% 0.00 perf-c2c.DRAM.local
13101 ± 15% -100.0% 0.00 perf-c2c.DRAM.remote
12067 ± 15% -100.0% 0.00 perf-c2c.HITM.local
8050 ± 16% -100.0% 0.00 perf-c2c.HITM.remote
20118 ± 15% -100.0% 0.00 perf-c2c.HITM.total
9403 ± 10% -86.8% 1240 ± 2% proc-vmstat.nr_active_anon
27583 -2.0% 27039 proc-vmstat.nr_mapped
30307 ± 3% -28.6% 21640 proc-vmstat.nr_shmem
9403 ± 10% -86.8% 1240 ± 2% proc-vmstat.nr_zone_active_anon
66223 -18.6% 53878 proc-vmstat.numa_other
263934 ± 3% +1e+05% 2.768e+08 ± 16% proc-vmstat.pgfault
67.43 ± 3% -14.6% 57.57 ± 9% stress-ng.dirmany.%_of_time_creating_files
32.57 ± 6% +30.3% 42.43 ± 12% stress-ng.dirmany.%_of_time_removing_file
703.32 -28.3% 504.06 ± 3% stress-ng.dirmany.files_removed_per_sec
63.61 ± 2% +22.9% 78.17 ± 13% stress-ng.time.elapsed_time
63.61 ± 2% +22.9% 78.17 ± 13% stress-ng.time.elapsed_time.max
422.17 ± 4% -35.4% 272.83 ± 17% stress-ng.time.percent_of_cpu_this_job_got
267.72 ± 5% -21.4% 210.45 ± 17% stress-ng.time.system_time
4177 ± 10% -86.6% 559.65 ± 49% numa-vmstat.node0.nr_active_anon
4177 ± 10% -86.6% 559.65 ± 49% numa-vmstat.node0.nr_zone_active_anon
5236 ± 15% -87.0% 681.46 ± 39% numa-vmstat.node1.nr_active_anon
345110 ± 75% +117.7% 751196 ± 40% numa-vmstat.node1.nr_file_pages
164075 ± 3% +16.8% 191570 ± 14% numa-vmstat.node1.nr_slab_reclaimable
166658 ±158% +224.1% 540181 ± 53% numa-vmstat.node1.nr_unevictable
5236 ± 15% -87.0% 681.46 ± 39% numa-vmstat.node1.nr_zone_active_anon
166658 ±158% +224.1% 540181 ± 53% numa-vmstat.node1.nr_zone_unevictable
76822 ± 10% +82.2% 140007 ± 19% sched_debug.cfs_rq:/.avg_vruntime.max
8327 ± 16% +138.0% 19815 ± 8% sched_debug.cfs_rq:/.avg_vruntime.stddev
2.25 ± 27% -100.0% 0.00 sched_debug.cfs_rq:/.load_avg.min
76822 ± 10% +82.2% 140007 ± 19% sched_debug.cfs_rq:/.min_vruntime.max
8327 ± 16% +138.0% 19815 ± 8% sched_debug.cfs_rq:/.min_vruntime.stddev
19.67 ± 57% -98.7% 0.25 ±223% sched_debug.cfs_rq:/.runnable_avg.min
217.81 ± 3% +27.8% 278.40 ± 4% sched_debug.cfs_rq:/.runnable_avg.stddev
1006 ± 7% +16.2% 1169 ± 6% sched_debug.cfs_rq:/.util_avg.max
19.75 ± 57% -99.2% 0.17 ±223% sched_debug.cfs_rq:/.util_avg.min
216.49 ± 3% +27.8% 276.68 ± 4% sched_debug.cfs_rq:/.util_avg.stddev
28.42 ± 11% +58.1% 44.94 ± 14% sched_debug.cfs_rq:/.util_est.avg
447.25 ± 12% +71.0% 764.67 sched_debug.cfs_rq:/.util_est.max
90.88 ± 9% +85.0% 168.09 ± 5% sched_debug.cfs_rq:/.util_est.stddev
415102 ± 6% +27.5% 529068 sched_debug.cpu.avg_idle.avg
752805 ± 25% +49.8% 1127548 ± 6% sched_debug.cpu.avg_idle.max
154552 ± 6% +53.2% 236733 ± 6% sched_debug.cpu.avg_idle.stddev
101993 ± 13% -40.9% 60236 ± 32% sched_debug.cpu.nr_switches.avg
113676 ± 14% -33.7% 75369 ± 27% sched_debug.cpu.nr_switches.max
93486 ± 14% -87.3% 11896 ± 26% sched_debug.cpu.nr_switches.min
3929 ± 34% +221.3% 12624 ± 28% sched_debug.cpu.nr_switches.stddev
2.27 ± 2% -64.5% 0.81 ± 11% perf-stat.i.MPKI
1.604e+09 ± 2% +138.1% 3.819e+09 ± 6% perf-stat.i.branch-instructions
2.33 ± 2% -1.6 0.72 ± 6% perf-stat.i.branch-miss-rate%
36868744 ± 3% -29.2% 26089842 ± 7% perf-stat.i.branch-misses
244416 ± 6% -24.1% 185618 ± 12% perf-stat.i.context-switches
1.90 ± 3% -39.4% 1.16 ± 3% perf-stat.i.cpi
10537 ± 6% -78.5% 2262 ± 46% perf-stat.i.cpu-migrations
1308 ± 11% +411.3% 6692 ± 26% perf-stat.i.cycles-between-cache-misses
8.205e+09 ± 2% +125.6% 1.851e+10 ± 6% perf-stat.i.instructions
0.88 ± 3% +11.6% 0.98 ± 2% perf-stat.i.ipc
3.78 ± 6% +2824.0% 110.64 ± 3% perf-stat.i.metric.K/sec
3035 ± 2% +1.1e+05% 3448996 ± 3% perf-stat.i.minor-faults
3035 ± 2% +1.1e+05% 3448996 ± 3% perf-stat.i.page-faults
2.82 ± 2% -60.8% 1.10 ± 7% perf-stat.overall.MPKI
2.30 ± 2% -1.6 0.68 ± 7% perf-stat.overall.branch-miss-rate%
2.58 ± 2% -51.0% 1.26 ± 3% perf-stat.overall.cpi
915.45 ± 4% +25.5% 1148 ± 5% perf-stat.overall.cycles-between-cache-misses
0.39 ± 2% +104.3% 0.79 ± 3% perf-stat.overall.ipc
1.579e+09 ± 2% +138.7% 3.769e+09 ± 6% perf-stat.ps.branch-instructions
36312783 ± 3% -29.1% 25735794 ± 7% perf-stat.ps.branch-misses
240154 ± 6% -23.7% 183188 ± 12% perf-stat.ps.context-switches
10351 ± 6% -78.4% 2237 ± 46% perf-stat.ps.cpu-migrations
8.078e+09 ± 2% +126.1% 1.827e+10 ± 6% perf-stat.ps.instructions
2990 ± 2% +1.1e+05% 3403503 ± 3% perf-stat.ps.minor-faults
2990 ± 2% +1.1e+05% 3403503 ± 3% perf-stat.ps.page-faults
5.224e+11 ± 2% +178.3% 1.454e+12 ± 15% perf-stat.total.instructions
24.93 ± 18% -24.9 0.00 perf-profile.calltrace.cycles-pp.common_startup_64
24.55 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction
24.55 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction
24.54 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.btree_write_cache_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.btrfs_write_marked_extents
24.51 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.fsync
24.50 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
24.50 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.fsync
24.49 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
24.49 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.btrfs_sync_file.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe.fsync
24.48 ± 56% -24.5 0.00 perf-profile.calltrace.cycles-pp.btrfs_commit_transaction.btrfs_sync_file.__x64_sys_fsync.do_syscall_64.entry_SYSCALL_64_after_hwframe
24.43 ± 57% -24.4 0.00 perf-profile.calltrace.cycles-pp.btrfs_write_and_wait_transaction.btrfs_commit_transaction.btrfs_sync_file.__x64_sys_fsync.do_syscall_64
24.43 ± 57% -24.4 0.00 perf-profile.calltrace.cycles-pp.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.btrfs_sync_file.__x64_sys_fsync
24.41 ± 57% -24.4 0.00 perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.btrfs_write_marked_extents.btrfs_write_and_wait_transaction.btrfs_commit_transaction.btrfs_sync_file
23.94 ± 56% -23.9 0.00 perf-profile.calltrace.cycles-pp.submit_eb_page.btree_write_cache_pages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
23.50 ± 16% -23.5 0.00 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
23.49 ± 16% -23.5 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
23.48 ± 16% -23.5 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
22.12 ± 17% -22.1 0.00 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
21.95 ± 19% -22.0 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
21.61 ± 20% -21.6 0.00 perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
20.64 ± 17% -20.6 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
20.29 ± 57% -20.3 0.00 perf-profile.calltrace.cycles-pp.btrfs_submit_bio.submit_eb_page.btree_write_cache_pages.do_writepages.filemap_fdatawrite_wbc
20.29 ± 57% -20.3 0.00 perf-profile.calltrace.cycles-pp.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page.btree_write_cache_pages.do_writepages
17.17 ± 57% -17.2 0.00 perf-profile.calltrace.cycles-pp.btree_csum_one_bio.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page.btree_write_cache_pages
14.62 ± 31% -14.6 0.00 perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
14.62 ± 31% -14.6 0.00 perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
14.62 ± 31% -14.6 0.00 perf-profile.calltrace.cycles-pp.ret_from_fork_asm
13.30 ± 27% -13.3 0.00 perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
12.54 ± 58% -12.5 0.00 perf-profile.calltrace.cycles-pp.btrfs_check_leaf.btree_csum_one_bio.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page
12.31 ± 58% -12.3 0.00 perf-profile.calltrace.cycles-pp.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio.btrfs_submit_chunk.btrfs_submit_bio
11.98 ± 32% -12.0 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
10.42 ± 37% -10.4 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
8.48 ± 57% -8.5 0.00 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
7.77 ± 62% -7.8 0.00 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
7.46 ± 58% -7.5 0.00 perf-profile.calltrace.cycles-pp.check_leaf_item.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio.btrfs_submit_chunk
5.40 ± 69% -5.4 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_posted_msi_notification.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
4.81 ± 57% -4.8 0.00 perf-profile.calltrace.cycles-pp.check_dir_item.check_leaf_item.__btrfs_check_leaf.btrfs_check_leaf.btree_csum_one_bio
4.62 ± 36% -4.6 0.00 perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state
4.52 ± 55% -4.5 0.00 perf-profile.calltrace.cycles-pp.csum_tree_block.btree_csum_one_bio.btrfs_submit_chunk.btrfs_submit_bio.submit_eb_page
4.51 ± 56% -4.5 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
4.49 ± 56% -4.5 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.46 ± 55% -4.5 0.00 perf-profile.calltrace.cycles-pp.crc32c_pcl_intel_update.csum_tree_block.btree_csum_one_bio.btrfs_submit_chunk.btrfs_submit_bio
4.46 ± 55% -4.5 0.00 perf-profile.calltrace.cycles-pp.crc_pcl.crc32c_pcl_intel_update.csum_tree_block.btree_csum_one_bio.btrfs_submit_chunk
55.50 ± 15% -55.5 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
55.46 ± 15% -55.5 0.00 perf-profile.children.cycles-pp.do_syscall_64
28.46 ± 53% -28.5 0.00 perf-profile.children.cycles-pp.btrfs_commit_transaction
24.93 ± 18% -24.9 0.00 perf-profile.children.cycles-pp.common_startup_64
24.93 ± 18% -24.9 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
24.92 ± 18% -24.9 0.00 perf-profile.children.cycles-pp.do_idle
24.61 ± 55% -24.6 0.00 perf-profile.children.cycles-pp.btrfs_write_and_wait_transaction
24.61 ± 55% -24.6 0.00 perf-profile.children.cycles-pp.btrfs_write_marked_extents
24.60 ± 55% -24.6 0.00 perf-profile.children.cycles-pp.__filemap_fdatawrite_range
24.60 ± 55% -24.6 0.00 perf-profile.children.cycles-pp.btree_write_cache_pages
24.60 ± 55% -24.6 0.00 perf-profile.children.cycles-pp.do_writepages
24.60 ± 55% -24.6 0.00 perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
24.51 ± 56% -24.5 0.00 perf-profile.children.cycles-pp.fsync
24.49 ± 56% -24.5 0.00 perf-profile.children.cycles-pp.__x64_sys_fsync
24.49 ± 56% -24.5 0.00 perf-profile.children.cycles-pp.btrfs_sync_file
24.00 ± 55% -24.0 0.00 perf-profile.children.cycles-pp.submit_eb_page
23.53 ± 19% -23.5 0.00 perf-profile.children.cycles-pp.cpuidle_idle_call
23.50 ± 16% -23.5 0.00 perf-profile.children.cycles-pp.start_secondary
22.05 ± 19% -22.1 0.00 perf-profile.children.cycles-pp.cpuidle_enter
22.00 ± 19% -22.0 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
21.66 ± 20% -21.7 0.00 perf-profile.children.cycles-pp.acpi_idle_enter
21.63 ± 20% -21.6 0.00 perf-profile.children.cycles-pp.acpi_safe_halt
20.41 ± 55% -20.4 0.00 perf-profile.children.cycles-pp.btrfs_submit_bio
20.41 ± 55% -20.4 0.00 perf-profile.children.cycles-pp.btrfs_submit_chunk
17.35 ± 55% -17.4 0.00 perf-profile.children.cycles-pp.btree_csum_one_bio
14.74 ± 31% -14.7 0.00 perf-profile.children.cycles-pp.ret_from_fork_asm
14.71 ± 31% -14.7 0.00 perf-profile.children.cycles-pp.ret_from_fork
14.62 ± 31% -14.6 0.00 perf-profile.children.cycles-pp.kthread
12.66 ± 56% -12.7 0.00 perf-profile.children.cycles-pp.__btrfs_check_leaf
12.66 ± 56% -12.7 0.00 perf-profile.children.cycles-pp.btrfs_check_leaf
8.52 ± 29% -8.5 0.00 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
8.48 ± 57% -8.5 0.00 perf-profile.children.cycles-pp.worker_thread
7.78 ± 62% -7.8 0.00 perf-profile.children.cycles-pp.process_one_work
7.62 ± 56% -7.6 0.00 perf-profile.children.cycles-pp.check_leaf_item
7.50 ± 34% -7.5 0.00 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
5.58 ± 53% -5.6 0.00 perf-profile.children.cycles-pp.crc32c_pcl_intel_update
5.20 ± 46% -5.2 0.00 perf-profile.children.cycles-pp.__irq_exit_rcu
5.11 ± 47% -5.1 0.00 perf-profile.children.cycles-pp.handle_softirqs
5.09 ± 54% -5.1 0.00 perf-profile.children.cycles-pp.check_dir_item
4.74 ± 34% -4.7 0.00 perf-profile.children.cycles-pp.sysvec_call_function_single
4.57 ± 53% -4.6 0.00 perf-profile.children.cycles-pp.csum_tree_block
4.54 ± 70% -4.5 0.00 perf-profile.children.cycles-pp.asm_sysvec_posted_msi_notification
4.49 ± 53% -4.5 0.00 perf-profile.children.cycles-pp.crc_pcl
4.45 ± 56% -4.5 0.00 perf-profile.children.cycles-pp.btrfs_get_32
4.06 ± 34% -4.1 0.00 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
8.89 ± 14% -8.9 0.00 perf-profile.self.cycles-pp.acpi_safe_halt
4.47 ± 53% -4.5 0.00 perf-profile.self.cycles-pp.crc_pcl

***************************************************************************************************
lkp-ivb-2ep2: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/x86_64-rhel-8.3/debian-12-x86_64-20240206.cgz/lkp-ivb-2ep2/exec_test/aim9/300s

commit:
cb1898f58e ("perf annotate-data: Support --skip-empty option")
a9ac22fb7a ("perf: Remove setting of page->index and ->mapping")

cb1898f58e0f175d a9ac22fb7ac3486a5bd89c25a2f
---------------- ---------------------------
%stddev %change %stddev
\ | \
4849014 +148.9% 12069624 cpuidle..usage
29121 ± 41% -62.2% 11007 ± 94% numa-numastat.node1.other_node
97.58 -6.9% 90.83 vmstat.cpu.id
2.30 +115.5% 4.95 vmstat.procs.r
8730 -13.2% 7575 vmstat.system.cs
14729 +456.7% 82001 vmstat.system.in
124849 ± 2% -91.8% 10221 meminfo.Active
124801 ± 2% -91.8% 10173 meminfo.Active(anon)
876134 -9.4% 793689 meminfo.Committed_AS
7544 +20.4% 9085 meminfo.PageTables
128860 ± 2% -91.1% 11481 ± 2% meminfo.Shmem
15.00 ± 19% -100.0% 0.00 perf-c2c.DRAM.local
36.00 ± 15% -100.0% 0.00 perf-c2c.DRAM.remote
2.67 ± 51% -100.0% 0.00 perf-c2c.HIT.remote
64.83 ± 3% -100.0% 0.00 perf-c2c.HITM.local
30.00 ± 15% -100.0% 0.00 perf-c2c.HITM.remote
1224 -12.3% 1074 aim9.exec_test.ops_per_sec
31706751 -12.9% 27603136 aim9.time.minor_page_faults
97.00 -1.0% 96.00 aim9.time.percent_of_cpu_this_job_got
176.72 +14.8% 202.90 aim9.time.system_time
116.68 -25.8% 86.55 ± 4% aim9.time.user_time
733520 -12.2% 643730 aim9.time.voluntary_context_switches
0.25 +1.2 1.42 mpstat.cpu.all.irq%
0.07 ± 2% +0.1 0.12 ± 2% mpstat.cpu.all.soft%
1.66 +3.8 5.46 mpstat.cpu.all.sys%
0.68 +1.7 2.36 mpstat.cpu.all.usr%
1.00 +14516.7% 146.17 ± 51% mpstat.max_utilization.seconds
8.64 +20.2% 10.38 mpstat.max_utilization_pct
9398 ± 74% -82.2% 1675 ± 57% numa-meminfo.node0.Active
9366 ± 74% -82.3% 1659 ± 58% numa-meminfo.node0.Active(anon)
9860 ± 70% -78.5% 2121 ± 48% numa-meminfo.node0.Shmem
115389 ± 6% -92.6% 8534 ± 11% numa-meminfo.node1.Active
115373 ± 6% -92.6% 8502 ± 11% numa-meminfo.node1.Active(anon)
522369 ± 30% -26.9% 381731 ± 30% numa-meminfo.node1.AnonPages.max
118964 ± 5% -92.1% 9348 ± 11% numa-meminfo.node1.Shmem
2345 ± 74% -82.3% 414.77 ± 58% numa-vmstat.node0.nr_active_anon
2466 ± 70% -78.5% 530.32 ± 48% numa-vmstat.node0.nr_shmem
2345 ± 74% -82.3% 414.77 ± 58% numa-vmstat.node0.nr_zone_active_anon
28843 ± 6% -92.6% 2125 ± 11% numa-vmstat.node1.nr_active_anon
29739 ± 5% -92.1% 2337 ± 11% numa-vmstat.node1.nr_shmem
28843 ± 6% -92.6% 2125 ± 11% numa-vmstat.node1.nr_zone_active_anon
29121 ± 41% -62.2% 11007 ± 94% numa-vmstat.node1.numa_other
519.91 ± 29% +39.3% 724.40 ± 3% sched_debug.IIkworker/R-0000
11208 ± 3% +249.5% 39175 ± 19% sched_debug.cfs_rq:/.avg_vruntime.avg
52206 ± 19% +419.6% 271275 ± 14% sched_debug.cfs_rq:/.avg_vruntime.max
9447 ± 10% +573.4% 63622 ± 17% sched_debug.cfs_rq:/.avg_vruntime.stddev
221530 ± 22% -59.4% 89942 ± 48% sched_debug.cfs_rq:/.load.max
39358 ± 19% -45.8% 21349 ± 35% sched_debug.cfs_rq:/.load.stddev
11208 ± 3% +249.5% 39175 ± 19% sched_debug.cfs_rq:/.min_vruntime.avg
52206 ± 19% +419.6% 271275 ± 14% sched_debug.cfs_rq:/.min_vruntime.max
9447 ± 10% +573.4% 63622 ± 17% sched_debug.cfs_rq:/.min_vruntime.stddev
7.04 ± 16% +352.0% 31.80 ± 20% sched_debug.cfs_rq:/.util_est.avg
121.39 ± 30% +212.8% 379.76 ± 23% sched_debug.cfs_rq:/.util_est.max
23.68 ± 20% +318.5% 99.12 ± 21% sched_debug.cfs_rq:/.util_est.stddev
66860 ± 13% -37.7% 41635 ± 27% sched_debug.cpu.avg_idle.stddev
2.40 ± 21% -70.1% 0.72 ± 21% sched_debug.cpu.clock.stddev
451.30 ± 3% -26.7% 330.98 ± 28% sched_debug.cpu.clock_task.stddev
90057 ± 3% -24.8% 67696 ± 23% sched_debug.cpu.curr->pid.max
16561 ± 9% -23.8% 12621 ± 22% sched_debug.cpu.curr->pid.stddev
13987 ± 3% -25.6% 10412 ± 23% sched_debug.cpu.nr_switches.avg
38759 ± 11% -30.0% 27133 ± 25% sched_debug.cpu.nr_switches.max
2680 ± 14% -72.4% 738.87 ± 24% sched_debug.cpu.nr_switches.min
31186 ± 2% -91.9% 2540 proc-vmstat.nr_active_anon
160386 +2.0% 163528 proc-vmstat.nr_anon_pages
797165 -3.7% 767817 proc-vmstat.nr_file_pages
161314 +1.5% 163804 proc-vmstat.nr_inactive_anon
11375 +3.6% 11782 proc-vmstat.nr_kernel_stack
8362 -2.8% 8125 proc-vmstat.nr_mapped
1829 +25.4% 2294 proc-vmstat.nr_page_table_pages
32208 ± 2% -91.1% 2867 ± 2% proc-vmstat.nr_shmem
20413 -3.4% 19710 proc-vmstat.nr_slab_reclaimable
28051 +0.8% 28289 proc-vmstat.nr_slab_unreclaimable
31186 ± 2% -91.9% 2540 proc-vmstat.nr_zone_active_anon
161314 +1.5% 163804 proc-vmstat.nr_zone_inactive_anon
20425 ± 33% -83.5% 3376 ±182% proc-vmstat.numa_hint_faults
26596010 -13.6% 22977970 proc-vmstat.numa_hit
26542993 -13.6% 22936126 proc-vmstat.numa_local
49790 -18.7% 40495 proc-vmstat.numa_other
11951 ± 68% -99.0% 123.17 ±160% proc-vmstat.numa_pages_migrated
91437 ± 43% -79.1% 19121 ±115% proc-vmstat.numa_pte_updates
35248 ± 4% -99.0% 364.17 ± 4% proc-vmstat.pgactivate
27497487 -13.5% 23788404 proc-vmstat.pgalloc_normal
32590981 +2227.8% 7.587e+08 proc-vmstat.pgfault
27449503 -13.7% 23694730 proc-vmstat.pgfree
11951 ± 68% -99.0% 123.17 ±160% proc-vmstat.pgmigrate_success
1549666 ± 2% -12.4% 1356755 ± 2% proc-vmstat.pgreuse
2.17 -61.6% 0.83 ± 2% perf-stat.i.MPKI
5.641e+08 +225.5% 1.836e+09 perf-stat.i.branch-instructions
5.50 -3.5 1.98 perf-stat.i.branch-miss-rate%
31671027 +15.1% 36440585 perf-stat.i.branch-misses
8.81 +1.4 10.23 ± 2% perf-stat.i.cache-miss-rate%
5351586 +35.8% 7269406 ± 2% perf-stat.i.cache-misses
60638638 +16.8% 70801331 perf-stat.i.cache-references
8748 -13.3% 7589 perf-stat.i.context-switches
1.85 -13.4% 1.60 perf-stat.i.cpi
4.765e+09 +193.1% 1.397e+10 perf-stat.i.cpu-cycles
890.01 +117.1% 1931 perf-stat.i.cycles-between-cache-misses
2.791e+09 +212.5% 8.722e+09 perf-stat.i.instructions
0.56 +12.6% 0.63 perf-stat.i.ipc
0.02 ± 27% +90.8% 0.03 ± 46% perf-stat.i.major-faults
4.40 +2276.8% 104.67 perf-stat.i.metric.K/sec
105697 +2276.8% 2512206 perf-stat.i.minor-faults
105697 +2276.8% 2512206 perf-stat.i.page-faults
1.92 -56.5% 0.83 ± 2% perf-stat.overall.MPKI
5.61 -3.6 1.98 perf-stat.overall.branch-miss-rate%
8.82 +1.4 10.27 ± 2% perf-stat.overall.cache-miss-rate%
1.71 -6.2% 1.60 perf-stat.overall.cpi
890.60 +115.8% 1922 perf-stat.overall.cycles-between-cache-misses
0.59 +6.6% 0.62 perf-stat.overall.ipc
5.623e+08 +225.4% 1.83e+09 perf-stat.ps.branch-instructions
31568362 +15.0% 36300756 perf-stat.ps.branch-misses
5333517 +35.9% 7245937 ± 2% perf-stat.ps.cache-misses
60439198 +16.8% 70570262 perf-stat.ps.cache-references
8719 -13.3% 7563 perf-stat.ps.context-switches
4.749e+09 +193.2% 1.392e+10 perf-stat.ps.cpu-cycles
2.782e+09 +212.5% 8.692e+09 perf-stat.ps.instructions
0.02 ± 27% +90.9% 0.03 ± 46% perf-stat.ps.major-faults
105344 +2277.4% 2504483 perf-stat.ps.minor-faults
105344 +2277.4% 2504483 perf-stat.ps.page-faults
8.38e+11 +212.6% 2.62e+12 perf-stat.total.instructions
66.96 -67.0 0.00 perf-profile.calltrace.cycles-pp.common_startup_64
65.60 -65.6 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
65.60 -65.6 0.00 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
65.54 -65.5 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
64.56 -64.6 0.00 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
62.17 -62.2 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
61.49 -61.5 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
49.76 -49.8 0.00 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
11.21 ± 2% -11.2 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
10.60 ± 2% -10.6 0.00 perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
8.26 ± 2% -8.3 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
8.24 ± 2% -8.2 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.96 ± 2% -8.0 0.00 perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
7.18 ± 2% -7.2 0.00 perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
7.01 ± 2% -7.0 0.00 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
6.33 -6.3 0.00 perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.02 -6.0 0.00 perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
6.00 -6.0 0.00 perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
5.75 -5.7 0.00 perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
5.35 -5.3 0.00 perf-profile.calltrace.cycles-pp.execve
5.34 -5.3 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
5.34 -5.3 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
5.34 -5.3 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
5.28 -5.3 0.00 perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
66.96 -67.0 0.00 perf-profile.children.cycles-pp.common_startup_64
66.96 -67.0 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
66.96 -67.0 0.00 perf-profile.children.cycles-pp.do_idle
65.92 -65.9 0.00 perf-profile.children.cycles-pp.cpuidle_idle_call
65.60 -65.6 0.00 perf-profile.children.cycles-pp.start_secondary
62.74 -62.7 0.00 perf-profile.children.cycles-pp.cpuidle_enter
62.59 -62.6 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
49.76 -49.8 0.00 perf-profile.children.cycles-pp.intel_idle
21.70 -21.7 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
21.66 -21.7 0.00 perf-profile.children.cycles-pp.do_syscall_64
11.84 ± 2% -11.8 0.00 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
11.19 ± 2% -11.2 0.00 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
8.12 ± 2% -8.1 0.00 perf-profile.children.cycles-pp.__x64_sys_execve
8.07 ± 2% -8.1 0.00 perf-profile.children.cycles-pp.execve
8.02 ± 2% -8.0 0.00 perf-profile.children.cycles-pp.do_execveat_common
7.57 ± 2% -7.6 0.00 perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
7.40 ± 2% -7.4 0.00 perf-profile.children.cycles-pp.hrtimer_interrupt
6.83 -6.8 0.00 perf-profile.children.cycles-pp.asm_exc_page_fault
6.43 -6.4 0.00 perf-profile.children.cycles-pp.bprm_execve
6.39 -6.4 0.00 perf-profile.children.cycles-pp.exc_page_fault
6.27 -6.3 0.00 perf-profile.children.cycles-pp.do_user_addr_fault
6.12 -6.1 0.00 perf-profile.children.cycles-pp.__hrtimer_run_queues
6.04 -6.0 0.00 perf-profile.children.cycles-pp.exec_binprm
6.02 -6.0 0.00 perf-profile.children.cycles-pp.search_binary_handler
5.64 -5.6 0.00 perf-profile.children.cycles-pp.handle_mm_fault
5.57 -5.6 0.00 perf-profile.children.cycles-pp.load_elf_binary
5.25 -5.2 0.00 perf-profile.children.cycles-pp.__handle_mm_fault
49.75 -49.8 0.00 perf-profile.self.cycles-pp.intel_idle

***************************************************************************************************
lkp-spr-r02: 224 threads 2 sockets Intel(R) Xeon(R) Platinum 8480CTDX (Sapphire Rapids) with 256G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-12/performance/x86_64-rhel-8.3/100%/debian-12-x86_64-20240206.cgz/lkp-spr-r02/sem/stress-ng/60s

commit:
cb1898f58e ("perf annotate-data: Support --skip-empty option")
a9ac22fb7a ("perf: Remove setting of page->index and ->mapping")

cb1898f58e0f175d a9ac22fb7ac3486a5bd89c25a2f
---------------- ---------------------------
%stddev %change %stddev
\ | \
257775 ± 22% +124.5% 578661 ± 21% cpuidle..usage
2202062 ± 22% -90.8% 202614 ± 38% numa-numastat.node1.local_node
2296320 ± 19% -85.0% 345152 ± 19% numa-numastat.node1.numa_hit
19.92 -52.0% 9.55 ± 2% vmstat.cpu.us
1.856e+08 ± 3% -58.4% 77159822 ± 4% vmstat.system.cs
556587 +42.9% 795511 vmstat.system.in
0.58 ± 2% +0.2 0.73 mpstat.cpu.all.irq%
0.02 ± 3% -0.0 0.01 ± 6% mpstat.cpu.all.soft%
75.65 +10.9 86.52 mpstat.cpu.all.sys%
20.45 -10.8 9.70 ± 3% mpstat.cpu.all.usr%
1531 ± 54% -100.0% 0.00 perf-c2c.DRAM.local
2246 ± 32% -100.0% 0.00 perf-c2c.DRAM.remote
1173 ± 69% -100.0% 0.00 perf-c2c.HITM.local
549.17 ± 31% -100.0% 0.00 perf-c2c.HITM.remote
25808257 ± 2% -58.1% 10805602 ± 4% sched_debug.cpu.nr_switches.avg
30207217 -57.0% 12980041 sched_debug.cpu.nr_switches.max
3103000 ± 72% -88.6% 354272 ±153% sched_debug.cpu.nr_switches.min
4087097 ± 21% -40.3% 2439826 ± 18% sched_debug.cpu.nr_switches.stddev
1.429e+10 ± 3% -53.6% 6.627e+09 ± 5% stress-ng.sem.ops
2.382e+08 ± 3% -53.6% 1.104e+08 ± 5% stress-ng.sem.ops_per_sec
327589 -56.8% 141462 stress-ng.sem.sem_timedwait_calls_per_sec
327403 -56.8% 141294 stress-ng.sem.sem_trywait_calls_per_sec
327558 -56.8% 141438 stress-ng.sem.sem_wait_calls_per_sec
1.178e+10 ± 2% -58.5% 4.886e+09 ± 4% stress-ng.time.involuntary_context_switches
10499 +13.5% 11913 stress-ng.time.system_time
2767 -53.0% 1301 ± 2% stress-ng.time.user_time
473599 ± 19% -63.1% 174673 ± 55% numa-meminfo.node0.Mapped
422997 ± 14% -99.3% 2940 ± 68% numa-meminfo.node1.Active
422988 ± 14% -99.3% 2932 ± 69% numa-meminfo.node1.Active(anon)
4625028 ± 25% -96.2% 176220 ± 42% numa-meminfo.node1.FilePages
4814124 ± 26% -84.9% 728255 ± 20% numa-meminfo.node1.Inactive
4814082 ± 26% -84.9% 728212 ± 20% numa-meminfo.node1.Inactive(anon)
1000373 ± 11% -92.9% 71080 ±136% numa-meminfo.node1.Mapped
6804306 ± 19% -67.0% 2242499 ± 5% numa-meminfo.node1.MemUsed
21291 ± 41% -41.3% 12504 ± 70% numa-meminfo.node1.PageTables
4556992 ± 26% -98.4% 74595 ±132% numa-meminfo.node1.Shmem
118547 ± 19% -63.1% 43793 ± 54% numa-vmstat.node0.nr_mapped
105773 ± 13% -99.3% 733.51 ± 69% numa-vmstat.node1.nr_active_anon
1156277 ± 25% -96.2% 44056 ± 42% numa-vmstat.node1.nr_file_pages
1203588 ± 26% -84.9% 182078 ± 20% numa-vmstat.node1.nr_inactive_anon
249177 ± 11% -92.8% 17833 ±135% numa-vmstat.node1.nr_mapped
5333 ± 41% -41.6% 3116 ± 70% numa-vmstat.node1.nr_page_table_pages
1139269 ± 26% -98.4% 18649 ±132% numa-vmstat.node1.nr_shmem
105772 ± 13% -99.3% 733.51 ± 69% numa-vmstat.node1.nr_zone_active_anon
1203585 ± 26% -84.9% 182077 ± 20% numa-vmstat.node1.nr_zone_inactive_anon
2295346 ± 19% -85.0% 344087 ± 19% numa-vmstat.node1.numa_hit
2201088 ± 22% -90.8% 201553 ± 38% numa-vmstat.node1.numa_local
426533 ± 13% -98.5% 6257 ± 20% meminfo.Active
426484 ± 13% -98.5% 6209 ± 21% meminfo.Active(anon)
7723073 ± 16% -57.1% 3309547 meminfo.Cached
10911723 ± 11% -41.3% 6400110 meminfo.Committed_AS
5441802 ± 22% -75.4% 1340267 meminfo.Inactive
5441575 ± 22% -75.4% 1340041 meminfo.Inactive(anon)
152735 -10.7% 136341 meminfo.KReclaimable
1472940 ± 7% -83.4% 245093 meminfo.Mapped
12189055 ± 10% -37.8% 7581637 meminfo.Memused
152735 -10.7% 136341 meminfo.SReclaimable
4639245 ± 27% -95.1% 225811 meminfo.Shmem
12381239 ± 9% -32.7% 8329522 meminfo.max_used_kB
0.05 ± 14% -52.4% 0.02 ± 18% perf-stat.i.MPKI
1.34e+11 +20.6% 1.616e+11 perf-stat.i.branch-instructions
1.17 ± 2% -0.7 0.49 ± 4% perf-stat.i.branch-miss-rate%
1.535e+09 ± 3% -53.0% 7.219e+08 ± 2% perf-stat.i.branch-misses
37.03 ± 7% -14.9 22.14 ± 15% perf-stat.i.cache-miss-rate%
21811787 ± 11% -63.8% 7902212 ± 21% perf-stat.i.cache-misses
57650952 ± 7% -39.5% 34876714 ± 11% perf-stat.i.cache-references
1.944e+08 ± 3% -58.3% 81049117 ± 4% perf-stat.i.context-switches
1.04 -17.7% 0.85 perf-stat.i.cpi
366.44 ± 3% -14.1% 314.89 ± 3% perf-stat.i.cpu-migrations
39918 ± 13% +167.9% 106957 ± 21% perf-stat.i.cycles-between-cache-misses
6.228e+11 +22.2% 7.609e+11 perf-stat.i.instructions
0.97 +22.5% 1.19 perf-stat.i.ipc
848.31 ± 2% -56.7% 367.24 ± 5% perf-stat.i.metric.K/sec
17694 ± 4% +7343.5% 1317088 ± 25% perf-stat.i.minor-faults
17694 ± 4% +7343.5% 1317089 ± 25% perf-stat.i.page-faults
24418 ± 46% +244.4% 84106 ± 20% perf-stat.overall.cycles-between-cache-misses
0.81 ± 44% +47.4% 1.19 perf-stat.overall.ipc
1.091e+11 ± 44% +44.9% 1.581e+11 perf-stat.ps.branch-instructions
5.071e+11 ± 44% +46.8% 7.443e+11 perf-stat.ps.instructions
14428 ± 44% +8787.7% 1282380 ± 24% perf-stat.ps.minor-faults
14428 ± 44% +8787.7% 1282380 ± 24% perf-stat.ps.page-faults
3.113e+13 ± 44% +46.7% 4.566e+13 perf-stat.total.instructions
107212 ± 13% -98.6% 1552 ± 21% proc-vmstat.nr_active_anon
309617 ± 2% -8.7% 282833 proc-vmstat.nr_anon_pages
6248796 +1.8% 6363802 proc-vmstat.nr_dirty_background_threshold
12512871 +1.8% 12743164 proc-vmstat.nr_dirty_threshold
1931563 ± 16% -57.2% 827520 proc-vmstat.nr_file_pages
62864244 +1.8% 64015988 proc-vmstat.nr_free_pages
1360577 ± 22% -75.3% 335437 proc-vmstat.nr_inactive_anon
367857 ± 7% -83.3% 61606 proc-vmstat.nr_mapped
8911 -10.3% 7992 ± 5% proc-vmstat.nr_page_table_pages
1160604 ± 27% -95.1% 56584 proc-vmstat.nr_shmem
38183 -10.7% 34085 proc-vmstat.nr_slab_reclaimable
107212 ± 13% -98.6% 1552 ± 21% proc-vmstat.nr_zone_active_anon
1360577 ± 22% -75.3% 335437 proc-vmstat.nr_zone_inactive_anon
250896 ± 6% -91.1% 22308 ± 29% proc-vmstat.numa_hint_faults
123468 ± 9% -93.7% 7808 ± 3% proc-vmstat.numa_hint_faults_local
2834379 ± 16% -72.1% 789773 proc-vmstat.numa_hit
2602586 ± 18% -76.9% 601311 proc-vmstat.numa_local
231785 -18.7% 188526 proc-vmstat.numa_other
74727 ± 8% -33.0% 50062 ± 21% proc-vmstat.numa_pages_migrated
923809 ± 3% -53.9% 425642 ± 10% proc-vmstat.numa_pte_updates
104416 ± 18% -48.0% 54266 proc-vmstat.pgactivate
2989481 ± 14% -66.7% 996881 proc-vmstat.pgalloc_normal
1207566 ± 2% +6626.3% 81224725 ± 24% proc-vmstat.pgfault
909425 ± 2% -27.7% 657444 ± 5% proc-vmstat.pgfree
74727 ± 8% -33.0% 50062 ± 21% proc-vmstat.pgmigrate_success
38727 ± 7% -34.0% 25562 proc-vmstat.pgreuse
90.17 -90.2 0.00 perf-profile.calltrace.cycles-pp.__sched_yield
69.08 -69.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__sched_yield
68.94 -68.9 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
46.48 -46.5 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
36.80 -36.8 0.00 perf-profile.calltrace.cycles-pp.schedule.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
34.49 -34.5 0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe
18.83 ± 2% -18.8 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
17.91 -17.9 0.00 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.__x64_sys_sched_yield.do_syscall_64
8.68 -8.7 0.00 perf-profile.calltrace.cycles-pp.do_sched_yield.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
8.13 ± 2% -8.1 0.00 perf-profile.calltrace.cycles-pp.__rseq_handle_notify_resume.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
7.60 ± 2% -7.6 0.00 perf-profile.calltrace.cycles-pp.switch_fpu_return.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sched_yield
7.04 ± 3% -7.0 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__sched_yield
6.38 ± 2% -6.4 0.00 perf-profile.calltrace.cycles-pp.yield_task_fair.do_sched_yield.__x64_sys_sched_yield.do_syscall_64.entry_SYSCALL_64_after_hwframe
6.24 ± 2% -6.2 0.00 perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__schedule.schedule.__x64_sys_sched_yield
6.09 ± 2% -6.1 0.00 perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.01 ± 2% -5.0 0.00 perf-profile.calltrace.cycles-pp.rseq_ip_fixup.__rseq_handle_notify_resume.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe
91.66 -91.7 0.00 perf-profile.children.cycles-pp.__sched_yield
70.10 -70.1 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
68.79 -68.8 0.00 perf-profile.children.cycles-pp.do_syscall_64
47.16 -47.2 0.00 perf-profile.children.cycles-pp.__x64_sys_sched_yield
36.94 -36.9 0.00 perf-profile.children.cycles-pp.schedule
35.62 -35.6 0.00 perf-profile.children.cycles-pp.__schedule
19.16 ± 2% -19.2 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
18.43 -18.4 0.00 perf-profile.children.cycles-pp.pick_next_task_fair
8.91 -8.9 0.00 perf-profile.children.cycles-pp.do_sched_yield
8.28 ± 2% -8.3 0.00 perf-profile.children.cycles-pp.__rseq_handle_notify_resume
7.72 ± 2% -7.7 0.00 perf-profile.children.cycles-pp.switch_fpu_return
7.15 ± 2% -7.1 0.00 perf-profile.children.cycles-pp.update_curr
6.61 ± 2% -6.6 0.00 perf-profile.children.cycles-pp.yield_task_fair
6.57 ± 2% -6.6 0.00 perf-profile.children.cycles-pp.put_prev_entity
6.32 ± 2% -6.3 0.00 perf-profile.children.cycles-pp.update_load_avg
6.14 ± 2% -6.1 0.00 perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
5.17 ± 2% -5.2 0.00 perf-profile.children.cycles-pp.rseq_ip_fixup
5.07 ± 2% -5.1 0.00 perf-profile.children.cycles-pp.prepare_task_switch
6.07 ± 2% -6.1 0.00 perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
5.19 ± 2% -5.2 0.00 perf-profile.self.cycles-pp.__schedule

***************************************************************************************************
lkp-icl-2sp9: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
=========================================================================================
cluster/compiler/cpufreq_governor/iterations/kconfig/nr_threads/rootfs/runtime/tbox_group/testcase:
cs-localhost/gcc-12/performance/1/x86_64-rhel-8.3/100%/debian-12-x86_64-20240206.cgz/600/lkp-icl-2sp9/tbench

commit:
cb1898f58e ("perf annotate-data: Support --skip-empty option")
a9ac22fb7a ("perf: Remove setting of page->index and ->mapping")

cb1898f58e0f175d a9ac22fb7ac3486a5bd89c25a2f
---------------- ---------------------------
%stddev %change %stddev
\ | \
7848524 -35.6% 5055935 vmstat.system.cs
99625 ± 16% +320.7% 419100 ± 2% vmstat.system.in
0.35 ± 9% +0.2 0.51 ± 7% mpstat.cpu.all.irq%
18.73 +13.5 32.27 mpstat.cpu.all.soft%
68.31 -8.5 59.81 mpstat.cpu.all.sys%
12.13 -5.0 7.17 mpstat.cpu.all.usr%
1.335e+08 -35.0% 86802870 numa-numastat.node0.local_node
1.335e+08 -35.0% 86830895 numa-numastat.node0.numa_hit
1.35e+08 -36.3% 85986323 numa-numastat.node1.local_node
1.35e+08 -36.3% 86011207 numa-numastat.node1.numa_hit
240.00 ± 11% -100.0% 0.00 perf-c2c.DRAM.local
7529 ± 6% -100.0% 0.00 perf-c2c.DRAM.remote
7785 ± 6% -100.0% 0.00 perf-c2c.HITM.local
4494 ± 4% -100.0% 0.00 perf-c2c.HITM.remote
12280 ± 3% -100.0% 0.00 perf-c2c.HITM.total
20996 -42.5% 12072 tbench.throughput_MB/s
20996 -42.5% 12072 tbench.throughput_MB/s_harmonic_mean
1.499e+09 -36.6% 9.505e+08 tbench.time.involuntary_context_switches
3512 -19.7% 2820 tbench.time.percent_of_cpu_this_job_got
20247 -14.4% 17328 tbench.time.system_time
5054 -40.8% 2991 tbench.time.user_time
1.328e+09 -34.3% 8.725e+08 tbench.time.voluntary_context_switches
80062 ± 51% -96.2% 3047 ± 70% numa-meminfo.node0.Active
80041 ± 51% -96.3% 2993 ± 71% numa-meminfo.node0.Active(anon)
94532 ± 44% -84.6% 14602 ± 24% numa-meminfo.node0.Shmem
3416559 ± 24% -99.9% 3490 ± 56% numa-meminfo.node1.Active
3416370 ± 24% -99.9% 3343 ± 55% numa-meminfo.node1.Active(anon)
179580 ± 58% +185.9% 513453 ± 32% numa-meminfo.node1.AnonPages.max
5327384 ± 30% -76.3% 1264854 ±100% numa-meminfo.node1.FilePages
106230 ± 45% -86.1% 14817 ±110% numa-meminfo.node1.Mapped
5947008 ± 28% -60.9% 2327541 ± 50% numa-meminfo.node1.MemUsed
5973 ± 84% +73.8% 10382 ± 47% numa-meminfo.node1.PageTables
3586379 ± 25% -99.8% 7080 ± 50% numa-meminfo.node1.Shmem
3495980 ± 23% -99.8% 6538 ± 18% meminfo.Active
3495769 ± 23% -99.8% 6337 ± 18% meminfo.Active(anon)
6793049 ± 13% -53.9% 3133909 meminfo.Cached
5711472 ± 15% -63.6% 2077036 meminfo.Committed_AS
860774 ± 10% -19.1% 696070 meminfo.Inactive
860058 ± 10% -19.2% 695354 meminfo.Inactive(anon)
100084 ± 2% -11.6% 88500 meminfo.KReclaimable
140027 ± 29% -69.4% 42890 meminfo.Mapped
8473548 ± 10% -40.4% 5048986 meminfo.Memused
100084 ± 2% -11.6% 88500 meminfo.SReclaimable
3680778 ± 24% -99.4% 21686 meminfo.Shmem
8658135 ± 10% -39.6% 5229555 meminfo.max_used_kB
20021 ± 51% -96.3% 748.82 ± 71% numa-vmstat.node0.nr_active_anon
23635 ± 44% -84.6% 3651 ± 24% numa-vmstat.node0.nr_shmem
20021 ± 51% -96.3% 748.82 ± 71% numa-vmstat.node0.nr_zone_active_anon
1.335e+08 -35.0% 86830064 numa-vmstat.node0.numa_hit
1.335e+08 -35.0% 86802040 numa-vmstat.node0.numa_local
854348 ± 24% -99.9% 835.66 ± 55% numa-vmstat.node1.nr_active_anon
1331772 ± 30% -76.3% 316213 ±100% numa-vmstat.node1.nr_file_pages
26385 ± 45% -85.6% 3786 ±110% numa-vmstat.node1.nr_mapped
1493 ± 84% +73.7% 2595 ± 47% numa-vmstat.node1.nr_page_table_pages
896521 ± 25% -99.8% 1770 ± 50% numa-vmstat.node1.nr_shmem
854348 ± 24% -99.9% 835.66 ± 55% numa-vmstat.node1.nr_zone_active_anon
1.35e+08 -36.3% 86010078 numa-vmstat.node1.numa_hit
1.35e+08 -36.3% 85985195 numa-vmstat.node1.numa_local
874132 ± 23% -99.8% 1584 ± 18% proc-vmstat.nr_active_anon
6344454 +1.3% 6429945 proc-vmstat.nr_dirty_background_threshold
12704422 +1.3% 12875612 proc-vmstat.nr_dirty_threshold
1698254 ± 13% -53.9% 783477 proc-vmstat.nr_file_pages
63822128 +1.3% 64678294 proc-vmstat.nr_free_pages
214803 ± 10% -19.1% 173832 proc-vmstat.nr_inactive_anon
35091 ± 29% -68.7% 10970 proc-vmstat.nr_mapped
920185 ± 24% -99.4% 5420 proc-vmstat.nr_shmem
25021 ± 2% -11.6% 22125 proc-vmstat.nr_slab_reclaimable
874132 ± 23% -99.8% 1584 ± 18% proc-vmstat.nr_zone_active_anon
214803 ± 10% -19.1% 173832 proc-vmstat.nr_zone_inactive_anon
119453 ± 26% -95.7% 5091 ± 59% proc-vmstat.numa_hint_faults
65444 ± 36% -98.4% 1036 ± 25% proc-vmstat.numa_hint_faults_local
2.685e+08 -35.6% 1.728e+08 proc-vmstat.numa_hit
2.685e+08 -35.6% 1.728e+08 proc-vmstat.numa_local
66380 -18.6% 54024 proc-vmstat.numa_other
252531 ± 14% -60.8% 99021 ± 17% proc-vmstat.numa_pte_updates
1163823 ± 24% -99.9% 776.50 ± 23% proc-vmstat.pgactivate
2.129e+09 -35.5% 1.373e+09 proc-vmstat.pgalloc_normal
2063471 ± 2% +45810.5% 9.474e+08 ± 9% proc-vmstat.pgfault
2.128e+09 -35.5% 1.373e+09 proc-vmstat.pgfree
85685 ± 9% -11.0% 76256 proc-vmstat.pgreuse
17143370 -10.1% 15417411 sched_debug.cfs_rq:/.avg_vruntime.avg
16788080 -10.5% 15023795 sched_debug.cfs_rq:/.avg_vruntime.min
147023 ± 26% +96.3% 288623 ± 26% sched_debug.cfs_rq:/.avg_vruntime.stddev
8563 ± 18% -47.1% 4529 ± 42% sched_debug.cfs_rq:/.load.min
17143370 -10.1% 15417411 sched_debug.cfs_rq:/.min_vruntime.avg
16788080 -10.5% 15023795 sched_debug.cfs_rq:/.min_vruntime.min
147023 ± 26% +96.3% 288623 ± 26% sched_debug.cfs_rq:/.min_vruntime.stddev
1805 ± 3% +15.6% 2086 sched_debug.cfs_rq:/.runnable_avg.max
1079 ± 7% -15.8% 908.85 ± 6% sched_debug.cfs_rq:/.runnable_avg.min
98.24 ± 12% +92.3% 188.89 ± 3% sched_debug.cfs_rq:/.runnable_avg.stddev
51.17 ± 16% +35.0% 69.10 ± 9% sched_debug.cfs_rq:/.util_avg.stddev
1176 ± 8% +41.6% 1665 ± 4% sched_debug.cfs_rq:/.util_est.max
241.66 ± 3% +16.9% 282.59 ± 2% sched_debug.cfs_rq:/.util_est.stddev
620439 ± 19% +40.4% 871331 ± 4% sched_debug.cpu.avg_idle.max
6766 ± 3% +94.0% 13129 ± 4% sched_debug.cpu.avg_idle.min
8.69 ± 4% +31.4% 11.41 ± 6% sched_debug.cpu.clock.stddev
327685 -12.4% 287198 sched_debug.cpu.clock_task.avg
331298 -10.1% 297931 sched_debug.cpu.clock_task.max
319890 -12.6% 279574 sched_debug.cpu.clock_task.min
1712 ± 10% +72.8% 2960 ± 11% sched_debug.cpu.clock_task.stddev
2339 ± 10% -48.6% 1202 ± 74% sched_debug.cpu.max_idle_balance_cost.stddev
0.00 ± 6% +27.9% 0.00 ± 20% sched_debug.cpu.next_balance.stddev
44202960 -28.6% 31561513 sched_debug.cpu.nr_switches.avg
45127928 -27.9% 32529217 sched_debug.cpu.nr_switches.max
42497993 -35.6% 27381916 ± 4% sched_debug.cpu.nr_switches.min
489510 ± 7% +87.5% 918047 ± 17% sched_debug.cpu.nr_switches.stddev
0.20 ± 12% -42.7% 0.12 ± 7% perf-stat.i.MPKI
3.212e+10 +38.9% 4.463e+10 perf-stat.i.branch-instructions
1.01 -0.4 0.64 perf-stat.i.branch-miss-rate%
3.214e+08 -15.3% 2.721e+08 perf-stat.i.branch-misses
32.93 ± 4% -2.2 30.71 ± 2% perf-stat.i.cache-miss-rate%
32055029 ± 12% -26.8% 23468576 ± 7% perf-stat.i.cache-misses
1.042e+08 ± 13% -24.7% 78443526 ± 5% perf-stat.i.cache-references
7869989 -35.6% 5069963 perf-stat.i.context-switches
1.40 -24.6% 1.06 perf-stat.i.cpi
7138 ± 12% +40.2% 10010 ± 7% perf-stat.i.cycles-between-cache-misses
1.601e+11 +34.5% 2.153e+11 perf-stat.i.instructions
0.71 +34.8% 0.96 perf-stat.i.ipc
2740 ± 2% +47765.7% 1311791 ± 9% perf-stat.i.minor-faults
2740 ± 2% +47765.5% 1311791 ± 9% perf-stat.i.page-faults
0.20 ± 12% -45.6% 0.11 ± 7% perf-stat.overall.MPKI
1.00 -0.4 0.61 perf-stat.overall.branch-miss-rate%
1.40 -25.9% 1.04 perf-stat.overall.cpi
7104 ± 12% +35.0% 9592 ± 8% perf-stat.overall.cycles-between-cache-misses
0.71 +34.9% 0.96 perf-stat.overall.ipc
3.207e+10 +38.9% 4.455e+10 perf-stat.ps.branch-instructions
3.209e+08 -15.3% 2.717e+08 perf-stat.ps.branch-misses
32011532 ± 12% -26.8% 23433423 ± 7% perf-stat.ps.cache-misses
1.042e+08 ± 13% -24.8% 78378749 ± 5% perf-stat.ps.cache-references
7857872 -35.6% 5062068 perf-stat.ps.context-switches
1.598e+11 +34.5% 2.149e+11 perf-stat.ps.instructions
2738 ± 2% +47722.7% 1309682 ± 9% perf-stat.ps.minor-faults
2738 ± 2% +47722.5% 1309682 ± 9% perf-stat.ps.page-faults
1.153e+14 +34.5% 1.551e+14 perf-stat.total.instructions
96.07 -96.1 0.00 perf-profile.calltrace.cycles-pp.main
60.65 -60.7 0.00 perf-profile.calltrace.cycles-pp.create_procs.main
42.12 -42.1 0.00 perf-profile.calltrace.cycles-pp.child_run.create_procs.main
40.64 -40.6 0.00 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send
39.19 -39.2 0.00 perf-profile.calltrace.cycles-pp.tcp_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
38.18 -38.2 0.00 perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
37.37 -37.4 0.00 perf-profile.calltrace.cycles-pp.child_op.child_run.create_procs.main
35.42 -35.4 0.00 perf-profile.calltrace.cycles-pp.listener.main
35.36 -35.4 0.00 perf-profile.calltrace.cycles-pp.server.listener.main
30.48 -30.5 0.00 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.__sys_sendto.__x64_sys_sendto
30.20 -30.2 0.00 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.__sys_sendto
27.92 -27.9 0.00 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg
24.93 -24.9 0.00 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked
23.36 -23.4 0.00 perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
22.77 -22.8 0.00 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit
21.45 -21.4 0.00 perf-profile.calltrace.cycles-pp.__send.server.listener.main
21.42 -21.4 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets
20.65 -20.7 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.server.listener.main
20.61 -20.6 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.server.listener
19.40 -19.4 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.server
18.50 -18.5 0.00 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb
18.31 ± 2% -18.3 0.00 perf-profile.calltrace.cycles-pp.kill.create_procs.main
18.30 -18.3 0.00 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
18.07 -18.1 0.00 perf-profile.calltrace.cycles-pp.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
17.70 ± 2% -17.7 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill.create_procs.main
17.67 ± 2% -17.7 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill.create_procs.main
17.14 ± 2% -17.1 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill.create_procs
17.05 -17.0 0.00 perf-profile.calltrace.cycles-pp.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
16.77 ± 3% -16.8 0.00 perf-profile.calltrace.cycles-pp.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill
16.50 ± 3% -16.5 0.00 perf-profile.calltrace.cycles-pp.kill_pid_info_type.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe
16.42 -16.4 0.00 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip
16.31 -16.3 0.00 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.handle_softirqs.do_softirq
15.90 -15.9 0.00 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv
15.76 -15.8 0.00 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.handle_softirqs
13.94 -13.9 0.00 perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
13.43 -13.4 0.00 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
13.38 -13.4 0.00 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
13.28 -13.3 0.00 perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
13.15 -13.2 0.00 perf-profile.calltrace.cycles-pp.recv.server.listener.main
13.15 -13.1 0.00 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
12.97 -13.0 0.00 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
12.60 ± 3% -12.6 0.00 perf-profile.calltrace.cycles-pp.security_task_kill.kill_pid_info_type.kill_something_info.__x64_sys_kill.do_syscall_64
12.57 ± 3% -12.6 0.00 perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.kill_pid_info_type.kill_something_info.__x64_sys_kill
11.84 -11.8 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.server.listener.main
11.76 -11.8 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.server.listener
11.29 -11.3 0.00 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
11.21 -11.2 0.00 perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom
10.98 -11.0 0.00 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
10.56 -10.6 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.server
9.59 -9.6 0.00 perf-profile.calltrace.cycles-pp.sio_readx.child_op.child_run.create_procs.main
8.73 -8.7 0.00 perf-profile.calltrace.cycles-pp.do_packets.sio_readx.child_op.child_run.create_procs
6.57 -6.6 0.00 perf-profile.calltrace.cycles-pp.__send.do_packets.sio_readx.child_op.child_run
6.36 -6.4 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_readx.child_op
6.35 -6.4 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send.do_packets.sio_readx
6.06 -6.1 0.00 perf-profile.calltrace.cycles-pp.sio_createx.child_op.child_run.create_procs.main
5.90 -5.9 0.00 perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg
5.57 -5.6 0.00 perf-profile.calltrace.cycles-pp.do_packets.sio_createx.child_op.child_run.create_procs
5.49 -5.5 0.00 perf-profile.calltrace.cycles-pp.sio_qpathinfo.child_op.child_run.create_procs.main
5.09 -5.1 0.00 perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg
5.04 -5.0 0.00 perf-profile.calltrace.cycles-pp.do_packets.sio_qpathinfo.child_op.child_run.create_procs
96.13 -96.1 0.00 perf-profile.children.cycles-pp.main
85.56 -85.6 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
84.98 -85.0 0.00 perf-profile.children.cycles-pp.do_syscall_64
60.65 -60.7 0.00 perf-profile.children.cycles-pp.create_procs
48.42 -48.4 0.00 perf-profile.children.cycles-pp.__send
42.26 -42.3 0.00 perf-profile.children.cycles-pp.child_run
41.59 -41.6 0.00 perf-profile.children.cycles-pp.__x64_sys_sendto
41.40 -41.4 0.00 perf-profile.children.cycles-pp.__sys_sendto
39.97 -40.0 0.00 perf-profile.children.cycles-pp.tcp_sendmsg
38.93 -38.9 0.00 perf-profile.children.cycles-pp.tcp_sendmsg_locked
37.51 -37.5 0.00 perf-profile.children.cycles-pp.child_op
35.42 -35.4 0.00 perf-profile.children.cycles-pp.listener
35.38 -35.4 0.00 perf-profile.children.cycles-pp.server
33.33 -33.3 0.00 perf-profile.children.cycles-pp.do_packets
32.27 -32.3 0.00 perf-profile.children.cycles-pp.tcp_write_xmit
32.00 -32.0 0.00 perf-profile.children.cycles-pp.__tcp_push_pending_frames
30.08 -30.1 0.00 perf-profile.children.cycles-pp.__tcp_transmit_skb
27.32 -27.3 0.00 perf-profile.children.cycles-pp.__ip_queue_xmit
25.59 -25.6 0.00 perf-profile.children.cycles-pp.ip_finish_output2
25.22 -25.2 0.00 perf-profile.children.cycles-pp.recv
24.96 -25.0 0.00 perf-profile.children.cycles-pp.__dev_queue_xmit
20.53 -20.5 0.00 perf-profile.children.cycles-pp.__local_bh_enable_ip
20.06 -20.1 0.00 perf-profile.children.cycles-pp.do_softirq
19.82 -19.8 0.00 perf-profile.children.cycles-pp.handle_softirqs
19.19 -19.2 0.00 perf-profile.children.cycles-pp.__x64_sys_recvfrom
18.92 -18.9 0.00 perf-profile.children.cycles-pp.__sys_recvfrom
18.67 -18.7 0.00 perf-profile.children.cycles-pp.net_rx_action
18.44 ± 2% -18.4 0.00 perf-profile.children.cycles-pp.kill
17.99 -18.0 0.00 perf-profile.children.cycles-pp.__napi_poll
17.88 -17.9 0.00 perf-profile.children.cycles-pp.process_backlog
17.27 -17.3 0.00 perf-profile.children.cycles-pp.__netif_receive_skb_one_core
17.16 ± 2% -17.2 0.00 perf-profile.children.cycles-pp.__x64_sys_kill
16.99 -17.0 0.00 perf-profile.children.cycles-pp.sock_recvmsg
16.78 ± 3% -16.8 0.00 perf-profile.children.cycles-pp.kill_something_info
16.51 ± 3% -16.5 0.00 perf-profile.children.cycles-pp.kill_pid_info_type
16.03 -16.0 0.00 perf-profile.children.cycles-pp.inet_recvmsg
15.70 -15.7 0.00 perf-profile.children.cycles-pp.tcp_recvmsg
15.25 -15.2 0.00 perf-profile.children.cycles-pp.ip_local_deliver_finish
15.20 -15.2 0.00 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
14.97 -15.0 0.00 perf-profile.children.cycles-pp.tcp_v4_rcv
13.87 -13.9 0.00 perf-profile.children.cycles-pp.tcp_recvmsg_locked
12.95 -12.9 0.00 perf-profile.children.cycles-pp.tcp_v4_do_rcv
12.66 -12.7 0.00 perf-profile.children.cycles-pp.tcp_rcv_established
12.61 ± 3% -12.6 0.00 perf-profile.children.cycles-pp.security_task_kill
12.58 ± 3% -12.6 0.00 perf-profile.children.cycles-pp.apparmor_task_kill
9.59 -9.6 0.00 perf-profile.children.cycles-pp.sio_readx
8.65 -8.6 0.00 perf-profile.children.cycles-pp.schedule
8.55 -8.6 0.00 perf-profile.children.cycles-pp.__schedule
7.75 -7.8 0.00 perf-profile.children.cycles-pp.sk_wait_data
6.60 -6.6 0.00 perf-profile.children.cycles-pp.wait_woken
6.23 -6.2 0.00 perf-profile.children.cycles-pp.schedule_timeout
6.07 -6.1 0.00 perf-profile.children.cycles-pp.sio_createx
5.70 -5.7 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
5.49 -5.5 0.00 perf-profile.children.cycles-pp.sio_qpathinfo
5.08 -5.1 0.00 perf-profile.children.cycles-pp.sock_def_readable
10.63 ± 3% -10.6 0.00 perf-profile.self.cycles-pp.apparmor_task_kill

***************************************************************************************************
lkp-icl-2sp9: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
=========================================================================================
cluster/compiler/cpufreq_governor/datasize/kconfig/nr_threads/protocol/rootfs/runtime/tbox_group/testcase/winsize:
cs-localhost/gcc-12/performance/64/x86_64-rhel-8.3/1/UDP/debian-12-x86_64-20240206.cgz/30s/lkp-icl-2sp9/uperf/32k

commit:
cb1898f58e ("perf annotate-data: Support --skip-empty option")
a9ac22fb7a ("perf: Remove setting of page->index and ->mapping")

cb1898f58e0f175d a9ac22fb7ac3486a5bd89c25a2f
---------------- ---------------------------
%stddev %change %stddev
\ | \
6808243 -25.8% 5054865 cpuidle..usage
34.51 -1.7% 33.94 boot-time.boot
2003 -1.8% 1966 boot-time.idle
313841 ± 3% -72.4% 86605 ± 25% numa-numastat.node1.local_node
334568 -67.5% 108757 ± 15% numa-numastat.node1.numa_hit
3.53 ± 3% +55.6% 5.49 vmstat.procs.r
367355 ± 3% -24.3% 277978 vmstat.system.cs
197397 ± 3% -12.2% 173405 vmstat.system.in
0.17 ± 2% -0.0 0.15 mpstat.cpu.all.irq%
0.27 ± 3% +0.2 0.47 mpstat.cpu.all.soft%
1.07 ± 5% +2.9 3.97 mpstat.cpu.all.sys%
5.63 ± 6% +38.5% 7.80 ± 5% mpstat.max_utilization_pct
11.33 ± 41% -100.0% 0.00 perf-c2c.DRAM.local
65.17 ± 64% -100.0% 0.00 perf-c2c.DRAM.remote
2717 ± 8% -100.0% 0.00 perf-c2c.HITM.local
32.17 ± 90% -100.0% 0.00 perf-c2c.HITM.remote
31117 ± 24% -54.1% 14280 ± 87% numa-meminfo.node0.Mapped
505070 -99.7% 1522 ± 24% numa-meminfo.node1.Active
504998 -99.7% 1490 ± 24% numa-meminfo.node1.Active(anon)
44594 ± 30% -60.3% 17722 ± 69% numa-meminfo.node1.Mapped
595059 -99.6% 2232 ± 16% numa-meminfo.node1.Shmem
151.45 -25.4% 112.99 uperf.Netstat_lo_in_Mbps
151.45 -25.4% 112.99 uperf.Netstat_lo_out_Mbps
105.36 -25.4% 78.60 uperf.Runstat_127.0.0.1_Mbps
105.36 -25.4% 78.60 uperf.Runstat_master_Mbps
25.17 -12.6% 22.00 uperf.time.percent_of_cpu_this_job_got
3325771 -25.8% 2467986 uperf.time.voluntary_context_switches
7923 ± 25% -53.9% 3654 ± 86% numa-vmstat.node0.nr_mapped
126255 -99.7% 373.42 ± 24% numa-vmstat.node1.nr_active_anon
11390 ± 30% -59.6% 4604 ± 68% numa-vmstat.node1.nr_mapped
148846 -99.6% 558.20 ± 16% numa-vmstat.node1.nr_shmem
126255 -99.7% 373.42 ± 24% numa-vmstat.node1.nr_zone_active_anon
334029 -67.7% 107918 ± 15% numa-vmstat.node1.numa_hit
313302 ± 3% -72.6% 85796 ± 25% numa-vmstat.node1.numa_local
506511 -99.5% 2716 meminfo.Active
506389 -99.5% 2652 meminfo.Active(anon)
3666247 -16.2% 3072894 meminfo.Cached
1380898 -41.7% 804785 meminfo.Committed_AS
730235 -11.6% 645257 meminfo.Inactive
729851 -11.6% 645012 meminfo.Inactive(anon)
75195 ± 17% -57.7% 31826 meminfo.Mapped
5569857 -9.0% 5068038 meminfo.Memused
606477 -97.8% 13355 meminfo.Shmem
5802899 -11.8% 5115559 meminfo.max_used_kB
126602 -99.5% 664.74 proc-vmstat.nr_active_anon
916622 -16.2% 768228 proc-vmstat.nr_file_pages
182518 -11.7% 161242 proc-vmstat.nr_inactive_anon
19151 ± 17% -57.1% 8207 proc-vmstat.nr_mapped
1519 +3.7% 1575 proc-vmstat.nr_page_table_pages
151678 -97.8% 3342 proc-vmstat.nr_shmem
22637 -3.8% 21768 proc-vmstat.nr_slab_reclaimable
126602 -99.5% 664.74 proc-vmstat.nr_zone_active_anon
182518 -11.7% 161242 proc-vmstat.nr_zone_inactive_anon
465325 -52.6% 220544 ± 2% proc-vmstat.numa_hit
399117 -58.2% 166673 ± 3% proc-vmstat.numa_local
66208 -18.7% 53838 proc-vmstat.numa_other
206736 -100.0% 33.17 ± 20% proc-vmstat.pgactivate
492731 -49.5% 248776 ± 2% proc-vmstat.pgalloc_normal
208304 ± 5% +54244.8% 1.132e+08 proc-vmstat.pgfault
275792 ± 2% -51.0% 135053 ± 4% proc-vmstat.pgfree
0.23 ± 3% -59.3% 0.09 ± 5% perf-stat.i.MPKI
1.982e+09 +106.5% 4.094e+09 perf-stat.i.branch-instructions
3.04 ± 2% -1.6 1.48 perf-stat.i.branch-miss-rate%
68611062 ± 2% -18.7% 55765171 perf-stat.i.branch-misses
7.33 ± 4% -1.3 6.00 ± 4% perf-stat.i.cache-miss-rate%
2783561 ± 3% -39.1% 1695747 ± 5% perf-stat.i.cache-misses
35458378 -19.3% 28615257 ± 2% perf-stat.i.cache-references
399282 ± 3% -24.1% 303251 perf-stat.i.context-switches
1.05 -20.3% 0.84 perf-stat.i.cpi
9.455e+09 ± 2% +74.1% 1.646e+10 perf-stat.i.cpu-cycles
137.55 -4.4% 131.46 perf-stat.i.cpu-migrations
11616 ± 17% +77.4% 20608 ± 4% perf-stat.i.cycles-between-cache-misses
9.825e+09 +101.0% 1.975e+10 perf-stat.i.instructions
1.01 +19.0% 1.20 perf-stat.i.ipc
6.21 ± 3% +1629.6% 107.36 perf-stat.i.metric.K/sec
4804 ± 6% +68281.0% 3285377 perf-stat.i.minor-faults
4804 ± 6% +68280.4% 3285377 perf-stat.i.page-faults
0.28 ± 2% -69.8% 0.09 ± 5% perf-stat.overall.MPKI
3.46 -2.1 1.36 perf-stat.overall.branch-miss-rate%
7.86 ± 2% -1.9 5.92 ± 6% perf-stat.overall.cache-miss-rate%
0.96 -13.4% 0.83 perf-stat.overall.cpi
3395 ± 2% +187.6% 9765 ± 5% perf-stat.overall.cycles-between-cache-misses
1.04 +15.5% 1.20 perf-stat.overall.ipc
1.925e+09 +106.2% 3.971e+09 perf-stat.ps.branch-instructions
66548387 -19.0% 53931766 perf-stat.ps.branch-misses
2708553 ± 3% -39.4% 1641008 ± 5% perf-stat.ps.cache-misses
34460219 -19.4% 27762399 ± 2% perf-stat.ps.cache-references
387430 ± 3% -24.1% 294073 perf-stat.ps.context-switches
9.192e+09 ± 2% +73.8% 1.598e+10 perf-stat.ps.cpu-cycles
133.55 -4.5% 127.51 perf-stat.ps.cpu-migrations
9.543e+09 +100.7% 1.916e+10 perf-stat.ps.instructions
4668 ± 6% +68177.7% 3187819 perf-stat.ps.minor-faults
4668 ± 6% +68177.1% 3187819 perf-stat.ps.page-faults
3.268e+11 +94.9% 6.368e+11 perf-stat.total.instructions
63.98 ± 3% -64.0 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
47.37 ± 2% -47.4 0.00 perf-profile.calltrace.cycles-pp.common_startup_64
45.61 ± 6% -45.6 0.00 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
45.57 ± 6% -45.6 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
45.47 ± 6% -45.5 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
41.88 ± 6% -41.9 0.00 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
41.79 ± 2% -41.8 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
41.07 ± 2% -41.1 0.00 perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
40.56 ± 6% -40.6 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
22.56 ± 2% -22.6 0.00 perf-profile.calltrace.cycles-pp.sendto
21.42 ± 2% -21.4 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto
21.36 ± 2% -21.4 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
21.01 ± 2% -21.0 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
20.87 ± 2% -20.9 0.00 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto
19.40 ± 2% -19.4 0.00 perf-profile.calltrace.cycles-pp.udpv6_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe
19.07 ± 3% -19.1 0.00 perf-profile.calltrace.cycles-pp.recvfrom
18.86 ± 2% -18.9 0.00 perf-profile.calltrace.cycles-pp.udp_sendmsg.udpv6_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64
16.78 ± 4% -16.8 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvfrom
16.72 ± 4% -16.7 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom
15.37 ± 4% -15.4 0.00 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom
15.24 ± 4% -15.2 0.00 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom
14.58 ± 5% -14.6 0.00 perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
13.60 ± 3% -13.6 0.00 perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe
13.43 ± 3% -13.4 0.00 perf-profile.calltrace.cycles-pp.inet6_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64
13.32 ± 3% -13.3 0.00 perf-profile.calltrace.cycles-pp.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom
12.52 ± 4% -12.5 0.00 perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.udpv6_sendmsg.__sys_sendto.__x64_sys_sendto
12.18 ± 4% -12.2 0.00 perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.udpv6_sendmsg.__sys_sendto
11.64 ± 4% -11.6 0.00 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.udpv6_sendmsg
11.35 ± 4% -11.3 0.00 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg
9.29 ± 4% -9.3 0.00 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb
9.14 ± 4% -9.1 0.00 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb
9.13 ± 9% -9.1 0.00 perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state
8.98 ± 4% -9.0 0.00 perf-profile.calltrace.cycles-pp.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
8.83 -8.8 0.00 perf-profile.calltrace.cycles-pp.__skb_recv_udp.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg.__sys_recvfrom
8.27 ± 4% -8.3 0.00 perf-profile.calltrace.cycles-pp.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
7.80 ± 4% -7.8 0.00 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip
7.68 ± 4% -7.7 0.00 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.handle_softirqs.do_softirq
7.37 ± 4% -7.4 0.00 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.handle_softirqs
6.77 ± 4% -6.8 0.00 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
6.73 ± 4% -6.7 0.00 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
6.54 ± 4% -6.5 0.00 perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
6.40 ± 2% -6.4 0.00 perf-profile.calltrace.cycles-pp.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg.inet6_recvmsg.sock_recvmsg
6.23 ± 9% -6.2 0.00 perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter
5.95 ± 4% -5.9 0.00 perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
5.89 ± 4% -5.9 0.00 perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
5.72 ± 9% -5.7 0.00 perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt
5.13 ± 5% -5.1 0.00 perf-profile.calltrace.cycles-pp.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udpv6_recvmsg.inet6_recvmsg
4.97 ± 3% -5.0 0.00 perf-profile.calltrace.cycles-pp.__udp_enqueue_schedule_skb.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu
47.37 ± 2% -47.4 0.00 perf-profile.children.cycles-pp.common_startup_64
47.37 ± 2% -47.4 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
47.27 ± 2% -47.3 0.00 perf-profile.children.cycles-pp.do_idle
45.61 ± 6% -45.6 0.00 perf-profile.children.cycles-pp.start_secondary
43.59 ± 2% -43.6 0.00 perf-profile.children.cycles-pp.cpuidle_idle_call
42.19 ± 2% -42.2 0.00 perf-profile.children.cycles-pp.cpuidle_enter
41.99 ± 2% -42.0 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
41.83 -41.8 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
41.55 -41.5 0.00 perf-profile.children.cycles-pp.do_syscall_64
41.23 ± 2% -41.2 0.00 perf-profile.children.cycles-pp.acpi_idle_enter
41.18 ± 2% -41.2 0.00 perf-profile.children.cycles-pp.acpi_safe_halt
37.04 ± 2% -37.0 0.00 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
22.75 ± 2% -22.7 0.00 perf-profile.children.cycles-pp.sendto
21.04 ± 2% -21.0 0.00 perf-profile.children.cycles-pp.__x64_sys_sendto
20.92 ± 2% -20.9 0.00 perf-profile.children.cycles-pp.__sys_sendto
19.44 ± 2% -19.4 0.00 perf-profile.children.cycles-pp.udpv6_sendmsg
19.25 ± 3% -19.3 0.00 perf-profile.children.cycles-pp.recvfrom
18.95 ± 2% -19.0 0.00 perf-profile.children.cycles-pp.udp_sendmsg
15.40 ± 4% -15.4 0.00 perf-profile.children.cycles-pp.__x64_sys_recvfrom
15.27 ± 4% -15.3 0.00 perf-profile.children.cycles-pp.__sys_recvfrom
13.62 ± 3% -13.6 0.00 perf-profile.children.cycles-pp.sock_recvmsg
13.43 ± 3% -13.4 0.00 perf-profile.children.cycles-pp.inet6_recvmsg
13.34 ± 3% -13.3 0.00 perf-profile.children.cycles-pp.udpv6_recvmsg
12.54 ± 4% -12.5 0.00 perf-profile.children.cycles-pp.udp_send_skb
12.20 ± 4% -12.2 0.00 perf-profile.children.cycles-pp.ip_send_skb
11.66 ± 4% -11.7 0.00 perf-profile.children.cycles-pp.ip_finish_output2
11.39 ± 4% -11.4 0.00 perf-profile.children.cycles-pp.__dev_queue_xmit
9.53 ± 4% -9.5 0.00 perf-profile.children.cycles-pp.handle_softirqs
9.37 ± 8% -9.4 0.00 perf-profile.children.cycles-pp.sysvec_call_function_single
9.35 ± 4% -9.3 0.00 perf-profile.children.cycles-pp.__local_bh_enable_ip
9.15 ± 4% -9.2 0.00 perf-profile.children.cycles-pp.do_softirq
8.88 -8.9 0.00 perf-profile.children.cycles-pp.__skb_recv_udp
8.28 ± 4% -8.3 0.00 perf-profile.children.cycles-pp.net_rx_action
7.82 ± 4% -7.8 0.00 perf-profile.children.cycles-pp.__napi_poll
7.70 ± 4% -7.7 0.00 perf-profile.children.cycles-pp.process_backlog
7.38 ± 4% -7.4 0.00 perf-profile.children.cycles-pp.__netif_receive_skb_one_core
7.05 -7.0 0.00 perf-profile.children.cycles-pp.__schedule
6.78 ± 4% -6.8 0.00 perf-profile.children.cycles-pp.ip_local_deliver_finish
6.75 ± 4% -6.7 0.00 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
6.56 ± 4% -6.6 0.00 perf-profile.children.cycles-pp.__udp4_lib_rcv
6.42 ± 2% -6.4 0.00 perf-profile.children.cycles-pp.__skb_wait_for_more_packets
6.31 ± 8% -6.3 0.00 perf-profile.children.cycles-pp.__sysvec_call_function_single
5.96 ± 4% -6.0 0.00 perf-profile.children.cycles-pp.udp_unicast_rcv_skb
5.92 ± 4% -5.9 0.00 perf-profile.children.cycles-pp.udp_queue_rcv_one_skb
5.80 ± 8% -5.8 0.00 perf-profile.children.cycles-pp.__flush_smp_call_function_queue
5.16 ± 4% -5.2 0.00 perf-profile.children.cycles-pp.schedule_timeout
4.98 ± 3% -5.0 0.00 perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb
29.86 ± 4% -29.9 0.00 perf-profile.self.cycles-pp.acpi_safe_halt

***************************************************************************************************
lkp-icl-2sp2: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/50%/debian-12-x86_64-20240206.cgz/300s/lkp-icl-2sp2/SCTP_RR/netperf

commit:
cb1898f58e ("perf annotate-data: Support --skip-empty option")
a9ac22fb7a ("perf: Remove setting of page->index and ->mapping")

cb1898f58e0f175d a9ac22fb7ac3486a5bd89c25a2f
---------------- ---------------------------
%stddev %change %stddev
\ | \
20146 -14.1% 17300 ± 2% uptime.idle
1.568e+10 -20.5% 1.246e+10 cpuidle..time
1.702e+09 -32.2% 1.154e+09 cpuidle..usage
3115754 -88.3% 365170 ± 16% numa-numastat.node1.local_node
3215262 -87.0% 419236 ± 8% numa-numastat.node1.numa_hit
37.40 -16.6% 31.18 vmstat.cpu.id
11031758 -31.8% 7527987 vmstat.system.cs
5666321 -23.4% 4342549 vmstat.system.in
37.03 -6.3 30.75 mpstat.cpu.all.idle%
3.69 -0.7 3.02 mpstat.cpu.all.irq%
18.00 +7.2 25.18 mpstat.cpu.all.soft%
2.07 -0.7 1.38 mpstat.cpu.all.usr%
81.67 ± 19% -100.0% 0.00 perf-c2c.DRAM.local
3324 ± 14% -100.0% 0.00 perf-c2c.DRAM.remote
246856 -100.0% 0.00 perf-c2c.HITM.local
1771 ± 11% -100.0% 0.00 perf-c2c.HITM.remote
248627 -100.0% 0.00 perf-c2c.HITM.total
2836525 -31.4% 1944538 netperf.ThroughputBoth_total_tps
44320 -31.4% 30383 netperf.ThroughputBoth_tps
2836525 -31.4% 1944538 netperf.Throughput_total_tps
44320 -31.4% 30383 netperf.Throughput_tps
15538 ± 2% +1516.1% 251120 ± 5% netperf.time.involuntary_context_switches
2180 -18.6% 1774 netperf.time.percent_of_cpu_this_job_got
6401 -17.7% 5264 netperf.time.system_time
253.51 -40.6% 150.54 netperf.time.user_time
8.508e+08 -31.5% 5.829e+08 netperf.time.voluntary_context_switches
8.51e+08 -31.4% 5.834e+08 netperf.workload
3568833 ± 6% -99.9% 5313 ± 2% meminfo.Active
3568785 ± 6% -99.9% 5273 ± 2% meminfo.Active(anon)
10615057 -71.1% 3072650 meminfo.Cached
8430375 -89.9% 851992 meminfo.Committed_AS
4707200 ± 4% -85.5% 681046 meminfo.Inactive
4706988 ± 4% -85.5% 680826 meminfo.Inactive(anon)
130914 -16.5% 109250 meminfo.KReclaimable
2445242 ± 6% -98.6% 33828 meminfo.Mapped
13265505 -55.6% 5893945 meminfo.Memused
16829 ± 2% -26.3% 12400 meminfo.PageTables
130914 -16.5% 109250 meminfo.SReclaimable
7553109 -99.9% 11212 meminfo.Shmem
13506404 -54.0% 6207695 meminfo.max_used_kB
225772 ± 64% -38.0% 139999 ±100% numa-meminfo.node0.AnonPages
227536 ± 64% -37.7% 141653 ± 99% numa-meminfo.node0.Inactive
227365 ± 64% -37.8% 141435 ± 99% numa-meminfo.node0.Inactive(anon)
122078 ± 8% -85.9% 17153 ± 74% numa-meminfo.node0.Mapped
9967 ± 5% -38.4% 6143 ± 8% numa-meminfo.node0.PageTables
3547939 ± 6% -99.9% 2434 ± 19% numa-meminfo.node1.Active
3547931 ± 6% -99.9% 2434 ± 19% numa-meminfo.node1.Active(anon)
10022700 ± 6% -84.9% 1510608 ± 82% numa-meminfo.node1.FilePages
4491145 ± 7% -88.0% 539362 ± 26% numa-meminfo.node1.Inactive
4491105 ± 7% -88.0% 539359 ± 26% numa-meminfo.node1.Inactive(anon)
2331352 ± 6% -99.3% 16689 ± 75% numa-meminfo.node1.Mapped
11420187 ± 6% -73.5% 3022233 ± 39% numa-meminfo.node1.MemUsed
7538938 -100.0% 3512 ± 13% numa-meminfo.node1.Shmem
56484 ± 64% -38.0% 34994 ±100% numa-vmstat.node0.nr_anon_pages
56882 ± 64% -37.8% 35353 ± 99% numa-vmstat.node0.nr_inactive_anon
30662 ± 9% -85.5% 4434 ± 73% numa-vmstat.node0.nr_mapped
2501 ± 5% -38.5% 1537 ± 8% numa-vmstat.node0.nr_page_table_pages
56882 ± 64% -37.8% 35353 ± 99% numa-vmstat.node0.nr_zone_inactive_anon
889164 ± 6% -99.9% 608.62 ± 19% numa-vmstat.node1.nr_active_anon
2505432 ± 6% -84.9% 377651 ± 82% numa-vmstat.node1.nr_file_pages
1120376 ± 7% -88.0% 134862 ± 26% numa-vmstat.node1.nr_inactive_anon
581125 ± 6% -99.3% 4276 ± 75% numa-vmstat.node1.nr_mapped
1884492 -100.0% 877.88 ± 13% numa-vmstat.node1.nr_shmem
889164 ± 6% -99.9% 608.62 ± 19% numa-vmstat.node1.nr_zone_active_anon
1120376 ± 7% -88.0% 134862 ± 26% numa-vmstat.node1.nr_zone_inactive_anon
3212683 -87.0% 417119 ± 8% numa-vmstat.node1.numa_hit
3113203 -88.3% 363053 ± 16% numa-vmstat.node1.numa_local
889503 ± 6% -99.9% 1318 ± 2% proc-vmstat.nr_active_anon
181558 -6.6% 169658 proc-vmstat.nr_anon_pages
6224451 +3.0% 6408360 proc-vmstat.nr_dirty_background_threshold
12464123 +3.0% 12832389 proc-vmstat.nr_dirty_threshold
2652787 -71.0% 768163 proc-vmstat.nr_file_pages
62620501 +2.9% 64462280 proc-vmstat.nr_free_pages
1178408 ± 4% -85.6% 170197 proc-vmstat.nr_inactive_anon
612759 ± 6% -98.6% 8708 proc-vmstat.nr_mapped
4212 ± 2% -26.5% 3098 proc-vmstat.nr_page_table_pages
1887299 -99.9% 2803 proc-vmstat.nr_shmem
32725 -16.5% 27312 proc-vmstat.nr_slab_reclaimable
889503 ± 6% -99.9% 1318 ± 2% proc-vmstat.nr_zone_active_anon
1178408 ± 4% -85.6% 170197 proc-vmstat.nr_zone_inactive_anon
290672 ± 45% -96.8% 9303 ± 41% proc-vmstat.numa_hint_faults
175901 ± 76% -97.5% 4457 ± 89% proc-vmstat.numa_hint_faults_local
3663440 -77.3% 830893 proc-vmstat.numa_hit
3530845 -79.5% 723050 proc-vmstat.numa_local
132548 -18.6% 107828 proc-vmstat.numa_other
467705 ± 25% -87.5% 58338 ± 63% proc-vmstat.numa_pte_updates
2487950 -100.0% 114.50 ± 16% proc-vmstat.pgactivate
3816007 -74.2% 983809 proc-vmstat.pgalloc_normal
1529029 ± 8% +34293.1% 5.259e+08 proc-vmstat.pgfault
1080189 -31.0% 745470 proc-vmstat.pgfree
50047 ± 16% -19.9% 40097 proc-vmstat.pgreuse
0.13 ± 12% -55.6% 0.06 ± 9% perf-stat.i.MPKI
3.129e+10 +64.8% 5.157e+10 perf-stat.i.branch-instructions
1.12 -0.5 0.63 perf-stat.i.branch-miss-rate%
3.419e+08 -14.1% 2.935e+08 perf-stat.i.branch-misses
19925383 ± 13% -32.6% 13438806 ± 9% perf-stat.i.cache-misses
1.336e+09 -24.7% 1.006e+09 perf-stat.i.cache-references
11135389 -31.8% 7599205 perf-stat.i.context-switches
1.55 -28.8% 1.10 perf-stat.i.cpi
2.551e+11 +3.4% 2.637e+11 perf-stat.i.cpu-cycles
86514 ± 4% +26.7% 109601 perf-stat.i.cpu-migrations
14145 ± 17% +43.8% 20343 ± 8% perf-stat.i.cycles-between-cache-misses
1.648e+11 +51.5% 2.497e+11 perf-stat.i.instructions
0.65 +45.3% 0.95 perf-stat.i.ipc
87.60 -1.3% 86.48 perf-stat.i.metric.K/sec
4794 ± 9% +35590.7% 1711366 perf-stat.i.minor-faults
4795 ± 9% +35590.6% 1711366 perf-stat.i.page-faults
0.12 ± 13% -55.5% 0.05 ± 9% perf-stat.overall.MPKI
1.09 -0.5 0.57 perf-stat.overall.branch-miss-rate%
1.55 -31.8% 1.06 perf-stat.overall.cpi
13042 ± 14% +51.7% 19779 ± 8% perf-stat.overall.cycles-between-cache-misses
0.65 +46.6% 0.95 perf-stat.overall.ipc
59142 +121.0% 130686 perf-stat.overall.path-length
3.118e+10 +64.8% 5.139e+10 perf-stat.ps.branch-instructions
3.407e+08 -14.1% 2.925e+08 perf-stat.ps.branch-misses
19880579 ± 13% -32.6% 13396005 ± 9% perf-stat.ps.cache-misses
1.332e+09 -24.7% 1.003e+09 perf-stat.ps.cache-references
11093234 -31.7% 7573495 perf-stat.ps.context-switches
2.542e+11 +3.4% 2.628e+11 perf-stat.ps.cpu-cycles
87621 ± 3% +24.9% 109407 perf-stat.ps.cpu-migrations
1.642e+11 +51.6% 2.489e+11 perf-stat.ps.instructions
4770 ± 9% +35666.9% 1706137 perf-stat.ps.minor-faults
4770 ± 9% +35666.8% 1706137 perf-stat.ps.page-faults
5.033e+13 +51.5% 7.624e+13 perf-stat.total.instructions
3403878 -9.6% 3076953 sched_debug.cfs_rq:/.avg_vruntime.avg
5906828 ± 7% +51.1% 8927261 ± 9% sched_debug.cfs_rq:/.avg_vruntime.max
3312785 -12.6% 2896386 sched_debug.cfs_rq:/.avg_vruntime.min
243929 ± 9% +164.4% 644860 ± 7% sched_debug.cfs_rq:/.avg_vruntime.stddev
1.31 ± 11% +31.9% 1.72 ± 17% sched_debug.cfs_rq:/.h_nr_running.max
267222 ± 12% -37.6% 166754 ± 11% sched_debug.cfs_rq:/.left_deadline.avg
3510445 -9.8% 3168156 ± 4% sched_debug.cfs_rq:/.left_deadline.max
902490 ± 6% -25.0% 677076 ± 4% sched_debug.cfs_rq:/.left_deadline.stddev
267208 ± 12% -37.6% 166745 ± 11% sched_debug.cfs_rq:/.left_vruntime.avg
3510259 -9.8% 3167981 ± 4% sched_debug.cfs_rq:/.left_vruntime.max
902441 ± 6% -25.0% 677037 ± 4% sched_debug.cfs_rq:/.left_vruntime.stddev
7978 ± 2% +21.3% 9675 ± 11% sched_debug.cfs_rq:/.load.avg
22814 ± 14% +289.5% 88865 ±147% sched_debug.cfs_rq:/.load.max
3403878 -9.6% 3076953 sched_debug.cfs_rq:/.min_vruntime.avg
5906828 ± 7% +51.1% 8927261 ± 9% sched_debug.cfs_rq:/.min_vruntime.max
3312785 -12.6% 2896386 sched_debug.cfs_rq:/.min_vruntime.min
243929 ± 9% +164.4% 644860 ± 7% sched_debug.cfs_rq:/.min_vruntime.stddev
267208 ± 12% -37.6% 166745 ± 11% sched_debug.cfs_rq:/.right_vruntime.avg
3510259 -9.8% 3167983 ± 4% sched_debug.cfs_rq:/.right_vruntime.max
902441 ± 6% -25.0% 677037 ± 4% sched_debug.cfs_rq:/.right_vruntime.stddev
124.18 ± 5% +19.4% 148.27 ± 2% sched_debug.cfs_rq:/.runnable_avg.stddev
410.44 -8.7% 374.59 sched_debug.cfs_rq:/.util_avg.avg
123.79 ± 5% +19.4% 147.85 ± 2% sched_debug.cfs_rq:/.util_avg.stddev
688.89 ± 8% +38.7% 955.28 ± 3% sched_debug.cfs_rq:/.util_est.max
451880 ± 32% +68.2% 759893 ± 15% sched_debug.cpu.avg_idle.max
6812 ± 2% +66.2% 11326 sched_debug.cpu.avg_idle.min
57452 ± 22% +59.0% 91375 ± 12% sched_debug.cpu.avg_idle.stddev
1511 ± 10% +166.4% 4026 ± 6% sched_debug.cpu.clock_task.stddev
1.31 ± 11% +34.0% 1.75 ± 9% sched_debug.cpu.nr_running.max
13041493 -24.7% 9825289 sched_debug.cpu.nr_switches.avg
13244004 -23.9% 10073573 sched_debug.cpu.nr_switches.max
8013647 ± 10% -84.4% 1247149 ±125% sched_debug.cpu.nr_switches.min
499070 ± 10% +82.7% 911672 ± 9% sched_debug.cpu.nr_switches.stddev
9.34 ± 3% -11.6% 8.26 ± 5% sched_debug.cpu.nr_uninterruptible.stddev
46.47 -46.5 0.00 perf-profile.calltrace.cycles-pp.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg
45.66 -45.7 0.00 perf-profile.calltrace.cycles-pp.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
45.53 ± 18% -45.5 0.00 perf-profile.calltrace.cycles-pp.main
43.65 -43.6 0.00 perf-profile.calltrace.cycles-pp.sctp_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg.do_syscall_64
42.48 -42.5 0.00 perf-profile.calltrace.cycles-pp.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg.___sys_sendmsg.__sys_sendmsg
37.52 -37.5 0.00 perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg.___sys_sendmsg
35.70 -35.7 0.00 perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc
29.47 -29.5 0.00 perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
29.39 -29.4 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
26.68 -26.7 0.00 perf-profile.calltrace.cycles-pp.common_startup_64
26.58 -26.6 0.00 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm
26.51 -26.5 0.00 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
26.48 -26.5 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
26.42 -26.4 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
25.57 -25.6 0.00 perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter
25.24 -25.2 0.00 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
24.20 -24.2 0.00 perf-profile.calltrace.cycles-pp.sendmsg.process_requests.spawn_child.accept_connection.accept_connections
24.11 -24.1 0.00 perf-profile.calltrace.cycles-pp.sendmsg.main
23.86 -23.9 0.00 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit
23.73 -23.7 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendmsg.process_requests.spawn_child.accept_connection
23.72 -23.7 0.00 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
23.71 -23.7 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg.process_requests.spawn_child
23.62 -23.6 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendmsg.main
23.60 -23.6 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg.main
23.59 -23.6 0.00 perf-profile.calltrace.cycles-pp.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
23.53 -23.5 0.00 perf-profile.calltrace.cycles-pp.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg.process_requests
23.43 -23.4 0.00 perf-profile.calltrace.cycles-pp.__sys_sendmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendmsg.main
22.88 -22.9 0.00 perf-profile.calltrace.cycles-pp.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
22.79 -22.8 0.00 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
22.46 -22.5 0.00 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.handle_softirqs.do_softirq.__local_bh_enable_ip
22.35 -22.3 0.00 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.handle_softirqs.do_softirq
22.02 -22.0 0.00 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.handle_softirqs
21.80 -21.8 0.00 perf-profile.calltrace.cycles-pp.recvmsg
21.50 -21.5 0.00 perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg
21.31 -21.3 0.00 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action
21.26 -21.3 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
21.24 -21.2 0.00 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll
21.08 -21.1 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
20.96 -21.0 0.00 perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog
20.92 ± 41% -20.9 0.00 perf-profile.calltrace.cycles-pp.accept_connection.accept_connections.main
20.92 ± 41% -20.9 0.00 perf-profile.calltrace.cycles-pp.accept_connections.main
20.92 ± 41% -20.9 0.00 perf-profile.calltrace.cycles-pp.process_requests.spawn_child.accept_connection.accept_connections.main
20.92 ± 41% -20.9 0.00 perf-profile.calltrace.cycles-pp.spawn_child.accept_connection.accept_connections.main
20.40 -20.4 0.00 perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
20.34 -20.3 0.00 perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg
20.05 -20.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvmsg
20.02 -20.0 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
18.81 -18.8 0.00 perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
18.35 -18.4 0.00 perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg
17.76 -17.8 0.00 perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core
17.53 -17.5 0.00 perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe
17.04 -17.0 0.00 perf-profile.calltrace.cycles-pp.sock_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64
16.61 -16.6 0.00 perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg
16.40 -16.4 0.00 perf-profile.calltrace.cycles-pp.sctp_recvmsg.inet_recvmsg.sock_recvmsg.____sys_recvmsg.___sys_recvmsg
14.32 ± 2% -14.3 0.00 perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish
13.68 ± 2% -13.7 0.00 perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu
9.63 -9.6 0.00 perf-profile.calltrace.cycles-pp.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.sock_recvmsg.____sys_recvmsg
9.28 ± 2% -9.3 0.00 perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
9.08 -9.1 0.00 perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
7.30 -7.3 0.00 perf-profile.calltrace.cycles-pp.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg.sock_recvmsg
7.05 -7.1 0.00 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg.inet_recvmsg
6.92 -6.9 0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.sctp_skb_recv_datagram.sctp_recvmsg
6.02 -6.0 0.00 perf-profile.calltrace.cycles-pp.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND
5.78 ± 3% -5.8 0.00 perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
5.48 -5.5 0.00 perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state
5.33 ± 4% -5.3 0.00 perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm
67.76 -67.8 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
67.49 -67.5 0.00 perf-profile.children.cycles-pp.do_syscall_64
48.53 -48.5 0.00 perf-profile.children.cycles-pp.sendmsg
47.03 -47.0 0.00 perf-profile.children.cycles-pp.__sys_sendmsg
46.55 -46.6 0.00 perf-profile.children.cycles-pp.___sys_sendmsg
45.75 -45.7 0.00 perf-profile.children.cycles-pp.____sys_sendmsg
45.59 ± 18% -45.6 0.00 perf-profile.children.cycles-pp.main
44.18 -44.2 0.00 perf-profile.children.cycles-pp.sctp_sendmsg
42.58 -42.6 0.00 perf-profile.children.cycles-pp.sctp_sendmsg_to_asoc
37.58 -37.6 0.00 perf-profile.children.cycles-pp.sctp_primitive_SEND
37.38 -37.4 0.00 perf-profile.children.cycles-pp.sctp_do_sm
36.84 -36.8 0.00 perf-profile.children.cycles-pp.sctp_cmd_interpreter
35.94 -35.9 0.00 perf-profile.children.cycles-pp.sctp_outq_flush
29.54 -29.5 0.00 perf-profile.children.cycles-pp.sctp_packet_transmit
26.68 -26.7 0.00 perf-profile.children.cycles-pp.common_startup_64
26.68 -26.7 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
26.63 -26.6 0.00 perf-profile.children.cycles-pp.__ip_queue_xmit
26.63 -26.6 0.00 perf-profile.children.cycles-pp.do_idle
26.51 -26.5 0.00 perf-profile.children.cycles-pp.start_secondary
25.62 -25.6 0.00 perf-profile.children.cycles-pp.ip_finish_output2
25.30 -25.3 0.00 perf-profile.children.cycles-pp.__dev_queue_xmit
24.78 -24.8 0.00 perf-profile.children.cycles-pp.accept_connection
24.78 -24.8 0.00 perf-profile.children.cycles-pp.accept_connections
24.78 -24.8 0.00 perf-profile.children.cycles-pp.process_requests
24.78 -24.8 0.00 perf-profile.children.cycles-pp.spawn_child
24.02 -24.0 0.00 perf-profile.children.cycles-pp.__local_bh_enable_ip
23.80 -23.8 0.00 perf-profile.children.cycles-pp.handle_softirqs
23.78 -23.8 0.00 perf-profile.children.cycles-pp.do_softirq
22.97 -23.0 0.00 perf-profile.children.cycles-pp.cpuidle_idle_call
22.92 -22.9 0.00 perf-profile.children.cycles-pp.net_rx_action
22.50 -22.5 0.00 perf-profile.children.cycles-pp.__napi_poll
22.38 -22.4 0.00 perf-profile.children.cycles-pp.process_backlog
22.06 -22.1 0.00 perf-profile.children.cycles-pp.__netif_receive_skb_one_core
21.93 -21.9 0.00 perf-profile.children.cycles-pp.recvmsg
21.41 -21.4 0.00 perf-profile.children.cycles-pp.cpuidle_enter
21.34 -21.3 0.00 perf-profile.children.cycles-pp.ip_local_deliver_finish
21.28 -21.3 0.00 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu
21.26 -21.3 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
21.03 -21.0 0.00 perf-profile.children.cycles-pp.sctp_rcv
20.54 -20.5 0.00 perf-profile.children.cycles-pp.acpi_idle_enter
20.42 -20.4 0.00 perf-profile.children.cycles-pp.acpi_safe_halt
18.83 -18.8 0.00 perf-profile.children.cycles-pp.__sys_recvmsg
18.36 -18.4 0.00 perf-profile.children.cycles-pp.___sys_recvmsg
17.81 -17.8 0.00 perf-profile.children.cycles-pp.sctp_assoc_bh_rcv
17.75 -17.8 0.00 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
17.55 -17.6 0.00 perf-profile.children.cycles-pp.____sys_recvmsg
17.06 -17.1 0.00 perf-profile.children.cycles-pp.sock_recvmsg
16.84 -16.8 0.00 perf-profile.children.cycles-pp.sctp_recvmsg
16.62 -16.6 0.00 perf-profile.children.cycles-pp.inet_recvmsg
9.68 -9.7 0.00 perf-profile.children.cycles-pp.sctp_skb_recv_datagram
9.54 -9.5 0.00 perf-profile.children.cycles-pp.__schedule
9.36 ± 2% -9.4 0.00 perf-profile.children.cycles-pp.sctp_outq_sack
8.24 ± 2% -8.2 0.00 perf-profile.children.cycles-pp.sctp_chunk_put
7.62 -7.6 0.00 perf-profile.children.cycles-pp.schedule_timeout
7.06 -7.1 0.00 perf-profile.children.cycles-pp.schedule
6.38 ± 3% -6.4 0.00 perf-profile.children.cycles-pp.consume_skb
6.14 ± 3% -6.1 0.00 perf-profile.children.cycles-pp.skb_release_head_state
6.14 -6.1 0.00 perf-profile.children.cycles-pp.sctp_outq_flush_data
5.54 -5.5 0.00 perf-profile.children.cycles-pp.sysvec_call_function_single
13.56 -13.6 0.00 perf-profile.self.cycles-pp.acpi_safe_halt

***************************************************************************************************
lkp-skl-fpga01: 104 threads 2 sockets (Skylake) with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-12/performance/x86_64-rhel-8.3/process/100%/debian-12-x86_64-20240206.cgz/lkp-skl-fpga01/context_switch1/will-it-scale

commit:
cb1898f58e ("perf annotate-data: Support --skip-empty option")
a9ac22fb7a ("perf: Remove setting of page->index and ->mapping")

cb1898f58e0f175d a9ac22fb7ac3486a5bd89c25a2f
---------------- ---------------------------
%stddev %change %stddev
\ | \
64399 ± 2% +268.0% 236970 ± 13% cpuidle..usage
1317337 ± 5% -75.7% 320761 ± 39% numa-numastat.node1.local_node
1363498 ± 5% -72.5% 374923 ± 32% numa-numastat.node1.numa_hit
27.91 -60.0% 11.18 ± 3% vmstat.cpu.us
18301349 -60.6% 7203175 ± 4% vmstat.system.cs
129807 +465.4% 733957 ± 5% vmstat.system.in
18547591 -61.9% 7060612 ± 4% will-it-scale.104.processes
178341 -61.9% 67890 ± 4% will-it-scale.per_process_ops
18547591 -61.9% 7060612 ± 4% will-it-scale.workload
0.41 ± 4% +0.9 1.33 mpstat.cpu.all.irq%
0.01 ± 3% -0.0 0.01 ± 2% mpstat.cpu.all.soft%
70.70 +16.1 86.78 mpstat.cpu.all.sys%
28.09 -17.0 11.11 ± 4% mpstat.cpu.all.usr%
197.00 ± 65% -100.0% 0.00 perf-c2c.DRAM.local
345.00 ± 13% -100.0% 0.00 perf-c2c.DRAM.remote
6145 ± 30% -100.0% 0.00 perf-c2c.HITM.local
207.50 ± 9% -100.0% 0.00 perf-c2c.HITM.remote
6353 ± 29% -100.0% 0.00 perf-c2c.HITM.total
2719746 ± 3% -99.9% 3450 meminfo.Active
2719698 ± 3% -99.9% 3402 meminfo.Active(anon)
6617563 ± 2% -53.4% 3084370 meminfo.Cached
6890243 ± 2% -51.3% 3357208 meminfo.Committed_AS
1534356 ± 7% -54.6% 696751 meminfo.Inactive
1534138 ± 7% -54.6% 696537 meminfo.Inactive(anon)
107936 -12.2% 94813 meminfo.KReclaimable
528224 ± 12% -93.0% 36788 meminfo.Mapped
8481255 ± 2% -36.9% 5355841 meminfo.Memused
107936 -12.2% 94813 meminfo.SReclaimable
3544842 ± 4% -99.7% 11703 ± 3% meminfo.Shmem
9043642 ± 2% -38.6% 5554552 meminfo.max_used_kB
322297 ± 32% -50.7% 159027 ± 76% numa-meminfo.node0.AnonPages
422986 ± 24% -59.1% 173110 ± 74% numa-meminfo.node0.AnonPages.max
323931 ± 32% -50.6% 160140 ± 76% numa-meminfo.node0.Inactive
323754 ± 32% -50.6% 159998 ± 76% numa-meminfo.node0.Inactive(anon)
75677 ± 11% -66.8% 25143 ± 48% numa-meminfo.node0.Mapped
2716604 ± 3% -99.9% 2155 ± 21% numa-meminfo.node1.Active
2716596 ± 3% -99.9% 2139 ± 21% numa-meminfo.node1.Active(anon)
4505110 ± 29% -71.2% 1298660 ± 98% numa-meminfo.node1.FilePages
1211376 ± 16% -55.7% 536704 ± 22% numa-meminfo.node1.Inactive
1211335 ± 16% -55.7% 536631 ± 22% numa-meminfo.node1.Inactive(anon)
451272 ± 15% -97.4% 11673 ±103% numa-meminfo.node1.Mapped
5431186 ± 24% -52.3% 2592950 ± 46% numa-meminfo.node1.MemUsed
3537633 ± 4% -99.8% 6102 ± 19% numa-meminfo.node1.Shmem
80524 ± 32% -50.6% 39762 ± 76% numa-vmstat.node0.nr_anon_pages
80887 ± 32% -50.5% 40003 ± 76% numa-vmstat.node0.nr_inactive_anon
19126 ± 11% -66.2% 6470 ± 47% numa-vmstat.node0.nr_mapped
80887 ± 32% -50.5% 40003 ± 76% numa-vmstat.node0.nr_zone_inactive_anon
679371 ± 3% -99.9% 535.24 ± 21% numa-vmstat.node1.nr_active_anon
1126311 ± 29% -71.2% 324665 ± 98% numa-vmstat.node1.nr_file_pages
302619 ± 16% -55.7% 134149 ± 22% numa-vmstat.node1.nr_inactive_anon
113050 ± 15% -97.4% 2961 ±103% numa-vmstat.node1.nr_mapped
884442 ± 4% -99.8% 1526 ± 19% numa-vmstat.node1.nr_shmem
679371 ± 3% -99.9% 535.23 ± 21% numa-vmstat.node1.nr_zone_active_anon
302620 ± 16% -55.7% 134149 ± 22% numa-vmstat.node1.nr_zone_inactive_anon
1362527 ± 5% -72.5% 374747 ± 32% numa-vmstat.node1.numa_hit
1316366 ± 5% -75.6% 320586 ± 39% numa-vmstat.node1.numa_local
120076 ± 10% +44.5% 173481 ± 10% sched_debug.cfs_rq:/.avg_vruntime.stddev
1.29 -14.3% 1.11 ± 2% sched_debug.cfs_rq:/.h_nr_running.avg
1093 ±141% +350.1% 4923 ± 22% sched_debug.cfs_rq:/.load.min
120076 ± 10% +44.5% 173481 ± 10% sched_debug.cfs_rq:/.min_vruntime.stddev
1815 ± 4% +13.8% 2065 sched_debug.cfs_rq:/.runnable_avg.max
1209 -12.4% 1059 ± 2% sched_debug.cfs_rq:/.runnable_avg.min
97.73 ± 12% +66.6% 162.81 ± 5% sched_debug.cfs_rq:/.runnable_avg.stddev
804.36 ± 3% -10.7% 718.36 ± 3% sched_debug.cfs_rq:/.util_avg.min
618.13 +16.3% 719.15 ± 2% sched_debug.cfs_rq:/.util_est.avg
1100 ± 5% +36.4% 1501 ± 7% sched_debug.cfs_rq:/.util_est.max
793984 ± 2% -34.7% 518681 ± 3% sched_debug.cpu.avg_idle.avg
1164653 ± 8% -19.1% 942294 ± 8% sched_debug.cpu.avg_idle.max
187692 ± 10% -18.2% 153546 ± 7% sched_debug.cpu.avg_idle.stddev
2817 ± 7% -26.8% 2061 ± 8% sched_debug.cpu.curr->pid.min
26222425 -55.7% 11622454 ± 3% sched_debug.cpu.nr_switches.avg
27158282 -55.2% 12180443 ± 4% sched_debug.cpu.nr_switches.max
24464466 ± 2% -57.5% 10402856 ± 4% sched_debug.cpu.nr_switches.min
605107 ± 12% -44.4% 336330 ± 6% sched_debug.cpu.nr_switches.stddev
679317 ± 3% -99.9% 850.20 proc-vmstat.nr_active_anon
178194 -2.9% 172994 proc-vmstat.nr_anon_pages
4677129 +1.7% 4755122 proc-vmstat.nr_dirty_background_threshold
9365694 +1.7% 9521870 proc-vmstat.nr_dirty_threshold
1654099 ± 2% -53.4% 771092 proc-vmstat.nr_file_pages
47057782 +1.7% 47838859 proc-vmstat.nr_free_pages
383856 ± 7% -54.6% 174141 proc-vmstat.nr_inactive_anon
132404 ± 12% -92.9% 9427 proc-vmstat.nr_mapped
4649 -1.8% 4567 proc-vmstat.nr_page_table_pages
885917 ± 4% -99.7% 2924 ± 3% proc-vmstat.nr_shmem
26984 -12.2% 23703 proc-vmstat.nr_slab_reclaimable
679317 ± 3% -99.9% 850.20 proc-vmstat.nr_zone_active_anon
383856 ± 7% -54.6% 174141 proc-vmstat.nr_zone_inactive_anon
201323 ± 11% -96.8% 6359 ± 91% proc-vmstat.numa_hint_faults
98999 ± 21% -99.5% 520.33 ± 5% proc-vmstat.numa_hint_faults_local
1907797 ± 2% -58.8% 785657 proc-vmstat.numa_hit
1805585 ± 2% -61.4% 697853 proc-vmstat.numa_local
107648 -18.6% 87641 proc-vmstat.numa_other
399248 ± 16% -88.1% 47657 ± 70% proc-vmstat.numa_pte_updates
958792 ± 5% -100.0% 204.50 ± 8% proc-vmstat.pgactivate
2040912 ± 2% -54.9% 920241 proc-vmstat.pgalloc_normal
1329369 ± 2% +15694.5% 2.1e+08 proc-vmstat.pgfault
1004379 -27.6% 727403 proc-vmstat.pgfree
49086 ± 20% -18.3% 40086 ± 4% proc-vmstat.pgreuse
0.03 -33.3% 0.02 ± 17% perf-stat.i.MPKI
2.522e+10 +69.5% 4.275e+10 ± 3% perf-stat.i.branch-instructions
1.55 -0.6 0.94 ± 18% perf-stat.i.branch-miss-rate%
18477740 -60.7% 7266076 ± 4% perf-stat.i.context-switches
2.20 -37.2% 1.38 ± 3% perf-stat.i.cpi
2.835e+11 -2.3% 2.771e+11 perf-stat.i.cpu-cycles
189.99 +3.7% 197.08 perf-stat.i.cpu-migrations
154844 ± 2% -35.7% 99550 ± 14% perf-stat.i.cycles-between-cache-misses
1.292e+11 +59.0% 2.054e+11 ± 3% perf-stat.i.instructions
0.46 +62.4% 0.74 ± 3% perf-stat.i.ipc
177.68 -53.2% 83.14 ± 3% perf-stat.i.metric.K/sec
4186 ± 2% +16391.7% 690372 perf-stat.i.minor-faults
4186 ± 2% +16391.2% 690372 perf-stat.i.page-faults
0.03 ± 2% -42.7% 0.01 ± 19% perf-stat.overall.MPKI
1.54 -0.6 0.91 ± 19% perf-stat.overall.branch-miss-rate%
2.19 -38.5% 1.35 ± 3% perf-stat.overall.cpi
0.46 +62.7% 0.74 ± 3% perf-stat.overall.ipc
2105030 +318.8% 8815433 perf-stat.overall.path-length
2.513e+10 +69.6% 4.262e+10 ± 3% perf-stat.ps.branch-instructions
18416035 -60.7% 7236360 ± 4% perf-stat.ps.context-switches
2.825e+11 -2.3% 2.761e+11 perf-stat.ps.cpu-cycles
189.25 +3.8% 196.41 perf-stat.ps.cpu-migrations
1.288e+11 +59.0% 2.048e+11 ± 3% perf-stat.ps.instructions
4169 ± 2% +16406.3% 688277 perf-stat.ps.minor-faults
4169 ± 2% +16405.8% 688277 perf-stat.ps.page-faults
3.904e+13 +59.3% 6.221e+13 ± 3% perf-stat.total.instructions
53.50 -53.5 0.00 perf-profile.calltrace.cycles-pp.read
45.69 -45.7 0.00 perf-profile.calltrace.cycles-pp.write
35.13 -35.1 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
31.89 -31.9 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
29.98 -30.0 0.00 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
26.86 -26.9 0.00 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
26.74 -26.7 0.00 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
25.88 -25.9 0.00 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
23.71 -23.7 0.00 perf-profile.calltrace.cycles-pp.pipe_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
21.52 -21.5 0.00 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
20.61 -20.6 0.00 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
18.62 ± 2% -18.6 0.00 perf-profile.calltrace.cycles-pp.pipe_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
16.92 -16.9 0.00 perf-profile.calltrace.cycles-pp.schedule.pipe_read.vfs_read.ksys_read.do_syscall_64
16.55 -16.6 0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.pipe_read.vfs_read.ksys_read
14.74 ± 2% -14.7 0.00 perf-profile.calltrace.cycles-pp.__wake_up_sync_key.pipe_write.vfs_write.ksys_write.do_syscall_64
14.31 ± 2% -14.3 0.00 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_sync_key.pipe_write.vfs_write.ksys_write
14.10 ± 2% -14.1 0.00 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_write.vfs_write
13.74 ± 2% -13.7 0.00 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key.pipe_write
9.02 -9.0 0.00 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
7.16 -7.2 0.00 perf-profile.calltrace.cycles-pp.enqueue_task_fair.ttwu_do_activate.try_to_wake_up.autoremove_wake_function.__wake_up_common
6.93 -6.9 0.00 perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.pipe_read.vfs_read
6.62 -6.6 0.00 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read
6.14 ± 2% -6.1 0.00 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
65.61 -65.6 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
58.74 -58.7 0.00 perf-profile.children.cycles-pp.do_syscall_64
53.92 -53.9 0.00 perf-profile.children.cycles-pp.read
46.12 -46.1 0.00 perf-profile.children.cycles-pp.write
26.87 -26.9 0.00 perf-profile.children.cycles-pp.ksys_read
25.90 -25.9 0.00 perf-profile.children.cycles-pp.vfs_read
23.87 -23.9 0.00 perf-profile.children.cycles-pp.pipe_read
21.55 -21.5 0.00 perf-profile.children.cycles-pp.ksys_write
20.65 -20.7 0.00 perf-profile.children.cycles-pp.vfs_write
19.74 -19.7 0.00 perf-profile.children.cycles-pp.schedule
19.36 -19.4 0.00 perf-profile.children.cycles-pp.__schedule
18.67 ± 2% -18.7 0.00 perf-profile.children.cycles-pp.pipe_write
14.75 ± 2% -14.8 0.00 perf-profile.children.cycles-pp.__wake_up_sync_key
14.32 ± 2% -14.3 0.00 perf-profile.children.cycles-pp.__wake_up_common
14.10 ± 2% -14.1 0.00 perf-profile.children.cycles-pp.autoremove_wake_function
13.79 ± 2% -13.8 0.00 perf-profile.children.cycles-pp.try_to_wake_up
12.86 -12.9 0.00 perf-profile.children.cycles-pp.syscall_return_via_sysret
10.00 -10.0 0.00 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
9.04 -9.0 0.00 perf-profile.children.cycles-pp.ttwu_do_activate
8.44 -8.4 0.00 perf-profile.children.cycles-pp.syscall_exit_to_user_mode
7.17 -7.2 0.00 perf-profile.children.cycles-pp.enqueue_task_fair
6.95 -6.9 0.00 perf-profile.children.cycles-pp.dequeue_task_fair
6.09 -6.1 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64
12.54 -12.5 0.00 perf-profile.self.cycles-pp.syscall_return_via_sysret
9.91 -9.9 0.00 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
6.97 -7.0 0.00 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
5.47 -5.5 0.00 perf-profile.self.cycles-pp.entry_SYSCALL_64

***************************************************************************************************
lkp-icl-2sp9: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
=========================================================================================
compiler/cpufreq_governor/disk/filesize/fs2/fs/iterations/kconfig/nr_files_per_directory/nr_threads/rootfs/sync_method/tbox_group/test_size/testcase:
gcc-12/performance/1BRD_32G/4K/nfsv4/ext4/1x/x86_64-rhel-8.3/1fpd/1t/debian-12-x86_64-20240206.cgz/fsyncBeforeClose/lkp-icl-2sp9/4G/fsmark

commit:
cb1898f58e ("perf annotate-data: Support --skip-empty option")
a9ac22fb7a ("perf: Remove setting of page->index and ->mapping")

cb1898f58e0f175d a9ac22fb7ac3486a5bd89c25a2f
---------------- ---------------------------
%stddev %change %stddev
\ | \
1.519e+10 +34.7% 2.046e+10 ± 3% cpuidle..time
97.75 -4.6% 93.24 iostat.cpu.idle
1.84 +220.2% 5.88 iostat.cpu.system
301.00 +33.4% 401.55 ± 3% uptime.boot
18660 +28.1% 23909 ± 3% uptime.idle
929516 ±111% +336.5% 4057213 ± 47% numa-numastat.node0.local_node
974371 ±104% +320.7% 4098811 ± 47% numa-numastat.node0.numa_hit
17513401 ± 5% -17.6% 14426818 ± 13% numa-numastat.node1.local_node
17535676 ± 5% -17.6% 14440697 ± 13% numa-numastat.node1.numa_hit
403587 -29.1% 286045 ± 3% vmstat.io.bo
2.25 +128.2% 5.14 vmstat.procs.r
324930 -27.4% 236047 ± 2% vmstat.system.cs
176422 +45.4% 256569 vmstat.system.in
227.83 ± 11% -100.0% 0.00 perf-c2c.DRAM.local
427.33 ± 83% -100.0% 0.00 perf-c2c.DRAM.remote
5358 ± 4% -100.0% 0.00 perf-c2c.HITM.local
236.50 ± 98% -100.0% 0.00 perf-c2c.HITM.remote
5595 ± 2% -100.0% 0.00 perf-c2c.HITM.total
76998903 +33.3% 1.027e+08 ± 3% fsmark.app_overhead
4243 -29.8% 2978 ± 3% fsmark.files_per_sec
239.76 +42.0% 340.44 ± 3% fsmark.time.elapsed_time
239.76 +42.0% 340.44 ± 3% fsmark.time.elapsed_time.max
31.30 +51.6% 47.44 ± 3% fsmark.time.system_time
5769288 -1.7% 5670289 fsmark.time.voluntary_context_switches
0.15 +0.0 0.20 ± 5% mpstat.cpu.all.irq%
0.19 +0.1 0.26 mpstat.cpu.all.soft%
1.50 +3.9 5.43 mpstat.cpu.all.sys%
0.09 ± 2% +0.5 0.58 mpstat.cpu.all.usr%
1.17 ± 31% +22328.6% 261.67 ± 2% mpstat.max_utilization.seconds
5.18 ± 3% +66.7% 8.63 mpstat.max_utilization_pct
379352 ± 59% +87.1% 709913 ± 32% numa-meminfo.node0.AnonPages.max
3104 ± 9% +28.1% 3975 ± 12% numa-meminfo.node0.PageTables
111563 ± 19% +62.9% 181723 ± 31% numa-meminfo.node0.SUnreclaim
517329 ± 2% -99.6% 2323 ± 44% numa-meminfo.node1.Active(anon)
277501 ± 6% -41.6% 162034 ± 18% numa-meminfo.node1.Dirty
526882 ± 2% -99.5% 2824 ± 42% numa-meminfo.node1.Shmem
643.83 ± 6% +39.5% 898.01 ± 20% numa-meminfo.node1.Writeback
3076105 -13.2% 2669099 meminfo.Active
524607 -98.8% 6139 ± 2% meminfo.Active(anon)
1246297 +11.7% 1391614 meminfo.AnonHugePages
3303829 -14.1% 2836865 meminfo.Committed_AS
284869 -28.9% 202566 ± 3% meminfo.Dirty
7768 +10.6% 8592 meminfo.PageTables
543894 -96.9% 16697 meminfo.Shmem
645.23 ± 5% +80.9% 1167 ± 18% meminfo.Writeback
87885 ±195% +615.6% 628897 ± 53% numa-vmstat.node0.nr_dirtied
1993 ±222% +420.2% 10370 ± 55% numa-vmstat.node0.nr_dirty
781.52 ± 8% +27.2% 994.01 ± 12% numa-vmstat.node0.nr_page_table_pages
27893 ± 19% +62.9% 45435 ± 31% numa-vmstat.node0.nr_slab_unreclaimable
87721 ±195% +600.0% 614086 ± 53% numa-vmstat.node0.nr_written
1998 ±222% +422.0% 10431 ± 55% numa-vmstat.node0.nr_zone_write_pending
973993 ±104% +320.8% 4098274 ± 47% numa-vmstat.node0.numa_hit
929139 ±111% +336.6% 4056676 ± 47% numa-vmstat.node0.numa_local
129283 ± 2% -99.6% 581.30 ± 44% numa-vmstat.node1.nr_active_anon
69155 ± 6% -41.7% 40315 ± 18% numa-vmstat.node1.nr_dirty
131705 ± 2% -99.5% 705.99 ± 42% numa-vmstat.node1.nr_shmem
158.63 ± 4% +35.6% 215.16 ± 15% numa-vmstat.node1.nr_writeback
129283 ± 2% -99.6% 581.30 ± 44% numa-vmstat.node1.nr_zone_active_anon
69315 ± 6% -41.5% 40531 ± 18% numa-vmstat.node1.nr_zone_write_pending
17534283 ± 5% -17.6% 14439527 ± 13% numa-vmstat.node1.numa_hit
17512007 ± 5% -17.6% 14425648 ± 13% numa-vmstat.node1.numa_local
1.74 ± 9% -65.6% 0.60 ± 15% perf-stat.i.MPKI
9.615e+08 +285.6% 3.707e+09 perf-stat.i.branch-instructions
0.69 -0.5 0.19 perf-stat.i.branch-miss-rate%
7798406 -11.5% 6900769 perf-stat.i.branch-misses
329459 -27.7% 238084 ± 2% perf-stat.i.context-switches
1.35 -30.1% 0.94 perf-stat.i.cpi
6.387e+09 +159.7% 1.659e+10 perf-stat.i.cpu-cycles
853.55 ± 3% +132.1% 1980 ± 17% perf-stat.i.cycles-between-cache-misses
4.847e+09 +266.3% 1.776e+10 perf-stat.i.instructions
0.75 +42.0% 1.07 perf-stat.i.ipc
5.14 +2508.3% 134.15 perf-stat.i.metric.K/sec
2541 +1.6e+05% 4173925 perf-stat.i.minor-faults
2541 +1.6e+05% 4173925 perf-stat.i.page-faults
673.07 ± 45% +148.1% 1669 ± 16% perf-stat.overall.cycles-between-cache-misses
0.63 ± 44% +69.6% 1.07 perf-stat.overall.ipc
7.964e+08 ± 44% +364.2% 3.697e+09 perf-stat.ps.branch-instructions
5.299e+09 ± 44% +212.2% 1.654e+10 perf-stat.ps.cpu-cycles
4.015e+09 ± 44% +341.1% 1.771e+10 perf-stat.ps.instructions
2107 ± 44% +2e+05% 4162878 perf-stat.ps.minor-faults
2107 ± 44% +2e+05% 4162878 perf-stat.ps.page-faults
9.689e+11 ± 44% +524.4% 6.05e+12 ± 3% perf-stat.total.instructions
131138 -98.8% 1536 ± 2% proc-vmstat.nr_active_anon
637951 +4.3% 665408 proc-vmstat.nr_active_file
412901 +5.0% 433490 proc-vmstat.nr_anon_pages
607.19 +11.8% 679.13 proc-vmstat.nr_anon_transparent_hugepages
71193 -28.7% 50743 ± 3% proc-vmstat.nr_dirty
415380 +4.4% 433762 proc-vmstat.nr_inactive_anon
1003928 +5.2% 1056379 proc-vmstat.nr_inactive_file
9194 -5.4% 8699 proc-vmstat.nr_mapped
1941 +10.8% 2151 proc-vmstat.nr_page_table_pages
135981 -96.9% 4176 proc-vmstat.nr_shmem
761716 +5.0% 799556 proc-vmstat.nr_slab_reclaimable
135618 +5.3% 142780 proc-vmstat.nr_slab_unreclaimable
165.40 ± 2% +72.0% 284.42 ± 19% proc-vmstat.nr_writeback
3050431 +2.5% 3128215 proc-vmstat.nr_written
131138 -98.8% 1536 ± 2% proc-vmstat.nr_zone_active_anon
637951 +4.3% 665408 proc-vmstat.nr_zone_active_file
415380 +4.4% 433762 proc-vmstat.nr_zone_inactive_anon
1003928 +5.2% 1056379 proc-vmstat.nr_zone_inactive_file
71359 -28.5% 51028 ± 3% proc-vmstat.nr_zone_write_pending
1065 ± 10% -19.5% 858.50 ± 5% proc-vmstat.numa_hint_faults
945.50 ± 10% -50.9% 464.50 ± 6% proc-vmstat.numa_hint_faults_local
668.33 ± 25% +47.3% 984.67 ± 2% proc-vmstat.numa_huge_pte_updates
66342 -18.5% 54068 proc-vmstat.numa_other
367076 ± 23% +38.1% 507046 ± 2% proc-vmstat.numa_pte_updates
150353 ± 2% -98.4% 2420 proc-vmstat.pgactivate
671082 +2.1e+05% 1.427e+09 ± 4% proc-vmstat.pgfault
29181 +34.0% 39102 ± 3% proc-vmstat.pgreuse
7675 ± 9% +401.5% 38488 ± 8% sched_debug.cfs_rq:/.avg_vruntime.avg
40528 ± 16% +958.7% 429065 ± 19% sched_debug.cfs_rq:/.avg_vruntime.max
705.29 ± 28% +155.6% 1802 ± 21% sched_debug.cfs_rq:/.avg_vruntime.min
7669 ± 14% +883.8% 75453 ± 12% sched_debug.cfs_rq:/.avg_vruntime.stddev
0.30 ± 5% +13.7% 0.34 ± 5% sched_debug.cfs_rq:/.h_nr_running.stddev
951.02 ± 14% -38.0% 590.09 ± 34% sched_debug.cfs_rq:/.load_avg.max
172.57 ± 9% -28.4% 123.61 ± 18% sched_debug.cfs_rq:/.load_avg.stddev
7675 ± 9% +401.5% 38488 ± 8% sched_debug.cfs_rq:/.min_vruntime.avg
40528 ± 16% +958.7% 429065 ± 19% sched_debug.cfs_rq:/.min_vruntime.max
705.29 ± 28% +155.6% 1802 ± 21% sched_debug.cfs_rq:/.min_vruntime.min
7669 ± 14% +883.8% 75453 ± 12% sched_debug.cfs_rq:/.min_vruntime.stddev
0.30 ± 5% +13.9% 0.34 ± 5% sched_debug.cfs_rq:/.nr_running.stddev
716.12 ± 10% +51.6% 1085 ± 3% sched_debug.cfs_rq:/.runnable_avg.max
147.84 ± 6% +64.0% 242.44 sched_debug.cfs_rq:/.runnable_avg.stddev
712.46 ± 10% +52.1% 1083 ± 3% sched_debug.cfs_rq:/.util_avg.max
147.41 ± 6% +64.2% 242.12 sched_debug.cfs_rq:/.util_avg.stddev
16.26 ± 17% +216.4% 51.45 ± 7% sched_debug.cfs_rq:/.util_est.avg
383.67 ± 19% +162.8% 1008 ± 3% sched_debug.cfs_rq:/.util_est.max
63.16 ± 17% +222.5% 203.68 ± 2% sched_debug.cfs_rq:/.util_est.stddev
165310 ± 8% +24.3% 205470 ± 5% sched_debug.cpu.clock.avg
165314 ± 8% +24.3% 205474 ± 5% sched_debug.cpu.clock.max
165279 ± 8% +24.3% 205454 ± 5% sched_debug.cpu.clock.min
5.18 ± 28% -36.4% 3.30 ± 18% sched_debug.cpu.clock.stddev
164698 ± 8% +24.2% 204576 ± 5% sched_debug.cpu.clock_task.avg
165287 ± 8% +24.3% 205370 ± 5% sched_debug.cpu.clock_task.max
155952 ± 9% +25.1% 195174 ± 5% sched_debug.cpu.clock_task.min
1185 ± 2% +8.9% 1290 ± 5% sched_debug.cpu.clock_task.stddev
274.22 ± 13% +32.4% 362.97 ± 7% sched_debug.cpu.curr->pid.avg
5453 ± 6% +18.4% 6455 ± 4% sched_debug.cpu.curr->pid.max
940.43 ± 9% +18.8% 1117 ± 2% sched_debug.cpu.curr->pid.stddev
0.11 ± 10% +25.0% 0.13 ± 9% sched_debug.cpu.nr_running.avg
0.30 ± 6% +13.5% 0.34 ± 4% sched_debug.cpu.nr_running.stddev
24.81 ± 14% +28.4% 31.85 ± 8% sched_debug.cpu.nr_uninterruptible.stddev
165310 ± 8% +24.3% 205469 ± 5% sched_debug.cpu_clk
164147 ± 8% +24.5% 204306 ± 5% sched_debug.ktime
165888 ± 8% +24.2% 206051 ± 5% sched_debug.sched_clk
50.27 -50.3 0.00 perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
50.27 -50.3 0.00 perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
50.27 -50.3 0.00 perf-profile.calltrace.cycles-pp.ret_from_fork_asm
43.03 ± 6% -43.0 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
37.89 ± 2% -37.9 0.00 perf-profile.calltrace.cycles-pp.common_startup_64
36.23 ± 2% -36.2 0.00 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
36.20 ± 2% -36.2 0.00 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
36.14 ± 2% -36.1 0.00 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
33.61 ± 2% -33.6 0.00 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
33.08 ± 2% -33.1 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
31.89 ± 2% -31.9 0.00 perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
31.81 ± 2% -31.8 0.00 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
27.02 -27.0 0.00 perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork.ret_from_fork_asm
26.83 -26.8 0.00 perf-profile.calltrace.cycles-pp.svc_recv.nfsd.kthread.ret_from_fork.ret_from_fork_asm
24.50 -24.5 0.00 perf-profile.calltrace.cycles-pp.svc_handle_xprt.svc_recv.nfsd.kthread.ret_from_fork
16.59 ± 3% -16.6 0.00 perf-profile.calltrace.cycles-pp.svc_process.svc_handle_xprt.svc_recv.nfsd.kthread
16.57 ± 3% -16.6 0.00 perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.svc_handle_xprt.svc_recv.nfsd
16.02 ± 2% -16.0 0.00 perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.svc_handle_xprt.svc_recv
15.33 ± 2% -15.3 0.00 perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.svc_handle_xprt
13.29 ± 5% -13.3 0.00 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
12.88 ± 4% -12.9 0.00 perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
10.99 ± 8% -11.0 0.00 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
9.54 ± 5% -9.5 0.00 perf-profile.calltrace.cycles-pp.kjournald2.kthread.ret_from_fork.ret_from_fork_asm
8.75 ± 5% -8.7 0.00 perf-profile.calltrace.cycles-pp.jbd2_journal_commit_transaction.kjournald2.kthread.ret_from_fork.ret_from_fork_asm
6.78 ± 4% -6.8 0.00 perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state
6.10 ± 3% -6.1 0.00 perf-profile.calltrace.cycles-pp.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames
5.97 ± 4% -6.0 0.00 perf-profile.calltrace.cycles-pp.nfsd4_create.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
5.93 ± 4% -5.9 0.00 perf-profile.calltrace.cycles-pp.ip6_finish_output.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb.tcp_write_xmit
5.88 ± 7% -5.9 0.00 perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
5.87 ± 5% -5.9 0.00 perf-profile.calltrace.cycles-pp.nfsd_create.nfsd4_create.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
5.86 ± 4% -5.9 0.00 perf-profile.calltrace.cycles-pp.ip6_finish_output2.ip6_finish_output.ip6_xmit.inet6_csk_xmit.__tcp_transmit_skb
5.85 ± 7% -5.9 0.00 perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
5.73 ± 4% -5.7 0.00 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip6_finish_output2.ip6_finish_output.ip6_xmit.inet6_csk_xmit
5.40 ± 15% -5.4 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
5.10 ± 8% -5.1 0.00 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip6_finish_output2.ip6_finish_output.ip6_xmit
5.08 ± 8% -5.1 0.00 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip6_finish_output2.ip6_finish_output
5.04 ± 9% -5.0 0.00 perf-profile.calltrace.cycles-pp.handle_softirqs.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip6_finish_output2
4.92 ± 6% -4.9 0.00 perf-profile.calltrace.cycles-pp.xs_tcp_send_request.xprt_request_transmit.xprt_transmit.call_transmit.__rpc_execute
50.28 -50.3 0.00 perf-profile.children.cycles-pp.ret_from_fork_asm
50.27 -50.3 0.00 perf-profile.children.cycles-pp.ret_from_fork
50.27 -50.3 0.00 perf-profile.children.cycles-pp.kthread
37.89 ± 2% -37.9 0.00 perf-profile.children.cycles-pp.common_startup_64
37.89 ± 2% -37.9 0.00 perf-profile.children.cycles-pp.cpu_startup_entry
37.83 ± 2% -37.8 0.00 perf-profile.children.cycles-pp.do_idle
36.23 ± 2% -36.2 0.00 perf-profile.children.cycles-pp.start_secondary
35.18 ± 2% -35.2 0.00 perf-profile.children.cycles-pp.cpuidle_idle_call
33.25 ± 2% -33.2 0.00 perf-profile.children.cycles-pp.cpuidle_enter
33.11 ± 2% -33.1 0.00 perf-profile.children.cycles-pp.cpuidle_enter_state
31.90 ± 2% -31.9 0.00 perf-profile.children.cycles-pp.acpi_idle_enter
31.85 ± 2% -31.8 0.00 perf-profile.children.cycles-pp.acpi_safe_halt
27.02 -27.0 0.00 perf-profile.children.cycles-pp.nfsd
26.84 -26.8 0.00 perf-profile.children.cycles-pp.svc_recv
25.33 ± 4% -25.3 0.00 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
24.51 -24.5 0.00 perf-profile.children.cycles-pp.svc_handle_xprt
16.59 ± 3% -16.6 0.00 perf-profile.children.cycles-pp.svc_process
16.57 ± 3% -16.6 0.00 perf-profile.children.cycles-pp.svc_process_common
16.03 ± 2% -16.0 0.00 perf-profile.children.cycles-pp.nfsd_dispatch
15.35 ± 2% -15.4 0.00 perf-profile.children.cycles-pp.nfsd4_proc_compound
13.31 ± 5% -13.3 0.00 perf-profile.children.cycles-pp.worker_thread
11.01 ± 8% -11.0 0.00 perf-profile.children.cycles-pp.process_one_work
9.63 ± 6% -9.6 0.00 perf-profile.children.cycles-pp.__rpc_execute
9.55 ± 3% -9.6 0.00 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
9.54 ± 5% -9.5 0.00 perf-profile.children.cycles-pp.kjournald2
9.54 ± 3% -9.5 0.00 perf-profile.children.cycles-pp.do_syscall_64
8.78 ± 5% -8.8 0.00 perf-profile.children.cycles-pp.jbd2_journal_commit_transaction
7.36 ± 3% -7.4 0.00 perf-profile.children.cycles-pp.handle_softirqs
7.32 ± 3% -7.3 0.00 perf-profile.children.cycles-pp.__tcp_push_pending_frames
7.29 ± 3% -7.3 0.00 perf-profile.children.cycles-pp.tcp_write_xmit
6.86 ± 2% -6.9 0.00 perf-profile.children.cycles-pp.vfs_mkdir
6.84 ± 3% -6.8 0.00 perf-profile.children.cycles-pp.__tcp_transmit_skb
6.82 ± 4% -6.8 0.00 perf-profile.children.cycles-pp.sysvec_call_function_single
6.30 ± 3% -6.3 0.00 perf-profile.children.cycles-pp.inet6_csk_xmit
6.10 ± 3% -6.1 0.00 perf-profile.children.cycles-pp.ip6_xmit
5.97 ± 4% -6.0 0.00 perf-profile.children.cycles-pp.nfsd4_create
5.93 ± 4% -5.9 0.00 perf-profile.children.cycles-pp.ip6_finish_output
5.88 ± 7% -5.9 0.00 perf-profile.children.cycles-pp.rpc_async_schedule
5.88 ± 5% -5.9 0.00 perf-profile.children.cycles-pp.nfsd_create
5.86 ± 4% -5.9 0.00 perf-profile.children.cycles-pp.ip6_finish_output2
5.74 ± 4% -5.7 0.00 perf-profile.children.cycles-pp.__dev_queue_xmit
5.53 ± 4% -5.5 0.00 perf-profile.children.cycles-pp.__local_bh_enable_ip
5.51 ± 3% -5.5 0.00 perf-profile.children.cycles-pp.sock_sendmsg
5.45 ± 4% -5.4 0.00 perf-profile.children.cycles-pp.do_softirq
5.38 ± 3% -5.4 0.00 perf-profile.children.cycles-pp.tcp_sendmsg
5.31 ± 4% -5.3 0.00 perf-profile.children.cycles-pp.__schedule
5.18 ± 6% -5.2 0.00 perf-profile.children.cycles-pp.call_transmit
5.14 ± 6% -5.1 0.00 perf-profile.children.cycles-pp.xprt_transmit
5.10 ± 6% -5.1 0.00 perf-profile.children.cycles-pp.xprt_request_transmit
5.08 ± 2% -5.1 0.00 perf-profile.children.cycles-pp.tcp_sendmsg_locked
5.02 ± 5% -5.0 0.00 perf-profile.children.cycles-pp.net_rx_action
4.93 ± 5% -4.9 0.00 perf-profile.children.cycles-pp.xs_tcp_send_request
21.35 ± 6% -21.4 0.00 perf-profile.self.cycles-pp.acpi_safe_halt

Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.

--
0-DAY CI Kernel Test Service