* [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression
@ 2024-08-08 7:35 kernel test robot
2024-08-12 11:21 ` Christoph Hellwig
0 siblings, 1 reply; 7+ messages in thread
From: kernel test robot @ 2024-08-08 7:35 UTC (permalink / raw)
To: Christoph Hellwig
Cc: oe-lkp, lkp, linux-kernel, Anna Schumaker, Sagi Grimberg,
linux-nfs, ying.huang, feng.tang, fengwei.yin, oliver.sang
Hello,
kernel test robot noticed a -85.6% regression of filebench.sum_operations/s on:
commit: 49b29a573da83b65d5f4ecf2db6619bab7aa910c ("nfs: add support for large folios")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
testcase: filebench
test machine: 128 threads 2 sockets Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz (Ice Lake) with 128G memory
parameters:
disk: 1HDD
fs: ext4
fs2: nfsv4
test: randomrw.f
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202408081514.106c770e-oliver.sang@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240808/202408081514.106c770e-oliver.sang@intel.com
=========================================================================================
compiler/cpufreq_governor/disk/fs2/fs/kconfig/rootfs/tbox_group/test/testcase:
gcc-13/performance/1HDD/nfsv4/ext4/x86_64-rhel-8.3/debian-12-x86_64-20240206.cgz/lkp-icl-2sp6/randomrw.f/filebench
commit:
9aac777aaf ("filemap: Convert generic_perform_write() to support large folios")
49b29a573d ("nfs: add support for large folios")
9aac777aaf945978 49b29a573da83b65d5f4ecf2db6
---------------- ---------------------------
%stddev %change %stddev
\ | \
98.36 -1.0% 97.34 iostat.cpu.idle
1.28 ± 21% +89.3% 2.42 ± 10% iostat.cpu.iowait
203868 ± 9% -25.7% 151373 ± 17% numa-meminfo.node0.SUnreclaim
1026057 ± 40% -76.5% 241071 ± 70% numa-meminfo.node1.Dirty
1693537 ± 24% -74.2% 437697 ± 27% sched_debug.cpu.nr_switches.max
191731 ± 18% -61.9% 73070 ± 25% sched_debug.cpu.nr_switches.stddev
1849853 ± 66% +175.6% 5097620 ± 32% numa-vmstat.node0.nr_dirtied
50982 ± 9% -25.8% 37842 ± 17% numa-vmstat.node0.nr_slab_unreclaimable
1849844 ± 66% +175.6% 5097604 ± 32% numa-vmstat.node0.nr_written
256902 ± 40% -76.9% 59457 ± 71% numa-vmstat.node1.nr_dirty
43810 +35.9% 59529 ± 5% vmstat.io.bo
1.65 ± 22% +93.6% 3.19 ± 9% vmstat.procs.b
1.54 ± 3% -11.2% 1.37 vmstat.procs.r
4341 ± 3% -18.4% 3543 ± 4% vmstat.system.in
3713739 ± 3% +10.7% 4110439 ± 3% meminfo.Active
3688730 ± 2% +10.7% 4085171 ± 3% meminfo.Active(file)
1631748 ± 3% -63.1% 602285 ± 12% meminfo.Dirty
398961 -23.8% 304003 meminfo.SUnreclaim
655165 -16.2% 549118 meminfo.Slab
1411642 ± 3% +56.0% 2202614 ± 5% meminfo.Writeback
1.29 ± 21% +1.2 2.44 ± 10% mpstat.cpu.all.iowait%
0.02 ± 2% +0.0 0.02 ± 5% mpstat.cpu.all.soft%
0.28 ± 5% -0.1 0.15 ± 6% mpstat.cpu.all.sys%
0.05 ± 2% -0.0 0.04 ± 6% mpstat.cpu.all.usr%
22.67 ±184% -86.8% 3.00 mpstat.max_utilization.seconds
7.61 ± 9% -12.7% 6.65 mpstat.max_utilization_pct
919.42 ± 16% -85.6% 132.50 ± 34% filebench.sum_bytes_mb/s
7061641 ± 16% -85.6% 1017633 ± 34% filebench.sum_operations
117684 ± 16% -85.6% 16959 ± 34% filebench.sum_operations/s
59443 ± 16% -85.7% 8524 ± 34% filebench.sum_reads/s
0.02 ± 13% +611.2% 0.12 ± 31% filebench.sum_time_ms/op
58241 ± 16% -85.5% 8435 ± 34% filebench.sum_writes/s
21810424 +37.4% 29966336 ± 11% filebench.time.file_system_outputs
14251 -5.6% 13458 ± 2% filebench.time.minor_page_faults
17.67 ± 5% -90.6% 1.67 ± 66% filebench.time.percent_of_cpu_this_job_got
41.30 ± 5% -88.1% 4.90 ± 43% filebench.time.system_time
3582819 ± 16% -85.8% 509229 ± 33% filebench.time.voluntary_context_switches
921497 ± 2% +11.0% 1022743 ± 3% proc-vmstat.nr_active_file
5452715 +37.4% 7491524 ± 11% proc-vmstat.nr_dirtied
407500 ± 2% -62.8% 151456 ± 13% proc-vmstat.nr_dirty
1653629 -5.4% 1564747 ± 2% proc-vmstat.nr_inactive_file
64005 -4.2% 61291 proc-vmstat.nr_slab_reclaimable
99730 -23.8% 75999 proc-vmstat.nr_slab_unreclaimable
352757 ± 3% +55.8% 549420 ± 5% proc-vmstat.nr_writeback
5452662 +37.4% 7491481 ± 11% proc-vmstat.nr_written
921497 ± 2% +11.0% 1022743 ± 3% proc-vmstat.nr_zone_active_file
1653629 -5.4% 1564747 ± 2% proc-vmstat.nr_zone_inactive_file
1655 ± 15% -42.7% 947.83 ± 33% proc-vmstat.numa_hint_faults
1546 ± 16% -53.2% 723.33 ± 38% proc-vmstat.numa_hint_faults_local
4381534 ± 6% -25.2% 3276231 ± 16% proc-vmstat.numa_hit
4248725 ± 6% -26.0% 3143651 ± 17% proc-vmstat.numa_local
10725 ± 54% -57.7% 4532 ± 35% proc-vmstat.numa_pte_updates
1271493 ± 2% +3.6% 1317523 proc-vmstat.pgactivate
10906410 +37.4% 14983963 ± 11% proc-vmstat.pgpgout
1.982e+08 ± 5% -35.7% 1.274e+08 ± 3% perf-stat.i.branch-instructions
7.83 ± 3% -1.4 6.40 ± 7% perf-stat.i.cache-miss-rate%
5551160 ± 7% -48.1% 2879590 ± 5% perf-stat.i.cache-misses
25389854 ± 6% -30.6% 17609995 ± 2% perf-stat.i.cache-references
1.92 +3.1% 1.98 perf-stat.i.cpi
1.685e+09 ± 4% -35.6% 1.086e+09 ± 5% perf-stat.i.cpu-cycles
175.05 ± 3% -15.6% 147.75 ± 2% perf-stat.i.cpu-migrations
1277 ± 3% +27.3% 1625 ± 8% perf-stat.i.cycles-between-cache-misses
8.992e+08 ± 5% -32.6% 6.062e+08 ± 3% perf-stat.i.instructions
0.57 -3.2% 0.55 perf-stat.i.ipc
0.24 ± 14% -27.6% 0.17 ± 23% perf-stat.i.metric.K/sec
6.17 ± 2% -22.8% 4.76 ± 3% perf-stat.overall.MPKI
1.95 ± 4% +0.8 2.76 ± 4% perf-stat.overall.branch-miss-rate%
21.85 ± 2% -5.5 16.37 ± 4% perf-stat.overall.cache-miss-rate%
304.50 ± 4% +23.8% 377.10 ± 4% perf-stat.overall.cycles-between-cache-misses
0.53 ± 2% +4.7% 0.56 ± 4% perf-stat.overall.ipc
1.975e+08 ± 5% -35.7% 1.27e+08 ± 3% perf-stat.ps.branch-instructions
3849663 -8.9% 3506691 ± 4% perf-stat.ps.branch-misses
5532498 ± 7% -48.0% 2879018 ± 5% perf-stat.ps.cache-misses
25301762 ± 6% -30.5% 17579581 perf-stat.ps.cache-references
1.68e+09 ± 4% -35.4% 1.084e+09 ± 5% perf-stat.ps.cpu-cycles
174.27 ± 3% -15.6% 147.10 ± 2% perf-stat.ps.cpu-migrations
8.961e+08 ± 5% -32.5% 6.045e+08 ± 3% perf-stat.ps.instructions
2.211e+11 ± 4% -32.0% 1.504e+11 ± 6% perf-stat.total.instructions
52.52 ± 12% -45.4 7.08 ± 20% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
51.90 ± 12% -45.0 6.86 ± 20% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
79.10 ± 9% -43.9 35.15 ± 29% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
79.10 ± 9% -43.9 35.15 ± 29% perf-profile.calltrace.cycles-pp.ret_from_fork_asm
79.10 ± 9% -43.9 35.15 ± 29% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
42.11 ± 12% -42.1 0.00 perf-profile.calltrace.cycles-pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_from_fork
42.11 ± 12% -42.1 0.00 perf-profile.calltrace.cycles-pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.kthread
28.67 ± 19% -28.7 0.00 perf-profile.calltrace.cycles-pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work.worker_thread
16.89 ± 31% -16.9 0.00 perf-profile.calltrace.cycles-pp.nfs_page_end_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
16.28 ± 32% -16.3 0.00 perf-profile.calltrace.cycles-pp.folio_end_writeback.nfs_page_end_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
15.82 ± 32% -15.8 0.00 perf-profile.calltrace.cycles-pp.__folio_end_writeback.folio_end_writeback.nfs_page_end_writeback.nfs_write_completion.rpc_free_task
13.99 ± 35% -14.0 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__folio_end_writeback.folio_end_writeback.nfs_page_end_writeback.nfs_write_completion
13.55 ± 35% -13.5 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__folio_end_writeback.folio_end_writeback.nfs_page_end_writeback
13.42 ± 15% -13.4 0.00 perf-profile.calltrace.cycles-pp.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work.worker_thread
13.42 ± 15% -13.4 0.00 perf-profile.calltrace.cycles-pp.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work
10.58 ± 16% -10.6 0.00 perf-profile.calltrace.cycles-pp.nfs_request_add_commit_list.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
10.48 ± 15% -10.5 0.00 perf-profile.calltrace.cycles-pp.nfs_inode_remove_request.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_async_release
8.89 ± 17% -8.9 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock.nfs_inode_remove_request.nfs_commit_release_pages.nfs_commit_release.rpc_free_task
8.19 ± 17% -8.2 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.nfs_inode_remove_request.nfs_commit_release_pages.nfs_commit_release
7.56 ± 24% -7.6 0.00 perf-profile.calltrace.cycles-pp.__mutex_lock.nfs_request_add_commit_list.nfs_write_completion.rpc_free_task.rpc_async_release
6.57 ± 27% -6.6 0.00 perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.nfs_request_add_commit_list.nfs_write_completion.rpc_free_task
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file.nfsd_commit.nfsd4_commit
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.calltrace.cycles-pp.ext4_do_writepages.ext4_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file.nfsd_commit
4.45 ± 19% -2.1 2.31 ± 25% perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_do_writepages.ext4_writepages.do_writepages.filemap_fdatawrite_wbc
4.05 ± 13% -1.3 2.74 ± 21% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.20 ±141% +0.6 0.75 ± 16% perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
0.20 ±141% +0.6 0.75 ± 16% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.start_kernel.x86_64_start_reservations
0.20 ±141% +0.6 0.75 ± 16% perf-profile.calltrace.cycles-pp.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
0.20 ±141% +0.6 0.75 ± 16% perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
0.20 ±141% +0.6 0.75 ± 16% perf-profile.calltrace.cycles-pp.x86_64_start_kernel.common_startup_64
0.20 ±141% +0.6 0.75 ± 16% perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
0.00 +0.6 0.59 ± 14% perf-profile.calltrace.cycles-pp.tick_nohz_idle_exit.do_idle.cpu_startup_entry.start_secondary.common_startup_64
0.00 +1.1 1.12 ± 34% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
0.00 +1.2 1.16 ± 32% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue
0.31 ±102% +1.4 1.68 ± 15% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
0.22 ±141% +1.4 1.62 ± 15% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.00 +1.6 1.60 ± 25% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle
0.12 ±223% +2.4 2.50 ± 23% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
0.23 ±146% +3.1 3.36 ± 23% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
0.00 +3.1 3.13 ± 95% perf-profile.calltrace.cycles-pp.folio_wait_bit_common.folio_wait_writeback.__filemap_fdatawait_range.file_write_and_wait_range.ext4_sync_file
0.00 +3.8 3.77 ± 94% perf-profile.calltrace.cycles-pp.folio_wait_writeback.__filemap_fdatawait_range.file_write_and_wait_range.ext4_sync_file.nfsd_commit
1.51 ± 30% +4.3 5.78 ± 48% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.common_startup_64
0.00 +4.5 4.53 ± 94% perf-profile.calltrace.cycles-pp.__filemap_fdatawait_range.file_write_and_wait_range.ext4_sync_file.nfsd_commit.nfsd4_commit
0.71 ± 24% +10.0 10.76 ± 40% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
10.00 ± 10% +12.6 22.59 ± 13% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
8.46 ± 10% +12.7 21.13 ± 14% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
8.27 ± 11% +13.2 21.46 ± 14% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
12.87 ± 10% +18.6 31.52 ± 18% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
12.89 ± 10% +18.7 31.56 ± 18% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
12.90 ± 10% +18.7 31.60 ± 18% perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
13.40 ± 10% +19.0 32.35 ± 17% perf-profile.calltrace.cycles-pp.common_startup_64
52.52 ± 12% -45.4 7.08 ± 20% perf-profile.children.cycles-pp.worker_thread
51.90 ± 12% -45.0 6.86 ± 20% perf-profile.children.cycles-pp.process_one_work
79.10 ± 9% -43.9 35.15 ± 29% perf-profile.children.cycles-pp.kthread
79.12 ± 9% -43.9 35.18 ± 29% perf-profile.children.cycles-pp.ret_from_fork
79.12 ± 9% -43.9 35.18 ± 29% perf-profile.children.cycles-pp.ret_from_fork_asm
42.11 ± 12% -42.0 0.08 ± 19% perf-profile.children.cycles-pp.rpc_async_release
42.11 ± 12% -42.0 0.08 ± 19% perf-profile.children.cycles-pp.rpc_free_task
28.68 ± 19% -28.6 0.06 ± 19% perf-profile.children.cycles-pp.nfs_write_completion
23.08 ± 18% -21.5 1.54 ±113% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
16.89 ± 31% -16.9 0.00 perf-profile.children.cycles-pp.nfs_page_end_writeback
16.54 ± 31% -15.9 0.62 ± 42% perf-profile.children.cycles-pp.__folio_end_writeback
17.05 ± 31% -14.9 2.20 ± 93% perf-profile.children.cycles-pp.folio_end_writeback
15.65 ± 32% -14.4 1.30 ± 49% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
13.42 ± 15% -13.4 0.00 perf-profile.children.cycles-pp.nfs_commit_release
13.42 ± 15% -13.4 0.00 perf-profile.children.cycles-pp.nfs_commit_release_pages
10.58 ± 16% -10.6 0.00 perf-profile.children.cycles-pp.nfs_request_add_commit_list
10.48 ± 15% -10.5 0.00 perf-profile.children.cycles-pp.nfs_inode_remove_request
9.54 ± 16% -7.8 1.70 ± 31% perf-profile.children.cycles-pp._raw_spin_lock
7.59 ± 24% -7.6 0.00 perf-profile.children.cycles-pp.__mutex_lock
7.33 ± 14% -4.7 2.62 ± 24% perf-profile.children.cycles-pp.do_writepages
2.73 ± 25% -2.5 0.18 ± 47% perf-profile.children.cycles-pp.__writeback_inodes_wb
2.73 ± 25% -2.5 0.18 ± 47% perf-profile.children.cycles-pp.__writeback_single_inode
2.73 ± 25% -2.5 0.18 ± 47% perf-profile.children.cycles-pp.nfs_writepages
2.73 ± 25% -2.5 0.18 ± 47% perf-profile.children.cycles-pp.wb_do_writeback
2.73 ± 25% -2.5 0.18 ± 47% perf-profile.children.cycles-pp.wb_writeback
2.73 ± 25% -2.5 0.18 ± 48% perf-profile.children.cycles-pp.write_cache_pages
2.73 ± 25% -2.5 0.18 ± 47% perf-profile.children.cycles-pp.writeback_sb_inodes
2.73 ± 25% -2.5 0.18 ± 48% perf-profile.children.cycles-pp.wb_workfn
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.children.cycles-pp.ext4_do_writepages
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.children.cycles-pp.ext4_writepages
4.60 ± 18% -2.2 2.44 ± 25% perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
4.46 ± 19% -2.1 2.32 ± 25% perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
2.27 ± 28% -2.1 0.16 ± 46% perf-profile.children.cycles-pp.nfs_writepages_callback
2.20 ± 29% -2.0 0.16 ± 46% perf-profile.children.cycles-pp.nfs_page_async_flush
4.16 ± 12% -1.2 2.94 ± 21% perf-profile.children.cycles-pp.intel_idle
1.48 ± 28% -0.8 0.68 ± 22% perf-profile.children.cycles-pp.__folio_start_writeback
1.30 ± 14% -0.7 0.58 ± 19% perf-profile.children.cycles-pp.sched_balance_newidle
0.85 ± 29% -0.7 0.15 ± 37% perf-profile.children.cycles-pp.kmem_cache_free
1.44 ± 13% -0.7 0.77 ± 19% perf-profile.children.cycles-pp.sched_balance_rq
1.31 ± 13% -0.7 0.65 ± 18% perf-profile.children.cycles-pp.sched_balance_find_src_group
1.30 ± 14% -0.6 0.64 ± 17% perf-profile.children.cycles-pp.update_sd_lb_stats
1.16 ± 13% -0.6 0.57 ± 19% perf-profile.children.cycles-pp.update_sg_lb_stats
0.79 ± 22% -0.6 0.22 ±121% perf-profile.children.cycles-pp.nfs_folio_find_private_request
1.25 ± 13% -0.5 0.77 ± 22% perf-profile.children.cycles-pp.pick_next_task_fair
0.55 ± 20% -0.3 0.22 ± 39% perf-profile.children.cycles-pp.tick_nohz_stop_tick
0.56 ± 20% -0.3 0.23 ± 37% perf-profile.children.cycles-pp.tick_nohz_idle_stop_tick
0.34 ± 29% -0.3 0.05 ± 76% perf-profile.children.cycles-pp.__slab_free
0.68 ± 21% -0.3 0.40 ± 24% perf-profile.children.cycles-pp.__lruvec_stat_mod_folio
0.34 ± 25% -0.3 0.08 ± 27% perf-profile.children.cycles-pp.__xa_clear_mark
0.54 ± 10% -0.2 0.30 ± 17% perf-profile.children.cycles-pp.percpu_counter_add_batch
0.22 ± 15% -0.2 0.03 ± 70% perf-profile.children.cycles-pp.seq_read
0.38 ± 11% -0.2 0.20 ± 12% perf-profile.children.cycles-pp.seq_read_iter
0.21 ± 15% -0.2 0.04 ±102% perf-profile.children.cycles-pp.blk_mq_run_work_fn
0.62 ± 8% -0.2 0.45 ± 9% perf-profile.children.cycles-pp.ksys_read
0.24 ± 12% -0.2 0.07 ± 55% perf-profile.children.cycles-pp.blk_mq_sched_dispatch_requests
0.23 ± 12% -0.2 0.07 ± 55% perf-profile.children.cycles-pp.__blk_mq_sched_dispatch_requests
0.25 ± 22% -0.2 0.10 ± 25% perf-profile.children.cycles-pp.submit_bio_noacct_nocheck
0.22 ± 15% -0.2 0.07 ± 55% perf-profile.children.cycles-pp.__blk_mq_do_dispatch_sched
0.66 ± 8% -0.1 0.51 ± 9% perf-profile.children.cycles-pp.read
0.24 ± 25% -0.1 0.10 ± 26% perf-profile.children.cycles-pp.__submit_bio
0.24 ± 25% -0.1 0.10 ± 26% perf-profile.children.cycles-pp.blk_mq_submit_bio
0.21 ± 19% -0.1 0.07 ± 31% perf-profile.children.cycles-pp.__common_interrupt
0.19 ± 21% -0.1 0.05 ± 50% perf-profile.children.cycles-pp.__handle_irq_event_percpu
0.21 ± 20% -0.1 0.07 ± 28% perf-profile.children.cycles-pp.handle_edge_irq
0.20 ± 21% -0.1 0.06 ± 53% perf-profile.children.cycles-pp.handle_irq_event
0.18 ± 22% -0.1 0.05 ± 50% perf-profile.children.cycles-pp.ahci_single_level_irq_intr
0.24 ± 33% -0.1 0.10 ± 13% perf-profile.children.cycles-pp.idle_cpu
0.24 ± 18% -0.1 0.12 ± 30% perf-profile.children.cycles-pp.__get_next_timer_interrupt
0.28 ± 17% -0.1 0.16 ± 26% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.16 ± 26% -0.1 0.04 ± 73% perf-profile.children.cycles-pp.ahci_handle_port_intr
0.28 ± 22% -0.1 0.17 ± 10% perf-profile.children.cycles-pp.svc_send
0.28 ± 24% -0.1 0.17 ± 9% perf-profile.children.cycles-pp.svc_tcp_sendto
0.23 ± 19% -0.1 0.12 ± 19% perf-profile.children.cycles-pp.__mod_lruvec_state
0.16 ± 11% -0.1 0.06 ± 47% perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list
0.27 ± 10% -0.1 0.17 ± 30% perf-profile.children.cycles-pp.tick_nohz_restart_sched_tick
0.14 ± 17% -0.1 0.04 ±112% perf-profile.children.cycles-pp.nfs_unlock_and_release_request
0.16 ± 13% -0.1 0.06 ± 47% perf-profile.children.cycles-pp.scsi_queue_rq
0.16 ± 22% -0.1 0.06 ± 50% perf-profile.children.cycles-pp.blk_mq_flush_plug_list
0.25 ± 26% -0.1 0.16 ± 4% perf-profile.children.cycles-pp.svc_tcp_sendmsg
0.19 ± 24% -0.1 0.10 ± 12% perf-profile.children.cycles-pp.__mod_node_page_state
0.16 ± 20% -0.1 0.06 ± 50% perf-profile.children.cycles-pp.blk_mq_dispatch_plug_list
0.21 ± 13% -0.1 0.13 ± 33% perf-profile.children.cycles-pp.tick_nohz_next_event
0.18 ± 25% -0.1 0.11 ± 25% perf-profile.children.cycles-pp.xas_start
0.11 ± 38% -0.1 0.04 ± 75% perf-profile.children.cycles-pp.inode_to_bdi
0.14 ± 23% -0.1 0.07 ± 52% perf-profile.children.cycles-pp.kick_pool
0.16 ± 27% -0.1 0.10 ± 15% perf-profile.children.cycles-pp.xs_stream_data_receive_workfn
0.19 ± 18% -0.1 0.12 ± 23% perf-profile.children.cycles-pp.__queue_work
0.10 ± 10% -0.1 0.04 ± 71% perf-profile.children.cycles-pp._find_next_and_bit
0.09 ± 11% -0.1 0.03 ±102% perf-profile.children.cycles-pp.hrtimer_cancel
0.17 ± 17% -0.1 0.12 ± 25% perf-profile.children.cycles-pp.sched_balance_update_blocked_averages
0.09 ± 18% -0.1 0.04 ±105% perf-profile.children.cycles-pp.__lock_sock
0.12 ± 10% -0.1 0.06 ± 45% perf-profile.children.cycles-pp.__hrtimer_start_range_ns
0.12 ± 19% -0.0 0.07 ± 34% perf-profile.children.cycles-pp.lock_sock_nested
0.04 ± 73% +0.0 0.09 ± 28% perf-profile.children.cycles-pp.copy_mc_enhanced_fast_string
0.07 ± 14% +0.1 0.12 ± 22% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited_flags
0.01 ±223% +0.1 0.06 ± 21% perf-profile.children.cycles-pp.pick_next_task_idle
0.00 +0.1 0.06 ± 19% perf-profile.children.cycles-pp.sched_clock_noinstr
0.08 ± 20% +0.1 0.14 ± 20% perf-profile.children.cycles-pp.ct_kernel_enter
0.03 ±143% +0.1 0.10 ± 23% perf-profile.children.cycles-pp.__collapse_huge_page_copy
0.00 +0.1 0.07 ± 15% perf-profile.children.cycles-pp.__dequeue_entity
0.02 ±141% +0.1 0.08 ± 23% perf-profile.children.cycles-pp.nohz_run_idle_balance
0.10 ± 17% +0.1 0.17 ± 18% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.12 ± 19% +0.1 0.19 ± 17% perf-profile.children.cycles-pp.update_curr
0.12 ± 15% +0.1 0.20 ± 12% perf-profile.children.cycles-pp.read_tsc
0.02 ±142% +0.1 0.11 ± 24% perf-profile.children.cycles-pp.call_cpuidle
0.01 ±223% +0.1 0.10 ± 20% perf-profile.children.cycles-pp.avg_vruntime
0.00 +0.1 0.10 ± 34% perf-profile.children.cycles-pp.place_entity
0.02 ±142% +0.1 0.12 ± 23% perf-profile.children.cycles-pp.ct_kernel_exit_state
0.06 ± 63% +0.1 0.17 ± 37% perf-profile.children.cycles-pp.__update_load_avg_se
0.12 ± 14% +0.1 0.23 ± 12% perf-profile.children.cycles-pp.___perf_sw_event
0.02 ±144% +0.1 0.16 ± 69% perf-profile.children.cycles-pp.llist_add_batch
0.08 ± 64% +0.2 0.23 ± 17% perf-profile.children.cycles-pp.set_next_entity
0.02 ±223% +0.2 0.18 ± 24% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.04 ±101% +0.2 0.22 ± 7% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.13 ± 20% +0.2 0.33 ± 15% perf-profile.children.cycles-pp.prepare_task_switch
0.15 ± 15% +0.2 0.35 ± 20% perf-profile.children.cycles-pp.sched_clock_cpu
0.11 ± 10% +0.2 0.30 ± 16% perf-profile.children.cycles-pp.local_clock_noinstr
0.40 ± 11% +0.2 0.62 ± 14% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.09 ± 23% +0.2 0.32 ± 21% perf-profile.children.cycles-pp.tick_nohz_stop_idle
0.07 ± 50% +0.2 0.31 ± 21% perf-profile.children.cycles-pp.nr_iowait_cpu
0.11 ± 24% +0.2 0.35 ± 6% perf-profile.children.cycles-pp.__switch_to_asm
0.15 ± 23% +0.2 0.39 ± 48% perf-profile.children.cycles-pp.update_rq_clock_task
0.50 ± 15% +0.2 0.75 ± 16% perf-profile.children.cycles-pp.rest_init
0.50 ± 15% +0.2 0.75 ± 16% perf-profile.children.cycles-pp.start_kernel
0.50 ± 15% +0.2 0.75 ± 16% perf-profile.children.cycles-pp.x86_64_start_kernel
0.50 ± 15% +0.2 0.75 ± 16% perf-profile.children.cycles-pp.x86_64_start_reservations
0.09 ± 28% +0.3 0.35 ± 18% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.10 ± 81% +0.3 0.36 ± 36% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
0.08 ± 32% +0.3 0.36 ± 46% perf-profile.children.cycles-pp.wake_affine
0.08 ± 33% +0.3 0.40 ± 44% perf-profile.children.cycles-pp.available_idle_cpu
0.22 ± 22% +0.4 0.60 ± 16% perf-profile.children.cycles-pp.dequeue_entity
0.21 ± 21% +0.4 0.61 ± 31% perf-profile.children.cycles-pp.select_task_rq_fair
0.12 ± 30% +0.4 0.52 ± 10% perf-profile.children.cycles-pp.__switch_to
0.26 ± 17% +0.4 0.67 ± 16% perf-profile.children.cycles-pp.dequeue_task_fair
0.22 ± 32% +0.5 0.68 ± 18% perf-profile.children.cycles-pp.update_load_avg
0.17 ± 44% +0.5 0.66 ± 19% perf-profile.children.cycles-pp.sched_clock
0.14 ± 25% +0.5 0.62 ± 18% perf-profile.children.cycles-pp.update_rq_clock
1.82 ± 9% +0.5 2.35 ± 12% perf-profile.children.cycles-pp.schedule
0.10 ± 50% +0.6 0.70 ± 20% perf-profile.children.cycles-pp.llist_reverse_order
0.22 ± 31% +0.6 0.83 ± 11% perf-profile.children.cycles-pp.native_sched_clock
0.19 ± 41% +0.6 0.83 ± 19% perf-profile.children.cycles-pp.select_task_rq
0.11 ± 30% +0.7 0.82 ± 40% perf-profile.children.cycles-pp.call_function_single_prep_ipi
0.24 ± 31% +0.8 1.01 ± 32% perf-profile.children.cycles-pp.enqueue_entity
0.15 ± 29% +0.8 1.00 ± 42% perf-profile.children.cycles-pp.__smp_call_single_queue
0.29 ± 26% +0.9 1.19 ± 31% perf-profile.children.cycles-pp.enqueue_task_fair
0.31 ± 23% +0.9 1.24 ± 30% perf-profile.children.cycles-pp.activate_task
0.18 ± 30% +1.0 1.16 ± 40% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.21 ± 26% +1.1 1.32 ±130% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.58 ± 17% +1.1 1.73 ± 14% perf-profile.children.cycles-pp.schedule_idle
0.35 ± 24% +1.3 1.66 ± 24% perf-profile.children.cycles-pp.ttwu_do_activate
2.40 ± 8% +1.6 3.98 ± 9% perf-profile.children.cycles-pp.__schedule
0.76 ± 14% +1.7 2.45 ± 30% perf-profile.children.cycles-pp.try_to_wake_up
0.43 ± 33% +2.1 2.53 ± 22% perf-profile.children.cycles-pp.sched_ttwu_pending
1.10 ± 19% +2.3 3.37 ± 85% perf-profile.children.cycles-pp.folio_wait_bit_common
4.11 ± 24% +2.7 6.81 ± 14% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.56 ± 32% +2.8 3.40 ± 22% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.93 ±159% +3.6 4.53 ± 89% perf-profile.children.cycles-pp.intel_idle_irq
0.00 +3.8 3.78 ± 94% perf-profile.children.cycles-pp.folio_wait_writeback
1.51 ± 30% +4.3 5.79 ± 48% perf-profile.children.cycles-pp.flush_smp_call_function_queue
0.00 +4.5 4.54 ± 94% perf-profile.children.cycles-pp.__filemap_fdatawait_range
0.72 ± 24% +10.0 10.77 ± 40% perf-profile.children.cycles-pp.poll_idle
10.42 ± 9% +12.8 23.27 ± 13% perf-profile.children.cycles-pp.cpuidle_idle_call
8.76 ± 10% +12.9 21.66 ± 14% perf-profile.children.cycles-pp.cpuidle_enter_state
8.81 ± 10% +12.9 21.73 ± 14% perf-profile.children.cycles-pp.cpuidle_enter
12.90 ± 10% +18.7 31.60 ± 18% perf-profile.children.cycles-pp.start_secondary
13.38 ± 10% +18.9 32.30 ± 17% perf-profile.children.cycles-pp.do_idle
13.40 ± 10% +19.0 32.35 ± 17% perf-profile.children.cycles-pp.common_startup_64
13.40 ± 10% +19.0 32.35 ± 17% perf-profile.children.cycles-pp.cpu_startup_entry
23.04 ± 18% -21.5 1.54 ±112% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
4.16 ± 12% -1.2 2.94 ± 21% perf-profile.self.cycles-pp.intel_idle
1.38 ± 16% -1.2 0.17 ± 70% perf-profile.self.cycles-pp.__folio_end_writeback
0.70 ± 22% -0.6 0.08 ± 84% perf-profile.self.cycles-pp.nfs_folio_find_private_request
0.87 ± 15% -0.4 0.44 ± 19% perf-profile.self.cycles-pp.update_sg_lb_stats
0.50 ± 21% -0.4 0.10 ±105% perf-profile.self.cycles-pp.folio_end_writeback
0.37 ± 29% -0.3 0.04 ±105% perf-profile.self.cycles-pp.kmem_cache_free
0.33 ± 30% -0.3 0.04 ±102% perf-profile.self.cycles-pp.__slab_free
0.49 ± 11% -0.2 0.26 ± 17% perf-profile.self.cycles-pp.percpu_counter_add_batch
0.26 ± 32% -0.2 0.10 ± 34% perf-profile.self.cycles-pp.mpage_prepare_extent_to_map
0.30 ± 26% -0.1 0.17 ± 33% perf-profile.self.cycles-pp.__folio_start_writeback
0.20 ± 20% -0.1 0.10 ± 14% perf-profile.self.cycles-pp.idle_cpu
0.18 ± 24% -0.1 0.09 ± 15% perf-profile.self.cycles-pp.__mod_node_page_state
0.20 ± 22% -0.1 0.11 ± 25% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.09 ± 17% -0.1 0.03 ±100% perf-profile.self.cycles-pp._find_next_and_bit
0.04 ± 73% +0.0 0.09 ± 28% perf-profile.self.cycles-pp.copy_mc_enhanced_fast_string
0.12 ± 16% +0.1 0.20 ± 15% perf-profile.self.cycles-pp.read_tsc
0.02 ±141% +0.1 0.09 ± 30% perf-profile.self.cycles-pp.sched_ttwu_pending
0.02 ±141% +0.1 0.09 ± 35% perf-profile.self.cycles-pp.balance_dirty_pages_ratelimited_flags
0.01 ±223% +0.1 0.08 ± 25% perf-profile.self.cycles-pp.schedule
0.01 ±223% +0.1 0.09 ± 30% perf-profile.self.cycles-pp.select_task_rq_fair
0.00 +0.1 0.08 ± 19% perf-profile.self.cycles-pp.nohz_run_idle_balance
0.03 ±101% +0.1 0.12 ± 20% perf-profile.self.cycles-pp.dequeue_entity
0.09 ± 26% +0.1 0.18 ± 16% perf-profile.self.cycles-pp.___perf_sw_event
0.01 ±223% +0.1 0.10 ± 17% perf-profile.self.cycles-pp.avg_vruntime
0.02 ±141% +0.1 0.10 ± 26% perf-profile.self.cycles-pp.call_cpuidle
0.00 +0.1 0.09 ± 24% perf-profile.self.cycles-pp.cpu_startup_entry
0.05 ± 88% +0.1 0.15 ± 36% perf-profile.self.cycles-pp.__update_load_avg_se
0.02 ±141% +0.1 0.12 ± 22% perf-profile.self.cycles-pp.ct_kernel_exit_state
0.06 ± 52% +0.1 0.17 ± 77% perf-profile.self.cycles-pp.ext4_finish_bio
0.02 ±142% +0.1 0.12 ± 36% perf-profile.self.cycles-pp.prepare_task_switch
0.02 ±141% +0.1 0.13 ± 27% perf-profile.self.cycles-pp.pick_next_task_fair
0.02 ±141% +0.1 0.13 ± 41% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.05 ± 79% +0.1 0.16 ± 20% perf-profile.self.cycles-pp.generic_perform_write
0.07 ± 15% +0.1 0.19 ± 24% perf-profile.self.cycles-pp.update_load_avg
0.09 ± 14% +0.1 0.22 ± 21% perf-profile.self.cycles-pp.cpuidle_idle_call
0.04 ± 73% +0.1 0.18 ± 45% perf-profile.self.cycles-pp.enqueue_task_fair
0.02 ±144% +0.1 0.16 ± 71% perf-profile.self.cycles-pp.llist_add_batch
0.01 ±223% +0.2 0.17 ± 47% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.02 ±223% +0.2 0.18 ± 24% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.05 ± 46% +0.2 0.23 ± 28% perf-profile.self.cycles-pp.enqueue_entity
0.05 ± 74% +0.2 0.26 ± 23% perf-profile.self.cycles-pp.do_idle
0.10 ± 23% +0.2 0.34 ± 8% perf-profile.self.cycles-pp.__switch_to_asm
0.07 ± 50% +0.2 0.31 ± 21% perf-profile.self.cycles-pp.nr_iowait_cpu
0.09 ± 28% +0.2 0.34 ± 17% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.11 ± 26% +0.2 0.36 ± 54% perf-profile.self.cycles-pp.update_rq_clock_task
0.03 ±163% +0.3 0.30 ± 35% perf-profile.self.cycles-pp.select_task_rq
0.07 ± 50% +0.3 0.36 ± 91% perf-profile.self.cycles-pp._raw_spin_lock_irq
0.00 +0.3 0.29 ± 98% perf-profile.self.cycles-pp.__filemap_fdatawait_range
0.08 ± 33% +0.3 0.39 ± 45% perf-profile.self.cycles-pp.available_idle_cpu
0.02 ±141% +0.3 0.34 ± 27% perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.06 ± 57% +0.4 0.42 ± 18% perf-profile.self.cycles-pp.update_rq_clock
0.14 ± 22% +0.4 0.50 ± 6% perf-profile.self.cycles-pp.__schedule
0.12 ± 33% +0.4 0.50 ± 11% perf-profile.self.cycles-pp.__switch_to
0.01 ±223% +0.4 0.41 ± 55% perf-profile.self.cycles-pp.ttwu_do_activate
0.22 ± 28% +0.6 0.80 ± 11% perf-profile.self.cycles-pp.native_sched_clock
0.10 ± 50% +0.6 0.70 ± 20% perf-profile.self.cycles-pp.llist_reverse_order
0.00 +0.6 0.62 ± 89% perf-profile.self.cycles-pp.folio_wait_writeback
0.11 ± 30% +0.7 0.82 ± 41% perf-profile.self.cycles-pp.call_function_single_prep_ipi
0.20 ± 20% +1.1 1.27 ± 73% perf-profile.self.cycles-pp.folio_wait_bit_common
0.91 ±163% +3.6 4.50 ± 89% perf-profile.self.cycles-pp.intel_idle_irq
0.71 ± 25% +9.9 10.64 ± 40% perf-profile.self.cycles-pp.poll_idle
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression
2024-08-08 7:35 [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression kernel test robot
@ 2024-08-12 11:21 ` Christoph Hellwig
2024-08-13 6:47 ` Oliver Sang
0 siblings, 1 reply; 7+ messages in thread
From: Christoph Hellwig @ 2024-08-12 11:21 UTC (permalink / raw)
To: kernel test robot
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Anna Schumaker,
Sagi Grimberg, linux-nfs, ying.huang, feng.tang, fengwei.yin
On Thu, Aug 08, 2024 at 03:35:20PM +0800, kernel test robot wrote:
>
>
> Hello,
>
> kernel test robot noticed a -85.6% regression of filebench.sum_operations/s on:
I'm not sure what operations this measures. But if it is what I think it
might be, does this regression go away with the following commit (which
already is in mainline)?
commit 39c910a430370fd25d5b5e4b2f4b24581a705499
Author: Christoph Hellwig <hch@lst.de>
Date: Fri Jul 5 07:42:51 2024 +0200
nfs: do not extend writes to the entire folio
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression
2024-08-12 11:21 ` Christoph Hellwig
@ 2024-08-13 6:47 ` Oliver Sang
2024-08-13 6:49 ` Christoph Hellwig
0 siblings, 1 reply; 7+ messages in thread
From: Oliver Sang @ 2024-08-13 6:47 UTC (permalink / raw)
To: Christoph Hellwig
Cc: oe-lkp, lkp, linux-kernel, Anna Schumaker, Sagi Grimberg,
linux-nfs, ying.huang, feng.tang, fengwei.yin, oliver.sang
hi, Christoph Hellwig,
On Mon, Aug 12, 2024 at 01:21:45PM +0200, Christoph Hellwig wrote:
> On Thu, Aug 08, 2024 at 03:35:20PM +0800, kernel test robot wrote:
> >
> >
> > Hello,
> >
> > kernel test robot noticed a -85.6% regression of filebench.sum_operations/s on:
>
> I'm not sure what operations this measures. But if it is what I think it
> might be, does this regression go away with the following commit (which
> already is in mainline)?
>
> commit 39c910a430370fd25d5b5e4b2f4b24581a705499
> Author: Christoph Hellwig <hch@lst.de>
> Date: Fri Jul 5 07:42:51 2024 +0200
>
> nfs: do not extend writes to the entire folio
>
no in our tests, becomes even worse actually. detail data is in [1]
=========================================================================================
compiler/cpufreq_governor/disk/fs2/fs/kconfig/rootfs/tbox_group/test/testcase:
gcc-12/performance/1HDD/nfsv4/ext4/x86_64-rhel-8.3/debian-12-x86_64-20240206.cgz/lkp-icl-2sp6/randomrw.f/filebench
commit:
9aac777aaf ("filemap: Convert generic_perform_write() to support large folios")
49b29a573d ("nfs: add support for large folios")
39c910a430 ("nfs: do not extend writes to the entire folio")
9aac777aaf945978 49b29a573da83b65d5f4ecf2db6 39c910a430370fd25d5b5e4b2f4
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
100309 ± 35% -81.7% 18307 ±120% -98.9% 1107 ± 9% filebench.sum_operations/s
since recently we convert to use gcc-12, above data for 9aac777aaf 49b29a573d
are different with those in our original report after rerun. but trend is
similar.
and data seem not very stable (though still distinct enough to each other among
different commits), so I list detail data here.
for 9aac777aaf
"filebench.sum_operations/s": [
71903.725,
107727.523,
145913.915,
145871.645,
15254.195,
110211.721,
103905.218,
105653.318,
105911.477,
90742.831
],
for 49b29a573d
"filebench.sum_operations/s": [
10868.825,
11376.494,
21319.105,
9556.2,
8885.713,
83420.582,
12523.666,
10912.477,
9162.987,
5051.223
],
for 39c910a430
"filebench.sum_operations/s": [
1019.414,
1016.246,
1116.425,
973.664,
1128.914,
1089.307,
1362.736,
1099.302,
1157.4,
1114.701
],
[1]
=========================================================================================
compiler/cpufreq_governor/disk/fs2/fs/kconfig/rootfs/tbox_group/test/testcase:
gcc-12/performance/1HDD/nfsv4/ext4/x86_64-rhel-8.3/debian-12-x86_64-20240206.cgz/lkp-icl-2sp6/randomrw.f/filebench
commit:
9aac777aaf ("filemap: Convert generic_perform_write() to support large folios")
49b29a573d ("nfs: add support for large folios")
39c910a430 ("nfs: do not extend writes to the entire folio")
9aac777aaf945978 49b29a573da83b65d5f4ecf2db6 39c910a430370fd25d5b5e4b2f4
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
27254 ± 22% -7.2% 25291 ± 22% -30.2% 19036 ± 5% numa-meminfo.node1.Active(anon)
400.70 ±108% -29.8% 281.30 ±152% -93.9% 24.60 ± 27% perf-c2c.DRAM.local
319.17 +2.7% 327.76 ± 2% -9.2% 289.89 uptime.boot
3.11e+10 +3.8% 3.228e+10 ± 3% -11.9% 2.741e+10 cpuidle..time
4984791 ± 28% -8.4% 4565787 ± 28% -82.2% 887365 cpuidle..usage
98.10 -0.8% 97.34 +1.2% 99.24 iostat.cpu.idle
1.56 ± 23% +55.3% 2.42 ± 8% -63.0% 0.58 iostat.cpu.iowait
6810 ± 22% -7.2% 6321 ± 22% -30.1% 4759 ± 5% numa-vmstat.node1.nr_active_anon
6810 ± 22% -7.2% 6321 ± 22% -30.1% 4759 ± 5% numa-vmstat.node1.nr_zone_active_anon
29209 ± 21% -5.9% 27485 ± 19% -28.0% 21029 ± 4% meminfo.Active(anon)
1488681 ± 15% -57.5% 632263 ± 38% -50.0% 744602 ± 6% meminfo.Dirty
393102 -20.6% 312102 ± 5% -23.4% 301066 meminfo.SUnreclaim
66112 ± 9% -1.6% 65039 ± 7% -16.2% 55394 meminfo.Shmem
648778 -13.8% 559100 ± 3% -15.8% 546419 meminfo.Slab
1.57 ± 23% +0.9 2.43 ± 8% -1.0 0.58 mpstat.cpu.all.iowait%
0.02 ± 15% +0.0 0.02 ± 5% -0.0 0.01 ± 4% mpstat.cpu.all.soft%
0.26 ± 14% -0.1 0.16 ± 16% -0.2 0.10 ± 2% mpstat.cpu.all.sys%
0.05 ± 8% -0.0 0.04 ± 4% -0.0 0.04 mpstat.cpu.all.usr%
4.00 -25.0% 3.00 -25.0% 3.00 mpstat.max_utilization.seconds
42296 ± 7% +32.4% 55991 ± 18% -41.5% 24748 vmstat.io.bo
2.02 ± 24% +54.7% 3.13 ± 8% -63.4% 0.74 ± 6% vmstat.procs.b
1.54 ± 4% -9.7% 1.39 ± 3% -11.7% 1.36 ± 4% vmstat.procs.r
36557 ± 29% -12.8% 31886 ± 32% -91.5% 3124 ± 3% vmstat.system.cs
4732 ± 15% -17.2% 3917 ± 29% -23.4% 3622 ± 2% vmstat.system.in
242.09 +3.6% 250.91 ± 3% -12.1% 212.83 time.elapsed_time
242.09 +3.6% 250.91 ± 3% -12.1% 212.83 time.elapsed_time.max
20761124 ± 8% +37.8% 28604910 ± 20% +62.0% 33640448 ± 6% time.file_system_outputs
89.80 ± 42% -84.0% 14.40 ± 31% -92.9% 6.40 ± 32% time.involuntary_context_switches
14207 -5.3% 13450 ± 2% -10.5% 12715 time.minor_page_faults
15.30 ± 34% -87.6% 1.90 ±129% -100.0% 0.00 time.percent_of_cpu_this_job_got
34.90 ± 32% -83.5% 5.75 ±101% -97.0% 1.04 ± 4% time.system_time
3.19 ± 40% -85.6% 0.46 ± 94% -97.7% 0.08 ± 12% time.user_time
3230390 ± 31% -78.5% 693676 ±130% -99.1% 28507 ± 7% time.voluntary_context_switches
783.66 ± 35% -81.7% 143.04 ±120% -98.9% 8.64 ± 9% filebench.sum_bytes_mb/s
6019202 ± 35% -81.7% 1098559 ±120% -98.9% 66474 ± 9% filebench.sum_operations
100309 ± 35% -81.7% 18307 ±120% -98.9% 1107 ± 9% filebench.sum_operations/s
52305 ± 33% -79.6% 10648 ±127% -98.2% 918.80 ± 10% filebench.sum_reads/s
0.03 ±110% +457.7% 0.16 ± 43% +6256.3% 1.82 ± 8% filebench.sum_time_ms/op
48003 ± 38% -84.0% 7659 ±114% -99.6% 188.70 ± 9% filebench.sum_writes/s
242.09 +3.6% 250.91 ± 3% -12.1% 212.83 filebench.time.elapsed_time
242.09 +3.6% 250.91 ± 3% -12.1% 212.83 filebench.time.elapsed_time.max
20761124 ± 8% +37.8% 28604910 ± 20% +62.0% 33640448 ± 6% filebench.time.file_system_outputs
14207 -5.3% 13450 ± 2% -10.5% 12715 filebench.time.minor_page_faults
15.30 ± 34% -87.6% 1.90 ±129% -100.0% 0.00 filebench.time.percent_of_cpu_this_job_got
34.90 ± 32% -83.5% 5.75 ±101% -97.0% 1.04 ± 4% filebench.time.system_time
3230390 ± 31% -78.5% 693676 ±130% -99.1% 28507 ± 7% filebench.time.voluntary_context_switches
5353 ± 9% -20.2% 4270 ± 12% -16.2% 4485 ± 10% sched_debug.cfs_rq:/.avg_vruntime.avg
5353 ± 9% -20.2% 4270 ± 12% -16.2% 4485 ± 10% sched_debug.cfs_rq:/.min_vruntime.avg
180949 ± 7% -0.2% 180620 ± 11% -13.2% 157009 ± 8% sched_debug.cpu.clock.avg
180963 ± 7% -0.2% 180632 ± 11% -13.2% 157021 ± 8% sched_debug.cpu.clock.max
180930 ± 7% -0.2% 180600 ± 11% -13.2% 156990 ± 8% sched_debug.cpu.clock.min
180778 ± 7% -0.2% 180445 ± 11% -13.2% 156850 ± 8% sched_debug.cpu.clock_task.avg
180934 ± 7% -0.2% 180601 ± 11% -13.2% 156995 ± 8% sched_debug.cpu.clock_task.max
172070 ± 8% -0.1% 171841 ± 11% -13.8% 148376 ± 8% sched_debug.cpu.clock_task.min
0.00 ± 23% -41.8% 0.00 ± 45% +17.4% 0.00 ± 63% sched_debug.cpu.next_balance.stddev
43241 ± 34% -21.6% 33913 ± 38% -90.6% 4053 ± 8% sched_debug.cpu.nr_switches.avg
1467016 ± 34% -69.4% 449490 ± 66% -97.8% 32538 ± 24% sched_debug.cpu.nr_switches.max
631.54 ± 11% +2.6% 647.79 ± 21% -16.9% 524.85 ± 11% sched_debug.cpu.nr_switches.min
164469 ± 29% -55.5% 73162 ± 41% -97.0% 4919 ± 10% sched_debug.cpu.nr_switches.stddev
180950 ± 7% -0.2% 180619 ± 11% -13.2% 157008 ± 8% sched_debug.cpu_clk
180224 ± 7% -0.2% 179894 ± 11% -13.3% 156282 ± 8% sched_debug.ktime
181759 ± 7% -0.2% 181428 ± 10% -13.2% 157819 ± 8% sched_debug.sched_clk
7299 ± 21% -5.9% 6871 ± 19% -28.0% 5256 ± 4% proc-vmstat.nr_active_anon
372843 ± 15% -57.5% 158372 ± 38% -50.0% 186326 ± 6% proc-vmstat.nr_dirty
13951 +0.9% 14083 -3.9% 13402 proc-vmstat.nr_mapped
16536 ± 9% -1.7% 16261 ± 7% -16.1% 13866 proc-vmstat.nr_shmem
63932 -3.4% 61741 -4.0% 61376 proc-vmstat.nr_slab_reclaimable
98280 -20.6% 78021 ± 5% -23.4% 75267 proc-vmstat.nr_slab_unreclaimable
7299 ± 21% -5.9% 6871 ± 19% -28.0% 5256 ± 4% proc-vmstat.nr_zone_active_anon
386619 ± 13% -4.7% 368327 ± 17% -49.3% 196170 ± 5% proc-vmstat.nr_zone_write_pending
2042 ± 61% -57.5% 867.90 ± 29% -92.2% 159.80 ± 30% proc-vmstat.numa_hint_faults
1503 ± 17% -56.3% 657.60 ± 30% -96.9% 45.90 ±102% proc-vmstat.numa_hint_faults_local
5231737 ± 35% -19.4% 4217786 ± 37% -55.8% 2311589 proc-vmstat.numa_hit
5099157 ± 36% -19.9% 4085262 ± 38% -57.3% 2179161 proc-vmstat.numa_local
16280 ±124% -74.2% 4194 ± 33% -96.0% 655.70 ± 47% proc-vmstat.numa_pte_updates
1160800 ± 27% +7.3% 1245889 ± 15% +13.4% 1315967 proc-vmstat.pgactivate
7695175 ± 24% +15.6% 8896125 ± 13% -36.3% 4899322 proc-vmstat.pgalloc_normal
750565 +2.6% 770119 ± 2% -9.3% 681069 proc-vmstat.pgfault
7645166 ± 24% -1.3% 7543115 ± 16% -53.5% 3557195 proc-vmstat.pgfree
10381756 ± 8% +37.8% 14303106 ± 20% -48.5% 5347677 proc-vmstat.pgpgout
36983 +2.6% 37938 ± 2% -8.4% 33875 proc-vmstat.pgreuse
2.71 ± 14% +8.7% 2.94 ± 16% -29.6% 1.91 ± 3% perf-stat.i.MPKI
1.967e+08 ± 11% -30.7% 1.363e+08 ± 20% -51.4% 95541508 perf-stat.i.branch-instructions
3.27 ± 5% -0.1 3.19 ± 5% +0.9 4.17 perf-stat.i.branch-miss-rate%
3983754 ± 4% -9.1% 3620399 ± 10% -4.8% 3791850 perf-stat.i.branch-misses
7.17 ± 16% -0.7 6.45 ± 13% -3.8 3.39 ± 3% perf-stat.i.cache-miss-rate%
5001882 ± 21% -41.6% 2921780 ± 17% -76.1% 1193694 ± 2% perf-stat.i.cache-misses
25150802 ± 10% -25.8% 18654089 ± 17% -48.8% 12876923 perf-stat.i.cache-references
36959 ± 29% -12.9% 32206 ± 32% -91.7% 3055 ± 3% perf-stat.i.context-switches
1.91 ± 2% +4.2% 1.99 ± 3% +5.1% 2.01 ± 2% perf-stat.i.cpi
1.61e+09 ± 12% -28.4% 1.152e+09 ± 14% -57.7% 6.811e+08 perf-stat.i.cpu-cycles
184.09 ± 6% -16.1% 154.40 ± 13% -23.2% 141.33 perf-stat.i.cpu-migrations
9.054e+08 ± 11% -27.9% 6.526e+08 ± 19% -48.2% 4.688e+08 perf-stat.i.instructions
0.57 ± 2% -3.8% 0.55 ± 2% -3.7% 0.55 perf-stat.i.ipc
0.25 ± 37% -24.6% 0.19 ± 45% -100.0% 0.00 perf-stat.i.metric.K/sec
5.50 ± 18% -17.4% 4.54 ± 11% -53.4% 2.56 ± 2% perf-stat.overall.MPKI
2.06 ± 15% +0.6 2.70 ± 8% +1.9 3.96 perf-stat.overall.branch-miss-rate%
19.72 ± 17% -4.0 15.76 ± 8% -10.4 9.31 ± 2% perf-stat.overall.cache-miss-rate%
1.78 ± 7% +0.2% 1.78 ± 5% -18.2% 1.46 perf-stat.overall.cpi
333.30 ± 16% +19.0% 396.63 ± 9% +70.8% 569.21 ± 3% perf-stat.overall.cycles-between-cache-misses
0.57 ± 7% -0.4% 0.56 ± 5% +21.6% 0.69 perf-stat.overall.ipc
1.96e+08 ± 11% -30.6% 1.359e+08 ± 19% -51.4% 95209820 perf-stat.ps.branch-instructions
3964680 ± 4% -9.1% 3604521 ± 10% -4.9% 3770958 perf-stat.ps.branch-misses
4984920 ± 21% -41.4% 2920752 ± 17% -76.0% 1195796 ± 2% perf-stat.ps.cache-misses
25058655 ± 10% -25.7% 18617868 ± 17% -48.7% 12845143 perf-stat.ps.cache-references
36798 ± 29% -12.8% 32082 ± 32% -91.7% 3044 ± 3% perf-stat.ps.context-switches
1.604e+09 ± 12% -28.3% 1.15e+09 ± 14% -57.6% 6.802e+08 perf-stat.ps.cpu-cycles
183.26 ± 6% -16.1% 153.73 ± 13% -23.3% 140.64 perf-stat.ps.cpu-migrations
9.019e+08 ± 11% -27.8% 6.509e+08 ± 19% -48.2% 4.672e+08 perf-stat.ps.instructions
2.196e+11 ± 12% -25.5% 1.636e+11 ± 18% -54.5% 9.985e+10 perf-stat.total.instructions
45.60 ± 18% -33.1 12.53 ± 83% -29.8 15.77 ± 21% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
44.60 ± 20% -32.7 11.92 ± 82% -30.5 14.13 ± 25% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
33.44 ± 30% -30.7 2.79 ±237% -33.3 0.17 ±154% perf-profile.calltrace.cycles-pp.rpc_async_release.process_one_work.worker_thread.kthread.ret_from_fork
33.44 ± 30% -30.6 2.79 ±238% -33.3 0.17 ±154% perf-profile.calltrace.cycles-pp.rpc_free_task.rpc_async_release.process_one_work.worker_thread.kthread
68.48 ± 14% -30.1 38.35 ± 30% -42.6 25.89 ± 11% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
68.48 ± 14% -30.1 38.35 ± 30% -42.6 25.89 ± 11% perf-profile.calltrace.cycles-pp.ret_from_fork_asm
68.48 ± 14% -30.1 38.35 ± 30% -42.6 25.89 ± 11% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
23.26 ± 33% -21.3 1.96 ±241% -23.2 0.06 ±299% perf-profile.calltrace.cycles-pp.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work.worker_thread
12.96 ± 45% -12.0 0.92 ±274% -13.0 0.00 perf-profile.calltrace.cycles-pp.nfs_page_end_writeback.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
12.46 ± 46% -11.6 0.87 ±276% -12.5 0.00 perf-profile.calltrace.cycles-pp.folio_end_writeback.nfs_page_end_writeback.nfs_write_completion.rpc_free_task.rpc_async_release
12.14 ± 47% -11.3 0.84 ±277% -12.1 0.00 perf-profile.calltrace.cycles-pp.__folio_end_writeback.folio_end_writeback.nfs_page_end_writeback.nfs_write_completion.rpc_free_task
10.63 ± 51% -10.0 0.68 ±300% -10.6 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__folio_end_writeback.folio_end_writeback.nfs_page_end_writeback.nfs_write_completion
10.28 ± 51% -9.6 0.66 ±300% -10.3 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__folio_end_writeback.folio_end_writeback.nfs_page_end_writeback
10.17 ± 30% -9.4 0.82 ±230% -10.2 0.00 perf-profile.calltrace.cycles-pp.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work.worker_thread
10.16 ± 30% -9.3 0.82 ±230% -10.2 0.00 perf-profile.calltrace.cycles-pp.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_async_release.process_one_work
9.26 ± 34% -8.4 0.91 ±223% -9.3 0.00 perf-profile.calltrace.cycles-pp.nfs_request_add_commit_list.nfs_write_completion.rpc_free_task.rpc_async_release.process_one_work
8.16 ± 32% -7.5 0.61 ±238% -8.2 0.00 perf-profile.calltrace.cycles-pp.nfs_inode_remove_request.nfs_commit_release_pages.nfs_commit_release.rpc_free_task.rpc_async_release
6.91 ± 33% -6.4 0.51 ±244% -6.9 0.00 perf-profile.calltrace.cycles-pp._raw_spin_lock.nfs_inode_remove_request.nfs_commit_release_pages.nfs_commit_release.rpc_free_task
6.83 ± 39% -6.2 0.67 ±223% -6.8 0.00 perf-profile.calltrace.cycles-pp.__mutex_lock.nfs_request_add_commit_list.nfs_write_completion.rpc_free_task.rpc_async_release
6.37 ± 33% -5.9 0.47 ±246% -6.4 0.00 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.nfs_inode_remove_request.nfs_commit_release_pages.nfs_commit_release
6.01 ± 41% -5.4 0.59 ±225% -6.0 0.00 perf-profile.calltrace.cycles-pp.osq_lock.__mutex_lock.nfs_request_add_commit_list.nfs_write_completion.rpc_free_task
3.19 ± 21% -2.6 0.55 ±225% -2.6 0.63 ± 12% perf-profile.calltrace.cycles-pp.nfs_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
3.19 ± 21% -2.6 0.55 ±225% -2.6 0.63 ± 11% perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
3.19 ± 21% -2.6 0.55 ±225% -2.6 0.63 ± 12% perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_do_writeback.wb_workfn.process_one_work
3.19 ± 21% -2.6 0.55 ±225% -2.6 0.63 ± 12% perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_do_writeback
3.19 ± 21% -2.6 0.55 ±225% -2.6 0.63 ± 12% perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_do_writeback.wb_workfn
3.19 ± 21% -2.6 0.55 ±225% -2.6 0.64 ± 12% perf-profile.calltrace.cycles-pp.wb_writeback.wb_do_writeback.wb_workfn.process_one_work.worker_thread
3.19 ± 21% -2.6 0.55 ±225% -2.6 0.64 ± 12% perf-profile.calltrace.cycles-pp.wb_do_writeback.wb_workfn.process_one_work.worker_thread.kthread
3.19 ± 21% -2.6 0.55 ±225% -2.6 0.64 ± 12% perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
3.18 ± 21% -2.6 0.55 ±225% -2.6 0.56 ± 35% perf-profile.calltrace.cycles-pp.write_cache_pages.nfs_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.91 ± 13% perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file.nfsd_commit.nfsd4_commit
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.90 ± 13% perf-profile.calltrace.cycles-pp.ext4_do_writepages.ext4_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.91 ± 13% perf-profile.calltrace.cycles-pp.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.91 ± 13% perf-profile.calltrace.cycles-pp.ext4_writepages.do_writepages.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.91 ± 13% perf-profile.calltrace.cycles-pp.filemap_fdatawrite_wbc.__filemap_fdatawrite_range.file_write_and_wait_range.ext4_sync_file.nfsd_commit
5.42 ± 13% -2.4 2.97 ± 44% -3.8 1.58 ± 9% perf-profile.calltrace.cycles-pp.mpage_prepare_extent_to_map.ext4_do_writepages.ext4_writepages.do_writepages.filemap_fdatawrite_wbc
2.60 ± 23% -2.2 0.43 ±226% -2.1 0.48 ± 51% perf-profile.calltrace.cycles-pp.nfs_writepages_callback.write_cache_pages.nfs_writepages.do_writepages.__writeback_single_inode
5.85 ± 58% -1.9 3.91 ± 80% +12.1 17.92 ± 7% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
1.48 ± 23% -1.2 0.32 ±201% -1.2 0.33 ±100% perf-profile.calltrace.cycles-pp.folio_wait_bit_common.mpage_prepare_extent_to_map.ext4_do_writepages.ext4_writepages.do_writepages
2.49 ± 13% -0.6 1.91 ± 24% -1.8 0.65 ± 19% perf-profile.calltrace.cycles-pp.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_do_writepages.ext4_writepages.do_writepages
0.61 ± 11% -0.5 0.12 ±200% +1.1 1.69 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
0.61 ± 11% -0.5 0.12 ±200% +1.1 1.69 ± 8% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
0.62 ± 12% -0.4 0.17 ±153% +1.2 1.77 ± 10% perf-profile.calltrace.cycles-pp.read
0.54 ± 35% -0.4 0.11 ±200% +1.0 1.57 ± 10% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
0.54 ± 35% -0.4 0.11 ±200% +1.1 1.61 ± 9% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
1.75 ± 15% -0.2 1.51 ± 25% -1.3 0.43 ± 68% perf-profile.calltrace.cycles-pp.mpage_submit_folio.mpage_process_page_bufs.mpage_prepare_extent_to_map.ext4_do_writepages.ext4_writepages
0.63 ± 35% -0.2 0.40 ± 84% +3.7 4.34 ± 12% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.47 ± 50% -0.2 0.26 ±123% +3.2 3.69 ± 12% perf-profile.calltrace.cycles-pp.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.60 ± 15% -0.2 0.39 ± 83% +2.1 2.68 ± 15% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
0.60 ± 15% -0.2 0.39 ± 83% +2.1 2.69 ± 15% perf-profile.calltrace.cycles-pp.execve
0.60 ± 15% -0.2 0.39 ± 83% +2.1 2.69 ± 14% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
0.60 ± 15% -0.2 0.39 ± 83% +2.1 2.69 ± 14% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve
2.07 ± 16% -0.2 1.89 ± 23% +9.6 11.68 ± 12% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.86 ± 13% -0.2 0.68 ± 41% +4.6 5.43 ± 10% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
1.94 ± 14% -0.2 1.78 ± 24% +8.6 10.49 ± 10% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
0.55 ± 36% -0.2 0.39 ± 83% +2.1 2.66 ± 15% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve
0.87 ± 22% -0.1 0.73 ± 17% +2.7 3.56 ± 11% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.87 ± 22% -0.1 0.74 ± 18% +2.7 3.58 ± 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
0.88 ± 13% -0.1 0.75 ± 24% +4.7 5.57 ± 9% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.97 ± 42% -0.1 0.85 ± 33% +2.0 2.93 ± 14% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
4.96 ± 13% -0.1 4.85 ± 15% -3.4 1.61 ± 14% perf-profile.calltrace.cycles-pp.xs_tcp_send_request.xprt_request_transmit.xprt_transmit.call_transmit.__rpc_execute
4.47 ± 18% -0.1 4.36 ± 18% -3.8 0.69 ± 18% perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.xprt_sock_sendmsg.xs_tcp_send_request
0.11 ±200% -0.1 0.00 +1.5 1.56 ± 8% perf-profile.calltrace.cycles-pp.asm_exc_page_fault
5.84 ± 19% -0.1 5.74 ± 34% -1.7 4.10 ± 10% perf-profile.calltrace.cycles-pp.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread.kthread
4.83 ± 16% -0.1 4.73 ± 17% -3.9 0.90 ± 18% perf-profile.calltrace.cycles-pp.tcp_sendmsg.sock_sendmsg.xprt_sock_sendmsg.xs_tcp_send_request.xprt_request_transmit
4.84 ± 16% -0.1 4.74 ± 17% -3.9 0.94 ± 17% perf-profile.calltrace.cycles-pp.sock_sendmsg.xprt_sock_sendmsg.xs_tcp_send_request.xprt_request_transmit.xprt_transmit
4.85 ± 16% -0.1 4.75 ± 17% -3.9 0.95 ± 17% perf-profile.calltrace.cycles-pp.xprt_sock_sendmsg.xs_tcp_send_request.xprt_request_transmit.xprt_transmit.call_transmit
5.09 ± 11% -0.1 4.99 ± 15% -3.1 2.00 ± 12% perf-profile.calltrace.cycles-pp.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work.worker_thread
2.85 ± 13% -0.1 2.76 ± 21% -2.0 0.85 ± 17% perf-profile.calltrace.cycles-pp.svc_tcp_recvfrom.svc_handle_xprt.svc_recv.nfsd.kthread
5.85 ± 19% -0.1 5.75 ± 34% -1.7 4.13 ± 10% perf-profile.calltrace.cycles-pp.rpc_async_schedule.process_one_work.worker_thread.kthread.ret_from_fork
5.05 ± 12% -0.1 4.96 ± 15% -3.2 1.88 ± 12% perf-profile.calltrace.cycles-pp.xprt_request_transmit.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule
5.06 ± 11% -0.1 4.97 ± 15% -3.1 1.94 ± 12% perf-profile.calltrace.cycles-pp.xprt_transmit.call_transmit.__rpc_execute.rpc_async_schedule.process_one_work
0.05 ±300% -0.1 0.00 +1.5 1.50 ± 16% perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve
0.05 ±300% -0.1 0.00 +1.5 1.51 ± 16% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64
0.05 ±299% -0.1 0.00 +1.4 1.45 ± 16% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common
0.69 ± 42% -0.0 0.66 ± 46% +2.2 2.92 ± 15% perf-profile.calltrace.cycles-pp.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.08 ±300% -0.0 0.06 ±300% +1.0 1.10 ± 16% perf-profile.calltrace.cycles-pp.tick_nohz_get_sleep_length.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry
0.00 +0.0 0.00 +0.7 0.65 ± 16% perf-profile.calltrace.cycles-pp.__x64_sys_vfork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
0.00 +0.0 0.00 +0.7 0.65 ± 15% perf-profile.calltrace.cycles-pp.kernel_clone.__x64_sys_vfork.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
0.00 +0.0 0.00 +0.7 0.66 ± 15% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__vfork
0.00 +0.0 0.00 +0.7 0.66 ± 15% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__vfork
0.00 +0.0 0.00 +0.7 0.72 ± 13% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
0.00 +0.0 0.00 +0.7 0.74 ± 13% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +0.7 0.74 ± 13% perf-profile.calltrace.cycles-pp.__vfork
0.00 +0.0 0.00 +0.8 0.85 ± 9% perf-profile.calltrace.cycles-pp.sched_balance_domains.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +0.0 0.00 +0.9 0.88 ± 11% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +0.9 0.88 ± 11% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +0.9 0.92 ± 19% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sched_setaffinity.evlist_cpu_iterator__next.read_counters
0.00 +0.0 0.00 +0.9 0.92 ± 16% perf-profile.calltrace.cycles-pp.setlocale
0.00 +0.0 0.00 +0.9 0.92 ± 18% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sched_setaffinity.evlist_cpu_iterator__next.read_counters.process_interval
0.00 +0.0 0.00 +0.9 0.94 ± 19% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.balance_fair
0.00 +0.0 0.00 +0.9 0.95 ± 19% perf-profile.calltrace.cycles-pp.sched_balance_find_src_group.sched_balance_rq.sched_balance_newidle.balance_fair.__schedule
0.00 +0.0 0.00 +1.0 0.96 ± 21% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +1.0 0.96 ± 21% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +1.0 0.96 ± 21% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.0 0.00 +1.0 0.99 ± 19% perf-profile.calltrace.cycles-pp.sched_setaffinity.evlist_cpu_iterator__next.read_counters.process_interval.dispatch_events
0.00 +0.0 0.00 +1.0 0.99 ± 20% perf-profile.calltrace.cycles-pp.sched_balance_rq.sched_balance_newidle.balance_fair.__schedule.schedule
0.00 +0.0 0.00 +1.1 1.06 ± 19% perf-profile.calltrace.cycles-pp.balance_fair.__schedule.schedule.smpboot_thread_fn.kthread
0.00 +0.0 0.00 +1.1 1.06 ± 19% perf-profile.calltrace.cycles-pp.sched_balance_newidle.balance_fair.__schedule.schedule.smpboot_thread_fn
0.00 +0.0 0.00 +1.1 1.06 ± 21% perf-profile.calltrace.cycles-pp.tick_irq_enter.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
0.00 +0.0 0.00 +1.1 1.12 ± 21% perf-profile.calltrace.cycles-pp.irq_enter_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.00 +0.0 0.00 +1.1 1.15 ± 11% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
0.00 +0.0 0.00 +1.2 1.19 ± 18% perf-profile.calltrace.cycles-pp.evlist_cpu_iterator__next.read_counters.process_interval.dispatch_events.cmd_stat
0.00 +0.0 0.00 +1.2 1.19 ± 12% perf-profile.calltrace.cycles-pp.sched_tick.update_process_times.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt
0.00 +0.0 0.00 +1.2 1.20 ± 8% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
0.00 +0.0 0.00 +1.3 1.28 ± 16% perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
0.00 +0.0 0.00 +1.3 1.28 ± 17% perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
0.00 +0.0 0.00 +1.3 1.32 ± 9% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
0.00 +0.0 0.00 +1.3 1.34 ± 9% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
0.06 ±300% +0.0 0.06 ±299% +1.7 1.78 ± 12% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.10 ±200% +0.0 0.11 ±200% +2.6 2.70 ± 11% perf-profile.calltrace.cycles-pp.update_process_times.tick_nohz_handler.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
0.06 ±300% +0.0 0.08 ±299% +0.7 0.73 ± 22% perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.72 ± 42% +0.0 0.74 ± 32% +2.4 3.13 ± 15% perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
0.09 ±299% +0.0 0.12 ±300% +0.9 0.96 ± 25% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irq.process_one_work.worker_thread.kthread
0.10 ±299% +0.0 0.14 ±299% +0.9 1.05 ± 24% perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.process_one_work.worker_thread.kthread.ret_from_fork
0.06 ±300% +0.2 0.31 ±101% +1.6 1.66 ± 15% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
4.56 ± 19% +0.3 4.87 ± 31% -4.2 0.40 ± 83% perf-profile.calltrace.cycles-pp.generic_perform_write.ext4_buffered_write_iter.do_iter_readv_writev.vfs_iter_write.nfsd_vfs_write
0.45 ± 72% +0.7 1.13 ± 48% +0.9 1.30 ± 16% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.rest_init.start_kernel.x86_64_start_reservations
0.45 ± 72% +0.7 1.13 ± 48% +0.8 1.30 ± 16% perf-profile.calltrace.cycles-pp.cpu_startup_entry.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel
0.45 ± 72% +0.7 1.13 ± 48% +0.8 1.30 ± 16% perf-profile.calltrace.cycles-pp.rest_init.start_kernel.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
0.45 ± 72% +0.7 1.13 ± 48% +0.8 1.30 ± 16% perf-profile.calltrace.cycles-pp.start_kernel.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
0.45 ± 72% +0.7 1.13 ± 48% +0.8 1.30 ± 16% perf-profile.calltrace.cycles-pp.x86_64_start_kernel.common_startup_64
0.45 ± 72% +0.7 1.13 ± 48% +0.8 1.30 ± 16% perf-profile.calltrace.cycles-pp.x86_64_start_reservations.x86_64_start_kernel.common_startup_64
0.05 ±300% +0.8 0.81 ± 52% +0.8 0.86 ± 22% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init
0.11 ±200% +0.8 0.94 ± 39% +0.9 1.05 ± 17% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.rest_init.start_kernel
0.21 ±154% +1.2 1.42 ± 46% +0.5 0.66 ± 36% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle
7.98 ± 70% +1.2 9.23 ± 79% -8.0 0.00 perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.ext4_buffered_write_iter.do_iter_readv_writev
8.00 ± 70% +1.2 9.24 ± 79% -8.0 0.00 perf-profile.calltrace.cycles-pp.down_write.ext4_buffered_write_iter.do_iter_readv_writev.vfs_iter_write.nfsd_vfs_write
7.99 ± 70% +1.2 9.24 ± 79% -8.0 0.00 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.ext4_buffered_write_iter.do_iter_readv_writev.vfs_iter_write
5.57 ± 13% +1.3 6.89 ± 39% -3.6 1.98 ± 13% perf-profile.calltrace.cycles-pp.file_write_and_wait_range.ext4_sync_file.nfsd_commit.nfsd4_commit.nfsd4_proc_compound
5.57 ± 13% +1.3 6.90 ± 39% -3.5 2.04 ± 13% perf-profile.calltrace.cycles-pp.ext4_sync_file.nfsd_commit.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch
5.57 ± 13% +1.3 6.90 ± 39% -3.5 2.04 ± 13% perf-profile.calltrace.cycles-pp.nfsd_commit.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
5.58 ± 13% +1.3 6.90 ± 39% -3.4 2.14 ± 12% perf-profile.calltrace.cycles-pp.nfsd4_commit.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
12.70 ± 49% +1.6 14.28 ± 60% -11.7 0.98 ± 23% perf-profile.calltrace.cycles-pp.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process
12.64 ± 49% +1.6 14.25 ± 61% -11.8 0.88 ± 25% perf-profile.calltrace.cycles-pp.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch.svc_process_common
12.61 ± 49% +1.6 14.22 ± 61% -11.8 0.85 ± 27% perf-profile.calltrace.cycles-pp.vfs_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound.nfsd_dispatch
12.60 ± 49% +1.6 14.21 ± 61% -11.8 0.82 ± 26% perf-profile.calltrace.cycles-pp.ext4_buffered_write_iter.do_iter_readv_writev.vfs_iter_write.nfsd_vfs_write.nfsd4_write
12.60 ± 49% +1.6 14.22 ± 61% -11.8 0.82 ± 27% perf-profile.calltrace.cycles-pp.do_iter_readv_writev.vfs_iter_write.nfsd_vfs_write.nfsd4_write.nfsd4_proc_compound
0.53 ± 74% +1.7 2.20 ± 39% +0.5 1.06 ± 20% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
0.82 ± 51% +2.1 2.88 ± 37% +0.5 1.32 ± 16% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
22.26 ± 25% +2.9 25.13 ± 38% -15.5 6.73 ± 10% perf-profile.calltrace.cycles-pp.nfsd.kthread.ret_from_fork.ret_from_fork_asm
22.24 ± 25% +2.9 25.12 ± 38% -15.5 6.72 ± 10% perf-profile.calltrace.cycles-pp.svc_recv.nfsd.kthread.ret_from_fork.ret_from_fork_asm
22.07 ± 25% +2.9 24.98 ± 38% -15.6 6.42 ± 11% perf-profile.calltrace.cycles-pp.svc_handle_xprt.svc_recv.nfsd.kthread.ret_from_fork
18.49 ± 32% +2.9 21.44 ± 45% -14.8 3.73 ± 14% perf-profile.calltrace.cycles-pp.nfsd4_proc_compound.nfsd_dispatch.svc_process_common.svc_process.svc_handle_xprt
18.59 ± 32% +2.9 21.53 ± 44% -14.7 3.88 ± 14% perf-profile.calltrace.cycles-pp.nfsd_dispatch.svc_process_common.svc_process.svc_handle_xprt.svc_recv
18.68 ± 32% +3.0 21.63 ± 44% -14.6 4.06 ± 13% perf-profile.calltrace.cycles-pp.svc_process.svc_handle_xprt.svc_recv.nfsd.kthread
18.67 ± 32% +3.0 21.63 ± 44% -14.6 4.06 ± 13% perf-profile.calltrace.cycles-pp.svc_process_common.svc_process.svc_handle_xprt.svc_recv.nfsd
1.77 ± 72% +3.0 4.76 ± 46% +0.5 2.22 ± 20% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.common_startup_64
0.86 ± 44% +7.5 8.36 ± 61% +2.3 3.20 ± 15% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
15.15 ± 42% +8.5 23.68 ± 18% +27.5 42.63 ± 7% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
13.02 ± 41% +9.0 22.02 ± 18% +24.6 37.65 ± 6% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
12.84 ± 40% +9.8 22.60 ± 18% +24.7 37.57 ± 6% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
19.12 ± 44% +12.3 31.47 ± 14% +29.8 48.88 ± 7% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
19.15 ± 44% +12.4 31.50 ± 14% +29.8 48.92 ± 7% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
19.16 ± 44% +12.4 31.53 ± 14% +29.8 48.92 ± 7% perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
19.74 ± 43% +12.9 32.66 ± 14% +30.5 50.23 ± 6% perf-profile.calltrace.cycles-pp.common_startup_64
45.60 ± 18% -33.1 12.54 ± 83% -29.8 15.77 ± 21% perf-profile.children.cycles-pp.worker_thread
44.60 ± 20% -32.7 11.92 ± 82% -30.5 14.14 ± 25% perf-profile.children.cycles-pp.process_one_work
33.44 ± 30% -30.6 2.83 ±234% -33.0 0.47 ± 17% perf-profile.children.cycles-pp.rpc_async_release
33.44 ± 30% -30.6 2.83 ±234% -33.0 0.47 ± 17% perf-profile.children.cycles-pp.rpc_free_task
68.51 ± 14% -30.1 38.38 ± 30% -42.5 26.01 ± 11% perf-profile.children.cycles-pp.ret_from_fork_asm
68.48 ± 14% -30.1 38.35 ± 30% -42.6 25.89 ± 11% perf-profile.children.cycles-pp.kthread
68.50 ± 14% -30.1 38.37 ± 30% -42.5 26.00 ± 11% perf-profile.children.cycles-pp.ret_from_fork
23.26 ± 33% -21.3 1.98 ±239% -22.9 0.39 ± 19% perf-profile.children.cycles-pp.nfs_write_completion
18.49 ± 33% -15.3 3.15 ±105% -15.3 3.22 ± 19% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
12.97 ± 45% -12.0 0.92 ±273% -12.8 0.16 ± 39% perf-profile.children.cycles-pp.nfs_page_end_writeback
12.93 ± 44% -11.5 1.46 ±162% -12.6 0.31 ± 29% perf-profile.children.cycles-pp.__folio_end_writeback
13.28 ± 43% -10.6 2.68 ± 88% -12.9 0.36 ± 25% perf-profile.children.cycles-pp.folio_end_writeback
12.41 ± 43% -10.3 2.06 ±108% -11.4 0.99 ± 14% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
10.17 ± 30% -9.4 0.82 ±230% -10.1 0.07 ± 41% perf-profile.children.cycles-pp.nfs_commit_release
10.17 ± 30% -9.4 0.82 ±230% -10.1 0.07 ± 41% perf-profile.children.cycles-pp.nfs_commit_release_pages
9.27 ± 34% -8.4 0.91 ±223% -9.2 0.08 ± 67% perf-profile.children.cycles-pp.nfs_request_add_commit_list
8.16 ± 32% -7.5 0.61 ±238% -8.1 0.01 ±203% perf-profile.children.cycles-pp.nfs_inode_remove_request
6.92 ± 38% -6.1 0.77 ±205% -6.6 0.33 ± 35% perf-profile.children.cycles-pp.__mutex_lock
8.24 ± 20% -5.6 2.68 ± 58% -4.8 3.41 ± 16% perf-profile.children.cycles-pp._raw_spin_lock
8.76 ± 11% -5.0 3.80 ± 62% -6.2 2.54 ± 12% perf-profile.children.cycles-pp.do_writepages
11.18 ± 47% -4.5 6.63 ± 80% -11.1 0.10 ± 49% perf-profile.children.cycles-pp.osq_lock
3.19 ± 21% -2.5 0.70 ±169% -2.6 0.63 ± 12% perf-profile.children.cycles-pp.nfs_writepages
3.19 ± 21% -2.5 0.70 ±169% -2.6 0.63 ± 12% perf-profile.children.cycles-pp.__writeback_inodes_wb
3.19 ± 21% -2.5 0.70 ±169% -2.6 0.63 ± 12% perf-profile.children.cycles-pp.__writeback_single_inode
3.19 ± 21% -2.5 0.70 ±169% -2.6 0.63 ± 12% perf-profile.children.cycles-pp.writeback_sb_inodes
3.19 ± 21% -2.5 0.70 ±169% -2.6 0.64 ± 12% perf-profile.children.cycles-pp.wb_writeback
3.19 ± 21% -2.5 0.70 ±169% -2.6 0.64 ± 12% perf-profile.children.cycles-pp.wb_do_writeback
3.19 ± 21% -2.5 0.70 ±169% -2.6 0.64 ± 12% perf-profile.children.cycles-pp.wb_workfn
3.19 ± 21% -2.5 0.70 ±169% -2.6 0.61 ± 12% perf-profile.children.cycles-pp.write_cache_pages
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.91 ± 13% perf-profile.children.cycles-pp.__filemap_fdatawrite_range
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.90 ± 13% perf-profile.children.cycles-pp.ext4_do_writepages
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.91 ± 13% perf-profile.children.cycles-pp.ext4_writepages
5.57 ± 13% -2.5 3.10 ± 43% -3.7 1.91 ± 13% perf-profile.children.cycles-pp.filemap_fdatawrite_wbc
5.43 ± 13% -2.5 2.98 ± 44% -3.9 1.58 ± 9% perf-profile.children.cycles-pp.mpage_prepare_extent_to_map
2.61 ± 23% -2.0 0.57 ±163% -2.0 0.57 ± 15% perf-profile.children.cycles-pp.nfs_writepages_callback
2.46 ± 28% -2.0 0.48 ±169% -2.3 0.15 ± 46% perf-profile.children.cycles-pp.nfs_page_async_flush
6.00 ± 56% -1.9 4.12 ± 77% +12.2 18.16 ± 6% perf-profile.children.cycles-pp.intel_idle
1.44 ± 29% -1.3 0.16 ±221% -1.3 0.13 ± 40% perf-profile.children.cycles-pp.mutex_lock
1.59 ± 29% -0.8 0.80 ± 37% -1.5 0.13 ± 52% perf-profile.children.cycles-pp.__folio_start_writeback
0.72 ± 19% -0.6 0.13 ±215% -0.5 0.22 ± 50% perf-profile.children.cycles-pp.mutex_spin_on_owner
2.50 ± 12% -0.6 1.92 ± 24% -1.9 0.65 ± 19% perf-profile.children.cycles-pp.mpage_process_page_bufs
0.59 ± 19% -0.5 0.06 ±210% -0.5 0.12 ± 49% perf-profile.children.cycles-pp.__nfs_commit_inode
0.59 ± 19% -0.5 0.07 ±205% -0.6 0.03 ±103% perf-profile.children.cycles-pp.nfs_io_completion_put
0.74 ± 12% -0.3 0.41 ± 18% -0.5 0.22 ± 22% perf-profile.children.cycles-pp.__lruvec_stat_mod_folio
1.28 ± 35% -0.3 0.99 ± 38% -0.7 0.63 ± 14% perf-profile.children.cycles-pp.io_schedule
0.77 ± 13% -0.2 0.52 ± 23% +1.5 2.26 ± 5% perf-profile.children.cycles-pp.ksys_read
0.83 ± 12% -0.2 0.59 ± 23% +1.8 2.58 ± 5% perf-profile.children.cycles-pp.read
0.58 ± 5% -0.2 0.33 ± 34% -0.4 0.21 ± 23% perf-profile.children.cycles-pp.percpu_counter_add_batch
1.76 ± 15% -0.2 1.51 ± 25% -1.2 0.55 ± 24% perf-profile.children.cycles-pp.mpage_submit_folio
1.37 ± 19% -0.2 1.14 ± 30% -0.9 0.49 ± 26% perf-profile.children.cycles-pp.ext4_bio_write_folio
0.63 ± 13% -0.2 0.41 ± 30% -0.5 0.08 ± 80% perf-profile.children.cycles-pp.folio_clear_dirty_for_io
0.34 ± 14% -0.2 0.13 ± 71% -0.3 0.07 ± 71% perf-profile.children.cycles-pp.__xa_clear_mark
0.87 ± 28% -0.2 0.66 ± 26% -0.6 0.32 ± 35% perf-profile.children.cycles-pp.xas_load
0.46 ± 11% -0.2 0.27 ± 34% +0.6 1.04 ± 9% perf-profile.children.cycles-pp.seq_read_iter
0.21 ± 15% -0.2 0.04 ±213% -0.1 0.07 ± 62% perf-profile.children.cycles-pp.get_slabinfo
0.21 ± 15% -0.2 0.04 ±213% -0.2 0.03 ±106% perf-profile.children.cycles-pp.count_partial_free_approx
1.18 ± 11% -0.2 1.02 ± 20% +4.8 5.96 ± 9% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
2.67 ± 12% -0.2 2.51 ± 19% +9.7 12.36 ± 9% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.39 ± 45% -0.2 0.23 ± 65% +0.8 1.15 ± 18% perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
1.16 ± 11% -0.2 1.01 ± 20% +4.7 5.83 ± 9% perf-profile.children.cycles-pp.hrtimer_interrupt
0.31 ± 12% -0.1 0.16 ± 31% -0.1 0.17 ± 32% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
2.44 ± 11% -0.1 2.29 ± 20% +8.9 11.37 ± 10% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.94 ± 11% -0.1 0.80 ± 21% +3.7 4.65 ± 10% perf-profile.children.cycles-pp.__hrtimer_run_queues
1.02 ± 42% -0.1 0.90 ± 30% +2.0 3.02 ± 14% perf-profile.children.cycles-pp.menu_select
0.24 ± 18% -0.1 0.13 ± 21% -0.2 0.09 ± 49% perf-profile.children.cycles-pp.__mod_lruvec_state
0.25 ± 25% -0.1 0.14 ± 43% -0.2 0.08 ± 47% perf-profile.children.cycles-pp.nfs_pageio_add_request
4.96 ± 13% -0.1 4.86 ± 15% -3.4 1.61 ± 14% perf-profile.children.cycles-pp.xs_tcp_send_request
2.94 ± 11% -0.1 2.84 ± 22% -1.8 1.17 ± 17% perf-profile.children.cycles-pp.sock_recvmsg
2.93 ± 11% -0.1 2.82 ± 22% -1.8 1.11 ± 16% perf-profile.children.cycles-pp.inet6_recvmsg
0.64 ± 13% -0.1 0.54 ± 21% +2.3 2.97 ± 9% perf-profile.children.cycles-pp.update_process_times
0.80 ± 11% -0.1 0.70 ± 21% +3.2 3.98 ± 10% perf-profile.children.cycles-pp.tick_nohz_handler
2.92 ± 10% -0.1 2.82 ± 22% -1.8 1.11 ± 16% perf-profile.children.cycles-pp.tcp_recvmsg
5.85 ± 19% -0.1 5.75 ± 34% -1.7 4.13 ± 10% perf-profile.children.cycles-pp.rpc_async_schedule
2.85 ± 13% -0.1 2.76 ± 21% -2.0 0.85 ± 16% perf-profile.children.cycles-pp.svc_tcp_recvfrom
4.85 ± 16% -0.1 4.75 ± 17% -3.9 0.95 ± 17% perf-profile.children.cycles-pp.xprt_sock_sendmsg
0.26 ± 50% -0.1 0.17 ± 66% +0.6 0.84 ± 22% perf-profile.children.cycles-pp.tick_nohz_next_event
5.09 ± 11% -0.1 5.00 ± 15% -3.1 2.00 ± 12% perf-profile.children.cycles-pp.call_transmit
5.05 ± 12% -0.1 4.96 ± 15% -3.2 1.88 ± 12% perf-profile.children.cycles-pp.xprt_request_transmit
0.20 ± 17% -0.1 0.11 ± 20% -0.1 0.08 ± 62% perf-profile.children.cycles-pp.__mod_node_page_state
5.06 ± 11% -0.1 4.98 ± 15% -3.1 1.94 ± 12% perf-profile.children.cycles-pp.xprt_transmit
5.84 ± 19% -0.1 5.75 ± 34% -1.7 4.11 ± 10% perf-profile.children.cycles-pp.__rpc_execute
0.22 ± 26% -0.1 0.14 ± 43% -0.1 0.08 ± 47% perf-profile.children.cycles-pp.__nfs_pageio_add_request
0.25 ± 36% -0.1 0.17 ± 52% +0.5 0.72 ± 18% perf-profile.children.cycles-pp.clockevents_program_event
5.22 ± 11% -0.1 5.14 ± 15% -3.3 1.90 ± 10% perf-profile.children.cycles-pp.tcp_sendmsg
5.24 ± 11% -0.1 5.16 ± 15% -3.3 1.96 ± 9% perf-profile.children.cycles-pp.sock_sendmsg
2.69 ± 14% -0.1 2.61 ± 22% -2.0 0.68 ± 16% perf-profile.children.cycles-pp.svc_tcp_sock_recv_cmsg
4.78 ± 12% -0.1 4.71 ± 13% -3.2 1.55 ± 13% perf-profile.children.cycles-pp.tcp_sendmsg_locked
0.22 ± 19% -0.1 0.16 ± 41% +0.4 0.61 ± 28% perf-profile.children.cycles-pp.sched_balance_update_blocked_averages
0.12 ± 24% -0.1 0.06 ± 61% -0.1 0.04 ±111% perf-profile.children.cycles-pp.xas_clear_mark
2.59 ± 20% -0.1 2.53 ± 26% -2.2 0.37 ± 23% perf-profile.children.cycles-pp.svc_tcp_read_msg
0.15 ± 51% -0.1 0.09 ± 78% +0.2 0.37 ± 27% perf-profile.children.cycles-pp.lapic_next_deadline
0.27 ± 20% -0.1 0.21 ± 23% +0.8 1.10 ± 23% perf-profile.children.cycles-pp.__mmput
0.09 ± 49% -0.1 0.03 ±106% +0.2 0.29 ± 30% perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.08 ± 64% -0.1 0.03 ±157% +0.2 0.31 ± 15% perf-profile.children.cycles-pp.timerqueue_del
0.26 ± 21% -0.1 0.21 ± 23% +0.8 1.08 ± 23% perf-profile.children.cycles-pp.exit_mmap
0.12 ± 42% -0.1 0.06 ± 67% +0.2 0.30 ± 41% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.24 ± 21% -0.1 0.19 ± 28% +0.2 0.45 ± 21% perf-profile.children.cycles-pp.__cond_resched
0.13 ± 12% -0.1 0.08 ± 27% -0.0 0.10 ± 73% perf-profile.children.cycles-pp.tag_pages_for_writeback
0.15 ± 34% -0.0 0.10 ± 83% +0.4 0.53 ± 15% perf-profile.children.cycles-pp.native_irq_return_iret
0.41 ± 17% -0.0 0.36 ± 28% +1.4 1.80 ± 12% perf-profile.children.cycles-pp.bprm_execve
0.58 ± 19% -0.0 0.53 ± 17% +1.8 2.38 ± 7% perf-profile.children.cycles-pp.asm_exc_page_fault
0.31 ± 14% -0.0 0.26 ± 27% +1.0 1.29 ± 10% perf-profile.children.cycles-pp.sched_tick
0.58 ± 20% -0.0 0.53 ± 21% +1.9 2.51 ± 11% perf-profile.children.cycles-pp.cmd_stat
0.58 ± 20% -0.0 0.53 ± 21% +1.9 2.51 ± 11% perf-profile.children.cycles-pp.dispatch_events
0.28 ± 28% -0.0 0.23 ± 18% +0.9 1.14 ± 20% perf-profile.children.cycles-pp.do_exit
0.24 ± 29% -0.0 0.19 ± 18% +0.8 1.02 ± 19% perf-profile.children.cycles-pp.do_group_exit
0.58 ± 21% -0.0 0.53 ± 21% +1.9 2.50 ± 11% perf-profile.children.cycles-pp.process_interval
0.07 ± 61% -0.0 0.02 ±162% +0.1 0.19 ± 26% perf-profile.children.cycles-pp.tmigr_update_events
0.17 ± 38% -0.0 0.12 ± 21% +0.5 0.64 ± 16% perf-profile.children.cycles-pp.do_vmi_munmap
0.24 ± 29% -0.0 0.19 ± 19% +0.8 1.02 ± 19% perf-profile.children.cycles-pp.__x64_sys_exit_group
0.07 ± 45% -0.0 0.02 ±165% +0.2 0.23 ± 49% perf-profile.children.cycles-pp.enqueue_hrtimer
0.33 ± 20% -0.0 0.29 ± 28% +1.1 1.45 ± 16% perf-profile.children.cycles-pp.load_elf_binary
0.34 ± 20% -0.0 0.30 ± 30% +1.2 1.50 ± 16% perf-profile.children.cycles-pp.search_binary_handler
0.35 ± 20% -0.0 0.30 ± 30% +1.2 1.52 ± 16% perf-profile.children.cycles-pp.exec_binprm
0.25 ± 16% -0.0 0.21 ± 17% -0.2 0.06 ± 61% perf-profile.children.cycles-pp.folio_account_dirtied
0.56 ± 20% -0.0 0.52 ± 21% +1.9 2.43 ± 11% perf-profile.children.cycles-pp.read_counters
0.15 ± 41% -0.0 0.11 ± 19% +0.4 0.58 ± 19% perf-profile.children.cycles-pp.do_vmi_align_munmap
0.11 ± 9% -0.0 0.07 ± 62% +0.3 0.45 ± 20% perf-profile.children.cycles-pp.begin_new_exec
0.08 ± 29% -0.0 0.04 ± 90% +0.2 0.26 ± 31% perf-profile.children.cycles-pp.update_irq_load_avg
0.12 ± 38% -0.0 0.08 ± 77% +0.3 0.41 ± 24% perf-profile.children.cycles-pp.irqtime_account_irq
0.23 ± 28% -0.0 0.19 ± 20% +0.8 0.99 ± 16% perf-profile.children.cycles-pp.kernel_clone
0.05 ± 69% -0.0 0.01 ±300% +0.2 0.21 ± 48% perf-profile.children.cycles-pp.__vm_munmap
0.21 ± 15% -0.0 0.17 ± 29% +0.6 0.82 ± 25% perf-profile.children.cycles-pp.readn
0.60 ± 15% -0.0 0.56 ± 22% +2.1 2.67 ± 15% perf-profile.children.cycles-pp.do_execveat_common
0.60 ± 15% -0.0 0.56 ± 22% +2.1 2.69 ± 15% perf-profile.children.cycles-pp.execve
0.31 ± 27% -0.0 0.27 ± 22% +1.0 1.30 ± 10% perf-profile.children.cycles-pp.do_mmap
0.48 ± 18% -0.0 0.44 ± 17% +1.5 2.01 ± 9% perf-profile.children.cycles-pp.do_user_addr_fault
0.60 ± 15% -0.0 0.56 ± 22% +2.1 2.69 ± 14% perf-profile.children.cycles-pp.__x64_sys_execve
0.29 ± 22% -0.0 0.25 ± 44% +1.0 1.27 ± 19% perf-profile.children.cycles-pp.irq_enter_rcu
0.32 ± 27% -0.0 0.29 ± 21% +1.0 1.37 ± 10% perf-profile.children.cycles-pp.vm_mmap_pgoff
0.18 ± 16% -0.0 0.15 ± 22% -0.2 0.04 ±122% perf-profile.children.cycles-pp.folio_unlock
0.07 ± 81% -0.0 0.03 ±146% +0.2 0.29 ± 19% perf-profile.children.cycles-pp.blk_mq_run_hw_queue
0.49 ± 18% -0.0 0.45 ± 16% +1.5 2.03 ± 8% perf-profile.children.cycles-pp.exc_page_fault
0.20 ± 22% -0.0 0.17 ± 22% +0.6 0.84 ± 24% perf-profile.children.cycles-pp.exit_mm
0.24 ± 29% -0.0 0.20 ± 24% +0.7 0.94 ± 14% perf-profile.children.cycles-pp.ksys_mmap_pgoff
0.05 ± 78% -0.0 0.02 ±154% +0.2 0.21 ± 36% perf-profile.children.cycles-pp.rb_next
0.04 ± 71% -0.0 0.01 ±300% +0.2 0.24 ± 30% perf-profile.children.cycles-pp.__run_timers
0.28 ± 22% -0.0 0.25 ± 43% +1.0 1.23 ± 19% perf-profile.children.cycles-pp.tick_irq_enter
0.05 ± 85% -0.0 0.02 ±156% +0.2 0.23 ± 30% perf-profile.children.cycles-pp.sync_regs
0.19 ± 43% -0.0 0.16 ± 60% +0.4 0.64 ± 26% perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
0.11 ± 27% -0.0 0.08 ± 33% +0.3 0.45 ± 39% perf-profile.children.cycles-pp.zap_pte_range
0.11 ± 28% -0.0 0.08 ± 36% +0.4 0.46 ± 37% perf-profile.children.cycles-pp.zap_pmd_range
0.17 ± 28% -0.0 0.14 ± 25% -0.1 0.06 ± 90% perf-profile.children.cycles-pp.xas_start
0.12 ± 29% -0.0 0.09 ± 35% +0.4 0.48 ± 34% perf-profile.children.cycles-pp.unmap_page_range
0.12 ± 30% -0.0 0.09 ± 36% +0.4 0.53 ± 34% perf-profile.children.cycles-pp.unmap_vmas
0.07 ± 38% -0.0 0.04 ± 88% +0.2 0.26 ± 28% perf-profile.children.cycles-pp.arch_scale_freq_tick
0.03 ±100% -0.0 0.00 +0.2 0.20 ± 24% perf-profile.children.cycles-pp.__vmalloc_node_range_noprof
0.08 ± 28% -0.0 0.05 ± 36% +0.2 0.29 ± 23% perf-profile.children.cycles-pp.tlb_finish_mmu
0.09 ± 35% -0.0 0.06 ± 58% +0.3 0.35 ± 29% perf-profile.children.cycles-pp.pipe_read
0.35 ± 22% -0.0 0.32 ± 19% +1.2 1.51 ± 10% perf-profile.children.cycles-pp.do_sys_openat2
0.07 ± 45% -0.0 0.04 ± 68% +0.2 0.22 ± 21% perf-profile.children.cycles-pp.mas_store_prealloc
0.03 ±125% -0.0 0.00 +0.1 0.13 ± 33% perf-profile.children.cycles-pp.irq_work_needs_cpu
0.08 ± 19% -0.0 0.05 ± 71% +0.3 0.36 ± 24% perf-profile.children.cycles-pp.exec_mmap
0.45 ± 17% -0.0 0.42 ± 18% +1.5 1.98 ± 10% perf-profile.children.cycles-pp.handle_mm_fault
0.21 ± 16% -0.0 0.18 ± 15% +0.6 0.83 ± 12% perf-profile.children.cycles-pp.link_path_walk
0.08 ± 36% -0.0 0.06 ± 54% +0.4 0.45 ± 23% perf-profile.children.cycles-pp.tmigr_requires_handle_remote
0.03 ±101% -0.0 0.01 ±299% +0.2 0.20 ± 25% perf-profile.children.cycles-pp.alloc_thread_stack_node
0.08 ± 22% -0.0 0.06 ± 58% +0.2 0.29 ± 28% perf-profile.children.cycles-pp.fold_vm_numa_events
0.34 ± 22% -0.0 0.31 ± 18% +1.1 1.48 ± 11% perf-profile.children.cycles-pp.__x64_sys_openat
0.04 ±106% -0.0 0.02 ±154% +0.1 0.16 ± 36% perf-profile.children.cycles-pp.cpuidle_reflect
0.08 ± 49% -0.0 0.05 ± 69% +0.6 0.66 ± 21% perf-profile.children.cycles-pp.rcu_gp_kthread
0.03 ±102% -0.0 0.01 ±300% +0.1 0.17 ± 35% perf-profile.children.cycles-pp.unmap_region
0.03 ±102% -0.0 0.01 ±300% +0.2 0.19 ± 37% perf-profile.children.cycles-pp.vsnprintf
0.05 ± 58% -0.0 0.03 ± 82% +0.2 0.27 ± 37% perf-profile.children.cycles-pp.wake_up_new_task
0.05 ± 70% -0.0 0.03 ±100% +0.2 0.22 ± 14% perf-profile.children.cycles-pp.free_pages_and_swap_cache
0.05 ± 70% -0.0 0.03 ±100% +0.2 0.22 ± 15% perf-profile.children.cycles-pp.__tlb_batch_free_encoded_pages
0.14 ± 20% -0.0 0.11 ± 16% +0.5 0.66 ± 15% perf-profile.children.cycles-pp.__x64_sys_vfork
0.16 ± 19% -0.0 0.14 ± 18% +0.6 0.74 ± 13% perf-profile.children.cycles-pp.__vfork
2.54 ± 16% -0.0 2.52 ± 23% -1.8 0.70 ± 14% perf-profile.children.cycles-pp.tcp_recvmsg_locked
3.70 ± 22% -0.0 3.68 ± 24% -3.3 0.39 ± 17% perf-profile.children.cycles-pp._copy_from_iter
0.11 ± 22% -0.0 0.09 ± 37% +0.3 0.43 ± 25% perf-profile.children.cycles-pp.perf_read
0.28 ± 27% -0.0 0.25 ± 22% +0.9 1.18 ± 10% perf-profile.children.cycles-pp.mmap_region
0.09 ± 50% -0.0 0.07 ± 39% +0.3 0.37 ± 28% perf-profile.children.cycles-pp.__split_vma
0.14 ± 29% -0.0 0.12 ± 27% +0.5 0.68 ± 18% perf-profile.children.cycles-pp.__set_cpus_allowed_ptr
0.42 ± 18% -0.0 0.40 ± 19% +1.5 1.88 ± 12% perf-profile.children.cycles-pp.__handle_mm_fault
0.09 ± 62% -0.0 0.07 ± 43% +0.2 0.34 ± 30% perf-profile.children.cycles-pp.__do_sys_clone
0.05 ± 54% -0.0 0.03 ±101% +0.3 0.31 ± 37% perf-profile.children.cycles-pp.__set_cpus_allowed_ptr_locked
0.10 ± 25% -0.0 0.08 ± 23% +0.3 0.39 ± 28% perf-profile.children.cycles-pp.sched_balance_softirq
0.03 ±107% -0.0 0.01 ±209% +0.2 0.24 ± 19% perf-profile.children.cycles-pp.__wait_for_common
0.07 ± 43% -0.0 0.05 ± 75% +0.3 0.36 ± 24% perf-profile.children.cycles-pp.affine_move_task
0.17 ± 27% -0.0 0.15 ± 24% +0.6 0.77 ± 21% perf-profile.children.cycles-pp.__sched_setaffinity
2.36 ± 19% -0.0 2.34 ± 26% -2.0 0.35 ± 25% perf-profile.children.cycles-pp.__skb_datagram_iter
0.03 ±102% -0.0 0.01 ±201% +0.2 0.18 ± 24% perf-profile.children.cycles-pp.perf_event_read
0.04 ± 84% -0.0 0.02 ±154% +0.1 0.16 ± 33% perf-profile.children.cycles-pp.dev_attr_show
0.02 ±156% -0.0 0.00 +0.1 0.14 ± 41% perf-profile.children.cycles-pp.__perf_read_group_add
0.04 ± 84% -0.0 0.02 ±154% +0.1 0.16 ± 31% perf-profile.children.cycles-pp.sysfs_kf_seq_show
0.03 ±103% -0.0 0.01 ±203% +0.1 0.18 ± 25% perf-profile.children.cycles-pp.smp_call_function_single
0.04 ± 68% -0.0 0.02 ±129% +0.2 0.27 ± 29% perf-profile.children.cycles-pp.run_posix_cpu_timers
0.41 ± 25% -0.0 0.40 ± 22% +1.5 1.92 ± 17% perf-profile.children.cycles-pp.sched_setaffinity
3.72 ± 22% -0.0 3.71 ± 23% -3.3 0.40 ± 17% perf-profile.children.cycles-pp.skb_do_copy_data_nocache
2.36 ± 19% -0.0 2.34 ± 26% -2.0 0.36 ± 24% perf-profile.children.cycles-pp.skb_copy_datagram_iter
0.05 ± 52% -0.0 0.04 ± 83% +0.2 0.25 ± 19% perf-profile.children.cycles-pp.dup_task_struct
0.02 ±123% -0.0 0.01 ±300% +0.1 0.17 ± 36% perf-profile.children.cycles-pp.mprotect_fixup
0.04 ± 71% -0.0 0.03 ±127% +0.2 0.22 ± 27% perf-profile.children.cycles-pp.free_pgtables
0.14 ± 20% -0.0 0.13 ± 24% +0.5 0.62 ± 29% perf-profile.children.cycles-pp.evsel__read_counter
0.63 ± 15% -0.0 0.62 ± 16% -0.4 0.18 ± 31% perf-profile.children.cycles-pp.__folio_mark_dirty
0.03 ±100% -0.0 0.01 ±200% +0.2 0.27 ± 31% perf-profile.children.cycles-pp.tmigr_handle_remote_cpu
0.16 ± 30% -0.0 0.14 ± 20% +0.5 0.66 ± 15% perf-profile.children.cycles-pp.copy_process
0.03 ±128% -0.0 0.01 ±200% +0.2 0.19 ± 45% perf-profile.children.cycles-pp.mod_objcg_state
0.10 ± 22% -0.0 0.08 ± 26% +0.3 0.44 ± 33% perf-profile.children.cycles-pp.sched_balance_find_dst_cpu
0.03 ±109% -0.0 0.02 ±152% +0.1 0.16 ± 33% perf-profile.children.cycles-pp.dup_mm
0.10 ± 50% -0.0 0.09 ± 28% +0.3 0.41 ± 21% perf-profile.children.cycles-pp._Fork
0.13 ± 34% -0.0 0.12 ± 22% +0.4 0.54 ± 19% perf-profile.children.cycles-pp.__open64_nocancel
0.06 ± 77% -0.0 0.04 ±110% +0.1 0.16 ± 42% perf-profile.children.cycles-pp._find_next_bit
0.05 ± 69% -0.0 0.04 ±104% +0.2 0.30 ± 27% perf-profile.children.cycles-pp.hrtimer_update_next_event
0.04 ± 86% -0.0 0.02 ±123% +0.2 0.19 ± 21% perf-profile.children.cycles-pp.__x64_sys_mprotect
0.04 ± 86% -0.0 0.02 ±123% +0.2 0.19 ± 21% perf-profile.children.cycles-pp.do_mprotect_pkey
0.08 ± 23% -0.0 0.07 ± 43% +0.3 0.42 ± 34% perf-profile.children.cycles-pp.update_sg_wakeup_stats
0.11 ± 25% -0.0 0.09 ± 30% +0.3 0.43 ± 25% perf-profile.children.cycles-pp.perf_evsel__read
0.18 ± 25% -0.0 0.17 ± 22% +0.7 0.86 ± 19% perf-profile.children.cycles-pp.__x64_sys_sched_setaffinity
0.06 ± 60% -0.0 0.04 ± 82% +0.2 0.23 ± 36% perf-profile.children.cycles-pp.pipe_write
0.02 ±154% -0.0 0.01 ±299% +0.1 0.12 ± 43% perf-profile.children.cycles-pp.tmigr_quick_check
0.08 ± 42% -0.0 0.06 ± 57% +0.2 0.27 ± 24% perf-profile.children.cycles-pp.cpu_stopper_thread
0.06 ± 57% -0.0 0.04 ± 83% +0.5 0.53 ± 16% perf-profile.children.cycles-pp.rcu_gp_fqs_loop
0.32 ± 23% -0.0 0.31 ± 19% +1.1 1.38 ± 11% perf-profile.children.cycles-pp.path_openat
0.33 ± 23% -0.0 0.31 ± 20% +1.1 1.41 ± 12% perf-profile.children.cycles-pp.do_filp_open
0.03 ±106% -0.0 0.02 ±154% +0.2 0.26 ± 31% perf-profile.children.cycles-pp.irqentry_enter
0.02 ±154% -0.0 0.01 ±300% +0.1 0.15 ± 23% perf-profile.children.cycles-pp.__perf_event_read
0.02 ±153% -0.0 0.01 ±300% +0.1 0.12 ± 23% perf-profile.children.cycles-pp.mas_wr_node_store
0.02 ±220% -0.0 0.01 ±299% +0.1 0.16 ± 40% perf-profile.children.cycles-pp.vma_complete
0.02 ±210% -0.0 0.01 ±299% +0.2 0.19 ± 33% perf-profile.children.cycles-pp.do_open
0.08 ± 23% -0.0 0.06 ± 36% +0.3 0.34 ± 19% perf-profile.children.cycles-pp.d_alloc
0.04 ±103% -0.0 0.03 ±132% +0.3 0.33 ± 26% perf-profile.children.cycles-pp.tick_check_oneshot_broadcast_this_cpu
0.01 ±201% -0.0 0.00 +0.1 0.12 ± 40% perf-profile.children.cycles-pp.__fdget_pos
0.03 ± 82% -0.0 0.02 ±153% +0.2 0.22 ± 39% perf-profile.children.cycles-pp.sched_exec
0.06 ± 56% -0.0 0.05 ± 68% +0.2 0.25 ± 24% perf-profile.children.cycles-pp.migration_cpu_stop
0.08 ± 49% -0.0 0.07 ± 48% +0.2 0.28 ± 30% perf-profile.children.cycles-pp.__memcg_slab_free_hook
0.24 ± 82% -0.0 0.23 ±122% +0.6 0.82 ± 21% perf-profile.children.cycles-pp.rcu_core
0.02 ±152% -0.0 0.01 ±299% +0.1 0.11 ± 42% perf-profile.children.cycles-pp.__x64_sys_munmap
0.02 ±123% -0.0 0.01 ±201% +0.1 0.14 ± 39% perf-profile.children.cycles-pp.error_entry
0.01 ±200% -0.0 0.00 +0.1 0.12 ± 34% perf-profile.children.cycles-pp.cpu_stop_queue_work
0.01 ±200% -0.0 0.00 +0.1 0.16 ± 27% perf-profile.children.cycles-pp._IO_fwrite
0.02 ±216% -0.0 0.01 ±300% +0.2 0.19 ± 41% perf-profile.children.cycles-pp.leave_mm
0.02 ±153% -0.0 0.01 ±300% +0.2 0.20 ± 41% perf-profile.children.cycles-pp.call_timer_fn
0.04 ± 70% -0.0 0.03 ± 82% +0.1 0.15 ± 26% perf-profile.children.cycles-pp.mas_wr_store_entry
0.06 ± 55% -0.0 0.05 ± 55% +0.2 0.27 ± 32% perf-profile.children.cycles-pp.wp_page_copy
0.09 ± 26% -0.0 0.08 ± 27% +0.3 0.43 ± 32% perf-profile.children.cycles-pp.sched_balance_find_dst_group
0.07 ±205% -0.0 0.06 ±299% +0.5 0.59 ± 21% perf-profile.children.cycles-pp.tcp_sock_set_cork
0.03 ±125% -0.0 0.02 ±153% +0.2 0.21 ± 34% perf-profile.children.cycles-pp.irqtime_account_process_tick
0.02 ±152% -0.0 0.01 ±300% +0.1 0.11 ± 51% perf-profile.children.cycles-pp.__wake_up_sync_key
0.03 ±102% -0.0 0.02 ±124% +0.2 0.24 ± 26% perf-profile.children.cycles-pp.alloc_anon_folio
0.01 ±300% -0.0 0.00 +0.1 0.15 ± 47% perf-profile.children.cycles-pp.vfs_open
0.06 ± 54% -0.0 0.05 ± 69% +0.2 0.23 ± 25% perf-profile.children.cycles-pp.move_queued_task
0.10 ± 58% -0.0 0.09 ± 61% +0.2 0.34 ± 30% perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.02 ±156% -0.0 0.01 ±201% +0.3 0.29 ± 29% perf-profile.children.cycles-pp.force_qs_rnp
0.03 ±104% -0.0 0.03 ±100% +0.2 0.22 ± 25% perf-profile.children.cycles-pp.get_jiffies_update
0.08 ± 36% -0.0 0.07 ± 42% +0.2 0.30 ± 40% perf-profile.children.cycles-pp.copy_strings
0.09 ± 31% -0.0 0.08 ± 41% +0.3 0.43 ± 30% perf-profile.children.cycles-pp.do_anonymous_page
0.21 ± 20% -0.0 0.20 ± 26% +0.8 0.98 ± 15% perf-profile.children.cycles-pp.do_fault
0.05 ± 57% -0.0 0.04 ± 68% +0.2 0.23 ± 21% perf-profile.children.cycles-pp.__cmd_record
0.05 ± 57% -0.0 0.04 ± 68% +0.2 0.23 ± 21% perf-profile.children.cycles-pp.cmd_record
0.13 ± 15% -0.0 0.12 ± 26% +0.2 0.34 ± 23% perf-profile.children.cycles-pp.alloc_pages_mpol_noprof
0.11 ± 19% -0.0 0.10 ± 20% +0.4 0.46 ± 18% perf-profile.children.cycles-pp.__lookup_slow
0.01 ±200% -0.0 0.01 ±300% +0.1 0.15 ± 47% perf-profile.children.cycles-pp.vma_alloc_folio_noprof
0.01 ±300% -0.0 0.00 +0.1 0.15 ± 47% perf-profile.children.cycles-pp.do_dentry_open
0.12 ± 31% -0.0 0.11 ± 31% +0.4 0.52 ± 25% perf-profile.children.cycles-pp.elf_load
0.12 ± 24% -0.0 0.12 ± 26% +0.2 0.33 ± 21% perf-profile.children.cycles-pp.__alloc_pages_noprof
0.22 ± 28% -0.0 0.22 ± 28% +0.7 0.92 ± 16% perf-profile.children.cycles-pp.setlocale
0.01 ±300% -0.0 0.00 +0.3 0.26 ± 22% perf-profile.children.cycles-pp.schedule_timeout
0.01 ±201% -0.0 0.01 ±299% +0.1 0.15 ± 46% perf-profile.children.cycles-pp.__mem_cgroup_charge
0.03 ±103% -0.0 0.02 ±123% +0.2 0.25 ± 33% perf-profile.children.cycles-pp.__do_set_cpus_allowed
0.09 ± 61% -0.0 0.08 ± 60% +0.2 0.32 ± 30% perf-profile.children.cycles-pp.tsc_verify_tsc_adjust
0.05 ± 67% -0.0 0.05 ± 51% +0.2 0.23 ± 30% perf-profile.children.cycles-pp.show_stat
0.02 ±152% -0.0 0.01 ±200% +0.2 0.18 ± 32% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
0.01 ±299% -0.0 0.00 +0.1 0.12 ± 23% perf-profile.children.cycles-pp.sclhi
0.09 ± 35% -0.0 0.08 ± 36% +0.3 0.35 ± 31% perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
0.02 ±213% -0.0 0.01 ±201% +0.1 0.15 ± 29% perf-profile.children.cycles-pp.__percpu_counter_sum
0.06 ± 36% -0.0 0.05 ± 37% +0.2 0.23 ± 24% perf-profile.children.cycles-pp.vmstat_start
0.09 ± 31% -0.0 0.08 ± 29% +0.2 0.33 ± 22% perf-profile.children.cycles-pp.__mmap
0.08 ± 24% -0.0 0.07 ± 47% +0.3 0.36 ± 21% perf-profile.children.cycles-pp.next_uptodate_folio
0.16 ± 16% -0.0 0.15 ± 17% +0.5 0.61 ± 15% perf-profile.children.cycles-pp.walk_component
0.28 ± 26% -0.0 0.28 ± 24% +1.0 1.26 ± 17% perf-profile.children.cycles-pp.evlist_cpu_iterator__next
0.01 ±206% -0.0 0.01 ±200% +0.1 0.14 ± 37% perf-profile.children.cycles-pp.pm_qos_read_value
0.02 ±156% -0.0 0.02 ±153% +0.1 0.12 ± 32% perf-profile.children.cycles-pp.dup_mmap
0.04 ± 84% -0.0 0.04 ± 84% +0.2 0.22 ± 16% perf-profile.children.cycles-pp.try_address
0.25 ± 13% -0.0 0.25 ± 32% +0.8 1.07 ± 19% perf-profile.children.cycles-pp.balance_fair
0.02 ±123% -0.0 0.02 ±124% +0.2 0.18 ± 18% perf-profile.children.cycles-pp.i2c_outb
0.04 ± 84% -0.0 0.03 ±102% +0.1 0.17 ± 20% perf-profile.children.cycles-pp.step_into
0.08 ± 25% -0.0 0.08 ± 18% +0.3 0.39 ± 16% perf-profile.children.cycles-pp.d_alloc_parallel
0.03 ±124% -0.0 0.03 ±100% +0.1 0.17 ± 37% perf-profile.children.cycles-pp.__pte_offset_map_lock
0.01 ±200% -0.0 0.01 ±300% +0.2 0.21 ± 30% perf-profile.children.cycles-pp.timekeeping_update
0.01 ±300% -0.0 0.01 ±299% +0.2 0.20 ± 24% perf-profile.children.cycles-pp.dyntick_save_progress_counter
0.09 ± 26% -0.0 0.08 ± 30% +0.2 0.24 ± 34% perf-profile.children.cycles-pp.get_page_from_freelist
0.00 +0.0 0.00 +0.1 0.09 ± 27% perf-profile.children.cycles-pp.seq_printf
0.00 +0.0 0.00 +0.1 0.10 ± 35% perf-profile.children.cycles-pp.task_tick_mm_cid
0.00 +0.0 0.00 +0.1 0.10 ± 40% perf-profile.children.cycles-pp.notifier_call_chain
0.00 +0.0 0.00 +0.1 0.11 ± 52% perf-profile.children.cycles-pp.part_in_flight
0.00 +0.0 0.00 +0.1 0.11 ± 34% perf-profile.children.cycles-pp.blk_mq_run_hw_queues
0.00 +0.0 0.00 +0.1 0.11 ± 34% perf-profile.children.cycles-pp.blk_mq_requeue_work
0.01 ±299% +0.0 0.01 ±300% +0.1 0.12 ± 23% perf-profile.children.cycles-pp.record__pushfn
0.01 ±299% +0.0 0.01 ±300% +0.1 0.12 ± 56% perf-profile.children.cycles-pp.vma_prepare
0.01 ±299% +0.0 0.01 ±300% +0.1 0.12 ± 20% perf-profile.children.cycles-pp.writen
0.01 ±299% +0.0 0.01 ±300% +0.1 0.13 ± 20% perf-profile.children.cycles-pp.shmem_file_write_iter
0.01 ±206% +0.0 0.01 ±200% +0.1 0.14 ± 28% perf-profile.children.cycles-pp.mas_walk
0.00 +0.0 0.00 +0.1 0.14 ± 38% perf-profile.children.cycles-pp.open64
0.00 +0.0 0.00 +0.2 0.16 ± 38% perf-profile.children.cycles-pp.nfs_wb_folio
0.00 +0.0 0.00 +0.2 0.16 ± 56% perf-profile.children.cycles-pp.irq_work_tick
0.01 ±200% +0.0 0.01 ±203% +0.2 0.19 ± 19% perf-profile.children.cycles-pp.setup_arg_pages
0.00 +0.0 0.00 +0.2 0.18 ± 35% perf-profile.children.cycles-pp.journal_submit_commit_record
0.00 +0.0 0.00 +0.2 0.18 ± 38% perf-profile.children.cycles-pp.blk_finish_plug
0.00 +0.0 0.00 +0.2 0.19 ± 41% perf-profile.children.cycles-pp.jbd2_journal_get_descriptor_buffer
0.00 +0.0 0.00 +0.2 0.21 ± 34% perf-profile.children.cycles-pp.check_cpu_stall
0.03 ±102% +0.0 0.03 ±100% +0.3 0.36 ± 23% perf-profile.children.cycles-pp.rcu_pending
0.00 +0.0 0.00 +0.6 0.59 ± 22% perf-profile.children.cycles-pp.jbd2_journal_commit_transaction
0.00 +0.0 0.00 +0.6 0.61 ± 20% perf-profile.children.cycles-pp.kjournald2
0.01 ±300% +0.0 0.01 ±300% +0.1 0.14 ± 41% perf-profile.children.cycles-pp.clear_page_erms
0.01 ±300% +0.0 0.01 ±300% +0.1 0.15 ± 30% perf-profile.children.cycles-pp.shift_arg_pages
0.03 ±132% +0.0 0.03 ±127% +0.2 0.18 ± 37% perf-profile.children.cycles-pp.ct_kernel_exit
0.01 ±201% +0.0 0.01 ±200% +0.2 0.17 ± 28% perf-profile.children.cycles-pp.mas_find
0.02 ±300% +0.0 0.02 ±300% +0.2 0.19 ± 37% perf-profile.children.cycles-pp.nfs4_proc_pgio_rpc_prepare
0.16 ± 15% +0.0 0.16 ± 29% +0.6 0.76 ± 16% perf-profile.children.cycles-pp.filemap_map_pages
0.01 ±299% +0.0 0.01 ±300% +0.1 0.08 ± 36% perf-profile.children.cycles-pp.__dentry_kill
0.08 ± 37% +0.0 0.08 ± 17% +0.3 0.34 ± 21% perf-profile.children.cycles-pp.proc_reg_read_iter
0.18 ± 34% +0.0 0.19 ± 18% +0.4 0.57 ± 34% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.52 ± 16% +0.0 0.53 ± 33% -0.3 0.23 ± 51% perf-profile.children.cycles-pp.filemap_get_folios_tag
0.01 ±200% +0.0 0.01 ±200% +0.2 0.20 ± 34% perf-profile.children.cycles-pp.perf_iterate_sb
0.04 ± 85% +0.0 0.04 ± 71% +0.2 0.23 ± 15% perf-profile.children.cycles-pp.__i2c_transfer
0.04 ± 85% +0.0 0.04 ± 71% +0.2 0.23 ± 15% perf-profile.children.cycles-pp.bit_xfer
0.04 ± 85% +0.0 0.04 ± 71% +0.2 0.23 ± 15% perf-profile.children.cycles-pp.drm_connector_helper_detect_from_ddc
0.04 ± 85% +0.0 0.04 ± 71% +0.2 0.23 ± 15% perf-profile.children.cycles-pp.drm_do_probe_ddc_edid
0.04 ± 85% +0.0 0.04 ± 71% +0.2 0.23 ± 15% perf-profile.children.cycles-pp.drm_helper_probe_detect_ctx
0.04 ± 85% +0.0 0.04 ± 71% +0.2 0.23 ± 15% perf-profile.children.cycles-pp.drm_probe_ddc
0.04 ± 85% +0.0 0.04 ± 71% +0.2 0.23 ± 15% perf-profile.children.cycles-pp.i2c_transfer
0.04 ± 85% +0.0 0.04 ± 71% +0.2 0.23 ± 15% perf-profile.children.cycles-pp.output_poll_execute
0.04 ± 84% +0.0 0.04 ± 82% +0.2 0.24 ± 30% perf-profile.children.cycles-pp.get_arg_page
0.14 ± 45% +0.0 0.14 ± 46% +0.1 0.28 ± 31% perf-profile.children.cycles-pp.__check_object_size
0.01 ±299% +0.0 0.01 ±201% +0.2 0.16 ± 48% perf-profile.children.cycles-pp.kmem_cache_alloc_lru_noprof
0.01 ±300% +0.0 0.01 ±200% +0.2 0.16 ± 35% perf-profile.children.cycles-pp.perf_event_task_tick
0.02 ±123% +0.0 0.02 ±125% +0.2 0.19 ± 36% perf-profile.children.cycles-pp.__get_user_pages
0.38 ± 29% +0.0 0.38 ± 39% -0.2 0.13 ± 39% perf-profile.children.cycles-pp.__release_sock
0.02 ±123% +0.0 0.03 ±124% +0.2 0.22 ± 39% perf-profile.children.cycles-pp.getname_flags
0.01 ±300% +0.0 0.01 ±201% +0.1 0.15 ± 32% perf-profile.children.cycles-pp.strncpy_from_user
0.00 +0.0 0.01 ±299% +0.1 0.10 ± 32% perf-profile.children.cycles-pp.simple_lookup
0.01 ±299% +0.0 0.01 ±200% +0.1 0.14 ± 20% perf-profile.children.cycles-pp.perf_mmap__push
0.01 ±299% +0.0 0.01 ±200% +0.1 0.14 ± 20% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.11 ± 39% +0.0 0.11 ± 31% +0.2 0.29 ± 39% perf-profile.children.cycles-pp.get_cpu_device
0.07 ± 26% +0.0 0.08 ± 27% +0.4 0.51 ± 20% perf-profile.children.cycles-pp.rcu_sched_clock_irq
0.17 ± 15% +0.0 0.18 ± 29% +0.7 0.83 ± 16% perf-profile.children.cycles-pp.do_read_fault
0.02 ±123% +0.0 0.03 ±103% +0.2 0.20 ± 37% perf-profile.children.cycles-pp.get_user_pages_remote
0.02 ±154% +0.0 0.03 ±127% +0.1 0.15 ± 55% perf-profile.children.cycles-pp.acpi_ps_execute_method
0.02 ±155% +0.0 0.03 ±100% +0.2 0.18 ± 34% perf-profile.children.cycles-pp.__d_alloc
0.02 ±154% +0.0 0.03 ±129% +0.1 0.16 ± 54% perf-profile.children.cycles-pp.acpi_ns_evaluate
0.01 ±299% +0.0 0.02 ±204% +0.1 0.14 ± 47% perf-profile.children.cycles-pp.check_tsc_unstable
0.01 ±300% +0.0 0.02 ±153% +0.1 0.11 ± 19% perf-profile.children.cycles-pp.x64_sys_call
0.02 ±153% +0.0 0.03 ±125% +0.2 0.20 ± 27% perf-profile.children.cycles-pp.wait4
0.00 +0.0 0.01 ±201% +0.1 0.11 ± 36% perf-profile.children.cycles-pp.acpi_ev_asynch_execute_gpe_method
0.00 +0.0 0.01 ±201% +0.1 0.12 ± 38% perf-profile.children.cycles-pp.acpi_os_execute_deferred
0.03 ±102% +0.0 0.04 ± 87% +0.2 0.27 ± 32% perf-profile.children.cycles-pp.alloc_empty_file
0.04 ± 85% +0.0 0.06 ± 65% +0.4 0.40 ± 31% perf-profile.children.cycles-pp.timekeeping_advance
0.01 ±300% +0.0 0.02 ±154% +0.2 0.16 ± 38% perf-profile.children.cycles-pp.copy_string_kernel
0.04 ± 85% +0.0 0.06 ± 65% +0.4 0.40 ± 31% perf-profile.children.cycles-pp.update_wall_time
0.00 +0.0 0.02 ±204% +0.1 0.10 ± 32% perf-profile.children.cycles-pp.ext4_io_submit
0.03 ±101% +0.0 0.04 ± 69% +0.1 0.16 ± 38% perf-profile.children.cycles-pp.lookup_fast
0.02 ±165% +0.0 0.04 ± 71% +0.2 0.21 ± 20% perf-profile.children.cycles-pp.__mmdrop
0.02 ±154% +0.0 0.03 ±106% +0.2 0.26 ± 27% perf-profile.children.cycles-pp.perf_event_mmap_event
0.02 ±126% +0.0 0.04 ± 88% +0.2 0.27 ± 28% perf-profile.children.cycles-pp.perf_event_mmap
0.01 ±206% +0.0 0.03 ±124% +0.2 0.22 ± 18% perf-profile.children.cycles-pp.vma_interval_tree_insert
0.00 +0.0 0.02 ±155% +0.2 0.18 ± 38% perf-profile.children.cycles-pp.set_pte_range
0.06 ± 44% +0.0 0.08 ± 26% +0.3 0.38 ± 36% perf-profile.children.cycles-pp.alloc_bprm
0.08 ± 71% +0.0 0.10 ± 23% +0.1 0.22 ± 27% perf-profile.children.cycles-pp.menu_reflect
0.54 ± 19% +0.0 0.56 ± 22% -0.4 0.16 ± 20% perf-profile.children.cycles-pp.ext4_da_write_begin
0.04 ±211% +0.0 0.06 ±182% +0.4 0.45 ± 17% perf-profile.children.cycles-pp.rpc_execute
0.10 ± 83% +0.0 0.13 ± 59% +0.3 0.43 ± 47% perf-profile.children.cycles-pp.delay_tsc
0.21 ± 29% +0.0 0.24 ± 42% +0.4 0.62 ± 43% perf-profile.children.cycles-pp.tmigr_handle_remote_up
0.22 ± 31% +0.0 0.25 ± 39% +0.5 0.68 ± 38% perf-profile.children.cycles-pp.tmigr_handle_remote
0.14 ± 41% +0.0 0.17 ± 20% +0.4 0.51 ± 24% perf-profile.children.cycles-pp.ct_idle_exit
0.41 ± 44% +0.0 0.45 ± 31% +0.6 0.98 ± 19% perf-profile.children.cycles-pp.ktime_get
0.05 ±226% +0.0 0.09 ±116% +0.4 0.48 ± 12% perf-profile.children.cycles-pp.nfs_initiate_pgio
0.10 ± 40% +0.0 0.14 ± 19% +0.3 0.40 ± 26% perf-profile.children.cycles-pp.ct_kernel_enter
0.04 ±220% +0.0 0.09 ±153% +0.5 0.51 ± 16% perf-profile.children.cycles-pp.rpc_run_task
0.02 ±154% +0.1 0.07 ± 54% +0.1 0.10 ± 39% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.02 ±156% +0.1 0.08 ± 51% +0.2 0.18 ± 33% perf-profile.children.cycles-pp.switch_mm_irqs_off
0.04 ± 85% +0.1 0.10 ± 21% +0.2 0.19 ± 26% perf-profile.children.cycles-pp.ct_kernel_exit_state
0.09 ± 35% +0.1 0.15 ± 45% +0.1 0.22 ± 28% perf-profile.children.cycles-pp.clear_bhb_loop
0.13 ± 29% +0.1 0.19 ± 19% +0.2 0.35 ± 25% perf-profile.children.cycles-pp.update_curr
0.14 ± 40% +0.1 0.20 ± 27% +0.2 0.29 ± 24% perf-profile.children.cycles-pp.set_next_entity
0.32 ± 16% +0.1 0.39 ± 13% +0.8 1.11 ± 7% perf-profile.children.cycles-pp.sched_balance_domains
0.14 ± 38% +0.1 0.22 ± 22% +0.3 0.46 ± 21% perf-profile.children.cycles-pp.local_clock_noinstr
0.41 ± 19% +0.1 0.49 ± 28% +1.3 1.66 ± 15% perf-profile.children.cycles-pp.smpboot_thread_fn
1.75 ± 36% +0.1 1.84 ± 55% +2.1 3.89 ± 14% perf-profile.children.cycles-pp.__irq_exit_rcu
0.06 ± 70% +0.1 0.16 ± 22% -0.0 0.04 ± 92% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.33 ± 27% +0.1 0.44 ± 21% +0.6 0.90 ± 22% perf-profile.children.cycles-pp._nohz_idle_balance
0.08 ± 50% +0.1 0.19 ± 19% +0.0 0.13 ± 38% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.19 ± 31% +0.1 0.32 ± 21% +0.3 0.51 ± 18% perf-profile.children.cycles-pp.sched_clock_cpu
0.15 ± 51% +0.1 0.29 ± 28% +0.2 0.40 ± 16% perf-profile.children.cycles-pp.syscall_exit_to_user_mode
1.86 ± 18% +0.2 2.02 ± 34% -1.6 0.24 ± 36% perf-profile.children.cycles-pp.mark_buffer_dirty
0.20 ± 30% +0.2 0.36 ± 28% +0.5 0.66 ± 21% perf-profile.children.cycles-pp.update_rq_clock_task
0.14 ± 55% +0.2 0.32 ± 27% +0.3 0.39 ± 31% perf-profile.children.cycles-pp.tick_nohz_stop_idle
0.12 ± 44% +0.2 0.30 ± 29% +0.2 0.27 ± 32% perf-profile.children.cycles-pp.nr_iowait_cpu
0.29 ± 38% +0.2 0.48 ± 35% +0.4 0.74 ± 18% perf-profile.children.cycles-pp.select_task_rq_fair
0.11 ± 67% +0.2 0.31 ± 46% +0.0 0.16 ± 26% perf-profile.children.cycles-pp.available_idle_cpu
0.32 ± 38% +0.2 0.52 ± 19% +0.4 0.76 ± 21% perf-profile.children.cycles-pp.dequeue_entity
0.44 ± 35% +0.2 0.64 ± 37% -0.3 0.17 ± 30% perf-profile.children.cycles-pp.filemap_get_entry
2.26 ± 19% +0.2 2.48 ± 34% -2.0 0.27 ± 37% perf-profile.children.cycles-pp.__block_commit_write
0.56 ± 30% +0.2 0.78 ± 35% -0.3 0.29 ± 26% perf-profile.children.cycles-pp.__filemap_get_folio
0.38 ± 35% +0.2 0.62 ± 19% +0.5 0.88 ± 22% perf-profile.children.cycles-pp.dequeue_task_fair
2.47 ± 20% +0.3 2.76 ± 35% -2.2 0.29 ± 36% perf-profile.children.cycles-pp.block_write_end
0.32 ± 42% +0.3 0.67 ± 30% +0.1 0.41 ± 15% perf-profile.children.cycles-pp.select_task_rq
2.58 ± 20% +0.4 2.93 ± 35% -2.3 0.31 ± 34% perf-profile.children.cycles-pp.ext4_da_write_end
1.05 ± 18% +0.4 1.43 ± 65% -0.8 0.22 ± 29% perf-profile.children.cycles-pp.folio_wake_bit
0.24 ± 27% +0.4 0.62 ± 28% +0.2 0.46 ± 13% perf-profile.children.cycles-pp.sched_clock
0.19 ± 35% +0.4 0.59 ± 32% +0.2 0.42 ± 28% perf-profile.children.cycles-pp.update_rq_clock
0.18 ± 61% +0.4 0.58 ± 29% +0.1 0.25 ± 17% perf-profile.children.cycles-pp.llist_reverse_order
0.29 ± 26% +0.4 0.71 ± 23% +0.4 0.65 ± 11% perf-profile.children.cycles-pp.native_sched_clock
0.21 ± 63% +0.5 0.68 ± 49% +0.1 0.30 ± 32% perf-profile.children.cycles-pp.call_function_single_prep_ipi
0.53 ± 20% +0.5 1.00 ± 71% -0.4 0.13 ± 36% perf-profile.children.cycles-pp.wake_page_function
0.42 ± 34% +0.5 0.91 ± 51% +0.3 0.69 ± 16% perf-profile.children.cycles-pp.enqueue_entity
0.30 ± 64% +0.5 0.83 ± 46% +0.1 0.41 ± 29% perf-profile.children.cycles-pp.__smp_call_single_queue
0.59 ± 28% +0.5 1.13 ± 48% +0.7 1.30 ± 16% perf-profile.children.cycles-pp.rest_init
0.59 ± 28% +0.5 1.13 ± 48% +0.7 1.30 ± 16% perf-profile.children.cycles-pp.start_kernel
0.59 ± 28% +0.5 1.13 ± 48% +0.7 1.30 ± 16% perf-profile.children.cycles-pp.x86_64_start_kernel
0.59 ± 28% +0.5 1.13 ± 48% +0.7 1.30 ± 16% perf-profile.children.cycles-pp.x86_64_start_reservations
0.49 ± 35% +0.6 1.04 ± 54% +0.4 0.85 ± 13% perf-profile.children.cycles-pp.enqueue_task_fair
0.52 ± 33% +0.6 1.09 ± 52% +0.4 0.92 ± 12% perf-profile.children.cycles-pp.activate_task
0.36 ± 46% +0.6 0.99 ± 43% +0.1 0.42 ± 20% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.12 ± 23% +0.8 0.93 ± 49% +3.0 3.09 ± 58% perf-profile.children.cycles-pp.write
0.11 ± 23% +0.8 0.91 ± 50% +2.9 3.03 ± 59% perf-profile.children.cycles-pp.ksys_write
1.49 ± 29% +0.9 2.35 ± 56% -0.6 0.94 ± 18% perf-profile.children.cycles-pp.blk_complete_reqs
0.59 ± 36% +0.9 1.51 ± 43% +0.4 0.96 ± 16% perf-profile.children.cycles-pp.ttwu_do_activate
1.36 ± 25% +0.9 2.30 ± 57% -0.6 0.77 ± 17% perf-profile.children.cycles-pp.scsi_io_completion
1.36 ± 25% +0.9 2.30 ± 57% -0.6 0.76 ± 17% perf-profile.children.cycles-pp.scsi_end_request
1.16 ± 16% +1.0 2.16 ± 63% -0.7 0.45 ± 23% perf-profile.children.cycles-pp.blk_update_request
1.05 ± 15% +1.0 2.08 ± 67% -0.8 0.27 ± 23% perf-profile.children.cycles-pp.ext4_end_bio
0.50 ± 85% +1.0 1.53 ± 81% +1.3 1.82 ± 22% perf-profile.children.cycles-pp._raw_spin_lock_irq
1.03 ± 15% +1.0 2.08 ± 68% -0.8 0.24 ± 24% perf-profile.children.cycles-pp.ext4_finish_bio
5.57 ± 13% +1.3 6.89 ± 39% -3.6 1.98 ± 13% perf-profile.children.cycles-pp.file_write_and_wait_range
5.57 ± 13% +1.3 6.90 ± 39% -3.5 2.04 ± 13% perf-profile.children.cycles-pp.ext4_sync_file
5.57 ± 13% +1.3 6.90 ± 39% -3.5 2.04 ± 13% perf-profile.children.cycles-pp.nfsd_commit
5.58 ± 13% +1.3 6.90 ± 39% -3.4 2.14 ± 12% perf-profile.children.cycles-pp.nfsd4_commit
0.73 ± 37% +1.6 2.28 ± 39% +0.4 1.14 ± 19% perf-profile.children.cycles-pp.sched_ttwu_pending
12.70 ± 49% +1.6 14.28 ± 60% -11.7 0.98 ± 22% perf-profile.children.cycles-pp.nfsd4_write
12.64 ± 49% +1.6 14.25 ± 61% -11.8 0.88 ± 25% perf-profile.children.cycles-pp.nfsd_vfs_write
12.61 ± 49% +1.6 14.22 ± 61% -11.8 0.85 ± 27% perf-profile.children.cycles-pp.vfs_iter_write
12.60 ± 49% +1.6 14.21 ± 61% -11.8 0.82 ± 26% perf-profile.children.cycles-pp.ext4_buffered_write_iter
12.60 ± 49% +1.6 14.22 ± 61% -11.8 0.82 ± 27% perf-profile.children.cycles-pp.do_iter_readv_writev
1.48 ± 23% +1.7 3.13 ± 65% -0.9 0.58 ± 21% perf-profile.children.cycles-pp.folio_wait_bit_common
1.78 ± 27% +1.8 3.54 ± 40% -1.5 0.26 ± 21% perf-profile.children.cycles-pp.copy_page_from_iter_atomic
0.98 ± 40% +2.0 3.02 ± 37% +0.5 1.45 ± 15% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
3.58 ± 35% +2.6 6.16 ± 35% -3.3 0.26 ± 26% perf-profile.children.cycles-pp.rwsem_spin_on_owner
6.38 ± 20% +2.7 9.07 ± 26% -5.1 1.30 ± 12% perf-profile.children.cycles-pp.generic_perform_write
22.26 ± 25% +2.9 25.13 ± 38% -15.5 6.73 ± 10% perf-profile.children.cycles-pp.nfsd
22.24 ± 25% +2.9 25.12 ± 38% -15.5 6.72 ± 10% perf-profile.children.cycles-pp.svc_recv
22.07 ± 25% +2.9 24.98 ± 38% -15.6 6.43 ± 11% perf-profile.children.cycles-pp.svc_handle_xprt
18.49 ± 32% +2.9 21.44 ± 45% -14.8 3.73 ± 14% perf-profile.children.cycles-pp.nfsd4_proc_compound
18.59 ± 32% +2.9 21.53 ± 44% -14.7 3.88 ± 14% perf-profile.children.cycles-pp.nfsd_dispatch
18.68 ± 32% +3.0 21.63 ± 44% -14.6 4.06 ± 13% perf-profile.children.cycles-pp.svc_process
18.67 ± 32% +3.0 21.63 ± 44% -14.6 4.06 ± 13% perf-profile.children.cycles-pp.svc_process_common
1.83 ± 67% +3.0 4.84 ± 47% +0.4 2.24 ± 19% perf-profile.children.cycles-pp.flush_smp_call_function_queue
0.91 ± 31% +7.5 8.37 ± 61% +2.3 3.21 ± 15% perf-profile.children.cycles-pp.poll_idle
10.41 ± 53% +8.4 18.84 ± 41% -9.7 0.73 ± 25% perf-profile.children.cycles-pp.rwsem_optimistic_spin
10.44 ± 52% +8.5 18.90 ± 41% -9.6 0.88 ± 25% perf-profile.children.cycles-pp.rwsem_down_write_slowpath
10.47 ± 52% +8.5 18.96 ± 41% -9.5 0.98 ± 22% perf-profile.children.cycles-pp.down_write
15.61 ± 40% +9.0 24.65 ± 17% +28.1 43.73 ± 6% perf-profile.children.cycles-pp.cpuidle_idle_call
13.36 ± 39% +9.4 22.80 ± 18% +24.9 38.30 ± 6% perf-profile.children.cycles-pp.cpuidle_enter_state
13.41 ± 39% +9.5 22.88 ± 18% +25.1 38.51 ± 6% perf-profile.children.cycles-pp.cpuidle_enter
19.16 ± 44% +12.4 31.53 ± 14% +29.8 48.92 ± 7% perf-profile.children.cycles-pp.start_secondary
19.72 ± 43% +12.9 32.63 ± 14% +30.5 50.20 ± 6% perf-profile.children.cycles-pp.do_idle
19.74 ± 43% +12.9 32.66 ± 14% +30.5 50.23 ± 6% perf-profile.children.cycles-pp.common_startup_64
19.74 ± 43% +12.9 32.66 ± 14% +30.5 50.23 ± 6% perf-profile.children.cycles-pp.cpu_startup_entry
9.76 ± 46% +15.3 25.03 ± 46% +8.8 18.57 ± 6% perf-profile.children.cycles-pp.do_syscall_64
9.78 ± 45% +15.3 25.10 ± 46% +8.8 18.62 ± 6% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
18.46 ± 33% -15.3 3.14 ±106% -15.2 3.22 ± 20% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
11.13 ± 47% -4.5 6.61 ± 80% -11.0 0.10 ± 49% perf-profile.self.cycles-pp.osq_lock
6.00 ± 56% -1.9 4.12 ± 77% +12.2 18.16 ± 6% perf-profile.self.cycles-pp.intel_idle
1.42 ± 30% -1.3 0.15 ±222% -1.3 0.11 ± 57% perf-profile.self.cycles-pp.mutex_lock
1.05 ± 25% -0.8 0.24 ± 87% -1.0 0.02 ±125% perf-profile.self.cycles-pp.__folio_end_writeback
0.72 ± 19% -0.6 0.13 ±215% -0.5 0.22 ± 50% perf-profile.self.cycles-pp.mutex_spin_on_owner
0.51 ± 6% -0.2 0.29 ± 37% -0.3 0.18 ± 24% perf-profile.self.cycles-pp.percpu_counter_add_batch
0.34 ± 15% -0.2 0.14 ± 73% -0.3 0.04 ±112% perf-profile.self.cycles-pp.mpage_prepare_extent_to_map
0.70 ± 30% -0.2 0.52 ± 35% -0.4 0.25 ± 35% perf-profile.self.cycles-pp.xas_load
0.23 ± 14% -0.1 0.12 ± 37% -0.1 0.13 ± 32% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.29 ± 12% -0.1 0.20 ± 34% -0.3 0.03 ±102% perf-profile.self.cycles-pp.folio_clear_dirty_for_io
0.19 ± 17% -0.1 0.10 ± 21% -0.1 0.06 ± 64% perf-profile.self.cycles-pp.__mod_node_page_state
0.21 ± 24% -0.1 0.14 ± 32% -0.2 0.05 ± 87% perf-profile.self.cycles-pp.__lruvec_stat_mod_folio
0.12 ± 26% -0.1 0.05 ± 76% -0.1 0.04 ±111% perf-profile.self.cycles-pp.xas_clear_mark
0.14 ± 53% -0.1 0.09 ± 78% +0.2 0.37 ± 27% perf-profile.self.cycles-pp.lapic_next_deadline
0.15 ± 35% -0.0 0.10 ± 83% +0.4 0.53 ± 15% perf-profile.self.cycles-pp.native_irq_return_iret
0.07 ± 49% -0.0 0.02 ±124% +0.2 0.25 ± 30% perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.18 ± 21% -0.0 0.14 ± 23% -0.1 0.03 ±112% perf-profile.self.cycles-pp.folio_unlock
0.05 ± 95% -0.0 0.01 ±200% +0.1 0.18 ± 45% perf-profile.self.cycles-pp.irqtime_account_irq
0.06 ± 61% -0.0 0.02 ±124% +0.1 0.17 ± 35% perf-profile.self.cycles-pp.__get_next_timer_interrupt
0.08 ± 42% -0.0 0.04 ± 90% +0.2 0.25 ± 32% perf-profile.self.cycles-pp.update_irq_load_avg
0.08 ± 22% -0.0 0.04 ± 84% +0.2 0.28 ± 31% perf-profile.self.cycles-pp.fold_vm_numa_events
0.16 ± 29% -0.0 0.12 ± 31% -0.1 0.06 ± 90% perf-profile.self.cycles-pp.xas_start
0.05 ± 85% -0.0 0.02 ±156% +0.2 0.23 ± 30% perf-profile.self.cycles-pp.sync_regs
0.07 ± 38% -0.0 0.04 ± 88% +0.2 0.26 ± 28% perf-profile.self.cycles-pp.arch_scale_freq_tick
0.04 ± 86% -0.0 0.01 ±300% +0.2 0.19 ± 34% perf-profile.self.cycles-pp.tick_nohz_next_event
0.04 ± 97% -0.0 0.02 ±154% +0.2 0.19 ± 35% perf-profile.self.cycles-pp.rb_next
0.08 ± 48% -0.0 0.05 ± 88% +0.2 0.27 ± 42% perf-profile.self.cycles-pp.kmem_cache_alloc_noprof
3.68 ± 22% -0.0 3.66 ± 24% -3.3 0.36 ± 21% perf-profile.self.cycles-pp._copy_from_iter
0.02 ±126% -0.0 0.00 +0.2 0.20 ± 29% perf-profile.self.cycles-pp.update_process_times
0.04 ±110% -0.0 0.02 ±154% +0.1 0.14 ± 38% perf-profile.self.cycles-pp.cpuidle_reflect
0.05 ± 76% -0.0 0.03 ±127% +0.1 0.15 ± 46% perf-profile.self.cycles-pp._find_next_bit
0.04 ± 68% -0.0 0.02 ±129% +0.2 0.27 ± 29% perf-profile.self.cycles-pp.run_posix_cpu_timers
0.03 ± 82% -0.0 0.02 ±153% +0.1 0.18 ± 31% perf-profile.self.cycles-pp.sched_tick
0.02 ±153% -0.0 0.00 +0.1 0.12 ± 33% perf-profile.self.cycles-pp.irq_work_needs_cpu
0.07 ± 17% -0.0 0.05 ± 74% +0.2 0.30 ± 25% perf-profile.self.cycles-pp.next_uptodate_folio
0.07 ± 41% -0.0 0.06 ± 69% +0.3 0.40 ± 35% perf-profile.self.cycles-pp.update_sg_wakeup_stats
0.06 ± 81% -0.0 0.04 ± 91% +0.1 0.18 ± 32% perf-profile.self.cycles-pp.tsc_verify_tsc_adjust
0.01 ±203% -0.0 0.00 +0.1 0.11 ± 39% perf-profile.self.cycles-pp.irqentry_enter
0.01 ±203% -0.0 0.00 +0.1 0.12 ± 34% perf-profile.self.cycles-pp.__percpu_counter_sum
0.04 ±103% -0.0 0.03 ±132% +0.3 0.32 ± 28% perf-profile.self.cycles-pp.tick_check_oneshot_broadcast_this_cpu
0.02 ±153% -0.0 0.01 ±300% +0.2 0.24 ± 34% perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
0.02 ±123% -0.0 0.01 ±201% +0.1 0.14 ± 40% perf-profile.self.cycles-pp.error_entry
0.01 ±200% -0.0 0.00 +0.2 0.25 ± 28% perf-profile.self.cycles-pp.tmigr_requires_handle_remote
0.03 ±103% -0.0 0.02 ±123% +0.2 0.26 ± 25% perf-profile.self.cycles-pp.__hrtimer_run_queues
0.22 ± 34% -0.0 0.22 ± 32% +0.3 0.55 ± 24% perf-profile.self.cycles-pp.ktime_get
0.03 ±125% -0.0 0.02 ±153% +0.2 0.21 ± 34% perf-profile.self.cycles-pp.irqtime_account_process_tick
0.01 ±200% -0.0 0.01 ±299% +0.1 0.12 ± 46% perf-profile.self.cycles-pp.tmigr_quick_check
0.01 ±206% -0.0 0.01 ±299% +0.1 0.14 ± 37% perf-profile.self.cycles-pp.pm_qos_read_value
0.03 ±104% -0.0 0.03 ±100% +0.2 0.21 ± 31% perf-profile.self.cycles-pp.get_jiffies_update
0.01 ±300% -0.0 0.00 +0.1 0.12 ± 40% perf-profile.self.cycles-pp.__fdget_pos
0.01 ±299% -0.0 0.00 +0.1 0.13 ± 74% perf-profile.self.cycles-pp.hrtimer_interrupt
0.01 ±299% -0.0 0.00 +0.1 0.14 ± 25% perf-profile.self.cycles-pp._IO_fwrite
0.01 ±299% -0.0 0.00 +0.2 0.16 ± 43% perf-profile.self.cycles-pp.tick_nohz_handler
0.04 ± 66% -0.0 0.04 ±103% +0.2 0.19 ± 34% perf-profile.self.cycles-pp.sched_balance_domains
0.01 ±300% -0.0 0.01 ±299% +0.2 0.20 ± 24% perf-profile.self.cycles-pp.dyntick_save_progress_counter
0.00 +0.0 0.00 +0.1 0.10 ± 35% perf-profile.self.cycles-pp.task_tick_mm_cid
0.01 ±299% +0.0 0.01 ±300% +0.1 0.11 ± 38% perf-profile.self.cycles-pp.clockevents_program_event
0.00 +0.0 0.00 +0.1 0.11 ± 46% perf-profile.self.cycles-pp.perf_event_task_tick
0.01 ±206% +0.0 0.01 ±200% +0.1 0.14 ± 29% perf-profile.self.cycles-pp.mas_walk
0.00 +0.0 0.00 +0.1 0.14 ± 60% perf-profile.self.cycles-pp.irq_work_tick
0.00 +0.0 0.00 +0.2 0.21 ± 34% perf-profile.self.cycles-pp.check_cpu_stall
0.01 ±200% +0.0 0.01 ±200% +0.2 0.17 ± 37% perf-profile.self.cycles-pp.tick_irq_enter
0.11 ± 37% +0.0 0.11 ± 34% +0.2 0.29 ± 39% perf-profile.self.cycles-pp.get_cpu_device
0.00 +0.0 0.01 ±300% +0.1 0.13 ± 42% perf-profile.self.cycles-pp.hrtimer_update_next_event
0.01 ±299% +0.0 0.01 ±200% +0.1 0.14 ± 37% perf-profile.self.cycles-pp.rcu_pending
0.32 ± 57% +0.0 0.32 ± 28% +0.9 1.20 ± 22% perf-profile.self.cycles-pp.cpuidle_enter_state
0.01 ±299% +0.0 0.01 ±206% +0.1 0.13 ± 45% perf-profile.self.cycles-pp.check_tsc_unstable
0.02 ±220% +0.0 0.03 ±127% +0.1 0.13 ± 33% perf-profile.self.cycles-pp.memset_orig
0.07 ± 29% +0.0 0.08 ± 34% +0.1 0.19 ± 24% perf-profile.self.cycles-pp.update_curr
0.02 ±123% +0.0 0.04 ± 83% +0.1 0.10 ± 31% perf-profile.self.cycles-pp.set_next_entity
0.02 ±124% +0.0 0.04 ± 86% +0.2 0.22 ± 44% perf-profile.self.cycles-pp.filemap_map_pages
0.01 ±206% +0.0 0.03 ±124% +0.2 0.22 ± 18% perf-profile.self.cycles-pp.vma_interval_tree_insert
0.05 ± 54% +0.0 0.07 ± 57% +0.1 0.20 ± 39% perf-profile.self.cycles-pp.cpuidle_enter
0.06 ± 62% +0.0 0.08 ± 20% +0.2 0.27 ± 29% perf-profile.self.cycles-pp.ct_kernel_enter
0.10 ± 83% +0.0 0.13 ± 59% +0.3 0.43 ± 47% perf-profile.self.cycles-pp.delay_tsc
0.43 ± 47% +0.0 0.46 ± 26% +0.8 1.25 ± 14% perf-profile.self.cycles-pp.menu_select
0.02 ±129% +0.0 0.06 ± 47% +0.1 0.12 ± 55% perf-profile.self.cycles-pp.sched_clock_idle_wakeup_event
0.05 ± 69% +0.0 0.10 ± 46% +0.1 0.11 ± 40% perf-profile.self.cycles-pp.dequeue_task_fair
0.02 ±154% +0.1 0.07 ± 55% +0.1 0.10 ± 41% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.01 ±202% +0.1 0.07 ± 62% +0.2 0.18 ± 35% perf-profile.self.cycles-pp.switch_mm_irqs_off
0.10 ± 35% +0.1 0.16 ± 30% +0.1 0.22 ± 28% perf-profile.self.cycles-pp.update_load_avg
0.13 ± 27% +0.1 0.19 ± 13% +0.1 0.28 ± 32% perf-profile.self.cycles-pp.___perf_sw_event
0.09 ± 33% +0.1 0.15 ± 45% +0.1 0.21 ± 27% perf-profile.self.cycles-pp.clear_bhb_loop
0.03 ±125% +0.1 0.10 ± 20% +0.2 0.18 ± 28% perf-profile.self.cycles-pp.ct_kernel_exit_state
0.12 ± 40% +0.1 0.21 ± 24% +0.2 0.36 ± 31% perf-profile.self.cycles-pp.cpuidle_idle_call
0.06 ± 70% +0.1 0.16 ± 22% -0.0 0.04 ± 94% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.10 ± 58% +0.1 0.23 ± 23% +0.1 0.18 ± 28% perf-profile.self.cycles-pp.do_idle
1.24 ± 20% +0.2 1.39 ± 43% -1.1 0.10 ± 67% perf-profile.self.cycles-pp.mark_buffer_dirty
0.11 ± 44% +0.2 0.30 ± 30% +0.2 0.27 ± 32% perf-profile.self.cycles-pp.nr_iowait_cpu
0.14 ± 34% +0.2 0.32 ± 35% +0.3 0.42 ± 22% perf-profile.self.cycles-pp.update_rq_clock_task
0.19 ± 32% +0.2 0.39 ± 44% -0.1 0.10 ± 32% perf-profile.self.cycles-pp.filemap_get_entry
0.10 ± 70% +0.2 0.30 ± 47% +0.0 0.16 ± 26% perf-profile.self.cycles-pp.available_idle_cpu
0.10 ± 35% +0.3 0.39 ± 41% +0.1 0.23 ± 50% perf-profile.self.cycles-pp.update_rq_clock
0.02 ±154% +0.3 0.35 ± 46% +0.0 0.06 ± 74% perf-profile.self.cycles-pp.ttwu_do_activate
0.18 ± 62% +0.4 0.58 ± 29% +0.1 0.24 ± 17% perf-profile.self.cycles-pp.llist_reverse_order
0.28 ± 26% +0.4 0.69 ± 23% +0.4 0.63 ± 12% perf-profile.self.cycles-pp.native_sched_clock
0.21 ± 62% +0.5 0.68 ± 49% +0.1 0.29 ± 33% perf-profile.self.cycles-pp.call_function_single_prep_ipi
3.13 ± 39% +1.3 4.38 ± 40% -3.0 0.14 ± 29% perf-profile.self.cycles-pp.rwsem_spin_on_owner
0.75 ± 29% +1.9 2.68 ± 47% -0.5 0.23 ± 27% perf-profile.self.cycles-pp.copy_page_from_iter_atomic
0.89 ± 30% +7.4 8.29 ± 61% +2.3 3.18 ± 14% perf-profile.self.cycles-pp.poll_idle
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression
2024-08-13 6:47 ` Oliver Sang
@ 2024-08-13 6:49 ` Christoph Hellwig
2024-08-13 6:59 ` Oliver Sang
0 siblings, 1 reply; 7+ messages in thread
From: Christoph Hellwig @ 2024-08-13 6:49 UTC (permalink / raw)
To: Oliver Sang
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Anna Schumaker,
Sagi Grimberg, linux-nfs, ying.huang, feng.tang, fengwei.yin
On Tue, Aug 13, 2024 at 02:47:48PM +0800, Oliver Sang wrote:
> no in our tests, becomes even worse actually. detail data is in [1]
Thanks. Do you have a good summary of what operations this counts?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression
2024-08-13 6:49 ` Christoph Hellwig
@ 2024-08-13 6:59 ` Oliver Sang
2024-08-14 13:04 ` Christoph Hellwig
0 siblings, 1 reply; 7+ messages in thread
From: Oliver Sang @ 2024-08-13 6:59 UTC (permalink / raw)
To: Christoph Hellwig
Cc: oe-lkp, lkp, linux-kernel, Anna Schumaker, Sagi Grimberg,
linux-nfs, ying.huang, feng.tang, fengwei.yin, oliver.sang
hi, Christoph Hellwig,
On Tue, Aug 13, 2024 at 08:49:55AM +0200, Christoph Hellwig wrote:
> On Tue, Aug 13, 2024 at 02:47:48PM +0800, Oliver Sang wrote:
> > no in our tests, becomes even worse actually. detail data is in [1]
>
> Thanks. Do you have a good summary of what operations this counts?
sorry I don't have many details. not sure if https://github.com/filebench/filebench/wiki
is helpful for you?
and a raw data run by this test looks like below
2024-08-13 02:24:56 filebench -f /lkp/benchmarks/filebench/share/filebench/workloads/randomrw.f
Filebench Version 1.5-alpha3
0.000: Allocated 177MB of shared memory
0.001: Random RW Version 3.0 personality successfully loaded
0.001: Populating and pre-allocating filesets
0.001: Removing largefile1 tree (if exists)
0.002: Pre-allocating directories in largefile1 tree
0.119: Pre-allocating files in largefile1 tree
49.469: Waiting for pre-allocation to finish (in case of a parallel pre-allocation)
49.469: Population and pre-allocation of filesets completed
49.469: Starting 1 rand-rw instances
50.472: Running...
110.478: Run took 60 seconds...
110.478: Per-Operation Breakdown
rand-write1 11769ops 196ops/s 1.5mb/s 5.096ms/op [0.002ms - 1567.054ms]
rand-read1 49402ops 823ops/s 6.4mb/s 1.214ms/op [0.001ms - 1567.058ms]
110.478: IO Summary: 61171 ops 1019.414 ops/s 823/196 rd/wr 8.0mb/s 1.961ms/op
110.478: Shutting down processes
2024-08-13 02:26:48 sleep 100
/nfs/sdd1/largefile1
/nfs/sdd1/lost+found
/nfs/sdd1/wait_for_nfs_grace_period
2024-08-13 02:28:28 rm -rf /nfs/sdd1/largefile1 /nfs/sdd1/lost+found /nfs/sdd1/wait_for_nfs_grace_period
our bot parse it as:
{
"filebench.sum_operations": [
61171
],
"filebench.sum_operations/s": [
1019.414
],
"filebench.sum_reads/s": [
823
],
"filebench.sum_writes/s": [
196
],
"filebench.sum_bytes_mb/s": [
8.0
],
"filebench.sum_time_ms/op": [
1.961
]
}
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression
2024-08-13 6:59 ` Oliver Sang
@ 2024-08-14 13:04 ` Christoph Hellwig
2024-08-16 2:11 ` Oliver Sang
0 siblings, 1 reply; 7+ messages in thread
From: Christoph Hellwig @ 2024-08-14 13:04 UTC (permalink / raw)
To: Oliver Sang
Cc: Christoph Hellwig, oe-lkp, lkp, linux-kernel, Anna Schumaker,
Sagi Grimberg, linux-nfs, ying.huang, feng.tang, fengwei.yin
> sorry I don't have many details. not sure if https://github.com/filebench/filebench/wiki
> is helpful for you?
Not too much. Especially as I'm not sure what you are actually
running. If I run the workloads/randomrw.f from the filebench git
repository, it fails to run due to a lack of a run statement, and it
also hardcodes /tmp. Can you share the actual randomrw.f used for the
test?
Also do you run this test on other local file systems exported by
NFS, e.g. XFS and do you have numbers for that?
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression
2024-08-14 13:04 ` Christoph Hellwig
@ 2024-08-16 2:11 ` Oliver Sang
0 siblings, 0 replies; 7+ messages in thread
From: Oliver Sang @ 2024-08-16 2:11 UTC (permalink / raw)
To: Christoph Hellwig
Cc: oe-lkp, lkp, linux-kernel, Anna Schumaker, Sagi Grimberg,
linux-nfs, ying.huang, feng.tang, fengwei.yin, oliver.sang
hi, Christoph Hellwig,
On Wed, Aug 14, 2024 at 03:04:15PM +0200, Christoph Hellwig wrote:
> > sorry I don't have many details. not sure if https://github.com/filebench/filebench/wiki
> > is helpful for you?
>
> Not too much. Especially as I'm not sure what you are actually
> running. If I run the workloads/randomrw.f from the filebench git
> repository, it fails to run due to a lack of a run statement, and it
> also hardcodes /tmp. Can you share the actual randomrw.f used for the
> test?
please refer to
https://download.01.org/0day-ci/archive/20240808/202408081514.106c770e-oliver.sang@intel.com/repro-script
for our bot setup [1]
for the 'run statement' issue, you should append a such like
run 60
in workload file workloads/randomrw.f end part
(some workload files under https://github.com/filebench/filebench/blob/master/workloads/
have this, some don't)
[1]
dmsetup remove_all
wipefs -a --force /dev/sda1
mkfs -t ext4 -q -E lazy_itable_init=0,lazy_journal_init=0 -F /dev/sda1
mkdir -p /fs/sda1
mount -t ext4 /dev/sda1 /fs/sda1
mkdir /export
mount -t tmpfs nfsv4_root_export /export
mkdir -p /export//fs/sda1
mount --bind /fs/sda1 /export//fs/sda1
echo '/export//fs/sda1 *(rw,no_subtree_check,no_root_squash)' >> /etc/exports
systemctl restart rpcbind
systemctl restart rpc-statd
systemctl restart nfs-idmapd
systemctl restart nfs-server
mkdir -p /nfs/sda1
timeout 5m mount -t nfs -o vers=4 localhost:/fs/sda1 /nfs/sda1
touch /nfs/sda1/wait_for_nfs_grace_period
sync
echo 3 > /proc/sys/vm/drop_caches
for cpu_dir in /sys/devices/system/cpu/cpu[0-9]*
do
online_file="$cpu_dir"/online
[ -f "$online_file" ] && [ "$(cat "$online_file")" -eq 0 ] && continue
file="$cpu_dir"/cpufreq/scaling_governor
[ -f "$file" ] && echo "performance" > "$file"
done
filebench -f /lkp/benchmarks/filebench/share/filebench/workloads/randomrw.f
sleep 100
rm -rf /nfs/sda1/largefile1 /nfs/sda1/lost+found /nfs/sda1/wait_for_nfs_grace_period
>
> Also do you run this test on other local file systems exported by
> NFS, e.g. XFS and do you have numbers for that?
>
we tested the xfs, seems no big diff between 9aac777aaf and 49b29a573d.
39c910a430 seems have a drop.
=========================================================================================
compiler/cpufreq_governor/disk/fs2/fs/kconfig/rootfs/tbox_group/test/testcase:
gcc-12/performance/1HDD/nfsv4/xfs/x86_64-rhel-8.3/debian-12-x86_64-20240206.cgz/lkp-icl-2sp6/randomrw.f/filebench
<---- only change part is 'xfs'
commit:
9aac777aaf ("filemap: Convert generic_perform_write() to support large folios")
49b29a573d ("nfs: add support for large folios")
39c910a430 ("nfs: do not extend writes to the entire folio")
9aac777aaf945978 49b29a573da83b65d5f4ecf2db6 39c910a430370fd25d5b5e4b2f4
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
25513 ± 22% +2.7% 26206 ± 22% -19.2% 20609 ± 9% filebench.sum_operations/s
but all above data is not stable. kernel test bot won't recognize performance
changes based on this kind of data.
below detail data just FYI.
for 9aac777aaf ("filemap: Convert generic_perform_write() to support large folios")
"filebench.sum_operations/s": [
24749.259,
24963.29,
39646.08,
19061.09,
21232.461,
23361.606,
24028.835,
27065.077
],
for 49b29a573d ("nfs: add support for large folios")
"filebench.sum_operations/s": [
22241.08,
23600.1,
36988.03,
23380.36,
18751.434,
25665.28,
35146.827,
23874.923
],
for 39c910a430 ("nfs: do not extend writes to the entire folio")
"filebench.sum_operations/s": [
22756.036,
19669.426,
22478.429,
18800.486,
22682.041,
17167.844,
19160.962,
22163.603
],
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2024-08-16 2:11 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-08-08 7:35 [linus:master] [nfs] 49b29a573d: filebench.sum_operations/s -85.6% regression kernel test robot
2024-08-12 11:21 ` Christoph Hellwig
2024-08-13 6:47 ` Oliver Sang
2024-08-13 6:49 ` Christoph Hellwig
2024-08-13 6:59 ` Oliver Sang
2024-08-14 13:04 ` Christoph Hellwig
2024-08-16 2:11 ` Oliver Sang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox