Re: [LKP] [lkp] [xfs] 68a9f5e700: aim7.jobs-per-min -13.6% regression

Huang, Ying
2016-08-11T00:11:20Z

"Huang, Ying" <ying.huang@intel.com> writes:

> Hi, Linus,
>
> Linus Torvalds <torvalds@linux-foundation.org> writes:
>
>> On Wed, Aug 10, 2016 at 4:08 PM, Dave Chinner <david@fromorbit.com> wrote:
>>>
>>> That, to me, says there's a change in lock contention behaviour in
>>> the workload (which we know aim7 is good at exposing). i.e. the
>>> iomap change shifted contention from a sleeping lock to a spinning
>>> lock, or maybe we now trigger optimistic spinning behaviour on a
>>> lock we previously didn't spin on at all.
>>
>> Hmm. Possibly. I reacted to the lower cpu load number, but yeah, I
>> could easily imagine some locking primitive difference too.
>>
>>> We really need instruction level perf profiles to understand
>>> this - I don't have a machine with this many cpu cores available
>>> locally, so I'm not sure I'm going to be able to make any progress
>>> tracking it down in the short term. Maybe the lkp team has more
>>> in-depth cpu usage profiles they can share?
>>
>> Yeah, I've occasionally wanted to see some kind of "top-25 kernel
>> functions in the profile" thing. That said, when the load isn't all
>> that familiar, the profiles usually are not all that easy to make
>> sense of either. But comparing the before and after state might give
>> us clues.
>
> I have started perf-profile data collection, will send out the
> comparison result soon.

Here is the comparison result with perf-profile data.

=========================================================================================
compiler/cpufreq_governor/debug-setup/disk/fs/kconfig/load/rootfs/tbox_group/test/testcase:
gcc-6/performance/profile/1BRD_48G/xfs/x86_64-rhel/3000/debian-x86_64-2015-02-07.cgz/ivb44/disk_wrt/aim7

commit:
f0c6bcba74ac51cb77aadb33ad35cb2dc1ad1506
68a9f5e7007c1afa2cf6830b690a90d0187c0684

f0c6bcba74ac51cb 68a9f5e7007c1afa2cf6830b69
---------------- --------------------------
%stddev %change %stddev
\ | \
484435 ± 0% -13.3% 420004 ± 0% aim7.jobs-per-min
37.37 ± 0% +15.3% 43.09 ± 0% aim7.time.elapsed_time
37.37 ± 0% +15.3% 43.09 ± 0% aim7.time.elapsed_time.max
6491 ± 3% +30.8% 8491 ± 0% aim7.time.involuntary_context_switches
376.89 ± 0% +28.4% 484.11 ± 0% aim7.time.system_time
430512 ± 0% -20.1% 343838 ± 0% aim7.time.voluntary_context_switches
26816 ± 8% +10.2% 29542 ± 1% interrupts.CAL:Function_call_interrupts
125122 ± 10% -10.7% 111758 ± 12% softirqs.SCHED
24772 ± 0% -28.6% 17675 ± 0% vmstat.system.cs
53477 ± 2% +5.6% 56453 ± 0% vmstat.system.in
15627 ± 0% +27.7% 19956 ± 1% meminfo.Active(file)
16103 ± 3% +14.3% 18405 ± 8% meminfo.AnonHugePages
132898 ± 9% +15.4% 153380 ± 1% meminfo.DirectMap4k
13777 ± 5% +43.1% 19709 ± 0% meminfo.Shmem
3906 ± 0% +28.8% 5032 ± 2% proc-vmstat.nr_active_file
919.33 ± 5% +14.8% 1055 ± 8% proc-vmstat.nr_dirty
3444 ± 5% +41.8% 4884 ± 0% proc-vmstat.nr_shmem
4092 ± 14% +61.2% 6595 ± 1% proc-vmstat.pgactivate
1975 ± 15% +63.2% 3224 ± 17% slabinfo.scsi_data_buffer.active_objs
1975 ± 15% +63.2% 3224 ± 17% slabinfo.scsi_data_buffer.num_objs
464.33 ± 15% +63.3% 758.33 ± 17% slabinfo.xfs_efd_item.active_objs
464.33 ± 15% +63.3% 758.33 ± 17% slabinfo.xfs_efd_item.num_objs
1724300 ± 27% -40.5% 1025538 ± 1% sched_debug.cfs_rq:/.load.max
96.36 ± 3% +18.6% 114.32 ± 15% sched_debug.cfs_rq:/.util_avg.stddev
1724300 ± 27% -40.5% 1025538 ± 1% sched_debug.cpu.load.max
2887 ± 30% -28.2% 2073 ± 48% sched_debug.cpu.nr_load_updates.min
7.66 ± 20% -24.9% 5.75 ± 15% sched_debug.cpu.nr_uninterruptible.stddev
37.37 ± 0% +15.3% 43.09 ± 0% time.elapsed_time
37.37 ± 0% +15.3% 43.09 ± 0% time.elapsed_time.max
6491 ± 3% +30.8% 8491 ± 0% time.involuntary_context_switches
1037 ± 0% +10.8% 1148 ± 0% time.percent_of_cpu_this_job_got
376.89 ± 0% +28.4% 484.11 ± 0% time.system_time
430512 ± 0% -20.1% 343838 ± 0% time.voluntary_context_switches
24.18 ± 0% +9.0% 26.35 ± 0% turbostat.%Busy
686.00 ± 0% +9.5% 751.00 ± 0% turbostat.Avg_MHz
0.28 ± 0% -25.0% 0.21 ± 0% turbostat.CPU%c3
93.33 ± 1% +3.0% 96.15 ± 0% turbostat.CorWatt
124.61 ± 0% +2.1% 127.17 ± 0% turbostat.PkgWatt
4.74 ± 0% -2.7% 4.61 ± 1% turbostat.RAMWatt
7723 ± 0% +32.6% 10238 ± 5% numa-meminfo.node0.Active(file)
1589 ± 17% +45.5% 2313 ± 24% numa-meminfo.node0.Dirty
56052 ± 3% +58.2% 88666 ± 17% numa-meminfo.node1.Active
48142 ± 4% +64.0% 78943 ± 19% numa-meminfo.node1.Active(anon)
7908 ± 1% +22.9% 9722 ± 3% numa-meminfo.node1.Active(file)
46721 ± 3% +55.9% 72837 ± 24% numa-meminfo.node1.AnonPages
4789 ± 69% +102.3% 9687 ± 9% numa-meminfo.node1.Shmem
52991525 ± 1% -19.4% 42687208 ± 0% cpuidle.C1-IVT.time
319584 ± 1% -26.5% 234868 ± 1% cpuidle.C1-IVT.usage
3468808 ± 2% -19.8% 2783341 ± 3% cpuidle.C1E-IVT.time
46760 ± 0% -22.4% 36298 ± 0% cpuidle.C1E-IVT.usage
12590471 ± 0% -22.3% 9788585 ± 1% cpuidle.C3-IVT.time
79965 ± 0% -19.0% 64749 ± 0% cpuidle.C3-IVT.usage
1.3e+09 ± 0% +13.3% 1.473e+09 ± 0% cpuidle.C6-IVT.time
352.33 ± 8% -24.7% 265.33 ± 1% cpuidle.POLL.usage
1930 ± 0% +33.9% 2585 ± 3% numa-vmstat.node0.nr_active_file
4468 ± 7% -8.5% 4089 ± 5% numa-vmstat.node0.nr_alloc_batch
466.67 ± 4% +29.3% 603.33 ± 14% numa-vmstat.node0.nr_dirty
12026 ± 4% +64.1% 19734 ± 20% numa-vmstat.node1.nr_active_anon
1977 ± 1% +23.6% 2444 ± 1% numa-vmstat.node1.nr_active_file
3809 ± 6% +16.1% 4422 ± 4% numa-vmstat.node1.nr_alloc_batch
11671 ± 3% +55.9% 18197 ± 24% numa-vmstat.node1.nr_anon_pages
1197 ± 69% +102.3% 2422 ± 9% numa-vmstat.node1.nr_shmem
456.33 ± 57% -75.6% 111.33 ± 86% numa-vmstat.node1.nr_written
2.658e+11 ± 4% +24.7% 3.316e+11 ± 2% perf-stat.branch-instructions
0.41 ± 1% -9.1% 0.37 ± 1% perf-stat.branch-miss-rate
1.09e+09 ± 3% +13.4% 1.237e+09 ± 1% perf-stat.branch-misses
981138 ± 0% -18.1% 803696 ± 0% perf-stat.context-switches
1.511e+12 ± 5% +23.4% 1.864e+12 ± 3% perf-stat.cpu-cycles
102600 ± 1% -7.3% 95075 ± 1% perf-stat.cpu-migrations
0.26 ± 12% -30.8% 0.18 ± 10% perf-stat.dTLB-load-miss-rate
3.164e+11 ± 1% +39.9% 4.426e+11 ± 4% perf-stat.dTLB-loads
0.03 ± 26% -41.3% 0.02 ± 13% perf-stat.dTLB-store-miss-rate
2.247e+11 ± 6% +26.4% 2.839e+11 ± 2% perf-stat.dTLB-stores
1.49e+12 ± 4% +30.1% 1.939e+12 ± 2% perf-stat.instructions
43348 ± 2% +34.2% 58161 ± 12% perf-stat.instructions-per-iTLB-miss
0.99 ± 0% +5.5% 1.04 ± 0% perf-stat.ipc
262799 ± 0% +4.4% 274251 ± 1% perf-stat.minor-faults
34.12 ± 1% +2.1% 34.83 ± 0% perf-stat.node-load-miss-rate
46476754 ± 2% +4.6% 48601269 ± 1% perf-stat.node-load-misses
9.96 ± 0% +13.4% 11.30 ± 0% perf-stat.node-store-miss-rate
24460859 ± 1% +14.4% 27971097 ± 1% perf-stat.node-store-misses
262780 ± 0% +4.4% 274227 ± 1% perf-stat.page-faults
11.31 ± 1% -18.1% 9.27 ± 0% perf-profile.cycles-pp.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
0.00 ± -1% +Inf% 1.68 ± 1% perf-profile.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
1.80 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__add_to_page_cache_locked.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
2.55 ± 3% -14.2% 2.19 ± 2% perf-profile.cycles-pp.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin
0.00 ± -1% +Inf% 4.45 ± 1% perf-profile.cycles-pp.__block_commit_write.isra.24.block_write_end.generic_write_end.iomap_write_actor.iomap_apply
5.93 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__block_commit_write.isra.24.block_write_end.generic_write_end.xfs_vm_write_end.generic_perform_write
13.71 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
10.36 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
0.00 ± -1% +Inf% 3.64 ± 0% perf-profile.cycles-pp.__block_write_begin_int.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
1.04 ± 2% -18.9% 0.84 ± 1% perf-profile.cycles-pp.__delete_from_page_cache.delete_from_page_cache.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final
11.24 ± 2% -18.1% 9.21 ± 0% perf-profile.cycles-pp.__dentry_kill.dput.__fput.____fput.task_work_run
11.31 ± 2% -18.1% 9.26 ± 0% perf-profile.cycles-pp.__fput.____fput.task_work_run.exit_to_usermode_loop.syscall_return_slowpath
0.00 ± -1% +Inf% 1.09 ± 2% perf-profile.cycles-pp.__mark_inode_dirty.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
1.32 ± 4% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__mark_inode_dirty.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
0.00 ± -1% +Inf% 2.68 ± 2% perf-profile.cycles-pp.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
3.04 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
1.00 ± 1% -18.0% 0.82 ± 1% perf-profile.cycles-pp.__radix_tree_lookup.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.grab_cache_page_write_begin
1.12 ± 2% -17.6% 0.92 ± 4% perf-profile.cycles-pp.__sb_start_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
1.38 ± 2% -13.3% 1.19 ± 1% perf-profile.cycles-pp.__set_page_dirty.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end
54.10 ± 1% +13.1% 61.20 ± 0% perf-profile.cycles-pp.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
6.34 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin
0.00 ± -1% +Inf% 3.69 ± 1% perf-profile.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
4.02 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
0.98 ± 5% -100.0% 0.00 ± -1% perf-profile.cycles-pp.alloc_page_buffers.create_empty_buffers.create_page_buffers.__block_write_begin_int.__block_write_begin
0.00 ± -1% +Inf% 2.56 ± 2% perf-profile.cycles-pp.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
2.91 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles-pp.alloc_pages_current.__page_cache_alloc.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
3.42 ± 0% -20.9% 2.71 ± 2% perf-profile.cycles-pp.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final
0.00 ± -1% +Inf% 4.69 ± 0% perf-profile.cycles-pp.block_write_end.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
6.24 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.block_write_end.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
19.18 ± 5% -9.3% 17.40 ± 0% perf-profile.cycles-pp.call_cpuidle.cpu_startup_entry.start_secondary
0.94 ± 4% -19.8% 0.76 ± 0% perf-profile.cycles-pp.cancel_dirty_page.try_to_free_buffers.xfs_vm_releasepage.try_to_release_page.block_invalidatepage
3.95 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.copy_user_enhanced_fast_string.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
0.00 ± -1% +Inf% 3.22 ± 0% perf-profile.cycles-pp.copy_user_enhanced_fast_string.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
19.75 ± 5% -9.8% 17.81 ± 0% perf-profile.cycles-pp.cpu_startup_entry.start_secondary
19.18 ± 5% -9.3% 17.40 ± 0% perf-profile.cycles-pp.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
18.45 ± 5% -9.2% 16.75 ± 0% perf-profile.cycles-pp.cpuidle_enter_state.cpuidle_enter.call_cpuidle.cpu_startup_entry.start_secondary
1.44 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles-pp.create_empty_buffers.create_page_buffers.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin
0.00 ± -1% +Inf% 1.18 ± 1% perf-profile.cycles-pp.create_empty_buffers.create_page_buffers.__block_write_begin_int.iomap_write_begin.iomap_write_actor
1.86 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.create_page_buffers.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write
0.00 ± -1% +Inf% 1.53 ± 1% perf-profile.cycles-pp.create_page_buffers.__block_write_begin_int.iomap_write_begin.iomap_write_actor.iomap_apply
1.74 ± 2% -19.9% 1.40 ± 3% perf-profile.cycles-pp.delete_from_page_cache.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict
1.27 ± 0% -22.5% 0.99 ± 4% perf-profile.cycles-pp.destroy_inode.evict.iput.__dentry_kill.dput
2.61 ± 1% -24.3% 1.98 ± 1% perf-profile.cycles-pp.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
2.66 ± 1% -24.3% 2.01 ± 1% perf-profile.cycles-pp.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
1.79 ± 2% -28.2% 1.28 ± 3% perf-profile.cycles-pp.do_unlinkat.sys_unlink.entry_SYSCALL_64_fastpath
1.07 ± 3% -23.3% 0.82 ± 3% perf-profile.cycles-pp.down_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
1.01 ± 3% -17.9% 0.83 ± 2% perf-profile.cycles-pp.down_write.xfs_ilock.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
11.26 ± 2% -18.1% 9.23 ± 0% perf-profile.cycles-pp.dput.__fput.____fput.task_work_run.exit_to_usermode_loop
11.21 ± 2% -18.1% 9.18 ± 0% perf-profile.cycles-pp.evict.iput.__dentry_kill.dput.__fput
11.34 ± 2% -18.1% 9.29 ± 0% perf-profile.cycles-pp.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
0.00 ± -1% +Inf% 1.55 ± 3% perf-profile.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor
1.83 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write
43.95 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
0.00 ± -1% +Inf% 7.91 ± 1% perf-profile.cycles-pp.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
10.68 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
1.91 ± 3% -16.4% 1.59 ± 1% perf-profile.cycles-pp.get_page_from_freelist.__alloc_pages_nodemask.alloc_pages_current.__page_cache_alloc.pagecache_get_page
0.00 ± -1% +Inf% 9.85 ± 0% perf-profile.cycles-pp.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
10.96 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter
0.00 ± -1% +Inf% 52.29 ± 0% perf-profile.cycles-pp.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
0.00 ± -1% +Inf% 52.94 ± 0% perf-profile.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
0.00 ± -1% +Inf% 34.35 ± 0% perf-profile.cycles-pp.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter
0.00 ± -1% +Inf% 16.48 ± 0% perf-profile.cycles-pp.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
11.22 ± 2% -18.1% 9.19 ± 0% perf-profile.cycles-pp.iput.__dentry_kill.dput.__fput.____fput
0.00 ± -1% +Inf% 1.55 ± 1% perf-profile.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin
1.72 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.lru_cache_add.add_to_page_cache_lru.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
0.00 ± -1% +Inf% 2.78 ± 0% perf-profile.cycles-pp.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end.iomap_write_actor
3.39 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.mark_buffer_dirty.__block_commit_write.isra.24.block_write_end.generic_write_end.xfs_vm_write_end
0.00 ± -1% +Inf% 3.44 ± 1% perf-profile.cycles-pp.mark_page_accessed.iomap_write_actor.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
3.03 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.memset_erms.__block_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
0.00 ± -1% +Inf% 2.43 ± 0% perf-profile.cycles-pp.memset_erms.iomap_write_begin.iomap_write_actor.iomap_apply.iomap_file_buffered_write
0.00 ± -1% +Inf% 9.25 ± 0% perf-profile.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.iomap_write_begin.iomap_write_actor.iomap_apply
10.37 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write
2.58 ± 1% -24.1% 1.96 ± 0% perf-profile.cycles-pp.path_openat.do_filp_open.do_sys_open.sys_creat.entry_SYSCALL_64_fastpath
1.17 ± 3% -100.0% 0.00 ± -1% perf-profile.cycles-pp.radix_tree_lookup_slot.find_get_entry.pagecache_get_page.grab_cache_page_write_begin.xfs_vm_write_begin
2.06 ± 3% -22.5% 1.60 ± 2% perf-profile.cycles-pp.rw_verify_area.vfs_write.sys_write.entry_SYSCALL_64_fastpath
1.79 ± 3% -22.2% 1.39 ± 0% perf-profile.cycles-pp.security_file_permission.rw_verify_area.vfs_write.sys_write.entry_SYSCALL_64_fastpath
1.32 ± 4% -21.4% 1.04 ± 0% perf-profile.cycles-pp.selinux_file_permission.security_file_permission.rw_verify_area.vfs_write.sys_write
19.79 ± 5% -9.9% 17.84 ± 0% perf-profile.cycles-pp.start_secondary
2.67 ± 1% -24.2% 2.02 ± 1% perf-profile.cycles-pp.sys_creat.entry_SYSCALL_64_fastpath
1.79 ± 3% -27.9% 1.29 ± 3% perf-profile.cycles-pp.sys_unlink.entry_SYSCALL_64_fastpath
60.98 ± 1% +9.5% 66.76 ± 0% perf-profile.cycles-pp.sys_write.entry_SYSCALL_64_fastpath
11.34 ± 1% -18.1% 9.29 ± 0% perf-profile.cycles-pp.syscall_return_slowpath.entry_SYSCALL_64_fastpath
11.32 ± 1% -18.0% 9.28 ± 0% perf-profile.cycles-pp.task_work_run.exit_to_usermode_loop.syscall_return_slowpath.entry_SYSCALL_64_fastpath
5.96 ± 1% -20.0% 4.77 ± 0% perf-profile.cycles-pp.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput
9.89 ± 2% -17.4% 8.17 ± 0% perf-profile.cycles-pp.truncate_inode_pages_final.evict.iput.__dentry_kill.dput
9.87 ± 2% -17.5% 8.15 ± 0% perf-profile.cycles-pp.truncate_inode_pages_range.truncate_inode_pages_final.evict.iput.__dentry_kill
2.07 ± 1% -20.4% 1.65 ± 2% perf-profile.cycles-pp.try_to_free_buffers.xfs_vm_releasepage.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage
2.40 ± 1% -21.0% 1.89 ± 2% perf-profile.cycles-pp.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range
0.00 ± -1% +Inf% 1.36 ± 1% perf-profile.cycles-pp.unlock_page.generic_write_end.iomap_write_actor.iomap_apply.iomap_file_buffered_write
1.72 ± 4% -100.0% 0.00 ± -1% perf-profile.cycles-pp.unlock_page.generic_write_end.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write
59.63 ± 1% +10.2% 65.72 ± 0% perf-profile.cycles-pp.vfs_write.sys_write.entry_SYSCALL_64_fastpath
0.00 ± -1% +Inf% 1.52 ± 2% perf-profile.cycles-pp.workingset_activation.mark_page_accessed.iomap_write_actor.iomap_apply.iomap_file_buffered_write
0.00 ± -1% +Inf% 1.73 ± 1% perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply
0.00 ± -1% +Inf% 1.97 ± 2% perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
0.00 ± -1% +Inf% 1.61 ± 2% perf-profile.cycles-pp.xfs_bmap_search_extents.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin
0.00 ± -1% +Inf% 1.24 ± 2% perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin
0.00 ± -1% +Inf% 1.46 ± 1% perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply
0.00 ± -1% +Inf% 1.21 ± 2% perf-profile.cycles-pp.xfs_bmap_search_multi_extents.xfs_bmap_search_extents.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay
1.25 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_bmapi_delay.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int
0.00 ± -1% +Inf% 3.06 ± 1% perf-profile.cycles-pp.xfs_bmapi_delay.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
1.04 ± 0% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_bmapi_read.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin
0.00 ± -1% +Inf% 3.04 ± 1% perf-profile.cycles-pp.xfs_bmapi_read.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
0.00 ± -1% +Inf% 3.05 ± 1% perf-profile.cycles-pp.xfs_bmapi_read.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply
1.32 ± 2% -21.5% 1.04 ± 1% perf-profile.cycles-pp.xfs_create.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat
51.83 ± 1% +14.3% 59.25 ± 0% perf-profile.cycles-pp.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write.sys_write
0.00 ± -1% +Inf% 16.05 ± 0% perf-profile.cycles-pp.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write.xfs_file_write_iter
53.16 ± 1% +13.6% 60.40 ± 0% perf-profile.cycles-pp.xfs_file_write_iter.__vfs_write.vfs_write.sys_write.entry_SYSCALL_64_fastpath
1.24 ± 1% -23.1% 0.95 ± 4% perf-profile.cycles-pp.xfs_fs_destroy_inode.destroy_inode.evict.iput.__dentry_kill
1.42 ± 2% -21.2% 1.12 ± 1% perf-profile.cycles-pp.xfs_generic_create.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open
6.46 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_get_blocks.__block_write_begin_int.__block_write_begin.xfs_vm_write_begin.generic_perform_write
1.29 ± 3% -18.9% 1.04 ± 1% perf-profile.cycles-pp.xfs_ilock.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write.vfs_write
0.00 ± -1% +Inf% 1.14 ± 3% perf-profile.cycles-pp.xfs_ilock.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
1.21 ± 1% -23.4% 0.93 ± 4% perf-profile.cycles-pp.xfs_inactive.xfs_fs_destroy_inode.destroy_inode.evict.iput
1.23 ± 4% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.6.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int
0.00 ± -1% +Inf% 4.14 ± 0% perf-profile.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.8.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write
3.28 ± 2% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_iomap_write_delay.__xfs_get_blocks.xfs_get_blocks.__block_write_begin_int.__block_write_begin
0.00 ± -1% +Inf% 9.08 ± 0% perf-profile.cycles-pp.xfs_iomap_write_delay.xfs_file_iomap_begin.iomap_apply.iomap_file_buffered_write.xfs_file_buffered_aio_write
3.54 ± 0% -20.8% 2.81 ± 1% perf-profile.cycles-pp.xfs_vm_invalidatepage.truncate_inode_page.truncate_inode_pages_range.truncate_inode_pages_final.evict
2.35 ± 1% -21.0% 1.86 ± 1% perf-profile.cycles-pp.xfs_vm_releasepage.try_to_release_page.block_invalidatepage.xfs_vm_invalidatepage.truncate_inode_page
25.10 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_vm_write_begin.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
11.03 ± 1% -100.0% 0.00 ± -1% perf-profile.cycles-pp.xfs_vm_write_end.generic_perform_write.xfs_file_buffered_aio_write.xfs_file_write_iter.__vfs_write
1.42 ± 2% -20.7% 1.13 ± 1% perf-profile.cycles-pp.xfs_vn_create.path_openat.do_filp_open.do_sys_open.sys_creat
1.42 ± 2% -20.5% 1.13 ± 1% perf-profile.cycles-pp.xfs_vn_mknod.xfs_vn_create.path_openat.do_filp_open.do_sys_open
2.27 ± 1% -10.6% 2.03 ± 0% perf-profile.func.cycles-pp.___might_sleep
2.49 ± 0% -34.5% 1.63 ± 1% perf-profile.func.cycles-pp.__block_commit_write.isra.24
1.51 ± 2% +15.4% 1.75 ± 1% perf-profile.func.cycles-pp.__block_write_begin_int
1.79 ± 4% -16.8% 1.49 ± 1% perf-profile.func.cycles-pp.__mark_inode_dirty
1.32 ± 0% -16.4% 1.10 ± 1% perf-profile.func.cycles-pp.__radix_tree_lookup
1.08 ± 2% -100.0% 0.00 ± -1% perf-profile.func.cycles-pp.__xfs_get_blocks
1.16 ± 0% -18.1% 0.95 ± 1% perf-profile.func.cycles-pp._raw_spin_lock
3.96 ± 2% -18.4% 3.23 ± 0% perf-profile.func.cycles-pp.copy_user_enhanced_fast_string
1.41 ± 3% -20.6% 1.12 ± 3% perf-profile.func.cycles-pp.entry_SYSCALL_64_fastpath
1.30 ± 2% -100.0% 0.00 ± -1% perf-profile.func.cycles-pp.generic_perform_write
1.31 ± 2% -46.7% 0.70 ± 0% perf-profile.func.cycles-pp.generic_write_end
18.43 ± 5% -9.1% 16.76 ± 0% perf-profile.func.cycles-pp.intel_idle
0.00 ± -1% +Inf% 1.12 ± 1% perf-profile.func.cycles-pp.iomap_write_actor
1.50 ± 1% -20.9% 1.19 ± 1% perf-profile.func.cycles-pp.mark_buffer_dirty
0.00 ± -1% +Inf% 1.91 ± 1% perf-profile.func.cycles-pp.mark_page_accessed
3.24 ± 0% -19.8% 2.60 ± 0% perf-profile.func.cycles-pp.memset_erms
1.75 ± 2% -18.9% 1.42 ± 1% perf-profile.func.cycles-pp.unlock_page
1.16 ± 1% -21.6% 0.91 ± 1% perf-profile.func.cycles-pp.vfs_write
0.37 ± 2% +243.6% 1.26 ± 2% perf-profile.func.cycles-pp.xfs_bmap_search_extents
0.41 ± 1% +198.4% 1.22 ± 2% perf-profile.func.cycles-pp.xfs_bmap_search_multi_extents
0.70 ± 5% +219.5% 2.24 ± 0% perf-profile.func.cycles-pp.xfs_bmapi_read
1.05 ± 2% -15.6% 0.88 ± 3% perf-profile.func.cycles-pp.xfs_file_write_iter
0.64 ± 1% +182.8% 1.81 ± 4% perf-profile.func.cycles-pp.xfs_iext_bno_to_ext
0.00 ± -1% +Inf% 1.10 ± 3% perf-profile.func.cycles-pp.xfs_iomap_eof_want_preallocate.constprop.8
0.46 ± 4% +161.6% 1.20 ± 1% perf-profile.func.cycles-pp.xfs_iomap_write_delay

Best Regards,
Huang, Ying