* [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression
@ 2023-08-10 13:24 kernel test robot
2023-08-11 1:11 ` Chen Yu
0 siblings, 1 reply; 30+ messages in thread
From: kernel test robot @ 2023-08-10 13:24 UTC (permalink / raw)
To: Peter Zijlstra
Cc: oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, ying.huang,
feng.tang, fengwei.yin, aubrey.li, yu.c.chen, oliver.sang
Hello,
kernel test robot noticed a -34.8% regression of phoronix-test-suite.blogbench.Write.final_score on:
commit: e0c2ff903c320d3fd3c2c604dc401b3b7c0a1d13 ("sched/fair: Remove sched_feat(START_DEBIT)")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/eevdf
testcase: phoronix-test-suite
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
parameters:
test: blogbench-1.1.0
option_a: Write
cpufreq_governor: performance
(
previously, we reported
"[tip:sched/eevdf] [sched/fair] e0c2ff903c: pft.faults_per_sec_per_cpu 7.0% improvement"
on
https://lore.kernel.org/all/202308091624.d97ae058-oliver.sang@intel.com/
since now we found a regression, so report again FYI
)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202308101628.7af4631a-oliver.sang@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20230810/202308101628.7af4631a-oliver.sang@intel.com
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/rootfs/tbox_group/test/testcase:
gcc-12/performance/x86_64-rhel-8.3/Write/debian-x86_64-phoronix/lkp-csl-2sp7/blogbench-1.1.0/phoronix-test-suite
commit:
af4cf40470 ("sched/fair: Add cfs_rq::avg_vruntime")
e0c2ff903c ("sched/fair: Remove sched_feat(START_DEBIT)")
af4cf40470c22efa e0c2ff903c320d3fd3c2c604dc4
---------------- ---------------------------
%stddev %change %stddev
\ | \
13.43 ± 6% +1.7 15.15 ± 5% mpstat.cpu.all.idle%
4.516e+09 ± 7% +13.6% 5.129e+09 ± 6% cpuidle..time
5386162 ± 6% +15.1% 6199901 ± 5% cpuidle..usage
5829408 ± 7% -8.7% 5320418 ± 4% numa-numastat.node0.local_node
5839930 ± 7% -8.7% 5333078 ± 4% numa-numastat.node0.numa_hit
6025065 ± 5% -12.9% 5245696 ± 7% numa-numastat.node1.local_node
57086 -28.2% 40989 vmstat.io.bo
11120354 -21.6% 8721798 vmstat.memory.cache
18750 +12.1% 21014 vmstat.system.cs
215070 ± 25% +29.4% 278379 ± 23% numa-meminfo.node1.AnonPages.max
5507703 ± 13% -31.6% 3766183 ± 23% numa-meminfo.node1.FilePages
2118171 ± 13% -26.7% 1551936 ± 25% numa-meminfo.node1.Inactive
6824965 ± 12% -26.7% 5005740 ± 18% numa-meminfo.node1.MemUsed
4960 -34.8% 3235 ± 2% phoronix-test-suite.blogbench.Write.final_score
35120853 -29.4% 24804986 phoronix-test-suite.time.file_system_outputs
8058 -1.9% 7908 phoronix-test-suite.time.percent_of_cpu_this_job_got
26517 -1.2% 26196 phoronix-test-suite.time.system_time
1445969 +10.6% 1599011 phoronix-test-suite.time.voluntary_context_switches
600079 ± 3% +26.4% 758665 ± 3% turbostat.C1E
0.75 ± 3% +0.1 0.84 ± 4% turbostat.C1E%
4372028 ± 7% +14.1% 4987180 ± 7% turbostat.C6
13.09 ± 6% +1.6 14.72 ± 6% turbostat.C6%
11.39 ± 5% +10.4% 12.57 ± 4% turbostat.CPU%c1
2330913 ± 13% -23.5% 1782895 ± 14% numa-vmstat.node0.nr_dirtied
2256547 ± 13% -23.8% 1719197 ± 14% numa-vmstat.node0.nr_written
5840154 ± 7% -8.7% 5333324 ± 4% numa-vmstat.node0.numa_hit
5829632 ± 7% -8.7% 5320664 ± 4% numa-vmstat.node0.numa_local
2287685 ± 11% -33.8% 1514622 ± 20% numa-vmstat.node1.nr_dirtied
1376658 ± 13% -31.6% 941500 ± 23% numa-vmstat.node1.nr_file_pages
127.17 ± 18% -52.4% 60.50 ± 31% numa-vmstat.node1.nr_writeback
2215841 ± 12% -34.1% 1460272 ± 20% numa-vmstat.node1.nr_written
6025232 ± 5% -12.9% 5245703 ± 7% numa-vmstat.node1.numa_local
74.24 ± 8% -38.1% 45.97 ± 11% sched_debug.cfs_rq:/.load_avg.avg
3167 ± 14% -78.6% 676.56 ± 45% sched_debug.cfs_rq:/.load_avg.max
339.56 ± 15% -74.2% 87.77 ± 38% sched_debug.cfs_rq:/.load_avg.stddev
1829 ± 6% -12.7% 1597 ± 9% sched_debug.cfs_rq:/.runnable_avg.max
1490 ± 7% -13.5% 1289 ± 7% sched_debug.cfs_rq:/.util_avg.max
1495 ± 7% -13.3% 1296 ± 8% sched_debug.cfs_rq:/.util_est_enqueued.max
33772 +15.4% 38971 sched_debug.cpu.nr_switches.avg
117253 ± 10% +106.6% 242187 ± 13% sched_debug.cpu.nr_switches.max
11889 ± 9% +96.8% 23399 ± 12% sched_debug.cpu.nr_switches.stddev
58611259 -50.0% 29305629 sched_debug.sysctl_sched.sysctl_sched_features
6885022 ± 2% -25.3% 5139844 ± 2% meminfo.Active
6735239 ± 2% -25.9% 4988483 ± 2% meminfo.Active(file)
10388292 -21.3% 8173968 meminfo.Cached
3953695 -11.9% 3483739 meminfo.Inactive
2793179 -16.8% 2323187 meminfo.Inactive(file)
716630 -21.7% 561030 meminfo.KReclaimable
12938216 -18.5% 10538792 meminfo.Memused
716630 -21.7% 561030 meminfo.SReclaimable
462556 -10.4% 414369 meminfo.SUnreclaim
1179187 -17.3% 975399 meminfo.Slab
13543613 -19.3% 10926031 meminfo.max_used_kB
1682930 ± 2% -25.9% 1246741 ± 2% proc-vmstat.nr_active_file
4618598 -28.6% 3297498 ± 2% proc-vmstat.nr_dirtied
115523 ± 2% -4.3% 110605 proc-vmstat.nr_dirty
2596313 -21.3% 2043401 proc-vmstat.nr_file_pages
698013 -16.8% 580677 proc-vmstat.nr_inactive_file
179075 -21.7% 140220 proc-vmstat.nr_slab_reclaimable
115576 -10.4% 103600 proc-vmstat.nr_slab_unreclaimable
200.00 ± 12% -60.2% 79.67 ± 27% proc-vmstat.nr_writeback
4472388 -28.9% 3179453 ± 2% proc-vmstat.nr_written
1682930 ± 2% -25.9% 1246741 ± 2% proc-vmstat.nr_zone_active_file
698013 -16.8% 580677 proc-vmstat.nr_zone_inactive_file
115652 ± 2% -4.4% 110571 proc-vmstat.nr_zone_write_pending
11888053 -10.8% 10598323 ± 2% proc-vmstat.numa_hit
11861225 -10.9% 10573027 ± 2% proc-vmstat.numa_local
3434596 -28.4% 2460772 proc-vmstat.pgactivate
18752650 -8.2% 17219808 ± 3% proc-vmstat.pgalloc_normal
18252683 -8.4% 16727157 ± 3% proc-vmstat.pgfree
19379562 -27.5% 14053587 ± 2% proc-vmstat.pgpgout
205988 +3.5% 213118 proc-vmstat.pgreuse
97.12 -1.1 96.06 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
97.08 -1.1 96.02 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
52.33 -1.0 51.36 perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
52.32 -1.0 51.35 perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
97.12 -1.1 96.06 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
97.08 -1.1 96.02 perf-profile.children.cycles-pp.do_syscall_64
85.21 -1.0 84.18 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
85.58 -1.0 84.56 perf-profile.children.cycles-pp._raw_spin_lock
52.33 -1.0 51.36 perf-profile.children.cycles-pp.__x64_sys_openat
52.32 -1.0 51.35 perf-profile.children.cycles-pp.do_sys_openat2
0.64 ± 2% +0.0 0.68 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.28 ± 5% +0.0 0.32 ± 5% perf-profile.children.cycles-pp.__irq_exit_rcu
0.02 ±141% +0.1 0.07 ± 15% perf-profile.children.cycles-pp.__x64_sys_rename
0.08 ± 29% +0.1 0.14 ± 19% perf-profile.children.cycles-pp.process_one_work
0.01 ±223% +0.1 0.07 ± 15% perf-profile.children.cycles-pp.do_renameat2
0.08 ± 29% +0.1 0.15 ± 20% perf-profile.children.cycles-pp.worker_thread
0.03 ±101% +0.1 0.10 ± 21% perf-profile.children.cycles-pp.__extent_writepage
0.03 ±103% +0.1 0.11 ± 20% perf-profile.children.cycles-pp.do_writepages
0.03 ±103% +0.1 0.11 ± 20% perf-profile.children.cycles-pp.extent_writepages
0.03 ±103% +0.1 0.11 ± 20% perf-profile.children.cycles-pp.extent_write_cache_pages
84.73 -1.0 83.69 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
5.487e+09 -2.2% 5.364e+09 perf-stat.i.branch-instructions
0.73 ± 8% +0.2 0.92 ± 12% perf-stat.i.branch-miss-rate%
18867 +12.0% 21136 perf-stat.i.context-switches
2.307e+11 -2.0% 2.261e+11 perf-stat.i.cpu-cycles
1352 ± 2% +3.5% 1399 perf-stat.i.cpu-migrations
4638095 ± 27% -38.2% 2865514 ± 23% perf-stat.i.dTLB-load-misses
6.107e+09 -2.3% 5.965e+09 perf-stat.i.dTLB-loads
2.391e+10 -2.3% 2.335e+10 perf-stat.i.instructions
0.17 -7.8% 0.16 ± 2% perf-stat.i.ipc
0.62 ± 2% +14.5% 0.71 ± 6% perf-stat.i.major-faults
2.40 -2.0% 2.35 perf-stat.i.metric.GHz
139.98 -2.3% 136.76 perf-stat.i.metric.M/sec
66.90 -1.8 65.11 perf-stat.i.node-load-miss-rate%
65029001 -5.5% 61464545 ± 3% perf-stat.i.node-load-misses
0.08 ± 27% -0.0 0.05 ± 24% perf-stat.overall.dTLB-load-miss-rate%
67.28 -2.3 64.93 perf-stat.overall.node-load-miss-rate%
5.474e+09 -2.2% 5.352e+09 perf-stat.ps.branch-instructions
18821 +12.1% 21105 perf-stat.ps.context-switches
2.301e+11 -2.0% 2.256e+11 perf-stat.ps.cpu-cycles
1349 ± 2% +3.6% 1398 perf-stat.ps.cpu-migrations
4630176 ± 27% -38.2% 2859589 ± 23% perf-stat.ps.dTLB-load-misses
6.092e+09 -2.3% 5.952e+09 perf-stat.ps.dTLB-loads
2.385e+10 -2.3% 2.33e+10 perf-stat.ps.instructions
0.61 ± 2% +14.6% 0.70 ± 6% perf-stat.ps.major-faults
64859061 -5.5% 61314167 ± 3% perf-stat.ps.node-load-misses
8.031e+12 -1.4% 7.921e+12 perf-stat.total.instructions
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 30+ messages in thread* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-10 13:24 [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression kernel test robot @ 2023-08-11 1:11 ` Chen Yu 2023-08-11 2:42 ` Chen Yu 2023-08-14 12:49 ` Peter Zijlstra 0 siblings, 2 replies; 30+ messages in thread From: Chen Yu @ 2023-08-11 1:11 UTC (permalink / raw) To: Peter Zijlstra Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On 2023-08-10 at 21:24:37 +0800, kernel test robot wrote: > > > Hello, > > kernel test robot noticed a -34.8% regression of phoronix-test-suite.blogbench.Write.final_score on: > > > commit: e0c2ff903c320d3fd3c2c604dc401b3b7c0a1d13 ("sched/fair: Remove sched_feat(START_DEBIT)") > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/eevdf > > testcase: phoronix-test-suite > test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory > parameters: > > test: blogbench-1.1.0 > option_a: Write > cpufreq_governor: performance > It seems that commit e0c2ff903c32 removed the sched_feat(START_DEBIT) for initial task, but also increases the vruntime for non-initial task: Before the e0c2ff903c32, the vruntime for a enqueued task is: cfs_rq->min_vruntime After the e0c2ff903c32, the vruntime for a enqueued task is: avg_vruntime(cfs_rq) = \Sum v_i * w_i / W = \Sum v_i / nr_tasks which is usually higher than cfs_rq->min_vruntime, and we give less sleep bonus to the wakee, which could bring more or less impact to different workloads. But since later we switched to lag based placement, this new vruntime will minus lag, which could mitigate this problem. thanks, Chenyu ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-11 1:11 ` Chen Yu @ 2023-08-11 2:42 ` Chen Yu 2023-08-14 13:29 ` Peter Zijlstra 2023-08-14 12:49 ` Peter Zijlstra 1 sibling, 1 reply; 30+ messages in thread From: Chen Yu @ 2023-08-11 2:42 UTC (permalink / raw) To: Peter Zijlstra Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar Hi Peter, On 2023-08-11 at 09:11:21 +0800, Chen Yu wrote: > On 2023-08-10 at 21:24:37 +0800, kernel test robot wrote: > > > > > > Hello, > > > > kernel test robot noticed a -34.8% regression of phoronix-test-suite.blogbench.Write.final_score on: > > > > > > commit: e0c2ff903c320d3fd3c2c604dc401b3b7c0a1d13 ("sched/fair: Remove sched_feat(START_DEBIT)") > > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/eevdf > > > > testcase: phoronix-test-suite > > test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory > > parameters: > > > > test: blogbench-1.1.0 > > option_a: Write > > cpufreq_governor: performance > > > > It seems that commit e0c2ff903c32 removed the sched_feat(START_DEBIT) for initial > task, but also increases the vruntime for non-initial task: > Before the e0c2ff903c32, the vruntime for a enqueued task is: > cfs_rq->min_vruntime > After the e0c2ff903c32, the vruntime for a enqueued task is: > avg_vruntime(cfs_rq) = \Sum v_i * w_i / W > = \Sum v_i / nr_tasks > which is usually higher than cfs_rq->min_vruntime, and we give less sleep bonus to > the wakee, which could bring more or less impact to different workloads. > But since later we switched to lag based placement, this new vruntime will minus > lag, which could mitigate this problem. > > Since previously lkp has reported that with eevdf policy enabled, there was a regression in hackbench, I did some experiments and found that, with eevdf enabled there are more preemptions, and this preemption could slow down the waker(each waker could wakes up 20 wakee in hackbench). The reason might be that, check_preempt_wakeup() is easier to preempt the current task in eevdf: baseline(no eevdf): sched/fair: Add cfs_rq::avg_vruntime compare(eevdf): sched/smp: Use lag to simplify cross-runqueue placement hackbench -g 112 -f 20 --threads -l 30000 -s 100 on a system with 1 socket 56C/112T online 1. Use the following bpf script to track the preemption count within 10 seconds: tracepoint:sched:sched_switch { if (args->prev_state == TASK_RUNNING) { @preempt = count(); } } baseline: bpftrace preemption.bt Attaching 4 probes... 10:54:45 Preemption count: @preempt: 2409638 compare: bpftrace preemption.bt Attaching 4 probes... 10:02:21 Preemption count: @preempt: 12147709 There are much more preemptions with eevdf enabled. 2. Add the following schedstats to track the ratio of successful preemption in check_preempt_wakeup(): diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 57e8bc14b06e..dfd4a6ebdf23 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8294,6 +8294,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ int next_buddy_marked = 0; int cse_is_idle, pse_is_idle; + schedstat_inc(rq->check_preempt_count); + if (unlikely(se == pse)) return; @@ -8358,8 +8360,12 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ /* * XXX pick_eevdf(cfs_rq) != se ? */ - if (pick_eevdf(cfs_rq) == pse) + if (pick_eevdf(cfs_rq) == pse) { + if (se->vruntime <= pse->vruntime + sysctl_sched_wakeup_granularity) + schedstat_inc(rq->low_gran_preempt_count); + goto preempt; + } return; } @@ -8377,6 +8383,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ return; preempt: + schedstat_inc(rq->need_preempt_count); + resched_curr(rq); /* * Only set the backward buddy when the current task is still diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index aa5b293ca4ed..58abd3d53f1d 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1128,6 +1128,9 @@ struct rq { /* try_to_wake_up() stats */ unsigned int ttwu_count; unsigned int ttwu_local; + unsigned int check_preempt_count; + unsigned int need_preempt_count; + unsigned int low_gran_preempt_count; #endif #ifdef CONFIG_CPU_IDLE diff --git a/kernel/sched/stats.c b/kernel/sched/stats.c index 857f837f52cb..99392cad0c07 100644 --- a/kernel/sched/stats.c +++ b/kernel/sched/stats.c @@ -133,12 +133,14 @@ static int show_schedstat(struct seq_file *seq, void *v) /* runqueue-specific stats */ seq_printf(seq, - "cpu%d %u 0 %u %u %u %u %llu %llu %lu", + "cpu%d %u 0 %u %u %u %u %llu %llu %lu %u %u %u", cpu, rq->yld_count, rq->sched_count, rq->sched_goidle, rq->ttwu_count, rq->ttwu_local, rq->rq_cpu_time, - rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount); + rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount, + rq->check_preempt_count, rq->need_preempt_count, + rq->low_gran_preempt_count); seq_printf(seq, "\n"); -- 2.25.1 Without eevdf enabled, the /proc/schedstat delta within 5 seconds on CPU8 is: Thu Aug 10 11:02:02 2023 cpu8 .stats.check_preempt_count 51973 <----- .stats.need_preempt_count 10514 <----- .stats.rq_cpu_time 5004068598 .stats.rq_sched_info.pcount 60374 .stats.rq_sched_info.run_delay 80405664582 .stats.sched_count 60609 .stats.sched_goidle 227 .stats.ttwu_count 56250 .stats.ttwu_local 14619 The preemption success ration is 10514 / 51973 = 20.23% ----------------------------------------------------------------------------- With eevdf enabled, the /proc/schedstat delta within 5 seconds on CPU8 is: Thu Aug 10 10:22:55 2023 cpu8 .stats.check_preempt_count 71673 <---- .stats.low_gran_preempt_count 57410 .stats.need_preempt_count 57413 <---- .stats.rq_cpu_time 5007778990 .stats.rq_sched_info.pcount 129233 .stats.rq_sched_info.run_delay 164830921362 .stats.sched_count 129233 .stats.ttwu_count 70222 .stats.ttwu_local 66847 The preemption success ration is 57413 / 71673 = 80.10% According to the low_gran_preempt_count, most successfully preemption happens when the current->vruntime is smaller than wakee->vruntime + sysctl_sched_wakeup_granularity, which will not happen in current cfs's wakeup_preempt_entity(). It seems that, eevdf does not inhit the wakeup preemption as much as cfs, and maybe it is because eevdf needs to consider fairness more? thanks, Chenyu ^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-11 2:42 ` Chen Yu @ 2023-08-14 13:29 ` Peter Zijlstra 2023-08-14 18:32 ` Mike Galbraith 2023-08-16 3:40 ` Chen Yu 0 siblings, 2 replies; 30+ messages in thread From: Peter Zijlstra @ 2023-08-14 13:29 UTC (permalink / raw) To: Chen Yu Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Mike Galbraith On Fri, Aug 11, 2023 at 10:42:09AM +0800, Chen Yu wrote: > Since previously lkp has reported that with eevdf policy enabled, there was > a regression in hackbench, I did some experiments and found that, with eevdf > enabled there are more preemptions, and this preemption could slow down > the waker(each waker could wakes up 20 wakee in hackbench). The reason might > be that, check_preempt_wakeup() is easier to preempt the current task in eevdf: This is true. > Without eevdf enabled, the /proc/schedstat delta within 5 seconds on CPU8 is: > Thu Aug 10 11:02:02 2023 cpu8 > .stats.check_preempt_count 51973 <----- > .stats.need_preempt_count 10514 <----- > .stats.rq_cpu_time 5004068598 > .stats.rq_sched_info.pcount 60374 > .stats.rq_sched_info.run_delay 80405664582 > .stats.sched_count 60609 > .stats.sched_goidle 227 > .stats.ttwu_count 56250 > .stats.ttwu_local 14619 > > The preemption success ration is 10514 / 51973 = 20.23% > ----------------------------------------------------------------------------- > > With eevdf enabled, the /proc/schedstat delta within 5 seconds on CPU8 is: > Thu Aug 10 10:22:55 2023 cpu8 > .stats.check_preempt_count 71673 <---- > .stats.low_gran_preempt_count 57410 > .stats.need_preempt_count 57413 <---- > .stats.rq_cpu_time 5007778990 > .stats.rq_sched_info.pcount 129233 > .stats.rq_sched_info.run_delay 164830921362 > .stats.sched_count 129233 > .stats.ttwu_count 70222 > .stats.ttwu_local 66847 > > The preemption success ration is 57413 / 71673 = 80.10% note: wakeup-preemption > According to the low_gran_preempt_count, most successfully preemption happens > when the current->vruntime is smaller than wakee->vruntime + sysctl_sched_wakeup_granularity, > which will not happen in current cfs's wakeup_preempt_entity(). > > It seems that, eevdf does not inhit the wakeup preemption as much as cfs, and > maybe it is because eevdf needs to consider fairness more? Not fairness, latency. Because it wants to honour the virtual deadline. Are these wakeup preemptions typically on runqueues that have only a single other task? That is, consider a single task running, then avg_vruntime will be it's vruntime, because the average of one variable must be the value of that one variable. Then the moment a second task joins, we get two options: - positive lag - negative lag When the new task has negative lag, it gets placed to the right of the currently running task (and avg_vruntime has a forward discontinuity). At this point the new task is not eligible and does not get to run. When the new task has positive lag, it gets placed to the left of the currently running task (and avg_vruntime has a backward discontinuity). At this point the currently running task is no longer eligible, and the new task must be selected -- irrespective of it's deadline. The paper doesn't (AFAIR) consider the case of wake-up-preemption explicitly. It only considers task selection and vruntime placement. One option I suppose would be to gate the wakeup preemption by virtual deadline, only allow when the new task has an earlier deadline than the currently running one, and otherwise rely on tick preemption. NOTE: poking at wakeup preemption is a double edged sword, some workloads love it, some hate it. Touching it is bound to upset the balance -- again. (also, did I get that the right way around? -- I've got a Monday brain that isn't willing to boot properly) --- diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index fe5be91c71c7..16d24e5dda8f 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8047,6 +8047,15 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ cfs_rq = cfs_rq_of(se); update_curr(cfs_rq); + if (sched_feat(WAKEUP_DEADLINE)) { + /* + * Only allow preemption if the virtual deadline of the new + * task is before the virtual deadline of the existing task. + */ + if (deadline_gt(deadline, pse, se)) + return; + } + /* * XXX pick_eevdf(cfs_rq) != se ? */ diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 61bcbf5e46a4..e733981b32aa 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -24,6 +24,7 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true) * Allow wakeup-time preemption of the current task: */ SCHED_FEAT(WAKEUP_PREEMPTION, true) +SCHED_FEAT(WAKEUP_DEADLINE, true) SCHED_FEAT(HRTICK, false) SCHED_FEAT(HRTICK_DL, false) ^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-14 13:29 ` Peter Zijlstra @ 2023-08-14 18:32 ` Mike Galbraith 2023-08-15 23:52 ` Peter Zijlstra 2023-08-16 12:37 ` Peter Zijlstra 2023-08-16 3:40 ` Chen Yu 1 sibling, 2 replies; 30+ messages in thread From: Mike Galbraith @ 2023-08-14 18:32 UTC (permalink / raw) To: Peter Zijlstra, Chen Yu Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Mon, 2023-08-14 at 15:29 +0200, Peter Zijlstra wrote: > > --- > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index fe5be91c71c7..16d24e5dda8f 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -8047,6 +8047,15 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ > cfs_rq = cfs_rq_of(se); > update_curr(cfs_rq); > > + if (sched_feat(WAKEUP_DEADLINE)) { > + /* > + * Only allow preemption if the virtual deadline of the new > + * task is before the virtual deadline of the existing task. > + */ > + if (deadline_gt(deadline, pse, se)) > + return; > + } > + > /* > * XXX pick_eevdf(cfs_rq) != se ? > */ > diff --git a/kernel/sched/features.h b/kernel/sched/features.h > index 61bcbf5e46a4..e733981b32aa 100644 > --- a/kernel/sched/features.h > +++ b/kernel/sched/features.h > @@ -24,6 +24,7 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true) > * Allow wakeup-time preemption of the current task: > */ > SCHED_FEAT(WAKEUP_PREEMPTION, true) > +SCHED_FEAT(WAKEUP_DEADLINE, true) > > SCHED_FEAT(HRTICK, false) > SCHED_FEAT(HRTICK_DL, false) I don't have that phoronix thingy, but the above didn't seem to do anything for hackbench here. I whack eevdf with the stick below to dampen its scheduling enthusiasm a little, and it did help the hackbench deficit some. (yup, it's all rob Peter to pay Paul.. this game has no winner:) hackbench -l 10000 avg 6.4.10-cfs 5.575 5.565 5.575 5.571 v cfs 6.4.10-eevdf 6.106 6.078 6.103 6.095 1.094 5.888 5.890 5.889 5.889 1.057 +RUN_TO_PARITY --- kernel/sched/fair.c | 6 ++++++ kernel/sched/features.h | 1 + 2 files changed, 7 insertions(+) --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -875,6 +875,12 @@ static struct sched_entity *pick_eevdf(s if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) curr = NULL; + /* + * Once selected, run the task to parity to avoid overscheduling. + */ + if (sched_feat(RUN_TO_PARITY) && curr) + return curr; + while (node) { struct sched_entity *se = __node_2_se(node); --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -7,6 +7,7 @@ SCHED_FEAT(PLACE_LAG, true) SCHED_FEAT(PLACE_FUDGE, true) SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) +SCHED_FEAT(RUN_TO_PARITY, true) /* * Prefer to schedule the task we woke last (assuming it failed ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-14 18:32 ` Mike Galbraith @ 2023-08-15 23:52 ` Peter Zijlstra 2023-08-16 3:54 ` Mike Galbraith 2023-08-16 12:37 ` Peter Zijlstra 1 sibling, 1 reply; 30+ messages in thread From: Peter Zijlstra @ 2023-08-15 23:52 UTC (permalink / raw) To: Mike Galbraith Cc: Chen Yu, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: > I don't have that phoronix thingy, but the above didn't seem to do > anything for hackbench here. Yeah, looking at it this is because the lag and the request/slice are on the same order of things. More thinking required, but perhaps not at 2am. > I whack eevdf with the stick below to dampen its scheduling enthusiasm > a little, and it did help the hackbench deficit some. I'm thinking there's an argument to be had the current WAKEUP_PREEMPTION knob is at the wrong place and should be more or less where you put the triddle at. ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-15 23:52 ` Peter Zijlstra @ 2023-08-16 3:54 ` Mike Galbraith 0 siblings, 0 replies; 30+ messages in thread From: Mike Galbraith @ 2023-08-16 3:54 UTC (permalink / raw) To: Peter Zijlstra Cc: Chen Yu, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Wed, 2023-08-16 at 01:52 +0200, Peter Zijlstra wrote: > On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: > > > I don't have that phoronix thingy, but the above didn't seem to do > > anything for hackbench here. > > Yeah, looking at it this is because the lag and the request/slice are on > the same order of things. > > More thinking required, but perhaps not at 2am. > > > I whack eevdf with the stick below to dampen its scheduling enthusiasm > > a little, and it did help the hackbench deficit some. > > I'm thinking there's an argument to be had the current WAKEUP_PREEMPTION > knob is at the wrong place and should be more or less where you put the > triddle at. FWIW, the full twiddle also incorporates your XXX wakeup preempt check as well, which turned too harsh into local keeper. It turned out that someone at SUSE likes blogbench enough to package it, so I installed it and took it for a spin. I don't have near enough box to be running the thing, the resulting numbers aren't interesting, but I think the perf sched summaries below are, show that preemption adjust of some sort (less mallet like) would help that load. -Mike for i in `seq 3` do rm -r /usr/local/blogbench/* blogbench -i 5 -r 10 -d /usr/local/blogbench 2>&1|grep Final done Full warmup, repeat for numbers, then perf sched record an instance. 6.4.10-cfs avg Final score for writes: 4834 4900 4945 4893 Final score for reads : 189432 202455 199513 197133 6.4.10-eevdf virgin vs cfs Final score for writes: 3729* 4354 4309 4130 .844 Final score for reads : 171679* 210588 192533 191600 .971 6.4.10-eevdf WAKEUP_DEADLINE Final score for writes: 3890* 4319 4473 4227 .863 Final score for reads : 200479 245808* 209308 218531 1.108 6.4.10-eevdf WAKEUP_DEADLINE XXX Final score for writes: 4913 4912 4993 4939 1.009 Final score for reads : 204454 198320 204883 202552 1.027 6.4.10-eevdf RUN_TO_PARITY (full hacklet below) Final score for writes: 4942 4786 4788 4838 .988 Final score for reads : 199547 226007 203031 209528 1.062 6.4.10-cfs ---------------------------------------------------------------------------------------------------------- Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms | ---------------------------------------------------------------------------------------------------------- blogbench:4184 | 18113.843 ms | 11542 | avg: 0.295 ms | max: 40.177 ms | sum: 3401.177 ms | blogbench:4185 | 16225.865 ms | 10951 | avg: 0.276 ms | max: 39.698 ms | sum: 3021.575 ms | blogbench:4181 | 16348.799 ms | 10064 | avg: 0.285 ms | max: 39.266 ms | sum: 2871.963 ms | blogbench:4176 | 15557.432 ms | 8826 | avg: 0.381 ms | max: 37.174 ms | sum: 3366.757 ms | blogbench:4182 | 19597.634 ms | 12680 | avg: 0.289 ms | max: 36.001 ms | sum: 3669.867 ms | blogbench:4179 | 17484.541 ms | 10899 | avg: 0.327 ms | max: 32.517 ms | sum: 3565.309 ms | blogbench:4183 | 15369.613 ms | 11474 | avg: 0.256 ms | max: 32.484 ms | sum: 2934.823 ms | blogbench:4180 | 17114.917 ms | 7941 | avg: 0.408 ms | max: 32.041 ms | sum: 3242.867 ms | blogbench:4175 | 10835.899 ms | 3204 | avg: 0.356 ms | max: 29.878 ms | sum: 1141.257 ms | blogbench:4173 | 11471.599 ms | 2938 | avg: 0.392 ms | max: 28.987 ms | sum: 1153.089 ms | ---------------------------------------------------------------------------------------------------------- TOTAL: | 265634.961 ms | 506626 | | 40.177 ms | 49808.651 ms | ---------------------------------------------------------------------------------------------------------- 6.4.10-eevdf virgin ---------------------------------------------------------------------------------------------------------- Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms | ---------------------------------------------------------------------------------------------------------- blogbench:6759 | 4969.658 ms | 3906 | avg: 0.732 ms | max: 27.500 ms | sum: 2860.151 ms | blogbench:6770 | 23275.253 ms | 14462 | avg: 0.321 ms | max: 20.348 ms | sum: 4642.321 ms | blogbench:6764 | 10269.476 ms | 3651 | avg: 0.444 ms | max: 20.153 ms | sum: 1621.928 ms | blogbench:6762 | 10399.828 ms | 3555 | avg: 0.475 ms | max: 20.087 ms | sum: 1689.091 ms | blogbench:6776 | 21011.106 ms | 15849 | avg: 0.290 ms | max: 19.485 ms | sum: 4598.075 ms | kworker/u16:1-e:5339 | 5160.247 ms | 82402 | avg: 0.056 ms | max: 19.024 ms | sum: 4626.026 ms | blogbench:6760 | 4907.866 ms | 3742 | avg: 0.716 ms | max: 18.101 ms | sum: 2679.361 ms | blogbench:6758 | 4937.399 ms | 3963 | avg: 0.677 ms | max: 17.450 ms | sum: 2682.504 ms | blogbench:6767 | 18588.108 ms | 10497 | avg: 0.430 ms | max: 17.372 ms | sum: 4515.504 ms | blogbench:6771 | 19485.111 ms | 10453 | avg: 0.391 ms | max: 17.009 ms | sum: 4084.894 ms | ---------------------------------------------------------------------------------------------------------- TOTAL: | 296257.549 ms | 586072 | | 27.500 ms | 78716.789 ms | ---------------------------------------------------------------------------------------------------------- 6.4.10-eevdf WAKEUP_DEADLINE ---------------------------------------------------------------------------------------------------------- Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms | ---------------------------------------------------------------------------------------------------------- blogbench:7137 | 25101.783 ms | 14411 | avg: 0.310 ms | max: 41.585 ms | sum: 4472.840 ms | blogbench:7143 | 22564.665 ms | 13603 | avg: 0.311 ms | max: 41.051 ms | sum: 4235.135 ms | blogbench:7138 | 23607.049 ms | 14775 | avg: 0.276 ms | max: 24.185 ms | sum: 4075.245 ms | blogbench:7133 | 10642.876 ms | 3114 | avg: 0.594 ms | max: 21.304 ms | sum: 1849.157 ms | blogbench:7126 | 4900.276 ms | 3188 | avg: 0.855 ms | max: 21.100 ms | sum: 2725.992 ms | blogbench:7131 | 10280.962 ms | 3209 | avg: 0.591 ms | max: 20.345 ms | sum: 1897.020 ms | kworker/1:0H-kb:5226 | 1059.870 ms | 89053 | avg: 0.019 ms | max: 18.226 ms | sum: 1657.880 ms | blogbench:7140 | 22845.026 ms | 13193 | avg: 0.345 ms | max: 18.215 ms | sum: 4548.112 ms | blogbench:7141 | 22977.420 ms | 14292 | avg: 0.330 ms | max: 17.376 ms | sum: 4717.075 ms | blogbench:7135 | 22996.398 ms | 12226 | avg: 0.337 ms | max: 16.682 ms | sum: 4118.248 ms | ---------------------------------------------------------------------------------------------------------- TOTAL: | 312915.682 ms | 571062 | | 41.585 ms | 81987.387 ms | ---------------------------------------------------------------------------------------------------------- 6.4.10-eevdf WAKEUP_DEADLINE XXX ---------------------------------------------------------------------------------------------------------- Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms | ---------------------------------------------------------------------------------------------------------- blogbench:4462 | 19047.513 ms | 10408 | avg: 0.374 ms | max: 23.974 ms | sum: 3894.010 ms | blogbench:4464 | 18542.851 ms | 8309 | avg: 0.461 ms | max: 22.786 ms | sum: 3833.045 ms | blogbench:4454 | 5168.071 ms | 1892 | avg: 0.774 ms | max: 19.453 ms | sum: 1463.996 ms | blogbench:4469 | 18260.551 ms | 9593 | avg: 0.390 ms | max: 18.606 ms | sum: 3742.130 ms | blogbench:4467 | 18285.308 ms | 9095 | avg: 0.444 ms | max: 18.176 ms | sum: 4041.126 ms | blogbench:4461 | 16543.137 ms | 8470 | avg: 0.432 ms | max: 18.155 ms | sum: 3656.709 ms | blogbench:4470 | 16395.913 ms | 8358 | avg: 0.454 ms | max: 17.316 ms | sum: 3795.385 ms | kworker/u16:6-e:67 | 3952.031 ms | 33202 | avg: 0.101 ms | max: 17.238 ms | sum: 3344.452 ms | blogbench:4453 | 5281.302 ms | 1755 | avg: 0.872 ms | max: 16.601 ms | sum: 1530.829 ms | blogbench:4465 | 17635.375 ms | 6986 | avg: 0.551 ms | max: 16.107 ms | sum: 3848.489 ms | ---------------------------------------------------------------------------------------------------------- TOTAL: | 272453.638 ms | 427507 | | 23.974 ms | 68015.737 ms | ---------------------------------------------------------------------------------------------------------- 6.4.10-eevdf RUN_TO_PARITY ---------------------------------------------------------------------------------------------------------- Task | Runtime ms | Switches | Avg delay ms | Max delay ms | Sum delay ms | ---------------------------------------------------------------------------------------------------------- kworker/7:2H-tt:4421 | 360.323 ms | 35601 | avg: 0.040 ms | max: 29.541 ms | sum: 1433.946 ms | kthreadd:4527 | 3680.986 ms | 33028 | avg: 0.095 ms | max: 29.180 ms | sum: 3147.411 ms | blogbench:4520 | 17032.741 ms | 7149 | avg: 0.360 ms | max: 20.841 ms | sum: 2575.967 ms | blogbench:4518 | 16433.272 ms | 6425 | avg: 0.462 ms | max: 20.177 ms | sum: 2967.848 ms | blogbench:4517 | 16930.930 ms | 7137 | avg: 0.443 ms | max: 18.468 ms | sum: 3158.274 ms | blogbench:4515 | 17921.101 ms | 6469 | avg: 0.454 ms | max: 18.389 ms | sum: 2937.433 ms | kworker/u16:4-e:65 | 4764.526 ms | 36137 | avg: 0.099 ms | max: 17.966 ms | sum: 3571.866 ms | blogbench:4523 | 18151.172 ms | 9356 | avg: 0.306 ms | max: 17.855 ms | sum: 2866.256 ms | blogbench:4506 | 5139.457 ms | 1653 | avg: 0.945 ms | max: 17.768 ms | sum: 1562.387 ms | blogbench:4524 | 16104.410 ms | 7232 | avg: 0.414 ms | max: 15.995 ms | sum: 2992.634 ms | ---------------------------------------------------------------------------------------------------------- TOTAL: | 269989.455 ms | 410693 | | 29.541 ms | 62477.554 ms | ---------------------------------------------------------------------------------------------------------- WAKEUP_DEADLINE XXX --- kernel/sched/fair.c | 10 ++++++++++ kernel/sched/features.h | 1 + 2 files changed, 11 insertions(+) --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7959,6 +7959,16 @@ static void check_preempt_wakeup(struct cfs_rq = cfs_rq_of(se); update_curr(cfs_rq); + if (sched_feat(WAKEUP_DEADLINE)) { + /* + * Only allow preemption if the virtual deadline of the next + * task in line is earlier than that of the current task. + */ + if (deadline_gt(deadline, se, pick_eevdf(cfs_rq))) + goto preempt; + return; + } + /* * XXX pick_eevdf(cfs_rq) != se ? */ --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -25,6 +25,7 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true) * Allow wakeup-time preemption of the current task: */ SCHED_FEAT(WAKEUP_PREEMPTION, true) +SCHED_FEAT(WAKEUP_DEADLINE, true) SCHED_FEAT(HRTICK, false) SCHED_FEAT(HRTICK_DL, false) RUN_TO_PARITY hacklet. --- kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/features.h | 1 + 2 files changed, 13 insertions(+) --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -875,6 +875,12 @@ static struct sched_entity *pick_eevdf(s if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) curr = NULL; + /* + * Once selected, run the task to parity to avoid overscheduling. + */ + if (sched_feat(RUN_TO_PARITY) && curr) + return curr; + while (node) { struct sched_entity *se = __node_2_se(node); @@ -7959,6 +7965,12 @@ static void check_preempt_wakeup(struct cfs_rq = cfs_rq_of(se); update_curr(cfs_rq); + if (sched_feat(RUN_TO_PARITY)) { + if (pick_eevdf(cfs_rq) != se) + goto preempt; + return; + } + /* * XXX pick_eevdf(cfs_rq) != se ? */ --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -7,6 +7,7 @@ SCHED_FEAT(PLACE_LAG, true) SCHED_FEAT(PLACE_FUDGE, true) SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) +SCHED_FEAT(RUN_TO_PARITY, true) /* * Prefer to schedule the task we woke last (assuming it failed ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-14 18:32 ` Mike Galbraith 2023-08-15 23:52 ` Peter Zijlstra @ 2023-08-16 12:37 ` Peter Zijlstra 2023-08-16 13:40 ` Peter Zijlstra 1 sibling, 1 reply; 30+ messages in thread From: Peter Zijlstra @ 2023-08-16 12:37 UTC (permalink / raw) To: Mike Galbraith Cc: Chen Yu, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -875,6 +875,12 @@ static struct sched_entity *pick_eevdf(s > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > curr = NULL; > > + /* > + * Once selected, run the task to parity to avoid overscheduling. > + */ > + if (sched_feat(RUN_TO_PARITY) && curr) > + return curr; > + > while (node) { > struct sched_entity *se = __node_2_se(node); > So I read it wrong last night... but I rather like this idea. But there's something missing. When curr starts a new slice it should probably do a full repick and not stick with it. Let me poke at this a bit.. nice ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-16 12:37 ` Peter Zijlstra @ 2023-08-16 13:40 ` Peter Zijlstra 2023-08-16 15:38 ` Mike Galbraith ` (2 more replies) 0 siblings, 3 replies; 30+ messages in thread From: Peter Zijlstra @ 2023-08-16 13:40 UTC (permalink / raw) To: Mike Galbraith Cc: Chen Yu, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Wed, Aug 16, 2023 at 02:37:16PM +0200, Peter Zijlstra wrote: > On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: > > > --- a/kernel/sched/fair.c > > +++ b/kernel/sched/fair.c > > @@ -875,6 +875,12 @@ static struct sched_entity *pick_eevdf(s > > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > > curr = NULL; > > > > + /* > > + * Once selected, run the task to parity to avoid overscheduling. > > + */ > > + if (sched_feat(RUN_TO_PARITY) && curr) > > + return curr; > > + > > while (node) { > > struct sched_entity *se = __node_2_se(node); > > > > So I read it wrong last night... but I rather like this idea. But > there's something missing. When curr starts a new slice it should > probably do a full repick and not stick with it. > > Let me poke at this a bit.. nice Something like so.. it shouldn't matter much now, but might make a difference once we start mixing different slice lengths. --- kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/features.h | 1 + 2 files changed, 13 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index fe5be91c71c7..128a78f3f264 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -873,6 +873,13 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq) if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) curr = NULL; + /* + * Once selected, run a task until it either becomes non-eligible or + * until it gets a new slice. See the HACK in set_next_entity(). + */ + if (sched_feat(RUN_TO_PARITY) && curr && curr->vlag == curr->deadline) + return curr; + while (node) { struct sched_entity *se = __node_2_se(node); @@ -5168,6 +5175,11 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) update_stats_wait_end_fair(cfs_rq, se); __dequeue_entity(cfs_rq, se); update_load_avg(cfs_rq, se, UPDATE_TG); + /* + * HACK, stash a copy of deadline at the point of pick in vlag, + * which isn't used until dequeue. + */ + se->vlag = se->deadline; } update_stats_curr_start(cfs_rq, se); diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 61bcbf5e46a4..f770168230ae 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -6,6 +6,7 @@ */ SCHED_FEAT(PLACE_LAG, true) SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) +SCHED_FEAT(RUN_TO_PARITY, true) /* * Prefer to schedule the task we woke last (assuming it failed ^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-16 13:40 ` Peter Zijlstra @ 2023-08-16 15:38 ` Mike Galbraith 2023-08-16 20:04 ` Peter Zijlstra 2023-08-17 15:10 ` [tip: sched/core] sched/eevdf: Curb wakeup-preemption tip-bot2 for Peter Zijlstra 2023-08-18 1:09 ` [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression Chen Yu 2 siblings, 1 reply; 30+ messages in thread From: Mike Galbraith @ 2023-08-16 15:38 UTC (permalink / raw) To: Peter Zijlstra Cc: Chen Yu, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Wed, 2023-08-16 at 15:40 +0200, Peter Zijlstra wrote: > On Wed, Aug 16, 2023 at 02:37:16PM +0200, Peter Zijlstra wrote: > > On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: > > > > > --- a/kernel/sched/fair.c > > > +++ b/kernel/sched/fair.c > > > @@ -875,6 +875,12 @@ static struct sched_entity *pick_eevdf(s > > > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > > > curr = NULL; > > > > > > + /* > > > + * Once selected, run the task to parity to avoid overscheduling. > > > + */ > > > + if (sched_feat(RUN_TO_PARITY) && curr) > > > + return curr; > > > + > > > while (node) { > > > struct sched_entity *se = __node_2_se(node); > > > > > > > So I read it wrong last night... but I rather like this idea. But > > there's something missing. When curr starts a new slice it should > > probably do a full repick and not stick with it. > > > > Let me poke at this a bit.. nice > > Something like so.. it shouldn't matter much now, but might make a > difference once we start mixing different slice lengths. Hm, that stash the deadline trick _seems_ to have cured the reason I was inspired to added that XXX hunk.. no 'ew, that's a tad harsh' latency penalty in sight <knocks wood>. Here's hoping test bots don't have a cow. > > --- > kernel/sched/fair.c | 12 ++++++++++++ > kernel/sched/features.h | 1 + > 2 files changed, 13 insertions(+) > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index fe5be91c71c7..128a78f3f264 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -873,6 +873,13 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq) > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > curr = NULL; > > + /* > + * Once selected, run a task until it either becomes non-eligible or > + * until it gets a new slice. See the HACK in set_next_entity(). > + */ > + if (sched_feat(RUN_TO_PARITY) && curr && curr->vlag == curr->deadline) > + return curr; > + > while (node) { > struct sched_entity *se = __node_2_se(node); > > @@ -5168,6 +5175,11 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) > update_stats_wait_end_fair(cfs_rq, se); > __dequeue_entity(cfs_rq, se); > update_load_avg(cfs_rq, se, UPDATE_TG); > + /* > + * HACK, stash a copy of deadline at the point of pick in vlag, > + * which isn't used until dequeue. > + */ > + se->vlag = se->deadline; > } > > update_stats_curr_start(cfs_rq, se); > diff --git a/kernel/sched/features.h b/kernel/sched/features.h > index 61bcbf5e46a4..f770168230ae 100644 > --- a/kernel/sched/features.h > +++ b/kernel/sched/features.h > @@ -6,6 +6,7 @@ > */ > SCHED_FEAT(PLACE_LAG, true) > SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) > +SCHED_FEAT(RUN_TO_PARITY, true) > > /* > * Prefer to schedule the task we woke last (assuming it failed ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-16 15:38 ` Mike Galbraith @ 2023-08-16 20:04 ` Peter Zijlstra 2023-08-17 1:25 ` Mike Galbraith 0 siblings, 1 reply; 30+ messages in thread From: Peter Zijlstra @ 2023-08-16 20:04 UTC (permalink / raw) To: Mike Galbraith Cc: Chen Yu, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Wed, Aug 16, 2023 at 05:38:20PM +0200, Mike Galbraith wrote: > On Wed, 2023-08-16 at 15:40 +0200, Peter Zijlstra wrote: > > On Wed, Aug 16, 2023 at 02:37:16PM +0200, Peter Zijlstra wrote: > > > On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: > > > > > > > --- a/kernel/sched/fair.c > > > > +++ b/kernel/sched/fair.c > > > > @@ -875,6 +875,12 @@ static struct sched_entity *pick_eevdf(s > > > > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > > > > curr = NULL; > > > > > > > > + /* > > > > + * Once selected, run the task to parity to avoid overscheduling. > > > > + */ > > > > + if (sched_feat(RUN_TO_PARITY) && curr) > > > > + return curr; > > > > + > > > > while (node) { > > > > struct sched_entity *se = __node_2_se(node); > > > > > > > > > > So I read it wrong last night... but I rather like this idea. But > > > there's something missing. When curr starts a new slice it should > > > probably do a full repick and not stick with it. > > > > > > Let me poke at this a bit.. nice > > > > Something like so.. it shouldn't matter much now, but might make a > > difference once we start mixing different slice lengths. > > Hm, that stash the deadline trick _seems_ to have cured the reason I > was inspired to added that XXX hunk.. no 'ew, that's a tad harsh' > latency penalty in sight <knocks wood>. > > Here's hoping test bots don't have a cow. You want to write up a Changelog for this, or should I attempt to write one? ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-16 20:04 ` Peter Zijlstra @ 2023-08-17 1:25 ` Mike Galbraith 0 siblings, 0 replies; 30+ messages in thread From: Mike Galbraith @ 2023-08-17 1:25 UTC (permalink / raw) To: Peter Zijlstra Cc: Chen Yu, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Wed, 2023-08-16 at 22:04 +0200, Peter Zijlstra wrote: > > Here's hoping test bots don't have a cow. > > You want to write up a Changelog for this, or should I attempt to write > one? You write way better changelogs, you took the sting out, taking the basic notion to solid ground.. and I'm at least as much of a lazy sod as the next guy, so am all for you making it a real patch :) -Mike ^ permalink raw reply [flat|nested] 30+ messages in thread
* [tip: sched/core] sched/eevdf: Curb wakeup-preemption 2023-08-16 13:40 ` Peter Zijlstra 2023-08-16 15:38 ` Mike Galbraith @ 2023-08-17 15:10 ` tip-bot2 for Peter Zijlstra 2023-08-21 10:39 ` K Prateek Nayak 2023-08-18 1:09 ` [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression Chen Yu 2 siblings, 1 reply; 30+ messages in thread From: tip-bot2 for Peter Zijlstra @ 2023-08-17 15:10 UTC (permalink / raw) To: linux-tip-commits Cc: Mike Galbraith, Peter Zijlstra (Intel), x86, linux-kernel The following commit has been merged into the sched/core branch of tip: Commit-ID: 63304558ba5dcaaff9e052ee43cfdcc7f9c29e85 Gitweb: https://git.kernel.org/tip/63304558ba5dcaaff9e052ee43cfdcc7f9c29e85 Author: Peter Zijlstra <peterz@infradead.org> AuthorDate: Wed, 16 Aug 2023 15:40:59 +02:00 Committer: Peter Zijlstra <peterz@infradead.org> CommitterDate: Thu, 17 Aug 2023 17:07:07 +02:00 sched/eevdf: Curb wakeup-preemption Mike and others noticed that EEVDF does like to over-schedule quite a bit -- which does hurt performance of a number of benchmarks / workloads. In particular, what seems to cause over-scheduling is that when lag is of the same order (or larger) than the request / slice then placement will not only cause the task to be placed left of current, but also with a smaller deadline than current, which causes immediate preemption. [ notably, lag bounds are relative to HZ ] Mike suggested we stick to picking 'current' for as long as it's eligible to run, giving it uninterrupted runtime until it reaches parity with the pack. Augment Mike's suggestion by only allowing it to exhaust it's initial request. One random data point: echo NO_RUN_TO_PARITY > /debug/sched/features perf stat -a -e context-switches --repeat 10 -- perf bench sched messaging -g 20 -t -l 5000 3,723,554 context-switches ( +- 0.56% ) 9.5136 +- 0.0394 seconds time elapsed ( +- 0.41% ) echo RUN_TO_PARITY > /debug/sched/features perf stat -a -e context-switches --repeat 10 -- perf bench sched messaging -g 20 -t -l 5000 2,556,535 context-switches ( +- 0.51% ) 9.2427 +- 0.0302 seconds time elapsed ( +- 0.33% ) Suggested-by: Mike Galbraith <umgwanakikbuti@gmail.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lkml.kernel.org/r/20230816134059.GC982867@hirez.programming.kicks-ass.net --- kernel/sched/fair.c | 12 ++++++++++++ kernel/sched/features.h | 1 + 2 files changed, 13 insertions(+) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f496cef..0b7445c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -873,6 +873,13 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq) if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) curr = NULL; + /* + * Once selected, run a task until it either becomes non-eligible or + * until it gets a new slice. See the HACK in set_next_entity(). + */ + if (sched_feat(RUN_TO_PARITY) && curr && curr->vlag == curr->deadline) + return curr; + while (node) { struct sched_entity *se = __node_2_se(node); @@ -5167,6 +5174,11 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) update_stats_wait_end_fair(cfs_rq, se); __dequeue_entity(cfs_rq, se); update_load_avg(cfs_rq, se, UPDATE_TG); + /* + * HACK, stash a copy of deadline at the point of pick in vlag, + * which isn't used until dequeue. + */ + se->vlag = se->deadline; } update_stats_curr_start(cfs_rq, se); diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 61bcbf5..f770168 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -6,6 +6,7 @@ */ SCHED_FEAT(PLACE_LAG, true) SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) +SCHED_FEAT(RUN_TO_PARITY, true) /* * Prefer to schedule the task we woke last (assuming it failed ^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [tip: sched/core] sched/eevdf: Curb wakeup-preemption 2023-08-17 15:10 ` [tip: sched/core] sched/eevdf: Curb wakeup-preemption tip-bot2 for Peter Zijlstra @ 2023-08-21 10:39 ` K Prateek Nayak 2023-08-21 15:30 ` Mike Galbraith 0 siblings, 1 reply; 30+ messages in thread From: K Prateek Nayak @ 2023-08-21 10:39 UTC (permalink / raw) To: Peter Zijlstra (Intel) Cc: Mike Galbraith, linux-kernel, linux-tip-commits, x86, Chen Yu, Gautham Shenoy Hello Peter, Sorry for being late to the party but couple of benchmarks are unhappy (very!) with eevdf, even with this optimization. I'll leave the results of testing on a dual socket 3rd Generation EPYC System (2 x 64C/128T) running in NPS1 mode below. tl;dr - Hackbench with medium load, tbench when overloaded, and DeathStarBench are not a fan of EEVDF so far :( - schbench, when system overloaded, sees great benefit in 99th%ile latency, but that is expected since the deadline is fixed to (vruntime + base_slice) but base_slice_ns is equal to the legacy min_granularity_ns in all cases. Some cases of unixbench see a good benefit too. - Others seem perf neutral. On 8/17/2023 8:40 PM, tip-bot2 for Peter Zijlstra wrote: > The following commit has been merged into the sched/core branch of tip: > > Commit-ID: 63304558ba5dcaaff9e052ee43cfdcc7f9c29e85 > Gitweb: https://git.kernel.org/tip/63304558ba5dcaaff9e052ee43cfdcc7f9c29e85 > Author: Peter Zijlstra <peterz@infradead.org> > AuthorDate: Wed, 16 Aug 2023 15:40:59 +02:00 > Committer: Peter Zijlstra <peterz@infradead.org> > CommitterDate: Thu, 17 Aug 2023 17:07:07 +02:00 > > sched/eevdf: Curb wakeup-preemption > > Mike and others noticed that EEVDF does like to over-schedule quite a > bit -- which does hurt performance of a number of benchmarks / > workloads. > > In particular, what seems to cause over-scheduling is that when lag is > of the same order (or larger) than the request / slice then placement > will not only cause the task to be placed left of current, but also > with a smaller deadline than current, which causes immediate > preemption. > > [ notably, lag bounds are relative to HZ ] > > Mike suggested we stick to picking 'current' for as long as it's > eligible to run, giving it uninterrupted runtime until it reaches > parity with the pack. > > Augment Mike's suggestion by only allowing it to exhaust it's initial > request. > > One random data point: > > echo NO_RUN_TO_PARITY > /debug/sched/features > perf stat -a -e context-switches --repeat 10 -- perf bench sched messaging -g 20 -t -l 5000 > > 3,723,554 context-switches ( +- 0.56% ) > 9.5136 +- 0.0394 seconds time elapsed ( +- 0.41% ) > > echo RUN_TO_PARITY > /debug/sched/features > perf stat -a -e context-switches --repeat 10 -- perf bench sched messaging -g 20 -t -l 5000 > > 2,556,535 context-switches ( +- 0.51% ) > 9.2427 +- 0.0302 seconds time elapsed ( +- 0.33% ) o System Details - 3rd Generation EPYC System - 2 x 64C/128T - NPS1 mode o Kernels base: tip:sched/core at commit 752182b24bf4 ("Merge tag 'v6.5-rc2' into sched/core, to pick up fixes") eevdf: tip:sched/core at commit c1fc6484e1fb ("sched/rt: sysctl_sched_rr_timeslice show default timeslice after reset") eevdf_curb: tip:sched/core at commit 63304558ba5d ("sched/eevdf: Curb wakeup-preemption") o Benchmark Results * - Regression ^ - Improvement ================================================================== Test : hackbench Units : Normalized time in seconds Interpretation: Lower is better Statistic : AMean ================================================================== Case: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) 1-groups 1.00 [ -0.00]( 2.51) 1.02 [ -1.69]( 1.89) 1.03 [ -2.54]( 2.42) 2-groups 1.00 [ -0.00]( 1.63) 1.05 [ -4.68]( 2.04) 1.04 [ -3.75]( 1.25) * 4-groups 1.00 [ -0.00]( 1.80) 1.07 [ -7.47]( 2.38) 1.07 [ -6.81]( 1.68) * 8-groups 1.00 [ -0.00]( 1.43) 1.06 [ -6.22]( 1.52) 1.06 [ -6.43]( 1.32) * 16-groups 1.00 [ -0.00]( 1.04) 1.01 [ -1.27]( 3.44) 1.02 [ -1.55]( 2.58) ================================================================== Test : tbench Units : Normalized throughput Interpretation: Higher is better Statistic : AMean ================================================================== Clients: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) 1 1.00 [ 0.00]( 0.49) 1.01 [ 0.97]( 0.18) 1.01 [ 0.52]( 0.06) 2 1.00 [ 0.00]( 1.94) 1.02 [ 2.36]( 0.63) 1.02 [ 1.62]( 0.63) 4 1.00 [ 0.00]( 1.07) 1.00 [ -0.19]( 0.86) 1.01 [ 0.76]( 1.19) 8 1.00 [ 0.00]( 1.41) 1.02 [ 1.69]( 0.22) 1.01 [ 1.48]( 0.73) 16 1.00 [ 0.00]( 1.31) 1.04 [ 3.72]( 1.99) 1.05 [ 4.67]( 1.36) 32 1.00 [ 0.00]( 5.31) 1.04 [ 3.53]( 4.29) 1.05 [ 4.52]( 2.21) 64 1.00 [ 0.00]( 3.08) 1.12 [ 12.12]( 1.71) 1.10 [ 10.19]( 3.06) 128 1.00 [ 0.00]( 1.54) 1.01 [ 1.02]( 0.65) 0.98 [ -2.23]( 0.62) 256 1.00 [ 0.00]( 1.09) 0.95 [ -5.42]( 0.19) 0.92 [ -7.86]( 0.50) * 512 1.00 [ 0.00]( 0.20) 0.91 [ -9.03]( 0.20) 0.90 [-10.25]( 0.29) * 1024 1.00 [ 0.00]( 0.22) 0.88 [-12.47]( 0.29) 0.87 [-13.46]( 0.49) * ================================================================== Test : stream-10 Units : Normalized Bandwidth, MB/s Interpretation: Higher is better Statistic : HMean ================================================================== Test: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) Copy 1.00 [ 0.00]( 3.95) 1.00 [ 0.03]( 4.32) 1.02 [ 2.26]( 2.73) Scale 1.00 [ 0.00]( 8.33) 1.05 [ 5.17]( 5.21) 1.05 [ 4.80]( 5.48) Add 1.00 [ 0.00]( 8.15) 1.05 [ 4.50]( 6.25) 1.04 [ 4.44]( 5.53) Triad 1.00 [ 0.00]( 3.11) 0.93 [ -6.55](10.74) 0.97 [ -2.86]( 7.14) ================================================================== Test : stream-100 Units : Normalized Bandwidth, MB/s Interpretation: Higher is better Statistic : HMean ================================================================== Test: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) Copy 1.00 [ 0.00]( 0.95) 1.00 [ 0.30]( 0.70) 1.00 [ 0.30]( 1.08) Scale 1.00 [ 0.00]( 0.73) 0.97 [ -2.93]( 6.55) 1.00 [ 0.15]( 0.82) Add 1.00 [ 0.00]( 1.69) 0.98 [ -2.19]( 6.53) 1.01 [ 0.88]( 1.08) Triad 1.00 [ 0.00]( 7.49) 1.02 [ 2.02]( 6.66) 1.05 [ 4.88]( 4.56) ================================================================== Test : netperf Units : Normalized Througput Interpretation: Higher is better Statistic : AMean ================================================================== Clients: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) 1-clients 1.00 [ 0.00]( 1.07) 1.00 [ 0.42]( 0.46) 1.01 [ 1.02]( 0.70) 2-clients 1.00 [ 0.00]( 0.78) 1.00 [ -0.26]( 0.38) 1.00 [ 0.40]( 0.92) 4-clients 1.00 [ 0.00]( 0.96) 1.01 [ 0.77]( 0.72) 1.01 [ 1.07]( 0.83) 8-clients 1.00 [ 0.00]( 0.53) 1.00 [ -0.30]( 0.98) 1.00 [ 0.15]( 0.82) 16-clients 1.00 [ 0.00]( 1.05) 1.00 [ 0.22]( 0.70) 1.01 [ 0.54]( 1.26) 32-clients 1.00 [ 0.00]( 1.29) 1.00 [ 0.12]( 0.74) 1.00 [ 0.16]( 1.24) 64-clients 1.00 [ 0.00]( 2.80) 1.00 [ -0.27]( 2.24) 1.00 [ 0.32]( 3.06) 128-clients 1.00 [ 0.00]( 1.57) 1.00 [ -0.42]( 1.72) 0.99 [ -0.63]( 1.64) 256-clients 1.00 [ 0.00]( 3.85) 1.02 [ 2.40]( 4.44) 1.00 [ 0.45]( 3.71) 512-clients 1.00 [ 0.00](45.83) 1.00 [ 0.12](52.42) 0.97 [ -2.75](57.69) ================================================================== Test : schbench (old) Units : Normalized 99th percentile latency in us Interpretation: Lower is better Statistic : Median ================================================================== #workers: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) 1 1.00 [ -0.00]( 2.28) 1.00 [ -0.00]( 2.28) 1.00 [ -0.00]( 2.28) 2 1.00 [ -0.00](11.27) 1.27 [-27.27]( 6.42) 1.14 [-13.64](11.02) * 4 1.00 [ -0.00]( 1.95) 1.00 [ -0.00]( 3.77) 0.93 [ 6.67]( 4.22) 8 1.00 [ -0.00]( 4.17) 1.03 [ -2.70](13.83) 0.95 [ 5.41]( 1.63) 16 1.00 [ -0.00]( 4.17) 0.98 [ 2.08]( 4.37) 1.04 [ -4.17]( 3.53) 32 1.00 [ -0.00]( 1.89) 1.00 [ -0.00]( 8.69) 0.96 [ 3.70]( 5.14) 64 1.00 [ -0.00]( 3.66) 1.03 [ -3.31]( 2.30) 1.06 [ -5.96]( 2.56) 128 1.00 [ -0.00]( 5.79) 0.85 [ 14.77](12.12) 0.97 [ 3.15]( 6.76) ^ 256 1.00 [ -0.00]( 8.50) 0.15 [ 84.84](26.04) 0.17 [ 83.43]( 8.04) ^ 512 1.00 [ -0.00]( 2.01) 0.28 [ 72.09]( 5.62) 0.28 [ 72.35]( 3.48) ^ ================================================================== Test : Unixbench Units : Various, Throughput Interpretation: Higher is better Statistic : AMean, Hmean (Specified) ================================================================== tip eevdf eevdf-curb Hmean unixbench-dhry2reg-1 41333812.04 ( 0.00%) 41248390.97 ( -0.21%) 41576959.80 ( 0.59%) Hmean unixbench-dhry2reg-512 6244993319.97 ( 0.00%) 6239969914.15 ( -0.08%) 6223263669.12 ( -0.35%) Amean unixbench-syscall-1 2932426.17 ( 0.00%) 2968518.27 * -1.23%* 2923093.63 * 0.32%* Amean unixbench-syscall-512 7670057.70 ( 0.00%) 7790656.20 * -1.57%* 8300980.77 * 8.23%* ^ Hmean unixbench-pipe-1 2571551.92 ( 0.00%) 2535689.01 * -1.39%* 2472718.52 * -3.84%* Hmean unixbench-pipe-512 366469338.93 ( 0.00%) 361385055.25 * -1.39%* 363215893.62 * -0.89%* Hmean unixbench-spawn-1 4263.51 ( 0.00%) 4506.26 * 5.69%* 4520.53 * 6.03%* ^ Hmean unixbench-spawn-512 67782.44 ( 0.00%) 69380.09 * 2.36%* 69709.04 * 2.84%* Hmean unixbench-execl-1 3829.47 ( 0.00%) 3824.57 ( -0.13%) 3835.20 ( 0.15%) Hmean unixbench-execl-512 11929.77 ( 0.00%) 12288.64 ( 3.01%) 13096.25 * 9.78%* ^ ================================================================== Test : ycsb-mongodb Units : Throughput Interpretation: Higher is better Statistic : AMean ================================================================== base 303129.00 (var: 0.68%) eevdf 309589.33 (var: 1.41%) (+2.13%) eevdf-curb 303940.00 (var: 1.09%) (+0.27%) ================================================================== Test : DeathStarBench Units : %diff of Throughput Interpretation: Higher is better Statistic : AMean ================================================================== base eevdf eevdf_curb 1CCD 0% -15.15% -16.55% 2CCD 0% -13.80% -16.23% 4CCD 0% -7.50% -10.11% 8CCD 0% -3.42% -3.68% -- I'll go back to profile hackbench, tbench, and DeathStarBench. Will keep the thread updated of any findings. Let me know if you have any pointers for the debug. I plan on using Chenyu's schedstats extension unless IBS or idle-info show some obvious problems - Thank you Chenyu for sharing the schedstats patch :) > > Suggested-by: Mike Galbraith <umgwanakikbuti@gmail.com> > Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> > Link: https://lkml.kernel.org/r/20230816134059.GC982867@hirez.programming.kicks-ass.net > --- > kernel/sched/fair.c | 12 ++++++++++++ > kernel/sched/features.h | 1 + > 2 files changed, 13 insertions(+) > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index f496cef..0b7445c 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -873,6 +873,13 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq) > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > curr = NULL; > > + /* > + * Once selected, run a task until it either becomes non-eligible or > + * until it gets a new slice. See the HACK in set_next_entity(). > + */ > + if (sched_feat(RUN_TO_PARITY) && curr && curr->vlag == curr->deadline) > + return curr; > + > while (node) { > struct sched_entity *se = __node_2_se(node); > > @@ -5167,6 +5174,11 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) > update_stats_wait_end_fair(cfs_rq, se); > __dequeue_entity(cfs_rq, se); > update_load_avg(cfs_rq, se, UPDATE_TG); > + /* > + * HACK, stash a copy of deadline at the point of pick in vlag, > + * which isn't used until dequeue. > + */ > + se->vlag = se->deadline; > } > > update_stats_curr_start(cfs_rq, se); > diff --git a/kernel/sched/features.h b/kernel/sched/features.h > index 61bcbf5..f770168 100644 > --- a/kernel/sched/features.h > +++ b/kernel/sched/features.h > @@ -6,6 +6,7 @@ > */ > SCHED_FEAT(PLACE_LAG, true) > SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) > +SCHED_FEAT(RUN_TO_PARITY, true) > > /* > * Prefer to schedule the task we woke last (assuming it failed -- Thanks and Regards, Prateek ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip: sched/core] sched/eevdf: Curb wakeup-preemption 2023-08-21 10:39 ` K Prateek Nayak @ 2023-08-21 15:30 ` Mike Galbraith 2023-08-22 3:03 ` K Prateek Nayak 0 siblings, 1 reply; 30+ messages in thread From: Mike Galbraith @ 2023-08-21 15:30 UTC (permalink / raw) To: K Prateek Nayak, Peter Zijlstra (Intel) Cc: linux-kernel, linux-tip-commits, x86, Chen Yu, Gautham Shenoy On Mon, 2023-08-21 at 16:09 +0530, K Prateek Nayak wrote: > Hello Peter, > > Sorry for being late to the party but couple of benchmarks are unhappy > (very!) with eevdf, even with this optimization. I'll leave the results > of testing on a dual socket 3rd Generation EPYC System (2 x 64C/128T) > running in NPS1 mode below. > > tl;dr > > - Hackbench with medium load, tbench when overloaded, and DeathStarBench > are not a fan of EEVDF so far :( FWIW, there are more tbench shards lying behind EEVDF than in front. tbench 8 on old i7-4790 box 4.4.302 4024 6.4.11 3668 6.4.11-eevdf 3522 I went a-hunting once, but it didn't go well. There were a couple identifiable sched related dips/recoveries, but the overall result was a useless downward trending mess. -Mike ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip: sched/core] sched/eevdf: Curb wakeup-preemption 2023-08-21 15:30 ` Mike Galbraith @ 2023-08-22 3:03 ` K Prateek Nayak 2023-08-22 6:09 ` Mike Galbraith 0 siblings, 1 reply; 30+ messages in thread From: K Prateek Nayak @ 2023-08-22 3:03 UTC (permalink / raw) To: Mike Galbraith, Peter Zijlstra (Intel) Cc: linux-kernel, linux-tip-commits, x86, Chen Yu, Gautham Shenoy Hello Mike, On 8/21/2023 9:00 PM, Mike Galbraith wrote: > On Mon, 2023-08-21 at 16:09 +0530, K Prateek Nayak wrote: >> Hello Peter, >> >> Sorry for being late to the party but couple of benchmarks are unhappy >> (very!) with eevdf, even with this optimization. I'll leave the results >> of testing on a dual socket 3rd Generation EPYC System (2 x 64C/128T) >> running in NPS1 mode below. >> >> tl;dr >> >> - Hackbench with medium load, tbench when overloaded, and DeathStarBench >> are not a fan of EEVDF so far :( > > FWIW, there are more tbench shards lying behind EEVDF than in front. > > tbench 8 on old i7-4790 box > 4.4.302 4024 > 6.4.11 3668 > 6.4.11-eevdf 3522 > I agree, but on servers, tbench has been useful to identify a variety of issues [1][2][3] and I believe it is better to pick some shards up than leave them lying around for others to step on :) Casting aside tbench, there are still more workloads that have regression and it'll be good to understand which property of those don't sit well with EEVDF. [1] https://lore.kernel.org/lkml/c50bdbfe-02ce-c1bc-c761-c95f8e216ca0@amd.com/ [2] https://lore.kernel.org/lkml/20220921063638.2489-1-kprateek.nayak@amd.com/ [3] https://lore.kernel.org/lkml/80956e8f-761e-b74-1c7a-3966f9e8d934@linutronix.de/ > I went a-hunting once, but it didn't go well. There were a couple > identifiable sched related dips/recoveries, but the overall result was > a useless downward trending mess. > > -Mike -- Thanks and Regards, Prateek ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip: sched/core] sched/eevdf: Curb wakeup-preemption 2023-08-22 3:03 ` K Prateek Nayak @ 2023-08-22 6:09 ` Mike Galbraith 2023-08-25 6:41 ` K Prateek Nayak 2023-09-19 9:02 ` [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks Ingo Molnar 0 siblings, 2 replies; 30+ messages in thread From: Mike Galbraith @ 2023-08-22 6:09 UTC (permalink / raw) To: K Prateek Nayak, Peter Zijlstra (Intel) Cc: linux-kernel, linux-tip-commits, x86, Chen Yu, Gautham Shenoy On Tue, 2023-08-22 at 08:33 +0530, K Prateek Nayak wrote: > Hello Mike, Greetings! > > FWIW, there are more tbench shards lying behind EEVDF than in front. > > > > tbench 8 on old i7-4790 box > > 4.4.302 4024 > > 6.4.11 3668 > > 6.4.11-eevdf 3522 > > > > I agree, but on servers, tbench has been useful to identify a variety of > issues [1][2][3] and I believe it is better to pick some shards up than > leave them lying around for others to step on :) Absolutely, but in this case it isn't due to various overheads wiggling about and/or bitrot, everything's identical except the scheduler, and its overhead essentially is too. taskset -c 3 pipe-test 6.4.11 1.420033 usecs/loop -- avg 1.420033 1408.4 KHz 6.4.11-eevdf 1.413024 usecs/loop -- avg 1.413024 1415.4 KHz Methinks these shards are due to tbench simply being one of those things that happens to like the CFS notion of short term fairness a bit better than the EEVDF notion, ie are inevitable fallout tied to the very thing that makes EEVDF service less spiky that CFS, and thus will be difficult to sweep up. Too bad I didn't save Peter's test hack to make EEVDF use the same notion of fair (not a keeper) as I think that would likely prove it. -Mike ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip: sched/core] sched/eevdf: Curb wakeup-preemption 2023-08-22 6:09 ` Mike Galbraith @ 2023-08-25 6:41 ` K Prateek Nayak 2023-09-19 9:02 ` [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks Ingo Molnar 1 sibling, 0 replies; 30+ messages in thread From: K Prateek Nayak @ 2023-08-25 6:41 UTC (permalink / raw) To: Mike Galbraith, Peter Zijlstra (Intel) Cc: linux-kernel, linux-tip-commits, x86, Chen Yu, Gautham Shenoy, Aaron Lu, Mathieu Desnoyers, Swapnil Sapkal Hello again, On 8/22/2023 11:39 AM, Mike Galbraith wrote: > Methinks these shards are due to tbench simply being one of those > things that happens to like the CFS notion of short term fairness a bit > better than the EEVDF notion, ie are inevitable fallout tied to the > very thing that makes EEVDF service less spiky that CFS, and thus will > be difficult to sweep up. Ah! That is pretty sad for some of these benchmarks that are not a fan of large number of context-switches. I went back and tried some solutions floating in the community (thank you Mathieu and Aaron) > > Too bad I didn't save Peter's test hack to make EEVDF use the same > notion of fair (not a keeper) as I think that would likely prove it. More benchmarking with Mathieu's [1] and Aaron's [2] proposed changes on top of EEVDF + Curb Wakeup-Preemption: tl;dr - Hackbench is now happy with Mathieu's and Aaron's patches (I'm not looking with CFS baseline for now since Swapnil and Gautham are testing those) - Some unixbench variants are pretty happy in overloaded system. - schbench is pretty happy throughout. - tbench and DeathStarBench are still pretty unhappy. Most likely because of greater number of context switches but I'll get more concrete evidence for this. - Other micro-benchmarks tested are more or less perf neutral to EEVDF. o System Details - 3rd Generation EPYC System - 2 x 64C/128T - NPS1 mode o Kernels base (updated): tip:sched/core at commit 88c56cfeaec4 ("sched/fair: Block nohz tick_stop when cfs bandwidth in use") eevdf: tip:sched/core at commit c1fc6484e1fb ("sched/rt: sysctl_sched_rr_timeslice show default timeslice after reset") eevdf_curb: tip:sched/core at commit 63304558ba5d ("sched/eevdf:Curb wakeup-preemption") eevdf-curb+mathieu: eevdf + cheery-pick commit e242aa146918 ("sched/eevdf: Curb wakeup-preemption") + Mathieu's patches [1] eevdf-curb+mathieu+aaron eevdf-curb+mathieu + Aaron's patch [2] [References] [1] "sched: Skip queued wakeups only when L2 is shared" (https://lore.kernel.org/all/20230822113133.643238-1-mathieu.desnoyers@efficios.com/) [2] "Reduce cost of accessing tg->load_avg" (https://lore.kernel.org/lkml/20230823060832.454842-1-aaron.lu@intel.com/) o Benchmark Results * - Interesting Regression ^ - Interesting Improvement ================================================================== Test : hackbench Units : Normalized time in seconds Interpretation: Lower is better Statistic : AMean ================================================================== Case: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) eevdf-curb+mathieu[pct imp](CV) eevdf-curb+mathieu+aaron[pct imp](CV) 1-groups 1.00 [ -0.00]( 2.64) 1.02 [ -1.98]( 1.89) 1.03 [ -2.83]( 2.42) 0.95 [ 5.10]( 2.65) 0.94 [ 5.67]( 2.07) ^ 2-groups 1.00 [ -0.00]( 2.97) 1.05 [ -5.18]( 2.04) 1.04 [ -4.24]( 1.25) 0.98 [ 1.88]( 1.99) 0.96 [ 3.76]( 2.64) ^ 4-groups 1.00 [ -0.00]( 1.84) 1.06 [ -6.07]( 2.38) 1.05 [ -5.42]( 1.68) 1.00 [ 0.43]( 1.50) 0.99 [ 0.65]( 0.97) 8-groups 1.00 [ -0.00]( 1.24) 1.07 [ -7.09]( 1.52) 1.07 [ -7.29]( 1.32) 0.91 [ 9.11]( 1.63) 0.93 [ 7.49]( 2.43) ^ 16-groups 1.00 [ -0.00]( 3.31) 1.05 [ -4.98]( 3.44) 1.05 [ -5.27]( 2.58) 0.74 [ 26.35]( 1.22) 0.69 [ 31.04]( 1.90) ^ ================================================================== Test : tbench Units : Normalized throughput Interpretation: Higher is better Statistic : AMean ================================================================== Clients: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) eevdf-curb+mathieu[pct imp](CV) eevdf-curb+mathieu+aaron[pct imp](CV) 1 1.00 [ 0.00]( 1.08) 1.00 [ 0.42]( 0.18) 1.00 [ -0.03]( 0.06) 0.99 [ -0.57]( 0.27) 0.99 [ -0.51]( 0.97) 2 1.00 [ 0.00]( 0.69) 1.00 [ -0.09]( 0.63) 0.99 [ -0.81]( 0.63) 1.00 [ -0.15]( 0.74) 0.99 [ -0.76]( 0.76) 4 1.00 [ 0.00]( 0.70) 0.99 [ -1.20]( 0.86) 1.00 [ -0.26]( 1.19) 0.99 [ -1.08]( 1.03) 0.99 [ -1.43]( 0.84) 8 1.00 [ 0.00]( 0.85) 1.01 [ 0.56]( 0.22) 1.00 [ 0.35]( 0.73) 1.00 [ -0.32]( 0.53) 1.00 [ -0.22]( 1.45) 16 1.00 [ 0.00]( 2.18) 0.98 [ -1.82]( 1.99) 0.99 [ -0.92]( 1.36) 0.99 [ -1.19]( 1.54) 0.97 [ -3.33]( 1.44) 32 1.00 [ 0.00]( 3.84) 0.99 [ -0.88]( 4.29) 1.00 [ 0.06]( 2.21) 0.98 [ -1.92]( 3.60) 0.96 [ -4.36]( 2.45) 64 1.00 [ 0.00]( 7.06) 1.03 [ 3.45]( 1.71) 1.02 [ 1.68]( 3.06) 1.03 [ 2.60]( 3.37) 1.01 [ 0.57]( 3.97) 128 1.00 [ 0.00]( 0.88) 1.02 [ 1.54]( 0.65) 0.98 [ -1.74]( 0.62) 0.98 [ -1.63]( 1.85) 0.97 [ -3.33]( 1.83) 256 1.00 [ 0.00]( 0.88) 0.94 [ -5.67]( 0.19) 0.92 [ -8.10]( 0.50) 0.95 [ -4.94]( 0.21) 0.91 [ -8.74]( 1.55) * 512 1.00 [ 0.00]( 0.07) 0.91 [ -9.17]( 0.20) 0.90 [-10.39]( 0.29) 0.92 [ -8.37]( 1.51) 0.90 [ -9.73]( 0.05) * 1024 1.00 [ 0.00]( 0.30) 0.87 [-12.66]( 0.29) 0.86 [-13.64]( 0.49) 0.86 [-14.47]( 0.35) 0.87 [-13.44]( 0.26) * ================================================================== Test : stream-10 Units : Normalized Bandwidth, MB/s Interpretation: Higher is better Statistic : HMean ================================================================== Test: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) eevdf-curb+mathieu[pct imp](CV) eevdf-curb+mathieu+aaron[pct imp](CV) Copy 1.00 [ 0.00]( 8.87) 1.03 [ 2.94]( 4.32) 1.05 [ 5.23]( 2.73) 0.96 [ -3.92]( 7.61) 0.97 [ -3.28]( 5.93) Scale 1.00 [ 0.00]( 6.80) 1.01 [ 1.38]( 5.21) 1.01 [ 1.03]( 5.48) 1.03 [ 2.98]( 1.04) 0.99 [ -1.24]( 7.22) Add 1.00 [ 0.00]( 7.24) 1.01 [ 1.45]( 6.25) 1.01 [ 1.39]( 5.53) 1.02 [ 1.51]( 2.51) 0.98 [ -2.02]( 6.99) Triad 1.00 [ 0.00]( 5.00) 0.96 [ -3.63](10.74) 1.00 [ 0.18]( 7.14) 0.99 [ -0.78]( 5.21) 1.00 [ 0.45]( 2.91) ================================================================== Test : stream-100 Units : Normalized Bandwidth, MB/s Interpretation: Higher is better Statistic : HMean ================================================================== Test: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) eevdf-curb+mathieu[pct imp](CV) eevdf-curb+mathieu+aaron[pct imp](CV) Copy 1.00 [ 0.00]( 0.45) 1.00 [ -0.27]( 0.70) 1.00 [ -0.27]( 1.08) 1.00 [ -0.48]( 1.44) 0.98 [ -2.02]( 4.84) Scale 1.00 [ 0.00]( 4.40) 0.98 [ -1.86]( 6.55) 1.01 [ 1.25]( 0.82) 0.99 [ -0.85]( 4.98) 1.01 [ 1.10]( 1.07) Add 1.00 [ 0.00]( 4.97) 0.99 [ -0.91]( 6.53) 1.02 [ 2.20]( 1.08) 1.00 [ 0.26]( 4.90) 1.00 [ 0.11]( 2.37) Triad 1.00 [ 0.00]( 0.24) 0.95 [ -4.85]( 6.66) 0.98 [ -2.19]( 4.56) 0.97 [ -2.80]( 5.56) 0.98 [ -1.58]( 2.15) ================================================================== Test : netperf Units : Normalized Througput Interpretation: Higher is better Statistic : AMean ================================================================== Clients: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) eevdf-curb+mathieu[pct imp](CV) eevdf-curb+mathieu+aaron[pct imp](CV) 1-clients 1.00 [ 0.00]( 0.46) 0.99 [ -1.26]( 0.46) 0.99 [ -0.67]( 0.70) 1.00 [ -0.13]( 0.30) 0.99 [ -1.46]( 0.37) 2-clients 1.00 [ 0.00]( 0.75) 0.99 [ -0.87]( 0.38) 1.00 [ -0.21]( 0.92) 1.00 [ 0.13]( 0.32) 0.99 [ -1.47]( 1.05) 4-clients 1.00 [ 0.00]( 0.84) 1.00 [ -0.45]( 0.72) 1.00 [ -0.14]( 0.83) 1.00 [ -0.10]( 0.60) 0.99 [ -0.70]( 0.78) 8-clients 1.00 [ 0.00]( 0.78) 0.99 [ -1.07]( 0.98) 0.99 [ -0.63]( 0.82) 1.00 [ -0.49]( 0.85) 0.99 [ -0.83]( 0.81) 16-clients 1.00 [ 0.00]( 2.56) 1.00 [ -0.29]( 0.70) 1.00 [ 0.03]( 1.26) 1.00 [ 0.17]( 0.66) 1.00 [ -0.43]( 0.80) 32-clients 1.00 [ 0.00]( 1.03) 0.99 [ -0.76]( 0.74) 0.99 [ -0.73]( 1.24) 1.00 [ -0.49]( 1.45) 0.99 [ -1.29]( 1.88) 64-clients 1.00 [ 0.00]( 2.69) 0.98 [ -1.96]( 2.24) 0.99 [ -1.39]( 3.06) 0.98 [ -1.78]( 3.00) 0.98 [ -2.31]( 3.00) 128-clients 1.00 [ 0.00]( 1.91) 1.00 [ -0.43]( 1.72) 0.99 [ -0.64]( 1.64) 0.97 [ -2.72]( 1.34) 1.00 [ -0.49]( 1.83) 256-clients 1.00 [ 0.00]( 2.21) 1.02 [ 2.23]( 4.44) 1.00 [ 0.29]( 3.71) 1.01 [ 1.49]( 3.72) 1.02 [ 2.17]( 2.50) 512-clients 1.00 [ 0.00](45.18) 1.00 [ 0.10](52.42) 0.97 [ -2.76](57.69) 0.99 [ -1.14](49.76) 0.99 [ -0.58](50.33) ================================================================== Test : schbench (old) Units : Normalized 99th percentile latency in us Interpretation: Lower is better Statistic : Median ================================================================== #workers: base[pct imp](CV) eevdf[pct imp](CV) eevdf-curb[pct imp](CV) eevdf-curb+mathieu[pct imp](CV) eevdf-curb+mathieu+aaron[pct imp](CV) 1 1.00 [ -0.00](12.03) 1.09 [ -8.70]( 2.28) 1.09 [ -8.70]( 2.28) 0.96 [ 4.35](17.83) 1.13 [-13.04]( 5.95) 2 1.00 [ -0.00]( 9.36) 1.08 [ -7.69]( 6.42) 0.96 [ 3.85](11.02) 1.00 [ -0.00]( 2.19) 0.92 [ 7.69]( 0.00) 4 1.00 [ -0.00]( 1.95) 1.00 [ -0.00]( 3.77) 0.93 [ 6.67]( 4.22) 1.03 [ -3.33]( 3.81) 0.93 [ 6.67]( 5.52) 8 1.00 [ -0.00]( 9.01) 1.03 [ -2.70](13.83) 0.95 [ 5.41]( 1.63) 1.08 [ -8.11]( 2.94) 1.05 [ -5.41]( 3.01) 16 1.00 [ -0.00]( 3.08) 0.94 [ 6.00]( 4.37) 1.00 [ -0.00]( 3.53) 1.06 [ -6.00]( 1.10) 0.98 [ 2.00]( 5.51) 32 1.00 [ -0.00]( 0.75) 1.05 [ -5.19]( 8.69) 1.01 [ -1.30]( 5.14) 1.01 [ -1.30]( 5.14) 0.95 [ 5.19]( 5.36) 64 1.00 [ -0.00]( 2.15) 0.96 [ 4.29]( 2.30) 0.98 [ 1.84]( 2.56) 0.93 [ 7.36](14.38) 0.90 [ 9.82](15.50) 128 1.00 [ -0.00]( 5.18) 1.01 [ -0.57](12.12) 1.14 [-14.29]( 6.76) 1.09 [ -9.43]( 8.88) 1.04 [ -4.29]( 4.46) 256 1.00 [ -0.00]( 4.18) 0.16 [ 83.86](26.04) 0.18 [ 82.36]( 8.04) 0.16 [ 83.69](21.07) 0.19 [ 81.29]( 2.30) ^ 512 1.00 [ -0.00]( 2.10) 0.29 [ 70.52]( 5.62) 0.29 [ 70.80]( 3.48) 0.30 [ 70.45]( 1.73) 0.30 [ 70.17]( 5.31) ^ ================================================================== Test : Unixbench Units : Various, Throughput Interpretation: Higher is better Statistic : AMean, Hmean (Specified) ================================================================== base eevdf eevdf_curb eevdf-curb+mathieu eevdf-curb+mathieu+aaron Hmean unixbench-dhry2reg-1 41407024.82 ( 0.00%) 41248390.97 ( -0.38%) 41576959.80 ( 0.41%) 41641988.85 ( 0.57%) 40989184.46 ( -1.01%) Hmean unixbench-dhry2reg-512 6249629291.88 ( 0.00%) 6239969914.15 ( -0.15%) 6223263669.12 ( -0.42%) 6258072789.83 ( 0.14%) 6243186854.90 ( -0.10%) Amean unixbench-syscall-1 2922580.63 ( 0.00%) 2968518.27 * -1.57%* 2923093.63 ( -0.02%) 2887947.73 * 1.19%* 2921456.17 ( 0.04%) Amean unixbench-syscall-512 7606400.73 ( 0.00%) 7790656.20 * 2.42%* 8300980.77 * 9.13%* 7743306.30 * 1.80%* 8284260.83 * 8.91%* ^ Hmean unixbench-pipe-1 2574942.54 ( 0.00%) 2535689.01 * -1.52%* 2472718.52 * -3.97%* 2558661.32 * -0.63%* 2567755.92 * -0.28%* Hmean unixbench-pipe-512 364489246.31 ( 0.00%) 361385055.25 * -0.85%* 363215893.62 * -0.35%* 368809317.33 * 1.19%* 370896020.69 * 1.76%* Hmean unixbench-spawn-1 4428.94 ( 0.00%) 4506.26 ( 1.75%) 4520.53 ( 2.07%) 4778.12 * 7.88%* 4353.00 ( -1.71%) Hmean unixbench-spawn-512 68883.47 ( 0.00%) 69380.09 ( 0.72%) 69709.04 * 1.20%* 70278.77 * 2.03%* 71187.47 * 3.34%* Hmean unixbench-execl-1 3878.47 ( 0.00%) 3824.57 * -1.39%* 3835.20 * -1.12%* 3802.46 * -1.96%* 3830.36 * -1.24%* Hmean unixbench-execl-512 11638.84 ( 0.00%) 12288.64 ( 5.58%) 13096.25 * 12.52%* 12976.96 * 11.50%* 12468.95 ( 7.13%) ^ ================================================================== Test : ycsb-mongodb Units : Throughput Interpretation: Higher is better Statistic : AMean ================================================================== tip: 301757.33 (var: 1.63%) (%diff: +0.00%) eevdf: 309589.33 (var: 1.41%) (%diff: +2.60%) eevdf_curb; 303940.00 (var: 1.09%) (%diff: +0.72%) eevdf-curb+mathieu: 303055.67 (var: 4.14%) (%diff: +0.43%) eevdf-curb+mathieu+aaron: 297810.00 (var: 0.45%) (%diff: -1.31%) ================================================================== Test : DeathStarBench Units : %diff of Throughput Interpretation: Higher is better Statistic : AMean ================================================================== Scaling base eevdf eevdf_curb eevdf-curb+mathieu eevdf-curb+mathieu+aaron 1CCD 0% -14.95% -16.35% -16.67% -16.22% * 2CCD 0% -13.50% -15.94% -18.03% -17.93% * 4CCD 0% -8.49% -11.07% -13.11% -14.57% * 8CCD 0% -4.61% -4.87% -5.72% -6.02% * -- I'll go back and dig further. -- Thanks and Regards, Prateek ^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks 2023-08-22 6:09 ` Mike Galbraith 2023-08-25 6:41 ` K Prateek Nayak @ 2023-09-19 9:02 ` Ingo Molnar 2023-09-19 9:48 ` Mike Galbraith 2023-09-22 10:00 ` kernel test robot 1 sibling, 2 replies; 30+ messages in thread From: Ingo Molnar @ 2023-09-19 9:02 UTC (permalink / raw) To: Mike Galbraith Cc: K Prateek Nayak, Peter Zijlstra (Intel), linux-kernel, linux-tip-commits, x86, Chen Yu, Gautham Shenoy * Mike Galbraith <efault@gmx.de> wrote: > On Tue, 2023-08-22 at 08:33 +0530, K Prateek Nayak wrote: > > Hello Mike, > > Greetings! > > > > FWIW, there are more tbench shards lying behind EEVDF than in front. > > > > > > tbench 8 on old i7-4790 box > > > 4.4.302 4024 > > > 6.4.11 3668 > > > 6.4.11-eevdf 3522 > > > > > > > I agree, but on servers, tbench has been useful to identify a variety of > > issues [1][2][3] and I believe it is better to pick some shards up than > > leave them lying around for others to step on :) > > Absolutely, but in this case it isn't due to various overheads wiggling > about and/or bitrot, everything's identical except the scheduler, and > its overhead essentially is too. > > taskset -c 3 pipe-test > 6.4.11 1.420033 usecs/loop -- avg 1.420033 1408.4 KHz > 6.4.11-eevdf 1.413024 usecs/loop -- avg 1.413024 1415.4 KHz > > Methinks these shards are due to tbench simply being one of those > things that happens to like the CFS notion of short term fairness a bit > better than the EEVDF notion, ie are inevitable fallout tied to the > very thing that makes EEVDF service less spiky that CFS, and thus will > be difficult to sweep up. > > Too bad I didn't save Peter's test hack to make EEVDF use the same > notion of fair (not a keeper) as I think that would likely prove it. BTW., if overscheduling is still an issue, I'm wondering whether we could go so far as to turn off wakeup preemption for same-prio SCHED_OTHER tasks altogether, as per the attached patch? What does this do to your various tests? Test booted only. Thanks, Ingo =============> From: Ingo Molnar <mingo@kernel.org> Date: Tue, 19 Sep 2023 10:49:51 +0200 Subject: [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks Reduce overscheduling some more: do not wakeup-preempt same-priority SCHED_OTHER tasks. Signed-off-by: Ingo Molnar <mingo@kernel.org> --- kernel/sched/fair.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a502e3255392..98efe01c8e4e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8042,7 +8042,7 @@ static void check_preempt_wakeup_fair(struct rq *rq, struct task_struct *p, int * Batch and idle tasks do not preempt non-idle tasks (their preemption * is driven by the tick): */ - if (unlikely(p->policy != SCHED_NORMAL) || !sched_feat(WAKEUP_PREEMPTION)) + if (unlikely(p->policy != SCHED_NORMAL) || likely(p->prio == curr->prio) || !sched_feat(WAKEUP_PREEMPTION)) return; find_matching_se(&se, &pse); ^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks 2023-09-19 9:02 ` [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks Ingo Molnar @ 2023-09-19 9:48 ` Mike Galbraith 2023-09-22 10:00 ` kernel test robot 1 sibling, 0 replies; 30+ messages in thread From: Mike Galbraith @ 2023-09-19 9:48 UTC (permalink / raw) To: Ingo Molnar Cc: K Prateek Nayak, Peter Zijlstra (Intel), linux-kernel, linux-tip-commits, x86, Chen Yu, Gautham Shenoy On Tue, 2023-09-19 at 11:02 +0200, Ingo Molnar wrote: > > BTW., if overscheduling is still an issue... The trouble I had with overscheduling was mostly about annoying insta- preempt creating useless overhead, and that the RUN_TO_PARITY thing ended. All I see now is the expected 60/40 indeed != 50/50, so things are gonna shift a bit. Most of the testing I've done lately has been due to fretting over the desktop perhaps getting the short end of that shift, but after much poking and prodding, everything seems just fine here. -Mike ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks 2023-09-19 9:02 ` [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks Ingo Molnar 2023-09-19 9:48 ` Mike Galbraith @ 2023-09-22 10:00 ` kernel test robot 2023-09-25 11:07 ` Ingo Molnar 1 sibling, 1 reply; 30+ messages in thread From: kernel test robot @ 2023-09-22 10:00 UTC (permalink / raw) To: Ingo Molnar Cc: oe-lkp, lkp, linux-kernel, ying.huang, feng.tang, fengwei.yin, aubrey.li, yu.c.chen, Mike Galbraith, K Prateek Nayak, Peter Zijlstra (Intel), linux-tip-commits, x86, Gautham Shenoy, oliver.sang Hello, kernel test robot noticed a -19.0% regression of stress-ng.filename.ops_per_sec on: commit: 8afa2eec7fb5de648561dba3b195cef09dab884d ("[PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") url: https://github.com/intel-lab-lkp/linux/commits/Ingo-Molnar/sched-fair-Do-not-wakeup-preempt-same-prio-SCHED_OTHER-tasks/20230919-170307 base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 7ad0354d18ae05e9c8885251e234cbcf141f8972 patch link: https://lore.kernel.org/all/ZQljoiSBhZLEFI/G@gmail.com/ patch subject: [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks testcase: stress-ng test machine: 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory parameters: nr_threads: 10% disk: 1SSD testtime: 60s fs: xfs class: filesystem test: filename cpufreq_governor: performance In addition to that, the commit also has significant impact on the following tests: +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.lockbus.ops_per_sec -6.0% regression | | test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory | | test parameters | class=memory | | | cpufreq_governor=performance | | | nr_threads=100% | | | test=lockbus | | | testtime=60s | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.sigfd.ops_per_sec 17.6% improvement | | test machine | 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory | | test parameters | class=interrupt | | | cpufreq_governor=performance | | | disk=1SSD | | | nr_threads=100% | | | test=sigfd | | | testtime=60s | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | phoronix-test-suite: phoronix-test-suite.darktable.Masskrug.CPU-only.seconds -5.3% improvement | | test machine | 12 threads 1 sockets Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Coffee Lake) with 16G memory | | test parameters | cpufreq_governor=performance | | | option_a=Masskrug | | | option_b=CPU-only | | | test=darktable-1.0.5 | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | lmbench3: lmbench3.TCP.socket.bandwidth.64B.MB/sec 11.5% improvement | | test machine | 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 112G memory | | test parameters | cpufreq_governor=performance | | | mode=development | | | nr_threads=100% | | | test=TCP | | | test_memory_size=50% | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | phoronix-test-suite: phoronix-test-suite.darktable.Boat.CPU-only.seconds -3.5% improvement | | test machine | 12 threads 1 sockets Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Coffee Lake) with 16G memory | | test parameters | cpufreq_governor=performance | | | option_a=Boat | | | option_b=CPU-only | | | test=darktable-1.0.5 | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.sigrt.ops_per_sec 100.2% improvement | | test machine | 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory | | test parameters | class=interrupt | | | cpufreq_governor=performance | | | disk=1SSD | | | nr_threads=100% | | | test=sigrt | | | testtime=60s | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.sigsuspend.ops_per_sec -93.9% regression | | test machine | 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory | | test parameters | class=interrupt | | | cpufreq_governor=performance | | | disk=1SSD | | | nr_threads=100% | | | test=sigsuspend | | | testtime=60s | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.sigsuspend.ops_per_sec -82.1% regression | | test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory | | test parameters | class=interrupt | | | cpufreq_governor=performance | | | disk=1HDD | | | nr_threads=100% | | | test=sigsuspend | | | testtime=60s | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.sock.ops_per_sec 59.4% improvement | | test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory | | test parameters | class=network | | | cpufreq_governor=performance | | | nr_threads=100% | | | test=sock | | | testtime=60s | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | blogbench: blogbench.write_score -35.9% regression | | test machine | 96 threads 2 sockets Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz (Cascade Lake) with 128G memory | | test parameters | cpufreq_governor=performance | | | disk=1SSD | | | fs=btrfs | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput -4.8% regression | | test machine | 224 threads 4 sockets Intel(R) Xeon(R) Platinum 8380H CPU @ 2.90GHz (Cooper Lake) with 192G memory | | test parameters | cpufreq_governor=performance | | | ipc=socket | | | iterations=4 | | | mode=threads | | | nr_threads=50% | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | blogbench: blogbench.write_score -59.3% regression | | test machine | 96 threads 2 sockets Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz (Cascade Lake) with 128G memory | | test parameters | cpufreq_governor=performance | | | disk=1SSD | | | fs=xfs | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.exec.ops_per_sec -34.6% regression | | test machine | 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory | | test parameters | class=exec_spawn | | | cpufreq_governor=performance | | | disk=1HDD | | | nr_threads=100% | | | test=exec | | | testtime=60s | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | netperf: netperf.Throughput_Mbps 60.6% improvement | | test machine | 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory | | test parameters | cluster=cs-localhost | | | cpufreq_governor=performance | | | ip=ipv4 | | | nr_threads=200% | | | runtime=300s | | | test=UDP_STREAM | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | hackbench: hackbench.throughput 19.1% improvement | | test machine | 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory | | test parameters | cpufreq_governor=performance | | | ipc=pipe | | | iterations=4 | | | mode=threads | | | nr_threads=50% | +------------------+----------------------------------------------------------------------------------------------------+ | testcase: change | stress-ng: stress-ng.dnotify.ops_per_sec -15.7% regression | | test machine | 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory | | test parameters | class=filesystem | | | cpufreq_governor=performance | | | disk=1SSD | | | fs=xfs | | | nr_threads=10% | | | test=dnotify | | | testtime=60s | +------------------+----------------------------------------------------------------------------------------------------+ If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags | Reported-by: kernel test robot <oliver.sang@intel.com> | Closes: https://lore.kernel.org/oe-lkp/202309221758.d655aa5b-oliver.sang@intel.com Details are as below: --------------------------------------------------------------------------------------------------> The kernel config and materials to reproduce are available at: https://download.01.org/0day-ci/archive/20230922/202309221758.d655aa5b-oliver.sang@intel.com ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: filesystem/gcc-12/performance/1SSD/xfs/x86_64-rhel-8.3/10%/debian-11.1-x86_64-20220510.cgz/lkp-skl-d08/filename/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 4641666 +19.5% 5545394 ± 2% cpuidle..usage 1.78 -0.2 1.57 mpstat.cpu.all.usr% 6.49 -5.6% 6.13 iostat.cpu.system 1.77 -11.3% 1.57 iostat.cpu.user 10560 ± 2% +56.5% 16523 vmstat.io.bo 90589 ± 2% +70.5% 154471 ± 2% vmstat.system.cs 55956 +4.2% 58304 vmstat.system.in 10705 ± 6% -18.7% 8702 ± 21% sched_debug.cfs_rq:/.avg_vruntime.avg 75917 ± 18% -24.3% 57455 ± 12% sched_debug.cfs_rq:/.avg_vruntime.max 14846 ± 13% -19.3% 11978 ± 15% sched_debug.cfs_rq:/.avg_vruntime.stddev 10705 ± 6% -18.7% 8702 ± 21% sched_debug.cfs_rq:/.min_vruntime.avg 75917 ± 18% -24.3% 57455 ± 12% sched_debug.cfs_rq:/.min_vruntime.max 14846 ± 13% -19.3% 11978 ± 15% sched_debug.cfs_rq:/.min_vruntime.stddev 36871 ± 14% -36.1% 23564 ± 45% sched_debug.cpu.nr_switches.min 628439 -19.2% 507711 stress-ng.filename.ops 10317 -19.0% 8355 stress-ng.filename.ops_per_sec 171981 -59.7% 69333 ± 3% stress-ng.time.involuntary_context_switches 221.00 -12.7% 193.00 stress-ng.time.percent_of_cpu_this_job_got 108.56 -11.7% 95.90 stress-ng.time.system_time 31.82 -16.9% 26.44 stress-ng.time.user_time 770691 ± 3% +200.9% 2319214 stress-ng.time.voluntary_context_switches 1138479 ± 3% +63.3% 1859024 ± 5% turbostat.C1 5.16 ± 2% +2.0 7.15 ± 4% turbostat.C1% 1194780 +10.1% 1315182 turbostat.C1E 7.10 +1.7 8.76 ± 2% turbostat.C1E% 404353 ± 4% +32.1% 534162 ± 5% turbostat.POLL 0.09 ± 4% +0.0 0.12 ± 3% turbostat.POLL% 86.42 -1.4% 85.17 turbostat.PkgWatt 157494 +97.3% 310687 meminfo.Active 157288 +97.4% 310497 meminfo.Active(anon) 714039 +24.1% 885851 meminfo.Committed_AS 759716 -17.0% 630878 meminfo.KReclaimable 48745 +20.5% 58737 ± 5% meminfo.Mapped 759716 -17.0% 630878 meminfo.SReclaimable 171371 +99.9% 342496 meminfo.Shmem 881795 -15.2% 747680 meminfo.Slab 39322 +97.4% 77624 proc-vmstat.nr_active_anon 740918 +5.8% 783713 proc-vmstat.nr_file_pages 79407 +5.9% 84102 proc-vmstat.nr_inactive_anon 12193 +20.5% 14691 ± 5% proc-vmstat.nr_mapped 42845 +99.9% 85627 proc-vmstat.nr_shmem 189913 -17.0% 157692 proc-vmstat.nr_slab_reclaimable 30518 -4.3% 29198 proc-vmstat.nr_slab_unreclaimable 39322 +97.4% 77624 proc-vmstat.nr_zone_active_anon 79407 +5.9% 84102 proc-vmstat.nr_zone_inactive_anon 734099 +9.4% 802836 proc-vmstat.numa_hit 734074 +9.4% 802787 proc-vmstat.numa_local 57736 +99.3% 115074 proc-vmstat.pgactivate 1010216 -7.8% 931388 ± 2% proc-vmstat.pgfree 707596 ± 2% +56.5% 1107112 proc-vmstat.pgpgout 2.279e+09 -6.6% 2.128e+09 perf-stat.i.branch-instructions 1.41 +0.1 1.52 perf-stat.i.branch-miss-rate% 8.25 ± 5% -1.3 7.00 ± 3% perf-stat.i.cache-miss-rate% 47892495 ± 2% +14.2% 54671049 ± 2% perf-stat.i.cache-references 95307 +70.9% 162853 ± 2% perf-stat.i.context-switches 1.06 +3.1% 1.09 perf-stat.i.cpi 1.238e+10 -3.8% 1.191e+10 perf-stat.i.cpu-cycles 1422 +51.5% 2155 ± 3% perf-stat.i.cpu-migrations 4209 +7.0% 4504 perf-stat.i.cycles-between-cache-misses 2.84e+09 -6.2% 2.665e+09 perf-stat.i.dTLB-loads 0.00 ± 16% +0.0 0.01 ± 15% perf-stat.i.dTLB-store-miss-rate% 60986 ± 16% +44.5% 88132 ± 14% perf-stat.i.dTLB-store-misses 1.734e+09 -7.5% 1.604e+09 perf-stat.i.dTLB-stores 23.19 ± 9% -5.9 17.32 ± 11% perf-stat.i.iTLB-load-miss-rate% 4381599 ± 10% +27.7% 5596623 ± 9% perf-stat.i.iTLB-loads 1.174e+10 -6.3% 1.1e+10 perf-stat.i.instructions 0.95 -2.8% 0.92 perf-stat.i.ipc 0.34 -3.8% 0.33 perf-stat.i.metric.GHz 191.62 -6.5% 179.14 perf-stat.i.metric.M/sec 1.44 +0.1 1.56 perf-stat.overall.branch-miss-rate% 7.87 ± 5% -1.1 6.74 perf-stat.overall.cache-miss-rate% 1.05 +2.7% 1.08 perf-stat.overall.cpi 0.00 ± 17% +0.0 0.01 ± 14% perf-stat.overall.dTLB-store-miss-rate% 23.16 ± 9% -5.9 17.22 ± 11% perf-stat.overall.iTLB-load-miss-rate% 0.95 -2.6% 0.92 perf-stat.overall.ipc 2.243e+09 -6.6% 2.095e+09 perf-stat.ps.branch-instructions 47146306 ± 2% +14.1% 53816827 ± 2% perf-stat.ps.cache-references 93818 +70.9% 160303 ± 2% perf-stat.ps.context-switches 1.218e+10 -3.8% 1.172e+10 perf-stat.ps.cpu-cycles 1400 +51.5% 2121 ± 3% perf-stat.ps.cpu-migrations 2.795e+09 -6.2% 2.623e+09 perf-stat.ps.dTLB-loads 60034 ± 16% +44.5% 86753 ± 14% perf-stat.ps.dTLB-store-misses 1.707e+09 -7.5% 1.579e+09 perf-stat.ps.dTLB-stores 4313198 ± 10% +27.7% 5509023 ± 9% perf-stat.ps.iTLB-loads 1.155e+10 -6.3% 1.082e+10 perf-stat.ps.instructions 7.437e+11 -6.3% 6.968e+11 perf-stat.total.instructions 0.00 ±223% +1200.0% 0.00 ± 31% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.__d_alloc.d_alloc 0.00 ±141% +650.0% 0.00 ± 19% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_createname 0.00 ±223% +1100.0% 0.00 ± 70% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_removename 0.00 ±223% +1000.0% 0.00 ± 37% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.single_open.do_dentry_open 0.00 ±223% +1000.0% 0.00 ± 37% perf-sched.sch_delay.avg.ms.__cond_resched.apparmor_file_alloc_security.security_file_alloc.init_file.alloc_empty_file 0.00 ±223% +1100.0% 0.00 ± 40% perf-sched.sch_delay.avg.ms.__cond_resched.down_read.walk_component.path_lookupat.filename_lookup 0.00 ± 70% +275.0% 0.00 ± 30% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vfs_unlink.do_unlinkat.__x64_sys_unlink 0.00 ±141% +550.0% 0.00 ± 31% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_create.xfs_generic_create 0.00 ±141% +600.0% 0.00 ± 58% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes.xfs_trans_alloc_dir 0.00 ±141% +500.0% 0.00 ± 40% perf-sched.sch_delay.avg.ms.__cond_resched.dput.__lookup_slow.walk_component.path_lookupat 0.00 +150.0% 0.00 ± 19% perf-sched.sch_delay.avg.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 0.00 ±223% +1300.0% 0.00 ± 40% perf-sched.sch_delay.avg.ms.__cond_resched.dput.lookup_open.isra.0 0.00 ±223% +3100.0% 0.01 ±140% perf-sched.sch_delay.avg.ms.__cond_resched.dput.step_into.path_lookupat.filename_lookup 0.00 ±223% +1000.0% 0.00 ± 58% perf-sched.sch_delay.avg.ms.__cond_resched.dput.terminate_walk.path_lookupat.filename_lookup 0.00 ±223% +800.0% 0.00 ± 33% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.security_file_alloc.init_file.alloc_empty_file 0.00 ±141% +550.0% 0.00 ± 81% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.seq_open.single_open.do_dentry_open 0.00 ± 17% +276.9% 0.02 ± 8% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_difree_finobt 0.00 ±223% +1800.0% 0.00 ± 76% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_iunlink_log_inode.xfs_iunlink.xfs_remove 0.37 ± 30% +260.3% 1.32 ± 7% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_ifree.xfs_inactive 0.00 ±223% +900.0% 0.00 ± 28% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate.xfs_create 0.00 ±141% +500.0% 0.00 ± 28% perf-sched.sch_delay.avg.ms.__cond_resched.mnt_want_write.do_unlinkat.__x64_sys_unlink.do_syscall_64 0.08 ± 28% +94.1% 0.16 ± 8% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xlog_cil_commit 0.03 ± 3% -36.7% 0.02 perf-sched.sch_delay.avg.ms.schedule_timeout.___down_common.__down.down 0.00 +166.7% 0.00 ± 17% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra 0.01 ± 42% +78.3% 0.02 ± 31% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_killable.__kthread_create_on_node 0.03 ± 10% +194.0% 0.08 ± 5% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.01 ± 34% +233.3% 0.02 ± 29% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_createname 0.00 ± 52% +432.0% 0.02 ± 63% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.single_open.do_dentry_open 0.00 ± 54% +203.7% 0.01 ± 36% perf-sched.sch_delay.max.ms.__cond_resched.apparmor_file_alloc_security.security_file_alloc.init_file.alloc_empty_file 0.02 ± 40% +23085.6% 4.29 ±221% perf-sched.sch_delay.max.ms.__cond_resched.down.xlog_write_iclog.xlog_state_release_iclog.xlog_write_get_more_iclog_space 0.01 ± 56% +204.1% 0.02 ± 56% perf-sched.sch_delay.max.ms.__cond_resched.down_read.walk_component.path_lookupat.filename_lookup 0.00 ±223% +900.0% 0.01 ± 28% perf-sched.sch_delay.max.ms.__cond_resched.down_write.open_last_lookups.path_openat.do_filp_open 0.01 ± 28% +135.6% 0.02 ± 54% perf-sched.sch_delay.max.ms.__cond_resched.down_write.vfs_unlink.do_unlinkat.__x64_sys_unlink 0.01 ± 37% +189.5% 0.02 ± 63% perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes.xfs_trans_alloc_dir 0.01 ± 18% +100.0% 0.02 ± 32% perf-sched.sch_delay.max.ms.__cond_resched.dput.path_put.vfs_statx.__do_sys_newstat 0.01 ± 32% +450.0% 0.03 ± 77% perf-sched.sch_delay.max.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open 0.00 ± 99% +500.0% 0.01 ± 68% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_dup_cursor.xfs_btree_dup_cursor 0.01 ± 27% +236.0% 0.03 ± 30% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_dialloc_ag 0.02 ± 48% +114.2% 0.04 ± 41% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_difree_finobt 0.00 ±152% +1250.0% 0.03 ±144% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_iunlink_log_inode.xfs_iunlink.xfs_remove 0.01 ± 28% +163.3% 0.01 ± 55% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_common_interrupt 0.01 ± 33% +3565.1% 0.26 ±153% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.02 ± 42% +162.1% 0.05 ± 58% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.02 ± 24% +71.3% 0.03 ± 29% perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__flush_work.isra.0 3.81 ± 4% +27.2% 4.85 ± 16% perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.02 ± 3% -13.8% 0.02 ± 2% perf-sched.total_sch_delay.average.ms 3.48 ± 2% -46.4% 1.86 perf-sched.total_wait_and_delay.average.ms 227601 +91.9% 436761 perf-sched.total_wait_and_delay.count.ms 3.45 ± 2% -46.6% 1.85 perf-sched.total_wait_time.average.ms 6.78 ± 14% -35.0% 4.40 ± 6% perf-sched.wait_and_delay.avg.ms.__cond_resched.down.xlog_write_iclog.xlog_state_release_iclog.xlog_write_get_more_iclog_space 0.10 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 3.78 ± 72% +281.9% 14.42 ± 29% perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_ifree.xfs_inactive 138.51 ± 3% -51.2% 67.56 ± 6% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 1.10 ± 4% -58.2% 0.46 perf-sched.wait_and_delay.avg.ms.schedule_timeout.___down_common.__down.down 60.79 ± 5% -30.7% 42.15 ± 3% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 14.81 ± 2% +29.7% 19.21 ± 2% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 7.15 ± 6% -33.2% 4.78 ± 9% perf-sched.wait_and_delay.avg.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread 318.17 ± 16% +50.1% 477.50 ± 7% perf-sched.wait_and_delay.count.__cond_resched.down.xlog_write_iclog.xlog_state_release_iclog.xlog_write_get_more_iclog_space 8265 -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 7.17 ± 64% +130.2% 16.50 ± 24% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 85.50 ± 27% -95.5% 3.83 ± 43% perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 108.17 ± 4% +103.9% 220.50 ± 2% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 2.50 ± 75% +193.3% 7.33 ± 47% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.xlog_cil_push_work 112747 ± 2% +194.7% 332296 perf-sched.wait_and_delay.count.schedule_timeout.___down_common.__down.down 5400 ± 5% +47.7% 7978 ± 3% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 14061 -26.8% 10296 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 11.17 ± 25% +129.9% 25.67 ± 14% perf-sched.wait_and_delay.count.xlog_state_get_iclog_space.xlog_write_get_more_iclog_space.xlog_write_partial.xlog_write 326.67 +34.9% 440.83 ± 2% perf-sched.wait_and_delay.count.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread 47.65 ± 11% -39.4% 28.86 ± 6% perf-sched.wait_and_delay.max.ms.__cond_resched.down.xlog_write_iclog.xlog_state_release_iclog.xlog_write_get_more_iclog_space 5.96 ± 29% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 1001 ± 55% -94.4% 56.12 ±117% perf-sched.wait_and_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 52.30 ± 15% -54.3% 23.88 ± 9% perf-sched.wait_and_delay.max.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread 0.04 ± 25% -76.3% 0.01 ± 31% perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc_node.seq_read_iter.seq_read 6.78 ± 14% -35.2% 4.39 ± 7% perf-sched.wait_time.avg.ms.__cond_resched.down.xlog_write_iclog.xlog_state_release_iclog.xlog_write_get_more_iclog_space 0.05 ± 24% +133.6% 0.11 ± 40% perf-sched.wait_time.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_remove 0.06 ± 34% -85.1% 0.01 ± 12% perf-sched.wait_time.avg.ms.__cond_resched.down_write.open_last_lookups.path_openat.do_filp_open 0.10 -41.5% 0.06 ± 14% perf-sched.wait_time.avg.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 0.04 ± 39% +185.4% 0.10 ± 82% perf-sched.wait_time.avg.ms.__cond_resched.dput.step_into.path_lookupat.filename_lookup 0.57 ± 52% +80.2% 1.03 ± 19% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_buf_item_init._xfs_trans_bjoin.xfs_trans_get_buf_map 0.37 ± 19% -93.2% 0.02 ± 8% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_difree_finobt 0.58 ± 52% +80.4% 1.05 ± 24% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_difree_inobt 4.11 ± 41% +218.5% 13.10 ± 32% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_ifree.xfs_inactive 0.23 ±104% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.perf_event_ctx_lock_nested.constprop.0 138.51 ± 3% -51.2% 67.56 ± 6% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 1.07 ± 4% -58.8% 0.44 perf-sched.wait_time.avg.ms.schedule_timeout.___down_common.__down.down 0.07 ± 28% +388.3% 0.36 ± 17% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra 60.77 ± 5% -30.8% 42.07 ± 3% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 14.74 ± 2% +29.8% 19.13 ± 2% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 7.14 ± 6% -33.2% 4.77 ± 9% perf-sched.wait_time.avg.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread 0.12 ± 28% -87.8% 0.01 ± 34% perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc_node.seq_read_iter.seq_read 47.64 ± 11% -39.4% 28.85 ± 6% perf-sched.wait_time.max.ms.__cond_resched.down.xlog_write_iclog.xlog_state_release_iclog.xlog_write_get_more_iclog_space 0.27 ± 69% -93.2% 0.02 ± 26% perf-sched.wait_time.max.ms.__cond_resched.down_write.open_last_lookups.path_openat.do_filp_open 0.22 ± 74% +458.4% 1.25 ± 95% perf-sched.wait_time.max.ms.__cond_resched.dput.step_into.path_lookupat.filename_lookup 1.32 ± 65% +462.3% 7.41 ± 59% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.getname_flags.part.0 1.16 ± 87% -94.6% 0.06 ± 36% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_difree_finobt 1.34 ± 84% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.perf_event_ctx_lock_nested.constprop.0 1000 ± 55% -94.5% 54.97 ±119% perf-sched.wait_time.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 4.70 ± 70% +184.0% 13.35 ± 38% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra 52.29 ± 15% -54.3% 23.87 ± 9% perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread 9.93 -1.4 8.56 perf-profile.calltrace.cycles-pp.__xstat64 17.34 -1.2 16.19 perf-profile.calltrace.cycles-pp.creat64 16.29 -1.1 15.18 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.creat64 15.99 -1.1 14.92 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64 7.66 -1.0 6.64 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__xstat64 15.38 -1.0 14.37 perf-profile.calltrace.cycles-pp.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64 15.32 -1.0 14.32 perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe.creat64 14.68 -0.9 13.76 perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_creat.do_syscall_64.entry_SYSCALL_64_after_hwframe 14.60 -0.9 13.68 perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_creat.do_syscall_64 6.91 ± 2% -0.9 6.03 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 4.72 ± 3% -0.7 4.02 ± 3% perf-profile.calltrace.cycles-pp.open64 12.76 ± 2% -0.7 12.10 perf-profile.calltrace.cycles-pp.open_last_lookups.path_openat.do_filp_open.do_sys_openat2.__x64_sys_creat 5.49 ± 2% -0.7 4.84 ± 2% perf-profile.calltrace.cycles-pp.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 12.44 ± 2% -0.6 11.83 perf-profile.calltrace.cycles-pp.lookup_open.open_last_lookups.path_openat.do_filp_open.do_sys_openat2 3.40 ± 5% -0.6 2.79 ± 2% perf-profile.calltrace.cycles-pp.read 3.97 ± 4% -0.6 3.40 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64 3.74 ± 4% -0.5 3.21 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 11.76 ± 2% -0.5 11.22 perf-profile.calltrace.cycles-pp.xfs_generic_create.lookup_open.open_last_lookups.path_openat.do_filp_open 4.19 ± 2% -0.5 3.68 ± 2% perf-profile.calltrace.cycles-pp.__close 4.18 ± 3% -0.5 3.69 ± 2% perf-profile.calltrace.cycles-pp.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 2.68 ± 6% -0.5 2.21 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 3.32 ± 4% -0.5 2.85 ± 5% perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 3.28 ± 4% -0.5 2.82 ± 4% perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 3.79 ± 2% -0.4 3.35 ± 2% perf-profile.calltrace.cycles-pp.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.85 ± 4% -0.4 2.44 ± 4% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.43 ± 6% -0.4 2.02 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 2.79 ± 4% -0.4 2.39 ± 4% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64 3.58 ± 2% -0.4 3.19 ± 2% perf-profile.calltrace.cycles-pp.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat.do_syscall_64 2.55 ± 4% -0.3 2.20 ± 3% perf-profile.calltrace.cycles-pp.stress_mwc32modn 2.88 ± 3% -0.3 2.54 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close 0.58 ± 8% -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.__entry_text_start.__xstat64 0.58 ± 5% -0.3 0.26 ±100% perf-profile.calltrace.cycles-pp.list_sort.xfs_trans_run_precommits.__xfs_trans_commit.xfs_create.xfs_generic_create 1.98 ± 6% -0.3 1.67 ± 4% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 4.39 ± 2% -0.3 4.09 perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_create.xfs_generic_create.lookup_open.open_last_lookups 1.91 ± 6% -0.3 1.62 ± 4% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 2.41 ± 3% -0.3 2.14 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 1.68 ± 7% -0.2 1.44 ± 3% perf-profile.calltrace.cycles-pp.seq_read_iter.seq_read.vfs_read.ksys_read.do_syscall_64 1.72 ± 7% -0.2 1.48 ± 4% perf-profile.calltrace.cycles-pp.seq_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.37 ± 3% -0.2 1.15 ± 4% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__xstat64 1.62 ± 2% -0.2 1.42 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 3.22 ± 2% -0.2 3.03 ± 2% perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_create.xfs_generic_create.lookup_open 1.24 ± 5% -0.2 1.05 ± 4% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 2.70 ± 3% -0.2 2.50 ± 3% perf-profile.calltrace.cycles-pp.xfs_dialloc_ag.xfs_dialloc.xfs_create.xfs_generic_create.lookup_open 1.21 ± 2% -0.2 1.03 ± 3% perf-profile.calltrace.cycles-pp.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 1.22 ± 3% -0.2 1.05 ± 3% perf-profile.calltrace.cycles-pp.link_path_walk.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat 0.97 ± 10% -0.2 0.81 ± 5% perf-profile.calltrace.cycles-pp.seq_show.seq_read_iter.seq_read.vfs_read.ksys_read 1.44 ± 3% -0.1 1.30 ± 3% perf-profile.calltrace.cycles-pp.xfs_init_new_inode.xfs_create.xfs_generic_create.lookup_open.open_last_lookups 0.60 ± 4% -0.1 0.45 ± 44% perf-profile.calltrace.cycles-pp.destroy_inode.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.69 ± 4% -0.1 1.56 ± 3% perf-profile.calltrace.cycles-pp.walk_component.path_lookupat.filename_lookup.vfs_statx.__do_sys_newstat 0.94 ± 7% -0.1 0.82 ± 6% perf-profile.calltrace.cycles-pp.getname_flags.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__xstat64 0.68 ± 7% -0.1 0.57 ± 7% perf-profile.calltrace.cycles-pp.strncpy_from_user.getname_flags.__do_sys_newstat.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.80 ± 3% -0.1 0.69 ± 5% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__close 1.11 ± 6% -0.1 1.02 ± 5% perf-profile.calltrace.cycles-pp.xfs_difree_finobt.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive 0.68 ± 5% -0.1 0.58 ± 2% perf-profile.calltrace.cycles-pp.alloc_empty_file.path_openat.do_filp_open.do_sys_openat2.__x64_sys_creat 0.64 ± 4% -0.1 0.55 ± 4% perf-profile.calltrace.cycles-pp.xfs_dir_createname.xfs_create.xfs_generic_create.lookup_open.open_last_lookups 0.75 ± 4% -0.1 0.66 ± 4% perf-profile.calltrace.cycles-pp.do_open.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat 0.99 ± 3% -0.1 0.90 ± 4% perf-profile.calltrace.cycles-pp.xfs_trans_run_precommits.__xfs_trans_commit.xfs_create.xfs_generic_create.lookup_open 0.67 ± 5% -0.1 0.59 ± 3% perf-profile.calltrace.cycles-pp.do_dentry_open.do_open.path_openat.do_filp_open.do_sys_openat2 0.71 ± 3% -0.1 0.64 ± 5% perf-profile.calltrace.cycles-pp.xfs_trans_alloc.xfs_trans_alloc_dir.xfs_remove.xfs_vn_unlink.vfs_unlink 0.61 ± 5% -0.0 0.56 ± 5% perf-profile.calltrace.cycles-pp.xfs_btree_lookup.xfs_difree_finobt.xfs_difree.xfs_ifree.xfs_inactive_ifree 1.01 ± 3% +0.1 1.09 ± 3% perf-profile.calltrace.cycles-pp.xfs_trans_run_precommits.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink 0.62 ± 8% +0.1 0.73 ± 3% perf-profile.calltrace.cycles-pp.load_balance.newidle_balance.pick_next_task_fair.__schedule.schedule 0.70 ± 9% +0.1 0.83 ± 2% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.do_nanosleep.hrtimer_nanosleep 0.69 ± 8% +0.1 0.82 ± 2% perf-profile.calltrace.cycles-pp.newidle_balance.pick_next_task_fair.__schedule.schedule.do_nanosleep 0.82 ± 7% +0.1 0.96 ± 2% perf-profile.calltrace.cycles-pp.xlog_cil_alloc_shadow_bufs.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive 0.84 ± 10% +0.1 0.98 ± 3% perf-profile.calltrace.cycles-pp.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.94 ± 9% +0.2 1.09 ± 3% perf-profile.calltrace.cycles-pp.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.83 ± 9% +0.2 0.98 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.do_nanosleep.hrtimer_nanosleep.common_nsleep 0.94 ± 9% +0.2 1.10 ± 3% perf-profile.calltrace.cycles-pp.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.97 ± 8% +0.2 1.12 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_clock_nanosleep.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.92 ± 8% +0.2 1.08 ± 2% perf-profile.calltrace.cycles-pp.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep.do_syscall_64 1.05 ± 8% +0.2 1.21 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.clock_nanosleep 1.09 ± 9% +0.2 1.25 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.clock_nanosleep 0.86 ± 5% +0.2 1.02 ± 5% perf-profile.calltrace.cycles-pp.xfs_trans_run_precommits.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker 1.23 ± 8% +0.2 1.40 ± 3% perf-profile.calltrace.cycles-pp.clock_nanosleep 10.12 +0.2 10.33 perf-profile.calltrace.cycles-pp.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 0.81 ± 10% +0.3 1.08 ± 7% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.58 ± 5% +0.3 0.86 ± 6% perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_create 0.62 ± 5% +0.3 0.90 ± 6% perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_create.xfs_generic_create 0.66 ± 4% +0.3 0.94 ± 6% perf-profile.calltrace.cycles-pp.xfs_ialloc_read_agi.xfs_dialloc.xfs_create.xfs_generic_create.lookup_open 1.50 ± 5% +0.3 1.80 ± 4% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_remove.xfs_vn_unlink 1.32 ± 6% +0.3 1.64 ± 4% perf-profile.calltrace.cycles-pp.xlog_cil_insert_format_items.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_remove 0.27 ±100% +0.3 0.59 ± 3% perf-profile.calltrace.cycles-pp.xfs_buf_item_size_segment.xfs_buf_item_size.xlog_cil_alloc_shadow_bufs.xlog_cil_commit.__xfs_trans_commit 0.64 ± 8% +0.3 0.96 ± 5% perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_remove.xfs_vn_unlink 0.68 ± 8% +0.3 1.01 ± 4% perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_iunlink.xfs_remove.xfs_vn_unlink.vfs_unlink 0.98 ± 7% +0.3 1.32 ± 4% perf-profile.calltrace.cycles-pp.xfs_iunlink.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat 2.60 ± 7% +0.4 2.98 ± 6% perf-profile.calltrace.cycles-pp.xfs_buf_item_format_segment.xfs_buf_item_format.xlog_cil_insert_format_items.xlog_cil_insert_items.xlog_cil_commit 2.97 ± 2% +0.4 3.36 ± 2% perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink 1.97 ± 2% +0.4 2.41 ± 5% perf-profile.calltrace.cycles-pp.xlog_cil_insert_format_items.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree 2.14 ± 2% +0.5 2.59 ± 5% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive 4.17 +0.5 4.63 perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat 0.36 ± 70% +0.5 0.84 ± 6% perf-profile.calltrace.cycles-pp.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink 0.38 ± 70% +0.5 0.85 ± 6% perf-profile.calltrace.cycles-pp.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_remove 3.57 ± 4% +0.5 4.08 ± 2% perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker 0.84 ± 7% +0.5 1.38 ± 6% perf-profile.calltrace.cycles-pp.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_difree 0.00 +0.5 0.54 ± 4% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.schedule_timeout.___down_common 4.16 ± 4% +0.5 4.71 ± 2% perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work 7.83 +0.6 8.39 perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.56 ± 8% perf-profile.calltrace.cycles-pp.up.xfs_buf_unlock.xfs_buf_item_release.xlog_cil_commit.__xfs_trans_commit 1.03 ± 6% +0.6 1.61 ± 6% perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_ialloc_read_agi.xfs_difree.xfs_ifree.xfs_inactive_ifree 0.98 ± 7% +0.6 1.55 ± 6% perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_difree.xfs_ifree 1.06 ± 6% +0.6 1.65 ± 6% perf-profile.calltrace.cycles-pp.xfs_ialloc_read_agi.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive 0.00 +0.6 0.59 ± 7% perf-profile.calltrace.cycles-pp.xfs_buf_unlock.xfs_buf_item_release.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree 7.31 +0.6 7.92 perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.__x64_sys_unlink.do_syscall_64 0.09 ±223% +0.6 0.71 ± 8% perf-profile.calltrace.cycles-pp.xfs_buf_item_release.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive 7.26 +0.6 7.88 perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.__x64_sys_unlink 0.00 +0.6 0.63 ± 11% perf-profile.calltrace.cycles-pp.memcpy_orig.xfs_buf_item_format_segment.xfs_buf_item_format.xlog_cil_insert_format_items.xlog_cil_insert_items 0.08 ±223% +0.7 0.75 ± 8% perf-profile.calltrace.cycles-pp.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc 3.01 ± 2% +0.7 3.76 ± 4% perf-profile.calltrace.cycles-pp.xfs_buf_item_format.xlog_cil_insert_format_items.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit 1.61 ± 9% +0.8 2.40 ± 8% perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 4.19 ± 3% +0.8 4.99 ± 3% perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker 5.29 ± 3% +1.0 6.26 ± 3% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work 3.39 ± 4% +1.1 4.45 ± 6% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.81 ± 8% +1.3 2.09 ± 6% perf-profile.calltrace.cycles-pp.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi 0.00 +1.4 1.36 ± 25% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.___down_common.__down 4.60 ± 9% +1.4 6.01 ± 7% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 10.49 ± 3% +1.5 11.94 ± 2% perf-profile.calltrace.cycles-pp.xfs_inodegc_worker.process_one_work.worker_thread.kthread.ret_from_fork 0.17 ±141% +1.5 1.64 ± 23% perf-profile.calltrace.cycles-pp.schedule_timeout.___down_common.__down.down.xfs_buf_lock 10.23 ± 3% +1.5 11.72 ± 2% perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread.kthread 10.17 ± 3% +1.5 11.66 ± 2% perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread 0.00 +1.5 1.51 ± 21% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.___down_common.__down.down 0.28 ±100% +1.5 1.81 ± 16% perf-profile.calltrace.cycles-pp.___down_common.__down.down.xfs_buf_lock.xfs_buf_find_lock 0.56 ± 8% +1.6 2.11 ± 11% perf-profile.calltrace.cycles-pp.xfs_buf_lock.xfs_buf_find_lock.xfs_buf_lookup.xfs_buf_get_map.xfs_buf_read_map 11.52 ± 3% +1.6 13.12 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 11.52 ± 3% +1.6 13.12 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 11.52 ± 3% +1.6 13.12 ± 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 0.46 ± 45% +1.6 2.06 ± 11% perf-profile.calltrace.cycles-pp.down.xfs_buf_lock.xfs_buf_find_lock.xfs_buf_lookup.xfs_buf_get_map 0.28 ±100% +1.6 1.91 ± 15% perf-profile.calltrace.cycles-pp.__down.down.xfs_buf_lock.xfs_buf_find_lock.xfs_buf_lookup 11.08 ± 3% +1.7 12.75 ± 2% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 10.98 ± 3% +1.7 12.65 ± 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.59 ± 8% +1.7 2.27 ± 10% perf-profile.calltrace.cycles-pp.xfs_buf_find_lock.xfs_buf_lookup.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map 0.65 ± 9% +1.9 2.56 ± 4% perf-profile.calltrace.cycles-pp.xfs_buf_lookup.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi 29.87 ± 3% +2.1 32.02 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 29.58 ± 2% +2.5 32.05 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 30.03 ± 2% +2.6 32.63 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 31.48 ± 2% +2.9 34.42 perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 30.91 ± 2% +3.0 33.88 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 30.93 ± 2% +3.0 33.90 perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 30.92 ± 2% +3.0 33.90 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 45.93 -3.2 42.69 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 43.64 -2.9 40.77 perf-profile.children.cycles-pp.do_syscall_64 18.64 ± 2% -1.5 17.17 perf-profile.children.cycles-pp.do_sys_openat2 10.02 -1.4 8.64 perf-profile.children.cycles-pp.__xstat64 17.54 -1.3 16.20 perf-profile.children.cycles-pp.do_filp_open 17.42 -1.3 16.09 perf-profile.children.cycles-pp.path_openat 17.39 -1.2 16.23 perf-profile.children.cycles-pp.creat64 15.38 -1.0 14.37 perf-profile.children.cycles-pp.__x64_sys_creat 13.27 ± 2% -0.7 12.54 perf-profile.children.cycles-pp.open_last_lookups 4.75 ± 3% -0.7 4.04 ± 3% perf-profile.children.cycles-pp.open64 5.50 ± 2% -0.7 4.84 ± 2% perf-profile.children.cycles-pp.__do_sys_newstat 4.08 ± 3% -0.6 3.46 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret 3.43 ± 5% -0.6 2.81 ± 2% perf-profile.children.cycles-pp.read 12.45 ± 2% -0.6 11.84 perf-profile.children.cycles-pp.lookup_open 11.77 ± 2% -0.5 11.23 perf-profile.children.cycles-pp.xfs_generic_create 4.26 ± 2% -0.5 3.73 ± 2% perf-profile.children.cycles-pp.__close 4.18 ± 2% -0.5 3.70 ± 2% perf-profile.children.cycles-pp.vfs_statx 3.84 ± 3% -0.5 3.35 ± 3% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 3.34 ± 4% -0.5 2.87 ± 4% perf-profile.children.cycles-pp.__x64_sys_openat 3.80 ± 2% -0.4 3.36 ± 2% perf-profile.children.cycles-pp.filename_lookup 3.60 ± 2% -0.4 3.21 ± 2% perf-profile.children.cycles-pp.path_lookupat 2.75 -0.4 2.37 ± 4% perf-profile.children.cycles-pp.link_path_walk 2.65 ± 3% -0.3 2.30 ± 3% perf-profile.children.cycles-pp.stress_mwc32modn 1.98 ± 6% -0.3 1.67 ± 4% perf-profile.children.cycles-pp.ksys_read 2.18 ± 3% -0.3 1.87 ± 4% perf-profile.children.cycles-pp.kmem_cache_alloc 2.17 ± 3% -0.3 1.88 ± 4% perf-profile.children.cycles-pp.__entry_text_start 1.92 ± 6% -0.3 1.63 ± 4% perf-profile.children.cycles-pp.vfs_read 1.73 ± 7% -0.2 1.48 ± 4% perf-profile.children.cycles-pp.seq_read 1.69 ± 7% -0.2 1.44 ± 3% perf-profile.children.cycles-pp.seq_read_iter 1.84 ± 6% -0.2 1.61 ± 3% perf-profile.children.cycles-pp.getname_flags 1.22 ± 5% -0.2 1.01 ± 3% perf-profile.children.cycles-pp.alloc_empty_file 1.62 ± 2% -0.2 1.42 ± 3% perf-profile.children.cycles-pp.__x64_sys_close 2.71 ± 3% -0.2 2.51 ± 3% perf-profile.children.cycles-pp.xfs_dialloc_ag 1.30 ± 6% -0.2 1.10 ± 3% perf-profile.children.cycles-pp.strncpy_from_user 1.95 ± 2% -0.2 1.77 ± 2% perf-profile.children.cycles-pp.xfs_trans_alloc 1.25 ± 2% -0.2 1.07 ± 2% perf-profile.children.cycles-pp.__fput 1.31 ± 4% -0.2 1.14 ± 2% perf-profile.children.cycles-pp.do_open 0.97 ± 10% -0.2 0.82 ± 5% perf-profile.children.cycles-pp.seq_show 1.08 ± 6% -0.1 0.94 ± 2% perf-profile.children.cycles-pp.do_dentry_open 1.45 ± 3% -0.1 1.31 ± 3% perf-profile.children.cycles-pp.xfs_init_new_inode 0.79 ± 8% -0.1 0.66 ± 4% perf-profile.children.cycles-pp.__check_object_size 0.88 ± 6% -0.1 0.75 ± 6% perf-profile.children.cycles-pp.xfs_inobt_update 0.93 ± 4% -0.1 0.80 ± 3% perf-profile.children.cycles-pp.inode_permission 1.40 ± 2% -0.1 1.27 ± 3% perf-profile.children.cycles-pp.xfs_log_reserve 1.54 ± 3% -0.1 1.41 ± 2% perf-profile.children.cycles-pp.xfs_trans_reserve 0.80 ± 3% -0.1 0.68 ± 5% perf-profile.children.cycles-pp.dput 0.34 ± 14% -0.1 0.22 ± 23% perf-profile.children.cycles-pp.run_ksoftirqd 0.58 ± 8% -0.1 0.48 ± 5% perf-profile.children.cycles-pp.__filename_parentat 1.12 ± 5% -0.1 1.03 ± 5% perf-profile.children.cycles-pp.xfs_difree_finobt 0.27 ± 12% -0.1 0.18 ± 18% perf-profile.children.cycles-pp.xfs_btree_update_keys 0.67 ± 6% -0.1 0.57 ± 8% perf-profile.children.cycles-pp.stress_filename 0.51 ± 5% -0.1 0.41 ± 10% perf-profile.children.cycles-pp.down_read 0.64 ± 4% -0.1 0.55 ± 4% perf-profile.children.cycles-pp.xfs_dir_createname 0.51 ± 6% -0.1 0.42 ± 8% perf-profile.children.cycles-pp.generic_permission 1.04 ± 4% -0.1 0.96 ± 3% perf-profile.children.cycles-pp.xfs_inode_item_format 0.54 ± 11% -0.1 0.46 ± 9% perf-profile.children.cycles-pp.xfs_inobt_init_cursor 0.42 ± 5% -0.1 0.34 ± 9% perf-profile.children.cycles-pp.xlog_grant_add_space 0.51 ± 7% -0.1 0.44 ± 4% perf-profile.children.cycles-pp.xfs_log_ticket_ungrant 0.46 ± 8% -0.1 0.38 ± 6% perf-profile.children.cycles-pp.check_heap_object 0.54 ± 7% -0.1 0.47 ± 7% perf-profile.children.cycles-pp.xfs_trans_log_inode 0.50 ± 6% -0.1 0.42 ± 6% perf-profile.children.cycles-pp.security_file_alloc 0.60 ± 4% -0.1 0.53 ± 5% perf-profile.children.cycles-pp.destroy_inode 0.16 ± 11% -0.1 0.10 ± 16% perf-profile.children.cycles-pp.xfs_btree_get_leaf_keys 0.24 ± 11% -0.1 0.19 ± 11% perf-profile.children.cycles-pp.__check_heap_object 0.38 ± 4% -0.1 0.33 ± 8% perf-profile.children.cycles-pp.__kmalloc_node 0.34 ± 7% -0.1 0.29 ± 7% perf-profile.children.cycles-pp.__kmem_cache_free 0.25 ± 16% -0.1 0.20 ± 11% perf-profile.children.cycles-pp.security_file_free 0.47 ± 4% -0.1 0.42 ± 5% perf-profile.children.cycles-pp.__cond_resched 0.17 ± 13% -0.1 0.12 ± 16% perf-profile.children.cycles-pp.mnt_want_write 0.09 ± 18% -0.0 0.04 ± 75% perf-profile.children.cycles-pp.apparmor_file_permission 0.19 ± 7% -0.0 0.14 ± 18% perf-profile.children.cycles-pp.fsnotify_destroy_marks 0.31 ± 10% -0.0 0.26 ± 5% perf-profile.children.cycles-pp.__virt_addr_valid 0.19 ± 12% -0.0 0.15 ± 13% perf-profile.children.cycles-pp.seq_open 0.36 ± 4% -0.0 0.33 ± 6% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 0.32 ± 6% -0.0 0.28 ± 5% perf-profile.children.cycles-pp.xfs_da_hashname 0.18 ± 10% -0.0 0.14 ± 10% perf-profile.children.cycles-pp.__get_obj_cgroup_from_memcg 0.06 ± 11% +0.0 0.09 ± 11% perf-profile.children.cycles-pp._find_next_and_bit 0.05 ± 8% +0.0 0.09 ± 17% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 0.02 ±141% +0.0 0.06 ± 8% perf-profile.children.cycles-pp.xfs_alloc_vextent_exact_bno 0.08 ± 19% +0.0 0.13 ± 12% perf-profile.children.cycles-pp.__smp_call_single_queue 0.10 ± 26% +0.0 0.15 ± 10% perf-profile.children.cycles-pp.update_rq_clock_task 0.11 ± 18% +0.0 0.16 ± 18% perf-profile.children.cycles-pp.crc_pcl 0.01 ±223% +0.0 0.06 ± 11% perf-profile.children.cycles-pp.rcu_sched_clock_irq 0.10 ± 13% +0.1 0.15 ± 12% perf-profile.children.cycles-pp.xfs_iunlink_lookup 0.12 ± 21% +0.1 0.17 ± 15% perf-profile.children.cycles-pp.xfs_iunlink_log_inode 0.01 ±223% +0.1 0.06 ± 9% perf-profile.children.cycles-pp.check_preempt_curr 0.08 ± 17% +0.1 0.14 ± 13% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.02 ±142% +0.1 0.07 ± 21% perf-profile.children.cycles-pp.xlog_cil_order_cmp 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.__wrgsbase_inactive 0.08 ± 17% +0.1 0.14 ± 12% perf-profile.children.cycles-pp.finish_task_switch 0.01 ±223% +0.1 0.06 ± 21% perf-profile.children.cycles-pp.update_cfs_group 0.06 ± 11% +0.1 0.12 ± 15% perf-profile.children.cycles-pp.update_rq_clock 0.07 ± 18% +0.1 0.13 ± 27% perf-profile.children.cycles-pp.switch_fpu_return 0.02 ±141% +0.1 0.08 ± 20% perf-profile.children.cycles-pp.sched_use_asym_prio 0.02 ±141% +0.1 0.07 ± 15% perf-profile.children.cycles-pp.sysvec_call_function_single 0.10 ± 7% +0.1 0.16 ± 14% perf-profile.children.cycles-pp.xlog_write 0.04 ± 45% +0.1 0.10 ± 13% perf-profile.children.cycles-pp.__update_load_avg_se 0.17 ± 15% +0.1 0.22 ± 13% perf-profile.children.cycles-pp.__list_del_entry_valid_or_report 0.15 ± 10% +0.1 0.21 ± 17% perf-profile.children.cycles-pp.xfs_trans_committed_bulk 0.08 ± 22% +0.1 0.14 ± 4% perf-profile.children.cycles-pp.__switch_to 0.12 ± 22% +0.1 0.18 ± 5% perf-profile.children.cycles-pp.__switch_to_asm 0.01 ±223% +0.1 0.07 ± 10% perf-profile.children.cycles-pp.os_xsave 0.13 ± 14% +0.1 0.19 ± 7% perf-profile.children.cycles-pp.native_sched_clock 0.07 ± 13% +0.1 0.13 ± 13% perf-profile.children.cycles-pp.___perf_sw_event 0.02 ±142% +0.1 0.08 ± 19% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate 0.14 ± 16% +0.1 0.20 ± 13% perf-profile.children.cycles-pp.idle_cpu 0.21 ± 9% +0.1 0.28 ± 9% perf-profile.children.cycles-pp.xfs_ialloc_ag_alloc 0.12 ± 6% +0.1 0.19 ± 9% perf-profile.children.cycles-pp._xfs_buf_obj_cmp 0.10 ± 22% +0.1 0.17 ± 12% perf-profile.children.cycles-pp.prepare_task_switch 0.08 ± 23% +0.1 0.16 ± 16% perf-profile.children.cycles-pp.update_curr 0.11 ± 6% +0.1 0.18 ± 2% perf-profile.children.cycles-pp.sched_clock 0.01 ±223% +0.1 0.09 ± 23% perf-profile.children.cycles-pp.avg_vruntime 0.33 ± 5% +0.1 0.41 ± 5% perf-profile.children.cycles-pp.menu_select 0.11 ± 13% +0.1 0.20 ± 12% perf-profile.children.cycles-pp.ttwu_queue_wakelist 0.12 ± 7% +0.1 0.21 ± 3% perf-profile.children.cycles-pp.sched_clock_cpu 0.13 ± 5% +0.1 0.21 ± 10% perf-profile.children.cycles-pp.update_blocked_averages 0.18 ± 16% +0.1 0.26 ± 11% perf-profile.children.cycles-pp.xfs_buf_item_init 0.01 ±223% +0.1 0.10 ± 13% perf-profile.children.cycles-pp.place_entity 0.10 ± 20% +0.1 0.18 ± 12% perf-profile.children.cycles-pp.set_next_entity 0.23 ± 10% +0.1 0.32 ± 12% perf-profile.children.cycles-pp.xlog_cil_process_committed 0.23 ± 10% +0.1 0.32 ± 12% perf-profile.children.cycles-pp.xlog_cil_committed 0.23 ± 9% +0.1 0.32 ± 11% perf-profile.children.cycles-pp.xlog_state_do_callback 0.23 ± 9% +0.1 0.32 ± 11% perf-profile.children.cycles-pp.xlog_state_do_iclog_callbacks 0.24 ± 10% +0.1 0.33 ± 11% perf-profile.children.cycles-pp.xlog_ioend_work 0.69 ± 4% +0.1 0.79 ± 4% perf-profile.children.cycles-pp.xlog_prepare_iovec 0.36 ± 2% +0.1 0.46 ± 5% perf-profile.children.cycles-pp.xfs_next_bit 0.37 ± 6% +0.1 0.47 ± 7% perf-profile.children.cycles-pp.xfs_buf_offset 0.21 ± 12% +0.1 0.32 ± 11% perf-profile.children.cycles-pp.crc32c_pcl_intel_update 1.26 ± 5% +0.1 1.38 ± 4% perf-profile.children.cycles-pp.xfs_buf_item_size 0.26 ± 14% +0.1 0.38 ± 12% perf-profile.children.cycles-pp.xfs_dinode_calc_crc 0.22 ± 3% +0.1 0.34 ± 8% perf-profile.children.cycles-pp.xlog_cil_push_work 0.40 ± 15% +0.1 0.53 ± 2% perf-profile.children.cycles-pp.schedule_idle 0.27 ± 14% +0.1 0.40 ± 12% perf-profile.children.cycles-pp.crc32c 0.60 ± 12% +0.1 0.74 ± 6% perf-profile.children.cycles-pp._xfs_trans_bjoin 0.24 ± 12% +0.1 0.38 ± 4% perf-profile.children.cycles-pp.sched_ttwu_pending 0.20 ± 11% +0.1 0.35 ± 6% perf-profile.children.cycles-pp.update_load_avg 0.94 ± 9% +0.2 1.09 ± 3% perf-profile.children.cycles-pp.hrtimer_nanosleep 0.97 ± 8% +0.2 1.12 ± 3% perf-profile.children.cycles-pp.__x64_sys_clock_nanosleep 0.94 ± 9% +0.2 1.10 ± 3% perf-profile.children.cycles-pp.common_nsleep 0.92 ± 8% +0.2 1.08 ± 2% perf-profile.children.cycles-pp.do_nanosleep 1.24 ± 8% +0.2 1.40 ± 3% perf-profile.children.cycles-pp.clock_nanosleep 1.00 ± 6% +0.2 1.17 ± 4% perf-profile.children.cycles-pp.xfs_buf_item_size_segment 0.46 ± 5% +0.2 0.63 ± 3% perf-profile.children.cycles-pp.xfs_contig_bits 0.30 ± 16% +0.2 0.47 ± 6% perf-profile.children.cycles-pp.flush_smp_call_function_queue 0.30 ± 12% +0.2 0.48 ± 5% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 0.32 ± 7% +0.2 0.50 ± 3% perf-profile.children.cycles-pp.xfs_imap_to_bp 2.00 ± 3% +0.2 2.19 ± 2% perf-profile.children.cycles-pp.xlog_cil_alloc_shadow_bufs 10.13 +0.2 10.34 perf-profile.children.cycles-pp.do_unlinkat 0.21 ± 8% +0.2 0.42 ± 6% perf-profile.children.cycles-pp.dequeue_entity 0.21 ± 10% +0.2 0.44 ± 6% perf-profile.children.cycles-pp.enqueue_entity 0.26 ± 4% +0.2 0.51 ± 6% perf-profile.children.cycles-pp.dequeue_task_fair 0.82 ± 9% +0.3 1.10 ± 6% perf-profile.children.cycles-pp.poll_idle 0.25 ± 6% +0.3 0.54 ± 6% perf-profile.children.cycles-pp.enqueue_task_fair 0.27 ± 8% +0.3 0.57 ± 5% perf-profile.children.cycles-pp.activate_task 0.54 ± 5% +0.3 0.85 ± 7% perf-profile.children.cycles-pp.xfs_iunlink_log_dinode 0.31 ± 8% +0.3 0.62 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate 0.60 ± 5% +0.3 0.94 ± 5% perf-profile.children.cycles-pp.xfs_iunlink_item_precommit 0.98 ± 7% +0.3 1.33 ± 4% perf-profile.children.cycles-pp.xfs_iunlink 1.07 ± 2% +0.4 1.43 ± 7% perf-profile.children.cycles-pp.memcpy_orig 0.85 ± 7% +0.4 1.23 ± 4% perf-profile.children.cycles-pp.update_sg_lb_stats 0.90 ± 10% +0.4 1.29 ± 3% perf-profile.children.cycles-pp.xfs_buf_item_release 0.98 ± 7% +0.4 1.38 ± 4% perf-profile.children.cycles-pp.update_sd_lb_stats 1.01 ± 7% +0.4 1.41 ± 4% perf-profile.children.cycles-pp.find_busiest_group 0.58 ± 10% +0.5 1.04 ± 4% perf-profile.children.cycles-pp.xfs_buf_unlock 0.51 ± 11% +0.5 0.98 ± 4% perf-profile.children.cycles-pp.up 0.42 ± 6% +0.5 0.90 ± 2% perf-profile.children.cycles-pp.try_to_wake_up 3.57 ± 4% +0.5 4.08 ± 2% perf-profile.children.cycles-pp.xfs_difree 1.21 ± 7% +0.5 1.72 ± 3% perf-profile.children.cycles-pp.load_balance 4.16 ± 4% +0.6 4.72 ± 2% perf-profile.children.cycles-pp.xfs_ifree 7.84 +0.6 8.39 perf-profile.children.cycles-pp.vfs_unlink 1.30 ± 7% +0.6 1.86 ± 2% perf-profile.children.cycles-pp.newidle_balance 7.31 +0.6 7.92 perf-profile.children.cycles-pp.xfs_vn_unlink 7.26 +0.6 7.89 perf-profile.children.cycles-pp.xfs_remove 1.47 ± 5% +0.7 2.18 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair 5.14 +0.7 5.84 ± 2% perf-profile.children.cycles-pp.xlog_cil_insert_items 4.62 ± 2% +0.7 5.34 ± 3% perf-profile.children.cycles-pp.xlog_cil_insert_format_items 3.04 ± 2% +0.7 3.78 ± 4% perf-profile.children.cycles-pp.xfs_buf_item_format 2.80 ± 2% +0.7 3.54 ± 4% perf-profile.children.cycles-pp.xfs_buf_item_format_segment 4.89 ± 4% +0.8 5.73 ± 6% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 1.72 ± 5% +0.9 2.59 ± 5% perf-profile.children.cycles-pp.xfs_ialloc_read_agi 10.46 +1.0 11.46 ± 2% perf-profile.children.cycles-pp.xlog_cil_commit 3.44 ± 5% +1.1 4.49 ± 6% perf-profile.children.cycles-pp.intel_idle 0.92 ± 6% +1.1 2.04 ± 4% perf-profile.children.cycles-pp.schedule_timeout 1.20 ± 6% +1.2 2.35 ± 4% perf-profile.children.cycles-pp.down 1.27 ± 6% +1.2 2.43 ± 4% perf-profile.children.cycles-pp.xfs_buf_lock 0.96 ± 7% +1.2 2.12 ± 4% perf-profile.children.cycles-pp.___down_common 0.97 ± 7% +1.2 2.13 ± 4% perf-profile.children.cycles-pp.__down 2.44 ± 5% +1.2 3.61 ± 3% perf-profile.children.cycles-pp.xfs_read_agi 13.92 +1.2 15.08 perf-profile.children.cycles-pp.__xfs_trans_commit 1.42 ± 7% +1.2 2.59 ± 4% perf-profile.children.cycles-pp.xfs_buf_find_lock 3.05 ± 4% +1.2 4.26 ± 2% perf-profile.children.cycles-pp.xfs_buf_read_map 2.92 ± 4% +1.2 4.15 ± 3% perf-profile.children.cycles-pp.xfs_buf_get_map 1.90 ± 5% +1.2 3.14 ± 3% perf-profile.children.cycles-pp.schedule 1.95 ± 5% +1.3 3.23 ± 4% perf-profile.children.cycles-pp.xfs_buf_lookup 2.30 ± 5% +1.3 3.59 ± 2% perf-profile.children.cycles-pp.__schedule 4.01 ± 4% +1.4 5.39 ± 2% perf-profile.children.cycles-pp.xfs_trans_read_buf_map 10.49 ± 3% +1.5 11.95 ± 2% perf-profile.children.cycles-pp.xfs_inodegc_worker 3.82 ± 9% +1.5 5.30 ± 7% perf-profile.children.cycles-pp.intel_idle_irq 10.24 ± 3% +1.5 11.72 ± 2% perf-profile.children.cycles-pp.xfs_inactive 10.17 ± 3% +1.5 11.66 ± 2% perf-profile.children.cycles-pp.xfs_inactive_ifree 11.52 ± 3% +1.6 13.12 ± 2% perf-profile.children.cycles-pp.ret_from_fork_asm 11.52 ± 3% +1.6 13.12 ± 2% perf-profile.children.cycles-pp.ret_from_fork 11.52 ± 3% +1.6 13.12 ± 2% perf-profile.children.cycles-pp.kthread 11.08 ± 3% +1.7 12.75 ± 2% perf-profile.children.cycles-pp.worker_thread 10.98 ± 3% +1.7 12.65 ± 2% perf-profile.children.cycles-pp.process_one_work 30.10 ± 2% +2.4 32.52 perf-profile.children.cycles-pp.cpuidle_enter_state 30.10 ± 2% +2.4 32.54 perf-profile.children.cycles-pp.cpuidle_enter 30.58 ± 2% +2.6 33.14 perf-profile.children.cycles-pp.cpuidle_idle_call 31.47 ± 2% +2.9 34.40 perf-profile.children.cycles-pp.do_idle 31.48 ± 2% +2.9 34.42 perf-profile.children.cycles-pp.secondary_startup_64_no_verify 31.48 ± 2% +2.9 34.42 perf-profile.children.cycles-pp.cpu_startup_entry 30.93 ± 2% +3.0 33.90 perf-profile.children.cycles-pp.start_secondary 4.07 ± 3% -0.6 3.45 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 3.56 ± 3% -0.5 3.06 ± 3% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 2.34 ± 4% -0.4 1.96 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 2.50 ± 3% -0.3 2.16 ± 4% perf-profile.self.cycles-pp.stress_mwc32modn 1.90 ± 4% -0.2 1.67 ± 4% perf-profile.self.cycles-pp.__entry_text_start 1.23 ± 2% -0.2 1.05 ± 4% perf-profile.self.cycles-pp.link_path_walk 0.65 ± 6% -0.1 0.56 ± 7% perf-profile.self.cycles-pp.stress_filename 0.45 ± 7% -0.1 0.36 ± 11% perf-profile.self.cycles-pp.down_read 0.44 ± 8% -0.1 0.35 ± 9% perf-profile.self.cycles-pp.generic_permission 0.42 ± 5% -0.1 0.34 ± 9% perf-profile.self.cycles-pp.xlog_grant_add_space 0.52 ± 6% -0.1 0.44 ± 8% perf-profile.self.cycles-pp.xfs_buf_get_map 0.47 ± 7% -0.1 0.41 ± 4% perf-profile.self.cycles-pp.xfs_log_ticket_ungrant 0.24 ± 10% -0.1 0.18 ± 11% perf-profile.self.cycles-pp.__check_heap_object 0.21 ± 8% -0.1 0.16 ± 11% perf-profile.self.cycles-pp.step_into 0.20 ± 6% -0.0 0.16 ± 8% perf-profile.self.cycles-pp.do_dentry_open 0.19 ± 7% -0.0 0.15 ± 14% perf-profile.self.cycles-pp.set_nlink 0.32 ± 6% -0.0 0.28 ± 5% perf-profile.self.cycles-pp.xfs_da_hashname 0.08 ± 17% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.mnt_want_write 0.17 ± 10% -0.0 0.14 ± 11% perf-profile.self.cycles-pp.__get_obj_cgroup_from_memcg 0.06 ± 11% +0.0 0.08 ± 17% perf-profile.self.cycles-pp.newidle_balance 0.06 ± 11% +0.0 0.08 ± 10% perf-profile.self.cycles-pp.load_balance 0.06 ± 14% +0.0 0.08 ± 11% perf-profile.self.cycles-pp._find_next_and_bit 0.15 ± 6% +0.0 0.18 ± 11% perf-profile.self.cycles-pp.read_tsc 0.06 ± 48% +0.0 0.10 ± 9% perf-profile.self.cycles-pp.update_rq_clock_task 0.08 ± 20% +0.0 0.13 ± 16% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.52 ± 7% +0.0 0.56 ± 4% perf-profile.self.cycles-pp.xfs_buf_item_size_segment 0.02 ±141% +0.1 0.07 ± 23% perf-profile.self.cycles-pp.xlog_cil_order_cmp 0.11 ± 15% +0.1 0.16 ± 18% perf-profile.self.cycles-pp.crc_pcl 0.04 ± 71% +0.1 0.10 ± 11% perf-profile.self.cycles-pp.pick_next_task_fair 0.06 ± 11% +0.1 0.12 ± 10% perf-profile.self.cycles-pp.___perf_sw_event 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.__wrgsbase_inactive 0.04 ± 44% +0.1 0.10 ± 8% perf-profile.self.cycles-pp.enqueue_entity 0.08 ± 14% +0.1 0.14 ± 7% perf-profile.self.cycles-pp.update_load_avg 0.03 ±100% +0.1 0.08 ± 14% perf-profile.self.cycles-pp.update_curr 0.12 ± 12% +0.1 0.18 ± 10% perf-profile.self.cycles-pp.native_sched_clock 0.15 ± 12% +0.1 0.21 ± 12% perf-profile.self.cycles-pp.__list_del_entry_valid_or_report 0.03 ± 70% +0.1 0.09 ± 9% perf-profile.self.cycles-pp.dequeue_task_fair 0.12 ± 21% +0.1 0.17 ± 5% perf-profile.self.cycles-pp.__switch_to_asm 0.08 ± 21% +0.1 0.13 ± 3% perf-profile.self.cycles-pp.__switch_to 0.04 ± 71% +0.1 0.09 ± 15% perf-profile.self.cycles-pp.__update_load_avg_se 0.13 ± 18% +0.1 0.19 ± 12% perf-profile.self.cycles-pp.idle_cpu 0.00 +0.1 0.06 ± 17% perf-profile.self.cycles-pp.update_cfs_group 0.02 ±142% +0.1 0.08 ± 19% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate 0.02 ±142% +0.1 0.08 ± 17% perf-profile.self.cycles-pp.try_to_wake_up 0.00 +0.1 0.07 ± 11% perf-profile.self.cycles-pp.os_xsave 0.12 ± 6% +0.1 0.19 ± 10% perf-profile.self.cycles-pp._xfs_buf_obj_cmp 0.18 ± 10% +0.1 0.26 ± 6% perf-profile.self.cycles-pp.menu_select 0.02 ±142% +0.1 0.09 ± 19% perf-profile.self.cycles-pp.enqueue_task_fair 0.30 ± 8% +0.1 0.38 ± 8% perf-profile.self.cycles-pp.xfs_buf_offset 0.14 ± 7% +0.1 0.22 ± 11% perf-profile.self.cycles-pp.__schedule 0.00 +0.1 0.08 ± 22% perf-profile.self.cycles-pp.avg_vruntime 0.67 ± 3% +0.1 0.76 ± 4% perf-profile.self.cycles-pp.xlog_prepare_iovec 0.16 ± 17% +0.1 0.25 ± 10% perf-profile.self.cycles-pp.xfs_buf_item_init 0.35 ± 3% +0.1 0.45 ± 6% perf-profile.self.cycles-pp.xfs_next_bit 0.41 ± 4% +0.1 0.56 ± 3% perf-profile.self.cycles-pp.xfs_contig_bits 0.56 ± 6% +0.3 0.82 ± 4% perf-profile.self.cycles-pp.update_sg_lb_stats 0.81 ± 10% +0.3 1.08 ± 7% perf-profile.self.cycles-pp.poll_idle 1.04 ± 2% +0.3 1.39 ± 7% perf-profile.self.cycles-pp.memcpy_orig 3.44 ± 5% +1.1 4.49 ± 6% perf-profile.self.cycles-pp.intel_idle 3.45 ± 9% +1.4 4.83 ± 8% perf-profile.self.cycles-pp.intel_idle_irq *************************************************************************************************** lkp-icl-2sp8: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory ========================================================================================= class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: memory/gcc-12/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp8/lockbus/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 13656 ± 9% -14.9% 11627 ± 9% numa-meminfo.node1.Active(anon) 60.52 +1.4% 61.36 turbostat.RAMWatt 3887 ± 4% -17.0% 3224 ± 11% vmstat.system.cs 3413 ± 9% -15.0% 2902 ± 9% numa-vmstat.node1.nr_active_anon 3413 ± 9% -15.0% 2901 ± 9% numa-vmstat.node1.nr_zone_active_anon 3966 ± 3% -6.2% 3721 ± 2% proc-vmstat.nr_active_anon 3966 ± 3% -6.2% 3721 ± 2% proc-vmstat.nr_zone_active_anon 69264 -6.0% 65120 stress-ng.lockbus.ops 1152 -6.0% 1083 stress-ng.lockbus.ops_per_sec 17830 -32.9% 11961 ± 7% stress-ng.time.involuntary_context_switches 36007 -32.3% 24373 ± 7% stress-ng.time.voluntary_context_switches 13.93 -0.6 13.32 perf-stat.i.cache-miss-rate% 3761 ± 4% -18.5% 3066 ± 12% perf-stat.i.context-switches 11.55 +4.1% 12.02 ± 2% perf-stat.i.cpi 27963 ± 2% -3.4% 27016 ± 2% perf-stat.i.dTLB-store-misses 142439 ± 2% -9.4% 128993 ± 4% perf-stat.i.node-load-misses 4.43 +2.9% 4.56 perf-stat.overall.cpi 0.23 -2.8% 0.22 perf-stat.overall.ipc 3706 ± 4% -18.5% 3019 ± 12% perf-stat.ps.context-switches 27533 ± 2% -3.5% 26568 ± 2% perf-stat.ps.dTLB-store-misses 140198 ± 2% -9.4% 126952 ± 4% perf-stat.ps.node-load-misses 0.01 ± 42% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.01 ± 11% +98.0% 0.02 ± 31% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.11 ± 7% +1058.4% 1.30 ± 10% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.02 ± 85% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.63 ± 42% +281.6% 2.41 ± 39% perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.03 ± 9% +992.3% 0.28 ± 14% perf-sched.total_sch_delay.average.ms 113.93 ± 10% +40.6% 160.23 ± 7% perf-sched.total_wait_and_delay.average.ms 8907 ± 12% -28.7% 6347 ± 9% perf-sched.total_wait_and_delay.count.ms 113.90 ± 10% +40.4% 159.95 ± 7% perf-sched.total_wait_time.average.ms 234.83 ± 3% -23.0% 180.82 ± 10% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 189.41 +49.2% 282.58 ± 5% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 141.64 ± 5% +46.2% 207.06 ± 3% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1500 -32.8% 1007 ± 5% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 1496 -32.6% 1008 ± 5% perf-sched.wait_and_delay.count.schedule_timeout.___down_common.__down_interruptible.down_interruptible 1495 -32.6% 1008 ± 5% perf-sched.wait_and_delay.count.schedule_timeout.msleep_interruptible.split_lock_warn.handle_user_split_lock 1837 -26.4% 1352 ± 4% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 216.51 ± 2% +53.6% 332.55 ± 5% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 355.47 ±118% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 234.82 ± 3% -23.0% 180.82 ± 10% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 189.41 +49.2% 282.57 ± 5% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 141.53 ± 5% +45.4% 205.77 ± 3% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1173 ±114% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 216.51 ± 2% +53.6% 332.55 ± 5% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 56.75 -3.5 53.26 ± 2% perf-profile.calltrace.cycles-pp.stress_lockbus 2.21 ± 4% +0.2 2.39 ± 5% perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter 2.02 ± 7% +0.4 2.45 ± 13% perf-profile.calltrace.cycles-pp.menu_select.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 20.55 ± 3% +1.5 22.06 ± 2% perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 46.79 +2.4 49.24 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter 37.90 +2.8 40.70 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 38.00 +3.2 41.16 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 37.58 +3.2 40.75 perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 40.36 +3.3 43.67 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 40.84 +3.3 44.18 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 40.84 +3.3 44.18 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 40.79 +3.4 44.14 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 41.44 +3.4 44.87 ± 2% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 56.78 -3.5 53.30 ± 2% perf-profile.children.cycles-pp.stress_lockbus 0.68 ± 9% -0.1 0.54 ± 7% perf-profile.children.cycles-pp.irq_work_run_list 0.68 ± 10% -0.1 0.53 ± 8% perf-profile.children.cycles-pp.asm_sysvec_irq_work 0.67 ± 9% -0.1 0.53 ± 8% perf-profile.children.cycles-pp.sysvec_irq_work 0.67 ± 9% -0.1 0.53 ± 8% perf-profile.children.cycles-pp.__sysvec_irq_work 0.67 ± 9% -0.1 0.53 ± 8% perf-profile.children.cycles-pp.irq_work_run 0.67 ± 9% -0.1 0.53 ± 8% perf-profile.children.cycles-pp.irq_work_single 0.67 ± 9% -0.1 0.53 ± 8% perf-profile.children.cycles-pp._printk 0.67 ± 9% -0.1 0.53 ± 8% perf-profile.children.cycles-pp.vprintk_emit 0.66 ± 10% -0.1 0.52 ± 7% perf-profile.children.cycles-pp.console_unlock 0.66 ± 10% -0.1 0.52 ± 7% perf-profile.children.cycles-pp.console_flush_all 0.63 ± 9% -0.1 0.50 ± 8% perf-profile.children.cycles-pp.serial8250_console_write 0.60 ± 8% -0.1 0.47 ± 8% perf-profile.children.cycles-pp.wait_for_lsr 0.49 ± 11% -0.1 0.36 ± 9% perf-profile.children.cycles-pp.io_serial_in 0.26 ± 15% -0.1 0.16 ± 21% perf-profile.children.cycles-pp.asm_exc_alignment_check 0.56 ± 9% -0.1 0.46 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.25 ± 15% -0.1 0.16 ± 20% perf-profile.children.cycles-pp.exc_alignment_check 0.20 ± 14% -0.1 0.12 ± 25% perf-profile.children.cycles-pp.msleep_interruptible 0.24 ± 15% -0.1 0.16 ± 20% perf-profile.children.cycles-pp.handle_user_split_lock 0.24 ± 15% -0.1 0.16 ± 20% perf-profile.children.cycles-pp.split_lock_warn 0.16 ± 17% -0.1 0.09 ± 20% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.14 ± 20% -0.1 0.07 ± 47% perf-profile.children.cycles-pp.irqentry_exit_to_user_mode 0.14 ± 18% -0.1 0.08 ± 20% perf-profile.children.cycles-pp.exit_to_user_mode_loop 0.13 ± 11% -0.0 0.08 ± 26% perf-profile.children.cycles-pp.nr_iowait_cpu 0.11 ± 15% -0.0 0.06 ± 50% perf-profile.children.cycles-pp.task_mm_cid_work 0.10 ± 15% -0.0 0.05 ± 46% perf-profile.children.cycles-pp.local_clock_noinstr 0.12 ± 12% -0.0 0.07 ± 21% perf-profile.children.cycles-pp.task_work_run 0.20 ± 14% +0.1 0.26 ± 7% perf-profile.children.cycles-pp.hrtimer_next_event_without 0.06 ± 47% +0.1 0.17 ± 30% perf-profile.children.cycles-pp.irq_work_needs_cpu 0.00 +0.2 0.16 ±122% perf-profile.children.cycles-pp.timekeeping_max_deferment 2.06 ± 7% +0.4 2.49 ± 13% perf-profile.children.cycles-pp.menu_select 30.75 +1.0 31.77 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 38.45 +2.9 41.35 perf-profile.children.cycles-pp.cpuidle_enter 38.32 +2.9 41.24 perf-profile.children.cycles-pp.cpuidle_enter_state 37.88 +2.9 40.82 perf-profile.children.cycles-pp.acpi_idle_enter 37.79 +2.9 40.73 perf-profile.children.cycles-pp.acpi_safe_halt 40.84 +3.3 44.18 ± 2% perf-profile.children.cycles-pp.start_secondary 40.98 +3.4 44.37 ± 2% perf-profile.children.cycles-pp.cpuidle_idle_call 41.44 +3.4 44.87 ± 2% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 41.44 +3.4 44.87 ± 2% perf-profile.children.cycles-pp.cpu_startup_entry 41.43 +3.4 44.87 ± 2% perf-profile.children.cycles-pp.do_idle 56.26 -3.3 52.94 ± 2% perf-profile.self.cycles-pp.stress_lockbus 0.47 ± 10% -0.1 0.35 ± 10% perf-profile.self.cycles-pp.io_serial_in 0.55 ± 9% -0.1 0.46 ± 9% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.09 ± 11% -0.0 0.05 ± 72% perf-profile.self.cycles-pp.task_mm_cid_work 0.03 ±101% +0.1 0.08 ± 14% perf-profile.self.cycles-pp.hrtimer_next_event_without 0.15 ± 9% +0.1 0.21 ± 12% perf-profile.self.cycles-pp.cpuidle_idle_call 0.05 ± 48% +0.1 0.13 ± 34% perf-profile.self.cycles-pp.irq_work_needs_cpu 0.01 ±223% +0.1 0.10 ± 20% perf-profile.self.cycles-pp.cpuidle_governor_latency_req 0.00 +0.2 0.16 ±127% perf-profile.self.cycles-pp.timekeeping_max_deferment 0.39 ± 5% +0.2 0.63 ± 10% perf-profile.self.cycles-pp.menu_select 23.01 ± 3% +3.5 26.48 ± 3% perf-profile.self.cycles-pp.acpi_safe_halt *************************************************************************************************** lkp-skl-d08: 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory ========================================================================================= class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: interrupt/gcc-12/performance/1SSD/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-skl-d08/sigfd/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 253708 ± 13% +136.2% 599134 ± 33% cpuidle..usage 35.43 +4.5 39.94 mpstat.cpu.all.usr% 56.93 -7.8% 52.48 iostat.cpu.system 34.43 +12.3% 38.67 iostat.cpu.user 166.50 ± 62% -87.9% 20.17 ± 37% perf-c2c.DRAM.local 2950 ± 4% +36.7% 4034 ± 3% perf-c2c.HITM.local 6322874 -53.5% 2941783 vmstat.memory.cache 25427591 +13.6% 28889500 vmstat.memory.free 55.00 +16.4% 64.00 vmstat.procs.r 3896193 -98.6% 53463 ± 21% vmstat.system.cs 81531 -30.6% 56593 vmstat.system.in 3.441e+08 +17.6% 4.047e+08 stress-ng.sigfd.ops 5734484 +17.6% 6745114 stress-ng.sigfd.ops_per_sec 1.277e+08 -99.7% 375150 stress-ng.time.involuntary_context_switches 3369 +1.1% 3406 stress-ng.time.percent_of_cpu_this_job_got 1343 -7.9% 1237 stress-ng.time.system_time 757.06 +18.0% 893.08 stress-ng.time.user_time 1.282e+08 -98.2% 2309843 ± 19% stress-ng.time.voluntary_context_switches 0.23 ± 58% +271.7% 0.86 ± 14% turbostat.CPU%c6 0.09 ± 4% -35.8% 0.06 ± 8% turbostat.IPC 5368468 -30.8% 3714484 turbostat.IRQ 31407 ± 43% +994.9% 343860 ± 49% turbostat.POLL 0.05 ±223% +1100.0% 0.64 ± 18% turbostat.Pkg%pc2 46.33 ± 3% -8.6% 42.33 turbostat.PkgTmp 179.76 -6.0% 169.06 turbostat.PkgWatt 1.01 -11.4% 0.90 turbostat.RAMWatt 574041 ± 79% -63.0% 212329 ± 79% meminfo.Active 573863 ± 79% -77.5% 128881 ± 17% meminfo.Active(anon) 354239 -15.1% 300649 meminfo.AnonPages 6204630 -53.9% 2863247 meminfo.Cached 4724536 -72.3% 1307972 ± 2% meminfo.Committed_AS 3273969 ± 14% -90.1% 323898 meminfo.Inactive 3273680 ± 14% -90.2% 319651 meminfo.Inactive(anon) 1289574 ± 12% -95.7% 54909 ± 5% meminfo.Mapped 25234814 +13.9% 28730239 meminfo.MemAvailable 25458262 +13.5% 28886614 meminfo.MemFree 7103772 -48.3% 3675420 ± 4% meminfo.Memused 10748 ± 3% -25.5% 8009 ± 2% meminfo.PageTables 3493681 -95.8% 148301 ± 18% meminfo.Shmem 7140927 -46.4% 3828694 ± 4% meminfo.max_used_kB 143464 ± 79% -77.5% 32218 ± 17% proc-vmstat.nr_active_anon 88568 -15.1% 75170 proc-vmstat.nr_anon_pages 628908 +14.0% 716679 proc-vmstat.nr_dirty_background_threshold 1259356 +14.0% 1435113 proc-vmstat.nr_dirty_threshold 1551266 -52.5% 736621 ± 5% proc-vmstat.nr_file_pages 6364467 +13.5% 7221660 proc-vmstat.nr_free_pages 818533 ± 14% -90.2% 79908 proc-vmstat.nr_inactive_anon 10060 -2.3% 9828 proc-vmstat.nr_kernel_stack 322934 ± 12% -95.8% 13709 ± 5% proc-vmstat.nr_mapped 2688 ± 3% -25.5% 2001 ± 2% proc-vmstat.nr_page_table_pages 873528 -95.8% 37060 ± 18% proc-vmstat.nr_shmem 143464 ± 79% -77.5% 32218 ± 17% proc-vmstat.nr_zone_active_anon 818533 ± 14% -90.2% 79908 proc-vmstat.nr_zone_inactive_anon 2677779 +2.6% 2747266 proc-vmstat.numa_hit 2677778 +2.6% 2747331 proc-vmstat.numa_local 28685 ± 38% +92.4% 55189 ± 12% proc-vmstat.pgactivate 2713395 +2.5% 2782484 proc-vmstat.pgalloc_normal 317818 ± 3% -21.8% 248483 proc-vmstat.pgfault 1394260 +92.6% 2685162 proc-vmstat.pgfree 8198 +3.3% 8467 proc-vmstat.pgreuse 938223 +14.4% 1073726 sched_debug.cfs_rq:/.avg_vruntime.avg 994157 +20.0% 1192780 ± 2% sched_debug.cfs_rq:/.avg_vruntime.max 19325 ± 9% +150.1% 48339 ± 13% sched_debug.cfs_rq:/.avg_vruntime.stddev 1.10 ± 3% +12.7% 1.24 sched_debug.cfs_rq:/.h_nr_running.avg 551.08 ± 4% +29.2% 712.17 ± 4% sched_debug.cfs_rq:/.load_avg.max 938223 +14.4% 1073726 sched_debug.cfs_rq:/.min_vruntime.avg 994157 +20.0% 1192790 ± 2% sched_debug.cfs_rq:/.min_vruntime.max 19325 ± 9% +150.1% 48340 ± 13% sched_debug.cfs_rq:/.min_vruntime.stddev 1201 +12.5% 1350 sched_debug.cfs_rq:/.runnable_avg.avg 2025 ± 4% +18.2% 2393 ± 4% sched_debug.cfs_rq:/.runnable_avg.max 276.13 ± 9% +61.0% 444.71 ± 8% sched_debug.cfs_rq:/.runnable_avg.stddev 476.67 ± 5% -22.2% 370.83 ± 9% sched_debug.cfs_rq:/.util_avg.min 203.19 ± 11% +32.4% 269.09 ± 10% sched_debug.cfs_rq:/.util_avg.stddev 174.48 ± 7% +16.4% 203.01 ± 8% sched_debug.cfs_rq:/.util_est_enqueued.avg 380990 ± 3% -31.8% 259898 ± 4% sched_debug.cpu.avg_idle.avg 307426 ± 4% -39.9% 184804 ± 9% sched_debug.cpu.avg_idle.stddev 3.81 ± 10% -38.1% 2.36 ± 29% sched_debug.cpu.clock.stddev 73.87 ± 4% -12.2% 64.86 sched_debug.cpu.clock_task.stddev 1.11 ± 3% +12.3% 1.25 sched_debug.cpu.nr_running.avg 3367531 -98.5% 50921 ± 19% sched_debug.cpu.nr_switches.avg 3625378 -94.0% 218664 ± 49% sched_debug.cpu.nr_switches.max 2489663 ± 13% -99.5% 12797 ± 5% sched_debug.cpu.nr_switches.min 268237 ± 21% -84.7% 40977 ± 40% sched_debug.cpu.nr_switches.stddev 0.13 ± 3% -43.8% 0.07 ± 12% perf-stat.i.MPKI 7.37e+09 -41.2% 4.333e+09 perf-stat.i.branch-instructions 1.75 +0.1 1.88 perf-stat.i.branch-miss-rate% 1.218e+08 -35.1% 79006703 perf-stat.i.branch-misses 4.54 ± 7% -2.7 1.89 ± 11% perf-stat.i.cache-miss-rate% 5337171 ± 4% -70.8% 1556171 ± 4% perf-stat.i.cache-misses 1.324e+08 ± 10% +17.0% 1.549e+08 ± 4% perf-stat.i.cache-references 4092256 -98.7% 55203 ± 21% perf-stat.i.context-switches 3.33 +56.7% 5.22 perf-stat.i.cpi 185.40 ± 4% +29.6% 240.27 ± 7% perf-stat.i.cpu-migrations 2.551e+12 ±223% -100.0% 25988310 perf-stat.i.dTLB-load-misses 1.054e+10 -37.7% 6.568e+09 perf-stat.i.dTLB-loads 0.01 ± 39% -0.0 0.00 ± 10% perf-stat.i.dTLB-store-miss-rate% 756875 ± 40% -90.4% 72744 ± 10% perf-stat.i.dTLB-store-misses 6.348e+09 -35.9% 4.07e+09 perf-stat.i.dTLB-stores 34.90 +47.5 82.41 ± 2% perf-stat.i.iTLB-load-miss-rate% 25434272 ± 2% +18.1% 30031173 ± 4% perf-stat.i.iTLB-load-misses 47308644 -88.6% 5405815 ± 13% perf-stat.i.iTLB-loads 3.885e+10 -37.4% 2.432e+10 perf-stat.i.instructions 3454 ± 3% -21.7% 2706 ± 14% perf-stat.i.instructions-per-iTLB-miss 0.33 -31.3% 0.23 perf-stat.i.ipc 267.83 ± 5% -26.7% 196.31 ± 9% perf-stat.i.metric.K/sec 678.76 -38.1% 419.96 perf-stat.i.metric.M/sec 3629 ± 4% -29.7% 2551 perf-stat.i.minor-faults 1176942 ± 19% -87.4% 148587 ± 6% perf-stat.i.node-loads 553456 ± 6% -48.3% 286312 ± 2% perf-stat.i.node-stores 3629 ± 4% -29.7% 2552 perf-stat.i.page-faults 0.14 ± 4% -54.1% 0.06 ± 5% perf-stat.overall.MPKI 1.65 +0.2 1.82 perf-stat.overall.branch-miss-rate% 4.07 ± 11% -3.1 0.99 ± 7% perf-stat.overall.cache-miss-rate% 3.35 +59.2% 5.34 perf-stat.overall.cpi 24445 ± 4% +246.8% 84772 ± 4% perf-stat.overall.cycles-between-cache-misses 0.01 ± 40% -0.0 0.00 ± 10% perf-stat.overall.dTLB-store-miss-rate% 34.96 +49.8 84.73 ± 2% perf-stat.overall.iTLB-load-miss-rate% 1528 ± 2% -47.0% 810.40 ± 4% perf-stat.overall.instructions-per-iTLB-miss 0.30 -37.2% 0.19 perf-stat.overall.ipc 7.255e+09 -41.2% 4.267e+09 perf-stat.ps.branch-instructions 1.199e+08 -35.3% 77595921 perf-stat.ps.branch-misses 5256369 ± 4% -71.2% 1512093 ± 4% perf-stat.ps.cache-misses 1.304e+08 ± 10% +17.3% 1.53e+08 ± 4% perf-stat.ps.cache-references 4027980 -98.6% 54625 ± 21% perf-stat.ps.context-switches 182.42 ± 4% +29.9% 237.02 ± 7% perf-stat.ps.cpu-migrations 2.504e+12 ±223% -100.0% 25641665 perf-stat.ps.dTLB-load-misses 1.038e+10 -37.6% 6.472e+09 perf-stat.ps.dTLB-loads 745124 ± 40% -90.4% 71588 ± 10% perf-stat.ps.dTLB-store-misses 6.249e+09 -35.8% 4.013e+09 perf-stat.ps.dTLB-stores 25036838 ± 2% +18.4% 29632745 ± 4% perf-stat.ps.iTLB-load-misses 46566778 -88.6% 5330136 ± 13% perf-stat.ps.iTLB-loads 3.824e+10 -37.4% 2.396e+10 perf-stat.ps.instructions 3569 ± 4% -30.8% 2469 perf-stat.ps.minor-faults 1158877 ± 19% -87.5% 144748 ± 6% perf-stat.ps.node-loads 545337 ± 6% -48.4% 281416 ± 2% perf-stat.ps.node-stores 3569 ± 4% -30.8% 2469 perf-stat.ps.page-faults 2.432e+12 -37.4% 1.524e+12 perf-stat.total.instructions 0.14 ± 87% +1555.7% 2.31 ± 15% perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.56 ± 17% +319.2% 2.36 ± 15% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.60 ±143% +1009.2% 6.60 ±109% perf-sched.sch_delay.avg.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.02 ± 4% +10979.7% 2.46 ± 48% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.29 ± 23% +499.9% 1.73 ± 16% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.11 ± 48% +849.5% 1.06 ± 23% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.01 ± 14% +123.3% 0.02 ± 29% perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.04 ± 17% +2052.2% 0.80 ± 19% perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.01 ± 42% +22451.8% 3.12 ± 6% perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 0.05 ± 90% +465.7% 0.27 ± 23% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.01 ± 11% +11390.4% 1.40 ± 22% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.48 ± 16% +174.2% 1.33 ± 8% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.01 ± 5% +12626.3% 1.21 ± 19% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.02 ± 77% +15778.6% 2.73 ± 5% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.02 ± 2% +4345.4% 0.96 ± 37% perf-sched.sch_delay.avg.ms.signalfd_dequeue.signalfd_read.vfs_read.ksys_read 0.01 ± 27% +1287.7% 0.17 ± 17% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.08 ± 80% +1825.3% 1.56 ± 61% perf-sched.sch_delay.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read 0.03 ±143% +3719.1% 1.16 ± 40% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.02 ± 26% +26270.4% 4.31 ±162% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1.92 ± 76% +173.4% 5.26 ± 7% perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 2.55 ± 25% +93.7% 4.93 ± 10% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.62 ±135% +959.9% 6.60 ±109% perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 1.50 ± 55% +200.2% 4.52 ± 26% perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 2.78 ± 22% -91.5% 0.24 ±142% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 3.02 ± 19% +49.5% 4.51 ± 9% perf-sched.sch_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.69 ±216% +772.6% 6.01 ± 53% perf-sched.sch_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 0.56 ± 98% +458.0% 3.10 ± 6% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.03 ± 43% +11384.2% 3.02 ± 14% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ± 7% +21834.2% 2.67 ± 27% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 2.86 ±221% +721.8% 23.50 ± 77% perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 1.15 ± 78% +245.8% 3.98 perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.12 ± 96% +1205.0% 1.56 ± 61% perf-sched.sch_delay.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read 0.54 ±165% +751.0% 4.60 ± 13% perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 1.18 ± 74% +3965.3% 47.99 ±196% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.01 +6693.9% 0.75 ± 27% perf-sched.total_sch_delay.average.ms 0.09 +6069.7% 5.43 ± 27% perf-sched.total_wait_and_delay.average.ms 8083586 -98.2% 147185 ± 27% perf-sched.total_wait_and_delay.count.ms 0.08 ± 2% +5967.6% 4.68 ± 27% perf-sched.total_wait_time.average.ms 1.48 ±126% +21665.4% 322.06 ± 50% perf-sched.wait_and_delay.avg.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 18.51 ± 7% +16.4% 21.54 ± 6% perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 1.51 ± 29% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 0.26 ± 83% +48951.0% 125.33 ± 40% perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.14 ± 81% +1.3e+05% 176.52 ± 57% perf-sched.wait_and_delay.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 374.91 ± 26% +114.5% 804.01 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.13 ± 13% +5456.7% 6.97 ± 47% perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.11 ± 16% +1515.9% 1.77 ± 2% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.02 ± 3% +10782.8% 1.80 ± 3% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 111.99 ± 28% +414.1% 575.79 ± 2% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 3.63 ± 5% +190.7% 10.54 ± 2% perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 586.45 ± 2% -68.0% 187.86 ± 18% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 387.39 ± 3% +11.5% 431.95 ± 6% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 8.40 ± 2% +22.7% 10.30 ± 3% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.05 +4315.4% 2.25 ± 38% perf-sched.wait_and_delay.avg.ms.signalfd_dequeue.signalfd_read.vfs_read.ksys_read 5.00 ± 40% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 54900 -94.3% 3143 ± 85% perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 47.17 ± 4% -74.2% 12.17 ±100% perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 27665 ± 3% -49.4% 13994 perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 3967007 -99.6% 17684 ± 2% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 461.67 ± 27% -81.7% 84.33 ± 2% perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64 264.33 ± 2% -23.2% 203.00 ± 5% perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 20.50 ± 2% +291.9% 80.33 ± 21% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 4015151 -97.2% 110553 ± 35% perf-sched.wait_and_delay.count.signalfd_dequeue.signalfd_read.vfs_read.ksys_read 5.77 ± 46% +9584.3% 559.22 ± 61% perf-sched.wait_and_delay.max.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 2.47 ± 13% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 6.64 ± 71% +5251.2% 355.39 ± 62% perf-sched.wait_and_delay.max.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 6.49 ± 72% +4657.3% 308.75 ± 59% perf-sched.wait_and_delay.max.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 1001 -96.6% 34.32 ± 10% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 5.51 ± 20% +191.4% 16.06 ± 40% perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 499.96 +11.9% 559.61 ± 6% perf-sched.wait_and_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 1.49 ±125% +21524.1% 322.06 ± 50% perf-sched.wait_time.avg.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 18.49 ± 7% +16.2% 21.48 ± 6% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.17 ±133% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 1.51 ± 29% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 0.27 ± 71% +45862.8% 125.33 ± 40% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.16 ± 60% +1.1e+05% 176.52 ± 57% perf-sched.wait_time.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 0.06 ±101% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 374.35 ± 26% +114.1% 801.65 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.10 ± 16% +4250.9% 4.52 ± 46% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.52 ± 32% +593.2% 3.63 ± 17% perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.10 ± 19% +1653.1% 1.74 ± 2% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.05 ± 46% +4462.5% 2.43 ± 33% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.02 ± 3% +10625.3% 1.77 ± 2% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 111.96 ± 28% +413.6% 574.99 ± 2% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 3.61 ± 5% +105.5% 7.42 perf-sched.wait_time.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 586.40 ± 2% -68.0% 187.59 ± 18% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 387.37 ± 3% +11.1% 430.56 ± 6% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 2.79 ± 10% +70.8% 4.76 ± 8% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.03 +4244.4% 1.29 ± 39% perf-sched.wait_time.avg.ms.signalfd_dequeue.signalfd_read.vfs_read.ksys_read 0.05 ± 77% +948.6% 0.51 ± 20% perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 6.95 ± 11% +7947.3% 559.22 ± 61% perf-sched.wait_time.max.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 2.05 ± 97% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 2.47 ± 13% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 9.86 ± 5% +3506.0% 355.39 ± 62% perf-sched.wait_time.max.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 9.04 ± 18% +3315.1% 308.75 ± 59% perf-sched.wait_time.max.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 0.33 ±129% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 3.45 ± 28% +210.4% 10.71 ± 14% perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 1000 -97.3% 26.58 ± 24% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 5.50 ± 20% +82.9% 10.06 ± 31% perf-sched.wait_time.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 499.95 +11.5% 557.31 ± 7% perf-sched.wait_time.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 8.72 ± 17% +24.3% 10.84 ± 7% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.37 ±137% +1006.1% 4.12 ± 8% perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 5.76 -5.8 0.00 perf-profile.calltrace.cycles-pp.schedule.signalfd_dequeue.signalfd_read.vfs_read.ksys_read 5.63 -5.6 0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.signalfd_dequeue.signalfd_read.vfs_read 5.32 -5.1 0.19 ±142% perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.__send_signal_locked.do_send_sig_info 5.70 -4.8 0.87 ± 6% perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.__send_signal_locked.do_send_sig_info.kill_pid_info 6.16 -4.8 1.36 ± 4% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.__send_signal_locked.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo 10.19 -4.4 5.80 ± 3% perf-profile.calltrace.cycles-pp.signalfd_dequeue.signalfd_read.vfs_read.ksys_read.do_syscall_64 11.77 -4.1 7.66 ± 2% perf-profile.calltrace.cycles-pp.signalfd_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.97 -3.8 6.17 perf-profile.calltrace.cycles-pp.__send_signal_locked.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo 8.10 -3.7 4.44 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigqueue 10.70 -3.6 7.12 perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64 29.25 -3.4 25.83 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigqueue 31.76 -3.4 28.38 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sigqueue 14.16 -3.2 10.95 ± 2% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 14.88 -3.0 11.83 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 19.48 -2.7 16.76 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 39.98 -2.6 37.35 perf-profile.calltrace.cycles-pp.sigqueue 22.09 -2.5 19.62 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 29.54 -1.6 27.98 perf-profile.calltrace.cycles-pp.read 0.86 +0.1 0.96 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_getuid.do_syscall_64.entry_SYSCALL_64_after_hwframe.getuid 0.52 ± 2% +0.1 0.62 perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.kmem_cache_alloc.__sigqueue_alloc.__send_signal_locked.do_send_sig_info 0.73 ± 2% +0.1 0.85 perf-profile.calltrace.cycles-pp.kmem_cache_free.dequeue_signal.signalfd_dequeue.signalfd_read.vfs_read 0.70 +0.1 0.84 perf-profile.calltrace.cycles-pp._copy_from_user.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigqueue 0.55 ± 6% +0.2 0.73 ± 9% perf-profile.calltrace.cycles-pp._copy_to_user.signalfd_copyinfo.signalfd_read.vfs_read.ksys_read 1.36 +0.3 1.62 perf-profile.calltrace.cycles-pp.__entry_text_start.getuid 1.12 ± 3% +0.3 1.40 ± 5% perf-profile.calltrace.cycles-pp.signalfd_copyinfo.signalfd_read.vfs_read.ksys_read.do_syscall_64 0.79 ± 10% +0.3 1.07 ± 9% perf-profile.calltrace.cycles-pp.profile_signal_perm.aa_may_signal.apparmor_task_kill.security_task_kill.kill_pid_info 1.40 +0.3 1.68 perf-profile.calltrace.cycles-pp.__entry_text_start.sigqueue 4.11 +0.3 4.40 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.38 +0.3 1.68 perf-profile.calltrace.cycles-pp.__entry_text_start.__getpid 1.33 +0.3 1.65 perf-profile.calltrace.cycles-pp.kmem_cache_alloc.__sigqueue_alloc.__send_signal_locked.do_send_sig_info.kill_pid_info 0.70 ± 2% +0.3 1.03 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_getpid.do_syscall_64.entry_SYSCALL_64_after_hwframe.__getpid 1.25 +0.4 1.62 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.getuid 1.51 +0.4 1.90 perf-profile.calltrace.cycles-pp.__entry_text_start.read 1.36 +0.4 1.77 perf-profile.calltrace.cycles-pp.inc_rlimit_get_ucounts.__sigqueue_alloc.__send_signal_locked.do_send_sig_info.kill_pid_info 0.58 ± 2% +0.4 1.00 ± 2% perf-profile.calltrace.cycles-pp.__task_pid_nr_ns.__x64_sys_getpid.do_syscall_64.entry_SYSCALL_64_after_hwframe.__getpid 3.64 +0.4 4.07 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__getpid 0.17 ±141% +0.4 0.60 ± 2% perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.21 ± 2% +0.5 1.75 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.read 0.00 +0.6 0.56 ± 2% perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.58 +0.6 4.21 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.getuid 0.97 ± 25% +0.7 1.62 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.__getpid 1.46 +0.7 2.18 perf-profile.calltrace.cycles-pp.do_dec_rlimit_put_ucounts.collect_signal.dequeue_signal.signalfd_dequeue.signalfd_read 2.82 +0.7 3.55 perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal_locked.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo 1.72 +0.8 2.55 perf-profile.calltrace.cycles-pp.collect_signal.dequeue_signal.signalfd_dequeue.signalfd_read.vfs_read 0.00 +0.9 0.86 ± 4% perf-profile.calltrace.cycles-pp.__slab_free.dequeue_signal.signalfd_dequeue.signalfd_read.vfs_read 0.55 ± 2% +0.9 1.42 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.sigqueue 4.89 +0.9 5.81 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.getuid 1.83 +0.9 2.77 ± 2% perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.kill_pid_info.do_rt_sigqueueinfo 2.09 ± 4% +1.0 3.05 ± 3% perf-profile.calltrace.cycles-pp.aa_may_signal.apparmor_task_kill.security_task_kill.kill_pid_info.do_rt_sigqueueinfo 1.72 +1.0 2.69 perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64 1.84 +1.0 2.82 perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.70 +1.0 5.68 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__getpid 4.16 +1.0 5.14 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.getuid 4.20 +1.0 5.19 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.sigqueue 4.13 +1.1 5.19 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read 4.19 +1.1 5.26 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__getpid 7.18 +1.3 8.45 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.getuid 7.03 +1.3 8.36 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__getpid 3.40 +1.5 4.89 perf-profile.calltrace.cycles-pp.dequeue_signal.signalfd_dequeue.signalfd_read.vfs_read.ksys_read 13.72 +2.8 16.48 perf-profile.calltrace.cycles-pp.getuid 13.55 +3.0 16.55 perf-profile.calltrace.cycles-pp.__getpid 7.36 +3.4 10.78 perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo 7.42 +3.4 10.85 perf-profile.calltrace.cycles-pp.security_task_kill.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64 9.68 -9.5 0.18 ± 35% perf-profile.children.cycles-pp.schedule 9.47 -9.3 0.20 ± 42% perf-profile.children.cycles-pp.__schedule 6.14 -5.3 0.85 ± 3% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 5.37 -4.9 0.52 ± 12% perf-profile.children.cycles-pp.try_to_wake_up 5.71 -4.8 0.88 ± 6% perf-profile.children.cycles-pp.__wake_up_common 6.20 -4.8 1.40 ± 4% perf-profile.children.cycles-pp.__wake_up_common_lock 10.23 -4.4 5.81 ± 3% perf-profile.children.cycles-pp.signalfd_dequeue 58.54 -4.3 54.25 perf-profile.children.cycles-pp.do_syscall_64 11.80 -4.1 7.68 ± 2% perf-profile.children.cycles-pp.signalfd_read 10.00 -3.8 6.22 perf-profile.children.cycles-pp.__send_signal_locked 10.72 -3.6 7.16 perf-profile.children.cycles-pp.do_send_sig_info 68.34 -3.2 65.11 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 14.17 -3.2 10.97 ± 2% perf-profile.children.cycles-pp.vfs_read 14.89 -3.0 11.84 perf-profile.children.cycles-pp.ksys_read 3.03 -3.0 0.06 ± 94% perf-profile.children.cycles-pp.ttwu_do_activate 40.22 -2.6 37.61 perf-profile.children.cycles-pp.sigqueue 2.45 -2.4 0.04 ±114% perf-profile.children.cycles-pp.activate_task 2.36 -2.3 0.04 ±114% perf-profile.children.cycles-pp.enqueue_task_fair 2.36 -2.3 0.06 ± 60% perf-profile.children.cycles-pp.dequeue_task_fair 19.51 -2.3 17.22 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 1.75 -1.7 0.06 ± 24% perf-profile.children.cycles-pp.update_load_avg 29.80 -1.5 28.27 perf-profile.children.cycles-pp.read 0.14 ± 6% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.__list_del_entry_valid_or_report 0.13 ± 4% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.__list_add_valid_or_report 0.39 ± 2% -0.0 0.35 ± 5% perf-profile.children.cycles-pp.recalc_sigpending 0.37 -0.0 0.33 ± 3% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.32 ± 2% -0.0 0.29 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.09 ± 10% -0.0 0.06 ± 11% perf-profile.children.cycles-pp.__libc_current_sigrtmin@plt 0.28 ± 2% -0.0 0.26 perf-profile.children.cycles-pp.hrtimer_interrupt 0.29 -0.0 0.26 ± 2% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 0.24 ± 2% -0.0 0.22 ± 3% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.05 ± 8% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.post_copy_siginfo_from_user 0.10 +0.0 0.12 ± 3% perf-profile.children.cycles-pp.complete_signal 0.06 ± 8% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.siginfo_layout 0.05 ± 7% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.fsnotify_perm 0.11 ± 4% +0.0 0.14 ± 8% perf-profile.children.cycles-pp.obj_cgroup_charge 0.32 ± 3% +0.0 0.35 ± 4% perf-profile.children.cycles-pp.__libc_current_sigrtmin_private 0.22 +0.0 0.27 ± 3% perf-profile.children.cycles-pp.mod_objcg_state 0.38 ± 3% +0.0 0.43 ± 2% perf-profile.children.cycles-pp.from_kuid_munged 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.__unfreeze_partials 0.05 ± 8% +0.1 0.11 ± 4% perf-profile.children.cycles-pp.allocate_slab 0.00 +0.1 0.07 ± 5% perf-profile.children.cycles-pp.shuffle_freelist 0.19 ± 3% +0.1 0.26 ± 7% perf-profile.children.cycles-pp.prepare_signal 0.25 +0.1 0.34 ± 2% perf-profile.children.cycles-pp.map_id_up 0.47 ± 3% +0.1 0.56 perf-profile.children.cycles-pp.__fget_light 0.53 ± 2% +0.1 0.63 perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook 0.50 ± 3% +0.1 0.61 ± 2% perf-profile.children.cycles-pp.__fdget_pos 0.10 ± 3% +0.1 0.22 ± 3% perf-profile.children.cycles-pp.___slab_alloc 0.92 ± 2% +0.1 1.04 ± 2% perf-profile.children.cycles-pp.__x64_sys_getuid 0.74 ± 2% +0.1 0.86 perf-profile.children.cycles-pp.kmem_cache_free 0.92 ± 2% +0.1 1.06 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.74 ± 2% +0.2 0.90 perf-profile.children.cycles-pp._copy_from_user 0.30 ± 3% +0.2 0.47 ± 6% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.62 ± 4% +0.2 0.80 perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.61 ± 5% +0.2 0.80 ± 8% perf-profile.children.cycles-pp.rep_movs_alternative 0.64 ± 5% +0.2 0.84 ± 7% perf-profile.children.cycles-pp._copy_to_user 0.32 ± 2% +0.2 0.55 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.00 +0.2 0.24 ± 11% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.20 ± 5% +0.3 0.47 perf-profile.children.cycles-pp.aa_file_perm 0.80 ± 10% +0.3 1.08 ± 9% perf-profile.children.cycles-pp.profile_signal_perm 1.13 ± 3% +0.3 1.42 ± 5% perf-profile.children.cycles-pp.signalfd_copyinfo 1.35 +0.3 1.68 perf-profile.children.cycles-pp.kmem_cache_alloc 1.36 +0.4 1.77 perf-profile.children.cycles-pp.inc_rlimit_get_ucounts 0.68 ± 2% +0.4 1.10 ± 3% perf-profile.children.cycles-pp.__task_pid_nr_ns 0.76 +0.4 1.19 ± 2% perf-profile.children.cycles-pp.__x64_sys_getpid 0.37 ± 3% +0.5 0.87 ± 4% perf-profile.children.cycles-pp.__slab_free 1.46 +0.7 2.18 perf-profile.children.cycles-pp.do_dec_rlimit_put_ucounts 2.82 +0.7 3.56 perf-profile.children.cycles-pp.__sigqueue_alloc 1.72 +0.8 2.56 perf-profile.children.cycles-pp.collect_signal 2.54 +0.9 3.47 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 1.83 +0.9 2.77 ± 2% perf-profile.children.cycles-pp.aa_get_task_label 2.10 ± 4% +1.0 3.07 ± 3% perf-profile.children.cycles-pp.aa_may_signal 1.74 +1.0 2.70 perf-profile.children.cycles-pp.apparmor_file_permission 1.85 +1.0 2.83 perf-profile.children.cycles-pp.security_file_permission 3.42 +1.5 4.90 perf-profile.children.cycles-pp.dequeue_signal 7.69 +2.1 9.83 perf-profile.children.cycles-pp.__entry_text_start 13.99 +2.8 16.81 perf-profile.children.cycles-pp.getuid 13.82 +3.1 16.88 perf-profile.children.cycles-pp.__getpid 7.38 +3.4 10.80 perf-profile.children.cycles-pp.apparmor_task_kill 7.42 +3.4 10.85 perf-profile.children.cycles-pp.security_task_kill 16.80 +4.1 20.92 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.48 ± 3% -0.2 0.28 ± 3% perf-profile.self.cycles-pp.signalfd_dequeue 0.32 ± 5% -0.1 0.19 ± 11% perf-profile.self.cycles-pp.try_to_wake_up 0.43 ± 3% -0.1 0.34 ± 10% perf-profile.self.cycles-pp.vfs_read 0.12 ± 4% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.__list_del_entry_valid_or_report 0.80 ± 2% -0.1 0.74 ± 2% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.12 ± 4% -0.1 0.07 ± 10% perf-profile.self.cycles-pp.__list_add_valid_or_report 0.38 ± 2% -0.0 0.34 ± 5% perf-profile.self.cycles-pp.recalc_sigpending 0.10 +0.0 0.12 ± 4% perf-profile.self.cycles-pp.complete_signal 0.14 ± 4% +0.0 0.16 ± 4% perf-profile.self.cycles-pp.send_signal_locked 0.11 ± 3% +0.0 0.13 ± 3% perf-profile.self.cycles-pp.kill_pid_info 0.04 ± 44% +0.0 0.06 ± 7% perf-profile.self.cycles-pp.siginfo_layout 0.49 +0.0 0.52 perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.09 ± 5% +0.0 0.12 ± 10% perf-profile.self.cycles-pp.obj_cgroup_charge 0.27 ± 4% +0.0 0.30 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.03 ± 70% +0.0 0.07 ± 8% perf-profile.self.cycles-pp.fsnotify_perm 0.22 ± 2% +0.0 0.26 ± 4% perf-profile.self.cycles-pp.mod_objcg_state 0.21 ± 3% +0.0 0.26 ± 5% perf-profile.self.cycles-pp.ksys_read 0.31 ± 4% +0.1 0.36 ± 3% perf-profile.self.cycles-pp.kmem_cache_alloc 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.__fdget_pos 0.26 ± 4% +0.1 0.32 ± 2% perf-profile.self.cycles-pp._copy_to_user 0.78 +0.1 0.84 perf-profile.self.cycles-pp.__x64_sys_rt_sigqueueinfo 0.27 ± 2% +0.1 0.33 ± 2% perf-profile.self.cycles-pp.getuid 0.27 ± 3% +0.1 0.33 ± 3% perf-profile.self.cycles-pp.__getpid 0.48 ± 3% +0.1 0.55 perf-profile.self.cycles-pp.kmem_cache_free 0.24 +0.1 0.30 ± 4% perf-profile.self.cycles-pp.__libc_current_sigrtmin_private 0.18 ± 4% +0.1 0.25 ± 7% perf-profile.self.cycles-pp.prepare_signal 0.45 +0.1 0.53 ± 3% perf-profile.self.cycles-pp.signalfd_copyinfo 0.42 ± 3% +0.1 0.51 perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook 0.46 ± 2% +0.1 0.56 perf-profile.self.cycles-pp.__fget_light 0.57 +0.1 0.67 ± 3% perf-profile.self.cycles-pp.__x64_sys_getuid 0.91 ± 2% +0.1 1.01 perf-profile.self.cycles-pp.sigqueue 0.22 ± 4% +0.1 0.33 ± 4% perf-profile.self.cycles-pp.collect_signal 0.23 ± 2% +0.1 0.33 ± 2% perf-profile.self.cycles-pp.map_id_up 0.36 ± 3% +0.1 0.47 ± 4% perf-profile.self.cycles-pp.read 0.22 ± 5% +0.1 0.34 ± 6% perf-profile.self.cycles-pp.__x64_sys_getpid 0.73 ± 3% +0.1 0.86 ± 2% perf-profile.self.cycles-pp.do_syscall_64 0.55 ± 2% +0.1 0.68 ± 2% perf-profile.self.cycles-pp._copy_from_user 0.80 ± 2% +0.1 0.94 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.62 ± 2% +0.1 0.76 ± 3% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.52 ± 6% +0.2 0.68 ± 9% perf-profile.self.cycles-pp.rep_movs_alternative 0.66 ± 3% +0.2 0.83 ± 3% perf-profile.self.cycles-pp.__send_signal_locked 0.53 ± 5% +0.2 0.71 perf-profile.self.cycles-pp.syscall_enter_from_user_mode 0.30 +0.2 0.51 ± 3% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare 0.00 +0.2 0.24 ± 10% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.20 ± 3% +0.3 0.46 perf-profile.self.cycles-pp.aa_file_perm 0.79 ± 10% +0.3 1.06 ± 9% perf-profile.self.cycles-pp.profile_signal_perm 0.62 ± 3% +0.3 0.94 ± 2% perf-profile.self.cycles-pp.__task_pid_nr_ns 1.36 +0.4 1.76 perf-profile.self.cycles-pp.inc_rlimit_get_ucounts 0.37 ± 3% +0.5 0.87 ± 4% perf-profile.self.cycles-pp.__slab_free 1.30 ± 2% +0.7 1.99 perf-profile.self.cycles-pp.aa_may_signal 1.53 ± 2% +0.7 2.23 perf-profile.self.cycles-pp.apparmor_file_permission 1.46 +0.7 2.18 perf-profile.self.cycles-pp.do_dec_rlimit_put_ucounts 1.82 +0.9 2.75 ± 2% perf-profile.self.cycles-pp.aa_get_task_label 9.98 +1.1 11.08 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 3.43 +1.5 4.95 perf-profile.self.cycles-pp.apparmor_task_kill 6.90 +2.0 8.92 perf-profile.self.cycles-pp.__entry_text_start 13.14 +2.7 15.88 perf-profile.self.cycles-pp.syscall_exit_to_user_mode 16.78 +4.1 20.90 perf-profile.self.cycles-pp.syscall_return_via_sysret *************************************************************************************************** lkp-cfl-d1: 12 threads 1 sockets Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Coffee Lake) with 16G memory ========================================================================================= compiler/cpufreq_governor/kconfig/option_a/option_b/rootfs/tbox_group/test/testcase: gcc-12/performance/x86_64-rhel-8.3/Masskrug/CPU-only/debian-x86_64-phoronix/lkp-cfl-d1/darktable-1.0.5/phoronix-test-suite commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 1213004 -17.6% 998925 turbostat.IRQ 1.63 +1.9% 1.66 turbostat.RAMWatt 7.02 -5.3% 6.64 phoronix-test-suite.darktable.Masskrug.CPU-only.seconds 204076 ± 3% -96.0% 8120 phoronix-test-suite.time.involuntary_context_switches 207.98 -2.7% 202.35 phoronix-test-suite.time.user_time 3569 ± 2% +4.5% 3731 vmstat.io.bi 27380 -29.3% 19366 vmstat.system.cs 24655 -14.9% 20990 vmstat.system.in 21.04 ± 38% -12.1 8.98 ± 75% perf-profile.calltrace.cycles-pp.asm_exc_page_fault 4.75 ±112% -3.4 1.35 ±150% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 4.15 ±105% -1.1 3.04 ±184% perf-profile.calltrace.cycles-pp.wp_page_copy.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 4.75 ±112% -3.4 1.35 ±150% perf-profile.children.cycles-pp.syscall_return_via_sysret 4.15 ±105% -1.1 3.04 ±184% perf-profile.children.cycles-pp.wp_page_copy 4.75 ±112% -3.4 1.35 ±150% perf-profile.self.cycles-pp.syscall_return_via_sysret 91846 ± 40% +155.2% 234436 ± 50% sched_debug.cfs_rq:/.load.avg 318475 ±114% +211.4% 991708 ± 36% sched_debug.cfs_rq:/.load.max 222871 ± 8% +19.9% 267301 ± 8% sched_debug.cpu.avg_idle.stddev 7765 ± 8% +13.6% 8821 ± 10% sched_debug.cpu.nr_switches.max 56.67 ± 36% +90.0% 107.67 ± 17% sched_debug.cpu.nr_uninterruptible.max 32.73 ± 45% +88.4% 61.69 ± 20% sched_debug.cpu.nr_uninterruptible.stddev 27757 -30.1% 19398 perf-stat.i.context-switches 53.71 -12.9% 46.77 ± 2% perf-stat.i.cpu-migrations 0.18 ± 3% -0.0 0.17 ± 3% perf-stat.i.dTLB-load-miss-rate% 0.03 -0.0 0.03 ± 2% perf-stat.i.dTLB-store-miss-rate% 36.96 +3.5 40.45 perf-stat.i.iTLB-load-miss-rate% 195744 -9.7% 176715 perf-stat.i.iTLB-load-misses 497587 ± 2% -9.7% 449446 ± 2% perf-stat.i.iTLB-loads 147371 ± 3% +156.2% 377598 ± 5% perf-stat.i.instructions-per-iTLB-miss 0.68 +3.0% 0.70 perf-stat.i.ipc 4.95 +0.1 5.01 perf-stat.overall.branch-miss-rate% 52.27 +0.8 53.06 perf-stat.overall.cache-miss-rate% 1.29 -3.1% 1.25 perf-stat.overall.cpi 152.38 -3.6% 146.92 perf-stat.overall.cycles-between-cache-misses 0.02 -0.0 0.02 ± 2% perf-stat.overall.dTLB-store-miss-rate% 87114 +14.8% 100039 ± 2% perf-stat.overall.instructions-per-iTLB-miss 0.77 +3.2% 0.80 perf-stat.overall.ipc 1.415e+08 +3.4% 1.463e+08 perf-stat.ps.cache-misses 27167 -30.8% 18787 perf-stat.ps.context-switches 52.57 -12.7% 45.87 ± 2% perf-stat.ps.cpu-migrations 191616 -10.4% 171736 perf-stat.ps.iTLB-load-misses 487106 ± 2% -10.8% 434393 ± 2% perf-stat.ps.iTLB-loads 16888296 +3.7% 17516532 perf-stat.ps.node-loads 13019327 ± 2% +4.1% 13557844 ± 2% perf-stat.ps.node-stores *************************************************************************************************** lkp-ivb-2ep1: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 112G memory ========================================================================================= compiler/cpufreq_governor/kconfig/mode/nr_threads/rootfs/tbox_group/test/test_memory_size/testcase: gcc-12/performance/x86_64-rhel-8.3/development/100%/debian-11.1-x86_64-20220510.cgz/lkp-ivb-2ep1/TCP/50%/lmbench3 commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 6.53 ± 2% -2.0 4.48 mpstat.cpu.all.soft% 0.15 ±141% -82.6% 0.03 ± 76% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 742523 ± 2% -65.0% 260147 vmstat.system.cs 97884842 ± 2% +15.3% 1.129e+08 ± 2% numa-numastat.node0.local_node 97897136 ± 2% +15.3% 1.129e+08 ± 2% numa-numastat.node0.numa_hit 0.10 -15.0% 0.08 ± 5% turbostat.IPC 26.66 +12.1% 29.90 turbostat.RAMWatt 510239 ± 3% +11.7% 569754 meminfo.AnonHugePages 645720 +11.3% 718985 meminfo.AnonPages 650621 +12.0% 728941 meminfo.Inactive 649977 +12.0% 728297 meminfo.Inactive(anon) 231411 ± 13% +28.8% 297976 ± 13% numa-meminfo.node1.AnonHugePages 298950 ± 16% +24.8% 373214 ± 14% numa-meminfo.node1.AnonPages 304547 ± 16% +24.3% 378608 ± 13% numa-meminfo.node1.Inactive 304010 ± 16% +24.4% 378071 ± 13% numa-meminfo.node1.Inactive(anon) 97897001 ± 2% +15.3% 1.129e+08 ± 2% numa-vmstat.node0.numa_hit 97884707 ± 2% +15.3% 1.129e+08 ± 2% numa-vmstat.node0.numa_local 74802 ± 16% +24.8% 93363 ± 14% numa-vmstat.node1.nr_anon_pages 76037 ± 16% +24.4% 94578 ± 13% numa-vmstat.node1.nr_inactive_anon 76037 ± 16% +24.4% 94578 ± 13% numa-vmstat.node1.nr_zone_inactive_anon 161396 +11.5% 179907 proc-vmstat.nr_anon_pages 162463 +12.1% 182130 proc-vmstat.nr_inactive_anon 162463 +12.1% 182130 proc-vmstat.nr_zone_inactive_anon 1.715e+08 ± 2% +9.0% 1.868e+08 ± 2% proc-vmstat.numa_hit 1.715e+08 ± 2% +9.0% 1.868e+08 ± 2% proc-vmstat.numa_local 1.367e+09 ± 2% +9.0% 1.49e+09 ± 2% proc-vmstat.pgalloc_normal 2337627 +3.7% 2423386 proc-vmstat.pgfault 1.367e+09 ± 2% +9.0% 1.49e+09 ± 2% proc-vmstat.pgfree 76598 ± 2% +4.7% 80163 proc-vmstat.pgreuse 25.32 +8.2% 27.39 lmbench3.TCP.localhost.latency 31811 -29.7% 22377 ± 10% lmbench3.TCP.socket.bandwidth.10MB.MB/sec 958.62 +11.5% 1068 lmbench3.TCP.socket.bandwidth.64B.MB/sec 662.76 +5.5% 699.02 lmbench3.time.elapsed_time 662.76 +5.5% 699.02 lmbench3.time.elapsed_time.max 27141817 -97.2% 751186 ± 5% lmbench3.time.involuntary_context_switches 1697 +12.3% 1905 lmbench3.time.percent_of_cpu_this_job_got 8586 ± 2% +22.4% 10512 ± 2% lmbench3.time.system_time 2663 ± 2% +5.6% 2811 ± 3% lmbench3.time.user_time 2.177e+08 ± 2% -59.4% 88495229 lmbench3.time.voluntary_context_switches 66942 ± 26% +533.5% 424063 ± 64% sched_debug.cfs_rq:/.avg_vruntime.stddev 0.22 ±223% +1.6e+07% 36129 ± 63% sched_debug.cfs_rq:/.left_vruntime.avg 10.55 ±223% +1.6e+07% 1641139 ± 65% sched_debug.cfs_rq:/.left_vruntime.max 1.51 ±223% +1.6e+07% 238933 ± 64% sched_debug.cfs_rq:/.left_vruntime.stddev 64627 ± 55% +324.8% 274564 ± 45% sched_debug.cfs_rq:/.load.max 13943 ± 63% +208.5% 43020 ± 42% sched_debug.cfs_rq:/.load.stddev 66942 ± 26% +533.5% 424063 ± 64% sched_debug.cfs_rq:/.min_vruntime.stddev 0.22 ±223% +1.6e+07% 36129 ± 63% sched_debug.cfs_rq:/.right_vruntime.avg 10.55 ±223% +1.6e+07% 1641139 ± 65% sched_debug.cfs_rq:/.right_vruntime.max 1.51 ±223% +1.6e+07% 238933 ± 64% sched_debug.cfs_rq:/.right_vruntime.stddev 5.99 ± 2% +132.9% 13.94 perf-stat.i.MPKI 7.894e+09 -14.1% 6.779e+09 perf-stat.i.branch-instructions 2.29 ± 3% -0.1 2.17 perf-stat.i.branch-miss-rate% 1.519e+08 -16.6% 1.268e+08 perf-stat.i.branch-misses 7.64 ± 3% +3.9 11.50 perf-stat.i.cache-miss-rate% 66250181 ± 2% +45.0% 96066421 perf-stat.i.cache-misses 6.213e+08 -13.5% 5.374e+08 perf-stat.i.cache-references 745606 ± 2% -63.5% 272354 perf-stat.i.context-switches 4.03 +80.0% 7.26 perf-stat.i.cpi 240.83 ± 25% +79.5% 432.40 ± 29% perf-stat.i.cpu-migrations 65304 ± 8% -15.8% 55003 ± 5% perf-stat.i.cycles-between-cache-misses 1.244e+10 -13.2% 1.079e+10 perf-stat.i.dTLB-loads 0.42 -0.0 0.40 perf-stat.i.dTLB-store-miss-rate% 45312253 ± 2% -7.3% 42014497 perf-stat.i.dTLB-store-misses 9.34e+09 -11.3% 8.282e+09 perf-stat.i.dTLB-stores 95.38 -1.4 93.98 perf-stat.i.iTLB-load-miss-rate% 39012477 ± 2% -44.5% 21654013 ± 2% perf-stat.i.iTLB-load-misses 4.083e+10 -14.0% 3.51e+10 perf-stat.i.instructions 2001 ± 3% +76.1% 3524 perf-stat.i.instructions-per-iTLB-miss 0.37 -12.0% 0.33 perf-stat.i.ipc 482.59 +17.9% 568.77 ± 3% perf-stat.i.metric.K/sec 634.08 -12.7% 553.57 perf-stat.i.metric.M/sec 37.86 -4.8 33.02 perf-stat.i.node-load-miss-rate% 8319154 +1.4e+06% 1.199e+11 ±223% perf-stat.i.node-load-misses 76634824 ± 2% +24.2% 95161297 perf-stat.i.node-loads 27.91 ± 4% -6.1 21.81 ± 3% perf-stat.i.node-store-miss-rate% 2370524 ± 3% +79.6% 4256478 ± 11% perf-stat.i.node-store-misses 34212304 ± 3% +65.5% 56621839 perf-stat.i.node-stores 1.63 ± 2% +63.3% 2.66 ± 2% perf-stat.overall.MPKI 1.92 -0.1 1.86 perf-stat.overall.branch-miss-rate% 10.69 +7.2 17.87 perf-stat.overall.cache-miss-rate% 2.85 +16.5% 3.33 perf-stat.overall.cpi 1754 -28.6% 1251 perf-stat.overall.cycles-between-cache-misses 0.48 +0.0 0.51 perf-stat.overall.dTLB-store-miss-rate% 1046 +56.1% 1632 perf-stat.overall.instructions-per-iTLB-miss 0.35 -14.2% 0.30 perf-stat.overall.ipc 9.79 +15.8 25.61 ±129% perf-stat.overall.node-load-miss-rate% 7.894e+09 -12.7% 6.889e+09 perf-stat.ps.branch-instructions 1.518e+08 -15.5% 1.283e+08 perf-stat.ps.branch-misses 66453715 ± 2% +42.6% 94782071 perf-stat.ps.cache-misses 6.216e+08 -14.7% 5.302e+08 perf-stat.ps.cache-references 743709 ± 2% -65.0% 260297 perf-stat.ps.context-switches 239.31 ± 25% +80.4% 431.62 ± 27% perf-stat.ps.cpu-migrations 1.244e+10 -12.0% 1.094e+10 perf-stat.ps.dTLB-loads 45329501 ± 2% -5.2% 42978604 perf-stat.ps.dTLB-store-misses 9.341e+09 -10.1% 8.397e+09 perf-stat.ps.dTLB-stores 39030275 -44.0% 21854369 ± 2% perf-stat.ps.iTLB-load-misses 4.083e+10 -12.6% 3.567e+10 perf-stat.ps.instructions 8343744 +1.5e+06% 1.253e+11 ±223% perf-stat.ps.node-load-misses 76921361 ± 2% +22.1% 93936207 perf-stat.ps.node-loads 2373023 ± 3% +77.1% 4203412 ± 11% perf-stat.ps.node-store-misses 34320713 ± 3% +62.8% 55873239 perf-stat.ps.node-stores 2.712e+13 -7.9% 2.498e+13 perf-stat.total.instructions *************************************************************************************************** lkp-cfl-d1: 12 threads 1 sockets Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz (Coffee Lake) with 16G memory ========================================================================================= compiler/cpufreq_governor/kconfig/option_a/option_b/rootfs/tbox_group/test/testcase: gcc-12/performance/x86_64-rhel-8.3/Boat/CPU-only/debian-x86_64-phoronix/lkp-cfl-d1/darktable-1.0.5/phoronix-test-suite commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 0.03 ± 4% +0.0 0.03 ± 2% mpstat.cpu.all.soft% 1813331 -22.3% 1409832 turbostat.IRQ 8137165 +2.3% 8323779 proc-vmstat.pgalloc_normal 8072998 +2.3% 8260434 proc-vmstat.pgfree 20617 -50.7% 10174 vmstat.system.cs 23085 -20.4% 18383 vmstat.system.in 16.65 -3.5% 16.07 phoronix-test-suite.darktable.Boat.CPU-only.seconds 414303 ± 2% -95.0% 20719 phoronix-test-suite.time.involuntary_context_switches 563.17 -1.8% 553.12 phoronix-test-suite.time.user_time 1.08 ± 22% +48.4% 1.60 ± 11% sched_debug.cfs_rq:/.h_nr_running.avg 0.54 ± 27% +64.0% 0.88 ± 28% sched_debug.cfs_rq:/.h_nr_running.stddev 87.44 ±148% +17938.3% 15772 ± 82% sched_debug.cfs_rq:/.left_vruntime.avg 858.21 ±170% +17391.2% 150111 ± 69% sched_debug.cfs_rq:/.left_vruntime.max 255.23 ±160% +17609.7% 45201 ± 72% sched_debug.cfs_rq:/.left_vruntime.stddev 111255 ± 45% +96.3% 218448 ± 21% sched_debug.cfs_rq:/.load.avg 441826 ± 80% +155.0% 1126499 ± 36% sched_debug.cfs_rq:/.load.max 117920 ± 84% +171.2% 319813 ± 34% sched_debug.cfs_rq:/.load.stddev 184.32 ± 14% +44.3% 266.03 ± 15% sched_debug.cfs_rq:/.load_avg.avg 0.89 ± 17% +25.0% 1.11 ± 3% sched_debug.cfs_rq:/.nr_running.avg 1.17 ± 20% +71.4% 2.00 ± 20% sched_debug.cfs_rq:/.nr_running.max 87.44 ±148% +17938.4% 15772 ± 82% sched_debug.cfs_rq:/.right_vruntime.avg 858.21 ±170% +17391.3% 150112 ± 69% sched_debug.cfs_rq:/.right_vruntime.max 255.23 ±160% +17609.8% 45201 ± 72% sched_debug.cfs_rq:/.right_vruntime.stddev 1070 ± 16% +20.9% 1295 sched_debug.cfs_rq:/.runnable_avg.avg 1660 ± 15% +25.7% 2087 ± 4% sched_debug.cfs_rq:/.runnable_avg.max 1.04 ± 22% +52.0% 1.58 ± 12% sched_debug.cpu.nr_running.avg 0.52 ± 32% +69.6% 0.88 ± 27% sched_debug.cpu.nr_running.stddev 71162 -48.2% 36847 sched_debug.cpu.nr_switches.avg 26317 ± 17% -69.5% 8014 ± 22% sched_debug.cpu.nr_switches.min 21322 -50.1% 10644 perf-stat.i.context-switches 52.41 -19.6% 42.12 perf-stat.i.cpu-migrations 47285 ± 2% -8.0% 43518 ± 2% perf-stat.i.dTLB-store-misses 36.71 ± 3% +5.1 41.78 ± 4% perf-stat.i.iTLB-load-miss-rate% 130170 ± 3% -16.5% 108627 ± 2% perf-stat.i.iTLB-load-misses 338792 -17.3% 280337 ± 2% perf-stat.i.iTLB-loads 337767 ± 6% +75.2% 591790 ± 3% perf-stat.i.instructions-per-iTLB-miss 0.64 +2.2% 0.65 perf-stat.i.ipc 0.03 ± 29% +65.5% 0.05 ± 6% perf-stat.i.metric.G/sec 5644 ± 2% +3.2% 5823 perf-stat.i.minor-faults 5649 ± 2% +3.2% 5829 perf-stat.i.page-faults 12.22 +1.4% 12.39 perf-stat.overall.MPKI 44.88 +1.0 45.91 perf-stat.overall.cache-miss-rate% 1.55 -2.2% 1.52 perf-stat.overall.cpi 127.03 -3.6% 122.51 perf-stat.overall.cycles-between-cache-misses 0.01 ± 2% -0.0 0.01 ± 2% perf-stat.overall.dTLB-store-miss-rate% 169606 ± 3% +23.8% 209958 ± 3% perf-stat.overall.instructions-per-iTLB-miss 0.64 +2.2% 0.66 perf-stat.overall.ipc 2.66e+08 +3.3% 2.747e+08 perf-stat.ps.cache-misses 21044 -51.1% 10292 perf-stat.ps.context-switches 51.73 -19.6% 41.58 perf-stat.ps.cpu-migrations 263944 ± 2% -3.8% 253811 perf-stat.ps.dTLB-load-misses 46679 ± 2% -8.7% 42611 ± 2% perf-stat.ps.dTLB-store-misses 128484 ± 3% -17.7% 105690 ± 2% perf-stat.ps.iTLB-load-misses 334309 -18.9% 271276 ± 2% perf-stat.ps.iTLB-loads 5363609 ± 2% +4.2% 5588414 perf-stat.ps.node-loads *************************************************************************************************** lkp-skl-d08: 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory ========================================================================================= class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: interrupt/gcc-12/performance/1SSD/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-skl-d08/sigrt/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 1.593e+08 ± 4% -9.9% 1.436e+08 ± 5% cpuidle..time 60.99 -6.3% 57.13 iostat.cpu.system 30.16 +15.2% 34.74 iostat.cpu.user 1.01 -0.2 0.81 ± 2% mpstat.cpu.all.irq% 31.02 +4.8 35.79 mpstat.cpu.all.usr% 1033 ± 27% -66.5% 346.33 ± 11% perf-c2c.DRAM.local 1475 ± 12% +301.4% 5921 ± 7% perf-c2c.HITM.local 1475 ± 12% +301.4% 5921 ± 7% perf-c2c.HITM.total 6685023 ± 5% -35.3% 4324007 vmstat.memory.cache 180.50 ± 3% +318.8% 756.00 vmstat.procs.r 4928645 -89.5% 519621 vmstat.system.cs 78059 ± 4% -18.3% 63798 vmstat.system.in 1.645e+08 +100.2% 3.294e+08 stress-ng.sigrt.ops 2741487 +100.2% 5488391 stress-ng.sigrt.ops_per_sec 1.04e+08 -99.7% 340761 stress-ng.time.involuntary_context_switches 1424 -5.4% 1347 stress-ng.time.system_time 672.86 +13.8% 765.82 stress-ng.time.user_time 2.21e+08 -85.0% 33196290 stress-ng.time.voluntary_context_switches 211047 ± 74% +387.2% 1028244 ± 14% meminfo.Active 200486 ± 77% +388.0% 978459 ± 10% meminfo.Active(anon) 410105 -12.8% 357737 meminfo.AnonPages 6573265 ± 5% -35.6% 4231363 meminfo.Cached 15173178 ± 2% -14.7% 12939277 meminfo.Committed_AS 4071852 ± 7% -77.9% 899527 ± 12% meminfo.Inactive 4067728 ± 7% -78.1% 891282 ± 12% meminfo.Inactive(anon) 983422 ± 10% -53.9% 453243 ± 10% meminfo.Mapped 7674294 ± 4% -31.5% 5255062 ± 2% meminfo.Memused 3858501 ± 8% -60.8% 1512401 meminfo.Shmem 7714943 ± 4% -30.4% 5370445 ± 2% meminfo.max_used_kB 30811 ± 32% +104.9% 63143 ± 23% turbostat.C1 0.03 ± 24% +0.0 0.06 ± 17% turbostat.C1% 53638 ± 6% +27.1% 68152 ± 9% turbostat.C1E 0.18 ± 8% +0.0 0.21 ± 6% turbostat.C1E% 149004 ± 5% -10.8% 132981 ± 6% turbostat.C6 6.57 ± 4% -0.7 5.86 ± 5% turbostat.C6% 6.58 ± 4% -19.7% 5.28 ± 5% turbostat.CPU%c1 0.18 ± 4% +346.8% 0.81 ± 11% turbostat.CPU%c6 0.10 ± 5% -26.3% 0.07 turbostat.IPC 5152367 ± 4% -19.1% 4167291 turbostat.IRQ 6243 ± 54% +265.3% 22809 ± 48% turbostat.POLL 47.00 ± 2% -9.2% 42.67 turbostat.PkgTmp 187.20 -5.9% 176.24 turbostat.PkgWatt 1.44 ± 6% -25.6% 1.08 turbostat.RAMWatt 50124 ± 77% +389.5% 245381 ± 10% proc-vmstat.nr_active_anon 102535 -12.8% 89425 proc-vmstat.nr_anon_pages 615018 +10.0% 676477 proc-vmstat.nr_dirty_background_threshold 1231542 +10.0% 1354609 proc-vmstat.nr_dirty_threshold 1646034 ± 5% -35.0% 1070730 ± 2% proc-vmstat.nr_file_pages 6221811 +9.7% 6826456 proc-vmstat.nr_free_pages 1017053 ± 8% -78.1% 222524 ± 12% proc-vmstat.nr_inactive_anon 246027 ± 10% -54.1% 113019 ± 10% proc-vmstat.nr_mapped 964741 ± 8% -60.8% 378578 proc-vmstat.nr_shmem 50124 ± 77% +389.5% 245381 ± 10% proc-vmstat.nr_zone_active_anon 1017053 ± 8% -78.1% 222524 ± 12% proc-vmstat.nr_zone_inactive_anon 1860386 ± 6% -50.6% 918734 ± 3% proc-vmstat.numa_hit 1852974 ± 6% -50.5% 917323 ± 2% proc-vmstat.numa_local 21127 ± 18% +1738.1% 388340 ± 6% proc-vmstat.pgactivate 1895717 ± 6% -49.5% 956438 ± 2% proc-vmstat.pgalloc_normal 352900 -4.5% 336903 proc-vmstat.pgfault 434980 -11.9% 383073 ± 2% proc-vmstat.pgfree 960649 +11.4% 1070530 sched_debug.cfs_rq:/.avg_vruntime.avg 1214116 ± 11% +57.6% 1913835 ± 11% sched_debug.cfs_rq:/.avg_vruntime.max 70132 ± 39% +286.2% 270868 ± 18% sched_debug.cfs_rq:/.avg_vruntime.stddev 2.47 ± 20% +396.0% 12.23 ± 9% sched_debug.cfs_rq:/.h_nr_running.avg 16.25 ± 10% +12.8% 18.33 ± 4% sched_debug.cfs_rq:/.h_nr_running.max 3.55 ± 19% +52.5% 5.41 ± 7% sched_debug.cfs_rq:/.h_nr_running.stddev 10.52 ±160% +53297.6% 5616 ±217% sched_debug.cfs_rq:/.left_vruntime.avg 271.36 ±146% +73728.8% 200341 ±219% sched_debug.cfs_rq:/.left_vruntime.max 49.42 ±149% +66598.7% 32963 ±219% sched_debug.cfs_rq:/.left_vruntime.stddev 37985 ± 40% +101.8% 76655 ± 29% sched_debug.cfs_rq:/.load.avg 78644 ± 83% +149.6% 196333 ± 27% sched_debug.cfs_rq:/.load.stddev 960649 +11.4% 1070532 sched_debug.cfs_rq:/.min_vruntime.avg 1214116 ± 11% +57.6% 1913835 ± 11% sched_debug.cfs_rq:/.min_vruntime.max 70132 ± 39% +286.2% 270869 ± 18% sched_debug.cfs_rq:/.min_vruntime.stddev 0.26 ± 13% +31.8% 0.35 ± 15% sched_debug.cfs_rq:/.nr_running.stddev 10.52 ±160% +53293.0% 5616 ±217% sched_debug.cfs_rq:/.right_vruntime.avg 271.39 ±146% +73719.9% 200341 ±219% sched_debug.cfs_rq:/.right_vruntime.max 49.43 ±149% +66592.6% 32963 ±219% sched_debug.cfs_rq:/.right_vruntime.stddev 3170 ± 2% +286.2% 12242 ± 3% sched_debug.cfs_rq:/.runnable_avg.avg 5697 ± 9% +160.0% 14816 ± 5% sched_debug.cfs_rq:/.runnable_avg.max 806.39 ± 11% +218.5% 2568 ± 17% sched_debug.cfs_rq:/.runnable_avg.stddev 211.14 ± 6% +63.8% 345.75 ± 4% sched_debug.cfs_rq:/.util_est_enqueued.avg 332013 ± 9% -17.5% 273981 ± 6% sched_debug.cpu.avg_idle.avg 27647 ± 17% -29.3% 19559 ± 11% sched_debug.cpu.avg_idle.min 211303 ± 10% -11.5% 186985 ± 4% sched_debug.cpu.avg_idle.stddev 2.39 ± 22% +410.0% 12.21 ± 9% sched_debug.cpu.nr_running.avg 3.47 ± 20% +56.1% 5.41 ± 7% sched_debug.cpu.nr_running.stddev 4282626 -89.5% 451524 sched_debug.cpu.nr_switches.avg 4490688 -82.0% 808072 ± 15% sched_debug.cpu.nr_switches.max 3803398 ± 4% -90.9% 345054 ± 3% sched_debug.cpu.nr_switches.min 157828 ± 24% -32.9% 105929 ± 23% sched_debug.cpu.nr_switches.stddev 0.89 ± 8% -60.1% 0.35 ± 3% perf-stat.i.MPKI 8.358e+09 -34.7% 5.458e+09 perf-stat.i.branch-instructions 1.79 -0.1 1.65 perf-stat.i.branch-miss-rate% 1.398e+08 -38.6% 85770592 perf-stat.i.branch-misses 9.88 ± 5% -4.5 5.42 ± 2% perf-stat.i.cache-miss-rate% 38787613 ± 6% -72.6% 10639896 ± 3% perf-stat.i.cache-misses 4.288e+08 ± 5% -46.2% 2.307e+08 perf-stat.i.cache-references 5186746 -89.6% 540527 perf-stat.i.context-switches 3.07 +42.2% 4.36 perf-stat.i.cpi 4563 ± 3% +99.9% 9123 ± 3% perf-stat.i.cpu-migrations 3504 ± 8% +313.6% 14493 ± 4% perf-stat.i.cycles-between-cache-misses 1.61 ± 3% -1.0 0.61 perf-stat.i.dTLB-load-miss-rate% 1.899e+08 ± 2% -73.2% 50962287 perf-stat.i.dTLB-load-misses 1.162e+10 -30.6% 8.059e+09 perf-stat.i.dTLB-loads 0.44 ± 2% -0.4 0.09 perf-stat.i.dTLB-store-miss-rate% 30729438 ± 2% -85.7% 4393879 perf-stat.i.dTLB-store-misses 6.881e+09 -27.9% 4.962e+09 perf-stat.i.dTLB-stores 57.28 +24.1 81.41 perf-stat.i.iTLB-load-miss-rate% 59373888 -45.8% 32152758 ± 3% perf-stat.i.iTLB-load-misses 42335180 -85.1% 6310611 ± 15% perf-stat.i.iTLB-loads 4.308e+10 -31.5% 2.949e+10 perf-stat.i.instructions 0.36 -27.2% 0.26 perf-stat.i.ipc 1429 ± 2% -67.8% 459.72 ± 5% perf-stat.i.metric.K/sec 759.02 -31.6% 519.50 perf-stat.i.metric.M/sec 4130 ± 2% -3.1% 4002 perf-stat.i.minor-faults 9282786 ± 7% -67.7% 2997133 ± 5% perf-stat.i.node-loads 3643109 ± 5% -56.4% 1587011 ± 4% perf-stat.i.node-stores 4130 ± 2% -3.1% 4002 perf-stat.i.page-faults 0.90 ± 7% -60.0% 0.36 ± 3% perf-stat.overall.MPKI 1.67 -0.1 1.57 perf-stat.overall.branch-miss-rate% 9.06 ± 6% -4.4 4.61 ± 3% perf-stat.overall.cache-miss-rate% 3.02 +46.4% 4.42 perf-stat.overall.cpi 3366 ± 7% +264.1% 12258 ± 3% perf-stat.overall.cycles-between-cache-misses 1.61 ± 3% -1.0 0.63 perf-stat.overall.dTLB-load-miss-rate% 0.44 ± 2% -0.4 0.09 perf-stat.overall.dTLB-store-miss-rate% 58.38 +25.3 83.66 ± 2% perf-stat.overall.iTLB-load-miss-rate% 725.64 +26.4% 917.19 ± 3% perf-stat.overall.instructions-per-iTLB-miss 0.33 -31.7% 0.23 perf-stat.overall.ipc 8.228e+09 -34.7% 5.374e+09 perf-stat.ps.branch-instructions 1.376e+08 -38.6% 84440936 perf-stat.ps.branch-misses 38188699 ± 6% -72.6% 10471539 ± 3% perf-stat.ps.cache-misses 4.222e+08 ± 5% -46.2% 2.273e+08 perf-stat.ps.cache-references 5105776 -89.6% 533193 perf-stat.ps.context-switches 4492 ± 3% +100.6% 9010 ± 3% perf-stat.ps.cpu-migrations 1.869e+08 ± 2% -73.1% 50231984 perf-stat.ps.dTLB-load-misses 1.144e+10 -30.6% 7.936e+09 perf-stat.ps.dTLB-loads 30250301 ± 2% -85.7% 4332582 perf-stat.ps.dTLB-store-misses 6.775e+09 -27.9% 4.887e+09 perf-stat.ps.dTLB-stores 58447403 -45.8% 31692746 ± 3% perf-stat.ps.iTLB-load-misses 41674419 -85.1% 6222297 ± 15% perf-stat.ps.iTLB-loads 4.241e+10 -31.5% 2.903e+10 perf-stat.ps.instructions 4064 -4.3% 3887 perf-stat.ps.minor-faults 9139873 ± 7% -67.7% 2950054 ± 6% perf-stat.ps.node-loads 3587370 ± 5% -56.6% 1558351 ± 4% perf-stat.ps.node-stores 4064 -4.3% 3887 perf-stat.ps.page-faults 2.7e+12 -32.2% 1.832e+12 perf-stat.total.instructions 0.01 ± 99% +333.3% 0.03 ± 41% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.05 ± 70% +3735.4% 1.84 ± 24% perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.22 ±115% +650.0% 1.67 ± 59% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.04 ± 49% +400.0% 0.18 ± 75% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.13 ±134% +245.4% 0.46 ± 40% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 1.64 ±128% +307.5% 6.68 ± 40% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select 0.11 ± 45% +1580.9% 1.77 ± 2% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64 0.04 ±146% +1699.2% 0.72 ± 36% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ± 45% +6406.5% 0.50 ± 43% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.01 ± 44% +882.5% 0.07 ± 19% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.03 ± 72% +2646.4% 0.95 ± 44% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.74 ± 96% +307.8% 3.02 ± 32% perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.46 ± 91% +1392.9% 6.84 ± 36% perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 2.94 ±104% +175.9% 8.10 ± 30% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select 0.22 ±183% +1048.2% 2.57 ± 26% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ± 49% +20581.0% 2.00 ± 40% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.05 ± 62% +6009.1% 2.81 ± 13% perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.16 ± 69% +4356.0% 7.20 ± 49% perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.07 ± 45% +2214.0% 1.73 ± 2% perf-sched.total_sch_delay.average.ms 0.76 ± 45% +486.9% 4.44 ± 2% perf-sched.total_wait_and_delay.average.ms 0.68 ± 45% +297.3% 2.71 ± 2% perf-sched.total_wait_time.average.ms 0.33 ±223% +1914.3% 6.71 ± 14% perf-sched.wait_and_delay.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 0.34 ±223% +1970.3% 6.96 ± 23% perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.23 ±223% +3657.4% 8.68 ± 27% perf-sched.wait_and_delay.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 0.16 ±223% +1722.4% 2.98 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.08 ± 45% +3795.1% 2.94 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 263.78 ± 47% +108.5% 550.09 ± 6% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 1.00 ± 45% +329.1% 4.31 ± 2% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64 123.67 ± 47% +74.8% 216.17 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 5.00 ± 74% +100.0% 10.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 31.00 ± 45% +106.5% 64.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 910.33 ±223% +1198.2% 11818 perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 10.00 ± 49% +110.0% 21.00 ± 19% perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 11.17 ± 47% +164.2% 29.50 ± 23% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 6.83 ± 46% +78.0% 12.17 ± 5% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 4.50 ± 52% +118.5% 9.83 ± 3% perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork 39.17 ± 56% +160.9% 102.17 ± 10% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 209.50 ± 46% +53.5% 321.50 ± 5% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 121.67 ± 49% +77.9% 216.50 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 13.51 ±223% +406.9% 68.50 ± 58% perf-sched.wait_and_delay.max.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.67 ± 98% +908.4% 6.71 ± 14% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 0.03 ± 89% +17207.2% 5.97 ± 39% perf-sched.wait_time.avg.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 0.64 ±101% +991.7% 6.96 ± 23% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.58 ±215% +686.2% 4.56 ± 28% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_write.do_syscall_64 0.03 ± 83% +14149.8% 4.87 ± 29% perf-sched.wait_time.avg.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 0.48 ± 90% +1710.8% 8.68 ± 27% perf-sched.wait_time.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 0.08 ±157% +6874.1% 5.35 ± 24% perf-sched.wait_time.avg.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.12 ± 50% +263.8% 0.44 ± 56% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 137.90 ± 47% +81.8% 250.73 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.69 ± 50% +212.7% 2.16 ± 12% perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.58 ± 52% +414.5% 2.98 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 1.76 ± 55% +198.4% 5.27 ± 40% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.07 ± 45% +3925.6% 2.93 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 237.09 ± 47% +131.8% 549.68 ± 6% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.90 ± 45% +182.3% 2.54 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64 0.09 ± 56% +13828.3% 13.21 ± 40% perf-sched.wait_time.max.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 1.56 ±170% +658.1% 11.85 ± 20% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_write.do_syscall_64 1.82 ±202% +661.3% 13.89 ± 29% perf-sched.wait_time.max.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 1.74 ±208% +584.0% 11.88 ± 14% perf-sched.wait_time.max.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 285.21 ±115% +251.8% 1003 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 17.87 -14.5 3.35 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigtimedwait 33.49 -13.3 20.16 perf-profile.calltrace.cycles-pp.__sigtimedwait 15.99 -13.0 3.04 ± 3% perf-profile.calltrace.cycles-pp.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigtimedwait 12.85 -10.6 2.26 ± 3% perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.48 -10.3 2.20 ± 3% perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64 12.13 -10.0 2.14 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait.__x64_sys_rt_sigtimedwait 9.86 -7.9 1.92 ± 4% perf-profile.calltrace.cycles-pp.complete_signal.__send_signal_locked.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo 9.40 -7.9 1.55 ± 5% perf-profile.calltrace.cycles-pp.try_to_wake_up.complete_signal.__send_signal_locked.do_send_sig_info.kill_pid_info 26.46 -7.4 19.10 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__sigtimedwait 13.01 -6.3 6.68 perf-profile.calltrace.cycles-pp.__send_signal_locked.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo 13.84 -6.2 7.59 perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64 28.54 -6.0 22.52 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigqueue 24.67 -5.8 18.86 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigtimedwait 30.44 -5.6 24.80 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sigqueue 37.34 -5.3 32.06 perf-profile.calltrace.cycles-pp.sigqueue 5.89 -5.0 0.88 ± 4% perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.complete_signal.__send_signal_locked.do_send_sig_info 4.89 -4.1 0.81 ± 4% perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.try_to_wake_up.complete_signal.__send_signal_locked 4.76 -4.0 0.78 ± 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.try_to_wake_up.complete_signal 4.25 -3.6 0.64 ± 4% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.schedule_hrtimeout_range_clock.do_sigtimedwait 7.00 -3.5 3.54 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigqueue 19.94 -2.8 17.18 perf-profile.calltrace.cycles-pp.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigqueue 19.02 -2.6 16.39 perf-profile.calltrace.cycles-pp.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe 21.15 -2.6 18.54 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigqueue 2.19 -1.6 0.60 ± 2% perf-profile.calltrace.cycles-pp.dequeue_signal.do_sigtimedwait.__x64_sys_rt_sigtimedwait.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.14 ± 30% -0.8 0.37 ±100% perf-profile.calltrace.cycles-pp.process_simple.reader__read_event.perf_session__process_events.record__finish_output.__cmd_record 1.10 ± 29% -0.8 0.34 ±100% perf-profile.calltrace.cycles-pp.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events.record__finish_output 1.09 ± 29% -0.8 0.34 ±100% perf-profile.calltrace.cycles-pp.queue_event.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events 1.94 ± 5% -0.4 1.52 ± 2% perf-profile.calltrace.cycles-pp.stress_sigrt 0.66 -0.3 0.37 ± 71% perf-profile.calltrace.cycles-pp.__radix_tree_lookup.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.82 ± 2% -0.2 0.64 perf-profile.calltrace.cycles-pp.__x64_sys_getuid.do_syscall_64.entry_SYSCALL_64_after_hwframe.getuid 0.75 ± 3% -0.1 0.66 ± 3% perf-profile.calltrace.cycles-pp.__x64_sys_getpid.do_syscall_64.entry_SYSCALL_64_after_hwframe.__getpid 0.64 ± 5% -0.1 0.57 ± 4% perf-profile.calltrace.cycles-pp.__task_pid_nr_ns.__x64_sys_getpid.do_syscall_64.entry_SYSCALL_64_after_hwframe.__getpid 1.25 ± 2% +0.2 1.41 perf-profile.calltrace.cycles-pp.__entry_text_start.sigqueue 0.44 ± 44% +0.2 0.64 ± 2% perf-profile.calltrace.cycles-pp._copy_from_user.__x64_sys_rt_sigqueueinfo.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigqueue 1.14 ± 2% +0.3 1.46 perf-profile.calltrace.cycles-pp.__entry_text_start.getuid 1.13 +0.3 1.48 perf-profile.calltrace.cycles-pp.__entry_text_start.__getpid 0.66 ± 2% +0.4 1.08 perf-profile.calltrace.cycles-pp.profile_signal_perm.aa_may_signal.apparmor_task_kill.security_task_kill.kill_pid_info 1.02 +0.4 1.45 perf-profile.calltrace.cycles-pp.inc_rlimit_get_ucounts.__sigqueue_alloc.__send_signal_locked.do_send_sig_info.kill_pid_info 0.00 +0.5 0.54 perf-profile.calltrace.cycles-pp._copy_from_user.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.5 0.54 perf-profile.calltrace.cycles-pp.signal_setup_done.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.__get_user_nocheck_8.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.58 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.sigqueue 0.00 +0.6 0.59 ± 2% perf-profile.calltrace.cycles-pp.check_xstate_in_sigframe.__fpu_restore_sig.fpu__restore_sig.restore_sigcontext.__x64_sys_rt_sigreturn 0.00 +0.6 0.62 ± 3% perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.kmem_cache_alloc.__sigqueue_alloc.__send_signal_locked.do_send_sig_info 3.76 +0.6 4.40 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.getuid 0.00 +0.7 0.70 ± 6% perf-profile.calltrace.cycles-pp.___slab_alloc.kmem_cache_alloc.__sigqueue_alloc.__send_signal_locked.do_send_sig_info 0.00 +0.7 0.72 perf-profile.calltrace.cycles-pp.restore_altstack.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigtimedwait 2.67 +0.7 3.40 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.getuid 3.63 +0.8 4.38 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__getpid 2.62 +0.8 3.38 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__getpid 3.23 +0.8 4.03 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.getuid 1.26 ± 5% +0.9 2.12 perf-profile.calltrace.cycles-pp.kmem_cache_alloc.__sigqueue_alloc.__send_signal_locked.do_send_sig_info.kill_pid_info 0.00 +0.9 0.87 perf-profile.calltrace.cycles-pp.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.26 +0.9 4.18 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.sigqueue 3.27 +0.9 4.21 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__getpid 1.18 ± 5% +1.0 2.18 perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.kill_pid_info.do_rt_sigqueueinfo 0.00 +1.0 1.04 perf-profile.calltrace.cycles-pp.__entry_text_start.stress_sighandler_nop 0.17 ±141% +1.1 1.23 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.__getpid 0.00 +1.1 1.06 perf-profile.calltrace.cycles-pp.save_xstate_epilog.get_sigframe.x64_setup_rt_frame.handle_signal.arch_do_signal_or_restart 0.00 +1.1 1.11 perf-profile.calltrace.cycles-pp.fpu__clear_user_states.handle_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare 5.56 +1.2 6.72 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.getuid 0.00 +1.2 1.17 ± 2% perf-profile.calltrace.cycles-pp.kmem_cache_free.dequeue_signal.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop 0.00 +1.2 1.18 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.getuid 0.00 +1.2 1.23 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.stress_sighandler_nop 5.44 +1.3 6.72 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__getpid 1.39 ± 4% +1.3 2.68 perf-profile.calltrace.cycles-pp.aa_may_signal.apparmor_task_kill.security_task_kill.kill_pid_info.do_rt_sigqueueinfo 2.37 ± 2% +1.3 3.69 perf-profile.calltrace.cycles-pp.__sigqueue_alloc.__send_signal_locked.do_send_sig_info.kill_pid_info.do_rt_sigqueueinfo 0.00 +1.5 1.47 perf-profile.calltrace.cycles-pp.__slab_free.dequeue_signal.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop 0.00 +1.5 1.53 perf-profile.calltrace.cycles-pp.do_dec_rlimit_put_ucounts.collect_signal.dequeue_signal.get_signal.arch_do_signal_or_restart 0.00 +1.5 1.55 ± 2% perf-profile.calltrace.cycles-pp.restore_fpregs_from_user.__fpu_restore_sig.fpu__restore_sig.restore_sigcontext.__x64_sys_rt_sigreturn 0.18 ±141% +1.7 1.89 perf-profile.calltrace.cycles-pp.copy_fpstate_to_sigframe.get_sigframe.x64_setup_rt_frame.handle_signal.arch_do_signal_or_restart 0.00 +2.2 2.23 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +2.3 2.35 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_sighandler_nop 0.00 +2.5 2.48 perf-profile.calltrace.cycles-pp.collect_signal.dequeue_signal.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop 0.00 +2.5 2.49 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.67 +2.5 13.20 perf-profile.calltrace.cycles-pp.getuid 0.79 ± 5% +2.6 3.36 perf-profile.calltrace.cycles-pp.get_sigframe.x64_setup_rt_frame.handle_signal.arch_do_signal_or_restart.exit_to_user_mode_loop 0.00 +2.7 2.66 perf-profile.calltrace.cycles-pp.__fpu_restore_sig.fpu__restore_sig.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64 0.17 ±141% +2.7 2.89 perf-profile.calltrace.cycles-pp.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigtimedwait 10.69 +2.7 13.42 perf-profile.calltrace.cycles-pp.__getpid 0.00 +2.8 2.75 perf-profile.calltrace.cycles-pp.fpu__restore_sig.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.63 ± 2% +3.1 3.74 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigtimedwait 0.91 ± 4% +3.1 4.05 perf-profile.calltrace.cycles-pp.x64_setup_rt_frame.handle_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare 0.64 ± 3% +3.1 3.78 perf-profile.calltrace.cycles-pp.__irqentry_text_end.stress_sighandler_nop 4.46 ± 4% +3.7 8.16 perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo 4.51 ± 4% +3.7 8.23 perf-profile.calltrace.cycles-pp.security_task_kill.kill_pid_info.do_rt_sigqueueinfo.__x64_sys_rt_sigqueueinfo.do_syscall_64 0.78 ± 3% +3.9 4.64 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 0.86 ± 3% +4.5 5.33 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_sighandler_nop 0.93 ± 5% +4.6 5.49 perf-profile.calltrace.cycles-pp.handle_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.87 ± 3% +4.6 5.46 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_sighandler_nop 0.89 ± 3% +4.6 5.51 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.stress_sighandler_nop 0.67 ± 3% +5.1 5.76 perf-profile.calltrace.cycles-pp.dequeue_signal.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare 0.82 ± 3% +5.7 6.56 perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 5.83 +5.8 11.68 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigtimedwait 3.63 +7.7 11.30 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigtimedwait 5.52 +7.8 13.36 perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.80 ± 3% +11.4 13.21 perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 2.10 ± 3% +11.9 13.96 perf-profile.calltrace.cycles-pp.stress_sighandler_nop 17.91 -14.6 3.35 ± 3% perf-profile.children.cycles-pp.__x64_sys_rt_sigtimedwait 16.11 -13.9 2.24 ± 3% perf-profile.children.cycles-pp.schedule 15.72 -13.5 2.20 ± 4% perf-profile.children.cycles-pp.__schedule 33.63 -13.4 20.18 perf-profile.children.cycles-pp.__sigtimedwait 16.02 -13.0 3.04 ± 3% perf-profile.children.cycles-pp.do_sigtimedwait 12.86 -10.6 2.26 ± 3% perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock 9.88 -7.9 1.93 ± 4% perf-profile.children.cycles-pp.complete_signal 9.46 -7.9 1.56 ± 5% perf-profile.children.cycles-pp.try_to_wake_up 13.03 -6.3 6.71 perf-profile.children.cycles-pp.__send_signal_locked 13.87 -6.3 7.61 perf-profile.children.cycles-pp.do_send_sig_info 37.52 -5.2 32.28 perf-profile.children.cycles-pp.sigqueue 5.92 -5.0 0.89 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate 4.91 -4.1 0.82 ± 4% perf-profile.children.cycles-pp.activate_task 4.79 -4.0 0.80 ± 4% perf-profile.children.cycles-pp.enqueue_task_fair 62.14 -3.8 58.30 perf-profile.children.cycles-pp.do_syscall_64 4.28 -3.6 0.66 ± 4% perf-profile.children.cycles-pp.dequeue_task_fair 3.32 -2.9 0.42 ± 4% perf-profile.children.cycles-pp.switch_mm_irqs_off 3.36 -2.9 0.48 ± 5% perf-profile.children.cycles-pp.pick_next_task_fair 19.95 -2.8 17.20 perf-profile.children.cycles-pp.do_rt_sigqueueinfo 19.04 -2.6 16.41 perf-profile.children.cycles-pp.kill_pid_info 21.22 -2.6 18.61 perf-profile.children.cycles-pp.__x64_sys_rt_sigqueueinfo 2.64 -2.3 0.36 ± 5% perf-profile.children.cycles-pp.update_curr 2.56 ± 2% -2.2 0.40 ± 4% perf-profile.children.cycles-pp.update_load_avg 2.22 ± 3% -2.1 0.10 ± 5% perf-profile.children.cycles-pp.switch_fpu_return 69.84 -2.1 67.75 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 2.51 -2.0 0.49 ± 6% perf-profile.children.cycles-pp.enqueue_entity 2.08 -1.8 0.31 ± 5% perf-profile.children.cycles-pp.dequeue_entity 1.90 ± 2% -1.7 0.22 ± 5% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate 1.64 ± 3% -1.4 0.24 ± 4% perf-profile.children.cycles-pp.reweight_entity 1.20 ± 2% -1.1 0.14 ± 7% perf-profile.children.cycles-pp.prepare_task_switch 1.20 ± 3% -1.0 0.15 ± 8% perf-profile.children.cycles-pp.__switch_to_asm 1.24 ± 2% -1.0 0.23 ± 5% perf-profile.children.cycles-pp.set_next_entity 1.18 ± 2% -0.9 0.24 ± 18% perf-profile.children.cycles-pp.select_task_rq 1.04 ± 7% -0.9 0.18 ± 4% perf-profile.children.cycles-pp.copy_siginfo_to_user 0.98 ± 3% -0.8 0.14 ± 19% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template 0.93 ± 2% -0.8 0.12 ± 7% perf-profile.children.cycles-pp.pick_eevdf 0.86 ± 2% -0.8 0.05 ± 45% perf-profile.children.cycles-pp.check_preempt_curr 0.89 ± 3% -0.8 0.10 ± 10% perf-profile.children.cycles-pp.___perf_sw_event 0.89 ± 3% -0.7 0.20 ± 20% perf-profile.children.cycles-pp.select_task_rq_fair 0.81 ± 2% -0.7 0.15 ± 5% perf-profile.children.cycles-pp.__update_load_avg_se 0.79 ± 10% -0.6 0.14 ± 5% perf-profile.children.cycles-pp._copy_to_user 0.75 ± 4% -0.6 0.12 ± 7% perf-profile.children.cycles-pp.update_cfs_group 0.74 ± 4% -0.6 0.12 ± 6% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 1.10 ± 29% -0.6 0.50 ± 35% perf-profile.children.cycles-pp.queue_event 1.10 ± 29% -0.6 0.52 ± 35% perf-profile.children.cycles-pp.ordered_events__queue 1.14 ± 30% -0.6 0.56 ± 33% perf-profile.children.cycles-pp.process_simple 0.62 ± 3% -0.6 0.07 ± 5% perf-profile.children.cycles-pp.__switch_to 0.62 ± 3% -0.5 0.08 perf-profile.children.cycles-pp.os_xsave 0.61 ± 4% -0.5 0.09 ± 18% perf-profile.children.cycles-pp.perf_tp_event 0.52 ± 2% -0.4 0.09 ± 8% perf-profile.children.cycles-pp.place_entity 1.98 ± 5% -0.4 1.57 ± 2% perf-profile.children.cycles-pp.stress_sigrt 0.48 ± 2% -0.4 0.08 ± 6% perf-profile.children.cycles-pp.avg_vruntime 0.46 ± 3% -0.4 0.08 ± 11% perf-profile.children.cycles-pp.update_rq_clock 0.49 ± 5% -0.4 0.12 ± 3% perf-profile.children.cycles-pp.__dequeue_entity 0.38 ± 6% -0.4 0.02 ± 99% perf-profile.children.cycles-pp.cr4_update_irqsoff 0.39 ± 2% -0.3 0.06 ± 7% perf-profile.children.cycles-pp.__calc_delta 0.34 ± 4% -0.3 0.03 ± 70% perf-profile.children.cycles-pp.update_min_vruntime 0.34 ± 3% -0.3 0.06 ± 14% perf-profile.children.cycles-pp.sched_clock_cpu 0.42 ± 20% -0.3 0.16 ± 9% perf-profile.children.cycles-pp._raw_spin_lock 0.29 ± 25% -0.3 0.03 ±100% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime 0.56 ± 3% -0.3 0.30 ± 5% perf-profile.children.cycles-pp.__libc_current_sigrtmin_private 0.31 ± 4% -0.3 0.05 ± 46% perf-profile.children.cycles-pp.sched_clock 0.35 ± 7% -0.2 0.10 ± 38% perf-profile.children.cycles-pp.select_idle_sibling 0.28 ± 3% -0.2 0.04 ± 45% perf-profile.children.cycles-pp.native_sched_clock 0.34 ± 10% -0.2 0.13 ± 8% perf-profile.children.cycles-pp.__enqueue_entity 0.26 ± 6% -0.2 0.06 ± 9% perf-profile.children.cycles-pp.update_rq_clock_task 0.28 ± 4% -0.2 0.11 ± 6% perf-profile.children.cycles-pp.__libc_current_sigrtmin@plt 0.84 ± 2% -0.2 0.67 perf-profile.children.cycles-pp.__x64_sys_getuid 0.47 ± 5% -0.1 0.34 ± 3% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.54 ± 4% -0.1 0.41 ± 4% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.67 -0.1 0.54 ± 8% perf-profile.children.cycles-pp.__radix_tree_lookup 0.44 ± 5% -0.1 0.30 ± 3% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 0.43 ± 5% -0.1 0.30 ± 3% perf-profile.children.cycles-pp.hrtimer_interrupt 0.44 ± 13% -0.1 0.32 ± 4% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 0.32 ± 19% -0.1 0.20 ± 7% perf-profile.children.cycles-pp.__get_obj_cgroup_from_memcg 0.38 ± 5% -0.1 0.26 ± 4% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.29 ± 5% -0.1 0.20 ± 4% perf-profile.children.cycles-pp.tick_sched_handle 0.30 ± 5% -0.1 0.22 ± 5% perf-profile.children.cycles-pp.tick_sched_timer 0.28 ± 4% -0.1 0.20 ± 4% perf-profile.children.cycles-pp.update_process_times 0.36 ± 6% -0.1 0.28 ± 2% perf-profile.children.cycles-pp.audit_signal_info_syscall 0.18 ± 5% -0.1 0.10 ± 6% perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context 0.18 ± 5% -0.1 0.10 ± 8% perf-profile.children.cycles-pp.perf_event_task_tick 0.24 ± 4% -0.1 0.17 ± 7% perf-profile.children.cycles-pp.scheduler_tick 0.77 ± 3% -0.1 0.71 ± 3% perf-profile.children.cycles-pp.__x64_sys_getpid 0.45 ± 6% -0.1 0.39 perf-profile.children.cycles-pp.check_kill_permission 0.20 ± 2% -0.1 0.14 ± 4% perf-profile.children.cycles-pp.__list_del_entry_valid_or_report 0.14 ± 7% -0.0 0.10 ± 6% perf-profile.children.cycles-pp.pid_task 0.07 ± 24% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.irqentry_exit_to_user_mode 0.10 ± 5% -0.0 0.07 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.10 ± 4% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.find_vpid 0.62 ± 2% +0.0 0.66 perf-profile.children.cycles-pp.recalc_sigpending 0.08 +0.0 0.12 ± 5% perf-profile.children.cycles-pp.obj_cgroup_charge 0.00 +0.1 0.05 ± 7% perf-profile.children.cycles-pp.__set_task_blocked 0.14 ± 5% +0.1 0.20 perf-profile.children.cycles-pp.__list_add_valid_or_report 0.31 ± 5% +0.1 0.37 ± 2% perf-profile.children.cycles-pp.send_signal_locked 0.16 ± 4% +0.1 0.26 ± 3% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.00 +0.1 0.10 ± 4% perf-profile.children.cycles-pp.__local_bh_enable_ip 0.12 ± 5% +0.1 0.22 ± 2% perf-profile.children.cycles-pp.map_id_up 0.15 ± 4% +0.1 0.26 ± 2% perf-profile.children.cycles-pp.from_kuid_munged 0.00 +0.1 0.11 ± 6% perf-profile.children.cycles-pp.fpu__alloc_mathframe 0.54 ± 7% +0.1 0.65 perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.36 ± 7% +0.1 0.48 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.19 ± 4% +0.1 0.33 ± 2% perf-profile.children.cycles-pp.mod_objcg_state 0.47 +0.2 0.62 ± 3% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook 0.60 ± 6% +0.2 0.83 ± 4% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.00 +0.2 0.24 ± 3% perf-profile.children.cycles-pp.do_sigaltstack 0.06 ± 8% +0.3 0.31 ± 2% perf-profile.children.cycles-pp.fpregs_mark_activate 0.00 +0.3 0.32 ± 5% perf-profile.children.cycles-pp.__unfreeze_partials 0.07 ± 5% +0.3 0.40 perf-profile.children.cycles-pp.set_current_blocked 12.57 +0.3 12.92 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.08 ± 5% +0.4 0.45 ± 2% perf-profile.children.cycles-pp.__put_user_nocheck_4 0.07 ± 10% +0.4 0.47 ± 2% perf-profile.children.cycles-pp.__set_current_blocked 0.09 ± 10% +0.4 0.50 ± 2% perf-profile.children.cycles-pp.__get_user_nocheck_4 0.00 +0.4 0.41 ± 7% perf-profile.children.cycles-pp.get_partial_node 0.07 ±123% +0.4 0.50 ± 7% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.66 ± 3% +0.4 1.09 perf-profile.children.cycles-pp.profile_signal_perm 1.02 +0.4 1.45 perf-profile.children.cycles-pp.inc_rlimit_get_ucounts 0.09 ± 6% +0.5 0.55 perf-profile.children.cycles-pp.signal_setup_done 0.12 ± 6% +0.5 0.63 ± 2% perf-profile.children.cycles-pp.check_xstate_in_sigframe 0.82 ± 3% +0.5 1.35 perf-profile.children.cycles-pp.kmem_cache_free 0.10 ± 3% +0.6 0.67 perf-profile.children.cycles-pp.__get_user_nocheck_8 1.16 ± 2% +0.6 1.75 perf-profile.children.cycles-pp.do_dec_rlimit_put_ucounts 1.08 ± 2% +0.6 1.71 perf-profile.children.cycles-pp._copy_from_user 0.05 ± 7% +0.7 0.70 ± 5% perf-profile.children.cycles-pp.___slab_alloc 0.13 ± 3% +0.7 0.79 perf-profile.children.cycles-pp.restore_altstack 1.91 ± 3% +0.8 2.66 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 1.28 ± 5% +0.9 2.14 perf-profile.children.cycles-pp.kmem_cache_alloc 0.22 ± 6% +0.9 1.13 perf-profile.children.cycles-pp.save_xstate_epilog 0.18 ± 4% +0.9 1.12 perf-profile.children.cycles-pp.fpu__clear_user_states 1.18 ± 4% +1.0 2.19 perf-profile.children.cycles-pp.aa_get_task_label 0.28 ± 4% +1.2 1.46 ± 2% perf-profile.children.cycles-pp.native_irq_return_iret 1.40 ± 4% +1.3 2.69 perf-profile.children.cycles-pp.aa_may_signal 0.27 ± 2% +1.3 1.59 perf-profile.children.cycles-pp.restore_fpregs_from_user 2.37 ± 2% +1.3 3.70 perf-profile.children.cycles-pp.__sigqueue_alloc 6.31 +1.4 7.69 perf-profile.children.cycles-pp.__entry_text_start 0.15 ± 14% +1.4 1.57 perf-profile.children.cycles-pp.__slab_free 1.50 +1.4 2.93 perf-profile.children.cycles-pp.collect_signal 0.52 ± 6% +1.5 1.97 perf-profile.children.cycles-pp.copy_fpstate_to_sigframe 0.47 ± 2% +2.3 2.78 perf-profile.children.cycles-pp.__fpu_restore_sig 0.49 ± 3% +2.4 2.84 perf-profile.children.cycles-pp.fpu__restore_sig 10.89 +2.6 13.47 perf-profile.children.cycles-pp.getuid 0.82 ± 5% +2.7 3.48 perf-profile.children.cycles-pp.get_sigframe 10.92 +2.8 13.67 perf-profile.children.cycles-pp.__getpid 0.65 ± 4% +3.1 3.79 perf-profile.children.cycles-pp.__irqentry_text_end 0.67 ± 2% +3.2 3.86 perf-profile.children.cycles-pp.restore_sigcontext 0.94 ± 5% +3.3 4.21 perf-profile.children.cycles-pp.x64_setup_rt_frame 2.89 +3.7 6.58 perf-profile.children.cycles-pp.dequeue_signal 4.47 ± 4% +3.7 8.17 perf-profile.children.cycles-pp.apparmor_task_kill 4.51 ± 4% +3.7 8.23 perf-profile.children.cycles-pp.security_task_kill 1.16 ± 4% +4.3 5.50 perf-profile.children.cycles-pp.handle_signal 1.04 ± 2% +5.1 6.12 perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn 8.54 +5.6 14.10 perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.85 ± 3% +6.0 6.81 perf-profile.children.cycles-pp.get_signal 5.94 +7.5 13.44 perf-profile.children.cycles-pp.exit_to_user_mode_loop 19.04 +8.4 27.42 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 2.24 ± 3% +10.9 13.14 perf-profile.children.cycles-pp.stress_sighandler_nop 2.16 ± 3% +11.1 13.26 perf-profile.children.cycles-pp.arch_do_signal_or_restart 3.20 -2.8 0.40 ± 3% perf-profile.self.cycles-pp.switch_mm_irqs_off 2.16 ± 2% -1.9 0.30 ± 3% perf-profile.self.cycles-pp.__schedule 1.90 ± 2% -1.7 0.22 ± 4% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate 1.19 ± 3% -1.0 0.15 ± 8% perf-profile.self.cycles-pp.__switch_to_asm 1.17 ± 4% -1.0 0.13 ± 8% perf-profile.self.cycles-pp.update_curr 1.02 ± 2% -0.9 0.14 ± 3% perf-profile.self.cycles-pp.update_load_avg 0.98 ± 8% -0.8 0.15 ± 7% perf-profile.self.cycles-pp.__sigtimedwait 0.94 ± 2% -0.8 0.15 ± 3% perf-profile.self.cycles-pp.reweight_entity 0.82 ± 4% -0.7 0.10 ± 12% perf-profile.self.cycles-pp.___perf_sw_event 0.79 ± 2% -0.6 0.14 ± 5% perf-profile.self.cycles-pp.__update_load_avg_se 0.74 ± 4% -0.6 0.11 ± 9% perf-profile.self.cycles-pp.update_cfs_group 0.69 ± 5% -0.6 0.08 ± 4% perf-profile.self.cycles-pp.enqueue_task_fair 0.68 -0.6 0.07 ± 5% perf-profile.self.cycles-pp.pick_next_task_fair 0.72 ± 3% -0.6 0.11 ± 7% perf-profile.self.cycles-pp.pick_eevdf 0.71 ± 4% -0.6 0.11 ± 6% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 1.07 ± 29% -0.6 0.49 ± 35% perf-profile.self.cycles-pp.queue_event 0.62 ± 2% -0.5 0.08 perf-profile.self.cycles-pp.os_xsave 0.56 ± 2% -0.5 0.06 ± 7% perf-profile.self.cycles-pp.__switch_to 0.53 ± 4% -0.5 0.07 perf-profile.self.cycles-pp.select_task_rq_fair 0.54 ± 3% -0.5 0.08 ± 5% perf-profile.self.cycles-pp.__x64_sys_rt_sigtimedwait 1.90 ± 5% -0.4 1.47 ± 3% perf-profile.self.cycles-pp.stress_sigrt 0.47 ± 6% -0.4 0.05 ± 7% perf-profile.self.cycles-pp.enqueue_entity 0.46 ± 2% -0.4 0.08 ± 6% perf-profile.self.cycles-pp.avg_vruntime 0.43 ± 4% -0.4 0.05 ± 47% perf-profile.self.cycles-pp.perf_tp_event 0.44 ± 3% -0.4 0.07 perf-profile.self.cycles-pp.dequeue_task_fair 0.39 ± 2% -0.4 0.03 ± 70% perf-profile.self.cycles-pp.schedule 0.39 ± 3% -0.3 0.06 ± 7% perf-profile.self.cycles-pp.__calc_delta 0.38 ± 4% -0.3 0.07 ± 8% perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock 0.38 ± 6% -0.3 0.08 ± 5% perf-profile.self.cycles-pp.__dequeue_entity 0.34 ± 5% -0.3 0.05 ± 7% perf-profile.self.cycles-pp.try_to_wake_up 0.33 ± 4% -0.3 0.04 ± 44% perf-profile.self.cycles-pp._raw_spin_lock 0.31 ± 5% -0.3 0.02 ± 99% perf-profile.self.cycles-pp.update_min_vruntime 0.34 ± 3% -0.3 0.06 ± 7% perf-profile.self.cycles-pp.do_sigtimedwait 0.29 ± 25% -0.3 0.03 ±100% perf-profile.self.cycles-pp.perf_trace_sched_stat_runtime 0.29 ± 2% -0.2 0.04 ± 45% perf-profile.self.cycles-pp.dequeue_entity 0.68 ± 2% -0.2 0.43 perf-profile.self.cycles-pp.__x64_sys_getuid 0.28 ± 4% -0.2 0.04 ± 45% perf-profile.self.cycles-pp.native_sched_clock 0.25 ± 6% -0.2 0.04 ± 71% perf-profile.self.cycles-pp.update_rq_clock_task 0.34 ± 9% -0.2 0.13 ± 8% perf-profile.self.cycles-pp.__enqueue_entity 0.26 ± 4% -0.2 0.08 ± 11% perf-profile.self.cycles-pp.__libc_current_sigrtmin@plt 0.59 ± 6% -0.2 0.44 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.96 ± 10% -0.2 0.81 ± 2% perf-profile.self.cycles-pp.rep_movs_alternative 0.17 ± 13% -0.1 0.03 ±100% perf-profile.self.cycles-pp.select_idle_sibling 0.67 ± 2% -0.1 0.53 ± 8% perf-profile.self.cycles-pp.__radix_tree_lookup 0.96 ± 2% -0.1 0.84 perf-profile.self.cycles-pp.sigqueue 0.32 ± 19% -0.1 0.20 ± 7% perf-profile.self.cycles-pp.__get_obj_cgroup_from_memcg 0.38 ± 4% -0.1 0.30 ± 5% perf-profile.self.cycles-pp.complete_signal 0.35 ± 7% -0.1 0.28 ± 2% perf-profile.self.cycles-pp.audit_signal_info_syscall 0.14 ± 8% -0.1 0.07 ± 10% perf-profile.self.cycles-pp.perf_adjust_freq_unthr_context 0.29 ± 9% -0.1 0.22 ± 5% perf-profile.self.cycles-pp.__libc_current_sigrtmin_private 0.68 ± 4% -0.1 0.63 ± 3% perf-profile.self.cycles-pp.__task_pid_nr_ns 0.14 ± 6% -0.0 0.10 ± 6% perf-profile.self.cycles-pp.pid_task 0.17 ± 4% -0.0 0.13 ± 4% perf-profile.self.cycles-pp.__list_del_entry_valid_or_report 0.16 ± 3% -0.0 0.13 ± 2% perf-profile.self.cycles-pp.exit_to_user_mode_loop 0.08 ± 5% -0.0 0.06 ± 6% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.06 ± 7% +0.0 0.08 ± 5% perf-profile.self.cycles-pp.check_kill_permission 0.08 ± 5% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.__sigqueue_alloc 0.62 +0.0 0.64 perf-profile.self.cycles-pp.recalc_sigpending 0.14 ± 4% +0.0 0.17 ± 4% perf-profile.self.cycles-pp.__x64_sys_getpid 0.06 ± 7% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.obj_cgroup_charge 0.08 ± 4% +0.0 0.12 ± 5% perf-profile.self.cycles-pp.__libc_current_sigrtmax_private 0.14 ± 3% +0.1 0.20 ± 2% perf-profile.self.cycles-pp.__list_add_valid_or_report 0.22 ± 4% +0.1 0.28 perf-profile.self.cycles-pp.getuid 0.00 +0.1 0.07 ± 5% perf-profile.self.cycles-pp.fpu__restore_sig 0.45 ± 5% +0.1 0.52 perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.15 ± 4% +0.1 0.22 ± 4% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare 0.00 +0.1 0.08 ± 8% perf-profile.self.cycles-pp.signal_setup_done 0.00 +0.1 0.09 ± 5% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.00 +0.1 0.09 ± 4% perf-profile.self.cycles-pp.set_current_blocked 0.00 +0.1 0.09 ± 5% perf-profile.self.cycles-pp.__set_current_blocked 0.12 ± 6% +0.1 0.22 ± 3% perf-profile.self.cycles-pp.map_id_up 0.39 +0.1 0.50 ± 3% perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook 0.00 +0.1 0.10 ± 4% perf-profile.self.cycles-pp.fpu__alloc_mathframe 0.00 +0.1 0.11 ± 3% perf-profile.self.cycles-pp.__unfreeze_partials 0.44 ± 9% +0.1 0.56 ± 2% perf-profile.self.cycles-pp.syscall_enter_from_user_mode 0.18 ± 3% +0.1 0.30 ± 2% perf-profile.self.cycles-pp.mod_objcg_state 0.34 ± 8% +0.1 0.46 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.00 +0.1 0.13 ± 4% perf-profile.self.cycles-pp.get_partial_node 0.00 +0.1 0.14 ± 6% perf-profile.self.cycles-pp.handle_signal 0.23 ± 7% +0.1 0.37 perf-profile.self.cycles-pp.kmem_cache_alloc 0.37 ± 2% +0.2 0.53 ± 2% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.35 ± 3% +0.2 0.54 ± 2% perf-profile.self.cycles-pp.__send_signal_locked 0.00 +0.2 0.20 ± 4% perf-profile.self.cycles-pp.do_sigaltstack 0.00 +0.2 0.20 ± 4% perf-profile.self.cycles-pp.restore_altstack 0.04 ± 45% +0.2 0.28 ± 4% perf-profile.self.cycles-pp.get_sigframe 0.06 ± 8% +0.2 0.30 ± 2% perf-profile.self.cycles-pp.check_xstate_in_sigframe 0.05 ± 7% +0.2 0.30 perf-profile.self.cycles-pp.fpregs_mark_activate 0.06 ± 11% +0.3 0.32 perf-profile.self.cycles-pp.save_xstate_epilog 0.00 +0.3 0.28 ± 5% perf-profile.self.cycles-pp.___slab_alloc 0.08 ± 6% +0.3 0.42 ± 2% perf-profile.self.cycles-pp.__x64_sys_rt_sigreturn 12.56 +0.3 12.90 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.06 ± 11% +0.4 0.42 perf-profile.self.cycles-pp.arch_do_signal_or_restart 0.08 ± 4% +0.4 0.44 ± 3% perf-profile.self.cycles-pp.__put_user_nocheck_4 0.07 ± 5% +0.4 0.43 ± 2% perf-profile.self.cycles-pp.restore_sigcontext 0.08 ± 11% +0.4 0.49 ± 2% perf-profile.self.cycles-pp.__get_user_nocheck_4 0.58 ± 3% +0.4 0.98 ± 2% perf-profile.self.cycles-pp.kmem_cache_free 0.65 ± 3% +0.4 1.07 perf-profile.self.cycles-pp.profile_signal_perm 0.06 ±145% +0.4 0.50 ± 7% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 1.01 +0.4 1.45 perf-profile.self.cycles-pp.inc_rlimit_get_ucounts 0.70 +0.4 1.15 perf-profile.self.cycles-pp._copy_from_user 0.10 ± 4% +0.5 0.58 perf-profile.self.cycles-pp.get_signal 0.09 ± 8% +0.5 0.58 perf-profile.self.cycles-pp.__fpu_restore_sig 0.13 ± 8% +0.5 0.63 ± 2% perf-profile.self.cycles-pp.stress_sighandler_nop 0.10 +0.6 0.66 perf-profile.self.cycles-pp.__get_user_nocheck_8 1.16 ± 2% +0.6 1.75 perf-profile.self.cycles-pp.do_dec_rlimit_put_ucounts 0.12 ± 7% +0.6 0.72 ± 3% perf-profile.self.cycles-pp.x64_setup_rt_frame 0.15 ± 4% +0.8 0.96 perf-profile.self.cycles-pp.fpu__clear_user_states 0.29 ± 10% +0.8 1.10 perf-profile.self.cycles-pp.collect_signal 0.75 ± 6% +0.9 1.61 perf-profile.self.cycles-pp.aa_may_signal 1.17 ± 5% +1.0 2.17 perf-profile.self.cycles-pp.aa_get_task_label 0.28 ± 4% +1.2 1.46 ± 2% perf-profile.self.cycles-pp.native_irq_return_iret 0.24 ± 3% +1.2 1.42 ± 2% perf-profile.self.cycles-pp.restore_fpregs_from_user 0.33 ± 4% +1.3 1.65 perf-profile.self.cycles-pp.copy_fpstate_to_sigframe 5.69 +1.4 7.08 perf-profile.self.cycles-pp.__entry_text_start 1.89 ± 4% +1.4 3.29 perf-profile.self.cycles-pp.apparmor_task_kill 0.15 ± 14% +1.4 1.57 perf-profile.self.cycles-pp.__slab_free 7.83 +1.8 9.62 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 10.42 +2.7 13.13 perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.65 ± 4% +3.1 3.79 perf-profile.self.cycles-pp.__irqentry_text_end *************************************************************************************************** lkp-skl-d08: 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory ========================================================================================= class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: interrupt/gcc-12/performance/1SSD/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-skl-d08/sigsuspend/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 249337 ± 6% +50.3% 374645 ± 8% cpuidle..usage 27.84 +3.0 30.79 mpstat.cpu.all.usr% 64.76 -5.0% 61.54 iostat.cpu.system 27.07 +10.4% 29.90 iostat.cpu.user 192.83 ± 30% -96.6% 6.50 ± 52% perf-c2c.DRAM.local 2321 ± 17% +75.1% 4065 ± 15% perf-c2c.HITM.local 2321 ± 17% +75.1% 4065 ± 15% perf-c2c.HITM.total 7134990 ± 4% -51.8% 3438618 ± 3% vmstat.memory.cache 24539145 +15.8% 28406554 vmstat.memory.free 83.17 +85.6% 154.33 vmstat.procs.r 5561442 -96.3% 203150 ± 4% vmstat.system.cs 2.077e+08 -93.9% 12639824 ± 5% stress-ng.sigsuspend.ops 3461868 -93.9% 210660 ± 5% stress-ng.sigsuspend.ops_per_sec 1.559e+08 -99.7% 514434 stress-ng.time.involuntary_context_switches 3389 +1.1% 3427 stress-ng.time.percent_of_cpu_this_job_got 1487 -3.8% 1430 stress-ng.time.system_time 617.36 +13.0% 697.67 stress-ng.time.user_time 2.076e+08 -94.1% 12311096 ± 5% stress-ng.time.voluntary_context_switches 19245 ± 36% +78.9% 34432 ± 15% turbostat.C1 0.17 ± 10% +558.3% 1.13 ± 32% turbostat.CPU%c6 47.83 -12.5% 41.83 turbostat.CoreTmp 0.11 ± 3% -53.8% 0.05 turbostat.IPC 5727321 ± 7% -8.9% 5218101 ± 2% turbostat.IRQ 45347 ± 20% +201.2% 136579 ± 14% turbostat.POLL 47.50 ± 2% -12.3% 41.67 ± 2% turbostat.PkgTmp 188.77 -17.4% 156.00 turbostat.PkgWatt 1.01 -11.6% 0.89 turbostat.RAMWatt 97263 ± 38% +497.2% 580860 ± 17% meminfo.Active 87518 ± 38% +563.5% 580681 ± 17% meminfo.Active(anon) 433224 -13.5% 374894 meminfo.AnonPages 7019086 ± 4% -52.2% 3358359 ± 3% meminfo.Cached 6553310 ± 5% -57.0% 2820190 ± 3% meminfo.Committed_AS 5307050 ± 21% +23.3% 6542336 ± 11% meminfo.DirectMap2M 4653770 ± 6% -90.5% 441481 ± 3% meminfo.Inactive 4647503 ± 6% -90.5% 441107 ± 3% meminfo.Inactive(anon) 85173 -10.2% 76471 meminfo.KReclaimable 1228127 ± 8% -92.0% 98369 ± 13% meminfo.Mapped 24353220 +15.7% 28178003 meminfo.MemAvailable 24567682 +15.6% 28404545 meminfo.MemFree 7994352 ± 4% -48.0% 4157489 ± 2% meminfo.Memused 16997 ± 3% -12.9% 14808 ± 4% meminfo.PageTables 85173 -10.2% 76471 meminfo.SReclaimable 4302180 ± 7% -85.0% 647263 ± 16% meminfo.Shmem 8034165 ± 4% -46.4% 4303791 ± 2% meminfo.max_used_kB 21833 ± 38% +565.0% 145182 ± 17% proc-vmstat.nr_active_anon 108372 -13.6% 93685 proc-vmstat.nr_anon_pages 607141 +15.7% 702469 proc-vmstat.nr_dirty_background_threshold 1215767 +15.7% 1406656 proc-vmstat.nr_dirty_threshold 1756412 ± 4% -52.2% 839582 ± 3% proc-vmstat.nr_file_pages 6142589 +15.6% 7101129 proc-vmstat.nr_free_pages 1161234 ± 6% -90.5% 110217 ± 3% proc-vmstat.nr_inactive_anon 305935 ± 9% -92.0% 24521 ± 13% proc-vmstat.nr_mapped 4252 ± 3% -13.0% 3701 ± 4% proc-vmstat.nr_page_table_pages 1074788 ± 7% -84.9% 161806 ± 16% proc-vmstat.nr_shmem 21290 -10.2% 19117 proc-vmstat.nr_slab_reclaimable 21833 ± 38% +565.0% 145182 ± 17% proc-vmstat.nr_zone_active_anon 1161234 ± 6% -90.5% 110217 ± 3% proc-vmstat.nr_zone_inactive_anon 1777928 ± 6% -73.1% 478826 ± 8% proc-vmstat.numa_hit 1784234 ± 6% -73.4% 474428 ± 7% proc-vmstat.numa_local 18962 ± 5% +684.1% 148679 ± 6% proc-vmstat.pgactivate 1830911 ± 6% -71.3% 525134 ± 7% proc-vmstat.pgalloc_normal 316769 -13.9% 272741 proc-vmstat.pgfault 216412 +27.0% 274858 proc-vmstat.pgfree 917478 +21.2% 1111830 sched_debug.cfs_rq:/.avg_vruntime.avg 976815 ± 2% +60.8% 1570651 ± 11% sched_debug.cfs_rq:/.avg_vruntime.max 19932 ± 22% +576.0% 134748 ± 15% sched_debug.cfs_rq:/.avg_vruntime.stddev 1.52 ± 4% +79.1% 2.72 ± 3% sched_debug.cfs_rq:/.h_nr_running.avg 3.25 ± 7% +35.9% 4.42 ± 7% sched_debug.cfs_rq:/.h_nr_running.max 0.95 ± 2% -14.7% 0.81 ± 14% sched_debug.cfs_rq:/.h_nr_running.stddev 917478 +21.2% 1111831 sched_debug.cfs_rq:/.min_vruntime.avg 976815 ± 2% +60.8% 1570651 ± 11% sched_debug.cfs_rq:/.min_vruntime.max 19932 ± 22% +576.0% 134748 ± 15% sched_debug.cfs_rq:/.min_vruntime.stddev 1652 +69.4% 2798 ± 2% sched_debug.cfs_rq:/.runnable_avg.avg 2748 ± 11% +60.4% 4409 ± 8% sched_debug.cfs_rq:/.runnable_avg.max 1097 ± 18% -37.3% 688.33 ± 23% sched_debug.cfs_rq:/.runnable_avg.min 341.89 ± 17% +111.6% 723.58 ± 15% sched_debug.cfs_rq:/.runnable_avg.stddev 295.42 ± 3% -22.5% 228.81 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.avg 1044 ± 9% -14.3% 894.42 ± 8% sched_debug.cfs_rq:/.util_est_enqueued.max 230.41 ± 4% -9.3% 208.98 ± 4% sched_debug.cfs_rq:/.util_est_enqueued.stddev 3.25 ± 8% -32.0% 2.21 ± 19% sched_debug.cpu.clock.stddev 81.33 ± 11% -17.0% 67.53 ± 5% sched_debug.cpu.clock_task.stddev 1.50 ± 5% +81.2% 2.72 ± 2% sched_debug.cpu.nr_running.avg 3.25 ± 7% +35.9% 4.42 ± 7% sched_debug.cpu.nr_running.max 4802408 -96.2% 181526 ± 5% sched_debug.cpu.nr_switches.avg 5003393 -83.9% 803946 ± 27% sched_debug.cpu.nr_switches.max 4149380 ± 7% -99.0% 42283 ± 9% sched_debug.cpu.nr_switches.min 9.351e+09 -55.6% 4.155e+09 perf-stat.i.branch-instructions 1.573e+08 -56.0% 69287057 perf-stat.i.branch-misses 4.23 ± 7% -2.2 2.01 ± 2% perf-stat.i.cache-miss-rate% 6751954 ± 5% -78.8% 1428725 ± 8% perf-stat.i.cache-misses 1.872e+08 ± 10% -41.4% 1.096e+08 ± 9% perf-stat.i.cache-references 5817774 -96.4% 211465 ± 5% perf-stat.i.context-switches 2.75 +115.9% 5.94 perf-stat.i.cpi 327628 ± 14% +373.6% 1551605 ± 4% perf-stat.i.cycles-between-cache-misses 1.34 -1.0 0.34 ± 2% perf-stat.i.dTLB-load-miss-rate% 1.787e+08 -88.8% 20010077 ± 2% perf-stat.i.dTLB-load-misses 1.302e+10 -54.8% 5.879e+09 perf-stat.i.dTLB-loads 0.70 ± 85% -0.7 0.05 ± 7% perf-stat.i.dTLB-store-miss-rate% 1.351e+12 ±223% -100.0% 1548664 ± 7% perf-stat.i.dTLB-store-misses 7.591e+09 -57.9% 3.197e+09 perf-stat.i.dTLB-stores 52.32 +31.9 84.24 perf-stat.i.iTLB-load-miss-rate% 77655104 -79.3% 16073669 ± 2% perf-stat.i.iTLB-load-misses 67953773 -96.2% 2583574 ± 6% perf-stat.i.iTLB-loads 4.781e+10 -54.6% 2.17e+10 perf-stat.i.instructions 2609 ± 7% +27.4% 3324 ± 3% perf-stat.i.instructions-per-iTLB-miss 0.40 -48.1% 0.21 perf-stat.i.ipc 691.54 ± 22% -78.0% 151.98 ± 7% perf-stat.i.metric.K/sec 839.73 -55.9% 370.40 perf-stat.i.metric.M/sec 3605 ± 2% -18.3% 2945 perf-stat.i.minor-faults 1917836 ± 14% -91.7% 158930 ± 5% perf-stat.i.node-loads 559746 ± 6% -78.7% 119446 ± 18% perf-stat.i.node-stores 3606 ± 2% -18.3% 2946 perf-stat.i.page-faults 0.14 ± 4% -53.9% 0.07 ± 7% perf-stat.overall.MPKI 3.65 ± 12% -2.4 1.30 ± 10% perf-stat.overall.cache-miss-rate% 2.73 +119.3% 5.98 perf-stat.overall.cpi 19357 ± 4% +377.8% 92490 ± 7% perf-stat.overall.cycles-between-cache-misses 1.35 -1.0 0.34 perf-stat.overall.dTLB-load-miss-rate% 17.01 ±217% -17.0 0.05 ± 7% perf-stat.overall.dTLB-store-miss-rate% 53.33 +32.9 86.19 perf-stat.overall.iTLB-load-miss-rate% 615.74 +119.1% 1348 perf-stat.overall.instructions-per-iTLB-miss 0.37 -54.4% 0.17 perf-stat.overall.ipc 9.204e+09 -55.5% 4.094e+09 perf-stat.ps.branch-instructions 1.548e+08 -56.1% 68037157 perf-stat.ps.branch-misses 6647486 ± 5% -79.1% 1392058 ± 8% perf-stat.ps.cache-misses 1.843e+08 ± 10% -41.3% 1.081e+08 ± 9% perf-stat.ps.cache-references 5726331 -96.4% 208210 ± 5% perf-stat.ps.context-switches 1.759e+08 -88.8% 19737632 ± 2% perf-stat.ps.dTLB-load-misses 1.281e+10 -54.8% 5.795e+09 perf-stat.ps.dTLB-loads 1.326e+12 ±223% -100.0% 1525476 ± 7% perf-stat.ps.dTLB-store-misses 7.472e+09 -57.8% 3.153e+09 perf-stat.ps.dTLB-stores 76434496 -79.2% 15860324 ± 2% perf-stat.ps.iTLB-load-misses 66885780 -96.2% 2543308 ± 6% perf-stat.ps.iTLB-loads 4.706e+10 -54.6% 2.139e+10 perf-stat.ps.instructions 3547 ± 2% -19.2% 2865 perf-stat.ps.minor-faults 1887790 ± 14% -91.7% 156314 ± 5% perf-stat.ps.node-loads 551228 ± 6% -78.8% 117021 ± 18% perf-stat.ps.node-stores 3548 ± 2% -19.2% 2865 perf-stat.ps.page-faults 2.99e+12 -54.5% 1.36e+12 perf-stat.total.instructions 0.01 ±148% +600.0% 0.04 ± 40% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.01 ± 71% +21401.7% 2.11 ± 52% perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.26 ±139% +652.5% 1.96 ± 28% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.02 ± 73% +3052.6% 0.60 ± 81% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.02 ± 99% +2003.0% 0.47 ± 29% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.16 ± 71% +219.9% 0.50 ± 6% perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.43 ±116% +412.5% 2.18 ± 11% perf-sched.sch_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 0.00 ± 71% +5040.0% 0.21 ± 15% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.08 ±120% +1422.8% 1.26 ± 44% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ± 71% +20254.5% 1.12 ± 13% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.03 ± 70% +2713.2% 0.71 ± 21% perf-sched.sch_delay.avg.ms.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.02 ± 95% +3400.8% 0.75 ± 54% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.04 ± 72% +25117.5% 10.09 ± 53% perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 1.16 ±142% +330.0% 4.98 ± 44% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1.08 ±128% +392.6% 5.32 ±124% perf-sched.sch_delay.max.ms.devkmsg_read.vfs_read.ksys_read.do_syscall_64 0.39 ±147% +729.0% 3.21 ± 30% perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.01 ± 72% +39108.7% 3.01 perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.76 ±142% +267.9% 2.79 ± 21% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ± 73% +34165.8% 2.17 ± 17% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.13 ± 81% +4389.0% 5.73 ± 51% perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.02 ± 71% +4307.6% 0.68 ± 20% perf-sched.total_sch_delay.average.ms 0.10 ± 70% +1928.4% 1.93 ± 20% perf-sched.total_wait_and_delay.average.ms 0.08 ± 70% +1473.4% 1.25 ± 20% perf-sched.total_wait_time.average.ms 0.35 ± 77% +2608.9% 9.56 ± 69% perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.28 ± 82% +7145.3% 19.96 ± 56% perf-sched.wait_and_delay.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 276.75 ± 85% +190.3% 803.34 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.05 ±141% +3263.9% 1.79 ± 76% perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.19 ± 76% +1535.7% 3.09 ± 9% perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.55 ± 72% +526.0% 3.46 ± 6% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.01 ± 70% +5118.5% 0.70 ± 10% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 137.43 ± 76% +302.6% 553.27 ± 8% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.11 ± 70% +1285.7% 1.50 ± 20% perf-sched.wait_and_delay.avg.ms.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe 4.00 ± 82% +150.0% 10.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 28.33 ± 74% +126.5% 64.17 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 4669 ± 71% +201.6% 14080 ± 2% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 8.33 ± 74% +206.0% 25.50 ± 10% perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 9.17 ± 77% +1574.5% 153.50 ± 13% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 19.67 ± 71% +113.6% 42.00 perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 4.67 ± 73% +114.3% 10.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork 41.33 ± 72% +150.8% 103.67 ± 12% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 171.00 ± 72% +78.3% 304.83 ± 5% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 18.91 ± 85% +719.3% 154.94 ±114% perf-sched.wait_and_delay.max.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 9.29 ± 91% +714.6% 75.71 ± 76% perf-sched.wait_and_delay.max.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 191.87 ±190% +425.1% 1007 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 1.29 ± 91% +1076.9% 15.18 ± 17% perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.25 ± 79% +2911.4% 7.59 ± 87% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 0.01 ±101% +27812.8% 1.81 ± 70% perf-sched.wait_time.avg.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 0.35 ± 77% +2608.9% 9.56 ± 69% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.02 ±100% +20233.7% 3.52 ± 52% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_write.do_syscall_64 0.01 ± 71% +29442.9% 3.10 ± 91% perf-sched.wait_time.avg.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 0.28 ± 82% +7145.3% 19.96 ± 56% perf-sched.wait_time.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 0.12 ±210% +2745.5% 3.41 ± 42% perf-sched.wait_time.avg.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 276.49 ± 85% +189.8% 801.38 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.08 ± 71% +1396.0% 1.19 ± 73% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 125.29 ± 74% +99.9% 250.43 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.17 ± 75% +1474.2% 2.62 ± 14% perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.40 ± 73% +647.9% 2.96 ± 6% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.59 ±109% +734.4% 4.91 ± 42% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.01 ± 70% +5183.8% 0.70 ± 10% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 125.95 ± 76% +338.8% 552.62 ± 8% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.08 ± 70% +850.6% 0.80 ± 19% perf-sched.wait_time.avg.ms.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.42 ±101% +201.7% 1.28 ± 34% perf-sched.wait_time.avg.ms.syslog_print.do_syslog.kmsg_read.vfs_read 0.01 ±103% +67159.1% 7.40 ±105% perf-sched.wait_time.max.ms.__cond_resched.down_write.shmem_file_write_iter.vfs_write.ksys_write 18.91 ± 85% +719.3% 154.94 ±114% perf-sched.wait_time.max.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.19 ±196% +18735.6% 35.32 ± 35% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_write.do_syscall_64 0.02 ± 73% +1.1e+05% 25.54 ±130% perf-sched.wait_time.max.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 9.29 ± 91% +714.6% 75.71 ± 76% perf-sched.wait_time.max.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 1.01 ±221% +2598.5% 27.34 ± 44% perf-sched.wait_time.max.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 210.40 ±168% +377.1% 1003 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.98 ± 84% +1376.7% 14.48 ± 21% perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.83 ±104% +208.9% 2.56 ± 34% perf-sched.wait_time.max.ms.syslog_print.do_syslog.kmsg_read.vfs_read 30.53 -28.4 2.16 ± 26% perf-profile.calltrace.cycles-pp.__sigsuspend 24.42 -22.6 1.82 ± 26% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__sigsuspend 22.97 -21.3 1.67 ± 26% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 13.62 -13.1 0.53 ± 47% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 11.04 -10.4 0.65 ± 47% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 10.39 -9.8 0.62 ± 47% perf-profile.calltrace.cycles-pp.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 11.06 -9.7 1.35 ± 11% perf-profile.calltrace.cycles-pp.__send_signal_locked.do_send_sig_info.kill_pid_info.kill_something_info.__x64_sys_kill 10.17 -9.6 0.61 ± 47% perf-profile.calltrace.cycles-pp.schedule.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.97 -9.4 0.60 ± 47% perf-profile.calltrace.cycles-pp.__schedule.schedule.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64 11.95 -9.2 2.73 ± 6% perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64 9.36 -8.8 0.59 ± 46% perf-profile.calltrace.cycles-pp.stress_sighandler_nop 9.04 -8.3 0.73 ± 47% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 8.57 -8.1 0.48 ± 46% perf-profile.calltrace.cycles-pp.complete_signal.__send_signal_locked.do_send_sig_info.kill_pid_info.kill_something_info 8.24 -7.9 0.37 ± 71% perf-profile.calltrace.cycles-pp.try_to_wake_up.complete_signal.__send_signal_locked.do_send_sig_info.kill_pid_info 7.91 -7.4 0.52 ± 47% perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 6.93 -6.4 0.53 ± 47% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 6.59 -5.9 0.72 ± 3% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 5.44 -5.4 0.00 perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 5.28 -5.3 0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 4.99 ± 2% -5.0 0.00 perf-profile.calltrace.cycles-pp.ttwu_do_activate.try_to_wake_up.complete_signal.__send_signal_locked.do_send_sig_info 1.06 ± 3% -0.5 0.57 ± 6% perf-profile.calltrace.cycles-pp.clock_gettime 0.58 ± 3% -0.1 0.44 ± 44% perf-profile.calltrace.cycles-pp.stress_time_now_timespec 0.60 ± 5% +0.2 0.77 ± 8% perf-profile.calltrace.cycles-pp.__radix_tree_lookup.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.56 ± 6% perf-profile.calltrace.cycles-pp.check_kill_permission.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64 0.98 ± 8% +0.6 1.58 ± 13% perf-profile.calltrace.cycles-pp.profile_signal_perm.aa_may_signal.apparmor_task_kill.security_task_kill.kill_pid_info 0.00 +0.7 0.75 ± 2% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.do_send_sig_info.kill_pid_info.kill_something_info.__x64_sys_kill 0.17 ±141% +1.0 1.22 ± 3% perf-profile.calltrace.cycles-pp.__task_pid_nr_ns.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 1.72 +1.2 2.87 perf-profile.calltrace.cycles-pp.__entry_text_start.kill 0.55 ± 2% +1.3 1.86 ± 3% perf-profile.calltrace.cycles-pp.stress_time_now 0.00 +1.5 1.48 ± 8% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 0.74 ± 26% +2.1 2.85 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_safe_stack.kill 3.12 ± 3% +4.2 7.35 ± 7% perf-profile.calltrace.cycles-pp.aa_may_signal.apparmor_task_kill.security_task_kill.kill_pid_info.kill_something_info 0.94 ± 7% +4.5 5.45 ± 3% perf-profile.calltrace.cycles-pp.stress_atomic_lock_acquire 4.76 +6.5 11.22 perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.kill 3.43 ± 2% +11.1 14.48 perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.kill_pid_info.kill_something_info 24.98 +25.4 50.35 perf-profile.calltrace.cycles-pp.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe 25.86 +25.8 51.63 perf-profile.calltrace.cycles-pp.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 27.18 +27.1 54.31 perf-profile.calltrace.cycles-pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 38.37 +27.7 66.05 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 41.43 +30.4 71.79 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill 12.29 ± 2% +34.3 46.54 perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.kill_pid_info.kill_something_info.__x64_sys_kill 12.37 ± 2% +34.3 46.70 perf-profile.calltrace.cycles-pp.security_task_kill.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64 50.05 +37.8 87.88 perf-profile.calltrace.cycles-pp.kill 30.84 -28.7 2.17 ± 26% perf-profile.children.cycles-pp.__sigsuspend 15.71 -15.0 0.70 ± 24% perf-profile.children.cycles-pp.schedule 15.41 -14.7 0.70 ± 25% perf-profile.children.cycles-pp.__schedule 14.90 -13.4 1.48 ± 13% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 13.70 -12.9 0.76 ± 25% perf-profile.children.cycles-pp.exit_to_user_mode_loop 23.14 -11.0 12.12 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 11.06 -10.4 0.71 ± 26% perf-profile.children.cycles-pp.__x64_sys_rt_sigsuspend 10.39 -9.7 0.68 ± 26% perf-profile.children.cycles-pp.sigsuspend 11.09 -9.7 1.39 ± 11% perf-profile.children.cycles-pp.__send_signal_locked 11.98 -9.2 2.79 ± 6% perf-profile.children.cycles-pp.do_send_sig_info 9.00 -8.3 0.69 ± 25% perf-profile.children.cycles-pp.stress_sighandler_nop 8.58 -8.1 0.52 ± 24% perf-profile.children.cycles-pp.complete_signal 8.29 -7.8 0.50 ± 24% perf-profile.children.cycles-pp.try_to_wake_up 7.93 -7.2 0.71 ± 27% perf-profile.children.cycles-pp.arch_do_signal_or_restart 5.02 ± 2% -4.8 0.25 ± 25% perf-profile.children.cycles-pp.ttwu_do_activate 4.86 -4.5 0.32 ± 26% perf-profile.children.cycles-pp.handle_signal 4.17 ± 4% -4.0 0.20 ± 24% perf-profile.children.cycles-pp.dequeue_task_fair 4.13 ± 2% -3.9 0.23 ± 25% perf-profile.children.cycles-pp.activate_task 3.99 ± 3% -3.8 0.22 ± 26% perf-profile.children.cycles-pp.enqueue_task_fair 4.01 -3.8 0.26 ± 27% perf-profile.children.cycles-pp.x64_setup_rt_frame 3.95 -3.7 0.27 ± 26% perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn 3.56 -3.4 0.12 ± 20% perf-profile.children.cycles-pp.pick_next_task_fair 3.54 -3.3 0.22 ± 26% perf-profile.children.cycles-pp.get_sigframe 3.07 -2.9 0.18 ± 25% perf-profile.children.cycles-pp.switch_mm_irqs_off 2.75 ± 4% -2.6 0.14 ± 25% perf-profile.children.cycles-pp.update_curr 2.56 -2.4 0.19 ± 24% perf-profile.children.cycles-pp.__irqentry_text_end 2.52 -2.3 0.17 ± 27% perf-profile.children.cycles-pp.restore_sigcontext 2.46 -2.3 0.16 ± 24% perf-profile.children.cycles-pp.copy_fpstate_to_sigframe 2.34 ± 3% -2.2 0.13 ± 18% perf-profile.children.cycles-pp.update_load_avg 2.52 -2.2 0.31 ± 29% perf-profile.children.cycles-pp.get_signal 2.15 -2.0 0.12 ± 27% perf-profile.children.cycles-pp.enqueue_entity 2.11 ± 4% -2.0 0.10 ± 26% perf-profile.children.cycles-pp.dequeue_entity 1.88 -1.8 0.12 ± 28% perf-profile.children.cycles-pp.fpu__restore_sig 1.83 -1.7 0.12 ± 27% perf-profile.children.cycles-pp.__fpu_restore_sig 1.87 -1.7 0.19 ± 29% perf-profile.children.cycles-pp.dequeue_signal 1.75 -1.7 0.09 ± 26% perf-profile.children.cycles-pp.__sigqueue_alloc 1.44 ± 4% -1.4 0.07 ± 47% perf-profile.children.cycles-pp.reweight_entity 1.34 -1.3 0.03 ±100% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate 1.36 ± 12% -1.3 0.09 ± 20% perf-profile.children.cycles-pp.select_task_rq 1.59 ± 2% -1.3 0.33 ± 2% perf-profile.children.cycles-pp.stress_sigsuspend 1.22 -1.2 0.04 ± 71% perf-profile.children.cycles-pp.set_next_entity 1.19 ± 3% -1.2 0.02 ± 99% perf-profile.children.cycles-pp.prepare_task_switch 1.15 ± 14% -1.1 0.07 ± 45% perf-profile.children.cycles-pp.select_task_rq_fair 1.05 ± 2% -1.0 0.06 ± 48% perf-profile.children.cycles-pp._copy_from_user 1.04 -1.0 0.06 ± 49% perf-profile.children.cycles-pp.collect_signal 1.02 ± 2% -0.9 0.07 ± 47% perf-profile.children.cycles-pp.native_irq_return_iret 1.00 -0.9 0.06 ± 45% perf-profile.children.cycles-pp.restore_fpregs_from_user 0.82 ± 3% -0.8 0.05 ± 46% perf-profile.children.cycles-pp.kmem_cache_alloc 0.79 -0.8 0.03 ±101% perf-profile.children.cycles-pp.do_dec_rlimit_put_ucounts 0.78 -0.7 0.05 ± 48% perf-profile.children.cycles-pp.save_xstate_epilog 0.71 -0.7 0.05 ± 45% perf-profile.children.cycles-pp.fpu__clear_user_states 0.66 ± 3% -0.6 0.05 ± 48% perf-profile.children.cycles-pp.kmem_cache_free 1.08 ± 3% -0.5 0.58 ± 6% perf-profile.children.cycles-pp.clock_gettime 0.43 ± 2% -0.4 0.07 ± 47% perf-profile.children.cycles-pp._raw_spin_lock_irq 4.62 -0.3 4.31 perf-profile.children.cycles-pp.__entry_text_start 0.37 ± 3% -0.3 0.06 ± 47% perf-profile.children.cycles-pp.signal_setup_done 0.42 ± 3% -0.3 0.11 ± 26% perf-profile.children.cycles-pp._raw_spin_lock 0.36 ± 2% -0.3 0.08 ± 48% perf-profile.children.cycles-pp.recalc_sigpending 0.31 ± 2% -0.2 0.06 ± 46% perf-profile.children.cycles-pp.__set_current_blocked 0.51 ± 4% -0.2 0.27 ± 10% perf-profile.children.cycles-pp.stress_lock_acquire 0.30 ± 7% -0.2 0.09 ± 10% perf-profile.children.cycles-pp.clock_gettime@plt 0.09 ± 9% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.map_id_up 0.02 ±141% +0.0 0.06 ± 7% perf-profile.children.cycles-pp.task_tick_fair 0.00 +0.1 0.05 perf-profile.children.cycles-pp.idr_find 0.01 ±223% +0.1 0.06 ± 13% perf-profile.children.cycles-pp.audit_signal_info 0.30 ± 2% +0.1 0.36 ± 3% perf-profile.children.cycles-pp.send_signal_locked 0.10 ± 6% +0.1 0.17 ± 4% perf-profile.children.cycles-pp.find_vpid 0.28 ± 2% +0.1 0.37 ± 9% perf-profile.children.cycles-pp.audit_signal_info_syscall 0.44 ± 3% +0.1 0.54 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.34 ± 2% +0.2 0.50 ± 2% perf-profile.children.cycles-pp.prepare_signal 0.60 +0.2 0.77 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.61 ± 5% +0.2 0.78 ± 8% perf-profile.children.cycles-pp.__radix_tree_lookup 0.39 ± 3% +0.2 0.57 ± 6% perf-profile.children.cycles-pp.check_kill_permission 1.31 ± 3% +0.3 1.57 ± 2% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.99 ± 8% +0.6 1.59 ± 13% perf-profile.children.cycles-pp.profile_signal_perm 0.64 ± 2% +0.7 1.32 ± 3% perf-profile.children.cycles-pp.stress_time_now 0.59 ± 3% +0.8 1.36 ± 3% perf-profile.children.cycles-pp.__task_pid_nr_ns 0.23 ± 3% +1.3 1.49 ± 8% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 66.38 +1.9 68.30 perf-profile.children.cycles-pp.do_syscall_64 72.50 +2.0 74.46 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 7.29 +4.2 11.45 perf-profile.children.cycles-pp.syscall_return_via_sysret 3.13 ± 3% +4.3 7.39 ± 7% perf-profile.children.cycles-pp.aa_may_signal 1.03 ± 7% +5.0 6.02 ± 3% perf-profile.children.cycles-pp.stress_atomic_lock_acquire 3.43 ± 2% +11.1 14.49 perf-profile.children.cycles-pp.aa_get_task_label 25.00 +25.4 50.38 perf-profile.children.cycles-pp.kill_pid_info 25.88 +25.8 51.67 perf-profile.children.cycles-pp.kill_something_info 27.21 +27.2 54.36 perf-profile.children.cycles-pp.__x64_sys_kill 12.30 ± 2% +34.3 46.57 perf-profile.children.cycles-pp.apparmor_task_kill 12.38 ± 2% +34.3 46.70 perf-profile.children.cycles-pp.security_task_kill 50.41 +37.9 88.35 perf-profile.children.cycles-pp.kill 3.01 -2.8 0.18 ± 25% perf-profile.self.cycles-pp.switch_mm_irqs_off 2.56 -2.4 0.19 ± 24% perf-profile.self.cycles-pp.__irqentry_text_end 1.90 ± 2% -1.8 0.07 ± 46% perf-profile.self.cycles-pp.__schedule 1.33 -1.3 0.02 ± 99% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate 1.30 ± 7% -1.2 0.06 ± 47% perf-profile.self.cycles-pp.update_curr 1.34 -1.2 0.10 ± 25% perf-profile.self.cycles-pp.copy_fpstate_to_sigframe 1.27 ± 2% -1.1 0.20 ± 19% perf-profile.self.cycles-pp.stress_sigsuspend 1.01 ± 2% -0.9 0.07 ± 47% perf-profile.self.cycles-pp.native_irq_return_iret 0.88 -0.8 0.05 ± 47% perf-profile.self.cycles-pp.restore_fpregs_from_user 0.79 -0.8 0.03 ±101% perf-profile.self.cycles-pp.do_dec_rlimit_put_ucounts 0.42 ± 2% -0.4 0.06 ± 46% perf-profile.self.cycles-pp._raw_spin_lock_irq 4.25 -0.3 3.94 perf-profile.self.cycles-pp.__entry_text_start 0.36 ± 2% -0.3 0.07 ± 48% perf-profile.self.cycles-pp.get_signal 0.79 ± 3% -0.3 0.52 ± 7% perf-profile.self.cycles-pp.clock_gettime 0.35 ± 3% -0.3 0.08 ± 48% perf-profile.self.cycles-pp.recalc_sigpending 0.50 ± 4% -0.3 0.24 ± 7% perf-profile.self.cycles-pp.stress_lock_acquire 0.50 ± 2% -0.2 0.35 perf-profile.self.cycles-pp.do_syscall_64 0.41 ± 2% -0.1 0.30 ± 4% perf-profile.self.cycles-pp.__send_signal_locked 0.34 ± 4% -0.1 0.24 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.07 ± 11% -0.0 0.03 ± 70% perf-profile.self.cycles-pp.perf_adjust_freq_unthr_context 0.10 ± 7% +0.0 0.12 ± 6% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.12 ± 4% +0.0 0.15 ± 4% perf-profile.self.cycles-pp.send_signal_locked 0.08 ± 10% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.map_id_up 0.51 ± 4% +0.0 0.56 ± 3% perf-profile.self.cycles-pp.stress_time_now_timespec 0.00 +0.1 0.05 perf-profile.self.cycles-pp.idr_find 0.14 ± 5% +0.1 0.20 ± 3% perf-profile.self.cycles-pp.kill_pid_info 0.09 ± 6% +0.1 0.16 ± 4% perf-profile.self.cycles-pp.find_vpid 0.07 ± 6% +0.1 0.14 ± 10% perf-profile.self.cycles-pp.security_task_kill 0.12 ± 5% +0.1 0.18 ± 4% perf-profile.self.cycles-pp.do_send_sig_info 0.07 ± 8% +0.1 0.14 ± 6% perf-profile.self.cycles-pp.check_kill_permission 0.39 ± 2% +0.1 0.47 ± 2% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.28 ± 3% +0.1 0.36 ± 9% perf-profile.self.cycles-pp.audit_signal_info_syscall 0.33 ± 2% +0.1 0.47 ± 2% perf-profile.self.cycles-pp.prepare_signal 0.14 ± 4% +0.1 0.28 ± 3% perf-profile.self.cycles-pp.kill_something_info 0.59 +0.2 0.75 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.54 ± 2% +0.2 0.71 ± 3% perf-profile.self.cycles-pp.stress_time_now 0.60 ± 5% +0.2 0.78 ± 8% perf-profile.self.cycles-pp.__radix_tree_lookup 0.42 ± 2% +0.2 0.65 ± 2% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.97 ± 8% +0.6 1.57 ± 13% perf-profile.self.cycles-pp.profile_signal_perm 0.69 +0.7 1.34 perf-profile.self.cycles-pp.__x64_sys_kill 0.57 ± 2% +0.8 1.33 ± 3% perf-profile.self.cycles-pp.__task_pid_nr_ns 8.10 +1.2 9.26 perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.21 ± 2% +1.3 1.47 ± 8% perf-profile.self.cycles-pp.syscall_exit_to_user_mode_prepare 2.14 ± 2% +3.7 5.80 ± 5% perf-profile.self.cycles-pp.aa_may_signal 7.28 +4.2 11.44 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.99 ± 7% +5.0 5.98 ± 3% perf-profile.self.cycles-pp.stress_atomic_lock_acquire 3.41 ± 2% +11.0 14.43 perf-profile.self.cycles-pp.aa_get_task_label 5.73 +18.9 24.62 perf-profile.self.cycles-pp.apparmor_task_kill *************************************************************************************************** lkp-icl-2sp8: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory ========================================================================================= class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: interrupt/gcc-12/performance/1HDD/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp8/sigsuspend/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 3.153e+08 ± 11% +40.9% 4.443e+08 ± 14% cpuidle..time 5481241 ± 18% +276.0% 20610803 ± 22% cpuidle..usage 9.61 ± 5% +27.1% 12.21 ± 10% iostat.cpu.idle 10.25 ± 2% -29.0% 7.28 ± 3% iostat.cpu.user 6.87 ± 7% +2.4 9.30 ± 13% mpstat.cpu.all.idle% 0.85 ± 2% +0.6 1.44 ± 18% mpstat.cpu.all.irq% 0.01 ± 7% +0.0 0.02 ± 14% mpstat.cpu.all.soft% 10.54 ± 2% -3.0 7.52 ± 3% mpstat.cpu.all.usr% 79719 ± 19% +2380.3% 1977310 ± 55% numa-meminfo.node1.Active 75233 ± 12% +2528.1% 1977238 ± 55% numa-meminfo.node1.Active(anon) 5606537 ± 46% -87.0% 730627 ± 29% numa-meminfo.node1.Inactive 5602487 ± 46% -87.0% 730499 ± 29% numa-meminfo.node1.Inactive(anon) 18805 ± 12% +2521.8% 493051 ± 55% numa-vmstat.node1.nr_active_anon 1400600 ± 46% -87.0% 182524 ± 29% numa-vmstat.node1.nr_inactive_anon 18805 ± 12% +2521.8% 493050 ± 55% numa-vmstat.node1.nr_zone_active_anon 1400600 ± 46% -87.0% 182525 ± 29% numa-vmstat.node1.nr_zone_inactive_anon 9472810 ± 11% -34.9% 6168068 ± 4% vmstat.memory.cache 126.17 +88.2% 237.50 vmstat.procs.r 11136388 -86.5% 1508689 ± 5% vmstat.system.cs 280720 ± 11% +88.4% 528750 ± 13% vmstat.system.in 445.50 ± 21% -56.0% 196.17 ± 24% perf-c2c.DRAM.local 1718 ± 7% +46.8% 2522 ± 11% perf-c2c.DRAM.remote 2903 ± 11% +329.9% 12482 ± 9% perf-c2c.HITM.local 812.00 ± 9% +25.9% 1022 ± 17% perf-c2c.HITM.remote 3715 ± 9% +263.5% 13505 ± 9% perf-c2c.HITM.total 4.119e+08 -82.1% 73702782 ± 12% stress-ng.sigsuspend.ops 6864693 -82.1% 1228335 ± 12% stress-ng.sigsuspend.ops_per_sec 3.171e+08 -99.7% 854068 stress-ng.time.involuntary_context_switches 25483 -4.4% 24363 stress-ng.time.minor_page_faults 5944 -3.5% 5735 ± 2% stress-ng.time.percent_of_cpu_this_job_got 410.58 ± 2% -28.9% 292.11 ± 4% stress-ng.time.user_time 4.117e+08 -82.2% 73220853 ± 12% stress-ng.time.voluntary_context_switches 5374378 ± 19% +278.1% 20318436 ± 23% turbostat.C1 6.99 ± 12% +3.2 10.15 ± 15% turbostat.C1% 6.60 ± 13% +26.9% 8.37 ± 13% turbostat.CPU%c1 63.83 ± 3% -7.8% 58.83 ± 2% turbostat.CoreTmp 0.14 ± 2% -63.5% 0.05 ± 7% turbostat.IPC 18533186 ± 11% +88.4% 34918066 ± 13% turbostat.IRQ 8.57 ± 13% -8.6 0.01 ±223% turbostat.PKG_% 63.50 ± 2% -6.8% 59.17 ± 2% turbostat.PkgTmp 396.02 -12.2% 347.63 turbostat.PkgWatt 65.65 -3.0% 63.68 turbostat.RAMWatt 106442 ± 27% +2425.0% 2687648 ± 4% meminfo.Active 91979 ± 4% +2821.9% 2687504 ± 4% meminfo.Active(anon) 584098 -14.9% 496908 meminfo.AnonPages 9314973 ± 11% -35.1% 6042275 ± 4% meminfo.Cached 10265181 ± 10% -33.0% 6878896 ± 4% meminfo.Committed_AS 7061821 ± 14% -84.3% 1105432 ± 15% meminfo.Inactive 7055660 ± 14% -84.3% 1105092 ± 15% meminfo.Inactive(anon) 1983205 ± 4% -64.8% 697466 ± 17% meminfo.Mapped 11236856 ± 9% -31.2% 7727784 ± 3% meminfo.Memused 28127 ± 2% -10.0% 25308 ± 4% meminfo.PageTables 6562876 ± 15% -49.8% 3295910 ± 7% meminfo.Shmem 11308526 ± 9% -29.8% 7937794 ± 3% meminfo.max_used_kB 22992 ± 4% +2823.8% 672263 ± 4% proc-vmstat.nr_active_anon 146027 -14.9% 124341 proc-vmstat.nr_anon_pages 6276102 +1.4% 6363190 proc-vmstat.nr_dirty_background_threshold 12567550 +1.4% 12741940 proc-vmstat.nr_dirty_threshold 2332556 ± 11% -35.2% 1510629 ± 4% proc-vmstat.nr_file_pages 63132681 +1.4% 64009876 proc-vmstat.nr_free_pages 1764143 ± 14% -84.4% 276059 ± 15% proc-vmstat.nr_inactive_anon 495967 ± 4% -65.0% 173781 ± 18% proc-vmstat.nr_mapped 7017 ± 2% -10.2% 6303 ± 4% proc-vmstat.nr_page_table_pages 1640948 ± 15% -49.8% 824035 ± 8% proc-vmstat.nr_shmem 27434 ± 2% -7.2% 25448 proc-vmstat.nr_slab_reclaimable 22992 ± 4% +2823.8% 672262 ± 4% proc-vmstat.nr_zone_active_anon 1764143 ± 14% -84.4% 276059 ± 15% proc-vmstat.nr_zone_inactive_anon 273202 ± 13% -65.8% 93554 ± 5% proc-vmstat.numa_hint_faults 212829 ± 17% -68.4% 67222 ± 14% proc-vmstat.numa_hint_faults_local 2834604 ± 14% -44.6% 1571597 ± 5% proc-vmstat.numa_hit 1442 -45.8% 781.17 ± 2% proc-vmstat.numa_huge_pte_updates 2694400 ± 13% -44.1% 1505222 ± 6% proc-vmstat.numa_local 1110690 ± 3% -46.9% 589549 ± 2% proc-vmstat.numa_pte_updates 45441 ± 3% +1228.5% 603669 ± 17% proc-vmstat.pgactivate 2843888 ± 12% -41.9% 1651423 ± 5% proc-vmstat.pgalloc_normal 778597 ± 6% -27.0% 568119 proc-vmstat.pgfault 379559 ± 3% +23.1% 467169 proc-vmstat.pgfree 1674425 +25.8% 2106274 ± 3% sched_debug.cfs_rq:/.avg_vruntime.avg 2070968 ± 2% +82.1% 3771832 ± 9% sched_debug.cfs_rq:/.avg_vruntime.max 1495530 -10.0% 1346187 ± 2% sched_debug.cfs_rq:/.avg_vruntime.min 102696 ± 14% +444.2% 558889 ± 14% sched_debug.cfs_rq:/.avg_vruntime.stddev 1.24 ± 4% +77.5% 2.20 ± 3% sched_debug.cfs_rq:/.h_nr_running.avg 3.33 ± 16% +45.0% 4.83 ± 9% sched_debug.cfs_rq:/.h_nr_running.max 0.90 ± 7% +27.5% 1.15 ± 7% sched_debug.cfs_rq:/.h_nr_running.stddev 4315 ±223% +1037.3% 49080 ± 33% sched_debug.cfs_rq:/.left_vruntime.avg 276181 ±223% +880.0% 2706646 ± 14% sched_debug.cfs_rq:/.left_vruntime.max 34251 ±223% +941.9% 356863 ± 18% sched_debug.cfs_rq:/.left_vruntime.stddev 1674426 +25.8% 2106275 ± 3% sched_debug.cfs_rq:/.min_vruntime.avg 2070968 ± 2% +82.1% 3771832 ± 9% sched_debug.cfs_rq:/.min_vruntime.max 1495530 -10.0% 1346187 ± 2% sched_debug.cfs_rq:/.min_vruntime.min 102696 ± 14% +444.2% 558890 ± 14% sched_debug.cfs_rq:/.min_vruntime.stddev 4315 ±223% +1037.3% 49080 ± 33% sched_debug.cfs_rq:/.right_vruntime.avg 276181 ±223% +880.0% 2706646 ± 14% sched_debug.cfs_rq:/.right_vruntime.max 34251 ±223% +941.9% 356863 ± 18% sched_debug.cfs_rq:/.right_vruntime.stddev 1349 +77.8% 2399 sched_debug.cfs_rq:/.runnable_avg.avg 2553 ± 12% +94.9% 4978 ± 8% sched_debug.cfs_rq:/.runnable_avg.max 334.47 ± 14% +218.1% 1064 ± 5% sched_debug.cfs_rq:/.runnable_avg.stddev 365.42 ± 31% -69.2% 112.50 ± 55% sched_debug.cfs_rq:/.util_avg.min 199.71 ± 10% +46.0% 291.64 ± 13% sched_debug.cfs_rq:/.util_avg.stddev 4.83 ± 27% -52.1% 2.32 ± 27% sched_debug.cpu.clock.stddev 862.45 ± 10% +36.4% 1176 ± 18% sched_debug.cpu.curr->pid.stddev 1.24 ± 6% +77.6% 2.20 ± 3% sched_debug.cpu.nr_running.avg 3.33 ± 16% +45.0% 4.83 ± 9% sched_debug.cpu.nr_running.max 0.90 ± 5% +29.5% 1.16 ± 8% sched_debug.cpu.nr_running.stddev 5401202 -86.3% 737648 ± 5% sched_debug.cpu.nr_switches.avg 5904940 -55.4% 2630889 ± 8% sched_debug.cpu.nr_switches.max 4207188 ± 5% -99.2% 35131 ± 3% sched_debug.cpu.nr_switches.min 306253 ± 9% +144.4% 748534 ± 3% sched_debug.cpu.nr_switches.stddev 5.90 ± 11% +22.7% 7.24 ± 12% sched_debug.cpu.nr_uninterruptible.stddev 0.32 +88.3% 0.61 ± 11% perf-stat.i.MPKI 1.772e+10 -63.1% 6.533e+09 ± 2% perf-stat.i.branch-instructions 0.86 ± 3% -0.3 0.55 perf-stat.i.branch-miss-rate% 1.374e+08 ± 2% -76.5% 32249763 ± 3% perf-stat.i.branch-misses 25.69 ± 5% -13.0 12.66 ± 5% perf-stat.i.cache-miss-rate% 29425206 -30.4% 20491507 ± 10% perf-stat.i.cache-misses 1.105e+08 ± 5% +46.2% 1.615e+08 ± 12% perf-stat.i.cache-references 11696956 -86.5% 1584573 ± 5% perf-stat.i.context-switches 2.49 +165.9% 6.61 ± 2% perf-stat.i.cpi 2682 ± 15% +3544.5% 97753 ± 73% perf-stat.i.cpu-migrations 7982 ± 2% +47.9% 11805 ± 10% perf-stat.i.cycles-between-cache-misses 0.24 -0.2 0.03 ± 19% perf-stat.i.dTLB-load-miss-rate% 58578477 -94.9% 2999773 ± 16% perf-stat.i.dTLB-load-misses 2.453e+10 -63.4% 8.967e+09 ± 3% perf-stat.i.dTLB-loads 0.11 ± 3% -0.1 0.01 ± 22% perf-stat.i.dTLB-store-miss-rate% 16110645 ± 3% -96.2% 618460 ± 20% perf-stat.i.dTLB-store-misses 1.441e+10 -65.4% 4.982e+09 ± 3% perf-stat.i.dTLB-stores 8.984e+10 -63.2% 3.304e+10 ± 3% perf-stat.i.instructions 0.43 ± 2% -58.2% 0.18 ± 4% perf-stat.i.ipc 771.99 ± 2% -67.7% 249.22 ± 8% perf-stat.i.metric.K/sec 886.72 -63.7% 322.11 ± 3% perf-stat.i.metric.M/sec 10557 ± 5% -28.9% 7503 ± 2% perf-stat.i.minor-faults 68.85 ± 6% +16.5 85.38 ± 2% perf-stat.i.node-load-miss-rate% 2492992 ± 16% -72.9% 675588 ± 13% perf-stat.i.node-loads 84.88 -5.9 79.01 ± 2% perf-stat.i.node-store-miss-rate% 10676104 ± 3% -50.4% 5294348 ± 5% perf-stat.i.node-store-misses 10558 ± 5% -28.9% 7504 ± 2% perf-stat.i.page-faults 0.33 +89.5% 0.62 ± 11% perf-stat.overall.MPKI 0.78 -0.3 0.49 perf-stat.overall.branch-miss-rate% 26.66 ± 5% -13.9 12.72 ± 5% perf-stat.overall.cache-miss-rate% 2.42 +170.5% 6.55 ± 2% perf-stat.overall.cpi 7386 +44.4% 10667 ± 10% perf-stat.overall.cycles-between-cache-misses 0.24 -0.2 0.03 ± 18% perf-stat.overall.dTLB-load-miss-rate% 0.11 ± 3% -0.1 0.01 ± 23% perf-stat.overall.dTLB-store-miss-rate% 0.41 -63.0% 0.15 ± 2% perf-stat.overall.ipc 65.40 ± 8% +21.0 86.35 ± 3% perf-stat.overall.node-load-miss-rate% 82.36 -10.7 71.70 ± 2% perf-stat.overall.node-store-miss-rate% 1.742e+10 -63.1% 6.434e+09 ± 2% perf-stat.ps.branch-instructions 1.352e+08 ± 2% -76.7% 31552895 ± 3% perf-stat.ps.branch-misses 28958605 -30.3% 20174319 ± 10% perf-stat.ps.cache-misses 1.089e+08 ± 5% +46.3% 1.593e+08 ± 12% perf-stat.ps.cache-references 11523298 -86.4% 1563817 ± 5% perf-stat.ps.context-switches 2620 ± 15% +3589.0% 96685 ± 74% perf-stat.ps.cpu-migrations 57759305 -94.9% 2957134 ± 16% perf-stat.ps.dTLB-load-misses 2.413e+10 -63.4% 8.835e+09 ± 3% perf-stat.ps.dTLB-loads 15880442 ± 3% -96.2% 609839 ± 20% perf-stat.ps.dTLB-store-misses 1.417e+10 -65.4% 4.911e+09 ± 3% perf-stat.ps.dTLB-stores 8.835e+10 -63.2% 3.254e+10 ± 3% perf-stat.ps.instructions 10344 ± 6% -29.6% 7280 ± 2% perf-stat.ps.minor-faults 2484298 ± 16% -73.1% 667951 ± 13% perf-stat.ps.node-loads 10513724 ± 3% -50.3% 5223127 ± 5% perf-stat.ps.node-store-misses 10345 ± 6% -29.6% 7281 ± 2% perf-stat.ps.page-faults 5.612e+12 -63.4% 2.055e+12 ± 3% perf-stat.total.instructions 0.00 ±223% +70400.0% 0.47 ± 39% perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.00 ±223% +79653.3% 1.99 ± 47% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.00 ±223% +6.6e+05% 2.20 ± 69% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.01 ±223% +15548.9% 1.23 ± 20% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ±223% +1.5e+05% 1.46 ± 17% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.31 ±223% +732.4% 2.55 ± 37% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select 0.00 ±223% +61558.3% 1.23 ± 56% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.03 ±223% +2718.5% 0.94 ± 19% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.00 ±223% +86033.3% 0.86 ± 25% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.01 ±223% +1805.1% 0.19 ± 10% perf-sched.sch_delay.avg.ms.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.03 ±223% +2747.4% 0.72 ± 36% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.01 ±223% +1.9e+05% 11.24 ±148% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.00 ±223% +3.1e+05% 4.18 ± 62% perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.00 ±223% +5.8e+05% 11.58 ± 42% perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.01 ±223% +49755.4% 4.65 ± 60% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.00 ±223% +3.3e+05% 4.89 ± 93% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.08 ±223% +21410.1% 17.39 ± 34% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ±223% +1.8e+05% 3.34 ± 14% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.00 ±223% +1.8e+05% 6.39 ± 63% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.31 ±223% +1225.7% 4.08 ± 9% perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.00 ±223% +2.1e+05% 2.84 ± 24% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.15 ±223% +2375.4% 3.82 ± 9% perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.49 ±223% +1.4e+05% 664.31 ±166% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.01 ±223% +3013.9% 0.19 ± 10% perf-sched.total_sch_delay.average.ms 0.03 ±223% +1560.1% 0.58 ± 10% perf-sched.total_wait_and_delay.average.ms 0.03 ±223% +1255.2% 0.39 ± 10% perf-sched.total_wait_time.average.ms 83.33 ±223% +864.1% 803.38 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 10.12 ±223% +2201.2% 232.95 ± 4% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 1.07 ±223% +802.7% 9.65 ± 26% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.01 ±223% +18894.3% 1.11 ± 7% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 41.00 ±223% +954.7% 432.37 ± 8% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.31 ±223% +782.3% 2.70 ± 33% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select 0.04 ±223% +869.8% 0.43 ± 10% perf-sched.wait_and_delay.avg.ms.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe 38.33 ±223% +735.7% 320.33 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.67 ±223% +1400.0% 10.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 5.50 ±223% +2193.9% 126.17 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 4.83 ±223% +1227.6% 64.17 ± 11% perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 84.50 ±223% +6033.5% 5182 ± 10% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 23.17 ±223% +624.5% 167.83 ± 8% perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64 2.83 ±223% +1641.2% 49.33 ± 17% perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 1.00 ±223% +1800.0% 19.00 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.50 ±223% +600.0% 3.50 ± 14% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select 1.00 ±223% +1083.3% 11.83 ± 9% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 4.67 ±223% +1775.0% 87.50 perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.67 ±223% +2850.0% 19.67 ± 2% perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork 16.00 ±223% +2991.7% 494.67 ± 17% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 52.67 ±223% +958.5% 557.50 ± 5% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 17.50 ±223% +1452.4% 271.67 ± 4% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 166.71 ±223% +806.2% 1510 ± 50% perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.00 ±223% +16555.6% 0.50 ±159% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 0.01 ±223% +1578.1% 0.20 ± 70% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.00 ±223% +7083.3% 0.07 ±128% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_write.do_syscall_64 0.00 ±223% +2850.0% 0.03 ± 26% perf-sched.wait_time.avg.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 0.00 ±223% +9166.7% 0.37 ± 94% perf-sched.wait_time.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 0.00 ±223% +6157.1% 0.07 ±136% perf-sched.wait_time.avg.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 83.33 ±223% +861.8% 801.38 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.00 ±223% +3.6e+07% 652.15 ± 5% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 10.12 ±223% +2190.9% 231.73 ± 4% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 1.00 ±223% +551.9% 6.52 ± 4% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.12 ±223% +6688.1% 8.34 ± 24% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.01 ±223% +19450.0% 1.11 ± 7% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 31.33 ±223% +1278.0% 431.77 ± 8% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.03 ±223% +605.3% 0.25 ± 10% perf-sched.wait_time.avg.ms.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 ±223% +81633.3% 0.82 ±176% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_write.do_syscall_64 0.00 ±223% +10075.0% 0.20 ± 49% perf-sched.wait_time.max.ms.__cond_resched.shmem_get_folio_gfp.shmem_write_begin.generic_perform_write.shmem_file_write_iter 0.00 ±223% +23508.3% 0.94 ±144% perf-sched.wait_time.max.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 ±223% +3.8e+07% 1003 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 166.71 ±223% +804.7% 1508 ± 50% perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 515.98 ±223% +628.1% 3756 ± 13% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 26.90 -20.0 6.93 ± 12% perf-profile.calltrace.cycles-pp.__sigsuspend 24.82 -18.2 6.58 ± 12% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__sigsuspend 24.72 -18.2 6.56 ± 12% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 18.74 -15.4 3.32 ± 9% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 13.10 -9.8 3.31 ± 9% perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 11.05 -8.4 2.60 ± 9% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 10.79 -8.4 2.38 ± 10% perf-profile.calltrace.cycles-pp.__send_signal_locked.do_send_sig_info.kill_pid_info.kill_something_info.__x64_sys_kill 10.95 -8.4 2.58 ± 9% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 11.57 -8.0 3.52 ± 5% perf-profile.calltrace.cycles-pp.do_send_sig_info.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64 10.32 -7.1 3.26 ± 14% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 9.86 -6.7 3.18 ± 14% perf-profile.calltrace.cycles-pp.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 6.64 ± 2% -6.6 0.00 perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 6.56 ± 2% -6.6 0.00 perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 9.66 -6.5 3.15 ± 14% perf-profile.calltrace.cycles-pp.schedule.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.46 -6.4 3.10 ± 14% perf-profile.calltrace.cycles-pp.__schedule.schedule.sigsuspend.__x64_sys_rt_sigsuspend.do_syscall_64 7.22 -5.5 1.73 ± 12% perf-profile.calltrace.cycles-pp.complete_signal.__send_signal_locked.do_send_sig_info.kill_pid_info.kill_something_info 5.37 -5.4 0.00 perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 5.21 ± 2% -5.2 0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 6.50 -5.0 1.55 ± 12% perf-profile.calltrace.cycles-pp.try_to_wake_up.complete_signal.__send_signal_locked.do_send_sig_info.kill_pid_info 6.61 -4.9 1.67 ± 10% perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 5.19 -4.3 0.87 ± 7% perf-profile.calltrace.cycles-pp.handle_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 4.88 -3.9 1.02 ± 11% perf-profile.calltrace.cycles-pp.stress_sighandler_nop 4.88 -3.8 1.09 ± 10% perf-profile.calltrace.cycles-pp.dequeue_signal.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare 3.98 -3.1 0.85 ± 7% perf-profile.calltrace.cycles-pp.x64_setup_rt_frame.handle_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare 3.52 -2.8 0.76 ± 6% perf-profile.calltrace.cycles-pp.get_sigframe.x64_setup_rt_frame.handle_signal.arch_do_signal_or_restart.exit_to_user_mode_loop 2.97 -2.4 0.59 ± 46% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 2.46 -2.1 0.38 ± 71% perf-profile.calltrace.cycles-pp.fpu__restore_sig.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.54 -2.0 0.50 ± 46% perf-profile.calltrace.cycles-pp.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.__sigsuspend 2.41 -2.0 0.38 ± 71% perf-profile.calltrace.cycles-pp.__fpu_restore_sig.fpu__restore_sig.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64 2.47 -2.0 0.46 ± 44% perf-profile.calltrace.cycles-pp.copy_fpstate_to_sigframe.get_sigframe.x64_setup_rt_frame.handle_signal.arch_do_signal_or_restart 2.53 -1.8 0.71 ± 9% perf-profile.calltrace.cycles-pp.dequeue_task_fair.__schedule.schedule.sigsuspend.__x64_sys_rt_sigsuspend 2.54 -1.8 0.78 ± 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.stress_sighandler_nop 2.39 -1.6 0.74 ± 9% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 2.36 -1.6 0.76 ± 10% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_sighandler_nop 2.36 -1.6 0.76 ± 10% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_sighandler_nop 2.33 -1.6 0.74 ± 8% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.34 -1.6 0.75 ± 10% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_sighandler_nop 2.23 -1.6 0.67 ± 10% perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.40 -1.1 0.26 ±100% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_task_fair.__schedule.schedule.sigsuspend 1.43 ± 3% -1.0 0.44 ± 72% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.sigsuspend.__x64_sys_rt_sigsuspend 1.19 -0.9 0.26 ±100% perf-profile.calltrace.cycles-pp.set_current_blocked.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.54 ± 2% +0.1 0.60 ± 3% perf-profile.calltrace.cycles-pp.profile_signal_perm.aa_may_signal.apparmor_task_kill.security_task_kill.kill_pid_info 0.10 ±223% +0.7 0.79 ± 6% perf-profile.calltrace.cycles-pp.stress_time_now 0.00 +0.8 0.80 ± 11% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.do_send_sig_info.kill_pid_info.kill_something_info.__x64_sys_kill 0.00 +1.5 1.51 ± 35% perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 1.62 ± 8% +1.8 3.46 ± 8% perf-profile.calltrace.cycles-pp.stress_atomic_lock_acquire 0.20 ±142% +3.1 3.28 ± 33% perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.20 ±142% +3.2 3.41 ± 33% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 0.21 ±142% +3.2 3.42 ± 33% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 0.22 ±142% +3.4 3.62 ± 33% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 0.46 ± 74% +4.1 4.61 ± 31% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 0.46 ± 74% +4.2 4.62 ± 31% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 0.46 ± 74% +4.2 4.62 ± 31% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 0.46 ± 74% +4.2 4.68 ± 32% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 0.09 ±223% +4.2 4.34 ± 35% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter 15.50 +9.7 25.24 ± 2% perf-profile.calltrace.cycles-pp.aa_get_task_label.apparmor_task_kill.security_task_kill.kill_pid_info.kill_something_info 3.27 +10.4 13.68 ± 5% perf-profile.calltrace.cycles-pp.aa_may_signal.apparmor_task_kill.security_task_kill.kill_pid_info.kill_something_info 59.43 +21.0 80.45 perf-profile.calltrace.cycles-pp.kill 57.84 +22.1 79.92 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.kill 57.49 +22.3 79.76 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 50.66 +28.8 79.47 perf-profile.calltrace.cycles-pp.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 49.95 +29.0 78.92 perf-profile.calltrace.cycles-pp.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe.kill 49.38 +29.1 78.46 perf-profile.calltrace.cycles-pp.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64.entry_SYSCALL_64_after_hwframe 37.42 +37.1 74.56 perf-profile.calltrace.cycles-pp.security_task_kill.kill_pid_info.kill_something_info.__x64_sys_kill.do_syscall_64 37.19 +37.3 74.46 perf-profile.calltrace.cycles-pp.apparmor_task_kill.security_task_kill.kill_pid_info.kill_something_info.__x64_sys_kill 27.08 -20.1 6.96 ± 11% perf-profile.children.cycles-pp.__sigsuspend 20.11 -16.6 3.55 ± 8% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 19.95 -16.5 3.44 ± 9% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 18.82 -15.5 3.36 ± 9% perf-profile.children.cycles-pp.exit_to_user_mode_loop 15.14 -12.0 3.18 ± 14% perf-profile.children.cycles-pp.schedule 14.94 -11.1 3.79 ± 8% perf-profile.children.cycles-pp.__schedule 13.12 -9.8 3.31 ± 9% perf-profile.children.cycles-pp.arch_do_signal_or_restart 10.83 -8.4 2.41 ± 9% perf-profile.children.cycles-pp.__send_signal_locked 11.62 -8.0 3.57 ± 5% perf-profile.children.cycles-pp.do_send_sig_info 10.53 -7.3 3.27 ± 14% perf-profile.children.cycles-pp.__x64_sys_rt_sigsuspend 9.88 -6.7 3.19 ± 14% perf-profile.children.cycles-pp.sigsuspend 7.24 -5.5 1.74 ± 11% perf-profile.children.cycles-pp.complete_signal 6.55 -5.0 1.55 ± 12% perf-profile.children.cycles-pp.try_to_wake_up 6.64 -5.0 1.68 ± 10% perf-profile.children.cycles-pp.get_signal 5.20 -4.1 1.13 ± 7% perf-profile.children.cycles-pp.handle_signal 5.24 -3.9 1.34 ± 12% perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn 4.90 -3.8 1.10 ± 10% perf-profile.children.cycles-pp.dequeue_signal 3.84 -3.3 0.58 ± 15% perf-profile.children.cycles-pp.switch_mm_irqs_off 4.23 -3.2 1.06 ± 11% perf-profile.children.cycles-pp.stress_sighandler_nop 4.02 -3.2 0.86 ± 7% perf-profile.children.cycles-pp.x64_setup_rt_frame 3.22 -2.9 0.32 ± 9% perf-profile.children.cycles-pp.__sigqueue_alloc 3.30 -2.8 0.46 ± 10% perf-profile.children.cycles-pp.collect_signal 3.54 -2.8 0.76 ± 6% perf-profile.children.cycles-pp.get_sigframe 3.48 -2.7 0.82 ± 11% perf-profile.children.cycles-pp.ttwu_do_activate 2.96 -2.6 0.34 ± 11% perf-profile.children.cycles-pp.do_dec_rlimit_put_ucounts 3.18 -2.5 0.68 ± 17% perf-profile.children.cycles-pp.restore_sigcontext 3.00 -2.3 0.68 ± 20% perf-profile.children.cycles-pp.pick_next_task_fair 2.46 ± 2% -2.3 0.15 ± 15% perf-profile.children.cycles-pp.inc_rlimit_get_ucounts 2.66 -2.0 0.69 ± 9% perf-profile.children.cycles-pp.activate_task 2.51 -2.0 0.55 ± 7% perf-profile.children.cycles-pp.copy_fpstate_to_sigframe 2.54 -1.9 0.60 ± 11% perf-profile.children.cycles-pp.enqueue_task_fair 2.47 -1.9 0.53 ± 15% perf-profile.children.cycles-pp.fpu__restore_sig 2.43 -1.9 0.52 ± 15% perf-profile.children.cycles-pp.__fpu_restore_sig 2.56 -1.8 0.72 ± 9% perf-profile.children.cycles-pp.dequeue_task_fair 2.01 -1.6 0.46 ± 13% perf-profile.children.cycles-pp.update_load_avg 1.80 -1.5 0.34 ± 20% perf-profile.children.cycles-pp.update_curr 2.00 -1.4 0.64 ± 8% perf-profile.children.cycles-pp.recalc_sigpending 1.65 -1.3 0.38 ± 15% perf-profile.children.cycles-pp.restore_fpregs_from_user 1.29 -1.1 0.15 ± 10% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate 1.27 ± 2% -1.1 0.18 ± 17% perf-profile.children.cycles-pp._copy_from_user 1.33 ± 2% -1.1 0.28 ± 7% perf-profile.children.cycles-pp.stress_sigsuspend 1.44 ± 2% -1.0 0.40 ± 10% perf-profile.children.cycles-pp.enqueue_entity 1.34 ± 7% -1.0 0.30 ± 60% perf-profile.children.cycles-pp.reader__read_event 1.34 ± 7% -1.0 0.32 ± 57% perf-profile.children.cycles-pp.record__finish_output 1.34 ± 7% -1.0 0.32 ± 57% perf-profile.children.cycles-pp.perf_session__process_events 1.36 ± 7% -1.0 0.36 ± 56% perf-profile.children.cycles-pp.__cmd_record 1.63 -1.0 0.62 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irq 1.45 -0.9 0.50 ± 8% perf-profile.children.cycles-pp.dequeue_entity 1.25 -0.9 0.32 ± 8% perf-profile.children.cycles-pp.__switch_to 1.14 ± 3% -0.9 0.29 ± 23% perf-profile.children.cycles-pp.set_next_entity 1.06 -0.8 0.26 ± 6% perf-profile.children.cycles-pp.native_irq_return_iret 1.05 -0.8 0.26 ± 11% perf-profile.children.cycles-pp.fpu__clear_user_states 1.10 ± 2% -0.8 0.33 ± 5% perf-profile.children.cycles-pp.prepare_task_switch 0.86 ± 4% -0.8 0.10 ± 55% perf-profile.children.cycles-pp.process_simple 0.84 ± 4% -0.8 0.09 ± 56% perf-profile.children.cycles-pp.queue_event 0.84 ± 4% -0.7 0.09 ± 58% perf-profile.children.cycles-pp.ordered_events__queue 1.21 -0.7 0.48 ± 8% perf-profile.children.cycles-pp.set_current_blocked 1.03 ± 3% -0.7 0.36 ± 25% perf-profile.children.cycles-pp.select_task_rq 1.44 -0.7 0.78 ± 11% perf-profile.children.cycles-pp._raw_spin_lock 0.74 -0.7 0.08 ± 48% perf-profile.children.cycles-pp.pick_eevdf 1.08 -0.7 0.42 ± 10% perf-profile.children.cycles-pp.__set_current_blocked 1.13 -0.7 0.48 ± 10% perf-profile.children.cycles-pp.signal_setup_done 0.78 ± 3% -0.6 0.14 ± 25% perf-profile.children.cycles-pp.reweight_entity 0.72 -0.6 0.08 ± 14% perf-profile.children.cycles-pp.os_xsave 0.74 -0.6 0.10 ± 7% perf-profile.children.cycles-pp.__switch_to_asm 0.76 -0.6 0.13 ± 24% perf-profile.children.cycles-pp.check_preempt_curr 0.74 ± 2% -0.6 0.13 ± 10% perf-profile.children.cycles-pp.__update_load_avg_se 0.77 -0.6 0.18 ± 7% perf-profile.children.cycles-pp.save_xstate_epilog 0.92 ± 3% -0.6 0.33 ± 27% perf-profile.children.cycles-pp.select_task_rq_fair 0.88 -0.5 0.33 ± 12% perf-profile.children.cycles-pp.fpregs_mark_activate 0.71 -0.5 0.16 ± 8% perf-profile.children.cycles-pp.kmem_cache_alloc 0.79 ± 3% -0.5 0.25 ± 7% perf-profile.children.cycles-pp.___perf_sw_event 0.56 -0.5 0.03 ±100% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template 0.64 ± 2% -0.5 0.14 ± 13% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.68 -0.5 0.19 ± 10% perf-profile.children.cycles-pp.kmem_cache_free 0.57 ± 6% -0.4 0.16 ± 47% perf-profile.children.cycles-pp.select_idle_sibling 0.46 -0.4 0.08 ± 20% perf-profile.children.cycles-pp.check_xstate_in_sigframe 0.55 ± 6% -0.3 0.20 ± 33% perf-profile.children.cycles-pp.__dequeue_entity 0.38 ± 2% -0.3 0.06 ± 14% perf-profile.children.cycles-pp.__get_user_nocheck_4 0.40 ± 2% -0.3 0.09 ± 7% perf-profile.children.cycles-pp.update_rq_clock 0.41 -0.3 0.10 ± 13% perf-profile.children.cycles-pp.restore_altstack 0.37 -0.3 0.06 ± 11% perf-profile.children.cycles-pp.__get_user_nocheck_8 0.56 ± 3% -0.3 0.26 ± 6% perf-profile.children.cycles-pp.clock_gettime 0.32 ± 4% -0.3 0.04 ± 71% perf-profile.children.cycles-pp.perf_trace_sched_switch 0.34 ± 4% -0.3 0.06 ± 13% perf-profile.children.cycles-pp.update_cfs_group 0.33 ± 5% -0.3 0.05 ± 47% perf-profile.children.cycles-pp.__cgroup_account_cputime 0.29 -0.3 0.03 ±100% perf-profile.children.cycles-pp.__put_user_nocheck_4 0.31 ± 6% -0.3 0.06 ± 46% perf-profile.children.cycles-pp.update_min_vruntime 0.42 ± 14% -0.2 0.17 ± 67% perf-profile.children.cycles-pp.perf_session__process_user_event 0.28 -0.2 0.04 ± 71% perf-profile.children.cycles-pp.place_entity 0.28 -0.2 0.05 ± 46% perf-profile.children.cycles-pp.__list_del_entry_valid_or_report 0.26 ± 5% -0.2 0.03 ± 70% perf-profile.children.cycles-pp.__enqueue_entity 0.42 ± 14% -0.2 0.19 ± 61% perf-profile.children.cycles-pp.__ordered_events__flush 0.32 ± 2% -0.2 0.10 ± 6% perf-profile.children.cycles-pp.sched_clock_cpu 0.62 ± 2% -0.2 0.40 perf-profile.children.cycles-pp.__entry_text_start 0.28 ± 2% -0.2 0.09 ± 5% perf-profile.children.cycles-pp.sched_clock 0.38 ± 6% -0.2 0.20 ± 19% perf-profile.children.cycles-pp.stress_lock_acquire 0.26 -0.2 0.10 ± 44% perf-profile.children.cycles-pp.resched_curr 0.26 -0.2 0.09 ± 7% perf-profile.children.cycles-pp.native_sched_clock 0.35 ± 15% -0.2 0.18 ± 62% perf-profile.children.cycles-pp.perf_session__deliver_event 0.23 -0.1 0.08 ± 11% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook 0.25 ± 5% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.from_kuid_munged 0.20 ± 5% -0.1 0.07 ± 14% perf-profile.children.cycles-pp.cpuacct_charge 0.18 ± 13% -0.1 0.04 ± 73% perf-profile.children.cycles-pp.evlist__parse_sample 0.22 ± 10% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.clock_gettime@plt 0.41 -0.1 0.29 ± 2% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.22 ± 2% -0.1 0.11 perf-profile.children.cycles-pp.syscall_return_via_sysret 0.13 ± 2% -0.1 0.02 ± 99% perf-profile.children.cycles-pp.refill_obj_stock 0.15 ± 7% -0.1 0.05 ± 73% perf-profile.children.cycles-pp.__list_add_valid_or_report 0.14 ± 7% -0.1 0.04 ± 73% perf-profile.children.cycles-pp.rb_next 0.12 ± 15% -0.1 0.05 ± 46% perf-profile.children.cycles-pp.perf_exclude_event 0.14 ± 3% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.audit_signal_info_syscall 0.09 ± 9% -0.1 0.02 ± 99% perf-profile.children.cycles-pp.perf_swevent_event 0.36 ± 4% -0.1 0.30 ± 6% perf-profile.children.cycles-pp.__radix_tree_lookup 0.17 ± 4% -0.1 0.11 ± 12% perf-profile.children.cycles-pp.update_rq_clock_task 0.15 ± 3% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.12 ± 3% -0.1 0.06 ± 7% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.10 ± 4% -0.1 0.05 ± 7% perf-profile.children.cycles-pp.find_vpid 0.12 ± 6% -0.0 0.08 ± 4% perf-profile.children.cycles-pp.map_id_up 0.09 ± 13% -0.0 0.05 ± 48% perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi 0.30 +0.0 0.33 ± 4% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.16 ± 3% +0.0 0.21 ± 4% perf-profile.children.cycles-pp.prepare_signal 0.55 ± 2% +0.1 0.62 ± 3% perf-profile.children.cycles-pp.profile_signal_perm 0.00 +0.1 0.08 ± 22% perf-profile.children.cycles-pp.menu_select 0.00 +0.1 0.08 ± 24% perf-profile.children.cycles-pp.task_h_load 0.12 ± 5% +0.1 0.20 ± 27% perf-profile.children.cycles-pp.ttwu_queue_wakelist 0.00 +0.1 0.08 ± 35% perf-profile.children.cycles-pp.irqentry_exit 0.00 +0.1 0.09 ± 10% perf-profile.children.cycles-pp.newidle_balance 0.00 +0.1 0.09 ± 38% perf-profile.children.cycles-pp.__smp_call_single_queue 0.00 +0.1 0.14 ± 18% perf-profile.children.cycles-pp.wake_affine 0.01 ±223% +0.2 0.21 ± 35% perf-profile.children.cycles-pp.ct_kernel_exit_state 0.04 ± 77% +0.2 0.25 ± 12% perf-profile.children.cycles-pp.__slab_free 0.60 ± 11% +0.3 0.86 ± 5% perf-profile.children.cycles-pp.stress_time_now 0.02 ±144% +0.3 0.34 ± 33% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 0.05 ± 75% +0.6 0.63 ± 41% perf-profile.children.cycles-pp.sched_ttwu_pending 0.06 ± 74% +0.6 0.66 ± 25% perf-profile.children.cycles-pp.schedule_idle 0.07 ± 53% +0.7 0.81 ± 38% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 0.07 ± 55% +0.7 0.82 ± 38% perf-profile.children.cycles-pp.__sysvec_call_function_single 0.09 ± 55% +0.9 0.96 ± 37% perf-profile.children.cycles-pp.sysvec_call_function_single 1.74 ± 9% +2.0 3.72 ± 8% perf-profile.children.cycles-pp.stress_atomic_lock_acquire 0.34 ± 46% +2.5 2.86 ± 35% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 0.38 ± 47% +2.9 3.32 ± 35% perf-profile.children.cycles-pp.acpi_safe_halt 0.38 ± 46% +2.9 3.33 ± 35% perf-profile.children.cycles-pp.acpi_idle_enter 0.40 ± 47% +3.1 3.46 ± 35% perf-profile.children.cycles-pp.cpuidle_enter_state 0.40 ± 47% +3.1 3.47 ± 35% perf-profile.children.cycles-pp.cpuidle_enter 0.43 ± 47% +3.2 3.66 ± 34% perf-profile.children.cycles-pp.cpuidle_idle_call 0.54 ± 47% +4.1 4.62 ± 31% perf-profile.children.cycles-pp.start_secondary 0.54 ± 47% +4.1 4.68 ± 32% perf-profile.children.cycles-pp.do_idle 0.54 ± 47% +4.1 4.68 ± 32% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 0.54 ± 47% +4.1 4.68 ± 32% perf-profile.children.cycles-pp.cpu_startup_entry 15.51 +9.7 25.25 ± 2% perf-profile.children.cycles-pp.aa_get_task_label 3.30 +10.4 13.71 ± 5% perf-profile.children.cycles-pp.aa_may_signal 59.72 +21.0 80.71 perf-profile.children.cycles-pp.kill 50.68 +28.8 79.50 perf-profile.children.cycles-pp.__x64_sys_kill 49.98 +29.0 78.95 perf-profile.children.cycles-pp.kill_something_info 49.41 +29.1 78.50 perf-profile.children.cycles-pp.kill_pid_info 37.43 +37.1 74.57 perf-profile.children.cycles-pp.security_task_kill 37.22 +37.3 74.50 perf-profile.children.cycles-pp.apparmor_task_kill 3.80 -3.2 0.58 ± 15% perf-profile.self.cycles-pp.switch_mm_irqs_off 2.95 -2.6 0.34 ± 11% perf-profile.self.cycles-pp.do_dec_rlimit_put_ucounts 2.46 ± 2% -2.3 0.14 ± 14% perf-profile.self.cycles-pp.inc_rlimit_get_ucounts 1.98 -1.4 0.63 ± 8% perf-profile.self.cycles-pp.recalc_sigpending 1.73 -1.2 0.54 ± 9% perf-profile.self.cycles-pp.__schedule 1.27 -1.1 0.14 ± 10% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate 1.40 -1.0 0.36 ± 8% perf-profile.self.cycles-pp.copy_fpstate_to_sigframe 1.61 -1.0 0.60 ± 9% perf-profile.self.cycles-pp._raw_spin_lock_irq 1.23 ± 2% -1.0 0.23 ± 5% perf-profile.self.cycles-pp.stress_sigsuspend 1.15 -1.0 0.19 ± 15% perf-profile.self.cycles-pp.restore_fpregs_from_user 1.39 ± 2% -0.9 0.46 ± 16% perf-profile.self.cycles-pp._raw_spin_lock 1.22 -0.9 0.31 ± 8% perf-profile.self.cycles-pp.__switch_to 1.07 -0.9 0.18 ± 16% perf-profile.self.cycles-pp._copy_from_user 1.06 -0.8 0.26 ± 6% perf-profile.self.cycles-pp.native_irq_return_iret 0.79 ± 4% -0.7 0.08 ± 59% perf-profile.self.cycles-pp.queue_event 0.72 -0.6 0.08 ± 15% perf-profile.self.cycles-pp.os_xsave 0.73 -0.6 0.10 ± 6% perf-profile.self.cycles-pp.__switch_to_asm 0.68 -0.6 0.11 ± 13% perf-profile.self.cycles-pp.fpu__clear_user_states 0.67 ± 2% -0.6 0.12 ± 11% perf-profile.self.cycles-pp.__update_load_avg_se 0.66 ± 2% -0.5 0.12 ± 19% perf-profile.self.cycles-pp.update_curr 0.68 -0.5 0.16 ± 18% perf-profile.self.cycles-pp.update_load_avg 0.83 -0.5 0.31 ± 13% perf-profile.self.cycles-pp.fpregs_mark_activate 0.63 ± 2% -0.5 0.13 ± 16% perf-profile.self.cycles-pp.complete_signal 0.77 -0.5 0.26 ± 10% perf-profile.self.cycles-pp.get_signal 0.64 -0.5 0.16 ± 3% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.60 ± 2% -0.5 0.14 ± 15% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.52 -0.5 0.06 ± 48% perf-profile.self.cycles-pp.pick_eevdf 0.50 ± 3% -0.4 0.07 ± 5% perf-profile.self.cycles-pp.__sigsuspend 0.58 -0.4 0.16 ± 14% perf-profile.self.cycles-pp.___perf_sw_event 0.48 -0.4 0.10 ± 21% perf-profile.self.cycles-pp.x64_setup_rt_frame 0.48 -0.3 0.15 ± 19% perf-profile.self.cycles-pp.pick_next_task_fair 0.41 ± 4% -0.3 0.10 ± 19% perf-profile.self.cycles-pp.reweight_entity 0.36 -0.3 0.06 ± 13% perf-profile.self.cycles-pp.__get_user_nocheck_4 0.35 ± 2% -0.3 0.06 ± 14% perf-profile.self.cycles-pp.__get_user_nocheck_8 0.32 -0.3 0.05 ± 47% perf-profile.self.cycles-pp.__fpu_restore_sig 0.31 ± 4% -0.3 0.04 ± 71% perf-profile.self.cycles-pp.perf_trace_sched_switch 0.32 ± 5% -0.3 0.06 ± 13% perf-profile.self.cycles-pp.update_cfs_group 0.27 -0.2 0.02 ± 99% perf-profile.self.cycles-pp.__put_user_nocheck_4 0.29 ± 6% -0.2 0.06 ± 47% perf-profile.self.cycles-pp.update_min_vruntime 0.35 ± 6% -0.2 0.12 ± 32% perf-profile.self.cycles-pp.__dequeue_entity 0.27 -0.2 0.05 ± 47% perf-profile.self.cycles-pp.check_xstate_in_sigframe 0.33 -0.2 0.11 ± 4% perf-profile.self.cycles-pp.do_syscall_64 0.25 ± 5% -0.2 0.03 ± 70% perf-profile.self.cycles-pp.__enqueue_entity 0.26 ± 2% -0.2 0.05 ± 46% perf-profile.self.cycles-pp.__list_del_entry_valid_or_report 0.30 ± 2% -0.2 0.09 ± 13% perf-profile.self.cycles-pp.prepare_task_switch 0.30 ± 2% -0.2 0.09 ± 20% perf-profile.self.cycles-pp.save_xstate_epilog 0.29 ± 4% -0.2 0.09 ± 9% perf-profile.self.cycles-pp.kmem_cache_free 0.25 -0.2 0.08 ± 8% perf-profile.self.cycles-pp.enqueue_task_fair 0.24 ± 2% -0.2 0.08 ± 8% perf-profile.self.cycles-pp.native_sched_clock 0.25 -0.2 0.08 ± 14% perf-profile.self.cycles-pp.enqueue_entity 0.38 ± 7% -0.2 0.21 ± 6% perf-profile.self.cycles-pp.clock_gettime 0.26 ± 2% -0.2 0.09 ± 45% perf-profile.self.cycles-pp.resched_curr 0.35 ± 5% -0.2 0.18 ± 20% perf-profile.self.cycles-pp.stress_lock_acquire 0.23 ± 2% -0.2 0.06 ± 47% perf-profile.self.cycles-pp.restore_sigcontext 0.18 ± 8% -0.1 0.04 ± 72% perf-profile.self.cycles-pp.set_next_entity 0.22 ± 2% -0.1 0.08 ± 7% perf-profile.self.cycles-pp.dequeue_task_fair 0.21 -0.1 0.07 ± 5% perf-profile.self.cycles-pp.security_task_kill 0.23 -0.1 0.09 ± 14% perf-profile.self.cycles-pp.dequeue_entity 0.19 ± 6% -0.1 0.06 ± 14% perf-profile.self.cycles-pp.cpuacct_charge 0.19 ± 4% -0.1 0.06 ± 7% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.40 ± 2% -0.1 0.28 ± 2% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.20 ± 2% -0.1 0.09 ± 16% perf-profile.self.cycles-pp.try_to_wake_up 0.22 ± 2% -0.1 0.11 perf-profile.self.cycles-pp.syscall_return_via_sysret 0.13 ± 2% -0.1 0.02 ± 99% perf-profile.self.cycles-pp.refill_obj_stock 0.23 ± 3% -0.1 0.13 ± 5% perf-profile.self.cycles-pp.__entry_text_start 0.12 ± 8% -0.1 0.04 ± 72% perf-profile.self.cycles-pp.rb_next 0.12 ± 10% -0.1 0.05 ± 75% perf-profile.self.cycles-pp.__list_add_valid_or_report 0.14 ± 3% -0.1 0.07 ± 15% perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook 0.10 ± 3% -0.1 0.03 ±100% perf-profile.self.cycles-pp.ttwu_queue_wakelist 0.36 ± 3% -0.1 0.28 ± 2% perf-profile.self.cycles-pp.kill 0.13 ± 5% -0.1 0.06 ± 9% perf-profile.self.cycles-pp.audit_signal_info_syscall 0.10 ± 4% -0.1 0.02 ± 99% perf-profile.self.cycles-pp.find_vpid 0.16 ± 3% -0.1 0.08 ± 13% perf-profile.self.cycles-pp.update_rq_clock_task 0.12 ± 13% -0.1 0.05 ± 46% perf-profile.self.cycles-pp.perf_exclude_event 0.20 ± 2% -0.1 0.14 perf-profile.self.cycles-pp.__send_signal_locked 0.35 ± 4% -0.1 0.29 ± 6% perf-profile.self.cycles-pp.__radix_tree_lookup 0.15 ± 2% -0.1 0.09 ± 19% perf-profile.self.cycles-pp.finish_task_switch 0.14 ± 3% -0.1 0.09 ± 20% perf-profile.self.cycles-pp.collect_signal 0.12 ± 3% -0.1 0.07 ± 5% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.13 ± 3% -0.1 0.08 ± 6% perf-profile.self.cycles-pp.syscall_enter_from_user_mode 0.09 ± 4% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.map_id_up 0.15 ± 2% +0.0 0.19 ± 4% perf-profile.self.cycles-pp.prepare_signal 0.08 ± 4% +0.1 0.14 ± 9% perf-profile.self.cycles-pp.check_kill_permission 0.53 ± 2% +0.1 0.60 ± 3% perf-profile.self.cycles-pp.profile_signal_perm 0.00 +0.1 0.08 ± 24% perf-profile.self.cycles-pp.task_h_load 0.46 ± 8% +0.1 0.57 ± 5% perf-profile.self.cycles-pp.stress_time_now 0.00 +0.1 0.13 ± 28% perf-profile.self.cycles-pp.__flush_smp_call_function_queue 0.00 +0.2 0.18 ± 32% perf-profile.self.cycles-pp.do_idle 0.01 ±223% +0.2 0.21 ± 35% perf-profile.self.cycles-pp.ct_kernel_exit_state 0.04 ± 77% +0.2 0.25 ± 12% perf-profile.self.cycles-pp.__slab_free 0.02 ±144% +0.3 0.34 ± 33% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 1.68 ± 9% +2.0 3.66 ± 8% perf-profile.self.cycles-pp.stress_atomic_lock_acquire 0.26 ± 48% +2.1 2.34 ± 31% perf-profile.self.cycles-pp.acpi_safe_halt 15.45 +9.7 25.17 ± 2% perf-profile.self.cycles-pp.aa_get_task_label 2.75 ± 2% +10.3 13.06 ± 5% perf-profile.self.cycles-pp.aa_may_signal 18.36 +17.1 35.44 perf-profile.self.cycles-pp.apparmor_task_kill *************************************************************************************************** lkp-icl-2sp8: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory ========================================================================================= class/compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: network/gcc-12/performance/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp8/sock/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 9220 ± 10% -25.4% 6878 ± 4% perf-c2c.DRAM.local 427.50 ± 15% +25.0% 534.50 ± 9% perf-c2c.DRAM.remote 8687407 ± 2% -60.8% 3401817 vmstat.memory.cache 9104079 -99.2% 76005 ± 3% vmstat.system.cs 1.42 +0.5 1.88 mpstat.cpu.all.irq% 21.66 -6.5 15.13 mpstat.cpu.all.soft% 66.59 +6.9 73.48 mpstat.cpu.all.sys% 4.74 -1.3 3.41 mpstat.cpu.all.usr% 36698401 +100.4% 73530845 numa-numastat.node0.local_node 36743603 +100.2% 73556209 numa-numastat.node0.numa_hit 39746823 +86.6% 74187371 numa-numastat.node1.local_node 39767761 +86.6% 74219438 numa-numastat.node1.numa_hit 3480 -4.1% 3337 turbostat.Bzy_MHz 0.27 ± 9% -23.9% 0.21 ± 7% turbostat.IPC 164.36 ± 2% +20.5 184.86 turbostat.PKG_% 111.21 +43.3% 159.31 turbostat.RAMWatt 179.40 ± 3% +10.4% 198.05 ± 6% sched_debug.cfs_rq:/.util_est_enqueued.stddev 6.93 ± 10% -44.8% 3.83 ± 3% sched_debug.cpu.clock.stddev 1010 ± 2% -11.1% 897.55 sched_debug.cpu.clock_task.stddev 4417021 -99.1% 40112 ± 2% sched_debug.cpu.nr_switches.avg 4633822 -98.0% 92783 ± 11% sched_debug.cpu.nr_switches.max 3155274 ± 5% -99.0% 31120 ± 7% sched_debug.cpu.nr_switches.min 257878 ± 8% -96.5% 8984 ± 22% sched_debug.cpu.nr_switches.stddev 1463496 +59.6% 2335111 stress-ng.sock.ops 24393 +59.4% 38886 stress-ng.sock.ops_per_sec 2.963e+08 -100.0% 23120 ± 24% stress-ng.time.involuntary_context_switches 4579 +9.4% 5011 stress-ng.time.percent_of_cpu_this_job_got 2704 +11.0% 3002 stress-ng.time.system_time 138.93 -18.5% 113.20 stress-ng.time.user_time 2.994e+08 -98.4% 4779638 stress-ng.time.voluntary_context_switches 13292 ± 22% +516.9% 81996 ± 49% numa-meminfo.node0.Active 13244 ± 22% +518.4% 81900 ± 49% numa-meminfo.node0.Active(anon) 127398 ± 11% -50.7% 62800 ± 29% numa-meminfo.node0.Mapped 19753 ± 17% +460.4% 110688 ± 50% numa-meminfo.node0.Shmem 95378 ± 4% +224.5% 309529 ± 11% numa-meminfo.node1.Active 95282 ± 4% +224.8% 309481 ± 11% numa-meminfo.node1.Active(anon) 7381585 ± 15% -67.7% 2382225 ± 39% numa-meminfo.node1.FilePages 5922737 ± 3% -93.6% 379656 ± 9% numa-meminfo.node1.Inactive 5922599 ± 3% -93.6% 379587 ± 9% numa-meminfo.node1.Inactive(anon) 1547876 ± 3% -92.1% 122727 ± 24% numa-meminfo.node1.Mapped 8342373 ± 13% -60.7% 3280359 ± 29% numa-meminfo.node1.MemUsed 5768899 ± 3% -92.5% 435300 ± 12% numa-meminfo.node1.Shmem 108530 ± 5% +260.3% 391084 ± 4% meminfo.Active 108386 ± 5% +260.7% 390940 ± 4% meminfo.Active(anon) 8531615 ± 2% -61.4% 3291104 meminfo.Cached 7676613 ± 2% -69.5% 2344739 meminfo.Committed_AS 6180671 ± 3% -90.5% 586262 ± 12% meminfo.Inactive 6180462 ± 3% -90.5% 586052 ± 12% meminfo.Inactive(anon) 113439 -10.2% 101831 meminfo.KReclaimable 1670440 ± 3% -89.0% 183788 ± 9% meminfo.Mapped 10473614 ± 2% -52.0% 5029882 meminfo.Memused 16325 -17.2% 13525 ± 2% meminfo.PageTables 113439 -10.2% 101831 meminfo.SReclaimable 5785468 ± 3% -90.6% 544897 meminfo.Shmem 10520180 ± 2% -50.7% 5182942 meminfo.max_used_kB 3322 ± 22% +515.8% 20463 ± 49% numa-vmstat.node0.nr_active_anon 31835 ± 11% -50.6% 15735 ± 28% numa-vmstat.node0.nr_mapped 4942 ± 17% +459.9% 27672 ± 50% numa-vmstat.node0.nr_shmem 3322 ± 22% +515.8% 20463 ± 49% numa-vmstat.node0.nr_zone_active_anon 36743650 +100.2% 73556294 numa-vmstat.node0.numa_hit 36698448 +100.4% 73530930 numa-vmstat.node0.numa_local 23867 ± 4% +224.0% 77337 ± 11% numa-vmstat.node1.nr_active_anon 1845745 ± 15% -67.7% 595578 ± 39% numa-vmstat.node1.nr_file_pages 1480927 ± 3% -93.6% 95003 ± 9% numa-vmstat.node1.nr_inactive_anon 387463 ± 3% -92.1% 30744 ± 24% numa-vmstat.node1.nr_mapped 1442574 ± 3% -92.5% 108847 ± 12% numa-vmstat.node1.nr_shmem 23867 ± 4% +224.0% 77337 ± 11% numa-vmstat.node1.nr_zone_active_anon 1480926 ± 3% -93.6% 95002 ± 9% numa-vmstat.node1.nr_zone_inactive_anon 39767712 +86.6% 74219234 numa-vmstat.node1.numa_hit 39746773 +86.6% 74187167 numa-vmstat.node1.numa_local 27114 ± 5% +261.7% 98078 ± 4% proc-vmstat.nr_active_anon 6294668 +2.2% 6430570 proc-vmstat.nr_dirty_background_threshold 12604727 +2.2% 12876865 proc-vmstat.nr_dirty_threshold 2132985 ± 2% -61.4% 823062 proc-vmstat.nr_file_pages 63323676 +2.1% 64684697 proc-vmstat.nr_free_pages 1545180 ± 3% -90.5% 146386 ± 12% proc-vmstat.nr_inactive_anon 417921 ± 3% -89.0% 45993 ± 10% proc-vmstat.nr_mapped 4079 -17.3% 3374 ± 2% proc-vmstat.nr_page_table_pages 1446446 ± 3% -90.6% 136509 proc-vmstat.nr_shmem 28359 -10.2% 25459 proc-vmstat.nr_slab_reclaimable 27114 ± 5% +261.7% 98078 ± 4% proc-vmstat.nr_zone_active_anon 1545180 ± 3% -90.5% 146386 ± 12% proc-vmstat.nr_zone_inactive_anon 191857 ± 8% -73.2% 51331 proc-vmstat.numa_hint_faults 141647 ± 11% -74.9% 35486 ± 6% proc-vmstat.numa_hint_faults_local 76513548 +93.1% 1.478e+08 proc-vmstat.numa_hit 76447407 +93.2% 1.477e+08 proc-vmstat.numa_local 56551 ± 8% +248.0% 196817 proc-vmstat.pgactivate 5.946e+08 +98.0% 1.178e+09 proc-vmstat.pgalloc_normal 653661 ± 2% -30.8% 452095 proc-vmstat.pgfault 5.924e+08 +98.7% 1.177e+09 proc-vmstat.pgfree 2.54 ± 2% +247.7% 8.83 perf-stat.i.MPKI 3.162e+10 -26.2% 2.333e+10 perf-stat.i.branch-instructions 1.01 -0.4 0.59 perf-stat.i.branch-miss-rate% 3.007e+08 -58.0% 1.264e+08 perf-stat.i.branch-misses 24.80 +12.5 37.26 perf-stat.i.cache-miss-rate% 4.154e+08 ± 2% +171.4% 1.127e+09 perf-stat.i.cache-misses 1.629e+09 +80.7% 2.943e+09 perf-stat.i.cache-references 9543474 -99.2% 79674 ± 3% perf-stat.i.context-switches 1.30 +24.7% 1.61 perf-stat.i.cpi 2.111e+11 -4.5% 2.015e+11 perf-stat.i.cpu-cycles 665.75 ± 2% -54.4% 303.29 ± 3% perf-stat.i.cycles-between-cache-misses 0.01 ± 3% -0.0 0.00 ± 5% perf-stat.i.dTLB-load-miss-rate% 5000927 ± 3% -81.4% 930683 ± 4% perf-stat.i.dTLB-load-misses 4.595e+10 -23.9% 3.498e+10 perf-stat.i.dTLB-loads 0.00 ± 2% +0.0 0.02 perf-stat.i.dTLB-store-miss-rate% 1050765 ± 2% +349.6% 4723978 perf-stat.i.dTLB-store-misses 2.687e+10 -24.6% 2.027e+10 perf-stat.i.dTLB-stores 1.642e+11 -24.7% 1.236e+11 perf-stat.i.instructions 0.78 -20.2% 0.62 perf-stat.i.ipc 3.30 -4.5% 3.15 perf-stat.i.metric.GHz 376.37 ± 2% -38.2% 232.62 ± 2% perf-stat.i.metric.K/sec 1661 -22.6% 1285 perf-stat.i.metric.M/sec 8755 ± 2% -36.2% 5589 perf-stat.i.minor-faults 5.44 ± 7% -1.6 3.84 ± 3% perf-stat.i.node-load-miss-rate% 3394831 ± 7% +56.4% 5310095 ± 3% perf-stat.i.node-load-misses 1.121e+08 ± 5% +175.7% 3.092e+08 perf-stat.i.node-loads 2.17 ± 14% -0.7 1.42 ± 11% perf-stat.i.node-store-miss-rate% 2041975 ± 10% +79.2% 3658647 ± 7% perf-stat.i.node-store-misses 1.731e+08 +173.3% 4.732e+08 perf-stat.i.node-stores 8756 ± 2% -36.2% 5589 perf-stat.i.page-faults 2.53 ± 2% +260.2% 9.13 perf-stat.overall.MPKI 0.95 -0.4 0.54 perf-stat.overall.branch-miss-rate% 25.51 +12.8 38.31 perf-stat.overall.cache-miss-rate% 1.29 +26.8% 1.63 perf-stat.overall.cpi 507.86 ± 2% -64.8% 178.74 perf-stat.overall.cycles-between-cache-misses 0.01 ± 3% -0.0 0.00 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 2% +0.0 0.02 perf-stat.overall.dTLB-store-miss-rate% 0.78 -21.2% 0.61 perf-stat.overall.ipc 2.95 ± 13% -1.3 1.69 ± 3% perf-stat.overall.node-load-miss-rate% 1.17 ± 10% -0.4 0.77 ± 7% perf-stat.overall.node-store-miss-rate% 3.11e+10 -26.1% 2.299e+10 perf-stat.ps.branch-instructions 2.959e+08 -58.0% 1.243e+08 perf-stat.ps.branch-misses 4.092e+08 ± 2% +171.7% 1.112e+09 perf-stat.ps.cache-misses 1.604e+09 +80.9% 2.902e+09 perf-stat.ps.cache-references 9393169 -99.2% 78457 ± 3% perf-stat.ps.context-switches 2.077e+11 -4.3% 1.987e+11 perf-stat.ps.cpu-cycles 4918971 ± 3% -81.6% 904508 ± 4% perf-stat.ps.dTLB-load-misses 4.521e+10 -23.7% 3.448e+10 perf-stat.ps.dTLB-loads 1036059 ± 2% +349.2% 4653866 perf-stat.ps.dTLB-store-misses 2.644e+10 -24.4% 1.998e+10 perf-stat.ps.dTLB-stores 1.615e+11 -24.6% 1.218e+11 perf-stat.ps.instructions 8588 -36.7% 5440 perf-stat.ps.minor-faults 3340433 ± 7% +56.8% 5237317 ± 3% perf-stat.ps.node-load-misses 1.105e+08 ± 5% +176.1% 3.051e+08 perf-stat.ps.node-loads 2010110 ± 10% +79.6% 3611038 ± 7% perf-stat.ps.node-store-misses 1.705e+08 +173.6% 4.664e+08 perf-stat.ps.node-stores 8588 -36.7% 5440 perf-stat.ps.page-faults 1.014e+13 -24.0% 7.704e+12 perf-stat.total.instructions 0.10 ±100% +720.2% 0.78 ± 8% perf-sched.sch_delay.avg.ms.__cond_resched.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect 0.00 ± 99% +28933.3% 0.15 ±194% perf-sched.sch_delay.avg.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg 0.01 ±188% +4331.1% 0.55 ± 3% perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.13 ±154% +661.6% 0.95 ± 14% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.05 ±134% +965.6% 0.56 ± 9% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.04 ±133% +363.9% 0.18 ± 69% perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.21 ±217% +343.4% 0.92 ± 61% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ±108% +174.2% 0.03 perf-sched.sch_delay.avg.ms.schedule_timeout.inet_csk_accept.inet_accept.do_accept 0.00 ±107% +9703.6% 0.46 ± 6% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.01 ±100% +5433.0% 0.81 perf-sched.sch_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 0.02 ±126% +396.9% 0.11 ± 23% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.10 ±201% +1603.5% 1.72 ± 2% perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.18 ±170% +757.6% 1.53 ± 8% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1.52 ±222% +366.0% 7.10 ± 89% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.01 ±121% +11961.0% 0.82 ± 2% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.01 ±100% +4938.0% 0.42 perf-sched.total_sch_delay.average.ms 0.06 ±100% +5069.9% 3.27 ± 3% perf-sched.total_wait_and_delay.average.ms 0.05 ±100% +5089.4% 2.85 ± 3% perf-sched.total_wait_time.average.ms 0.15 ±111% +78953.3% 119.63 ± 77% perf-sched.wait_and_delay.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 0.74 ±143% +7023.2% 52.77 ± 99% perf-sched.wait_and_delay.avg.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 0.24 ±104% +18409.1% 44.58 ± 39% perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.16 ±113% +24208.5% 38.81 ± 90% perf-sched.wait_and_delay.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 67.31 ±222% +892.6% 668.08 perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.22 ±100% +304.9% 0.87 perf-sched.wait_and_delay.avg.ms.schedule_timeout.inet_csk_accept.inet_accept.do_accept 0.04 ±100% +4498.1% 1.64 perf-sched.wait_and_delay.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 16.14 ±102% +3168.2% 527.60 ± 4% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.67 ±145% +500.0% 10.00 ± 45% perf-sched.wait_and_delay.count.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 104.50 ±101% +267.5% 384.00 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 1.00 ±141% +400.0% 5.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1.33 ±134% +350.0% 6.00 perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 23.83 ±100% +428.7% 126.00 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 31.17 ±100% +400.5% 156.00 perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64 4.17 ±105% +2900.0% 125.00 ± 9% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 4.00 ±107% +212.5% 12.50 ± 4% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 6559 ±100% +2807.6% 190728 perf-sched.wait_and_delay.count.schedule_timeout.inet_csk_accept.inet_accept.do_accept 3.00 ±105% +566.7% 20.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork 378.83 ±101% +315.5% 1574 ± 4% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 68.33 ±100% +460.0% 382.67 ± 6% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 3.06 ±100% +9424.2% 291.66 ± 85% perf-sched.wait_and_delay.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 1.93 ±142% +24932.3% 483.87 ± 98% perf-sched.wait_and_delay.max.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 3.74 ±100% +21791.0% 817.67 ± 21% perf-sched.wait_and_delay.max.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 3.73 ±100% +10126.5% 381.06 ± 82% perf-sched.wait_and_delay.max.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 0.15 ±111% +78953.3% 119.63 ± 77% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 0.74 ±142% +6999.3% 52.77 ± 99% perf-sched.wait_time.avg.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 0.03 ±100% +2490.1% 0.74 ± 47% perf-sched.wait_time.avg.ms.__cond_resched.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg 0.24 ±104% +18409.1% 44.58 ± 39% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 0.01 ±100% +6623.7% 0.66 ± 39% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.tcp_stream_alloc_skb.tcp_sendmsg_locked 0.16 ±113% +24208.5% 38.81 ± 90% perf-sched.wait_time.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 66.90 ±223% +897.3% 667.24 perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 71.60 ±110% +211.6% 223.12 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.14 ±119% +11827.5% 16.48 ±203% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.01 ±100% +5690.1% 0.69 ± 36% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.21 ±100% +311.4% 0.84 perf-sched.wait_time.avg.ms.schedule_timeout.inet_csk_accept.inet_accept.do_accept 0.02 ±100% +3845.2% 0.83 perf-sched.wait_time.avg.ms.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 16.12 ±102% +3171.9% 527.50 ± 4% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.05 ±109% +874.8% 0.49 ± 11% perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 3.06 ±100% +9424.2% 291.66 ± 85% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 1.94 ±141% +24837.7% 483.87 ± 98% perf-sched.wait_time.max.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 3.74 ±100% +21791.0% 817.67 ± 21% perf-sched.wait_time.max.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 3.73 ±100% +10126.5% 381.06 ± 82% perf-sched.wait_time.max.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 0.64 ±106% +166.1% 1.72 ± 2% perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 18.01 -16.1 1.90 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg 16.53 -14.7 1.82 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked 22.91 -14.7 8.24 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 22.58 -14.7 7.92 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 15.60 -11.8 3.80 perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames 15.54 ± 2% -11.4 4.10 ± 6% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit 11.16 -10.4 0.71 ± 44% perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 13.15 ± 85% -9.9 3.21 ±223% perf-profile.calltrace.cycles-pp.__send 11.96 ± 92% -8.9 3.10 ±223% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__send 11.87 ± 92% -8.8 3.08 ±223% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send 7.82 -7.8 0.00 perf-profile.calltrace.cycles-pp.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg 7.04 ± 2% -7.0 0.00 perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 10.06 ±100% -7.0 3.04 ±223% perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.__send 6.78 -6.8 0.00 perf-profile.calltrace.cycles-pp.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg 10.53 -6.7 3.88 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg 6.51 -6.5 0.00 perf-profile.calltrace.cycles-pp.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked.tcp_recvmsg 21.45 -6.4 15.08 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb 6.27 -6.3 0.00 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.wait_woken.sk_wait_data.tcp_recvmsg_locked 21.30 -6.3 15.05 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit 21.08 -6.1 14.98 perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2 6.10 -6.1 0.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.wait_woken.sk_wait_data 5.50 ± 3% -5.5 0.00 perf-profile.calltrace.cycles-pp.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.tcp_v4_rcv 5.39 ± 3% -5.4 0.00 perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv 19.80 -5.2 14.59 perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit 5.19 ± 3% -5.2 0.00 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_data_queue.tcp_rcv_established 5.01 ± 3% -5.0 0.00 perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.sock_def_readable.tcp_data_queue 19.32 -4.9 14.42 perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip 19.10 -4.7 14.37 perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq 9.78 -4.7 5.06 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg 18.28 -4.2 14.09 perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq 16.41 -3.9 12.54 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 15.94 -3.7 12.22 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll 15.48 -3.6 11.88 perf-profile.calltrace.cycles-pp.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog 8.98 -3.5 5.44 perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked.tcp_recvmsg 13.12 -3.2 9.96 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core 8.66 -3.1 5.52 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_recvmsg_locked 31.20 -2.9 28.35 perf-profile.calltrace.cycles-pp.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom 3.66 -0.6 3.08 perf-profile.calltrace.cycles-pp.ioctl.stress_sock 2.87 -0.5 2.33 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.ioctl.stress_sock 2.71 -0.5 2.20 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl.stress_sock 2.41 -0.5 1.94 perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl.stress_sock 1.92 -0.4 1.51 perf-profile.calltrace.cycles-pp.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl 1.75 -0.4 1.34 perf-profile.calltrace.cycles-pp.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.64 ± 5% -0.3 1.30 ± 2% perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto 0.59 -0.3 0.25 ±100% perf-profile.calltrace.cycles-pp.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 1.51 -0.3 1.20 perf-profile.calltrace.cycles-pp.sock_do_ioctl.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl.do_syscall_64 1.38 -0.3 1.10 perf-profile.calltrace.cycles-pp.inet_ioctl.sock_do_ioctl.sock_ioctl.do_vfs_ioctl.__x64_sys_ioctl 38.50 -0.3 38.25 perf-profile.calltrace.cycles-pp.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 2.01 ± 4% -0.2 1.76 perf-profile.calltrace.cycles-pp.release_sock.tcp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 0.90 -0.1 0.76 ± 2% perf-profile.calltrace.cycles-pp.sk_ioctl.inet_ioctl.sock_do_ioctl.sock_ioctl.do_vfs_ioctl 1.24 +0.1 1.30 perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg 1.05 +0.1 1.18 perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked 0.81 +0.1 0.95 perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 0.54 +0.1 0.68 perf-profile.calltrace.cycles-pp.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl 0.55 +0.1 0.70 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl 0.56 +0.2 0.71 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.ioctl 0.58 +0.2 0.75 perf-profile.calltrace.cycles-pp.ioctl 0.26 ±100% +0.3 0.54 perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_try_rmem_schedule.tcp_data_queue 0.34 ± 70% +0.3 0.63 perf-profile.calltrace.cycles-pp.sock_ioctl.__x64_sys_ioctl.do_syscall_64.entry_SYSCALL_64_after_hwframe.ioctl 0.52 +0.3 0.86 perf-profile.calltrace.cycles-pp.skb_attempt_defer_free.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg 0.90 +0.3 1.25 perf-profile.calltrace.cycles-pp.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64 0.91 ± 2% +0.3 1.26 perf-profile.calltrace.cycles-pp.inet_stream_connect.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.93 ± 2% +0.4 1.28 perf-profile.calltrace.cycles-pp.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect.stress_sock 0.93 ± 2% +0.4 1.28 perf-profile.calltrace.cycles-pp.__sys_connect.__x64_sys_connect.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect 0.93 +0.4 1.29 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.connect.stress_sock 0.93 ± 2% +0.4 1.29 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.connect.stress_sock 0.94 +0.4 1.30 perf-profile.calltrace.cycles-pp.connect.stress_sock 0.00 +0.5 0.53 perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect 0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.tcp_rcv_synsent_state_process.tcp_rcv_state_process.tcp_v4_do_rcv.__release_sock.release_sock 0.00 +0.6 0.58 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect 0.00 +0.6 0.58 perf-profile.calltrace.cycles-pp.tcp_rcv_space_adjust.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg 0.00 +0.6 0.58 perf-profile.calltrace.cycles-pp.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect 0.00 +0.6 0.58 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.__inet_stream_connect.inet_stream_connect.__sys_connect 0.00 +0.6 0.65 perf-profile.calltrace.cycles-pp.tcp_v4_connect.__inet_stream_connect.inet_stream_connect.__sys_connect.__x64_sys_connect 0.00 +0.7 0.69 ± 2% perf-profile.calltrace.cycles-pp.inet_shutdown.__sys_shutdown.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.7 0.70 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe.shutdown.stress_sock 0.00 +0.7 0.70 ± 2% perf-profile.calltrace.cycles-pp.__sys_shutdown.__x64_sys_shutdown.do_syscall_64.entry_SYSCALL_64_after_hwframe.shutdown 0.00 +0.7 0.70 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.shutdown.stress_sock 0.00 +0.7 0.70 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.shutdown.stress_sock 0.00 +0.7 0.71 ± 2% perf-profile.calltrace.cycles-pp.shutdown.stress_sock 0.00 +0.8 0.75 perf-profile.calltrace.cycles-pp.skb_try_coalesce.tcp_try_coalesce.tcp_queue_rcv.tcp_data_queue.tcp_rcv_established 0.00 +0.8 0.83 perf-profile.calltrace.cycles-pp.tcp_try_rmem_schedule.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock 0.00 +0.8 0.84 perf-profile.calltrace.cycles-pp.skb_release_data.skb_attempt_defer_free.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg 0.00 +0.8 0.84 perf-profile.calltrace.cycles-pp.tcp_try_coalesce.tcp_queue_rcv.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv 0.00 +0.9 0.89 ± 16% perf-profile.calltrace.cycles-pp.__entry_text_start.__send.stress_sock 0.39 ±100% +0.9 1.29 ± 2% perf-profile.calltrace.cycles-pp.__check_object_size.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 0.00 +0.9 0.91 perf-profile.calltrace.cycles-pp.tcp_queue_rcv.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock 0.53 ± 74% +1.0 1.51 perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 0.00 +1.0 1.02 ± 15% perf-profile.calltrace.cycles-pp.tcp_mtu_probe.tcp_write_xmit.__tcp_push_pending_frames.tcp_sendmsg_locked.tcp_sendmsg 0.56 ± 74% +1.1 1.64 perf-profile.calltrace.cycles-pp.sk_page_frag_refill.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 1.15 ± 16% +1.1 2.26 perf-profile.calltrace.cycles-pp.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked.tcp_sendmsg 0.38 ±100% +1.1 1.50 perf-profile.calltrace.cycles-pp.mem_cgroup_charge_skmem.__sk_mem_raise_allocated.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked 1.20 ± 16% +1.2 2.43 perf-profile.calltrace.cycles-pp.__sk_mem_schedule.tcp_wmem_schedule.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 1.36 +1.2 2.60 perf-profile.calltrace.cycles-pp.tcp_wmem_schedule.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 2.12 +1.2 3.37 perf-profile.calltrace.cycles-pp.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 1.48 +1.4 2.88 perf-profile.calltrace.cycles-pp.__alloc_skb.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 0.36 ±100% +1.5 1.83 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.tcp_stream_alloc_skb.tcp_sendmsg_locked.tcp_sendmsg 0.00 +1.7 1.72 perf-profile.calltrace.cycles-pp.tcp_clean_rtx_queue.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv 0.00 +2.2 2.19 perf-profile.calltrace.cycles-pp.tcp_ack.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 0.00 +2.2 2.21 perf-profile.calltrace.cycles-pp.tcp_data_queue.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock 0.00 +2.7 2.75 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_state_process 0.00 +2.9 2.90 perf-profile.calltrace.cycles-pp.tcp_rcv_established.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg 0.00 +4.9 4.90 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_state_process.tcp_v4_do_rcv 0.00 +5.2 5.25 perf-profile.calltrace.cycles-pp.__tcp_transmit_skb.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg 5.08 +5.6 10.67 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg 0.51 +5.7 6.26 perf-profile.calltrace.cycles-pp.tcp_write_xmit.__tcp_push_pending_frames.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv 0.51 +5.8 6.29 perf-profile.calltrace.cycles-pp.__tcp_push_pending_frames.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu 0.00 +5.9 5.94 perf-profile.calltrace.cycles-pp.__ip_queue_xmit.__tcp_transmit_skb.tcp_v4_do_rcv.__release_sock.release_sock 5.41 +5.9 11.35 perf-profile.calltrace.cycles-pp._copy_from_iter.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg 0.00 +6.0 6.00 perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_v4_do_rcv.__release_sock 0.00 +6.0 6.00 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.__tcp_transmit_skb.tcp_v4_do_rcv 0.00 +6.4 6.38 perf-profile.calltrace.cycles-pp.tcp_v4_do_rcv.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg 6.41 +6.6 13.04 perf-profile.calltrace.cycles-pp.skb_do_copy_data_nocache.tcp_sendmsg_locked.tcp_sendmsg.sock_sendmsg.__sys_sendto 34.83 +6.9 41.69 perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv 36.29 +6.9 43.24 perf-profile.calltrace.cycles-pp.recv.stress_sock 35.24 +7.0 42.21 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recv.stress_sock 33.81 +7.0 40.85 perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 34.94 +7.1 42.05 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.stress_sock 33.00 +7.3 40.26 perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 32.76 +7.4 40.16 perf-profile.calltrace.cycles-pp.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 33.86 +7.9 41.75 perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recv.stress_sock 0.94 +8.1 9.08 perf-profile.calltrace.cycles-pp.tcp_rcv_state_process.tcp_v4_do_rcv.tcp_v4_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 6.69 ± 2% +9.0 15.66 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked 6.89 ± 2% +9.1 15.98 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg 8.65 +9.7 18.30 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg 8.72 +9.7 18.39 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.tcp_recvmsg_locked.tcp_recvmsg.inet_recvmsg.sock_recvmsg 0.68 ± 5% +10.4 11.08 perf-profile.calltrace.cycles-pp.release_sock.tcp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom 0.08 ±223% +10.7 10.81 perf-profile.calltrace.cycles-pp.__release_sock.release_sock.tcp_recvmsg.inet_recvmsg.sock_recvmsg 31.60 -12.6 18.95 perf-profile.children.cycles-pp.__tcp_transmit_skb 28.75 -10.5 18.21 perf-profile.children.cycles-pp.__ip_queue_xmit 10.22 -9.9 0.34 perf-profile.children.cycles-pp.__schedule 9.72 -9.4 0.34 ± 2% perf-profile.children.cycles-pp.schedule 26.70 -9.1 17.61 perf-profile.children.cycles-pp.ip_finish_output2 25.92 -8.5 17.40 perf-profile.children.cycles-pp.__dev_queue_xmit 24.85 -8.5 16.34 perf-profile.children.cycles-pp.tcp_write_xmit 24.00 -8.4 15.62 perf-profile.children.cycles-pp.__tcp_push_pending_frames 7.87 -7.7 0.18 ± 2% perf-profile.children.cycles-pp.sk_wait_data 12.76 -7.6 5.21 perf-profile.children.cycles-pp.tcp_rcv_established 6.79 -6.6 0.16 ± 3% perf-profile.children.cycles-pp.wait_woken 6.63 -6.3 0.34 perf-profile.children.cycles-pp.schedule_timeout 23.02 -6.2 16.85 perf-profile.children.cycles-pp.__local_bh_enable_ip 22.53 -6.1 16.46 perf-profile.children.cycles-pp.do_softirq 22.50 -5.9 16.62 perf-profile.children.cycles-pp.__do_softirq 5.71 ± 3% -5.3 0.37 ± 2% perf-profile.children.cycles-pp.sock_def_readable 5.60 ± 3% -5.3 0.30 ± 3% perf-profile.children.cycles-pp.__wake_up_common_lock 5.38 ± 3% -5.1 0.26 ± 3% perf-profile.children.cycles-pp.__wake_up_common 20.96 -5.0 15.92 perf-profile.children.cycles-pp.net_rx_action 5.23 ± 3% -5.0 0.22 ± 3% perf-profile.children.cycles-pp.try_to_wake_up 7.71 ± 2% -5.0 2.72 perf-profile.children.cycles-pp.tcp_data_queue 5.04 -4.8 0.28 ± 2% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 5.47 -4.7 0.73 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 20.45 -4.7 15.73 perf-profile.children.cycles-pp.__napi_poll 20.32 -4.6 15.68 perf-profile.children.cycles-pp.process_backlog 51.63 -4.5 47.16 perf-profile.children.cycles-pp.__send 19.56 -4.2 15.37 perf-profile.children.cycles-pp.__netif_receive_skb_one_core 17.67 -3.5 14.15 perf-profile.children.cycles-pp.ip_local_deliver_finish 17.20 -3.4 13.80 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 16.78 -3.2 13.56 perf-profile.children.cycles-pp.tcp_v4_rcv 31.28 -2.9 28.42 perf-profile.children.cycles-pp.tcp_recvmsg_locked 2.75 -2.7 0.10 ± 5% perf-profile.children.cycles-pp.ttwu_do_activate 2.42 -2.4 0.06 perf-profile.children.cycles-pp.pick_next_task_fair 2.24 -2.2 0.08 perf-profile.children.cycles-pp.dequeue_task_fair 2.22 -2.1 0.08 ± 7% perf-profile.children.cycles-pp.activate_task 2.14 -2.1 0.08 ± 8% perf-profile.children.cycles-pp.enqueue_task_fair 1.84 -1.8 0.07 perf-profile.children.cycles-pp.update_load_avg 1.57 -1.5 0.06 ± 9% perf-profile.children.cycles-pp.update_curr 1.45 ± 8% -1.3 0.12 ±122% perf-profile.children.cycles-pp.__cmd_record 1.32 ± 2% -1.3 0.05 perf-profile.children.cycles-pp.switch_fpu_return 1.20 -1.2 0.03 ± 70% perf-profile.children.cycles-pp.enqueue_entity 1.17 ± 2% -1.1 0.04 ± 44% perf-profile.children.cycles-pp.dequeue_entity 2.03 -0.9 1.14 perf-profile.children.cycles-pp.dev_hard_start_xmit 1.84 -0.8 1.02 perf-profile.children.cycles-pp.loopback_xmit 0.76 -0.7 0.05 ± 8% perf-profile.children.cycles-pp.__switch_to 1.39 -0.7 0.72 perf-profile.children.cycles-pp.__cond_resched 3.82 -0.7 3.16 perf-profile.children.cycles-pp.tcp_ack 0.79 -0.6 0.17 ± 4% perf-profile.children.cycles-pp.sched_clock_cpu 0.78 -0.6 0.22 ± 2% perf-profile.children.cycles-pp.irqtime_account_irq 0.68 -0.5 0.14 ± 2% perf-profile.children.cycles-pp.sched_clock 0.93 -0.5 0.41 perf-profile.children.cycles-pp.__tcp_send_ack 0.60 -0.5 0.12 ± 3% perf-profile.children.cycles-pp.native_sched_clock 0.77 ± 9% -0.5 0.30 perf-profile.children.cycles-pp._raw_spin_lock 1.06 ± 2% -0.5 0.61 ± 2% perf-profile.children.cycles-pp.ip_rcv 4.51 -0.4 4.09 perf-profile.children.cycles-pp.ioctl 1.94 -0.4 1.53 perf-profile.children.cycles-pp.do_vfs_ioctl 0.70 -0.4 0.32 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 1.26 -0.4 0.89 perf-profile.children.cycles-pp.tcp_try_rmem_schedule 0.82 -0.3 0.48 perf-profile.children.cycles-pp.__netif_rx 0.78 ± 2% -0.3 0.45 perf-profile.children.cycles-pp.netif_rx_internal 3.00 -0.3 2.67 perf-profile.children.cycles-pp.__x64_sys_ioctl 1.54 -0.3 1.23 perf-profile.children.cycles-pp.sock_do_ioctl 0.69 -0.3 0.38 perf-profile.children.cycles-pp.enqueue_to_backlog 1.01 -0.3 0.72 perf-profile.children.cycles-pp.__inet_lookup_skb 2.27 -0.3 2.00 perf-profile.children.cycles-pp.sock_ioctl 1.40 -0.3 1.12 perf-profile.children.cycles-pp.inet_ioctl 0.83 -0.3 0.55 perf-profile.children.cycles-pp.__inet_lookup_established 41.35 -0.3 41.09 perf-profile.children.cycles-pp.tcp_sendmsg 0.49 -0.2 0.26 ± 2% perf-profile.children.cycles-pp.__list_del_entry_valid_or_report 0.35 -0.2 0.13 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.44 ± 7% -0.2 0.22 ± 5% perf-profile.children.cycles-pp.ip_rcv_finish_core 0.62 ± 2% -0.2 0.40 ± 2% perf-profile.children.cycles-pp.validate_xmit_skb 0.52 ± 3% -0.2 0.31 ± 2% perf-profile.children.cycles-pp.mem_cgroup_uncharge_skmem 0.47 -0.2 0.29 ± 2% perf-profile.children.cycles-pp.ip_rcv_core 1.10 -0.2 0.92 perf-profile.children.cycles-pp.__sk_mem_reduce_allocated 0.72 -0.2 0.56 perf-profile.children.cycles-pp.kmalloc_reserve 1.03 -0.2 0.86 perf-profile.children.cycles-pp.kmem_cache_free 0.54 -0.2 0.38 ± 2% perf-profile.children.cycles-pp.ip_output 0.38 ± 5% -0.2 0.23 ± 9% perf-profile.children.cycles-pp.cgroup_rstat_updated 0.37 -0.1 0.22 ± 2% perf-profile.children.cycles-pp.bpf_skops_write_hdr_opt 0.92 -0.1 0.78 ± 2% perf-profile.children.cycles-pp.sk_ioctl 0.58 -0.1 0.45 ± 3% perf-profile.children.cycles-pp.tcp_ioctl 0.27 -0.1 0.14 ± 2% perf-profile.children.cycles-pp.is_vmalloc_addr 0.42 -0.1 0.29 ± 2% perf-profile.children.cycles-pp.ip_local_out 0.25 -0.1 0.13 ± 2% perf-profile.children.cycles-pp.tcp_ack_update_rtt 0.34 ± 4% -0.1 0.22 ± 2% perf-profile.children.cycles-pp.netif_skb_features 2.31 -0.1 2.19 perf-profile.children.cycles-pp.tcp_clean_rtx_queue 0.48 -0.1 0.37 ± 4% perf-profile.children.cycles-pp.stress_sock_client 0.35 -0.1 0.24 perf-profile.children.cycles-pp.__ip_local_out 0.57 -0.1 0.46 perf-profile.children.cycles-pp.__netif_receive_skb_core 0.20 ± 3% -0.1 0.09 ± 4% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.40 ± 2% -0.1 0.30 ± 2% perf-profile.children.cycles-pp.ktime_get_with_offset 0.18 ± 3% -0.1 0.08 perf-profile.children.cycles-pp.__fdget 0.16 ± 3% -0.1 0.06 ± 8% perf-profile.children.cycles-pp.__list_add_valid_or_report 0.61 ± 2% -0.1 0.51 ± 2% perf-profile.children.cycles-pp.security_socket_recvmsg 0.22 ± 4% -0.1 0.12 ± 3% perf-profile.children.cycles-pp.__sk_dst_check 0.15 ± 3% -0.1 0.06 ± 6% perf-profile.children.cycles-pp.tcp_rtt_estimator 0.27 -0.1 0.19 perf-profile.children.cycles-pp.import_single_range 0.23 ± 2% -0.1 0.15 ± 2% perf-profile.children.cycles-pp.ip_send_check 0.33 -0.1 0.25 perf-profile.children.cycles-pp.__lock_sock_fast 0.80 -0.1 0.72 perf-profile.children.cycles-pp.__kfree_skb 0.11 ± 4% -0.1 0.04 ± 44% perf-profile.children.cycles-pp.rb_next 0.24 ± 2% -0.1 0.18 ± 2% perf-profile.children.cycles-pp.sk_filter_trim_cap 0.30 ± 3% -0.1 0.23 ± 5% perf-profile.children.cycles-pp.__ip_finish_output 0.15 ± 3% -0.1 0.08 ± 4% perf-profile.children.cycles-pp.tcp_options_write 0.16 ± 6% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.ipv4_dst_check 0.12 ± 6% -0.1 0.06 ± 9% perf-profile.children.cycles-pp.raw_local_deliver 0.17 ± 2% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.nf_hook_slow 0.13 ± 3% -0.1 0.07 ± 7% perf-profile.children.cycles-pp.skb_push 0.12 ± 3% -0.1 0.06 perf-profile.children.cycles-pp.tcp_rearm_rto 0.16 ± 4% -0.1 0.10 ± 3% perf-profile.children.cycles-pp.skb_network_protocol 0.20 ± 3% -0.1 0.15 ± 3% perf-profile.children.cycles-pp.inet_ehashfn 0.14 ± 3% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.security_sock_rcv_skb 0.16 ± 3% -0.0 0.11 ± 3% perf-profile.children.cycles-pp.sock_put 0.18 ± 5% -0.0 0.13 ± 5% perf-profile.children.cycles-pp.ip_skb_dst_mtu 0.16 ± 4% -0.0 0.11 ± 6% perf-profile.children.cycles-pp.ip_finish_output 0.12 ± 4% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.eth_type_trans 0.17 ± 2% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.tcp_update_skb_after_send 0.13 ± 5% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.raw_v4_input 0.81 -0.0 0.78 perf-profile.children.cycles-pp.tcp_mstamp_refresh 0.42 -0.0 0.38 perf-profile.children.cycles-pp.tcp_schedule_loss_probe 0.10 ± 3% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__usecs_to_jiffies 0.17 -0.0 0.14 ± 3% perf-profile.children.cycles-pp.dst_release 0.14 ± 2% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.skb_release_head_state 0.09 ± 5% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.tcp_rate_skb_sent 0.11 ± 7% -0.0 0.08 perf-profile.children.cycles-pp.neigh_hh_output 0.12 -0.0 0.09 perf-profile.children.cycles-pp.tcp_event_data_recv 0.09 ± 4% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.validate_xmit_xfrm 0.09 ± 7% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.cubictcp_cong_avoid 0.12 ± 4% -0.0 0.09 ± 4% perf-profile.children.cycles-pp.__build_skb_around 0.09 ± 4% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__xfrm_policy_check2 0.13 ± 3% -0.0 0.11 perf-profile.children.cycles-pp.tcp_v4_fill_cb 0.08 ± 7% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.tcp_cleanup_rbuf 0.08 ± 5% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.ip_local_deliver 0.08 ± 6% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.netdev_core_pick_tx 0.24 ± 2% -0.0 0.22 ± 3% perf-profile.children.cycles-pp.tcp_check_space 0.08 ± 7% -0.0 0.06 perf-profile.children.cycles-pp.rb_first 0.07 -0.0 0.05 perf-profile.children.cycles-pp.tcp_small_queue_check 0.14 ± 2% -0.0 0.12 ± 6% perf-profile.children.cycles-pp.tcp_inbound_md5_hash 0.10 ± 4% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.tcp_recv_timestamp 0.08 ± 6% -0.0 0.06 perf-profile.children.cycles-pp.tcp_v4_send_check 0.12 ± 4% -0.0 0.10 ± 3% perf-profile.children.cycles-pp.__copy_skb_header 0.07 ± 6% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.security_file_ioctl 0.07 +0.0 0.08 perf-profile.children.cycles-pp.tcp_urg 0.06 ± 6% +0.0 0.07 perf-profile.children.cycles-pp.__tcp_close 0.06 ± 6% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.slab_pre_alloc_hook 0.06 ± 9% +0.0 0.07 perf-profile.children.cycles-pp.ip_route_output_flow 0.07 ± 5% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.do_tcp_getsockopt 0.07 +0.0 0.09 ± 5% perf-profile.children.cycles-pp.do_tcp_setsockopt 0.12 ± 3% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.tcp_update_recv_tstamps 0.05 ± 8% +0.0 0.07 ± 5% perf-profile.children.cycles-pp.seq_show 0.05 +0.0 0.07 ± 5% perf-profile.children.cycles-pp.task_tick_fair 0.13 ± 3% +0.0 0.15 ± 3% perf-profile.children.cycles-pp.sock_rfree 0.06 +0.0 0.08 perf-profile.children.cycles-pp.__sk_destruct 0.04 ± 44% +0.0 0.06 ± 6% perf-profile.children.cycles-pp.tcp_v4_send_synack 0.06 ± 6% +0.0 0.08 ± 7% perf-profile.children.cycles-pp.__d_alloc 0.21 ± 2% +0.0 0.24 ± 3% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook 0.07 ± 6% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.__tcp_ack_snd_check 0.06 ± 8% +0.0 0.08 ± 7% perf-profile.children.cycles-pp.rcu_cblist_dequeue 0.06 ± 8% +0.0 0.08 perf-profile.children.cycles-pp.__kmem_cache_alloc_node 0.11 ± 3% +0.0 0.13 ± 2% perf-profile.children.cycles-pp.inet_release 0.07 ± 9% +0.0 0.10 ± 5% perf-profile.children.cycles-pp.tcp_getsockopt 0.06 ± 9% +0.0 0.08 ± 7% perf-profile.children.cycles-pp.tcp_create_openreq_child 0.05 ± 8% +0.0 0.08 ± 4% perf-profile.children.cycles-pp.walk_component 0.12 ± 3% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.__sock_release 0.12 ± 3% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.sock_close 0.05 +0.0 0.08 ± 6% perf-profile.children.cycles-pp.tcp_reset 0.07 +0.0 0.10 ± 3% perf-profile.children.cycles-pp.init_file 0.07 ± 5% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.__inet_hash_connect 0.06 ± 6% +0.0 0.09 perf-profile.children.cycles-pp.evict 0.08 ± 5% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.link_path_walk 0.22 +0.0 0.25 ± 3% perf-profile.children.cycles-pp.check_stack_object 0.08 +0.0 0.11 perf-profile.children.cycles-pp.alloc_file_pseudo 0.12 ± 3% +0.0 0.15 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.07 +0.0 0.10 ± 3% perf-profile.children.cycles-pp.do_open 0.04 ± 44% +0.0 0.07 ± 6% perf-profile.children.cycles-pp._copy_from_user 0.09 ± 4% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.tcp_v4_send_reset 0.08 ± 5% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.sock_alloc_file 0.03 ± 70% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.inet_csk_clone_lock 0.11 +0.0 0.14 ± 3% perf-profile.children.cycles-pp.do_dentry_open 0.24 ± 3% +0.0 0.28 ± 2% perf-profile.children.cycles-pp.rcu_all_qs 0.12 ± 3% +0.0 0.16 ± 3% perf-profile.children.cycles-pp.dentry_open 0.11 +0.0 0.15 ± 3% perf-profile.children.cycles-pp.seq_read_iter 0.08 ± 6% +0.0 0.11 ± 6% perf-profile.children.cycles-pp.mod_objcg_state 0.08 ± 6% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.inode_init_always 0.06 ± 7% +0.0 0.10 ± 3% perf-profile.children.cycles-pp.tcp_set_state 0.10 ± 3% +0.0 0.14 ± 5% perf-profile.children.cycles-pp.autoremove_wake_function 0.29 ± 3% +0.0 0.33 ± 3% perf-profile.children.cycles-pp.tcp_established_options 0.12 ± 4% +0.0 0.16 ± 3% perf-profile.children.cycles-pp.seq_read 0.07 ± 5% +0.0 0.11 ± 3% perf-profile.children.cycles-pp.tcp_done 0.14 +0.0 0.18 ± 2% perf-profile.children.cycles-pp.__sys_setsockopt 0.02 ± 99% +0.0 0.07 ± 7% perf-profile.children.cycles-pp.sk_clone_lock 0.09 ± 4% +0.0 0.13 ± 4% perf-profile.children.cycles-pp.lookup_fast 0.27 +0.0 0.32 perf-profile.children.cycles-pp.syscall_enter_from_user_mode 0.12 ± 4% +0.0 0.16 perf-profile.children.cycles-pp.ip_send_unicast_reply 0.06 ± 8% +0.0 0.10 perf-profile.children.cycles-pp.rb_erase 1.13 +0.0 1.17 perf-profile.children.cycles-pp._raw_spin_lock_bh 0.14 +0.0 0.18 ± 2% perf-profile.children.cycles-pp.ksys_read 0.10 ± 5% +0.0 0.14 ± 6% perf-profile.children.cycles-pp.tcp_v4_syn_recv_sock 0.02 ± 99% +0.0 0.07 perf-profile.children.cycles-pp.prep_compound_page 0.14 ± 3% +0.0 0.19 ± 3% perf-profile.children.cycles-pp.__x64_sys_setsockopt 0.12 ± 4% +0.0 0.17 ± 4% perf-profile.children.cycles-pp.tcp_check_req 0.13 ± 2% +0.0 0.18 ± 2% perf-profile.children.cycles-pp.tcp_conn_request 0.13 ± 2% +0.0 0.18 ± 4% perf-profile.children.cycles-pp.vfs_read 0.15 +0.0 0.20 ± 4% perf-profile.children.cycles-pp.read 0.00 +0.1 0.05 perf-profile.children.cycles-pp.enqueue_timer 0.00 +0.1 0.05 perf-profile.children.cycles-pp.sk_leave_memory_pressure 0.00 +0.1 0.05 perf-profile.children.cycles-pp.vsnprintf 0.00 +0.1 0.05 perf-profile.children.cycles-pp.seq_printf 0.00 +0.1 0.05 perf-profile.children.cycles-pp.get_nohz_timer_target 0.00 +0.1 0.05 perf-profile.children.cycles-pp.__kmalloc_node 0.00 +0.1 0.05 perf-profile.children.cycles-pp.apparmor_socket_sendmsg 0.00 +0.1 0.05 ± 7% perf-profile.children.cycles-pp.tcp_leave_memory_pressure 0.00 +0.1 0.05 ± 7% perf-profile.children.cycles-pp.tcp_v4_send_ack 0.02 ±141% +0.1 0.07 ± 5% perf-profile.children.cycles-pp.getsockname 0.15 ± 4% +0.1 0.20 ± 3% perf-profile.children.cycles-pp.__dentry_kill 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.__mkroute_output 0.00 +0.1 0.06 ± 9% perf-profile.children.cycles-pp.free_unref_page_commit 0.02 ±141% +0.1 0.07 ± 5% perf-profile.children.cycles-pp.alloc_file 0.16 ± 2% +0.1 0.21 ± 2% perf-profile.children.cycles-pp.dentry_kill 0.14 ± 3% +0.1 0.19 ± 4% perf-profile.children.cycles-pp.tcp_child_process 0.13 ± 2% +0.1 0.18 ± 2% perf-profile.children.cycles-pp.kmem_cache_alloc_lru 0.10 ± 5% +0.1 0.16 ± 3% perf-profile.children.cycles-pp.inet_sendmsg 0.02 ± 99% +0.1 0.08 ± 4% perf-profile.children.cycles-pp.propagate_protected_usage 0.00 +0.1 0.06 ± 8% perf-profile.children.cycles-pp.memcmp 0.00 +0.1 0.06 ± 6% perf-profile.children.cycles-pp.open_last_lookups 0.01 ±223% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.__d_lookup_rcu 0.18 ± 4% +0.1 0.24 ± 3% perf-profile.children.cycles-pp.__x64_sys_getsockopt 0.17 ± 4% +0.1 0.23 ± 2% perf-profile.children.cycles-pp.__sys_getsockopt 0.17 ± 5% +0.1 0.23 perf-profile.children.cycles-pp.__sock_create 0.19 ± 3% +0.1 0.25 perf-profile.children.cycles-pp.__ns_get_path 0.06 +0.1 0.12 perf-profile.children.cycles-pp._raw_spin_trylock 0.00 +0.1 0.06 perf-profile.children.cycles-pp.single_open 0.00 +0.1 0.06 perf-profile.children.cycles-pp.sock_alloc_inode 0.00 +0.1 0.06 perf-profile.children.cycles-pp.security_file_alloc 0.00 +0.1 0.06 perf-profile.children.cycles-pp.free_tail_page_prepare 0.00 +0.1 0.06 perf-profile.children.cycles-pp.alloc_pages 0.00 +0.1 0.06 perf-profile.children.cycles-pp.inet_send_prepare 0.11 ± 5% +0.1 0.17 perf-profile.children.cycles-pp.sock_alloc 1.26 +0.1 1.32 perf-profile.children.cycles-pp.simple_copy_to_iter 0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.inet_put_port 0.22 ± 3% +0.1 0.28 ± 2% perf-profile.children.cycles-pp.rcu_core 0.19 +0.1 0.26 ± 2% perf-profile.children.cycles-pp.dput 0.58 +0.1 0.65 ± 2% perf-profile.children.cycles-pp.__virt_addr_valid 0.18 ± 2% +0.1 0.25 ± 2% perf-profile.children.cycles-pp.rcu_do_batch 0.18 ± 4% +0.1 0.25 ± 4% perf-profile.children.cycles-pp.__irq_exit_rcu 0.14 ± 3% +0.1 0.21 ± 2% perf-profile.children.cycles-pp.alloc_empty_file 0.07 ± 5% +0.1 0.14 ± 2% perf-profile.children.cycles-pp.rb_insert_color 0.17 ± 4% +0.1 0.24 ± 2% perf-profile.children.cycles-pp.alloc_inode 0.14 ± 3% +0.1 0.21 perf-profile.children.cycles-pp.__x64_sys_accept 0.13 ± 3% +0.1 0.20 perf-profile.children.cycles-pp.kmem_cache_alloc 0.23 ± 4% +0.1 0.30 perf-profile.children.cycles-pp.__x64_sys_socket 0.23 ± 3% +0.1 0.30 perf-profile.children.cycles-pp.__sys_socket 0.20 ± 2% +0.1 0.28 ± 2% perf-profile.children.cycles-pp.setsockopt 0.00 +0.1 0.08 perf-profile.children.cycles-pp.___slab_alloc 1.04 +0.1 1.12 ± 3% perf-profile.children.cycles-pp.__fget_light 0.53 ± 2% +0.1 0.62 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret 0.10 ± 4% +0.1 0.18 ± 3% perf-profile.children.cycles-pp.tcp_validate_incoming 0.14 ± 3% +0.1 0.22 ± 2% perf-profile.children.cycles-pp.__x64_sys_accept4 0.20 ± 4% +0.1 0.28 ± 7% perf-profile.children.cycles-pp.ipv4_mtu 0.25 ± 4% +0.1 0.34 perf-profile.children.cycles-pp.__socket 0.30 +0.1 0.40 perf-profile.children.cycles-pp.__skb_clone 0.20 ± 3% +0.1 0.29 perf-profile.children.cycles-pp.new_inode_pseudo 0.16 ± 3% +0.1 0.25 perf-profile.children.cycles-pp.accept 0.49 ± 2% +0.1 0.59 perf-profile.children.cycles-pp.tcp_rcv_space_adjust 0.25 ± 2% +0.1 0.35 ± 2% perf-profile.children.cycles-pp.getsockopt 0.35 +0.1 0.45 perf-profile.children.cycles-pp.open_related_ns 0.20 ± 2% +0.1 0.30 perf-profile.children.cycles-pp.rmqueue 0.24 ± 2% +0.1 0.34 perf-profile.children.cycles-pp.path_openat 0.24 ± 2% +0.1 0.34 perf-profile.children.cycles-pp.do_filp_open 0.15 ± 6% +0.1 0.26 ± 2% perf-profile.children.cycles-pp.inet_csk_accept 0.41 +0.1 0.52 perf-profile.children.cycles-pp.refill_stock 0.16 ± 3% +0.1 0.27 perf-profile.children.cycles-pp.accept4 0.00 +0.1 0.11 ± 4% perf-profile.children.cycles-pp.kfree_skb_partial 0.36 ± 6% +0.1 0.48 ± 3% perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context 1.03 +0.1 1.15 perf-profile.children.cycles-pp.read_tsc 0.36 ± 6% +0.1 0.48 ± 4% perf-profile.children.cycles-pp.perf_event_task_tick 0.30 +0.1 0.42 perf-profile.children.cycles-pp.tcp_connect 0.00 +0.1 0.12 ± 4% perf-profile.children.cycles-pp.tcp_stream_memory_free 0.16 ± 4% +0.1 0.27 perf-profile.children.cycles-pp.inet_accept 0.39 +0.1 0.50 perf-profile.children.cycles-pp.__fput 0.27 +0.1 0.39 perf-profile.children.cycles-pp.do_sys_openat2 0.28 +0.1 0.40 perf-profile.children.cycles-pp.__x64_sys_openat 0.17 ± 4% +0.1 0.30 ± 2% perf-profile.children.cycles-pp.tcp_push 0.29 ± 2% +0.1 0.42 perf-profile.children.cycles-pp.open64 0.43 ± 2% +0.1 0.56 perf-profile.children.cycles-pp.__x64_sys_close 0.00 +0.1 0.13 ± 3% perf-profile.children.cycles-pp.lock_timer_base 0.46 ± 6% +0.1 0.60 ± 3% perf-profile.children.cycles-pp.scheduler_tick 0.14 ± 4% +0.1 0.28 ± 2% perf-profile.children.cycles-pp.page_counter_uncharge 0.14 ± 3% +0.1 0.28 perf-profile.children.cycles-pp.free_unref_page_prepare 0.15 ± 3% +0.1 0.29 perf-profile.children.cycles-pp.drain_stock 0.57 ± 7% +0.1 0.71 ± 4% perf-profile.children.cycles-pp.tick_sched_timer 0.38 ± 2% +0.1 0.52 perf-profile.children.cycles-pp.sk_reset_timer 0.54 ± 7% +0.1 0.69 ± 3% perf-profile.children.cycles-pp.update_process_times 0.61 ± 2% +0.1 0.76 ± 2% perf-profile.children.cycles-pp.security_socket_sendmsg 0.64 ± 6% +0.1 0.78 ± 3% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.55 ± 6% +0.2 0.70 ± 4% perf-profile.children.cycles-pp.tick_sched_handle 0.26 +0.2 0.40 perf-profile.children.cycles-pp.do_accept 0.46 +0.2 0.62 perf-profile.children.cycles-pp.__close 0.41 +0.2 0.56 perf-profile.children.cycles-pp.tcp_rcv_synsent_state_process 0.04 ± 44% +0.2 0.20 ± 2% perf-profile.children.cycles-pp.tcp_rbtree_insert 0.27 ± 2% +0.2 0.43 perf-profile.children.cycles-pp.__sys_accept4 0.22 ± 3% +0.2 0.38 perf-profile.children.cycles-pp.tcp_tso_segs 0.20 ± 3% +0.2 0.36 perf-profile.children.cycles-pp.page_counter_try_charge 0.68 ± 5% +0.2 0.85 ± 3% perf-profile.children.cycles-pp.hrtimer_interrupt 0.32 +0.2 0.49 perf-profile.children.cycles-pp.__mod_timer 0.29 +0.2 0.47 perf-profile.children.cycles-pp.get_page_from_freelist 0.55 +0.2 0.73 perf-profile.children.cycles-pp.lock_sock_nested 0.70 ± 6% +0.2 0.88 ± 2% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 0.47 +0.2 0.65 perf-profile.children.cycles-pp.tcp_v4_connect 0.18 ± 2% +0.2 0.37 perf-profile.children.cycles-pp.tcp_skb_entail 0.89 +0.2 1.08 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.54 +0.2 0.74 perf-profile.children.cycles-pp.tcp_event_new_data_sent 0.21 +0.2 0.42 perf-profile.children.cycles-pp.free_unref_page 0.34 +0.2 0.58 perf-profile.children.cycles-pp.__alloc_pages 0.44 +0.2 0.69 ± 2% perf-profile.children.cycles-pp.inet_shutdown 0.45 +0.2 0.70 ± 2% perf-profile.children.cycles-pp.__x64_sys_shutdown 0.44 ± 2% +0.3 0.70 ± 2% perf-profile.children.cycles-pp.__sys_shutdown 0.89 ± 4% +0.3 1.14 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.46 ± 2% +0.3 0.71 ± 2% perf-profile.children.cycles-pp.shutdown 0.93 ± 4% +0.3 1.20 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.68 +0.3 0.97 ± 2% perf-profile.children.cycles-pp.tcp_send_mss 0.54 +0.3 0.84 ± 2% perf-profile.children.cycles-pp.tcp_current_mss 0.08 ± 5% +0.3 0.39 ± 2% perf-profile.children.cycles-pp.tcp_add_backlog 0.11 ± 5% +0.3 0.43 ± 2% perf-profile.children.cycles-pp.skb_clone 0.53 +0.3 0.87 perf-profile.children.cycles-pp.skb_attempt_defer_free 0.90 +0.3 1.25 perf-profile.children.cycles-pp.__inet_stream_connect 0.91 ± 2% +0.3 1.26 perf-profile.children.cycles-pp.inet_stream_connect 0.93 ± 2% +0.4 1.28 perf-profile.children.cycles-pp.__x64_sys_connect 0.93 ± 2% +0.4 1.28 perf-profile.children.cycles-pp.__sys_connect 0.94 +0.4 1.30 perf-profile.children.cycles-pp.connect 1.97 +0.4 2.35 perf-profile.children.cycles-pp.mem_cgroup_charge_skmem 0.75 +0.4 1.13 perf-profile.children.cycles-pp.try_charge_memcg 1.35 ± 2% +0.4 1.75 perf-profile.children.cycles-pp.ktime_get 1.11 +0.4 1.54 perf-profile.children.cycles-pp.skb_release_data 0.00 +0.4 0.45 perf-profile.children.cycles-pp.__slab_free 1.47 +0.5 1.97 perf-profile.children.cycles-pp.check_heap_object 1.27 +0.6 1.84 ± 2% perf-profile.children.cycles-pp.__entry_text_start 2.13 +0.6 2.75 perf-profile.children.cycles-pp.__check_object_size 0.90 +0.6 1.54 perf-profile.children.cycles-pp.skb_page_frag_refill 0.41 +0.7 1.13 perf-profile.children.cycles-pp.tcp_mtu_probe 0.96 +0.7 1.68 perf-profile.children.cycles-pp.sk_page_frag_refill 1.66 +0.8 2.47 perf-profile.children.cycles-pp.kmem_cache_alloc_node 0.08 ± 4% +0.8 0.90 perf-profile.children.cycles-pp.skb_try_coalesce 0.23 ± 2% +0.8 1.08 perf-profile.children.cycles-pp.tcp_queue_rcv 0.09 ± 4% +0.9 0.99 perf-profile.children.cycles-pp.tcp_try_coalesce 2.46 +0.9 3.41 perf-profile.children.cycles-pp.__sk_mem_raise_allocated 2.37 +1.0 3.33 perf-profile.children.cycles-pp.__alloc_skb 2.54 +1.1 3.60 perf-profile.children.cycles-pp.__sk_mem_schedule 1.38 +1.3 2.64 perf-profile.children.cycles-pp.tcp_wmem_schedule 2.16 +1.3 3.42 perf-profile.children.cycles-pp.tcp_stream_alloc_skb 92.39 +2.5 94.93 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 91.61 +2.8 94.41 perf-profile.children.cycles-pp.do_syscall_64 15.61 +3.5 19.09 perf-profile.children.cycles-pp.tcp_v4_do_rcv 38.74 +4.8 43.56 perf-profile.children.cycles-pp.recv 5.17 +5.7 10.85 perf-profile.children.cycles-pp.copyin 5.44 +6.0 11.40 perf-profile.children.cycles-pp._copy_from_iter 6.46 +6.7 13.13 perf-profile.children.cycles-pp.skb_do_copy_data_nocache 35.00 +6.8 41.78 perf-profile.children.cycles-pp.__x64_sys_recvfrom 34.87 +6.9 41.74 perf-profile.children.cycles-pp.__sys_recvfrom 33.83 +7.1 40.89 perf-profile.children.cycles-pp.sock_recvmsg 33.03 +7.3 40.29 perf-profile.children.cycles-pp.inet_recvmsg 32.84 +7.4 40.22 perf-profile.children.cycles-pp.tcp_recvmsg 1.61 +8.4 10.00 perf-profile.children.cycles-pp.tcp_rcv_state_process 6.74 ± 2% +9.0 15.75 perf-profile.children.cycles-pp.copyout 6.91 ± 2% +9.1 16.02 perf-profile.children.cycles-pp._copy_to_iter 8.74 +9.7 18.41 perf-profile.children.cycles-pp.skb_copy_datagram_iter 8.67 +9.7 18.34 perf-profile.children.cycles-pp.__skb_datagram_iter 3.51 +10.2 13.68 perf-profile.children.cycles-pp.release_sock 2.66 +10.2 12.86 perf-profile.children.cycles-pp.__release_sock 1.17 ± 2% -1.1 0.05 ± 7% perf-profile.self.cycles-pp.__schedule 0.73 -0.7 0.05 ± 7% perf-profile.self.cycles-pp.__switch_to 0.74 ± 9% -0.5 0.28 perf-profile.self.cycles-pp._raw_spin_lock 0.58 -0.5 0.12 ± 4% perf-profile.self.cycles-pp.native_sched_clock 0.54 -0.4 0.17 perf-profile.self.cycles-pp.loopback_xmit 0.66 -0.4 0.30 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.74 ± 3% -0.4 0.38 ± 2% perf-profile.self.cycles-pp.ip_finish_output2 0.99 -0.4 0.64 perf-profile.self.cycles-pp.tcp_ack 0.52 -0.3 0.18 ± 2% perf-profile.self.cycles-pp.__do_softirq 0.44 -0.3 0.16 perf-profile.self.cycles-pp.net_rx_action 0.67 -0.3 0.40 perf-profile.self.cycles-pp.__ip_queue_xmit 0.81 -0.3 0.56 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.81 -0.2 0.56 perf-profile.self.cycles-pp.tcp_recvmsg_locked 0.42 -0.2 0.19 perf-profile.self.cycles-pp.process_backlog 0.62 -0.2 0.41 ± 2% perf-profile.self.cycles-pp.__inet_lookup_established 0.33 -0.2 0.12 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irq 0.44 -0.2 0.24 ± 2% perf-profile.self.cycles-pp.__list_del_entry_valid_or_report 0.42 ± 7% -0.2 0.22 ± 6% perf-profile.self.cycles-pp.ip_rcv_finish_core 0.44 ± 2% -0.2 0.25 ± 2% perf-profile.self.cycles-pp.mem_cgroup_uncharge_skmem 0.25 -0.2 0.07 ± 5% perf-profile.self.cycles-pp.irqtime_account_irq 0.54 -0.2 0.37 perf-profile.self.cycles-pp.do_syscall_64 0.74 -0.2 0.56 ± 2% perf-profile.self.cycles-pp.__sys_recvfrom 0.97 -0.2 0.80 perf-profile.self.cycles-pp.kmem_cache_free 0.45 -0.2 0.28 ± 2% perf-profile.self.cycles-pp.ip_rcv_core 0.52 -0.2 0.36 perf-profile.self.cycles-pp.tcp_clean_rtx_queue 0.32 ± 2% -0.2 0.16 ± 2% perf-profile.self.cycles-pp.enqueue_to_backlog 0.22 ± 3% -0.1 0.08 perf-profile.self.cycles-pp.__x64_sys_sendto 0.25 ± 3% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.is_vmalloc_addr 0.36 ± 2% -0.1 0.21 perf-profile.self.cycles-pp.bpf_skops_write_hdr_opt 0.22 ± 3% -0.1 0.08 ± 7% perf-profile.self.cycles-pp.do_softirq 0.35 ± 5% -0.1 0.21 ± 10% perf-profile.self.cycles-pp.cgroup_rstat_updated 0.57 -0.1 0.43 perf-profile.self.cycles-pp.tcp_v4_rcv 0.45 -0.1 0.31 perf-profile.self.cycles-pp.tcp_recvmsg 0.47 ± 2% -0.1 0.34 ± 2% perf-profile.self.cycles-pp.inet_ioctl 0.18 ± 2% -0.1 0.07 perf-profile.self.cycles-pp.inet_recvmsg 0.20 -0.1 0.09 ± 6% perf-profile.self.cycles-pp.sock_recvmsg 0.56 -0.1 0.45 perf-profile.self.cycles-pp.__netif_receive_skb_core 0.22 ± 2% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.tcp_schedule_loss_probe 1.60 -0.1 1.50 perf-profile.self.cycles-pp.__tcp_transmit_skb 0.37 -0.1 0.27 perf-profile.self.cycles-pp.ip_output 0.16 ± 3% -0.1 0.07 ± 5% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.43 ± 8% -0.1 0.34 ± 4% perf-profile.self.cycles-pp.recv 0.25 -0.1 0.16 ± 4% perf-profile.self.cycles-pp.import_single_range 0.21 -0.1 0.12 ± 3% perf-profile.self.cycles-pp.tcp_event_new_data_sent 0.25 ± 4% -0.1 0.17 ± 2% perf-profile.self.cycles-pp.sock_ioctl 0.14 ± 3% -0.1 0.06 perf-profile.self.cycles-pp.tcp_rtt_estimator 0.12 ± 4% -0.1 0.04 ± 44% perf-profile.self.cycles-pp.__list_add_valid_or_report 0.45 -0.1 0.36 ± 2% perf-profile.self.cycles-pp.__local_bh_enable_ip 0.19 ± 2% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.__netif_receive_skb_one_core 0.24 -0.1 0.16 ± 3% perf-profile.self.cycles-pp.validate_xmit_skb 0.13 ± 2% -0.1 0.05 perf-profile.self.cycles-pp.__napi_poll 0.30 -0.1 0.22 perf-profile.self.cycles-pp.tcp_rcv_established 0.19 -0.1 0.11 ± 4% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu 0.39 ± 2% -0.1 0.31 ± 2% perf-profile.self.cycles-pp.stress_sock_client 0.22 ± 2% -0.1 0.14 ± 2% perf-profile.self.cycles-pp.ip_send_check 0.11 ± 4% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.raw_local_deliver 0.20 ± 2% -0.1 0.13 ± 4% perf-profile.self.cycles-pp.dev_hard_start_xmit 0.17 ± 4% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.ip_rcv 0.13 ± 3% -0.1 0.06 ± 6% perf-profile.self.cycles-pp.tcp_options_write 0.18 ± 2% -0.1 0.11 ± 3% perf-profile.self.cycles-pp.__sk_mem_reduce_allocated 0.18 ± 7% -0.1 0.12 ± 4% perf-profile.self.cycles-pp.netif_skb_features 0.15 ± 7% -0.1 0.09 perf-profile.self.cycles-pp.skb_network_protocol 0.14 ± 7% -0.1 0.08 ± 6% perf-profile.self.cycles-pp.ipv4_dst_check 0.11 ± 3% -0.1 0.05 perf-profile.self.cycles-pp.skb_push 0.24 ± 2% -0.1 0.18 ± 4% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.89 -0.1 0.84 perf-profile.self.cycles-pp.aa_sk_perm 0.18 ± 2% -0.1 0.13 ± 10% perf-profile.self.cycles-pp.tcp_ioctl 0.18 ± 2% -0.1 0.13 ± 3% perf-profile.self.cycles-pp.inet_ehashfn 0.22 ± 2% -0.0 0.18 ± 2% perf-profile.self.cycles-pp.tcp_data_queue 0.15 ± 2% -0.0 0.10 perf-profile.self.cycles-pp.sock_put 0.13 ± 2% -0.0 0.08 perf-profile.self.cycles-pp.tcp_queue_rcv 0.19 ± 3% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.tcp_sendmsg 0.11 ± 6% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.eth_type_trans 0.16 ± 3% -0.0 0.11 ± 5% perf-profile.self.cycles-pp.ktime_get_with_offset 0.12 ± 3% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.__sk_dst_check 0.11 ± 3% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.kmalloc_reserve 0.15 ± 6% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.ip_finish_output 0.12 ± 6% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.raw_v4_input 0.17 ± 4% -0.0 0.12 ± 4% perf-profile.self.cycles-pp.ip_skb_dst_mtu 0.14 ± 2% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.sock_do_ioctl 0.11 ± 4% -0.0 0.08 ± 4% perf-profile.self.cycles-pp.nf_hook_slow 0.25 -0.0 0.22 ± 2% perf-profile.self.cycles-pp.refill_stock 0.17 -0.0 0.14 ± 5% perf-profile.self.cycles-pp.__inet_lookup_skb 0.10 ± 3% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.security_sock_rcv_skb 0.15 ± 2% -0.0 0.12 ± 3% perf-profile.self.cycles-pp.dst_release 0.22 ± 2% -0.0 0.20 ± 2% perf-profile.self.cycles-pp.__mod_timer 0.12 ± 4% -0.0 0.10 ± 5% perf-profile.self.cycles-pp.__ip_local_out 0.09 ± 4% -0.0 0.06 perf-profile.self.cycles-pp.tcp_rate_skb_sent 0.10 ± 6% -0.0 0.07 perf-profile.self.cycles-pp.neigh_hh_output 0.09 ± 5% -0.0 0.06 perf-profile.self.cycles-pp.netif_rx_internal 0.11 ± 4% -0.0 0.09 ± 5% perf-profile.self.cycles-pp.tcp_event_data_recv 0.10 ± 3% -0.0 0.08 ± 6% perf-profile.self.cycles-pp.__build_skb_around 0.12 ± 3% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.__ip_finish_output 0.12 ± 3% -0.0 0.10 perf-profile.self.cycles-pp.tcp_v4_fill_cb 0.23 -0.0 0.21 ± 3% perf-profile.self.cycles-pp.tcp_check_space 0.10 ± 4% -0.0 0.08 ± 4% perf-profile.self.cycles-pp.sk_filter_trim_cap 0.09 -0.0 0.07 ± 5% perf-profile.self.cycles-pp.ip_local_deliver_finish 0.09 ± 4% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.tcp_update_skb_after_send 0.08 ± 4% -0.0 0.06 perf-profile.self.cycles-pp.validate_xmit_xfrm 0.07 ± 5% -0.0 0.05 perf-profile.self.cycles-pp.__usecs_to_jiffies 0.07 ± 9% -0.0 0.05 perf-profile.self.cycles-pp.__xfrm_policy_check2 0.10 -0.0 0.08 perf-profile.self.cycles-pp.sock_def_readable 0.10 ± 4% -0.0 0.08 ± 4% perf-profile.self.cycles-pp.tcp_recv_timestamp 0.11 ± 3% -0.0 0.09 perf-profile.self.cycles-pp.__copy_skb_header 0.07 ± 5% -0.0 0.05 ± 7% perf-profile.self.cycles-pp.tcp_v4_send_check 0.07 ± 6% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.ip_local_deliver 0.18 ± 2% -0.0 0.17 ± 2% perf-profile.self.cycles-pp.sock_sendmsg 0.07 ± 10% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.tcp_cleanup_rbuf 0.06 ± 7% -0.0 0.05 perf-profile.self.cycles-pp.rb_first 0.06 ± 6% -0.0 0.05 perf-profile.self.cycles-pp.ip_local_out 0.07 +0.0 0.08 perf-profile.self.cycles-pp.tcp_update_recv_tstamps 0.06 +0.0 0.07 perf-profile.self.cycles-pp.tcp_urg 0.19 +0.0 0.20 ± 2% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.18 ± 3% +0.0 0.20 ± 3% perf-profile.self.cycles-pp.rcu_all_qs 0.18 ± 2% +0.0 0.20 ± 3% perf-profile.self.cycles-pp.check_stack_object 0.06 ± 9% +0.0 0.08 ± 4% perf-profile.self.cycles-pp.rcu_cblist_dequeue 0.22 ± 2% +0.0 0.25 perf-profile.self.cycles-pp.syscall_enter_from_user_mode 0.12 ± 5% +0.0 0.14 ± 3% perf-profile.self.cycles-pp.sock_rfree 0.09 ± 4% +0.0 0.12 perf-profile.self.cycles-pp.lock_sock_nested 0.06 ± 6% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.mod_objcg_state 0.11 ± 3% +0.0 0.14 ± 4% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.06 ± 9% +0.0 0.09 ± 4% perf-profile.self.cycles-pp.inet_sendmsg 0.06 +0.0 0.10 ± 5% perf-profile.self.cycles-pp.kmem_cache_alloc 1.04 +0.0 1.08 perf-profile.self.cycles-pp._raw_spin_lock_bh 0.03 ± 70% +0.0 0.07 perf-profile.self.cycles-pp._copy_from_user 0.26 ± 4% +0.0 0.30 ± 3% perf-profile.self.cycles-pp.tcp_established_options 0.05 +0.0 0.09 perf-profile.self.cycles-pp.rb_erase 0.14 ± 3% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.sockfd_lookup_light 0.09 ± 4% +0.0 0.13 ± 3% perf-profile.self.cycles-pp.tcp_rcv_space_adjust 0.02 ±141% +0.0 0.06 ± 7% perf-profile.self.cycles-pp.simple_copy_to_iter 0.43 ± 2% +0.0 0.48 perf-profile.self.cycles-pp.skb_release_data 0.14 ± 2% +0.1 0.19 ± 3% perf-profile.self.cycles-pp.tcp_send_mss 0.08 +0.1 0.13 ± 3% perf-profile.self.cycles-pp.security_socket_sendmsg 0.00 +0.1 0.05 ± 8% perf-profile.self.cycles-pp.memcmp 0.06 +0.1 0.12 ± 4% perf-profile.self.cycles-pp._raw_spin_trylock 0.09 ± 5% +0.1 0.15 ± 4% perf-profile.self.cycles-pp.tcp_stream_alloc_skb 0.01 ±223% +0.1 0.06 ± 7% perf-profile.self.cycles-pp.__d_lookup_rcu 0.06 +0.1 0.12 ± 4% perf-profile.self.cycles-pp.sk_page_frag_refill 0.01 ±223% +0.1 0.07 ± 5% perf-profile.self.cycles-pp.prep_compound_page 0.02 ±141% +0.1 0.08 perf-profile.self.cycles-pp.propagate_protected_usage 0.04 ± 44% +0.1 0.11 ± 3% perf-profile.self.cycles-pp.__alloc_pages 0.00 +0.1 0.07 ± 5% perf-profile.self.cycles-pp.get_page_from_freelist 1.00 +0.1 1.07 ± 3% perf-profile.self.cycles-pp.__fget_light 0.06 ± 6% +0.1 0.14 ± 3% perf-profile.self.cycles-pp.rb_insert_color 0.18 ± 5% +0.1 0.26 ± 8% perf-profile.self.cycles-pp.ipv4_mtu 0.00 +0.1 0.08 ± 6% perf-profile.self.cycles-pp.rmqueue 0.00 +0.1 0.08 ± 6% perf-profile.self.cycles-pp.___slab_alloc 0.00 +0.1 0.08 ± 4% perf-profile.self.cycles-pp.tcp_rcv_state_process 0.99 ± 3% +0.1 1.06 ± 2% perf-profile.self.cycles-pp.mem_cgroup_charge_skmem 0.52 +0.1 0.60 ± 2% perf-profile.self.cycles-pp.__virt_addr_valid 0.52 ± 2% +0.1 0.61 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret 0.00 +0.1 0.09 ± 4% perf-profile.self.cycles-pp.tcp_validate_incoming 0.08 ± 4% +0.1 0.17 ± 4% perf-profile.self.cycles-pp.tcp_wmem_schedule 0.00 +0.1 0.09 perf-profile.self.cycles-pp.__release_sock 0.10 ± 5% +0.1 0.20 ± 2% perf-profile.self.cycles-pp.free_unref_page_prepare 0.96 +0.1 1.06 perf-profile.self.cycles-pp.read_tsc 0.46 +0.1 0.57 ± 2% perf-profile.self.cycles-pp.__check_object_size 0.00 +0.1 0.10 ± 4% perf-profile.self.cycles-pp.tcp_stream_memory_free 0.19 +0.1 0.30 perf-profile.self.cycles-pp.__skb_clone 0.08 ± 5% +0.1 0.19 perf-profile.self.cycles-pp.__sk_mem_schedule 0.16 ± 2% +0.1 0.27 ± 2% perf-profile.self.cycles-pp._copy_to_iter 0.16 ± 4% +0.1 0.28 ± 2% perf-profile.self.cycles-pp.tcp_push 0.25 ± 6% +0.1 0.37 ± 3% perf-profile.self.cycles-pp.perf_adjust_freq_unthr_context 0.12 ± 4% +0.1 0.24 ± 3% perf-profile.self.cycles-pp.page_counter_uncharge 0.22 +0.1 0.34 ± 3% perf-profile.self.cycles-pp.tcp_current_mss 0.00 +0.1 0.12 ± 4% perf-profile.self.cycles-pp.tcp_try_coalesce 0.17 ± 4% +0.1 0.31 perf-profile.self.cycles-pp.page_counter_try_charge 0.89 ± 2% +0.1 1.04 perf-profile.self.cycles-pp.__sk_mem_raise_allocated 0.21 ± 2% +0.2 0.36 perf-profile.self.cycles-pp.tcp_tso_segs 0.05 +0.2 0.21 perf-profile.self.cycles-pp.tcp_add_backlog 0.17 +0.2 0.34 ± 2% perf-profile.self.cycles-pp.skb_do_copy_data_nocache 0.01 ±223% +0.2 0.19 perf-profile.self.cycles-pp.tcp_rbtree_insert 0.14 ± 3% +0.2 0.33 perf-profile.self.cycles-pp.tcp_skb_entail 0.45 ± 2% +0.2 0.65 ± 3% perf-profile.self.cycles-pp.__send 0.86 +0.2 1.05 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.50 +0.2 0.70 perf-profile.self.cycles-pp.__alloc_skb 0.56 ± 4% +0.2 0.77 perf-profile.self.cycles-pp.ktime_get 0.51 +0.2 0.72 perf-profile.self.cycles-pp.try_charge_memcg 0.92 +0.2 1.17 perf-profile.self.cycles-pp.__sys_sendto 0.27 +0.3 0.55 perf-profile.self.cycles-pp._copy_from_iter 0.10 ± 5% +0.3 0.42 ± 2% perf-profile.self.cycles-pp.skb_clone 0.50 +0.4 0.86 perf-profile.self.cycles-pp.skb_page_frag_refill 0.88 +0.4 1.26 perf-profile.self.cycles-pp.__dev_queue_xmit 0.42 ± 3% +0.4 0.82 ± 5% perf-profile.self.cycles-pp.__entry_text_start 0.00 +0.4 0.44 ± 2% perf-profile.self.cycles-pp.__slab_free 0.45 ± 2% +0.5 0.92 perf-profile.self.cycles-pp.__skb_datagram_iter 0.65 ± 2% +0.5 1.17 perf-profile.self.cycles-pp.check_heap_object 0.38 +0.7 1.04 perf-profile.self.cycles-pp.tcp_mtu_probe 1.38 +0.8 2.18 perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.08 ± 4% +0.8 0.88 perf-profile.self.cycles-pp.skb_try_coalesce 2.20 +1.4 3.62 perf-profile.self.cycles-pp.tcp_write_xmit 2.54 +4.1 6.62 perf-profile.self.cycles-pp.tcp_sendmsg_locked 5.10 +5.6 10.68 perf-profile.self.cycles-pp.copyin 6.67 ± 2% +8.9 15.54 perf-profile.self.cycles-pp.copyout *************************************************************************************************** lkp-csl-2sp3: 96 threads 2 sockets Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz (Cascade Lake) with 128G memory ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/testcase: gcc-12/performance/1SSD/btrfs/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp3/blogbench commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 6.695e+08 ± 7% +11.6% 7.468e+08 ± 6% cpuidle..time 2.75 ± 5% +15.0% 3.17 ± 5% iostat.cpu.idle 2.12 ± 7% +0.4 2.55 ± 7% mpstat.cpu.all.idle% 1.64 ± 6% -0.3 1.34 ± 8% mpstat.cpu.all.soft% 206451 ± 19% -28.6% 147387 ± 10% numa-meminfo.node1.Active(anon) 1438479 ± 23% -35.7% 925625 ± 38% numa-meminfo.node1.Inactive 7482580 ± 10% -22.8% 5778855 ± 14% numa-numastat.node1.local_node 7539650 ± 10% -22.9% 5815892 ± 14% numa-numastat.node1.numa_hit 521731 ± 9% +19.7% 624301 ± 7% turbostat.C6 1.98 ± 6% +16.0% 2.29 ± 5% turbostat.CPU%c1 51613 ± 19% -28.6% 36831 ± 10% numa-vmstat.node1.nr_active_anon 51613 ± 19% -28.6% 36831 ± 10% numa-vmstat.node1.nr_zone_active_anon 7539552 ± 10% -22.9% 5815411 ± 14% numa-vmstat.node1.numa_hit 7482483 ± 10% -22.8% 5778375 ± 14% numa-vmstat.node1.numa_local 87888 -28.9% 62448 ± 2% vmstat.io.bo 15888177 -19.9% 12718486 vmstat.memory.cache 30035 -49.7% 15097 ± 3% vmstat.system.cs 115535 -6.2% 108398 vmstat.system.in 2243545 ± 2% -4.1% 2152228 blogbench.read_score 52412617 -28.3% 37571769 blogbench.time.file_system_outputs 2682930 -74.1% 694136 blogbench.time.involuntary_context_switches 2369329 -50.0% 1184098 ± 5% blogbench.time.voluntary_context_switches 5851 -35.9% 3752 ± 2% blogbench.write_score 9556530 -23.7% 7288507 meminfo.Active 335058 ± 2% -11.0% 298303 meminfo.Active(anon) 9221471 -24.2% 6990203 meminfo.Active(file) 14822026 -19.8% 11882120 meminfo.Cached 2813168 ± 2% -23.9% 2141166 ± 2% meminfo.Inactive 2495481 ± 2% -26.9% 1823137 ± 3% meminfo.Inactive(file) 1027330 -21.5% 806837 meminfo.KReclaimable 17291320 -18.7% 14055876 meminfo.Memused 1027330 -21.5% 806837 meminfo.SReclaimable 567705 -11.8% 500491 meminfo.SUnreclaim 347747 ± 2% -10.4% 311626 meminfo.Shmem 1595036 -18.0% 1307328 meminfo.Slab 17296989 -18.7% 14063076 meminfo.max_used_kB 83751 ± 2% -10.9% 74593 proc-vmstat.nr_active_anon 2305037 -24.2% 1747654 proc-vmstat.nr_active_file 6910506 -28.1% 4965836 proc-vmstat.nr_dirtied 3705052 -19.8% 2970624 proc-vmstat.nr_file_pages 28598624 +2.8% 29406900 proc-vmstat.nr_free_pages 623755 ± 2% -26.9% 455767 ± 3% proc-vmstat.nr_inactive_file 86928 ± 2% -10.4% 77904 proc-vmstat.nr_shmem 256791 -21.4% 201717 proc-vmstat.nr_slab_reclaimable 141880 -11.9% 125045 proc-vmstat.nr_slab_unreclaimable 6683552 -29.1% 4740612 ± 2% proc-vmstat.nr_written 83750 ± 2% -10.9% 74593 proc-vmstat.nr_zone_active_anon 2305037 -24.2% 1747654 proc-vmstat.nr_zone_active_file 623755 ± 2% -26.9% 455767 ± 3% proc-vmstat.nr_zone_inactive_file 10239 ± 31% -43.9% 5745 ± 8% proc-vmstat.numa_hint_faults 14790318 -18.1% 12115224 proc-vmstat.numa_hit 14690001 -18.2% 12015656 proc-vmstat.numa_local 5041659 -27.1% 3676364 proc-vmstat.pgactivate 22801122 -16.1% 19140499 proc-vmstat.pgalloc_normal 941255 -2.0% 922358 proc-vmstat.pgfault 16727716 ± 2% -10.1% 15042005 ± 2% proc-vmstat.pgfree 26688664 -29.1% 18932460 ± 2% proc-vmstat.pgpgout 0.34 ± 4% +26.3% 0.43 ± 16% sched_debug.cfs_rq:/.h_nr_running.stddev 143848 ± 73% +503.5% 868094 ± 46% sched_debug.cfs_rq:/.left_vruntime.avg 5267401 ± 54% +140.4% 12664767 ± 15% sched_debug.cfs_rq:/.left_vruntime.max 775016 ± 57% +264.4% 2824325 ± 29% sched_debug.cfs_rq:/.left_vruntime.stddev 19797 ± 25% +273.1% 73865 ± 46% sched_debug.cfs_rq:/.load.avg 432533 ± 44% +248.2% 1506268 ± 38% sched_debug.cfs_rq:/.load.max 59670 ± 38% +295.8% 236179 ± 38% sched_debug.cfs_rq:/.load.stddev 1.36 ± 10% +73.5% 2.36 ± 24% sched_debug.cfs_rq:/.nr_running.max 0.12 ± 19% +131.1% 0.28 ± 26% sched_debug.cfs_rq:/.nr_running.stddev 143848 ± 73% +503.5% 868094 ± 46% sched_debug.cfs_rq:/.right_vruntime.avg 5267401 ± 54% +140.4% 12664767 ± 15% sched_debug.cfs_rq:/.right_vruntime.max 775016 ± 57% +264.4% 2824326 ± 29% sched_debug.cfs_rq:/.right_vruntime.stddev 2050 ± 5% +13.3% 2324 ± 7% sched_debug.cfs_rq:/.runnable_avg.max 144.28 ± 52% -60.9% 56.44 ± 45% sched_debug.cfs_rq:/.util_est_enqueued.min 715372 ± 5% +18.2% 845717 sched_debug.cpu.avg_idle.avg 1295617 ± 9% -14.2% 1112152 ± 4% sched_debug.cpu.avg_idle.max 75779 ± 21% +81.8% 137779 ± 26% sched_debug.cpu.avg_idle.min 305540 ± 10% -31.2% 210201 ± 4% sched_debug.cpu.avg_idle.stddev 39563 ± 64% -65.5% 13646 ± 54% sched_debug.cpu.max_idle_balance_cost.stddev 55267 ± 2% -39.7% 33351 ± 4% sched_debug.cpu.nr_switches.avg 117367 ± 8% -32.0% 79860 ± 6% sched_debug.cpu.nr_switches.max 26070 ± 4% -40.7% 15457 ± 9% sched_debug.cpu.nr_switches.min 21097 ± 7% -34.0% 13918 ± 13% sched_debug.cpu.nr_switches.stddev 0.11 ± 7% +14.2% 0.13 ± 9% sched_debug.cpu.nr_uninterruptible.avg 76.11 +1.5 77.58 perf-stat.i.cache-miss-rate% 4.955e+08 -4.6% 4.728e+08 ± 2% perf-stat.i.cache-references 30207 -47.5% 15845 ± 4% perf-stat.i.context-switches 1977 ± 4% -18.7% 1606 ± 2% perf-stat.i.cpu-migrations 7.754e+09 -1.4% 7.649e+09 perf-stat.i.dTLB-loads 1.822e+09 ± 2% -4.0% 1.749e+09 perf-stat.i.dTLB-stores 3.046e+10 -1.3% 3.005e+10 perf-stat.i.instructions 177.30 -1.6% 174.44 perf-stat.i.metric.M/sec 2470 -1.7% 2428 perf-stat.i.minor-faults 68.87 -1.9 66.92 perf-stat.i.node-load-miss-rate% 83930000 ± 2% -5.8% 79071089 perf-stat.i.node-load-misses 4258826 ± 2% -4.9% 4049629 perf-stat.i.node-stores 2470 -1.7% 2428 perf-stat.i.page-faults 76.23 +1.5 77.74 perf-stat.overall.cache-miss-rate% 68.87 -2.0 66.89 perf-stat.overall.node-load-miss-rate% 4.937e+08 -4.6% 4.711e+08 ± 2% perf-stat.ps.cache-references 30138 -50.3% 14978 ± 4% perf-stat.ps.context-switches 1972 ± 4% -19.2% 1594 ± 2% perf-stat.ps.cpu-migrations 7.725e+09 -1.4% 7.618e+09 perf-stat.ps.dTLB-loads 1.816e+09 ± 2% -4.2% 1.74e+09 perf-stat.ps.dTLB-stores 3.034e+10 -1.4% 2.993e+10 perf-stat.ps.instructions 2464 -2.6% 2402 perf-stat.ps.minor-faults 83612770 ± 2% -5.7% 78809861 perf-stat.ps.node-load-misses 4243163 ± 2% -4.9% 4033510 perf-stat.ps.node-stores 2465 -2.6% 2402 perf-stat.ps.page-faults 9.141e+12 -1.4% 9.017e+12 perf-stat.total.instructions 2.10 ± 5% -1.2 0.86 ± 19% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 2.10 ± 5% -1.2 0.86 ± 19% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 2.10 ± 5% -1.2 0.86 ± 19% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 1.82 ± 6% -1.1 0.69 ± 17% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.82 ± 7% -1.1 0.68 ± 17% perf-profile.calltrace.cycles-pp.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork 1.82 ± 7% -1.1 0.68 ± 17% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.54 ± 6% -1.1 0.42 ± 72% perf-profile.calltrace.cycles-pp.file_free_rcu.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd 1.81 ± 7% -1.1 0.68 ± 17% perf-profile.calltrace.cycles-pp.rcu_core.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread 1.81 ± 7% -1.1 0.68 ± 17% perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd.smpboot_thread_fn 7.52 -0.2 7.28 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read 7.50 -0.2 7.27 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read 7.62 -0.2 7.39 perf-profile.calltrace.cycles-pp.__libc_read 6.98 -0.2 6.77 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read 6.41 -0.2 6.21 perf-profile.calltrace.cycles-pp.filemap_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.74 -0.2 6.54 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read 5.23 -0.2 5.06 perf-profile.calltrace.cycles-pp._copy_to_iter.copy_page_to_iter.filemap_read.vfs_read.ksys_read 4.92 -0.2 4.75 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.copy_page_to_iter.filemap_read.vfs_read 5.34 -0.2 5.17 perf-profile.calltrace.cycles-pp.copy_page_to_iter.filemap_read.vfs_read.ksys_read.do_syscall_64 4.79 -0.2 4.63 perf-profile.calltrace.cycles-pp.rep_movs_alternative.copyout._copy_to_iter.copy_page_to_iter.filemap_read 0.00 +0.5 0.54 ± 2% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.alloc_fd 0.00 +0.6 0.55 ± 2% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.alloc_fd.do_sys_openat2 37.34 +0.8 38.13 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 37.41 +0.8 38.20 perf-profile.calltrace.cycles-pp.__close 37.35 +0.8 38.14 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close 37.13 +0.8 37.92 perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 36.51 +0.8 37.31 perf-profile.calltrace.cycles-pp.close_fd_get_file.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 36.21 +0.8 37.01 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.close_fd_get_file.__x64_sys_close.do_syscall_64 36.39 +0.8 37.20 perf-profile.calltrace.cycles-pp._raw_spin_lock.close_fd_get_file.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.10 ± 5% -1.2 0.86 ± 19% perf-profile.children.cycles-pp.ret_from_fork_asm 2.10 ± 5% -1.2 0.86 ± 19% perf-profile.children.cycles-pp.ret_from_fork 2.10 ± 5% -1.2 0.86 ± 19% perf-profile.children.cycles-pp.kthread 1.82 ± 6% -1.1 0.69 ± 17% perf-profile.children.cycles-pp.smpboot_thread_fn 1.82 ± 7% -1.1 0.68 ± 17% perf-profile.children.cycles-pp.run_ksoftirqd 2.07 ± 6% -0.5 1.59 ± 8% perf-profile.children.cycles-pp.rcu_do_batch 2.08 ± 6% -0.5 1.60 ± 8% perf-profile.children.cycles-pp.rcu_core 2.09 ± 6% -0.5 1.61 ± 8% perf-profile.children.cycles-pp.__do_softirq 1.76 ± 6% -0.4 1.35 ± 9% perf-profile.children.cycles-pp.file_free_rcu 7.64 -0.2 7.42 perf-profile.children.cycles-pp.__libc_read 6.99 -0.2 6.78 perf-profile.children.cycles-pp.ksys_read 6.42 -0.2 6.21 perf-profile.children.cycles-pp.filemap_read 6.74 -0.2 6.55 perf-profile.children.cycles-pp.vfs_read 5.34 -0.2 5.18 perf-profile.children.cycles-pp.copy_page_to_iter 4.82 -0.2 4.65 perf-profile.children.cycles-pp.rep_movs_alternative 5.30 -0.2 5.13 perf-profile.children.cycles-pp._copy_to_iter 4.94 -0.2 4.78 perf-profile.children.cycles-pp.copyout 0.27 ± 4% -0.1 0.17 ± 35% perf-profile.children.cycles-pp.process_one_work 0.27 ± 4% -0.1 0.17 ± 36% perf-profile.children.cycles-pp.worker_thread 0.18 ± 8% -0.1 0.10 ± 10% perf-profile.children.cycles-pp.btrfs_search_slot 0.23 ± 6% -0.1 0.16 ± 15% perf-profile.children.cycles-pp.rename 0.22 ± 7% -0.1 0.16 ± 16% perf-profile.children.cycles-pp.do_renameat2 0.22 ± 8% -0.1 0.16 ± 15% perf-profile.children.cycles-pp.__x64_sys_rename 0.15 ± 8% -0.1 0.09 ± 24% perf-profile.children.cycles-pp.btrfs_work_helper 0.15 ± 8% -0.0 0.11 ± 15% perf-profile.children.cycles-pp.btrfs_rename2 0.14 ± 8% -0.0 0.10 ± 16% perf-profile.children.cycles-pp.__libc_write 0.14 ± 8% -0.0 0.09 ± 17% perf-profile.children.cycles-pp.vfs_write 0.16 ± 6% -0.0 0.12 ± 16% perf-profile.children.cycles-pp.vfs_rename 0.14 ± 6% -0.0 0.10 ± 12% perf-profile.children.cycles-pp.ksys_write 0.12 ± 6% -0.0 0.08 ± 13% perf-profile.children.cycles-pp.btrfs_do_write_iter 0.12 ± 6% -0.0 0.08 ± 13% perf-profile.children.cycles-pp.btrfs_buffered_write 0.07 ± 9% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.lookup_open 0.12 ± 5% -0.0 0.09 ± 15% perf-profile.children.cycles-pp.btrfs_rename 0.10 ± 6% -0.0 0.07 ± 12% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited_flags 0.46 -0.0 0.44 ± 2% perf-profile.children.cycles-pp.open_last_lookups 0.09 ± 7% -0.0 0.07 ± 9% perf-profile.children.cycles-pp.dput 0.23 ± 29% +0.1 0.37 ± 27% perf-profile.children.cycles-pp.start_secondary 0.23 ± 29% +0.1 0.37 ± 28% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 0.23 ± 29% +0.1 0.37 ± 28% perf-profile.children.cycles-pp.cpu_startup_entry 0.23 ± 29% +0.1 0.37 ± 28% perf-profile.children.cycles-pp.do_idle 0.23 ± 29% +0.1 0.37 ± 28% perf-profile.children.cycles-pp.cpuidle_idle_call 0.23 ± 29% +0.1 0.37 ± 28% perf-profile.children.cycles-pp.cpuidle_enter 0.23 ± 29% +0.1 0.37 ± 28% perf-profile.children.cycles-pp.cpuidle_enter_state 0.21 ± 28% +0.1 0.36 ± 27% perf-profile.children.cycles-pp.intel_idle 0.28 ± 4% +0.7 0.94 ± 3% perf-profile.children.cycles-pp.__irq_exit_rcu 0.65 ± 3% +0.7 1.30 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.62 ± 2% +0.7 1.28 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 37.42 +0.8 38.21 perf-profile.children.cycles-pp.__close 37.13 +0.8 37.92 perf-profile.children.cycles-pp.__x64_sys_close 36.51 +0.8 37.31 perf-profile.children.cycles-pp.close_fd_get_file 97.16 +1.1 98.25 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 97.12 +1.1 98.22 perf-profile.children.cycles-pp.do_syscall_64 84.64 +1.6 86.20 perf-profile.children.cycles-pp._raw_spin_lock 84.20 +1.6 85.77 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.76 ± 6% -0.4 1.35 ± 9% perf-profile.self.cycles-pp.file_free_rcu 4.77 -0.2 4.58 perf-profile.self.cycles-pp.rep_movs_alternative 0.21 ± 28% +0.1 0.36 ± 27% perf-profile.self.cycles-pp.intel_idle 83.71 +1.0 84.66 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.26 ± 55% +300.3% 1.03 ± 29% perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages.folio_alloc.__filemap_get_folio.pagecache_get_page 0.01 ± 12% +563.8% 0.09 ± 31% perf-sched.sch_delay.avg.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.20 ± 73% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.btrfs_wq_run_delayed_node.btrfs_btree_balance_dirty 0.00 ±223% +9900.0% 0.02 ± 44% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.03 ± 49% +4632.4% 1.48 ±109% perf-sched.sch_delay.avg.ms.__cond_resched.btrfs_alloc_path.__btrfs_unlink_inode.btrfs_rename.btrfs_rename2 0.44 ±210% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.btrfs_alloc_path.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work 0.02 ± 45% +3572.2% 0.81 ± 55% perf-sched.sch_delay.avg.ms.__cond_resched.btrfs_alloc_path.btrfs_insert_dir_item.btrfs_add_link.btrfs_create_new_inode 0.04 ± 44% -93.6% 0.00 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.btrfs_alloc_path.btrfs_lookup_dentry.btrfs_lookup.lookup_one_qstr_excl 0.01 ±159% +4251.0% 0.36 ±166% perf-sched.sch_delay.avg.ms.__cond_resched.btrfs_search_slot.btrfs_del_csums.cleanup_ref_head.__btrfs_run_delayed_refs 0.34 ±124% -91.3% 0.03 ±144% perf-sched.sch_delay.avg.ms.__cond_resched.down_read.find_free_extent.btrfs_reserve_extent.btrfs_alloc_tree_block 0.09 ±107% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_delete_delayed_items 0.05 ± 90% -98.1% 0.00 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_file_extent 0.38 ± 38% -81.6% 0.07 ±137% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_truncate_inode_items 0.36 ±124% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 0.10 ± 37% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.dput.step_into.open_last_lookups.path_openat 0.01 ±116% +381.8% 0.03 ± 77% perf-sched.sch_delay.avg.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open 0.42 ± 40% +365.1% 1.96 ±110% perf-sched.sch_delay.avg.ms.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 0.02 ± 24% +375.8% 0.08 ± 29% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 0.16 ± 75% +1051.5% 1.85 ± 78% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.alloc_extent_state.__clear_extent_bit.btrfs_dirty_pages 0.00 ± 91% +1495.2% 0.06 ±101% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.security_file_alloc.init_file.alloc_empty_file 0.15 ± 49% +395.9% 0.75 ± 65% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.start_transaction.btrfs_create_common.lookup_open 0.08 ±182% +1182.8% 1.04 ±129% perf-sched.sch_delay.avg.ms.__cond_resched.mnt_want_write.open_last_lookups.path_openat.do_filp_open 0.08 ± 28% +1245.8% 1.02 ± 55% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.__btrfs_release_delayed_node.part.0 0.14 ± 44% +1386.6% 2.03 ± 85% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.btrfs_delayed_delete_inode_ref.__btrfs_unlink_inode.btrfs_rename 0.01 ± 47% -94.0% 0.00 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.btrfs_delete_delayed_items.btrfs_async_run_delayed_root.btrfs_work_helper 0.03 ± 54% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.workqueue_set_max_active.btrfs_work_helper.process_one_work 0.01 ± 42% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.start_transaction.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work 0.78 ± 61% +153.0% 1.97 ± 21% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.10 ± 33% +114.8% 0.23 ± 72% perf-sched.sch_delay.avg.ms.btrfs_balance_delayed_items.btrfs_btree_balance_dirty.btrfs_rename2.vfs_rename 0.09 ± 39% +266.4% 0.34 ± 25% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.01 ±101% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single 0.10 ± 83% +392.8% 0.49 ± 28% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.49 ± 65% +740.0% 4.12 ±130% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.16 ± 38% +288.6% 0.63 ± 24% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_async_run_delayed_root 0.21 ± 11% +152.5% 0.52 ± 13% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_delete_delayed_dir_index 0.12 ± 75% +328.8% 0.52 ± 48% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_delete_delayed_items 0.13 ± 28% +95.9% 0.26 ± 30% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_insert_delayed_dir_index 0.10 ± 7% +290.3% 0.38 ± 23% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.__btrfs_tree_read_lock 0.07 ± 5% +240.2% 0.24 ± 37% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.__btrfs_tree_lock 0.25 ± 24% +123.8% 0.56 ± 33% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.do_renameat2 0.30 ± 18% +147.2% 0.75 ± 23% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.open_last_lookups 0.29 ± 19% +112.6% 0.61 ± 30% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.02 ± 39% +9932.6% 2.41 ± 32% perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq 0.15 ± 85% +999.8% 1.65 ± 14% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.07 ± 16% +1038.8% 0.77 ± 23% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.17 ± 13% +1740.5% 3.10 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.09 ± 26% +787.9% 0.78 ± 11% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 9.30 ± 80% +605.4% 65.59 ± 42% perf-sched.sch_delay.max.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.66 ±105% -95.1% 0.13 ±151% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.btrfs_alloc_delayed_item.btrfs_delete_delayed_dir_index 1.42 ± 68% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.btrfs_wq_run_delayed_node.btrfs_btree_balance_dirty 0.12 ±167% +2268.0% 2.85 ± 41% perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 9.24 ±215% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.btrfs_alloc_path.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work 5.22 ± 41% -50.0% 2.61 ± 47% perf-sched.sch_delay.max.ms.__cond_resched.btrfs_alloc_path.btrfs_del_inode_ref.__btrfs_unlink_inode.btrfs_rename 0.69 ±108% +342.1% 3.07 ± 32% perf-sched.sch_delay.max.ms.__cond_resched.btrfs_alloc_path.btrfs_insert_dir_item.btrfs_add_link.btrfs_create_new_inode 0.13 ± 27% -96.2% 0.00 ±223% perf-sched.sch_delay.max.ms.__cond_resched.btrfs_alloc_path.btrfs_lookup_dentry.btrfs_lookup.lookup_one_qstr_excl 0.03 ±169% +2465.1% 0.71 ±166% perf-sched.sch_delay.max.ms.__cond_resched.btrfs_search_slot.btrfs_del_csums.cleanup_ref_head.__btrfs_run_delayed_refs 21.47 ± 83% -82.7% 3.72 ± 54% perf-sched.sch_delay.max.ms.__cond_resched.down_read.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 10.14 ±138% -92.1% 0.81 ±165% perf-sched.sch_delay.max.ms.__cond_resched.down_read.__btrfs_tree_read_lock.btrfs_search_slot.btrfs_lookup_dir_item 1.24 ±133% -96.8% 0.04 ±143% perf-sched.sch_delay.max.ms.__cond_resched.down_read.find_free_extent.btrfs_reserve_extent.btrfs_alloc_tree_block 6.24 ± 49% -70.5% 1.84 ± 95% perf-sched.sch_delay.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot 8.65 ± 19% -75.8% 2.09 ± 92% perf-sched.sch_delay.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_del_inode_ref 5.30 ± 59% -76.0% 1.28 ±140% perf-sched.sch_delay.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_del_orphan_item 2.06 ±138% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_delete_delayed_items 7.27 ± 27% -67.4% 2.37 ± 71% perf-sched.sch_delay.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_dir_item 1.58 ±101% -99.9% 0.00 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_file_extent 5.08 ± 20% -85.9% 0.72 ±180% perf-sched.sch_delay.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_inode 5.66 ± 39% -98.6% 0.08 ±139% perf-sched.sch_delay.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_truncate_inode_items 5.43 ±173% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 1.90 ± 61% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.dput.step_into.open_last_lookups.path_openat 0.12 ± 59% -68.8% 0.04 ±126% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.btrfs_add_delayed_tree_ref.btrfs_free_tree_block.__btrfs_cow_block 0.09 ± 92% -98.5% 0.00 ±223% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.btrfs_delete_delayed_items.btrfs_async_run_delayed_root.btrfs_work_helper 0.10 ±112% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.workqueue_set_max_active.btrfs_work_helper.process_one_work 0.02 ± 58% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.start_transaction.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work 1.18 ±128% -97.1% 0.03 ±150% perf-sched.sch_delay.max.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.04 ± 33% +292.7% 11.95 ±131% perf-sched.sch_delay.max.ms.btrfs_balance_delayed_items.btrfs_btree_balance_dirty.btrfs_create_common.lookup_open.isra 2.65 ± 31% +69.4% 4.49 ± 22% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 2.22 ± 23% +115.0% 4.78 ± 48% perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.73 ±107% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single 0.96 ± 84% -99.6% 0.00 ±223% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 48.23 ± 79% +760.5% 415.00 ± 80% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.__btrfs_tree_read_lock 4.33 ± 21% +293.8% 17.05 ± 88% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.open_last_lookups 14.68 ± 51% +622.8% 106.09 ± 65% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.__btrfs_tree_lock 4.13 ± 8% +161.6% 10.80 ± 78% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.open_last_lookups 0.03 ± 40% +11011.0% 3.20 ± 27% perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq 1.57 ± 83% +157.5% 4.04 perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.08 +763.2% 0.73 ± 11% perf-sched.total_sch_delay.average.ms 96.59 ±108% +725.2% 797.08 ± 40% perf-sched.total_sch_delay.max.ms 15.18 ± 5% +108.2% 31.60 ± 8% perf-sched.total_wait_and_delay.average.ms 109382 ± 5% -51.4% 53178 ± 6% perf-sched.total_wait_and_delay.count.ms 15.09 ± 5% +104.5% 30.87 ± 8% perf-sched.total_wait_time.average.ms 1.45 ± 34% +171.3% 3.94 ± 31% perf-sched.wait_and_delay.avg.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.59 ±100% +171.3% 9.75 ± 7% perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 2.23 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.down_read.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 1.62 ± 41% -91.5% 0.14 ±223% perf-sched.wait_and_delay.avg.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open 49.97 ± 54% +513.1% 306.37 ± 70% perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 1.56 ± 22% +184.1% 4.42 ± 25% perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 346.80 ± 15% +135.0% 814.99 ± 3% perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 52.56 ± 54% +97.8% 103.98 ± 10% perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 237.56 ± 3% +9.8% 260.81 ± 6% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 178.06 ± 3% +86.0% 331.14 ± 8% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 328.16 ± 4% +10.0% 360.96 ± 2% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 2.55 ± 11% +156.3% 6.54 ± 17% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.__btrfs_tree_read_lock 1.43 ± 11% +123.8% 3.19 ± 27% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.__btrfs_tree_lock 4.48 +30.7% 5.85 ± 5% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 59.30 ± 4% +61.0% 95.49 ± 8% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 9337 ± 11% -64.6% 3303 ± 6% perf-sched.wait_and_delay.count.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 1708 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.down_read.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 755.17 ± 5% -94.1% 44.67 ±223% perf-sched.wait_and_delay.count.__cond_resched.dput.terminate_walk.path_openat.do_filp_open 17.67 ± 32% -71.7% 5.00 ± 62% perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 11011 ± 12% -64.9% 3864 ± 7% perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 17.83 ± 7% -95.3% 0.83 ±107% perf-sched.wait_and_delay.count.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 105.83 ± 20% -98.9% 1.17 ± 76% perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 4522 ± 14% -74.5% 1152 ± 6% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 75.67 ± 4% -45.4% 41.33 ± 4% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 42893 ± 11% -50.9% 21042 ± 12% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.__btrfs_tree_read_lock 12748 ± 15% -52.9% 6008 ± 11% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.__btrfs_tree_lock 1126 -11.8% 993.33 ± 2% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 7303 ± 5% -37.1% 4592 ± 6% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 39.77 ± 84% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.down_read.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 672.58 ± 69% -99.6% 2.81 ±223% perf-sched.wait_and_delay.max.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open 350.82 ± 66% -86.6% 47.13 ±100% perf-sched.wait_and_delay.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 385.08 ± 13% -80.5% 74.94 ± 72% perf-sched.wait_and_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1077 ± 6% +31.8% 1419 ± 18% perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 1084 ± 6% +68.8% 1830 ± 30% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.42 ± 26% +228.5% 4.68 ± 44% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.folio_alloc.__filemap_get_folio.pagecache_get_page 1.44 ± 35% +167.7% 3.85 ± 32% perf-sched.wait_time.avg.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.11 ± 9% +267.3% 4.06 ± 38% perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.btrfs_alloc_delayed_item.btrfs_insert_delayed_dir_index 1.24 ± 41% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.btrfs_wq_run_delayed_node.btrfs_btree_balance_dirty 6.88 ± 5% +41.5% 9.73 ± 7% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 19.67 ± 44% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.btrfs_alloc_path.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work 1.11 ± 87% -98.6% 0.02 ±223% perf-sched.wait_time.avg.ms.__cond_resched.btrfs_alloc_path.btrfs_csum_file_blocks.btrfs_finish_one_ordered.btrfs_work_helper 0.62 ± 24% +738.4% 5.21 ± 30% perf-sched.wait_time.avg.ms.__cond_resched.btrfs_alloc_path.btrfs_del_inode_ref.__btrfs_unlink_inode.btrfs_rename 0.60 ± 14% +599.5% 4.19 ± 69% perf-sched.wait_time.avg.ms.__cond_resched.btrfs_alloc_path.btrfs_insert_dir_item.btrfs_add_link.btrfs_create_new_inode 1.03 ±149% -97.4% 0.03 ±169% perf-sched.wait_time.avg.ms.__cond_resched.btrfs_alloc_path.btrfs_lookup_dentry.btrfs_lookup.lookup_one_qstr_excl 0.68 ± 93% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.btrfs_search_slot.btrfs_del_inode_ref.__btrfs_unlink_inode.btrfs_rename 1.36 ± 72% -90.9% 0.12 ±156% perf-sched.wait_time.avg.ms.__cond_resched.btrfs_search_slot.btrfs_lookup_dir_item.__btrfs_unlink_inode.btrfs_rename 1.90 ± 9% +269.8% 7.02 ± 70% perf-sched.wait_time.avg.ms.__cond_resched.down_read.__btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot 0.92 ± 8% +366.3% 4.31 ± 36% perf-sched.wait_time.avg.ms.__cond_resched.down_read.__btrfs_tree_read_lock.btrfs_search_slot.btrfs_lookup_dir_item 2.00 ± 41% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_delete_delayed_items 1.03 ± 11% +227.6% 3.36 ± 42% perf-sched.wait_time.avg.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_insert_empty_items 0.94 ± 39% +284.1% 3.61 ± 49% perf-sched.wait_time.avg.ms.__cond_resched.dput.open_last_lookups.path_openat.do_filp_open 1.27 ± 27% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.dput.step_into.open_last_lookups.path_openat 1.36 ± 24% +325.9% 5.79 ± 59% perf-sched.wait_time.avg.ms.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 49.97 ± 54% +513.1% 306.37 ± 70% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 1.54 ± 22% +182.1% 4.34 ± 26% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 1.31 ± 34% +218.6% 4.18 ± 43% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.alloc_extent_state.__clear_extent_bit.btrfs_dirty_pages 59.90 ± 50% -91.6% 5.04 ± 53% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.alloc_extent_state.__set_extent_bit.lock_extent 1.25 ± 24% +260.1% 4.52 ± 32% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.start_transaction.btrfs_create_common.lookup_open 0.80 ± 19% +740.1% 6.72 ± 37% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.start_transaction.evict_refill_and_join.btrfs_evict_inode 0.76 ± 77% +525.0% 4.74 ± 55% perf-sched.wait_time.avg.ms.__cond_resched.mnt_want_write.do_renameat2.__x64_sys_rename.do_syscall_64 1.08 ± 11% +300.0% 4.32 ± 26% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.__btrfs_release_delayed_node.part.0 0.92 ± 17% +395.3% 4.57 ± 49% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.btrfs_delayed_delete_inode_ref.__btrfs_unlink_inode.btrfs_rename 22.56 ± 54% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.workqueue_set_max_active.btrfs_work_helper.process_one_work 2.71 ± 69% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.start_transaction.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work 1.08 ± 47% +4057.6% 44.81 ±207% perf-sched.wait_time.avg.ms.__cond_resched.truncate_inode_pages_range.btrfs_evict_inode.evict.__dentry_kill 6.94 ±135% -93.9% 0.42 ±188% perf-sched.wait_time.avg.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 346.02 ± 15% +135.0% 813.02 ± 3% perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.76 ± 29% +123.7% 1.70 ± 65% perf-sched.wait_time.avg.ms.btrfs_balance_delayed_items.btrfs_btree_balance_dirty.btrfs_create_common.lookup_open.isra 0.71 ± 21% +169.6% 1.91 ± 64% perf-sched.wait_time.avg.ms.btrfs_balance_delayed_items.btrfs_btree_balance_dirty.btrfs_rename2.vfs_rename 52.62 ± 53% +96.9% 103.64 ± 10% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 237.10 ± 3% +8.6% 257.41 ± 5% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.05 ± 69% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_common_interrupt 0.24 ±121% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 3.51 ±123% -100.0% 0.00 ±223% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single 177.96 ± 3% +85.8% 330.65 ± 8% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 327.67 ± 3% +8.9% 356.84 perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.83 ± 20% +147.4% 2.04 ± 18% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.__btrfs_release_delayed_node 2.45 ± 30% +174.6% 6.73 ± 27% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_async_run_delayed_root 1.47 ± 12% +136.3% 3.47 ± 17% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_delayed_update_inode 1.39 ± 13% +126.7% 3.16 ± 21% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_delete_delayed_dir_index 1.06 ± 21% +167.0% 2.83 ± 25% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_delete_delayed_items 1.44 ± 19% +123.9% 3.22 ± 20% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_insert_delayed_dir_index 2.45 ± 12% +151.0% 6.15 ± 17% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.__btrfs_tree_read_lock 1.36 ± 12% +117.8% 2.96 ± 26% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.__btrfs_tree_lock 1.51 ± 38% +99.2% 3.02 ± 31% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.do_renameat2 1.54 ± 13% +154.8% 3.91 ± 24% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.open_last_lookups 4.41 +15.2% 5.08 ± 3% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 59.22 ± 4% +59.9% 94.70 ± 8% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 18.06 ± 56% -92.2% 1.41 ±141% perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.btrfs_alloc_delayed_item.btrfs_delete_delayed_dir_index 7.21 ± 73% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.btrfs_wq_run_delayed_node.btrfs_btree_balance_dirty 6.08 ± 54% -60.5% 2.40 ± 53% perf-sched.wait_time.max.ms.__cond_resched.apparmor_file_alloc_security.security_file_alloc.init_file.alloc_empty_file 127.33 ± 76% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.btrfs_alloc_path.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work 5.83 ±169% -99.7% 0.02 ±223% perf-sched.wait_time.max.ms.__cond_resched.btrfs_alloc_path.btrfs_csum_file_blocks.btrfs_finish_one_ordered.btrfs_work_helper 2.04 ±149% -98.5% 0.03 ±174% perf-sched.wait_time.max.ms.__cond_resched.btrfs_alloc_path.btrfs_lookup_dentry.btrfs_lookup.lookup_one_qstr_excl 6.35 ± 32% -41.9% 3.69 ± 46% perf-sched.wait_time.max.ms.__cond_resched.btrfs_buffered_write.btrfs_do_write_iter.vfs_write.ksys_write 3.72 ±106% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.btrfs_search_slot.btrfs_del_inode_ref.__btrfs_unlink_inode.btrfs_rename 5.15 ± 64% -97.6% 0.12 ±156% perf-sched.wait_time.max.ms.__cond_resched.btrfs_search_slot.btrfs_lookup_dir_item.__btrfs_unlink_inode.btrfs_rename 16.69 ±103% -81.6% 3.08 ±114% perf-sched.wait_time.max.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats 14.90 ± 32% -82.2% 2.65 ±140% perf-sched.wait_time.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_del_orphan_item 12.39 ± 56% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_delete_delayed_items 33.98 ±105% -76.7% 7.93 ± 86% perf-sched.wait_time.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_dir_item 14.96 ± 33% -87.3% 1.91 ±223% perf-sched.wait_time.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_file_extent 22.42 ± 48% -91.6% 1.89 ±113% perf-sched.wait_time.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_lookup_inode 13.01 ± 24% -71.3% 3.74 ±101% perf-sched.wait_time.max.ms.__cond_resched.down_write.__btrfs_tree_lock.btrfs_search_slot.btrfs_truncate_inode_items 14.14 ± 68% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.dput.step_into.open_last_lookups.path_openat 9.34 ± 32% +332.2% 40.36 ±116% perf-sched.wait_time.max.ms.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 942.33 ± 53% -98.8% 10.95 ± 52% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.alloc_extent_state.__set_extent_bit.lock_extent 5.66 ± 60% -86.9% 0.74 ±215% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.btrfs_get_or_create_delayed_node.btrfs_delayed_update_inode.btrfs_update_inode 350.82 ± 66% -86.6% 47.13 ±100% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 40.53 ± 44% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.workqueue_set_max_active.btrfs_work_helper.process_one_work 385.03 ± 13% -81.0% 73.31 ± 72% perf-sched.wait_time.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 7.13 ± 83% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.start_transaction.btrfs_async_run_delayed_root.btrfs_work_helper.process_one_work 49.86 ±156% -99.1% 0.42 ±188% perf-sched.wait_time.max.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 9.87 ± 36% +191.4% 28.77 ± 43% perf-sched.wait_time.max.ms.btrfs_balance_delayed_items.btrfs_dirty_inode.touch_atime.filemap_read 0.36 ± 86% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_common_interrupt 0.61 ±108% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 345.76 ±134% -100.0% 0.00 ±223% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single 677.96 ± 68% -99.8% 1.30 ±223% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 1076 ± 6% +20.7% 1300 perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 28.80 ± 35% +213.0% 90.13 ± 52% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_async_run_delayed_root 7.71 ± 35% +83.8% 14.17 ± 29% perf-sched.wait_time.max.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.btrfs_delayed_update_inode 1084 ± 6% +68.8% 1830 ± 30% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm *************************************************************************************************** lkp-cpl-4sp2: 224 threads 4 sockets Intel(R) Xeon(R) Platinum 8380H CPU @ 2.90GHz (Cooper Lake) with 192G memory ========================================================================================= compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase: gcc-12/performance/socket/4/x86_64-rhel-8.3/threads/50%/debian-11.1-x86_64-20220510.cgz/lkp-cpl-4sp2/hackbench commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 30767113 ± 3% +30.0% 39986173 cpuidle..usage 46513 ± 24% -56.2% 20388 ± 63% numa-vmstat.node1.nr_kernel_stack 10994 ± 44% -47.7% 5753 ± 39% numa-vmstat.node2.nr_slab_reclaimable 46474 ± 24% -56.1% 20420 ± 63% numa-meminfo.node1.KernelStack 43970 ± 44% -47.7% 23008 ± 39% numa-meminfo.node2.KReclaimable 43970 ± 44% -47.7% 23008 ± 39% numa-meminfo.node2.SReclaimable 5106 ± 5% +45.4% 7425 ± 6% perf-c2c.DRAM.local 3600 ± 10% +75.1% 6304 ± 3% perf-c2c.DRAM.remote 1412 ± 16% +133.9% 3304 ± 4% perf-c2c.HITM.remote 7611598 ± 13% +24.6% 9486023 ± 4% vmstat.memory.cache 5412487 -60.4% 2142060 vmstat.system.cs 1257846 -48.9% 642429 vmstat.system.in 3591 +1.1% 3631 turbostat.Bzy_MHz 30443383 ± 4% +30.3% 39652818 ± 2% turbostat.C1 2.037e+08 -41.9% 1.183e+08 turbostat.IRQ 24.15 +10.0% 26.56 turbostat.RAMWatt 247.86 ± 3% +9.6% 271.71 ± 3% sched_debug.cfs_rq:/.util_avg.stddev 1443002 -63.3% 529509 ± 20% sched_debug.cpu.nr_switches.avg 1732432 ± 5% -60.4% 685443 ± 18% sched_debug.cpu.nr_switches.max 1259291 ± 4% -62.9% 466859 ± 20% sched_debug.cpu.nr_switches.min 58428 ± 12% -49.5% 29525 ± 28% sched_debug.cpu.nr_switches.stddev 197915 ± 4% +10.7% 219104 meminfo.Active 197828 ± 4% +10.7% 219020 meminfo.Active(anon) 7433766 ± 14% +25.1% 9300322 ± 5% meminfo.Cached 5681250 ± 18% +33.0% 7558762 ± 6% meminfo.Committed_AS 5147786 ± 20% +36.2% 7010760 ± 6% meminfo.Inactive 5147589 ± 20% +36.2% 7010560 ± 6% meminfo.Inactive(anon) 11052237 ± 9% +17.1% 12940406 ± 3% meminfo.Memused 4718879 ± 22% +39.6% 6585374 ± 7% meminfo.Shmem 11219273 ± 9% +17.0% 13129328 ± 3% meminfo.max_used_kB 739563 -4.8% 703970 hackbench.throughput 728804 -4.2% 698518 hackbench.throughput_avg 739563 -4.8% 703970 hackbench.throughput_best 703747 -1.6% 692698 hackbench.throughput_worst 145.81 +4.2% 151.93 hackbench.time.elapsed_time 145.81 +4.2% 151.93 hackbench.time.elapsed_time.max 1.617e+08 -99.5% 820613 hackbench.time.involuntary_context_switches 177419 +5.0% 186254 hackbench.time.minor_page_faults 29800 +4.4% 31096 hackbench.time.system_time 1370 -5.7% 1292 hackbench.time.user_time 6.081e+08 -52.7% 2.879e+08 hackbench.time.voluntary_context_switches 49436 ± 4% +10.7% 54736 proc-vmstat.nr_active_anon 156662 +2.8% 161046 proc-vmstat.nr_anon_pages 1857775 ± 14% +25.1% 2324649 ± 5% proc-vmstat.nr_file_pages 1286180 ± 20% +36.2% 1752228 ± 6% proc-vmstat.nr_inactive_anon 1179052 ± 22% +39.6% 1645911 ± 7% proc-vmstat.nr_shmem 152254 +4.6% 159244 proc-vmstat.nr_slab_unreclaimable 49436 ± 4% +10.7% 54736 proc-vmstat.nr_zone_active_anon 1286180 ± 20% +36.2% 1752228 ± 6% proc-vmstat.nr_zone_inactive_anon 542581 ± 15% +28.9% 699338 ± 4% proc-vmstat.numa_hint_faults 2979572 ± 11% +23.5% 3680914 ± 4% proc-vmstat.numa_hit 2631931 ± 13% +26.5% 3330495 ± 4% proc-vmstat.numa_local 82333 ± 2% +6.5% 87675 ± 2% proc-vmstat.pgactivate 4134461 ± 8% +29.6% 5358720 ± 3% proc-vmstat.pgalloc_normal 1713524 ± 4% +10.2% 1889151 proc-vmstat.pgfault 2097823 +30.8% 2744388 proc-vmstat.pgfree 0.68 +45.7% 0.99 perf-stat.i.MPKI 7.654e+10 -7.5% 7.083e+10 perf-stat.i.branch-instructions 7.88 +3.4 11.29 perf-stat.i.cache-miss-rate% 2.713e+08 +37.6% 3.733e+08 perf-stat.i.cache-misses 3.515e+09 -5.6% 3.319e+09 perf-stat.i.cache-references 5504254 -60.8% 2158477 perf-stat.i.context-switches 1.91 +8.8% 2.08 perf-stat.i.cpi 733816 -13.7% 633594 perf-stat.i.cpu-migrations 2890 -26.9% 2112 perf-stat.i.cycles-between-cache-misses 0.01 ± 8% +0.0 0.02 ± 4% perf-stat.i.dTLB-load-miss-rate% 13018959 ± 7% +25.1% 16280772 ± 4% perf-stat.i.dTLB-load-misses 1.111e+11 -7.3% 1.03e+11 perf-stat.i.dTLB-loads 0.01 ± 15% +0.0 0.01 ± 5% perf-stat.i.dTLB-store-miss-rate% 3972585 ± 14% +40.9% 5597716 ± 5% perf-stat.i.dTLB-store-misses 6.844e+10 -7.3% 6.344e+10 perf-stat.i.dTLB-stores 2.533e+08 ± 8% -19.9% 2.03e+08 perf-stat.i.iTLB-load-misses 5777737 +7.5% 6211845 ± 2% perf-stat.i.iTLB-loads 4.075e+11 -7.3% 3.778e+11 perf-stat.i.instructions 0.52 -8.3% 0.48 perf-stat.i.ipc 3.46 +1.2% 3.50 perf-stat.i.metric.GHz 806.15 +43.5% 1156 perf-stat.i.metric.K/sec 1158 -7.4% 1073 perf-stat.i.metric.M/sec 42534701 +44.0% 61247922 perf-stat.i.node-load-misses 48366476 ± 2% +45.0% 70147043 perf-stat.i.node-loads 36.75 +5.4 42.11 perf-stat.i.node-store-miss-rate% 27919430 +71.2% 47811964 perf-stat.i.node-store-misses 46266177 +42.3% 65859572 perf-stat.i.node-stores 0.67 +48.4% 0.99 perf-stat.overall.MPKI 7.72 +3.5 11.25 perf-stat.overall.cache-miss-rate% 1.90 +9.2% 2.08 perf-stat.overall.cpi 2858 -26.5% 2102 perf-stat.overall.cycles-between-cache-misses 0.01 ± 7% +0.0 0.02 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.01 ± 14% +0.0 0.01 ± 5% perf-stat.overall.dTLB-store-miss-rate% 0.53 -8.4% 0.48 perf-stat.overall.ipc 37.36 +4.5 41.90 perf-stat.overall.node-store-miss-rate% 7.604e+10 -7.6% 7.027e+10 perf-stat.ps.branch-instructions 2.695e+08 +37.4% 3.703e+08 perf-stat.ps.cache-misses 3.49e+09 -5.7% 3.293e+09 perf-stat.ps.cache-references 5460869 -60.7% 2145220 perf-stat.ps.context-switches 727414 -13.5% 629220 perf-stat.ps.cpu-migrations 13347934 ± 7% +24.1% 16558649 ± 4% perf-stat.ps.dTLB-load-misses 1.104e+11 -7.5% 1.022e+11 perf-stat.ps.dTLB-loads 3960394 ± 14% +40.1% 5546955 ± 5% perf-stat.ps.dTLB-store-misses 6.8e+10 -7.4% 6.294e+10 perf-stat.ps.dTLB-stores 2.516e+08 ± 8% -20.0% 2.012e+08 perf-stat.ps.iTLB-load-misses 5624655 +9.7% 6169172 ± 2% perf-stat.ps.iTLB-loads 4.049e+11 -7.4% 3.748e+11 perf-stat.ps.instructions 42012686 +44.0% 60495775 perf-stat.ps.node-load-misses 48376582 ± 2% +44.7% 69980029 perf-stat.ps.node-loads 27483719 +71.8% 47215357 perf-stat.ps.node-store-misses 46080326 +42.1% 65466159 perf-stat.ps.node-stores 5.906e+13 -3.4% 5.707e+13 perf-stat.total.instructions 3.74 -2.5 1.20 ± 3% perf-profile.calltrace.cycles-pp.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_write_iter.vfs_write 5.21 -2.5 2.69 perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_write_iter.vfs_write.ksys_write 2.78 -1.8 0.93 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg.sock_write_iter 2.68 -1.8 0.89 perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable.unix_stream_sendmsg 2.64 -1.8 0.88 perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.sock_def_readable 2.81 -1.6 1.19 perf-profile.calltrace.cycles-pp.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter 2.41 -1.4 1.03 perf-profile.calltrace.cycles-pp.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg 2.32 -1.3 1.00 perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic.unix_stream_recvmsg 2.30 -1.3 0.99 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.unix_stream_data_wait.unix_stream_read_generic 46.71 -1.3 45.43 perf-profile.calltrace.cycles-pp.__libc_write.start_thread 45.23 -1.2 43.99 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_write.start_thread 44.99 -1.2 43.77 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.start_thread 39.77 -1.0 38.79 perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write 44.22 -1.0 43.24 perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write.start_thread 37.63 -0.9 36.74 perf-profile.calltrace.cycles-pp.sock_write_iter.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 35.89 -0.8 35.05 perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_write_iter.vfs_write.ksys_write.do_syscall_64 1.12 ± 20% -0.7 0.42 ± 71% perf-profile.calltrace.cycles-pp.get_obj_cgroup_from_current.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 2.16 ± 8% -0.6 1.55 ± 15% perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter 1.53 ± 7% -0.5 1.01 ± 14% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 1.54 ± 7% -0.5 1.02 ± 14% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 1.53 ± 7% -0.5 1.01 ± 14% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 1.53 ± 7% -0.5 1.01 ± 14% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 3.68 ± 5% -0.5 3.19 perf-profile.calltrace.cycles-pp.__kmem_cache_free.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg 1.43 ± 7% -0.5 0.96 ± 14% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 1.39 ± 7% -0.5 0.94 ± 14% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 1.39 ± 7% -0.5 0.94 ± 14% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 1.38 ± 7% -0.4 0.93 ± 14% perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.55 ± 3% -0.2 0.34 ± 70% perf-profile.calltrace.cycles-pp.mod_objcg_state.__kmem_cache_free.skb_release_data.consume_skb.unix_stream_read_generic 1.32 ± 2% -0.2 1.16 perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 1.32 ± 3% -0.1 1.18 perf-profile.calltrace.cycles-pp.memcg_slab_post_alloc_hook.__kmem_cache_alloc_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb 2.60 -0.1 2.49 perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_write_iter.vfs_write.ksys_write 0.97 ± 2% -0.1 0.88 ± 3% perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.84 -0.1 0.75 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg 0.92 -0.1 0.83 perf-profile.calltrace.cycles-pp.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter 0.83 ± 4% -0.1 0.75 ± 5% perf-profile.calltrace.cycles-pp.apparmor_file_permission.security_file_permission.vfs_read.ksys_read.do_syscall_64 0.76 ± 3% -0.1 0.69 perf-profile.calltrace.cycles-pp.security_socket_recvmsg.sock_recvmsg.sock_read_iter.vfs_read.ksys_read 0.64 ± 3% -0.1 0.58 perf-profile.calltrace.cycles-pp.fput.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_write 0.71 ± 2% -0.1 0.65 ± 2% perf-profile.calltrace.cycles-pp.__pthread_disable_asynccancel.start_thread 1.28 -0.1 1.22 perf-profile.calltrace.cycles-pp.__check_object_size.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_write_iter.vfs_write 1.11 -0.0 1.06 perf-profile.calltrace.cycles-pp._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_write_iter.vfs_write 0.63 -0.0 0.59 ± 2% perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_write_iter 0.64 -0.0 0.61 perf-profile.calltrace.cycles-pp.__pthread_enable_asynccancel.start_thread 0.78 -0.0 0.75 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_write_iter 0.55 ± 2% -0.0 0.53 perf-profile.calltrace.cycles-pp.__check_heap_object.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 1.67 +0.1 1.74 perf-profile.calltrace.cycles-pp.__build_skb_around.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg 0.54 ± 4% +0.1 0.61 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.unix_stream_sendmsg.sock_write_iter.vfs_write 2.96 +0.1 3.05 perf-profile.calltrace.cycles-pp.__slab_free.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg 3.01 +0.1 3.11 perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic 2.92 +0.1 3.03 perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor 2.16 +0.1 2.29 perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 3.05 +0.1 3.19 perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_sendmsg.sock_write_iter.vfs_write.ksys_write 2.52 +0.2 2.73 perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.consume_skb.unix_stream_read_generic 2.64 +0.2 2.88 perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic 2.71 +0.2 2.96 perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.consume_skb.unix_stream_read_generic.unix_stream_recvmsg 2.84 +0.2 3.09 perf-profile.calltrace.cycles-pp.skb_release_head_state.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg 2.34 +0.3 2.60 perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor 2.11 +0.3 2.38 perf-profile.calltrace.cycles-pp.rep_movs_alternative.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter 5.91 +0.3 6.24 perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter 5.86 +0.3 6.20 perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg 5.79 +0.3 6.13 perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg 98.04 +0.5 98.54 perf-profile.calltrace.cycles-pp.start_thread 6.88 ± 3% +0.7 7.59 ± 2% perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg 2.53 ± 2% +1.2 3.75 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.kmem_cache_alloc_node 2.59 ± 2% +1.2 3.82 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.kmem_cache_alloc_node.__alloc_skb 2.88 ± 2% +1.3 4.19 ± 5% perf-profile.calltrace.cycles-pp.get_partial_node.___slab_alloc.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags 7.84 ± 2% +1.3 9.18 perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg 7.01 ± 2% +1.3 8.35 ± 2% perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 6.98 ± 2% +1.4 8.34 ± 2% perf-profile.calltrace.cycles-pp.__kmem_cache_alloc_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags 3.42 +1.4 4.81 ± 4% perf-profile.calltrace.cycles-pp.___slab_alloc.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 2.66 ± 7% +1.4 4.06 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_node 2.71 ± 6% +1.4 4.13 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.get_partial_node.___slab_alloc.__kmem_cache_alloc_node.__kmalloc_node_track_caller 3.00 ± 6% +1.5 4.50 ± 4% perf-profile.calltrace.cycles-pp.get_partial_node.___slab_alloc.__kmem_cache_alloc_node.__kmalloc_node_track_caller.kmalloc_reserve 3.54 ± 5% +1.6 5.14 ± 4% perf-profile.calltrace.cycles-pp.___slab_alloc.__kmem_cache_alloc_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb 20.34 +1.7 22.08 perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter.vfs_write.ksys_write 4.01 ± 2% +1.9 5.89 ± 5% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__unfreeze_partials.unix_stream_read_generic.unix_stream_recvmsg 49.14 +1.9 51.05 perf-profile.calltrace.cycles-pp.__libc_read.start_thread 4.12 ± 2% +1.9 6.03 ± 5% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__unfreeze_partials.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg 4.36 ± 2% +2.0 6.34 ± 5% perf-profile.calltrace.cycles-pp.__unfreeze_partials.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter 12.34 ± 2% +2.0 14.35 ± 2% perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg 17.40 +2.1 19.47 perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter.vfs_write 17.25 +2.1 19.32 perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_write_iter 47.43 +2.1 49.54 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__libc_read.start_thread 47.19 +2.1 49.33 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read.start_thread 4.18 ± 7% +2.2 6.37 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__unfreeze_partials.skb_release_data.consume_skb 4.29 ± 7% +2.2 6.51 ± 4% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__unfreeze_partials.skb_release_data.consume_skb.unix_stream_read_generic 15.54 +2.2 17.79 perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter 4.54 ± 6% +2.3 6.82 ± 4% perf-profile.calltrace.cycles-pp.__unfreeze_partials.skb_release_data.consume_skb.unix_stream_read_generic.unix_stream_recvmsg 45.06 +2.3 47.40 perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read 46.29 +2.4 48.67 perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__libc_read.start_thread 42.45 +2.5 44.99 perf-profile.calltrace.cycles-pp.sock_read_iter.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe 41.82 +2.6 44.42 perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.vfs_read.ksys_read.do_syscall_64 40.90 +2.7 43.60 perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_recvmsg.sock_read_iter.vfs_read.ksys_read 40.69 +2.7 43.42 perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_recvmsg.sock_read_iter.vfs_read 5.22 -2.5 2.70 perf-profile.children.cycles-pp.sock_def_readable 4.10 -2.5 1.58 ± 2% perf-profile.children.cycles-pp.__wake_up_common_lock 3.12 -1.8 1.30 perf-profile.children.cycles-pp.__wake_up_common 3.16 -1.8 1.36 perf-profile.children.cycles-pp.__schedule 3.02 -1.8 1.26 perf-profile.children.cycles-pp.autoremove_wake_function 2.98 -1.7 1.23 perf-profile.children.cycles-pp.try_to_wake_up 3.05 -1.7 1.33 perf-profile.children.cycles-pp.schedule 2.82 -1.6 1.19 perf-profile.children.cycles-pp.unix_stream_data_wait 2.78 -1.4 1.37 perf-profile.children.cycles-pp.schedule_timeout 47.15 -1.3 45.86 perf-profile.children.cycles-pp.__libc_write 39.82 -1.0 38.84 perf-profile.children.cycles-pp.vfs_write 44.25 -1.0 43.28 perf-profile.children.cycles-pp.ksys_write 1.51 -0.9 0.59 perf-profile.children.cycles-pp.ttwu_do_activate 37.65 -0.9 36.76 perf-profile.children.cycles-pp.sock_write_iter 35.97 -0.8 35.13 perf-profile.children.cycles-pp.unix_stream_sendmsg 1.38 -0.8 0.60 perf-profile.children.cycles-pp.activate_task 1.33 -0.8 0.58 perf-profile.children.cycles-pp.enqueue_task_fair 1.69 ± 17% -0.6 1.06 ± 10% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 1.16 ± 24% -0.6 0.54 ± 13% perf-profile.children.cycles-pp.__get_obj_cgroup_from_memcg 1.08 -0.6 0.51 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.88 -0.6 0.32 ± 2% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 1.54 ± 7% -0.5 1.02 ± 14% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 1.54 ± 7% -0.5 1.02 ± 14% perf-profile.children.cycles-pp.cpu_startup_entry 1.53 ± 7% -0.5 1.01 ± 14% perf-profile.children.cycles-pp.start_secondary 1.54 ± 7% -0.5 1.02 ± 14% perf-profile.children.cycles-pp.do_idle 0.99 -0.5 0.48 perf-profile.children.cycles-pp.dequeue_task_fair 3.68 ± 5% -0.5 3.19 perf-profile.children.cycles-pp.__kmem_cache_free 1.43 ± 7% -0.5 0.96 ± 14% perf-profile.children.cycles-pp.cpuidle_idle_call 1.40 ± 7% -0.5 0.94 ± 14% perf-profile.children.cycles-pp.cpuidle_enter_state 0.80 -0.5 0.34 perf-profile.children.cycles-pp.enqueue_entity 1.40 ± 7% -0.5 0.94 ± 14% perf-profile.children.cycles-pp.cpuidle_enter 1.38 ± 7% -0.4 0.93 ± 14% perf-profile.children.cycles-pp.acpi_safe_halt 1.38 ± 7% -0.4 0.94 ± 14% perf-profile.children.cycles-pp.acpi_idle_enter 0.94 -0.4 0.49 perf-profile.children.cycles-pp.pick_next_task_fair 0.57 ± 3% -0.4 0.18 ± 3% perf-profile.children.cycles-pp.prepare_to_wait 0.77 -0.4 0.39 perf-profile.children.cycles-pp.update_load_avg 1.22 ± 7% -0.4 0.85 ± 15% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 0.60 ± 4% -0.3 0.28 ± 2% perf-profile.children.cycles-pp.select_task_rq 0.53 ± 2% -0.3 0.21 perf-profile.children.cycles-pp.update_curr 4.53 -0.3 4.22 perf-profile.children.cycles-pp._raw_spin_lock 0.56 ± 4% -0.3 0.26 ± 2% perf-profile.children.cycles-pp.select_task_rq_fair 2.65 ± 3% -0.3 2.36 perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook 0.51 -0.2 0.26 perf-profile.children.cycles-pp.dequeue_entity 0.27 -0.2 0.09 ± 5% perf-profile.children.cycles-pp.set_next_entity 0.31 -0.2 0.13 ± 2% perf-profile.children.cycles-pp.reweight_entity 0.24 ± 2% -0.2 0.08 ± 6% perf-profile.children.cycles-pp.__enqueue_entity 0.21 ± 2% -0.2 0.05 perf-profile.children.cycles-pp.pick_eevdf 0.36 -0.2 0.20 ± 2% perf-profile.children.cycles-pp.switch_fpu_return 0.24 -0.2 0.08 ± 5% perf-profile.children.cycles-pp.prepare_task_switch 0.28 ± 9% -0.2 0.12 ± 6% perf-profile.children.cycles-pp.select_idle_sibling 0.49 ± 2% -0.1 0.34 ± 2% perf-profile.children.cycles-pp.update_cfs_group 0.18 ± 12% -0.1 0.05 ± 7% perf-profile.children.cycles-pp.finish_task_switch 0.22 -0.1 0.10 ± 3% perf-profile.children.cycles-pp.__update_load_avg_se 0.20 ± 2% -0.1 0.08 ± 4% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 1.94 ± 2% -0.1 1.81 ± 2% perf-profile.children.cycles-pp.security_file_permission 2.62 -0.1 2.51 perf-profile.children.cycles-pp.skb_copy_datagram_from_iter 0.21 -0.1 0.10 perf-profile.children.cycles-pp.wake_affine 1.70 ± 2% -0.1 1.60 ± 3% perf-profile.children.cycles-pp.apparmor_file_permission 0.26 -0.1 0.16 ± 2% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate 0.20 ± 2% -0.1 0.10 perf-profile.children.cycles-pp.__switch_to_asm 0.23 -0.1 0.13 perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template 0.91 -0.1 0.81 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.15 ± 2% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.___perf_sw_event 0.51 ± 6% -0.1 0.42 perf-profile.children.cycles-pp.obj_cgroup_uncharge_pages 0.14 ± 2% -0.1 0.06 ± 9% perf-profile.children.cycles-pp.__dequeue_entity 0.11 ± 3% -0.1 0.02 ± 99% perf-profile.children.cycles-pp.update_rq_clock_task 0.92 -0.1 0.84 perf-profile.children.cycles-pp.skb_unlink 0.19 ± 5% -0.1 0.10 ± 11% perf-profile.children.cycles-pp.sysvec_call_function_single 0.93 ± 2% -0.1 0.85 perf-profile.children.cycles-pp.fput 0.13 ± 10% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.available_idle_cpu 0.14 ± 18% -0.1 0.07 ± 11% perf-profile.children.cycles-pp.select_idle_cpu 0.17 ± 5% -0.1 0.09 ± 14% perf-profile.children.cycles-pp.__sysvec_call_function_single 0.17 ± 4% -0.1 0.10 ± 10% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 0.77 ± 2% -0.1 0.70 perf-profile.children.cycles-pp.security_socket_recvmsg 0.14 ± 2% -0.1 0.08 ± 6% perf-profile.children.cycles-pp.__switch_to 1.71 -0.1 1.65 perf-profile.children.cycles-pp.mod_objcg_state 0.67 -0.1 0.61 perf-profile.children.cycles-pp.__cond_resched 0.14 ± 4% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.sched_ttwu_pending 0.13 -0.1 0.07 ± 5% perf-profile.children.cycles-pp.task_h_load 0.71 ± 2% -0.1 0.65 perf-profile.children.cycles-pp.__pthread_disable_asynccancel 0.56 -0.1 0.50 perf-profile.children.cycles-pp.mutex_lock 0.11 ± 4% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.perf_tp_event 1.12 -0.0 1.07 perf-profile.children.cycles-pp._copy_from_iter 1.42 -0.0 1.38 perf-profile.children.cycles-pp.__entry_text_start 0.64 -0.0 0.61 perf-profile.children.cycles-pp.__pthread_enable_asynccancel 0.92 -0.0 0.89 perf-profile.children.cycles-pp.copyin 0.33 -0.0 0.30 perf-profile.children.cycles-pp.__get_task_ioprio 0.91 -0.0 0.88 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack 0.09 -0.0 0.07 ± 6% perf-profile.children.cycles-pp.syscall_exit_to_user_mode_prepare 0.35 -0.0 0.33 perf-profile.children.cycles-pp.refill_obj_stock 0.07 ± 7% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.__irq_exit_rcu 0.07 ± 5% -0.0 0.06 perf-profile.children.cycles-pp.set_task_cpu 0.12 ± 3% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.put_pid 0.06 ± 7% +0.0 0.08 perf-profile.children.cycles-pp.find_busiest_group 0.06 ± 7% +0.0 0.08 ± 4% perf-profile.children.cycles-pp.update_sd_lb_stats 0.06 ± 9% +0.0 0.07 perf-profile.children.cycles-pp.update_sg_lb_stats 0.12 ± 3% +0.0 0.14 perf-profile.children.cycles-pp.put_cpu_partial 0.10 ± 3% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.detach_tasks 0.08 ± 13% +0.0 0.10 ± 10% perf-profile.children.cycles-pp.cgroup_rstat_updated 0.24 +0.0 0.28 ± 2% perf-profile.children.cycles-pp.load_balance 0.24 +0.1 0.29 ± 2% perf-profile.children.cycles-pp.newidle_balance 1.68 +0.1 1.75 perf-profile.children.cycles-pp.__build_skb_around 0.11 ± 6% +0.1 0.20 ± 2% perf-profile.children.cycles-pp.__x64_sys_read 2.83 +0.1 2.92 perf-profile.children.cycles-pp.check_heap_object 5.70 +0.1 5.80 perf-profile.children.cycles-pp.__slab_free 3.02 +0.1 3.12 perf-profile.children.cycles-pp.simple_copy_to_iter 0.30 ± 3% +0.1 0.41 ± 12% perf-profile.children.cycles-pp.get_any_partial 2.52 +0.2 2.73 perf-profile.children.cycles-pp.sock_wfree 2.65 +0.2 2.89 perf-profile.children.cycles-pp._copy_to_iter 2.86 +0.2 3.10 perf-profile.children.cycles-pp.skb_release_head_state 2.68 +0.2 2.92 perf-profile.children.cycles-pp.rep_movs_alternative 2.80 +0.2 3.05 perf-profile.children.cycles-pp.unix_destruct_scm 2.46 +0.3 2.72 perf-profile.children.cycles-pp.copyout 5.91 +0.3 6.24 perf-profile.children.cycles-pp.unix_stream_read_actor 5.87 +0.3 6.20 perf-profile.children.cycles-pp.skb_copy_datagram_iter 5.80 +0.3 6.14 perf-profile.children.cycles-pp.__skb_datagram_iter 98.04 +0.5 98.54 perf-profile.children.cycles-pp.start_thread 6.94 ± 3% +0.7 7.64 ± 2% perf-profile.children.cycles-pp.kmem_cache_alloc_node 92.75 +0.9 93.61 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 92.26 +0.9 93.16 perf-profile.children.cycles-pp.do_syscall_64 7.88 ± 2% +1.3 9.21 perf-profile.children.cycles-pp.kmalloc_reserve 7.06 ± 2% +1.3 8.41 ± 2% perf-profile.children.cycles-pp.__kmem_cache_alloc_node 7.37 ± 2% +1.4 8.72 ± 2% perf-profile.children.cycles-pp.__kmalloc_node_track_caller 20.35 +1.7 22.09 perf-profile.children.cycles-pp.sock_alloc_send_pskb 49.57 +1.9 51.46 perf-profile.children.cycles-pp.__libc_read 12.35 ± 2% +2.0 14.36 ± 2% perf-profile.children.cycles-pp.skb_release_data 17.42 +2.1 19.49 perf-profile.children.cycles-pp.alloc_skb_with_frags 17.28 +2.1 19.36 perf-profile.children.cycles-pp.__alloc_skb 15.57 +2.2 17.82 perf-profile.children.cycles-pp.consume_skb 45.09 +2.3 47.42 perf-profile.children.cycles-pp.vfs_read 46.32 +2.4 48.69 perf-profile.children.cycles-pp.ksys_read 42.47 +2.5 45.01 perf-profile.children.cycles-pp.sock_read_iter 41.85 +2.6 44.45 perf-profile.children.cycles-pp.sock_recvmsg 40.92 +2.7 43.61 perf-profile.children.cycles-pp.unix_stream_recvmsg 40.76 +2.7 43.48 perf-profile.children.cycles-pp.unix_stream_read_generic 6.17 ± 3% +2.9 9.10 ± 2% perf-profile.children.cycles-pp.get_partial_node 6.96 ± 2% +3.0 9.94 ± 2% perf-profile.children.cycles-pp.___slab_alloc 8.90 ± 3% +4.3 13.17 ± 2% perf-profile.children.cycles-pp.__unfreeze_partials 18.14 ± 2% +5.7 23.84 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 15.76 ± 2% +5.7 21.46 ± 2% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.14 ± 25% -0.6 0.53 ± 13% perf-profile.self.cycles-pp.__get_obj_cgroup_from_memcg 1.20 ± 7% -0.4 0.84 ± 14% perf-profile.self.cycles-pp.acpi_safe_halt 2.58 ± 6% -0.4 2.23 perf-profile.self.cycles-pp.__kmem_cache_free 1.96 ± 4% -0.3 1.66 perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook 0.48 ± 3% -0.3 0.20 perf-profile.self.cycles-pp.__schedule 3.28 -0.2 3.07 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.24 ± 2% -0.2 0.07 ± 6% perf-profile.self.cycles-pp.__enqueue_entity 0.26 ± 2% -0.2 0.10 ± 5% perf-profile.self.cycles-pp.update_curr 0.49 ± 2% -0.1 0.34 ± 2% perf-profile.self.cycles-pp.update_cfs_group 0.32 -0.1 0.19 ± 2% perf-profile.self.cycles-pp.update_load_avg 0.20 -0.1 0.08 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.21 -0.1 0.09 ± 4% perf-profile.self.cycles-pp.__update_load_avg_se 1.53 -0.1 1.42 perf-profile.self.cycles-pp.mod_objcg_state 0.26 -0.1 0.16 ± 2% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate 0.20 ± 2% -0.1 0.10 perf-profile.self.cycles-pp.__switch_to_asm 0.18 -0.1 0.08 perf-profile.self.cycles-pp.reweight_entity 3.58 -0.1 3.49 perf-profile.self.cycles-pp._raw_spin_lock 0.18 ± 22% -0.1 0.09 ± 8% perf-profile.self.cycles-pp.obj_cgroup_uncharge_pages 2.26 -0.1 2.17 perf-profile.self.cycles-pp.unix_stream_sendmsg 0.14 ± 2% -0.1 0.05 perf-profile.self.cycles-pp.___perf_sw_event 0.90 ± 2% -0.1 0.82 perf-profile.self.cycles-pp.fput 0.13 ± 8% -0.1 0.05 ± 8% perf-profile.self.cycles-pp.available_idle_cpu 1.05 -0.1 0.97 perf-profile.self.cycles-pp.vfs_read 0.10 ± 3% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.schedule_timeout 0.14 ± 2% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.__switch_to 0.11 ± 4% -0.1 0.05 perf-profile.self.cycles-pp.try_to_wake_up 0.43 -0.1 0.37 perf-profile.self.cycles-pp.mutex_lock 0.13 ± 2% -0.1 0.07 perf-profile.self.cycles-pp.enqueue_task_fair 0.13 -0.1 0.07 ± 5% perf-profile.self.cycles-pp.task_h_load 0.62 -0.1 0.56 perf-profile.self.cycles-pp.sock_read_iter 0.11 -0.1 0.05 ± 7% perf-profile.self.cycles-pp.enqueue_entity 0.69 ± 2% -0.1 0.63 ± 2% perf-profile.self.cycles-pp.__pthread_disable_asynccancel 0.72 ± 5% -0.1 0.67 perf-profile.self.cycles-pp.__libc_read 1.04 -0.0 1.00 perf-profile.self.cycles-pp.vfs_write 0.34 -0.0 0.30 perf-profile.self.cycles-pp.unix_write_space 0.10 -0.0 0.06 perf-profile.self.cycles-pp.perf_trace_sched_wakeup_template 0.09 ± 4% -0.0 0.05 perf-profile.self.cycles-pp.perf_tp_event 0.50 -0.0 0.47 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.63 -0.0 0.60 ± 2% perf-profile.self.cycles-pp.__pthread_enable_asynccancel 0.87 -0.0 0.84 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack 0.31 -0.0 0.28 ± 2% perf-profile.self.cycles-pp.__get_task_ioprio 0.16 ± 3% -0.0 0.14 ± 4% perf-profile.self.cycles-pp.unix_stream_recvmsg 0.79 -0.0 0.77 perf-profile.self.cycles-pp.__alloc_skb 0.12 -0.0 0.10 ± 3% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.40 -0.0 0.38 perf-profile.self.cycles-pp.copyout 0.34 -0.0 0.32 perf-profile.self.cycles-pp.refill_obj_stock 0.18 ± 2% -0.0 0.16 perf-profile.self.cycles-pp.sock_recvmsg 0.14 ± 3% -0.0 0.13 ± 2% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.20 ± 2% -0.0 0.18 perf-profile.self.cycles-pp.do_syscall_64 0.36 -0.0 0.35 perf-profile.self.cycles-pp.consume_skb 0.20 ± 2% -0.0 0.18 ± 2% perf-profile.self.cycles-pp._copy_from_iter 0.19 -0.0 0.18 ± 2% perf-profile.self.cycles-pp.skb_copy_datagram_from_iter 0.19 -0.0 0.18 perf-profile.self.cycles-pp._copy_to_iter 0.08 ± 5% +0.0 0.10 perf-profile.self.cycles-pp.put_pid 0.12 +0.0 0.14 perf-profile.self.cycles-pp.put_cpu_partial 0.52 +0.0 0.54 ± 2% perf-profile.self.cycles-pp.__kmalloc_node_track_caller 0.08 ± 12% +0.0 0.10 ± 8% perf-profile.self.cycles-pp.cgroup_rstat_updated 0.18 ± 2% +0.0 0.21 ± 3% perf-profile.self.cycles-pp.ksys_read 0.00 +0.1 0.06 ± 8% perf-profile.self.cycles-pp.update_sg_lb_stats 0.76 +0.1 0.82 perf-profile.self.cycles-pp.___slab_alloc 1.66 +0.1 1.73 perf-profile.self.cycles-pp.__build_skb_around 0.10 ± 5% +0.1 0.19 ± 3% perf-profile.self.cycles-pp.__x64_sys_read 0.39 +0.1 0.48 perf-profile.self.cycles-pp.__unfreeze_partials 5.66 +0.1 5.75 perf-profile.self.cycles-pp.__slab_free 0.42 ± 2% +0.1 0.54 perf-profile.self.cycles-pp.get_partial_node 0.94 ± 2% +0.1 1.06 ± 2% perf-profile.self.cycles-pp.skb_release_data 1.88 +0.1 2.03 perf-profile.self.cycles-pp.check_heap_object 1.76 +0.2 1.97 perf-profile.self.cycles-pp.sock_wfree 2.51 +0.3 2.77 perf-profile.self.cycles-pp.rep_movs_alternative 15.74 ± 2% +5.7 21.44 ± 2% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath *************************************************************************************************** lkp-csl-2sp3: 96 threads 2 sockets Intel(R) Xeon(R) Platinum 8260L CPU @ 2.40GHz (Cascade Lake) with 128G memory ========================================================================================= compiler/cpufreq_governor/disk/fs/kconfig/rootfs/tbox_group/testcase: gcc-12/performance/1SSD/xfs/x86_64-rhel-8.3/debian-11.1-x86_64-20220510.cgz/lkp-csl-2sp3/blogbench commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 7499 ± 5% +13.3% 8495 ± 2% perf-c2c.DRAM.local 3.527e+08 ± 9% +54.6% 5.453e+08 ± 6% cpuidle..time 415301 ± 8% +58.9% 659888 ± 7% cpuidle..usage 1.66 ± 9% +46.7% 2.43 ± 5% iostat.cpu.idle 1.74 ± 2% -4.9% 1.65 iostat.cpu.user 1.01 ± 15% +0.8 1.79 ± 8% mpstat.cpu.all.idle% 0.00 ± 27% +0.0 0.00 ± 22% mpstat.cpu.all.iowait% 1.78 ± 9% -0.4 1.37 ± 5% mpstat.cpu.all.soft% 1.151e+08 -62.0% 43742493 blogbench.time.file_system_outputs 1810540 -55.7% 801888 blogbench.time.involuntary_context_switches 1287050 -56.5% 560205 blogbench.time.voluntary_context_switches 13606 -59.3% 5540 blogbench.write_score 9989139 ± 20% -31.4% 6849227 ± 6% numa-numastat.node0.local_node 10015980 ± 20% -31.2% 6894845 ± 6% numa-numastat.node0.numa_hit 13369217 ± 13% -52.5% 6349579 ± 5% numa-numastat.node1.local_node 13425354 ± 13% -52.3% 6403904 ± 5% numa-numastat.node1.numa_hit 182611 -61.1% 70997 vmstat.io.bo 30794675 -50.4% 15267505 vmstat.memory.cache 97798124 +17.0% 1.144e+08 vmstat.memory.free 15769 -50.3% 7831 vmstat.system.cs 113890 -6.3% 106679 vmstat.system.in 23666 ± 16% +55.2% 36720 ± 25% turbostat.C1 47654 ± 13% +52.6% 72730 ± 7% turbostat.C1E 0.04 ± 16% +0.0 0.09 ± 7% turbostat.C1E% 307535 ± 11% +64.9% 506978 ± 6% turbostat.C6 1.13 ± 11% +0.6 1.74 ± 6% turbostat.C6% 0.96 ± 10% +63.3% 1.57 ± 5% turbostat.CPU%c1 6890 ± 17% +111.9% 14603 ± 7% turbostat.POLL 19729599 -54.9% 8903829 meminfo.Active 19418799 -55.6% 8619704 meminfo.Active(file) 28310960 -50.1% 14140405 meminfo.Cached 5407969 -62.9% 2004774 ± 2% meminfo.Dirty 6129889 -54.6% 2783669 ± 2% meminfo.Inactive 5814371 -57.6% 2466804 ± 2% meminfo.Inactive(file) 2388126 -54.5% 1086718 meminfo.KReclaimable 27378 ± 2% -20.0% 21915 meminfo.KernelStack 97898878 +16.9% 1.144e+08 meminfo.MemFree 33785080 -48.9% 17257562 meminfo.Memused 2388126 -54.5% 1086718 meminfo.SReclaimable 1528111 -43.5% 862709 meminfo.SUnreclaim 3916238 -50.2% 1949429 meminfo.Slab 33786668 -48.9% 17265013 meminfo.max_used_kB 12164579 ± 25% -32.6% 8196575 ± 15% numa-meminfo.node0.FilePages 14656 ± 10% -20.6% 11638 ± 9% numa-meminfo.node0.KernelStack 14648986 ± 24% -32.7% 9851554 ± 14% numa-meminfo.node0.MemUsed 11750087 ± 19% -67.7% 3793215 ± 18% numa-meminfo.node1.Active 11512623 ± 19% -69.0% 3564771 ± 20% numa-meminfo.node1.Active(file) 3393154 ± 20% -73.2% 907764 ± 14% numa-meminfo.node1.Dirty 16142152 ± 18% -63.2% 5940861 ± 21% numa-meminfo.node1.FilePages 3616437 ± 18% -66.7% 1204429 ± 16% numa-meminfo.node1.Inactive 3461709 ± 18% -71.2% 997578 ± 18% numa-meminfo.node1.Inactive(file) 1363742 ± 18% -66.7% 454332 ± 19% numa-meminfo.node1.KReclaimable 46878383 ± 7% +25.0% 58606546 ± 2% numa-meminfo.node1.MemFree 19130463 ± 18% -61.3% 7402301 ± 18% numa-meminfo.node1.MemUsed 1363742 ± 18% -66.7% 454332 ± 19% numa-meminfo.node1.SReclaimable 2165606 ± 21% -60.4% 856793 ± 15% numa-meminfo.node1.Slab 3042027 ± 25% -32.7% 2048741 ± 15% numa-vmstat.node0.nr_file_pages 14657 ± 10% -20.6% 11640 ± 9% numa-vmstat.node0.nr_kernel_stack 10016173 ± 20% -31.2% 6894532 ± 6% numa-vmstat.node0.numa_hit 9989331 ± 20% -31.4% 6848914 ± 6% numa-vmstat.node0.numa_local 2879703 ± 19% -69.1% 890805 ± 20% numa-vmstat.node1.nr_active_file 8866075 ± 19% -72.6% 2432986 ± 15% numa-vmstat.node1.nr_dirtied 848292 ± 20% -73.2% 227046 ± 14% numa-vmstat.node1.nr_dirty 4037556 ± 18% -63.2% 1484735 ± 21% numa-vmstat.node1.nr_file_pages 11717275 ± 7% +25.0% 14652208 ± 2% numa-vmstat.node1.nr_free_pages 865897 ± 18% -71.2% 249317 ± 18% numa-vmstat.node1.nr_inactive_file 341110 ± 18% -66.7% 113538 ± 19% numa-vmstat.node1.nr_slab_reclaimable 7216977 ± 21% -72.2% 2009178 ± 16% numa-vmstat.node1.nr_written 2879702 ± 19% -69.1% 890805 ± 20% numa-vmstat.node1.nr_zone_active_file 865897 ± 18% -71.2% 249317 ± 18% numa-vmstat.node1.nr_zone_inactive_file 848345 ± 20% -73.2% 227062 ± 14% numa-vmstat.node1.nr_zone_write_pending 13425546 ± 13% -52.3% 6403736 ± 5% numa-vmstat.node1.numa_hit 13369409 ± 13% -52.5% 6349410 ± 5% numa-vmstat.node1.numa_local 237454 ± 14% -46.2% 127707 ± 6% sched_debug.cfs_rq:/.avg_vruntime.stddev 281333 ± 72% +245.3% 971329 ± 25% sched_debug.cfs_rq:/.left_vruntime.avg 1513017 ± 40% +100.4% 3032618 ± 16% sched_debug.cfs_rq:/.left_vruntime.stddev 10.08 ± 25% +99.0% 20.07 ± 29% sched_debug.cfs_rq:/.load_avg.min 237455 ± 14% -46.2% 127707 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev 0.15 ± 22% +69.2% 0.25 ± 14% sched_debug.cfs_rq:/.nr_running.stddev 281334 ± 72% +245.3% 971329 ± 25% sched_debug.cfs_rq:/.right_vruntime.avg 1513017 ± 40% +100.4% 3032618 ± 16% sched_debug.cfs_rq:/.right_vruntime.stddev 2582 ± 4% -15.5% 2181 ± 8% sched_debug.cfs_rq:/.runnable_avg.max 316.33 ± 5% -24.3% 239.31 ± 6% sched_debug.cfs_rq:/.runnable_avg.stddev 148.04 ± 11% -17.1% 122.69 ± 8% sched_debug.cfs_rq:/.util_avg.stddev 687989 ± 4% +26.2% 868129 ± 2% sched_debug.cpu.avg_idle.avg 278391 ± 6% -22.8% 215039 ± 5% sched_debug.cpu.avg_idle.stddev 13.15 ± 9% -29.1% 9.32 ± 14% sched_debug.cpu.clock.stddev 888.73 ± 5% -11.6% 786.08 ± 4% sched_debug.cpu.curr->pid.stddev 27314 -46.1% 14723 ± 5% sched_debug.cpu.nr_switches.avg 78618 ± 9% -42.4% 45244 ± 18% sched_debug.cpu.nr_switches.max 13072 ± 19% -36.9% 8247 ± 11% sched_debug.cpu.nr_switches.min 11267 ± 15% -51.4% 5472 ± 8% sched_debug.cpu.nr_switches.stddev 69.78 ± 26% -37.7% 43.48 ± 14% sched_debug.cpu.nr_uninterruptible.stddev 77702 -8.6% 71011 ± 2% proc-vmstat.nr_active_anon 4856684 -55.6% 2154541 proc-vmstat.nr_active_file 14382743 -62.0% 5467742 proc-vmstat.nr_dirtied 1352052 -62.9% 501053 ± 2% proc-vmstat.nr_dirty 3058358 +1.9% 3117839 proc-vmstat.nr_dirty_background_threshold 6124195 +1.9% 6243302 proc-vmstat.nr_dirty_threshold 7080320 -50.1% 3534557 proc-vmstat.nr_file_pages 24471767 +16.9% 28607216 proc-vmstat.nr_free_pages 1454177 -57.6% 616549 ± 2% proc-vmstat.nr_inactive_file 27379 ± 2% -20.0% 21914 proc-vmstat.nr_kernel_stack 10068 -3.1% 9757 proc-vmstat.nr_mapped 80678 -7.9% 74291 ± 2% proc-vmstat.nr_shmem 597268 -54.5% 271633 proc-vmstat.nr_slab_reclaimable 382038 -43.6% 215628 proc-vmstat.nr_slab_unreclaimable 11806249 -60.5% 4665462 proc-vmstat.nr_written 77702 -8.6% 71011 ± 2% proc-vmstat.nr_zone_active_anon 4856684 -55.6% 2154541 proc-vmstat.nr_zone_active_file 1454177 -57.6% 616549 ± 2% proc-vmstat.nr_zone_inactive_file 1352125 -62.9% 501087 ± 2% proc-vmstat.nr_zone_write_pending 23443764 -43.3% 13300295 proc-vmstat.numa_hit 23360786 -43.5% 13200349 proc-vmstat.numa_local 101119 -1.5% 99647 proc-vmstat.numa_other 11447594 -61.5% 4402707 proc-vmstat.pgactivate 32942601 -36.0% 21069699 proc-vmstat.pgalloc_normal 19182964 -19.2% 15502671 proc-vmstat.pgfree 55214729 -61.3% 21390778 proc-vmstat.pgpgout 11.59 +5.9% 12.28 perf-stat.i.MPKI 7.491e+09 -5.4% 7.087e+09 perf-stat.i.branch-instructions 0.41 -0.1 0.34 perf-stat.i.branch-miss-rate% 30427014 -21.1% 24011532 ± 2% perf-stat.i.branch-misses 75.23 +2.9 78.08 perf-stat.i.cache-miss-rate% 5.097e+08 -4.9% 4.846e+08 perf-stat.i.cache-references 15802 -50.3% 7853 perf-stat.i.context-switches 8.89 +6.3% 9.45 perf-stat.i.cpi 806.13 ± 3% -59.9% 323.03 ± 2% perf-stat.i.cpu-migrations 0.08 ± 13% -0.0 0.04 ± 21% perf-stat.i.dTLB-load-miss-rate% 6792913 ± 14% -56.3% 2969547 ± 21% perf-stat.i.dTLB-load-misses 8.472e+09 -7.3% 7.851e+09 perf-stat.i.dTLB-loads 0.03 ± 21% -0.0 0.01 ± 29% perf-stat.i.dTLB-store-miss-rate% 581113 ± 21% -52.1% 278227 ± 30% perf-stat.i.dTLB-store-misses 2.264e+09 -16.3% 1.895e+09 perf-stat.i.dTLB-stores 73.39 +12.8 86.24 perf-stat.i.iTLB-load-miss-rate% 7089683 ± 2% -12.3% 6220677 ± 4% perf-stat.i.iTLB-load-misses 2589801 -60.2% 1030982 ± 6% perf-stat.i.iTLB-loads 3.317e+10 -6.9% 3.089e+10 perf-stat.i.instructions 4730 ± 2% +7.0% 5062 ± 4% perf-stat.i.instructions-per-iTLB-miss 0.12 -5.1% 0.11 perf-stat.i.ipc 194.68 -7.5% 180.12 perf-stat.i.metric.M/sec 73.66 -4.4 69.27 perf-stat.i.node-load-miss-rate% 92452040 -8.4% 84721005 ± 2% perf-stat.i.node-load-misses 33026451 +13.3% 37421493 ± 2% perf-stat.i.node-loads 72.82 +2.3 75.13 perf-stat.i.node-store-miss-rate% 14065332 -4.2% 13479612 perf-stat.i.node-store-misses 5237884 -15.3% 4436059 ± 2% perf-stat.i.node-stores 11.57 +6.2% 12.29 perf-stat.overall.MPKI 0.41 -0.1 0.34 perf-stat.overall.branch-miss-rate% 75.34 +3.0 78.30 perf-stat.overall.cache-miss-rate% 8.87 +6.6% 9.46 perf-stat.overall.cpi 0.08 ± 13% -0.0 0.04 ± 21% perf-stat.overall.dTLB-load-miss-rate% 0.03 ± 20% -0.0 0.01 ± 29% perf-stat.overall.dTLB-store-miss-rate% 73.19 +12.7 85.85 perf-stat.overall.iTLB-load-miss-rate% 4690 ± 2% +6.3% 4983 ± 3% perf-stat.overall.instructions-per-iTLB-miss 0.11 -6.2% 0.11 perf-stat.overall.ipc 73.68 -4.3 69.36 perf-stat.overall.node-load-miss-rate% 72.85 +2.4 75.26 perf-stat.overall.node-store-miss-rate% 7.454e+09 -5.3% 7.057e+09 perf-stat.ps.branch-instructions 30346141 -21.5% 23821173 ± 2% perf-stat.ps.branch-misses 5.068e+08 -4.7% 4.828e+08 perf-stat.ps.cache-references 15735 -50.7% 7755 perf-stat.ps.context-switches 799.77 ± 3% -59.9% 320.39 ± 2% perf-stat.ps.cpu-migrations 6797855 ± 13% -56.4% 2960817 ± 21% perf-stat.ps.dTLB-load-misses 8.434e+09 -7.3% 7.816e+09 perf-stat.ps.dTLB-loads 580794 ± 21% -52.2% 277399 ± 30% perf-stat.ps.dTLB-store-misses 2.257e+09 -16.5% 1.884e+09 perf-stat.ps.dTLB-stores 7044215 ± 2% -12.2% 6181385 ± 4% perf-stat.ps.iTLB-load-misses 2579465 -60.6% 1017233 ± 6% perf-stat.ps.iTLB-loads 3.301e+10 -6.8% 3.075e+10 perf-stat.ps.instructions 2402 -1.1% 2376 perf-stat.ps.minor-faults 91948654 -8.2% 84415047 ± 2% perf-stat.ps.node-load-misses 32841685 +13.6% 37299426 ± 2% perf-stat.ps.node-loads 13989660 -4.0% 13428744 perf-stat.ps.node-store-misses 5212115 -15.3% 4413635 ± 2% perf-stat.ps.node-stores 2402 -1.1% 2376 perf-stat.ps.page-faults 9.94e+12 -6.8% 9.268e+12 perf-stat.total.instructions 2.51 ± 14% -1.6 0.92 ± 10% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 2.51 ± 14% -1.6 0.92 ± 10% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 2.51 ± 14% -1.6 0.92 ± 10% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 1.97 ± 11% -1.3 0.71 ± 8% perf-profile.calltrace.cycles-pp.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork 1.97 ± 11% -1.3 0.71 ± 8% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.97 ± 11% -1.3 0.70 ± 8% perf-profile.calltrace.cycles-pp.rcu_core.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread 1.98 ± 11% -1.3 0.71 ± 8% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.96 ± 11% -1.3 0.70 ± 8% perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd.smpboot_thread_fn 1.66 ± 12% -1.1 0.59 ± 9% perf-profile.calltrace.cycles-pp.file_free_rcu.rcu_do_batch.rcu_core.__do_softirq.run_ksoftirqd 2.70 ± 4% -0.3 2.36 ± 2% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64 2.81 ± 3% -0.3 2.48 perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.73 ± 2% -0.1 0.66 perf-profile.calltrace.cycles-pp.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 0.82 ± 3% -0.1 0.76 perf-profile.calltrace.cycles-pp.do_open.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat 5.82 +0.4 6.26 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.put_unused_fd.do_sys_openat2.__x64_sys_openat 5.84 +0.4 6.29 perf-profile.calltrace.cycles-pp._raw_spin_lock.put_unused_fd.do_sys_openat2.__x64_sys_openat.do_syscall_64 5.86 +0.4 6.30 perf-profile.calltrace.cycles-pp.put_unused_fd.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe 37.67 +0.5 38.16 perf-profile.calltrace.cycles-pp.__close 37.62 +0.5 38.11 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close 37.61 +0.5 38.10 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 37.38 +0.5 37.89 perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 0.00 +0.5 0.52 ± 2% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.alloc_fd 0.00 +0.5 0.53 ± 2% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.native_queued_spin_lock_slowpath._raw_spin_lock.alloc_fd.do_sys_openat2 36.40 +0.6 36.98 perf-profile.calltrace.cycles-pp.close_fd_get_file.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 36.10 +0.6 36.68 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.close_fd_get_file.__x64_sys_close.do_syscall_64 36.28 +0.6 36.86 perf-profile.calltrace.cycles-pp._raw_spin_lock.close_fd_get_file.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 41.35 +1.2 42.56 perf-profile.calltrace.cycles-pp._raw_spin_lock.alloc_fd.do_sys_openat2.__x64_sys_openat.do_syscall_64 41.16 +1.2 42.37 perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.alloc_fd.do_sys_openat2.__x64_sys_openat 41.65 +1.2 42.86 perf-profile.calltrace.cycles-pp.alloc_fd.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe 51.11 +1.3 52.42 perf-profile.calltrace.cycles-pp.__open64 51.02 +1.3 52.33 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64 51.03 +1.3 52.34 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__open64 50.60 +1.3 51.93 perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64 50.59 +1.3 51.92 perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.__open64 2.51 ± 14% -1.6 0.92 ± 10% perf-profile.children.cycles-pp.ret_from_fork_asm 2.51 ± 14% -1.6 0.92 ± 10% perf-profile.children.cycles-pp.ret_from_fork 2.51 ± 14% -1.6 0.92 ± 10% perf-profile.children.cycles-pp.kthread 1.97 ± 11% -1.3 0.71 ± 8% perf-profile.children.cycles-pp.run_ksoftirqd 1.98 ± 11% -1.3 0.71 ± 8% perf-profile.children.cycles-pp.smpboot_thread_fn 2.21 ± 11% -0.6 1.62 ± 3% perf-profile.children.cycles-pp.__do_softirq 2.19 ± 11% -0.6 1.60 ± 3% perf-profile.children.cycles-pp.rcu_do_batch 2.20 ± 11% -0.6 1.61 ± 3% perf-profile.children.cycles-pp.rcu_core 1.85 ± 11% -0.5 1.36 ± 3% perf-profile.children.cycles-pp.file_free_rcu 2.70 ± 4% -0.3 2.37 perf-profile.children.cycles-pp.path_openat 0.54 ± 26% -0.3 0.21 ± 25% perf-profile.children.cycles-pp.worker_thread 0.53 ± 26% -0.3 0.21 ± 25% perf-profile.children.cycles-pp.process_one_work 2.81 ± 3% -0.3 2.49 perf-profile.children.cycles-pp.do_filp_open 0.37 ± 37% -0.2 0.16 ± 31% perf-profile.children.cycles-pp.wb_workfn 0.37 ± 37% -0.2 0.16 ± 31% perf-profile.children.cycles-pp.wb_do_writeback 0.37 ± 37% -0.2 0.16 ± 31% perf-profile.children.cycles-pp.wb_writeback 0.37 ± 37% -0.2 0.16 ± 31% perf-profile.children.cycles-pp.__writeback_inodes_wb 0.37 ± 37% -0.2 0.16 ± 31% perf-profile.children.cycles-pp.writeback_sb_inodes 0.36 ± 37% -0.2 0.16 ± 31% perf-profile.children.cycles-pp.__writeback_single_inode 0.36 ± 37% -0.2 0.16 ± 31% perf-profile.children.cycles-pp.do_writepages 0.36 ± 37% -0.2 0.16 ± 31% perf-profile.children.cycles-pp.xfs_vm_writepages 0.35 ± 38% -0.2 0.15 ± 32% perf-profile.children.cycles-pp.iomap_writepages 0.35 ± 38% -0.2 0.15 ± 32% perf-profile.children.cycles-pp.write_cache_pages 0.32 ± 6% -0.2 0.12 ± 12% perf-profile.children.cycles-pp.__libc_write 0.32 ± 39% -0.2 0.14 ± 32% perf-profile.children.cycles-pp.xfs_map_blocks 0.31 ± 6% -0.2 0.12 ± 14% perf-profile.children.cycles-pp.ksys_write 0.32 ± 39% -0.2 0.14 ± 32% perf-profile.children.cycles-pp.xfs_bmapi_convert_delalloc 0.30 ± 5% -0.2 0.12 ± 14% perf-profile.children.cycles-pp.vfs_write 0.29 ± 6% -0.2 0.11 ± 14% perf-profile.children.cycles-pp.xfs_file_buffered_write 0.29 ± 5% -0.2 0.11 ± 14% perf-profile.children.cycles-pp.iomap_file_buffered_write 0.29 ± 42% -0.2 0.12 ± 36% perf-profile.children.cycles-pp.xfs_bmapi_allocate 0.29 ± 42% -0.2 0.12 ± 36% perf-profile.children.cycles-pp.xfs_bmap_btalloc 0.29 ± 42% -0.2 0.12 ± 36% perf-profile.children.cycles-pp.xfs_bmap_btalloc_best_length 0.29 ± 41% -0.2 0.12 ± 34% perf-profile.children.cycles-pp.xfs_alloc_vextent_start_ag 0.27 ± 5% -0.2 0.10 ± 15% perf-profile.children.cycles-pp.iomap_write_iter 0.27 ± 40% -0.2 0.11 ± 31% perf-profile.children.cycles-pp.xfs_alloc_ag_vextent_near 0.59 ± 5% -0.1 0.45 ± 4% perf-profile.children.cycles-pp.open_last_lookups 0.20 ± 10% -0.1 0.06 ± 19% perf-profile.children.cycles-pp.lookup_open 0.20 ± 12% -0.1 0.07 ± 13% perf-profile.children.cycles-pp.__xfs_trans_commit 0.18 ± 9% -0.1 0.05 ± 47% perf-profile.children.cycles-pp.rename 0.17 ± 9% -0.1 0.05 ± 49% perf-profile.children.cycles-pp.xfs_generic_create 0.17 ± 12% -0.1 0.05 ± 46% perf-profile.children.cycles-pp.__x64_sys_rename 0.16 ± 10% -0.1 0.05 ± 46% perf-profile.children.cycles-pp.do_renameat2 0.16 ± 11% -0.1 0.05 ± 49% perf-profile.children.cycles-pp.xfs_create 0.16 ± 11% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.xlog_cil_commit 0.17 ± 7% -0.1 0.07 ± 12% perf-profile.children.cycles-pp.balance_dirty_pages_ratelimited_flags 0.16 ± 5% -0.1 0.07 ± 14% perf-profile.children.cycles-pp.do_flush_stats 0.16 ± 5% -0.1 0.07 ± 14% perf-profile.children.cycles-pp.balance_dirty_pages 0.16 ± 5% -0.1 0.07 ± 14% perf-profile.children.cycles-pp.mem_cgroup_wb_stats 0.16 ± 5% -0.1 0.07 ± 14% perf-profile.children.cycles-pp.cgroup_rstat_flush 0.16 ± 5% -0.1 0.07 ± 14% perf-profile.children.cycles-pp.cgroup_rstat_flush_locked 0.11 ± 6% -0.1 0.03 ±100% perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush 0.76 ± 2% -0.1 0.69 perf-profile.children.cycles-pp.__fput 0.32 ± 8% -0.1 0.25 ± 6% perf-profile.children.cycles-pp.inode_permission 0.29 ± 8% -0.1 0.22 ± 8% perf-profile.children.cycles-pp.generic_permission 0.31 ± 5% -0.1 0.25 ± 3% perf-profile.children.cycles-pp.link_path_walk 0.14 ± 15% -0.1 0.08 ± 11% perf-profile.children.cycles-pp.__slab_free 0.82 ± 3% -0.1 0.76 perf-profile.children.cycles-pp.do_open 0.49 ± 2% -0.1 0.44 ± 2% perf-profile.children.cycles-pp.xfs_release 1.10 ± 2% -0.1 1.05 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 0.38 ± 4% -0.0 0.34 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc 0.07 ± 8% -0.0 0.04 ± 44% perf-profile.children.cycles-pp.touch_atime 0.29 ± 2% -0.0 0.26 ± 2% perf-profile.children.cycles-pp.xfs_can_free_eofblocks 0.23 ± 4% -0.0 0.20 ± 3% perf-profile.children.cycles-pp.may_open 0.15 ± 3% -0.0 0.13 ± 4% perf-profile.children.cycles-pp.xfs_bmapi_read 0.13 ± 2% -0.0 0.11 ± 3% perf-profile.children.cycles-pp.xfs_iext_lookup_extent 0.10 ± 5% -0.0 0.08 ± 4% perf-profile.children.cycles-pp.kmem_cache_free 0.07 ± 6% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.___slab_alloc 0.11 ± 3% -0.0 0.09 ± 5% perf-profile.children.cycles-pp.down_read 0.08 ± 5% -0.0 0.07 ± 9% perf-profile.children.cycles-pp.errseq_sample 0.01 ±223% +0.2 0.20 ± 44% perf-profile.children.cycles-pp.intel_idle 0.01 ±223% +0.2 0.20 ± 45% perf-profile.children.cycles-pp.cpuidle_enter 0.01 ±223% +0.2 0.20 ± 45% perf-profile.children.cycles-pp.cpuidle_enter_state 0.01 ±223% +0.2 0.20 ± 47% perf-profile.children.cycles-pp.start_secondary 0.01 ±223% +0.2 0.21 ± 46% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 0.01 ±223% +0.2 0.21 ± 46% perf-profile.children.cycles-pp.cpu_startup_entry 0.01 ±223% +0.2 0.21 ± 46% perf-profile.children.cycles-pp.do_idle 0.01 ±223% +0.2 0.21 ± 46% perf-profile.children.cycles-pp.cpuidle_idle_call 5.86 +0.4 6.30 perf-profile.children.cycles-pp.put_unused_fd 37.68 +0.5 38.18 perf-profile.children.cycles-pp.__close 37.38 +0.5 37.89 perf-profile.children.cycles-pp.__x64_sys_close 36.40 +0.6 36.98 perf-profile.children.cycles-pp.close_fd_get_file 0.59 ± 4% +0.7 1.24 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.61 ± 4% +0.7 1.27 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.25 ± 10% +0.7 0.92 ± 3% perf-profile.children.cycles-pp.__irq_exit_rcu 41.65 +1.2 42.86 perf-profile.children.cycles-pp.alloc_fd 51.13 +1.3 52.43 perf-profile.children.cycles-pp.__open64 50.61 +1.3 51.94 perf-profile.children.cycles-pp.__x64_sys_openat 50.60 +1.3 51.92 perf-profile.children.cycles-pp.do_sys_openat2 96.92 +1.4 98.35 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 96.89 +1.4 98.32 perf-profile.children.cycles-pp.do_syscall_64 83.67 +2.2 85.87 perf-profile.children.cycles-pp._raw_spin_lock 83.10 +2.2 85.33 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 1.85 ± 11% -0.5 1.36 ± 3% perf-profile.self.cycles-pp.file_free_rcu 0.28 ± 8% -0.1 0.21 ± 7% perf-profile.self.cycles-pp.generic_permission 0.14 ± 13% -0.1 0.08 ± 11% perf-profile.self.cycles-pp.__slab_free 1.02 ± 2% -0.0 0.97 perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.58 ± 2% -0.0 0.54 ± 2% perf-profile.self.cycles-pp._raw_spin_lock 0.13 ± 3% -0.0 0.11 ± 4% perf-profile.self.cycles-pp.xfs_iext_lookup_extent 0.10 ± 3% -0.0 0.08 ± 5% perf-profile.self.cycles-pp.down_read 0.40 +0.0 0.43 perf-profile.self.cycles-pp._copy_to_iter 0.01 ±223% +0.2 0.20 ± 44% perf-profile.self.cycles-pp.intel_idle 82.65 +1.6 84.27 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath 0.03 ± 17% -41.0% 0.02 ± 14% perf-sched.sch_delay.avg.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.06 ± 77% -86.0% 0.01 ±181% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_createname 0.12 ± 47% -94.3% 0.01 ±205% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_lookup 0.15 ±101% -84.6% 0.02 ±157% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_removename 0.01 ± 72% +183.3% 0.02 ± 20% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.10 ± 25% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats 0.08 ±101% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_free_eofblocks 0.03 ± 85% -91.9% 0.00 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_create.xfs_generic_create 0.12 ±102% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_free_eofblocks.xfs_release 0.14 ± 31% -85.3% 0.02 ±141% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_iget_cache_hit.xfs_iget 0.22 ±115% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 0.65 ± 59% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.dput.do_mkdirat.__x64_sys_mkdir.do_syscall_64 0.23 ± 3% -76.9% 0.05 ± 77% perf-sched.sch_delay.avg.ms.__cond_resched.dput.open_last_lookups.path_openat.do_filp_open 0.08 ± 86% -97.0% 0.00 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.getname_flags.part.0 0.13 ± 48% -86.7% 0.02 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_inode_item_init.xfs_trans_ijoin.xfs_init_new_inode 0.07 ± 73% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_free_eofblocks.xfs_release 0.08 ± 69% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_rename.xfs_vn_rename 0.04 ±111% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate.xfs_create 0.07 ±115% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve 0.07 ± 92% -90.1% 0.01 ±152% perf-sched.sch_delay.avg.ms.__cond_resched.mnt_want_write.do_renameat2.__x64_sys_rename.do_syscall_64 0.04 ± 74% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mnt_want_write.open_last_lookups.path_openat.do_filp_open 0.27 ± 70% +7399.7% 19.94 ±189% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.04 ±152% +18500.8% 7.32 ±212% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.17 ± 31% +159.0% 0.44 ± 26% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.04 ±103% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.05 ±133% +4237.8% 2.10 ± 60% perf-sched.sch_delay.avg.ms.kthreadd.ret_from_fork.ret_from_fork_asm 0.03 ± 61% +4840.1% 1.62 ±151% perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.11 ± 41% +867.3% 1.03 ± 32% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 1.08 ± 58% +114.8% 2.32 ± 27% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.25 ± 27% +155.1% 0.64 ± 18% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.do_renameat2 0.30 ± 29% +574.6% 2.05 ±142% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.open_last_lookups 0.16 ± 5% +207.2% 0.48 ± 22% perf-sched.sch_delay.avg.ms.schedule_timeout.___down_common.__down.down 0.21 ± 33% +266.2% 0.76 ± 18% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.03 ± 65% +6017.7% 2.02 ± 48% perf-sched.sch_delay.avg.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq 0.06 ±104% +3279.9% 1.99 ± 14% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.04 ± 22% +2605.6% 1.06 ± 17% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.08 ± 46% +1680.3% 1.39 ± 13% perf-sched.sch_delay.avg.ms.schedule_timeout.xfsaild.kthread.ret_from_fork 0.10 ± 11% +3086.2% 3.05 perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.05 ± 27% +4472.4% 2.45 ± 24% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 6.77 ± 40% -50.6% 3.34 ± 40% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.folio_alloc.__filemap_get_folio.iomap_write_begin 7.14 ± 12% -22.4% 5.54 ± 15% perf-sched.sch_delay.max.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.74 ± 66% -96.8% 0.02 ±206% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_createname 3.30 ± 47% -99.2% 0.03 ±197% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_lookup 2.51 ±110% -97.1% 0.07 ±147% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_removename 1.73 ±119% -97.4% 0.04 ±130% perf-sched.sch_delay.max.ms.__cond_resched.begin_current_label_crit_section.apparmor_path_rename.part.0 5.07 ± 33% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats 5.86 ± 70% -84.4% 0.91 ±138% perf-sched.sch_delay.max.ms.__cond_resched.down.xfs_buf_lock.xfs_buf_find_lock.xfs_buf_lookup 2.36 ±126% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_free_eofblocks 0.23 ±124% -98.9% 0.00 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_create.xfs_generic_create 1.42 ±106% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_free_eofblocks.xfs_release 1.64 ± 38% -97.1% 0.05 ±143% perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_iget_cache_hit.xfs_iget 0.90 ±125% -85.9% 0.13 ±201% perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_ilock_for_iomap.xfs_buffered_write_iomap_begin 0.23 ± 93% -82.1% 0.04 ±149% perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_lock_inodes.constprop 0.86 ± 98% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 5.26 ± 52% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.dput.do_mkdirat.__x64_sys_mkdir.do_syscall_64 10.75 ± 26% -97.1% 0.31 ±110% perf-sched.sch_delay.max.ms.__cond_resched.dput.open_last_lookups.path_openat.do_filp_open 6.74 ± 34% -92.5% 0.50 ±117% perf-sched.sch_delay.max.ms.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 1.92 ± 82% -97.9% 0.04 ±212% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.getname_flags.part.0 0.39 ±139% -93.3% 0.03 ±223% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_dup_cursor.xfs_btree_dup_cursor 1.99 ± 65% -97.6% 0.05 ±223% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_inode_item_init.xfs_trans_ijoin.xfs_init_new_inode 0.30 ±114% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_free_eofblocks.xfs_release 0.74 ± 91% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_rename.xfs_vn_rename 0.53 ±125% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate.xfs_create 0.94 ±122% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve 1.30 ±121% -97.9% 0.03 ±183% perf-sched.sch_delay.max.ms.__cond_resched.mnt_want_write.do_renameat2.__x64_sys_rename.do_syscall_64 0.71 ±110% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mnt_want_write.open_last_lookups.path_openat.do_filp_open 0.25 ± 42% -78.7% 0.05 ±126% perf-sched.sch_delay.max.ms.__cond_resched.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.46 ± 70% +7019.5% 175.26 ±211% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.82 ±158% +20809.2% 170.93 ±217% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 2.21 ±122% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.17 ±180% +1920.1% 3.51 ± 40% perf-sched.sch_delay.max.ms.kthreadd.ret_from_fork.ret_from_fork_asm 19.18 ± 23% +468.8% 109.07 ± 31% perf-sched.sch_delay.max.ms.schedule_timeout.___down_common.__down.down 0.06 ± 94% +4671.5% 2.76 ± 28% perf-sched.sch_delay.max.ms.schedule_timeout.io_schedule_timeout.__wait_for_common.blk_execute_rq 0.74 ±152% +502.7% 4.47 ± 13% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 5.84 ± 71% +1283.7% 80.87 ±100% perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 2.38 ± 58% +128.8% 5.44 ± 28% perf-sched.sch_delay.max.ms.schedule_timeout.xfsaild.kthread.ret_from_fork 54.67 ±201% +1136.3% 675.85 ± 53% perf-sched.sch_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.10 ± 5% +818.1% 0.93 ± 10% perf-sched.total_sch_delay.average.ms 74.20 ±137% +1036.6% 843.38 ± 39% perf-sched.total_sch_delay.max.ms 23.12 ± 8% +83.1% 42.32 ± 8% perf-sched.total_wait_and_delay.average.ms 67776 ± 10% -45.6% 36844 ± 8% perf-sched.total_wait_and_delay.count.ms 23.02 ± 8% +79.8% 41.39 ± 8% perf-sched.total_wait_time.average.ms 7.15 +25.0% 8.94 ± 3% perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 1.51 ± 9% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.dput.open_last_lookups.path_openat.do_filp_open 1.47 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 2.32 ± 13% +28.6% 2.98 ± 19% perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 190.08 ± 51% +328.2% 814.00 ± 5% perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 58.27 ± 21% +104.8% 119.35 ± 25% perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 230.25 ± 4% +8.5% 249.75 ± 2% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 251.07 ± 9% +73.5% 435.71 ± 17% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 2.80 ± 9% +137.2% 6.64 ± 20% perf-sched.wait_and_delay.avg.ms.schedule_timeout.___down_common.__down.down 4.38 +43.4% 6.28 ± 6% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 144.32 ± 5% -13.6% 124.67 ± 8% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 7271 ± 12% -47.1% 3849 ± 8% perf-sched.wait_and_delay.count.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 576.33 -16.8% 479.50 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 2795 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.dput.open_last_lookups.path_openat.do_filp_open 622.33 ± 21% -61.3% 240.83 ± 6% perf-sched.wait_and_delay.count.__cond_resched.dput.terminate_walk.path_openat.do_filp_open 944.67 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 8111 ± 14% -45.0% 4458 ± 7% perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 99.00 ± 22% -99.2% 0.83 ±107% perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 3732 ± 19% -63.0% 1381 ± 8% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 52.83 ± 10% -41.6% 30.83 ± 14% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 26865 ± 6% -45.9% 14540 ± 27% perf-sched.wait_and_delay.count.schedule_timeout.___down_common.__down.down 1147 -16.0% 964.67 ± 4% perf-sched.wait_and_delay.count.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 6233 ± 5% +16.1% 7233 ± 6% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 32.01 ± 22% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.dput.open_last_lookups.path_openat.do_filp_open 23.43 ± 30% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 317.91 ± 6% -76.5% 74.78 ±104% perf-sched.wait_and_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1032 +32.3% 1366 ± 10% perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 15.60 ± 52% +554.0% 102.01 ± 69% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 1034 +63.2% 1688 ± 26% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 2.22 ± 10% +58.1% 3.51 ± 27% perf-sched.wait_time.avg.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.39 ± 37% -82.6% 0.24 ±165% perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_iext_insert 1.34 ± 55% -70.4% 0.40 ±147% perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc_node.memcg_alloc_slab_cgroups.allocate_slab 7.14 +24.9% 8.92 ± 3% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 1.34 ± 10% -41.3% 0.78 ± 32% perf-sched.wait_time.avg.ms.__cond_resched.down_read.xfs_can_free_eofblocks.xfs_release.__fput 0.98 ± 56% -71.5% 0.28 ±142% perf-sched.wait_time.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_vn_update_time 1.62 ± 52% -90.7% 0.15 ±216% perf-sched.wait_time.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_create.xfs_generic_create 1.28 ± 37% -73.0% 0.34 ±149% perf-sched.wait_time.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_ilock_for_iomap.xfs_buffered_write_iomap_begin 0.13 ±136% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 2.43 ± 40% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.dput.do_mkdirat.__x64_sys_mkdir.do_syscall_64 1.20 ± 55% -97.3% 0.03 ±168% perf-sched.wait_time.avg.ms.__cond_resched.dput.do_renameat2.__x64_sys_rename.do_syscall_64 0.75 ± 85% -90.1% 0.07 ±160% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc._xfs_buf_alloc.xfs_buf_find_insert.constprop 2.30 ± 13% +28.9% 2.96 ± 20% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 1.58 ± 10% -41.7% 0.92 ± 12% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.security_file_alloc.init_file.alloc_empty_file 0.73 ± 69% -96.2% 0.03 ±168% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_iunlink_log_inode.xfs_iunlink.xfs_rename 1.68 ± 46% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_free_eofblocks.xfs_release 1.46 ± 23% -84.6% 0.22 ±103% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate.xfs_create 1.81 ± 61% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve 1.50 ± 17% -41.5% 0.88 ± 16% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_read.do_syscall_64 1.07 ± 45% -93.4% 0.07 ±201% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_write.do_syscall_64 1.54 ± 29% -54.6% 0.70 ± 61% perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 1.99 ± 94% -97.5% 0.05 ±223% perf-sched.wait_time.avg.ms.__cond_resched.xfs_trans_alloc.xfs_free_eofblocks.xfs_release.__fput 189.81 ± 51% +318.3% 794.07 ± 2% perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 58.23 ± 21% +92.4% 112.03 ± 19% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 2.16 ± 12% -33.2% 1.44 ± 23% perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 250.97 ± 9% +73.2% 434.68 ± 17% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 1.56 ± 15% +182.8% 4.40 ± 53% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.open_last_lookups 1.78 ± 24% +255.3% 6.31 ± 18% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.do_renameat2 1.62 ± 32% +433.7% 8.65 ± 97% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.open_last_lookups 2.64 ± 9% +133.0% 6.16 ± 20% perf-sched.wait_time.avg.ms.schedule_timeout.___down_common.__down.down 1.20 ±141% +299.1% 4.79 ± 75% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.__flush_work.isra.0 4.34 +20.4% 5.23 ± 4% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 144.23 ± 5% -15.7% 121.62 ± 8% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 7.12 ± 18% -76.3% 1.68 ±159% perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_removename 6.25 ± 47% -90.4% 0.60 ±175% perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_iext_insert 3.96 ± 50% -89.9% 0.40 ±146% perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc_node.memcg_alloc_slab_cgroups.allocate_slab 9.08 ± 49% -68.6% 2.85 ± 42% perf-sched.wait_time.max.ms.__cond_resched.apparmor_file_alloc_security.security_file_alloc.init_file.alloc_empty_file 11.70 ± 13% -66.1% 3.97 ± 77% perf-sched.wait_time.max.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats 38.74 ±134% -97.7% 0.88 ±175% perf-sched.wait_time.max.ms.__cond_resched.dentry_kill.dput.do_renameat2.__x64_sys_rename 7.04 ± 29% -42.9% 4.02 perf-sched.wait_time.max.ms.__cond_resched.down_read.xfs_can_free_eofblocks.xfs_release.__fput 7.87 ± 34% -82.5% 1.38 ± 89% perf-sched.wait_time.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_create 10.82 ± 23% -59.2% 4.42 ± 52% perf-sched.wait_time.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_rename 2.57 ± 61% -84.1% 0.41 ±147% perf-sched.wait_time.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_vn_update_time 6.42 ± 56% -97.7% 0.15 ±216% perf-sched.wait_time.max.ms.__cond_resched.down_write.xfs_ilock.xfs_create.xfs_generic_create 5.92 ± 52% -84.6% 0.91 ±218% perf-sched.wait_time.max.ms.__cond_resched.down_write.xfs_ilock.xfs_free_eofblocks.xfs_release 7.63 ± 36% -95.1% 0.37 ±142% perf-sched.wait_time.max.ms.__cond_resched.down_write.xfs_ilock.xfs_ilock_for_iomap.xfs_buffered_write_iomap_begin 0.67 ±157% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 11.40 ± 41% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.dput.do_mkdirat.__x64_sys_mkdir.do_syscall_64 5.50 ± 57% -99.4% 0.03 ±168% perf-sched.wait_time.max.ms.__cond_resched.dput.do_renameat2.__x64_sys_rename.do_syscall_64 25.24 ± 23% -74.8% 6.36 ± 58% perf-sched.wait_time.max.ms.__cond_resched.dput.open_last_lookups.path_openat.do_filp_open 20.87 ± 17% -70.5% 6.16 ± 50% perf-sched.wait_time.max.ms.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 1.84 ± 79% -95.8% 0.08 ±155% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc._xfs_buf_alloc.xfs_buf_find_insert.constprop 4.87 ± 44% -79.3% 1.01 ±140% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_dup_cursor.xfs_btree_dup_cursor 1.38 ± 70% -98.0% 0.03 ±168% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_iunlink_log_inode.xfs_iunlink.xfs_rename 5.49 ± 37% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_free_eofblocks.xfs_release 8.34 ± 32% -95.6% 0.36 ±113% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate.xfs_create 4.34 ± 26% -66.7% 1.44 ±142% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_vn_update_time.kiocb_modified 9.40 ± 43% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xlog_ticket_alloc.xfs_log_reserve.xfs_trans_reserve 10.08 ± 30% -61.2% 3.91 ± 98% perf-sched.wait_time.max.ms.__cond_resched.mnt_want_write.do_renameat2.__x64_sys_rename.do_syscall_64 6.53 ± 48% -86.8% 0.86 ±165% perf-sched.wait_time.max.ms.__cond_resched.mnt_want_write.open_last_lookups.path_openat.do_filp_open 9.41 ± 17% -47.4% 4.95 ± 23% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_read.do_syscall_64 7.34 ± 84% -98.2% 0.13 ±211% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.__fdget_pos.ksys_write.do_syscall_64 10.79 ± 60% -83.8% 1.75 ±157% perf-sched.wait_time.max.ms.__cond_resched.slab_pre_alloc_hook.constprop.0.kmem_cache_alloc_lru 317.83 ± 6% -74.6% 80.68 ± 86% perf-sched.wait_time.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 5.50 ± 30% -53.7% 2.55 ± 56% perf-sched.wait_time.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 3.45 ± 47% -98.5% 0.05 ±223% perf-sched.wait_time.max.ms.__cond_resched.xfs_trans_alloc.xfs_free_eofblocks.xfs_release.__fput 6.53 ± 53% -88.0% 0.79 ±223% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single 172.54 ±214% -99.6% 0.69 ±151% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 1032 +32.3% 1366 ± 10% perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 13.16 ± 22% +79.6% 23.63 ± 20% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.do_renameat2 9.36 ± 21% +2228.4% 218.04 ±165% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.open_last_lookups 3.58 ±141% +369.9% 16.83 ± 70% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.__flush_work.isra.0 11.22 ± 49% +308.4% 45.82 ± 84% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 1034 +63.2% 1688 ± 26% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm *************************************************************************************************** lkp-icl-2sp8: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory ========================================================================================= class/compiler/cpufreq_governor/disk/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: exec_spawn/gcc-12/performance/1HDD/x86_64-rhel-8.3/100%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp8/exec/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 82.25 +3.6% 85.24 iostat.cpu.system 8.48 -32.5% 5.73 iostat.cpu.user 1.19 -0.2 1.01 mpstat.cpu.all.irq% 0.53 +0.1 0.67 ± 2% mpstat.cpu.all.soft% 8.73 -2.8 5.89 mpstat.cpu.all.usr% 151.83 ± 4% +79.1% 272.00 ± 5% vmstat.procs.r 88287 -45.7% 47970 ± 2% vmstat.system.cs 117994 -13.4% 102190 vmstat.system.in 79317172 -34.2% 52217838 ± 3% numa-numastat.node0.local_node 79360983 -34.2% 52240348 ± 3% numa-numastat.node0.numa_hit 77971050 -33.2% 52068168 ± 3% numa-numastat.node1.local_node 78009071 -33.2% 52089987 ± 3% numa-numastat.node1.numa_hit 6311 ± 2% -12.6% 5512 ± 3% perf-c2c.DRAM.remote 5297 -33.4% 3527 ± 4% perf-c2c.HITM.local 3611 ± 2% -37.0% 2276 ± 4% perf-c2c.HITM.remote 8908 -34.8% 5804 ± 4% perf-c2c.HITM.total 361953 ± 8% -17.5% 298533 ± 9% turbostat.C1 0.14 ± 5% -30.2% 0.10 turbostat.IPC 7803781 -13.6% 6739655 turbostat.IRQ 13.85 ± 18% -13.8 0.09 ±123% turbostat.PKG_% 272.00 ± 43% -58.6% 112.67 ± 28% turbostat.POLL 400.63 -4.7% 381.62 turbostat.PkgWatt 97.98 -11.2% 87.05 turbostat.RAMWatt 943488 -34.6% 617225 ± 3% stress-ng.exec.ops 15709 -34.6% 10274 ± 3% stress-ng.exec.ops_per_sec 1152325 -45.6% 627349 stress-ng.time.involuntary_context_switches 367062 -41.4% 215251 ± 3% stress-ng.time.major_page_faults 2.058e+08 -33.3% 1.372e+08 ± 3% stress-ng.time.minor_page_faults 3144 +4.3% 3281 stress-ng.time.system_time 301.60 -37.2% 189.54 ± 2% stress-ng.time.user_time 2253503 -38.6% 1384317 ± 3% stress-ng.time.voluntary_context_switches 391400 ± 4% +13.2% 443196 ± 3% meminfo.Active 391256 ± 4% +13.2% 443060 ± 3% meminfo.Active(anon) 2301202 ± 2% -44.5% 1277982 ± 4% meminfo.AnonPages 3028901 ± 2% -45.9% 1638075 ± 4% meminfo.Inactive 3028698 ± 2% -45.9% 1637865 ± 4% meminfo.Inactive(anon) 733055 -50.6% 362045 ± 4% meminfo.Mapped 7451969 -18.3% 6086630 meminfo.Memused 249614 +13.3% 282720 meminfo.SUnreclaim 1084892 -28.8% 772281 ± 2% meminfo.Shmem 7905146 -15.7% 6660361 meminfo.max_used_kB 585668 ± 36% -37.4% 366632 ± 19% numa-meminfo.node0.Inactive 585629 ± 36% -37.4% 366493 ± 19% numa-meminfo.node0.Inactive(anon) 12587 ± 4% +6.6% 13419 ± 3% numa-meminfo.node0.KernelStack 241837 ± 15% -54.9% 108956 ± 14% numa-meminfo.node0.Mapped 131373 ± 4% +14.2% 150063 numa-meminfo.node0.SUnreclaim 321414 ± 28% -46.4% 172158 ± 21% numa-meminfo.node0.Shmem 268099 ± 8% +27.7% 342267 ± 4% numa-meminfo.node1.Active 267979 ± 8% +27.7% 342221 ± 4% numa-meminfo.node1.Active(anon) 1927265 ± 7% -48.5% 991743 ± 11% numa-meminfo.node1.AnonPages 6126082 ± 9% -60.6% 2411821 ± 3% numa-meminfo.node1.AnonPages.max 2441426 ± 7% -48.1% 1267741 ± 9% numa-meminfo.node1.Inactive 2441262 ± 7% -48.1% 1267671 ± 9% numa-meminfo.node1.Inactive(anon) 491202 ± 7% -49.3% 248842 ± 9% numa-meminfo.node1.Mapped 763522 ± 11% -21.3% 600904 ± 5% numa-meminfo.node1.Shmem 146263 ± 36% -37.7% 91143 ± 18% numa-vmstat.node0.nr_inactive_anon 60515 ± 15% -54.9% 27318 ± 15% numa-vmstat.node0.nr_mapped 80250 ± 28% -46.7% 42760 ± 21% numa-vmstat.node0.nr_shmem 32997 ± 4% +13.7% 37503 numa-vmstat.node0.nr_slab_unreclaimable 146263 ± 36% -37.7% 91144 ± 18% numa-vmstat.node0.nr_zone_inactive_anon 79361000 -34.2% 52240236 ± 3% numa-vmstat.node0.numa_hit 79317189 -34.2% 52217726 ± 3% numa-vmstat.node0.numa_local 66840 ± 8% +26.7% 84693 ± 5% numa-vmstat.node1.nr_active_anon 483705 ± 6% -48.6% 248549 ± 11% numa-vmstat.node1.nr_anon_pages 612356 ± 7% -48.1% 317581 ± 9% numa-vmstat.node1.nr_inactive_anon 122952 ± 7% -49.4% 62247 ± 8% numa-vmstat.node1.nr_mapped 190826 ± 11% -21.7% 149413 ± 5% numa-vmstat.node1.nr_shmem 66840 ± 8% +26.7% 84693 ± 5% numa-vmstat.node1.nr_zone_active_anon 612357 ± 7% -48.1% 317580 ± 9% numa-vmstat.node1.nr_zone_inactive_anon 78008757 -33.2% 52089958 ± 3% numa-vmstat.node1.numa_hit 77970736 -33.2% 52068139 ± 3% numa-vmstat.node1.numa_local 97892 ± 2% +14.4% 112017 proc-vmstat.nr_active_anon 576196 ± 2% -44.8% 317799 ± 4% proc-vmstat.nr_anon_pages 958147 -8.1% 880525 proc-vmstat.nr_file_pages 758356 -46.3% 407365 ± 4% proc-vmstat.nr_inactive_anon 24263 +6.0% 25716 proc-vmstat.nr_kernel_stack 183853 -51.1% 89950 ± 3% proc-vmstat.nr_mapped 271480 -28.6% 193835 ± 2% proc-vmstat.nr_shmem 62515 +12.6% 70382 proc-vmstat.nr_slab_unreclaimable 97892 ± 2% +14.4% 112017 proc-vmstat.nr_zone_active_anon 758356 -46.3% 407365 ± 4% proc-vmstat.nr_zone_inactive_anon 1.574e+08 -33.7% 1.043e+08 ± 3% proc-vmstat.numa_hit 1.573e+08 -33.7% 1.043e+08 ± 3% proc-vmstat.numa_local 360578 -12.9% 313907 ± 2% proc-vmstat.pgactivate 1.646e+08 -33.8% 1.09e+08 ± 3% proc-vmstat.pgalloc_normal 2.177e+08 -33.2% 1.453e+08 ± 3% proc-vmstat.pgfault 1.632e+08 -33.5% 1.086e+08 ± 3% proc-vmstat.pgfree 2457518 -30.9% 1697138 ± 2% proc-vmstat.pgreuse 3067 -31.7% 2095 ± 2% proc-vmstat.thp_fault_alloc 3027 ± 2% -33.5% 2012 ± 5% proc-vmstat.thp_split_pmd 177531 ± 3% -39.1% 108181 ± 4% proc-vmstat.unevictable_pgs_culled 1773286 +41.2% 2504364 sched_debug.cfs_rq:/.avg_vruntime.avg 2257333 ± 5% +35.3% 3053999 ± 4% sched_debug.cfs_rq:/.avg_vruntime.max 1312190 ± 2% +61.8% 2122831 ± 5% sched_debug.cfs_rq:/.avg_vruntime.min 94694 ± 11% +44.6% 136926 ± 6% sched_debug.cfs_rq:/.avg_vruntime.stddev 257547 ± 33% +195.6% 761221 ± 24% sched_debug.cfs_rq:/.left_vruntime.avg 1827692 +57.3% 2875024 ± 6% sched_debug.cfs_rq:/.left_vruntime.max 611191 ± 14% +85.5% 1133544 ± 8% sched_debug.cfs_rq:/.left_vruntime.stddev 118843 ± 17% +93.5% 230014 ± 21% sched_debug.cfs_rq:/.load.avg 1151341 ± 16% +48.9% 1713855 ± 20% sched_debug.cfs_rq:/.load.max 269930 ± 6% +40.0% 377785 ± 14% sched_debug.cfs_rq:/.load.stddev 1773288 +41.2% 2504386 sched_debug.cfs_rq:/.min_vruntime.avg 2257333 ± 5% +35.3% 3053992 ± 4% sched_debug.cfs_rq:/.min_vruntime.max 1312190 ± 2% +61.8% 2122809 ± 5% sched_debug.cfs_rq:/.min_vruntime.min 94694 ± 11% +44.6% 136925 ± 6% sched_debug.cfs_rq:/.min_vruntime.stddev 0.71 ± 5% +14.2% 0.81 ± 8% sched_debug.cfs_rq:/.nr_running.avg 31.58 ± 16% -37.1% 19.86 ± 28% sched_debug.cfs_rq:/.removed.util_avg.avg 257547 ± 33% +195.6% 761225 ± 24% sched_debug.cfs_rq:/.right_vruntime.avg 1827692 +57.3% 2875024 ± 6% sched_debug.cfs_rq:/.right_vruntime.max 611191 ± 14% +85.5% 1133548 ± 8% sched_debug.cfs_rq:/.right_vruntime.stddev 916.68 ± 8% +142.2% 2220 ± 92% sched_debug.cfs_rq:/.runnable_avg.avg 2254 ± 24% +262.5% 8171 ± 87% sched_debug.cfs_rq:/.runnable_avg.max 421.39 ± 20% +353.5% 1910 ±114% sched_debug.cfs_rq:/.runnable_avg.stddev 153.00 ± 22% -51.5% 74.17 ± 55% sched_debug.cfs_rq:/.util_avg.min 6.19 ± 8% +40.0% 8.67 ± 11% sched_debug.cpu.clock.stddev 532031 ± 4% -37.6% 332043 ± 8% sched_debug.cpu.curr->pid.avg 700350 -34.9% 455732 ± 2% sched_debug.cpu.curr->pid.max 296839 ± 3% -33.0% 198864 ± 6% sched_debug.cpu.curr->pid.stddev 46273 -43.3% 26252 sched_debug.cpu.nr_switches.avg 91990 ± 12% -25.4% 68587 ± 4% sched_debug.cpu.nr_switches.max 32267 ± 4% -38.5% 19850 ± 4% sched_debug.cpu.nr_switches.min 228.08 ± 10% -47.9% 118.83 ± 25% sched_debug.cpu.nr_uninterruptible.max -825.92 -72.4% -227.75 sched_debug.cpu.nr_uninterruptible.min 133.85 ± 14% -58.5% 55.54 ± 7% sched_debug.cpu.nr_uninterruptible.stddev 3.77 -5.9% 3.54 perf-stat.i.MPKI 1.835e+10 -26.6% 1.346e+10 perf-stat.i.branch-instructions 1.76 -0.2 1.58 perf-stat.i.branch-miss-rate% 3.211e+08 -34.1% 2.116e+08 ± 2% perf-stat.i.branch-misses 23.67 +0.8 24.51 perf-stat.i.cache-miss-rate% 3.485e+08 -32.2% 2.364e+08 ± 2% perf-stat.i.cache-misses 1.441e+09 -34.6% 9.418e+08 ± 2% perf-stat.i.cache-references 93021 -46.0% 50257 ± 2% perf-stat.i.context-switches 2.37 +40.4% 3.33 ± 2% perf-stat.i.cpi 24503 -34.2% 16111 ± 2% perf-stat.i.cpu-migrations 720.98 ± 2% +44.2% 1039 ± 2% perf-stat.i.cycles-between-cache-misses 0.11 ± 5% -0.0 0.10 ± 4% perf-stat.i.dTLB-load-miss-rate% 25736302 ± 4% -35.8% 16517247 ± 3% perf-stat.i.dTLB-load-misses 2.313e+10 -28.4% 1.657e+10 perf-stat.i.dTLB-loads 22962749 -34.3% 15087029 ± 2% perf-stat.i.dTLB-store-misses 1.078e+10 -34.4% 7.074e+09 ± 2% perf-stat.i.dTLB-stores 8.975e+10 -28.1% 6.454e+10 perf-stat.i.instructions 0.43 -26.5% 0.32 perf-stat.i.ipc 5967 -41.7% 3481 ± 3% perf-stat.i.major-faults 1399 +76.9% 2475 ± 2% perf-stat.i.metric.K/sec 840.84 -29.4% 593.80 perf-stat.i.metric.M/sec 3446130 -33.3% 2299865 ± 2% perf-stat.i.minor-faults 69.34 -6.6 62.78 perf-stat.i.node-load-miss-rate% 67098069 -26.7% 49212970 perf-stat.i.node-load-misses 24.81 -2.2 22.64 perf-stat.i.node-store-miss-rate% 25783451 -45.6% 14015341 ± 3% perf-stat.i.node-store-misses 78809232 -38.0% 48884286 ± 3% perf-stat.i.node-stores 3452098 -33.3% 2303346 ± 2% perf-stat.i.page-faults 3.88 -5.7% 3.66 perf-stat.overall.MPKI 1.75 -0.2 1.56 perf-stat.overall.branch-miss-rate% 24.24 +0.9 25.15 perf-stat.overall.cache-miss-rate% 2.42 +40.4% 3.39 ± 2% perf-stat.overall.cpi 622.01 +49.0% 926.59 ± 3% perf-stat.overall.cycles-between-cache-misses 0.11 ± 4% -0.0 0.10 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.41 -28.8% 0.29 ± 2% perf-stat.overall.ipc 68.86 -6.7 62.16 perf-stat.overall.node-load-miss-rate% 24.60 -2.3 22.25 perf-stat.overall.node-store-miss-rate% 1.811e+10 -26.9% 1.323e+10 perf-stat.ps.branch-instructions 3.163e+08 -34.6% 2.069e+08 ± 2% perf-stat.ps.branch-misses 3.439e+08 -32.5% 2.322e+08 ± 2% perf-stat.ps.cache-misses 1.419e+09 -34.9% 9.233e+08 ± 2% perf-stat.ps.cache-references 91435 -46.2% 49192 perf-stat.ps.context-switches 24084 -34.5% 15781 ± 2% perf-stat.ps.cpu-migrations 25583545 ± 4% -36.5% 16239719 ± 3% perf-stat.ps.dTLB-load-misses 2.283e+10 -28.7% 1.628e+10 perf-stat.ps.dTLB-loads 22666931 -34.7% 14802539 ± 2% perf-stat.ps.dTLB-store-misses 1.065e+10 -34.8% 6.945e+09 ± 2% perf-stat.ps.dTLB-stores 8.855e+10 -28.4% 6.34e+10 perf-stat.ps.instructions 5864 -41.9% 3405 ± 2% perf-stat.ps.major-faults 3397769 -33.6% 2255265 ± 2% perf-stat.ps.minor-faults 66094286 -26.7% 48443106 perf-stat.ps.node-load-misses 25346139 -45.9% 13711174 ± 3% perf-stat.ps.node-store-misses 77676625 -38.3% 47895043 ± 3% perf-stat.ps.node-stores 3403634 -33.6% 2258671 ± 2% perf-stat.ps.page-faults 5.608e+12 -28.2% 4.028e+12 ± 2% perf-stat.total.instructions 0.91 ± 16% -69.5% 0.28 ±102% perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.wp_page_copy 0.54 ± 35% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages.__get_free_pages.pgd_alloc.mm_init 1.60 ± 78% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages.get_zeroed_page.__p4d_alloc.__handle_mm_fault 0.38 ±102% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages.pte_alloc_one.__pte_alloc.move_page_tables 0.93 ± 15% -94.4% 0.05 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.__alloc_pages.pte_alloc_one.do_read_fault.do_fault 0.44 ± 52% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 0.39 ± 63% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__fput.task_work_run.do_exit.do_group_exit 0.32 ± 30% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.load_elf_binary.search_binary_handler 0.42 ± 37% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.load_elf_phdrs.load_elf_binary 0.66 ± 38% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.security_prepare_creds.prepare_creds 1.10 ± 15% -79.7% 0.22 ±102% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.security_task_alloc.copy_process 0.95 ± 50% -97.6% 0.02 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.alloc_bprm.do_execveat_common 0.00 ± 83% +511.1% 0.02 ± 20% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.20 ± 89% +615.2% 1.42 ± 6% perf-sched.sch_delay.avg.ms.__cond_resched.__wait_for_common.wait_for_completion_state.kernel_clone.__x64_sys_vfork 0.58 ± 36% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.apparmor_file_alloc_security.security_file_alloc.init_file.alloc_empty_file 0.28 ± 17% -48.9% 0.14 ± 49% perf-sched.sch_delay.avg.ms.__cond_resched.copy_page_range.dup_mmap.dup_mm.constprop 1.46 ± 54% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.count.constprop.0.isra 1.14 ± 59% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.dentry_kill.dput.step_into.path_lookupat 0.48 ± 41% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_read.acct_collect.do_exit.do_group_exit 1.31 ± 49% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_read.exit_mm.do_exit.do_group_exit 0.63 ± 65% -96.9% 0.02 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.down_read.exit_mmap.__mmput.exec_mmap 0.89 ± 78% -79.8% 0.18 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.down_read.exit_mmap.__mmput.exit_mm 0.05 ±126% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_read.get_arg_page.copy_string_kernel.copy_strings_kernel 0.61 ±112% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_timeout.__request_module.search_binary_handler.exec_binprm 0.63 ± 45% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault 0.46 ± 57% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.__anon_vma_prepare.do_cow_fault.do_fault 0.32 ± 27% -94.1% 0.02 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.__split_vma.do_vmi_align_munmap.do_vmi_munmap 0.19 ± 69% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.do_vmi_align_munmap.do_vmi_munmap.mmap_region 1.35 ±124% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vfs_unlink.do_unlinkat.__x64_sys_unlink 0.59 ± 57% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vma_expand.mmap_region.do_mmap 0.47 ± 94% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vma_expand.shift_arg_pages.setup_arg_pages 0.34 ±146% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vma_link.insert_vm_struct.__install_special_mapping 0.18 ± 73% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vma_prepare.vma_expand.mmap_region 0.59 ± 64% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.__vm_munmap.elf_map.load_elf_binary 0.58 ± 37% -96.0% 0.02 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.create_elf_tables.load_elf_binary.search_binary_handler 0.53 ± 45% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64 1.21 ±160% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.50 ± 19% -84.5% 0.08 ±142% perf-sched.sch_delay.avg.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.elf_map.load_elf_interp 0.12 ±115% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.dput.nd_jump_root.pick_link.step_into 1.12 ± 46% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.dput.path_openat.do_filp_open.do_sys_openat2 0.65 ± 79% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.dput.step_into.open_last_lookups.path_openat 0.76 ± 77% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.exit_signals.do_exit.do_group_exit.get_signal 0.34 ± 83% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.filemap_read.__kernel_read.load_elf_binary.search_binary_handler 1.51 ± 39% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write 0.69 ± 29% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault 0.53 ± 34% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_cow_fault.do_fault 1.21 ± 11% -77.1% 0.28 ±101% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 0.29 ±108% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.copy_sighand.copy_process.kernel_clone 0.08 ± 96% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.dup_mm.constprop.0 0.74 ± 70% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.expand_downwards 0.74 ± 28% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.vma_link 1.29 ± 27% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.mm_alloc.alloc_bprm.do_execveat_common 1.34 ± 76% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.prepare_creds.__do_sys_capset.do_syscall_64 1.57 ± 70% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.prepare_creds.prepare_exec_creds.bprm_execve 0.04 ± 71% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.prepare_kernel_cred.call_usermodehelper_exec_async.ret_from_fork 0.48 ± 15% -92.6% 0.04 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.security_file_alloc.init_file.alloc_empty_file 0.49 ± 24% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.taskstats_exit.do_exit.do_group_exit 0.09 ±185% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.vm_area_alloc.alloc_bprm.kernel_execve 0.40 ± 43% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_preallocate.__split_vma 0.33 ± 41% -86.6% 0.04 ±223% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc_node.dup_task_struct.copy_process.kernel_clone 0.25 ± 63% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mmput.exec_mmap.begin_new_exec.load_elf_binary 1.73 ± 4% +567.7% 11.52 ± 9% perf-sched.sch_delay.avg.ms.__cond_resched.mmput.exit_mm.do_exit.do_group_exit 0.64 ± 37% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mmput.free_bprm.do_execveat_common.isra 0.62 ± 51% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mnt_want_write.do_unlinkat.__x64_sys_unlink.do_syscall_64 1.92 ± 55% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mnt_want_write.open_last_lookups.path_openat.do_filp_open 1.23 ± 23% -84.7% 0.19 ±160% perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock.futex_exit_release.exit_mm_release.exit_mm 1.24 ± 47% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock_interruptible.bprm_execve.do_execveat_common.isra 0.75 ± 22% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.mutex_lock_killable.pcpu_alloc.mm_init.alloc_bprm 0.20 ± 37% +698.6% 1.60 ± 9% perf-sched.sch_delay.avg.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork 2.09 ± 4% +305.1% 8.46 ± 13% perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.36 ± 70% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 0.50 ± 6% +102.5% 1.01 ± 40% perf-sched.sch_delay.avg.ms.__cond_resched.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput 0.46 ± 44% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.tlb_batch_pages_flush.tlb_finish_mmu.shift_arg_pages.setup_arg_pages 0.57 ± 78% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill 0.47 ± 62% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.truncate_inode_pages_range.evict.do_unlinkat.__x64_sys_unlink 0.74 ± 34% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.unmap_vmas.exit_mmap.__mmput.exec_mmap 0.11 ±113% +1453.8% 1.66 ± 25% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.53 ± 3% +169.2% 1.43 ± 7% perf-sched.sch_delay.avg.ms.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 1.09 ± 7% +333.5% 4.73 ± 15% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 0.56 ± 12% +183.7% 1.59 ± 8% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 0.60 ± 9% +552.7% 3.92 ± 18% perf-sched.sch_delay.avg.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 0.17 ± 27% +778.2% 1.49 ± 8% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait.call_usermodehelper_exec_work.process_one_work 0.44 ± 27% +112.2% 0.94 ± 20% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 0.52 ± 3% +81.3% 0.94 ± 9% perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 1.59 ± 4% +298.5% 6.35 ± 18% perf-sched.sch_delay.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 0.47 ± 7% +133.6% 1.11 ± 8% perf-sched.sch_delay.avg.ms.io_schedule.folio_wait_bit_common.filemap_fault.__do_fault 0.22 ± 23% +348.6% 0.99 ± 8% perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.01 ± 48% +981.2% 0.12 ± 33% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 0.31 ±121% +335.9% 1.35 ± 36% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 0.54 ± 31% -100.0% 0.00 perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.__put_anon_vma 0.39 ± 6% +223.2% 1.26 ± 88% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.mmap_region 0.55 ± 11% +221.7% 1.76 ± 24% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma 0.39 ± 17% +87.5% 0.73 ± 27% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.vma_prepare 0.55 ± 3% +157.1% 1.41 ± 13% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec 0.35 ± 10% +208.8% 1.07 ± 15% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.19 ± 64% +536.1% 1.21 ± 16% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 0.17 ± 11% +961.9% 1.85 ± 8% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.06 ± 3% +775.3% 0.55 ± 6% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.04 ± 63% +720.5% 0.35 ± 17% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.06 ± 8% +1352.1% 0.88 ± 4% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 87.88 ±100% -88.8% 9.89 ±113% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.wp_page_copy 4.86 ± 17% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.__get_free_pages.pgd_alloc.mm_init 3.85 ± 33% -86.5% 0.52 ±161% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.__pmd_alloc.__handle_mm_fault.handle_mm_fault 3.12 ± 44% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.get_zeroed_page.__p4d_alloc.__handle_mm_fault 5.82 ± 34% -83.1% 0.98 ±206% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.pte_alloc_one.__pte_alloc.do_anonymous_page 1.31 ±109% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.pte_alloc_one.__pte_alloc.move_page_tables 9.59 ± 36% -97.3% 0.26 ±223% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.pte_alloc_one.do_read_fault.do_fault 4.08 ± 45% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__do_fault.do_read_fault.do_fault.__handle_mm_fault 1.43 ± 46% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__fput.task_work_run.do_exit.do_group_exit 6.99 ± 43% -90.5% 0.66 ±223% perf-sched.sch_delay.max.ms.__cond_resched.__get_user_pages.get_user_pages_remote.get_arg_page.copy_string_kernel 4.27 ± 56% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.load_elf_binary.search_binary_handler 5.95 ± 33% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.load_elf_phdrs.load_elf_binary 4.81 ± 13% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.security_prepare_creds.prepare_creds 8.56 ± 16% -79.8% 1.72 ±107% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.security_task_alloc.copy_process 2.18 ± 60% -99.0% 0.02 ±223% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.alloc_bprm.do_execveat_common 0.74 ± 81% +319.2% 3.12 ± 25% perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.45 ± 98% +584.3% 3.10 ± 11% perf-sched.sch_delay.max.ms.__cond_resched.__wait_for_common.wait_for_completion_state.kernel_clone.__x64_sys_vfork 3.28 ± 38% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.apparmor_file_alloc_security.security_file_alloc.init_file.alloc_empty_file 33.39 ± 80% -77.1% 7.66 ± 54% perf-sched.sch_delay.max.ms.__cond_resched.copy_page_range.dup_mmap.dup_mm.constprop 3.29 ± 55% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.count.constprop.0.isra 3.51 ± 68% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.dentry_kill.dput.step_into.path_lookupat 3.75 ± 36% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_read.acct_collect.do_exit.do_group_exit 6.09 ± 65% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_read.exit_mm.do_exit.do_group_exit 4.96 ± 34% -99.6% 0.02 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_read.exit_mmap.__mmput.exec_mmap 2.91 ± 94% -87.7% 0.36 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_read.exit_mmap.__mmput.exit_mm 0.07 ±103% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_read.get_arg_page.copy_string_kernel.copy_strings_kernel 4.08 ± 33% -86.9% 0.54 ±141% perf-sched.sch_delay.max.ms.__cond_resched.down_read.kernfs_iop_permission.inode_permission.link_path_walk 1.98 ± 90% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_timeout.__request_module.search_binary_handler.exec_binprm 3.93 ± 47% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault 3.73 ± 59% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.__anon_vma_prepare.do_cow_fault.do_fault 5.51 ± 27% -97.2% 0.15 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_write.__split_vma.do_vmi_align_munmap.do_vmi_munmap 5.68 ± 86% -89.1% 0.62 ±166% perf-sched.sch_delay.max.ms.__cond_resched.down_write.anon_vma_clone.__split_vma.mprotect_fixup 1.53 ± 52% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.do_vmi_align_munmap.do_vmi_munmap.mmap_region 2.96 ± 98% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.vfs_unlink.do_unlinkat.__x64_sys_unlink 3.06 ± 50% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.vma_expand.mmap_region.do_mmap 1.97 ± 85% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.vma_expand.shift_arg_pages.setup_arg_pages 0.69 ±145% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.vma_link.insert_vm_struct.__install_special_mapping 2.26 ± 75% -69.7% 0.69 ±141% perf-sched.sch_delay.max.ms.__cond_resched.down_write.vma_prepare.__split_vma.do_vmi_align_munmap 1.81 ± 68% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write.vma_prepare.vma_expand.mmap_region 3.46 ± 64% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.__vm_munmap.elf_map.load_elf_binary 4.32 ± 28% -99.5% 0.02 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.create_elf_tables.load_elf_binary.search_binary_handler 4.59 ± 44% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64 4.05 ± 23% -82.5% 0.71 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.setup_arg_pages.load_elf_binary.search_binary_handler 4.32 ±112% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 5.84 ± 42% -89.0% 0.64 ±141% perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.elf_map.load_elf_interp 3.66 ± 26% -86.8% 0.48 ±223% perf-sched.sch_delay.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 0.51 ±123% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.dput.nd_jump_root.pick_link.step_into 2.20 ± 67% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.dput.path_openat.do_filp_open.do_sys_openat2 2.45 ± 84% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.dput.step_into.open_last_lookups.path_openat 13.60 ± 23% -77.4% 3.08 ±109% perf-sched.sch_delay.max.ms.__cond_resched.exit_signals.do_exit.do_group_exit.__x64_sys_exit_group 5.17 ± 64% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.exit_signals.do_exit.do_group_exit.get_signal 3.11 ± 70% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.filemap_read.__kernel_read.load_elf_binary.search_binary_handler 1.54 ± 89% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.filemap_read.__kernel_read.load_elf_phdrs.load_elf_binary 6.70 ± 27% -81.3% 1.25 ±223% perf-sched.sch_delay.max.ms.__cond_resched.filemap_read.__kernel_read.search_binary_handler.exec_binprm 3.70 ± 55% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.generic_perform_write.generic_file_write_iter.vfs_write.ksys_write 6.70 ± 53% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_anonymous_page.__handle_mm_fault 4.49 ± 35% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_cow_fault.do_fault 8.44 ± 12% -82.6% 1.47 ± 92% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 2.49 ±129% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.copy_sighand.copy_process.kernel_clone 1.11 ± 89% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.dup_mm.constprop.0 2.68 ± 55% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.expand_downwards 3.73 ± 40% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.vma_link 3.51 ± 27% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.mm_alloc.alloc_bprm.do_execveat_common 2.48 ± 63% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.prepare_creds.__do_sys_capset.do_syscall_64 1.90 ± 70% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.prepare_creds.prepare_exec_creds.bprm_execve 0.83 ± 81% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.prepare_kernel_cred.call_usermodehelper_exec_async.ret_from_fork 5.60 ± 44% -94.9% 0.28 ±223% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.security_file_alloc.init_file.alloc_empty_file 2.85 ± 38% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.taskstats_exit.do_exit.do_group_exit 6.97 ± 35% -81.3% 1.30 ±111% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.vm_area_alloc.alloc_bprm.do_execveat_common 0.26 ±184% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.vm_area_alloc.alloc_bprm.kernel_execve 9.57 ± 14% -47.3% 5.04 ± 55% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.vm_area_dup.__split_vma.mprotect_fixup 1.74 ± 37% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_preallocate.__split_vma 2.73 ± 49% -93.5% 0.18 ±223% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc_node.dup_task_struct.copy_process.kernel_clone 4.09 ± 44% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mmput.exec_mmap.begin_new_exec.load_elf_binary 26.55 ± 18% +272.2% 98.83 ± 10% perf-sched.sch_delay.max.ms.__cond_resched.mmput.exit_mm.do_exit.do_group_exit 3.77 ± 29% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mmput.free_bprm.do_execveat_common.isra 2.34 ± 43% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mnt_want_write.do_unlinkat.__x64_sys_unlink.do_syscall_64 3.38 ± 67% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mnt_want_write.open_last_lookups.path_openat.do_filp_open 11.39 ± 40% -88.0% 1.37 ±155% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.futex_exit_release.exit_mm_release.exit_mm 2.51 ± 55% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock_interruptible.bprm_execve.do_execveat_common.isra 7.66 ± 34% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock_killable.pcpu_alloc.mm_init.alloc_bprm 3.29 ± 35% -87.1% 0.43 ±223% perf-sched.sch_delay.max.ms.__cond_resched.remove_vma.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap 5.81 ± 68% -81.0% 1.11 ±143% perf-sched.sch_delay.max.ms.__cond_resched.remove_vma.exit_mmap.__mmput.exec_mmap 4.97 ± 37% -71.6% 1.41 ±108% perf-sched.sch_delay.max.ms.__cond_resched.remove_vma.exit_mmap.__mmput.exit_mm 5.55 ± 14% -98.7% 0.07 ±202% perf-sched.sch_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 37.98 ± 17% +285.9% 146.59 ± 14% perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 2.00 ± 64% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 4.43 ± 48% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.tlb_batch_pages_flush.tlb_finish_mmu.shift_arg_pages.setup_arg_pages 2.63 ± 37% -76.1% 0.63 ±141% perf-sched.sch_delay.max.ms.__cond_resched.tlb_batch_pages_flush.tlb_finish_mmu.unmap_region.constprop 1.65 ± 94% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill 2.32 ± 96% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.truncate_inode_pages_range.evict.do_unlinkat.__x64_sys_unlink 1.82 ± 61% -89.2% 0.20 ±223% perf-sched.sch_delay.max.ms.__cond_resched.unmap_page_range.unmap_vmas.exit_mmap.__mmput 6.32 ± 45% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.unmap_vmas.exit_mmap.__mmput.exec_mmap 0.52 ±102% +663.3% 3.99 ± 19% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 24.57 ± 29% +275.9% 92.36 ± 15% perf-sched.sch_delay.max.ms.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 22.84 ± 20% +344.8% 101.59 ± 13% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 37.30 ± 18% +368.1% 174.59 ± 21% perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 10.72 ± 60% -67.3% 3.50 ± 5% perf-sched.sch_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 5.70 ± 21% -87.5% 0.71 ±175% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single 8.53 ± 22% -92.4% 0.65 ±223% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.ret_from_fork_asm 33.90 ± 14% +246.7% 117.54 ± 13% perf-sched.sch_delay.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 2.78 ± 29% +139.7% 6.66 ± 20% perf-sched.sch_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 1.44 ± 44% +150.1% 3.59 ± 37% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 3.80 ± 39% -100.0% 0.00 perf-sched.sch_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.__put_anon_vma 7.53 ± 35% -86.5% 1.02 ±223% perf-sched.sch_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_anon_vmas 16.03 ± 29% +324.3% 68.00 ± 22% perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec 2.66 ± 20% +87.4% 4.98 ± 34% perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 1.64 ± 63% +122.2% 3.65 ± 8% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 7.89 ± 29% +443.0% 42.85 ±122% perf-sched.sch_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.89 ± 99% +180.2% 2.50 ± 14% perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.57 ± 5% +365.1% 2.63 ± 12% perf-sched.total_sch_delay.average.ms 6.14 +124.8% 13.80 ± 5% perf-sched.total_wait_and_delay.average.ms 461279 -51.8% 222478 ± 7% perf-sched.total_wait_and_delay.count.ms 5.57 +100.4% 11.16 ± 5% perf-sched.total_wait_time.average.ms 6.94 ± 7% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.down_read.kernfs_fop_readdir.iterate_dir.__x64_sys_getdents64 13.84 ± 8% -83.6% 2.27 ±223% perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write.dup_mmap.dup_mm.constprop 18.61 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.down_write.free_pgtables.exit_mmap.__mmput 8.64 ± 26% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.exit_mmap.__mmput.exit_mm.do_exit 119.50 ± 12% +298.7% 476.42 ± 58% perf-sched.wait_and_delay.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 1.02 ±223% +3125.3% 32.92 ± 10% perf-sched.wait_and_delay.avg.ms.__cond_resched.mmput.exit_mm.do_exit.do_group_exit 16.55 ± 35% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 15.61 ± 28% -81.8% 2.84 ±223% perf-sched.wait_and_delay.avg.ms.__cond_resched.remove_vma.exit_mmap.__mmput.exec_mmap 2.14 ± 4% +296.1% 8.46 ± 13% perf-sched.wait_and_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 1.32 ± 6% +317.9% 5.53 ± 11% perf-sched.wait_and_delay.avg.ms.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 1.39 ± 6% +275.8% 5.22 ± 13% perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 9.15 +88.6% 17.25 ± 12% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 2.16 ± 8% +469.8% 12.33 ± 17% perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 12.81 ± 8% +56.4% 20.04 ± 11% perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait.call_usermodehelper_exec_work.process_one_work 14.22 ± 18% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 4.55 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 9.00 ± 5% +31.9% 11.87 ± 4% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 1.79 ± 5% +276.1% 6.72 ± 15% perf-sched.wait_and_delay.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 6.38 ± 2% +82.8% 11.66 ± 20% perf-sched.wait_and_delay.avg.ms.io_schedule.folio_wait_bit_common.filemap_fault.__do_fault 366.82 ± 7% +15.5% 423.79 ± 2% perf-sched.wait_and_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 49.80 ± 5% +91.9% 95.57 ± 14% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 9.89 ± 11% +182.3% 27.93 ± 39% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.mmap_region 8.40 ± 8% +275.0% 31.49 ± 66% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma 8.56 ± 11% +261.6% 30.96 ± 83% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.vma_prepare 1.29 ± 6% +276.5% 4.85 ± 13% perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec 7.91 ± 2% +98.8% 15.72 ± 10% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 6.92 +276.0% 26.03 ± 10% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1153 ± 4% -98.9% 12.83 ±223% perf-sched.wait_and_delay.count.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.perf_event_mmap_event.perf_event_mmap 679.83 ± 5% -98.5% 10.33 ±223% perf-sched.wait_and_delay.count.__cond_resched.change_pmd_range.change_p4d_range.change_protection_range.mprotect_fixup 695.67 ± 5% -95.8% 29.00 ±223% perf-sched.wait_and_delay.count.__cond_resched.copy_page_range.dup_mmap.dup_mm.constprop 912.00 ± 4% -96.1% 35.83 ±223% perf-sched.wait_and_delay.count.__cond_resched.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap 239.17 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.down_read.kernfs_fop_readdir.iterate_dir.__x64_sys_getdents64 171.67 ± 7% -87.1% 22.17 ±102% perf-sched.wait_and_delay.count.__cond_resched.down_write.anon_vma_clone.anon_vma_fork.dup_mmap 1745 ± 2% -95.1% 84.67 ±223% perf-sched.wait_and_delay.count.__cond_resched.down_write.dup_mmap.dup_mm.constprop 337.33 ± 6% -94.8% 17.67 ±223% perf-sched.wait_and_delay.count.__cond_resched.down_write.dup_userfaultfd.dup_mmap.dup_mm 2326 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.down_write.free_pgtables.exit_mmap.__mmput 248.67 ± 7% -98.0% 5.00 ±223% perf-sched.wait_and_delay.count.__cond_resched.down_write.vma_prepare.__split_vma.mprotect_fixup 367.33 ± 8% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.exit_mmap.__mmput.exit_mm.do_exit 368.50 ± 9% -99.9% 0.50 ±223% perf-sched.wait_and_delay.count.__cond_resched.exit_signals.do_exit.do_group_exit.__x64_sys_exit_group 19.67 ± 19% -69.5% 6.00 ± 63% perf-sched.wait_and_delay.count.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 1297 ± 4% -92.5% 97.33 ±141% perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc.anon_vma_fork.dup_mmap.dup_mm 481.33 ± 4% -96.5% 16.83 ±223% perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc.vm_area_dup.dup_mmap.dup_mm 34.67 ± 19% -98.1% 0.67 ±223% perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_node.dup_task_struct.copy_process.kernel_clone 39.33 ± 14% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 38.17 ± 22% -99.6% 0.17 ±223% perf-sched.wait_and_delay.count.__cond_resched.remove_vma.exit_mmap.__mmput.exec_mmap 7.00 ± 31% -90.5% 0.67 ±111% perf-sched.wait_and_delay.count.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 32372 -100.0% 0.50 ±223% perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 37242 ± 3% -26.2% 27502 ± 10% perf-sched.wait_and_delay.count.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 1235 ± 4% -95.3% 58.00 ±143% perf-sched.wait_and_delay.count.__cond_resched.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput 3090 ± 3% -98.3% 53.67 ±223% perf-sched.wait_and_delay.count.__cond_resched.zap_pmd_range.isra.0.unmap_page_range 21317 -36.8% 13480 ± 9% perf-sched.wait_and_delay.count.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 14310 -42.6% 8210 ± 7% perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 3760 ± 4% -45.0% 2067 ± 7% perf-sched.wait_and_delay.count.do_task_dead.do_exit.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe 81460 -42.7% 46707 ± 7% perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 34640 -42.4% 19964 ± 7% perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 4361 ± 14% -41.8% 2536 ± 12% perf-sched.wait_and_delay.count.do_wait.kernel_wait.call_usermodehelper_exec_work.process_one_work 190.50 ± 6% -100.0% 0.00 perf-sched.wait_and_delay.count.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 6841 ± 2% -100.0% 0.00 perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 18858 -44.7% 10428 ± 10% perf-sched.wait_and_delay.count.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 17352 -55.5% 7724 ± 9% perf-sched.wait_and_delay.count.io_schedule.folio_wait_bit_common.filemap_fault.__do_fault 185.00 ± 6% -15.9% 155.67 ± 2% perf-sched.wait_and_delay.count.pipe_read.vfs_read.ksys_read.do_syscall_64 258.67 ± 3% -45.0% 142.17 ± 12% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 3489 ± 5% -90.8% 320.00 ±100% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.dup_mmap 2106 ± 7% -91.2% 186.00 ± 17% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.mmap_region 8439 ± 5% -82.7% 1461 ± 13% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma 2609 ± 8% -90.4% 251.00 ± 11% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.vma_prepare 4297 ± 14% -41.0% 2537 ± 12% perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec 63957 ± 2% -35.8% 41042 ± 9% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 48844 -72.1% 13616 ± 9% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 248.37 ± 20% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.down_read.kernfs_fop_readdir.iterate_dir.__x64_sys_getdents64 471.01 ± 24% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.down_write.free_pgtables.exit_mmap.__mmput 275.69 ± 31% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.exit_mmap.__mmput.exit_mm.do_exit 133.38 ± 66% -95.8% 5.61 ±223% perf-sched.wait_and_delay.max.ms.__cond_resched.exit_signals.do_exit.do_group_exit.__x64_sys_exit_group 272.22 ±133% -91.8% 22.24 ±223% perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc_node.dup_task_struct.copy_process.kernel_clone 25.72 ±223% +763.3% 222.05 ± 11% perf-sched.wait_and_delay.max.ms.__cond_resched.mmput.exit_mm.do_exit.do_group_exit 200.89 ± 29% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 270.60 ± 24% -99.0% 2.84 ±223% perf-sched.wait_and_delay.max.ms.__cond_resched.remove_vma.exit_mmap.__mmput.exec_mmap 124.17 ± 22% -93.7% 7.87 ±223% perf-sched.wait_and_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 37.98 ± 17% +285.9% 146.59 ± 14% perf-sched.wait_and_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 33.61 ± 18% +358.2% 153.97 ± 15% perf-sched.wait_and_delay.max.ms.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 1009 +30.6% 1319 ± 29% perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 79.71 ± 12% +328.0% 341.16 ± 13% perf-sched.wait_and_delay.max.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 389.95 ± 71% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 428.59 ± 23% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 33.90 ± 14% +255.8% 120.62 ± 10% perf-sched.wait_and_delay.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 473.25 ± 27% +164.0% 1249 ± 21% perf-sched.wait_and_delay.max.ms.io_schedule.folio_wait_bit_common.filemap_fault.__do_fault 460.89 ± 27% +146.1% 1134 ± 35% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma 345.08 ± 16% +156.5% 885.16 ± 52% perf-sched.wait_and_delay.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.vma_prepare 180.36 ± 15% +290.7% 704.70 ± 93% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 22.25 ± 97% -98.2% 0.40 ±154% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages.pte_alloc_one.__pte_alloc.move_page_tables 5.27 ±119% -91.5% 0.45 ±195% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages_bulk.__vmalloc_area_node.__vmalloc_node_range.alloc_thread_stack_node 4.06 ± 96% -85.5% 0.59 ±139% perf-sched.wait_time.avg.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 18.63 ±106% -87.8% 2.28 ±123% perf-sched.wait_time.avg.ms.__cond_resched.__fput.task_work_run.do_exit.do_group_exit 0.17 ±171% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__get_user_pages.get_user_pages_remote.get_arg_page.remove_arg_zero 0.24 ± 64% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.load_elf_binary.search_binary_handler 0.32 ± 20% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.load_elf_phdrs.load_elf_binary 2.62 ± 42% +142.0% 6.34 ± 33% perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.security_task_alloc.copy_process 0.76 ± 89% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.load_elf_binary.search_binary_handler 3.74 ± 9% +20.9% 4.52 ± 9% perf-sched.wait_time.avg.ms.__cond_resched.__put_anon_vma.unlink_anon_vmas.free_pgtables.exit_mmap 0.65 ± 11% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec.__request_module 0.99 ± 24% +1541.5% 16.24 ±169% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.wait_for_completion_state.kernel_clone.__x64_sys_vfork 9.18 ± 6% +32.5% 12.17 ± 30% perf-sched.wait_time.avg.ms.__cond_resched.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap 0.12 ±116% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.copy_strings.isra.0.do_execveat_common 0.64 ± 55% -94.8% 0.03 ±223% perf-sched.wait_time.avg.ms.__cond_resched.down_read.exit_mmap.__mmput.exec_mmap 0.45 ± 71% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_read.get_arg_page.copy_string_kernel.load_script 0.62 ± 76% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_read.get_arg_page.remove_arg_zero.load_script 3.16 ± 61% +299.2% 12.63 ± 64% perf-sched.wait_time.avg.ms.__cond_resched.down_read.kernfs_dop_revalidate.lookup_fast.walk_component 0.80 ±144% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_read_killable.exec_mmap.begin_new_exec.load_elf_binary 2.99 ± 49% -92.3% 0.23 ±222% perf-sched.wait_time.avg.ms.__cond_resched.down_read_killable.iterate_dir.__x64_sys_getdents64.do_syscall_64 0.86 ± 97% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_timeout.__request_module.search_binary_handler.exec_binprm 13.55 ± 8% -25.4% 10.10 ± 16% perf-sched.wait_time.avg.ms.__cond_resched.down_write.dup_mmap.dup_mm.constprop 18.06 ± 4% -63.0% 6.67 ± 28% perf-sched.wait_time.avg.ms.__cond_resched.down_write.free_pgtables.exit_mmap.__mmput 2.34 ± 29% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_write.vfs_unlink.do_unlinkat.__x64_sys_unlink 8.04 ±163% -95.6% 0.35 ±223% perf-sched.wait_time.avg.ms.__cond_resched.down_write.vma_expand.shift_arg_pages.setup_arg_pages 15.39 ±132% -100.0% 0.00 ±223% perf-sched.wait_time.avg.ms.__cond_resched.down_write.vma_prepare.vma_expand.shift_arg_pages 0.30 ± 84% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 6.73 ± 96% -93.3% 0.45 ±223% perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.vm_brk_flags.load_elf_interp.constprop 0.99 ± 85% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.down_write_killable.vm_brk_flags.set_brk.load_elf_binary 1.54 ±108% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 2.77 ± 30% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.exit_signals.do_exit.do_group_exit.get_signal 0.26 ± 98% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.filemap_read.__kernel_read.load_elf_binary.search_binary_handler 0.48 ± 64% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.filemap_read.__kernel_read.load_elf_phdrs.load_elf_binary 0.33 ± 39% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.filemap_read.__kernel_read.search_binary_handler.exec_binprm 119.50 ± 12% +298.7% 476.42 ± 58% perf-sched.wait_time.avg.ms.__cond_resched.generic_perform_write.shmem_file_write_iter.vfs_write.ksys_write 1.20 ± 58% +276.6% 4.53 ± 50% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 1.83 ± 54% +351.4% 8.26 ± 47% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.getname_flags.part.0 0.45 ± 67% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.getname_kernel.open_exec.load_elf_binary 2.36 ± 16% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.taskstats_exit.do_exit.do_group_exit 0.87 ± 29% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.mmput.exec_mmap.begin_new_exec.load_elf_binary 3.97 ± 5% +439.6% 21.40 ± 11% perf-sched.wait_time.avg.ms.__cond_resched.mmput.exit_mm.do_exit.do_group_exit 2.42 ± 22% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.mmput.free_bprm.do_execveat_common.isra 3.31 ± 70% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.mnt_want_write.do_unlinkat.__x64_sys_unlink.do_syscall_64 12.75 ±105% -96.2% 0.49 ±222% perf-sched.wait_time.avg.ms.__cond_resched.move_page_tables.shift_arg_pages.setup_arg_pages.load_elf_binary 1.57 +361.6% 7.23 ± 12% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.futex_exec_release.exec_mm_release.exec_mmap 18.93 ±212% -98.8% 0.23 ±210% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.kernfs_fop_open.do_dentry_open.do_open 9.50 ± 85% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 4.26 ± 37% +224.9% 13.84 ± 62% perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock_killable.pcpu_alloc.__percpu_counter_init_many.mm_init 15.97 ± 35% -65.8% 5.46 ± 64% perf-sched.wait_time.avg.ms.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 15.04 ± 28% -69.0% 4.67 ±122% perf-sched.wait_time.avg.ms.__cond_resched.remove_vma.exit_mmap.__mmput.exec_mmap 0.66 ±137% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.slab_pre_alloc_hook.constprop.0.kmem_cache_alloc_lru 0.05 ± 6% -91.4% 0.00 ± 42% perf-sched.wait_time.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 3.28 ± 44% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill 2.13 ± 52% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.truncate_inode_pages_range.evict.do_unlinkat.__x64_sys_unlink 3.84 ± 69% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.unmap_page_range.unmap_vmas.unmap_region.constprop 3.58 ± 6% +114.1% 7.68 ± 46% perf-sched.wait_time.avg.ms.__cond_resched.zap_pmd_range.isra.0.unmap_page_range 0.79 ± 9% +417.7% 4.10 ± 13% perf-sched.wait_time.avg.ms.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 0.30 +64.6% 0.49 ± 7% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 8.59 +82.4% 15.66 ± 12% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 1.56 ± 8% +438.0% 8.42 ± 17% perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 12.64 ± 8% +46.8% 18.55 ± 12% perf-sched.wait_time.avg.ms.do_wait.kernel_wait.call_usermodehelper_exec_work.process_one_work 13.78 ± 18% -73.4% 3.67 ± 30% perf-sched.wait_time.avg.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 3.87 ± 5% +95.9% 7.59 ± 25% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_exc_page_fault 8.48 ± 5% +28.9% 10.92 ± 4% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.01 ± 62% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.ret_from_fork_asm 0.19 ± 11% +92.2% 0.37 ± 27% perf-sched.wait_time.avg.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 5.90 ± 2% +78.7% 10.55 ± 21% perf-sched.wait_time.avg.ms.io_schedule.folio_wait_bit_common.filemap_fault.__do_fault 366.60 ± 7% +15.3% 422.81 ± 2% perf-sched.wait_time.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 49.79 ± 5% +91.7% 95.44 ± 14% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 9.50 ± 12% +180.7% 26.68 ± 38% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.mmap_region 7.85 ± 8% +278.8% 29.73 ± 69% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma 8.18 ± 12% +269.8% 30.23 ± 85% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.vma_prepare 0.02 ± 23% -89.8% 0.00 ±223% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.stop_one_cpu.sched_exec 0.74 ± 10% +365.1% 3.44 ± 14% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec 1.44 ± 24% +161.3% 3.77 ± 56% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 3.84 ± 2% +48.0% 5.68 ± 3% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 7.84 ± 2% +93.4% 15.17 ± 10% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.06 ± 53% +1136.6% 0.79 ± 27% perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 6.86 +266.4% 25.15 ± 10% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 75.72 ± 57% -77.2% 17.24 ±120% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages.__pmd_alloc.__handle_mm_fault.handle_mm_fault 41.72 ±182% -96.4% 1.52 ±154% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages.allocate_slab.___slab_alloc.kmem_cache_alloc_node 97.63 ±110% -99.6% 0.40 ±154% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages.pte_alloc_one.__pte_alloc.move_page_tables 18.50 ±137% -97.6% 0.45 ±195% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages_bulk.__vmalloc_area_node.__vmalloc_node_range.alloc_thread_stack_node 249.77 ± 36% -93.9% 15.28 ± 29% perf-sched.wait_time.max.ms.__cond_resched.__anon_vma_prepare.do_cow_fault.do_fault.__handle_mm_fault 18.44 ±113% -94.4% 1.03 ±158% perf-sched.wait_time.max.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 84.47 ± 94% -95.7% 3.67 ±159% perf-sched.wait_time.max.ms.__cond_resched.__fput.task_work_run.do_exit.do_group_exit 0.21 ±143% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__get_user_pages.get_user_pages_remote.get_arg_page.remove_arg_zero 2.34 ± 35% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.load_elf_binary.search_binary_handler 4.94 ± 47% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.load_elf_phdrs.load_elf_binary 92.00 ± 62% -78.4% 19.88 ± 36% perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.security_task_alloc.copy_process 3.61 ± 45% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__kmem_cache_alloc_node.kmalloc_trace.load_elf_binary.search_binary_handler 290.60 ± 19% -88.7% 32.76 ± 34% perf-sched.wait_time.max.ms.__cond_resched.__put_anon_vma.unlink_anon_vmas.free_pgtables.exit_mmap 14.96 ± 32% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.wait_for_completion_state.call_usermodehelper_exec.__request_module 1.49 ± 26% +8603.7% 129.38 ±209% perf-sched.wait_time.max.ms.__cond_resched.__wait_for_common.wait_for_completion_state.kernel_clone.__x64_sys_vfork 314.82 ± 13% +76.5% 555.60 ± 30% perf-sched.wait_time.max.ms.__cond_resched.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap 1.99 ± 91% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.copy_strings.isra.0.do_execveat_common 129.47 ± 81% -97.8% 2.81 ±172% perf-sched.wait_time.max.ms.__cond_resched.down_read.exit_mm.do_exit.__x64_sys_exit 25.23 ±106% -90.0% 2.51 ±145% perf-sched.wait_time.max.ms.__cond_resched.down_read.exit_mm.do_exit.do_group_exit 3.33 ± 55% -99.0% 0.03 ±223% perf-sched.wait_time.max.ms.__cond_resched.down_read.exit_mmap.__mmput.exec_mmap 1.26 ± 72% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_read.get_arg_page.copy_string_kernel.load_script 1.98 ± 77% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_read.get_arg_page.remove_arg_zero.load_script 246.51 ± 20% -76.1% 58.86 ± 30% perf-sched.wait_time.max.ms.__cond_resched.down_read.kernfs_fop_readdir.iterate_dir.__x64_sys_getdents64 1.32 ±171% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_read_killable.exec_mmap.begin_new_exec.load_elf_binary 6.24 ± 40% -96.3% 0.23 ±222% perf-sched.wait_time.max.ms.__cond_resched.down_read_killable.iterate_dir.__x64_sys_getdents64.do_syscall_64 3.65 ± 95% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_timeout.__request_module.search_binary_handler.exec_binprm 35.42 ± 87% -65.4% 12.26 ±158% perf-sched.wait_time.max.ms.__cond_resched.down_write.__anon_vma_prepare.do_cow_fault.do_fault 158.45 ± 74% -90.8% 14.51 ± 23% perf-sched.wait_time.max.ms.__cond_resched.down_write.__split_vma.do_vmi_align_munmap.do_vmi_munmap 92.81 ± 48% -87.7% 11.39 ± 47% perf-sched.wait_time.max.ms.__cond_resched.down_write.__split_vma.mprotect_fixup.do_mprotect_pkey 118.04 ± 42% -91.8% 9.64 ± 64% perf-sched.wait_time.max.ms.__cond_resched.down_write.anon_vma_clone.__split_vma.mprotect_fixup 288.08 ± 19% -89.8% 29.50 ± 47% perf-sched.wait_time.max.ms.__cond_resched.down_write.mmap_region.do_mmap.vm_mmap_pgoff 205.15 ± 12% -87.8% 25.07 ± 52% perf-sched.wait_time.max.ms.__cond_resched.down_write.unlink_anon_vmas.free_pgtables.exit_mmap 240.47 ± 41% -82.4% 42.21 ± 40% perf-sched.wait_time.max.ms.__cond_resched.down_write.unlink_file_vma.free_pgtables.exit_mmap 99.37 ± 54% -93.7% 6.21 ±109% perf-sched.wait_time.max.ms.__cond_resched.down_write.unlink_file_vma.free_pgtables.unmap_region 4.78 ± 38% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_write.vfs_unlink.do_unlinkat.__x64_sys_unlink 46.19 ±170% -99.2% 0.35 ±223% perf-sched.wait_time.max.ms.__cond_resched.down_write.vma_expand.shift_arg_pages.setup_arg_pages 39.69 ±139% -100.0% 0.00 ±223% perf-sched.wait_time.max.ms.__cond_resched.down_write.vma_prepare.vma_expand.shift_arg_pages 55.96 ±134% -97.2% 1.56 ±223% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.__vm_munmap.elf_map.load_elf_binary 114.17 ± 94% -93.8% 7.13 ±153% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.create_elf_tables.load_elf_binary.search_binary_handler 1.37 ± 78% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.exec_mmap.begin_new_exec.load_elf_binary 13.56 ±121% -95.1% 0.67 ±131% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.map_vdso.load_elf_binary.search_binary_handler 156.40 ± 50% -87.2% 20.01 ±161% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.setup_arg_pages.load_elf_binary.search_binary_handler 44.77 ±127% -99.0% 0.45 ±223% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.vm_brk_flags.load_elf_interp.constprop 3.88 ± 63% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.vm_brk_flags.set_brk.load_elf_binary 133.01 ± 42% -92.6% 9.90 ±100% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.elf_map.load_elf_binary 196.01 ± 42% -84.3% 30.69 ± 69% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.elf_map.load_elf_interp 157.78 ± 46% -92.8% 11.29 ± 70% perf-sched.wait_time.max.ms.__cond_resched.down_write_killable.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 3.80 ± 78% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 45.13 ±182% -98.3% 0.76 ±141% perf-sched.wait_time.max.ms.__cond_resched.dput.step_into.path_lookupat.filename_lookup 131.32 ± 68% -92.6% 9.74 ±117% perf-sched.wait_time.max.ms.__cond_resched.exit_signals.do_exit.do_group_exit.__x64_sys_exit_group 8.79 ± 41% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.exit_signals.do_exit.do_group_exit.get_signal 1.52 ± 89% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.filemap_read.__kernel_read.load_elf_binary.search_binary_handler 3.76 ± 71% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.filemap_read.__kernel_read.load_elf_phdrs.load_elf_binary 7.02 ± 39% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.filemap_read.__kernel_read.search_binary_handler.exec_binprm 120.56 ± 79% -87.5% 15.12 ±102% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.__anon_vma_prepare.do_cow_fault.do_fault 2.58 ± 78% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.getname_kernel.open_exec.load_elf_binary 208.87 ± 25% -88.2% 24.63 ± 87% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.__split_vma 171.23 ± 33% -91.7% 14.18 ± 32% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.mmap_region 60.37 ±201% -98.8% 0.72 ±223% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.mas_alloc_nodes.mas_preallocate.vma_link 12.35 ± 45% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.taskstats_exit.do_exit.do_group_exit 73.47 ±131% -99.4% 0.45 ±223% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.vm_area_alloc.__install_special_mapping.map_vdso 387.68 ± 32% -66.8% 128.69 ± 86% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.vm_area_alloc.mmap_region.do_mmap 5.03 ± 31% -95.2% 0.24 ±217% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_preallocate.__split_vma 29.39 ±217% -99.4% 0.16 ±208% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_bulk.mas_alloc_nodes.mas_preallocate.mmap_region 272.00 ±133% -90.9% 24.84 ±193% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_node.dup_task_struct.copy_process.kernel_clone 6.30 ± 42% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.mmput.exec_mmap.begin_new_exec.load_elf_binary 7.35 ± 17% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.mmput.free_bprm.do_execveat_common.isra 10.71 ± 79% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.mnt_want_write.do_unlinkat.__x64_sys_unlink.do_syscall_64 68.74 ±104% -98.6% 0.98 ±223% perf-sched.wait_time.max.ms.__cond_resched.move_page_tables.shift_arg_pages.setup_arg_pages.load_elf_binary 25.02 ± 30% +275.5% 93.93 ± 21% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.futex_exec_release.exec_mm_release.exec_mmap 224.32 ± 73% -94.8% 11.57 ± 73% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.futex_exit_release.exit_mm_release.exit_mm 38.07 ±206% -99.4% 0.23 ±210% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.kernfs_fop_open.do_dentry_open.do_open 9.34 ±166% -93.2% 0.63 ±132% perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.perf_event_exit_task.do_exit.do_group_exit 18.43 ± 63% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.perf_poll.do_poll.constprop 259.41 ± 25% -83.0% 44.12 ±158% perf-sched.wait_time.max.ms.__cond_resched.process_one_work.worker_thread.kthread.ret_from_fork 198.89 ± 29% -92.3% 15.31 ± 49% perf-sched.wait_time.max.ms.__cond_resched.put_files_struct.do_exit.do_group_exit.__x64_sys_exit_group 79.54 ± 69% -89.4% 8.40 ±129% perf-sched.wait_time.max.ms.__cond_resched.remove_vma.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap 336.72 ± 27% -89.8% 34.34 ± 34% perf-sched.wait_time.max.ms.__cond_resched.remove_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region 269.35 ± 24% -97.6% 6.35 ± 90% perf-sched.wait_time.max.ms.__cond_resched.remove_vma.exit_mmap.__mmput.exec_mmap 3.69 ±157% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.slab_pre_alloc_hook.constprop.0.kmem_cache_alloc_lru 123.54 ± 22% -91.0% 11.14 ±150% perf-sched.wait_time.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 144.86 ± 54% -82.4% 25.46 ± 42% perf-sched.wait_time.max.ms.__cond_resched.task_work_run.do_exit.do_group_exit.__x64_sys_exit_group 99.26 ± 64% -95.6% 4.35 ±100% perf-sched.wait_time.max.ms.__cond_resched.tlb_batch_pages_flush.tlb_finish_mmu.shift_arg_pages.setup_arg_pages 6.41 ± 37% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.truncate_inode_pages_range.evict.__dentry_kill.dentry_kill 6.38 ± 48% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.truncate_inode_pages_range.evict.do_unlinkat.__x64_sys_unlink 7.06 ± 60% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.unmap_page_range.unmap_vmas.unmap_region.constprop 5.74 ± 47% -88.7% 0.65 ±153% perf-sched.wait_time.max.ms.__cond_resched.unmap_vmas.exit_mmap.__mmput.exec_mmap 346.66 ± 27% -77.3% 78.62 ± 59% perf-sched.wait_time.max.ms.__cond_resched.unmap_vmas.unmap_region.constprop.0 26.95 ± 26% +308.5% 110.09 ± 11% perf-sched.wait_time.max.ms.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 1002 +31.1% 1314 ± 29% perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 45.81 ± 13% +334.5% 199.05 ± 9% perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 388.29 ± 72% -87.1% 50.21 ±124% perf-sched.wait_time.max.ms.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 117.03 ± 28% -94.4% 6.56 ± 99% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_call_function_single 0.95 ± 59% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.ret_from_fork_asm 22.11 ± 14% +360.9% 101.90 ± 21% perf-sched.wait_time.max.ms.futex_wait_queue.futex_wait.do_futex.__x64_sys_futex 471.08 ± 27% +164.0% 1243 ± 21% perf-sched.wait_time.max.ms.io_schedule.folio_wait_bit_common.filemap_fault.__do_fault 37.56 ±123% -97.9% 0.78 ±142% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.__put_anon_vma 371.15 ± 11% +170.6% 1004 ± 42% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.dup_mmap 458.19 ± 27% +145.9% 1126 ± 36% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.unlink_file_vma 341.64 ± 16% +158.5% 883.18 ± 52% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.vma_prepare 0.03 ± 37% -92.9% 0.00 ±223% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.stop_one_cpu.sched_exec 180.35 ± 15% +289.9% 703.25 ± 93% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.14 ± 41% +242.4% 3.90 ± 26% perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 10.30 ± 2% -5.6 4.73 ± 10% perf-profile.calltrace.cycles-pp.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.25 ± 2% -5.5 4.71 ± 10% perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.16 ± 2% -5.5 4.66 ± 10% perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe 10.01 ± 2% -5.4 4.58 ± 10% perf-profile.calltrace.cycles-pp.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff.do_syscall_64 10.78 -5.1 5.69 ± 7% perf-profile.calltrace.cycles-pp.dup_mm.copy_process.kernel_clone.__do_sys_clone.do_syscall_64 10.51 -5.0 5.53 ± 7% perf-profile.calltrace.cycles-pp.dup_mmap.dup_mm.copy_process.kernel_clone.__do_sys_clone 8.54 -4.6 3.96 ± 8% perf-profile.calltrace.cycles-pp.__mmput.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group 8.52 -4.6 3.95 ± 8% perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exit_mm.do_exit.do_group_exit 9.01 -4.3 4.73 ± 6% perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 7.87 -3.9 3.96 ± 8% perf-profile.calltrace.cycles-pp.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm 7.66 -3.8 3.84 ± 8% perf-profile.calltrace.cycles-pp.__mmput.exec_mmap.begin_new_exec.load_elf_binary.search_binary_handler 7.64 -3.8 3.82 ± 8% perf-profile.calltrace.cycles-pp.exit_mmap.__mmput.exec_mmap.begin_new_exec.load_elf_binary 8.16 -3.7 4.49 ± 7% perf-profile.calltrace.cycles-pp.asm_exc_page_fault 7.53 -3.4 4.13 ± 7% perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault 7.49 -3.4 4.10 ± 7% perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 6.16 ± 2% -3.4 2.79 ± 10% perf-profile.calltrace.cycles-pp.do_vmi_munmap.mmap_region.do_mmap.vm_mmap_pgoff.ksys_mmap_pgoff 6.09 ± 2% -3.3 2.75 ± 10% perf-profile.calltrace.cycles-pp.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap.vm_mmap_pgoff 3.60 ± 3% -3.3 0.30 ±100% perf-profile.calltrace.cycles-pp.down_write.unlink_file_vma.free_pgtables.exit_mmap.__mmput 6.73 -3.1 3.67 ± 7% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 6.44 -2.9 3.50 ± 7% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 3.93 ± 2% -2.5 1.48 ± 34% perf-profile.calltrace.cycles-pp.zap_pte_range.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap 4.22 ± 2% -2.4 1.86 ± 35% perf-profile.calltrace.cycles-pp.zap_pmd_range.unmap_page_range.unmap_vmas.exit_mmap.__mmput 3.78 ± 2% -2.4 1.42 ± 11% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.__mmput.exit_mm.do_exit 4.72 -2.2 2.55 ± 7% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 3.90 -2.1 1.76 ± 8% perf-profile.calltrace.cycles-pp.free_pgtables.exit_mmap.__mmput.exec_mmap.begin_new_exec 3.84 ± 2% -2.0 1.81 ± 10% perf-profile.calltrace.cycles-pp.__split_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap 2.76 ± 3% -1.7 1.01 ± 14% perf-profile.calltrace.cycles-pp.down_write.dup_mmap.dup_mm.copy_process.kernel_clone 2.52 ± 3% -1.7 0.83 ± 12% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.exit_mmap.__mmput.exit_mm 1.92 ± 3% -1.6 0.27 ±100% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.vma_prepare.__split_vma 2.50 ± 3% -1.6 0.88 ± 15% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.dup_mmap.dup_mm.copy_process 12.08 -1.6 10.52 ± 2% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common 2.40 ± 3% -1.6 0.84 ± 15% perf-profile.calltrace.cycles-pp.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.dup_mmap.dup_mm 3.31 -1.5 1.78 ± 7% perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 3.00 -1.5 1.47 ± 8% perf-profile.calltrace.cycles-pp.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.00 -1.5 1.48 ± 8% perf-profile.calltrace.cycles-pp.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.17 -1.5 1.70 ± 7% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault 2.77 -1.4 1.35 ± 8% perf-profile.calltrace.cycles-pp.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.69 ± 6% -1.4 0.29 ±100% perf-profile.calltrace.cycles-pp.record__finish_output.__cmd_record 1.69 ± 6% -1.4 0.29 ±100% perf-profile.calltrace.cycles-pp.perf_session__process_events.record__finish_output.__cmd_record 37.56 -1.4 36.21 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 37.53 -1.3 36.19 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.34 ± 2% -1.3 1.01 ± 11% perf-profile.calltrace.cycles-pp.vma_prepare.__split_vma.do_vmi_align_munmap.do_vmi_munmap.mmap_region 1.79 ± 4% -1.3 0.52 ± 46% perf-profile.calltrace.cycles-pp.osq_lock.rwsem_optimistic_spin.rwsem_down_write_slowpath.down_write.dup_mmap 2.36 -1.2 1.12 ± 9% perf-profile.calltrace.cycles-pp.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect.do_syscall_64 1.96 ± 2% -1.2 0.78 ± 12% perf-profile.calltrace.cycles-pp.unmap_region.do_vmi_align_munmap.do_vmi_munmap.mmap_region.do_mmap 3.22 -1.2 2.05 ± 6% perf-profile.calltrace.cycles-pp.anon_vma_fork.dup_mmap.dup_mm.copy_process.kernel_clone 1.46 ± 4% -1.2 0.30 ±100% perf-profile.calltrace.cycles-pp.down_write.vma_prepare.__split_vma.do_vmi_align_munmap.do_vmi_munmap 2.55 -1.1 1.42 ± 7% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exit_mm.do_exit 1.41 ± 3% -1.1 0.30 ±100% perf-profile.calltrace.cycles-pp.unlink_file_vma.free_pgtables.unmap_region.do_vmi_align_munmap.do_vmi_munmap 1.39 ± 3% -1.1 0.28 ±100% perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write.vma_prepare.__split_vma.do_vmi_align_munmap 2.40 -1.1 1.33 ± 7% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exit_mm 1.57 ± 3% -1.1 0.50 ± 46% perf-profile.calltrace.cycles-pp.free_pgtables.unmap_region.do_vmi_align_munmap.do_vmi_munmap.mmap_region 1.93 ± 4% -1.0 0.91 ± 45% perf-profile.calltrace.cycles-pp.unmap_page_range.unmap_vmas.exit_mmap.__mmput.exec_mmap 6.54 -1.0 5.56 ± 2% perf-profile.calltrace.cycles-pp.execve 6.53 -1.0 5.55 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve 6.53 -1.0 5.55 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 6.53 -1.0 5.55 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.69 ± 6% -1.0 0.71 ± 45% perf-profile.calltrace.cycles-pp.__cmd_record 6.52 -1.0 5.54 ± 2% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 6.40 -0.9 5.46 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.syscall 6.40 -0.9 5.46 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall 6.40 -0.9 5.46 ± 2% perf-profile.calltrace.cycles-pp.__x64_sys_execveat.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall 6.41 -0.9 5.47 ± 2% perf-profile.calltrace.cycles-pp.syscall 6.39 -0.9 5.46 ± 2% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execveat.do_syscall_64.entry_SYSCALL_64_after_hwframe.syscall 2.13 ± 4% -0.9 1.21 ± 7% perf-profile.calltrace.cycles-pp.unmap_vmas.exit_mmap.__mmput.exec_mmap.begin_new_exec 6.28 -0.9 5.41 ± 2% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.23 ± 2% -0.9 0.37 ± 71% perf-profile.calltrace.cycles-pp.vma_prepare.__split_vma.mprotect_fixup.do_mprotect_pkey.__x64_sys_mprotect 10.44 -0.8 9.60 perf-profile.calltrace.cycles-pp.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve 2.21 -0.8 1.38 ± 6% perf-profile.calltrace.cycles-pp.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm.copy_process 1.09 -0.8 0.27 ±100% perf-profile.calltrace.cycles-pp.release_pages.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput 6.13 -0.8 5.31 perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64 6.14 -0.8 5.32 ± 2% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execveat.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.13 -0.8 5.31 perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve 6.01 -0.8 5.24 ± 2% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execveat.do_syscall_64 6.01 -0.8 5.24 ± 2% perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execveat 1.59 ± 8% -0.7 0.85 ± 6% perf-profile.calltrace.cycles-pp.copy_page_range.dup_mmap.dup_mm.copy_process.kernel_clone 1.54 ± 8% -0.7 0.82 ± 7% perf-profile.calltrace.cycles-pp.copy_p4d_range.copy_page_range.dup_mmap.dup_mm.copy_process 1.44 -0.7 0.72 ± 7% perf-profile.calltrace.cycles-pp.tlb_finish_mmu.exit_mmap.__mmput.exit_mm.do_exit 1.39 -0.7 0.68 ± 8% perf-profile.calltrace.cycles-pp.tlb_batch_pages_flush.tlb_finish_mmu.exit_mmap.__mmput.exit_mm 1.45 ± 2% -0.7 0.79 ± 9% perf-profile.calltrace.cycles-pp.next_uptodate_folio.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault 1.36 ± 9% -0.6 0.71 ± 7% perf-profile.calltrace.cycles-pp.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap.dup_mm 1.39 -0.6 0.75 ± 7% perf-profile.calltrace.cycles-pp.do_cow_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault 1.08 -0.6 0.45 ± 45% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 17.18 -0.5 16.64 perf-profile.calltrace.cycles-pp.__libc_fork 0.98 -0.5 0.46 ± 45% perf-profile.calltrace.cycles-pp.wp_page_copy.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault 1.06 ± 2% -0.4 0.65 ± 6% perf-profile.calltrace.cycles-pp.anon_vma_interval_tree_insert.anon_vma_clone.anon_vma_fork.dup_mmap.dup_mm 0.46 ± 44% +0.5 0.97 ± 6% perf-profile.calltrace.cycles-pp.do_exit.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.46 ± 44% +0.5 0.97 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.44 ± 44% +0.5 0.97 ± 6% perf-profile.calltrace.cycles-pp.exit_notify.do_exit.__x64_sys_exit.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.00 +0.6 0.57 ± 6% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.exit_notify.do_exit.__x64_sys_exit.do_syscall_64 1.11 +0.7 1.84 ± 2% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.ret_from_fork_asm 1.11 +0.7 1.84 ± 2% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.ret_from_fork_asm 1.11 +0.7 1.84 ± 2% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.ret_from_fork_asm 1.11 +0.7 1.84 ± 2% perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.ret_from_fork_asm 0.00 +0.8 0.76 ± 4% perf-profile.calltrace.cycles-pp.mm_update_next_owner.exit_mm.do_exit.do_group_exit.__x64_sys_exit_group 0.26 ±100% +0.8 1.05 ± 12% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.do_wait.kernel_wait.call_usermodehelper_exec_work.process_one_work 0.17 ±141% +0.9 1.04 ± 12% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_read_lock_slowpath.do_wait.kernel_wait.call_usermodehelper_exec_work 0.69 ± 18% +0.9 1.57 ± 12% perf-profile.calltrace.cycles-pp.do_wait.kernel_wait.call_usermodehelper_exec_work.process_one_work.worker_thread 0.69 ± 18% +0.9 1.57 ± 12% perf-profile.calltrace.cycles-pp.kernel_wait.call_usermodehelper_exec_work.process_one_work.worker_thread.kthread 1.24 ± 15% +1.1 2.36 ± 12% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1.22 ± 15% +1.1 2.34 ± 12% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1.07 ± 17% +1.2 2.26 ± 13% perf-profile.calltrace.cycles-pp.call_usermodehelper_exec_work.process_one_work.worker_thread.kthread.ret_from_fork 3.58 ± 5% +1.2 4.82 ± 6% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 2.69 +1.3 4.02 perf-profile.calltrace.cycles-pp.exit_mm.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 2.37 +1.4 3.80 ± 2% perf-profile.calltrace.cycles-pp.mm_update_next_owner.exit_mm.do_exit.do_group_exit.get_signal 0.87 ± 2% +1.5 2.34 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.release_task.exit_notify.do_exit 0.91 ± 2% +1.5 2.41 ± 3% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.release_task.exit_notify.do_exit.do_group_exit 1.30 ± 2% +1.5 2.80 ± 10% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.mm_update_next_owner.exit_mm.do_exit.do_group_exit 0.94 ± 2% +1.5 2.44 ± 3% perf-profile.calltrace.cycles-pp.release_task.exit_notify.do_exit.do_group_exit.get_signal 1.29 +1.5 2.80 ± 10% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_read_lock_slowpath.mm_update_next_owner.exit_mm.do_exit 15.50 +1.7 17.24 perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 15.50 +1.7 17.24 perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 15.50 +1.7 17.24 perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.34 ± 2% +1.9 3.21 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.de_thread.begin_new_exec.load_elf_binary 1.45 ± 2% +1.9 3.38 ± 3% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 0.00 +2.0 2.03 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.release_task.de_thread.begin_new_exec 0.00 +2.1 2.10 ± 3% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.release_task.de_thread.begin_new_exec.load_elf_binary 0.00 +2.1 2.12 ± 3% perf-profile.calltrace.cycles-pp.release_task.de_thread.begin_new_exec.load_elf_binary.search_binary_handler 19.38 +2.4 21.79 perf-profile.calltrace.cycles-pp.copy_process.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.21 +2.5 5.72 ± 3% perf-profile.calltrace.cycles-pp.__clone 3.13 +2.5 5.67 ± 3% perf-profile.calltrace.cycles-pp.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__clone 3.13 +2.5 5.68 ± 3% perf-profile.calltrace.cycles-pp.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe.__clone 3.13 +2.5 5.68 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__clone 3.13 +2.5 5.68 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__clone 2.58 +3.0 5.56 ± 2% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.exit_notify.do_exit.do_group_exit.get_signal 2.47 +3.1 5.56 ± 3% perf-profile.calltrace.cycles-pp.de_thread.begin_new_exec.load_elf_binary.search_binary_handler.exec_binprm 4.29 +3.8 8.04 ± 2% perf-profile.calltrace.cycles-pp.exit_notify.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 3.37 ± 2% +4.1 7.45 ± 4% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 3.35 ± 2% +4.1 7.44 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_read_lock_slowpath.do_wait.kernel_wait4.__do_sys_wait4 3.44 ± 2% +4.2 7.67 ± 5% perf-profile.calltrace.cycles-pp.wait_task_zombie.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64 3.42 ± 2% +4.2 7.65 ± 5% perf-profile.calltrace.cycles-pp.release_task.wait_task_zombie.do_wait.kernel_wait4.__do_sys_wait4 3.30 ± 2% +4.3 7.55 ± 5% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.release_task.wait_task_zombie.do_wait.kernel_wait4 6.39 +4.4 10.77 ± 2% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.36 +4.4 10.75 ± 2% perf-profile.calltrace.cycles-pp.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.35 +4.4 10.75 ± 2% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.32 +4.4 10.73 ± 2% perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 3.08 ± 2% +4.6 7.64 ± 8% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.release_task.wait_task_zombie.do_wait 7.43 +5.1 12.57 ± 2% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop 7.43 +5.1 12.57 ± 2% perf-profile.calltrace.cycles-pp.do_group_exit.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare 7.43 +5.1 12.57 ± 2% perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 5.93 ± 2% +6.2 12.16 ± 4% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.exit_notify.do_exit.do_group_exit.__x64_sys_exit_group 6.01 ± 2% +6.2 12.24 ± 4% perf-profile.calltrace.cycles-pp.exit_notify.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 7.21 +7.8 15.04 ± 4% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.copy_process.kernel_clone.__do_sys_clone 7.65 +7.8 15.50 ± 4% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.copy_process.kernel_clone.__do_sys_clone.do_syscall_64 6.85 ± 2% +8.3 15.15 ± 4% perf-profile.calltrace.cycles-pp.wait4 6.84 ± 2% +8.3 15.13 ± 4% perf-profile.calltrace.cycles-pp.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait4 6.84 ± 2% +8.3 15.13 ± 4% perf-profile.calltrace.cycles-pp.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait4 6.83 ± 2% +8.3 15.13 ± 4% perf-profile.calltrace.cycles-pp.do_wait.kernel_wait4.__do_sys_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe 6.84 ± 2% +8.3 15.14 ± 4% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.wait4 6.84 ± 2% +8.3 15.14 ± 4% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.wait4 8.05 ± 2% +9.1 17.19 ± 3% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.exit_notify.do_exit.do_group_exit 16.62 -8.6 8.06 ± 8% perf-profile.children.cycles-pp.__mmput 16.58 -8.5 8.04 ± 8% perf-profile.children.cycles-pp.exit_mmap 13.55 ± 3% -8.3 5.21 ± 11% perf-profile.children.cycles-pp.down_write 12.02 ± 3% -7.6 4.42 ± 12% perf-profile.children.cycles-pp.rwsem_down_write_slowpath 11.58 ± 3% -7.4 4.23 ± 12% perf-profile.children.cycles-pp.rwsem_optimistic_spin 11.65 -6.2 5.43 ± 9% perf-profile.children.cycles-pp.vm_mmap_pgoff 11.51 -6.2 5.36 ± 9% perf-profile.children.cycles-pp.do_mmap 11.29 -6.1 5.24 ± 9% perf-profile.children.cycles-pp.mmap_region 13.35 -5.8 7.59 ± 7% perf-profile.children.cycles-pp.asm_exc_page_fault 9.68 ± 2% -5.7 3.95 ± 10% perf-profile.children.cycles-pp.free_pgtables 10.32 ± 2% -5.6 4.75 ± 10% perf-profile.children.cycles-pp.ksys_mmap_pgoff 7.92 ± 4% -5.3 2.59 ± 14% perf-profile.children.cycles-pp.osq_lock 12.13 -5.3 6.85 ± 7% perf-profile.children.cycles-pp.exc_page_fault 12.07 -5.3 6.82 ± 7% perf-profile.children.cycles-pp.do_user_addr_fault 10.78 -5.1 5.69 ± 7% perf-profile.children.cycles-pp.dup_mm 10.53 -5.0 5.55 ± 7% perf-profile.children.cycles-pp.dup_mmap 11.04 -4.8 6.21 ± 7% perf-profile.children.cycles-pp.handle_mm_fault 10.61 -4.7 5.96 ± 7% perf-profile.children.cycles-pp.__handle_mm_fault 6.16 ± 3% -4.0 2.16 ± 12% perf-profile.children.cycles-pp.unlink_file_vma 7.88 -3.9 3.98 ± 8% perf-profile.children.cycles-pp.exec_mmap 6.96 -3.8 3.18 ± 10% perf-profile.children.cycles-pp.do_vmi_munmap 6.87 ± 2% -3.7 3.13 ± 10% perf-profile.children.cycles-pp.do_vmi_align_munmap 7.74 -3.6 4.18 ± 7% perf-profile.children.cycles-pp.do_fault 6.42 -3.4 3.04 ± 9% perf-profile.children.cycles-pp.__split_vma 6.25 ± 2% -2.9 3.37 ± 7% perf-profile.children.cycles-pp.do_read_fault 12.10 -2.9 9.22 ± 2% perf-profile.children.cycles-pp.exit_mm 5.98 ± 2% -2.8 3.22 ± 7% perf-profile.children.cycles-pp.filemap_map_pages 3.93 ± 2% -2.2 1.69 ± 10% perf-profile.children.cycles-pp.vma_prepare 5.02 -2.2 2.83 ± 7% perf-profile.children.cycles-pp.unmap_vmas 4.63 ± 2% -2.0 2.59 ± 7% perf-profile.children.cycles-pp.unmap_page_range 4.49 ± 2% -2.0 2.50 ± 7% perf-profile.children.cycles-pp.zap_pmd_range 4.38 ± 2% -1.9 2.44 ± 7% perf-profile.children.cycles-pp.zap_pte_range 12.95 -1.9 11.04 ± 2% perf-profile.children.cycles-pp.do_execveat_common 12.58 -1.7 10.84 ± 2% perf-profile.children.cycles-pp.bprm_execve 3.03 ± 2% -1.7 1.38 ± 10% perf-profile.children.cycles-pp.rwsem_spin_on_owner 12.26 -1.6 10.64 ± 2% perf-profile.children.cycles-pp.exec_binprm 12.26 -1.6 10.64 ± 2% perf-profile.children.cycles-pp.search_binary_handler 2.71 ± 2% -1.6 1.12 ± 9% perf-profile.children.cycles-pp.up_write 12.19 -1.6 10.60 ± 2% perf-profile.children.cycles-pp.load_elf_binary 3.06 -1.6 1.51 ± 8% perf-profile.children.cycles-pp.do_mprotect_pkey 3.06 -1.5 1.51 ± 8% perf-profile.children.cycles-pp.__x64_sys_mprotect 2.83 -1.4 1.38 ± 8% perf-profile.children.cycles-pp.mprotect_fixup 2.44 ± 2% -1.4 1.00 ± 11% perf-profile.children.cycles-pp.unmap_region 1.68 ± 6% -1.3 0.42 ± 41% perf-profile.children.cycles-pp.reader__read_event 1.69 ± 6% -1.2 0.46 ± 37% perf-profile.children.cycles-pp.record__finish_output 1.69 ± 6% -1.2 0.46 ± 37% perf-profile.children.cycles-pp.perf_session__process_events 2.53 ± 2% -1.2 1.30 ± 6% perf-profile.children.cycles-pp.tlb_finish_mmu 2.60 -1.2 1.40 ± 8% perf-profile.children.cycles-pp.next_uptodate_folio 3.23 -1.2 2.06 ± 6% perf-profile.children.cycles-pp.anon_vma_fork 2.33 ± 2% -1.2 1.16 ± 7% perf-profile.children.cycles-pp.tlb_batch_pages_flush 2.43 -1.1 1.38 ± 7% perf-profile.children.cycles-pp.kmem_cache_alloc 2.23 -1.0 1.24 ± 8% perf-profile.children.cycles-pp.vma_interval_tree_insert 6.54 -1.0 5.56 ± 2% perf-profile.children.cycles-pp.execve 6.55 -1.0 5.57 ± 2% perf-profile.children.cycles-pp.__x64_sys_execve 1.26 ± 9% -1.0 0.30 ± 39% perf-profile.children.cycles-pp.process_simple 1.24 ± 8% -1.0 0.29 ± 39% perf-profile.children.cycles-pp.ordered_events__queue 1.24 ± 9% -1.0 0.29 ± 39% perf-profile.children.cycles-pp.queue_event 1.75 ± 6% -1.0 0.80 ± 26% perf-profile.children.cycles-pp.__cmd_record 1.83 ± 2% -0.9 0.88 ± 7% perf-profile.children.cycles-pp.release_pages 6.41 -0.9 5.47 ± 2% perf-profile.children.cycles-pp.syscall 6.42 -0.9 5.48 ± 2% perf-profile.children.cycles-pp.__x64_sys_execveat 1.85 -0.9 0.94 ± 9% perf-profile.children.cycles-pp.vma_complete 2.29 -0.9 1.43 ± 6% perf-profile.children.cycles-pp.anon_vma_clone 1.93 -0.8 1.10 ± 6% perf-profile.children.cycles-pp.unlink_anon_vmas 10.47 -0.8 9.64 perf-profile.children.cycles-pp.begin_new_exec 1.02 ± 4% -0.8 0.26 ± 13% perf-profile.children.cycles-pp.rwsem_wake 1.60 ± 7% -0.7 0.85 ± 6% perf-profile.children.cycles-pp.copy_page_range 1.56 ± 3% -0.7 0.82 ± 7% perf-profile.children.cycles-pp.page_remove_rmap 1.09 ± 3% -0.7 0.35 ± 10% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 1.54 ± 8% -0.7 0.82 ± 7% perf-profile.children.cycles-pp.copy_p4d_range 1.46 -0.7 0.79 ± 7% perf-profile.children.cycles-pp.do_cow_fault 1.37 ± 9% -0.7 0.72 ± 7% perf-profile.children.cycles-pp.copy_pte_range 1.39 -0.6 0.79 ± 6% perf-profile.children.cycles-pp.__do_softirq 1.34 -0.6 0.76 ± 6% perf-profile.children.cycles-pp.rcu_core 1.33 -0.6 0.76 ± 9% perf-profile.children.cycles-pp._compound_head 1.31 -0.6 0.74 ± 7% perf-profile.children.cycles-pp.rcu_do_batch 1.31 -0.6 0.74 ± 7% perf-profile.children.cycles-pp.vm_area_dup 17.21 -0.6 16.66 perf-profile.children.cycles-pp.__libc_fork 1.08 -0.5 0.53 ± 6% perf-profile.children.cycles-pp.smpboot_thread_fn 1.57 -0.5 1.04 ± 5% perf-profile.children.cycles-pp.wp_page_copy 1.26 -0.5 0.73 ± 7% perf-profile.children.cycles-pp.kmem_cache_free 1.20 -0.5 0.68 ± 5% perf-profile.children.cycles-pp.__alloc_pages 1.18 -0.5 0.66 ± 6% perf-profile.children.cycles-pp.__x64_sys_openat 1.17 -0.5 0.66 ± 6% perf-profile.children.cycles-pp.do_sys_openat2 1.15 -0.5 0.64 ± 6% perf-profile.children.cycles-pp.set_pte_range 0.98 -0.5 0.48 ± 6% perf-profile.children.cycles-pp.run_ksoftirqd 1.10 ± 2% -0.5 0.59 ± 8% perf-profile.children.cycles-pp.elf_map 1.13 -0.5 0.62 ± 7% perf-profile.children.cycles-pp.__slab_free 1.02 ± 2% -0.5 0.56 ± 7% perf-profile.children.cycles-pp.vma_interval_tree_remove 1.06 -0.5 0.59 ± 6% perf-profile.children.cycles-pp.perf_event_mmap 1.07 -0.5 0.62 ± 8% perf-profile.children.cycles-pp.mas_store_prealloc 1.02 -0.4 0.57 ± 6% perf-profile.children.cycles-pp.perf_event_mmap_event 0.93 ± 13% -0.4 0.48 ± 6% perf-profile.children.cycles-pp.copy_present_pte 1.02 -0.4 0.58 ± 6% perf-profile.children.cycles-pp.do_filp_open 1.02 ± 2% -0.4 0.58 ± 7% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook 1.01 -0.4 0.57 ± 6% perf-profile.children.cycles-pp.path_openat 1.08 ± 2% -0.4 0.66 ± 6% perf-profile.children.cycles-pp.anon_vma_interval_tree_insert 0.78 ± 2% -0.4 0.41 ± 6% perf-profile.children.cycles-pp._raw_spin_lock 0.86 -0.4 0.49 ± 7% perf-profile.children.cycles-pp.mas_wr_store_entry 0.83 -0.4 0.46 ± 5% perf-profile.children.cycles-pp.get_page_from_freelist 0.99 -0.3 0.68 ± 5% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.70 -0.3 0.40 ± 6% perf-profile.children.cycles-pp.mas_wr_bnode 0.66 ± 2% -0.3 0.36 ± 5% perf-profile.children.cycles-pp.perf_iterate_sb 0.95 -0.3 0.66 ± 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.44 ± 2% -0.3 0.14 ± 12% perf-profile.children.cycles-pp.folio_lruvec_lock_irqsave 0.42 ± 3% -0.3 0.13 ± 11% perf-profile.children.cycles-pp.try_to_wake_up 0.38 ± 4% -0.3 0.10 ± 14% perf-profile.children.cycles-pp.rwsem_mark_wake 0.62 -0.3 0.34 ± 6% perf-profile.children.cycles-pp.mod_objcg_state 0.52 ± 3% -0.3 0.24 ± 11% perf-profile.children.cycles-pp.vma_expand 0.39 ± 9% -0.3 0.12 ± 52% perf-profile.children.cycles-pp.perf_session__process_user_event 0.62 -0.3 0.35 ± 6% perf-profile.children.cycles-pp.mas_split 0.39 ± 9% -0.3 0.12 ± 51% perf-profile.children.cycles-pp.__ordered_events__flush 0.50 -0.3 0.23 ± 9% perf-profile.children.cycles-pp.folio_batch_move_lru 0.35 ± 4% -0.3 0.09 ± 14% perf-profile.children.cycles-pp.wake_up_q 0.66 -0.3 0.40 ± 6% perf-profile.children.cycles-pp.sync_regs 0.42 ± 6% -0.3 0.17 ± 10% perf-profile.children.cycles-pp.osq_unlock 0.54 -0.3 0.28 ± 7% perf-profile.children.cycles-pp.__schedule 0.56 -0.3 0.30 ± 5% perf-profile.children.cycles-pp.do_anonymous_page 0.60 -0.3 0.35 ± 7% perf-profile.children.cycles-pp.lock_vma_under_rcu 0.54 ± 4% -0.2 0.29 ± 7% perf-profile.children.cycles-pp.free_pages_and_swap_cache 0.54 -0.2 0.29 ± 7% perf-profile.children.cycles-pp.load_elf_interp 0.54 -0.2 0.30 ± 5% perf-profile.children.cycles-pp.vma_alloc_folio 0.33 ± 3% -0.2 0.08 ± 14% perf-profile.children.cycles-pp.wake_q_add 0.61 -0.2 0.36 ± 7% perf-profile.children.cycles-pp._dl_addr 0.35 ± 9% -0.2 0.11 ± 51% perf-profile.children.cycles-pp.perf_session__deliver_event 0.50 ± 2% -0.2 0.27 ± 9% perf-profile.children.cycles-pp.dup_task_struct 0.51 ± 4% -0.2 0.28 ± 6% perf-profile.children.cycles-pp.free_swap_cache 0.50 ± 2% -0.2 0.27 ± 8% perf-profile.children.cycles-pp.__vm_munmap 0.48 ± 2% -0.2 0.26 ± 7% perf-profile.children.cycles-pp.getdents64 0.48 ± 2% -0.2 0.26 ± 5% perf-profile.children.cycles-pp.open64 0.52 ± 2% -0.2 0.30 ± 7% perf-profile.children.cycles-pp.mas_store 0.49 -0.2 0.27 ± 8% perf-profile.children.cycles-pp.__vm_area_free 0.48 ± 2% -0.2 0.26 ± 6% perf-profile.children.cycles-pp.__x64_sys_getdents64 0.47 ± 2% -0.2 0.26 ± 6% perf-profile.children.cycles-pp.iterate_dir 0.56 -0.2 0.35 ± 7% perf-profile.children.cycles-pp.native_irq_return_iret 0.49 ± 2% -0.2 0.27 ± 8% perf-profile.children.cycles-pp.__rb_insert_augmented 0.47 -0.2 0.26 ± 5% perf-profile.children.cycles-pp.__folio_alloc 0.48 ± 2% -0.2 0.27 ± 6% perf-profile.children.cycles-pp.perf_event_mmap_output 0.46 ± 3% -0.2 0.25 ± 5% perf-profile.children.cycles-pp.kernfs_fop_readdir 0.46 ± 2% -0.2 0.25 ± 9% perf-profile.children.cycles-pp.pte_alloc_one 0.47 ± 2% -0.2 0.26 ± 8% perf-profile.children.cycles-pp.__pte_offset_map_lock 0.47 ± 2% -0.2 0.26 ± 8% perf-profile.children.cycles-pp.mm_init 0.49 -0.2 0.29 ± 7% perf-profile.children.cycles-pp.__cond_resched 0.47 -0.2 0.26 ± 4% perf-profile.children.cycles-pp.clear_page_erms 0.36 ± 2% -0.2 0.16 ± 11% perf-profile.children.cycles-pp.lru_add_drain 0.44 -0.2 0.24 ± 7% perf-profile.children.cycles-pp.__rb_erase_color 0.36 ± 2% -0.2 0.16 ± 11% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.48 -0.2 0.28 ± 9% perf-profile.children.cycles-pp.mas_wr_node_store 0.43 ± 3% -0.2 0.23 ± 7% perf-profile.children.cycles-pp.folio_add_file_rmap_range 0.46 ± 2% -0.2 0.27 ± 6% perf-profile.children.cycles-pp.__wp_page_copy_user 0.46 ± 2% -0.2 0.26 ± 6% perf-profile.children.cycles-pp.copy_mc_enhanced_fast_string 0.42 -0.2 0.23 ± 7% perf-profile.children.cycles-pp.copy_page 0.41 ± 2% -0.2 0.23 ± 7% perf-profile.children.cycles-pp.__mmdrop 0.40 -0.2 0.22 ± 6% perf-profile.children.cycles-pp.__mem_cgroup_charge 0.40 ± 3% -0.2 0.22 ± 6% perf-profile.children.cycles-pp.vm_area_alloc 0.38 ± 3% -0.2 0.20 ± 10% perf-profile.children.cycles-pp.alloc_thread_stack_node 0.43 -0.2 0.25 ± 7% perf-profile.children.cycles-pp.mas_preallocate 0.40 ± 3% -0.2 0.22 ± 7% perf-profile.children.cycles-pp.___slab_alloc 0.42 -0.2 0.25 ± 8% perf-profile.children.cycles-pp.mas_walk 0.39 -0.2 0.22 ± 8% perf-profile.children.cycles-pp.__list_del_entry_valid_or_report 0.42 -0.2 0.25 ± 8% perf-profile.children.cycles-pp.mtree_range_walk 0.36 -0.2 0.19 ± 7% perf-profile.children.cycles-pp.finish_fault 0.39 -0.2 0.23 ± 8% perf-profile.children.cycles-pp.__anon_vma_interval_tree_remove 0.38 -0.2 0.22 ± 6% perf-profile.children.cycles-pp.mas_next_slot 0.40 ± 2% -0.2 0.24 ± 7% perf-profile.children.cycles-pp.__perf_sw_event 0.54 -0.2 0.38 ± 3% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 0.37 -0.2 0.21 ± 6% perf-profile.children.cycles-pp.select_task_rq_fair 0.34 ± 2% -0.2 0.19 ± 7% perf-profile.children.cycles-pp.cgroup_rstat_updated 0.53 -0.2 0.38 ± 3% perf-profile.children.cycles-pp.hrtimer_interrupt 0.36 ± 3% -0.2 0.21 ± 7% perf-profile.children.cycles-pp.__sysconf 0.34 -0.2 0.20 ± 7% perf-profile.children.cycles-pp.rmqueue 0.31 ± 5% -0.2 0.16 ± 10% perf-profile.children.cycles-pp.__vmalloc_node_range 0.36 ± 2% -0.1 0.21 ± 8% perf-profile.children.cycles-pp.percpu_counter_add_batch 0.32 -0.1 0.18 ± 6% perf-profile.children.cycles-pp.charge_memcg 0.31 ± 3% -0.1 0.17 ± 12% perf-profile.children.cycles-pp.__pte_alloc 0.34 ± 2% -0.1 0.20 ± 6% perf-profile.children.cycles-pp.find_idlest_cpu 0.33 ± 2% -0.1 0.19 ± 7% perf-profile.children.cycles-pp.__do_fault 0.48 -0.1 0.35 ± 3% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.30 -0.1 0.16 ± 8% perf-profile.children.cycles-pp.schedule 0.33 ± 2% -0.1 0.19 ± 8% perf-profile.children.cycles-pp.___perf_sw_event 0.40 ± 2% -0.1 0.27 ± 8% perf-profile.children.cycles-pp.__irq_exit_rcu 0.30 ± 2% -0.1 0.16 ± 8% perf-profile.children.cycles-pp.pcpu_alloc 0.30 ± 3% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.__mod_lruvec_page_state 0.31 ± 2% -0.1 0.18 ± 8% perf-profile.children.cycles-pp.stress_exec 0.31 ± 2% -0.1 0.18 ± 6% perf-profile.children.cycles-pp.find_idlest_group 0.29 ± 2% -0.1 0.16 ± 7% perf-profile.children.cycles-pp.do_open 0.30 -0.1 0.18 ± 7% perf-profile.children.cycles-pp.mas_alloc_nodes 0.30 -0.1 0.18 ± 4% perf-profile.children.cycles-pp.wake_up_new_task 0.27 ± 3% -0.1 0.15 ± 9% perf-profile.children.cycles-pp.alloc_bprm 0.29 ± 2% -0.1 0.17 ± 7% perf-profile.children.cycles-pp.update_sg_wakeup_stats 0.26 ± 3% -0.1 0.14 ± 6% perf-profile.children.cycles-pp.__count_memcg_events 0.27 ± 3% -0.1 0.16 ± 8% perf-profile.children.cycles-pp.__memcg_kmem_charge_page 0.26 ± 5% -0.1 0.15 ± 8% perf-profile.children.cycles-pp.stress_mwc1 0.23 ± 7% -0.1 0.12 ± 12% perf-profile.children.cycles-pp.__get_vm_area_node 0.41 -0.1 0.29 ± 3% perf-profile.children.cycles-pp.tick_sched_timer 0.27 ± 3% -0.1 0.16 ± 8% perf-profile.children.cycles-pp.rcu_cblist_dequeue 0.26 ± 2% -0.1 0.15 ± 9% perf-profile.children.cycles-pp.__put_anon_vma 0.26 ± 2% -0.1 0.14 ± 8% perf-profile.children.cycles-pp.get_obj_cgroup_from_current 0.22 ± 7% -0.1 0.11 ± 13% perf-profile.children.cycles-pp.alloc_vmap_area 0.28 ± 2% -0.1 0.17 ± 10% perf-profile.children.cycles-pp.unmap_single_vma 0.20 ± 2% -0.1 0.09 ± 8% perf-profile.children.cycles-pp.pick_next_task_fair 0.23 -0.1 0.12 ± 5% perf-profile.children.cycles-pp.link_path_walk 0.24 ± 2% -0.1 0.14 ± 7% perf-profile.children.cycles-pp.finish_task_switch 0.24 ± 3% -0.1 0.13 ± 7% perf-profile.children.cycles-pp.remove_vma 0.39 -0.1 0.28 ± 3% perf-profile.children.cycles-pp.tick_sched_handle 0.39 -0.1 0.28 ± 3% perf-profile.children.cycles-pp.update_process_times 0.24 ± 2% -0.1 0.14 ± 7% perf-profile.children.cycles-pp.__anon_vma_prepare 0.24 -0.1 0.14 ± 9% perf-profile.children.cycles-pp.alloc_empty_file 0.13 ± 8% -0.1 0.03 ±100% perf-profile.children.cycles-pp.machine__process_fork_event 0.25 -0.1 0.14 ± 9% perf-profile.children.cycles-pp.vm_area_free_rcu_cb 0.23 ± 2% -0.1 0.12 ± 11% perf-profile.children.cycles-pp.vma_interval_tree_augment_rotate 0.24 ± 2% -0.1 0.14 ± 9% perf-profile.children.cycles-pp.obj_cgroup_charge 0.23 ± 2% -0.1 0.13 ± 6% perf-profile.children.cycles-pp.mas_store_gfp 0.23 ± 2% -0.1 0.13 ± 7% perf-profile.children.cycles-pp.mas_push_data 0.22 ± 3% -0.1 0.13 ± 8% perf-profile.children.cycles-pp.__cpu_indicator_init 0.22 ± 2% -0.1 0.12 ± 8% perf-profile.children.cycles-pp.__call_rcu_common 0.20 ± 5% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.prctl 0.21 ± 4% -0.1 0.12 ± 8% perf-profile.children.cycles-pp.filemap_fault 0.20 ± 2% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.schedule_tail 0.21 ± 2% -0.1 0.12 ± 7% perf-profile.children.cycles-pp.__open64_nocancel 0.22 ± 2% -0.1 0.13 ± 8% perf-profile.children.cycles-pp.mas_update_gap 0.23 -0.1 0.14 ± 6% perf-profile.children.cycles-pp.open_last_lookups 0.21 ± 10% -0.1 0.12 ± 9% perf-profile.children.cycles-pp.vm_normal_page 0.22 ± 2% -0.1 0.13 ± 7% perf-profile.children.cycles-pp.vfs_read 0.21 ± 4% -0.1 0.12 ± 7% perf-profile.children.cycles-pp.getname_flags 0.33 -0.1 0.24 ± 3% perf-profile.children.cycles-pp.scheduler_tick 0.23 ± 2% -0.1 0.14 ± 8% perf-profile.children.cycles-pp.ksys_read 0.21 ± 4% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.__x64_sys_munmap 0.12 ± 4% -0.1 0.02 ± 99% perf-profile.children.cycles-pp.put_cred_rcu 0.20 ± 2% -0.1 0.12 ± 10% perf-profile.children.cycles-pp.__memcpy 0.20 ± 2% -0.1 0.11 ± 10% perf-profile.children.cycles-pp.task_work_run 0.19 ± 6% -0.1 0.10 ± 10% perf-profile.children.cycles-pp.xas_load 0.18 ± 2% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.folio_add_lru_vma 0.22 ± 3% -0.1 0.13 ± 5% perf-profile.children.cycles-pp.flush_tlb_func 0.20 -0.1 0.12 ± 8% perf-profile.children.cycles-pp.mas_find 0.20 -0.1 0.11 ± 9% perf-profile.children.cycles-pp.lru_add_fn 0.20 -0.1 0.12 ± 6% perf-profile.children.cycles-pp.d_path 0.20 ± 3% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.do_dentry_open 0.17 ± 7% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.xas_find 0.19 ± 3% -0.1 0.11 ± 9% perf-profile.children.cycles-pp.__percpu_counter_sum 0.14 ± 4% -0.1 0.06 ± 13% perf-profile.children.cycles-pp.newidle_balance 0.18 ± 3% -0.1 0.10 ± 6% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state 0.19 ± 2% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.mas_wr_walk 0.18 ± 2% -0.1 0.09 ± 5% perf-profile.children.cycles-pp.fput 0.20 ± 3% -0.1 0.12 ± 6% perf-profile.children.cycles-pp.__fput 0.19 ± 6% -0.1 0.11 ± 9% perf-profile.children.cycles-pp.strncpy_from_user 0.16 ± 6% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.do_task_dead 0.20 ± 2% -0.1 0.12 ± 5% perf-profile.children.cycles-pp.vma_interval_tree_insert_after 0.18 ± 4% -0.1 0.10 ± 9% perf-profile.children.cycles-pp.__filemap_get_folio 0.13 ± 4% -0.1 0.06 ± 13% perf-profile.children.cycles-pp.load_balance 0.19 ± 2% -0.1 0.11 ± 6% perf-profile.children.cycles-pp.native_flush_tlb_one_user 0.17 ± 2% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.get_unmapped_area 0.17 ± 3% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.filemap_read 0.16 ± 4% -0.1 0.08 ± 11% perf-profile.children.cycles-pp.__percpu_counter_init_many 0.16 ± 4% -0.1 0.08 ± 11% perf-profile.children.cycles-pp.folio_add_new_anon_rmap 0.16 ± 2% -0.1 0.09 ± 4% perf-profile.children.cycles-pp.mas_topiary_replace 0.16 ± 2% -0.1 0.09 ± 10% perf-profile.children.cycles-pp.mas_leaf_max_gap 0.16 ± 4% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.try_charge_memcg 0.16 ± 2% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.lookup_fast 0.20 ± 3% -0.1 0.13 ± 7% perf-profile.children.cycles-pp.sched_exec 0.17 ± 2% -0.1 0.10 ± 6% perf-profile.children.cycles-pp.dup_userfaultfd 0.16 ± 4% -0.1 0.10 ± 7% perf-profile.children.cycles-pp.filemap_get_entry 0.15 ± 4% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.memset_orig 0.14 ± 3% -0.1 0.07 ± 10% perf-profile.children.cycles-pp.up_read 0.16 ± 3% -0.1 0.10 ± 5% perf-profile.children.cycles-pp.setup_arg_pages 0.12 ± 4% -0.1 0.05 ± 7% perf-profile.children.cycles-pp.delayed_vfree_work 0.09 -0.1 0.03 ±100% perf-profile.children.cycles-pp._init 0.14 ± 3% -0.1 0.08 ± 8% perf-profile.children.cycles-pp._exit 0.16 ± 3% -0.1 0.09 ± 5% perf-profile.children.cycles-pp.step_into 0.14 ± 2% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.arch_get_unmapped_area_topdown 0.14 ± 2% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.get_user_pages_remote 0.16 ± 3% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.update_load_avg 0.14 ± 3% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.down_read_trylock 0.14 -0.1 0.08 ± 8% perf-profile.children.cycles-pp.free_unref_page 0.13 ± 3% -0.1 0.06 ± 7% perf-profile.children.cycles-pp.dequeue_task_fair 0.15 ± 3% -0.1 0.09 ± 7% perf-profile.children.cycles-pp.memcg_account_kmem 0.15 ± 3% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.get_arg_page 0.11 ± 4% -0.1 0.05 ± 7% perf-profile.children.cycles-pp.vfree 0.09 ± 5% -0.1 0.02 ± 99% perf-profile.children.cycles-pp.dput 0.14 ± 5% -0.1 0.08 ± 9% perf-profile.children.cycles-pp.__get_obj_cgroup_from_memcg 0.14 ± 3% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.do_brk_flags 0.14 ± 3% -0.1 0.08 ± 6% perf-profile.children.cycles-pp.copy_string_kernel 0.14 -0.1 0.08 ± 10% perf-profile.children.cycles-pp.__get_user_pages 0.14 ± 2% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.mas_wr_append 0.12 -0.1 0.06 ± 13% perf-profile.children.cycles-pp.__unfreeze_partials 0.09 ± 5% -0.1 0.03 ±100% perf-profile.children.cycles-pp.free_percpu 0.14 ± 2% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.handle_pte_fault 0.14 ± 15% -0.1 0.08 ± 14% perf-profile.children.cycles-pp.prctl@plt 0.14 ± 3% -0.1 0.08 ± 7% perf-profile.children.cycles-pp.rcu_all_qs 0.14 ± 13% -0.1 0.08 ± 14% perf-profile.children.cycles-pp.call_usermodehelper_exec_async 0.13 ± 5% -0.1 0.08 ± 9% perf-profile.children.cycles-pp.kmem_cache_alloc_bulk 0.09 ± 4% -0.1 0.03 ± 70% perf-profile.children.cycles-pp.update_curr 0.12 ± 3% -0.1 0.06 ± 7% perf-profile.children.cycles-pp.change_protection_range 0.08 -0.1 0.02 ± 99% perf-profile.children.cycles-pp.mas_split_final_node 0.14 ± 3% -0.1 0.09 ± 6% perf-profile.children.cycles-pp.free_unref_page_list 0.14 ± 15% -0.1 0.08 ± 14% perf-profile.children.cycles-pp.kernel_execve 0.11 ± 6% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.do_wp_page 0.11 ± 4% -0.1 0.05 ± 45% perf-profile.children.cycles-pp.__nptl_set_robust 0.13 ± 8% -0.1 0.08 ± 14% perf-profile.children.cycles-pp.main 0.09 ± 7% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.get_partial_node 0.08 -0.1 0.03 ±100% perf-profile.children.cycles-pp.pte_offset_map_nolock 0.13 ± 4% -0.1 0.08 ± 8% perf-profile.children.cycles-pp.prepend_path 0.12 ± 3% -0.1 0.07 ± 5% perf-profile.children.cycles-pp.init_file 0.08 ± 4% -0.1 0.03 ±100% perf-profile.children.cycles-pp.walk_system_ram_range 0.12 ± 4% -0.1 0.07 ± 6% perf-profile.children.cycles-pp.refill_obj_stock 0.11 ± 3% -0.1 0.06 ± 11% perf-profile.children.cycles-pp.change_p4d_range 0.12 ± 6% -0.1 0.08 ± 10% perf-profile.children.cycles-pp.__tlb_remove_page_size 0.08 ± 5% -0.1 0.04 ± 71% perf-profile.children.cycles-pp.memmove 0.12 ± 3% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.activate_task 0.12 ± 3% -0.0 0.07 ± 6% perf-profile.children.cycles-pp.enqueue_task_fair 0.12 ± 4% -0.0 0.07 ± 11% perf-profile.children.cycles-pp.__kmem_cache_alloc_node 0.08 ± 6% -0.0 0.02 ± 99% perf-profile.children.cycles-pp.find_next_iomem_res 0.08 ± 6% -0.0 0.02 ± 99% perf-profile.children.cycles-pp.stress_cpu_bitops.resolver 0.18 ± 4% -0.0 0.13 ± 5% perf-profile.children.cycles-pp.perf_event_task_tick 0.12 ± 3% -0.0 0.08 ± 6% perf-profile.children.cycles-pp.shift_arg_pages 0.11 ± 3% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.__list_add_valid_or_report 0.09 ± 4% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.free_p4d_range 0.11 ± 5% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.mab_mas_cp 0.09 ± 5% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.__getrlimit 0.12 ± 4% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.pick_link 0.08 ± 8% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.__gettimeofday 0.18 ± 2% -0.0 0.13 ± 5% perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context 0.11 ± 3% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.brk 0.10 ± 3% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.free_pgd_range 0.08 -0.0 0.03 ± 70% perf-profile.children.cycles-pp.__mod_node_page_state 0.08 -0.0 0.03 ± 70% perf-profile.children.cycles-pp.find_vma 0.11 -0.0 0.06 ± 11% perf-profile.children.cycles-pp.__mod_lruvec_state 0.10 ± 4% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.vm_unmapped_area 0.12 ± 4% -0.0 0.07 ± 10% perf-profile.children.cycles-pp.__kmem_cache_alloc_bulk 0.11 ± 4% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.__do_sys_brk 0.10 ± 3% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.pgd_alloc 0.10 ± 3% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.change_pmd_range 0.09 ± 5% -0.0 0.04 ± 44% perf-profile.children.cycles-pp.inode_permission 0.11 ± 3% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.rep_stos_alternative 0.11 ± 7% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.do_open_execat 0.08 -0.0 0.04 ± 71% perf-profile.children.cycles-pp.pat_pagerange_is_ram 0.09 ± 4% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.mas_pop_node 0.10 -0.0 0.06 ± 8% perf-profile.children.cycles-pp.__perf_event_header__init_id 0.10 ± 3% -0.0 0.06 ± 6% perf-profile.children.cycles-pp._find_next_bit 0.17 ± 4% -0.0 0.13 ± 6% perf-profile.children.cycles-pp.rb_next 0.11 ± 3% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.padzero 0.11 ± 3% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.free_pcppages_bulk 0.10 ± 4% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.mt_find 0.10 ± 4% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.acct_collect 0.10 ± 4% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.lock_mm_and_find_vma 0.12 ± 4% -0.0 0.08 ± 9% perf-profile.children.cycles-pp.clock_gettime 0.10 ± 3% -0.0 0.06 ± 8% perf-profile.children.cycles-pp.__exit_signal 0.11 ± 3% -0.0 0.07 ± 7% perf-profile.children.cycles-pp.__pmd_alloc 0.10 ± 3% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.__get_free_pages 0.11 ± 4% -0.0 0.06 ± 7% perf-profile.children.cycles-pp.mast_fill_bnode 0.10 ± 3% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.__pte_offset_map 0.09 ± 4% -0.0 0.05 ± 7% perf-profile.children.cycles-pp.__libc_pthread_init 0.10 ± 4% -0.0 0.06 ± 6% perf-profile.children.cycles-pp.__mem_cgroup_uncharge_list 0.07 ± 9% -0.0 0.03 ± 70% perf-profile.children.cycles-pp.error_entry 0.08 ± 4% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.track_pfn_insert 0.08 ± 4% -0.0 0.04 ± 45% perf-profile.children.cycles-pp.lookup_memtype 0.09 ± 4% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.sched_move_task 0.07 -0.0 0.03 ± 70% perf-profile.children.cycles-pp.__output_copy 0.09 ± 4% -0.0 0.05 ± 7% perf-profile.children.cycles-pp.__put_user_4 0.09 ± 6% -0.0 0.06 ± 13% perf-profile.children.cycles-pp.__pud_alloc 0.09 ± 5% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.mas_prev_slot 0.09 -0.0 0.06 ± 6% perf-profile.children.cycles-pp.__x64_sys_close 0.09 ± 5% -0.0 0.06 ± 9% perf-profile.children.cycles-pp.vmf_insert_pfn_prot 0.08 ± 5% -0.0 0.05 ± 7% perf-profile.children.cycles-pp.__anon_vma_interval_tree_augment_rotate 0.09 ± 5% -0.0 0.06 ± 8% perf-profile.children.cycles-pp.folio_mark_accessed 0.11 ± 3% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.stress_exec_remove_pid 0.10 -0.0 0.07 ± 5% perf-profile.children.cycles-pp.task_tick_fair 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.__local_bh_enable_ip 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.do_softirq 0.21 ± 7% +0.1 0.27 ± 5% perf-profile.children.cycles-pp.setproctitle 0.00 +0.1 0.09 ± 10% perf-profile.children.cycles-pp.map__put 0.21 ± 7% +0.1 0.32 ± 4% perf-profile.children.cycles-pp.smp_call_function_many_cond 0.21 ± 7% +0.1 0.32 ± 4% perf-profile.children.cycles-pp.on_each_cpu_cond_mask 0.00 +0.1 0.12 ± 10% perf-profile.children.cycles-pp.rb_erase 0.00 +0.3 0.26 ± 12% perf-profile.children.cycles-pp.maps__put 0.00 +0.3 0.29 ± 12% perf-profile.children.cycles-pp.thread__delete 0.00 +0.3 0.30 ± 12% perf-profile.children.cycles-pp.machine__delete_threads 0.00 +0.3 0.30 ± 12% perf-profile.children.cycles-pp.perf_session__delete 0.39 ± 17% +0.3 0.69 ± 14% perf-profile.children.cycles-pp.user_mode_thread 0.55 ± 4% +0.4 0.98 ± 6% perf-profile.children.cycles-pp.__x64_sys_exit 0.69 ± 18% +0.9 1.57 ± 12% perf-profile.children.cycles-pp.kernel_wait 1.24 ± 15% +1.1 2.36 ± 12% perf-profile.children.cycles-pp.worker_thread 1.22 ± 15% +1.1 2.34 ± 12% perf-profile.children.cycles-pp.process_one_work 3.81 ± 5% +1.1 4.95 ± 6% perf-profile.children.cycles-pp.ret_from_fork_asm 1.07 ± 17% +1.2 2.26 ± 13% perf-profile.children.cycles-pp.call_usermodehelper_exec_work 15.64 +1.7 17.32 perf-profile.children.cycles-pp.__x64_sys_exit_group 3.13 +1.9 5.00 ± 2% perf-profile.children.cycles-pp.mm_update_next_owner 19.70 +2.3 21.98 perf-profile.children.cycles-pp.__do_sys_clone 3.21 +2.5 5.72 ± 3% perf-profile.children.cycles-pp.__clone 20.09 +2.6 22.68 perf-profile.children.cycles-pp.kernel_clone 19.76 +2.7 22.50 perf-profile.children.cycles-pp.copy_process 2.48 +3.1 5.58 ± 3% perf-profile.children.cycles-pp.de_thread 3.67 ± 3% +4.5 8.19 ± 5% perf-profile.children.cycles-pp.wait_task_zombie 7.63 +5.1 12.68 ± 2% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 7.60 +5.1 12.68 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 7.57 +5.1 12.64 ± 2% perf-profile.children.cycles-pp.exit_to_user_mode_loop 7.49 +5.1 12.60 ± 2% perf-profile.children.cycles-pp.arch_do_signal_or_restart 7.49 +5.1 12.60 ± 2% perf-profile.children.cycles-pp.get_signal 5.52 ± 3% +6.3 11.83 ± 4% perf-profile.children.cycles-pp.queued_read_lock_slowpath 79.06 +6.5 85.52 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 79.03 +6.5 85.50 perf-profile.children.cycles-pp.do_syscall_64 23.12 +6.8 29.91 perf-profile.children.cycles-pp.do_group_exit 6.07 ± 2% +7.1 13.14 ± 4% perf-profile.children.cycles-pp.release_task 23.67 +7.2 30.88 perf-profile.children.cycles-pp.do_exit 6.84 ± 2% +8.3 15.13 ± 4% perf-profile.children.cycles-pp.__do_sys_wait4 6.85 ± 2% +8.3 15.15 ± 4% perf-profile.children.cycles-pp.wait4 6.84 ± 2% +8.3 15.13 ± 4% perf-profile.children.cycles-pp.kernel_wait4 7.52 ± 3% +9.2 16.70 ± 4% perf-profile.children.cycles-pp.do_wait 10.83 +10.4 21.26 ± 3% perf-profile.children.cycles-pp.exit_notify 24.57 ± 2% +26.3 50.85 ± 4% perf-profile.children.cycles-pp.queued_write_lock_slowpath 29.35 ± 2% +32.0 61.32 ± 4% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath 7.83 ± 4% -5.3 2.57 ± 14% perf-profile.self.cycles-pp.osq_lock 2.99 ± 2% -1.6 1.36 ± 10% perf-profile.self.cycles-pp.rwsem_spin_on_owner 2.35 ± 2% -1.1 1.24 ± 7% perf-profile.self.cycles-pp.filemap_map_pages 2.38 -1.1 1.28 ± 9% perf-profile.self.cycles-pp.next_uptodate_folio 2.21 -1.0 1.23 ± 8% perf-profile.self.cycles-pp.vma_interval_tree_insert 1.23 ± 9% -0.9 0.29 ± 39% perf-profile.self.cycles-pp.queue_event 1.67 ± 2% -0.8 0.85 ± 7% perf-profile.self.cycles-pp.up_write 1.44 ± 4% -0.7 0.75 ± 7% perf-profile.self.cycles-pp.page_remove_rmap 1.37 ± 2% -0.7 0.70 ± 7% perf-profile.self.cycles-pp.down_write 1.22 ± 3% -0.6 0.59 ± 8% perf-profile.self.cycles-pp.release_pages 1.46 ± 3% -0.6 0.83 ± 8% perf-profile.self.cycles-pp.zap_pte_range 1.20 ± 2% -0.5 0.68 ± 9% perf-profile.self.cycles-pp._compound_head 1.11 -0.5 0.62 ± 7% perf-profile.self.cycles-pp.__slab_free 1.01 ± 2% -0.5 0.55 ± 7% perf-profile.self.cycles-pp.vma_interval_tree_remove 1.07 ± 2% -0.4 0.65 ± 6% perf-profile.self.cycles-pp.anon_vma_interval_tree_insert 0.57 ± 3% -0.4 0.20 ± 10% perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.78 -0.3 0.45 ± 7% perf-profile.self.cycles-pp.kmem_cache_free 0.63 ± 19% -0.3 0.31 ± 6% perf-profile.self.cycles-pp.copy_present_pte 0.72 -0.3 0.42 ± 7% perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook 0.65 -0.3 0.40 ± 6% perf-profile.self.cycles-pp.sync_regs 0.42 ± 7% -0.3 0.16 ± 11% perf-profile.self.cycles-pp.osq_unlock 0.56 -0.2 0.31 ± 5% perf-profile.self.cycles-pp._raw_spin_lock 0.54 -0.2 0.30 ± 6% perf-profile.self.cycles-pp.mod_objcg_state 0.32 ± 4% -0.2 0.08 ± 14% perf-profile.self.cycles-pp.wake_q_add 0.57 -0.2 0.33 ± 6% perf-profile.self.cycles-pp.kmem_cache_alloc 0.54 -0.2 0.32 ± 7% perf-profile.self.cycles-pp._dl_addr 0.46 ± 3% -0.2 0.24 ± 7% perf-profile.self.cycles-pp.free_swap_cache 0.39 ± 3% -0.2 0.18 ± 13% perf-profile.self.cycles-pp.rwsem_down_write_slowpath 0.56 -0.2 0.35 ± 7% perf-profile.self.cycles-pp.native_irq_return_iret 0.46 ± 2% -0.2 0.25 ± 7% perf-profile.self.cycles-pp.__rb_insert_augmented 0.46 -0.2 0.26 ± 4% perf-profile.self.cycles-pp.clear_page_erms 0.46 ± 2% -0.2 0.26 ± 5% perf-profile.self.cycles-pp.copy_mc_enhanced_fast_string 0.41 -0.2 0.23 ± 7% perf-profile.self.cycles-pp.copy_page 0.39 -0.2 0.21 ± 8% perf-profile.self.cycles-pp.__rb_erase_color 0.38 ± 3% -0.2 0.21 ± 8% perf-profile.self.cycles-pp.folio_add_file_rmap_range 0.42 -0.2 0.25 ± 8% perf-profile.self.cycles-pp.mtree_range_walk 0.38 -0.2 0.22 ± 7% perf-profile.self.cycles-pp.__list_del_entry_valid_or_report 0.38 -0.2 0.22 ± 7% perf-profile.self.cycles-pp.__anon_vma_interval_tree_remove 0.35 ± 5% -0.2 0.20 ± 11% perf-profile.self.cycles-pp.__handle_mm_fault 0.34 ± 2% -0.1 0.20 ± 5% perf-profile.self.cycles-pp.mas_next_slot 0.33 ± 2% -0.1 0.18 ± 5% perf-profile.self.cycles-pp.dup_mmap 0.32 -0.1 0.19 ± 9% perf-profile.self.cycles-pp.percpu_counter_add_batch 0.31 ± 2% -0.1 0.18 ± 5% perf-profile.self.cycles-pp.set_pte_range 0.23 ± 9% -0.1 0.10 ± 19% perf-profile.self.cycles-pp.rwsem_optimistic_spin 0.26 ± 2% -0.1 0.14 ± 5% perf-profile.self.cycles-pp.cgroup_rstat_updated 0.26 ± 3% -0.1 0.14 ± 5% perf-profile.self.cycles-pp.perf_event_mmap_output 0.27 -0.1 0.16 ± 7% perf-profile.self.cycles-pp.___perf_sw_event 0.27 ± 2% -0.1 0.16 ± 7% perf-profile.self.cycles-pp.rcu_cblist_dequeue 0.26 -0.1 0.15 ± 7% perf-profile.self.cycles-pp.__cond_resched 0.26 ± 2% -0.1 0.16 ± 9% perf-profile.self.cycles-pp.unmap_single_vma 0.24 ± 2% -0.1 0.14 ± 7% perf-profile.self.cycles-pp.update_sg_wakeup_stats 0.22 -0.1 0.12 ± 9% perf-profile.self.cycles-pp.vma_interval_tree_augment_rotate 0.23 -0.1 0.13 ± 7% perf-profile.self.cycles-pp.anon_vma_clone 0.22 ± 2% -0.1 0.12 ± 6% perf-profile.self.cycles-pp.vm_area_dup 0.22 ± 3% -0.1 0.13 ± 5% perf-profile.self.cycles-pp.unlink_anon_vmas 0.21 ± 3% -0.1 0.12 ± 7% perf-profile.self.cycles-pp.___slab_alloc 0.23 ± 2% -0.1 0.14 ± 10% perf-profile.self.cycles-pp.mas_wr_node_store 0.19 ± 2% -0.1 0.11 ± 10% perf-profile.self.cycles-pp.obj_cgroup_charge 0.19 -0.1 0.11 ± 9% perf-profile.self.cycles-pp.lock_vma_under_rcu 0.19 ± 2% -0.1 0.11 ± 7% perf-profile.self.cycles-pp.native_flush_tlb_one_user 0.18 ± 2% -0.1 0.10 ± 10% perf-profile.self.cycles-pp.__memcpy 0.18 ± 2% -0.1 0.10 ± 8% perf-profile.self.cycles-pp.mas_wr_walk 0.16 ± 2% -0.1 0.09 ± 7% perf-profile.self.cycles-pp.mmap_region 0.19 ± 2% -0.1 0.12 ± 5% perf-profile.self.cycles-pp.vma_interval_tree_insert_after 0.13 ± 2% -0.1 0.06 ± 11% perf-profile.self.cycles-pp.up_read 0.15 ± 3% -0.1 0.08 ± 5% perf-profile.self.cycles-pp.fput 0.15 ± 5% -0.1 0.08 ± 12% perf-profile.self.cycles-pp.memset_orig 0.16 ± 10% -0.1 0.09 ± 8% perf-profile.self.cycles-pp.vm_normal_page 0.16 ± 3% -0.1 0.09 ± 8% perf-profile.self.cycles-pp.__percpu_counter_sum 0.16 ± 4% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.handle_mm_fault 0.16 ± 2% -0.1 0.09 ± 9% perf-profile.self.cycles-pp.mas_leaf_max_gap 0.14 ± 3% -0.1 0.07 ± 8% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state 0.14 ± 4% -0.1 0.08 ± 8% perf-profile.self.cycles-pp.__mod_lruvec_page_state 0.13 ± 14% -0.1 0.07 ± 7% perf-profile.self.cycles-pp.copy_pte_range 0.14 ± 2% -0.1 0.08 ± 7% perf-profile.self.cycles-pp.down_read_trylock 0.14 ± 4% -0.1 0.08 ± 10% perf-profile.self.cycles-pp.__split_vma 0.14 ± 4% -0.1 0.08 ± 10% perf-profile.self.cycles-pp.get_obj_cgroup_from_current 0.13 ± 5% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.filemap_get_entry 0.08 ± 5% -0.1 0.03 ±100% perf-profile.self.cycles-pp.__pte_offset_map 0.13 ± 6% -0.1 0.07 ± 8% perf-profile.self.cycles-pp.__get_obj_cgroup_from_memcg 0.08 ± 5% -0.1 0.02 ± 99% perf-profile.self.cycles-pp.lru_add_fn 0.13 ± 2% -0.1 0.08 ± 10% perf-profile.self.cycles-pp.mas_topiary_replace 0.17 ± 4% -0.1 0.11 ± 6% perf-profile.self.cycles-pp.rb_next 0.12 ± 3% -0.1 0.07 ± 8% perf-profile.self.cycles-pp.__call_rcu_common 0.12 ± 4% -0.1 0.07 ± 6% perf-profile.self.cycles-pp.try_charge_memcg 0.10 ± 4% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.perf_iterate_sb 0.12 ± 3% -0.0 0.07 ± 10% perf-profile.self.cycles-pp.do_user_addr_fault 0.11 ± 3% -0.0 0.06 ± 11% perf-profile.self.cycles-pp.memcg_account_kmem 0.07 ± 5% -0.0 0.02 ± 99% perf-profile.self.cycles-pp.find_next_iomem_res 0.12 -0.0 0.07 ± 6% perf-profile.self.cycles-pp.unmap_page_range 0.12 ± 4% -0.0 0.07 ± 8% perf-profile.self.cycles-pp.refill_obj_stock 0.10 -0.0 0.06 ± 9% perf-profile.self.cycles-pp.perf_event_mmap_event 0.09 -0.0 0.04 ± 45% perf-profile.self.cycles-pp.mas_update_gap 0.10 ± 3% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.rcu_all_qs 0.10 ± 4% -0.0 0.06 ± 9% perf-profile.self.cycles-pp.__list_add_valid_or_report 0.10 ± 4% -0.0 0.06 ± 6% perf-profile.self.cycles-pp.perf_adjust_freq_unthr_context 0.09 ± 5% -0.0 0.05 ± 7% perf-profile.self.cycles-pp.mas_wr_store_entry 0.09 ± 7% -0.0 0.05 perf-profile.self.cycles-pp.__alloc_pages 0.10 -0.0 0.06 ± 9% perf-profile.self.cycles-pp.__pte_offset_map_lock 0.10 ± 4% -0.0 0.06 ± 13% perf-profile.self.cycles-pp.mas_preallocate 0.08 ± 5% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.mas_pop_node 0.09 ± 4% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.mas_walk 0.08 ± 4% -0.0 0.04 ± 45% perf-profile.self.cycles-pp.mas_prev_slot 0.09 ± 4% -0.0 0.05 perf-profile.self.cycles-pp.mab_mas_cp 0.09 -0.0 0.05 ± 8% perf-profile.self.cycles-pp.anon_vma_fork 0.08 ± 5% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.asm_exc_page_fault 0.08 ± 5% -0.0 0.05 ± 8% perf-profile.self.cycles-pp.__perf_sw_event 0.08 ± 4% -0.0 0.05 ± 7% perf-profile.self.cycles-pp.__anon_vma_interval_tree_augment_rotate 0.00 +0.1 0.06 ± 11% perf-profile.self.cycles-pp.map__put 0.00 +0.1 0.10 ± 12% perf-profile.self.cycles-pp.rb_erase 0.21 ± 6% +0.1 0.32 ± 3% perf-profile.self.cycles-pp.smp_call_function_many_cond 1.44 +0.1 1.58 perf-profile.self.cycles-pp.queued_write_lock_slowpath 1.41 +0.2 1.65 perf-profile.self.cycles-pp.mm_update_next_owner 29.30 ± 2% +32.0 61.25 ± 4% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath *************************************************************************************************** lkp-icl-2sp2: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/200%/debian-11.1-x86_64-20220510.cgz/300s/lkp-icl-2sp2/UDP_STREAM/netperf commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 1004672 ± 11% +24.9% 1254792 ± 11% cpuidle..usage 1.47 -0.3 1.17 mpstat.cpu.all.irq% 3.23 -0.9 2.33 mpstat.cpu.all.usr% 781.17 ± 2% -19.2% 630.83 ± 6% perf-c2c.DRAM.remote 604.50 ± 7% -69.0% 187.67 ± 6% perf-c2c.HITM.local 114.83 ± 10% -45.3% 62.83 ± 18% perf-c2c.HITM.remote 5146557 ± 8% -32.9% 3453221 vmstat.memory.cache 7185524 -99.1% 67189 vmstat.system.cs 170126 -14.0% 146355 vmstat.system.in 1.685e+09 +58.0% 2.662e+09 numa-numastat.node0.local_node 1.685e+09 +58.0% 2.663e+09 numa-numastat.node0.numa_hit 1.678e+09 +59.1% 2.67e+09 numa-numastat.node1.local_node 1.678e+09 +59.2% 2.671e+09 numa-numastat.node1.numa_hit 2265 +10.7% 2508 turbostat.Avg_MHz 2325 +10.9% 2578 turbostat.Bzy_MHz 970892 ± 11% +25.8% 1221367 ± 12% turbostat.C1 52445416 -13.8% 45226755 turbostat.IRQ 46.73 -18.5% 38.09 turbostat.RAMWatt 79579 ± 49% -56.4% 34716 ± 52% numa-meminfo.node0.Mapped 21986 ± 92% +333.7% 95357 ± 88% numa-meminfo.node0.Shmem 1618649 ± 8% -68.2% 514719 ± 16% numa-meminfo.node1.Active 1618211 ± 8% -68.2% 514388 ± 16% numa-meminfo.node1.Active(anon) 1063350 ± 47% -63.5% 387904 ± 22% numa-meminfo.node1.Inactive 1063088 ± 47% -63.5% 387713 ± 22% numa-meminfo.node1.Inactive(anon) 468998 ± 82% -84.7% 71668 ± 30% numa-meminfo.node1.Mapped 2317831 ± 18% -76.0% 555548 ± 14% numa-meminfo.node1.Shmem 1636051 ± 8% -63.6% 595731 ± 3% meminfo.Active 1635395 ± 8% -63.6% 595065 ± 3% meminfo.Active(anon) 5018135 ± 8% -33.6% 3330447 meminfo.Cached 3337774 ± 13% -51.9% 1604797 meminfo.Committed_AS 1382280 ± 40% -49.2% 701859 ± 3% meminfo.Inactive 1381892 ± 41% -49.3% 700895 ± 3% meminfo.Inactive(anon) 548575 ± 77% -80.7% 105894 ± 29% meminfo.Mapped 7471815 ± 6% -24.8% 5621352 meminfo.Memused 2338897 ± 18% -72.2% 650557 meminfo.Shmem 7892406 ± 6% -23.4% 6045488 meminfo.max_used_kB 30399 +19.5% 36340 netperf.ThroughputBoth_Mbps 7751914 +20.0% 9303206 netperf.ThroughputBoth_total_Mbps 7975 -95.8% 332.42 ± 2% netperf.ThroughputRecv_Mbps 2033643 -95.8% 85098 ± 2% netperf.ThroughputRecv_total_Mbps 22424 +60.6% 36008 netperf.Throughput_Mbps 5718270 +61.2% 9218107 netperf.Throughput_total_Mbps 1.106e+09 -99.1% 9520520 netperf.time.involuntary_context_switches 8317 +30.3% 10837 netperf.time.percent_of_cpu_this_job_got 24609 +31.2% 32299 netperf.time.system_time 835.50 +3.2% 861.97 netperf.time.user_time 4.438e+09 +20.0% 5.326e+09 netperf.workload 19860 ± 49% -56.3% 8683 ± 51% numa-vmstat.node0.nr_mapped 5500 ± 92% +333.6% 23849 ± 88% numa-vmstat.node0.nr_shmem 1.685e+09 +58.0% 2.663e+09 numa-vmstat.node0.numa_hit 1.685e+09 +58.0% 2.662e+09 numa-vmstat.node0.numa_local 404554 ± 8% -68.2% 128549 ± 16% numa-vmstat.node1.nr_active_anon 265645 ± 47% -63.5% 97018 ± 22% numa-vmstat.node1.nr_inactive_anon 117175 ± 82% -84.6% 18054 ± 28% numa-vmstat.node1.nr_mapped 579350 ± 18% -76.0% 138956 ± 14% numa-vmstat.node1.nr_shmem 404554 ± 8% -68.2% 128549 ± 16% numa-vmstat.node1.nr_zone_active_anon 265645 ± 47% -63.5% 97018 ± 22% numa-vmstat.node1.nr_zone_inactive_anon 1.678e+09 +59.2% 2.671e+09 numa-vmstat.node1.numa_hit 1.678e+09 +59.1% 2.67e+09 numa-vmstat.node1.numa_local 408871 ± 8% -63.6% 148824 ± 3% proc-vmstat.nr_active_anon 169563 ± 3% -4.8% 161384 proc-vmstat.nr_anon_pages 1254896 ± 8% -33.6% 832715 proc-vmstat.nr_file_pages 345783 ± 40% -49.3% 175209 ± 3% proc-vmstat.nr_inactive_anon 137462 ± 77% -80.7% 26555 ± 29% proc-vmstat.nr_mapped 585085 ± 18% -72.2% 162739 proc-vmstat.nr_shmem 31187 -3.0% 30254 proc-vmstat.nr_slab_reclaimable 408871 ± 8% -63.6% 148824 ± 3% proc-vmstat.nr_zone_active_anon 345783 ± 40% -49.3% 175209 ± 3% proc-vmstat.nr_zone_inactive_anon 226345 ± 23% -77.1% 51839 ± 17% proc-vmstat.numa_hint_faults 186550 ± 31% -81.5% 34594 ± 25% proc-vmstat.numa_hint_faults_local 3.363e+09 +58.6% 5.334e+09 proc-vmstat.numa_hit 3.363e+09 +58.6% 5.331e+09 proc-vmstat.numa_local 472352 ± 12% -41.9% 274372 ± 9% proc-vmstat.numa_pte_updates 800993 ± 17% -71.5% 228526 ± 3% proc-vmstat.pgactivate 2.687e+10 +58.6% 4.261e+10 proc-vmstat.pgalloc_normal 1460411 ± 4% -16.3% 1222576 proc-vmstat.pgfault 2.687e+10 +58.6% 4.261e+10 proc-vmstat.pgfree 17657434 +14.3% 20178457 sched_debug.cfs_rq:/.avg_vruntime.avg 2081801 ± 15% +71.2% 3564504 ± 8% sched_debug.cfs_rq:/.avg_vruntime.stddev 4.17 ± 3% +29.3% 5.39 ± 5% sched_debug.cfs_rq:/.h_nr_running.max 0.89 ± 8% +43.8% 1.28 ± 14% sched_debug.cfs_rq:/.h_nr_running.min 0.67 +37.6% 0.92 ± 6% sched_debug.cfs_rq:/.h_nr_running.stddev 2.58 ± 88% +3.7e+06% 96092 ± 75% sched_debug.cfs_rq:/.left_vruntime.avg 267.58 ± 91% +3.3e+06% 8722171 ± 66% sched_debug.cfs_rq:/.left_vruntime.max 25.27 ± 87% +3.4e+06% 869559 ± 67% sched_debug.cfs_rq:/.left_vruntime.stddev 9616 ± 18% +44.3% 13876 ± 25% sched_debug.cfs_rq:/.load.avg 188854 ± 74% +215.5% 595845 ± 52% sched_debug.cfs_rq:/.load.max 18292 ± 75% +233.6% 61017 ± 47% sched_debug.cfs_rq:/.load.stddev 17657434 +14.3% 20178460 sched_debug.cfs_rq:/.min_vruntime.avg 2081802 ± 15% +71.2% 3564506 ± 8% sched_debug.cfs_rq:/.min_vruntime.stddev 0.06 ± 12% +70.6% 0.10 ± 26% sched_debug.cfs_rq:/.nr_running.stddev 2.58 ± 88% +3.7e+06% 96092 ± 75% sched_debug.cfs_rq:/.right_vruntime.avg 267.58 ± 91% +3.3e+06% 8722171 ± 66% sched_debug.cfs_rq:/.right_vruntime.max 25.27 ± 87% +3.4e+06% 869559 ± 67% sched_debug.cfs_rq:/.right_vruntime.stddev 3668 ± 3% +48.1% 5432 ± 5% sched_debug.cfs_rq:/.runnable_avg.max 416.43 ± 8% +126.1% 941.39 ± 8% sched_debug.cfs_rq:/.runnable_avg.stddev 616.47 ± 2% -21.8% 481.86 ± 14% sched_debug.cfs_rq:/.util_avg.min 503.78 ± 3% -15.5% 425.51 sched_debug.cfs_rq:/.util_est_enqueued.avg 1365 ± 5% +12.8% 1540 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.max 255.94 +16.8% 298.85 ± 3% sched_debug.cfs_rq:/.util_est_enqueued.stddev 626590 ± 11% -28.4% 448406 ± 3% sched_debug.cpu.avg_idle.avg 1549078 ± 42% -34.7% 1012313 sched_debug.cpu.avg_idle.max 4009 ± 16% +138.1% 9547 ± 7% sched_debug.cpu.avg_idle.min 386914 ± 24% -23.0% 297912 sched_debug.cpu.avg_idle.stddev 37.85 ± 11% -50.4% 18.76 ± 15% sched_debug.cpu.clock.stddev 0.00 ± 4% -40.8% 0.00 ± 9% sched_debug.cpu.next_balance.stddev 4.08 ± 5% +32.0% 5.39 ± 5% sched_debug.cpu.nr_running.max 0.67 ± 2% +37.0% 0.92 ± 6% sched_debug.cpu.nr_running.stddev 8520448 -99.0% 83495 ± 2% sched_debug.cpu.nr_switches.avg 10903805 ± 3% -98.3% 186011 ± 22% sched_debug.cpu.nr_switches.max 7314094 ± 3% -99.1% 64483 ± 14% sched_debug.cpu.nr_switches.min 607096 ± 18% -97.2% 16789 ± 27% sched_debug.cpu.nr_switches.stddev 1.80 ± 5% -62.3% 0.68 ± 2% perf-stat.i.MPKI 2.394e+10 +9.0% 2.609e+10 perf-stat.i.branch-instructions 1.09 -0.1 0.95 perf-stat.i.branch-miss-rate% 2.591e+08 -4.2% 2.483e+08 perf-stat.i.branch-misses 5.47 ± 5% +12.7 18.16 ± 2% perf-stat.i.cache-miss-rate% 2.337e+08 ± 5% -57.7% 98986630 ± 2% perf-stat.i.cache-misses 4.279e+09 -87.1% 5.541e+08 perf-stat.i.cache-references 7253512 -99.1% 67518 perf-stat.i.context-switches 2.24 -2.1% 2.19 perf-stat.i.cpi 2.887e+11 +10.6% 3.193e+11 perf-stat.i.cpu-cycles 1377 ± 4% +143.6% 3356 ± 2% perf-stat.i.cycles-between-cache-misses 0.01 ± 9% -0.0 0.00 ± 2% perf-stat.i.dTLB-load-miss-rate% 4032948 ± 10% -70.6% 1186470 ± 3% perf-stat.i.dTLB-load-misses 3.57e+10 +13.1% 4.037e+10 perf-stat.i.dTLB-loads 0.01 ± 9% -0.0 0.01 ± 2% perf-stat.i.dTLB-store-miss-rate% 2761122 ± 10% -21.3% 2172944 perf-stat.i.dTLB-store-misses 2.163e+10 +13.0% 2.445e+10 perf-stat.i.dTLB-stores 1.299e+11 +12.4% 1.459e+11 perf-stat.i.instructions 0.45 +1.7% 0.46 perf-stat.i.ipc 2.26 +10.5% 2.49 perf-stat.i.metric.GHz 1219 ± 5% -56.7% 528.18 ± 2% perf-stat.i.metric.K/sec 668.28 +6.9% 714.19 perf-stat.i.metric.M/sec 4230 ± 4% -16.2% 3544 perf-stat.i.minor-faults 33627191 ± 6% -47.6% 17606499 ± 3% perf-stat.i.node-load-misses 860333 ± 28% -62.0% 327037 ± 17% perf-stat.i.node-loads 55972753 ± 5% -61.2% 21707486 ± 6% perf-stat.i.node-store-misses 55165937 ± 6% -54.2% 25284256 ± 8% perf-stat.i.node-stores 4230 ± 4% -16.2% 3544 perf-stat.i.page-faults 1.80 ± 5% -62.3% 0.68 ± 2% perf-stat.overall.MPKI 1.08 -0.1 0.95 perf-stat.overall.branch-miss-rate% 5.47 ± 5% +12.5 17.93 ± 2% perf-stat.overall.cache-miss-rate% 2.22 -1.6% 2.19 perf-stat.overall.cpi 1237 ± 5% +160.5% 3224 ± 2% perf-stat.overall.cycles-between-cache-misses 0.01 ± 9% -0.0 0.00 ± 3% perf-stat.overall.dTLB-load-miss-rate% 0.01 ± 9% -0.0 0.01 perf-stat.overall.dTLB-store-miss-rate% 0.45 +1.6% 0.46 perf-stat.overall.ipc 8952 -5.9% 8424 perf-stat.overall.path-length 2.386e+10 +9.4% 2.61e+10 perf-stat.ps.branch-instructions 2.583e+08 -3.8% 2.484e+08 perf-stat.ps.branch-misses 2.33e+08 ± 5% -57.5% 99108939 ± 2% perf-stat.ps.cache-misses 4.265e+09 -87.0% 5.527e+08 perf-stat.ps.cache-references 7229443 -99.1% 67558 perf-stat.ps.context-switches 2.878e+11 +11.0% 3.194e+11 perf-stat.ps.cpu-cycles 4022939 ± 10% -70.2% 1200154 ± 3% perf-stat.ps.dTLB-load-misses 3.557e+10 +13.5% 4.038e+10 perf-stat.ps.dTLB-loads 2750827 ± 10% -21.0% 2173887 perf-stat.ps.dTLB-store-misses 2.156e+10 +13.4% 2.446e+10 perf-stat.ps.dTLB-stores 1.294e+11 +12.8% 1.46e+11 perf-stat.ps.instructions 0.03 ± 55% +95.5% 0.05 ± 32% perf-stat.ps.major-faults 4199 ± 4% -18.7% 3415 perf-stat.ps.minor-faults 33544663 ± 6% -47.4% 17639108 ± 3% perf-stat.ps.node-load-misses 857620 ± 28% -61.9% 326891 ± 17% perf-stat.ps.node-loads 55803283 ± 5% -61.1% 21731314 ± 6% perf-stat.ps.node-store-misses 54985365 ± 6% -53.9% 25321155 ± 8% perf-stat.ps.node-stores 4199 ± 4% -18.7% 3415 perf-stat.ps.page-faults 3.973e+13 +12.9% 4.487e+13 perf-stat.total.instructions 0.05 ± 13% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom 0.01 ± 19% -100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.01 ± 61% +17477.2% 1.67 ± 17% perf-sched.sch_delay.avg.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.01 ±142% +14366.7% 1.01 ± 7% perf-sched.sch_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.17 ±192% -100.0% 0.00 perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 127.29 ± 84% -96.7% 4.21 ± 76% perf-sched.sch_delay.avg.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.01 ± 43% +7000.0% 0.59 ± 37% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 1.18 ± 76% +242.8% 4.04 ± 29% perf-sched.sch_delay.avg.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select 0.26 ± 17% +1330.0% 3.65 perf-sched.sch_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 0.25 ± 53% +557.4% 1.63 ± 5% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.01 ± 20% +10098.5% 1.14 ± 34% perf-sched.sch_delay.avg.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 34.60 ± 57% -94.0% 2.08 ± 20% perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.01 ±198% +11439.2% 0.98 ± 13% perf-sched.sch_delay.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 94.28 ±181% -97.7% 2.14 ± 86% perf-sched.sch_delay.max.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.udp_recvmsg.inet_recvmsg 0.09 ± 20% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom 0.02 ± 25% -100.0% 0.00 perf-sched.sch_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.06 ± 63% +6835.1% 4.18 ± 13% perf-sched.sch_delay.max.ms.__cond_resched.stop_one_cpu.sched_exec.bprm_execve.part 0.01 ±142% +19200.0% 1.35 ± 33% perf-sched.sch_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.48 ±223% +456.2% 2.64 ± 51% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 5.94 ±219% -100.0% 0.00 perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 613.00 ± 98% -99.6% 2.57 ± 60% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 3205 ± 28% -84.1% 508.83 ± 99% perf-sched.sch_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 0.02 ± 88% +13173.1% 3.21 ± 11% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 2.83 ± 72% +131.2% 6.54 ± 28% perf-sched.sch_delay.max.ms.schedule_hrtimeout_range_clock.do_select.core_sys_select.kern_select 2534 ± 24% -94.1% 148.88 ± 11% perf-sched.sch_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 2.18 ± 66% +169.8% 5.89 ± 49% perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 0.01 ± 31% +18121.6% 2.67 ± 27% perf-sched.sch_delay.max.ms.schedule_timeout.kcompactd.kthread.ret_from_fork 1034 ± 47% -98.0% 20.51 ±179% perf-sched.sch_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 0.03 ±191% +15454.5% 4.04 ± 4% perf-sched.sch_delay.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 0.13 ± 15% +1300.9% 1.88 perf-sched.total_sch_delay.average.ms 3754 ± 19% -65.6% 1292 ± 49% perf-sched.total_sch_delay.max.ms 1.07 ± 15% +1093.0% 12.71 perf-sched.total_wait_and_delay.average.ms 3239167 ± 19% -89.9% 328252 perf-sched.total_wait_and_delay.count.ms 6639 ± 15% -37.2% 4170 ± 8% perf-sched.total_wait_and_delay.max.ms 0.93 ± 15% +1063.2% 10.82 perf-sched.total_wait_time.average.ms 5297 ± 5% -21.3% 4170 ± 8% perf-sched.total_wait_time.max.ms 15.94 ± 10% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 19.65 ± 20% -79.1% 4.10 ± 2% perf-sched.wait_and_delay.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 3.36 ± 38% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 20.96 ± 23% -80.3% 4.14 perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 25.51 ± 31% -83.7% 4.15 perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc_node.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags 313.39 ± 33% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.06 ±223% +75297.4% 801.22 perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1.24 ±117% +52898.3% 656.47 ± 3% perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 40.73 ±100% +446.1% 222.42 perf-sched.wait_and_delay.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 23.95 ± 42% -83.0% 4.06 ± 2% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.55 ± 16% +653.1% 4.11 perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 168.76 ±151% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 0.96 ± 16% +1064.0% 11.18 perf-sched.wait_and_delay.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 108.37 ±124% -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 103.74 ± 35% -84.0% 16.62 ± 46% perf-sched.wait_and_delay.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 300.52 ± 15% +56.5% 470.21 ± 5% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 6.17 ± 70% -97.3% 0.17 ±223% perf-sched.wait_and_delay.count.__cond_resched.__alloc_pages.__folio_alloc.vma_alloc_folio.shmem_alloc_folio 315.83 ± 3% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 418.50 ± 12% +1405.4% 6300 perf-sched.wait_and_delay.count.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 1.33 ± 35% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 698.50 ± 18% +1271.6% 9580 perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 107.83 ± 17% +1624.6% 1859 ± 2% perf-sched.wait_and_delay.count.__cond_resched.kmem_cache_alloc_node.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags 24.67 ± 24% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.17 ±223% +2900.0% 5.00 perf-sched.wait_and_delay.count.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 0.83 ±107% +600.0% 5.83 ± 6% perf-sched.wait_and_delay.count.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 11.33 ± 78% +998.5% 124.50 perf-sched.wait_and_delay.count.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 284.33 ± 24% +1338.0% 4088 perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 1612428 ± 19% -91.6% 135646 perf-sched.wait_and_delay.count.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 11.00 ± 56% -100.0% 0.00 perf-sched.wait_and_delay.count.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 1.50 ±171% +4700.0% 72.00 ± 26% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 3.67 ± 20% +259.1% 13.17 ± 5% perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.ep_poll.do_epoll_wait.__x64_sys_epoll_wait 1621410 ± 19% -89.7% 166227 perf-sched.wait_and_delay.count.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 12.00 ± 47% -100.0% 0.00 perf-sched.wait_and_delay.count.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 3.00 ± 66% +566.7% 20.00 perf-sched.wait_and_delay.count.schedule_timeout.kcompactd.kthread.ret_from_fork 2837 ± 16% -23.4% 2173 ± 9% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.50 ± 33% -77.8% 0.33 ±141% perf-sched.wait_and_delay.count.syslog_print.do_syslog.kmsg_read.vfs_read 104.67 ± 22% +585.2% 717.17 ± 2% perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1043 -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 1721 ± 45% -96.2% 65.67 ± 53% perf-sched.wait_and_delay.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 3.67 ± 44% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 1922 ± 39% -95.8% 79.93 ± 65% perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 1053 ± 21% -96.1% 41.26 ± 74% perf-sched.wait_and_delay.max.ms.__cond_resched.kmem_cache_alloc_node.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags 1996 ± 26% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.06 ±223% +94154.0% 1001 perf-sched.wait_and_delay.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1.63 ±102% +61175.4% 1001 perf-sched.wait_and_delay.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 1628 ± 41% -97.3% 43.26 ± 46% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 2855 ± 14% -95.3% 135.14 ± 17% perf-sched.wait_and_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 4723 ± 33% -69.6% 1436 ± 18% perf-sched.wait_and_delay.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 744.50 ±112% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.rcu_gp_kthread.kthread.ret_from_fork.ret_from_fork_asm 5093 ± 23% -95.5% 229.27 ± 10% perf-sched.wait_and_delay.max.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 1354 ±102% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone 2276 ± 46% -88.5% 262.05 ± 45% perf-sched.wait_and_delay.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 5245 ± 6% -25.2% 3922 ± 17% perf-sched.wait_and_delay.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 5717 ± 20% -47.9% 2976 ± 22% perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 15.93 ± 10% -57.8% 6.73 ± 6% perf-sched.wait_time.avg.ms.__cond_resched.__wait_for_common.affine_move_task.__set_cpus_allowed_ptr.__sched_setaffinity 0.10 ± 10% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom 19.65 ± 20% -79.1% 4.10 ± 2% perf-sched.wait_time.avg.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 3.36 ± 38% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 20.96 ± 23% -80.3% 4.14 perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 25.51 ± 31% -83.7% 4.15 perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc_node.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags 8.24 ± 21% +788.2% 73.17 ± 82% perf-sched.wait_time.avg.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 313.38 ± 33% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.23 ±130% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 1.07 ±221% +74662.7% 800.21 perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1.00 ±152% +65480.4% 655.80 ± 3% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 22.48 ±152% +875.7% 219.36 perf-sched.wait_time.avg.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 21.79 ± 38% -81.4% 4.06 ± 2% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 8.98 ±136% -100.0% 0.00 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.54 ± 16% +653.5% 4.11 perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 0.70 ± 16% +967.4% 7.53 perf-sched.wait_time.avg.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 69.14 ± 27% -76.0% 16.61 ± 18% perf-sched.wait_time.avg.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 300.10 ± 15% +56.7% 470.14 ± 5% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.02 ±166% +4177.8% 0.83 ± 35% perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 383.82 ± 71% -98.6% 5.34 ± 93% perf-sched.wait_time.max.ms.__cond_resched.__skb_datagram_iter.skb_copy_datagram_iter.udp_recvmsg.inet_recvmsg 0.14 ± 9% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.__sys_recvfrom 1721 ± 45% -96.2% 65.67 ± 53% perf-sched.wait_time.max.ms.__cond_resched.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto 3.67 ± 44% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 1922 ± 39% -95.8% 79.93 ± 65% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 1053 ± 21% -96.1% 41.26 ± 74% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc_node.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags 11.92 ± 19% +7785.8% 940.02 ± 75% perf-sched.wait_time.max.ms.__cond_resched.shmem_inode_acct_block.shmem_alloc_and_acct_folio.shmem_get_folio_gfp.shmem_write_begin 1996 ± 26% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.38 ±179% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.task_work_run.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode 1.08 ±220% +92833.7% 1000 perf-sched.wait_time.max.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown] 1.16 ±143% +86225.1% 1000 perf-sched.wait_time.max.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep 335.65 ±139% +199.3% 1004 perf-sched.wait_time.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64 1346 ± 19% -96.8% 43.26 ± 46% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 232.14 ±116% -100.0% 0.00 perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 2829 ± 14% -95.2% 135.14 ± 17% perf-sched.wait_time.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.syscall_exit_to_user_mode.do_syscall_64 3221 ± 21% -59.1% 1318 perf-sched.wait_time.max.ms.pipe_read.vfs_read.ksys_read.do_syscall_64 2695 ± 15% -93.9% 165.66 ± 4% perf-sched.wait_time.max.ms.schedule_timeout.__skb_wait_for_more_packets.__skb_recv_udp.udp_recvmsg 1309 ± 36% -75.8% 317.35 ± 7% perf-sched.wait_time.max.ms.schedule_timeout.rcu_gp_fqs_loop.rcu_gp_kthread.kthread 5157 ± 4% -23.9% 3922 ± 17% perf-sched.wait_time.max.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.09 ±184% +3992.1% 3.87 perf-sched.wait_time.max.ms.wait_for_partner.fifo_open.do_dentry_open.do_open 4853 ± 18% -38.7% 2975 ± 22% perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 23.97 -22.4 1.53 ± 6% perf-profile.calltrace.cycles-pp.recv_omni.process_requests.spawn_child.accept_connection.accept_connections 23.48 -22.0 1.49 ± 6% perf-profile.calltrace.cycles-pp.recvfrom.recv_omni.process_requests.spawn_child.accept_connection 23.26 -21.8 1.48 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni.process_requests.spawn_child 23.12 -21.6 1.48 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni.process_requests 23.04 -21.6 1.45 ± 6% perf-profile.calltrace.cycles-pp.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom 22.51 -21.0 1.46 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvfrom.recv_omni 22.41 ± 15% -20.9 1.53 ± 6% perf-profile.calltrace.cycles-pp.accept_connections.main.__libc_start_main 22.41 ± 15% -20.9 1.53 ± 6% perf-profile.calltrace.cycles-pp.accept_connection.accept_connections.main.__libc_start_main 22.41 ± 15% -20.9 1.53 ± 6% perf-profile.calltrace.cycles-pp.spawn_child.accept_connection.accept_connections.main.__libc_start_main 22.41 ± 15% -20.9 1.53 ± 6% perf-profile.calltrace.cycles-pp.process_requests.spawn_child.accept_connection.accept_connections.main 22.14 -20.7 1.40 ± 7% perf-profile.calltrace.cycles-pp.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64.entry_SYSCALL_64_after_hwframe 21.97 -20.6 1.40 ± 7% perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom.do_syscall_64 21.92 -20.5 1.40 ± 7% perf-profile.calltrace.cycles-pp.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom.__x64_sys_recvfrom 15.63 -14.5 1.17 ± 7% perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.udp_recvmsg.inet_recvmsg.sock_recvmsg.__sys_recvfrom 15.59 -14.4 1.17 ± 7% perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.udp_recvmsg.inet_recvmsg.sock_recvmsg 15.00 -13.9 1.14 ± 7% perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.udp_recvmsg.inet_recvmsg 14.69 -13.6 1.13 ± 7% perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.udp_recvmsg 5.70 ± 3% -0.8 4.86 perf-profile.calltrace.cycles-pp.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 5.80 ± 3% -0.8 5.01 perf-profile.calltrace.cycles-pp.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core 6.84 ± 2% -0.3 6.50 perf-profile.calltrace.cycles-pp.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog 7.13 ± 2% -0.2 6.89 perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll 7.20 ± 2% -0.2 6.98 perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 0.61 ± 5% +0.1 0.74 ± 3% perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages.skb_page_frag_refill.sk_page_frag_refill 0.62 ± 2% +0.2 0.82 perf-profile.calltrace.cycles-pp.move_addr_to_kernel.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.09 ± 4% +0.2 1.31 perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.skb_page_frag_refill.sk_page_frag_refill.__ip_append_data 0.57 +0.2 0.82 perf-profile.calltrace.cycles-pp.irqtime_account_irq.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit 1.21 +0.3 1.46 ± 2% perf-profile.calltrace.cycles-pp.fib_table_lookup.ip_route_output_key_hash_rcu.ip_route_output_flow.udp_sendmsg.sock_sendmsg 0.61 ± 5% +0.3 0.87 perf-profile.calltrace.cycles-pp.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit 0.73 ± 7% +0.3 1.00 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data 0.68 ± 2% +0.3 0.95 perf-profile.calltrace.cycles-pp.ip_rcv.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 0.64 ± 4% +0.3 0.92 perf-profile.calltrace.cycles-pp.__netif_rx.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 1.36 ± 3% +0.3 1.66 perf-profile.calltrace.cycles-pp.__alloc_pages.skb_page_frag_refill.sk_page_frag_refill.__ip_append_data.ip_make_skb 0.69 +0.3 1.00 perf-profile.calltrace.cycles-pp.__udp4_lib_lookup.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core 0.67 ± 2% +0.3 0.98 perf-profile.calltrace.cycles-pp.__check_object_size.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg 0.47 ± 44% +0.3 0.79 perf-profile.calltrace.cycles-pp.enqueue_to_backlog.netif_rx_internal.__netif_rx.loopback_xmit.dev_hard_start_xmit 1.39 +0.3 1.71 ± 2% perf-profile.calltrace.cycles-pp.ip_route_output_key_hash_rcu.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto 0.47 ± 44% +0.3 0.79 perf-profile.calltrace.cycles-pp.security_socket_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 1.99 +0.5 2.53 perf-profile.calltrace.cycles-pp.ip_route_output_flow.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 0.00 +0.5 0.54 perf-profile.calltrace.cycles-pp.send_data.send_omni_inner.send_udp_stream.main.__libc_start_main 1.32 ± 3% +0.5 1.85 perf-profile.calltrace.cycles-pp.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb 0.00 +0.5 0.55 ± 2% perf-profile.calltrace.cycles-pp._copy_from_user.move_addr_to_kernel.__sys_sendto.__x64_sys_sendto.do_syscall_64 0.00 +0.6 0.56 ± 3% perf-profile.calltrace.cycles-pp.sock_wfree.loopback_xmit.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2 0.00 +0.6 0.56 perf-profile.calltrace.cycles-pp.sched_clock_cpu.irqtime_account_irq.__do_softirq.do_softirq.__local_bh_enable_ip 0.00 +0.6 0.57 perf-profile.calltrace.cycles-pp.__netif_receive_skb_core.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 1.47 ± 2% +0.6 2.06 perf-profile.calltrace.cycles-pp.dev_hard_start_xmit.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 0.08 ±223% +0.6 0.68 perf-profile.calltrace.cycles-pp.check_heap_object.__check_object_size.ip_generic_getfrag.__ip_append_data.ip_make_skb 0.00 +0.6 0.60 ± 2% perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 2.02 ± 2% +0.6 2.64 perf-profile.calltrace.cycles-pp.skb_page_frag_refill.sk_page_frag_refill.__ip_append_data.ip_make_skb.udp_sendmsg 0.17 ±141% +0.6 0.79 perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data 0.00 +0.6 0.64 ± 2% perf-profile.calltrace.cycles-pp.udp4_lib_lookup2.__udp4_lib_lookup.__udp4_lib_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 2.10 ± 2% +0.7 2.76 perf-profile.calltrace.cycles-pp.sk_page_frag_refill.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg 0.00 +0.7 0.66 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb 0.00 +0.7 0.67 ± 2% perf-profile.calltrace.cycles-pp.validate_xmit_skb.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 10.81 +0.7 11.51 perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb 10.57 +0.7 11.28 perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2 10.72 +0.7 11.43 perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.ip_send_skb 1.64 ± 3% +0.8 2.42 perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb 1.73 ± 3% +0.9 2.58 perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg 0.26 ±100% +0.9 1.13 perf-profile.calltrace.cycles-pp.free_unref_page_prepare.free_unref_page.skb_release_data.kfree_skb_reason.udp_queue_rcv_one_skb 2.20 ± 2% +0.9 3.14 perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg 0.83 ± 2% +0.9 1.78 perf-profile.calltrace.cycles-pp.free_unref_page.skb_release_data.kfree_skb_reason.udp_queue_rcv_one_skb.udp_unicast_rcv_skb 1.34 +1.6 2.99 perf-profile.calltrace.cycles-pp.skb_release_data.kfree_skb_reason.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv 1.39 +1.7 3.08 perf-profile.calltrace.cycles-pp.kfree_skb_reason.udp_queue_rcv_one_skb.udp_unicast_rcv_skb.__udp4_lib_rcv.ip_protocol_deliver_rcu 13.20 +1.7 14.94 perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg 13.81 +1.9 15.67 perf-profile.calltrace.cycles-pp.ip_finish_output2.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg 14.84 +2.3 17.12 perf-profile.calltrace.cycles-pp.ip_send_skb.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 15.31 +2.4 17.70 perf-profile.calltrace.cycles-pp.udp_send_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 94.98 ± 3% +4.2 99.17 perf-profile.calltrace.cycles-pp.__libc_start_main 94.90 ± 3% +4.2 99.14 perf-profile.calltrace.cycles-pp.main.__libc_start_main 4.80 ± 7% +5.9 10.72 perf-profile.calltrace.cycles-pp.__ip_select_ident.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg 5.23 ± 6% +6.0 11.27 perf-profile.calltrace.cycles-pp.__ip_make_skb.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 31.86 +12.6 44.43 perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.ip_generic_getfrag.__ip_append_data.ip_make_skb 32.48 +12.8 45.28 perf-profile.calltrace.cycles-pp._copy_from_iter.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg 33.38 +13.2 46.60 perf-profile.calltrace.cycles-pp.ip_generic_getfrag.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg 39.94 +15.7 55.68 perf-profile.calltrace.cycles-pp.__ip_append_data.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto 45.69 +22.0 67.73 perf-profile.calltrace.cycles-pp.ip_make_skb.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto 70.52 +24.5 95.06 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner.send_udp_stream.main 70.32 +24.6 94.94 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner.send_udp_stream 71.13 +24.8 95.89 perf-profile.calltrace.cycles-pp.sendto.send_omni_inner.send_udp_stream.main.__libc_start_main 72.45 +25.1 97.56 perf-profile.calltrace.cycles-pp.send_omni_inner.send_udp_stream.main.__libc_start_main 72.48 +25.1 97.61 perf-profile.calltrace.cycles-pp.send_udp_stream.main.__libc_start_main 64.78 +25.3 90.08 perf-profile.calltrace.cycles-pp.udp_sendmsg.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64 65.70 +25.7 91.42 perf-profile.calltrace.cycles-pp.sock_sendmsg.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe 67.28 +26.6 93.84 perf-profile.calltrace.cycles-pp.__sys_sendto.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto 67.49 +26.7 94.14 perf-profile.calltrace.cycles-pp.__x64_sys_sendto.do_syscall_64.entry_SYSCALL_64_after_hwframe.sendto.send_omni_inner 24.71 -23.2 1.55 ± 3% perf-profile.children.cycles-pp.recvfrom 23.97 -22.4 1.57 ± 3% perf-profile.children.cycles-pp.accept_connections 23.97 -22.4 1.57 ± 3% perf-profile.children.cycles-pp.accept_connection 23.97 -22.4 1.57 ± 3% perf-profile.children.cycles-pp.spawn_child 23.97 -22.4 1.57 ± 3% perf-profile.children.cycles-pp.process_requests 23.97 -22.4 1.57 ± 3% perf-profile.children.cycles-pp.recv_omni 23.12 -21.6 1.50 ± 3% perf-profile.children.cycles-pp.__x64_sys_recvfrom 23.06 -21.6 1.50 ± 3% perf-profile.children.cycles-pp.__sys_recvfrom 22.14 -20.7 1.45 ± 3% perf-profile.children.cycles-pp.sock_recvmsg 21.98 -20.5 1.44 ± 3% perf-profile.children.cycles-pp.inet_recvmsg 21.93 -20.5 1.44 ± 3% perf-profile.children.cycles-pp.udp_recvmsg 15.63 -14.4 1.20 ± 3% perf-profile.children.cycles-pp.skb_copy_datagram_iter 15.60 -14.4 1.20 ± 3% perf-profile.children.cycles-pp.__skb_datagram_iter 15.01 -13.8 1.17 ± 4% perf-profile.children.cycles-pp._copy_to_iter 14.80 -13.6 1.16 ± 4% perf-profile.children.cycles-pp.copyout 5.70 -5.6 0.12 ± 3% perf-profile.children.cycles-pp.schedule 5.58 -5.5 0.13 ± 2% perf-profile.children.cycles-pp.__schedule 4.92 -4.8 0.13 perf-profile.children.cycles-pp.__skb_recv_udp 3.91 -3.8 0.08 ± 4% perf-profile.children.cycles-pp.__skb_wait_for_more_packets 3.50 -3.4 0.07 ± 5% perf-profile.children.cycles-pp.schedule_timeout 3.46 -3.3 0.20 ± 2% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 3.63 -3.2 0.44 perf-profile.children.cycles-pp.syscall_exit_to_user_mode 3.45 ± 5% -3.1 0.34 ± 4% perf-profile.children.cycles-pp.__udp_enqueue_schedule_skb 2.70 ± 6% -2.6 0.05 ± 8% perf-profile.children.cycles-pp.sock_def_readable 2.63 ± 7% -2.6 0.05 perf-profile.children.cycles-pp.__wake_up_common_lock 2.50 ± 7% -2.5 0.04 ± 44% perf-profile.children.cycles-pp.__wake_up_common 2.45 ± 7% -2.4 0.02 ± 99% perf-profile.children.cycles-pp.autoremove_wake_function 2.50 -2.4 0.09 perf-profile.children.cycles-pp.exit_to_user_mode_loop 2.41 ± 7% -2.4 0.02 ± 99% perf-profile.children.cycles-pp.try_to_wake_up 0.97 ± 4% -0.9 0.09 ± 5% perf-profile.children.cycles-pp.__consume_stateless_skb 5.84 ± 2% -0.9 4.98 perf-profile.children.cycles-pp.udp_queue_rcv_one_skb 5.91 ± 2% -0.8 5.06 perf-profile.children.cycles-pp.udp_unicast_rcv_skb 6.99 ± 2% -0.4 6.59 perf-profile.children.cycles-pp.__udp4_lib_rcv 7.27 ± 2% -0.3 6.98 perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 7.34 ± 2% -0.3 7.05 perf-profile.children.cycles-pp.ip_local_deliver_finish 0.80 ± 2% -0.2 0.61 ± 2% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 0.58 ± 2% -0.2 0.40 perf-profile.children.cycles-pp._raw_spin_lock_irqsave 0.70 ± 3% -0.2 0.53 ± 3% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 0.67 ± 3% -0.2 0.51 ± 3% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 0.65 ± 3% -0.2 0.49 ± 3% perf-profile.children.cycles-pp.hrtimer_interrupt 0.75 -0.2 0.59 perf-profile.children.cycles-pp.sockfd_lookup_light 0.60 ± 3% -0.1 0.45 ± 4% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.53 ± 2% -0.1 0.40 ± 5% perf-profile.children.cycles-pp.tick_sched_timer 0.50 ± 3% -0.1 0.38 ± 4% perf-profile.children.cycles-pp.tick_sched_handle 0.50 ± 3% -0.1 0.38 ± 4% perf-profile.children.cycles-pp.update_process_times 0.29 ± 6% -0.1 0.18 ± 5% perf-profile.children.cycles-pp.perf_event_task_tick 0.29 ± 6% -0.1 0.18 ± 5% perf-profile.children.cycles-pp.perf_adjust_freq_unthr_context 0.43 ± 3% -0.1 0.34 ± 5% perf-profile.children.cycles-pp.scheduler_tick 0.59 -0.1 0.51 perf-profile.children.cycles-pp.__fget_light 0.14 ± 2% -0.1 0.08 ± 4% perf-profile.children.cycles-pp.import_single_range 0.45 ± 2% -0.0 0.41 ± 3% perf-profile.children.cycles-pp.__virt_addr_valid 0.19 -0.0 0.15 perf-profile.children.cycles-pp.__list_del_entry_valid_or_report 0.14 ± 3% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.__list_add_valid_or_report 0.12 ± 4% -0.0 0.09 ± 9% perf-profile.children.cycles-pp.__intel_pmu_enable_all 0.09 -0.0 0.07 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore 0.07 ± 5% +0.0 0.08 perf-profile.children.cycles-pp.netdev_core_pick_tx 0.13 +0.0 0.14 ± 3% perf-profile.children.cycles-pp.check_stack_object 0.44 +0.0 0.46 perf-profile.children.cycles-pp.native_sched_clock 0.11 +0.0 0.13 ± 3% perf-profile.children.cycles-pp.ip_finish_output 0.06 ± 8% +0.0 0.08 ± 4% perf-profile.children.cycles-pp.demo_interval_tick 0.49 +0.0 0.51 perf-profile.children.cycles-pp.sched_clock 0.22 ± 4% +0.0 0.24 ± 3% perf-profile.children.cycles-pp.__cond_resched 0.20 ± 3% +0.0 0.22 perf-profile.children.cycles-pp.ipv4_mtu 0.14 ± 3% +0.0 0.16 ± 5% perf-profile.children.cycles-pp.security_sock_rcv_skb 0.06 ± 6% +0.0 0.09 perf-profile.children.cycles-pp.policy_node 0.18 ± 2% +0.0 0.20 ± 2% perf-profile.children.cycles-pp.ip_send_check 0.06 ± 6% +0.0 0.09 ± 4% perf-profile.children.cycles-pp.skb_pull_rcsum 0.17 ± 2% +0.0 0.20 ± 2% perf-profile.children.cycles-pp.compute_score 0.07 +0.0 0.10 ± 3% perf-profile.children.cycles-pp.neigh_hh_output 0.06 ± 8% +0.0 0.09 ± 4% perf-profile.children.cycles-pp.ip_local_deliver 0.09 ± 5% +0.0 0.12 perf-profile.children.cycles-pp.__folio_put 0.58 +0.0 0.61 perf-profile.children.cycles-pp.sched_clock_cpu 0.07 ± 7% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.should_failslab 0.07 ± 5% +0.0 0.11 ± 3% perf-profile.children.cycles-pp.inet_sendmsg 0.06 +0.0 0.10 perf-profile.children.cycles-pp.validate_xmit_xfrm 0.09 ± 4% +0.0 0.13 ± 3% perf-profile.children.cycles-pp.raw_v4_input 0.08 ± 13% +0.0 0.12 ± 6% perf-profile.children.cycles-pp.eth_type_trans 0.09 +0.0 0.14 ± 3% perf-profile.children.cycles-pp.udp4_hwcsum 1.23 +0.0 1.28 perf-profile.children.cycles-pp.__check_object_size 0.06 ± 45% +0.0 0.11 perf-profile.children.cycles-pp.apparmor_socket_sendmsg 0.09 ± 4% +0.1 0.14 ± 2% perf-profile.children.cycles-pp.ip_skb_dst_mtu 0.02 ±141% +0.1 0.07 ± 7% perf-profile.children.cycles-pp.raw_local_deliver 0.00 +0.1 0.05 perf-profile.children.cycles-pp.__raise_softirq_irqoff 0.00 +0.1 0.05 perf-profile.children.cycles-pp.sendto@plt 0.10 ± 4% +0.1 0.15 perf-profile.children.cycles-pp.nf_hook_slow 0.01 ±223% +0.1 0.06 ± 7% perf-profile.children.cycles-pp.__page_cache_release 0.08 ± 5% +0.1 0.14 perf-profile.children.cycles-pp.__build_skb_around 0.18 ± 2% +0.1 0.24 ± 2% perf-profile.children.cycles-pp.udp4_csum_init 0.02 ± 99% +0.1 0.08 ± 5% perf-profile.children.cycles-pp.__mem_cgroup_uncharge 0.18 ± 2% +0.1 0.24 perf-profile.children.cycles-pp.ipv4_pktinfo_prepare 0.00 +0.1 0.06 perf-profile.children.cycles-pp.find_exception 0.00 +0.1 0.06 perf-profile.children.cycles-pp.skb_put 0.13 +0.1 0.19 perf-profile.children.cycles-pp.xfrm_lookup_with_ifid 0.24 +0.1 0.30 perf-profile.children.cycles-pp.ip_setup_cork 0.25 +0.1 0.31 ± 2% perf-profile.children.cycles-pp.__ip_local_out 0.08 ± 5% +0.1 0.15 ± 4% perf-profile.children.cycles-pp.destroy_large_folio 0.28 ± 3% +0.1 0.35 ± 6% perf-profile.children.cycles-pp.skb_set_owner_w 0.18 +0.1 0.25 perf-profile.children.cycles-pp.alloc_pages 0.00 +0.1 0.07 ± 5% perf-profile.children.cycles-pp.inet_lookup_reuseport 0.16 +0.1 0.23 perf-profile.children.cycles-pp.prep_compound_page 0.44 +0.1 0.51 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irq 0.24 ± 2% +0.1 0.32 perf-profile.children.cycles-pp.sk_filter_trim_cap 0.16 ± 3% +0.1 0.23 ± 2% perf-profile.children.cycles-pp.xfrm_lookup_route 0.19 +0.1 0.27 perf-profile.children.cycles-pp.dst_release 0.51 +0.1 0.59 perf-profile.children.cycles-pp.send_data 0.21 ± 2% +0.1 0.30 perf-profile.children.cycles-pp.siphash_3u32 0.23 ± 2% +0.1 0.32 ± 3% perf-profile.children.cycles-pp.get_pfnblock_flags_mask 0.40 ± 4% +0.1 0.48 perf-profile.children.cycles-pp.__mkroute_output 0.16 ± 3% +0.1 0.25 perf-profile.children.cycles-pp.__ip_finish_output 0.10 ± 6% +0.1 0.19 ± 2% perf-profile.children.cycles-pp.skb_network_protocol 0.48 ± 3% +0.1 0.57 ± 3% perf-profile.children.cycles-pp._raw_spin_trylock 0.28 ± 4% +0.1 0.38 perf-profile.children.cycles-pp.ip_rcv_core 0.34 ± 4% +0.1 0.46 ± 4% perf-profile.children.cycles-pp.ip_rcv_finish_core 0.10 ± 10% +0.1 0.22 ± 4% perf-profile.children.cycles-pp.memcg_slab_post_alloc_hook 0.65 ± 5% +0.1 0.77 ± 2% perf-profile.children.cycles-pp.rmqueue 0.50 ± 10% +0.1 0.63 ± 2% perf-profile.children.cycles-pp.aa_sk_perm 0.22 ± 2% +0.1 0.35 perf-profile.children.cycles-pp.free_tail_page_prepare 0.46 ± 2% +0.1 0.60 perf-profile.children.cycles-pp._copy_from_user 0.44 +0.1 0.59 perf-profile.children.cycles-pp.__netif_receive_skb_core 0.50 +0.2 0.68 ± 2% perf-profile.children.cycles-pp.udp4_lib_lookup2 0.22 ± 3% +0.2 0.41 ± 2% perf-profile.children.cycles-pp.netif_skb_features 0.34 +0.2 0.54 perf-profile.children.cycles-pp.ip_output 0.66 ± 2% +0.2 0.88 perf-profile.children.cycles-pp.move_addr_to_kernel 0.35 ± 2% +0.2 0.57 ± 2% perf-profile.children.cycles-pp.sock_wfree 0.60 ± 4% +0.2 0.84 perf-profile.children.cycles-pp.enqueue_to_backlog 1.14 ± 4% +0.2 1.38 perf-profile.children.cycles-pp.get_page_from_freelist 1.24 +0.2 1.50 ± 2% perf-profile.children.cycles-pp.fib_table_lookup 0.55 ± 12% +0.3 0.81 perf-profile.children.cycles-pp.security_socket_sendmsg 0.62 +0.3 0.88 perf-profile.children.cycles-pp.irqtime_account_irq 0.64 ± 4% +0.3 0.90 perf-profile.children.cycles-pp.netif_rx_internal 0.58 +0.3 0.85 perf-profile.children.cycles-pp.kmem_cache_free 0.71 ± 2% +0.3 0.98 perf-profile.children.cycles-pp.ip_rcv 0.67 ± 4% +0.3 0.94 perf-profile.children.cycles-pp.__netif_rx 1.42 ± 4% +0.3 1.70 perf-profile.children.cycles-pp.__alloc_pages 0.42 ± 2% +0.3 0.71 perf-profile.children.cycles-pp.validate_xmit_skb 0.51 ± 2% +0.3 0.81 perf-profile.children.cycles-pp.kmalloc_reserve 0.73 +0.3 1.04 perf-profile.children.cycles-pp.__udp4_lib_lookup 1.42 +0.3 1.74 ± 2% perf-profile.children.cycles-pp.ip_route_output_key_hash_rcu 0.87 +0.4 1.30 perf-profile.children.cycles-pp.free_unref_page_prepare 1.43 ± 3% +0.5 1.89 perf-profile.children.cycles-pp.free_unref_page 1.22 ± 5% +0.5 1.75 perf-profile.children.cycles-pp.kmem_cache_alloc_node 2.04 +0.5 2.58 perf-profile.children.cycles-pp.ip_route_output_flow 1.38 ± 2% +0.5 1.92 perf-profile.children.cycles-pp.loopback_xmit 11.03 +0.6 11.61 perf-profile.children.cycles-pp.__local_bh_enable_ip 1.51 ± 2% +0.6 2.10 perf-profile.children.cycles-pp.dev_hard_start_xmit 2.07 ± 3% +0.6 2.68 perf-profile.children.cycles-pp.skb_page_frag_refill 10.79 +0.6 11.41 perf-profile.children.cycles-pp.__do_softirq 10.91 +0.6 11.54 perf-profile.children.cycles-pp.do_softirq 2.17 ± 3% +0.7 2.83 perf-profile.children.cycles-pp.sk_page_frag_refill 2.34 ± 2% +0.8 3.12 perf-profile.children.cycles-pp.skb_release_data 1.70 ± 3% +0.8 2.49 perf-profile.children.cycles-pp.__alloc_skb 1.78 ± 3% +0.9 2.63 perf-profile.children.cycles-pp.alloc_skb_with_frags 2.25 +0.9 3.19 perf-profile.children.cycles-pp.sock_alloc_send_pskb 96.16 +1.2 97.35 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 95.65 +1.3 96.92 perf-profile.children.cycles-pp.do_syscall_64 13.46 +1.7 15.12 perf-profile.children.cycles-pp.__dev_queue_xmit 1.42 ± 2% +1.7 3.13 perf-profile.children.cycles-pp.kfree_skb_reason 14.06 +1.7 15.81 perf-profile.children.cycles-pp.ip_finish_output2 15.14 +2.2 17.30 perf-profile.children.cycles-pp.ip_send_skb 15.58 +2.3 17.87 perf-profile.children.cycles-pp.udp_send_skb 95.06 ± 3% +4.1 99.17 perf-profile.children.cycles-pp.main 94.98 ± 3% +4.2 99.17 perf-profile.children.cycles-pp.__libc_start_main 4.88 ± 7% +5.9 10.81 ± 2% perf-profile.children.cycles-pp.__ip_select_ident 5.34 ± 6% +6.1 11.41 perf-profile.children.cycles-pp.__ip_make_skb 32.67 +12.4 45.06 perf-profile.children.cycles-pp.copyin 33.03 +12.5 45.58 perf-profile.children.cycles-pp._copy_from_iter 33.96 +13.0 46.92 perf-profile.children.cycles-pp.ip_generic_getfrag 40.53 +15.5 56.05 perf-profile.children.cycles-pp.__ip_append_data 46.26 +21.7 67.96 perf-profile.children.cycles-pp.ip_make_skb 73.02 +23.8 96.79 perf-profile.children.cycles-pp.sendto 65.26 +25.1 90.40 perf-profile.children.cycles-pp.udp_sendmsg 72.89 +25.2 98.06 perf-profile.children.cycles-pp.send_omni_inner 72.82 +25.2 98.02 perf-profile.children.cycles-pp.send_udp_stream 66.10 +25.5 91.62 perf-profile.children.cycles-pp.sock_sendmsg 68.21 +26.0 94.25 perf-profile.children.cycles-pp.__sys_sendto 68.43 +26.1 94.54 perf-profile.children.cycles-pp.__x64_sys_sendto 14.69 -13.5 1.16 ± 4% perf-profile.self.cycles-pp.copyout 0.55 -0.2 0.39 perf-profile.self.cycles-pp._raw_spin_lock_irqsave 0.18 ± 12% -0.1 0.09 ± 4% perf-profile.self.cycles-pp.perf_adjust_freq_unthr_context 0.58 -0.1 0.50 perf-profile.self.cycles-pp.__fget_light 0.52 -0.1 0.44 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 0.13 ± 2% -0.1 0.07 perf-profile.self.cycles-pp.import_single_range 0.12 ± 3% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.exit_to_user_mode_prepare 0.43 ± 2% -0.0 0.39 ± 3% perf-profile.self.cycles-pp.__virt_addr_valid 0.10 ± 4% -0.0 0.07 perf-profile.self.cycles-pp.__local_bh_enable_ip 0.16 ± 3% -0.0 0.13 ± 2% perf-profile.self.cycles-pp.__list_del_entry_valid_or_report 0.35 ± 3% -0.0 0.32 perf-profile.self.cycles-pp.do_syscall_64 0.11 ± 4% -0.0 0.09 ± 7% perf-profile.self.cycles-pp.__list_add_valid_or_report 0.12 ± 4% -0.0 0.09 ± 9% perf-profile.self.cycles-pp.__intel_pmu_enable_all 0.09 ± 4% -0.0 0.07 ± 5% perf-profile.self.cycles-pp.sockfd_lookup_light 0.12 -0.0 0.10 ± 3% perf-profile.self.cycles-pp.__netif_receive_skb_one_core 0.07 -0.0 0.06 ± 6% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore 0.05 +0.0 0.07 ± 7% perf-profile.self.cycles-pp.ipv4_pktinfo_prepare 0.19 ± 2% +0.0 0.20 ± 3% perf-profile.self.cycles-pp.ipv4_mtu 0.05 +0.0 0.07 ± 5% perf-profile.self.cycles-pp.demo_interval_tick 0.26 +0.0 0.28 perf-profile.self.cycles-pp.check_heap_object 0.09 ± 4% +0.0 0.11 ± 6% perf-profile.self.cycles-pp.security_sock_rcv_skb 0.06 +0.0 0.08 ± 4% perf-profile.self.cycles-pp.security_socket_sendmsg 0.19 ± 2% +0.0 0.22 ± 3% perf-profile.self.cycles-pp.sock_alloc_send_pskb 0.31 ± 2% +0.0 0.33 perf-profile.self.cycles-pp.net_rx_action 0.16 ± 3% +0.0 0.18 ± 3% perf-profile.self.cycles-pp.compute_score 0.17 ± 2% +0.0 0.19 ± 2% perf-profile.self.cycles-pp.ip_send_check 0.05 +0.0 0.08 ± 4% perf-profile.self.cycles-pp.ip_local_deliver 0.05 +0.0 0.08 ± 4% perf-profile.self.cycles-pp.skb_pull_rcsum 0.06 ± 6% +0.0 0.09 ± 5% perf-profile.self.cycles-pp.neigh_hh_output 0.14 ± 3% +0.0 0.18 ± 8% perf-profile.self.cycles-pp.ip_protocol_deliver_rcu 0.08 ± 6% +0.0 0.11 ± 3% perf-profile.self.cycles-pp.__ip_local_out 0.08 ± 4% +0.0 0.11 ± 4% perf-profile.self.cycles-pp.udp4_hwcsum 0.07 ± 5% +0.0 0.10 ± 4% perf-profile.self.cycles-pp.kmalloc_reserve 0.09 +0.0 0.13 ± 3% perf-profile.self.cycles-pp.alloc_pages 0.06 ± 9% +0.0 0.09 ± 7% perf-profile.self.cycles-pp.should_failslab 0.14 ± 4% +0.0 0.18 ± 2% perf-profile.self.cycles-pp.move_addr_to_kernel 0.05 ± 7% +0.0 0.09 perf-profile.self.cycles-pp.validate_xmit_xfrm 0.27 ± 2% +0.0 0.31 ± 2% perf-profile.self.cycles-pp.__alloc_pages 0.08 ± 9% +0.0 0.12 ± 8% perf-profile.self.cycles-pp.eth_type_trans 0.08 ± 5% +0.0 0.12 ± 4% perf-profile.self.cycles-pp.raw_v4_input 0.07 +0.0 0.11 perf-profile.self.cycles-pp.nf_hook_slow 0.30 ± 2% +0.0 0.34 perf-profile.self.cycles-pp.process_backlog 0.07 ± 7% +0.0 0.11 ± 3% perf-profile.self.cycles-pp.__ip_finish_output 0.12 ± 5% +0.0 0.17 ± 6% perf-profile.self.cycles-pp.sk_filter_trim_cap 0.12 ± 4% +0.0 0.16 perf-profile.self.cycles-pp.free_unref_page 0.13 ± 3% +0.0 0.18 perf-profile.self.cycles-pp.ip_setup_cork 0.07 ± 6% +0.0 0.12 ± 3% perf-profile.self.cycles-pp.__build_skb_around 0.08 +0.0 0.13 ± 2% perf-profile.self.cycles-pp.ip_skb_dst_mtu 0.00 +0.1 0.05 perf-profile.self.cycles-pp.policy_node 0.00 +0.1 0.05 perf-profile.self.cycles-pp.find_exception 0.10 ± 3% +0.1 0.15 perf-profile.self.cycles-pp.ip_rcv 0.05 ± 45% +0.1 0.10 ± 4% perf-profile.self.cycles-pp.apparmor_socket_sendmsg 0.17 ± 2% +0.1 0.22 perf-profile.self.cycles-pp.udp4_csum_init 0.16 ± 3% +0.1 0.21 perf-profile.self.cycles-pp.dst_release 0.13 ± 3% +0.1 0.19 perf-profile.self.cycles-pp.dev_hard_start_xmit 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.__folio_put 0.00 +0.1 0.06 ± 9% perf-profile.self.cycles-pp.__page_cache_release 0.08 ± 5% +0.1 0.14 ± 4% perf-profile.self.cycles-pp.alloc_skb_with_frags 0.00 +0.1 0.06 ± 8% perf-profile.self.cycles-pp.inet_lookup_reuseport 0.25 +0.1 0.30 perf-profile.self.cycles-pp.ip_send_skb 0.01 ±223% +0.1 0.06 ± 7% perf-profile.self.cycles-pp.netif_rx_internal 0.00 +0.1 0.06 perf-profile.self.cycles-pp.kfree_skb_reason 0.17 ± 2% +0.1 0.23 perf-profile.self.cycles-pp.ip_route_output_key_hash_rcu 0.12 +0.1 0.18 perf-profile.self.cycles-pp.xfrm_lookup_with_ifid 0.53 ± 3% +0.1 0.59 ± 2% perf-profile.self.cycles-pp.ip_finish_output2 0.01 ±223% +0.1 0.07 perf-profile.self.cycles-pp.inet_sendmsg 0.10 ± 4% +0.1 0.17 ± 4% perf-profile.self.cycles-pp.sk_page_frag_refill 0.36 ± 5% +0.1 0.42 ± 2% perf-profile.self.cycles-pp.__mkroute_output 0.14 ± 3% +0.1 0.21 perf-profile.self.cycles-pp.prep_compound_page 0.42 +0.1 0.48 perf-profile.self.cycles-pp._raw_spin_lock_irq 0.28 ± 3% +0.1 0.34 ± 5% perf-profile.self.cycles-pp.skb_set_owner_w 0.00 +0.1 0.07 ± 5% perf-profile.self.cycles-pp.destroy_large_folio 0.39 +0.1 0.46 ± 2% perf-profile.self.cycles-pp.send_data 0.31 ± 3% +0.1 0.38 ± 3% perf-profile.self.cycles-pp.__check_object_size 0.22 ± 2% +0.1 0.29 perf-profile.self.cycles-pp.__x64_sys_sendto 0.21 ± 2% +0.1 0.28 perf-profile.self.cycles-pp.get_page_from_freelist 0.00 +0.1 0.07 ± 6% perf-profile.self.cycles-pp.__mem_cgroup_uncharge 0.16 ± 3% +0.1 0.23 perf-profile.self.cycles-pp.ip_make_skb 0.26 ± 8% +0.1 0.33 perf-profile.self.cycles-pp.rmqueue 0.36 ± 2% +0.1 0.44 perf-profile.self.cycles-pp.__do_softirq 0.20 ± 2% +0.1 0.28 perf-profile.self.cycles-pp.ip_generic_getfrag 0.24 ± 11% +0.1 0.32 ± 3% perf-profile.self.cycles-pp.enqueue_to_backlog 0.20 ± 2% +0.1 0.29 ± 2% perf-profile.self.cycles-pp.udp_queue_rcv_one_skb 0.46 ± 4% +0.1 0.54 ± 3% perf-profile.self.cycles-pp._raw_spin_trylock 0.07 ± 6% +0.1 0.16 perf-profile.self.cycles-pp.syscall_exit_to_user_mode 0.09 ± 5% +0.1 0.18 ± 3% perf-profile.self.cycles-pp.skb_network_protocol 0.20 ± 2% +0.1 0.29 perf-profile.self.cycles-pp.siphash_3u32 0.22 ± 2% +0.1 0.31 ± 3% perf-profile.self.cycles-pp.get_pfnblock_flags_mask 0.36 ± 2% +0.1 0.45 perf-profile.self.cycles-pp.udp_send_skb 0.20 +0.1 0.29 ± 2% perf-profile.self.cycles-pp.irqtime_account_irq 0.12 ± 4% +0.1 0.22 ± 3% perf-profile.self.cycles-pp.netif_skb_features 0.23 ± 2% +0.1 0.32 perf-profile.self.cycles-pp.sock_sendmsg 0.19 ± 3% +0.1 0.28 perf-profile.self.cycles-pp.__udp4_lib_rcv 0.14 ± 2% +0.1 0.23 perf-profile.self.cycles-pp.free_tail_page_prepare 0.16 ± 3% +0.1 0.26 ± 3% perf-profile.self.cycles-pp.validate_xmit_skb 0.27 ± 3% +0.1 0.37 perf-profile.self.cycles-pp.ip_rcv_core 0.35 +0.1 0.45 perf-profile.self.cycles-pp.sendto 0.09 ± 10% +0.1 0.20 ± 3% perf-profile.self.cycles-pp.memcg_slab_post_alloc_hook 0.34 ± 2% +0.1 0.45 perf-profile.self.cycles-pp.__ip_make_skb 0.34 ± 4% +0.1 0.45 ± 4% perf-profile.self.cycles-pp.ip_rcv_finish_core 0.50 ± 2% +0.1 0.62 perf-profile.self.cycles-pp.skb_release_data 0.45 ± 2% +0.1 0.58 perf-profile.self.cycles-pp._copy_from_user 0.28 +0.1 0.41 ± 3% perf-profile.self.cycles-pp.udp4_lib_lookup2 0.23 +0.1 0.36 perf-profile.self.cycles-pp.ip_route_output_flow 0.43 ± 3% +0.1 0.57 ± 2% perf-profile.self.cycles-pp.__dev_queue_xmit 0.43 ± 12% +0.1 0.56 ± 2% perf-profile.self.cycles-pp.aa_sk_perm 0.24 +0.1 0.38 perf-profile.self.cycles-pp.__udp4_lib_lookup 0.43 +0.1 0.57 perf-profile.self.cycles-pp.__netif_receive_skb_core 0.25 +0.1 0.39 perf-profile.self.cycles-pp.ip_output 0.36 +0.2 0.51 perf-profile.self.cycles-pp._copy_from_iter 0.34 ± 2% +0.2 0.52 ± 2% perf-profile.self.cycles-pp.__alloc_skb 0.62 +0.2 0.79 perf-profile.self.cycles-pp.send_omni_inner 0.34 ± 2% +0.2 0.56 ± 2% perf-profile.self.cycles-pp.sock_wfree 0.50 +0.2 0.73 perf-profile.self.cycles-pp.free_unref_page_prepare 0.47 +0.2 0.71 perf-profile.self.cycles-pp.skb_page_frag_refill 0.57 +0.3 0.84 perf-profile.self.cycles-pp.kmem_cache_free 0.92 ± 2% +0.3 1.19 ± 2% perf-profile.self.cycles-pp.fib_table_lookup 0.95 ± 5% +0.3 1.28 ± 2% perf-profile.self.cycles-pp.kmem_cache_alloc_node 0.78 +0.3 1.12 perf-profile.self.cycles-pp.__sys_sendto 1.16 +0.5 1.71 perf-profile.self.cycles-pp.udp_sendmsg 2.13 +0.9 3.08 perf-profile.self.cycles-pp.__ip_append_data 4.64 ± 7% +5.8 10.46 ± 2% perf-profile.self.cycles-pp.__ip_select_ident 32.40 +12.4 44.77 perf-profile.self.cycles-pp.copyin *************************************************************************************************** lkp-icl-2sp2: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory ========================================================================================= compiler/cpufreq_governor/ipc/iterations/kconfig/mode/nr_threads/rootfs/tbox_group/testcase: gcc-12/performance/pipe/4/x86_64-rhel-8.3/threads/50%/debian-11.1-x86_64-20220510.cgz/lkp-icl-2sp2/hackbench commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 11501929 ± 3% +68.8% 19412513 cpuidle..usage 5.00 ± 14% +1.5 6.49 ± 7% mpstat.cpu.all.idle% 0.03 +0.0 0.04 mpstat.cpu.all.soft% 12.06 +1.3 13.33 mpstat.cpu.all.usr% 5326554 ± 9% +14.0% 6069773 ± 2% vmstat.memory.cache 8815288 -69.6% 2682569 vmstat.system.cs 1343184 -59.7% 541194 vmstat.system.in 61966805 +14.7% 71085124 numa-numastat.node0.local_node 62024282 +14.7% 71118693 numa-numastat.node0.numa_hit 62376162 +16.2% 72483066 numa-numastat.node1.local_node 62445945 +16.2% 72585175 numa-numastat.node1.numa_hit 457.17 ± 19% -98.9% 5.17 ± 47% perf-c2c.DRAM.local 5400 ± 5% -93.2% 369.00 ± 50% perf-c2c.DRAM.remote 36324 ± 3% -99.8% 65.83 ± 37% perf-c2c.HITM.local 593.50 ± 9% -90.5% 56.50 ± 55% perf-c2c.HITM.remote 36918 ± 3% -99.7% 122.33 ± 43% perf-c2c.HITM.total 11462642 ± 3% +69.1% 19384557 turbostat.C1 5.94 ± 7% +1.5 7.45 ± 6% turbostat.C1% 5.53 ± 7% +21.5% 6.72 ± 7% turbostat.CPU%c1 2.55 ±199% +2201.3% 58.61 ± 49% turbostat.IPC 91841576 -66.0% 31216183 turbostat.IRQ 33171 ± 10% -24.6% 25015 ± 8% turbostat.POLL 40.82 +3.5% 42.26 turbostat.RAMWatt 22245 ± 20% -50.8% 10952 ± 12% meminfo.Active 21589 ± 21% -52.3% 10296 ± 13% meminfo.Active(anon) 5166665 ± 9% +13.3% 5851434 ± 2% meminfo.Cached 3325001 ± 15% +19.9% 3985812 ± 3% meminfo.Committed_AS 3014195 ± 16% +22.3% 3687514 ± 3% meminfo.Inactive 3013807 ± 16% +22.3% 3687126 ± 3% meminfo.Inactive(anon) 787068 ± 7% -26.8% 576258 ± 4% meminfo.Mapped 2485474 ± 20% +27.6% 3171140 ± 4% meminfo.Shmem 9587 ± 58% -76.9% 2211 ± 42% numa-meminfo.node0.Active 9259 ± 60% -79.7% 1883 ± 37% numa-meminfo.node0.Active(anon) 522086 ±138% -98.8% 6399 ± 9% numa-meminfo.node0.Shmem 2285325 ± 29% +50.7% 3443925 ± 5% numa-meminfo.node1.Inactive 2285131 ± 29% +50.7% 3443731 ± 5% numa-meminfo.node1.Inactive(anon) 630807 ± 12% -28.5% 450860 ± 7% numa-meminfo.node1.Mapped 4583 ± 12% -21.5% 3598 ± 20% numa-meminfo.node1.PageTables 1953932 ± 32% +62.1% 3166983 ± 4% numa-meminfo.node1.Shmem 982091 +19.1% 1169196 hackbench.throughput 937326 +19.1% 1116209 hackbench.throughput_avg 982091 +19.1% 1169196 hackbench.throughput_best 892204 +17.7% 1050479 hackbench.throughput_worst 64.72 -15.9% 54.42 hackbench.time.elapsed_time 64.72 -15.9% 54.42 hackbench.time.elapsed_time.max 1.258e+08 -99.9% 135322 ± 5% hackbench.time.involuntary_context_switches 51327 -13.6% 44325 hackbench.time.minor_page_faults 12140 -1.4% 11968 hackbench.time.percent_of_cpu_this_job_got 6871 -18.6% 5591 hackbench.time.system_time 986.79 -6.5% 922.72 hackbench.time.user_time 4.596e+08 -71.4% 1.315e+08 hackbench.time.voluntary_context_switches 2328 ± 59% -79.8% 470.67 ± 37% numa-vmstat.node0.nr_active_anon 130558 ±138% -98.8% 1599 ± 9% numa-vmstat.node0.nr_shmem 2328 ± 59% -79.8% 470.67 ± 37% numa-vmstat.node0.nr_zone_active_anon 62024201 +14.7% 71118659 numa-vmstat.node0.numa_hit 61966726 +14.7% 71085090 numa-vmstat.node0.numa_local 571427 ± 29% +51.0% 862801 ± 5% numa-vmstat.node1.nr_inactive_anon 157801 ± 12% -28.3% 113172 ± 7% numa-vmstat.node1.nr_mapped 1145 ± 12% -21.3% 901.33 ± 20% numa-vmstat.node1.nr_page_table_pages 488630 ± 32% +62.4% 793580 ± 4% numa-vmstat.node1.nr_shmem 571426 ± 29% +51.0% 862799 ± 5% numa-vmstat.node1.nr_zone_inactive_anon 62445598 +16.2% 72584990 numa-vmstat.node1.numa_hit 62375815 +16.2% 72482881 numa-vmstat.node1.numa_local 17.47 ± 35% -15.6 1.84 ±166% perf-profile.calltrace.cycles-pp.task_work_run.do_exit.do_group_exit.get_signal.arch_do_signal_or_restart 17.47 ± 35% -15.6 1.84 ±166% perf-profile.calltrace.cycles-pp.__fput.task_work_run.do_exit.do_group_exit.get_signal 15.20 ± 32% -13.4 1.84 ±166% perf-profile.calltrace.cycles-pp.perf_release.__fput.task_work_run.do_exit.do_group_exit 15.20 ± 32% -13.4 1.84 ±166% perf-profile.calltrace.cycles-pp.perf_event_release_kernel.perf_release.__fput.task_work_run.do_exit 7.20 ± 78% -6.0 1.19 ±146% perf-profile.calltrace.cycles-pp.number.vsnprintf.seq_printf.show_interrupts.seq_read_iter 6.20 ± 57% -5.3 0.93 ±223% perf-profile.calltrace.cycles-pp._free_event.perf_event_release_kernel.perf_release.__fput.task_work_run 6.37 ± 55% -5.0 1.32 ±157% perf-profile.calltrace.cycles-pp.smp_call_function_single.event_function_call.perf_event_release_kernel.perf_release.__fput 6.37 ± 55% -5.0 1.32 ±157% perf-profile.calltrace.cycles-pp.event_function_call.perf_event_release_kernel.perf_release.__fput.task_work_run 6.10 ± 92% -3.3 2.82 ±169% perf-profile.calltrace.cycles-pp.fault_in_readable.fault_in_iov_iter_readable.generic_perform_write.shmem_file_write_iter.vfs_write 17.47 ± 35% -15.6 1.90 ±159% perf-profile.children.cycles-pp.task_work_run 17.47 ± 35% -14.6 2.88 ±111% perf-profile.children.cycles-pp.__fput 15.20 ± 32% -13.3 1.85 ±165% perf-profile.children.cycles-pp.perf_release 15.20 ± 32% -13.3 1.85 ±165% perf-profile.children.cycles-pp.perf_event_release_kernel 6.37 ± 55% -5.0 1.32 ±157% perf-profile.children.cycles-pp.smp_call_function_single 6.37 ± 55% -5.0 1.32 ±157% perf-profile.children.cycles-pp.event_function_call 5.12 ± 52% -3.8 1.32 ±157% perf-profile.self.cycles-pp.smp_call_function_single 5525 ± 19% -53.5% 2569 ± 13% proc-vmstat.nr_active_anon 137664 -4.5% 131536 proc-vmstat.nr_anon_pages 1293208 ± 9% +12.9% 1460660 ± 2% proc-vmstat.nr_file_pages 754874 ± 16% +21.8% 919494 ± 3% proc-vmstat.nr_inactive_anon 197541 ± 7% -27.3% 143595 ± 5% proc-vmstat.nr_mapped 1756 -6.0% 1651 proc-vmstat.nr_page_table_pages 622910 ± 20% +26.9% 790585 ± 4% proc-vmstat.nr_shmem 5525 ± 19% -53.5% 2569 ± 13% proc-vmstat.nr_zone_active_anon 754874 ± 16% +21.8% 919494 ± 3% proc-vmstat.nr_zone_inactive_anon 146005 ± 11% -25.2% 109177 ± 15% proc-vmstat.numa_hint_faults_local 1.245e+08 +15.5% 1.437e+08 proc-vmstat.numa_hit 1.243e+08 +15.5% 1.436e+08 proc-vmstat.numa_local 5859 ± 39% -63.3% 2153 ± 53% proc-vmstat.numa_pages_migrated 422264 ± 12% -29.3% 298674 ± 9% proc-vmstat.numa_pte_updates 14681 ± 58% -97.0% 442.00 ± 12% proc-vmstat.pgactivate 1.245e+08 +15.4% 1.438e+08 proc-vmstat.pgalloc_normal 810834 ± 6% -18.6% 659998 ± 4% proc-vmstat.pgfault 1.229e+08 +15.1% 1.414e+08 proc-vmstat.pgfree 5859 ± 39% -63.3% 2153 ± 53% proc-vmstat.pgmigrate_success 16433 ± 7% -23.3% 12605 proc-vmstat.pgreuse 601472 -16.7% 500736 proc-vmstat.unevictable_pgs_scanned 0.37 -0.2 0.19 perf-stat.i.branch-miss-rate% 1.878e+08 -47.7% 98129053 perf-stat.i.branch-misses 1.594e+09 +4.1% 1.659e+09 perf-stat.i.cache-references 9235551 -69.5% 2820801 perf-stat.i.context-switches 1.23 -1.4% 1.21 perf-stat.i.cpi 558021 -10.3% 500432 perf-stat.i.cpu-migrations 1761 ± 2% -7.1% 1637 ± 3% perf-stat.i.cycles-between-cache-misses 0.01 ± 6% -0.0 0.01 ± 5% perf-stat.i.dTLB-load-miss-rate% 7.519e+10 +2.3% 7.692e+10 perf-stat.i.dTLB-loads 0.00 ± 4% +0.0 0.00 ± 2% perf-stat.i.dTLB-store-miss-rate% 320069 ± 4% +13.5% 363267 perf-stat.i.dTLB-store-misses 4.394e+10 +2.4% 4.499e+10 perf-stat.i.dTLB-stores 2.585e+11 +1.3% 2.62e+11 perf-stat.i.instructions 0.82 +1.4% 0.83 perf-stat.i.ipc 541.12 -10.3% 485.16 ± 3% perf-stat.i.metric.K/sec 1337 +1.9% 1363 perf-stat.i.metric.M/sec 80.81 +6.1 86.94 perf-stat.i.node-store-miss-rate% 22165236 +7.1% 23737654 ± 2% perf-stat.i.node-store-misses 5569243 ± 2% -34.1% 3670117 ± 5% perf-stat.i.node-stores 0.37 -0.2 0.19 perf-stat.overall.branch-miss-rate% 1.22 -1.5% 1.20 perf-stat.overall.cpi 1723 ± 2% -5.7% 1624 ± 3% perf-stat.overall.cycles-between-cache-misses 0.01 ± 6% -0.0 0.01 ± 4% perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 4% +0.0 0.00 perf-stat.overall.dTLB-store-miss-rate% 0.82 +1.5% 0.83 perf-stat.overall.ipc 79.96 +6.6 86.61 perf-stat.overall.node-store-miss-rate% 1.846e+08 -47.9% 96187705 perf-stat.ps.branch-misses 1.566e+09 +3.9% 1.627e+09 perf-stat.ps.cache-references 9086394 -69.6% 2766280 perf-stat.ps.context-switches 547994 -10.5% 490624 perf-stat.ps.cpu-migrations 7.396e+10 +2.0% 7.546e+10 perf-stat.ps.dTLB-loads 313926 ± 4% +13.2% 355487 perf-stat.ps.dTLB-store-misses 4.322e+10 +2.1% 4.413e+10 perf-stat.ps.dTLB-stores 2.543e+11 +1.1% 2.57e+11 perf-stat.ps.instructions 5454114 ± 2% -34.0% 3600130 ± 5% perf-stat.ps.node-stores 1.659e+13 -14.4% 1.421e+13 perf-stat.total.instructions 3539626 -99.9% 2518 ± 10% sched_debug.cfs_rq:/.avg_vruntime.avg 5096880 ± 6% -99.4% 30625 ± 21% sched_debug.cfs_rq:/.avg_vruntime.max 3382807 -100.0% 77.16 ± 29% sched_debug.cfs_rq:/.avg_vruntime.min 169658 ± 13% -97.5% 4318 ± 18% sched_debug.cfs_rq:/.avg_vruntime.stddev 6.56 ± 2% -98.2% 0.12 ± 19% sched_debug.cfs_rq:/.h_nr_running.avg 14.83 ± 9% -84.3% 2.33 ± 31% sched_debug.cfs_rq:/.h_nr_running.max 3.40 ± 4% -88.9% 0.38 ± 15% sched_debug.cfs_rq:/.h_nr_running.stddev 110634 ±177% +1184.3% 1420878 ± 54% sched_debug.cfs_rq:/.load.max 11881 ±140% +1102.8% 142905 ± 52% sched_debug.cfs_rq:/.load.stddev 515.16 ± 66% +310.8% 2116 ± 57% sched_debug.cfs_rq:/.load_avg.avg 30912 ± 62% +182.5% 87339 ± 2% sched_debug.cfs_rq:/.load_avg.max 1.08 ± 31% -100.0% 0.00 sched_debug.cfs_rq:/.load_avg.min 3468 ± 65% +235.4% 11631 ± 32% sched_debug.cfs_rq:/.load_avg.stddev 3539626 -99.9% 2519 ± 10% sched_debug.cfs_rq:/.min_vruntime.avg 5096880 ± 6% -99.4% 30625 ± 21% sched_debug.cfs_rq:/.min_vruntime.max 3382807 -100.0% 77.16 ± 29% sched_debug.cfs_rq:/.min_vruntime.min 169658 ± 13% -97.5% 4318 ± 18% sched_debug.cfs_rq:/.min_vruntime.stddev 0.55 -77.9% 0.12 ± 19% sched_debug.cfs_rq:/.nr_running.avg 0.20 ± 20% +86.0% 0.38 ± 15% sched_debug.cfs_rq:/.nr_running.stddev 512.00 +5811.2% 30265 ±136% sched_debug.cfs_rq:/.removed.load_avg.max 262.50 +98.2% 520.33 sched_debug.cfs_rq:/.removed.runnable_avg.max 262.50 +98.2% 520.33 sched_debug.cfs_rq:/.removed.util_avg.max 6520 ± 2% -96.1% 255.07 ± 3% sched_debug.cfs_rq:/.runnable_avg.avg 12582 ± 5% -90.3% 1220 ± 16% sched_debug.cfs_rq:/.runnable_avg.max 956.83 ± 51% -100.0% 0.00 sched_debug.cfs_rq:/.runnable_avg.min 1986 ± 3% -84.6% 306.55 ± 5% sched_debug.cfs_rq:/.runnable_avg.stddev 611.45 -59.0% 250.81 ± 3% sched_debug.cfs_rq:/.util_avg.avg 161.75 ± 22% -100.0% 0.00 sched_debug.cfs_rq:/.util_avg.min 318.54 ± 2% -92.0% 25.53 ± 26% sched_debug.cfs_rq:/.util_est_enqueued.avg 1178 ± 5% -29.7% 828.00 sched_debug.cfs_rq:/.util_est_enqueued.max 215.82 ± 5% -47.1% 114.24 ± 9% sched_debug.cfs_rq:/.util_est_enqueued.stddev 480833 ± 2% +76.4% 848127 sched_debug.cpu.avg_idle.avg 801726 ± 7% +36.7% 1096160 ± 12% sched_debug.cpu.avg_idle.max 157976 ± 4% +41.5% 223472 ± 4% sched_debug.cpu.avg_idle.stddev 83176 ± 2% -34.4% 54588 ± 4% sched_debug.cpu.clock.avg 83225 ± 2% -34.4% 54594 ± 4% sched_debug.cpu.clock.max 83121 ± 2% -34.3% 54578 ± 4% sched_debug.cpu.clock.min 28.92 ± 25% -88.0% 3.48 ± 3% sched_debug.cpu.clock.stddev 82606 ± 2% -34.1% 54459 ± 4% sched_debug.cpu.clock_task.avg 82808 ± 2% -34.1% 54585 ± 4% sched_debug.cpu.clock_task.max 74892 ± 3% -37.6% 46731 ± 5% sched_debug.cpu.clock_task.min 7846 -94.9% 403.47 ± 11% sched_debug.cpu.curr->pid.avg 10604 -60.1% 4227 sched_debug.cpu.curr->pid.max 0.00 ± 23% -44.8% 0.00 ± 23% sched_debug.cpu.next_balance.stddev 6.54 ± 2% -98.1% 0.12 ± 17% sched_debug.cpu.nr_running.avg 14.83 ± 9% -84.3% 2.33 ± 31% sched_debug.cpu.nr_running.max 3.39 ± 4% -88.8% 0.38 ± 14% sched_debug.cpu.nr_running.stddev 2125675 -99.9% 1749 sched_debug.cpu.nr_switches.avg 2505734 ± 5% -98.5% 36714 ± 30% sched_debug.cpu.nr_switches.max 1744096 ± 3% -100.0% 134.33 ± 6% sched_debug.cpu.nr_switches.min 99374 ± 12% -95.8% 4130 ± 14% sched_debug.cpu.nr_switches.stddev 142.50 ± 21% -62.9% 52.83 ± 18% sched_debug.cpu.nr_uninterruptible.max -141.58 -90.7% -13.17 sched_debug.cpu.nr_uninterruptible.min 42.75 ± 11% -85.3% 6.27 ± 7% sched_debug.cpu.nr_uninterruptible.stddev 83123 ± 2% -34.3% 54584 ± 4% sched_debug.cpu_clk 81893 ± 2% -34.8% 53354 ± 4% sched_debug.ktime 84041 ± 2% -34.0% 55457 ± 4% sched_debug.sched_clk *************************************************************************************************** lkp-skl-d08: 36 threads 1 sockets Intel(R) Core(TM) i9-9980XE CPU @ 3.00GHz (Skylake) with 32G memory ========================================================================================= class/compiler/cpufreq_governor/disk/fs/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime: filesystem/gcc-12/performance/1SSD/xfs/x86_64-rhel-8.3/10%/debian-11.1-x86_64-20220510.cgz/lkp-skl-d08/dnotify/stress-ng/60s commit: 7ad0354d18 ("sched/headers: Remove duplicated includes in kernel/sched/sched.h") 8afa2eec7f ("sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks") 7ad0354d18ae05e9 8afa2eec7fb5de648561dba3b19 ---------------- --------------------------- %stddev %change %stddev \ | \ 4116086 +22.7% 5049549 cpuidle..usage 7.17 -4.6% 6.84 iostat.cpu.system 1.51 -13.1% 1.31 iostat.cpu.user 0.09 +0.0 0.10 mpstat.cpu.all.soft% 1.51 -0.2 1.31 mpstat.cpu.all.usr% 6999 ± 2% +90.6% 13341 vmstat.io.bo 76876 +75.1% 134583 vmstat.system.cs 56176 +6.1% 59603 vmstat.system.in 140962 ± 2% +89.6% 267323 meminfo.Active 140740 ± 2% +89.8% 267104 meminfo.Active(anon) 669736 +20.2% 805324 meminfo.Committed_AS 154066 ± 2% +88.3% 290124 meminfo.Shmem 589958 ± 2% +102.9% 1197072 turbostat.C1 3.20 ± 2% +2.4 5.54 turbostat.C1% 1208014 +15.2% 1391274 turbostat.C1E 8.55 +2.7 11.27 turbostat.C1E% 394730 ± 2% +61.2% 636419 turbostat.POLL 0.10 ± 4% +0.1 0.17 turbostat.POLL% 914019 -15.7% 770790 stress-ng.dnotify.ops 15233 -15.7% 12846 stress-ng.dnotify.ops_per_sec 36560733 -15.7% 30831585 stress-ng.time.file_system_outputs 158471 -51.9% 76226 stress-ng.time.involuntary_context_switches 233.67 -9.7% 211.00 stress-ng.time.percent_of_cpu_this_job_got 122.05 -9.2% 110.86 stress-ng.time.system_time 437727 ± 2% +274.8% 1640596 stress-ng.time.voluntary_context_switches 35185 ± 2% +89.8% 66776 proc-vmstat.nr_active_anon 4569227 -15.7% 3852767 proc-vmstat.nr_dirtied 736600 +4.6% 770633 proc-vmstat.nr_file_pages 79569 +2.5% 81525 proc-vmstat.nr_inactive_anon 9245 -1.0% 9154 proc-vmstat.nr_kernel_stack 12204 +4.0% 12688 proc-vmstat.nr_mapped 38518 ± 2% +88.3% 72534 proc-vmstat.nr_shmem 35185 ± 2% +89.8% 66776 proc-vmstat.nr_zone_active_anon 79569 +2.5% 81525 proc-vmstat.nr_zone_inactive_anon 5126131 -12.2% 4501912 proc-vmstat.numa_hit 5128956 -12.3% 4499112 proc-vmstat.numa_local 52552 ± 2% +86.8% 98141 proc-vmstat.pgactivate 5244883 -12.4% 4596666 proc-vmstat.pgalloc_normal 5174307 -13.4% 4478515 proc-vmstat.pgfree 456141 ± 2% +91.6% 873837 proc-vmstat.pgpgout 11233 ± 2% -13.7% 9699 ± 4% sched_debug.cfs_rq:/.avg_vruntime.avg 2572 ± 12% -23.1% 1977 ± 9% sched_debug.cfs_rq:/.avg_vruntime.min 15056 ± 6% -19.2% 12165 ± 9% sched_debug.cfs_rq:/.avg_vruntime.stddev 1.58 ± 21% +47.4% 2.33 ± 15% sched_debug.cfs_rq:/.h_nr_running.max 0.48 ± 6% +23.2% 0.59 ± 11% sched_debug.cfs_rq:/.h_nr_running.stddev 11233 ± 2% -13.7% 9700 ± 4% sched_debug.cfs_rq:/.min_vruntime.avg 2572 ± 12% -23.1% 1977 ± 9% sched_debug.cfs_rq:/.min_vruntime.min 15056 ± 6% -19.2% 12165 ± 9% sched_debug.cfs_rq:/.min_vruntime.stddev 1.58 ± 21% +47.4% 2.33 ± 15% sched_debug.cfs_rq:/.nr_running.max 0.48 ± 6% +22.4% 0.59 ± 11% sched_debug.cfs_rq:/.nr_running.stddev 0.49 ± 6% +20.8% 0.59 ± 11% sched_debug.cpu.nr_running.stddev 73121 +67.4% 122381 sched_debug.cpu.nr_switches.avg 157331 ± 2% +197.2% 467519 ± 3% sched_debug.cpu.nr_switches.max 25739 ± 3% +251.8% 90541 sched_debug.cpu.nr_switches.stddev 1152 +202.9% 3490 ± 6% sched_debug.cpu.nr_uninterruptible.max -280.33 +109.0% -585.92 sched_debug.cpu.nr_uninterruptible.min 318.24 ± 2% +165.1% 843.69 ± 3% sched_debug.cpu.nr_uninterruptible.stddev 0.13 ± 2% +16.1% 0.15 perf-stat.i.MPKI 1.957e+09 -3.0% 1.898e+09 perf-stat.i.branch-instructions 1.48 +0.1 1.61 perf-stat.i.branch-miss-rate% 30093641 +4.8% 31541589 perf-stat.i.branch-misses 1543470 +17.1% 1807270 perf-stat.i.cache-misses 65963562 ± 3% +6.3% 70112140 perf-stat.i.cache-references 79898 +76.0% 140612 perf-stat.i.context-switches 1.23 +2.3% 1.25 perf-stat.i.cpi 1.291e+10 -1.2% 1.276e+10 perf-stat.i.cpu-cycles 1469 +69.6% 2492 perf-stat.i.cpu-migrations 14595 ± 2% -7.9% 13446 perf-stat.i.cycles-between-cache-misses 0.04 +0.0 0.05 perf-stat.i.dTLB-load-miss-rate% 2.809e+09 -4.2% 2.69e+09 perf-stat.i.dTLB-loads 0.00 ± 6% +0.0 0.00 ± 7% perf-stat.i.dTLB-store-miss-rate% 53064 ± 6% +28.2% 68019 ± 7% perf-stat.i.dTLB-store-misses 1.735e+09 -5.4% 1.642e+09 perf-stat.i.dTLB-stores 19.72 ± 4% -1.9 17.79 ± 5% perf-stat.i.iTLB-load-miss-rate% 4467778 ± 7% +23.8% 5532316 ± 3% perf-stat.i.iTLB-loads 1.056e+10 -3.2% 1.022e+10 perf-stat.i.instructions 0.82 -2.2% 0.81 perf-stat.i.ipc 0.36 -1.2% 0.35 perf-stat.i.metric.GHz 165.56 ± 5% +19.3% 197.48 ± 3% perf-stat.i.metric.K/sec 182.39 -4.1% 174.97 perf-stat.i.metric.M/sec 174277 ± 2% +19.7% 208636 ± 2% perf-stat.i.node-loads 226393 +19.8% 271319 ± 2% perf-stat.i.node-stores 0.15 +21.1% 0.18 perf-stat.overall.MPKI 1.54 +0.1 1.66 perf-stat.overall.branch-miss-rate% 2.34 ± 2% +0.2 2.58 perf-stat.overall.cache-miss-rate% 1.22 +2.1% 1.25 perf-stat.overall.cpi 8365 -15.6% 7056 perf-stat.overall.cycles-between-cache-misses 0.04 +0.0 0.05 perf-stat.overall.dTLB-load-miss-rate% 0.00 ± 6% +0.0 0.00 ± 7% perf-stat.overall.dTLB-store-miss-rate% 19.58 ± 4% -2.0 17.55 ± 5% perf-stat.overall.iTLB-load-miss-rate% 9759 ± 4% -10.6% 8728 ± 7% perf-stat.overall.instructions-per-iTLB-miss 0.82 -2.1% 0.80 perf-stat.overall.ipc 1.926e+09 -3.0% 1.868e+09 perf-stat.ps.branch-instructions 29616558 +4.8% 31047367 perf-stat.ps.branch-misses 1519046 +17.2% 1779751 perf-stat.ps.cache-misses 64920277 ± 3% +6.3% 69002916 perf-stat.ps.cache-references 78634 +76.0% 138383 perf-stat.ps.context-switches 1.27e+10 -1.2% 1.256e+10 perf-stat.ps.cpu-cycles 1446 +69.6% 2452 perf-stat.ps.cpu-migrations 2.765e+09 -4.2% 2.648e+09 perf-stat.ps.dTLB-loads 52223 ± 6% +28.2% 66941 ± 7% perf-stat.ps.dTLB-store-misses 1.708e+09 -5.4% 1.616e+09 perf-stat.ps.dTLB-stores 4397091 ± 7% +23.8% 5444632 ± 3% perf-stat.ps.iTLB-loads 1.04e+10 -3.2% 1.006e+10 perf-stat.ps.instructions 171521 ± 2% +19.8% 205431 ± 2% perf-stat.ps.node-loads 222827 +19.9% 267072 ± 2% perf-stat.ps.node-stores 6.572e+11 -3.2% 6.361e+11 perf-stat.total.instructions 0.00 ±223% +900.0% 0.00 ± 44% perf-sched.sch_delay.avg.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats 0.00 ±223% +1100.0% 0.00 perf-sched.sch_delay.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_create 0.71 ± 42% +262.2% 2.57 ± 28% perf-sched.sch_delay.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_truncate 0.00 ±223% +1000.0% 0.00 ± 37% perf-sched.sch_delay.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_remove 0.00 ±141% +450.0% 0.00 ± 20% perf-sched.sch_delay.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_rename 0.00 ±223% +900.0% 0.00 ± 28% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.vfs_unlink.do_unlinkat.__x64_sys_unlink 0.00 ±223% +900.0% 0.00 ± 44% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_free_eofblocks.xfs_release 0.00 ±141% +750.0% 0.00 ± 62% perf-sched.sch_delay.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_iget_cache_hit.xfs_iget 0.00 ±223% +900.0% 0.00 ± 44% perf-sched.sch_delay.avg.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 0.00 ±223% +1100.0% 0.00 ± 28% perf-sched.sch_delay.avg.ms.__cond_resched.dput.__fsnotify_parent.notify_change.chmod_common 0.00 ±223% +1400.0% 0.00 ± 50% perf-sched.sch_delay.avg.ms.__cond_resched.dput.vfs_rename.do_renameat2.__x64_sys_rename 0.00 ±223% +800.0% 0.00 ± 33% perf-sched.sch_delay.avg.ms.__cond_resched.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.vfs_write 0.00 ±223% +1000.0% 0.00 ± 37% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_dialloc_ag 0.00 ± 33% +1736.8% 0.06 ±164% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_difree_finobt 0.17 ± 71% +762.5% 1.44 ± 52% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_ifree.xfs_inactive 0.33 ± 42% +319.5% 1.37 ± 21% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_truncate.xfs_inactive 0.00 ±141% +500.0% 0.00 ± 70% perf-sched.sch_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_ichange.xfs_setattr_nonsize 0.00 ±141% +650.0% 0.00 ± 50% perf-sched.sch_delay.avg.ms.__cond_resched.mnt_want_write.do_renameat2.__x64_sys_rename.do_syscall_64 0.00 ±152% +466.7% 0.00 ± 51% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.__mutex_lock.constprop.0.dnotify_flush 0.39 ± 4% +28.8% 0.50 ± 3% perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xlog_cil_commit 0.03 ± 4% -36.2% 0.02 perf-sched.sch_delay.avg.ms.schedule_timeout.___down_common.__down.down 0.00 ± 82% +175.0% 0.00 ± 12% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__flush_work.isra.0 0.00 +116.7% 0.00 ± 17% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra 0.03 ± 10% +128.2% 0.07 ± 4% perf-sched.sch_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.01 ± 23% +91.7% 0.01 ± 17% perf-sched.sch_delay.max.ms.__cond_resched.__alloc_pages.folio_alloc.__filemap_get_folio.iomap_write_begin 0.01 ± 31% +931.4% 0.09 ±150% perf-sched.sch_delay.max.ms.__cond_resched.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.01 ± 35% +142.2% 0.02 ± 30% perf-sched.sch_delay.max.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_createname 0.00 ±145% +610.0% 0.01 ± 61% perf-sched.sch_delay.max.ms.__cond_resched.cgroup_rstat_flush_locked.cgroup_rstat_flush.do_flush_stats.mem_cgroup_wb_stats 0.00 ±104% +250.0% 0.01 ± 36% perf-sched.sch_delay.max.ms.__cond_resched.dentry_kill.dput.do_renameat2.__x64_sys_rename 0.00 ± 72% +428.6% 0.01 ± 26% perf-sched.sch_delay.max.ms.__cond_resched.down_read.xfs_can_free_eofblocks.xfs_release.__fput 0.01 ± 22% +244.6% 0.03 ± 21% perf-sched.sch_delay.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_create 0.02 ± 68% -75.7% 0.00 ± 52% perf-sched.sch_delay.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree 0.01 ± 66% +135.6% 0.03 ± 42% perf-sched.sch_delay.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_remove 0.00 ± 21% +1092.6% 0.05 ±144% perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_create.xfs_generic_create 0.01 ± 67% +1102.9% 0.14 ±104% perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_iget_cache_hit.xfs_iget 0.00 ± 65% +467.9% 0.03 ± 62% perf-sched.sch_delay.max.ms.__cond_resched.down_write.xfs_ilock.xfs_lock_two_inodes.xfs_trans_alloc_dir 0.01 ± 52% +129.4% 0.01 ± 36% perf-sched.sch_delay.max.ms.__cond_resched.dput.__fsnotify_parent.notify_change.chmod_common 0.01 ± 24% +198.0% 0.03 ± 34% perf-sched.sch_delay.max.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open 0.01 ± 27% +362.5% 0.04 ± 64% perf-sched.sch_delay.max.ms.__cond_resched.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.vfs_write 0.01 ± 24% +143.6% 0.02 ± 27% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 0.01 ± 26% +643.4% 0.07 ±117% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.getname_flags.part.0 0.01 ± 23% +639.5% 0.05 ± 65% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_dialloc_ag 0.01 +11161.1% 0.68 ±213% perf-sched.sch_delay.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_difree_finobt 0.01 ± 43% +250.0% 0.03 ± 54% perf-sched.sch_delay.max.ms.__cond_resched.mnt_want_write.do_unlinkat.__x64_sys_unlink.do_syscall_64 0.01 ± 41% +245.5% 0.02 ± 44% perf-sched.sch_delay.max.ms.__cond_resched.mutex_lock.dnotify_flush.filp_flush.__x64_sys_close 0.00 ±145% +480.0% 0.00 ± 32% perf-sched.sch_delay.max.ms.__cond_resched.slab_pre_alloc_hook.constprop.0.kmem_cache_alloc_lru 0.63 ± 5% -90.8% 0.06 ±200% perf-sched.sch_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.01 ± 23% +1292.9% 0.10 ±140% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt 0.07 ± 34% -59.8% 0.03 ± 28% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.exit_to_user_mode_prepare.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi 0.01 ±110% +324.4% 0.03 ± 21% perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__flush_work.isra.0 0.01 ± 19% +624.3% 0.08 ±150% perf-sched.sch_delay.max.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra 3.96 ± 2% -41.6% 2.32 perf-sched.total_wait_and_delay.average.ms 207803 +79.8% 373538 perf-sched.total_wait_and_delay.count.ms 3.94 ± 2% -41.8% 2.29 perf-sched.total_wait_time.average.ms 0.13 -100.0% 0.00 perf-sched.wait_and_delay.avg.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 1.12 ±223% +1462.1% 17.43 ± 74% perf-sched.wait_and_delay.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_ifree.xfs_inactive 149.63 ± 2% -47.0% 79.32 ± 4% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 13.77 ± 13% -21.5% 10.81 ± 4% perf-sched.wait_and_delay.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xlog_cil_commit 1.05 ± 5% -56.6% 0.46 perf-sched.wait_and_delay.avg.ms.schedule_timeout.___down_common.__down.down 60.07 ± 3% -40.0% 36.03 ± 4% perf-sched.wait_and_delay.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 13.00 +19.3% 15.51 ± 2% perf-sched.wait_and_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 7293 ± 4% -100.0% 0.00 perf-sched.wait_and_delay.count.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 59.17 ± 28% -98.0% 1.17 ±125% perf-sched.wait_and_delay.count.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 100.33 ± 2% +89.0% 189.67 perf-sched.wait_and_delay.count.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 807.17 ± 3% +72.0% 1388 ± 2% perf-sched.wait_and_delay.count.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xlog_cil_commit 85849 ± 4% +198.3% 256077 perf-sched.wait_and_delay.count.schedule_timeout.___down_common.__down.down 5487 ± 2% +70.7% 9364 ± 3% perf-sched.wait_and_delay.count.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 19743 -13.5% 17076 perf-sched.wait_and_delay.count.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 6.26 ± 54% -100.0% 0.00 perf-sched.wait_and_delay.max.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 714.57 ± 59% -80.5% 139.66 ±207% perf-sched.wait_and_delay.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.16 ±223% +534.8% 1.05 ± 49% perf-sched.wait_time.avg.ms.__cond_resched.__alloc_pages_bulk.xfs_buf_alloc_pages.xfs_buf_find_insert.constprop 0.05 ± 21% -33.1% 0.04 ± 34% perf-sched.wait_time.avg.ms.__cond_resched.__kmem_cache_alloc_node.__kmalloc.kmem_alloc.xfs_dir_createname 0.05 ± 47% -76.5% 0.01 ± 20% perf-sched.wait_time.avg.ms.__cond_resched.begin_current_label_crit_section.apparmor_path_rename.part.0 0.26 ± 32% -97.0% 0.01 ± 34% perf-sched.wait_time.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree 0.93 ± 31% +202.3% 2.81 ± 14% perf-sched.wait_time.avg.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_truncate 0.07 ± 47% -53.5% 0.03 ± 41% perf-sched.wait_time.avg.ms.__cond_resched.down_write.xfs_ilock.xfs_create.xfs_generic_create 0.05 ± 5% +83.2% 0.09 ± 28% perf-sched.wait_time.avg.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 0.13 -60.8% 0.05 ± 22% perf-sched.wait_time.avg.ms.__cond_resched.dput.do_unlinkat.__x64_sys_unlink.do_syscall_64 1.86 ±117% +759.0% 15.99 ± 79% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_inactive_ifree.xfs_inactive 0.09 ± 86% -68.7% 0.03 ± 42% perf-sched.wait_time.avg.ms.__cond_resched.kmem_cache_alloc.xfs_trans_alloc.xfs_trans_alloc_icreate.xfs_create 0.03 ± 33% -100.0% 0.00 perf-sched.wait_time.avg.ms.__cond_resched.mutex_lock.perf_event_ctx_lock_nested.constprop.0 149.62 ± 2% -47.0% 79.32 ± 4% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.do_poll.constprop.0.do_sys_poll 13.38 ± 13% -22.9% 10.31 ± 4% perf-sched.wait_time.avg.ms.schedule_preempt_disabled.rwsem_down_read_slowpath.down_read.xlog_cil_commit 1.02 ± 5% -57.3% 0.43 perf-sched.wait_time.avg.ms.schedule_timeout.___down_common.__down.down 0.07 ± 15% +297.2% 0.26 ± 4% perf-sched.wait_time.avg.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra 60.04 ± 3% -40.1% 35.96 ± 4% perf-sched.wait_time.avg.ms.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 12.96 +19.3% 15.46 ± 2% perf-sched.wait_time.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 0.96 ± 44% +229.1% 3.15 ± 45% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages.folio_alloc.__filemap_get_folio.iomap_write_begin 0.16 ±223% +611.0% 1.17 ± 58% perf-sched.wait_time.max.ms.__cond_resched.__alloc_pages_bulk.xfs_buf_alloc_pages.xfs_buf_find_insert.constprop 0.13 ± 29% -85.0% 0.02 ± 24% perf-sched.wait_time.max.ms.__cond_resched.begin_current_label_crit_section.apparmor_path_rename.part.0 1.51 ± 54% -99.2% 0.01 ± 54% perf-sched.wait_time.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree 0.77 ± 48% +202.7% 2.33 ± 53% perf-sched.wait_time.max.ms.__cond_resched.down_read.xlog_cil_commit.__xfs_trans_commit.xfs_remove 0.32 ± 21% +434.9% 1.73 ± 65% perf-sched.wait_time.max.ms.__cond_resched.down_write.vfs_unlink.do_unlinkat.__x64_sys_unlink 0.26 ± 44% +653.2% 1.92 ± 80% perf-sched.wait_time.max.ms.__cond_resched.down_write.xfs_ilock.xfs_free_eofblocks.xfs_release 0.80 ± 53% +377.0% 3.80 ±109% perf-sched.wait_time.max.ms.__cond_resched.down_write.xfs_ilock.xfs_iget_cache_hit.xfs_iget 0.52 ± 24% +708.7% 4.24 ± 67% perf-sched.wait_time.max.ms.__cond_resched.dput.__fput.__x64_sys_close.do_syscall_64 1.43 ± 78% +158.0% 3.68 ± 57% perf-sched.wait_time.max.ms.__cond_resched.dput.terminate_walk.path_openat.do_filp_open 0.61 ± 43% +210.5% 1.90 ± 49% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.alloc_empty_file.path_openat.do_filp_open 0.03 ±167% +3200.6% 0.95 ±102% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_allocbt_init_common.xfs_allocbt_init_cursor.xfs_alloc_ag_vextent_exact 0.67 ± 59% +790.2% 5.97 ±103% perf-sched.wait_time.max.ms.__cond_resched.kmem_cache_alloc.xfs_inobt_init_common.xfs_inobt_init_cursor.xfs_dialloc_ag 0.69 ± 86% +234.1% 2.31 ± 80% perf-sched.wait_time.max.ms.__cond_resched.mnt_want_write.do_unlinkat.__x64_sys_unlink.do_syscall_64 0.58 ± 5% +292.4% 2.27 ± 55% perf-sched.wait_time.max.ms.__cond_resched.mnt_want_write.open_last_lookups.path_openat.do_filp_open 0.05 ± 41% -100.0% 0.00 perf-sched.wait_time.max.ms.__cond_resched.mutex_lock.perf_event_ctx_lock_nested.constprop.0 714.27 ± 59% -80.4% 139.65 ±207% perf-sched.wait_time.max.ms.__cond_resched.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 0.69 ± 58% +265.0% 2.52 ± 46% perf-sched.wait_time.max.ms.schedule_preempt_disabled.rwsem_down_write_slowpath.down_write.xlog_cil_push_work 0.74 ±123% +639.9% 5.46 ± 45% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.__flush_work.isra.0 1.97 ± 48% +501.3% 11.82 ± 22% perf-sched.wait_time.max.ms.schedule_timeout.__wait_for_common.__flush_workqueue.xlog_cil_push_now.isra 4.90 ± 12% +42.2% 6.96 ± 13% perf-sched.wait_time.max.ms.xlog_wait_on_iclog.xlog_cil_push_work.process_one_work.worker_thread 20.19 -1.4 18.83 ± 2% perf-profile.calltrace.cycles-pp.intel_idle_ibrs.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 15.82 -1.1 14.68 perf-profile.calltrace.cycles-pp.open64 14.49 -1.0 13.47 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.open64 14.05 -1.0 13.08 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 7.65 -0.9 6.74 perf-profile.calltrace.cycles-pp.__close 13.25 -0.9 12.39 perf-profile.calltrace.cycles-pp.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 13.17 -0.8 12.33 perf-profile.calltrace.cycles-pp.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 6.30 -0.8 5.52 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__close 12.29 -0.7 11.55 ± 2% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe 12.15 -0.7 11.43 ± 2% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64 5.83 ± 2% -0.7 5.13 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 6.06 ± 2% -0.6 5.42 perf-profile.calltrace.cycles-pp.write 5.02 ± 2% -0.6 4.41 perf-profile.calltrace.cycles-pp.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 5.45 ± 3% -0.5 4.91 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write 5.26 ± 2% -0.5 4.74 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 13.67 -0.5 13.20 perf-profile.calltrace.cycles-pp.unlink 3.37 ± 4% -0.5 2.92 ± 2% perf-profile.calltrace.cycles-pp.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 4.44 ± 3% -0.4 4.02 ± 2% perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 4.50 ± 3% -0.4 4.08 ± 2% perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write 9.12 -0.4 8.75 perf-profile.calltrace.cycles-pp.open_last_lookups.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat 3.09 ± 2% -0.4 2.73 ± 2% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe 3.80 ± 3% -0.4 3.44 ± 2% perf-profile.calltrace.cycles-pp.xfs_file_buffered_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.51 ± 4% -0.3 2.18 ± 5% perf-profile.calltrace.cycles-pp.evict.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe 3.41 ± 3% -0.3 3.09 ± 3% perf-profile.calltrace.cycles-pp.iomap_file_buffered_write.xfs_file_buffered_write.vfs_write.ksys_write.do_syscall_64 2.56 ± 4% -0.3 2.25 ± 2% perf-profile.calltrace.cycles-pp.xfs_release.__fput.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.50 ± 3% -0.3 2.20 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.92 ± 3% -0.3 1.65 ± 2% perf-profile.calltrace.cycles-pp.rename 1.81 ± 3% -0.3 1.56 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.rename 1.97 ± 3% -0.3 1.72 ± 5% perf-profile.calltrace.cycles-pp.truncate_inode_pages_range.evict.do_unlinkat.__x64_sys_unlink.do_syscall_64 1.78 ± 3% -0.3 1.52 ± 2% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.rename 2.10 ± 5% -0.2 1.86 ± 3% perf-profile.calltrace.cycles-pp.xfs_free_eofblocks.xfs_release.__fput.__x64_sys_close.do_syscall_64 2.41 ± 2% -0.2 2.18 ± 3% perf-profile.calltrace.cycles-pp.iomap_write_iter.iomap_file_buffered_write.xfs_file_buffered_write.vfs_write.ksys_write 1.52 ± 4% -0.2 1.31 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_rename.do_syscall_64.entry_SYSCALL_64_after_hwframe.rename 1.44 ± 4% -0.2 1.24 ± 4% perf-profile.calltrace.cycles-pp.do_renameat2.__x64_sys_rename.do_syscall_64.entry_SYSCALL_64_after_hwframe.rename 3.75 -0.2 3.56 ± 3% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_create.xfs_generic_create.lookup_open.open_last_lookups 2.72 ± 2% -0.2 2.53 ± 2% perf-profile.calltrace.cycles-pp.xfs_inactive_truncate.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread 0.62 ± 3% -0.2 0.44 ± 44% perf-profile.calltrace.cycles-pp.notify_change.chmod_common.__x64_sys_chmod.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.64 ± 5% -0.2 0.47 ± 44% perf-profile.calltrace.cycles-pp.__filename_parentat.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.05 ± 4% -0.2 0.88 ± 6% perf-profile.calltrace.cycles-pp.vfs_rename.do_renameat2.__x64_sys_rename.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.46 ± 4% -0.2 1.30 ± 4% perf-profile.calltrace.cycles-pp.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe 2.30 ± 2% -0.2 2.14 ± 3% perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_create.xfs_generic_create.lookup_open 1.04 ± 4% -0.2 0.89 ± 6% perf-profile.calltrace.cycles-pp.__folio_batch_release.truncate_inode_pages_range.evict.do_unlinkat.__x64_sys_unlink 1.34 ± 4% -0.1 1.20 ± 4% perf-profile.calltrace.cycles-pp.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.99 ± 2% -0.1 0.85 ± 4% perf-profile.calltrace.cycles-pp.alloc_empty_file.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat 1.30 ± 2% -0.1 1.17 ± 5% perf-profile.calltrace.cycles-pp.do_open.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat 1.21 ± 2% -0.1 1.08 ± 3% perf-profile.calltrace.cycles-pp.chmod 1.11 ± 2% -0.1 0.98 ± 3% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.chmod 1.08 ± 2% -0.1 0.95 ± 3% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.chmod 1.38 ± 3% -0.1 1.26 ± 4% perf-profile.calltrace.cycles-pp.dnotify_handler 1.29 ± 4% -0.1 1.16 ± 4% perf-profile.calltrace.cycles-pp.fcntl_dirnotify.do_fcntl.__x64_sys_fcntl.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.09 ± 4% -0.1 0.96 ± 6% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_create.xfs_generic_create 0.96 ± 3% -0.1 0.84 ± 3% perf-profile.calltrace.cycles-pp.do_dentry_open.do_open.path_openat.do_filp_open.do_sys_openat2 0.70 ± 5% -0.1 0.58 ± 7% perf-profile.calltrace.cycles-pp.xfs_vn_rename.vfs_rename.do_renameat2.__x64_sys_rename.do_syscall_64 0.68 ± 4% -0.1 0.56 ± 7% perf-profile.calltrace.cycles-pp.xfs_rename.xfs_vn_rename.vfs_rename.do_renameat2.__x64_sys_rename 0.82 ± 2% -0.1 0.70 perf-profile.calltrace.cycles-pp.__x64_sys_chmod.do_syscall_64.entry_SYSCALL_64_after_hwframe.chmod 2.00 ± 5% -0.1 1.88 perf-profile.calltrace.cycles-pp.xfs_dialloc_ag.xfs_dialloc.xfs_create.xfs_generic_create.lookup_open 1.32 ± 4% -0.1 1.20 ± 3% perf-profile.calltrace.cycles-pp.filp_flush.__x64_sys_close.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 0.76 ± 3% -0.1 0.66 ± 7% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.open64 1.05 ± 5% -0.1 0.96 ± 6% perf-profile.calltrace.cycles-pp.xfs_init_new_inode.xfs_create.xfs_generic_create.lookup_open.open_last_lookups 0.80 ± 4% -0.1 0.70 ± 5% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.__close 0.70 ± 5% -0.1 0.61 ± 7% perf-profile.calltrace.cycles-pp.syscall_return_via_sysret 0.66 ± 3% -0.1 0.57 ± 2% perf-profile.calltrace.cycles-pp.chmod_common.__x64_sys_chmod.do_syscall_64.entry_SYSCALL_64_after_hwframe.chmod 0.89 ± 6% -0.1 0.80 ± 4% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_free_eofblocks.xfs_release.__fput.__x64_sys_close 1.39 ± 2% -0.1 1.30 ± 3% perf-profile.calltrace.cycles-pp.xfs_itruncate_extents_flags.xfs_inactive_truncate.xfs_inactive.xfs_inodegc_worker.process_one_work 0.74 ± 6% -0.1 0.65 ± 5% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__close 0.72 ± 3% -0.1 0.64 ± 2% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.open64 0.95 ± 7% -0.1 0.86 ± 3% perf-profile.calltrace.cycles-pp.iomap_iter.iomap_file_buffered_write.xfs_file_buffered_write.vfs_write.ksys_write 0.84 ± 5% -0.1 0.77 ± 5% perf-profile.calltrace.cycles-pp.xfs_trans_alloc_dir.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat 0.98 -0.1 0.92 ± 2% perf-profile.calltrace.cycles-pp.xfs_defer_finish.xfs_itruncate_extents_flags.xfs_inactive_truncate.xfs_inactive.xfs_inodegc_worker 0.80 ± 4% -0.1 0.74 ± 3% perf-profile.calltrace.cycles-pp.read 0.95 -0.1 0.89 ± 2% perf-profile.calltrace.cycles-pp.xfs_defer_trans_roll.xfs_defer_finish.xfs_itruncate_extents_flags.xfs_inactive_truncate.xfs_inactive 0.64 ± 4% -0.0 0.59 ± 3% perf-profile.calltrace.cycles-pp.syscall_exit_to_user_mode.do_syscall_64.entry_SYSCALL_64_after_hwframe.unlink 1.29 ± 3% +0.1 1.44 ± 4% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_remove.xfs_vn_unlink 0.60 ± 9% +0.2 0.76 ± 6% perf-profile.calltrace.cycles-pp.xlog_cil_alloc_shadow_bufs.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive 1.12 ± 4% +0.2 1.31 ± 3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_format_items.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_remove 0.64 ± 3% +0.2 0.83 ± 6% perf-profile.calltrace.cycles-pp.xfs_trans_run_precommits.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker 2.47 ± 3% +0.2 2.71 ± 2% perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink 0.55 ± 5% +0.3 0.84 ± 4% perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_iunlink.xfs_remove.xfs_vn_unlink.vfs_unlink 3.47 ± 2% +0.3 3.78 ± 2% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat 0.27 ±100% +0.3 0.58 ± 7% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair 0.77 ± 6% +0.3 1.09 ± 3% perf-profile.calltrace.cycles-pp.xfs_iunlink.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat 0.27 ±100% +0.3 0.60 ± 7% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.newidle_balance.pick_next_task_fair.__schedule 6.70 +0.4 7.07 perf-profile.calltrace.cycles-pp.vfs_unlink.do_unlinkat.__x64_sys_unlink.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.63 ± 2% +0.4 2.02 ± 3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive 1.49 ± 2% +0.4 1.88 ± 3% perf-profile.calltrace.cycles-pp.xlog_cil_insert_format_items.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree 5.98 +0.5 6.45 perf-profile.calltrace.cycles-pp.xfs_remove.xfs_vn_unlink.vfs_unlink.do_unlinkat.__x64_sys_unlink 6.03 +0.5 6.50 perf-profile.calltrace.cycles-pp.xfs_vn_unlink.vfs_unlink.do_unlinkat.__x64_sys_unlink.do_syscall_64 0.09 ±223% +0.5 0.60 ± 8% perf-profile.calltrace.cycles-pp.xfs_buf_item_size.xlog_cil_alloc_shadow_bufs.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree 0.95 ± 4% +0.5 1.48 ± 2% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.27 ±100% +0.5 0.80 ± 4% perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_remove.xfs_vn_unlink 1.91 ± 12% +0.6 2.49 ± 3% perf-profile.calltrace.cycles-pp.xfs_buf_item_format.xlog_cil_insert_format_items.xlog_cil_insert_items.xlog_cil_commit.__xfs_trans_commit 2.91 ± 4% +0.6 3.51 ± 2% perf-profile.calltrace.cycles-pp.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker 0.64 ± 4% +0.6 1.24 ± 2% perf-profile.calltrace.cycles-pp.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_difree 0.00 +0.6 0.61 ± 8% perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_create 0.75 ± 3% +0.6 1.38 ± 4% perf-profile.calltrace.cycles-pp.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi.xfs_difree.xfs_ifree 0.79 ± 2% +0.6 1.42 ± 3% perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_ialloc_read_agi.xfs_difree.xfs_ifree.xfs_inactive_ifree 1.63 ± 3% +0.6 2.26 ± 3% perf-profile.calltrace.cycles-pp.xfs_buf_item_format_segment.xfs_buf_item_format.xlog_cil_insert_format_items.xlog_cil_insert_items.xlog_cil_commit 0.81 ± 3% +0.6 1.46 ± 3% perf-profile.calltrace.cycles-pp.xfs_ialloc_read_agi.xfs_difree.xfs_ifree.xfs_inactive_ifree.xfs_inactive 0.72 ± 9% +0.6 1.37 ± 4% perf-profile.calltrace.cycles-pp.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 0.00 +0.6 0.65 ± 7% perf-profile.calltrace.cycles-pp.xfs_read_agi.xfs_ialloc_read_agi.xfs_dialloc.xfs_create.xfs_generic_create 0.00 +0.7 0.66 ± 3% perf-profile.calltrace.cycles-pp.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink 0.00 +0.7 0.67 ± 7% perf-profile.calltrace.cycles-pp.xfs_ialloc_read_agi.xfs_dialloc.xfs_create.xfs_generic_create.lookup_open 3.40 ± 3% +0.7 4.08 ± 2% perf-profile.calltrace.cycles-pp.xfs_ifree.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work 0.00 +0.7 0.68 ± 4% perf-profile.calltrace.cycles-pp.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_iunlink.xfs_remove 3.22 ± 2% +0.8 4.00 ± 2% perf-profile.calltrace.cycles-pp.xlog_cil_commit.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker 0.00 +0.8 0.79 ± 3% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.___down_common.__down 0.00 +0.8 0.84 ± 3% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.___down_common.__down.down 0.62 ± 3% +0.9 1.49 ± 19% perf-profile.calltrace.cycles-pp.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi.xfs_ialloc_read_agi 0.00 +0.9 0.87 ± 4% perf-profile.calltrace.cycles-pp.schedule_timeout.___down_common.__down.down.xfs_buf_lock 0.00 +0.9 0.92 ± 3% perf-profile.calltrace.cycles-pp.___down_common.__down.down.xfs_buf_lock.xfs_buf_find_lock 0.00 +0.9 0.93 ± 3% perf-profile.calltrace.cycles-pp.__down.down.xfs_buf_lock.xfs_buf_find_lock.xfs_buf_lookup 0.00 +1.0 0.96 ± 3% perf-profile.calltrace.cycles-pp.down.xfs_buf_lock.xfs_buf_find_lock.xfs_buf_lookup.xfs_buf_get_map 0.00 +1.0 0.98 ± 3% perf-profile.calltrace.cycles-pp.xfs_buf_lock.xfs_buf_find_lock.xfs_buf_lookup.xfs_buf_get_map.xfs_buf_read_map 4.06 +1.0 5.05 ± 2% perf-profile.calltrace.cycles-pp.__xfs_trans_commit.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work 0.08 ±223% +1.0 1.09 ± 19% perf-profile.calltrace.cycles-pp.xfs_buf_find_lock.xfs_buf_lookup.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map 0.35 ± 70% +1.3 1.65 ± 3% perf-profile.calltrace.cycles-pp.xfs_buf_lookup.xfs_buf_get_map.xfs_buf_read_map.xfs_trans_read_buf_map.xfs_read_agi 10.80 +1.5 12.28 ± 2% perf-profile.calltrace.cycles-pp.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread.kthread 11.01 +1.5 12.52 ± 2% perf-profile.calltrace.cycles-pp.xfs_inodegc_worker.process_one_work.worker_thread.kthread.ret_from_fork 12.13 +1.5 13.65 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 12.13 +1.5 13.65 ± 2% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 12.13 +1.5 13.65 ± 2% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 11.60 +1.6 13.22 ± 2% perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 11.76 +1.6 13.39 ± 2% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 8.02 +1.7 9.68 ± 2% perf-profile.calltrace.cycles-pp.xfs_inactive_ifree.xfs_inactive.xfs_inodegc_worker.process_one_work.worker_thread 3.17 ± 2% +1.7 4.85 ± 2% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 2.77 ± 3% +1.9 4.63 ± 9% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.intel_idle_irq.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 28.56 +2.3 30.91 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 29.00 +2.4 31.40 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 28.83 +2.4 31.26 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry 29.64 +2.8 32.41 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 29.65 +2.8 32.44 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 29.66 +2.8 32.44 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify 30.26 +2.8 33.04 ± 2% perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify 50.13 -3.7 46.48 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe 47.98 -3.4 44.59 perf-profile.children.cycles-pp.do_syscall_64 20.49 -1.4 19.10 ± 2% perf-profile.children.cycles-pp.intel_idle_ibrs 15.87 -1.1 14.73 perf-profile.children.cycles-pp.open64 15.48 -1.0 14.52 perf-profile.children.cycles-pp.do_sys_openat2 7.72 -0.9 6.80 perf-profile.children.cycles-pp.__close 13.27 -0.9 12.40 perf-profile.children.cycles-pp.__x64_sys_openat 14.52 -0.9 13.66 perf-profile.children.cycles-pp.do_filp_open 14.40 -0.9 13.54 perf-profile.children.cycles-pp.path_openat 6.09 ± 2% -0.6 5.45 perf-profile.children.cycles-pp.write 5.02 ± 2% -0.6 4.41 perf-profile.children.cycles-pp.__x64_sys_close 4.76 ± 2% -0.5 4.22 ± 2% perf-profile.children.cycles-pp.syscall_exit_to_user_mode 13.71 -0.5 13.24 perf-profile.children.cycles-pp.unlink 11.12 -0.5 10.66 perf-profile.children.cycles-pp.open_last_lookups 3.39 ± 4% -0.5 2.93 ± 2% perf-profile.children.cycles-pp.__fput 4.45 ± 3% -0.4 4.04 ± 2% perf-profile.children.cycles-pp.vfs_write 4.50 ± 3% -0.4 4.10 ± 2% perf-profile.children.cycles-pp.ksys_write 3.40 ± 2% -0.4 3.00 ± 3% perf-profile.children.cycles-pp.syscall_return_via_sysret 3.80 ± 3% -0.4 3.44 ± 2% perf-profile.children.cycles-pp.xfs_file_buffered_write 2.84 ± 3% -0.4 2.48 ± 4% perf-profile.children.cycles-pp.xfs_trans_alloc 2.52 ± 4% -0.3 2.18 ± 5% perf-profile.children.cycles-pp.evict 3.41 ± 3% -0.3 3.09 ± 3% perf-profile.children.cycles-pp.iomap_file_buffered_write 2.57 ± 4% -0.3 2.26 ± 2% perf-profile.children.cycles-pp.xfs_release 1.93 ± 3% -0.3 1.65 ± 2% perf-profile.children.cycles-pp.rename 2.23 ± 4% -0.3 1.96 ± 3% perf-profile.children.cycles-pp.xfs_trans_reserve 1.99 ± 4% -0.3 1.72 ± 4% perf-profile.children.cycles-pp.xfs_log_reserve 1.98 ± 3% -0.2 1.73 ± 5% perf-profile.children.cycles-pp.truncate_inode_pages_range 2.11 ± 5% -0.2 1.86 ± 3% perf-profile.children.cycles-pp.xfs_free_eofblocks 2.31 ± 2% -0.2 2.08 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc 2.41 ± 2% -0.2 2.18 ± 3% perf-profile.children.cycles-pp.iomap_write_iter 1.84 ± 2% -0.2 1.62 ± 3% perf-profile.children.cycles-pp.__entry_text_start 1.52 ± 4% -0.2 1.31 ± 4% perf-profile.children.cycles-pp.__x64_sys_rename 1.44 ± 4% -0.2 1.24 ± 3% perf-profile.children.cycles-pp.do_renameat2 2.72 ± 2% -0.2 2.53 ± 2% perf-profile.children.cycles-pp.xfs_inactive_truncate 0.63 ± 12% -0.2 0.46 ± 7% perf-profile.children.cycles-pp.rcu_do_batch 0.68 ± 13% -0.2 0.50 ± 6% perf-profile.children.cycles-pp.rcu_core 1.05 ± 4% -0.2 0.88 ± 5% perf-profile.children.cycles-pp.vfs_rename 0.26 ± 13% -0.2 0.09 ± 11% perf-profile.children.cycles-pp.run_ksoftirqd 1.34 ± 4% -0.2 1.18 ± 2% perf-profile.children.cycles-pp.fsnotify 1.46 ± 4% -0.2 1.30 ± 4% perf-profile.children.cycles-pp.__x64_sys_fcntl 1.34 ± 4% -0.2 1.18 ± 3% perf-profile.children.cycles-pp.xfs_inode_item_format 1.04 ± 4% -0.2 0.89 ± 6% perf-profile.children.cycles-pp.__folio_batch_release 2.44 ± 5% -0.2 2.29 perf-profile.children.cycles-pp.xfs_dialloc_ag 1.07 ± 2% -0.2 0.92 ± 3% perf-profile.children.cycles-pp.alloc_empty_file 1.40 ± 2% -0.2 1.25 ± 5% perf-profile.children.cycles-pp.do_open 0.92 ± 10% -0.1 0.78 ± 7% perf-profile.children.cycles-pp.__do_softirq 1.34 ± 4% -0.1 1.20 ± 4% perf-profile.children.cycles-pp.do_fcntl 0.98 ± 6% -0.1 0.84 ± 5% perf-profile.children.cycles-pp.kmem_cache_free 1.03 ± 3% -0.1 0.90 ± 3% perf-profile.children.cycles-pp.do_dentry_open 0.77 ± 6% -0.1 0.64 ± 5% perf-profile.children.cycles-pp.down_read 1.78 ± 2% -0.1 1.65 ± 2% perf-profile.children.cycles-pp.xfs_itruncate_extents_flags 1.02 -0.1 0.89 ± 8% perf-profile.children.cycles-pp.link_path_walk 1.21 ± 2% -0.1 1.08 ± 3% perf-profile.children.cycles-pp.chmod 1.30 ± 4% -0.1 1.17 ± 4% perf-profile.children.cycles-pp.fcntl_dirnotify 1.39 ± 3% -0.1 1.26 ± 4% perf-profile.children.cycles-pp.dnotify_handler 0.67 ± 5% -0.1 0.54 ± 5% perf-profile.children.cycles-pp.xfs_trans_alloc_icreate 0.70 ± 5% -0.1 0.58 ± 7% perf-profile.children.cycles-pp.xfs_vn_rename 0.68 ± 4% -0.1 0.56 ± 7% perf-profile.children.cycles-pp.xfs_rename 0.82 ± 2% -0.1 0.70 perf-profile.children.cycles-pp.__x64_sys_chmod 1.32 ± 4% -0.1 1.20 ± 3% perf-profile.children.cycles-pp.filp_flush 0.95 ± 5% -0.1 0.83 ± 3% perf-profile.children.cycles-pp.__fsnotify_parent 0.98 ± 2% -0.1 0.87 ± 4% perf-profile.children.cycles-pp.getname_flags 0.61 ± 6% -0.1 0.50 ± 10% perf-profile.children.cycles-pp.__cond_resched 0.34 ± 8% -0.1 0.23 ± 9% perf-profile.children.cycles-pp.smpboot_thread_fn 0.71 ± 4% -0.1 0.61 ± 2% perf-profile.children.cycles-pp.dnotify_handle_event 0.77 ± 7% -0.1 0.67 ± 6% perf-profile.children.cycles-pp.xlog_ticket_alloc 0.54 ± 6% -0.1 0.44 ± 3% perf-profile.children.cycles-pp.init_file 1.15 ± 3% -0.1 1.05 ± 4% perf-profile.children.cycles-pp.exit_to_user_mode_loop 0.83 ± 4% -0.1 0.73 ± 3% perf-profile.children.cycles-pp.send_to_group 0.78 ± 5% -0.1 0.68 ± 4% perf-profile.children.cycles-pp.__filename_parentat 0.62 ± 3% -0.1 0.52 ± 3% perf-profile.children.cycles-pp.notify_change 1.12 ± 3% -0.1 1.03 ± 3% perf-profile.children.cycles-pp.arch_do_signal_or_restart 0.66 ± 3% -0.1 0.57 perf-profile.children.cycles-pp.chmod_common 0.53 ± 5% -0.1 0.44 ± 2% perf-profile.children.cycles-pp.do_send_sig_info 0.61 ± 4% -0.1 0.53 ± 4% perf-profile.children.cycles-pp.send_sigio 0.95 ± 7% -0.1 0.87 ± 2% perf-profile.children.cycles-pp.iomap_iter 0.68 ± 7% -0.1 0.60 ± 5% perf-profile.children.cycles-pp.path_parentat 0.57 ± 4% -0.1 0.49 ± 4% perf-profile.children.cycles-pp.send_sigio_to_task 0.75 ± 2% -0.1 0.67 ± 6% perf-profile.children.cycles-pp.xfs_trans_log_inode 0.50 ± 4% -0.1 0.42 ± 10% perf-profile.children.cycles-pp.lru_add_drain_cpu 0.46 ± 6% -0.1 0.38 ± 3% perf-profile.children.cycles-pp.__send_signal_locked 0.57 ± 7% -0.1 0.49 ± 5% perf-profile.children.cycles-pp.release_pages 0.87 ± 8% -0.1 0.79 ± 3% perf-profile.children.cycles-pp.xfs_buffered_write_iomap_begin 0.40 ± 10% -0.1 0.32 ± 10% perf-profile.children.cycles-pp.xlog_space_left 0.57 ± 6% -0.1 0.50 ± 5% perf-profile.children.cycles-pp.filemap_dirty_folio 0.48 ± 3% -0.1 0.41 ± 11% perf-profile.children.cycles-pp.folio_batch_move_lru 0.55 ± 5% -0.1 0.48 ± 6% perf-profile.children.cycles-pp.fsnotify_add_mark_locked 0.44 ± 8% -0.1 0.36 ± 3% perf-profile.children.cycles-pp.security_file_alloc 1.30 ± 3% -0.1 1.23 ± 3% perf-profile.children.cycles-pp.exit_to_user_mode_prepare 0.50 ± 8% -0.1 0.44 ± 6% perf-profile.children.cycles-pp.dput 0.40 ± 8% -0.1 0.33 ± 10% perf-profile.children.cycles-pp.xlog_grant_push_threshold 0.66 ± 4% -0.1 0.59 ± 3% perf-profile.children.cycles-pp.xfs_log_ticket_ungrant 0.65 ± 6% -0.1 0.58 ± 5% perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn 0.81 ± 4% -0.1 0.74 ± 4% perf-profile.children.cycles-pp.read 0.32 ± 6% -0.1 0.26 ± 8% perf-profile.children.cycles-pp.security_file_free 0.74 ± 3% -0.1 0.68 perf-profile.children.cycles-pp.handle_signal 1.02 -0.1 0.95 ± 2% perf-profile.children.cycles-pp.xfs_defer_finish 0.60 ± 4% -0.1 0.53 ± 4% perf-profile.children.cycles-pp.xlog_grant_add_space 0.25 ± 15% -0.1 0.19 ± 18% perf-profile.children.cycles-pp.xlog_calc_unit_res 0.49 ± 4% -0.1 0.43 ± 5% perf-profile.children.cycles-pp.xfs_dir_removename 0.36 ± 8% -0.1 0.30 ± 10% perf-profile.children.cycles-pp.generic_permission 0.59 ± 3% -0.1 0.52 ± 5% perf-profile.children.cycles-pp.fsnotify_put_mark 0.31 ± 5% -0.1 0.25 ± 8% perf-profile.children.cycles-pp.apparmor_file_free_security 0.31 ± 9% -0.1 0.25 ± 6% perf-profile.children.cycles-pp.apparmor_file_alloc_security 0.58 ± 4% -0.1 0.52 ± 2% perf-profile.children.cycles-pp.x64_setup_rt_frame 0.34 ± 6% -0.1 0.29 ± 8% perf-profile.children.cycles-pp.xfs_setattr_nonsize 0.31 ± 11% -0.1 0.26 ± 3% perf-profile.children.cycles-pp.xfs_ilock 0.46 ± 8% -0.0 0.41 ± 4% perf-profile.children.cycles-pp.xfs_bmapi_reserve_delalloc 0.21 ± 10% -0.0 0.17 ± 11% perf-profile.children.cycles-pp.locked_inode_to_wb_and_lock_list 0.41 ± 3% -0.0 0.36 ± 6% perf-profile.children.cycles-pp.__irqentry_text_end 0.18 ± 10% -0.0 0.13 ± 12% perf-profile.children.cycles-pp.rcu_all_qs 0.98 ± 2% -0.0 0.93 ± 2% perf-profile.children.cycles-pp.xfs_defer_trans_roll 0.34 ± 9% -0.0 0.30 ± 4% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack 0.27 ± 7% -0.0 0.23 ± 7% perf-profile.children.cycles-pp.__sigqueue_alloc 0.29 ± 6% -0.0 0.26 ± 8% perf-profile.children.cycles-pp.xfs_trans_del_item 0.30 ± 6% -0.0 0.27 ± 5% perf-profile.children.cycles-pp.xfs_can_free_eofblocks 0.10 ± 11% -0.0 0.08 ± 20% perf-profile.children.cycles-pp.security_path_unlink 0.10 ± 11% -0.0 0.07 ± 23% perf-profile.children.cycles-pp.xfs_iext_lookup_extent_before 0.13 ± 9% -0.0 0.10 ± 4% perf-profile.children.cycles-pp.xfs_btree_get_leaf_keys 0.13 ± 7% -0.0 0.11 ± 8% perf-profile.children.cycles-pp.mem_cgroup_css_rstat_flush 0.06 ± 11% +0.0 0.09 ± 12% perf-profile.children.cycles-pp.__update_blocked_fair 0.04 ± 45% +0.0 0.08 ± 14% perf-profile.children.cycles-pp.sched_use_asym_prio 0.08 ± 17% +0.0 0.12 ± 8% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq 0.06 ± 17% +0.0 0.10 ± 13% perf-profile.children.cycles-pp.asm_sysvec_call_function_single 0.09 ± 11% +0.0 0.13 ± 14% perf-profile.children.cycles-pp.update_rq_clock_task 0.03 ±100% +0.0 0.07 ± 14% perf-profile.children.cycles-pp.kernel_fpu_begin_mask 0.05 ± 48% +0.0 0.10 ± 18% perf-profile.children.cycles-pp.xlog_write 0.06 ± 49% +0.0 0.11 ± 8% perf-profile.children.cycles-pp.xfs_iunlink_lookup 0.09 ± 14% +0.0 0.14 ± 11% perf-profile.children.cycles-pp.cpu_util 0.08 ± 20% +0.0 0.12 ± 7% perf-profile.children.cycles-pp.xfs_iunlink_log_inode 0.04 ± 72% +0.0 0.09 ± 17% perf-profile.children.cycles-pp.llist_add_batch 0.04 ± 71% +0.0 0.08 ± 20% perf-profile.children.cycles-pp.xfs_btree_dup_cursor 0.10 ± 16% +0.1 0.16 ± 8% perf-profile.children.cycles-pp.sched_clock 0.14 ± 6% +0.1 0.20 ± 12% perf-profile.children.cycles-pp.idle_cpu 0.02 ±142% +0.1 0.07 ± 16% perf-profile.children.cycles-pp.select_task_rq 0.08 ± 18% +0.1 0.13 ± 4% perf-profile.children.cycles-pp.update_curr 0.00 +0.1 0.05 ± 8% perf-profile.children.cycles-pp.balance_fair 0.01 ±223% +0.1 0.06 ± 7% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate 0.09 ± 20% +0.1 0.14 ± 6% perf-profile.children.cycles-pp.__switch_to_asm 0.12 ± 19% +0.1 0.18 ± 7% perf-profile.children.cycles-pp.native_sched_clock 0.02 ±141% +0.1 0.08 ± 16% perf-profile.children.cycles-pp.sysvec_call_function_single 0.31 ± 6% +0.1 0.37 ± 6% perf-profile.children.cycles-pp.xfs_next_bit 0.15 ± 16% +0.1 0.20 ± 6% perf-profile.children.cycles-pp.xfs_buf_item_init 0.03 ±100% +0.1 0.09 ± 18% perf-profile.children.cycles-pp.switch_fpu_return 0.05 ± 51% +0.1 0.11 ± 6% perf-profile.children.cycles-pp.__switch_to 0.09 ± 12% +0.1 0.15 ± 17% perf-profile.children.cycles-pp.prepare_task_switch 0.08 ± 10% +0.1 0.14 ± 12% perf-profile.children.cycles-pp.finish_task_switch 0.04 ± 71% +0.1 0.10 ± 15% perf-profile.children.cycles-pp.update_rq_clock 0.12 ± 15% +0.1 0.18 ± 9% perf-profile.children.cycles-pp.sched_clock_cpu 0.07 ± 24% +0.1 0.14 ± 11% perf-profile.children.cycles-pp.set_next_entity 0.04 ± 72% +0.1 0.11 ± 14% perf-profile.children.cycles-pp.__smp_call_single_queue 0.00 +0.1 0.07 ± 18% perf-profile.children.cycles-pp.place_entity 0.16 ± 14% +0.1 0.23 ± 8% perf-profile.children.cycles-pp.xfs_ialloc_ag_alloc 0.40 ± 6% +0.1 0.48 ± 4% perf-profile.children.cycles-pp.xfs_perag_get 0.08 ± 12% +0.1 0.16 ± 14% perf-profile.children.cycles-pp.ttwu_queue_wakelist 0.31 ± 11% +0.1 0.39 ± 4% perf-profile.children.cycles-pp.xfs_buf_offset 0.13 ± 12% +0.1 0.23 ± 10% perf-profile.children.cycles-pp.xlog_cil_push_work 0.18 ± 11% +0.1 0.29 ± 9% perf-profile.children.cycles-pp.sched_ttwu_pending 0.49 ± 12% +0.1 0.61 ± 8% perf-profile.children.cycles-pp._xfs_trans_bjoin 0.19 ± 10% +0.1 0.31 ± 5% perf-profile.children.cycles-pp.update_load_avg 0.20 ± 10% +0.1 0.33 ± 5% perf-profile.children.cycles-pp.enqueue_entity 0.21 ± 9% +0.1 0.36 ± 9% perf-profile.children.cycles-pp.flush_smp_call_function_queue 0.23 ± 12% +0.1 0.38 ± 5% perf-profile.children.cycles-pp.xfs_imap_to_bp 0.32 ± 18% +0.2 0.47 ± 9% perf-profile.children.cycles-pp.xfs_contig_bits 0.22 ± 13% +0.2 0.38 ± 7% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 0.20 ± 9% +0.2 0.36 ± 5% perf-profile.children.cycles-pp.dequeue_entity 0.24 ± 7% +0.2 0.40 ± 5% perf-profile.children.cycles-pp.enqueue_task_fair 0.25 ± 7% +0.2 0.42 ± 4% perf-profile.children.cycles-pp.activate_task 0.29 ± 13% +0.2 0.46 ± 6% perf-profile.children.cycles-pp.schedule_idle 1.00 ± 4% +0.2 1.18 ± 7% perf-profile.children.cycles-pp.xfs_buf_item_size 0.28 ± 8% +0.2 0.45 ± 4% perf-profile.children.cycles-pp.ttwu_do_activate 0.23 ± 8% +0.2 0.42 ± 6% perf-profile.children.cycles-pp.dequeue_task_fair 0.79 ± 4% +0.2 0.98 ± 9% perf-profile.children.cycles-pp.xfs_buf_item_size_segment 2.66 ± 3% +0.2 2.90 ± 2% perf-profile.children.cycles-pp.xfs_trans_run_precommits 0.39 ± 9% +0.3 0.65 ± 13% perf-profile.children.cycles-pp.xfs_iunlink_log_dinode 1.73 ± 3% +0.3 2.00 ± 4% perf-profile.children.cycles-pp.xlog_cil_alloc_shadow_bufs 0.44 ± 7% +0.3 0.72 ± 13% perf-profile.children.cycles-pp.xfs_iunlink_item_precommit 5.10 ± 2% +0.3 5.40 ± 2% perf-profile.children.cycles-pp.xlog_cil_insert_items 0.44 ± 9% +0.3 0.76 ± 5% perf-profile.children.cycles-pp.try_to_wake_up 0.75 ± 3% +0.3 1.08 ± 3% perf-profile.children.cycles-pp.memcpy_orig 0.77 ± 6% +0.3 1.09 ± 3% perf-profile.children.cycles-pp.xfs_iunlink 0.82 ± 5% +0.4 1.17 ± 5% perf-profile.children.cycles-pp.update_sg_lb_stats 6.70 +0.4 7.08 perf-profile.children.cycles-pp.vfs_unlink 0.38 ± 7% +0.4 0.77 ± 7% perf-profile.children.cycles-pp.up 0.43 ± 8% +0.4 0.83 ± 7% perf-profile.children.cycles-pp.xfs_buf_unlock 0.66 ± 7% +0.4 1.06 ± 4% perf-profile.children.cycles-pp.xfs_buf_item_release 0.94 ± 6% +0.4 1.34 ± 5% perf-profile.children.cycles-pp.update_sd_lb_stats 0.96 ± 5% +0.4 1.38 ± 5% perf-profile.children.cycles-pp.find_busiest_group 4.29 ± 3% +0.4 4.74 perf-profile.children.cycles-pp.xlog_cil_insert_format_items 5.99 +0.5 6.46 perf-profile.children.cycles-pp.xfs_remove 6.03 +0.5 6.50 perf-profile.children.cycles-pp.xfs_vn_unlink 1.18 ± 5% +0.5 1.68 ± 4% perf-profile.children.cycles-pp.load_balance 0.98 ± 4% +0.5 1.53 ± 2% perf-profile.children.cycles-pp.poll_idle 2.91 ± 4% +0.6 3.51 ± 2% perf-profile.children.cycles-pp.xfs_difree 1.21 ± 5% +0.6 1.81 ± 2% perf-profile.children.cycles-pp.newidle_balance 2.40 ± 4% +0.6 3.00 ± 2% perf-profile.children.cycles-pp.xfs_buf_item_format 2.19 ± 4% +0.6 2.82 ± 2% perf-profile.children.cycles-pp.xfs_buf_item_format_segment 10.46 +0.7 11.13 perf-profile.children.cycles-pp.xlog_cil_commit 3.41 ± 3% +0.7 4.08 ± 2% perf-profile.children.cycles-pp.xfs_ifree 1.32 ± 6% +0.7 2.04 ± 2% perf-profile.children.cycles-pp.pick_next_task_fair 3.93 ± 2% +0.8 4.76 ± 3% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 14.08 +0.8 14.93 perf-profile.children.cycles-pp.__xfs_trans_commit 1.37 +0.9 2.26 ± 3% perf-profile.children.cycles-pp.xfs_ialloc_read_agi 0.68 ± 8% +1.0 1.72 perf-profile.children.cycles-pp.schedule_timeout 0.86 ± 7% +1.1 1.94 ± 2% perf-profile.children.cycles-pp.down 0.67 ± 8% +1.1 1.76 ± 2% perf-profile.children.cycles-pp.___down_common 0.91 ± 7% +1.1 2.01 ± 2% perf-profile.children.cycles-pp.xfs_buf_lock 0.67 ± 8% +1.1 1.77 ± 2% perf-profile.children.cycles-pp.__down 1.02 ± 6% +1.1 2.14 ± 2% perf-profile.children.cycles-pp.xfs_buf_find_lock 1.52 ± 3% +1.1 2.66 ± 3% perf-profile.children.cycles-pp.xfs_buf_lookup 1.96 ± 2% +1.2 3.12 ± 3% perf-profile.children.cycles-pp.xfs_read_agi 1.70 ± 4% +1.2 2.87 ± 2% perf-profile.children.cycles-pp.schedule 2.29 ± 3% +1.2 3.51 ± 3% perf-profile.children.cycles-pp.xfs_buf_get_map 2.39 ± 2% +1.2 3.62 ± 3% perf-profile.children.cycles-pp.xfs_buf_read_map 2.00 ± 4% +1.3 3.28 ± 2% perf-profile.children.cycles-pp.__schedule 3.20 ± 2% +1.4 4.56 ± 3% perf-profile.children.cycles-pp.xfs_trans_read_buf_map 10.80 +1.5 12.28 ± 2% perf-profile.children.cycles-pp.xfs_inactive 12.13 +1.5 13.65 ± 2% perf-profile.children.cycles-pp.ret_from_fork_asm 12.13 +1.5 13.65 ± 2% perf-profile.children.cycles-pp.ret_from_fork 12.13 +1.5 13.65 ± 2% perf-profile.children.cycles-pp.kthread 11.01 +1.5 12.52 ± 2% perf-profile.children.cycles-pp.xfs_inodegc_worker 2.07 ± 5% +1.6 3.64 ± 6% perf-profile.children.cycles-pp.intel_idle_irq 11.60 +1.6 13.22 ± 2% perf-profile.children.cycles-pp.process_one_work 11.76 +1.6 13.39 ± 2% perf-profile.children.cycles-pp.worker_thread 8.02 +1.7 9.68 ± 2% perf-profile.children.cycles-pp.xfs_inactive_ifree 3.27 ± 2% +1.7 4.94 perf-profile.children.cycles-pp.intel_idle 29.12 +2.3 31.45 ± 2% perf-profile.children.cycles-pp.cpuidle_enter_state 29.12 +2.3 31.45 ± 2% perf-profile.children.cycles-pp.cpuidle_enter 29.59 +2.4 31.97 ± 2% perf-profile.children.cycles-pp.cpuidle_idle_call 30.25 +2.8 33.03 ± 2% perf-profile.children.cycles-pp.do_idle 29.66 +2.8 32.44 ± 2% perf-profile.children.cycles-pp.start_secondary 30.26 +2.8 33.04 ± 2% perf-profile.children.cycles-pp.secondary_startup_64_no_verify 30.26 +2.8 33.04 ± 2% perf-profile.children.cycles-pp.cpu_startup_entry 20.48 -1.4 19.10 ± 2% perf-profile.self.cycles-pp.intel_idle_ibrs 3.42 ± 2% -0.5 2.95 ± 2% perf-profile.self.cycles-pp.syscall_exit_to_user_mode 3.40 ± 2% -0.4 2.99 ± 3% perf-profile.self.cycles-pp.syscall_return_via_sysret 2.20 ± 3% -0.3 1.93 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe 1.66 ± 3% -0.2 1.47 ± 2% perf-profile.self.cycles-pp.__entry_text_start 2.76 ± 3% -0.2 2.57 ± 3% perf-profile.self.cycles-pp._raw_spin_lock 0.81 ± 7% -0.2 0.66 ± 6% perf-profile.self.cycles-pp.xlog_cil_insert_items 0.66 ± 5% -0.1 0.54 ± 5% perf-profile.self.cycles-pp.down_read 0.78 ± 5% -0.1 0.68 ± 5% perf-profile.self.cycles-pp.kmem_cache_free 1.14 -0.1 1.05 ± 4% perf-profile.self.cycles-pp.kmem_cache_alloc 0.44 ± 5% -0.1 0.36 ± 12% perf-profile.self.cycles-pp.link_path_walk 0.39 ± 10% -0.1 0.32 ± 11% perf-profile.self.cycles-pp.xlog_space_left 0.45 ± 8% -0.1 0.38 ± 8% perf-profile.self.cycles-pp.fsnotify 0.62 ± 5% -0.1 0.55 ± 4% perf-profile.self.cycles-pp.xfs_log_ticket_ungrant 0.59 ± 5% -0.1 0.52 ± 5% perf-profile.self.cycles-pp.xlog_grant_add_space 0.24 ± 15% -0.1 0.18 ± 17% perf-profile.self.cycles-pp.xlog_calc_unit_res 0.30 ± 5% -0.1 0.25 ± 8% perf-profile.self.cycles-pp.apparmor_file_free_security 0.30 ± 9% -0.1 0.24 ± 7% perf-profile.self.cycles-pp.apparmor_file_alloc_security 0.41 ± 3% -0.0 0.36 ± 6% perf-profile.self.cycles-pp.__irqentry_text_end 0.24 ± 11% -0.0 0.20 ± 9% perf-profile.self.cycles-pp._atomic_dec_and_lock 0.15 ± 8% -0.0 0.12 ± 7% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack 0.13 ± 8% -0.0 0.10 ± 15% perf-profile.self.cycles-pp.rcu_all_qs 0.12 ± 6% -0.0 0.10 ± 9% perf-profile.self.cycles-pp.mem_cgroup_css_rstat_flush 0.10 ± 22% -0.0 0.07 ± 18% perf-profile.self.cycles-pp.getname_flags 0.06 ± 13% +0.0 0.08 ± 11% perf-profile.self.cycles-pp.load_balance 0.08 ± 14% +0.0 0.11 ± 11% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq 0.10 ± 7% +0.0 0.13 ± 11% perf-profile.self.cycles-pp.update_sd_lb_stats 0.05 ± 47% +0.0 0.09 ± 10% perf-profile.self.cycles-pp.update_rq_clock_task 0.14 ± 5% +0.0 0.18 ± 11% perf-profile.self.cycles-pp.idle_cpu 0.09 ± 14% +0.0 0.14 ± 11% perf-profile.self.cycles-pp.cpu_util 0.04 ± 72% +0.0 0.09 ± 17% perf-profile.self.cycles-pp.llist_add_batch 0.08 ± 14% +0.1 0.13 ± 6% perf-profile.self.cycles-pp.update_load_avg 0.02 ±141% +0.1 0.07 ± 14% perf-profile.self.cycles-pp.do_idle 0.11 ± 18% +0.1 0.16 ± 5% perf-profile.self.cycles-pp.native_sched_clock 0.01 ±223% +0.1 0.06 ± 17% perf-profile.self.cycles-pp.dequeue_task_fair 0.01 ±223% +0.1 0.06 ± 7% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate 0.09 ± 20% +0.1 0.14 ± 6% perf-profile.self.cycles-pp.__switch_to_asm 0.26 ± 10% +0.1 0.32 ± 9% perf-profile.self.cycles-pp.xfs_perag_put 0.41 ± 3% +0.1 0.47 ± 3% perf-profile.self.cycles-pp.xlog_cil_alloc_shadow_bufs 0.13 ± 16% +0.1 0.19 ± 9% perf-profile.self.cycles-pp.xfs_buf_item_init 0.05 ± 53% +0.1 0.11 ± 8% perf-profile.self.cycles-pp.__switch_to 0.30 ± 6% +0.1 0.36 ± 6% perf-profile.self.cycles-pp.xfs_next_bit 0.26 ± 14% +0.1 0.32 ± 6% perf-profile.self.cycles-pp.xfs_buf_offset 0.24 ± 11% +0.1 0.30 ± 11% perf-profile.self.cycles-pp.xfs_perag_get 0.01 ±223% +0.1 0.08 ± 21% perf-profile.self.cycles-pp.pick_next_task_fair 0.39 ± 7% +0.1 0.48 ± 8% perf-profile.self.cycles-pp.xfs_buf_item_size_segment 0.71 ± 6% +0.1 0.80 ± 4% perf-profile.self.cycles-pp.xfs_buf_item_format_segment 0.09 ± 7% +0.1 0.20 ± 14% perf-profile.self.cycles-pp.__schedule 0.29 ± 17% +0.1 0.43 ± 8% perf-profile.self.cycles-pp.xfs_contig_bits 0.53 ± 7% +0.2 0.76 ± 5% perf-profile.self.cycles-pp.update_sg_lb_stats 0.73 ± 3% +0.3 1.04 ± 3% perf-profile.self.cycles-pp.memcpy_orig 0.96 ± 4% +0.5 1.51 ± 2% perf-profile.self.cycles-pp.poll_idle 1.84 ± 5% +1.5 3.34 ± 7% perf-profile.self.cycles-pp.intel_idle_irq 3.27 ± 2% +1.7 4.94 perf-profile.self.cycles-pp.intel_idle Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. -- 0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks 2023-09-22 10:00 ` kernel test robot @ 2023-09-25 11:07 ` Ingo Molnar 2023-09-25 16:45 ` Chen Yu 0 siblings, 1 reply; 30+ messages in thread From: Ingo Molnar @ 2023-09-25 11:07 UTC (permalink / raw) To: kernel test robot, Mel Gorman, Peter Zijlstra Cc: oe-lkp, lkp, linux-kernel, ying.huang, feng.tang, fengwei.yin, aubrey.li, yu.c.chen, Mike Galbraith, K Prateek Nayak, Peter Zijlstra (Intel), linux-tip-commits, x86, Gautham Shenoy * kernel test robot <oliver.sang@intel.com> wrote: > Hello, > > kernel test robot noticed a -19.0% regression of stress-ng.filename.ops_per_sec on: Thanks for the testing, this is useful! So I've tabulated the results into a much easier to read format: > | testcase: change | stress-ng: stress-ng.filename.ops_per_sec -19.0% regression > | testcase: change | stress-ng: stress-ng.lockbus.ops_per_sec -6.0% regression > | testcase: change | stress-ng: stress-ng.sigfd.ops_per_sec 17.6% improvement > | testcase: change | phoronix-test-suite: phoronix-test-suite.darktable.Masskrug.CPU-only.seconds -5.3% improvement > | testcase: change | lmbench3: lmbench3.TCP.socket.bandwidth.64B.MB/sec 11.5% improvement > | testcase: change | phoronix-test-suite: phoronix-test-suite.darktable.Boat.CPU-only.seconds -3.5% improvement > | testcase: change | stress-ng: stress-ng.sigrt.ops_per_sec 100.2% improvement > | testcase: change | stress-ng: stress-ng.sigsuspend.ops_per_sec -93.9% regression > | testcase: change | stress-ng: stress-ng.sigsuspend.ops_per_sec -82.1% regression > | testcase: change | stress-ng: stress-ng.sock.ops_per_sec 59.4% improvement > | testcase: change | blogbench: blogbench.write_score -35.9% regression > | testcase: change | hackbench: hackbench.throughput -4.8% regression > | testcase: change | blogbench: blogbench.write_score -59.3% regression > | testcase: change | stress-ng: stress-ng.exec.ops_per_sec -34.6% regression > | testcase: change | netperf: netperf.Throughput_Mbps 60.6% improvement > | testcase: change | hackbench: hackbench.throughput 19.1% improvement > | testcase: change | stress-ng: stress-ng.dnotify.ops_per_sec -15.7% regression And then sorted them along the regression/improvement axis: > | testcase: change | stress-ng: stress-ng.sigsuspend.ops_per_sec -93.9% regression > | testcase: change | stress-ng: stress-ng.sigsuspend.ops_per_sec -82.1% regression > | testcase: change | blogbench: blogbench.write_score -59.3% regression > | testcase: change | blogbench: blogbench.write_score -35.9% regression > | testcase: change | stress-ng: stress-ng.exec.ops_per_sec -34.6% regression > | testcase: change | stress-ng: stress-ng.filename.ops_per_sec -19.0% regression > | testcase: change | stress-ng: stress-ng.dnotify.ops_per_sec -15.7% regression > | testcase: change | stress-ng: stress-ng.lockbus.ops_per_sec -6.0% regression > | testcase: change | hackbench: hackbench.throughput -4.8% regression > | testcase: change | phoronix-test-suite: phoronix-test-suite.darktable.Masskrug.CPU-only.seconds +5.3% improvement > | testcase: change | phoronix-test-suite: phoronix-test-suite.darktable.Boat.CPU-only.seconds +3.5% improvement > | testcase: change | lmbench3: lmbench3.TCP.socket.bandwidth.64B.MB/sec 11.5% improvement > | testcase: change | stress-ng: stress-ng.sigfd.ops_per_sec 17.6% improvement > | testcase: change | hackbench: hackbench.throughput 19.1% improvement > | testcase: change | stress-ng: stress-ng.sock.ops_per_sec 59.4% improvement > | testcase: change | netperf: netperf.Throughput_Mbps 60.6% improvement > | testcase: change | stress-ng: stress-ng.sigrt.ops_per_sec 100.2% improvement Testing results notes: - the '+' denotes an inverted improvement. The mixing of signs in the output of the ktest robot is arguably confusing. - Any hope getting similar summary format by default? It's much more informative than just picking up the biggest regression, which wasn't even done correctly AFAICT. Summary: While there's a lot of improvements, it is primarily the nature of performance regressions that dictate the way forward: - stress-ng.sigsuspend.ops_per_sec regressions, -93%: Clearly signal delivery performance hurts from delayed preemption, but that should be straightforward to resolve, if we are willing to commit to adding a high-prio insta-wakeup variant API ... - stress-ng.exec.ops_per_sec -34% regression: Likewise this possibly expresses that it's better to immediately reschedule during exec() - but maybe it's more and reflects some unfavorable migration, as suggested by the NUMA locality figures: %change %stddev | \ 79317172 -34.2% 52217838 ± 3% numa-numastat.node0.local_node 79360983 -34.2% 52240348 ± 3% numa-numastat.node0.numa_hit 77971050 -33.2% 52068168 ± 3% numa-numastat.node1.local_node 78009071 -33.2% 52089987 ± 3% numa-numastat.node1.numa_hit 88287 -45.7% 47970 ± 2% vmstat.system.cs - 'blogbench' regression of -59%: It too has a very large reduction in context switches: %stddev %change %stddev \ | \ 30035 -49.7% 15097 ± 3% vmstat.system.cs 2243545 ± 2% -4.1% 2152228 blogbench.read_score 52412617 -28.3% 37571769 blogbench.time.file_system_outputs 2682930 -74.1% 694136 blogbench.time.involuntary_context_switches 2369329 -50.0% 1184098 ± 5% blogbench.time.voluntary_context_switches 5851 -35.9% 3752 ± 2% blogbench.write_score It's unclear to me what's happening with this one, just from these stats, but it's "write_score" that hurts most. - 'stress-ng.filename.ops_per_sec' regression of -19%: This test suffered from an *increase* in context-switching, and a large increase in CPU-idle: %stddev %change %stddev \ | \ 4641666 +19.5% 5545394 ± 2% cpuidle..usage 90589 ± 2% +70.5% 154471 ± 2% vmstat.system.cs 628439 -19.2% 507711 stress-ng.filename.ops 10317 -19.0% 8355 stress-ng.filename.ops_per_sec 171981 -59.7% 69333 ± 3% stress-ng.time.involuntary_context_switches 770691 ± 3% +200.9% 2319214 stress-ng.time.voluntary_context_switches Anyway, it's clear from these results that while many workloads hurt from our notion of wake-preemption, there's several ones that benefit from it, especially generic ones like phoronix-test-suite - which have no good way to turn off wakeup preemption (SCHED_BATCH might help though). One way to approach this would be to instead of always doing wakeup-preemption (our current default), we could turn it around and only use it when it is clearly beneficial - such as signal delivery, or exec(). The canonical way to solve this would be give *userspace* a way to signal that it's beneficial to preempt immediately, ie. yield(), but right now that interface is hurting tasks that only want to give other tasks a chance to run, without necessarily giving up their own right to run: se->deadline += calc_delta_fair(se->slice, se); Anyway, my patch is obviously a no-go as-is, and this clearly needs more work. Thanks, Ingo ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks 2023-09-25 11:07 ` Ingo Molnar @ 2023-09-25 16:45 ` Chen Yu 0 siblings, 0 replies; 30+ messages in thread From: Chen Yu @ 2023-09-25 16:45 UTC (permalink / raw) To: Ingo Molnar Cc: Peter Zijlstra, linux-kernel, Mike Galbraith, linux-tip-commits, x86 Hi Ingo, On 2023-09-25 at 13:07:08 +0200, Ingo Molnar wrote: > > Anyway, it's clear from these results that while many workloads hurt > from our notion of wake-preemption, there's several ones that benefit > from it, especially generic ones like phoronix-test-suite - which have > no good way to turn off wakeup preemption (SCHED_BATCH might help though). > > One way to approach this would be to instead of always doing > wakeup-preemption (our current default), we could turn it around and > only use it when it is clearly beneficial - such as signal delivery, > or exec(). > > The canonical way to solve this would be give *userspace* a way to > signal that it's beneficial to preempt immediately, ie. yield(), > but right now that interface is hurting tasks that only want to > give other tasks a chance to run, without necessarily giving up > their own right to run: > > se->deadline += calc_delta_fair(se->slice, se); > Do you mean we want to give hint to the scheduler that, in most cases the tasks do not want to be wakeup-preempted, but some other tasks want to keep current wakeup-preempt strategy? If we want the current task to be preempted easier, we have to shrink the current task's slice, which gives up its own right to run. If this is the case, how about introducing negative se->slice, tasks has a negative slice indicates that it wants to be wakeup-preempted, and its real slice is the absolute value of se->slice. diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index f7c0d9cad6e0..019576da9737 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -8120,7 +8120,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ * Batch and idle tasks do not preempt non-idle tasks (their preemption * is driven by the tick): */ - if (unlikely(p->policy != SCHED_NORMAL) || !sched_feat(WAKEUP_PREEMPTION)) + if (unlikely(p->policy != SCHED_NORMAL) || || se->slice > 0 || !sched_feat(WAKEUP_PREEMPTION)) return; find_matching_se(&se, &pse); thanks, Chenyu ^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-16 13:40 ` Peter Zijlstra 2023-08-16 15:38 ` Mike Galbraith 2023-08-17 15:10 ` [tip: sched/core] sched/eevdf: Curb wakeup-preemption tip-bot2 for Peter Zijlstra @ 2023-08-18 1:09 ` Chen Yu 2023-08-22 6:48 ` Chen Yu 2 siblings, 1 reply; 30+ messages in thread From: Chen Yu @ 2023-08-18 1:09 UTC (permalink / raw) To: Peter Zijlstra Cc: Mike Galbraith, kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On 2023-08-16 at 15:40:59 +0200, Peter Zijlstra wrote: > On Wed, Aug 16, 2023 at 02:37:16PM +0200, Peter Zijlstra wrote: > > On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: > > > > > --- a/kernel/sched/fair.c > > > +++ b/kernel/sched/fair.c > > > @@ -875,6 +875,12 @@ static struct sched_entity *pick_eevdf(s > > > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > > > curr = NULL; > > > > > > + /* > > > + * Once selected, run the task to parity to avoid overscheduling. > > > + */ > > > + if (sched_feat(RUN_TO_PARITY) && curr) > > > + return curr; > > > + > > > while (node) { > > > struct sched_entity *se = __node_2_se(node); > > > > > > > So I read it wrong last night... but I rather like this idea. But > > there's something missing. When curr starts a new slice it should > > probably do a full repick and not stick with it. > > > > Let me poke at this a bit.. nice > > Something like so.. it shouldn't matter much now, but might make a > difference once we start mixing different slice lengths. > > --- > kernel/sched/fair.c | 12 ++++++++++++ > kernel/sched/features.h | 1 + > 2 files changed, 13 insertions(+) > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index fe5be91c71c7..128a78f3f264 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -873,6 +873,13 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq) > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > curr = NULL; > > + /* > + * Once selected, run a task until it either becomes non-eligible or > + * until it gets a new slice. See the HACK in set_next_entity(). > + */ > + if (sched_feat(RUN_TO_PARITY) && curr && curr->vlag == curr->deadline) > + return curr; > + > while (node) { > struct sched_entity *se = __node_2_se(node); > > @@ -5168,6 +5175,11 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) > update_stats_wait_end_fair(cfs_rq, se); > __dequeue_entity(cfs_rq, se); > update_load_avg(cfs_rq, se, UPDATE_TG); > + /* > + * HACK, stash a copy of deadline at the point of pick in vlag, > + * which isn't used until dequeue. > + */ > + se->vlag = se->deadline; > } > > update_stats_curr_start(cfs_rq, se); > diff --git a/kernel/sched/features.h b/kernel/sched/features.h > index 61bcbf5e46a4..f770168230ae 100644 > --- a/kernel/sched/features.h > +++ b/kernel/sched/features.h > @@ -6,6 +6,7 @@ > */ > SCHED_FEAT(PLACE_LAG, true) > SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) > +SCHED_FEAT(RUN_TO_PARITY, true) > > /* > * Prefer to schedule the task we woke last (assuming it failed I had a test on top of this, and it restored some performance: uname -r 6.5.0-rc2-mixed-slice-run-parity echo NO_RUN_TO_PARITY > /sys/kernel/debug/sched/features hackbench -g 112 -f 20 --threads -l 30000 -s 100 Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) Each sender will pass 30000 messages of 100 bytes Time: 118.220 <--- echo RUN_TO_PARITY > /sys/kernel/debug/sched/features hackbench -g 112 -f 20 --threads -l 30000 -s 100 Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) Each sender will pass 30000 messages of 100 bytes Time: 100.873 <--- Then I switched to the first commit of EEVDF, that is to introduce the avg_runtime for cfs_rq(but not use it yet) It seems that there is still 50% performance to restored: uname -r 6.5.0-rc2-add-cfs-avruntime hackbench -g 112 -f 20 --threads -l 30000 -s 100 Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) Each sender will pass 30000 messages of 100 bytes Time: 52.569 <--- I'll collect the statistic to check what's remainning there. [Besides, with the $subject benchmark, I tested it with PLACE_DEADLINE_INITIAL diabled, and it did not show much difference. I'll work with lkp to test it with RUN_TO_PARITY enabled.] thanks, Chenyu thanks, Chenyu ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-18 1:09 ` [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression Chen Yu @ 2023-08-22 6:48 ` Chen Yu 2023-08-22 7:07 ` Chen Yu 0 siblings, 1 reply; 30+ messages in thread From: Chen Yu @ 2023-08-22 6:48 UTC (permalink / raw) To: Peter Zijlstra, Mike Galbraith Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, K Prateek Nayak, Feng Tang Hi Peter, Mike, On 2023-08-18 at 09:09:29 +0800, Chen Yu wrote: > On 2023-08-16 at 15:40:59 +0200, Peter Zijlstra wrote: > > On Wed, Aug 16, 2023 at 02:37:16PM +0200, Peter Zijlstra wrote: > > > On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: > > > > > > > --- a/kernel/sched/fair.c > > > > +++ b/kernel/sched/fair.c > > > > @@ -875,6 +875,12 @@ static struct sched_entity *pick_eevdf(s > > > > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > > > > curr = NULL; > > > > > > > > + /* > > > > + * Once selected, run the task to parity to avoid overscheduling. > > > > + */ > > > > + if (sched_feat(RUN_TO_PARITY) && curr) > > > > + return curr; > > > > + > > > > while (node) { > > > > struct sched_entity *se = __node_2_se(node); > > > > > > > > > > So I read it wrong last night... but I rather like this idea. But > > > there's something missing. When curr starts a new slice it should > > > probably do a full repick and not stick with it. > > > > > > Let me poke at this a bit.. nice > > > > Something like so.. it shouldn't matter much now, but might make a > > difference once we start mixing different slice lengths. > > > > --- > > kernel/sched/fair.c | 12 ++++++++++++ > > kernel/sched/features.h | 1 + > > 2 files changed, 13 insertions(+) > > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > > index fe5be91c71c7..128a78f3f264 100644 > > --- a/kernel/sched/fair.c > > +++ b/kernel/sched/fair.c > > @@ -873,6 +873,13 @@ static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq) > > if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) > > curr = NULL; > > > > + /* > > + * Once selected, run a task until it either becomes non-eligible or > > + * until it gets a new slice. See the HACK in set_next_entity(). > > + */ > > + if (sched_feat(RUN_TO_PARITY) && curr && curr->vlag == curr->deadline) > > + return curr; > > + > > while (node) { > > struct sched_entity *se = __node_2_se(node); > > > > @@ -5168,6 +5175,11 @@ set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se) > > update_stats_wait_end_fair(cfs_rq, se); > > __dequeue_entity(cfs_rq, se); > > update_load_avg(cfs_rq, se, UPDATE_TG); > > + /* > > + * HACK, stash a copy of deadline at the point of pick in vlag, > > + * which isn't used until dequeue. > > + */ > > + se->vlag = se->deadline; > > } > > > > update_stats_curr_start(cfs_rq, se); > > diff --git a/kernel/sched/features.h b/kernel/sched/features.h > > index 61bcbf5e46a4..f770168230ae 100644 > > --- a/kernel/sched/features.h > > +++ b/kernel/sched/features.h > > @@ -6,6 +6,7 @@ > > */ > > SCHED_FEAT(PLACE_LAG, true) > > SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) > > +SCHED_FEAT(RUN_TO_PARITY, true) > > > > /* > > * Prefer to schedule the task we woke last (assuming it failed > > I had a test on top of this, and it restored some performance: > > uname -r > 6.5.0-rc2-mixed-slice-run-parity > > echo NO_RUN_TO_PARITY > /sys/kernel/debug/sched/features > hackbench -g 112 -f 20 --threads -l 30000 -s 100 > Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) > Each sender will pass 30000 messages of 100 bytes > Time: 118.220 <--- > > > echo RUN_TO_PARITY > /sys/kernel/debug/sched/features > hackbench -g 112 -f 20 --threads -l 30000 -s 100 > Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) > Each sender will pass 30000 messages of 100 bytes > Time: 100.873 <--- > > Then I switched to the first commit of EEVDF, that is to > introduce the avg_runtime for cfs_rq(but not use it yet) > It seems that there is still 50% performance to restored: > > uname -r > 6.5.0-rc2-add-cfs-avruntime > > hackbench -g 112 -f 20 --threads -l 30000 -s 100 > Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) > Each sender will pass 30000 messages of 100 bytes > Time: 52.569 <--- > > I'll collect the statistic to check what's remainning there. > > [Besides, with the $subject benchmark, I tested it with PLACE_DEADLINE_INITIAL > diabled, and it did not show much difference. I'll work with lkp > to test it with RUN_TO_PARITY enabled.] > > To figure out the remainning hackbench performance gap, I pulled the tip/sched/core and launched Flamegraph to track the perf profile. The bottleneck according to flamegraph profile is as followed: Baseline: "sched/fair: Add cfs_rq::avg_vruntime" unix_stream_sendmsg() -> slab_alloc_node() -> memcg_slab_pre_alloc_hook() -> obj_cgroup_charge_pages() -> atomic_add() The per memory cgroup counter atomic write is the bottleneck[1] Compare: "sched/eevdf: Curb wakeup-preemption" unix_stream_sendmsg() -> slab_alloc_node() -> ___slab_alloc() -> get_partial() -> get_partial_node() -> spinlock_irq() The per kmem_cache_node's list_lock spinlock is the bottleneck[2] In theory the negative impact of atomic write should be less than the spinlock, because the latter would increase the busy wait time a lot. And this might be the reason why we saw lower throughput with EEVDF. Since EEVDF does not change any code in mm, I've no idea why the bottleneck switches from atomic write to spinlock. One reason I can guess is the timing. Before EEVDF, tasks enter spinlock critical section in difference time slots, while after EEVDF, tasks enter the critical section at the same time. Reducing the spinlock contention could help improve the situation. Feng Tang told me that there is a discussion on the LKML related to high slub spinlock contention[2]. And Feng provides a hack patch which enlarges the slab percpu partial list order so as to reduce the lock contention on the node(if I understand the patch correctly) With this patch applied, the spinlock contention has been reduced a lot[3], and the performance has been improved to: hackbench -g 112 -f 20 --threads -l 30000 -s 100 Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) Each sender will pass 30000 messages of 100 bytes Time: 71.838 I guess if we enlarge the percpu slab partial list further, it can reduce the spinlock contention further, and get more improvement. Meanwhile I'm still wondering what the reason to cause multiple tasks enter the spinlock section at the same time. BTW, with RUN_TO_PARITY enabled, the blogbench has also restored some of its performance. [1] https://raw.githubusercontent.com/yu-chen-surf/schedtests/master/results/eevdf/flamegraph_add_avg_runtime.svg [2] https://raw.githubusercontent.com/yu-chen-surf/schedtests/master/results/eevdf/flamegraph_eevdf_parity.svg [3] https://raw.githubusercontent.com/yu-chen-surf/schedtests/master/results/eevdf/flamegraph_slab_enlarge_order_eevdf_parity.svg thanks, Chenyu ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-22 6:48 ` Chen Yu @ 2023-08-22 7:07 ` Chen Yu 0 siblings, 0 replies; 30+ messages in thread From: Chen Yu @ 2023-08-22 7:07 UTC (permalink / raw) To: Peter Zijlstra, Mike Galbraith Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, K Prateek Nayak, Feng Tang On 2023-08-22 at 14:48:50 +0800, Chen Yu wrote: > Hi Peter, Mike, > > On 2023-08-18 at 09:09:29 +0800, Chen Yu wrote: > > On 2023-08-16 at 15:40:59 +0200, Peter Zijlstra wrote: > > > On Wed, Aug 16, 2023 at 02:37:16PM +0200, Peter Zijlstra wrote: > > > > On Mon, Aug 14, 2023 at 08:32:55PM +0200, Mike Galbraith wrote: [snip...] > Reducing the spinlock contention could help improve the situation. > Feng Tang told me that there is a discussion on the LKML related to high > slub spinlock contention[2]. And Feng provides a hack patch which The discussion is at https://lore.kernel.org/all/ZL+R5kJpnHMUgGY2@feng-clx/ thanks, Chenyu ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-14 13:29 ` Peter Zijlstra 2023-08-14 18:32 ` Mike Galbraith @ 2023-08-16 3:40 ` Chen Yu 2023-08-16 9:20 ` Peter Zijlstra 1 sibling, 1 reply; 30+ messages in thread From: Chen Yu @ 2023-08-16 3:40 UTC (permalink / raw) To: Peter Zijlstra Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Mike Galbraith On 2023-08-14 at 15:29:35 +0200, Peter Zijlstra wrote: > On Fri, Aug 11, 2023 at 10:42:09AM +0800, Chen Yu wrote: > > > Since previously lkp has reported that with eevdf policy enabled, there was > > a regression in hackbench, I did some experiments and found that, with eevdf > > enabled there are more preemptions, and this preemption could slow down > > the waker(each waker could wakes up 20 wakee in hackbench). The reason might > > be that, check_preempt_wakeup() is easier to preempt the current task in eevdf: > > This is true. > > > Without eevdf enabled, the /proc/schedstat delta within 5 seconds on CPU8 is: > > Thu Aug 10 11:02:02 2023 cpu8 > > .stats.check_preempt_count 51973 <----- > > .stats.need_preempt_count 10514 <----- > > .stats.rq_cpu_time 5004068598 > > .stats.rq_sched_info.pcount 60374 > > .stats.rq_sched_info.run_delay 80405664582 > > .stats.sched_count 60609 > > .stats.sched_goidle 227 > > .stats.ttwu_count 56250 > > .stats.ttwu_local 14619 > > > > The preemption success ration is 10514 / 51973 = 20.23% > > ----------------------------------------------------------------------------- > > > > With eevdf enabled, the /proc/schedstat delta within 5 seconds on CPU8 is: > > Thu Aug 10 10:22:55 2023 cpu8 > > .stats.check_preempt_count 71673 <---- > > .stats.low_gran_preempt_count 57410 > > .stats.need_preempt_count 57413 <---- > > .stats.rq_cpu_time 5007778990 > > .stats.rq_sched_info.pcount 129233 > > .stats.rq_sched_info.run_delay 164830921362 > > .stats.sched_count 129233 > > .stats.ttwu_count 70222 > > .stats.ttwu_local 66847 > > > > The preemption success ration is 57413 / 71673 = 80.10% > > note: wakeup-preemption > > > According to the low_gran_preempt_count, most successfully preemption happens > > when the current->vruntime is smaller than wakee->vruntime + sysctl_sched_wakeup_granularity, > > which will not happen in current cfs's wakeup_preempt_entity(). > > > > It seems that, eevdf does not inhit the wakeup preemption as much as cfs, and > > maybe it is because eevdf needs to consider fairness more? > > Not fairness, latency. Because it wants to honour the virtual deadline. > > > Are these wakeup preemptions typically on runqueues that have only a > single other task? > There are 112 groups of hackbench waker/wakee on a 112 CPUs system. Each group has 1 waker and 20 wakees. It seems that there are more than 1 other task on each runqueue. I collected the statistics below. > That is, consider a single task running, then avg_vruntime will be it's > vruntime, because the average of one variable must be the value of that > one variable. > > Then the moment a second task joins, we get two options: > > - positive lag > - negative lag > > When the new task has negative lag, it gets placed to the right of the > currently running task (and avg_vruntime has a forward discontinuity). > At this point the new task is not eligible and does not get to run. > > When the new task has positive lag, it gets placed to the left of the > currently running task (and avg_vruntime has a backward discontinuity). > At this point the currently running task is no longer eligible, and the > new task must be selected -- irrespective of it's deadline. > > The paper doesn't (AFAIR) consider the case of wake-up-preemption > explicitly. It only considers task selection and vruntime placement. > > One option I suppose would be to gate the wakeup preemption by virtual > deadline, only allow when the new task has an earlier deadline than the > currently running one, and otherwise rely on tick preemption. > > NOTE: poking at wakeup preemption is a double edged sword, some > workloads love it, some hate it. Touching it is bound to upset the > balance -- again. > > (also, did I get that the right way around? -- I've got a Monday brain > that isn't willing to boot properly) > > --- > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c > index fe5be91c71c7..16d24e5dda8f 100644 > --- a/kernel/sched/fair.c > +++ b/kernel/sched/fair.c > @@ -8047,6 +8047,15 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ > cfs_rq = cfs_rq_of(se); > update_curr(cfs_rq); > > + if (sched_feat(WAKEUP_DEADLINE)) { > + /* > + * Only allow preemption if the virtual deadline of the new > + * task is before the virtual deadline of the existing task. > + */ > + if (deadline_gt(deadline, pse, se)) > + return; > + } > + > /* > * XXX pick_eevdf(cfs_rq) != se ? > */ > diff --git a/kernel/sched/features.h b/kernel/sched/features.h > index 61bcbf5e46a4..e733981b32aa 100644 > --- a/kernel/sched/features.h > +++ b/kernel/sched/features.h > @@ -24,6 +24,7 @@ SCHED_FEAT(CACHE_HOT_BUDDY, true) > * Allow wakeup-time preemption of the current task: > */ > SCHED_FEAT(WAKEUP_PREEMPTION, true) > +SCHED_FEAT(WAKEUP_DEADLINE, true) > > SCHED_FEAT(HRTICK, false) > SCHED_FEAT(HRTICK_DL, false) Added more schedstats to track the counter, when the wakee succeed to preempt the current task: 1. curr's deadline is smaller than the wakee's deadline 2. curr is not eligible 3. the cfs_rq->nr_running <= 2 Also applied Mike's RUN_TO_PARITY patch with some minor change to only apply RUN_TO_PARITY for wakeup: diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 57e8bc14b06e..da7260ddd7e7 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -931,14 +931,20 @@ static struct sched_entity *pick_cfs(struct cfs_rq *cfs_rq, struct sched_entity * * Which allows an EDF like search on (sub)trees. */ -static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq) +static struct sched_entity *pick_eevdf(struct cfs_rq *cfs_rq, int wake) { struct rb_node *node = cfs_rq->tasks_timeline.rb_root.rb_node; struct sched_entity *curr = cfs_rq->curr; struct sched_entity *best = NULL; - if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) + if (curr && (!curr->on_rq || !entity_eligible(cfs_rq, curr))) { curr = NULL; + if (wake) + schedstat_inc(cfs_rq->rq->curr_noeli_preempt_count); + } + + if (sched_feat(RUN_TO_PARITY) && wake && curr) + return curr; while (node) { struct sched_entity *se = __node_2_se(node); @@ -5441,7 +5447,7 @@ pick_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *curr) cfs_rq->next && entity_eligible(cfs_rq, cfs_rq->next)) return cfs_rq->next; - return pick_eevdf(cfs_rq); + return pick_eevdf(cfs_rq, 0); } se = left = pick_cfs(cfs_rq, curr); @@ -8294,6 +8300,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ int next_buddy_marked = 0; int cse_is_idle, pse_is_idle; + schedstat_inc(rq->check_preempt_count); + if (unlikely(se == pse)) return; @@ -8358,8 +8366,19 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ /* * XXX pick_eevdf(cfs_rq) != se ? */ - if (pick_eevdf(cfs_rq) == pse) + if (pick_eevdf(cfs_rq, 1) == pse) { + if (cfs_rq->curr->vruntime < + pse->vruntime + sysctl_sched_wakeup_granularity) { + schedstat_inc(rq->curr_lowgran_preempt_count); + } + if (cfs_rq->curr->deadline < pse->deadline) { + schedstat_inc(rq->curr_lowdl_preempt_count); + } + if (cfs_rq->nr_running <= 2) { + schedstat_inc(rq->curr_lownr_preempt_count); + } goto preempt; + } return; } @@ -8377,6 +8396,8 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_ return; preempt: + schedstat_inc(rq->need_preempt_count); + resched_curr(rq); /* * Only set the backward buddy when the current task is still diff --git a/kernel/sched/features.h b/kernel/sched/features.h index 2a830eccda3e..d021dd0e0226 100644 --- a/kernel/sched/features.h +++ b/kernel/sched/features.h @@ -6,6 +6,7 @@ */ SCHED_FEAT(PLACE_LAG, true) SCHED_FEAT(PLACE_DEADLINE_INITIAL, true) +SCHED_FEAT(RUN_TO_PARITY, true) /* * Prefer to schedule the task we woke last (assuming it failed diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index aa5b293ca4ed..b6e245d2bba5 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1128,6 +1128,12 @@ struct rq { /* try_to_wake_up() stats */ unsigned int ttwu_count; unsigned int ttwu_local; + unsigned int check_preempt_count; + unsigned int need_preempt_count; + unsigned int curr_lowgran_preempt_count; + unsigned int curr_lowdl_preempt_count; + unsigned int curr_noeli_preempt_count; + unsigned int curr_lownr_preempt_count; #endif #ifdef CONFIG_CPU_IDLE diff --git a/kernel/sched/stats.c b/kernel/sched/stats.c index 857f837f52cb..e7f02e7bfff2 100644 --- a/kernel/sched/stats.c +++ b/kernel/sched/stats.c @@ -133,12 +133,16 @@ static int show_schedstat(struct seq_file *seq, void *v) /* runqueue-specific stats */ seq_printf(seq, - "cpu%d %u 0 %u %u %u %u %llu %llu %lu", + "cpu%d %u 0 %u %u %u %u %llu %llu %lu %u %u %u %u %u %u", cpu, rq->yld_count, rq->sched_count, rq->sched_goidle, rq->ttwu_count, rq->ttwu_local, rq->rq_cpu_time, - rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount); + rq->rq_sched_info.run_delay, rq->rq_sched_info.pcount, + rq->check_preempt_count, rq->need_preempt_count, rq->curr_lowgran_preempt_count, + rq->curr_lowdl_preempt_count, + rq->curr_noeli_preempt_count, + rq->curr_lownr_preempt_count); seq_printf(seq, "\n"); -- 2.25.1 NO_RUN_TO_PARITY: Tue Aug 15 19:16:30 2023 cpu8 .stats.check_preempt_count 85754 .stats.curr_lowdl_preempt_count 5189 .stats.curr_lowgran_preempt_count 70063 .stats.curr_lownr_preempt_count 31 .stats.curr_noeli_preempt_count 30252 .stats.need_preempt_count 70066 .stats.rq_cpu_time 5032932195 .stats.rq_sched_info.pcount 155748 .stats.rq_sched_info.run_delay 159730175645 .stats.sched_count 155748 .stats.ttwu_count 84568 .stats.ttwu_local 81029 When the wakee succeeds to preempt the current task: 1. curr's deadline is smaller than wakee's deadline: 5189 / 70066 = 7.41% 2. curr is not eligible 30252 / 70066 = 43.18% 3. the cfs_rq->nr_running <= 2 31 / 70066 = 0.04% According to above data, maybe comparing the deadline does not take much effect in this case(per 1). And the scenario of only 1 other task seldom happens(per 3). And considering the current task's eligibility without checking deadline might help(per 2, there are (70066 - 30252) times that the current is eligible but preempted, and Mike's patch addressed that) RUN_TO_PARITY: Tue Aug 15 19:19:07 2023 cpu8 .stats.check_preempt_count 20147 .stats.curr_lowdl_preempt_count 334 .stats.curr_lowgran_preempt_count 4637 .stats.curr_lownr_preempt_count 8 .stats.curr_noeli_preempt_count 5451 .stats.need_preempt_count 4645 .stats.rq_cpu_time 5052091287 .stats.rq_sched_info.pcount 25486 .stats.rq_sched_info.run_delay 166261574881 .stats.sched_count 25486 .stats.ttwu_count 20146 .stats.ttwu_local 20146 With Mike's change, the wakeup preemption success ratio drops to 4645 / 20147 = 23.06%, which is very close to cfs wakeup preemption ratio 20.23%. And the throughput has been increased: echo NO_RUN_TO_PARITY > /sys/kernel/debug/sched/features hackbench -g 112 -f 20 --threads -l 30000 -s 100 Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) Each sender will pass 30000 messages of 100 bytes Time: 117.487 echo RUN_TO_PARITY > /sys/kernel/debug/sched/features hackbench -g 112 -f 20 --threads -l 30000 -s 100 Running in threaded mode with 112 groups using 40 file descriptors each (== 4480 tasks) Each sender will pass 30000 messages of 100 bytes Time: 101.285 <--- lower the better Also tested schbench with/without RUN_TO_PARITY, no much difference(especially wakeup latency) is observed: echo NO_RUN_TO_PARITY > /sys/kernel/debug/sched/features ./schbench -m 14 -t 16 -r 100 Wakeup Latencies percentiles (usec) runtime 100 (s) (941339 total samples) 50.0th: 4 (53382 samples) 90.0th: 3772 (353480 samples) * 99.0th: 7448 (84115 samples) 99.9th: 7976 (8244 samples) min=1, max=15186 Request Latencies percentiles (usec) runtime 100 (s) (941598 total samples) 50.0th: 23264 (283891 samples) 90.0th: 32672 (375363 samples) * 99.0th: 52288 (83963 samples) 99.9th: 95360 (8449 samples) min=4487, max=241201 RPS percentiles (requests) runtime 100 (s) (101 total samples) 20.0th: 9264 (22 samples) * 50.0th: 9392 (33 samples) 90.0th: 9584 (36 samples) min=9055, max=10176 average rps: 9415.98 echo RUN_TO_PARITY > /sys/kernel/debug/sched/features ./schbench -m 14 -t 16 -r 100 Wakeup Latencies percentiles (usec) runtime 100 (s) (943005 total samples) 50.0th: 4 (53961 samples) 90.0th: 3772 (366473 samples) * 99.0th: 7496 (84902 samples) 99.9th: 9296 (8441 samples) min=1, max=20467 Request Latencies percentiles (usec) runtime 100 (s) (943206 total samples) 50.0th: 23072 (283181 samples) 90.0th: 32096 (376473 samples) * 99.0th: 53568 (84586 samples) 99.9th: 99456 (8443 samples) min=4078, max=219330 RPS percentiles (requests) runtime 100 (s) (101 total samples) 20.0th: 9296 (22 samples) * 50.0th: 9392 (34 samples) 90.0th: 9584 (38 samples) min=9065, max=10188 average rps: 9432.06 thanks, Chenyu ^ permalink raw reply related [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-16 3:40 ` Chen Yu @ 2023-08-16 9:20 ` Peter Zijlstra 0 siblings, 0 replies; 30+ messages in thread From: Peter Zijlstra @ 2023-08-16 9:20 UTC (permalink / raw) To: Chen Yu Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Mike Galbraith On Wed, Aug 16, 2023 at 11:40:33AM +0800, Chen Yu wrote: > > Are these wakeup preemptions typically on runqueues that have only a > > single other task? > > > > There are 112 groups of hackbench waker/wakee on a 112 CPUs system. > Each group has 1 waker and 20 wakees. It seems that there are more than 1 > other task on each runqueue. I collected the statistics below. Oh, I thought we were still talking about $subject benchmark. Yeah, hackbench has a ton of tasks. > According to above data, maybe comparing the deadline does not take much > effect in this case(per 1). Yeah, I stuck a trace_printk() in last night and quicly realized the lag is on the same order of request size, which then totally makes the whole thing moot. ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-11 1:11 ` Chen Yu 2023-08-11 2:42 ` Chen Yu @ 2023-08-14 12:49 ` Peter Zijlstra 2023-08-18 1:54 ` Chen Yu 1 sibling, 1 reply; 30+ messages in thread From: Peter Zijlstra @ 2023-08-14 12:49 UTC (permalink / raw) To: Chen Yu; +Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar On Fri, Aug 11, 2023 at 09:11:21AM +0800, Chen Yu wrote: > On 2023-08-10 at 21:24:37 +0800, kernel test robot wrote: > > > > > > Hello, > > > > kernel test robot noticed a -34.8% regression of phoronix-test-suite.blogbench.Write.final_score on: > > > > > > commit: e0c2ff903c320d3fd3c2c604dc401b3b7c0a1d13 ("sched/fair: Remove sched_feat(START_DEBIT)") > > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/eevdf > > > > testcase: phoronix-test-suite > > test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory > > parameters: > > > > test: blogbench-1.1.0 > > option_a: Write > > cpufreq_governor: performance > > Is this benchmark fork() heavy? > It seems that commit e0c2ff903c32 removed the sched_feat(START_DEBIT) for initial > task, but also increases the vruntime for non-initial task: > Before the e0c2ff903c32, the vruntime for a enqueued task is: > cfs_rq->min_vruntime > After the e0c2ff903c32, the vruntime for a enqueued task is: > avg_vruntime(cfs_rq) = \Sum v_i * w_i / W > = \Sum v_i / nr_tasks > which is usually higher than cfs_rq->min_vruntime, and we give less sleep bonus to > the wakee, which could bring more or less impact to different workloads. > But since later we switched to lag based placement, this new vruntime will minus > lag, which could mitigate this problem. Right.. but given this problem was bisected through the lag based placement to this commit, I wondered about fork() / pthread_create(). If this is indeed fork()/pthread_create() heavy, could you please see if disabling PLACE_DEADLINE_INITIAL helps? ^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression 2023-08-14 12:49 ` Peter Zijlstra @ 2023-08-18 1:54 ` Chen Yu 0 siblings, 0 replies; 30+ messages in thread From: Chen Yu @ 2023-08-18 1:54 UTC (permalink / raw) To: Peter Zijlstra Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86, Ingo Molnar, Mike Galbraith On 2023-08-14 at 14:49:14 +0200, Peter Zijlstra wrote: > On Fri, Aug 11, 2023 at 09:11:21AM +0800, Chen Yu wrote: > > On 2023-08-10 at 21:24:37 +0800, kernel test robot wrote: > > > > > > > > > Hello, > > > > > > kernel test robot noticed a -34.8% regression of phoronix-test-suite.blogbench.Write.final_score on: > > > > > > > > > commit: e0c2ff903c320d3fd3c2c604dc401b3b7c0a1d13 ("sched/fair: Remove sched_feat(START_DEBIT)") > > > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/eevdf > > > > > > testcase: phoronix-test-suite > > > test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory > > > parameters: > > > > > > test: blogbench-1.1.0 > > > option_a: Write > > > cpufreq_governor: performance > > > > > Is this benchmark fork() heavy? > It is not fork() heavy. After created the threads, it becomes a loop to write to some files. > > It seems that commit e0c2ff903c32 removed the sched_feat(START_DEBIT) for initial > > task, but also increases the vruntime for non-initial task: > > Before the e0c2ff903c32, the vruntime for a enqueued task is: > > cfs_rq->min_vruntime > > After the e0c2ff903c32, the vruntime for a enqueued task is: > > avg_vruntime(cfs_rq) = \Sum v_i * w_i / W > > = \Sum v_i / nr_tasks > > which is usually higher than cfs_rq->min_vruntime, and we give less sleep bonus to > > the wakee, which could bring more or less impact to different workloads. > > But since later we switched to lag based placement, this new vruntime will minus > > lag, which could mitigate this problem. > > Right.. but given this problem was bisected through the lag based > placement to this commit, I wondered about fork() / pthread_create(). > > If this is indeed fork()/pthread_create() heavy, could you please see if > disabling PLACE_DEADLINE_INITIAL helps? Tested with PLACE_DEADLINE_INITIAL disabled, no much difference is observed. The baseline is Commit 246c6d7ab4d0 ("sched/eevdf: Better handle mixed slice length") PLACE_DEADLINE_I NO_PLACE_DEADLINE_INITIAL ---------------- --------------------------- fail:runs %reproduction fail:runs | | | 4166 -4.7% 3969 phoronix-test-suite.blogbench.Write.final_score 330.88 +4.4% 345.49 phoronix-test-suite.time.elapsed_time 330.88 +4.4% 345.49 phoronix-test-suite.time.elapsed_time.max 150672 -0.0% 150640 phoronix-test-suite.time.file_system_inputs 29947344 -2.2% 29277840 phoronix-test-suite.time.file_system_outputs 1954038 -0.3% 1947949 phoronix-test-suite.time.involuntary_context_switches 163.00 +1.2% 165.00 phoronix-test-suite.time.major_page_faults 32256 +0.7% 32472 phoronix-test-suite.time.maximum_resident_set_size 152607 -1.1% 150874 phoronix-test-suite.time.minor_page_faults 4096 +0.0% 4096 phoronix-test-suite.time.page_size 8169 -5.0% 7764 phoronix-test-suite.time.percent_of_cpu_this_job_got 26616 -0.9% 26374 phoronix-test-suite.time.system_time 416.59 +8.3% 450.98 phoronix-test-suite.time.user_time 1764497 -0.8% 1749992 phoronix-test-suite.time.voluntary_context_switches blogbench.Write.final_score on different commits in eevdf branch: sched/fair: Add cfs_rq::avg_vruntime 5217 sched/fair: Remove sched_feat(START_DEBIT) 3223 sched/fair: Add lag based placement 2736 sched/fair: Implement an EEVDF-like scheduling policy 3942 sched/fair: Commit to EEVDF 3957 sched/eevdf: Better handle mixed slice length 3836 It seems that, "Remove sched_feat(START_DEBIT)" brings some impact and "Implement an EEVDF-like scheduling policy" restores some throughput. The score from "sched/fair: Add lag based placement" might not be reliable that, in place_entity() it scales the vlag based on se->load.weight directly, while "Implement an EEVDF-like scheduling policy" fixes that by using scale_load_down(). I'll check what RUN_TO_PARITY brings to blogbench. thanks, Chenyu ^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2023-09-25 16:46 UTC | newest] Thread overview: 30+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-08-10 13:24 [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression kernel test robot 2023-08-11 1:11 ` Chen Yu 2023-08-11 2:42 ` Chen Yu 2023-08-14 13:29 ` Peter Zijlstra 2023-08-14 18:32 ` Mike Galbraith 2023-08-15 23:52 ` Peter Zijlstra 2023-08-16 3:54 ` Mike Galbraith 2023-08-16 12:37 ` Peter Zijlstra 2023-08-16 13:40 ` Peter Zijlstra 2023-08-16 15:38 ` Mike Galbraith 2023-08-16 20:04 ` Peter Zijlstra 2023-08-17 1:25 ` Mike Galbraith 2023-08-17 15:10 ` [tip: sched/core] sched/eevdf: Curb wakeup-preemption tip-bot2 for Peter Zijlstra 2023-08-21 10:39 ` K Prateek Nayak 2023-08-21 15:30 ` Mike Galbraith 2023-08-22 3:03 ` K Prateek Nayak 2023-08-22 6:09 ` Mike Galbraith 2023-08-25 6:41 ` K Prateek Nayak 2023-09-19 9:02 ` [PATCH] sched/fair: Do not wakeup-preempt same-prio SCHED_OTHER tasks Ingo Molnar 2023-09-19 9:48 ` Mike Galbraith 2023-09-22 10:00 ` kernel test robot 2023-09-25 11:07 ` Ingo Molnar 2023-09-25 16:45 ` Chen Yu 2023-08-18 1:09 ` [tip:sched/eevdf] [sched/fair] e0c2ff903c: phoronix-test-suite.blogbench.Write.final_score -34.8% regression Chen Yu 2023-08-22 6:48 ` Chen Yu 2023-08-22 7:07 ` Chen Yu 2023-08-16 3:40 ` Chen Yu 2023-08-16 9:20 ` Peter Zijlstra 2023-08-14 12:49 ` Peter Zijlstra 2023-08-18 1:54 ` Chen Yu
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
