* [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression
@ 2024-03-01 8:09 kernel test robot
2024-03-04 0:32 ` Frederic Weisbecker
2024-03-12 23:57 ` Frederic Weisbecker
0 siblings, 2 replies; 10+ messages in thread
From: kernel test robot @ 2024-03-01 8:09 UTC (permalink / raw)
To: Anna-Maria Behnsen
Cc: oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner,
Frederic Weisbecker, ying.huang, feng.tang, fengwei.yin,
oliver.sang
Hello,
kernel test robot noticed a -1.2% regression of netperf.Throughput_Mbps on:
commit: 7ee988770326fca440472200c3eb58935fe712f6 ("timers: Implement the hierarchical pull model")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git timers/core
testcase: netperf
test machine: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory
parameters:
ip: ipv4
runtime: 300s
nr_threads: 200%
cluster: cs-localhost
test: SCTP_STREAM
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202403011511.24defbbd-oliver.sang@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20240301/202403011511.24defbbd-oliver.sang@intel.com
=========================================================================================
cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase:
cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/200%/debian-12-x86_64-20240206.cgz/300s/lkp-icl-2sp2/SCTP_STREAM/netperf
commit:
57e95a5c41 ("timers: Introduce function to check timer base is_idle flag")
7ee9887703 ("timers: Implement the hierarchical pull model")
57e95a5c4117dc6a 7ee988770326fca440472200c3e
---------------- ---------------------------
%stddev %change %stddev
\ | \
5232 +6.8% 5589 vmstat.system.in
0.03 +0.0 0.04 mpstat.cpu.all.soft%
0.04 +0.0 0.08 mpstat.cpu.all.sys%
201.33 ± 3% -34.1% 132.67 ± 7% perf-c2c.DRAM.remote
187.83 ± 3% -28.7% 133.83 ± 12% perf-c2c.HITM.local
40.67 ± 7% -57.4% 17.33 ± 26% perf-c2c.HITM.remote
4984 +4.9% 5227 proc-vmstat.nr_shmem
100999 +3.6% 104640 proc-vmstat.nr_slab_unreclaimable
8114 ± 2% +11.0% 9005 ± 2% proc-vmstat.pgactivate
20.00 +1907.5% 401.50 ± 79% proc-vmstat.unevictable_pgs_culled
965754 -10.3% 866147 sched_debug.cpu.avg_idle.avg
101797 ± 12% +41.8% 144376 ± 4% sched_debug.cpu.avg_idle.stddev
0.00 ± 22% -40.7% 0.00 ± 18% sched_debug.cpu.next_balance.stddev
886.06 ± 18% +339.7% 3895 ± 8% sched_debug.cpu.nr_switches.min
5474 ± 12% -23.9% 4164 ± 10% sched_debug.cpu.nr_switches.stddev
4.10 -1.2% 4.05 netperf.ThroughputBoth_Mbps
1049 -1.2% 1037 netperf.ThroughputBoth_total_Mbps
4.10 -1.2% 4.05 netperf.Throughput_Mbps
1049 -1.2% 1037 netperf.Throughput_total_Mbps
102.17 ± 8% +887.4% 1008 ± 3% netperf.time.involuntary_context_switches
2.07 ± 3% +388.2% 10.11 netperf.time.system_time
15.14 +18.8% 17.99 perf-stat.i.MPKI
1.702e+08 +1.6% 1.729e+08 perf-stat.i.branch-instructions
1.68 +0.0 1.71 perf-stat.i.branch-miss-rate%
18.46 +3.1 21.57 perf-stat.i.cache-miss-rate%
4047319 +19.7% 4846291 perf-stat.i.cache-misses
22007366 +3.2% 22707969 perf-stat.i.cache-references
1.84 +17.0% 2.15 perf-stat.i.cpi
9.159e+08 +10.4% 1.011e+09 perf-stat.i.cpu-cycles
161.08 +183.7% 456.91 ± 2% perf-stat.i.cpu-migrations
190.71 -1.6% 187.66 perf-stat.i.cycles-between-cache-misses
8.434e+08 +1.2% 8.535e+08 perf-stat.i.instructions
0.61 -10.6% 0.54 perf-stat.i.ipc
4.79 +18.4% 5.66 perf-stat.overall.MPKI
4.21 -0.1 4.14 perf-stat.overall.branch-miss-rate%
18.39 +2.9 21.33 perf-stat.overall.cache-miss-rate%
1.09 +9.1% 1.19 perf-stat.overall.cpi
227.07 -7.8% 209.29 perf-stat.overall.cycles-between-cache-misses
0.92 -8.3% 0.84 perf-stat.overall.ipc
1379820 +2.3% 1411871 perf-stat.overall.path-length
1.702e+08 +1.5% 1.728e+08 perf-stat.ps.branch-instructions
4035389 +19.7% 4830446 perf-stat.ps.cache-misses
21948285 +3.2% 22642774 perf-stat.ps.cache-references
9.163e+08 +10.3% 1.011e+09 perf-stat.ps.cpu-cycles
160.61 +183.5% 455.30 ± 2% perf-stat.ps.cpu-migrations
8.433e+08 +1.1% 8.529e+08 perf-stat.ps.instructions
2.551e+11 +1.5% 2.589e+11 perf-stat.total.instructions
31.82 ± 3% -12.9 18.91 ± 5% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
36.90 ± 2% -12.1 24.83 ± 7% perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
27.61 ± 3% -11.9 15.68 ± 8% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state
31.75 ± 2% -11.5 20.25 ± 8% perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
24.39 ± 3% -10.0 14.38 ± 8% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter
24.33 ± 3% -10.0 14.36 ± 8% perf-profile.calltrace.cycles-pp.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt
21.00 ± 3% -9.2 11.84 ± 9% perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
20.97 ± 3% -9.1 11.83 ± 9% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.irq_exit_rcu
20.97 ± 3% -9.1 11.84 ± 9% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt
4.92 ± 10% -4.0 0.91 ± 28% perf-profile.calltrace.cycles-pp._nohz_idle_balance.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
1.81 ± 13% -1.5 0.29 ±100% perf-profile.calltrace.cycles-pp.run_timer_softirq.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
9.76 ± 7% -1.5 8.26 ± 8% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv
5.08 ± 7% -1.0 4.12 ± 12% perf-profile.calltrace.cycles-pp.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
2.99 ± 4% -0.8 2.22 ± 16% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
2.98 ± 4% -0.8 2.22 ± 16% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.66 ± 7% -0.7 0.95 ± 11% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter
1.58 ± 5% -0.7 0.90 ± 10% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt
1.05 ± 9% -0.5 0.52 ± 49% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_generate_timeout_event.call_timer_fn.__run_timers
0.95 ± 14% -0.5 0.50 ± 46% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.90 ± 14% -0.4 0.53 ± 47% perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
0.90 ± 14% -0.4 0.52 ± 47% perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork
0.64 ± 17% -0.4 0.27 ±100% perf-profile.calltrace.cycles-pp.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.64 ± 17% -0.4 0.27 ±100% perf-profile.calltrace.cycles-pp.path_openat.do_filp_open.do_sys_openat2.__x64_sys_openat.do_syscall_64
0.96 ± 8% -0.3 0.65 ± 45% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.96 ± 8% -0.3 0.64 ± 45% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.96 ± 8% -0.3 0.65 ± 45% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.53 ± 4% -0.3 1.22 ± 12% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
1.53 ± 4% -0.3 1.24 ± 12% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
1.62 ± 4% -0.3 1.33 ± 12% perf-profile.calltrace.cycles-pp.read
1.57 ± 5% -0.3 1.28 ± 13% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
1.57 ± 5% -0.3 1.28 ± 13% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
1.23 ± 8% -0.2 1.03 ± 10% perf-profile.calltrace.cycles-pp.tick_nohz_stop_tick.tick_nohz_idle_stop_tick.cpuidle_idle_call.do_idle.cpu_startup_entry
0.50 ± 45% +0.4 0.92 ± 14% perf-profile.calltrace.cycles-pp.rebalance_domains._nohz_idle_balance.__do_softirq.irq_exit_rcu.sysvec_call_function_single
0.45 ± 72% +0.5 0.95 ± 17% perf-profile.calltrace.cycles-pp.skb_release_data.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.sock_recvmsg
0.75 ± 46% +0.5 1.26 ± 24% perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.sctp_chunk_put.sctp_ulpevent_free.sctp_recvmsg
0.82 ± 45% +0.5 1.36 ± 22% perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg
1.08 ± 12% +0.6 1.68 ± 14% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter
1.08 ± 12% +0.6 1.68 ± 14% perf-profile.calltrace.cycles-pp._nohz_idle_balance.__do_softirq.irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single
1.08 ± 12% +0.6 1.68 ± 14% perf-profile.calltrace.cycles-pp.__do_softirq.irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt
0.18 ±141% +0.6 0.83 ± 14% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains._nohz_idle_balance.__do_softirq.irq_exit_rcu
0.00 +0.7 0.66 ± 18% perf-profile.calltrace.cycles-pp.sctp_generate_timeout_event.call_timer_fn.__run_timers.timer_expire_remote.tmigr_handle_remote_up
0.30 ±101% +0.7 0.97 ± 28% perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.consume_skb.sctp_chunk_put.sctp_ulpevent_free
0.00 +0.8 0.77 ± 18% perf-profile.calltrace.cycles-pp.call_timer_fn.__run_timers.timer_expire_remote.tmigr_handle_remote_up.tmigr_handle_remote
0.00 +0.8 0.80 ± 17% perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.consume_skb.sctp_chunk_put.sctp_datamsg_put
3.45 ± 14% +0.8 4.26 ± 11% perf-profile.calltrace.cycles-pp.__memcpy.skb_copy_bits.skb_copy.sctp_make_reassembled_event.sctp_ulpq_tail_data
3.46 ± 14% +0.8 4.28 ± 10% perf-profile.calltrace.cycles-pp.skb_copy_bits.skb_copy.sctp_make_reassembled_event.sctp_ulpq_tail_data.sctp_cmd_interpreter
0.00 +0.8 0.82 ± 18% perf-profile.calltrace.cycles-pp.__run_timers.timer_expire_remote.tmigr_handle_remote_up.tmigr_handle_remote.__do_softirq
0.00 +0.8 0.83 ± 17% perf-profile.calltrace.cycles-pp.timer_expire_remote.tmigr_handle_remote_up.tmigr_handle_remote.__do_softirq.irq_exit_rcu
0.00 +0.8 0.84 ± 31% perf-profile.calltrace.cycles-pp.free_one_page.__free_pages_ok.skb_release_data.consume_skb.sctp_chunk_put
4.44 ± 14% +0.9 5.36 ± 10% perf-profile.calltrace.cycles-pp.sctp_make_reassembled_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv
0.00 +0.9 0.94 ± 16% perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.sctp_chunk_put.sctp_datamsg_put.sctp_chunk_free
4.33 ± 14% +0.9 5.27 ± 10% perf-profile.calltrace.cycles-pp.skb_copy.sctp_make_reassembled_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm
2.42 ± 9% +1.1 3.56 ± 17% perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit
2.41 ± 9% +1.1 3.55 ± 17% perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2
2.42 ± 9% +1.2 3.58 ± 17% perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit
2.34 ± 10% +1.2 3.50 ± 17% perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit
2.56 ± 8% +1.2 3.74 ± 17% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush
1.72 ± 15% +1.2 2.91 ± 19% perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_assoc_rwnd_increase
1.76 ± 14% +1.2 2.97 ± 18% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_assoc_rwnd_increase.sctp_ulpevent_free
1.88 ± 13% +1.2 3.10 ± 18% perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_assoc_rwnd_increase.sctp_ulpevent_free.sctp_recvmsg
2.34 ± 9% +1.2 3.57 ± 19% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq
2.34 ± 10% +1.2 3.57 ± 19% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip
1.98 ± 13% +1.3 3.23 ± 19% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_assoc_rwnd_increase.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg
2.23 ± 10% +1.3 3.50 ± 19% perf-profile.calltrace.cycles-pp.sctp_assoc_rwnd_increase.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.sock_recvmsg
0.00 +1.4 1.37 ± 9% perf-profile.calltrace.cycles-pp.tmigr_handle_remote_up.tmigr_handle_remote.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt
0.00 +1.4 1.38 ± 10% perf-profile.calltrace.cycles-pp.tmigr_handle_remote.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
3.52 ± 7% +1.8 5.30 ± 15% perf-profile.calltrace.cycles-pp.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.sock_recvmsg.____sys_recvmsg
4.94 ± 5% +5.6 10.58 ± 16% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
4.94 ± 5% +5.6 10.58 ± 16% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm
4.94 ± 5% +5.6 10.58 ± 16% perf-profile.calltrace.cycles-pp.ret_from_fork_asm
2.83 ± 9% +5.9 8.73 ± 18% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
1.73 ± 12% +6.2 7.93 ± 18% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
1.73 ± 12% +6.2 7.93 ± 18% perf-profile.calltrace.cycles-pp.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
1.70 ± 13% +6.2 7.91 ± 18% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.run_ksoftirqd
1.70 ± 13% +6.2 7.91 ± 18% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn
1.70 ± 12% +6.2 7.92 ± 18% perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread
0.00 +10.2 10.24 ± 89% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
36.91 ± 2% -12.1 24.83 ± 7% perf-profile.children.cycles-pp.acpi_idle_enter
36.80 ± 2% -12.0 24.80 ± 7% perf-profile.children.cycles-pp.acpi_safe_halt
29.95 ± 3% -11.8 18.19 ± 7% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
27.78 ± 3% -11.4 16.39 ± 8% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
25.75 ± 2% -9.1 16.65 ± 9% perf-profile.children.cycles-pp.irq_exit_rcu
5.07 ± 3% -3.4 1.71 ± 12% perf-profile.children.cycles-pp._raw_spin_lock
6.04 ± 8% -3.3 2.70 ± 17% perf-profile.children.cycles-pp._nohz_idle_balance
2.80 ± 6% -2.3 0.51 ± 31% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
1.83 ± 13% -1.3 0.51 ± 30% perf-profile.children.cycles-pp.run_timer_softirq
5.62 ± 5% -1.1 4.52 ± 10% perf-profile.children.cycles-pp.sctp_outq_flush_data
1.40 ± 13% -1.0 0.38 ± 25% perf-profile.children.cycles-pp.tick_irq_enter
1.44 ± 13% -1.0 0.44 ± 18% perf-profile.children.cycles-pp.irq_enter_rcu
1.77 ± 11% -1.0 0.78 ± 20% perf-profile.children.cycles-pp.__mod_timer
1.36 ± 15% -0.9 0.48 ± 24% perf-profile.children.cycles-pp.get_nohz_timer_target
0.97 ± 15% -0.9 0.10 ± 38% perf-profile.children.cycles-pp.tick_do_update_jiffies64
1.46 ± 11% -0.9 0.60 ± 23% perf-profile.children.cycles-pp.sctp_transport_reset_t3_rtx
1.53 ± 7% -0.6 0.92 ± 21% perf-profile.children.cycles-pp.update_blocked_averages
1.00 ± 27% -0.5 0.45 ± 32% perf-profile.children.cycles-pp.update_rq_clock_task
1.73 ± 7% -0.5 1.21 ± 7% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
1.65 ± 5% -0.5 1.14 ± 5% perf-profile.children.cycles-pp.hrtimer_interrupt
1.94 ± 6% -0.4 1.56 ± 13% perf-profile.children.cycles-pp.vfs_read
1.16 ± 10% -0.3 0.84 ± 19% perf-profile.children.cycles-pp.sctp_generate_timeout_event
0.66 ± 12% -0.3 0.37 ± 17% perf-profile.children.cycles-pp.update_rq_clock
0.99 ± 8% -0.3 0.73 ± 20% perf-profile.children.cycles-pp.__x64_sys_exit_group
0.99 ± 8% -0.3 0.73 ± 20% perf-profile.children.cycles-pp.do_group_exit
0.99 ± 8% -0.3 0.73 ± 20% perf-profile.children.cycles-pp.do_exit
0.34 ± 17% -0.2 0.12 ± 46% perf-profile.children.cycles-pp.run_rebalance_domains
0.28 ± 23% -0.2 0.07 ± 83% perf-profile.children.cycles-pp.__release_sock
1.24 ± 9% -0.2 1.03 ± 10% perf-profile.children.cycles-pp.tick_nohz_stop_tick
0.54 ± 17% -0.2 0.33 ± 20% perf-profile.children.cycles-pp.idle_cpu
0.30 ± 16% -0.2 0.10 ± 32% perf-profile.children.cycles-pp.ktime_get_update_offsets_now
1.09 ± 5% -0.2 0.88 ± 15% perf-profile.children.cycles-pp.seq_read_iter
0.26 ± 26% -0.2 0.07 ± 83% perf-profile.children.cycles-pp.sctp_backlog_rcv
0.35 ± 25% -0.2 0.15 ± 41% perf-profile.children.cycles-pp.release_sock
0.39 ± 19% -0.2 0.20 ± 11% perf-profile.children.cycles-pp.__memcg_slab_free_hook
0.72 ± 13% -0.2 0.54 ± 11% perf-profile.children.cycles-pp.ktime_get
0.50 ± 14% -0.2 0.33 ± 18% perf-profile.children.cycles-pp.do_vmi_align_munmap
0.39 ± 18% -0.2 0.23 ± 14% perf-profile.children.cycles-pp.sctp_inq_pop
0.51 ± 15% -0.2 0.35 ± 20% perf-profile.children.cycles-pp.do_vmi_munmap
0.46 ± 14% -0.1 0.32 ± 18% perf-profile.children.cycles-pp.proc_reg_read_iter
0.26 ± 20% -0.1 0.14 ± 29% perf-profile.children.cycles-pp.tlb_finish_mmu
0.46 ± 10% -0.1 0.34 ± 20% perf-profile.children.cycles-pp.sched_clock_cpu
0.22 ± 17% -0.1 0.12 ± 13% perf-profile.children.cycles-pp.should_we_balance
0.21 ± 13% -0.1 0.10 ± 20% perf-profile.children.cycles-pp.lookup_fast
0.36 ± 16% -0.1 0.25 ± 24% perf-profile.children.cycles-pp.__split_vma
0.29 ± 10% -0.1 0.20 ± 27% perf-profile.children.cycles-pp.wp_page_copy
0.54 ± 11% -0.1 0.45 ± 8% perf-profile.children.cycles-pp.balance_fair
0.14 ± 34% -0.1 0.06 ± 48% perf-profile.children.cycles-pp.vma_prepare
0.21 ± 28% -0.1 0.13 ± 27% perf-profile.children.cycles-pp.vsnprintf
0.14 ± 31% -0.1 0.05 ± 75% perf-profile.children.cycles-pp.irqentry_exit
0.15 ± 17% -0.1 0.07 ± 45% perf-profile.children.cycles-pp.__x64_sys_munmap
0.19 ± 19% -0.1 0.12 ± 21% perf-profile.children.cycles-pp.__vm_munmap
0.10 ± 22% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.dev_attr_show
0.10 ± 22% -0.1 0.05 ± 72% perf-profile.children.cycles-pp.sysfs_kf_seq_show
0.22 ± 8% -0.0 0.17 ± 17% perf-profile.children.cycles-pp.diskstats_show
0.05 ± 72% +0.1 0.12 ± 23% perf-profile.children.cycles-pp.__fdget_pos
0.07 ± 52% +0.1 0.16 ± 44% perf-profile.children.cycles-pp.local_clock_noinstr
0.06 ±106% +0.1 0.18 ± 32% perf-profile.children.cycles-pp.timerqueue_add
0.00 +0.2 0.17 ± 24% perf-profile.children.cycles-pp.tmigr_inactive_up
0.00 +0.2 0.18 ± 23% perf-profile.children.cycles-pp.tmigr_cpu_deactivate
0.06 ± 23% +0.2 0.25 ± 15% perf-profile.children.cycles-pp.task_work_run
0.00 +0.2 0.19 ± 19% perf-profile.children.cycles-pp.tmigr_cpu_activate
0.00 +0.2 0.20 ± 20% perf-profile.children.cycles-pp.task_mm_cid_work
0.00 +0.4 0.38 ± 17% perf-profile.children.cycles-pp.tmigr_update_events
0.07 ± 24% +0.4 0.49 ± 13% perf-profile.children.cycles-pp.__get_next_timer_interrupt
2.29 ± 4% +0.7 3.02 ± 12% perf-profile.children.cycles-pp.__kmalloc_large_node
2.32 ± 4% +0.7 3.05 ± 13% perf-profile.children.cycles-pp.__kmalloc_node_track_caller
2.09 ± 8% +0.7 2.82 ± 13% perf-profile.children.cycles-pp.get_page_from_freelist
0.00 +0.9 0.86 ± 16% perf-profile.children.cycles-pp.timer_expire_remote
1.25 ± 10% +0.9 2.15 ± 14% perf-profile.children.cycles-pp.rmqueue
1.48 ± 14% +1.0 2.51 ± 13% perf-profile.children.cycles-pp.__free_pages_ok
0.58 ± 29% +1.1 1.66 ± 14% perf-profile.children.cycles-pp.free_one_page
3.00 ± 6% +1.1 4.11 ± 17% perf-profile.children.cycles-pp.ip_finish_output2
2.88 ± 6% +1.1 4.00 ± 17% perf-profile.children.cycles-pp.__dev_queue_xmit
2.23 ± 10% +1.3 3.50 ± 19% perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase
0.00 +1.4 1.40 ± 8% perf-profile.children.cycles-pp.tmigr_handle_remote_up
0.00 +1.4 1.43 ± 9% perf-profile.children.cycles-pp.tmigr_handle_remote
2.43 ± 9% +1.5 3.91 ± 18% perf-profile.children.cycles-pp.do_softirq
2.46 ± 9% +1.5 3.95 ± 18% perf-profile.children.cycles-pp.__local_bh_enable_ip
2.16 ± 9% +1.7 3.87 ± 13% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
3.53 ± 7% +1.8 5.30 ± 14% perf-profile.children.cycles-pp.sctp_ulpevent_free
5.02 ± 5% +5.6 10.65 ± 15% perf-profile.children.cycles-pp.ret_from_fork_asm
5.00 ± 4% +5.6 10.64 ± 15% perf-profile.children.cycles-pp.ret_from_fork
4.94 ± 5% +5.6 10.58 ± 16% perf-profile.children.cycles-pp.kthread
2.83 ± 9% +5.9 8.74 ± 18% perf-profile.children.cycles-pp.smpboot_thread_fn
1.73 ± 12% +6.2 7.93 ± 18% perf-profile.children.cycles-pp.run_ksoftirqd
0.02 ±141% +10.3 10.31 ± 88% perf-profile.children.cycles-pp.poll_idle
2.41 ± 5% -1.1 1.32 ± 10% perf-profile.self.cycles-pp._raw_spin_lock
1.11 ± 19% -0.7 0.37 ± 20% perf-profile.self.cycles-pp.get_nohz_timer_target
0.84 ± 28% -0.5 0.34 ± 30% perf-profile.self.cycles-pp.update_rq_clock_task
0.65 ± 23% -0.5 0.16 ± 34% perf-profile.self.cycles-pp._nohz_idle_balance
0.29 ± 16% -0.2 0.07 ± 62% perf-profile.self.cycles-pp.ktime_get_update_offsets_now
0.52 ± 19% -0.2 0.31 ± 21% perf-profile.self.cycles-pp.idle_cpu
0.43 ± 9% -0.2 0.23 ± 23% perf-profile.self.cycles-pp.update_rq_clock
0.25 ± 28% -0.1 0.12 ± 50% perf-profile.self.cycles-pp.__memcg_slab_free_hook
0.16 ± 34% -0.1 0.03 ±101% perf-profile.self.cycles-pp.need_update
0.16 ± 32% -0.1 0.06 ± 80% perf-profile.self.cycles-pp.call_cpuidle
0.12 ± 19% -0.1 0.04 ±101% perf-profile.self.cycles-pp.sctp_inq_pop
0.23 ± 22% -0.1 0.15 ± 42% perf-profile.self.cycles-pp.sctp_check_transmitted
0.13 ± 47% -0.1 0.06 ± 55% perf-profile.self.cycles-pp.all_vm_events
0.19 ± 14% -0.1 0.12 ± 25% perf-profile.self.cycles-pp.filemap_map_pages
0.05 ± 72% +0.1 0.11 ± 24% perf-profile.self.cycles-pp.__fdget_pos
0.08 ± 54% +0.1 0.16 ± 22% perf-profile.self.cycles-pp.sctp_skb_recv_datagram
0.00 +0.1 0.09 ± 31% perf-profile.self.cycles-pp.tmigr_cpu_activate
0.00 +0.2 0.18 ± 26% perf-profile.self.cycles-pp.task_mm_cid_work
0.02 ±141% +10.1 10.08 ± 90% perf-profile.self.cycles-pp.poll_idle
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-01 8:09 [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression kernel test robot @ 2024-03-04 0:32 ` Frederic Weisbecker 2024-03-04 2:13 ` Oliver Sang 2024-03-12 23:57 ` Frederic Weisbecker 1 sibling, 1 reply; 10+ messages in thread From: Frederic Weisbecker @ 2024-03-04 0:32 UTC (permalink / raw) To: kernel test robot Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, ying.huang, feng.tang, fengwei.yin Le Fri, Mar 01, 2024 at 04:09:24PM +0800, kernel test robot a écrit : > > > Hello, > > kernel test robot noticed a -1.2% regression of netperf.Throughput_Mbps on: > > > commit: 7ee988770326fca440472200c3eb58935fe712f6 ("timers: Implement the hierarchical pull model") > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git timers/core > > testcase: netperf > test machine: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory > parameters: > > ip: ipv4 > runtime: 300s > nr_threads: 200% > cluster: cs-localhost > test: SCTP_STREAM > cpufreq_governor: performance > > > > > If you fix the issue in a separate patch/commit (i.e. not just a new version of > the same patch/commit), kindly add following tags > | Reported-by: kernel test robot <oliver.sang@intel.com> > | Closes: https://lore.kernel.org/oe-lkp/202403011511.24defbbd-oliver.sang@intel.com > > > Details are as below: > --------------------------------------------------------------------------------------------------> > > > The kernel config and materials to reproduce are available at: > https://download.01.org/0day-ci/archive/20240301/202403011511.24defbbd-oliver.sang@intel.com > > ========================================================================================= > cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: > cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/200%/debian-12-x86_64-20240206.cgz/300s/lkp-icl-2sp2/SCTP_STREAM/netperf > > commit: > 57e95a5c41 ("timers: Introduce function to check timer base is_idle flag") > 7ee9887703 ("timers: Implement the hierarchical pull model") Is this something that is observed also with the commits that follow in this branch? Ie: would it be possible to compare instead: 57e95a5c4117 (timers: Introduce function to check timer base is_idle flag) VS b2cf7507e186 (timers: Always queue timers on the local CPU) Because the improvements introduced by 7ee9887703 are mostly relevant after b2cf7507e186. Thanks. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-04 0:32 ` Frederic Weisbecker @ 2024-03-04 2:13 ` Oliver Sang 2024-03-04 11:28 ` Frederic Weisbecker 0 siblings, 1 reply; 10+ messages in thread From: Oliver Sang @ 2024-03-04 2:13 UTC (permalink / raw) To: Frederic Weisbecker Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, ying.huang, feng.tang, fengwei.yin, oliver.sang hi, Frederic Weisbecker, On Mon, Mar 04, 2024 at 01:32:45AM +0100, Frederic Weisbecker wrote: > Le Fri, Mar 01, 2024 at 04:09:24PM +0800, kernel test robot a écrit : > > > > > > Hello, > > > > kernel test robot noticed a -1.2% regression of netperf.Throughput_Mbps on: > > > > > > commit: 7ee988770326fca440472200c3eb58935fe712f6 ("timers: Implement the hierarchical pull model") > > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git timers/core > > > > testcase: netperf > > test machine: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory > > parameters: > > > > ip: ipv4 > > runtime: 300s > > nr_threads: 200% > > cluster: cs-localhost > > test: SCTP_STREAM > > cpufreq_governor: performance > > > > > > > > > > If you fix the issue in a separate patch/commit (i.e. not just a new version of > > the same patch/commit), kindly add following tags > > | Reported-by: kernel test robot <oliver.sang@intel.com> > > | Closes: https://lore.kernel.org/oe-lkp/202403011511.24defbbd-oliver.sang@intel.com > > > > > > Details are as below: > > --------------------------------------------------------------------------------------------------> > > > > > > The kernel config and materials to reproduce are available at: > > https://download.01.org/0day-ci/archive/20240301/202403011511.24defbbd-oliver.sang@intel.com > > > > ========================================================================================= > > cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: > > cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/200%/debian-12-x86_64-20240206.cgz/300s/lkp-icl-2sp2/SCTP_STREAM/netperf > > > > commit: > > 57e95a5c41 ("timers: Introduce function to check timer base is_idle flag") > > 7ee9887703 ("timers: Implement the hierarchical pull model") > > Is this something that is observed also with the commits that follow in this > branch? when this bisect done, we also tested the tip of timers/core branch at that time 8b3843ae3634b vdso/datapage: Quick fix - use asm/page-def.h for ARM64 the regression still exists on it: 57e95a5c4117dc6a 7ee988770326fca440472200c3e 8b3843ae3634b472530fb69c386 ---------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 4.10 -1.2% 4.05 -1.2% 4.05 netperf.ThroughputBoth_Mbps 1049 -1.2% 1037 -1.2% 1036 netperf.ThroughputBoth_total_Mbps 4.10 -1.2% 4.05 -1.2% 4.05 netperf.Throughput_Mbps 1049 -1.2% 1037 -1.2% 1036 netperf.Throughput_total_Mbps > > Ie: would it be possible to compare instead: > > 57e95a5c4117 (timers: Introduce function to check timer base is_idle flag) > VS > b2cf7507e186 (timers: Always queue timers on the local CPU) > > Because the improvements introduced by 7ee9887703 are mostly relevant after > b2cf7507e186. got it. will test. at the same time, we noticed current tip of timers/core is a184d9835a0a6 (tip/timers/core) tick/sched: Fix build failure for CONFIG_NO_HZ_COMMON=n though it seems irelevant, we will still get data for it. > > Thanks. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-04 2:13 ` Oliver Sang @ 2024-03-04 11:28 ` Frederic Weisbecker 2024-03-05 2:17 ` Oliver Sang 0 siblings, 1 reply; 10+ messages in thread From: Frederic Weisbecker @ 2024-03-04 11:28 UTC (permalink / raw) To: Oliver Sang Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, ying.huang, feng.tang, fengwei.yin Le Mon, Mar 04, 2024 at 10:13:00AM +0800, Oliver Sang a écrit : > On Mon, Mar 04, 2024 at 01:32:45AM +0100, Frederic Weisbecker wrote: > > Le Fri, Mar 01, 2024 at 04:09:24PM +0800, kernel test robot a écrit : > > > commit: > > > 57e95a5c41 ("timers: Introduce function to check timer base is_idle flag") > > > 7ee9887703 ("timers: Implement the hierarchical pull model") > > > > Is this something that is observed also with the commits that follow in this > > branch? > > when this bisect done, we also tested the tip of timers/core branch at that time > 8b3843ae3634b vdso/datapage: Quick fix - use asm/page-def.h for ARM64 > > the regression still exists on it: > > 57e95a5c4117dc6a 7ee988770326fca440472200c3e 8b3843ae3634b472530fb69c386 > ---------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev > \ | \ | \ > 4.10 -1.2% 4.05 -1.2% 4.05 netperf.ThroughputBoth_Mbps > 1049 -1.2% 1037 -1.2% 1036 netperf.ThroughputBoth_total_Mbps > 4.10 -1.2% 4.05 -1.2% 4.05 netperf.Throughput_Mbps > 1049 -1.2% 1037 -1.2% 1036 netperf.Throughput_total_Mbps Oh, I see... :-/ > > Ie: would it be possible to compare instead: > > > > 57e95a5c4117 (timers: Introduce function to check timer base is_idle flag) > > VS > > b2cf7507e186 (timers: Always queue timers on the local CPU) > > > > Because the improvements introduced by 7ee9887703 are mostly relevant after > > b2cf7507e186. > > got it. will test. > > at the same time, we noticed current tip of timers/core is > a184d9835a0a6 (tip/timers/core) tick/sched: Fix build failure for > CONFIG_NO_HZ_COMMON=n Shouldn't be a problem as it fixes an issue introduced after: b2cf7507e186 (timers: Always queue timers on the local CPU) > > though it seems irelevant, we will still get data for it. Thanks a lot, this will be very helpful. Especially with all the perf diff details like in the initial email report. Because I'm having some troubles running those lkp tests. How is it working BTW? I've seen it downloading two kernel trees but I haven't noticed a kernel build. Are the two compared instances running through kvm? Thanks. > > > > > Thanks. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-04 11:28 ` Frederic Weisbecker @ 2024-03-05 2:17 ` Oliver Sang 2024-03-05 10:46 ` Frederic Weisbecker ` (2 more replies) 0 siblings, 3 replies; 10+ messages in thread From: Oliver Sang @ 2024-03-05 2:17 UTC (permalink / raw) To: Frederic Weisbecker Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, ying.huang, feng.tang, fengwei.yin hi, Frederic Weisbecker, On Mon, Mar 04, 2024 at 12:28:33PM +0100, Frederic Weisbecker wrote: > Le Mon, Mar 04, 2024 at 10:13:00AM +0800, Oliver Sang a écrit : > > On Mon, Mar 04, 2024 at 01:32:45AM +0100, Frederic Weisbecker wrote: > > > Le Fri, Mar 01, 2024 at 04:09:24PM +0800, kernel test robot a écrit : > > > > commit: > > > > 57e95a5c41 ("timers: Introduce function to check timer base is_idle flag") > > > > 7ee9887703 ("timers: Implement the hierarchical pull model") > > > > > > Is this something that is observed also with the commits that follow in this > > > branch? > > > > when this bisect done, we also tested the tip of timers/core branch at that time > > 8b3843ae3634b vdso/datapage: Quick fix - use asm/page-def.h for ARM64 > > > > the regression still exists on it: > > > > 57e95a5c4117dc6a 7ee988770326fca440472200c3e 8b3843ae3634b472530fb69c386 > > ---------------- --------------------------- --------------------------- > > %stddev %change %stddev %change %stddev > > \ | \ | \ > > 4.10 -1.2% 4.05 -1.2% 4.05 netperf.ThroughputBoth_Mbps > > 1049 -1.2% 1037 -1.2% 1036 netperf.ThroughputBoth_total_Mbps > > 4.10 -1.2% 4.05 -1.2% 4.05 netperf.Throughput_Mbps > > 1049 -1.2% 1037 -1.2% 1036 netperf.Throughput_total_Mbps > > Oh, I see... :-/ > > > > Ie: would it be possible to compare instead: > > > > > > 57e95a5c4117 (timers: Introduce function to check timer base is_idle flag) > > > VS > > > b2cf7507e186 (timers: Always queue timers on the local CPU) > > > > > > Because the improvements introduced by 7ee9887703 are mostly relevant after > > > b2cf7507e186. > > > > got it. will test. > > > > at the same time, we noticed current tip of timers/core is > > a184d9835a0a6 (tip/timers/core) tick/sched: Fix build failure for > > CONFIG_NO_HZ_COMMON=n > > Shouldn't be a problem as it fixes an issue introduced after: > > b2cf7507e186 (timers: Always queue timers on the local CPU) > > > > > though it seems irelevant, we will still get data for it. > > Thanks a lot, this will be very helpful. Especially with all the perf diff > details like in the initial email report. the regression still exists on b2cf7507e186 and current tip of the branch: ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/200%/debian-12-x86_64-20240206.cgz/300s/lkp-icl-2sp2/SCTP_STREAM/netperf commit: 57e95a5c4117 (timers: Introduce function to check timer base is_idle flag) b2cf7507e186 (timers: Always queue timers on the local CPU) a184d9835a0a (tick/sched: Fix build failure for CONFIG_NO_HZ_COMMON=n) a184d9835a0a689261ea6a4a8dbc18173a031b77 57e95a5c4117dc6a b2cf7507e18649a30512515ec0c a184d9835a0a689261ea6a4a8db ---------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 4.10 -1.4% 4.04 -1.5% 4.04 netperf.ThroughputBoth_Mbps 1049 -1.4% 1034 -1.5% 1033 netperf.ThroughputBoth_total_Mbps 4.10 -1.4% 4.04 -1.5% 4.04 netperf.Throughput_Mbps 1049 -1.4% 1034 -1.5% 1033 netperf.Throughput_total_Mbps details are in below [1] > Because I'm having some troubles > running those lkp tests. How is it working BTW? I've seen it downloading > two kernel trees but I haven't noticed a kernel build. you need build 7ee9887703 and its parent kernel with config in https://download.01.org/0day-ci/archive/20240301/202403011511.24defbbd-oliver.sang@intel.com then boot into kernel. after that, you could run netperf in each kernel by following https://download.01.org/0day-ci/archive/20240301/202403011511.24defbbd-oliver.sang@intel.com/reproduce to get data. the results will store in different path according the kernel commit, then you could compare the results from both kernels. what's your OS BTW? we cannot support all distributions so far... > Are the two compared > instances running through kvm? we run performance tests on bare mental. for netperf, we just test on one machine so the test is really upon local net. > > Thanks. > > > > > > > > > Thanks. > [1] ========================================================================================= cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/200%/debian-12-x86_64-20240206.cgz/300s/lkp-icl-2sp2/SCTP_STREAM/netperf commit: 57e95a5c4117 (timers: Introduce function to check timer base is_idle flag) b2cf7507e186 (timers: Always queue timers on the local CPU) a184d9835a0a (tick/sched: Fix build failure for CONFIG_NO_HZ_COMMON=n) a184d9835a0a689261ea6a4a8dbc18173a031b77 57e95a5c4117dc6a b2cf7507e18649a30512515ec0c a184d9835a0a689261ea6a4a8db ---------------- --------------------------- --------------------------- %stddev %change %stddev %change %stddev \ | \ | \ 1364607 +11.8% 1525991 +10.3% 1504946 cpuidle..usage 45.86 ± 4% +8.4% 49.70 ± 5% +3.5% 47.46 ± 6% boot-time.boot 5430 ± 5% +9.0% 5921 ± 5% +3.8% 5636 ± 6% boot-time.idle 0.03 +0.0 0.04 +0.0 0.04 mpstat.cpu.all.soft% 0.04 +0.0 0.08 +0.0 0.08 ± 2% mpstat.cpu.all.sys% 4.14 -8.9% 3.77 ± 2% -8.3% 3.79 ± 2% mpstat.max_utilization_pct 20726 ± 63% +246.1% 71744 ± 53% +68.2% 34867 ± 72% numa-numastat.node0.other_node 1431327 ± 7% +13.9% 1630876 ± 3% +14.8% 1643375 ± 7% numa-numastat.node1.numa_hit 37532 ± 35% +62.1% 60841 ± 63% +160.8% 97891 ± 25% numa-numastat.node1.other_node 201.33 ± 3% -28.5% 144.00 ± 8% -26.8% 147.40 ± 11% perf-c2c.DRAM.remote 187.83 ± 3% -21.4% 147.67 ± 6% -37.0% 118.40 ± 11% perf-c2c.HITM.local 40.67 ± 7% -54.5% 18.50 ± 19% -59.7% 16.40 ± 11% perf-c2c.HITM.remote 1.36 ± 4% +10.7% 1.51 ± 3% +12.7% 1.53 ± 5% vmstat.procs.r 5654 -1.9% 5549 -2.8% 5497 vmstat.system.cs 5232 +10.7% 5790 +8.8% 5690 vmstat.system.in 15247 ± 6% -5.0% 14490 ± 9% -8.8% 13903 ± 5% numa-meminfo.node0.PageTables 12499 ± 6% +115.6% 26951 ± 3% +118.3% 27288 ± 2% numa-meminfo.node1.Active 12489 ± 6% +115.7% 26940 ± 3% +118.2% 27249 ± 2% numa-meminfo.node1.Active(anon) 12488 ± 6% +114.2% 26754 ± 3% +118.2% 27255 ± 2% numa-meminfo.node1.Shmem 102.17 ± 8% +906.2% 1028 ± 5% +910.1% 1032 ± 5% time.involuntary_context_switches 2.07 ± 3% +388.3% 10.12 +391.0% 10.17 ± 2% time.system_time 8.02 -74.1% 2.08 ± 3% -74.1% 2.08 ± 3% time.user_time 186981 -1.2% 184713 -1.4% 184450 time.voluntary_context_switches 16672 ± 2% +86.5% 31090 +87.5% 31255 ± 2% meminfo.Active 16611 ± 2% +86.8% 31026 +87.8% 31191 ± 2% meminfo.Active(anon) 183431 ± 28% +10.7% 203091 ± 25% +36.3% 249972 meminfo.AnonHugePages 29722 +10.5% 32854 +11.0% 32990 meminfo.Mapped 19919 +72.9% 34441 +73.8% 34613 meminfo.Shmem 4.10 -1.4% 4.04 -1.5% 4.04 netperf.ThroughputBoth_Mbps 1049 -1.4% 1034 -1.5% 1033 netperf.ThroughputBoth_total_Mbps 4.10 -1.4% 4.04 -1.5% 4.04 netperf.Throughput_Mbps 1049 -1.4% 1034 -1.5% 1033 netperf.Throughput_total_Mbps 102.17 ± 8% +906.2% 1028 ± 5% +910.1% 1032 ± 5% netperf.time.involuntary_context_switches 2.07 ± 3% +388.3% 10.12 +391.0% 10.17 ± 2% netperf.time.system_time 186981 -1.2% 184713 -1.4% 184450 netperf.time.voluntary_context_switches 3820 ± 6% -5.1% 3624 ± 9% -8.9% 3481 ± 5% numa-vmstat.node0.nr_page_table_pages 20726 ± 63% +246.1% 71744 ± 53% +68.2% 34867 ± 72% numa-vmstat.node0.numa_other 3103 ± 6% +116.4% 6718 ± 3% +118.7% 6786 ± 2% numa-vmstat.node1.nr_active_anon 3103 ± 6% +115.0% 6672 ± 3% +118.7% 6788 ± 2% numa-vmstat.node1.nr_shmem 3103 ± 6% +116.4% 6718 ± 3% +118.7% 6786 ± 2% numa-vmstat.node1.nr_zone_active_anon 1429007 ± 7% +14.0% 1628956 ± 3% +14.8% 1640988 ± 7% numa-vmstat.node1.numa_hit 37530 ± 35% +62.1% 60838 ± 63% +160.8% 97889 ± 25% numa-vmstat.node1.numa_other 256.26 ± 33% +65.6% 424.24 ± 16% +96.5% 503.61 ± 17% sched_debug.cfs_rq:/.avg_vruntime.min 256.26 ± 33% +65.6% 424.24 ± 16% +96.5% 503.61 ± 17% sched_debug.cfs_rq:/.min_vruntime.min 965754 -9.5% 874314 -9.7% 872201 sched_debug.cpu.avg_idle.avg 101797 ± 12% +41.8% 144325 ± 5% +45.3% 147947 sched_debug.cpu.avg_idle.stddev 129.98 ± 6% -4.0% 124.77 ± 8% -8.9% 118.35 ± 7% sched_debug.cpu.curr->pid.avg 0.00 ± 22% -43.6% 0.00 ± 5% -37.0% 0.00 ± 11% sched_debug.cpu.next_balance.stddev 0.03 ± 6% -8.6% 0.03 ± 7% -14.7% 0.03 ± 10% sched_debug.cpu.nr_running.avg 39842 ± 27% -32.4% 26922 ± 11% -30.4% 27714 ± 9% sched_debug.cpu.nr_switches.max 886.06 ± 18% +347.6% 3965 ± 4% +366.2% 4130 ± 4% sched_debug.cpu.nr_switches.min 5474 ± 12% -32.0% 3724 ± 8% -32.9% 3672 ± 3% sched_debug.cpu.nr_switches.stddev 4178 ± 2% +86.3% 7784 +87.3% 7826 ± 2% proc-vmstat.nr_active_anon 7649 +10.3% 8436 +10.7% 8469 proc-vmstat.nr_mapped 4984 +72.9% 8620 +73.8% 8663 proc-vmstat.nr_shmem 28495 +1.7% 28987 +1.8% 29014 proc-vmstat.nr_slab_reclaimable 100999 +4.5% 105530 +4.6% 105612 proc-vmstat.nr_slab_unreclaimable 4178 ± 2% +86.3% 7784 +87.3% 7826 ± 2% proc-vmstat.nr_zone_active_anon 3064698 +4.4% 3200938 +4.2% 3193539 proc-vmstat.numa_hit 3006439 +2.1% 3068397 +1.8% 3060785 proc-vmstat.numa_local 58258 +127.6% 132587 +127.9% 132758 proc-vmstat.numa_other 8114 ± 2% +63.2% 13244 ± 4% +62.6% 13190 ± 2% proc-vmstat.pgactivate 986600 +1.2% 998606 +0.9% 995307 proc-vmstat.pgfault 20.00 +1905.0% 401.00 ± 79% +2050.0% 430.00 ± 79% proc-vmstat.unevictable_pgs_culled 15.14 +17.0% 17.72 +17.4% 17.77 perf-stat.i.MPKI 1.702e+08 +3.5% 1.762e+08 +3.3% 1.758e+08 perf-stat.i.branch-instructions 1.68 +0.1 1.80 +0.1 1.81 perf-stat.i.branch-miss-rate% 7174339 +1.2% 7262760 +1.4% 7276699 perf-stat.i.branch-misses 18.46 +3.4 21.86 +3.4 21.87 perf-stat.i.cache-miss-rate% 4047319 +20.6% 4880009 +20.4% 4874638 perf-stat.i.cache-misses 22007366 +2.6% 22586331 +2.5% 22565036 perf-stat.i.cache-references 5620 -1.6% 5532 -2.5% 5482 perf-stat.i.context-switches 1.84 +17.0% 2.15 +16.5% 2.14 perf-stat.i.cpi 9.159e+08 +12.8% 1.033e+09 +12.4% 1.03e+09 perf-stat.i.cpu-cycles 161.08 +193.1% 472.19 ± 2% +192.1% 470.47 ± 4% perf-stat.i.cpu-migrations 8.434e+08 +3.1% 8.692e+08 +2.9% 8.677e+08 perf-stat.i.instructions 0.61 -8.5% 0.56 -8.2% 0.56 perf-stat.i.ipc 4.79 +17.0% 5.60 +17.2% 5.61 perf-stat.overall.MPKI 4.21 -0.1 4.12 -0.1 4.14 perf-stat.overall.branch-miss-rate% 18.39 +3.2 21.60 +3.2 21.59 perf-stat.overall.cache-miss-rate% 1.09 +9.4% 1.19 +9.3% 1.19 perf-stat.overall.cpi 227.07 -6.6% 212.18 -6.7% 211.77 perf-stat.overall.cycles-between-cache-misses 0.92 -8.6% 0.84 -8.5% 0.84 perf-stat.overall.ipc 1379820 +4.4% 1440927 +4.4% 1440267 perf-stat.overall.path-length 1.702e+08 +3.4% 1.76e+08 +3.1% 1.755e+08 perf-stat.ps.branch-instructions 7172548 +1.2% 7256062 +1.3% 7264905 perf-stat.ps.branch-misses 4035389 +20.5% 4864355 +20.4% 4858306 perf-stat.ps.cache-misses 21948285 +2.6% 22521305 +2.5% 22497443 perf-stat.ps.cache-references 5603 -1.6% 5514 -2.5% 5463 perf-stat.ps.context-switches 9.163e+08 +12.6% 1.032e+09 +12.3% 1.029e+09 perf-stat.ps.cpu-cycles 160.61 +193.0% 470.58 ± 2% +191.9% 468.79 ± 4% perf-stat.ps.cpu-migrations 8.433e+08 +3.0% 8.685e+08 +2.7% 8.665e+08 perf-stat.ps.instructions 2.551e+11 +3.3% 2.636e+11 +3.1% 2.631e+11 perf-stat.total.instructions 31.82 ± 3% -13.0 18.83 ± 12% -13.2 18.65 ± 6% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter 36.90 ± 2% -12.6 24.32 ± 10% -12.3 24.62 ± 5% perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 27.61 ± 3% -12.5 15.09 ± 14% -12.4 15.24 ± 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state 31.75 ± 2% -12.0 19.79 ± 12% -11.6 20.11 ± 5% perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call 24.39 ± 3% -10.7 13.65 ± 15% -10.4 13.94 ± 5% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter 24.33 ± 3% -10.7 13.60 ± 15% -10.4 13.92 ± 5% perf-profile.calltrace.cycles-pp.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt 21.00 ± 3% -9.8 11.23 ± 16% -9.6 11.40 ± 4% perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 20.97 ± 3% -9.7 11.23 ± 16% -9.6 11.40 ± 4% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt 20.97 ± 3% -9.7 11.23 ± 16% -9.6 11.39 ± 4% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.irq_exit_rcu 4.92 ± 10% -4.3 0.64 ± 17% -4.2 0.71 ± 10% perf-profile.calltrace.cycles-pp._nohz_idle_balance.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify 24.46 ± 3% -3.2 21.26 ± 9% -2.6 21.84 ± 6% perf-profile.calltrace.cycles-pp.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog 24.91 ± 3% -3.2 21.73 ± 8% -2.6 22.28 ± 6% perf-profile.calltrace.cycles-pp.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action.__do_softirq 24.57 ± 3% -3.2 21.42 ± 8% -2.6 21.93 ± 6% perf-profile.calltrace.cycles-pp.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll.net_rx_action 24.55 ± 3% -3.1 21.41 ± 8% -2.6 21.91 ± 6% perf-profile.calltrace.cycles-pp.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core.process_backlog.__napi_poll 21.78 ± 3% -2.7 19.06 ± 8% -2.4 19.42 ± 7% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu 22.48 ± 3% -2.6 19.90 ± 9% -2.2 20.33 ± 8% perf-profile.calltrace.cycles-pp.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish.__netif_receive_skb_one_core 21.92 ± 3% -2.6 19.35 ± 8% -2.1 19.85 ± 8% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv.ip_protocol_deliver_rcu.ip_local_deliver_finish 9.76 ± 7% -2.3 7.48 ± 9% -2.1 7.71 ± 7% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv 5.08 ± 7% -1.7 3.36 ± 9% -1.7 3.39 ± 9% perf-profile.calltrace.cycles-pp.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv 10.73 ± 3% -1.6 9.09 ± 7% -1.2 9.56 ± 6% perf-profile.calltrace.cycles-pp._copy_from_iter.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg 10.86 ± 3% -1.6 9.29 ± 7% -1.1 9.71 ± 6% perf-profile.calltrace.cycles-pp.sctp_user_addto_chunk.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg 12.22 ± 3% -1.2 11.01 ± 6% -0.7 11.55 ± 10% perf-profile.calltrace.cycles-pp.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg.___sys_sendmsg 3.42 ± 9% -0.7 2.75 ± 7% -0.5 2.94 ± 10% perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_packet_transmit_chunk.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter 3.09 ± 10% -0.7 2.43 ± 7% -0.5 2.61 ± 9% perf-profile.calltrace.cycles-pp.sctp_packet_pack.sctp_packet_transmit.sctp_packet_transmit_chunk.sctp_outq_flush_data.sctp_outq_flush 3.05 ± 11% -0.7 2.39 ± 7% -0.5 2.57 ± 10% perf-profile.calltrace.cycles-pp.__memcpy.sctp_packet_pack.sctp_packet_transmit.sctp_packet_transmit_chunk.sctp_outq_flush_data 3.44 ± 9% -0.7 2.79 ± 8% -0.4 3.00 ± 9% perf-profile.calltrace.cycles-pp.sctp_packet_transmit_chunk.sctp_outq_flush_data.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm 1.05 ± 9% -0.6 0.41 ± 72% -0.5 0.59 ± 9% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_generate_timeout_event.call_timer_fn.__run_timers 1.58 ± 5% -0.5 1.04 ± 33% -0.6 0.97 ± 27% perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt 1.66 ± 7% -0.5 1.12 ± 34% -0.6 1.01 ± 28% perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter 0.71 ± 11% -0.5 0.20 ±144% -0.3 0.37 ± 82% perf-profile.calltrace.cycles-pp.setlocale 0.95 ± 14% -0.5 0.50 ± 76% -0.6 0.38 ± 84% perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 2.49 ± 7% -0.4 2.10 ± 14% -0.4 2.08 ± 9% perf-profile.calltrace.cycles-pp.__memcpy.skb_copy_bits.skb_copy.sctp_make_reassembled_event.sctp_ulpq_partial_delivery 2.49 ± 7% -0.4 2.10 ± 14% -0.4 2.12 ± 9% perf-profile.calltrace.cycles-pp.skb_copy_bits.skb_copy.sctp_make_reassembled_event.sctp_ulpq_partial_delivery.sctp_cmd_interpreter 0.90 ± 14% -0.4 0.52 ± 45% -0.5 0.37 ± 86% perf-profile.calltrace.cycles-pp.__schedule.schedule.smpboot_thread_fn.kthread.ret_from_fork 2.40 ± 8% -0.4 2.04 ± 11% -0.4 2.04 ± 9% perf-profile.calltrace.cycles-pp.sctp_outq_sack.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv.sctp_rcv 1.62 ± 4% -0.3 1.28 ± 16% -0.3 1.35 ± 12% perf-profile.calltrace.cycles-pp.read 1.70 ± 11% -0.3 1.36 ± 16% -0.4 1.29 ± 13% perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm 1.53 ± 4% -0.3 1.19 ± 15% -0.3 1.27 ± 10% perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 2.81 ± 5% -0.3 2.48 ± 13% -0.3 2.49 ± 11% perf-profile.calltrace.cycles-pp.sctp_make_reassembled_event.sctp_ulpq_partial_delivery.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv 1.53 ± 4% -0.3 1.20 ± 16% -0.3 1.27 ± 10% perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 2.74 ± 6% -0.3 2.42 ± 13% -0.3 2.43 ± 10% perf-profile.calltrace.cycles-pp.skb_copy.sctp_make_reassembled_event.sctp_ulpq_partial_delivery.sctp_cmd_interpreter.sctp_do_sm 1.57 ± 5% -0.3 1.24 ± 16% -0.3 1.30 ± 11% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read 1.57 ± 5% -0.3 1.25 ± 16% -0.3 1.30 ± 11% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read 0.90 ± 14% -0.3 0.62 ± 9% -0.5 0.37 ± 86% perf-profile.calltrace.cycles-pp.schedule.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.76 ± 8% -0.3 1.47 ± 8% -0.3 1.50 ± 6% perf-profile.calltrace.cycles-pp.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.77 ± 7% -0.3 1.48 ± 9% -0.3 1.50 ± 6% perf-profile.calltrace.cycles-pp.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.77 ± 7% -0.3 1.48 ± 9% -0.3 1.50 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.execve 1.77 ± 7% -0.3 1.48 ± 9% -0.3 1.50 ± 6% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.execve 1.77 ± 7% -0.3 1.48 ± 9% -0.3 1.51 ± 6% perf-profile.calltrace.cycles-pp.execve 1.07 ± 14% -0.3 0.80 ± 20% -0.2 0.83 ± 14% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault 0.96 ± 8% -0.2 0.71 ± 18% -0.2 0.79 ± 17% perf-profile.calltrace.cycles-pp.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.96 ± 8% -0.2 0.71 ± 18% -0.2 0.79 ± 17% perf-profile.calltrace.cycles-pp.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 0.96 ± 8% -0.2 0.71 ± 18% -0.2 0.79 ± 17% perf-profile.calltrace.cycles-pp.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.03 ± 11% -0.2 0.79 ± 8% -0.2 0.81 ± 5% perf-profile.calltrace.cycles-pp.load_elf_binary.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common 1.06 ± 9% -0.2 0.83 ± 6% -0.2 0.83 ± 6% perf-profile.calltrace.cycles-pp.search_binary_handler.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve 1.06 ± 9% -0.2 0.83 ± 6% -0.2 0.84 ± 5% perf-profile.calltrace.cycles-pp.exec_binprm.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64 1.24 ± 9% -0.2 1.00 ± 10% -0.1 1.16 ± 15% perf-profile.calltrace.cycles-pp.read_counters.process_interval.dispatch_events.cmd_stat.run_builtin 1.12 ± 9% -0.2 0.89 ± 13% -0.2 0.87 ± 11% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single 1.38 ± 6% -0.2 1.16 ± 12% -0.2 1.14 ± 10% perf-profile.calltrace.cycles-pp.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter 1.31 ± 7% -0.2 1.09 ± 14% -0.2 1.09 ± 9% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt 1.14 ± 10% -0.2 0.95 ± 5% -0.2 0.95 ± 3% perf-profile.calltrace.cycles-pp.bprm_execve.do_execveat_common.__x64_sys_execve.do_syscall_64.entry_SYSCALL_64_after_hwframe 1.25 ± 9% -0.1 1.10 ± 7% -0.3 0.95 ± 4% perf-profile.calltrace.cycles-pp.tick_nohz_idle_stop_tick.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary 1.23 ± 8% -0.1 1.09 ± 7% -0.3 0.94 ± 4% perf-profile.calltrace.cycles-pp.tick_nohz_stop_tick.tick_nohz_idle_stop_tick.cpuidle_idle_call.do_idle.cpu_startup_entry 0.85 ± 8% -0.1 0.75 ± 14% -0.1 0.74 ± 14% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.__sysvec_call_function_single.sysvec_call_function_single 0.85 ± 17% +0.2 1.02 ± 10% +0.3 1.18 ± 14% perf-profile.calltrace.cycles-pp.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.sock_recvmsg.____sys_recvmsg 1.04 ± 5% +0.2 1.28 ± 7% +0.1 1.15 ± 10% perf-profile.calltrace.cycles-pp.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc 1.16 ± 5% +0.3 1.42 ± 7% +0.4 1.57 ± 39% perf-profile.calltrace.cycles-pp._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg 0.92 ± 8% +0.3 1.19 ± 7% +0.1 1.05 ± 10% perf-profile.calltrace.cycles-pp.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb._sctp_make_chunk 0.92 ± 7% +0.3 1.20 ± 7% +0.1 1.06 ± 10% perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty 0.93 ± 7% +0.3 1.22 ± 8% +0.1 1.08 ± 10% perf-profile.calltrace.cycles-pp.kmalloc_reserve.__alloc_skb._sctp_make_chunk.sctp_make_datafrag_empty.sctp_datamsg_from_user 1.21 ± 7% +0.3 1.52 ± 7% +0.4 1.65 ± 36% perf-profile.calltrace.cycles-pp.sctp_make_datafrag_empty.sctp_datamsg_from_user.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg 0.84 ± 13% +0.3 1.17 ± 2% +0.5 1.38 ± 7% perf-profile.calltrace.cycles-pp.sctp_wait_for_sndbuf.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg.___sys_sendmsg 1.14 ± 21% +0.4 1.50 ± 19% +0.4 1.55 ± 10% perf-profile.calltrace.cycles-pp.sctp_chunk_put.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.sock_recvmsg 0.45 ± 72% +0.4 0.82 ± 12% +0.5 0.97 ± 11% perf-profile.calltrace.cycles-pp.skb_release_data.kfree_skb_reason.sctp_recvmsg.inet_recvmsg.sock_recvmsg 0.00 +0.4 0.42 ± 71% +0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_generate_timeout_event.call_timer_fn.__run_timers.timer_expire_remote 0.82 ± 45% +0.4 1.26 ± 22% +0.5 1.31 ± 10% perf-profile.calltrace.cycles-pp.consume_skb.sctp_chunk_put.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg 0.75 ± 46% +0.4 1.18 ± 22% +0.5 1.22 ± 11% perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.sctp_chunk_put.sctp_ulpevent_free.sctp_recvmsg 0.09 ±223% +0.5 0.55 ± 62% +0.7 0.74 ± 30% perf-profile.calltrace.cycles-pp.update_sg_lb_stats.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains 0.50 ± 45% +0.5 0.98 ± 33% +0.6 1.07 ± 25% perf-profile.calltrace.cycles-pp.rebalance_domains._nohz_idle_balance.__do_softirq.irq_exit_rcu.sysvec_call_function_single 0.30 ±101% +0.6 0.86 ± 25% +0.6 0.93 ± 19% perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.consume_skb.sctp_chunk_put.sctp_ulpevent_free 2.88 ± 8% +0.6 3.45 ± 16% +0.7 3.60 ± 12% perf-profile.calltrace.cycles-pp.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state 0.08 ±223% +0.6 0.67 ± 6% +0.7 0.81 ± 21% perf-profile.calltrace.cycles-pp.schedule.schedule_timeout.sctp_wait_for_sndbuf.sctp_sendmsg_to_asoc.sctp_sendmsg 0.00 +0.6 0.62 ± 12% +0.4 0.35 ± 82% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.rmqueue.get_page_from_freelist.__alloc_pages.__kmalloc_large_node 1.08 ± 12% +0.6 1.70 ± 21% +0.8 1.91 ± 15% perf-profile.calltrace.cycles-pp.irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt.acpi_idle_enter 1.08 ± 12% +0.6 1.70 ± 21% +0.8 1.89 ± 15% perf-profile.calltrace.cycles-pp._nohz_idle_balance.__do_softirq.irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single 0.00 +0.6 0.63 ± 60% +0.8 0.84 ± 26% perf-profile.calltrace.cycles-pp.update_sd_lb_stats.find_busiest_group.load_balance.rebalance_domains._nohz_idle_balance 1.08 ± 12% +0.6 1.70 ± 21% +0.8 1.91 ± 15% perf-profile.calltrace.cycles-pp.__do_softirq.irq_exit_rcu.sysvec_call_function_single.asm_sysvec_call_function_single.acpi_safe_halt 0.00 +0.6 0.64 ± 16% +0.7 0.67 ± 4% perf-profile.calltrace.cycles-pp.sctp_generate_timeout_event.call_timer_fn.__run_timers.timer_expire_remote.tmigr_handle_remote_cpu 0.00 +0.6 0.64 ± 8% +0.8 0.78 ± 20% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_timeout.sctp_wait_for_sndbuf.sctp_sendmsg_to_asoc 0.00 +0.6 0.64 ± 61% +0.9 0.86 ± 26% perf-profile.calltrace.cycles-pp.find_busiest_group.load_balance.rebalance_domains._nohz_idle_balance.__do_softirq 3.03 ± 7% +0.6 3.67 ± 16% +0.7 3.70 ± 8% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc 2.81 ± 7% +0.6 3.45 ± 17% +0.6 3.43 ± 9% perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND 2.71 ± 8% +0.7 3.36 ± 16% +0.6 3.34 ± 6% perf-profile.calltrace.cycles-pp.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg 3.12 ± 7% +0.7 3.77 ± 16% +0.7 3.85 ± 7% perf-profile.calltrace.cycles-pp.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg.___sys_sendmsg 0.00 +0.7 0.66 ± 12% +0.6 0.57 ± 53% perf-profile.calltrace.cycles-pp.__free_pages_ok.skb_release_data.consume_skb.sctp_chunk_put.sctp_datamsg_put 2.79 ± 7% +0.7 3.47 ± 16% +0.7 3.48 ± 6% perf-profile.calltrace.cycles-pp.sctp_cmd_interpreter.sctp_do_sm.sctp_primitive_SEND.sctp_sendmsg_to_asoc.sctp_sendmsg 0.00 +0.7 0.70 ± 6% +0.8 0.83 ± 18% perf-profile.calltrace.cycles-pp.schedule_timeout.sctp_wait_for_sndbuf.sctp_sendmsg_to_asoc.sctp_sendmsg.____sys_sendmsg 0.00 +0.7 0.71 ± 17% +0.7 0.73 ± 6% perf-profile.calltrace.cycles-pp.call_timer_fn.__run_timers.timer_expire_remote.tmigr_handle_remote_cpu.tmigr_handle_remote_up 0.18 ±141% +0.7 0.90 ± 37% +0.8 0.98 ± 25% perf-profile.calltrace.cycles-pp.load_balance.rebalance_domains._nohz_idle_balance.__do_softirq.irq_exit_rcu 3.45 ± 14% +0.7 4.18 ± 8% +1.1 4.55 ± 10% perf-profile.calltrace.cycles-pp.__memcpy.skb_copy_bits.skb_copy.sctp_make_reassembled_event.sctp_ulpq_tail_data 0.00 +0.7 0.74 ± 18% +0.8 0.76 ± 8% perf-profile.calltrace.cycles-pp.__run_timers.timer_expire_remote.tmigr_handle_remote_cpu.tmigr_handle_remote_up.tmigr_handle_remote 3.46 ± 14% +0.7 4.21 ± 8% +1.1 4.56 ± 10% perf-profile.calltrace.cycles-pp.skb_copy_bits.skb_copy.sctp_make_reassembled_event.sctp_ulpq_tail_data.sctp_cmd_interpreter 0.00 +0.8 0.75 ± 18% +0.8 0.77 ± 8% perf-profile.calltrace.cycles-pp.timer_expire_remote.tmigr_handle_remote_cpu.tmigr_handle_remote_up.tmigr_handle_remote.__do_softirq 0.77 ± 6% +0.8 1.53 ± 46% +0.1 0.91 ± 9% perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve 0.80 ± 8% +0.8 1.57 ± 44% +0.2 0.96 ± 8% perf-profile.calltrace.cycles-pp.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller.kmalloc_reserve.__alloc_skb 4.44 ± 14% +0.8 5.24 ± 8% +1.1 5.54 ± 9% perf-profile.calltrace.cycles-pp.sctp_make_reassembled_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm.sctp_assoc_bh_rcv 4.33 ± 14% +0.8 5.14 ± 8% +1.1 5.46 ± 9% perf-profile.calltrace.cycles-pp.skb_copy.sctp_make_reassembled_event.sctp_ulpq_tail_data.sctp_cmd_interpreter.sctp_do_sm 0.00 +0.8 0.82 ± 10% +0.8 0.79 ± 19% perf-profile.calltrace.cycles-pp.skb_release_data.consume_skb.sctp_chunk_put.sctp_datamsg_put.sctp_chunk_free 2.34 ± 10% +1.0 3.30 ± 12% +1.1 3.44 ± 7% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip 2.34 ± 9% +1.0 3.30 ± 12% +1.1 3.44 ± 7% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.do_softirq 2.56 ± 8% +1.0 3.52 ± 12% +1.2 3.77 ± 5% perf-profile.calltrace.cycles-pp.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush 2.41 ± 9% +1.0 3.38 ± 12% +1.1 3.55 ± 5% perf-profile.calltrace.cycles-pp.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2 2.42 ± 9% +1.0 3.39 ± 12% +1.1 3.56 ± 5% perf-profile.calltrace.cycles-pp.do_softirq.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit 1.76 ± 14% +1.0 2.74 ± 12% +1.1 2.87 ± 3% perf-profile.calltrace.cycles-pp.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_assoc_rwnd_increase.sctp_ulpevent_free 2.34 ± 10% +1.0 3.32 ± 11% +1.1 3.47 ± 6% perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.do_softirq.__local_bh_enable_ip.__dev_queue_xmit 2.42 ± 9% +1.0 3.40 ± 12% +1.2 3.58 ± 5% perf-profile.calltrace.cycles-pp.__local_bh_enable_ip.__dev_queue_xmit.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit 1.72 ± 15% +1.0 2.70 ± 12% +1.1 2.81 ± 3% perf-profile.calltrace.cycles-pp.ip_finish_output2.__ip_queue_xmit.sctp_packet_transmit.sctp_outq_flush.sctp_assoc_rwnd_increase 0.09 ±223% +1.0 1.10 ± 51% +0.6 0.65 ± 13% perf-profile.calltrace.cycles-pp.rmqueue.get_page_from_freelist.__alloc_pages.__kmalloc_large_node.__kmalloc_node_track_caller 1.88 ± 13% +1.0 2.90 ± 12% +1.2 3.03 ± 4% perf-profile.calltrace.cycles-pp.sctp_packet_transmit.sctp_outq_flush.sctp_assoc_rwnd_increase.sctp_ulpevent_free.sctp_recvmsg 2.23 ± 10% +1.0 3.26 ± 13% +1.2 3.46 ± 6% perf-profile.calltrace.cycles-pp.sctp_assoc_rwnd_increase.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.sock_recvmsg 1.98 ± 13% +1.0 3.02 ± 12% +1.2 3.16 ± 5% perf-profile.calltrace.cycles-pp.sctp_outq_flush.sctp_assoc_rwnd_increase.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg 0.00 +1.2 1.17 ± 17% +1.1 1.13 ± 16% perf-profile.calltrace.cycles-pp.tmigr_handle_remote_cpu.tmigr_handle_remote_up.tmigr_handle_remote.__do_softirq.irq_exit_rcu 0.00 +1.3 1.32 ± 16% +1.4 1.41 ± 19% perf-profile.calltrace.cycles-pp.tmigr_handle_remote_up.tmigr_handle_remote.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt 0.00 +1.3 1.34 ± 17% +1.4 1.42 ± 19% perf-profile.calltrace.cycles-pp.tmigr_handle_remote.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt 3.52 ± 7% +1.4 4.90 ± 15% +1.6 5.11 ± 4% perf-profile.calltrace.cycles-pp.sctp_ulpevent_free.sctp_recvmsg.inet_recvmsg.sock_recvmsg.____sys_recvmsg 15.56 ± 7% +2.0 17.52 ± 8% +2.7 18.21 ± 7% perf-profile.calltrace.cycles-pp.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe 15.19 ± 7% +2.0 17.18 ± 8% +2.7 17.90 ± 7% perf-profile.calltrace.cycles-pp.inet_recvmsg.sock_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg 15.70 ± 6% +2.0 17.71 ± 7% +2.6 18.35 ± 7% perf-profile.calltrace.cycles-pp.___sys_recvmsg.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg 15.14 ± 7% +2.0 17.15 ± 8% +2.7 17.88 ± 7% perf-profile.calltrace.cycles-pp.sctp_recvmsg.inet_recvmsg.sock_recvmsg.____sys_recvmsg.___sys_recvmsg 15.29 ± 7% +2.0 17.31 ± 8% +2.7 18.02 ± 7% perf-profile.calltrace.cycles-pp.sock_recvmsg.____sys_recvmsg.___sys_recvmsg.__sys_recvmsg.do_syscall_64 15.87 ± 6% +2.0 17.89 ± 7% +2.7 18.55 ± 7% perf-profile.calltrace.cycles-pp.__sys_recvmsg.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg 16.56 ± 6% +2.1 18.68 ± 7% +2.7 19.26 ± 7% perf-profile.calltrace.cycles-pp.recvmsg 16.24 ± 6% +2.2 18.40 ± 7% +2.8 19.00 ± 7% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.recvmsg 16.22 ± 6% +2.2 18.39 ± 7% +2.8 18.99 ± 7% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.recvmsg 4.94 ± 5% +5.0 9.93 ± 9% +5.2 10.09 ± 13% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm 4.94 ± 5% +5.0 9.93 ± 9% +5.2 10.09 ± 13% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm 4.94 ± 5% +5.0 9.93 ± 9% +5.2 10.09 ± 13% perf-profile.calltrace.cycles-pp.ret_from_fork_asm 2.83 ± 9% +5.3 8.09 ± 13% +5.4 8.24 ± 15% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.70 ± 12% +5.6 7.28 ± 14% +5.8 7.54 ± 15% perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread 1.70 ± 13% +5.6 7.28 ± 14% +5.8 7.54 ± 15% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn 1.70 ± 13% +5.6 7.28 ± 14% +5.8 7.54 ± 15% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.run_ksoftirqd 1.73 ± 12% +5.6 7.31 ± 14% +5.8 7.57 ± 15% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm 1.73 ± 12% +5.6 7.31 ± 14% +5.8 7.57 ± 15% perf-profile.calltrace.cycles-pp.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork 0.00 +13.2 13.15 ± 49% +11.3 11.25 ± 55% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 36.91 ± 2% -12.6 24.32 ± 10% -12.3 24.63 ± 5% perf-profile.children.cycles-pp.acpi_idle_enter 36.80 ± 2% -12.5 24.31 ± 10% -12.2 24.58 ± 5% perf-profile.children.cycles-pp.acpi_safe_halt 29.95 ± 3% -12.1 17.81 ± 11% -12.0 17.96 ± 5% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt 27.78 ± 3% -12.0 15.78 ± 13% -11.7 16.06 ± 5% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt 25.75 ± 2% -9.7 16.02 ± 13% -9.2 16.53 ± 5% perf-profile.children.cycles-pp.irq_exit_rcu 6.04 ± 8% -3.5 2.52 ± 16% -3.3 2.72 ± 13% perf-profile.children.cycles-pp._nohz_idle_balance 5.07 ± 3% -3.5 1.60 ± 18% -3.4 1.63 ± 11% perf-profile.children.cycles-pp._raw_spin_lock 29.77 ± 2% -3.0 26.73 ± 8% -2.0 27.76 ± 7% perf-profile.children.cycles-pp.__do_softirq 25.22 ± 3% -2.9 22.31 ± 8% -2.2 23.01 ± 6% perf-profile.children.cycles-pp.net_rx_action 25.18 ± 3% -2.9 22.28 ± 8% -2.2 22.98 ± 6% perf-profile.children.cycles-pp.__napi_poll 25.14 ± 3% -2.9 22.26 ± 8% -2.2 22.96 ± 6% perf-profile.children.cycles-pp.process_backlog 24.71 ± 3% -2.9 21.86 ± 8% -2.2 22.47 ± 7% perf-profile.children.cycles-pp.ip_protocol_deliver_rcu 25.04 ± 3% -2.9 22.19 ± 8% -2.2 22.87 ± 6% perf-profile.children.cycles-pp.__netif_receive_skb_one_core 24.72 ± 3% -2.9 21.86 ± 8% -2.2 22.49 ± 7% perf-profile.children.cycles-pp.ip_local_deliver_finish 24.62 ± 3% -2.8 21.80 ± 8% -2.2 22.42 ± 7% perf-profile.children.cycles-pp.sctp_rcv 23.24 ± 3% -2.6 20.63 ± 8% -2.0 21.20 ± 7% perf-profile.children.cycles-pp.sctp_assoc_bh_rcv 2.80 ± 6% -2.4 0.45 ± 33% -2.4 0.44 ± 18% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested 26.58 ± 2% -2.2 24.38 ± 8% -1.6 25.02 ± 7% perf-profile.children.cycles-pp.sctp_do_sm 5.62 ± 5% -1.9 3.71 ± 10% -1.8 3.80 ± 8% perf-profile.children.cycles-pp.sctp_outq_flush_data 10.75 ± 3% -1.6 9.14 ± 7% -1.1 9.63 ± 6% perf-profile.children.cycles-pp._copy_from_iter 10.86 ± 3% -1.6 9.29 ± 7% -1.1 9.71 ± 6% perf-profile.children.cycles-pp.sctp_user_addto_chunk 1.77 ± 11% -1.5 0.25 ± 25% -1.5 0.22 ± 24% perf-profile.children.cycles-pp.__mod_timer 1.83 ± 13% -1.4 0.45 ± 16% -1.3 0.49 ± 12% perf-profile.children.cycles-pp.run_timer_softirq 1.46 ± 11% -1.3 0.12 ± 31% -1.4 0.09 ± 66% perf-profile.children.cycles-pp.sctp_transport_reset_t3_rtx 12.22 ± 3% -1.2 11.02 ± 6% -0.7 11.55 ± 10% perf-profile.children.cycles-pp.sctp_datamsg_from_user 1.40 ± 13% -1.0 0.35 ± 16% -1.0 0.40 ± 24% perf-profile.children.cycles-pp.tick_irq_enter 1.44 ± 13% -1.0 0.40 ± 11% -1.0 0.47 ± 13% perf-profile.children.cycles-pp.irq_enter_rcu 0.97 ± 15% -0.9 0.10 ± 52% -0.9 0.07 ± 52% perf-profile.children.cycles-pp.tick_do_update_jiffies64 1.53 ± 7% -0.8 0.69 ± 28% -0.7 0.83 ± 32% perf-profile.children.cycles-pp.update_blocked_averages 3.73 ± 8% -0.7 3.01 ± 8% -0.6 3.15 ± 8% perf-profile.children.cycles-pp.sctp_packet_transmit_chunk 1.00 ± 27% -0.6 0.43 ± 18% -0.6 0.39 ± 15% perf-profile.children.cycles-pp.update_rq_clock_task 2.33 ± 11% -0.5 1.79 ± 12% -0.6 1.77 ± 7% perf-profile.children.cycles-pp.asm_exc_page_fault 1.74 ± 12% -0.5 1.20 ± 12% -0.5 1.29 ± 5% perf-profile.children.cycles-pp.__run_timers 1.61 ± 9% -0.5 1.12 ± 12% -0.4 1.23 ± 4% perf-profile.children.cycles-pp.call_timer_fn 3.33 ± 3% -0.4 2.89 ± 11% -0.4 2.89 ± 10% perf-profile.children.cycles-pp.sctp_ulpq_partial_delivery 1.94 ± 6% -0.4 1.52 ± 11% -0.3 1.63 ± 7% perf-profile.children.cycles-pp.vfs_read 1.88 ± 10% -0.4 1.46 ± 11% -0.4 1.50 ± 7% perf-profile.children.cycles-pp.handle_mm_fault 1.65 ± 5% -0.4 1.24 ± 25% -0.4 1.22 ± 17% perf-profile.children.cycles-pp.hrtimer_interrupt 1.73 ± 7% -0.4 1.32 ± 26% -0.5 1.27 ± 18% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt 2.13 ± 5% -0.4 1.75 ± 13% -0.3 1.86 ± 8% perf-profile.children.cycles-pp.read 0.60 ± 19% -0.4 0.23 ± 22% -0.3 0.34 ± 75% perf-profile.children.cycles-pp._find_next_and_bit 1.98 ± 5% -0.4 1.62 ± 13% -0.2 1.75 ± 6% perf-profile.children.cycles-pp.ksys_read 0.66 ± 12% -0.4 0.31 ± 15% -0.3 0.33 ± 21% perf-profile.children.cycles-pp.update_rq_clock 1.70 ± 11% -0.3 1.36 ± 16% -0.4 1.29 ± 13% perf-profile.children.cycles-pp.worker_thread 1.16 ± 10% -0.3 0.83 ± 10% -0.2 0.92 ± 2% perf-profile.children.cycles-pp.sctp_generate_timeout_event 0.92 ± 14% -0.3 0.62 ± 24% -0.2 0.73 ± 14% perf-profile.children.cycles-pp.exit_mmap 0.93 ± 15% -0.3 0.63 ± 24% -0.2 0.74 ± 13% perf-profile.children.cycles-pp.__mmput 1.77 ± 7% -0.3 1.48 ± 9% -0.3 1.51 ± 6% perf-profile.children.cycles-pp.execve 0.34 ± 17% -0.3 0.06 ± 75% -0.3 0.09 ± 21% perf-profile.children.cycles-pp.run_rebalance_domains 1.77 ± 8% -0.3 1.48 ± 8% -0.3 1.51 ± 6% perf-profile.children.cycles-pp.do_execveat_common 1.78 ± 8% -0.3 1.50 ± 8% -0.3 1.51 ± 6% perf-profile.children.cycles-pp.__x64_sys_execve 0.52 ± 21% -0.3 0.24 ± 29% -0.3 0.25 ± 6% perf-profile.children.cycles-pp.__update_blocked_fair 1.03 ± 11% -0.2 0.79 ± 8% -0.2 0.81 ± 5% perf-profile.children.cycles-pp.load_elf_binary 1.06 ± 10% -0.2 0.83 ± 6% -0.2 0.83 ± 6% perf-profile.children.cycles-pp.search_binary_handler 0.73 ± 11% -0.2 0.50 ± 23% -0.1 0.62 ± 14% perf-profile.children.cycles-pp.exit_mm 1.24 ± 9% -0.2 1.01 ± 10% -0.1 1.16 ± 15% perf-profile.children.cycles-pp.read_counters 1.06 ± 10% -0.2 0.83 ± 6% -0.2 0.84 ± 5% perf-profile.children.cycles-pp.exec_binprm 0.79 ± 14% -0.2 0.56 ± 18% -0.1 0.71 ± 25% perf-profile.children.cycles-pp.sched_setaffinity 0.87 ± 13% -0.2 0.65 ± 14% -0.2 0.63 ± 13% perf-profile.children.cycles-pp.enqueue_task_fair 0.35 ± 25% -0.2 0.13 ± 19% -0.2 0.11 ± 60% perf-profile.children.cycles-pp.release_sock 1.50 ± 3% -0.2 1.28 ± 9% -0.3 1.18 ± 19% perf-profile.children.cycles-pp.kmem_cache_free 0.71 ± 11% -0.2 0.49 ± 22% -0.2 0.55 ± 16% perf-profile.children.cycles-pp.setlocale 1.09 ± 5% -0.2 0.87 ± 14% -0.2 0.91 ± 8% perf-profile.children.cycles-pp.seq_read_iter 1.36 ± 6% -0.2 1.14 ± 15% -0.2 1.16 ± 7% perf-profile.children.cycles-pp.__flush_smp_call_function_queue 1.15 ± 9% -0.2 0.93 ± 14% -0.2 0.91 ± 10% perf-profile.children.cycles-pp.sched_ttwu_pending 1.02 ± 12% -0.2 0.81 ± 18% -0.2 0.78 ± 13% perf-profile.children.cycles-pp.__hrtimer_run_queues 0.75 ± 13% -0.2 0.55 ± 11% -0.2 0.56 ± 12% perf-profile.children.cycles-pp.enqueue_entity 0.53 ± 13% -0.2 0.33 ± 23% -0.2 0.28 ± 18% perf-profile.children.cycles-pp.refresh_cpu_vm_stats 0.28 ± 23% -0.2 0.09 ± 10% -0.2 0.08 ± 68% perf-profile.children.cycles-pp.__release_sock 0.24 ± 32% -0.2 0.05 ± 74% -0.2 0.06 ± 54% perf-profile.children.cycles-pp.hrtimer_try_to_cancel 0.25 ± 28% -0.2 0.06 ± 53% -0.2 0.06 ± 54% perf-profile.children.cycles-pp.hrtimer_cancel 0.26 ± 26% -0.2 0.08 ± 16% -0.2 0.08 ± 68% perf-profile.children.cycles-pp.sctp_backlog_rcv 1.00 ± 8% -0.2 0.82 ± 14% -0.1 0.87 ± 14% perf-profile.children.cycles-pp.ttwu_do_activate 0.39 ± 19% -0.2 0.21 ± 23% -0.1 0.24 ± 22% perf-profile.children.cycles-pp.__memcg_slab_free_hook 0.30 ± 16% -0.2 0.13 ± 29% -0.2 0.09 ± 49% perf-profile.children.cycles-pp.ktime_get_update_offsets_now 0.46 ± 10% -0.1 0.32 ± 28% -0.1 0.32 ± 23% perf-profile.children.cycles-pp.sched_clock_cpu 0.40 ± 18% -0.1 0.26 ± 26% -0.1 0.30 ± 24% perf-profile.children.cycles-pp.sched_clock 0.56 ± 12% -0.1 0.42 ± 12% -0.0 0.51 ± 28% perf-profile.children.cycles-pp.evlist_cpu_iterator__next 1.24 ± 9% -0.1 1.11 ± 7% -0.3 0.95 ± 4% perf-profile.children.cycles-pp.tick_nohz_stop_tick 1.25 ± 10% -0.1 1.12 ± 7% -0.3 0.95 ± 4% perf-profile.children.cycles-pp.tick_nohz_idle_stop_tick 0.80 ± 12% -0.1 0.68 ± 17% -0.2 0.65 ± 3% perf-profile.children.cycles-pp.do_read_fault 0.31 ± 21% -0.1 0.19 ± 28% -0.1 0.25 ± 31% perf-profile.children.cycles-pp.read_tsc 0.13 ± 34% -0.1 0.01 ±223% -0.1 0.03 ± 82% perf-profile.children.cycles-pp.sctp_generate_heartbeat_event 0.32 ± 23% -0.1 0.20 ± 25% -0.2 0.15 ± 20% perf-profile.children.cycles-pp.need_update 0.39 ± 18% -0.1 0.28 ± 14% -0.1 0.25 ± 23% perf-profile.children.cycles-pp.sctp_inq_pop 0.35 ± 19% -0.1 0.24 ± 26% -0.0 0.31 ± 26% perf-profile.children.cycles-pp.__x64_sys_sched_setaffinity 0.24 ± 20% -0.1 0.12 ± 63% -0.1 0.13 ± 22% perf-profile.children.cycles-pp.free_pgtables 0.32 ± 25% -0.1 0.21 ± 25% -0.2 0.16 ± 19% perf-profile.children.cycles-pp.quiet_vmstat 0.16 ± 33% -0.1 0.05 ± 72% -0.1 0.06 ± 60% perf-profile.children.cycles-pp.call_cpuidle 0.54 ± 7% -0.1 0.44 ± 21% -0.0 0.50 ± 5% perf-profile.children.cycles-pp.update_load_avg 0.37 ± 18% -0.1 0.26 ± 15% -0.1 0.24 ± 5% perf-profile.children.cycles-pp.mem_cgroup_charge_skmem 0.23 ± 23% -0.1 0.13 ± 36% -0.0 0.22 ± 27% perf-profile.children.cycles-pp.__set_cpus_allowed_ptr 0.23 ± 31% -0.1 0.13 ± 19% -0.1 0.11 ± 16% perf-profile.children.cycles-pp.timekeeping_advance 0.23 ± 31% -0.1 0.13 ± 19% -0.1 0.11 ± 16% perf-profile.children.cycles-pp.update_wall_time 0.26 ± 20% -0.1 0.16 ± 22% -0.1 0.16 ± 34% perf-profile.children.cycles-pp.tlb_finish_mmu 0.20 ± 38% -0.1 0.10 ± 22% -0.1 0.10 ± 33% perf-profile.children.cycles-pp.__smp_call_single_queue 0.13 ± 26% -0.1 0.04 ±104% -0.1 0.08 ± 17% perf-profile.children.cycles-pp.open_last_lookups 0.40 ± 16% -0.1 0.31 ± 15% -0.1 0.26 ± 8% perf-profile.children.cycles-pp.__sk_mem_schedule 0.27 ± 7% -0.1 0.18 ± 15% -0.1 0.17 ± 38% perf-profile.children.cycles-pp.update_curr 0.38 ± 24% -0.1 0.29 ± 18% -0.1 0.27 ± 11% perf-profile.children.cycles-pp.lapic_next_deadline 0.14 ± 31% -0.1 0.05 ± 77% -0.0 0.09 ± 64% perf-profile.children.cycles-pp.irqentry_exit 0.25 ± 15% -0.1 0.17 ± 26% -0.1 0.16 ± 45% perf-profile.children.cycles-pp._find_next_bit 0.21 ± 13% -0.1 0.14 ± 17% -0.1 0.11 ± 41% perf-profile.children.cycles-pp.lookup_fast 0.34 ± 23% -0.1 0.27 ± 17% -0.1 0.23 ± 33% perf-profile.children.cycles-pp.rcu_core 0.54 ± 11% -0.1 0.48 ± 16% -0.2 0.38 ± 32% perf-profile.children.cycles-pp.balance_fair 0.14 ± 34% -0.1 0.08 ± 72% -0.1 0.05 ± 90% perf-profile.children.cycles-pp.vma_prepare 0.31 ± 22% -0.1 0.24 ± 35% -0.1 0.19 ± 19% perf-profile.children.cycles-pp.begin_new_exec 0.15 ± 36% -0.1 0.08 ± 52% -0.1 0.08 ± 24% perf-profile.children.cycles-pp.set_next_entity 0.12 ± 32% -0.1 0.06 ± 50% -0.0 0.08 ± 82% perf-profile.children.cycles-pp.setup_arg_pages 0.25 ± 37% -0.1 0.19 ± 42% -0.1 0.14 ± 19% perf-profile.children.cycles-pp.irqentry_enter 0.50 ± 14% -0.1 0.45 ± 30% -0.1 0.37 ± 11% perf-profile.children.cycles-pp.do_vmi_align_munmap 0.19 ± 19% -0.1 0.14 ± 37% -0.1 0.11 ± 23% perf-profile.children.cycles-pp.__vm_munmap 0.08 ± 13% -0.1 0.03 ±103% -0.1 0.01 ±200% perf-profile.children.cycles-pp.__call_rcu_common 0.22 ± 8% -0.0 0.17 ± 15% -0.1 0.17 ± 13% perf-profile.children.cycles-pp.diskstats_show 0.10 ± 26% -0.0 0.05 ± 45% -0.0 0.07 ± 82% perf-profile.children.cycles-pp.shift_arg_pages 0.15 ± 17% -0.0 0.10 ± 53% -0.1 0.05 ± 86% perf-profile.children.cycles-pp.__x64_sys_munmap 0.22 ± 17% -0.0 0.19 ± 38% -0.1 0.12 ± 34% perf-profile.children.cycles-pp.should_we_balance 0.13 ± 7% -0.0 0.11 ± 8% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.perf_mmap__push 0.13 ± 7% -0.0 0.11 ± 8% -0.0 0.10 ± 9% perf-profile.children.cycles-pp.record__mmap_read_evlist 0.10 ± 22% -0.0 0.09 ± 38% -0.0 0.06 ± 57% perf-profile.children.cycles-pp.down_write 0.00 +0.0 0.00 +0.7 0.67 ± 13% perf-profile.children.cycles-pp.tick_nohz_handler 0.02 ±141% +0.1 0.07 ± 62% +0.1 0.07 ± 32% perf-profile.children.cycles-pp.rb_erase 0.06 ±106% +0.1 0.11 ± 26% +0.1 0.18 ± 36% perf-profile.children.cycles-pp.timerqueue_add 0.01 ±223% +0.1 0.08 ± 68% +0.1 0.10 ± 25% perf-profile.children.cycles-pp.sched_mm_cid_migrate_to 0.07 ± 52% +0.1 0.15 ± 33% +0.1 0.16 ± 38% perf-profile.children.cycles-pp.local_clock_noinstr 0.16 ± 29% +0.1 0.25 ± 21% +0.1 0.25 ± 31% perf-profile.children.cycles-pp.select_task_rq 0.25 ± 25% +0.1 0.37 ± 23% +0.1 0.36 ± 17% perf-profile.children.cycles-pp.select_task_rq_fair 0.00 +0.1 0.12 ± 39% +0.1 0.12 ± 38% perf-profile.children.cycles-pp.__tmigr_cpu_activate 0.00 +0.2 0.16 ± 41% +0.2 0.16 ± 19% perf-profile.children.cycles-pp.tmigr_cpu_activate 0.00 +0.2 0.18 ± 22% +0.2 0.19 ± 38% perf-profile.children.cycles-pp.tmigr_inactive_up 0.00 +0.2 0.21 ± 49% +0.2 0.22 ± 29% perf-profile.children.cycles-pp.task_mm_cid_work 1.38 ± 5% +0.2 1.60 ± 3% +0.4 1.77 ± 34% perf-profile.children.cycles-pp._sctp_make_chunk 0.00 +0.2 0.22 ± 18% +0.2 0.19 ± 38% perf-profile.children.cycles-pp.tmigr_cpu_deactivate 0.06 ± 23% +0.2 0.29 ± 30% +0.2 0.25 ± 25% perf-profile.children.cycles-pp.task_work_run 1.21 ± 7% +0.3 1.52 ± 7% +0.4 1.66 ± 36% perf-profile.children.cycles-pp.sctp_make_datafrag_empty 0.84 ± 13% +0.3 1.18 ± 2% +0.5 1.38 ± 7% perf-profile.children.cycles-pp.sctp_wait_for_sndbuf 0.07 ± 24% +0.4 0.47 ± 9% +0.4 0.42 ± 9% perf-profile.children.cycles-pp.__get_next_timer_interrupt 0.00 +0.5 0.49 ± 18% +0.5 0.49 ± 35% perf-profile.children.cycles-pp.tmigr_update_events 2.73 ± 8% +0.5 3.26 ± 14% +0.5 3.24 ± 7% perf-profile.children.cycles-pp.consume_skb 2.91 ± 8% +0.6 3.49 ± 16% +0.7 3.63 ± 12% perf-profile.children.cycles-pp.sysvec_call_function_single 3.12 ± 7% +0.7 3.77 ± 16% +0.7 3.85 ± 7% perf-profile.children.cycles-pp.sctp_primitive_SEND 1.48 ± 14% +0.7 2.18 ± 15% +0.9 2.35 ± 15% perf-profile.children.cycles-pp.__free_pages_ok 0.58 ± 29% +0.8 1.34 ± 12% +0.8 1.39 ± 19% perf-profile.children.cycles-pp.free_one_page 2.46 ± 11% +0.8 3.23 ± 12% +0.9 3.35 ± 9% perf-profile.children.cycles-pp.skb_release_data 0.00 +0.8 0.78 ± 17% +0.8 0.81 ± 9% perf-profile.children.cycles-pp.timer_expire_remote 3.32 ± 5% +0.8 4.14 ± 11% +1.1 4.42 ± 6% perf-profile.children.cycles-pp.__ip_queue_xmit 2.44 ± 4% +0.8 3.27 ± 19% +0.7 3.10 ± 9% perf-profile.children.cycles-pp.kmalloc_reserve 2.86 ± 5% +0.8 3.70 ± 17% +0.6 3.48 ± 8% perf-profile.children.cycles-pp.__alloc_skb 3.00 ± 6% +0.8 3.84 ± 11% +1.1 4.09 ± 6% perf-profile.children.cycles-pp.ip_finish_output2 2.88 ± 6% +0.9 3.73 ± 11% +1.1 3.97 ± 6% perf-profile.children.cycles-pp.__dev_queue_xmit 2.26 ± 9% +0.9 3.12 ± 19% +0.6 2.90 ± 10% perf-profile.children.cycles-pp.__alloc_pages 2.32 ± 4% +0.9 3.19 ± 20% +0.7 3.01 ± 8% perf-profile.children.cycles-pp.__kmalloc_node_track_caller 2.29 ± 4% +0.9 3.18 ± 19% +0.7 2.98 ± 8% perf-profile.children.cycles-pp.__kmalloc_large_node 2.09 ± 8% +0.9 2.97 ± 20% +0.6 2.67 ± 9% perf-profile.children.cycles-pp.get_page_from_freelist 1.25 ± 10% +0.9 2.20 ± 24% +0.7 1.92 ± 10% perf-profile.children.cycles-pp.rmqueue 2.23 ± 10% +1.0 3.27 ± 13% +1.2 3.46 ± 6% perf-profile.children.cycles-pp.sctp_assoc_rwnd_increase 2.43 ± 9% +1.1 3.52 ± 11% +1.3 3.75 ± 8% perf-profile.children.cycles-pp.do_softirq 2.46 ± 9% +1.1 3.56 ± 11% +1.3 3.76 ± 7% perf-profile.children.cycles-pp.__local_bh_enable_ip 0.00 +1.2 1.20 ± 16% +1.2 1.18 ± 16% perf-profile.children.cycles-pp.tmigr_handle_remote_cpu 0.00 +1.4 1.36 ± 15% +1.5 1.47 ± 19% perf-profile.children.cycles-pp.tmigr_handle_remote_up 3.53 ± 7% +1.4 4.91 ± 15% +1.6 5.12 ± 4% perf-profile.children.cycles-pp.sctp_ulpevent_free 0.00 +1.4 1.38 ± 16% +1.5 1.48 ± 19% perf-profile.children.cycles-pp.tmigr_handle_remote 2.16 ± 9% +1.4 3.56 ± 9% +1.4 3.53 ± 9% perf-profile.children.cycles-pp._raw_spin_lock_irqsave 15.32 ± 7% +2.0 17.29 ± 8% +2.7 18.03 ± 7% perf-profile.children.cycles-pp.sctp_recvmsg 15.56 ± 7% +2.0 17.53 ± 8% +2.7 18.21 ± 7% perf-profile.children.cycles-pp.____sys_recvmsg 15.19 ± 7% +2.0 17.18 ± 8% +2.7 17.90 ± 7% perf-profile.children.cycles-pp.inet_recvmsg 15.70 ± 6% +2.0 17.72 ± 7% +2.7 18.36 ± 7% perf-profile.children.cycles-pp.___sys_recvmsg 15.29 ± 7% +2.0 17.31 ± 8% +2.7 18.02 ± 7% perf-profile.children.cycles-pp.sock_recvmsg 15.87 ± 6% +2.0 17.89 ± 7% +2.7 18.55 ± 7% perf-profile.children.cycles-pp.__sys_recvmsg 16.60 ± 6% +2.1 18.70 ± 8% +2.7 19.29 ± 7% perf-profile.children.cycles-pp.recvmsg 5.02 ± 5% +5.0 9.99 ± 9% +5.1 10.16 ± 13% perf-profile.children.cycles-pp.ret_from_fork_asm 5.00 ± 4% +5.0 9.97 ± 9% +5.2 10.16 ± 13% perf-profile.children.cycles-pp.ret_from_fork 4.94 ± 5% +5.0 9.93 ± 9% +5.2 10.09 ± 13% perf-profile.children.cycles-pp.kthread 2.83 ± 9% +5.3 8.09 ± 13% +5.4 8.24 ± 15% perf-profile.children.cycles-pp.smpboot_thread_fn 1.73 ± 12% +5.6 7.31 ± 14% +5.8 7.57 ± 15% perf-profile.children.cycles-pp.run_ksoftirqd 0.02 ±141% +13.2 13.22 ± 49% +11.3 11.35 ± 54% perf-profile.children.cycles-pp.poll_idle 10.64 ± 4% -1.6 9.04 ± 7% -1.2 9.43 ± 6% perf-profile.self.cycles-pp._copy_from_iter 2.41 ± 5% -1.2 1.18 ± 19% -1.3 1.12 ± 10% perf-profile.self.cycles-pp._raw_spin_lock 0.65 ± 23% -0.6 0.07 ± 81% -0.5 0.17 ± 11% perf-profile.self.cycles-pp._nohz_idle_balance 0.84 ± 28% -0.5 0.34 ± 19% -0.5 0.32 ± 22% perf-profile.self.cycles-pp.update_rq_clock_task 0.46 ± 27% -0.2 0.22 ± 18% -0.3 0.20 ± 47% perf-profile.self.cycles-pp._find_next_and_bit 0.43 ± 9% -0.2 0.20 ± 19% -0.2 0.21 ± 22% perf-profile.self.cycles-pp.update_rq_clock 0.52 ± 19% -0.2 0.33 ± 33% -0.2 0.36 ± 26% perf-profile.self.cycles-pp.idle_cpu 0.29 ± 16% -0.2 0.11 ± 28% -0.2 0.08 ± 71% perf-profile.self.cycles-pp.ktime_get_update_offsets_now 0.33 ± 32% -0.2 0.15 ± 24% -0.2 0.15 ± 24% perf-profile.self.cycles-pp.__update_blocked_fair 0.35 ± 17% -0.2 0.20 ± 48% -0.2 0.20 ± 13% perf-profile.self.cycles-pp.refresh_cpu_vm_stats 0.25 ± 28% -0.1 0.12 ± 74% -0.1 0.13 ± 31% perf-profile.self.cycles-pp.__memcg_slab_free_hook 0.16 ± 34% -0.1 0.04 ±156% -0.1 0.06 ± 81% perf-profile.self.cycles-pp.need_update 0.16 ± 32% -0.1 0.04 ±100% -0.1 0.06 ± 56% perf-profile.self.cycles-pp.call_cpuidle 0.30 ± 21% -0.1 0.19 ± 26% -0.0 0.25 ± 31% perf-profile.self.cycles-pp.read_tsc 0.20 ± 21% -0.1 0.09 ± 50% -0.1 0.13 ± 38% perf-profile.self.cycles-pp._raw_spin_lock_bh 0.38 ± 24% -0.1 0.29 ± 18% -0.1 0.27 ± 11% perf-profile.self.cycles-pp.lapic_next_deadline 0.12 ± 19% -0.1 0.04 ±116% -0.1 0.04 ± 83% perf-profile.self.cycles-pp.sctp_inq_pop 0.19 ± 14% -0.1 0.12 ± 15% -0.0 0.16 ± 25% perf-profile.self.cycles-pp.filemap_map_pages 0.12 ± 21% -0.1 0.06 ± 48% -0.1 0.07 ± 74% perf-profile.self.cycles-pp.rebalance_domains 0.10 ± 29% -0.0 0.06 ± 50% -0.1 0.05 ± 88% perf-profile.self.cycles-pp.release_pages 0.01 ±223% +0.1 0.08 ± 68% +0.1 0.10 ± 25% perf-profile.self.cycles-pp.sched_mm_cid_migrate_to 0.00 +0.2 0.18 ± 48% +0.2 0.19 ± 24% perf-profile.self.cycles-pp.task_mm_cid_work 0.02 ±141% +13.0 13.04 ± 49% +11.1 11.11 ± 54% perf-profile.self.cycles-pp.poll_idle ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-05 2:17 ` Oliver Sang @ 2024-03-05 10:46 ` Frederic Weisbecker 2024-03-05 11:21 ` Frederic Weisbecker 2024-03-05 11:35 ` Frederic Weisbecker 2 siblings, 0 replies; 10+ messages in thread From: Frederic Weisbecker @ 2024-03-05 10:46 UTC (permalink / raw) To: Oliver Sang Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, ying.huang, feng.tang, fengwei.yin On Tue, Mar 05, 2024 at 10:17:43AM +0800, Oliver Sang wrote: > hi, Frederic Weisbecker, > > On Mon, Mar 04, 2024 at 12:28:33PM +0100, Frederic Weisbecker wrote: > > Le Mon, Mar 04, 2024 at 10:13:00AM +0800, Oliver Sang a écrit : > > > On Mon, Mar 04, 2024 at 01:32:45AM +0100, Frederic Weisbecker wrote: > > > > Le Fri, Mar 01, 2024 at 04:09:24PM +0800, kernel test robot a écrit : > > > > > commit: > > > > > 57e95a5c41 ("timers: Introduce function to check timer base is_idle flag") > > > > > 7ee9887703 ("timers: Implement the hierarchical pull model") > > > > > > > > Is this something that is observed also with the commits that follow in this > > > > branch? > > > > > > when this bisect done, we also tested the tip of timers/core branch at that time > > > 8b3843ae3634b vdso/datapage: Quick fix - use asm/page-def.h for ARM64 > > > > > > the regression still exists on it: > > > > > > 57e95a5c4117dc6a 7ee988770326fca440472200c3e 8b3843ae3634b472530fb69c386 > > > ---------------- --------------------------- --------------------------- > > > %stddev %change %stddev %change %stddev > > > \ | \ | \ > > > 4.10 -1.2% 4.05 -1.2% 4.05 netperf.ThroughputBoth_Mbps > > > 1049 -1.2% 1037 -1.2% 1036 netperf.ThroughputBoth_total_Mbps > > > 4.10 -1.2% 4.05 -1.2% 4.05 netperf.Throughput_Mbps > > > 1049 -1.2% 1037 -1.2% 1036 netperf.Throughput_total_Mbps > > > > Oh, I see... :-/ > > > > > > Ie: would it be possible to compare instead: > > > > > > > > 57e95a5c4117 (timers: Introduce function to check timer base is_idle flag) > > > > VS > > > > b2cf7507e186 (timers: Always queue timers on the local CPU) > > > > > > > > Because the improvements introduced by 7ee9887703 are mostly relevant after > > > > b2cf7507e186. > > > > > > got it. will test. > > > > > > at the same time, we noticed current tip of timers/core is > > > a184d9835a0a6 (tip/timers/core) tick/sched: Fix build failure for > > > CONFIG_NO_HZ_COMMON=n > > > > Shouldn't be a problem as it fixes an issue introduced after: > > > > b2cf7507e186 (timers: Always queue timers on the local CPU) > > > > > > > > though it seems irelevant, we will still get data for it. > > > > Thanks a lot, this will be very helpful. Especially with all the perf diff > > details like in the initial email report. > > the regression still exists on b2cf7507e186 and current tip of the branch: > > ========================================================================================= > cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: > cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/200%/debian-12-x86_64-20240206.cgz/300s/lkp-icl-2sp2/SCTP_STREAM/netperf > > commit: > 57e95a5c4117 (timers: Introduce function to check timer base is_idle flag) > b2cf7507e186 (timers: Always queue timers on the local CPU) > a184d9835a0a (tick/sched: Fix build failure for CONFIG_NO_HZ_COMMON=n) > > a184d9835a0a689261ea6a4a8dbc18173a031b77 > > 57e95a5c4117dc6a b2cf7507e18649a30512515ec0c a184d9835a0a689261ea6a4a8db > ---------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev > \ | \ | \ > 4.10 -1.4% 4.04 -1.5% 4.04 netperf.ThroughputBoth_Mbps > 1049 -1.4% 1034 -1.5% 1033 netperf.ThroughputBoth_total_Mbps > 4.10 -1.4% 4.04 -1.5% 4.04 netperf.Throughput_Mbps > 1049 -1.4% 1034 -1.5% 1033 netperf.Throughput_total_Mbps > > details are in below [1] Thanks a lot! > > > Because I'm having some troubles > > running those lkp tests. How is it working BTW? I've seen it downloading > > two kernel trees but I haven't noticed a kernel build. > > you need build 7ee9887703 and its parent kernel with config in > https://download.01.org/0day-ci/archive/20240301/202403011511.24defbbd-oliver.sang@intel.com > then boot into kernel. > > after that, you could run netperf in each kernel by following > https://download.01.org/0day-ci/archive/20240301/202403011511.24defbbd-oliver.sang@intel.com/reproduce > to get data. > > the results will store in different path according the kernel commit, then you > could compare the results from both kernels. Oh I see now. > > what's your OS BTW? we cannot support all distributions so far... Opensuse, but it failed to find a lot of equivalent packages. Then I tried Ubuntu 22.04.4 LTS but it failed saying perf didn't have the "sched" subcommand. Which distro do you recommand using? > > > Are the two compared > > instances running through kvm? > > we run performance tests on bare mental. for netperf, we just test on one > machine so the test is really upon local net. Ok. Thanks! ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-05 2:17 ` Oliver Sang 2024-03-05 10:46 ` Frederic Weisbecker @ 2024-03-05 11:21 ` Frederic Weisbecker 2024-03-05 11:35 ` Frederic Weisbecker 2 siblings, 0 replies; 10+ messages in thread From: Frederic Weisbecker @ 2024-03-05 11:21 UTC (permalink / raw) To: Oliver Sang Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, ying.huang, feng.tang, fengwei.yin On Tue, Mar 05, 2024 at 10:17:43AM +0800, Oliver Sang wrote: > 57e95a5c4117dc6a b2cf7507e18649a30512515ec0c a184d9835a0a689261ea6a4a8db > ---------------- --------------------------- --------------------------- > %stddev %change %stddev %change %stddev > \ | \ | \ > 1364607 +11.8% 1525991 +10.3% 1504946 cpuidle..usage Does it mean more time spent in idle/C-states? That's unclear... > 45.86 ± 4% +8.4% 49.70 ± 5% +3.5% 47.46 ± 6% boot-time.boot > 5430 ± 5% +9.0% 5921 ± 5% +3.8% 5636 ± 6% boot-time.idle > 0.03 +0.0 0.04 +0.0 0.04 mpstat.cpu.all.soft% > 0.04 +0.0 0.08 +0.0 0.08 ± 2% mpstat.cpu.all.sys% > 4.14 -8.9% 3.77 ± 2% -8.3% 3.79 ± 2% mpstat.max_utilization_pct > 20726 ± 63% +246.1% 71744 ± 53% +68.2% 34867 ± 72% numa-numastat.node0.other_node > 1431327 ± 7% +13.9% 1630876 ± 3% +14.8% 1643375 ± 7% numa-numastat.node1.numa_hit > 37532 ± 35% +62.1% 60841 ± 63% +160.8% 97891 ± 25% numa-numastat.node1.other_node > 201.33 ± 3% -28.5% 144.00 ± 8% -26.8% 147.40 ± 11% perf-c2c.DRAM.remote > 187.83 ± 3% -21.4% 147.67 ± 6% -37.0% 118.40 ± 11% perf-c2c.HITM.local > 40.67 ± 7% -54.5% 18.50 ± 19% -59.7% 16.40 ± 11% perf-c2c.HITM.remote > 1.36 ± 4% +10.7% 1.51 ± 3% +12.7% 1.53 ± 5% vmstat.procs.r > 5654 -1.9% 5549 -2.8% 5497 vmstat.system.cs > 5232 +10.7% 5790 +8.8% 5690 vmstat.system.in > 15247 ± 6% -5.0% 14490 ± 9% -8.8% 13903 ± 5% numa-meminfo.node0.PageTables > 12499 ± 6% +115.6% 26951 ± 3% +118.3% 27288 ± 2% numa-meminfo.node1.Active > 12489 ± 6% +115.7% 26940 ± 3% +118.2% 27249 ± 2% numa-meminfo.node1.Active(anon) > 12488 ± 6% +114.2% 26754 ± 3% +118.2% 27255 ± 2% numa-meminfo.node1.Shmem > 102.17 ± 8% +906.2% 1028 ± 5% +910.1% 1032 ± 5% time.involuntary_context_switches There is a lot more involuntary context switches. This could be due to timers performing wake ups expiring more often on busy CPUs. [...] > 4178 ± 2% +86.3% 7784 +87.3% 7826 ± 2% proc-vmstat.nr_zone_active_anon > 3064698 +4.4% 3200938 +4.2% 3193539 proc-vmstat.numa_hit > 3006439 +2.1% 3068397 +1.8% 3060785 proc-vmstat.numa_local > 58258 +127.6% 132587 +127.9% 132758 proc-vmstat.numa_other > 8114 ± 2% +63.2% 13244 ± 4% +62.6% 13190 ± 2% proc-vmstat.pgactivate > 986600 +1.2% 998606 +0.9% 995307 proc-vmstat.pgfault > 20.00 +1905.0% 401.00 ± 79% +2050.0% 430.00 ± 79% proc-vmstat.unevictable_pgs_culled > 15.14 +17.0% 17.72 +17.4% 17.77 perf-stat.i.MPKI > 1.702e+08 +3.5% 1.762e+08 +3.3% 1.758e+08 perf-stat.i.branch-instructions > 1.68 +0.1 1.80 +0.1 1.81 perf-stat.i.branch-miss-rate% > 7174339 +1.2% 7262760 +1.4% 7276699 perf-stat.i.branch-misses > 18.46 +3.4 21.86 +3.4 21.87 perf-stat.i.cache-miss-rate% > 4047319 +20.6% 4880009 +20.4% 4874638 perf-stat.i.cache-misses > 22007366 +2.6% 22586331 +2.5% 22565036 perf-stat.i.cache-references > 5620 -1.6% 5532 -2.5% 5482 perf-stat.i.context-switches > 1.84 +17.0% 2.15 +16.5% 2.14 perf-stat.i.cpi > 9.159e+08 +12.8% 1.033e+09 +12.4% 1.03e+09 perf-stat.i.cpu-cycles > 161.08 +193.1% 472.19 ± 2% +192.1% 470.47 ± 4% perf-stat.i.cpu-migrations A lot more task migrations. Not sure how to explain that. [...] > 160.61 +193.0% 470.58 ± 2% +191.9% 468.79 ± 4% perf-stat.ps.cpu-migrations > 8.433e+08 +3.0% 8.685e+08 +2.7% 8.665e+08 perf-stat.ps.instructions > 2.551e+11 +3.3% 2.636e+11 +3.1% 2.631e+11 perf-stat.total.instructions > 31.82 ± 3% -13.0 18.83 ± 12% -13.2 18.65 ± 6% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter > 36.90 ± 2% -12.6 24.32 ± 10% -12.3 24.62 ± 5% perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle > 27.61 ± 3% -12.5 15.09 ± 14% -12.4 15.24 ± 5% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state > 31.75 ± 2% -12.0 19.79 ± 12% -11.6 20.11 ± 5% perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call Fewer C3 entries. [...] > 4.94 ± 5% +5.0 9.93 ± 9% +5.2 10.09 ± 13% perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm > 4.94 ± 5% +5.0 9.93 ± 9% +5.2 10.09 ± 13% perf-profile.calltrace.cycles-pp.ret_from_fork.ret_from_fork_asm > 4.94 ± 5% +5.0 9.93 ± 9% +5.2 10.09 ± 13% perf-profile.calltrace.cycles-pp.ret_from_fork_asm > 2.83 ± 9% +5.3 8.09 ± 13% +5.4 8.24 ± 15% perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm > 1.70 ± 12% +5.6 7.28 ± 14% +5.8 7.54 ± 15% perf-profile.calltrace.cycles-pp.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread > 1.70 ± 13% +5.6 7.28 ± 14% +5.8 7.54 ± 15% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn > 1.70 ± 13% +5.6 7.28 ± 14% +5.8 7.54 ± 15% perf-profile.calltrace.cycles-pp.process_backlog.__napi_poll.net_rx_action.__do_softirq.run_ksoftirqd > 1.73 ± 12% +5.6 7.31 ± 14% +5.8 7.57 ± 15% perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm > 1.73 ± 12% +5.6 7.31 ± 14% +5.8 7.57 ± 15% perf-profile.calltrace.cycles-pp.__do_softirq.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork More time spent in ksoftirqd. One theory is that remote timer expiry delay napi polling, or the other way around... > 0.00 +13.2 13.15 ± 49% +11.3 11.25 ± 55% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle But a lot more poll_idle time spent. Which is suprising. Also this should impact power and not throughput... Thanks. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-05 2:17 ` Oliver Sang 2024-03-05 10:46 ` Frederic Weisbecker 2024-03-05 11:21 ` Frederic Weisbecker @ 2024-03-05 11:35 ` Frederic Weisbecker 2 siblings, 0 replies; 10+ messages in thread From: Frederic Weisbecker @ 2024-03-05 11:35 UTC (permalink / raw) To: Oliver Sang Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, ying.huang, feng.tang, fengwei.yin On Tue, Mar 05, 2024 at 10:17:43AM +0800, Oliver Sang wrote: > 20.97 ± 3% -9.7 11.23 ± 16% -9.6 11.40 ± 4% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.irq_exit_rcu.sysvec_apic_timer_interrupt And yes, fewer time spent in IRQ-tail softirq processing. > 1.70 ± 13% +5.6 7.28 ± 14% +5.8 7.54 ± 15% perf-profile.calltrace.cycles-pp.__napi_poll.net_rx_action.__do_softirq.run_ksoftirqd.smpboot_thread_fn And more time spent in ksoftirqd softirq processing. This can match the increase in involuntary context switches: IRQ-tail softirq processing takes too much time, possibly due to long lasting remote timer expiring, ksoftirqd is then scheduled, preempting netperf (through might_resched()/cond_resched()) as this is a voluntary preemption kernel). Thanks. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-01 8:09 [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression kernel test robot 2024-03-04 0:32 ` Frederic Weisbecker @ 2024-03-12 23:57 ` Frederic Weisbecker 2024-03-13 8:25 ` Thomas Gleixner 1 sibling, 1 reply; 10+ messages in thread From: Frederic Weisbecker @ 2024-03-12 23:57 UTC (permalink / raw) To: kernel test robot Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, ying.huang, feng.tang, fengwei.yin Le Fri, Mar 01, 2024 at 04:09:24PM +0800, kernel test robot a écrit : > > > Hello, > > kernel test robot noticed a -1.2% regression of netperf.Throughput_Mbps on: > > > commit: 7ee988770326fca440472200c3eb58935fe712f6 ("timers: Implement the hierarchical pull model") > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git timers/core > > testcase: netperf > test machine: 128 threads 2 sockets Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz (Ice Lake) with 256G memory > parameters: > > ip: ipv4 > runtime: 300s > nr_threads: 200% > cluster: cs-localhost > test: SCTP_STREAM > cpufreq_governor: performance > > > > > If you fix the issue in a separate patch/commit (i.e. not just a new version of > the same patch/commit), kindly add following tags > | Reported-by: kernel test robot <oliver.sang@intel.com> > | Closes: https://lore.kernel.org/oe-lkp/202403011511.24defbbd-oliver.sang@intel.com > > > Details are as below: > --------------------------------------------------------------------------------------------------> > > > The kernel config and materials to reproduce are available at: > https://download.01.org/0day-ci/archive/20240301/202403011511.24defbbd-oliver.sang@intel.com > > ========================================================================================= > cluster/compiler/cpufreq_governor/ip/kconfig/nr_threads/rootfs/runtime/tbox_group/test/testcase: > cs-localhost/gcc-12/performance/ipv4/x86_64-rhel-8.3/200%/debian-12-x86_64-20240206.cgz/300s/lkp-icl-2sp2/SCTP_STREAM/netperf > > commit: > 57e95a5c41 ("timers: Introduce function to check timer base is_idle flag") > 7ee9887703 ("timers: Implement the hierarchical pull model") So I can reproduce. And after hours staring at traces I haven't really found the real cause of this. 1% difference is not always easy to track down. But here are some sort of conclusion so far: _ There is an increase of ksoftirqd use (+13%) but if I boot with threadirqs before and after the patch (which means that ksoftirqd is used all the time for softirq handling) I still see the performance regression. So this shouldn't play a role here. _ I suspected that timer migrators handling big queues of timers on behalf of idle CPUs would delay NET_RX softirqs but it doesn't seem to be the case. I don't see TIMER vector delaying NET_RX vector after the hierarchical pull model, quite the opposite actually, they are less delayed overall. _ I suspected that timer migrators handling big queues would add scheduling latency. But it doesn't seem to be the case. Quite the opposite again, surprisingly. _ I have observed that, in average, timers execute later with the hierarchical pull model. The following delta: time of callback execution - bucket_expiry is 3 times higher with the hierarchical pull model. Whether that plays a role is unclear. It might still be interesting to investigate. _ The initial perf profile seem to suggest a big increase of task migration. Is it the result of ping-pong wakeup? Does that play a role? Thanks. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression 2024-03-12 23:57 ` Frederic Weisbecker @ 2024-03-13 8:25 ` Thomas Gleixner 0 siblings, 0 replies; 10+ messages in thread From: Thomas Gleixner @ 2024-03-13 8:25 UTC (permalink / raw) To: Frederic Weisbecker, kernel test robot Cc: Anna-Maria Behnsen, oe-lkp, lkp, linux-kernel, x86, ying.huang, feng.tang, fengwei.yin On Wed, Mar 13 2024 at 00:57, Frederic Weisbecker wrote: > So I can reproduce. And after hours staring at traces I haven't really found > the real cause of this. 1% difference is not always easy to track down. > But here are some sort of conclusion so far: > > _ There is an increase of ksoftirqd use (+13%) but if I boot with threadirqs > before and after the patch (which means that ksoftirqd is used all the time > for softirq handling) I still see the performance regression. So this > shouldn't play a role here. > > _ I suspected that timer migrators handling big queues of timers on behalf of > idle CPUs would delay NET_RX softirqs but it doesn't seem to be the case. I > don't see TIMER vector delaying NET_RX vector after the hierarchical pull > model, quite the opposite actually, they are less delayed overall. > > _ I suspected that timer migrators handling big queues would add scheduling > latency. But it doesn't seem to be the case. Quite the opposite again, > surprisingly. > > _ I have observed that, in average, timers execute later with the hierarchical > pull model. The following delta: > time of callback execution - bucket_expiry > is 3 times higher with the hierarchical pull model. Whether that plays a role > is unclear. It might still be interesting to investigate. > > _ The initial perf profile seem to suggest a big increase of task migration. Is > it the result of ping-pong wakeup? Does that play a role? Migration is not cheap. The interesting question is whether this is caused by remote timer expiry. Looking at the perf data there are significant changes vs. idle too: perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle 36.91 ± 2% -12.6 24.32 ± 10% -12.3 24.63 ± 5% That indicates that cpuidle is spending less time in idle polling, which means that wakeup latency increases. That obviously might be a result of the timer migration properties. Do you have traces (before and after) handy to share? Thanks, tglx ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2024-03-13 8:25 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-03-01 8:09 [tip:timers/core] [timers] 7ee9887703: netperf.Throughput_Mbps -1.2% regression kernel test robot 2024-03-04 0:32 ` Frederic Weisbecker 2024-03-04 2:13 ` Oliver Sang 2024-03-04 11:28 ` Frederic Weisbecker 2024-03-05 2:17 ` Oliver Sang 2024-03-05 10:46 ` Frederic Weisbecker 2024-03-05 11:21 ` Frederic Weisbecker 2024-03-05 11:35 ` Frederic Weisbecker 2024-03-12 23:57 ` Frederic Weisbecker 2024-03-13 8:25 ` Thomas Gleixner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox