From: kernel test robot <oliver.sang@intel.com>
To: Thomas Gleixner <tglx@kernel.org>
Cc: <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
<linux-kernel@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
<aubrey.li@linux.intel.com>, <yu.c.chen@intel.com>,
<oliver.sang@intel.com>
Subject: [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression
Date: Fri, 24 Apr 2026 15:10:07 +0800 [thread overview]
Message-ID: <202604241448.365c92f9-lkp@intel.com> (raw)
Hello,
kernel test robot noticed a 2.8% regression of will-it-scale.per_process_ops on:
commit: 96d1610e0b20b5a627773874b4514ae922ad98f6 ("sched: Optimize hrtimer handling")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
[still regression on linus/master 1d51b370a0f8f642f4fc84c795fbedac0fcdbbd2]
[still regression on linux-next/master 936c21068d7ade00325e40d82bfd2f3f29d9f659]
[still regression on fix commit eef9f648fb0e92618041f019d4bdcf7ae17cb743]
testcase: will-it-scale
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
parameters:
nr_task: 100%
mode: process
test: context_switch1
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202604241448.365c92f9-lkp@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20260424/202604241448.365c92f9-lkp@intel.com
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/process/100%/debian-13-x86_64-20250902.cgz/lkp-ivb-2ep2/context_switch1/will-it-scale
commit:
c3a92213eb ("sched: Use hrtimer_highres_enabled()")
96d1610e0b ("sched: Optimize hrtimer handling")
c3a92213eb3dd8ea 96d1610e0b20b5a627773874b45
---------------- ---------------------------
%stddev %change %stddev
\ | \
11427219 -2.8% 11104140 will-it-scale.48.processes
238066 -2.8% 231335 will-it-scale.per_process_ops
11427219 -2.8% 11104140 will-it-scale.workload
11248988 -2.7% 10943054 vmstat.system.cs
1.394e+10 -2.3% 1.362e+10 perf-stat.i.branch-instructions
1.94e+08 -1.8% 1.906e+08 perf-stat.i.branch-misses
98548808 ± 5% +24.2% 1.224e+08 ± 5% perf-stat.i.cache-references
11359604 -2.7% 11053971 perf-stat.i.context-switches
6.86e+10 -2.3% 6.705e+10 perf-stat.i.instructions
0.49 -2.4% 0.48 perf-stat.i.ipc
236.66 -2.7% 230.29 perf-stat.i.metric.K/sec
2.04 +2.5% 2.09 perf-stat.overall.cpi
0.49 -2.5% 0.48 perf-stat.overall.ipc
1.389e+10 -2.3% 1.357e+10 perf-stat.ps.branch-instructions
1.933e+08 -1.8% 1.899e+08 perf-stat.ps.branch-misses
98259743 ± 5% +24.2% 1.22e+08 ± 5% perf-stat.ps.cache-references
11321664 -2.7% 11016986 perf-stat.ps.context-switches
6.837e+10 -2.3% 6.683e+10 perf-stat.ps.instructions
2.077e+13 -2.4% 2.028e+13 perf-stat.total.instructions
3.02 -0.1 2.90 perf-profile.calltrace.cycles-pp.pick_task_fair.pick_next_task_fair.__pick_next_task.__schedule.schedule
3.79 -0.1 3.68 perf-profile.calltrace.cycles-pp.perf_pmu_sched_task.__perf_event_task_sched_out.prepare_task_switch.__schedule.schedule
5.13 -0.1 5.04 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.exit_to_user_mode_loop
2.72 -0.1 2.64 perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__pick_next_task.__schedule.schedule
2.87 -0.1 2.79 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.anon_pipe_read.vfs_read
5.27 -0.1 5.19 perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.exit_to_user_mode_loop.do_syscall_64
1.82 -0.1 1.76 perf-profile.calltrace.cycles-pp.perf_pmu_sched_task.finish_task_switch.__schedule.schedule.anon_pipe_read
1.41 -0.1 1.35 perf-profile.calltrace.cycles-pp.__switch_to
0.84 -0.0 0.79 ± 2% perf-profile.calltrace.cycles-pp.anon_pipe_buf_release.anon_pipe_read.vfs_read.ksys_read.do_syscall_64
1.64 -0.0 1.59 perf-profile.calltrace.cycles-pp.intel_pmu_disable_all.perf_pmu_sched_task.__perf_event_task_sched_out.prepare_task_switch.__schedule
0.56 -0.0 0.52 ± 2% perf-profile.calltrace.cycles-pp.rw_verify_area.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.38 -0.0 1.34 perf-profile.calltrace.cycles-pp.update_curr.pick_task_fair.pick_next_task_fair.__pick_next_task.__schedule
0.64 +0.0 0.67 perf-profile.calltrace.cycles-pp.os_xsave
1.25 +0.0 1.30 perf-profile.calltrace.cycles-pp.__switch_to_asm
2.62 +0.1 2.68 perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__pick_next_task.__schedule.schedule
3.19 +0.1 3.27 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.exit_to_user_mode_loop.do_syscall_64
0.62 ± 3% +0.1 0.70 ± 3% perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.exit_to_user_mode_loop
0.65 ± 3% +0.1 0.73 ± 5% perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.anon_pipe_read
1.78 ± 2% +0.1 1.86 perf-profile.calltrace.cycles-pp.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.58 ± 3% +0.1 0.67 ± 2% perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.anon_pipe_write.vfs_write.ksys_write
3.27 +0.1 3.37 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.anon_pipe_read.vfs_read
1.38 +0.1 1.48 ± 2% perf-profile.calltrace.cycles-pp.copy_page_from_iter.anon_pipe_write.vfs_write.ksys_write.do_syscall_64
1.34 ± 3% +0.2 1.51 ± 7% perf-profile.calltrace.cycles-pp.update_curr.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up.autoremove_wake_function
1.08 ± 7% +0.2 1.29 ± 10% perf-profile.calltrace.cycles-pp.update_se.update_curr.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up
1.78 ± 3% +0.2 2.00 ± 6% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task
2.18 +0.2 2.41 ± 3% perf-profile.calltrace.cycles-pp.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up.autoremove_wake_function.__wake_up_common
2.58 +0.2 2.82 ± 2% perf-profile.calltrace.cycles-pp.wakeup_preempt.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
1.33 ± 6% +0.3 1.59 ± 9% perf-profile.calltrace.cycles-pp.update_se.update_curr.dequeue_entity.dequeue_entities.dequeue_task_fair
3.38 ± 2% +0.3 3.65 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule
4.35 ± 2% +0.3 4.66 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule
5.12 +0.3 5.44 ± 2% perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.anon_pipe_read.vfs_read
4.79 ± 2% +0.3 5.12 ± 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.anon_pipe_read
20.64 +0.4 21.01 perf-profile.calltrace.cycles-pp.schedule.anon_pipe_read.vfs_read.ksys_read.do_syscall_64
0.00 +0.6 0.57 ± 7% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_stat_runtime.perf_trace_sched_stat_runtime.update_se.update_curr.wakeup_preempt_fair
0.00 +0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_se.update_curr.wakeup_preempt_fair.wakeup_preempt
0.00 +0.6 0.63 ± 12% perf-profile.calltrace.cycles-pp.read
0.00 +0.7 0.69 ± 2% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_stat_runtime.perf_trace_sched_stat_runtime.update_se.update_curr.dequeue_entity
0.00 +0.7 0.71 ± 2% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_se.update_curr.dequeue_entity.dequeue_entities
7.45 -0.2 7.26 perf-profile.children.cycles-pp.perf_pmu_sched_task
9.62 -0.1 9.48 perf-profile.children.cycles-pp.__pick_next_task
3.09 -0.1 2.98 perf-profile.children.cycles-pp.pick_task_fair
5.78 -0.1 5.68 perf-profile.children.cycles-pp.finish_task_switch
2.88 -0.1 2.78 perf-profile.children.cycles-pp.put_prev_entity
1.03 -0.1 0.94 perf-profile.children.cycles-pp.__perf_event_task_sched_in
0.29 ± 3% -0.1 0.21 ± 4% perf-profile.children.cycles-pp.__rdgsbase_inactive
3.20 -0.1 3.12 perf-profile.children.cycles-pp.intel_pmu_disable_all
0.30 -0.1 0.22 ± 4% perf-profile.children.cycles-pp.propagate_entity_load_avg
0.40 -0.1 0.34 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.67 -0.1 0.60 ± 3% perf-profile.children.cycles-pp.security_file_permission
1.60 -0.1 1.54 perf-profile.children.cycles-pp.__switch_to
1.01 -0.1 0.96 ± 2% perf-profile.children.cycles-pp.rw_verify_area
1.18 -0.1 1.12 ± 2% perf-profile.children.cycles-pp.__pick_eevdf
1.73 ± 2% -0.1 1.67 perf-profile.children.cycles-pp.avg_vruntime
1.00 ± 2% -0.1 0.95 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.65 ± 4% -0.1 0.60 ± 2% perf-profile.children.cycles-pp.update_entity_lag
0.84 -0.0 0.80 ± 2% perf-profile.children.cycles-pp.anon_pipe_buf_release
0.60 ± 2% -0.0 0.56 perf-profile.children.cycles-pp.vruntime_eligible
0.42 ± 2% -0.0 0.39 ± 2% perf-profile.children.cycles-pp.__x64_sys_read
0.56 -0.0 0.53 perf-profile.children.cycles-pp.update_rq_clock_task
0.18 ± 3% -0.0 0.15 ± 3% perf-profile.children.cycles-pp.perf_cgroup_switch
0.11 ± 6% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.save_fpregs_to_fpstate
0.81 -0.0 0.78 perf-profile.children.cycles-pp._raw_spin_lock
0.60 ± 2% -0.0 0.57 perf-profile.children.cycles-pp.native_sched_clock
0.34 ± 2% -0.0 0.31 ± 4% perf-profile.children.cycles-pp.child
0.17 ± 4% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.pick_next_task
0.97 -0.0 0.95 perf-profile.children.cycles-pp.update_rq_clock
0.16 ± 3% -0.0 0.14 ± 4% perf-profile.children.cycles-pp.dequeue_task
0.05 +0.0 0.06 ± 7% perf-profile.children.cycles-pp.__x86_indirect_thunk_rdx
0.27 ± 2% +0.0 0.29 ± 3% perf-profile.children.cycles-pp.place_entity
0.65 +0.0 0.67 ± 2% perf-profile.children.cycles-pp.os_xsave
0.08 ± 6% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.raw_spin_rq_unlock
0.09 ± 5% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.rb_next
0.11 ± 16% +0.0 0.15 ± 15% perf-profile.children.cycles-pp.perf_swevent_event
0.35 ± 2% +0.0 0.40 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
2.72 +0.1 2.77 perf-profile.children.cycles-pp.set_next_entity
1.18 +0.1 1.24 perf-profile.children.cycles-pp.update_load_avg
0.09 ± 6% +0.1 0.16 ± 6% perf-profile.children.cycles-pp.kill_fasync
1.31 +0.1 1.38 perf-profile.children.cycles-pp.__switch_to_asm
1.78 ± 2% +0.1 1.87 perf-profile.children.cycles-pp.switch_fpu_return
0.59 ± 3% +0.1 0.68 ± 2% perf-profile.children.cycles-pp._copy_from_iter
1.40 +0.1 1.49 ± 2% perf-profile.children.cycles-pp.copy_page_from_iter
1.04 +0.1 1.14 perf-profile.children.cycles-pp.__update_load_avg_se
1.29 ± 3% +0.2 1.45 ± 4% perf-profile.children.cycles-pp.___perf_sw_event
6.50 +0.2 6.69 perf-profile.children.cycles-pp.prepare_task_switch
2.34 +0.2 2.59 ± 2% perf-profile.children.cycles-pp.wakeup_preempt_fair
2.78 +0.3 3.03 ± 2% perf-profile.children.cycles-pp.wakeup_preempt
3.43 ± 2% +0.3 3.70 ± 2% perf-profile.children.cycles-pp.dequeue_entity
4.39 ± 2% +0.3 4.70 ± 2% perf-profile.children.cycles-pp.dequeue_entities
5.13 +0.3 5.45 ± 2% perf-profile.children.cycles-pp.try_to_block_task
4.80 ± 2% +0.3 5.14 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair
5.64 +0.3 5.99 ± 4% perf-profile.children.cycles-pp.update_curr
0.34 ± 9% +0.3 0.68 ± 11% perf-profile.children.cycles-pp.read
0.64 +0.4 1.06 perf-profile.children.cycles-pp.strnlen
1.38 ± 4% +0.4 1.81 ± 3% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
1.30 ± 4% +0.4 1.73 ± 3% perf-profile.children.cycles-pp.do_perf_trace_sched_stat_runtime
3.76 ± 5% +0.5 4.24 ± 7% perf-profile.children.cycles-pp.update_se
36.56 +0.5 37.10 perf-profile.children.cycles-pp.schedule
1.01 ± 3% -0.2 0.86 ± 7% perf-profile.self.cycles-pp.try_to_wake_up
1.82 ± 4% -0.1 1.68 ± 6% perf-profile.self.cycles-pp.vfs_write
0.60 -0.1 0.52 ± 2% perf-profile.self.cycles-pp.put_prev_entity
3.18 -0.1 3.10 perf-profile.self.cycles-pp.intel_pmu_disable_all
0.28 ± 3% -0.1 0.20 ± 3% perf-profile.self.cycles-pp.__rdgsbase_inactive
1.54 -0.1 1.46 perf-profile.self.cycles-pp.__switch_to
0.25 -0.1 0.18 ± 4% perf-profile.self.cycles-pp.propagate_entity_load_avg
0.30 ± 4% -0.1 0.23 ± 3% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.65 -0.1 0.58 ± 4% perf-profile.self.cycles-pp.security_file_permission
1.70 -0.1 1.64 perf-profile.self.cycles-pp.avg_vruntime
0.82 -0.1 0.76 perf-profile.self.cycles-pp.__perf_event_task_sched_in
0.31 ± 3% -0.1 0.25 ± 2% perf-profile.self.cycles-pp.__wake_up_common
0.96 -0.0 0.91 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.70 -0.0 0.65 ± 2% perf-profile.self.cycles-pp.__pick_eevdf
0.80 -0.0 0.75 ± 2% perf-profile.self.cycles-pp.anon_pipe_buf_release
0.56 ± 2% -0.0 0.52 ± 2% perf-profile.self.cycles-pp.vruntime_eligible
0.59 ± 2% -0.0 0.55 perf-profile.self.cycles-pp.native_sched_clock
0.50 -0.0 0.46 perf-profile.self.cycles-pp.update_rq_clock_task
0.53 -0.0 0.50 ± 2% perf-profile.self.cycles-pp.pick_task_fair
0.88 -0.0 0.85 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.18 ± 3% -0.0 0.15 ± 3% perf-profile.self.cycles-pp.perf_cgroup_switch
0.67 ± 2% -0.0 0.64 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.11 ± 6% -0.0 0.08 perf-profile.self.cycles-pp.save_fpregs_to_fpstate
0.40 -0.0 0.37 ± 2% perf-profile.self.cycles-pp.__x64_sys_read
0.30 ± 3% -0.0 0.27 ± 5% perf-profile.self.cycles-pp.child
0.18 ± 5% -0.0 0.15 ± 4% perf-profile.self.cycles-pp.update_rq_clock
0.15 ± 3% -0.0 0.13 ± 5% perf-profile.self.cycles-pp.pick_next_task
0.08 ± 5% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.enqueue_entity
0.16 ± 3% -0.0 0.14 ± 4% perf-profile.self.cycles-pp.dequeue_task
0.07 ± 6% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.rb_next
0.07 ± 7% +0.0 0.09 ± 6% perf-profile.self.cycles-pp.raw_spin_rq_unlock
0.24 ± 3% +0.0 0.27 ± 2% perf-profile.self.cycles-pp.ksys_read
0.64 +0.0 0.67 ± 2% perf-profile.self.cycles-pp.os_xsave
0.12 ± 6% +0.0 0.15 ± 4% perf-profile.self.cycles-pp.enqueue_task
0.71 ± 2% +0.0 0.75 perf-profile.self.cycles-pp.pick_next_task_fair
0.55 +0.0 0.59 ± 2% perf-profile.self.cycles-pp.__enqueue_entity
0.02 ±141% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.__x86_indirect_thunk_rdx
0.10 ± 15% +0.0 0.14 ± 12% perf-profile.self.cycles-pp.perf_swevent_event
0.45 ± 3% +0.0 0.50 ± 3% perf-profile.self.cycles-pp.switch_fpu_return
0.30 ± 3% +0.0 0.35 ± 2% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.46 +0.1 0.51 perf-profile.self.cycles-pp.dequeue_entities
0.06 ± 7% +0.1 0.13 ± 6% perf-profile.self.cycles-pp.kill_fasync
1.30 +0.1 1.37 perf-profile.self.cycles-pp.__switch_to_asm
0.38 ± 4% +0.1 0.46 ± 2% perf-profile.self.cycles-pp._copy_from_iter
0.80 ± 4% +0.1 0.89 ± 2% perf-profile.self.cycles-pp.prepare_task_switch
0.67 ± 2% +0.1 0.78 ± 3% perf-profile.self.cycles-pp.dequeue_entity
1.01 +0.1 1.11 perf-profile.self.cycles-pp.__update_load_avg_se
0.57 ± 2% +0.1 0.68 ± 2% perf-profile.self.cycles-pp.wakeup_preempt_fair
0.48 ± 2% +0.2 0.70 ± 2% perf-profile.self.cycles-pp.schedule
0.21 ± 7% +0.3 0.52 ± 5% perf-profile.self.cycles-pp.read
0.63 +0.4 1.05 perf-profile.self.cycles-pp.strnlen
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
next reply other threads:[~2026-04-24 7:10 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-24 7:10 kernel test robot [this message]
2026-04-24 9:55 ` [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression Peter Zijlstra
2026-04-27 5:59 ` Oliver Sang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=202604241448.365c92f9-lkp@intel.com \
--to=oliver.sang@intel.com \
--cc=aubrey.li@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lkp@intel.com \
--cc=oe-lkp@lists.linux.dev \
--cc=peterz@infradead.org \
--cc=tglx@kernel.org \
--cc=yu.c.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox