From: kernel test robot <oliver.sang@intel.com>
To: Thomas Gleixner <tglx@kernel.org>
Cc: <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
<linux-kernel@vger.kernel.org>,
Peter Zijlstra <peterz@infradead.org>,
<aubrey.li@linux.intel.com>, <yu.c.chen@intel.com>,
<oliver.sang@intel.com>
Subject: [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression
Date: Fri, 24 Apr 2026 15:10:07 +0800 [thread overview]
Message-ID: <202604241448.365c92f9-lkp@intel.com> (raw)
Hello,
kernel test robot noticed a 2.8% regression of will-it-scale.per_process_ops on:
commit: 96d1610e0b20b5a627773874b4514ae922ad98f6 ("sched: Optimize hrtimer handling")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
[still regression on linus/master 1d51b370a0f8f642f4fc84c795fbedac0fcdbbd2]
[still regression on linux-next/master 936c21068d7ade00325e40d82bfd2f3f29d9f659]
[still regression on fix commit eef9f648fb0e92618041f019d4bdcf7ae17cb743]
testcase: will-it-scale
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
parameters:
nr_task: 100%
mode: process
test: context_switch1
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202604241448.365c92f9-lkp@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20260424/202604241448.365c92f9-lkp@intel.com
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/process/100%/debian-13-x86_64-20250902.cgz/lkp-ivb-2ep2/context_switch1/will-it-scale
commit:
c3a92213eb ("sched: Use hrtimer_highres_enabled()")
96d1610e0b ("sched: Optimize hrtimer handling")
c3a92213eb3dd8ea 96d1610e0b20b5a627773874b45
---------------- ---------------------------
%stddev %change %stddev
\ | \
11427219 -2.8% 11104140 will-it-scale.48.processes
238066 -2.8% 231335 will-it-scale.per_process_ops
11427219 -2.8% 11104140 will-it-scale.workload
11248988 -2.7% 10943054 vmstat.system.cs
1.394e+10 -2.3% 1.362e+10 perf-stat.i.branch-instructions
1.94e+08 -1.8% 1.906e+08 perf-stat.i.branch-misses
98548808 ± 5% +24.2% 1.224e+08 ± 5% perf-stat.i.cache-references
11359604 -2.7% 11053971 perf-stat.i.context-switches
6.86e+10 -2.3% 6.705e+10 perf-stat.i.instructions
0.49 -2.4% 0.48 perf-stat.i.ipc
236.66 -2.7% 230.29 perf-stat.i.metric.K/sec
2.04 +2.5% 2.09 perf-stat.overall.cpi
0.49 -2.5% 0.48 perf-stat.overall.ipc
1.389e+10 -2.3% 1.357e+10 perf-stat.ps.branch-instructions
1.933e+08 -1.8% 1.899e+08 perf-stat.ps.branch-misses
98259743 ± 5% +24.2% 1.22e+08 ± 5% perf-stat.ps.cache-references
11321664 -2.7% 11016986 perf-stat.ps.context-switches
6.837e+10 -2.3% 6.683e+10 perf-stat.ps.instructions
2.077e+13 -2.4% 2.028e+13 perf-stat.total.instructions
3.02 -0.1 2.90 perf-profile.calltrace.cycles-pp.pick_task_fair.pick_next_task_fair.__pick_next_task.__schedule.schedule
3.79 -0.1 3.68 perf-profile.calltrace.cycles-pp.perf_pmu_sched_task.__perf_event_task_sched_out.prepare_task_switch.__schedule.schedule
5.13 -0.1 5.04 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.exit_to_user_mode_loop
2.72 -0.1 2.64 perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__pick_next_task.__schedule.schedule
2.87 -0.1 2.79 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.anon_pipe_read.vfs_read
5.27 -0.1 5.19 perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.exit_to_user_mode_loop.do_syscall_64
1.82 -0.1 1.76 perf-profile.calltrace.cycles-pp.perf_pmu_sched_task.finish_task_switch.__schedule.schedule.anon_pipe_read
1.41 -0.1 1.35 perf-profile.calltrace.cycles-pp.__switch_to
0.84 -0.0 0.79 ± 2% perf-profile.calltrace.cycles-pp.anon_pipe_buf_release.anon_pipe_read.vfs_read.ksys_read.do_syscall_64
1.64 -0.0 1.59 perf-profile.calltrace.cycles-pp.intel_pmu_disable_all.perf_pmu_sched_task.__perf_event_task_sched_out.prepare_task_switch.__schedule
0.56 -0.0 0.52 ± 2% perf-profile.calltrace.cycles-pp.rw_verify_area.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.38 -0.0 1.34 perf-profile.calltrace.cycles-pp.update_curr.pick_task_fair.pick_next_task_fair.__pick_next_task.__schedule
0.64 +0.0 0.67 perf-profile.calltrace.cycles-pp.os_xsave
1.25 +0.0 1.30 perf-profile.calltrace.cycles-pp.__switch_to_asm
2.62 +0.1 2.68 perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__pick_next_task.__schedule.schedule
3.19 +0.1 3.27 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.exit_to_user_mode_loop.do_syscall_64
0.62 ± 3% +0.1 0.70 ± 3% perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.exit_to_user_mode_loop
0.65 ± 3% +0.1 0.73 ± 5% perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.anon_pipe_read
1.78 ± 2% +0.1 1.86 perf-profile.calltrace.cycles-pp.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.58 ± 3% +0.1 0.67 ± 2% perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.anon_pipe_write.vfs_write.ksys_write
3.27 +0.1 3.37 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.anon_pipe_read.vfs_read
1.38 +0.1 1.48 ± 2% perf-profile.calltrace.cycles-pp.copy_page_from_iter.anon_pipe_write.vfs_write.ksys_write.do_syscall_64
1.34 ± 3% +0.2 1.51 ± 7% perf-profile.calltrace.cycles-pp.update_curr.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up.autoremove_wake_function
1.08 ± 7% +0.2 1.29 ± 10% perf-profile.calltrace.cycles-pp.update_se.update_curr.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up
1.78 ± 3% +0.2 2.00 ± 6% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task
2.18 +0.2 2.41 ± 3% perf-profile.calltrace.cycles-pp.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up.autoremove_wake_function.__wake_up_common
2.58 +0.2 2.82 ± 2% perf-profile.calltrace.cycles-pp.wakeup_preempt.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
1.33 ± 6% +0.3 1.59 ± 9% perf-profile.calltrace.cycles-pp.update_se.update_curr.dequeue_entity.dequeue_entities.dequeue_task_fair
3.38 ± 2% +0.3 3.65 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule
4.35 ± 2% +0.3 4.66 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule
5.12 +0.3 5.44 ± 2% perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.anon_pipe_read.vfs_read
4.79 ± 2% +0.3 5.12 ± 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.anon_pipe_read
20.64 +0.4 21.01 perf-profile.calltrace.cycles-pp.schedule.anon_pipe_read.vfs_read.ksys_read.do_syscall_64
0.00 +0.6 0.57 ± 7% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_stat_runtime.perf_trace_sched_stat_runtime.update_se.update_curr.wakeup_preempt_fair
0.00 +0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_se.update_curr.wakeup_preempt_fair.wakeup_preempt
0.00 +0.6 0.63 ± 12% perf-profile.calltrace.cycles-pp.read
0.00 +0.7 0.69 ± 2% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_stat_runtime.perf_trace_sched_stat_runtime.update_se.update_curr.dequeue_entity
0.00 +0.7 0.71 ± 2% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_se.update_curr.dequeue_entity.dequeue_entities
7.45 -0.2 7.26 perf-profile.children.cycles-pp.perf_pmu_sched_task
9.62 -0.1 9.48 perf-profile.children.cycles-pp.__pick_next_task
3.09 -0.1 2.98 perf-profile.children.cycles-pp.pick_task_fair
5.78 -0.1 5.68 perf-profile.children.cycles-pp.finish_task_switch
2.88 -0.1 2.78 perf-profile.children.cycles-pp.put_prev_entity
1.03 -0.1 0.94 perf-profile.children.cycles-pp.__perf_event_task_sched_in
0.29 ± 3% -0.1 0.21 ± 4% perf-profile.children.cycles-pp.__rdgsbase_inactive
3.20 -0.1 3.12 perf-profile.children.cycles-pp.intel_pmu_disable_all
0.30 -0.1 0.22 ± 4% perf-profile.children.cycles-pp.propagate_entity_load_avg
0.40 -0.1 0.34 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.67 -0.1 0.60 ± 3% perf-profile.children.cycles-pp.security_file_permission
1.60 -0.1 1.54 perf-profile.children.cycles-pp.__switch_to
1.01 -0.1 0.96 ± 2% perf-profile.children.cycles-pp.rw_verify_area
1.18 -0.1 1.12 ± 2% perf-profile.children.cycles-pp.__pick_eevdf
1.73 ± 2% -0.1 1.67 perf-profile.children.cycles-pp.avg_vruntime
1.00 ± 2% -0.1 0.95 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.65 ± 4% -0.1 0.60 ± 2% perf-profile.children.cycles-pp.update_entity_lag
0.84 -0.0 0.80 ± 2% perf-profile.children.cycles-pp.anon_pipe_buf_release
0.60 ± 2% -0.0 0.56 perf-profile.children.cycles-pp.vruntime_eligible
0.42 ± 2% -0.0 0.39 ± 2% perf-profile.children.cycles-pp.__x64_sys_read
0.56 -0.0 0.53 perf-profile.children.cycles-pp.update_rq_clock_task
0.18 ± 3% -0.0 0.15 ± 3% perf-profile.children.cycles-pp.perf_cgroup_switch
0.11 ± 6% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.save_fpregs_to_fpstate
0.81 -0.0 0.78 perf-profile.children.cycles-pp._raw_spin_lock
0.60 ± 2% -0.0 0.57 perf-profile.children.cycles-pp.native_sched_clock
0.34 ± 2% -0.0 0.31 ± 4% perf-profile.children.cycles-pp.child
0.17 ± 4% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.pick_next_task
0.97 -0.0 0.95 perf-profile.children.cycles-pp.update_rq_clock
0.16 ± 3% -0.0 0.14 ± 4% perf-profile.children.cycles-pp.dequeue_task
0.05 +0.0 0.06 ± 7% perf-profile.children.cycles-pp.__x86_indirect_thunk_rdx
0.27 ± 2% +0.0 0.29 ± 3% perf-profile.children.cycles-pp.place_entity
0.65 +0.0 0.67 ± 2% perf-profile.children.cycles-pp.os_xsave
0.08 ± 6% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.raw_spin_rq_unlock
0.09 ± 5% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.rb_next
0.11 ± 16% +0.0 0.15 ± 15% perf-profile.children.cycles-pp.perf_swevent_event
0.35 ± 2% +0.0 0.40 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
2.72 +0.1 2.77 perf-profile.children.cycles-pp.set_next_entity
1.18 +0.1 1.24 perf-profile.children.cycles-pp.update_load_avg
0.09 ± 6% +0.1 0.16 ± 6% perf-profile.children.cycles-pp.kill_fasync
1.31 +0.1 1.38 perf-profile.children.cycles-pp.__switch_to_asm
1.78 ± 2% +0.1 1.87 perf-profile.children.cycles-pp.switch_fpu_return
0.59 ± 3% +0.1 0.68 ± 2% perf-profile.children.cycles-pp._copy_from_iter
1.40 +0.1 1.49 ± 2% perf-profile.children.cycles-pp.copy_page_from_iter
1.04 +0.1 1.14 perf-profile.children.cycles-pp.__update_load_avg_se
1.29 ± 3% +0.2 1.45 ± 4% perf-profile.children.cycles-pp.___perf_sw_event
6.50 +0.2 6.69 perf-profile.children.cycles-pp.prepare_task_switch
2.34 +0.2 2.59 ± 2% perf-profile.children.cycles-pp.wakeup_preempt_fair
2.78 +0.3 3.03 ± 2% perf-profile.children.cycles-pp.wakeup_preempt
3.43 ± 2% +0.3 3.70 ± 2% perf-profile.children.cycles-pp.dequeue_entity
4.39 ± 2% +0.3 4.70 ± 2% perf-profile.children.cycles-pp.dequeue_entities
5.13 +0.3 5.45 ± 2% perf-profile.children.cycles-pp.try_to_block_task
4.80 ± 2% +0.3 5.14 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair
5.64 +0.3 5.99 ± 4% perf-profile.children.cycles-pp.update_curr
0.34 ± 9% +0.3 0.68 ± 11% perf-profile.children.cycles-pp.read
0.64 +0.4 1.06 perf-profile.children.cycles-pp.strnlen
1.38 ± 4% +0.4 1.81 ± 3% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
1.30 ± 4% +0.4 1.73 ± 3% perf-profile.children.cycles-pp.do_perf_trace_sched_stat_runtime
3.76 ± 5% +0.5 4.24 ± 7% perf-profile.children.cycles-pp.update_se
36.56 +0.5 37.10 perf-profile.children.cycles-pp.schedule
1.01 ± 3% -0.2 0.86 ± 7% perf-profile.self.cycles-pp.try_to_wake_up
1.82 ± 4% -0.1 1.68 ± 6% perf-profile.self.cycles-pp.vfs_write
0.60 -0.1 0.52 ± 2% perf-profile.self.cycles-pp.put_prev_entity
3.18 -0.1 3.10 perf-profile.self.cycles-pp.intel_pmu_disable_all
0.28 ± 3% -0.1 0.20 ± 3% perf-profile.self.cycles-pp.__rdgsbase_inactive
1.54 -0.1 1.46 perf-profile.self.cycles-pp.__switch_to
0.25 -0.1 0.18 ± 4% perf-profile.self.cycles-pp.propagate_entity_load_avg
0.30 ± 4% -0.1 0.23 ± 3% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.65 -0.1 0.58 ± 4% perf-profile.self.cycles-pp.security_file_permission
1.70 -0.1 1.64 perf-profile.self.cycles-pp.avg_vruntime
0.82 -0.1 0.76 perf-profile.self.cycles-pp.__perf_event_task_sched_in
0.31 ± 3% -0.1 0.25 ± 2% perf-profile.self.cycles-pp.__wake_up_common
0.96 -0.0 0.91 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.70 -0.0 0.65 ± 2% perf-profile.self.cycles-pp.__pick_eevdf
0.80 -0.0 0.75 ± 2% perf-profile.self.cycles-pp.anon_pipe_buf_release
0.56 ± 2% -0.0 0.52 ± 2% perf-profile.self.cycles-pp.vruntime_eligible
0.59 ± 2% -0.0 0.55 perf-profile.self.cycles-pp.native_sched_clock
0.50 -0.0 0.46 perf-profile.self.cycles-pp.update_rq_clock_task
0.53 -0.0 0.50 ± 2% perf-profile.self.cycles-pp.pick_task_fair
0.88 -0.0 0.85 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.18 ± 3% -0.0 0.15 ± 3% perf-profile.self.cycles-pp.perf_cgroup_switch
0.67 ± 2% -0.0 0.64 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.11 ± 6% -0.0 0.08 perf-profile.self.cycles-pp.save_fpregs_to_fpstate
0.40 -0.0 0.37 ± 2% perf-profile.self.cycles-pp.__x64_sys_read
0.30 ± 3% -0.0 0.27 ± 5% perf-profile.self.cycles-pp.child
0.18 ± 5% -0.0 0.15 ± 4% perf-profile.self.cycles-pp.update_rq_clock
0.15 ± 3% -0.0 0.13 ± 5% perf-profile.self.cycles-pp.pick_next_task
0.08 ± 5% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.enqueue_entity
0.16 ± 3% -0.0 0.14 ± 4% perf-profile.self.cycles-pp.dequeue_task
0.07 ± 6% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.rb_next
0.07 ± 7% +0.0 0.09 ± 6% perf-profile.self.cycles-pp.raw_spin_rq_unlock
0.24 ± 3% +0.0 0.27 ± 2% perf-profile.self.cycles-pp.ksys_read
0.64 +0.0 0.67 ± 2% perf-profile.self.cycles-pp.os_xsave
0.12 ± 6% +0.0 0.15 ± 4% perf-profile.self.cycles-pp.enqueue_task
0.71 ± 2% +0.0 0.75 perf-profile.self.cycles-pp.pick_next_task_fair
0.55 +0.0 0.59 ± 2% perf-profile.self.cycles-pp.__enqueue_entity
0.02 ±141% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.__x86_indirect_thunk_rdx
0.10 ± 15% +0.0 0.14 ± 12% perf-profile.self.cycles-pp.perf_swevent_event
0.45 ± 3% +0.0 0.50 ± 3% perf-profile.self.cycles-pp.switch_fpu_return
0.30 ± 3% +0.0 0.35 ± 2% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.46 +0.1 0.51 perf-profile.self.cycles-pp.dequeue_entities
0.06 ± 7% +0.1 0.13 ± 6% perf-profile.self.cycles-pp.kill_fasync
1.30 +0.1 1.37 perf-profile.self.cycles-pp.__switch_to_asm
0.38 ± 4% +0.1 0.46 ± 2% perf-profile.self.cycles-pp._copy_from_iter
0.80 ± 4% +0.1 0.89 ± 2% perf-profile.self.cycles-pp.prepare_task_switch
0.67 ± 2% +0.1 0.78 ± 3% perf-profile.self.cycles-pp.dequeue_entity
1.01 +0.1 1.11 perf-profile.self.cycles-pp.__update_load_avg_se
0.57 ± 2% +0.1 0.68 ± 2% perf-profile.self.cycles-pp.wakeup_preempt_fair
0.48 ± 2% +0.2 0.70 ± 2% perf-profile.self.cycles-pp.schedule
0.21 ± 7% +0.3 0.52 ± 5% perf-profile.self.cycles-pp.read
0.63 +0.4 1.05 perf-profile.self.cycles-pp.strnlen
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
next reply other threads:[~2026-04-24 7:10 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-04-24 7:10 kernel test robot [this message]
2026-04-24 9:55 ` [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression Peter Zijlstra
2026-04-27 5:59 ` Oliver Sang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=202604241448.365c92f9-lkp@intel.com \
--to=oliver.sang@intel.com \
--cc=aubrey.li@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lkp@intel.com \
--cc=oe-lkp@lists.linux.dev \
--cc=peterz@infradead.org \
--cc=tglx@kernel.org \
--cc=yu.c.chen@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.