* [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression
@ 2026-04-24 7:10 kernel test robot
2026-04-24 9:55 ` Peter Zijlstra
0 siblings, 1 reply; 3+ messages in thread
From: kernel test robot @ 2026-04-24 7:10 UTC (permalink / raw)
To: Thomas Gleixner
Cc: oe-lkp, lkp, linux-kernel, Peter Zijlstra, aubrey.li, yu.c.chen,
oliver.sang
Hello,
kernel test robot noticed a 2.8% regression of will-it-scale.per_process_ops on:
commit: 96d1610e0b20b5a627773874b4514ae922ad98f6 ("sched: Optimize hrtimer handling")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
[still regression on linus/master 1d51b370a0f8f642f4fc84c795fbedac0fcdbbd2]
[still regression on linux-next/master 936c21068d7ade00325e40d82bfd2f3f29d9f659]
[still regression on fix commit eef9f648fb0e92618041f019d4bdcf7ae17cb743]
testcase: will-it-scale
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
parameters:
nr_task: 100%
mode: process
test: context_switch1
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202604241448.365c92f9-lkp@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20260424/202604241448.365c92f9-lkp@intel.com
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/process/100%/debian-13-x86_64-20250902.cgz/lkp-ivb-2ep2/context_switch1/will-it-scale
commit:
c3a92213eb ("sched: Use hrtimer_highres_enabled()")
96d1610e0b ("sched: Optimize hrtimer handling")
c3a92213eb3dd8ea 96d1610e0b20b5a627773874b45
---------------- ---------------------------
%stddev %change %stddev
\ | \
11427219 -2.8% 11104140 will-it-scale.48.processes
238066 -2.8% 231335 will-it-scale.per_process_ops
11427219 -2.8% 11104140 will-it-scale.workload
11248988 -2.7% 10943054 vmstat.system.cs
1.394e+10 -2.3% 1.362e+10 perf-stat.i.branch-instructions
1.94e+08 -1.8% 1.906e+08 perf-stat.i.branch-misses
98548808 ± 5% +24.2% 1.224e+08 ± 5% perf-stat.i.cache-references
11359604 -2.7% 11053971 perf-stat.i.context-switches
6.86e+10 -2.3% 6.705e+10 perf-stat.i.instructions
0.49 -2.4% 0.48 perf-stat.i.ipc
236.66 -2.7% 230.29 perf-stat.i.metric.K/sec
2.04 +2.5% 2.09 perf-stat.overall.cpi
0.49 -2.5% 0.48 perf-stat.overall.ipc
1.389e+10 -2.3% 1.357e+10 perf-stat.ps.branch-instructions
1.933e+08 -1.8% 1.899e+08 perf-stat.ps.branch-misses
98259743 ± 5% +24.2% 1.22e+08 ± 5% perf-stat.ps.cache-references
11321664 -2.7% 11016986 perf-stat.ps.context-switches
6.837e+10 -2.3% 6.683e+10 perf-stat.ps.instructions
2.077e+13 -2.4% 2.028e+13 perf-stat.total.instructions
3.02 -0.1 2.90 perf-profile.calltrace.cycles-pp.pick_task_fair.pick_next_task_fair.__pick_next_task.__schedule.schedule
3.79 -0.1 3.68 perf-profile.calltrace.cycles-pp.perf_pmu_sched_task.__perf_event_task_sched_out.prepare_task_switch.__schedule.schedule
5.13 -0.1 5.04 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.exit_to_user_mode_loop
2.72 -0.1 2.64 perf-profile.calltrace.cycles-pp.put_prev_entity.pick_next_task_fair.__pick_next_task.__schedule.schedule
2.87 -0.1 2.79 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.anon_pipe_read.vfs_read
5.27 -0.1 5.19 perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.exit_to_user_mode_loop.do_syscall_64
1.82 -0.1 1.76 perf-profile.calltrace.cycles-pp.perf_pmu_sched_task.finish_task_switch.__schedule.schedule.anon_pipe_read
1.41 -0.1 1.35 perf-profile.calltrace.cycles-pp.__switch_to
0.84 -0.0 0.79 ± 2% perf-profile.calltrace.cycles-pp.anon_pipe_buf_release.anon_pipe_read.vfs_read.ksys_read.do_syscall_64
1.64 -0.0 1.59 perf-profile.calltrace.cycles-pp.intel_pmu_disable_all.perf_pmu_sched_task.__perf_event_task_sched_out.prepare_task_switch.__schedule
0.56 -0.0 0.52 ± 2% perf-profile.calltrace.cycles-pp.rw_verify_area.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.38 -0.0 1.34 perf-profile.calltrace.cycles-pp.update_curr.pick_task_fair.pick_next_task_fair.__pick_next_task.__schedule
0.64 +0.0 0.67 perf-profile.calltrace.cycles-pp.os_xsave
1.25 +0.0 1.30 perf-profile.calltrace.cycles-pp.__switch_to_asm
2.62 +0.1 2.68 perf-profile.calltrace.cycles-pp.set_next_entity.pick_next_task_fair.__pick_next_task.__schedule.schedule
3.19 +0.1 3.27 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.exit_to_user_mode_loop.do_syscall_64
0.62 ± 3% +0.1 0.70 ± 3% perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.exit_to_user_mode_loop
0.65 ± 3% +0.1 0.73 ± 5% perf-profile.calltrace.cycles-pp.___perf_sw_event.prepare_task_switch.__schedule.schedule.anon_pipe_read
1.78 ± 2% +0.1 1.86 perf-profile.calltrace.cycles-pp.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.58 ± 3% +0.1 0.67 ± 2% perf-profile.calltrace.cycles-pp._copy_from_iter.copy_page_from_iter.anon_pipe_write.vfs_write.ksys_write
3.27 +0.1 3.37 perf-profile.calltrace.cycles-pp.prepare_task_switch.__schedule.schedule.anon_pipe_read.vfs_read
1.38 +0.1 1.48 ± 2% perf-profile.calltrace.cycles-pp.copy_page_from_iter.anon_pipe_write.vfs_write.ksys_write.do_syscall_64
1.34 ± 3% +0.2 1.51 ± 7% perf-profile.calltrace.cycles-pp.update_curr.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up.autoremove_wake_function
1.08 ± 7% +0.2 1.29 ± 10% perf-profile.calltrace.cycles-pp.update_se.update_curr.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up
1.78 ± 3% +0.2 2.00 ± 6% perf-profile.calltrace.cycles-pp.update_curr.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task
2.18 +0.2 2.41 ± 3% perf-profile.calltrace.cycles-pp.wakeup_preempt_fair.wakeup_preempt.try_to_wake_up.autoremove_wake_function.__wake_up_common
2.58 +0.2 2.82 ± 2% perf-profile.calltrace.cycles-pp.wakeup_preempt.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_sync_key
1.33 ± 6% +0.3 1.59 ± 9% perf-profile.calltrace.cycles-pp.update_se.update_curr.dequeue_entity.dequeue_entities.dequeue_task_fair
3.38 ± 2% +0.3 3.65 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule
4.35 ± 2% +0.3 4.66 ± 2% perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule
5.12 +0.3 5.44 ± 2% perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.anon_pipe_read.vfs_read
4.79 ± 2% +0.3 5.12 ± 2% perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.anon_pipe_read
20.64 +0.4 21.01 perf-profile.calltrace.cycles-pp.schedule.anon_pipe_read.vfs_read.ksys_read.do_syscall_64
0.00 +0.6 0.57 ± 7% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_stat_runtime.perf_trace_sched_stat_runtime.update_se.update_curr.wakeup_preempt_fair
0.00 +0.6 0.60 ± 7% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_se.update_curr.wakeup_preempt_fair.wakeup_preempt
0.00 +0.6 0.63 ± 12% perf-profile.calltrace.cycles-pp.read
0.00 +0.7 0.69 ± 2% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_stat_runtime.perf_trace_sched_stat_runtime.update_se.update_curr.dequeue_entity
0.00 +0.7 0.71 ± 2% perf-profile.calltrace.cycles-pp.perf_trace_sched_stat_runtime.update_se.update_curr.dequeue_entity.dequeue_entities
7.45 -0.2 7.26 perf-profile.children.cycles-pp.perf_pmu_sched_task
9.62 -0.1 9.48 perf-profile.children.cycles-pp.__pick_next_task
3.09 -0.1 2.98 perf-profile.children.cycles-pp.pick_task_fair
5.78 -0.1 5.68 perf-profile.children.cycles-pp.finish_task_switch
2.88 -0.1 2.78 perf-profile.children.cycles-pp.put_prev_entity
1.03 -0.1 0.94 perf-profile.children.cycles-pp.__perf_event_task_sched_in
0.29 ± 3% -0.1 0.21 ± 4% perf-profile.children.cycles-pp.__rdgsbase_inactive
3.20 -0.1 3.12 perf-profile.children.cycles-pp.intel_pmu_disable_all
0.30 -0.1 0.22 ± 4% perf-profile.children.cycles-pp.propagate_entity_load_avg
0.40 -0.1 0.34 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.67 -0.1 0.60 ± 3% perf-profile.children.cycles-pp.security_file_permission
1.60 -0.1 1.54 perf-profile.children.cycles-pp.__switch_to
1.01 -0.1 0.96 ± 2% perf-profile.children.cycles-pp.rw_verify_area
1.18 -0.1 1.12 ± 2% perf-profile.children.cycles-pp.__pick_eevdf
1.73 ± 2% -0.1 1.67 perf-profile.children.cycles-pp.avg_vruntime
1.00 ± 2% -0.1 0.95 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.65 ± 4% -0.1 0.60 ± 2% perf-profile.children.cycles-pp.update_entity_lag
0.84 -0.0 0.80 ± 2% perf-profile.children.cycles-pp.anon_pipe_buf_release
0.60 ± 2% -0.0 0.56 perf-profile.children.cycles-pp.vruntime_eligible
0.42 ± 2% -0.0 0.39 ± 2% perf-profile.children.cycles-pp.__x64_sys_read
0.56 -0.0 0.53 perf-profile.children.cycles-pp.update_rq_clock_task
0.18 ± 3% -0.0 0.15 ± 3% perf-profile.children.cycles-pp.perf_cgroup_switch
0.11 ± 6% -0.0 0.08 ± 5% perf-profile.children.cycles-pp.save_fpregs_to_fpstate
0.81 -0.0 0.78 perf-profile.children.cycles-pp._raw_spin_lock
0.60 ± 2% -0.0 0.57 perf-profile.children.cycles-pp.native_sched_clock
0.34 ± 2% -0.0 0.31 ± 4% perf-profile.children.cycles-pp.child
0.17 ± 4% -0.0 0.14 ± 3% perf-profile.children.cycles-pp.pick_next_task
0.97 -0.0 0.95 perf-profile.children.cycles-pp.update_rq_clock
0.16 ± 3% -0.0 0.14 ± 4% perf-profile.children.cycles-pp.dequeue_task
0.05 +0.0 0.06 ± 7% perf-profile.children.cycles-pp.__x86_indirect_thunk_rdx
0.27 ± 2% +0.0 0.29 ± 3% perf-profile.children.cycles-pp.place_entity
0.65 +0.0 0.67 ± 2% perf-profile.children.cycles-pp.os_xsave
0.08 ± 6% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.raw_spin_rq_unlock
0.09 ± 5% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.rb_next
0.11 ± 16% +0.0 0.15 ± 15% perf-profile.children.cycles-pp.perf_swevent_event
0.35 ± 2% +0.0 0.40 perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
2.72 +0.1 2.77 perf-profile.children.cycles-pp.set_next_entity
1.18 +0.1 1.24 perf-profile.children.cycles-pp.update_load_avg
0.09 ± 6% +0.1 0.16 ± 6% perf-profile.children.cycles-pp.kill_fasync
1.31 +0.1 1.38 perf-profile.children.cycles-pp.__switch_to_asm
1.78 ± 2% +0.1 1.87 perf-profile.children.cycles-pp.switch_fpu_return
0.59 ± 3% +0.1 0.68 ± 2% perf-profile.children.cycles-pp._copy_from_iter
1.40 +0.1 1.49 ± 2% perf-profile.children.cycles-pp.copy_page_from_iter
1.04 +0.1 1.14 perf-profile.children.cycles-pp.__update_load_avg_se
1.29 ± 3% +0.2 1.45 ± 4% perf-profile.children.cycles-pp.___perf_sw_event
6.50 +0.2 6.69 perf-profile.children.cycles-pp.prepare_task_switch
2.34 +0.2 2.59 ± 2% perf-profile.children.cycles-pp.wakeup_preempt_fair
2.78 +0.3 3.03 ± 2% perf-profile.children.cycles-pp.wakeup_preempt
3.43 ± 2% +0.3 3.70 ± 2% perf-profile.children.cycles-pp.dequeue_entity
4.39 ± 2% +0.3 4.70 ± 2% perf-profile.children.cycles-pp.dequeue_entities
5.13 +0.3 5.45 ± 2% perf-profile.children.cycles-pp.try_to_block_task
4.80 ± 2% +0.3 5.14 ± 2% perf-profile.children.cycles-pp.dequeue_task_fair
5.64 +0.3 5.99 ± 4% perf-profile.children.cycles-pp.update_curr
0.34 ± 9% +0.3 0.68 ± 11% perf-profile.children.cycles-pp.read
0.64 +0.4 1.06 perf-profile.children.cycles-pp.strnlen
1.38 ± 4% +0.4 1.81 ± 3% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
1.30 ± 4% +0.4 1.73 ± 3% perf-profile.children.cycles-pp.do_perf_trace_sched_stat_runtime
3.76 ± 5% +0.5 4.24 ± 7% perf-profile.children.cycles-pp.update_se
36.56 +0.5 37.10 perf-profile.children.cycles-pp.schedule
1.01 ± 3% -0.2 0.86 ± 7% perf-profile.self.cycles-pp.try_to_wake_up
1.82 ± 4% -0.1 1.68 ± 6% perf-profile.self.cycles-pp.vfs_write
0.60 -0.1 0.52 ± 2% perf-profile.self.cycles-pp.put_prev_entity
3.18 -0.1 3.10 perf-profile.self.cycles-pp.intel_pmu_disable_all
0.28 ± 3% -0.1 0.20 ± 3% perf-profile.self.cycles-pp.__rdgsbase_inactive
1.54 -0.1 1.46 perf-profile.self.cycles-pp.__switch_to
0.25 -0.1 0.18 ± 4% perf-profile.self.cycles-pp.propagate_entity_load_avg
0.30 ± 4% -0.1 0.23 ± 3% perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
0.65 -0.1 0.58 ± 4% perf-profile.self.cycles-pp.security_file_permission
1.70 -0.1 1.64 perf-profile.self.cycles-pp.avg_vruntime
0.82 -0.1 0.76 perf-profile.self.cycles-pp.__perf_event_task_sched_in
0.31 ± 3% -0.1 0.25 ± 2% perf-profile.self.cycles-pp.__wake_up_common
0.96 -0.0 0.91 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.70 -0.0 0.65 ± 2% perf-profile.self.cycles-pp.__pick_eevdf
0.80 -0.0 0.75 ± 2% perf-profile.self.cycles-pp.anon_pipe_buf_release
0.56 ± 2% -0.0 0.52 ± 2% perf-profile.self.cycles-pp.vruntime_eligible
0.59 ± 2% -0.0 0.55 perf-profile.self.cycles-pp.native_sched_clock
0.50 -0.0 0.46 perf-profile.self.cycles-pp.update_rq_clock_task
0.53 -0.0 0.50 ± 2% perf-profile.self.cycles-pp.pick_task_fair
0.88 -0.0 0.85 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.18 ± 3% -0.0 0.15 ± 3% perf-profile.self.cycles-pp.perf_cgroup_switch
0.67 ± 2% -0.0 0.64 ± 2% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.11 ± 6% -0.0 0.08 perf-profile.self.cycles-pp.save_fpregs_to_fpstate
0.40 -0.0 0.37 ± 2% perf-profile.self.cycles-pp.__x64_sys_read
0.30 ± 3% -0.0 0.27 ± 5% perf-profile.self.cycles-pp.child
0.18 ± 5% -0.0 0.15 ± 4% perf-profile.self.cycles-pp.update_rq_clock
0.15 ± 3% -0.0 0.13 ± 5% perf-profile.self.cycles-pp.pick_next_task
0.08 ± 5% -0.0 0.07 ± 7% perf-profile.self.cycles-pp.enqueue_entity
0.16 ± 3% -0.0 0.14 ± 4% perf-profile.self.cycles-pp.dequeue_task
0.07 ± 6% +0.0 0.10 ± 5% perf-profile.self.cycles-pp.rb_next
0.07 ± 7% +0.0 0.09 ± 6% perf-profile.self.cycles-pp.raw_spin_rq_unlock
0.24 ± 3% +0.0 0.27 ± 2% perf-profile.self.cycles-pp.ksys_read
0.64 +0.0 0.67 ± 2% perf-profile.self.cycles-pp.os_xsave
0.12 ± 6% +0.0 0.15 ± 4% perf-profile.self.cycles-pp.enqueue_task
0.71 ± 2% +0.0 0.75 perf-profile.self.cycles-pp.pick_next_task_fair
0.55 +0.0 0.59 ± 2% perf-profile.self.cycles-pp.__enqueue_entity
0.02 ±141% +0.0 0.06 ± 9% perf-profile.self.cycles-pp.__x86_indirect_thunk_rdx
0.10 ± 15% +0.0 0.14 ± 12% perf-profile.self.cycles-pp.perf_swevent_event
0.45 ± 3% +0.0 0.50 ± 3% perf-profile.self.cycles-pp.switch_fpu_return
0.30 ± 3% +0.0 0.35 ± 2% perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
0.46 +0.1 0.51 perf-profile.self.cycles-pp.dequeue_entities
0.06 ± 7% +0.1 0.13 ± 6% perf-profile.self.cycles-pp.kill_fasync
1.30 +0.1 1.37 perf-profile.self.cycles-pp.__switch_to_asm
0.38 ± 4% +0.1 0.46 ± 2% perf-profile.self.cycles-pp._copy_from_iter
0.80 ± 4% +0.1 0.89 ± 2% perf-profile.self.cycles-pp.prepare_task_switch
0.67 ± 2% +0.1 0.78 ± 3% perf-profile.self.cycles-pp.dequeue_entity
1.01 +0.1 1.11 perf-profile.self.cycles-pp.__update_load_avg_se
0.57 ± 2% +0.1 0.68 ± 2% perf-profile.self.cycles-pp.wakeup_preempt_fair
0.48 ± 2% +0.2 0.70 ± 2% perf-profile.self.cycles-pp.schedule
0.21 ± 7% +0.3 0.52 ± 5% perf-profile.self.cycles-pp.read
0.63 +0.4 1.05 perf-profile.self.cycles-pp.strnlen
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression
2026-04-24 7:10 [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression kernel test robot
@ 2026-04-24 9:55 ` Peter Zijlstra
2026-04-27 5:59 ` Oliver Sang
0 siblings, 1 reply; 3+ messages in thread
From: Peter Zijlstra @ 2026-04-24 9:55 UTC (permalink / raw)
To: kernel test robot
Cc: Thomas Gleixner, oe-lkp, lkp, linux-kernel, aubrey.li, yu.c.chen
On Fri, Apr 24, 2026 at 03:10:07PM +0800, kernel test robot wrote:
>
>
> Hello,
>
> kernel test robot noticed a 2.8% regression of will-it-scale.per_process_ops on:
>
>
> commit: 96d1610e0b20b5a627773874b4514ae922ad98f6 ("sched: Optimize hrtimer handling")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>
> [still regression on linus/master 1d51b370a0f8f642f4fc84c795fbedac0fcdbbd2]
> [still regression on linux-next/master 936c21068d7ade00325e40d82bfd2f3f29d9f659]
> [still regression on fix commit eef9f648fb0e92618041f019d4bdcf7ae17cb743]
>
> testcase: will-it-scale
> config: x86_64-rhel-9.4
> compiler: gcc-14
> test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
> parameters:
So this commit has lived in private trees (tglx and mine), has been in
tip, in next etc. *WHY* do we only get this report now?
This isn't the first time I see a regression report after things have
hit Linus' tree. This is starting to get really annoying. The point of
having this robot was to catch issues *before* they hit Linus, no?
That said, looking back at will-it-scale reports, 2.8 seems to be pretty
much in the noise range, no?
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression
2026-04-24 9:55 ` Peter Zijlstra
@ 2026-04-27 5:59 ` Oliver Sang
0 siblings, 0 replies; 3+ messages in thread
From: Oliver Sang @ 2026-04-27 5:59 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Thomas Gleixner, oe-lkp, lkp, linux-kernel, aubrey.li, yu.c.chen,
oliver.sang
hi, Peter Zijlstra,
On Fri, Apr 24, 2026 at 11:55:23AM +0200, Peter Zijlstra wrote:
> On Fri, Apr 24, 2026 at 03:10:07PM +0800, kernel test robot wrote:
> >
> >
> > Hello,
> >
> > kernel test robot noticed a 2.8% regression of will-it-scale.per_process_ops on:
> >
> >
> > commit: 96d1610e0b20b5a627773874b4514ae922ad98f6 ("sched: Optimize hrtimer handling")
> > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> >
> > [still regression on linus/master 1d51b370a0f8f642f4fc84c795fbedac0fcdbbd2]
> > [still regression on linux-next/master 936c21068d7ade00325e40d82bfd2f3f29d9f659]
> > [still regression on fix commit eef9f648fb0e92618041f019d4bdcf7ae17cb743]
> >
> > testcase: will-it-scale
> > config: x86_64-rhel-9.4
> > compiler: gcc-14
> > test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory
> > parameters:
>
> So this commit has lived in private trees (tglx and mine), has been in
> tip, in next etc. *WHY* do we only get this report now?
>
> This isn't the first time I see a regression report after things have
> hit Linus' tree. This is starting to get really annoying. The point of
> having this robot was to catch issues *before* they hit Linus, no?
there are some difference between kbuild/boot tests and functional/performance
tests by kernel test bot.
kbuild/boot tests could be triggered directly by repo/branch changes.
functional/performance depends on so-called hourly kernel merge to trigger tests
and bisection (due to resource constraints).
for this commit and related branches, we did generate kbuild/boot reports.
including e.g. kbuild report [1] and boot report [2].
(
for [2], the reported fbc is acefed487add3, the commit "sched: Optimize
hrtimer handling" is the tip commit of tglx-devel/sched/hrtick branch at that
time
* 1f963bc350a6f sched: Optimize hrtimer handling <----
* 8bc0da68c41a3 sched: Default enable HRTICK
...
* acefed487add3 x86/apic: Enable TSC coupled programming mode
* c903d7114698b x86/apic: Avoid the PVOPS indirection for the TSC deadline timer
)
after reviewing the history, we discovered that both reports originated from
direct repository/branch testing, meaning these branches were not merged into
hourly kernels, causing functional/performance tests to be skipped.
most likely, merge conflicts prevented this change from going through the
functional/performance pipeline until it reached mainline.
since mainline serves as the base for the hourly kernels mentioned above,
the commit will have more opportunity to go through functional/performance
testing after merging into mainline.
Thank you for bringing this to our attention. We are now considering
improvements to address this gap in our testing coverage.
>
> That said, looking back at will-it-scale reports, 2.8 seems to be pretty
> much in the noise range, no?
this test supplies stable results. for parent commit:
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json: "will-it-scale.per_process_ops": [
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json- 238895,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json- 237238,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json- 237558,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json- 238616,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json- 238299,
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json- 237793
c3a92213eb3dd8ea6f664d16a08eda800e34eaad/matrix.json- ],
for this commit:
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json: "will-it-scale.per_process_ops": [
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json- 231051,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json- 232241,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json- 231346,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json- 231058,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json- 230174,
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json- 232143
96d1610e0b20b5a627773874b4514ae922ad98f6/matrix.json- ],
We will conduct additional verification.
[1] https://lore.kernel.org/oe-kbuild-all/202602171013.VyR8FIZ2-lkp@intel.com/
[2] https://lore.kernel.org/all/202602181118.929e4694-lkp@intel.com/
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2026-04-27 5:59 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-04-24 7:10 [linus:master] [sched] 96d1610e0b: will-it-scale.per_process_ops 2.8% regression kernel test robot
2026-04-24 9:55 ` Peter Zijlstra
2026-04-27 5:59 ` Oliver Sang
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox