* [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
@ 2026-03-10 14:46 kernel test robot
2026-03-10 15:23 ` Peter Zijlstra
0 siblings, 1 reply; 10+ messages in thread
From: kernel test robot @ 2026-03-10 14:46 UTC (permalink / raw)
To: Peter Zijlstra
Cc: oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner, oliver.sang
Hello,
kernel test robot noticed a 30.1% regression of stress-ng.timermix.ops_per_sec on:
commit: 2889243848560b6b0211aba401d2fc122070ba2f ("hrtimer: Re-arrange hrtimer_interrupt()")
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/hrtick
testcase: stress-ng
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
parameters:
nr_threads: 100%
testtime: 60s
test: timermix
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202603102229.74b9dee4-lkp@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-14/performance/x86_64-rhel-9.4/100%/debian-13-x86_64-20250902.cgz/lkp-icl-2sp8/timermix/stress-ng/60s
commit:
8e10f6b81a ("hrtimer: Add hrtimer_rearm tracepoint")
2889243848 ("hrtimer: Re-arrange hrtimer_interrupt()")
8e10f6b81afbf60e 2889243848560b6b0211aba401d
---------------- ---------------------------
%stddev %change %stddev
\ | \
232123 ± 7% -78.8% 49228 ± 9% stress-ng.time.involuntary_context_switches
12201 +11.1% 13550 stress-ng.time.minor_page_faults
5205 -32.9% 3490 ± 2% stress-ng.time.percent_of_cpu_this_job_got
2849 -33.0% 1909 ± 2% stress-ng.time.system_time
276.59 -32.7% 186.28 ± 2% stress-ng.time.user_time
3067734 ± 5% +37.1% 4205161 ± 8% stress-ng.time.voluntary_context_switches
117087 -38.8% 71712 ± 2% stress-ng.timermix.CLOCK_BOOTTIME_ticks_per_sec
117127 -38.8% 71656 ± 2% stress-ng.timermix.CLOCK_MONOTONIC_ticks_per_sec
978.58 -39.3% 593.62 stress-ng.timermix.CLOCK_PROCESS_CPUTIME_ID_ticks_per_sec
117335 -38.7% 71961 stress-ng.timermix.CLOCK_REALTIME_ticks_per_sec
117066 -38.8% 71645 ± 2% stress-ng.timermix.CLOCK_TAI_ticks_per_sec
978.68 -39.3% 593.67 stress-ng.timermix.CLOCK_THREAD_CPUTIME_ID_ticks_per_sec
10.68 ± 2% -15.3% 9.05 ± 10% stress-ng.timermix.ITIMER_PROF_ticks_per_sec
1263 ± 2% -35.2% 818.29 ± 52% stress-ng.timermix.ITIMER_REAL_ticks_per_sec
9.68 ± 2% -15.1% 8.21 ± 11% stress-ng.timermix.ITIMER_VIRTUAL_ticks_per_sec
1.809e+09 -30.1% 1.264e+09 ± 2% stress-ng.timermix.ops
30158157 -30.1% 21078045 ± 2% stress-ng.timermix.ops_per_sec
59.00 ± 24% -42.7% 33.83 ± 27% perf-c2c.DRAM.local
2972 ± 9% +39.3% 4140 ± 6% uptime.idle
1.45e+08 ± 11% +812.5% 1.323e+09 ± 4% cpuidle..time
2492037 ± 7% +118.6% 5447859 ± 5% cpuidle..usage
2.68 ± 28% +29.9 32.60 ± 5% mpstat.cpu.all.idle%
16.10 -4.1 12.04 ± 2% mpstat.cpu.all.irq%
0.00 ± 21% +0.0 0.01 ± 12% mpstat.cpu.all.soft%
72.82 -23.7 49.15 ± 2% mpstat.cpu.all.sys%
8.39 -2.2 6.20 mpstat.cpu.all.usr%
5.67 ± 19% +847.1% 53.67 ± 5% mpstat.max_utilization.seconds
5.69 ± 12% +510.7% 34.76 ± 4% vmstat.cpu.id
86.13 -31.3% 59.19 ± 2% vmstat.cpu.sy
4175472 +9.8% 4585895 ± 2% vmstat.memory.cache
62.71 -32.1% 42.60 ± 2% vmstat.procs.r
99312 ± 6% +36.2% 135233 ± 8% vmstat.system.cs
28455658 -29.9% 19958244 ± 3% vmstat.system.in
1.58 ± 2% -99.7% 0.00 ± 46% perf-sched.sch_delay.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
1.58 ± 2% -99.7% 0.00 ± 46% perf-sched.total_sch_delay.average.ms
88.93 ± 9% -98.2% 1.61 ± 29% perf-sched.total_wait_and_delay.average.ms
6699 ± 9% +8221.6% 557489 ± 28% perf-sched.total_wait_and_delay.count.ms
87.36 ± 9% -98.2% 1.60 ± 29% perf-sched.total_wait_time.average.ms
88.93 ± 9% -98.2% 1.61 ± 29% perf-sched.wait_and_delay.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
6699 ± 9% +8221.6% 557489 ± 28% perf-sched.wait_and_delay.count.[unknown].[unknown].[unknown].[unknown].[unknown]
87.36 ± 9% -98.2% 1.60 ± 29% perf-sched.wait_time.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
3410 -27.3% 2477 ± 2% turbostat.Avg_MHz
96.56 -26.8 69.72 ± 2% turbostat.Busy%
3.62 ± 11% +26.9 30.49 ± 5% turbostat.C1%
3.16 ± 13% +572.0% 21.21 ± 9% turbostat.CPU%c1
0.58 +4.9% 0.61 turbostat.IPC
1.807e+09 -30.4% 1.257e+09 ± 3% turbostat.IRQ
113.70 ± 5% -60.5 53.16 ± 13% turbostat.PKG_%
404.64 -8.2% 371.64 turbostat.PkgWatt
0.02 ± 33% +114.8% 0.05 ± 17% perf-stat.i.MPKI
2.909e+10 -26.6% 2.136e+10 ± 2% perf-stat.i.branch-instructions
0.36 +0.4 0.80 ± 12% perf-stat.i.branch-miss-rate%
92789452 -13.2% 80563145 perf-stat.i.branch-misses
13395966 ± 7% -19.0% 10855385 ± 2% perf-stat.i.cache-references
106482 ± 6% +37.3% 146175 ± 9% perf-stat.i.context-switches
1.71 -4.2% 1.63 perf-stat.i.cpi
2.219e+11 -28.8% 1.58e+11 ± 2% perf-stat.i.cpu-cycles
1654 ± 20% -80.8% 317.81 ± 2% perf-stat.i.cpu-migrations
1.294e+11 -26.0% 9.585e+10 ± 2% perf-stat.i.instructions
0.59 +5.8% 0.62 perf-stat.i.ipc
1.68 ± 6% +30.1% 2.18 ± 10% perf-stat.i.metric.K/sec
0.31 +0.1 0.37 ± 2% perf-stat.overall.branch-miss-rate%
1.71 -3.7% 1.65 perf-stat.overall.cpi
0.58 +3.9% 0.61 perf-stat.overall.ipc
2.864e+10 -25.5% 2.133e+10 ± 2% perf-stat.ps.branch-instructions
90036611 -13.5% 77841621 perf-stat.ps.branch-misses
13516993 ± 6% -22.0% 10547777 ± 2% perf-stat.ps.cache-references
102237 ± 6% +35.7% 138762 ± 8% perf-stat.ps.context-switches
2.184e+11 -27.7% 1.579e+11 ± 2% perf-stat.ps.cpu-cycles
1592 ± 19% -80.8% 304.93 ± 3% perf-stat.ps.cpu-migrations
1.275e+11 -24.9% 9.574e+10 ± 2% perf-stat.ps.instructions
7.842e+12 -25.2% 5.869e+12 ± 2% perf-stat.total.instructions
34.10 ± 12% -25.9 8.18 ± 3% perf-profile.calltrace.cycles-pp.sigpending
29.70 ± 13% -25.7 3.96 ± 5% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.sigpending
29.34 ± 14% -25.7 3.62 ± 6% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigpending
22.76 ± 16% -22.7 0.10 ±223% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigpending
22.62 ± 16% -22.5 0.10 ±223% perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigpending
33.06 -16.9 16.21 perf-profile.calltrace.cycles-pp.x64_setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
28.31 -16.0 12.29 perf-profile.calltrace.cycles-pp.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64
18.06 ± 2% -11.2 6.89 perf-profile.calltrace.cycles-pp.copy_fpstate_to_sigframe.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_loop
8.72 ± 3% -8.7 0.00 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.copy_fpstate_to_sigframe.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart
4.88 ± 4% -1.7 3.15 perf-profile.calltrace.cycles-pp.save_xstate_epilog.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_loop
1.72 ± 2% -1.4 0.28 ±100% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_loop
7.86 ± 10% -1.0 6.84 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
7.57 ± 10% -1.0 6.57 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.78 ± 3% -1.0 0.79 perf-profile.calltrace.cycles-pp.__get_user_nocheck_8.copy_fpstate_to_sigframe.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart
18.06 -0.7 17.32 perf-profile.calltrace.cycles-pp.stress_timermix_timer_action
6.24 -0.4 5.81 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.stress_timermix_timer_action
6.20 -0.4 5.77 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_timermix_timer_action
5.58 -0.4 5.16 perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_timermix_timer_action
0.97 ± 23% -0.4 0.56 perf-profile.calltrace.cycles-pp.__put_user_nocheck_8.copy_fpstate_to_sigframe.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart
5.33 -0.4 4.92 perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_timermix_timer_action
0.89 ± 23% -0.3 0.55 perf-profile.calltrace.cycles-pp.__put_user_nocheck_4.copy_fpstate_to_sigframe.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart
6.23 -0.3 5.90 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.36 -0.3 3.06 perf-profile.calltrace.cycles-pp.fpu__clear_user_states.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.94 -0.1 1.82 perf-profile.calltrace.cycles-pp.clear_bhb_loop
1.52 ± 5% -0.1 1.40 perf-profile.calltrace.cycles-pp.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.62 -0.1 1.52 perf-profile.calltrace.cycles-pp.__get_user_nocheck_8.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.16 -0.1 2.08 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigpending.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigpending
2.64 -0.1 2.58 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.stress_timermix_timer_action
1.98 -0.1 1.92 perf-profile.calltrace.cycles-pp.clear_bhb_loop.sigpending
0.99 -0.1 0.94 perf-profile.calltrace.cycles-pp.set_current_blocked.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.30 -0.0 1.25 perf-profile.calltrace.cycles-pp.signal_setup_done.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.64 ± 2% -0.0 0.60 perf-profile.calltrace.cycles-pp.fpregs_mark_activate.fpu__clear_user_states.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64
1.12 -0.0 1.08 perf-profile.calltrace.cycles-pp._copy_from_user.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.56 -0.0 0.54 perf-profile.calltrace.cycles-pp.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe.sigpending
0.64 ± 9% +0.1 0.77 perf-profile.calltrace.cycles-pp.__get_user_nocheck_8.save_xstate_epilog.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart
0.59 ± 9% +0.3 0.92 ± 10% perf-profile.calltrace.cycles-pp._copy_to_user.copy_siginfo_to_user.x64_setup_rt_frame.arch_do_signal_or_restart.exit_to_user_mode_loop
0.36 ± 70% +0.5 0.84 perf-profile.calltrace.cycles-pp.__hrtimer_start_range_ns.hrtimer_start_range_ns.posixtimer_deliver_signal.dequeue_signal.get_signal
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.native_sched_clock.sched_clock.sched_clock_cpu.irqtime_account_irq.__irq_exit_rcu
0.00 +0.6 0.57 perf-profile.calltrace.cycles-pp.__put_user_nocheck_8.save_xstate_epilog.get_sigframe.x64_setup_rt_frame.arch_do_signal_or_restart
0.00 +0.6 0.57 ± 6% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +0.6 0.57 ± 2% perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
0.00 +0.6 0.58 perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.00 +0.6 0.63 perf-profile.calltrace.cycles-pp.sched_clock.sched_clock_cpu.irqtime_account_irq.__irq_exit_rcu.sysvec_apic_timer_interrupt
0.08 ±223% +0.6 0.72 perf-profile.calltrace.cycles-pp.__get_user_nocheck_4.check_xstate_in_sigframe.__fpu_restore_sig.fpu__restore_sig.restore_sigcontext
0.08 ±223% +0.6 0.72 perf-profile.calltrace.cycles-pp._raw_spin_lock.posixtimer_deliver_signal.dequeue_signal.get_signal.arch_do_signal_or_restart
0.00 +0.6 0.65 perf-profile.calltrace.cycles-pp.irqtime_account_irq.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt._raw_spin_unlock_irq.get_signal
0.00 +0.7 0.71 perf-profile.calltrace.cycles-pp.sched_clock_cpu.irqtime_account_irq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +0.8 0.84 ± 2% perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64
0.00 +0.9 0.88 perf-profile.calltrace.cycles-pp.complete_signal.posixtimer_send_sigqueue.posix_timer_fn.__hrtimer_run_queues.hrtimer_interrupt
0.00 +0.9 0.93 perf-profile.calltrace.cycles-pp.irqtime_account_irq.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt._raw_spin_unlock_irq
0.00 +1.0 0.99 perf-profile.calltrace.cycles-pp.__hrtimer_next_event_base.hrtimer_update_next_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.00 +1.3 1.27 perf-profile.calltrace.cycles-pp.__irq_exit_rcu.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt._raw_spin_unlock_irq.get_signal
2.26 ± 3% +1.3 3.55 perf-profile.calltrace.cycles-pp.posix_timer_fn.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt
0.58 ± 8% +1.4 1.97 perf-profile.calltrace.cycles-pp.native_apic_msr_eoi.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt._raw_spin_unlock_irq
0.00 +1.5 1.49 perf-profile.calltrace.cycles-pp.hrtimer_update_next_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
0.00 +1.6 1.56 perf-profile.calltrace.cycles-pp.ktime_get_update_offsets_now.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
3.64 ± 7% +1.6 5.29 perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
1.34 ± 17% +1.7 3.01 perf-profile.calltrace.cycles-pp.clockevents_program_event.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
1.12 ± 25% +1.7 2.82 perf-profile.calltrace.cycles-pp.posixtimer_send_sigqueue.posix_timer_fn.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
4.86 ± 9% +2.9 7.76 perf-profile.calltrace.cycles-pp.restore_sigcontext.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_timermix_itimer_action
4.94 ± 9% +3.0 7.88 perf-profile.calltrace.cycles-pp.__x64_sys_rt_sigreturn.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_timermix_itimer_action
5.12 ± 2% +7.5 12.62 perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt._raw_spin_unlock_irq
6.45 ± 2% +9.1 15.56 perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt._raw_spin_unlock_irq.get_signal
7.62 ± 2% +10.4 18.04 perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt._raw_spin_unlock_irq.get_signal.arch_do_signal_or_restart
12.20 ± 2% +15.5 27.70 perf-profile.calltrace.cycles-pp._raw_spin_unlock_irq.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64
25.74 +17.6 43.33 perf-profile.calltrace.cycles-pp.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
14.91 ± 2% +18.7 33.65 perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt._raw_spin_unlock_irq.get_signal.arch_do_signal_or_restart.exit_to_user_mode_loop
36.59 ± 8% +24.4 60.98 perf-profile.calltrace.cycles-pp.arch_do_signal_or_restart.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_timermix_itimer_action
36.80 ± 8% +24.6 61.39 perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_timermix_itimer_action
41.81 ± 8% +27.6 69.41 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.stress_timermix_itimer_action
41.82 ± 8% +27.6 69.43 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.stress_timermix_itimer_action
41.91 ± 8% +27.7 69.58 perf-profile.calltrace.cycles-pp.stress_timermix_itimer_action
34.62 ± 11% -25.9 8.69 ± 2% perf-profile.children.cycles-pp.sigpending
33.66 -17.0 16.63 perf-profile.children.cycles-pp.x64_setup_rt_frame
29.22 -16.6 12.66 perf-profile.children.cycles-pp.get_sigframe
19.90 -12.6 7.28 perf-profile.children.cycles-pp.copy_fpstate_to_sigframe
5.40 ± 2% -2.1 3.32 perf-profile.children.cycles-pp.save_xstate_epilog
5.00 -1.8 3.17 perf-profile.children.cycles-pp.__get_user_nocheck_8
2.42 ± 2% -1.2 1.18 perf-profile.children.cycles-pp.__put_user_nocheck_4
2.31 -1.1 1.18 perf-profile.children.cycles-pp.__put_user_nocheck_8
13.88 -0.7 13.23 perf-profile.children.cycles-pp.stress_timermix_timer_action
1.23 -0.6 0.61 perf-profile.children.cycles-pp.rep_stos_alternative
14.48 -0.5 14.01 perf-profile.children.cycles-pp.__x64_sys_rt_sigreturn
0.97 ± 2% -0.4 0.59 perf-profile.children.cycles-pp.__local_bh_enable_ip
3.58 ± 2% -0.3 3.25 perf-profile.children.cycles-pp.fpu__clear_user_states
0.57 -0.3 0.30 ± 5% perf-profile.children.cycles-pp.fpu__alloc_mathframe
9.50 -0.2 9.26 perf-profile.children.cycles-pp.restore_sigcontext
6.77 -0.2 6.58 perf-profile.children.cycles-pp.clockevents_program_event
7.79 -0.2 7.61 perf-profile.children.cycles-pp.fpu__restore_sig
8.73 -0.2 8.55 perf-profile.children.cycles-pp.native_irq_return_iret
3.97 -0.2 3.79 perf-profile.children.cycles-pp.clear_bhb_loop
9.92 -0.2 9.77 perf-profile.children.cycles-pp.dequeue_signal
0.26 ± 4% -0.1 0.11 ± 4% perf-profile.children.cycles-pp.get_xsave_addr_user
5.33 -0.1 5.21 perf-profile.children.cycles-pp.restore_fpregs_from_user
2.60 -0.1 2.50 perf-profile.children.cycles-pp.native_apic_msr_eoi
4.09 -0.1 3.99 perf-profile.children.cycles-pp.posix_timer_fn
8.64 -0.1 8.54 perf-profile.children.cycles-pp.posixtimer_deliver_signal
4.40 -0.1 4.30 perf-profile.children.cycles-pp.__restore_fpregs_from_user
3.34 -0.1 3.25 perf-profile.children.cycles-pp.posixtimer_send_sigqueue
1.90 -0.1 1.83 perf-profile.children.cycles-pp.irqtime_account_irq
6.10 -0.1 6.03 perf-profile.children.cycles-pp.__hrtimer_run_queues
2.29 -0.1 2.22 perf-profile.children.cycles-pp.__x64_sys_rt_sigpending
0.79 -0.1 0.71 perf-profile.children.cycles-pp.ktime_expiry_to_cycles
1.56 -0.1 1.49 perf-profile.children.cycles-pp.__irq_exit_rcu
1.06 -0.1 1.00 perf-profile.children.cycles-pp.set_current_blocked
2.09 -0.1 2.03 perf-profile.children.cycles-pp._raw_spin_lock_irq
3.47 -0.1 3.40 perf-profile.children.cycles-pp.its_return_thunk
2.25 -0.1 2.20 perf-profile.children.cycles-pp.__fpu_restore_sig
1.20 ± 2% -0.1 1.14 perf-profile.children.cycles-pp.recalc_sigpending
1.39 -0.1 1.34 perf-profile.children.cycles-pp.signal_setup_done
1.45 -0.1 1.40 perf-profile.children.cycles-pp.sched_clock_cpu
2.14 -0.0 2.09 perf-profile.children.cycles-pp._copy_from_user
2.07 -0.0 2.02 perf-profile.children.cycles-pp.check_xstate_in_sigframe
1.09 -0.0 1.04 perf-profile.children.cycles-pp.fpregs_mark_activate
1.50 -0.0 1.46 perf-profile.children.cycles-pp.common_hrtimer_rearm
2.08 -0.0 2.04 perf-profile.children.cycles-pp.entry_SYSCALL_64
1.27 -0.0 1.23 perf-profile.children.cycles-pp.sched_clock
0.47 ± 2% -0.0 0.44 ± 2% perf-profile.children.cycles-pp.collect_signal
1.09 -0.0 1.06 perf-profile.children.cycles-pp.complete_signal
0.77 -0.0 0.74 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.77 -0.0 0.74 perf-profile.children.cycles-pp.__get_user_nocheck_4
1.05 -0.0 1.03 perf-profile.children.cycles-pp.arch_exit_to_user_mode_prepare
0.70 -0.0 0.68 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.45 -0.0 0.43 perf-profile.children.cycles-pp.ktime_get_with_offset
0.31 -0.0 0.29 perf-profile.children.cycles-pp.__check_object_size
0.22 ± 2% -0.0 0.21 ± 2% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.22 -0.0 0.21 perf-profile.children.cycles-pp.__irqentry_text_end
0.22 -0.0 0.21 perf-profile.children.cycles-pp._raw_spin_unlock
0.16 ± 2% +0.0 0.18 ± 2% perf-profile.children.cycles-pp.rb_erase
0.10 ± 3% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.tick_nohz_handler
0.00 +0.1 0.09 perf-profile.children.cycles-pp.__hrtimer_reprogram
0.00 +0.4 0.44 ± 12% perf-profile.children.cycles-pp.poll_idle
0.00 +0.5 0.48 ± 11% perf-profile.children.cycles-pp.cpuidle_enter_state
0.00 +0.5 0.48 ± 11% perf-profile.children.cycles-pp.cpuidle_enter
0.00 +0.5 0.48 ± 9% perf-profile.children.cycles-pp.start_secondary
0.00 +0.5 0.48 ± 11% perf-profile.children.cycles-pp.cpuidle_idle_call
0.00 +0.5 0.49 ± 11% perf-profile.children.cycles-pp.common_startup_64
0.00 +0.5 0.49 ± 11% perf-profile.children.cycles-pp.cpu_startup_entry
0.00 +0.5 0.49 ± 11% perf-profile.children.cycles-pp.do_idle
0.56 +0.6 1.17 perf-profile.children.cycles-pp.__hrtimer_next_event_base
0.88 +0.9 1.80 perf-profile.children.cycles-pp.ktime_get_update_offsets_now
0.85 +0.9 1.78 perf-profile.children.cycles-pp.hrtimer_update_next_event
27.92 +1.3 29.20 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
18.65 +1.5 20.16 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
15.68 +1.6 17.32 perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
12.58 +1.6 14.22 perf-profile.children.cycles-pp.hrtimer_interrupt
14.73 +17.1 31.82 perf-profile.children.cycles-pp._raw_spin_unlock_irq
26.30 +17.5 43.81 perf-profile.children.cycles-pp.get_signal
41.91 ± 8% +27.7 69.58 perf-profile.children.cycles-pp.stress_timermix_itimer_action
6.86 -2.3 4.53 perf-profile.self.cycles-pp.copy_fpstate_to_sigframe
3.61 -0.6 3.00 perf-profile.self.cycles-pp.__get_user_nocheck_8
1.78 -0.4 1.36 perf-profile.self.cycles-pp.get_sigframe
1.46 -0.4 1.09 perf-profile.self.cycles-pp.__put_user_nocheck_4
1.58 ± 3% -0.4 1.23 perf-profile.self.cycles-pp.save_xstate_epilog
1.44 -0.3 1.10 perf-profile.self.cycles-pp.__put_user_nocheck_8
2.93 -0.2 2.73 perf-profile.self.cycles-pp.fpu__clear_user_states
8.72 -0.2 8.54 perf-profile.self.cycles-pp.native_irq_return_iret
3.93 -0.2 3.75 perf-profile.self.cycles-pp.clear_bhb_loop
0.57 -0.2 0.39 ± 2% perf-profile.self.cycles-pp.rep_stos_alternative
6.00 -0.1 5.88 perf-profile.self.cycles-pp.clockevents_program_event
0.63 -0.1 0.52 perf-profile.self.cycles-pp.__local_bh_enable_ip
2.58 -0.1 2.48 perf-profile.self.cycles-pp.native_apic_msr_eoi
4.36 -0.1 4.27 perf-profile.self.cycles-pp.__restore_fpregs_from_user
0.73 -0.1 0.66 perf-profile.self.cycles-pp.ktime_expiry_to_cycles
0.29 -0.1 0.22 ± 2% perf-profile.self.cycles-pp.fpu__alloc_mathframe
1.96 -0.1 1.90 perf-profile.self.cycles-pp._raw_spin_lock_irq
2.09 -0.0 2.04 perf-profile.self.cycles-pp._copy_from_user
0.12 ± 5% -0.0 0.07 ± 6% perf-profile.self.cycles-pp.get_xsave_addr_user
0.52 -0.0 0.48 perf-profile.self.cycles-pp.__x64_sys_rt_sigreturn
0.87 -0.0 0.83 perf-profile.self.cycles-pp.fpregs_mark_activate
1.27 -0.0 1.23 perf-profile.self.cycles-pp.sigpending
0.73 -0.0 0.70 perf-profile.self.cycles-pp.copy_siginfo_to_user
1.21 -0.0 1.18 perf-profile.self.cycles-pp.do_syscall_64
0.45 ± 2% -0.0 0.42 ± 2% perf-profile.self.cycles-pp.collect_signal
0.41 -0.0 0.38 perf-profile.self.cycles-pp.sysvec_apic_timer_interrupt
0.75 -0.0 0.72 perf-profile.self.cycles-pp.__get_user_nocheck_4
0.81 -0.0 0.78 perf-profile.self.cycles-pp.posixtimer_send_sigqueue
0.44 -0.0 0.41 perf-profile.self.cycles-pp.irqtime_account_irq
0.31 ± 2% -0.0 0.28 perf-profile.self.cycles-pp.__irq_exit_rcu
0.74 -0.0 0.72 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.51 -0.0 0.49 perf-profile.self.cycles-pp.error_entry
0.55 -0.0 0.53 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.38 -0.0 0.36 perf-profile.self.cycles-pp.restore_fpregs_from_user
0.49 -0.0 0.47 perf-profile.self.cycles-pp.restore_sigcontext
0.46 -0.0 0.44 perf-profile.self.cycles-pp.asm_sysvec_apic_timer_interrupt
0.34 -0.0 0.33 perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
0.44 -0.0 0.42 perf-profile.self.cycles-pp.ktime_get_with_offset
0.77 -0.0 0.75 perf-profile.self.cycles-pp.posixtimer_deliver_signal
0.22 -0.0 0.20 ± 2% perf-profile.self.cycles-pp.sched_clock
0.45 -0.0 0.43 perf-profile.self.cycles-pp.__x64_sys_rt_sigpending
0.22 ± 2% -0.0 0.21 ± 2% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.12 +0.0 0.13 ± 2% perf-profile.self.cycles-pp.rb_erase
0.17 +0.0 0.19 ± 2% perf-profile.self.cycles-pp.hrtimer_reprogram
0.58 +0.1 0.64 perf-profile.self.cycles-pp.arch_do_signal_or_restart
0.00 +0.1 0.07 perf-profile.self.cycles-pp.__hrtimer_reprogram
0.58 +0.1 0.69 perf-profile.self.cycles-pp.hrtimer_interrupt
1.38 +0.2 1.55 perf-profile.self.cycles-pp.get_signal
0.26 +0.3 0.55 perf-profile.self.cycles-pp.hrtimer_update_next_event
0.00 +0.4 0.44 ± 12% perf-profile.self.cycles-pp.poll_idle
0.52 +0.5 1.06 perf-profile.self.cycles-pp.__hrtimer_next_event_base
0.86 +0.9 1.76 perf-profile.self.cycles-pp.ktime_get_update_offsets_now
4.31 +4.1 8.42 perf-profile.self.cycles-pp._raw_spin_unlock_irq
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
2026-03-10 14:46 [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression kernel test robot
@ 2026-03-10 15:23 ` Peter Zijlstra
2026-03-10 17:11 ` Joe Talbott
0 siblings, 1 reply; 10+ messages in thread
From: Peter Zijlstra @ 2026-03-10 15:23 UTC (permalink / raw)
To: kernel test robot; +Cc: oe-lkp, lkp, linux-kernel, x86, Thomas Gleixner
On Tue, Mar 10, 2026 at 10:46:16PM +0800, kernel test robot wrote:
>
>
> Hello,
>
> kernel test robot noticed a 30.1% regression of stress-ng.timermix.ops_per_sec on:
>
>
> commit: 2889243848560b6b0211aba401d2fc122070ba2f ("hrtimer: Re-arrange hrtimer_interrupt()")
> https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/hrtick
>
>
> testcase: stress-ng
> config: x86_64-rhel-9.4
> compiler: gcc-14
> test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> parameters:
>
> nr_threads: 100%
> testtime: 60s
> test: timermix
> cpufreq_governor: performance
>
>
>
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <oliver.sang@intel.com>
> | Closes: https://lore.kernel.org/oe-lkp/202603102229.74b9dee4-lkp@intel.com
>
>
> Details are as below:
> -------------------------------------------------------------------------------------------------->
>
>
> The kernel config and materials to reproduce are available at:
> https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com
As per always, I find it incredibly hard to deduce how the test is
actually ran.
I suppose this is: stress-ng --timermix ....
but what arguments exactly?
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
2026-03-10 15:23 ` Peter Zijlstra
@ 2026-03-10 17:11 ` Joe Talbott
2026-03-10 18:16 ` Peter Zijlstra
0 siblings, 1 reply; 10+ messages in thread
From: Joe Talbott @ 2026-03-10 17:11 UTC (permalink / raw)
To: Peter Zijlstra
Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86,
Thomas Gleixner
On Tue, Mar 10, 2026 at 11:58 AM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Tue, Mar 10, 2026 at 10:46:16PM +0800, kernel test robot wrote:
> >
> >
> > Hello,
> >
> > kernel test robot noticed a 30.1% regression of stress-ng.timermix.ops_per_sec on:
> >
> >
> > commit: 2889243848560b6b0211aba401d2fc122070ba2f ("hrtimer: Re-arrange hrtimer_interrupt()")
> > https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git sched/hrtick
> >
> >
> > testcase: stress-ng
> > config: x86_64-rhel-9.4
> > compiler: gcc-14
> > test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory
> > parameters:
> >
> > nr_threads: 100%
> > testtime: 60s
> > test: timermix
> > cpufreq_governor: performance
> >
> >
> >
> > If you fix the issue in a separate patch/commit (i.e. not just a new version of
> > the same patch/commit), kindly add following tags
> > | Reported-by: kernel test robot <oliver.sang@intel.com>
> > | Closes: https://lore.kernel.org/oe-lkp/202603102229.74b9dee4-lkp@intel.com
> >
> >
> > Details are as below:
> > -------------------------------------------------------------------------------------------------->
> >
> >
> > The kernel config and materials to reproduce are available at:
> > https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com
>
> As per always, I find it incredibly hard to deduce how the test is
> actually ran.
>
> I suppose this is: stress-ng --timermix ....
> but what arguments exactly?
>
It looks like it can be found here:
https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com/repro-script
stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64
Joe
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
2026-03-10 17:11 ` Joe Talbott
@ 2026-03-10 18:16 ` Peter Zijlstra
2026-03-10 18:50 ` Peter Zijlstra
0 siblings, 1 reply; 10+ messages in thread
From: Peter Zijlstra @ 2026-03-10 18:16 UTC (permalink / raw)
To: Joe Talbott
Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86,
Thomas Gleixner
On Tue, Mar 10, 2026 at 01:11:09PM -0400, Joe Talbott wrote:
> It looks like it can be found here:
> https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com/repro-script
>
> stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64
Thanks, that does indeed work and show the regression.
Lets see if I can spot the fail...
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
2026-03-10 18:16 ` Peter Zijlstra
@ 2026-03-10 18:50 ` Peter Zijlstra
2026-03-10 19:02 ` Peter Zijlstra
0 siblings, 1 reply; 10+ messages in thread
From: Peter Zijlstra @ 2026-03-10 18:50 UTC (permalink / raw)
To: Joe Talbott
Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86,
Thomas Gleixner
On Tue, Mar 10, 2026 at 07:16:51PM +0100, Peter Zijlstra wrote:
> On Tue, Mar 10, 2026 at 01:11:09PM -0400, Joe Talbott wrote:
>
> > It looks like it can be found here:
> > https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com/repro-script
> >
> > stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64
>
> Thanks, that does indeed work and show the regression.
>
> Lets see if I can spot the fail...
It looks like that benchmark manages to trip significant nr_hangs, and
yes, I made those more expensive because those were not expected to
actually happen at any sane rate.
Lets see if we can cure that without making a giant mess of things.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
2026-03-10 18:50 ` Peter Zijlstra
@ 2026-03-10 19:02 ` Peter Zijlstra
2026-03-11 9:40 ` Thomas Gleixner
0 siblings, 1 reply; 10+ messages in thread
From: Peter Zijlstra @ 2026-03-10 19:02 UTC (permalink / raw)
To: Joe Talbott
Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86,
Thomas Gleixner
On Tue, Mar 10, 2026 at 07:50:06PM +0100, Peter Zijlstra wrote:
> On Tue, Mar 10, 2026 at 07:16:51PM +0100, Peter Zijlstra wrote:
> > On Tue, Mar 10, 2026 at 01:11:09PM -0400, Joe Talbott wrote:
> >
> > > It looks like it can be found here:
> > > https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com/repro-script
> > >
> > > stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64
> >
> > Thanks, that does indeed work and show the regression.
> >
> > Lets see if I can spot the fail...
>
> It looks like that benchmark manages to trip significant nr_hangs, and
> yes, I made those more expensive because those were not expected to
> actually happen at any sane rate.
>
> Lets see if we can cure that without making a giant mess of things.
Ha! This seems to work just fine.
---
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index b94bd56b739f..9872dd53f761 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -2031,7 +2031,8 @@ static void hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t expires_nex
* Give the system a chance to do something else than looping
* on hrtimer interrupts.
*/
- expires_next = ktime_add_ns(ktime_get(), 100 * NSEC_PER_MSEC);
+ expires_next = ktime_add_ns(ktime_get(),
+ min(100 * NSEC_PER_MSEC, cpu_base->max_hang_time));
cpu_base->hang_detected = false;
}
hrtimer_rearm_event(expires_next, deferred);
^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
2026-03-10 19:02 ` Peter Zijlstra
@ 2026-03-11 9:40 ` Thomas Gleixner
2026-03-11 10:58 ` Peter Zijlstra
0 siblings, 1 reply; 10+ messages in thread
From: Thomas Gleixner @ 2026-03-11 9:40 UTC (permalink / raw)
To: Peter Zijlstra, Joe Talbott
Cc: kernel test robot, oe-lkp, lkp, linux-kernel, x86
On Tue, Mar 10 2026 at 20:02, Peter Zijlstra wrote:
> On Tue, Mar 10, 2026 at 07:50:06PM +0100, Peter Zijlstra wrote:
>> On Tue, Mar 10, 2026 at 07:16:51PM +0100, Peter Zijlstra wrote:
>> > On Tue, Mar 10, 2026 at 01:11:09PM -0400, Joe Talbott wrote:
>> >
>> > > It looks like it can be found here:
>> > > https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com/repro-script
>> > >
>> > > stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64
>> >
>> > Thanks, that does indeed work and show the regression.
>> >
>> > Lets see if I can spot the fail...
>>
>> It looks like that benchmark manages to trip significant nr_hangs, and
>> yes, I made those more expensive because those were not expected to
>> actually happen at any sane rate.
>>
>> Lets see if we can cure that without making a giant mess of things.
>
> Ha! This seems to work just fine.
>
> ---
> diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
> index b94bd56b739f..9872dd53f761 100644
> --- a/kernel/time/hrtimer.c
> +++ b/kernel/time/hrtimer.c
> @@ -2031,7 +2031,8 @@ static void hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t expires_nex
> * Give the system a chance to do something else than looping
> * on hrtimer interrupts.
> */
> - expires_next = ktime_add_ns(ktime_get(), 100 * NSEC_PER_MSEC);
> + expires_next = ktime_add_ns(ktime_get(),
> + min(100 * NSEC_PER_MSEC, cpu_base->max_hang_time));
> cpu_base->hang_detected = false;
Hmm. The original code preserved hang_detected until the next timer
interrupt to prevent rearming when a new timer is queued.
Thanks,
tglx
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
2026-03-11 9:40 ` Thomas Gleixner
@ 2026-03-11 10:58 ` Peter Zijlstra
2026-03-11 12:15 ` Peter Zijlstra
0 siblings, 1 reply; 10+ messages in thread
From: Peter Zijlstra @ 2026-03-11 10:58 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Joe Talbott, kernel test robot, oe-lkp, lkp, linux-kernel, x86
On Wed, Mar 11, 2026 at 10:40:20AM +0100, Thomas Gleixner wrote:
> On Tue, Mar 10 2026 at 20:02, Peter Zijlstra wrote:
>
> > On Tue, Mar 10, 2026 at 07:50:06PM +0100, Peter Zijlstra wrote:
> >> On Tue, Mar 10, 2026 at 07:16:51PM +0100, Peter Zijlstra wrote:
> >> > On Tue, Mar 10, 2026 at 01:11:09PM -0400, Joe Talbott wrote:
> >> >
> >> > > It looks like it can be found here:
> >> > > https://download.01.org/0day-ci/archive/20260310/202603102229.74b9dee4-lkp@intel.com/repro-script
> >> > >
> >> > > stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64
> >> >
> >> > Thanks, that does indeed work and show the regression.
> >> >
> >> > Lets see if I can spot the fail...
> >>
> >> It looks like that benchmark manages to trip significant nr_hangs, and
> >> yes, I made those more expensive because those were not expected to
> >> actually happen at any sane rate.
> >>
> >> Lets see if we can cure that without making a giant mess of things.
> >
> > Ha! This seems to work just fine.
> >
> > ---
> > diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
> > index b94bd56b739f..9872dd53f761 100644
> > --- a/kernel/time/hrtimer.c
> > +++ b/kernel/time/hrtimer.c
> > @@ -2031,7 +2031,8 @@ static void hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t expires_nex
> > * Give the system a chance to do something else than looping
> > * on hrtimer interrupts.
> > */
> > - expires_next = ktime_add_ns(ktime_get(), 100 * NSEC_PER_MSEC);
> > + expires_next = ktime_add_ns(ktime_get(),
> > + min(100 * NSEC_PER_MSEC, cpu_base->max_hang_time));
> > cpu_base->hang_detected = false;
>
> Hmm. The original code preserved hang_detected until the next timer
> interrupt to prevent rearming when a new timer is queued.
Oh indeed. And that avoids __hrtimer_reprogram() from coming in and
'destroying' the delay I suppose.
Let me poke at this a little more then.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression
2026-03-11 10:58 ` Peter Zijlstra
@ 2026-03-11 12:15 ` Peter Zijlstra
2026-03-11 20:16 ` [tip: sched/hrtick] hrtimer: Less agressive interrupt 'hang' handling tip-bot2 for Peter Zijlstra
0 siblings, 1 reply; 10+ messages in thread
From: Peter Zijlstra @ 2026-03-11 12:15 UTC (permalink / raw)
To: Thomas Gleixner
Cc: Joe Talbott, kernel test robot, oe-lkp, lkp, linux-kernel, x86
On Wed, Mar 11, 2026 at 11:58:19AM +0100, Peter Zijlstra wrote:
> > Hmm. The original code preserved hang_detected until the next timer
> > interrupt to prevent rearming when a new timer is queued.
>
> Oh indeed. And that avoids __hrtimer_reprogram() from coming in and
> 'destroying' the delay I suppose.
>
> Let me poke at this a little more then.
How's this then?
---
Subject: hrtimer: Less agressive interrupt 'hang' handling
From: Peter Zijlstra <peterz@infradead.org>
Date: Tue, 10 Mar 2026 20:02:21 +0100
When the hrtimer_interrupt needs to restart more than 3 times and
still has expired timers, the interrupt is considered hung. To give
the system a little time to recover, the hardware timer is programmed
a little into the future.
Prior to commit 288924384856 ("hrtimer: Re-arrange
hrtimer_interrupt()"), this was relative to the amount of time spend
serving the interrupt with a max of 100 msec.
However, in order to simplify, and because this condition 'should' not
happen, the timeout was unconditionally set to 100 msec.
'Obviously' there is a benchmark that hits this hard, by programming a
ton of very short timers :-/
Since reprogramming is decoupled from the interrupt handling, the
actual execution time is lost, however the code does track
max_hang_time. Using that, rather than the 100 ms max restores
performance.
stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64
bogo ops/s
288924384856^1: 23715979.93
288924384856: 11550049.77
patched: 23361116.78
Additionally, Thomas noted that we should not clear ->hang_detected
until the next interrupt, such that __hrtimer_reprogram() won't undo
the extra delay.
Fixes: 288924384856 ("hrtimer: Re-arrange hrtimer_interrupt()")
Closes: https://lore.kernel.org/oe-lkp/202603102229.74b9dee4-lkp@intel.com
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/time/hrtimer.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -2031,8 +2031,8 @@ static void hrtimer_rearm(struct hrtimer
* Give the system a chance to do something else than looping
* on hrtimer interrupts.
*/
- expires_next = ktime_add_ns(ktime_get(), 100 * NSEC_PER_MSEC);
- cpu_base->hang_detected = false;
+ expires_next = ktime_add_ns(ktime_get(),
+ min(100 * NSEC_PER_MSEC, cpu_base->max_hang_time));
}
hrtimer_rearm_event(expires_next, deferred);
}
@@ -2121,6 +2121,7 @@ void hrtimer_interrupt(struct clock_even
*/
now = hrtimer_update_base(cpu_base);
expires_next = hrtimer_update_next_event(cpu_base);
+ cpu_base->hang_detected = false;
if (expires_next < now) {
if (++retries < 3)
goto retry;
^ permalink raw reply [flat|nested] 10+ messages in thread
* [tip: sched/hrtick] hrtimer: Less agressive interrupt 'hang' handling
2026-03-11 12:15 ` Peter Zijlstra
@ 2026-03-11 20:16 ` tip-bot2 for Peter Zijlstra
0 siblings, 0 replies; 10+ messages in thread
From: tip-bot2 for Peter Zijlstra @ 2026-03-11 20:16 UTC (permalink / raw)
To: linux-tip-commits
Cc: kernel test robot, Peter Zijlstra (Intel), Thomas Gleixner, x86,
linux-kernel
The following commit has been merged into the sched/hrtick branch of tip:
Commit-ID: 92f7ee408c23fcc074e4952fb6ffacdf3eb86b56
Gitweb: https://git.kernel.org/tip/92f7ee408c23fcc074e4952fb6ffacdf3eb86b56
Author: Peter Zijlstra <peterz@infradead.org>
AuthorDate: Tue, 10 Mar 2026 20:02:21 +01:00
Committer: Thomas Gleixner <tglx@kernel.org>
CommitterDate: Wed, 11 Mar 2026 21:13:55 +01:00
hrtimer: Less agressive interrupt 'hang' handling
When the hrtimer_interrupt needs to restart more than 3 times and still has
expired timers, the interrupt is considered hung. To give the system a
little time to recover, the hardware timer is programmed a little into the
future.
Prior to commit 288924384856 ("hrtimer: Re-arrange hrtimer_interrupt()"),
this was relative to the amount of time spend serving the interrupt with a
max of 100 msec.
However, in order to simplify, and because this condition 'should' not
happen, the timeout was unconditionally set to 100 msec.
'Obviously' there is a benchmark that hits this hard, by programming a
ton of very short timers :-/
Since reprogramming is decoupled from the interrupt handling, the actual
execution time is lost, however the code does track max_hang_time. Using
that, rather than the 100 ms max restores performance.
stress-ng --timeout 60 --times --verify --metrics --no-rand-seed --timermix 64
bogo ops/s
288924384856^1: 23715979.93
288924384856: 11550049.77
patched: 23361116.78
Additionally, Thomas noted that cpu_base->hang_detected should not be
cleared until the next interrupt, such that __hrtimer_reprogram() won't
undo the extra delay.
Fixes: 288924384856 ("hrtimer: Re-arrange hrtimer_interrupt()")
Reported-by: kernel test robot <oliver.sang@intel.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Thomas Gleixner <tglx@kernel.org>
Link: https://patch.msgid.link/20260311121500.GF652779@noisy.programming.kicks-ass.net
Closes: https://lore.kernel.org/oe-lkp/202603102229.74b9dee4-lkp@intel.com
---
kernel/time/hrtimer.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index b94bd56..cafd677 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -2031,8 +2031,8 @@ static void hrtimer_rearm(struct hrtimer_cpu_base *cpu_base, ktime_t expires_nex
* Give the system a chance to do something else than looping
* on hrtimer interrupts.
*/
- expires_next = ktime_add_ns(ktime_get(), 100 * NSEC_PER_MSEC);
- cpu_base->hang_detected = false;
+ expires_next = ktime_add_ns(ktime_get(),
+ min(100 * NSEC_PER_MSEC, cpu_base->max_hang_time));
}
hrtimer_rearm_event(expires_next, deferred);
}
@@ -2121,6 +2121,7 @@ retry:
*/
now = hrtimer_update_base(cpu_base);
expires_next = hrtimer_update_next_event(cpu_base);
+ cpu_base->hang_detected = false;
if (expires_next < now) {
if (++retries < 3)
goto retry;
^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2026-03-11 20:16 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-03-10 14:46 [tip:sched/hrtick] [hrtimer] 2889243848: stress-ng.timermix.ops_per_sec 30.1% regression kernel test robot
2026-03-10 15:23 ` Peter Zijlstra
2026-03-10 17:11 ` Joe Talbott
2026-03-10 18:16 ` Peter Zijlstra
2026-03-10 18:50 ` Peter Zijlstra
2026-03-10 19:02 ` Peter Zijlstra
2026-03-11 9:40 ` Thomas Gleixner
2026-03-11 10:58 ` Peter Zijlstra
2026-03-11 12:15 ` Peter Zijlstra
2026-03-11 20:16 ` [tip: sched/hrtick] hrtimer: Less agressive interrupt 'hang' handling tip-bot2 for Peter Zijlstra
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.