From: kernel test robot <oliver.sang@intel.com>
To: Mel Gorman <mgorman@techsingularity.net>
Cc: <oe-lkp@lists.linux.dev>, <lkp@intel.com>, <oliver.sang@intel.com>
Subject: [mel:sched-preemptnext-v2r1] [sched/fair] 09f89cc1ef: pts.schbench.32.usec,_99.9th_latency_percentile 20.3% regression
Date: Thu, 20 Nov 2025 14:05:30 +0800 [thread overview]
Message-ID: <202511201346.cd26911a-lkp@intel.com> (raw)
Hello,
kernel test robot noticed a 20.3% regression of pts.schbench.32.usec,_99.9th_latency_percentile on:
commit: 09f89cc1ef832eaf976764d34e781d0dcacc9b67 ("sched/fair: Reimplement NEXT_BUDDY to align with EEVDF goals")
https://git.kernel.org/cgit/linux/kernel/git/mel/linux.git sched-preemptnext-v2r1
testcase: pts
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 96 threads 2 sockets Intel(R) Xeon(R) Gold 6252 CPU @ 2.10GHz (Cascade Lake) with 512G memory
parameters:
test: schbench-1.1.0
option_a: 32
option_b: 32
cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202511201346.cd26911a-lkp@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251120/202511201346.cd26911a-lkp@intel.com
=========================================================================================
compiler/cpufreq_governor/kconfig/option_a/option_b/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/32/32/debian-12-x86_64-phoronix/lkp-csl-2sp7/schbench-1.1.0/pts
commit:
472046dff7 ("sched/fair: Enable scheduler feature NEXT_BUDDY")
09f89cc1ef ("sched/fair: Reimplement NEXT_BUDDY to align with EEVDF goals")
472046dff71dceab 09f89cc1ef832eaf976764d34e7
---------------- ---------------------------
%stddev %change %stddev
\ | \
19468 +22.1% 23767 vmstat.system.cs
2.59e+09 -9.7% 2.338e+09 cpuidle..time
2776948 -9.7% 2506500 cpuidle..usage
48.54 -4.7 43.80 mpstat.cpu.all.idle%
22.40 ± 17% +65.6% 37.10 ± 21% mpstat.max_utilization.seconds
37872 +3.6% 39253 ± 2% proc-vmstat.nr_mapped
15004 ± 44% +85.2% 27793 ± 17% proc-vmstat.numa_hint_faults
938678 +2.3% 960265 proc-vmstat.numa_hit
838158 +2.7% 860739 proc-vmstat.numa_local
610583 +3.0% 628976 proc-vmstat.pgfault
1810 ± 14% -60.0% 723.87 ± 54% sched_debug.cfs_rq:/.avg_vruntime.min
1810 ± 14% -60.0% 723.87 ± 54% sched_debug.cfs_rq:/.min_vruntime.min
1055 ± 36% -51.4% 513.39 ± 61% sched_debug.cfs_rq:/system.slice/containerd.service.se->vruntime.min
1643718 ± 5% -13.1% 1428339 ± 6% sched_debug.cpu.avg_idle.avg
992216 ± 4% -15.3% 840210 ± 7% sched_debug.cpu.max_idle_balance_cost.avg
298.10 ± 9% -22.7% 230.30 ± 14% sched_debug.cpu.nr_switches.min
3004 +213.8% 9427 pts.schbench.32.usec,_50.0th_latency_percentile
9660 ± 3% +60.9% 15542 pts.schbench.32.usec,_75.0th_latency_percentile
16155 +36.3% 22016 pts.schbench.32.usec,_90.0th_latency_percentile
38323 +20.3% 46118 ± 4% pts.schbench.32.usec,_99.9th_latency_percentile
646375 +41.2% 912804 pts.time.involuntary_context_switches
4835 +9.6% 5298 pts.time.percent_of_cpu_this_job_got
2582 +9.6% 2829 pts.time.user_time
264445 -1.5% 260481 pts.time.voluntary_context_switches
1.722e+10 +9.0% 1.878e+10 perf-stat.i.branch-instructions
20009 +23.8% 24762 perf-stat.i.context-switches
1.383e+11 +9.7% 1.516e+11 perf-stat.i.cpu-cycles
2203 ± 2% +25.3% 2760 perf-stat.i.cpu-migrations
51064 ± 6% +31.5% 67135 ± 7% perf-stat.i.cycles-between-cache-misses
2.038e+10 +9.1% 2.225e+10 perf-stat.i.dTLB-loads
6.848e+09 +9.1% 7.471e+09 perf-stat.i.dTLB-stores
39.27 ± 6% +15.3 54.55 ± 5% perf-stat.i.iTLB-load-miss-rate%
8.795e+10 +9.0% 9.591e+10 perf-stat.i.instructions
1.44 +9.7% 1.58 perf-stat.i.metric.GHz
463.04 +9.1% 505.10 perf-stat.i.metric.M/sec
8701 +4.1% 9058 perf-stat.i.minor-faults
477970 ± 3% -20.0% 382287 ± 5% perf-stat.i.node-load-misses
212470 ± 8% -18.1% 174001 ± 10% perf-stat.i.node-store-misses
8705 +4.1% 9063 perf-stat.i.page-faults
0.30 ± 2% -0.0 0.27 perf-stat.overall.branch-miss-rate%
71.60 -4.8 66.83 ± 3% perf-stat.overall.node-load-miss-rate%
55.21 ± 5% -6.6 48.63 ± 6% perf-stat.overall.node-store-miss-rate%
1.713e+10 +8.3% 1.855e+10 perf-stat.ps.branch-instructions
19876 +23.0% 24457 perf-stat.ps.context-switches
1.376e+11 +8.9% 1.499e+11 perf-stat.ps.cpu-cycles
2193 ± 2% +24.4% 2728 perf-stat.ps.cpu-migrations
2.028e+10 +8.4% 2.199e+10 perf-stat.ps.dTLB-loads
6.813e+09 +8.4% 7.383e+09 perf-stat.ps.dTLB-stores
8.749e+10 +8.3% 9.478e+10 perf-stat.ps.instructions
8447 +4.7% 8847 perf-stat.ps.minor-faults
470926 ± 3% -20.3% 375205 ± 5% perf-stat.ps.node-load-misses
210067 ± 8% -18.5% 171191 ± 10% perf-stat.ps.node-store-misses
8451 +4.7% 8852 perf-stat.ps.page-faults
4.733e+12 +8.0% 5.11e+12 perf-stat.total.instructions
0.11 ± 25% -76.7% 0.03 ± 30% perf-sched.sch_delay.avg.ms.anon_pipe_read.fifo_pipe_read.vfs_read.ksys_read
0.02 ± 25% -51.2% 0.01 ± 13% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
0.02 ± 55% -65.5% 0.01 ± 68% perf-sched.sch_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
0.26 ± 34% -63.2% 0.10 ± 40% perf-sched.sch_delay.avg.ms.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.28 ± 20% -63.3% 0.10 ± 9% perf-sched.sch_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
0.05 ± 6% -87.4% 0.01 ± 48% perf-sched.sch_delay.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex
1.44 ± 5% -86.0% 0.20 ± 5% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
1.44 ± 4% -86.0% 0.20 ± 3% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
0.31 ± 10% -62.4% 0.12 ± 11% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
0.30 ± 12% -61.3% 0.12 ± 14% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
0.27 ± 10% -60.8% 0.11 ± 8% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
0.27 ± 9% -60.9% 0.11 ± 7% perf-sched.sch_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown]
0.83 ± 19% -52.3% 0.40 ± 47% perf-sched.sch_delay.avg.ms.schedule_timeout.__wait_for_common.wait_for_completion_state.kernel_clone
0.01 ± 13% +46.4% 0.02 ± 21% perf-sched.sch_delay.avg.ms.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
1.46 ± 38% -95.3% 0.07 ±142% perf-sched.sch_delay.max.ms.do_nanosleep.hrtimer_nanosleep.__x64_sys_nanosleep.do_syscall_64
9.27 ± 45% +119.8% 20.37 ± 20% perf-sched.sch_delay.max.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
48.28 ± 49% -39.5% 29.22 ± 12% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
21.85 ± 25% -35.6% 14.08 ± 13% perf-sched.sch_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
0.89 ± 4% -84.9% 0.13 ± 3% perf-sched.total_sch_delay.average.ms
30.16 ± 2% -26.1% 22.29 perf-sched.total_wait_and_delay.average.ms
156088 +29.6% 202359 perf-sched.total_wait_and_delay.count.ms
29.28 -24.3% 22.16 perf-sched.total_wait_time.average.ms
714.37 ± 14% -66.2% 241.51 ± 72% perf-sched.wait_and_delay.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
98.02 ± 20% -54.2% 44.90 ± 25% perf-sched.wait_and_delay.avg.ms.anon_pipe_read.fifo_pipe_read.vfs_read.ksys_read
10.18 ± 5% -14.4% 8.71 ± 6% perf-sched.wait_and_delay.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
17.04 ± 17% -55.0% 7.66 ± 4% perf-sched.wait_and_delay.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
11.48 ± 2% -32.8% 7.71 ± 2% perf-sched.wait_and_delay.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex
30.26 -27.0% 22.09 perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
30.42 -26.7% 22.29 perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
31.81 ± 7% -41.4% 18.64 ± 4% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
32.87 ± 7% -45.2% 18.02 ± 3% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
31.78 ± 7% -26.2% 23.45 perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
31.88 ± 6% -26.1% 23.55 ± 2% perf-sched.wait_and_delay.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown]
139.45 ± 5% -9.9% 125.64 ± 4% perf-sched.wait_and_delay.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_poll
2.10 ± 72% +247.6% 7.30 ± 44% perf-sched.wait_and_delay.count.__cond_resched.kfree_rcu_monitor.process_one_work.worker_thread.kthread
0.40 ±165% +1050.0% 4.60 ± 70% perf-sched.wait_and_delay.count.__cond_resched.migrate_pages_batch.migrate_pages.migrate_misplaced_folio.do_numa_page
431.30 ± 21% +123.3% 963.20 ± 24% perf-sched.wait_and_delay.count.anon_pipe_read.fifo_pipe_read.vfs_read.ksys_read
2165 ± 20% +477.8% 12510 ± 4% perf-sched.wait_and_delay.count.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
46617 ± 2% +19.0% 55460 perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
45256 ± 2% +18.6% 53674 perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
1532 ± 12% +58.9% 2435 ± 19% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
1472 ± 13% +54.8% 2278 ± 19% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
4347 ± 6% +200.5% 13063 ± 3% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
4151 ± 5% +201.2% 12503 ± 3% perf-sched.wait_and_delay.count.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown]
16.16 ±153% +361.1% 74.50 ± 16% perf-sched.wait_and_delay.max.ms.__cond_resched.migrate_pages_batch.migrate_pages.migrate_misplaced_folio.do_numa_page
110.98 ± 6% +20.5% 133.75 ± 7% perf-sched.wait_and_delay.max.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
712.75 ± 14% -66.2% 241.20 ± 72% perf-sched.wait_time.avg.ms.__x64_sys_pause.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
97.90 ± 20% -54.2% 44.87 ± 25% perf-sched.wait_time.avg.ms.anon_pipe_read.fifo_pipe_read.vfs_read.ksys_read
10.16 ± 5% -14.3% 8.71 ± 6% perf-sched.wait_time.avg.ms.do_nanosleep.hrtimer_nanosleep.common_nsleep.__x64_sys_clock_nanosleep
16.76 ± 17% -54.9% 7.56 ± 4% perf-sched.wait_time.avg.ms.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.[unknown]
11.44 ± 2% -32.6% 7.70 ± 2% perf-sched.wait_time.avg.ms.futex_do_wait.__futex_wait.futex_wait.do_futex
28.82 -24.0% 21.89 perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown]
28.98 -23.8% 22.08 perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_apic_timer_interrupt.[unknown].[unknown]
31.49 ± 7% -41.2% 18.52 ± 4% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown]
32.57 ± 7% -45.0% 17.91 ± 3% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
31.50 ± 7% -25.9% 23.35 perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
31.61 ± 6% -25.8% 23.45 ± 2% perf-sched.wait_time.avg.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown].[unknown]
139.31 ± 5% -9.9% 125.52 ± 4% perf-sched.wait_time.avg.ms.schedule_hrtimeout_range_clock.poll_schedule_timeout.constprop.0.do_poll
0.39 ± 58% -66.4% 0.13 ± 39% perf-sched.wait_time.avg.ms.wait_for_partner.fifo_open.do_dentry_open.vfs_open
16.15 ±153% +361.2% 74.49 ± 16% perf-sched.wait_time.max.ms.__cond_resched.migrate_pages_batch.migrate_pages.migrate_misplaced_folio.do_numa_page
98.82 ± 5% +16.0% 114.63 ± 4% perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_call_function_single.[unknown].[unknown]
11.73 ±200% +240.8% 39.96 ± 92% perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_irq_work.[unknown]
100.51 ± 4% +24.2% 124.84 ± 6% perf-sched.wait_time.max.ms.irqentry_exit_to_user_mode.asm_sysvec_reschedule_ipi.[unknown]
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
reply other threads:[~2025-11-20 6:05 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=202511201346.cd26911a-lkp@intel.com \
--to=oliver.sang@intel.com \
--cc=lkp@intel.com \
--cc=mgorman@techsingularity.net \
--cc=oe-lkp@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.