All of lore.kernel.org
 help / color / mirror / Atom feed
From: "Mohamed Abuelfotoh, Hazem" <abuehaze@amazon.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Mario Roy <marioeroy@gmail.com>, Chris Mason <clm@meta.com>,
	"Joseph Salisbury" <joseph.salisbury@oracle.com>,
	Adam Li <adamli@os.amperecomputing.com>,
	Josh Don <joshdon@google.com>, <mingo@redhat.com>,
	<juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
	<dietmar.eggemann@arm.com>, <rostedt@goodmis.org>,
	<bsegall@google.com>, <mgorman@suse.de>, <vschneid@redhat.com>,
	<linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 4/4] sched/fair: Proportional newidle balance
Date: Wed, 28 Jan 2026 16:24:37 +0000	[thread overview]
Message-ID: <2eefea7c-2962-4e50-a8cd-3cb101bedd56@amazon.com> (raw)
In-Reply-To: <20260127091343.GC217302@noisy.programming.kicks-ass.net>

On 27/01/2026 09:13, Peter Zijlstra wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> On Tue, Jan 27, 2026 at 09:50:25AM +0100, Peter Zijlstra wrote:
>> On Sun, Jan 25, 2026 at 12:22:21PM +0000, Mohamed Abuelfotoh, Hazem wrote:
>>
>>> I can confirm that we are seeing a 4-11% performance regression in v6.12.66
>>> on multiple benchmarks running on c7a.4xlarge AWS EC2 instances that are
>>> powered by AMD EPYC 9R14-series CPU (code-named Genoa) and c7i.4xlarge which
>>> is powered by 4th-Generation Intel Xeon Scalable processor (code-named
>>> Sapphire Rapids). The regression is caused by the commit 33cf66d88306
>>> ("sched/fair: Proportional newidle balance"). We were able to reclaim the
>>> performance back after reverting this commit. We also noticed that the
>>> impact is higher on AMD vs Intel.
>>>
>>> Benchmark Name |  Description                                   | Unit
>>> postgresql     |  HammerDB workload (TPC-C-like benchmark)  | NOPM
>>> nginx_lb       |  Testing NGINX as a load balancer      | RPS
>>> memcached      |  Testing using Lancet load generator       | QPS
>>>
>>> **Results on v6.12.66**
>>>
>>> Benchmark name | SUT EC2 Instance | Regression percentage
>>> postgresql     | c7a.4xlarge      | -4.0%
>>> postgresql     | c7i.4xlarge      | -4.0%
>>> nginx_lb       | c7a.4xlarge      | -5.0%
>>> memcached      | c7a.4xlarge      | -11.0%
>>
>> So only postgres has a regression on Intel? Memcached doesn't show
>> anything?
> 
> And just to be sure, v6.12.43-v6.12.65 have no problem?
> 
> That is, afaict those are the kernels that have:
> 
>    fc4289233e4b sched/fair: Bump sd->max_newidle_lb_cost when newidle balance fails
> 
> But not yet have:
> 
>    1b9c118fe318 sched/fair: Proportional newidle balance
>    c6ae271bc5fd sched/fair: Small cleanup to update_newidle_cost()
>    52aa889c6f57 sched/fair: Small cleanup to sched_balance_newidle()
>    81343616e712 sched/fair: Revert max_newidle_lb_cost bump
> 
> Because fc4289233e4b was also causing a ton of regressions (but also
> improving some workloads). 81343616e712 then reverts this and
> 1b9c118fe318 is supposed to be a compromise between these two.
> 
> So if your workloads are not affected by fc4289233e4b and 81343616e712,
> but somehow 1b9c118fe318 is causing fail, then I'm a little puzzled.
> 
We have definitely seen significant performance regression specifically 
on DB workloads because of fc4289233e4b ("sched/fair: Bump 
sd->max_newidle_lb_cost when newidle balance fails") which we reported 
in [1]. We were able to reclaim the performance back with ("81343616e712 
sched/fair: Revert max_newidle_lb_cost bump") before we start seeing 
negative impact from 1b9c118fe318 sched/fair: Proportional newidle balance.



[1] 
https://lore.kernel.org/all/006c9df2-b691-47f1-82e6-e233c3f91faf@oracle.com/T/#mb96105e4a320659b5aa68ec112bbeafaae37e769

  reply	other threads:[~2026-01-28 16:25 UTC|newest]

Thread overview: 69+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-11-07 16:06 [PATCH 0/4] sched: The newidle balance regression Peter Zijlstra
2025-11-07 16:06 ` [PATCH 1/4] sched/fair: Revert max_newidle_lb_cost bump Peter Zijlstra
2025-11-14 12:19   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2025-11-17 16:23   ` tip-bot2 for Peter Zijlstra
2025-11-07 16:06 ` [PATCH 2/4] sched/fair: Small cleanup to sched_balance_newidle() Peter Zijlstra
2025-11-10 13:55   ` Dietmar Eggemann
2025-11-10 14:04     ` Peter Zijlstra
2025-11-12 14:37   ` Shrikanth Hegde
2025-11-12 14:42     ` Peter Zijlstra
2025-11-12 15:08       ` Peter Zijlstra
2025-11-12 15:28         ` Shrikanth Hegde
2025-11-14  9:49           ` Peter Zijlstra
2025-11-14 10:22             ` Vincent Guittot
2025-11-14 11:05               ` Peter Zijlstra
2025-11-14 13:11                 ` Vincent Guittot
2025-11-14 12:19   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2025-11-17 16:23   ` tip-bot2 for Peter Zijlstra
2025-11-07 16:06 ` [PATCH 3/4] sched/fair: Small cleanup to update_newidle_cost() Peter Zijlstra
2025-11-14 12:19   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2025-11-17 16:23   ` tip-bot2 for Peter Zijlstra
2025-11-07 16:06 ` [PATCH 4/4] sched/fair: Proportional newidle balance Peter Zijlstra
2025-11-10 13:55   ` Dietmar Eggemann
2025-11-11  9:07   ` Adam Li
2025-11-11  9:20     ` Peter Zijlstra
2025-11-12 12:04       ` Adam Li
2025-11-12 13:41         ` Peter Zijlstra
2025-11-12 15:42   ` Shrikanth Hegde
2025-11-14  9:35     ` Peter Zijlstra
2025-11-14 12:18   ` [tip: sched/core] " tip-bot2 for Peter Zijlstra
2025-11-17 16:23   ` tip-bot2 for Peter Zijlstra
2026-01-18 20:46   ` [PATCH 4/4] " Mario Roy
2026-01-23 10:50     ` Peter Zijlstra
2026-01-23 11:03       ` Peter Zijlstra
2026-01-23 12:24         ` K Prateek Nayak
2026-01-28  4:08           ` K Prateek Nayak
2026-01-27  4:15         ` Mario Roy
2026-01-27 10:40         ` Peter Zijlstra
2026-01-27 15:17           ` Peter Zijlstra
2026-01-30  1:44             ` Mario Roy
2026-01-30  4:14             ` Mario Roy
2026-02-24  9:13             ` [tip: sched/core] sched/fair: More complex proportional " tip-bot2 for Peter Zijlstra
2026-01-25 12:22     ` [PATCH 4/4] sched/fair: Proportional " Mohamed Abuelfotoh, Hazem
2026-01-27  8:44       ` Peter Zijlstra
2026-01-28 15:48         ` Mohamed Abuelfotoh, Hazem
2026-01-29  9:19           ` Peter Zijlstra
2026-01-29  9:24             ` Peter Zijlstra
2026-01-30 16:12               ` Mohamed Abuelfotoh, Hazem
2026-01-30 13:16             ` Mohamed Abuelfotoh, Hazem
2026-02-02 10:51               ` Peter Zijlstra
2026-02-02 11:07                 ` Mohamed Abuelfotoh, Hazem
2026-02-04 12:45                   ` Mohamed Abuelfotoh, Hazem
2026-02-04 13:27                     ` Peter Zijlstra
2026-02-04 13:59                       ` Mohamed Abuelfotoh, Hazem
2026-02-04 14:05                         ` Peter Zijlstra
2026-02-04 22:48                           ` Mohamed Abuelfotoh, Hazem
2026-01-27  8:50       ` Peter Zijlstra
2026-01-27  9:13         ` Peter Zijlstra
2026-01-28 16:24           ` Mohamed Abuelfotoh, Hazem [this message]
2026-01-28 16:03         ` Mohamed Abuelfotoh, Hazem
2026-04-29  8:51   ` Qing Wang
2026-05-06  2:44     ` [PATCH] sched/fair: Replace random newidle_balance with Bresenham accumulator Qing Wang
2026-05-13  2:21       ` Qing Wang
2026-05-13  5:29       ` K Prateek Nayak
2026-05-14  2:22         ` Qing Wang
2026-06-02  5:46       ` [PATCH v2] " Qing Wang
2025-11-10 19:47 ` [PATCH 0/4] sched: The newidle balance regression Chris Mason
2025-11-11 19:08 ` Josh Don
2025-11-12 21:59 ` Chris Mason
2025-11-14  9:37   ` Peter Zijlstra

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2eefea7c-2962-4e50-a8cd-3cb101bedd56@amazon.com \
    --to=abuehaze@amazon.com \
    --cc=adamli@os.amperecomputing.com \
    --cc=bsegall@google.com \
    --cc=clm@meta.com \
    --cc=dietmar.eggemann@arm.com \
    --cc=joseph.salisbury@oracle.com \
    --cc=joshdon@google.com \
    --cc=juri.lelli@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=marioeroy@gmail.com \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.