From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fra-out-015.esa.eu-central-1.outbound.mail-perimeter.amazon.com (fra-out-015.esa.eu-central-1.outbound.mail-perimeter.amazon.com [18.158.153.154]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15DA029D288 for ; Wed, 28 Jan 2026 16:25:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=18.158.153.154 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769617511; cv=none; b=nBUP2FzypsU1bUkY5tm5kWWPGTuloxATya4SdRoAiSAF8+9aMvrLXIEcwirldiKQ822cvpbYrYca3it9EQ6s6CUlWdZS8BVCoOH2Mi3/D6ORU0WJ2OwHiE3XO0u3laqLUHZ7cxsGVJrhOJ63ANyQIE9GjIOgrwGqYEk2lQZt76U= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1769617511; c=relaxed/simple; bh=W5rmQj7Pl4su909uJ7EYpQj8x6LJrYYScBYFFqBTedo=; h=Subject:Message-ID:Date:MIME-Version:To:CC:References:From: In-Reply-To:Content-Type; b=h7CIs8UICEAH83JmIahCYog3SRmfNTujovPhXB8N+MQxjdMHYL1sekZpve35AuRSgBra4oSF5HBWNJzEmbu+gD+GT9B+vwpUKKT2VkhTZZIG6AHnsAa7HUICvDTWIIz76C0ZwCVNGPhKS6UNaL9lz+gfs8FjktQkfeUbCOvn1aQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com; spf=pass smtp.mailfrom=amazon.com; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b=WnoC9ZAg; arc=none smtp.client-ip=18.158.153.154 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amazon.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=amazon.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=amazon.com header.i=@amazon.com header.b="WnoC9ZAg" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazoncorp2; t=1769617510; x=1801153510; h=message-id:date:mime-version:to:cc:references:from: in-reply-to:content-transfer-encoding:subject; bh=2ckEUcKKM2wjqoBb2pTr8hBO5m3zZ+Ky6EUsQl6xdqw=; b=WnoC9ZAg9yDgV34nmKsq3uUGLKdnWGqyGt+aklNYRQRjYgJdEQLr3doA v7Np1fTuhK6ghEYQHmcxdaGiwagYZngoHRs42wE5Oe5Nwp0+PL4hxpPYK zmIkr9lLCgE5RA6NeGGIRb8DBeuqQCEN7e2eLMUAZIlrAWLxs9qL7bLQ0 D6kyJWdqgBxisClL0C+ez7Z8Bfbzk4JdDGCeDAuUpuSQThkUMK6ci59+R /ZqF3/O0HDEgwU9FB0m2uVar+7Jn9jCdq4PJ/ZTBIDLgQ/ZNcPv8VIe91 UFWdnMo0F/RPSz5m5cJrgghayMG3sTBz8pCFHH2AE8WZuWbBSQisNOz/g w==; X-CSE-ConnectionGUID: Z9j39i/tSV6HprbonT8QoA== X-CSE-MsgGUID: fEluMKEZSxCu9gfTvVl5kQ== X-IronPort-AV: E=Sophos;i="6.21,258,1763424000"; d="scan'208";a="8444847" Subject: Re: [PATCH 4/4] sched/fair: Proportional newidle balance Received: from ip-10-6-6-97.eu-central-1.compute.internal (HELO smtpout.naws.eu-central-1.prod.farcaster.email.amazon.dev) ([10.6.6.97]) by internal-fra-out-015.esa.eu-central-1.outbound.mail-perimeter.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jan 2026 16:24:45 +0000 Received: from EX19MTAEUC001.ant.amazon.com [54.240.197.225:8127] by smtpin.naws.eu-central-1.prod.farcaster.email.amazon.dev [10.0.11.34:2525] with esmtp (Farcaster) id 064d91c9-a03b-499d-afd8-47376a3b4c13; Wed, 28 Jan 2026 16:24:45 +0000 (UTC) X-Farcaster-Flow-ID: 064d91c9-a03b-499d-afd8-47376a3b4c13 Received: from EX19D018EUA004.ant.amazon.com (10.252.50.85) by EX19MTAEUC001.ant.amazon.com (10.252.51.193) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.35; Wed, 28 Jan 2026 16:24:39 +0000 Received: from [192.168.19.37] (10.106.82.9) by EX19D018EUA004.ant.amazon.com (10.252.50.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.2562.35; Wed, 28 Jan 2026 16:24:38 +0000 Message-ID: <2eefea7c-2962-4e50-a8cd-3cb101bedd56@amazon.com> Date: Wed, 28 Jan 2026 16:24:37 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: Peter Zijlstra CC: Mario Roy , Chris Mason , "Joseph Salisbury" , Adam Li , Josh Don , , , , , , , , , References: <20251107160645.929564468@infradead.org> <20251107161739.770122091@infradead.org> <8760001e-0274-454c-a4e4-1f38a9695b88@gmail.com> <3ef2a178-db96-4a2c-9e74-44ba81bef030@amazon.com> <20260127085025.GW171111@noisy.programming.kicks-ass.net> <20260127091343.GC217302@noisy.programming.kicks-ass.net> Content-Language: en-US From: "Mohamed Abuelfotoh, Hazem" In-Reply-To: <20260127091343.GC217302@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit X-ClientProxiedBy: EX19D015EUA003.ant.amazon.com (10.252.50.252) To EX19D018EUA004.ant.amazon.com (10.252.50.85) On 27/01/2026 09:13, Peter Zijlstra wrote: > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > On Tue, Jan 27, 2026 at 09:50:25AM +0100, Peter Zijlstra wrote: >> On Sun, Jan 25, 2026 at 12:22:21PM +0000, Mohamed Abuelfotoh, Hazem wrote: >> >>> I can confirm that we are seeing a 4-11% performance regression in v6.12.66 >>> on multiple benchmarks running on c7a.4xlarge AWS EC2 instances that are >>> powered by AMD EPYC 9R14-series CPU (code-named Genoa) and c7i.4xlarge which >>> is powered by 4th-Generation Intel Xeon Scalable processor (code-named >>> Sapphire Rapids). The regression is caused by the commit 33cf66d88306 >>> ("sched/fair: Proportional newidle balance"). We were able to reclaim the >>> performance back after reverting this commit. We also noticed that the >>> impact is higher on AMD vs Intel. >>> >>> Benchmark Name | Description | Unit >>> postgresql | HammerDB workload (TPC-C-like benchmark) | NOPM >>> nginx_lb | Testing NGINX as a load balancer | RPS >>> memcached | Testing using Lancet load generator | QPS >>> >>> **Results on v6.12.66** >>> >>> Benchmark name | SUT EC2 Instance | Regression percentage >>> postgresql | c7a.4xlarge | -4.0% >>> postgresql | c7i.4xlarge | -4.0% >>> nginx_lb | c7a.4xlarge | -5.0% >>> memcached | c7a.4xlarge | -11.0% >> >> So only postgres has a regression on Intel? Memcached doesn't show >> anything? > > And just to be sure, v6.12.43-v6.12.65 have no problem? > > That is, afaict those are the kernels that have: > > fc4289233e4b sched/fair: Bump sd->max_newidle_lb_cost when newidle balance fails > > But not yet have: > > 1b9c118fe318 sched/fair: Proportional newidle balance > c6ae271bc5fd sched/fair: Small cleanup to update_newidle_cost() > 52aa889c6f57 sched/fair: Small cleanup to sched_balance_newidle() > 81343616e712 sched/fair: Revert max_newidle_lb_cost bump > > Because fc4289233e4b was also causing a ton of regressions (but also > improving some workloads). 81343616e712 then reverts this and > 1b9c118fe318 is supposed to be a compromise between these two. > > So if your workloads are not affected by fc4289233e4b and 81343616e712, > but somehow 1b9c118fe318 is causing fail, then I'm a little puzzled. > We have definitely seen significant performance regression specifically on DB workloads because of fc4289233e4b ("sched/fair: Bump sd->max_newidle_lb_cost when newidle balance fails") which we reported in [1]. We were able to reclaim the performance back with ("81343616e712 sched/fair: Revert max_newidle_lb_cost bump") before we start seeing negative impact from 1b9c118fe318 sched/fair: Proportional newidle balance. [1] https://lore.kernel.org/all/006c9df2-b691-47f1-82e6-e233c3f91faf@oracle.com/T/#mb96105e4a320659b5aa68ec112bbeafaae37e769