From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CY7PR03CU001.outbound.protection.outlook.com (mail-westcentralusazon11010052.outbound.protection.outlook.com [40.93.198.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3001F371D1C for ; Fri, 27 Mar 2026 17:08:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.198.52 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774631322; cv=fail; b=TTb3farYj+QKGbiMgI4NnE5unEcfYXd2zq+7KPTGqTPAegPWh151TETHmmN2t72WQnesr+J/xsjz3ILK5RnWm2UPoHZIzuNuwT9SOCxa9UBwByHjXVR2mu0xoomYdn9V5PLnnG25U12GooPoQA4PYg7YJUA4Pr/w0QyCLOejzcM= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774631322; c=relaxed/simple; bh=NDEXynma5Kf+xj93ysjXO4qp6GjwVVwwJpOCxfZH0ys=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=hvbgfHT8qkO26qpphp7NxxPpbuvzZD1/gJsHtSDloUgewzNNDIHLL4dBKU97hanOeKWVkQB2XwyGzDeUmIks8dta++UEbfXAexi2LbqjusIdG5JTKJ5yN5shXcBqZzu9L/fYqJ+oRzl0XAnDtxVgQT7e3c0cA1sM1cYX1WTgPZo= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=H4UeblKT; arc=fail smtp.client-ip=40.93.198.52 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="H4UeblKT" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=p3j2qHmhT4/YMSthM5bsXOsNlvTkcmYGcztkTI4x3a1U+OtLD5DXnjTX7wS846FkNSlwasVd3a1/XgqSqC7Vz2N9bFZ1+1wcrL5D6p0LQxTXuxObV0KJ+L6EOcWqkZ6882tVDXJ+izHhH+rxXyWXwE4Mo8fkpiTQP9L0fGRBJpvJhtWytBPwGrUNXOEp1tQWIkz6YiWM1NVxnUFffVLDrc6g8u0QE+2IHuRxcn2tXJcueUsTiHT9Q4Nm921kSiDpRmfu8R15x3YN9Q5LH4x31D/DCbUnCkbaw1IdUQPsta0CT8OOLXNRvZmg5T27YfqKKqSwWWBxt4U2sjdosdrfww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kaajzYmVkom6f51cOm7fmXVE8iMcRkUf/uyLS9zg10c=; b=ZAGo7O0INvSF1B+ZprV30Vh4gpxIfwlBUcmofNT249DDPhqKTCeTVLm4v07VO/FSvDqmFzf4V3BZMj7zFrRm1NvvRVVjgkDIVBJB5epFo5hl9B7RAAoqPQ4040r65DXQZGfgQebwNnayikxguB4Sz139xnDh0171j8rrFJKOH94ufS7ChkgAlgl/Ab8u+XJ8tU4VPJqMgoAGrFPKl28VSRYWSlWQyNGd4hWX1RoxmiSSsX3l9zNR2QOTb8RDrNOWBI0sKwPuIVeAadguYGljtePkh8VR8Zer3V8bcgVI8yKct9k1pLImoUg3wtziX1APYl/xMfnA4q+Ad8+Jq6VZyQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kaajzYmVkom6f51cOm7fmXVE8iMcRkUf/uyLS9zg10c=; b=H4UeblKT75tAIGcj/k4vm/DeyKkv+dPjyUrTAg/BHDjdPoyf+H3+DRfCOHN0cfDFNsFyzVrp700YrarqY+eI42+VdPWzY5ZcJEMWauSgjfaAY+0+bE4eVyc8CJ2OSRyeQjK/CNdoSv7rkU+7Mp5LknCQbqFcNp7CgWbs705XU5fSPiNuUwrc7BUNL/HbUWcmCJO8hD/otvT5sCI1QYil+q4ohjzLVuGL03tlDQGpnB+XJlJIsYSRUQ5zai8mcSsMWgYdozcrQP8ETrJSdBgxHTHywXMmw5ai8Yxp0oIDKZLa2dHCtfy9hEkgQcX+zfLy6yqwUsjeqJUZcimmVEs3cg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by DS2PR12MB9821.namprd12.prod.outlook.com (2603:10b6:8:270::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.10; Fri, 27 Mar 2026 17:08:37 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9769.006; Fri, 27 Mar 2026 17:08:37 +0000 Date: Fri, 27 Mar 2026 18:08:28 +0100 From: Andrea Righi To: Shrikanth Hegde Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , linux-kernel@vger.kernel.org Subject: Re: [PATCH 0/4] sched/fair: SMT-aware asymmetric CPU capacity Message-ID: References: <20260326151211.1862600-1-arighi@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: MI2PEPF00000B87.ITAP293.PROD.OUTLOOK.COM (2603:10a6:298:1::405) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|DS2PR12MB9821:EE_ X-MS-Office365-Filtering-Correlation-Id: 7114e925-4244-4149-f432-08de8c237bf4 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014|18002099003|22082099003|56012099003; X-Microsoft-Antispam-Message-Info: 9zT0gSb3FN46QV7yW3+JomCBvvJr8ZDKhdVuxqYjF4iE2hWCPUe8ZIHKeAjiSnIfQnz9OGTAgQ4iTxE4BIsDy2qnp6z2h5r8HqO6O1Z+qXvfpCjG/SRdhzH11iU9IHMqp+isBds/L5GoglZ66C+I6hO4/TNECkHD8H+hSRiLTt1hBrEgunb3KBAfWXJeqlOV9BFx3Vpn47bw8CpfaW6f3uB6o23MEY9PxnnMZ3doMcq20BRabwtmwmo6itRgmyR0lLhi/q0PXUKp++frxwceqBsckYUepZ0+D6ss3C2PHKUHeTEtyDTClp7zLblkU09yIZ1hE1DoD7zm13cXkRB35BZyT02Q9lw+FmMS9i7axonuZJGP3z7lTBMWOCMH8EKMxcfWckDnjHE88d4GIc5/t47qXhYCSEu65QSdp1vCHD7Tn475k6KnNUs+R8Jw/6ezmu7UGFPgir07JgyWLH5V3dGf/GnUaKUxRn/cdPNUFbKXiP4OboPUuhcvkcg9HCWEguzgfFvlLQALQr9BA58R7mdpSx3IOXmFW9oZhgzrHrqK9sdIn9m7aC6CrUvXtjjyLt6AeLt7vGZ7qOVxCTdV1JAIOISpKOpuXFfsU+R1KePT6HqYsn7StQC9KmveED+ce5mlPcPxnWGqH2/d+QMbNcsefUuvDGt1yDot/c9rcSrvhCD8s6euNoaxtlURl0QcwBWSGR1mq5bEOqRnLcRjtHh/Z2NxOIomFYKe5GMutz0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(366016)(376014)(7416014)(18002099003)(22082099003)(56012099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?96xAJq8lhlLkfesr15FQkXyebjH8CNBPpu2LXXmc6/9C1sRIl3Jt6oCtKZzY?= =?us-ascii?Q?oiVDXN4NEDmNZjZjHyuL+o9SrDZEkkmkAwjPaPW0JBKXKXeDQhVOy7rjfYGv?= =?us-ascii?Q?MbXwfWeXmXIL6Wa2yFpns6XzAYdbkyuuqLm9o/s2pzmLpZojTpv45hZXpF1+?= =?us-ascii?Q?bGZ/gATOKKCJcOPdMi6QfdLGmqg7NjIFmKkKy/MTA8alWsxFtiFoZjbOhMHz?= =?us-ascii?Q?rXQrfbFQwnsIdBAlETRf5XXLfPU1cFAFjc35X2eQ2doICmvDdYFqnogH+/Ni?= =?us-ascii?Q?ueDXBByISoMuU7Uftk2IN0ec+0EmQ7tHCiN1SzfYCWALcVjcLukycGSncjZ7?= =?us-ascii?Q?3SFoSZoDs1g5yGHmJxd7ka73GCz+3Ht2b5c+P0LGuVxMIHUI9FvfLK79kFNd?= =?us-ascii?Q?XAv+vPSXhNrsy8VJiQPpCWyCaOzjMarPvfkduRTyZp+sYLhdVeLxw9yKpKkC?= =?us-ascii?Q?/vgfoFTG0ELmxF8UQ+luUh2ycX0iMlzXdzX2yVhf7/8xtEmla/jWqBgawq6R?= =?us-ascii?Q?tTnyz4wUtW4fWKw12R7n+t22vkam8hNAFYo/0CfsTR4DPEVcLMO2sBQxhOzt?= =?us-ascii?Q?bKH87Kx+4gKQ2tSOSv0t9auW6oKPnyCW+v+WXBPbOgVjlBFR+Gxziej11Ccd?= =?us-ascii?Q?AuBPOZY6qXr+HQv/sEVktROo940HX5UBHx5vncNVhtx9pBgYpjEzN1v/aedy?= =?us-ascii?Q?iAuv8MS9q4KZmV1Hp11CJ14ZkEtROp+gFhJtYtO8BEQeHIqoo7r67eWHDq0C?= =?us-ascii?Q?Ru6WX+07Xim71nvVe+uhMNWg8+l/kzeH7oXhAsIO1vUfxusLRntL1lZCc0SR?= =?us-ascii?Q?S49k7stGaCb6ffBMXUdeP+3WES2tm9/FT/5i0QXq9E1l7g8iU6OCdL8tVbf5?= =?us-ascii?Q?lNJHC4wTSZ0TKZB/vXzQpvovdiXImG+jE1ZPKjdoeVxO9DWHUCb9jtlX/CAC?= =?us-ascii?Q?JuEvMsGN98q77RZfQwACnxFcJ9XyCqzHMBzcISAqT6jm8WM2P4OANGbUJfq2?= =?us-ascii?Q?JOAFxKET9tCeqK7n69zxc0TCYO0TioprHlK1/LFIudW9FJQoRFKmsmpw7EEk?= =?us-ascii?Q?n08Epsz5Phf8xQ84ZNbalPSK1BCvWFDryDA9+wVCqKCymjxKHfHxtacTCM8Q?= =?us-ascii?Q?OoY1jVCYAqn3g5mVDAEPjeKQ60qdBgLc8nd/YRLzHFict1f5DpxB4pa0Syo6?= =?us-ascii?Q?AKHBa1Q0YujRXXGSca4au2bJghFoOpcS5eZ5FBWj4miO201X2Kn3R4F7BZVN?= =?us-ascii?Q?yT2UbnzLBS7tapZ4mclyUp05EwnrrX1MhcMRC/0m88LD3DQSKcSNiS5bQIik?= =?us-ascii?Q?Ni6BgP890N88WOoHaNaKEo777zhdczEdNzIrr8B6hcmYcx9sLWtqxsdxK8rQ?= =?us-ascii?Q?OR8UMDu+Ww3r7j2XWl8fkOBwH1+NMlccaOwFSDc8GevpC+wp8pF5H5Z02VLT?= =?us-ascii?Q?ejkWT8AhlPmzitIUO3QGVV6OGnPXD9qqItpz1KfU79xoFWIrtcEpVsEXjhCI?= =?us-ascii?Q?FlB1RItpaYe86aK7rcWHcPAG/mHuCMVrzK8UTiobKiNF38Jpu2buxa2w2Giw?= =?us-ascii?Q?yXB/kitFvl7qfdU5qrozEtTmwTagNBpuAjtv0hGm/PdovSu8RSRFzJ1ts+kN?= =?us-ascii?Q?Zm+9f2LzyG4F9P1v4s7iQ9pP6qYzYb2jVRhEnx/pzlAtftDJ+a3C0XvJUAzp?= =?us-ascii?Q?LJrJF3GosLAX1xl+8kByHjMDHH+JB09XvAxExU5lMbU9wBMBi6TOa8Aj2Q5Z?= =?us-ascii?Q?XCSeRluccg=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7114e925-4244-4149-f432-08de8c237bf4 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 27 Mar 2026 17:08:36.9986 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8UnMhhxNFuvFo7InXwdoDCxGBLd7g7FkAm+WnbL/xyPs5ptKb6bCmRkKC7kCLquHQDO2gHzpDnAXLVUkLisbCw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS2PR12MB9821 On Fri, Mar 27, 2026 at 10:01:03PM +0530, Shrikanth Hegde wrote: > Hi Andrea. > > On 3/26/26 8:32 PM, Andrea Righi wrote: > > This series attempts to improve SD_ASYM_CPUCAPACITY scheduling by > > introducing SMT awareness. > > > > = Problem = > > > > Nominal per-logical-CPU capacity can overstate usable compute when an SMT > > sibling is busy, because the physical core doesn't deliver its full nominal > > capacity. So, several SD_ASYM_CPUCAPACITY paths may pick high capacity CPUs > > that are not actually good destinations. > > > > How does energy model define the opp for SMT? For now, as suggested by Vincent, we should probably ignore EAS / energy model and keep it as it is (not compatible with SMT). I'll drop PATCH 3/4 and focus only at SD_ASYM_CPUCAPACITY + SMT. > > SMT systems have multiple of different functional blocks, a few ALU(arithmetic), > LSU(load store unit) etc. If same/similar workload runs on sibling, it would affect the > performance, but sibling is using different functional blocks, then it would > not. > > So underlying actual CPU Capacity of each thread depends on what each sibling is running. > I don't understand how does the firmware/energy models define this. They don't and they probably shouldn't. I don't think it's possible to model CPU capacity with a static nominal value when SMT is enabled, since the effective capacity changes if the corresponding sibling is busy or not. It should be up to the scheduler to figure out a reasonable way to estimate the actual capacity, considering the status of the other sibling (e.g., prioritizing the fully-idle SMT cores over the partially-idle SMT cores, like we do in other parts of the scheduler code). > > > = Proposed Solution = > > > > This patch set aligns those paths with a simple rule already used > > elsewhere: when SMT is active, prefer fully idle cores and avoid treating > > partially idle SMT siblings as full-capacity targets where that would > > mislead load balance. > > > > Patch set summary: > > > > - [PATCH 1/4] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection > > > > Prefer fully-idle SMT cores in asym-capacity idle selection. In the > > wakeup fast path, extend select_idle_capacity() / asym_fits_cpu() so > > idle selection can prefer CPUs on fully idle cores, with a safe fallback. > > > > - [PATCH 2/4] sched/fair: Reject misfit pulls onto busy SMT siblings on asym-capacity > > > > Reject misfit pulls onto busy SMT siblings on SD_ASYM_CPUCAPACITY. > > Provided for consistency with PATCH 1/4. > > > > - [PATCH 3/4] sched/fair: Enable EAS with SMT on SD_ASYM_CPUCAPACITY systems > > > > Enable EAS with SD_ASYM_CPUCAPACITY and SMT. Also provided for > > consistency with PATCH 1/4. I've also tested with/without > > /proc/sys/kernel/sched_energy_aware enabled (same platform) and haven't > > noticed any regression. > > > > - [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer > > > > When choosing the housekeeping CPU that runs the idle load balancer, > > prefer an idle CPU on a fully idle core so migrated work lands where > > effective capacity is available. > > > > The change is still consistent with the same "avoid CPUs with busy > > sibling" logic and it shows some benefits on Vera, but could have > > negative impact on other systems, I'm including it for completeness > > (feedback is appreciated). > > > > This patch set has been tested on the new NVIDIA Vera Rubin platform, where > > SMT is enabled and the firmware exposes small frequency variations (+/-~5%) > > as differences in CPU capacity, resulting in SD_ASYM_CPUCAPACITY being set. > > > > I assume the CPU_CAPACITY values fixed? > first sibling has max, while other has less? The firmware is exposing the same capacity for both siblings. SMT cores may have different capacity, but siblings within the same SMT core have the same capacity. There was an idea to expose a higher capacity for all the 1st sibling and a lower capacity for all the 2nd siblings, but I don't think it's a good idea, since that would just confuse the scheduler (and the 2nd sibling doesn't really have a lower nominal capacity if it's running alone). > > > Without these patches, performance can drop up to ~2x with CPU-intensive > > workloads, because the SD_ASYM_CPUCAPACITY idle selection policy does not > > account for busy SMT siblings. > > > > How is the performance measured here? Which benchmark? I've used an internal NVIDIA suite (based on NVBLAS), I also tried Linpack and got similar results. I'm planning to repeat the tests using public benchmarks and share the results as soon as I can. > By any chance you are running number_running_task <= (nr_cpus / smt_threads_per_core), > so it is all fitting nicely? That's the case that gives me the optimal results. > > If you increase those numbers, how does the performance numbers compare? I tried different number of tasks. The more I approach system saturation the smaller the benefits are. When I completely saturate the system I don't see any benefit with this changes, neither regressions, but I guess that's expected. > > Also, whats the system is like? SMT level? 2 siblings for each SMT core. Thanks, -Andrea