From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from DM1PR04CU001.outbound.protection.outlook.com (mail-centralusazon11010037.outbound.protection.outlook.com [52.101.61.37]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5334E33938E for ; Thu, 6 Aug 2026 13:44:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.61.37 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786023849; cv=fail; b=P9EuOKC1soFx156f7oeFX3o34MulZEoA6sJFEciS7GqFcyPtL9koc+CBwrXZU3s5zad56MTZzRge86DZzKiR3GYatKU2dW68hltwm9N8hBvVwD5/6ApTFCdSKqgOL+GQbqdWMfN7P7/OjvR56R8i7Ytow2Jt/LeSHBZ5kc1o4JA= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786023849; c=relaxed/simple; bh=X3JOTOZk6yQL/yBT/WmcV9REoqnN9gbA3JapWKT0M/U=; h=From:To:Cc:Subject:Date:Message-ID:Content-Type:MIME-Version; b=S1JdV8uAKejvMZK/wgrERuq+Bn+z6cSp/i/cH1CrnDMNvUBsrYiCiPJ4afm8yslgX0+2M1QNL+0PceCCb19H1DzN3Uj5fmYeaE5X7ITUD/4KZPYecvA0arEA24QmOqOrCBgNFLQesygHhYQVAeY0movK0nthlfr5dXFYnycuVd0= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Ynq8yo1i; arc=fail smtp.client-ip=52.101.61.37 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Ynq8yo1i" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=r7xGvoxKjWl+clsVCgTbrno5goMOAhbRS9seU7xRcDC7u9j1/weUpbnFq3fMBAIZs21VDTKeS1/FXlj/p9xw6Xfhft5gY8Q9+13ID2gWuMhX69TNr/i+02cyKz5TLATFMuJn4/j0BkwMjed0RHgimJxMqoXzuwuuduJnOoyFJzm2hM1Js6Rfb+xDTFZwOtIxd7reggFGl+tTKRSNyFgF3+vl5HtEo/T+DuJWs9S58d7QS9fficyEqPZ6hbEeWJUOGx6mjm4i2DUzDqZUd+iMTAXhMczGG7jlEHVW1jT1Rdzgq3ltGSx+mxFvYqbwu+ZKxSQqGcNIpEgOrH/VsQ3MTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G60r6E5359lAm8jLBPsqd28Bx3W72N8olLQj4wt77IE=; b=R/tsedAnsJjpXCYLMy7NZ3suFZm2qOmlNjbivhBMOXF/E4GFOFsffxtq9F/mMkAlit+UEUneHi7sLn9sOuPLZYz/nn31O5d6syr88+1DAq4J+jzzXA6RazreGyXkpCO6k1mYZDPHkEbtGjWMuGMUJ1CufrNqoPz9IHsObIlFt6tTwchvNTYxcxUBoO5TAuYHOwdWh5ER9iNAsQqxyGcV5xl8x5kIpn93m+XS1dHznepDNKohQc4lJQ43rkSD8zWCBejgy7azEeRs9NF52+oiw9dn4MH9TVOwhRum20JISuVEDbWbJUZR9Pid86gPJUX5IbbKJLudZfAlVopRAn7hog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G60r6E5359lAm8jLBPsqd28Bx3W72N8olLQj4wt77IE=; b=Ynq8yo1i1Xhwt3Jqejd7+wXY9vEYvKKgtE2q+5PXvkJIMqhDd/ZPjRVIE3Zz0FfmoBk1rLQ5dh9raDHshusFkTvyaM7HtWN65LsEJqapqAFmOLkpgn9Y8fT8+eCIS8hGDoZZH8MEHff2LAoktHhpNN1xWtlj742IrvtnIRy39L3RPTzxKt1AwuVd0nifjRgc2IyIDUZWLq+DhORoTFJevJr2B4xLHzFzos7z04MbErUA4rcYSNAUCTEzGADOFM252WidQcvrh0ACurh9SLfO2DNKNNCZ8O9t/WuWk3gbzFH8gWtlInF1zG5FCOvO+M3J022SlVxIMNEEGz9bK1pNaw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB4827.namprd12.prod.outlook.com (2603:10b6:5:1d6::14) by CY8PR12MB8411.namprd12.prod.outlook.com (2603:10b6:930:6e::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.292.19; Thu, 6 Aug 2026 13:44:03 +0000 Received: from DM6PR12MB4827.namprd12.prod.outlook.com ([fe80::6261:3040:864b:159c]) by DM6PR12MB4827.namprd12.prod.outlook.com ([fe80::6261:3040:864b:159c%5]) with mapi id 15.21.0292.019; Thu, 6 Aug 2026 13:44:03 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Shrikanth Hegde , Phil Auld , Mete Durlu , linux-kernel@vger.kernel.org Subject: [PATCH v5] sched/fair: Prefer fully idle cores for NOHZ balancing Date: Thu, 6 Aug 2026 15:43:55 +0200 Message-ID: <20260806134355.592145-1-arighi@nvidia.com> X-Mailer: git-send-email 2.55.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: ZR0P278CA0035.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:1c::22) To DM6PR12MB4827.namprd12.prod.outlook.com (2603:10b6:5:1d6::14) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB4827:EE_|CY8PR12MB8411:EE_ X-MS-Office365-Filtering-Correlation-Id: c1fc73ab-f953-4eb3-1eaf-08def3c0c749 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|23010399003|366016|1800799024|10067099003|11063799006|56012099006|18002099003; X-Microsoft-Antispam-Message-Info: dWhzdavVKG1LSalwKLHZIKQ0Rfd742K4R/UNhhnEane0JMxAUEoZG9BknFy4oQO3SfSquT8eSBE3MNm10hCTsdAqERs87WAXjaHywhvUZI5coocdFaGK3OP7a83SSCbNkth8Toj42dsA3u0gJ1r1SFq/8z7fEfR0rdx/GbZGl6eKec7u0aHywJeffhPNGxndmtvIpQmw+eHfCkFrxUJ9VAiRLCfULNGawSYixJIinwpRLTarX7rPwKzIZN+/tHgzftmZADNiwz9h2ZY1iFsvoal/f8CmrUTahJJs4I3y6SzaIRA3LK3H1HxvnUdr/uT8jqhlRl4Qm9UQp7zBdSrCxIi84KHw9j2zHKAuulrU4I6lgcBEai60pO2aA0TIlrdZEuFa26p1qYvvXoI5lTQ3sTudZ29wiAYZiDhfnFWvFgBi5ZqcQYJONb79oUKgKLeJAc3FRzRa4V+dB1ygLV2sfd8KR56VBJXKN0kmbO7K+JgeGkG/cVGZyJcPvdxsv904e5vXTf5ne2BKJl5hZwel38t5l241L5vosu9ppXi1hKlGbkIIMSCn3MN8ndDQG9Qv/zcLWQYwo+hhDrKqD9gJEnmIwgNb66CeoMpV5a4vUkxs+HK/ONrMskktcoOKO+Xjf1wp4fsEPmfsyGKbxFV2JLpL/YNpcSASHUFXuuzfvAU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB4827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(23010399003)(366016)(1800799024)(10067099003)(11063799006)(56012099006)(18002099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?saMlqZ1apJ6/E6jxwtyjUb7M3wxYAkjLElrMc+H010hW4I6a2tYKzYEM4mY2?= =?us-ascii?Q?LCEVs/v8vmUVRLLKP8u1jchEmU0PgRMN7MIUYsvpDhb0WT36ulb+L7r4iSMI?= =?us-ascii?Q?VHg953NlSWZeVt11+4cl3jMFF13/f+OVhpHewU8HoXv8dpBCJysercVMuDq8?= =?us-ascii?Q?V6ksuzCLpf1GveDGoUKn3/Z19OH5Mii39sc7XuWo0G/5YJVAeLcEZeqLPCaZ?= =?us-ascii?Q?9Hz9dsDVRHV21GgbPUwpJS0vX6hn8PHWyXYBSRqKQAPhpADHcqPxUuSEwcDU?= =?us-ascii?Q?aBLMbPnBoG8nyBX6DVBJRlnLVx9Aj1f0rn8PiGtEzXoesI/VI50qu8LMLpCK?= =?us-ascii?Q?6XYKl3DxpgqewPKA8MUTQDUAxO4Y0fJmrTCtlqmBllnqDQsxvN/QQWpQ7Qtk?= =?us-ascii?Q?bhOFDhBtzeJzn7pf/IELHzo+Ql0FYskq1RR7TlTS91uuiWUYBBBExYDnCdSH?= =?us-ascii?Q?qpap6FWiVFkMiLL7sK1jHpp1UzPDcCDuPvgK898wvmKTAptPPv68nXdx6eqS?= =?us-ascii?Q?ptq0HSOLGoy677Rb3gkrih72Vm/u5Bu52dBjD8KrvFVU9ijr7wCfmUVhxJkH?= =?us-ascii?Q?3mfiVeUPikhOEdc5IoP2FnR3ph+Wjl79Zsd+06inaUZsgxTjM3bC306JRwnE?= =?us-ascii?Q?fRrID/uOy7ga4+4CAzeNc/mVjHLKFQb2Sk8jUdHChUrmkeNBPM/xC2p/kj+R?= =?us-ascii?Q?s3LcHLCxFZHnOtC7MvVYYZ2rwehSEapIZgKMiHTRpMWy5Ojw5QPCPvjA2kbT?= =?us-ascii?Q?7LEzBGBNDvDeUz7nlaroGbTnm03OKNHbvOBSOMHL5yuRKpNmEQd7AfpJLceA?= =?us-ascii?Q?+VwMWHTzhabEdQBigTwp/0GVZjOyRPvTFBxg3TVCsyD32f0pxILjjHLB5MWR?= =?us-ascii?Q?5uTxgG+c1Dlramk+4hsoI8QGLT89yNtTauZE7ZwBMBBGx2kp/GchGrV+8d+Y?= =?us-ascii?Q?+RJvhbbqjztxqpt+aXuZt48xgrGLl/svQbJvYKCb3cvP9pZSj8SI+TZXbif6?= =?us-ascii?Q?rhqGtQkwuB0SCOAGkX1A4W4Xn0Ce5oVmuAEjQbnMz5xD1Z+36P5ry0e7mi43?= =?us-ascii?Q?Ev8qusmr9EDnyiNIGukIj3dD1SxklOxnRWBgcfLFjGm+HZsD195zK+8q1oGj?= =?us-ascii?Q?u6IpLzpV1vGr2mYIJerddsIga2VbfjH+V43d3O6ixHo4gEAEWWakkOr5vnL7?= =?us-ascii?Q?D5KV4E8JyyIV3XL88bn1IilcZtfKzUHm4yk20kffpcPAd9nzVnjVEh4VNjMf?= =?us-ascii?Q?4bB0sJJppbucu862L7izWLWGp5T3tvhmiS6vdEDqfwLk70A2laY/ZEYvFEQj?= =?us-ascii?Q?bkWVWfuMtjDFeTgDO3tL/ktvZttufow/95E/OHOEn8GEiSwg33vqQa8sDBVo?= =?us-ascii?Q?tUoTKWxnGGonkPn1pG2LQidLb9m8le1PZ9MiuD+7Pg5h8QoR2qMYYm5GyciT?= =?us-ascii?Q?UVqP2I1LWHXaqHPTnZojCapdyYPseq6mSjWwvpEFgHxwtC0xjEzvayA/HKZe?= =?us-ascii?Q?Hi3GVYpXBIg5sciRvroDrXZ4oVXsU/tLnelZJBejcdCip36eRiY7e/djEhhU?= =?us-ascii?Q?8KFOvpmIxNQQH0vA2fJIpY4C0yl4UpWPjLrG3qkKvoa7Mz7+UTKHzrSnlRR6?= =?us-ascii?Q?/GGmSbtyFNlSh53X+YHzUceW6IV2CquZ8AmyBF9LwKfa4XQcX0rkxS3Q3pXS?= =?us-ascii?Q?0VmoqZpDmh4lYz+RQAMHl33M8LRQF4iVSXrUiG4qfJux8cbYZ/VUEmczsazj?= =?us-ascii?Q?zyHbGnSw9g=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c1fc73ab-f953-4eb3-1eaf-08def3c0c749 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB4827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Aug 2026 13:44:03.7432 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SuQaWGu5q8zfth+PbKVHxCF9mmrB2vSQFNffyFpkEL1hR2yxfi2S0CQNQSwQCZG8/EWXanQBgEfc/lyX2Jfq6A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB8411 find_new_ilb() selects the first idle housekeeping CPU without considering whether another thread is running on the same physical core. On an SMT system, the idle load balancer can therefore activate both siblings even when another housekeeping CPU has an entirely idle core. On most SMT systems, this is not problematic because the idle load balancer is a short-lived activity and the transient wakeup of a sibling has negligible performance impact. However, this can be particularly costly on NVIDIA Olympus cores used in Vera. Briefly activating an otherwise idle sibling can reduce the performance available to the other sibling and this effect does not necessarily end once the activated sibling becomes idle: after the ILB finishes and its CPU enters WFI, full single-thread performance is restored only after the sibling has remained idle for a qualification interval (10 Ki cycles on the tested Vera system). Repeated short sibling wakeups can therefore sustain the interference even with little actual overlap. Prevent this by preferring an idle housekeeping CPU whose entire SMT core is idle. Retain the first idle CPU as a fallback when no fully idle core is available, so NOHZ balancing continues to make forward progress. Once a partially busy core has been examined, skip its remaining SMT siblings to avoid repeating the core-idle check on wide SMT systems. Tests performed using an ad hoc GEMM benchmark running one CPU-intensive task per SMT core within its CPU affinity mask improved from approximately 6.2 TFLOP/s to 9.4 TFLOP/s. Note that this preference may wake a fully idle physical core instead of using an idle sibling of an active core, potentially increasing ILB wakeup latency or energy consumption on some architectures. It may also scan additional CPUs before selecting the one to run the ILB. The selection falls back to the first idle CPU when no fully idle SMT core is available. Non-SMT systems continue to select the first idle housekeeping CPU. Tested-by: K Prateek Nayak Reviewed-by: K Prateek Nayak Reviewed-by: Mete Durlu Reviewed-by: Vincent Guittot Reviewed-by: Shrikanth Hegde Signed-off-by: Andrea Righi --- Changes in v5: - Collect Tested-by and Reviewed-by tags - Reorder local variable declarations (Prateek Nayak) - Link to v4: https://lore.kernel.org/all/20260804151324.918020-1-arighi@nvidia.com/ Changes in v4: - Remove redundant this_cpu check (Prateek Nayak, Vincent Guittot) - Link to v3: https://lore.kernel.org/all/20260731191957.3199642-1-arighi@nvidia.com/ Changes in v3: - After finding an idle fallback, skip all siblings when a busy CPU is encountered, avoiding per-CPU traversal of known-busy cores (Mete Durlu) - Link to v2: https://lore.kernel.org/all/20260729163225.1987068-1-arighi@nvidia.com/ Changes in v2: - Avoid repeated is_core_idle() checks on wide SMT systems by pruning the remaining siblings of a partially busy core (Prateek Nayak) - Link to v1: https://lore.kernel.org/r/20260728214442.1648483-1-arighi@nvidia.com/ kernel/sched/fair.c | 55 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 44 insertions(+), 11 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 37001c63452e5..89bec68622db9 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -13964,29 +13964,62 @@ static inline int on_null_domain(struct rq *rq) */ static inline int find_new_ilb(void) { - int this_cpu = smp_processor_id(); - const struct cpumask *hk_mask; - int ilb_cpu; + int ilb_cpu, fallback = -1; + struct cpumask *ilb_cpus; + + lockdep_assert_irqs_disabled(); + + /* + * Reuse the per-CPU select_rq_mask, which is protected from concurrent + * use on this CPU by having interrupts disabled. + */ + ilb_cpus = this_cpu_cpumask_var_ptr(select_rq_mask); + cpumask_and(ilb_cpus, nohz.idle_cpus_mask, + housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)); + + for_each_cpu(ilb_cpu, ilb_cpus) { + if (!idle_cpu(ilb_cpu)) { + /* + * Once an idle fallback exists, a busy CPU proves that + * this core cannot be fully idle. Skip its siblings. + */ + if (sched_smt_active() && fallback >= 0) + cpumask_andnot(ilb_cpus, ilb_cpus, cpu_smt_mask(ilb_cpu)); + continue; + } - hk_mask = housekeeping_cpumask(HK_TYPE_KERNEL_NOISE); + /* + * Running the idle load balancer on an idle sibling of a busy + * SMT core can reduce the capacity available to its sibling. Prefer + * a CPU whose entire core is idle, but retain the first idle CPU as + * a fallback so idle balancing can still make progress when no fully + * idle core exists. + */ + if (sched_smt_active() && !is_core_idle(ilb_cpu)) { + if (fallback < 0) + fallback = ilb_cpu; - for_each_cpu_and(ilb_cpu, nohz.idle_cpus_mask, hk_mask) { - if (ilb_cpu == this_cpu) + /* + * The core is not idle, so there is no need to check + * any of its other SMT siblings. + */ + cpumask_andnot(ilb_cpus, ilb_cpus, + cpu_smt_mask(ilb_cpu)); continue; + } - if (idle_cpu(ilb_cpu)) - return ilb_cpu; + return ilb_cpu; } - return -1; + return fallback; } /* * Kick a CPU to do the NOHZ balancing, if it is time for it, via a cross-CPU * SMP function call (IPI). * - * We pick the first idle CPU in the HK_TYPE_KERNEL_NOISE housekeeping set - * (if there is one). + * Prefer a CPU on a fully idle core in the HK_TYPE_KERNEL_NOISE housekeeping + * set. Fall back to the first idle CPU when no fully idle core exists. */ static void kick_ilb(unsigned int flags) { -- 2.55.0