From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BN8PR05CU002.outbound.protection.outlook.com (mail-eastus2azon11011012.outbound.protection.outlook.com [52.101.57.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52BE33DC4AA for ; Mon, 30 Mar 2026 17:29:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.57.12 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774891769; cv=fail; b=cDy6QeS/A8TaC/7SZELkNyMD6vHhZ2cuKbS7kyZ17WyiZA5ETU8G/7CoVAB8vgtXRiyfRvqtyYsggQHUU00zvDwrD83pqaM+PwtKsg8m5X9HrnQC/CIE6xXnXNokDMtDr7P3eg6TOUqi3peBS0JY8Ooo8qs/Kk7xwjOQI5HhaDU= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774891769; c=relaxed/simple; bh=GjYpd5KL3B6LZd8B+80zhJ+Q/5a/VmK+mOPp3wAKM7g=; h=Date:From:To:Cc:Subject:Message-ID:References:Content-Type: Content-Disposition:In-Reply-To:MIME-Version; b=CTSpOP/Q2++VQ2uvrdonFENDBuNixq7wBbFql4EcsKaPGhEvwl54I/szh9g1AikcXet4SbeAaXIGkEor5s7AmR+q5yKSBziQVjnH71XHZNPd+J2BtcIjCaTnpJShOnpB0YErNARupnywaB+K2UgIe73askWU0Z+ydoap1E2K2ng= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=Z9ujwrjS; arc=fail smtp.client-ip=52.101.57.12 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="Z9ujwrjS" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=CfIwQKj9loc8kfIo47ZVtbrDk54VQyJlSgAqHrGhbSzA3A4NVPFlRdtVYOzF2HbN5TrSCuFB8N4xwYem8MQLnEM4OO4t8PDbKAzka/fdIsfBGaFD9ra7U6/y6kvCY4F8tN+3f5cjZBpNhX3dlcl2Xynxg4oeqVeXFkZcL3gjbGiYvaXO1lqCfuydWi/DfJT+MhYrWeha08aissO50Bz4Yb3LqJyixQPKQI1LfF1fo1gwYvY+zInQNpTw/UY5zX5VLhxk2XH3MLx9ng9fXMzWJ19KljVgHVKQPuiUuLUama0MOE6m1odrbJkpO3WYAzd8ORHXAXg0JlBVaUsaRDIdDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yjJfi7fVQOhVbXQtHDJgDRoZMxbinXEyvSeOVfbNv6M=; b=kVOr8Pjcb2AMN6yWmyAVmDVjnqgft6FPPLOh+SH5GvKhZBZBVYeLaCXlivSIx+64FO+Y8n6JkrRApSiMXaKPbRa9+vknVEU0k3kB2u0caHileUrDi4rLJ/JznLaoJPf5/f2hVtKtmvsdfeJ3HdfWh5vtJOqTtq1yNa/5+dJs0r9gL+3CJlRvXubuu31igq/cbWn3VY6/Y5UEdacWY5qfTw+xdz25uLGGDeG6i0Tij8uz4DIhaq/0LSCr1r2bcqOmAi3P5EnwL66UrK9KpyM3KlE1GE6OfN4m3V6lwCvjqUoMVMebgALPygq6q5TAmVDt33ntNIBIItDwIgIn7lGatg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yjJfi7fVQOhVbXQtHDJgDRoZMxbinXEyvSeOVfbNv6M=; b=Z9ujwrjSWwF+ifHk4CXTy52ZpKETo1xmPv0+8bUNLf9ODf7/zCebkn3PoD+kMRTcU9ukwhDP+m2dbJWuFhvviQOPkLZx3Eq43Nc7kW14XUlxIkNRMXHXBwlO12gsZuNedYHPr0imwgx7vyAbZNpm9od7D2ZUWCTBJnO/ZGfJKhuRvtsUG+LWj0e085Yz+aBgTNa35NYgm03gYRfNPmfCjIWVuSQw7SgGRA7dkg6dlot4BG8Aycd0r677suB/3U7Yva3rupjokcgqvVeiPNKsGSFgq9G/UZXDG1wMScVLWYbCN9VsrfUr4NvBf7eCoehGEmVbFOgRJ71iTiTMxdRGrg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by PH7PR12MB5854.namprd12.prod.outlook.com (2603:10b6:510:1d5::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9769.7; Mon, 30 Mar 2026 17:29:23 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9769.014; Mon, 30 Mar 2026 17:29:22 +0000 Date: Mon, 30 Mar 2026 19:29:13 +0200 From: Andrea Righi To: K Prateek Nayak Cc: Vincent Guittot , Ingo Molnar , Peter Zijlstra , Juri Lelli , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , linux-kernel@vger.kernel.org Subject: Re: [PATCH 4/4] sched/fair: Prefer fully-idle SMT core for NOHZ idle load balancer Message-ID: References: <20260326151211.1862600-1-arighi@nvidia.com> <20260326151211.1862600-5-arighi@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: MI1P293CA0010.ITAP293.PROD.OUTLOOK.COM (2603:10a6:290:2::7) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|PH7PR12MB5854:EE_ X-MS-Office365-Filtering-Correlation-Id: 1dbd2b51-7508-4299-0ff1-08de8e81e1da X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|18002099003|56012099003|22082099003; X-Microsoft-Antispam-Message-Info: MzdbP69m5D2JHz5NOHBv0IjAnRzeIfKGLAzLv2LM450QSuykLp5M1zG1/qviolwEgciGAEUZZSk2xZtkIIhDUWO2MjXfcaO2osWCegXRug8nHl23h3nyX+kWBvqHddqYmMicN8skicwjgUOlbNODxAQULy4OrfVvzcsXWzsuRv35OMI4NhjtsbhhN1pI7ThjChslT5XV6TGgVcH8NoR3fntkV6ag07+Q+eCqmkwbx1UYdv3j73Nx/wT4HAdEptjJEU73dTts6Frzsht6Rl7/i/0Zcu5gO1gQwXVXRlKq8WsTdR1F71KiQAMFO+dONZzxoLPikO7tqbiEakxvy1O0ITMz+Nyu04Z0Pvec8h94HAYUoHr/R5PvQCDrDdyxam7OubI3smarbaa7y6ji8szkGFjfgg0VK3MzMVTXCkIzAmEe0MFkzRDthqHizkY7FVKll46w4SKySlWJvDIyzhP8khmS1/B/i9+6ElY3BsW5gen5y4mGHx5mV7y/yScfHBmEPhXTd8UGWbGMrt8BZfITFUgpRn3LcgaRRU9mJfjISTBNbwKaUQPdAlmDNG6gP4Prs0BwhKYnnqki5MHH3d8A5DDJ51UaXPwUrl8ZGLtqw3nOzXwsi44EkFIe1h+EpWK/K90LXj6QCgkJN3ay2YbcICGKA9ropoYPnBylZUsE/jV8uihj7TsmtUt22qMHuvj00SiLZeJhlx43Adp1zgYdeoVg5K46BgbsNsBrB4TkIgU= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(18002099003)(56012099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?KiQEeJlKnbVpDE1K6fO1nK7OqrAOmYrMYVz4m+Sm167DLajAkFBoHiZWFAb4?= =?us-ascii?Q?U1dv28NxmE1v/dDe1T0/QnEbjHiQxd/IQNgf+M6iB+/r7OPJxQahTq7FMnuC?= =?us-ascii?Q?gL79+VDkrxS/vEhH3MNkYt8kYP9EY9ESfVfq0gyYOdRyNu748WFi1WE6HyyY?= =?us-ascii?Q?q/RF7Zn0JJy2v81IOkkawdv5J4KQsnJEN0iuK7mxr6Jj8wBM5snMF55aTtHY?= =?us-ascii?Q?7HgFelzR49eQkER8a7OuuEm3GW8upOasXvEde6fnMkWEPAx/CwZood1VmqlM?= =?us-ascii?Q?HZ/GfijxcqPL+A7bVa9SGIM3nZVVy1yaIEtYreJ8Nt+a/iPdeekcFf9oX6j3?= =?us-ascii?Q?nYzC3SBlDHvKt7l/H9VQJEbqMflMNPakw47n/md0wnRaoK8Gl78bQmhpyJdm?= =?us-ascii?Q?XlLEP2ou57DNQ6NCdp7b2Yfedwn4TcYDlhWMgf8jSmfW1roaF1Wi02/xiaBU?= =?us-ascii?Q?X3xaM3DvpOJDRvD+3t9TLKqfT5OPDBfNISuiKzxp8TPqUCW2qbRbOAK49k8z?= =?us-ascii?Q?nSQE32KgHR07VaQd48l26HAZuEPaBELyxqciLlXSbL8QaIOAw81zbEAmiKE8?= =?us-ascii?Q?1x9b+2gO9hW0D94ZXsHkJNbmRBPX4SH7vp56617OfJOxZM4B/e1d8HFpPWZ9?= =?us-ascii?Q?5lTqhH7c7s16DDmM5Zr8k81XFL+XT3FpvcFrUgyfphL+Z0LPfTzZqCvM3Jlc?= =?us-ascii?Q?klK+jDgHLQhEH+9ychE4pIc88fnqSnH8dYpqEzcRlsPs3Ol4qphpCbhEnc8z?= =?us-ascii?Q?oylDL3Mi4thnrolG8dMXkGXAW2GpxEF7sej6BS7DoFNjzhIQJn+Y66lQJSfF?= =?us-ascii?Q?GE4VLiqWE0nE6zCRY6ltGRXFTJCswbf97pWKYv+AjeiY2cl3TZAidNSCpVkF?= =?us-ascii?Q?T4KfkMCzxmy+0317Jw73qT7tLd2My+7V9psjjs7hTtz2AkVtsfLE//o6no72?= =?us-ascii?Q?ukxU6gO8QttupbD23WXXQjgpSz1XRHc27hzDZ4InjIsFekre+zGVSDIvQCAt?= =?us-ascii?Q?ySBt0NMIm5J+WVAPRuOoNiiIUmv7ng+uljxYqNYKiMR/GgYeToQT+vwg9IDL?= =?us-ascii?Q?kRFDq1+nnmK+LGawyDTf6pWgnT1+dh9NViBJXbbvyHZZntmR1Usw9nxW7Joa?= =?us-ascii?Q?opB3If3jwYalLh5d4Ta2U26Nys5f2DnNvNebeS/RlFmZIQd5LxZ5HKLe8hxJ?= =?us-ascii?Q?UDAIQlS62YjQTFSt0yvcKIpmg9mF1l0AxUGX7vykkJAaPyZtK0SKxEOEsa5j?= =?us-ascii?Q?PmRCBF6LEW030A7vFV5ANxHYrSQReY1IPVPrh2wQH20gURvib4yOCtvp1hp4?= =?us-ascii?Q?Ye86IZV4H0Jxi8ovvwOd5qdIFVB9PKbz4p0B1eelAup6Ej3aYpY8wz55NwdS?= =?us-ascii?Q?Fd/9XFh92JEiLrHlIEOuhQGTjOgpTJxbHShOeKh/GSwwlnfT0w+1SiL9rLXM?= =?us-ascii?Q?2uuq0R8ul/4Tujbc5R4N8XIadvUgOrQjygmVpvezm0r8enDJtXncKDwwLSbx?= =?us-ascii?Q?K/OrGtVcc6r2v5Bin+wYawsk/UZTVEeszL6Qg79Alkf0m0zSf/Y+jwdeKvy+?= =?us-ascii?Q?NTixRmLQTTCnJ8PH6eohOtSvAju41FlPVkwNXsWnzelj9ylzOOoSHSchn/GD?= =?us-ascii?Q?6JT2d0HNJYXiB8Oi6MOUWLGAXq0WWv/KY8as/NI2i9/BI0lf0bg1lji24aKi?= =?us-ascii?Q?FofGd5Yq1QlnuMHIXKJBvf5JW3XH3m3AAiqkdiY9KGnCGrcu3YLSvWbxsu+x?= =?us-ascii?Q?Vi0qtDKlXA=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1dbd2b51-7508-4299-0ff1-08de8e81e1da X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 30 Mar 2026 17:29:22.6894 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: b6z2qI2wfikhRC8y3EEdjJKTYA4x9utJZBx7tOILCGmhmXVxZgm3AdvCsXLL0HG+vXwUwMLyZMNfPhsOIr8ucA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5854 On Fri, Mar 27, 2026 at 05:04:23PM +0530, K Prateek Nayak wrote: > Hello Andrea, > > On 3/27/2026 3:14 PM, Andrea Righi wrote: > > Hi Vincent, > > > > On Fri, Mar 27, 2026 at 09:45:56AM +0100, Vincent Guittot wrote: > >> On Thu, 26 Mar 2026 at 16:12, Andrea Righi wrote: > >>> > >>> When choosing which idle housekeeping CPU runs the idle load balancer, > >>> prefer one on a fully idle core if SMT is active, so balance can migrate > >>> work onto a CPU that still offers full effective capacity. Fall back to > >>> any idle candidate if none qualify. > >> > >> This one isn't straightforward for me. The ilb cpu will check all > >> other idle CPUs 1st and finish with itself so unless the next CPU in > >> the idle_cpus_mask is a sibling, this should not make a difference > >> > >> Did you see any perf diff ? > > > > I actually see a benefit, in particular, with the first patch applied I see > > a ~1.76x speedup, if I add this on top I get ~1.9x speedup vs baseline, > > which seems pretty consistent across runs (definitely not in error range). > > > > The intention with this change was to minimize SMT noise running the ILB > > code on a fully-idle core when possible, but I also didn't expect to see > > such big difference. > > > > I'll investigate more to better understand what's happening. > > Interesting! Either this "CPU-intensive workload" hates SMT turning > busy (but to an extent where performance drops visibly?) or ILB > keeps getting interrupted on an SMT sibling that is burdened by > interrupts leading to slower balance (or IRQs driving the workload > being delayed by rq_lock disabling them) Alright, I dug a bit deeper into what's going on. In this case, the workload showing the large benefit (the NVBLAS benchmark) is running exactly one task per SMT core, all pinned to NUMA node 0. The system has two nodes, so node 1 remains mostly idle. With the SMT-aware select_idle_capacity(), tasks get distributed across SMT cores in a way that avoids placing them on busy siblings, which is nice and it's the part that gives most of the speedup. However, without this ILB patch, find_new_ilb() always picks a CPU with a busy sibling on node 0, because for_each_cpu_and() always starts from the lower CPU IDs. As a result, the ILB always ends up running on CPUs with a CPU-intensive worker running on its sibling, disrupting each other's performance. As an experiment, I tried something silly like the following, biasing the ILB selection toward node 1 (node0 = 0-87,176-263, node1 = 88-177,264-351): struct cpumask tmp; cpumask_and(&tmp, nohz.idle_cpus_mask, hk_mask); for_each_cpu_wrap(ilb_cpu, &tmp, nr_cpu_ids / 4) { if (ilb_cpu == smp_processor_id()) continue; if (idle_cpu(ilb_cpu)) return ilb_cpu; } And I get pretty much the same speedup (slighly better actually, because I always get an idle CPU in one step, since node 1 is always idle with this particular benchmark). So, in this particular scenario this patch makes sense, because we avoid the "SMT contention" at very low cost. In general, I think the benefit can be quite situational. I could still make sense to have it, the extra overhead is limited to an additional is_core_idle() check over idle & HK candidates (worst case), which could be worthwhile if it reduces interference from busy SMT siblings. What do you think? Thanks, -Andrea