From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from BN1PR04CU002.outbound.protection.outlook.com (mail-eastus2azon11010031.outbound.protection.outlook.com [52.101.56.31]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B339B429813 for ; Tue, 28 Apr 2026 14:44:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.56.31 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777387475; cv=fail; b=Z8mUqr87+ahkvl1i6TWr659B2lRYDB19wOEAK/2rf2kSy+GyC/XXSRz6cY4NzXo64Sv9gRLO8EUePhj7GGkEvvdnahRmfz306HIEZGgLRfcwoOGukZG0WHt+isrfZaj5YL+yl0wVChoGtHL8sVN/x9c+Pwu2FlHvFhIQlcrVeik= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777387475; c=relaxed/simple; bh=yvWgrlQVyKhrRl8iQ1xi9cY2xCWHgh+IyXmh8Y4CPmc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=ruNDCef9NMdF4DoyeWcwy6G77ABVXxEmudoYjnMLIz/NnNyEBS0080jmxecwTS2GuJ6Ow5mb4vQYzw9lBgQ+lxkE8r0NRtWI8yhhLOObId229k+gm7qHIISHIAHU1XWYTE1rTehue5mSh4TIL1X2fQ/ZTwSaq2+4PUrmF4liEJk= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=CbRPi/X8; arc=fail smtp.client-ip=52.101.56.31 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="CbRPi/X8" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=mq1l1DKEYDj3VScQJgdTU8PP1R7LXfM3kD1DtSEmXqdXlqp+9SEDYTP1AIt7GQZZZ9r520/dcfLV3yXlFKt/Q9epjHf0SlUtPTEofZGGLYT5PIee+ibbE4t/MtyNPwrTzM9M6OqpAnp+FK0S8g7Bn55WiBltAnSWmpU96MOJP6DVFaHJRXlhHmAjINUAk7s5XrsbQ4eWBsfdql5xMNOi9jUVImgz++/guWxcUmd8pYLBJbjH4E2jrRIB/lpW1Hr7bOrWq50ZfMXc0NSALdfzssry6HXLyK6FhkOViaNQiK1bn24gEud6RVS9Q0EgnOS6T2rcHSeShmI6UOsDZss0lQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jcwVjI1k39DqCy+S3uKGE2jAi9UQYvQaR8gKqZ2cH4A=; b=rxVshoVYa932818mQuY1JeYUQuXmIxdGRYU/Pt8gwhcEj8rDsWy1dbV5F49saljBiGDd3ED7TNkl3meIsMpDOsom8UVLDbSQRlDwEiUjPat96HD5WWXHweY/VFBnJiE7KYUbThhk6q3BfTbYMRLAPyMBp4oP1NN8zgxTcmFaX3ZSiTzSchULBSJoePKLNQEddeBlbKp/4e8+cBTsS4v0vciyz3FAn21uVNeq5xP7TCTrK8XdCMSq9JcKAqZxyuaHdr0A+tqRDHxx8C6/160CTVSnsX+z4nD8lZmrBMDWCLYFJw5f+NK5djZ9DR9YhNSWrbbepCzla9JxiCh4OwWopg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jcwVjI1k39DqCy+S3uKGE2jAi9UQYvQaR8gKqZ2cH4A=; b=CbRPi/X8zHU4+gS6QS+AiBC/iO1E6ZgfzxGaq93CZ+u9VmEyH3lCLUHjGRuW5+NvSp0Jp2x6HbWRBYE6rrtPDHWACwd1uelG6yyXZlnVclt8pbHxtflKi0tCBEKi+SqgIcY163B1ROs9U3dGufG/9MPV3RX8oo47gl7Q5SdEuW5j3StNz6bWSa9Ls1JJb04H76hX6y+KSmfethAXg2i5Bx//XOu0FAhnSVtKDSVAyOSugQNX3jPufLww2LO0MhpmWRsT4UHLL8HFcXvG9XCU1x5E8bHse/SvbgqiPn/J9R6Zrc8mmJAZ3DGtYM1paRBlikSbpFZNc0yruYMQXyvu/Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by IA1PR12MB6211.namprd12.prod.outlook.com (2603:10b6:208:3e5::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9870.15; Tue, 28 Apr 2026 14:44:27 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9870.013; Tue, 28 Apr 2026 14:44:27 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 3/5] sched/fair: Prefer fully-idle SMT cores in asym-capacity idle selection Date: Tue, 28 Apr 2026 16:41:09 +0200 Message-ID: <20260428144352.3575863-4-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260428144352.3575863-1-arighi@nvidia.com> References: <20260428144352.3575863-1-arighi@nvidia.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: ZR0P278CA0198.CHEP278.PROD.OUTLOOK.COM (2603:10a6:910:44::12) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|IA1PR12MB6211:EE_ X-MS-Office365-Filtering-Correlation-Id: 4f781f1c-230e-497b-50c3-08dea534a5c5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|1800799024|376014|366016|18002099003|22082099003|56012099003; X-Microsoft-Antispam-Message-Info: tQaWww0TPiGnlbHzYFL7IuO+3vLoG0xUzZN4nZ1OHiuw8XRs2fGtibrAtvB4lOGXz1Rb9kXIzML/6dRPZZCmnTqFqc2A1jy5Pvn5dy8u7LB2R5uFUf+En8btEYnvafRSFUmlWNTtroQPRvEh6IDCVFom19AyHHNb18+M9XaXF/tn2g+8x8v9Wve5GQGdIZYRERJe0nhuE/E73llTYoVIw/J+P+rtFOCZbhjL0ToVwpPHr6Zl6uM+XvMUgiRZQ3u18h3N9oSgz2MPkSJbMRE8375mejwOWDwAF+a/Aja70bLZ4WGswwsRBKrFieIedF8reTOu6VZq7ty9XiQGMK5zV/f94iDXp1jBAc3c+HsuWHX7xuEj4s/6MEORHKtSVe8YBYJJ9cuLxMhDdJGnz7TYxR4fghFEsb4tMoFzVLGqQgAn2iurTKogtxHL/oGgD2JYWj6JhRUSYd/BDLT9zgF8/s7U0qmdPByUhkaTrxEKC7u7AWpRcjDOw9LZiXXpHOcLZ9y3SyfYdqFHGT1G2UvRMpWh7VXYkxfoIPpNHAlF0fZVIpJ5PMm33QEoYUVU6ZEA8aZwqL9dB42yM3LHJfDeKk8Yk2BxwDBSqagY2gYJl+idFOC8z3lwrriJqqFUCPz6mwUgqqfocXpyzh1GJJ0Hg5PhmBonYH/ncDzSn9j+uLZt2w/8fMLyftdrwmQOAMwM58oEaPyFIFG9QuCxrfOodsOC3wUZRD2nVTDZWXMqNWo= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(1800799024)(376014)(366016)(18002099003)(22082099003)(56012099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?0CvaYzqWb9+pew7uhz8AwOmWwMnMufxQMLyUBdF+4xLMsRKfA5NKJhqo/H3T?= =?us-ascii?Q?O9RTC6rajWvTDM3g/TzC3tXvOugTTHL7n70YKkIw1tbFpnqtpdb3ph/pmHtY?= =?us-ascii?Q?rsk013Z9WDxrZzishmMw3rtjbd1fzUd6nyW0p8FIqHyMpmNqSaBY/HIBW2nI?= =?us-ascii?Q?4Wzx48bEv3RZt5yB7GteMcp6/mXpZclHEMHtGHSvKMGYA0uVBXqEd9TO8swW?= =?us-ascii?Q?fND38++Jg3UIWz3ZERu1oGyGPY7lqxjRCwwLa+HUMm4zL+aoLoKwBmoVye8I?= =?us-ascii?Q?3pBqAYb/75oqSh6srJOQq55LwHkPkXEARb64qtep5xa2q86BecQxhQp+SBr2?= =?us-ascii?Q?QUYayDvYM7221uvMj6m9bmyHutZj4PVmQCJZ84s7bE46BsWt6ARZ5TodK23M?= =?us-ascii?Q?5o1DVKD417bmKmV/gH5gcyYxKH0nWJbjFOWBMNyv1dBVUn3hhiQ4VlH581/m?= =?us-ascii?Q?lIDtTFVK9T8TjVi4V4UDEcx0rZBYsfwTIRDh8o+tyApLVsP1bpbPBU/l+fQb?= =?us-ascii?Q?i8JPPdRecbSaHS4P31ncJHKDMhB69a/dTc2tQRsj5g4HJK9KYdWi2LVc6iWF?= =?us-ascii?Q?JGwc/X+4KDMnGADMNaF5egZTJRl3boAi/0owfGgLb8EebL7H790T6mASO9xn?= =?us-ascii?Q?MCncOqHGuXxlOncNGIF17rQ2XTNBJoWHl7mkZ7DP3t8LwO6I+dhuzD54dkpN?= =?us-ascii?Q?7bnvpzui3Ga9bfjYfUmO4WMIGodKSk8BJj+KkyixJz37LKQq3WdDyRJvhUHG?= =?us-ascii?Q?U2j8IiJGnUykaGgC9Hf/oVIz3/7nnJvuT/0KtHsvIpTAUcOT1MfJi/S8Ooxx?= =?us-ascii?Q?Z9NH6g/QNM8tFMvBEAVWAqAiaOwrVuGTDf5LLahBaxLUi27ZXcMJ7w/9JRun?= =?us-ascii?Q?4Yop0SREuziHGcsvLEckTFEV8nhNhllOuB1sKaO1Z4Lu5+QOY9HPrsHswR/u?= =?us-ascii?Q?zspZOVaRhtcOhsTvMtZC7YrLXKWJ19+XOKUi6VC+lULOCXWWSs5V5A93JXd3?= =?us-ascii?Q?4S74hTYiUsn8HBAOjly2/ffnasMOx4MIb7Pe7Cat8XElwyjvRxwR1AXRRou+?= =?us-ascii?Q?MXeZIxge9EgBNGrEIctoem7itctShT9TfXNDjKeNpUTHGxY7SeW3zV+eTjY5?= =?us-ascii?Q?sAR84hvIJDwb49DfzcKlvXaiuHqY80muUIL9NrcW0T9U/GizUGF1zX239mOw?= =?us-ascii?Q?4jGers5IYN5IAJPCR/u9ELvt+N6od8NvRcEggJxj7+jygCbZoc4EsR1QehBf?= =?us-ascii?Q?jH1hDlX6l9W9rwnoXHRhrVv79Xt7tRkzNmHzSjzyO5r4Lro2ow1h3/tNLjnK?= =?us-ascii?Q?FvNJACl1qhphm2kZV9cwv/hpKWcxwT6LcG3ncdYNXxbo7d3ReNf3/5fNDDy3?= =?us-ascii?Q?uF4wf2UtMwLKqWGZv8FJPyu3C0WofeGcVF/nUkRJfmJwXuqLxRMxBw5JVuJX?= =?us-ascii?Q?ziZzK8/fHXR5nMMeghth9q82pA7L6+PG1iobuBjYj/ozIarW02h2Z4LBtlBH?= =?us-ascii?Q?bObeMy6dZzLjFzoYTWVV+in3uhXErINJyM9x3tYEG9V4s1a2CnLuyxdEQyJg?= =?us-ascii?Q?CB+Mo70ElgiM5Pa5uOjTPybJW64+N5kGCOF1TujdB2oVKOsgqqY7RTN+rDvR?= =?us-ascii?Q?xxtwTOrtvQGWCzk5mAST64H9BPtnCViEMReDOojhnkSwNmTowKEXIf2m6Tgt?= =?us-ascii?Q?toAOVuNExdZ9zQCIiSvB2dpOwuji6jVvpONL9Kmj3T4mRwr5hMazF3j+mJ+X?= =?us-ascii?Q?8EI+P0qgMA=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4f781f1c-230e-497b-50c3-08dea534a5c5 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Apr 2026 14:44:27.3593 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ukk+2g61hWnnoA/7JnPzY5WjziutvRHKbqlkZrbmmlOgQQT67mjtpZr7uB/Lb94QmVEQ73jsENexQroepLT47g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR12MB6211 On systems with asymmetric CPU capacity (e.g., ACPI/CPPC reporting different per-core frequencies), the wakeup path uses select_idle_capacity() and prioritizes idle CPUs with higher capacity for better task placement. However, when those CPUs belong to SMT cores, their effective capacity can be much lower than the nominal capacity when the sibling thread is busy: SMT siblings compete for shared resources, so a "high capacity" CPU that is idle but whose sibling is busy does not deliver its full capacity. This effective capacity reduction cannot be modeled by the static capacity value alone. Introduce SMT awareness in the asym-capacity idle selection policy: when SMT is active, always prefer fully-idle SMT cores over partially-idle ones. Prioritizing fully-idle SMT cores yields better task placement because the effective capacity of partially-idle SMT cores is reduced; always preferring them when available leads to more accurate capacity usage on task wakeup. On an SMT system with asymmetric CPU capacities, SMT-aware idle selection has been shown to improve throughput by around 15-18% for CPU-bound workloads, running an amount of tasks equal to the amount of SMT cores. Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Christian Loehle Cc: Koba Ko Reviewed-by: K Prateek Nayak Reported-by: Felix Abecassis Signed-off-by: Andrea Righi --- kernel/sched/fair.c | 70 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 65 insertions(+), 5 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index bbdf537f61154..6a7e4943804b5 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7989,6 +7989,22 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool return idle_cpu; } +/* + * Idle-capacity scan ranks transformed util_fits_cpu() outcomes; lower values + * are more preferred (see select_idle_capacity()). + */ +enum asym_fits_state { + /* In descending order of preference */ + ASYM_IDLE_CORE_UCLAMP_MISFIT = -4, + ASYM_IDLE_CORE_COMPLETE_MISFIT, + ASYM_IDLE_THREAD_FITS, + ASYM_IDLE_THREAD_UCLAMP_MISFIT, + ASYM_IDLE_COMPLETE_MISFIT, + + /* util_fits_cpu() bias for an idle core. */ + ASYM_IDLE_CORE_BIAS = -3, +}; + /* * Scan the asym_capacity domain for idle CPUs; pick the first idle one on which * the task fits. If no CPU is big enough, but there are idle ones, try to @@ -7997,8 +8013,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool static int select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) { + bool prefers_idle_core = sched_smt_active() && test_idle_cores(target); unsigned long task_util, util_min, util_max, best_cap = 0; - int fits, best_fits = 0; + int fits, best_fits = ASYM_IDLE_COMPLETE_MISFIT; int cpu, best_cpu = -1; struct cpumask *cpus; @@ -8010,6 +8027,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) util_max = uclamp_eff_value(p, UCLAMP_MAX); for_each_cpu_wrap(cpu, cpus, target) { + bool preferred_core = !prefers_idle_core || is_core_idle(cpu); unsigned long cpu_cap = capacity_of(cpu); if (!choose_idle_cpu(cpu, p)) @@ -8018,7 +8036,7 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) fits = util_fits_cpu(task_util, util_min, util_max, cpu); /* This CPU fits with all requirements */ - if (fits > 0) + if (fits > 0 && preferred_core) return cpu; /* * Only the min performance hint (i.e. uclamp_min) doesn't fit. @@ -8026,9 +8044,33 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) */ else if (fits < 0) cpu_cap = get_actual_cpu_capacity(cpu); + /* + * fits > 0 implies we are not on a preferred core + * but the util fits CPU capacity. Set fits to ASYM_IDLE_THREAD_FITS + * so the effective range becomes + * [ASYM_IDLE_THREAD_FITS, ASYM_IDLE_COMPLETE_MISFIT], where: + * ASYM_IDLE_COMPLETE_MISFIT - does not fit + * ASYM_IDLE_THREAD_UCLAMP_MISFIT - fits with the exception of UCLAMP_MIN + * ASYM_IDLE_THREAD_FITS - fits with the exception of preferred_core + */ + else if (fits > 0) + fits = ASYM_IDLE_THREAD_FITS; + + /* + * If we are on a preferred core, translate the range of fits + * of [ASYM_IDLE_THREAD_UCLAMP_MISFIT, ASYM_IDLE_COMPLETE_MISFIT] to + * [ASYM_IDLE_CORE_UCLAMP_MISFIT, ASYM_IDLE_CORE_COMPLETE_MISFIT]. + * This ensures that an idle core is always given priority over + * (partially) busy core. + * + * A fully fitting idle core would have returned early and hence + * fits > 0 for preferred_core need not be dealt with. + */ + if (preferred_core) + fits += ASYM_IDLE_CORE_BIAS; /* - * First, select CPU which fits better (-1 being better than 0). + * First, select CPU which fits better (lower is more preferred). * Then, select the one with best capacity at same level. */ if ((fits < best_fits) || @@ -8039,6 +8081,19 @@ select_idle_capacity(struct task_struct *p, struct sched_domain *sd, int target) } } + /* + * A value in the [ASYM_IDLE_CORE_UCLAMP_MISFIT, ASYM_IDLE_CORE_BIAS] + * range means the chosen CPU is in a fully idle SMT core. Values above + * ASYM_IDLE_CORE_BIAS mean we never ranked such a CPU best. + * + * The asym-capacity wakeup path returns from select_idle_sibling() + * after this function and never runs select_idle_cpu(), so the usual + * select_idle_cpu() tail that clears idle cores must live here when the + * idle-core preference did not win. + */ + if (prefers_idle_core && best_fits > ASYM_IDLE_CORE_BIAS) + set_idle_cores(target, false); + return best_cpu; } @@ -8047,12 +8102,17 @@ static inline bool asym_fits_cpu(unsigned long util, unsigned long util_max, int cpu) { - if (sched_asym_cpucap_active()) + if (sched_asym_cpucap_active()) { /* * Return true only if the cpu fully fits the task requirements * which include the utilization and the performance hints. + * + * When SMT is active, also require that the core has no busy + * siblings. */ - return (util_fits_cpu(util, util_min, util_max, cpu) > 0); + return (!sched_smt_active() || is_core_idle(cpu)) && + (util_fits_cpu(util, util_min, util_max, cpu) > 0); + } return true; } -- 2.54.0