From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from MW6PR02CU001.outbound.protection.outlook.com (mail-westus2azon11012011.outbound.protection.outlook.com [52.101.48.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6872F30FC23 for ; Thu, 23 Apr 2026 07:41:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=52.101.48.11 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776930118; cv=fail; b=JKlIRXbRQNN8kgj+rrhBIeK8yCpjx2gtq8F19jr6iPNrdR+WxmB72+Q4ByXiRCuwOXYQREKgIlbIvI0GE0lOwXTWFwbquZOfMcaM20LgB8slC7HfTfdWpd+Ih56P0DmxbR6zi63/G+XTfdRU6gQbyMSAOCuLwVsn1oSgETM24Tc= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1776930118; c=relaxed/simple; bh=Wxs7YN+UKH3ZCiiquU+sxRAgTFXmhOwMZNbeojNZaXU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: Content-Type:MIME-Version; b=Z9Ex/Y7ZZR2dMoXhzImOQ5NV/wU5IX5A7YJNS64q13QudzVROgVUXDFN3szBXAyKRSR2655PMurMlQyXXLaxM9t2pfJfx/tfPPkhIyIkSIdzyOsMw4UxgjdmYgyDEtZ9jPYZORqgX3bMwySgZSbjidewpCrsrRFxUXorhv8HUN0= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com; spf=fail smtp.mailfrom=nvidia.com; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b=bMgia/Qx; arc=fail smtp.client-ip=52.101.48.11 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=nvidia.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=nvidia.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=Nvidia.com header.i=@Nvidia.com header.b="bMgia/Qx" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uRlFpkzCABlZ2Cwn7M8l/ZNPZLIzJjE25vdMKDbDX+8v1JOz9IwOQqqUdKei3yaUjB00YGq5y72d8L7CFMvAO21ZEzVTeNtkClpnb5n/laSCSb4cS7IwwmHh0yECLL1UgEYVcUWCv0fLehOn6m8/HwYGzwDK2ouPkBRPmfBMZ8CAk4vZJ7f1uX8dOzRNmScH3cCjOg6ghp+durKjj1xp4l1Zyf2CpofM6x+K69JeJOU8/Cwqu8UpuU0rZX2easCV/un/T6gS8Si+wf84lxhN0MlDRM9Y5Nrvtjx9qwrZol2qcVNxI9DNvdhWT/fswNEV1/iEy4eVvqQmPPgCgMHNsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xIzr8bdkeS8xgPpaQMvzqlkHXpIMNq6LXkA342cv/LQ=; b=thHDBSNgIVHD0SzKLpdTFibK8zIdy2Ywi4F9wUdzmnfRYC4YH3gBL/29188/PTVnvbW7pJuAnBG+S8r6hWnAJaBbGnHvmHXbNxAmTn5jikumYtb1QuyLicU9ABtl/Zl5NkQJHD1ONXtRJABx/cSqtinYJK8TBM+iRVRlkABnkt3VzX2FQZTxkmyf2vTF8cpFi0elvaFRx/z4YAZo4ueJ+fILv91NzVR1/kAdva4K2ZR9OQ8O32xsu9Q3ch48M/5bphXjbntbOiEudw4iuZEI4prjw7cX68+XiH2hcxBm10auo/WCV3Xvblq9d1ojC3uNZmKyvIq/qwhG5QGGavoODw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xIzr8bdkeS8xgPpaQMvzqlkHXpIMNq6LXkA342cv/LQ=; b=bMgia/QxDVppJeWxKCYfOXh/ZrLS3vDT7N7MoMTXq/vcNSHYerxDzMUpatlPEW7C/QF7m6L6eqx4Uzo5HZjapZdgrw/QvnJdrazrIYJa0NnWsM7fZTAc7KdZcs7P8Tud4C2GPVfYK+42fNNPCqYUW73xLowpbCi4yx2AugqB9O1GinyjKDEYrfi/gWCRDkABsYVnOvzQTpYDbSwnlq1t7TZJnce1NJL3DLFKZWTYj42jqxS+TZoDVNvKjtrdwu8o6uiRUAAAJaJAGMgeAl6UJBY6Ggcx0wDtFukV72caOQjXDKyCEURuDGWqhSXki0ddJTx1aKGz5jrkwVhmNghFUw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) by BL4PR12MB9534.namprd12.prod.outlook.com (2603:10b6:208:58f::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.21; Thu, 23 Apr 2026 07:41:53 +0000 Received: from LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528]) by LV8PR12MB9620.namprd12.prod.outlook.com ([fe80::299d:f5e0:3550:1528%5]) with mapi id 15.20.9846.019; Thu, 23 Apr 2026 07:41:53 +0000 From: Andrea Righi To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot Cc: Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , linux-kernel@vger.kernel.org Subject: [PATCH 1/5] sched/fair: Attach sched_domain_shared to sd_asym_cpucapacity Date: Thu, 23 Apr 2026 09:36:45 +0200 Message-ID: <20260423074135.380390-2-arighi@nvidia.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260423074135.380390-1-arighi@nvidia.com> References: <20260423074135.380390-1-arighi@nvidia.com> Content-Transfer-Encoding: 8bit Content-Type: text/plain X-ClientProxiedBy: BYAPR01CA0051.prod.exchangelabs.com (2603:10b6:a03:94::28) To LV8PR12MB9620.namprd12.prod.outlook.com (2603:10b6:408:2a1::19) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV8PR12MB9620:EE_|BL4PR12MB9534:EE_ X-MS-Office365-Filtering-Correlation-Id: 554394ec-b9f1-4a35-a7d3-08dea10bc956 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|376014|7416014|366016|56012099003|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: mi49edeLtpzgAIuz3MfrolZbPNqUKfc9ToC2jTzYRi5Gb1IfCGsg3zI2CYfpXwMux1yt45sQUvCVtF+q+/EdH4Cy2UE0bg13z8/XAt589cOKFsWrJGy9cDKq6ZPr3J/i7PR3pk5/rI25xqUKLO6kOD+74NZOy+I0OVIEGFHMYw6WPifsMeZtu2IxtB3tOXpECQJDWKFKUYgQ9M0dLv2eoM9Be7tErTDjN8034cJLs0PGbvEV2OG+glqI+3fQYkJzUnGZp/CKWFskXbS4Ot3cDOwoABYVQD/tceBk/NvboyxbK922hRZfgl1q3euO4Yj/VWz3RNAclpY/g66PakUTPfbED1CNmCr/7qlIzEP1bGZE5gC+sOtJgmNwMOJJxGepPzYH3knybqXwive+S/v/O4Ws/zTnoU41yUGt9d7AxvfAravsTXvymZrCyTuMT8saHMGipD5wES5kxPSXUZe3MJ278MGqJ0dEw5JZZR9h/jgnH71kqiNdplw0r6Bz7lLsAdHaDo8Q5PRSYTsJWXSaaJgLe92ijgZ1sEOZUpXyGIzknXxrmt5enat1l9Ue16bnLp3znUKmDvO0+gNEYMb6LsCtifkajs5V8Fb2CXpdo7kNKEhxqrEukIBjki2VoDrpYpgZe+UjHjuYnfOwgfVxt+/9yJ9MLxta5G2HcAKroCl6DqcsoWGnZdpKRY/IGqdkr7K8Vm3eORBtGNXTfzXohHGHVuRjOvop9KYBZxcOWXI= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV8PR12MB9620.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(1800799024)(376014)(7416014)(366016)(56012099003)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?p7jJGNjnZhoOKsgIhJslXgwvk4G6GjKnml9Khlwq34MRmAcQ2H3zfQG8X8Kj?= =?us-ascii?Q?eqEh42qlgd6G6+StKy/lQKEKms7mWm3Yn5S6EQlrXCm1L+N/h3NHb2Jjw4b1?= =?us-ascii?Q?m+KPvrmxydSm4NVLQ4a8j2Qc0jnGbYC32+enA6KP6Fxp2Ul9xFOQlpyVYQvs?= =?us-ascii?Q?GpyIdLIxTOnTjmHUkyeC/yfbT2QvIIS8u4Qu5E/AJEl0JJsT50IH2dy7RSSv?= =?us-ascii?Q?L6HSEflFiFpg/om+12BpJEWXx6QvXKSqgyzzb0laRHmQlt5+ehIjXHllFLnu?= =?us-ascii?Q?FrSaFZZILTrfP1q3/bSVa9pGFyVfNBcGjgKoT+oBpwYIbh8Ty+1P0BrBSdYN?= =?us-ascii?Q?nnloB/KewsCrC0VNObAcqEwTiT3AFkCIZiN/mh9a71FPZIDr2pzGbHzg8H5t?= =?us-ascii?Q?JRR5xqvXlokIXDy3Fiz2gqHC5C5y68bsQmDg4eP2ovgpbpfxdBl/TCNTvCgt?= =?us-ascii?Q?SQhiAsAVhVjaHaWbHttEnEX0izXu1zBQfZy9tO+T3Hargy6R46ngVLoCmdxJ?= =?us-ascii?Q?fxoVlxO0YGZ7+6ezz02FfCi1GF8iF1eXeQr92MzwiMy1WvmXlZTuOR6gVaYp?= =?us-ascii?Q?1p0SaL3/E7F/Tuxo0a9oSy1b1VVCqwps+5z0U+AyXVYTZJZNOagj6dsfOjz3?= =?us-ascii?Q?M7fud4mxefqaM1EOswnZVm66h+NyO+KjEw6VP5Y6O6nQ+9UccochOK3ywtLa?= =?us-ascii?Q?zNgD+v5j88cPYaKSuB5NxINoLO7LN2IXUS0J+FMx6/tI3O2uVYYIVzvJDT93?= =?us-ascii?Q?qZ8ivWBnEuYJQJtipoDPmA1el67qMUAUUYHkKJfqWb8/uV2Psr1d8S0jLXm3?= =?us-ascii?Q?CKtHT1hpH5WDKhaUO/3xkPftE1XphKDIVnF+YPenn8v2+YIzYcdwJnFoce1y?= =?us-ascii?Q?a4XhC4sI355yMx/l/Yq307GZFXGjZkiJOhT1YN3iXvU5LuXBkw2+M6jldUyY?= =?us-ascii?Q?9h3JpJ2cTOFUdo4JXV9EzU/+oRIWAExVVWF0TN20eOF3Flm6PSP3i+ISUYE6?= =?us-ascii?Q?mc15ZuKyuMdDB1dN9XjH5qYtFFqS9+0c4X/sAwl/r7wRaUH7+de1v5vIjETI?= =?us-ascii?Q?27bC4gYrB5PjKUrpZsCF+UZ7p2s/K25lP6aNc2aEZUVJKMNV0EoCK44+6rQK?= =?us-ascii?Q?4HrphaPPeSs5dTekYQcNri+RWqEjvIf71WD6bxgKT90NQWhuhoNo4C1aquSB?= =?us-ascii?Q?oqabgBJ86reuxrWFHrCmRGtdska/4WkoHtTqEyEle5Nukagt+csMJMbc1AVk?= =?us-ascii?Q?jMLRJC/N2OTTcB+SiMLapajWeU1Aqx0AyDR9Ao1oHuB8828ZEkeswDab3UWO?= =?us-ascii?Q?H/PIpqbQ+9wWzXk/UAEco3vxb8Rdu1vLslgHVqIqj+fEsbpJaUJPflOhFBQ1?= =?us-ascii?Q?0u6JNYmg7pc38V4efeBlsEyRgo5YH7cU8vGb3ivEknL/cTBgB7SqjXJSbAci?= =?us-ascii?Q?iBUkF2XvxhbPL7R56sLfC5zm3bjQabM79wv95ukOB4fdp39Ogf7gpkhfchyZ?= =?us-ascii?Q?qowFYxurR64ZeqZoskKYNbsJ8P2kLCOG7BcmvmCRxeeBghfPUjLjEcvpuWUd?= =?us-ascii?Q?+EdNUJUT3S4DirQVtzwL+YE0o9BGe/d97x6Rra3BYzAEVKvT2xdfZCHBfH84?= =?us-ascii?Q?XoxGprkiZprmAyVRPJPE2S3/GHuhP5oLycceEnQirLW933js4HI2NifRKMP4?= =?us-ascii?Q?zMm/zinUKLpJ0fyC8rv3rH7bNm4oxgY54vu9AjY5pufPiuueJv9wu3MgwZ9U?= =?us-ascii?Q?3o7DLDbgyA=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 554394ec-b9f1-4a35-a7d3-08dea10bc956 X-MS-Exchange-CrossTenant-AuthSource: LV8PR12MB9620.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 Apr 2026 07:41:52.9299 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: p5Q2A1pTyxDaGkIlJadI1gMAqlL8jdKtV9xnyv/j5MO6jkXkxL6d1+fscx3vOcXQDz4qR8kJzT6aw4sSdUBB9g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL4PR12MB9534 From: K Prateek Nayak On asymmetric CPU capacity systems, the wakeup path uses select_idle_capacity(), which scans the span of sd_asym_cpucapacity rather than sd_llc. The has_idle_cores hint however lives on sd_llc->shared, so the wakeup-time read of has_idle_cores operates on an LLC-scoped blob while the actual scan/decision spans the asym domain; nr_busy_cpus also lives in the same shared sched_domain data, but it's never used in the asym CPU capacity scenario. Therefore, move the sched_domain_shared object to sd_asym_cpucapacity whenever the CPU has a SD_ASYM_CPUCAPACITY_FULL ancestor and that ancestor is non-overlapping (i.e., not built from SD_NUMA). In that case the scope of has_idle_cores matches the scope of the wakeup scan. Fall back to attaching the shared object to sd_llc in three cases: 1) plain symmetric systems (no SD_ASYM_CPUCAPACITY_FULL anywhere); 2) CPUs in an exclusive cpuset that carves out a symmetric capacity island: has_asym is system-wide but those CPUs have no SD_ASYM_CPUCAPACITY_FULL ancestor in their hierarchy and follow the symmetric LLC path in select_idle_sibling(); 3) exotic topologies where SD_ASYM_CPUCAPACITY_FULL lands on an SD_NUMA-built domain. init_sched_domain_shared() keys the shared blob off cpumask_first(span), which on overlapping NUMA domains would alias unrelated spans onto the same blob. Keep the shared object on the LLC there; select_idle_capacity() gracefully skips the has_idle_cores preference when sd->shared is NULL. Co-developed-by: Andrea Righi Signed-off-by: Andrea Righi Signed-off-by: K Prateek Nayak --- kernel/sched/fair.c | 8 ++-- kernel/sched/topology.c | 81 +++++++++++++++++++++++++++++++++++------ 2 files changed, 75 insertions(+), 14 deletions(-) diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 69361c63353ad..934eb663f445e 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -7925,7 +7925,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool struct cpumask *cpus = this_cpu_cpumask_var_ptr(select_rq_mask); int i, cpu, idle_cpu = -1, nr = INT_MAX; - if (sched_feat(SIS_UTIL)) { + if (sched_feat(SIS_UTIL) && sd->shared) { /* * Increment because !--nr is the condition to stop scan. * @@ -12840,7 +12840,8 @@ static void set_cpu_sd_state_busy(int cpu) goto unlock; sd->nohz_idle = 0; - atomic_inc(&sd->shared->nr_busy_cpus); + if (sd->shared) + atomic_inc(&sd->shared->nr_busy_cpus); unlock: rcu_read_unlock(); } @@ -12869,7 +12870,8 @@ static void set_cpu_sd_state_idle(int cpu) goto unlock; sd->nohz_idle = 1; - atomic_dec(&sd->shared->nr_busy_cpus); + if (sd->shared) + atomic_dec(&sd->shared->nr_busy_cpus); unlock: rcu_read_unlock(); } diff --git a/kernel/sched/topology.c b/kernel/sched/topology.c index 5847b83d9d552..dc50193b198c6 100644 --- a/kernel/sched/topology.c +++ b/kernel/sched/topology.c @@ -680,19 +680,39 @@ static void update_top_cache_domain(int cpu) int id = cpu; int size = 1; + sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); + /* + * The shared object is attached to sd_asym_cpucapacity only when the + * asym domain is non-overlapping (i.e., not built from SD_NUMA). + * On overlapping (NUMA) asym domains we fall back to letting the + * SD_SHARE_LLC path own the shared object, so sd->shared may be NULL + * here. + */ + if (sd && sd->shared) + sds = sd->shared; + + rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); + sd = highest_flag_domain(cpu, SD_SHARE_LLC); if (sd) { id = cpumask_first(sched_domain_span(sd)); size = cpumask_weight(sched_domain_span(sd)); - /* If sd_llc exists, sd_llc_shared should exist too. */ - WARN_ON_ONCE(!sd->shared); - sds = sd->shared; + /* + * If sd_asym_cpucapacity didn't claim the shared object, + * sd_llc must have one linked. + */ + if (!sds) { + WARN_ON_ONCE(!sd->shared); + sds = sd->shared; + } } rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); per_cpu(sd_llc_size, cpu) = size; per_cpu(sd_llc_id, cpu) = id; + + /* TODO: Rename sd_llc_shared to fit the new role. */ rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); sd = lowest_flag_domain(cpu, SD_CLUSTER); @@ -711,9 +731,6 @@ static void update_top_cache_domain(int cpu) sd = highest_flag_domain(cpu, SD_ASYM_PACKING); rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd); - - sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY_FULL); - rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd); } /* @@ -2650,6 +2667,15 @@ static void adjust_numa_imbalance(struct sched_domain *sd_llc) } } +static void init_sched_domain_shared(struct s_data *d, struct sched_domain *sd) +{ + int sd_id = cpumask_first(sched_domain_span(sd)); + + sd->shared = *per_cpu_ptr(d->sds, sd_id); + atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); + atomic_inc(&sd->shared->ref); +} + /* * Build sched domains for a given set of CPUs and attach the sched domains * to the individual CPUs @@ -2708,20 +2734,53 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att } for_each_cpu(i, cpu_map) { + struct sched_domain *sd_asym = NULL; + bool asym_claimed = false; + sd = *per_cpu_ptr(d.sd, i); if (!sd) continue; + /* + * In case of ASYM_CPUCAPACITY, attach sd->shared to + * sd_asym_cpucapacity for wakeup stat tracking. + * + * Caveats: + * + * 1) has_asym is system-wide, but a given CPU may still + * lack an SD_ASYM_CPUCAPACITY_FULL ancestor (e.g., an + * exclusive cpuset carving out a symmetric capacity island). + * Such CPUs must fall through to the LLC seeding path below. + * + * 2) Skip the asym attach if the asym ancestor is an + * overlapping domain (SD_NUMA). On those topologies let the + * LLC path own the shared object instead. + * + * XXX: This assumes SD_ASYM_CPUCAPACITY_FULL domain + * always has more than one group else it is prone to + * degeneration. + */ + sd_asym = sd; + while (sd_asym && !(sd_asym->flags & SD_ASYM_CPUCAPACITY_FULL)) + sd_asym = sd_asym->parent; + + if (sd_asym && !(sd_asym->flags & SD_NUMA)) { + init_sched_domain_shared(&d, sd_asym); + asym_claimed = true; + } + /* First, find the topmost SD_SHARE_LLC domain */ + sd = *per_cpu_ptr(d.sd, i); while (sd->parent && (sd->parent->flags & SD_SHARE_LLC)) sd = sd->parent; if (sd->flags & SD_SHARE_LLC) { - int sd_id = cpumask_first(sched_domain_span(sd)); - - sd->shared = *per_cpu_ptr(d.sds, sd_id); - atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); - atomic_inc(&sd->shared->ref); + /* + * Initialize the sd->shared for SD_SHARE_LLC unless + * the asym path above already claimed it. + */ + if (!asym_claimed) + init_sched_domain_shared(&d, sd); /* * In presence of higher domains, adjust the -- 2.54.0