From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from CY7PR03CU001.outbound.protection.outlook.com (mail-westcentralusazon11010014.outbound.protection.outlook.com [40.93.198.14]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1816E318ED9 for ; Fri, 24 Apr 2026 11:19:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.198.14 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777029556; cv=fail; b=axDEmSxYLhsi4NetLDyUbEGDqvWzrVPR5zyjyIT8JlmKcGYF0sM4gqhQS+2lS5KdJBlWeoe1Ryy/7N2lTP83YzgMGwhnFFIplLRpPy1CxejVxjcipThatG+iMTy82Mq96TwNTQGY9nG9uqynHmvPMRi5S90jLM0WUhG66Vg/kqo= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777029556; c=relaxed/simple; bh=cucxsnKgIIOSiLkXqjYG3oofXFERBASPRp6HiWWG5Ro=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=kdFHJsllaCgbaXQ4H+acPvWlYrFVaSBE8DgcRIVXFoi7qD9dGG++TjXoBTq311Ce3OC/xrPnLGvInCnA0MR8lF+KCDl//jtuV+Pv2OuNmxkPyD3pg15AnPEuwLhQig6VRn0h4yBeFSYkVwpattql/VESdSBfrBDCu4Fsh7Agd8I= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=KCE1gbQX; arc=fail smtp.client-ip=40.93.198.14 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="KCE1gbQX" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KnRs+qhL7FfZm8OHKLADRTYHhO3GbPetN5HnSLZypQpQm7g78iyn3rR+mchdTdCQPoJXpD2dDi4/c7p8slGiRk2WIWCuKghvAjUwhQQs7Rldc5ck6ZY7K3jj9NNfIX7KRTfvlQoZf4qcyVFzruw56tdpjtw6XGsYGg4i5wKvzLXFb9r3B3nnVE+dzcst+kc+mPSreErC52loONUX4Eu+8x0arNb8yUXXCblVh4GNOBcd4xj6IA0AW7uw3zsUCLB5/ulBts7ICxtCfWsHzCJoZ59LlwC5aqAm7dOrYpwjuRlzPo+yubuBTzHcVK3OBgLHZTa6AVo3V/B34/Uq0SdOEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fMZEY/+6m3Tiu63Px0T9C8IrpXDpg77uOkq14cPiNAM=; b=dlfIpGFWSlDbjiKEVs0OftjbtJNz3SOjKiZJL1Ga+g8Z6b4eVMLM70qIM+rD7a5Ax4bh/6Hb/26bfbxLiB3mkxMX/ueISvxQaVemVs4pjMNsFKdOF+G73PQmnnVMW36GKPsu5wQBSxBzwcKhIUQ2TGm8oTvdeBm9cAn5qSK7uy3DykNq6raGHp/OCg4ldkPDR2d0OoM58ZSAs6ykzrsVWO4hE5+aslXlUscoPpC25FQkZ2Hii87cv3pDVWhuNf1fOerr1jtAKyk2tMhChivT2NpL9KQNx7VosSU953ECORi4lp3OhhlxrQvACOw8+2XdYN1tEkAUpJ79ib/y4q+H9w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=nvidia.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fMZEY/+6m3Tiu63Px0T9C8IrpXDpg77uOkq14cPiNAM=; b=KCE1gbQXp6WLPHRlydCeEoc25RCxEG4RDHKQIcrl/OkFTdo50pGX2TJBnjyu7UI66lCvkIg3cy38hxcFRcRZzyTwyj8RtQe2vs5HWv86fFYApiJCF3XPAmRcj+mPAUQ+VEYx8fMTOF/MD8RpcNSmLACGXIVV+VCfhcKiQyG//i4= Received: from BN9PR03CA0872.namprd03.prod.outlook.com (2603:10b6:408:13c::7) by SA1PR12MB7104.namprd12.prod.outlook.com (2603:10b6:806:29e::7) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.15; Fri, 24 Apr 2026 11:19:08 +0000 Received: from BN2PEPF000044A4.namprd02.prod.outlook.com (2603:10b6:408:13c:cafe::f5) by BN9PR03CA0872.outlook.office365.com (2603:10b6:408:13c::7) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9846.21 via Frontend Transport; Fri, 24 Apr 2026 11:19:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BN2PEPF000044A4.mail.protection.outlook.com (10.167.243.155) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9846.18 via Frontend Transport; Fri, 24 Apr 2026 11:19:08 +0000 Received: from satlexmb08.amd.com (10.181.42.217) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.2562.17; Fri, 24 Apr 2026 06:19:05 -0500 Received: from [172.31.184.125] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Fri, 24 Apr 2026 06:19:00 -0500 Message-ID: Date: Fri, 24 Apr 2026 16:48:59 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 1/5] sched/fair: Attach sched_domain_shared to sd_asym_cpucapacity To: Andrea Righi CC: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , Christian Loehle , Koba Ko , Felix Abecassis , Balbir Singh , Joel Fernandes , Shrikanth Hegde , References: <20260423074135.380390-1-arighi@nvidia.com> <20260423074135.380390-2-arighi@nvidia.com> <75cf4fd1-2e80-4167-9113-954015ba63e1@amd.com> Content-Language: en-US From: K Prateek Nayak In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN2PEPF000044A4:EE_|SA1PR12MB7104:EE_ X-MS-Office365-Filtering-Correlation-Id: 1f54421c-2b2f-4bec-865c-08dea1f34d82 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|36860700016|82310400026|376014|7416014|1800799024|18096099003|18002099003|56012099003|22082099003; X-Microsoft-Antispam-Message-Info: dfTNDGoLkPks1YXc/kwMoeLIVoOl0SyCWLiSM0FlR9UzP8Sx7thoGsegSF+dMlUSwh6RqbQkOxnhGAgoU+NLjS9+IhXjx7AENoJvQbyLZiCTMUxv0cEmroStS8GrP4zw0cqPAwCZx/7l9m2gWXj2aqW4EUhbJ848QNV1Xo4Wc+ZgInKUJEOIiGGPJ3yI1a6xJvR3CB4XGUoAg2RV9ODkRA2M7Q1vhkrAaKM7svjCuWdV6lxwzlRosL7HVshzjb7L9h3tKr/nEWTvZ2bQKwSxzk3GK9PkyDtSfrmYcjBAYA1QoJ0tJYm8B6YKO6bus1qi9rccMGTtW9Qzim75iCicvHuxuiL79/9g6dnFpBJ03YHvL8JNXTcXs3CzHnU5501ouuI5QcqAeSib9PFbVrVD67Q3uLyGwb01+nFc6Ce00Y33IWyrw2uWdTeqLeEI/2z1m1xNQQgJUNV+jgKcdWaUzHhbbQGSvc3XinbmCknj+afTKD1/YDzr5mbTTF0LDTvz/TCFIByVQlNmv/bPIQZSaJ2S3WP9R9r00bFldyxsjAY5pMoAZ58Nq5373eNaRS1UXN8PADw7EZV3YLdo9c5vWyzzhQB0jKpxMm9xoYHFjpLZ5xU/a4ab2bWu1EZYojHMqjAv+ROHs4/EB+5vLPlG/FLJ5mVhs6Gy/Wu9tc0/68wYA6qfvNbX+d3OByi0zgNTCSXVJbny7UYiDbpZUdZJlzX4z7Wl9fY2q3Z/8q58sAJ/lsGWjMiv3NK7B6gvrcel1kbYHzctWj21SgJnHIVRDA== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(36860700016)(82310400026)(376014)(7416014)(1800799024)(18096099003)(18002099003)(56012099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9wbdvM5ydfbrW0fu0TmnZQdX2+O3dg8xJNuu6vGFFlJAqEmy+UmXNSSOCrFvowPdhU4fThlBtd2iG28VllLGu7aknRLRYVNPO4+IgzM+wD7lPlZ76FELWxfqXOCtTyNYklUKUDqtb/RVQcg6E+yPiMjoDA0WO+uLNWWiE7sSMXnY1LDIpHILW1OmgU66CxT3u0FAVyf5h3Vb8+N2+Te7ro2QreWQNeG5nJG/HoOPwjZ0lrm/uJyOZ1oAv2C9tkrjqUAJnY7vp5VASOhIhAriGELLpGsX2619HD8rIvnvFpo22pk4FH7FPlaOibrXvXM646KS2QeF3bZAcULQu3POrbZ1eAxmzLsEhtf4DoQYoAFjndVTYZ96/QqKyCo6loC7xon5hiSuVQW0rDwIwHtv5pfDUyRrfvjNM6aAyi5DYjkOKyX6B0QHUWCdPMB5rNWN X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Apr 2026 11:19:08.2272 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 1f54421c-2b2f-4bec-865c-08dea1f34d82 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN2PEPF000044A4.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR12MB7104 Hello Andrea, On 4/24/2026 2:16 PM, Andrea Righi wrote: >>> rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); >>> per_cpu(sd_llc_size, cpu) = size; >>> per_cpu(sd_llc_id, cpu) = id; >>> + >>> + /* TODO: Rename sd_llc_shared to fit the new role. */ >>> rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); >> >> Would love for folks to chime in but IMO "sd_wakeup_shared" sounds >> pretty reasonable since it is mainly the wakeup path that depends on >> this except for one !ASYM load balancing trigger. > > sd_wakeup_shared captures the bigger consumer (wakeup), but not the nohz > balancer kick logic. > > Maybe "sd_balance_shared" (balance in a broad sense, wakeup is still affecting > balancing at the end) or "sd_effective_shared" (if we want to stress that > topology may move: LLC vs asym)? Works for me! I don't have any strong feelings on this. [..snip..] >>> + /* >>> + * In case of ASYM_CPUCAPACITY, attach sd->shared to >>> + * sd_asym_cpucapacity for wakeup stat tracking. >>> + * >>> + * Caveats: >>> + * >>> + * 1) has_asym is system-wide, but a given CPU may still >>> + * lack an SD_ASYM_CPUCAPACITY_FULL ancestor (e.g., an >>> + * exclusive cpuset carving out a symmetric capacity island). >>> + * Such CPUs must fall through to the LLC seeding path below. >>> + * >>> + * 2) Skip the asym attach if the asym ancestor is an >>> + * overlapping domain (SD_NUMA). On those topologies let the >>> + * LLC path own the shared object instead. >>> + * >>> + * XXX: This assumes SD_ASYM_CPUCAPACITY_FULL domain >>> + * always has more than one group else it is prone to >>> + * degeneration. >> >> I looked into this and we only set SD_ASYM_CPUCAPACITY if we find more >> than one capacity and SD_ASYM_CPUCAPACITY_FULL implies there are atleast >> two CPUs covering differnt capcities in the span. >> >> The very first SD_ASYM_CPUCAPACITY_FULL domain should be safe from >> degeneration when it is non-overlapping. > > Makes sense, maybe we can replace the XXX part with note like this: > > * Note: SD_ASYM_CPUCAPACITY_FULL is only set when multiple distinct > * capacities exist in the domain span, so the asym domain we attach > * to cannot degenerate into a single-capacity group. The relevant > * edge cases are instead covered by the caveats above. Ack! That should make it clear. Thank you. [..snip..] >>> if (sd->flags & SD_SHARE_LLC) { >>> - int sd_id = cpumask_first(sched_domain_span(sd)); >>> - >>> - sd->shared = *per_cpu_ptr(d.sds, sd_id); >>> - atomic_set(&sd->shared->nr_busy_cpus, sd->span_weight); >>> - atomic_inc(&sd->shared->ref); >>> + /* >>> + * Initialize the sd->shared for SD_SHARE_LLC unless >>> + * the asym path above already claimed it. >>> + */ >>> + if (!asym_claimed) >>> + init_sched_domain_shared(&d, sd); >> >> Tbh, if "has_asym" is true, we probabaly don't even need this since the >> nr_busy_cpus accounting gets us nothing. >> >> Might save a little overhead and space on those systems but I would >> love to hear if there are any concerns if we just drop the >> sd_llc->shared when we detect asym capacities. > > Hm... but "has_asym" is global, we may still need LLC-owned shared for symmetric > islands and NUMA-overlap cases, no? "has_asym" is local to build_sched_domains() right? So it should operate per cpuset partition since we call build_sched_domains() for every "cpu_map". Ack to everything that was snipped off! -- Thanks and Regards, Prateek