From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from SN4PR0501CU005.outbound.protection.outlook.com (mail-southcentralusazon11011050.outbound.protection.outlook.com [40.93.194.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEAB83FFABF for ; Mon, 2 Mar 2026 15:35:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=fail smtp.client-ip=40.93.194.50 ARC-Seal:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772465721; cv=fail; b=mwKddFIJdhQDYgCuDMBYDjf4iKmHdGUcGvgIy+djuSyfHdlMHIhDZN2b9twCZNSQaPMvucELAvXjSguqsKxwodYzig/PsoPpOLDEW2dDcRIVLDicEQN5V6EQr3ZIifutV/PIfhOJxLjAacQSeZflSQpkjfH5txm1VibvY2EKUWU= ARC-Message-Signature:i=2; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772465721; c=relaxed/simple; bh=k2lQWzAwTJpVVOu/VGJKQa68LuSkARRDzYkXku6llLU=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=UCSih+nD7/uOvK7hAW/3mYzjbius5kjtqB/g/UZgMDigrrezwZQE09J83nhQxrBZhIYyCds9aVsq+yP8LFg680MN2CDZLIIlFSH0d0WPU3feDDr6w8BhoD2M9K1utVj3kkptq9VIJKLeWkDnIzyzzph0emPjbPAtIAUEPmqUzxI= ARC-Authentication-Results:i=2; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com; spf=fail smtp.mailfrom=amd.com; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b=Y/IHkLMr; arc=fail smtp.client-ip=40.93.194.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=amd.com Authentication-Results: smtp.subspace.kernel.org; spf=fail smtp.mailfrom=amd.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=amd.com header.i=@amd.com header.b="Y/IHkLMr" ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=n2RQqQkuvwoEbqWXSczDmr15U8nkCoyhhePO9v//o26JlPDQBYRXJeosbbI7GbOHyopmc7oJSHyXUmcuv5arlh80OdPDX3c7w11pXVIKdei6XB/o2A9QIc4ybgBf3K81PUMR4lsMBfFmj0WJvfmbElZpiwn3uq8q3G6rDX+FHQaOoUkR02P0qMWcJbzJNqCU4v3HYzZng666QiMq18NQQNs4HwpDMn4dnrzIEjLDxGRolkiQKKZRDVr4Ec65yZ6McMRCbzCWmn0/NVOxzCwsyOlJ6nKm/CaREDC41t8O4gtynQ3kBEgVG2UecjZLWXcrP99xZ+S7k5UJwTY+Q+xJcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=crYoo36/yIhWNYJrB1lyWx5s+hy6uGTz0fiOQWXPs+Q=; b=Ob+Fs34Zp3Fhpc8yWXTPGbhZvxSqS71P77qOwIfOx/SwP+ZVDju5LS8XEyuo8jveWw3gaTqsK1b+TizFgSy9MF7LhiFnzDod4tRcFI/PoUXZ6mDg0NLx7XPqhG5Y6CY9K4Bo6iGEIkYx1YBC3+LKHf20esEXP2Gg2tx49jhkEOwFfP7krkFF2Fewf9jmYasx+RKaNHvTqpTlPQ9cPQaGpc0eF8bxOY+d7sPy7/+hTdfSGbu9waW3j1zjRoVxG0zem4Qi9ZND82SVMwr9zR7SnhLp72PwMph2mvqwCil/sWFfhBSXozLxTcnbpMZlF2n4Lw54MCX2H4IAQaj2hgWc9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=infradead.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=crYoo36/yIhWNYJrB1lyWx5s+hy6uGTz0fiOQWXPs+Q=; b=Y/IHkLMr/91lplxOTccYHau81Y/lpNNEPa/Y/9rUnsMsZnkWyYWFrADQW95Iw1P4S12PDpgtLIuzUgUi9ubC/XQebaUH8Z447GVaG3bg7sQV8xpogIwv1/Pw/oZGwrsUxfvgDKA98fljlJmg8/yhq6p8/8NSGbtZCw01mVTGpyk= Received: from MN0PR04CA0028.namprd04.prod.outlook.com (2603:10b6:208:52d::27) by PH7PR12MB5757.namprd12.prod.outlook.com (2603:10b6:510:1d0::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9654.13; Mon, 2 Mar 2026 15:35:09 +0000 Received: from BL02EPF00021F69.namprd02.prod.outlook.com (2603:10b6:208:52d:cafe::2b) by MN0PR04CA0028.outlook.office365.com (2603:10b6:208:52d::27) with Microsoft SMTP Server (version=TLS1_3, cipher=TLS_AES_256_GCM_SHA384) id 15.20.9654.21 via Frontend Transport; Mon, 2 Mar 2026 15:35:07 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=satlexmb07.amd.com; pr=C Received: from satlexmb07.amd.com (165.204.84.17) by BL02EPF00021F69.mail.protection.outlook.com (10.167.249.5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.9654.16 via Frontend Transport; Mon, 2 Mar 2026 15:35:08 +0000 Received: from SATLEXMB06.amd.com (10.181.40.147) by satlexmb07.amd.com (10.181.42.216) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.2.2562.17; Mon, 2 Mar 2026 09:35:07 -0600 Received: from satlexmb08.amd.com (10.181.42.217) by SATLEXMB06.amd.com (10.181.40.147) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.39; Mon, 2 Mar 2026 09:35:07 -0600 Received: from [172.31.184.125] (10.180.168.240) by satlexmb08.amd.com (10.181.42.217) with Microsoft SMTP Server id 15.2.2562.17 via Frontend Transport; Mon, 2 Mar 2026 09:35:03 -0600 Message-ID: Date: Mon, 2 Mar 2026 21:05:03 +0530 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC][PATCH 2/6] x86/topo: Add TOPO_NUMA_DOMAIN To: Peter Zijlstra CC: , , , , , , , , , , , , , References: <20260226104909.675623579@infradead.org> <20260226105052.355756378@infradead.org> <76776da9-2552-49de-91c7-41e767c248fa@amd.com> <20260227140614.GF1282955@noisy.programming.kicks-ass.net> <8b8e7b6f-c874-48d5-8402-77eca5425b1d@amd.com> <20260302151034.GO1282955@noisy.programming.kicks-ass.net> Content-Language: en-US From: K Prateek Nayak In-Reply-To: <20260302151034.GO1282955@noisy.programming.kicks-ass.net> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BL02EPF00021F69:EE_|PH7PR12MB5757:EE_ X-MS-Office365-Filtering-Correlation-Id: eef80c92-d5c3-4b4c-e382-08de7871494f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|82310400026|36860700013|1800799024|7416014|376014; X-Microsoft-Antispam-Message-Info: jB1LI0mQazGTmbDmFvCDsI31rfJrs1iWm1Mas25pTWXGW0fkHdyo7v7t8aow7hMm3euMAfWA2CB6zLcBI/pXmi406p1A8DJxrAWLtpBDsRKZu6shog8JHmVbX0c09J1eeDbJSTW/DKI1/cvexjTQJwezgc7we+IqOgGReoM20WFxAQgr7uHdkP/TKoBttcudoNYVwmsZtR2ysCuJMn7rW9RadOvSGlxdHaVpC3t8NnmPF+AHadHyc8AhWwQH5JW6wLSWEJ1p/MXuOP9s7CFITQc2ialNb6H9Kn2AOVDRpcVk/CiyNzsG8cF4pmQCkNP0M4/qG++Nn4CKw9/ar32s1orhPXmUY4tGrqxcsYW/+nLMMxJ2z71w4k7XgHNKMTmOV1kkPj54YrRLS5R1jjJgBCCSB/sey7OSb1Fpjg2RmlBPZMo7H2zjmycgdeY8lS4nipYYtMYNLHX8WGePHUGup06aVRR0a6QwuXs/LdBkmfU+RBKxSrW8TyzyPFqS85T7p8nDxfZZVqpRZkbILRyvwOtRgs9m/MUKPWCuYaJs6NJaS5rLO2QO9VsszK/1TeAvc8QbHTOmHYNcZODMVv+6US/7JRizHz2eMeaKy+DwFRZ2Favhxugwe1lUEda4sgPaRpG5S6qhnZUdH2SAClwI8LMLdYGSqzdVNCyjygQ0MdAXJ/d86MtjNTSdnHlVg0t4cQfdFxIdTTPWcQeGdPdBRT3iy+l2SVg6ghCeGPh7F9fo61ebyHc/JBMkuD1g52qrWcb9Hf9hX5WixXClwB/b2Fy5Jzz8HKaWvzo4qdDChwpxwSts+kYvHYnjAzkC1zuxX67QiuOP+FWXozHfl6dhtg== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:satlexmb07.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230040)(82310400026)(36860700013)(1800799024)(7416014)(376014);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 7H6x65Bg+hYhM6Tnr4Jilo3uedpejySP0sk+2pC1TDgwFIHtba7SwSSOWDx4fQ+f9cdmVWqRXOTLOhnq0eCSmolFo/LA3MuHGk6U6ky8co82JV4QbuTpbd//L9yvK6LrodToLQ5GcEBNs57bI5rVexScNZoyyQLUDtjjtA9ReXt6DplI0sGe3mThZ02pnaIyE5Uq/rn63pJVZbPVy8IeU/MudFn1YdlFmSk/7JbdwA6/piTn04MyJCYTbKBmqkLCDpU27+9o+AxXT50n93tVu7BixgRtae0JLdCkQfPWzzlQtFl8KY1HHx4n3HA3JS107cOsQvs6szaO8wnlVOdTKHBBZ19fh9W4RcITaD8XWHZW2J8mgehDfT1v5++Z2ft9I4K8uym+7cbzZgOJWXUpyxsYGMHuPy5uyAELAarIeThQP+FlAiJY6b3VrO2dgK2L X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Mar 2026 15:35:08.9431 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: eef80c92-d5c3-4b4c-e382-08de7871494f X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[satlexmb07.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BL02EPF00021F69.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB5757 Hello Peter, On 3/2/2026 8:40 PM, Peter Zijlstra wrote: > On Mon, Mar 02, 2026 at 09:46:57AM +0530, K Prateek Nayak wrote: >> Hello Peter, >> >> On 2/27/2026 7:36 PM, Peter Zijlstra wrote: >>>> Looking at the series, all we need is an equivalent of: >>>> >>>> domain_weight(TOPO_NUMA_DOMAIN) >>> >>> Fair enough; but then lets replace patch 1 and 2 with something like >>> that. >>> >>> But I must note that the nodemask API is crap; it has both node_set() and >>> __node_set() be the atomic version :-( >>> >>> Let me go rework the other patches to fit on this. >> >> Boots fine with a s/domain_weight(TOPO_NUMA_DOMAIN)/num_phys_nodes()/ >> applied to Patch 3. >> >> Topology looks fine for NPS4 on my 3rd Generation EPYC with 2 sockets, >> and I haven't triggered any warning even with "L3 as NUMA" turned on. >> Feel free to include: >> >> Tested-by: K Prateek Nayak > > Thanks! > > I had a quick look at this NPS stuff, and that is more or less the same > as the intel SNC thing. With two notable exceptions: > > - you've stuck to power-of-two numbers (good!) Yeah but "L3 as NUMA" on a 6CCX machines doesn't follow that :-( Is there any implicit dependency there? P.S. All these configs are symmetric so those divisions should give the correct results. > > - NPS0; I don't think Intel has anything like that (although I could be > mistaken). > > Now, the __num_nodes_per_package is obviously not going to work for > NPS0 (it bottoms out at 1). > > Should we look at adding something for NPS0, or has that not been needed > (yet) ? Let me go boot into NPS0 to see what my machine thinks. But it shouldn't do any harm right because of the DIV_ROUND_UP() right? __num_nodes_per_package will be 1 (which is technically correct since the whole package is indeed one node) and then we retain the PKG domain so as far as those bits are concerned, it should be fine. This is from my dual socket Zen3 booted into NPS0: CPU topo: Max. logical packages: 2 CPU topo: Max. logical nodes: 1 CPU topo: Num. nodes per package: 1 CPU topo: Max. logical dies: 2 CPU topo: Max. dies per package: 1 CPU topo: Max. threads per core: 2 CPU topo: Num. cores per package: 64 CPU topo: Num. threads per package: 128 CPU topo: Allowing 256 present CPUs plus 0 hotplug CPUs CPU0's scheduler topology looks like: CPU0 attaching sched-domain(s): domain-0: span=0,128 level=SMT groups: 0:{ span=0 }, 128:{ span=128 } domain-1: span=0-7,128-135 level=MC groups: 0:{ span=0,128 cap=2048 }, 1:{ span=1,129 cap=2048 }, 2:{ span=2,130 cap=2048 }, 3:{ span=3,131 cap=2048 }, 4:{ span=4,132 cap=2048 }, 5:{ span=5,133 cap=2048 }, 6:{ span=6,134 cap=2048 }, 7:{ span=7,135 cap=2048 } domain-2: span=0-255 level=PKG groups: 0:{ span=0-7,128-135 cap=16384 }, 8:{ span=8-15,136-143 cap=16384 }, 16:{ span=16-23,144-151 cap=16384 }, 24:{ span=24-31,152-159 cap=16384 }, 32:{ span=32-39,160-167 cap=16384 }, 40:{ span=40-47,168-175 cap=16384 }, 48:{ span=48-55,176-183 cap=16384 }, 56:{ span=56-63,184-191 cap=16384 }, 64:{ span=64-71,192-199 cap=16384 }, 72:{ span=72-79,200-207 cap=16384 }, 80:{ span=80-87,208-215 cap=16384 }, 88:{ span=88-95,216-223 cap=16384 }, 96:{ span=96-103,224-231 cap=16384 }, 104:{ span=104-111,232-239 cap=16384 }, 112:{ span=112-119,240-247 cap=16384 }, 120:{ span=120-127,248-255 cap=16384 } ... root domain span: 0-255 The PKG domain covers both the sockets since it uses the node mask which covers the entire system. -- Thanks and Regards, Prateek