From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 23CC4C88E4C for ; Fri, 11 Sep 2026 09:11:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To: Content-Type:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=W25+wRpoT5FmrPKRz7FlvVkM5WlWhH4E1k/AYUHT/yU=; b=ewdNMzXv7W63iafNcydaBBTmLU ssu9s7OeSVNST5nXr1IFNSc2jgBneQKPDhGgvvI3fCZZpnHhQe4lEEg0GnvioUOc8ufSzmOsRQwc5 jYlVZS8NEQKXJTFqaEq5vOdcdZ0R/rQWqzQhqlH/PedJzxgCMWDCTOTZ/1kKq7TzUm3WI7snGz81S 1ouPitmLEn0RKrJZA4jb7ObEy/BjXCrkIwZJnoW5VxuNpDo6EfsIIBs5I/UA0OhJiz9BjNh3+V8Gz iez1NECdGB6gfEH1WVFxTs8ak5/8U8os5PcsuCTx3CD+W+WE00L1uiV6pD8CT7squTXxMdx5Tiw7t MxJgsb0Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1x4xI8-0000000GEQe-2nsH; Fri, 11 Sep 2026 09:11:28 +0000 Received: from mail-westus3azlp170120001.outbound.protection.outlook.com ([2a01:111:f403:c107::1] helo=PH8PR06CU001.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1x4xI6-0000000GEQ4-0LX1 for linux-arm-kernel@lists.infradead.org; Fri, 11 Sep 2026 09:11:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Q5NQb1guXagVllRooLwDCyjyV/msURwZuRqksPyVS8VBrD3FNVwpCck4H9DTHOziaoLjueLBPupGIXL4AMUjiPFyCkWKaPATkTjm2gwH6+BCfCNcH+29ZxOlzgFh7A1hH3UT7o2DF0e43ghVtSnaPooEfB8MU6jmvUXcbXr6QvmMHckHkleW4Cu3tXdpRUal6qIuf2lD7p9j6ip+iO7VpvbFwNe9PaD+FAMtkXaeMPIwNtlf7syBzSvsiYNtXEcRk6U6X1GiFDkgkAaWN4mtLG96n+0m1cb/ryLQHtb/ZbRpaCRNUFhspHTmUeK8MIaakICVt14VYceHfLNkBj6dEg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=W25+wRpoT5FmrPKRz7FlvVkM5WlWhH4E1k/AYUHT/yU=; b=J9xMO1Udu5nFwWoAacVO4I+t0oXTf59KAC4JmBvGSOe6d5RqVWMsbeAPKFA2jW1Sg4PA9w2dfNm1+lPqd9ZZkyw7WZY9tbOr1T8zeToXdJfL3c/hdqTWGFh/DdK6hO2MnB7xkZVSXIMwgjQMOigZ9PI+z2m4X5N63T65wsVJI47O48tvkiKjpgAm0QvigqYJX+PAi1t08SOWxU36b5T31AaLXRKHvJNcqqCfU064KWylIVTSdBhiSAwr2p6gOmVCMqonosnwsEhfiwe7vPRsEDbdJ3Ie7z0mx0go4df9Ih0RgD4v4IRS7mWuIetQKoS+i8aP6TWx+xuW4WwcAa1yFQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W25+wRpoT5FmrPKRz7FlvVkM5WlWhH4E1k/AYUHT/yU=; b=MvmbZt3xcHuQ8lwRDnkRaAcxdcc68qMII6h60ZrS3SuifH3w5WdO4lEjhYhQ9d13We+B0988UkL7AWHpUG3dSaxptOhuXI9GT6NfqXpLNSoa+ufbrRJ+ShV1GghBd8jCtSx7IZRrge+fsG2VaryS+4TKuzuZkAPsBv27V4ZLowByHnq4FpCtDFrp0lH6MIGIqb4AkfxOpKvsy1hbeK8eAu2M7/tcvhjiE9lpWrJDXDUVKB4Dt13lGhOk8VMdC1uOikcxfGeDw5te+SLL1O6NpPGxRFCRxaodTZzJlHUJdgfWrjKS7mYpPKxgWI8UdgTX8djkxv5hRpHeo+kjX9FVcA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from DM6PR12MB4827.namprd12.prod.outlook.com (2603:10b6:5:1d6::14) by DS2PR12MB425616.namprd12.prod.outlook.com (2603:10b6:8:401::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.21.406.10; Fri, 11 Sep 2026 09:11:12 +0000 Received: from DM6PR12MB4827.namprd12.prod.outlook.com ([fe80::6261:3040:864b:159c]) by DM6PR12MB4827.namprd12.prod.outlook.com ([fe80::6261:3040:864b:159c%7]) with mapi id 15.21.0406.007; Fri, 11 Sep 2026 09:11:11 +0000 Date: Fri, 11 Sep 2026 11:11:01 +0200 From: Andrea Righi To: Dietmar Eggemann Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Catalin Marinas , Will Deacon , Steven Rostedt , Ben Segall , Mel Gorman , Valentin Schneider , K Prateek Nayak , Mark Rutland , Christian Loehle , Shrikanth Hegde , Phil Auld , Breno Leitao , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH 1/2] arm64: topology: Prefer PE0 on NVIDIA Olympus SMT cores Message-ID: References: <20260909062649.469633-1-arighi@nvidia.com> <20260909062649.469633-2-arighi@nvidia.com> <89694e14-4be8-4c9e-9d7b-2808f87dd95f@arm.com> <1528999e-eb59-416f-b18f-2a7667e61a90@arm.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1528999e-eb59-416f-b18f-2a7667e61a90@arm.com> X-ClientProxiedBy: MI3PEPF0000753E.ITAP293.PROD.OUTLOOK.COM (2603:10a6:298:1::4da) To DM6PR12MB4827.namprd12.prod.outlook.com (2603:10b6:5:1d6::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6PR12MB4827:EE_|DS2PR12MB425616:EE_ X-MS-Office365-Filtering-Correlation-Id: 565ca042-366f-4a4d-0884-08df0fe49fa1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|23010399003|376014|1800799024|366016|3023799007|10067099003|4143699003|11063799006|56012099006|18002099003|22082099003; X-Microsoft-Antispam-Message-Info: W6RiSzKuCal3su+JkY9UJ9Wq7zoOLwx2rtoYgPujiVT7FJDrKTGjPWeCHN4zvpsORRoXj3TWlouQ2JMWbJSrTllTGadbFdgOm/J8JTtPsOBCJD29aGgg7aiG0BSztjOykbL5fF3IjwugHOQ8XdYTzlRCzxYvLL3QeGE18113ItVPzlmznN7XofKlMaUD6oWnE8YSnFbruzTXxMrGdLCBg7mCu8VP/q5JIr0scbUFOWfyZVmgArMMx9Mj5zXjRx8EB66DaMS//TtOo6RK9AAxIM7lHMMF7nYPa/jv/9HTj16Xz3r7phy1tPWcSO6jd9YyoBRg+T5aUj6v6i9cAaPhpx8yUI182XT3S/pMbwK5eqppZOESULmRvtCb+D8AyKsHLzYmOr5jnNX9s0qHsfpAiy8SBwb0ENipOjtWlKTUQt0oLgaOc11rO8g0++cHnWL8gE6wWKyyS3zErVuCO/Pm0Sab7GDvwHwt6LEh8s5YtUk3PlI67PR2WL3k+VmNv5orhaBrwsr+HpztBraESmGm/FiFTP+H6F0JQPo0Qu4fR5M1oc9RWRJ9hVgeotCxN/Ay8vxKzSllajPWsEV0cmwM7th/pfuGMH+WLCYq5Lgdl5CqLFTa0/pdOcYE+jv8FoANHQRn4Qm7pqfOHjphz8B/qEa17L6/+1Ib1b4Bd2Y2g3c= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM6PR12MB4827.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(23010399003)(376014)(1800799024)(366016)(3023799007)(10067099003)(4143699003)(11063799006)(56012099006)(18002099003)(22082099003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?gzGV4haMmds1pC7NoOiJVTw1VbLCtyN+moal+B2U4KLBQVqx33C9HBC0D3Yf?= =?us-ascii?Q?F/AK9G5d0GfS25J3SYxj9Nua551lypLDWoxx4AOuGaMlf74H+Bpqnuo4P7Zj?= =?us-ascii?Q?xL1fPZqI/qjUz1VUZ+hsWqVYvZFJUf9AulTKR0W1fkCTi/qM0ksm/ggicRgP?= =?us-ascii?Q?B9xdI5CbOq1m8lahCjPJHAuddSSL2jsGCjMH6+7r5j55sPcDvCGBXhsapmGP?= =?us-ascii?Q?oyEadFnUBnRtPmjL0z8ecD8jeZwKsQx8x3d9AQbsZW33ZgMsMOuoA9WOJpV5?= =?us-ascii?Q?z2Fj7IZwkD07/F7kRCAe9gugAXR+e+YIvCODamR+2f5QHDO9+7X8gQaZ9zZb?= =?us-ascii?Q?7QuKYAsIsRQtit6OtJbLxofD7fW/vzzl13xU4eqHpRFNY1WD5BhS9l6oz0T4?= =?us-ascii?Q?LL5gX+vFLy4EY2K2Au1evRRa95uO5xYaRRtGdMCM/bHi+rjWiATFtuutkZkU?= =?us-ascii?Q?v+xOP5dvgzy6Dsq9eSsXYsP7GIlCE+SX5XpHvhEqC2zolboW1hloqT3yBc53?= =?us-ascii?Q?R+NQSquXpZq9iNC3kpcErVdsfK2e9MnC4aHNMm9hKm+7p3cPp28Rtw5NrNOu?= =?us-ascii?Q?pPUSfPJz1B9hmsc2Tx+gAv11nMC19+/UFTMZxpTeJ+WQDak5fCnpoF06KA2n?= =?us-ascii?Q?XbwEnDQe8vpX4kUXVqHRscJX0sHoxujAofj0lwPkFhuJygk0TwDWNDdpxNkk?= =?us-ascii?Q?1Ugx/jpmyrHZB1GWU0szwVGhzMAEHrabfFnj9Qmy0KmVvF9G9eqTiXJwaSSt?= =?us-ascii?Q?NYGePZdqHH5H4mFrnX80qEv7thnC6VnfxZgVMVA00PXHBbOIjPWAliciy+Gc?= =?us-ascii?Q?xgBGhG+lohHFxkS3mz91GGVEOgU8H27plWZ2OMDRp3X0E/l0HObTPgkZO9cE?= =?us-ascii?Q?ByT93cWjUt7h7Za7p9lx+wvkU9UlUm1IXJu5uEeAaTHumJfkUU8IwFkMHkMi?= =?us-ascii?Q?iDk9+SNXyU+CZ6ZFUvcN9j0L609LwvVpN6YJ1VrjijloMtOHOrFNuWZHGNt7?= =?us-ascii?Q?CBWD+jZLfOTIaXnKZOGda6UfbbSHUqrGz7wTzdqJiDc1VtLlfSVpLpGQnjQC?= =?us-ascii?Q?dZPjj/iYVWJRFebaQzpQpNRTnzaTE8xalnhxKFm2M7icsdjhm33Y6q7vCDtr?= =?us-ascii?Q?pqBBMQlbKDadDmyj0r3UX6DpYbo+XwLmZu/AXyUOKLLpbJcYy8wfy0jGtqD+?= =?us-ascii?Q?Ii0AQkwK1p4Lg/RNiXOApJj2rLhyLbNqAAX6EQeeGMSTjVnN8gETa6+pw/4a?= =?us-ascii?Q?LsNMdK90Xk2NZwvA1MaTgsWKcE6JGVE3noqigtwri7ZmeRmqXUxE20JM3qMi?= =?us-ascii?Q?ND2L07wGvqMO6P6IgKVRh/ttbKsCVdGatM8TyIVffbEzQa2KSqkd3c/fWwhm?= =?us-ascii?Q?h5rJLba5cB1X1FVo2yDc8FuUFdAWmPg+A2fF8M3I+bn9r3BY2IrRMGjBT4D6?= =?us-ascii?Q?wRWbutNrrX8oFxQBiO/8jzfPQ/9t0HCyOpDD/VG1lPLfdromjPhdWpVrlD8w?= =?us-ascii?Q?MxL32+tgQjF5oEodJa65Aezv8OgrNrDmDag5Zl3O5mwTgcVWn5lOUTCnuAPd?= =?us-ascii?Q?0Y2XPrvFiAMcXyURacB7+eKh/n9a11IFNftX3qSO5zlSmQI4BEjOEY6Rv5JM?= =?us-ascii?Q?R1AERa+nGZh4fKRcNUH7ciONOapl3N7GOhkOPGi4wP4Y0mFwAKkPQ2fvORA5?= =?us-ascii?Q?X6f1/BjzHMUO44jqv/1cwzARpxUVB69vfIvQSoRmL/1Zb9V3AiXqG3fsAdQV?= =?us-ascii?Q?BTFn2lwflA=3D=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 565ca042-366f-4a4d-0884-08df0fe49fa1 X-MS-Exchange-CrossTenant-AuthSource: DM6PR12MB4827.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Sep 2026 09:11:11.7376 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1dL2hMPO+NAVpKVxoVvHq7KRSKkdvXzl3+xC8VDtxsnkD5HM1OWcdAAGmBYN53zIoU3rof+PisdY+z+9648YSg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS2PR12MB425616 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260911_021126_143498_CB6B3798 X-CRM114-Status: GOOD ( 28.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Dietmar, On Thu, Sep 10, 2026 at 12:11:06PM +0200, Dietmar Eggemann wrote: > On 09.09.26 18:17, Andrea Righi wrote: > > On Wed, Sep 09, 2026 at 05:19:08PM +0200, Dietmar Eggemann wrote: > >> On 09.09.26 08:26, Andrea Righi wrote: > >> > >> [...] > >> > >>> +void __init arm64_init_sched_topology(void) > >>> +{ > >>> + if (!IS_ENABLED(CONFIG_SCHED_SMT)) > >>> + return; > >>> + > >>> + if ((read_cpuid_id() & MIDR_CPU_MODEL_MASK) != MIDR_NVIDIA_OLYMPUS) > >>> + return; > >>> + > >>> + if (!topology_core_has_smt(smp_processor_id())) > >>> + return; > >>> + > >>> + set_sched_topology(arm64_asym_smt_topology); > >>> + pr_info("Enabling PE0 SMT preference for NVIDIA Olympus\n"); > >> > >> I'm not really a big fan of using this arm64 setup for NVIDIA Olympus > >> alone here. > >> > >>> +} > >>> + > >>> +int arch_asym_cpu_priority(int cpu) > >>> +{ > >>> + return MPIDR_AFFINITY_LEVEL(cpu_logical_map(cpu), 0) == 0; > >>> +} > >>> + > >> > >> This is done since it will only be called for NVIDIA Olympus since it's > >> the only CPU model enabling this. I think it will work but it's still > >> ugly :-) > >> > >> [...] > >> > > > > Agreed on both points. As mentioned in the other email, I'll coordinate with the > > firmware team on a proper ABI for describing the PE0 preference. For the systems > > with the current firmware, I don't have many ideas on how to enable this... a > > special scheduler facility enabled via kernel boot option? Other ideas? > With the explicit kernel boot time option discussed in the other thread, > how do you want to code arch_asym_cpu_priority()? Good point. I think we can avoid an arm64 override entirely and rely on the default implementation: int __weak arch_asym_cpu_priority(int cpu) { return -cpu; } The boot option would only add SD_ASYM_PACKING to the SMT domain. A rough prototype could look like this in kernel/sched/topology.c: static bool sched_smt_asym_packing __read_mostly; static int __init setup_sched_smt_asym_packing(char *str) { sched_smt_asym_packing = true; return 1; } __setup("sched_smt_asym_packing", setup_sched_smt_asym_packing); int cpu_smt_flags(void) { int flags = SD_SHARE_CPUCAPACITY | SD_SHARE_LLC; if (sched_smt_asym_packing) flags |= SD_ASYM_PACKING; return flags; } This would make the option generic rather than specific to arm64 or Olympus. > > AFAIK, Arm architecture does not require the first/primary HW thread of > an SMT core to have MPIDR_EL1.Aff0 == 0. Agreed. Using the default -cpu ordering avoids making any assumptions about MPIDR and consistently prefers the lowest-numbered logical CPU in each SMT domain. > > The closest I came to you setup in Arm64 is a ThunderX2 SMT-4 machine on > which I can run 'OpenBLAS benchmark/sgemm.goto'. This machine has > symmetric CPU capacity though like Power7. > It has MPIDR_EL1.Aff0 == {0,1,2,3} for HW threads but '== 0' wouldn't > work here. > > [...] Right, -cpu provides a total ordering for all threads, so this should provide the expected sibling ordering on the ThunderX2 machine as well. If you're willing to test the next version there, that would be very useful! Thanks, -Andrea