From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E4CFC433F5 for ; Thu, 3 Mar 2022 02:20:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:In-Reply-To:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=iNspjWhni7MLZAUORNpuaZEsRvBkmUjsNYi8BgSiTTM=; b=Na//9v53hg1Ths IBmbYzJ5Q1T3wwRhg2/ANkaYF1UqUt9XGYrRZWkdx2qx3dlu3VlTJFEY0XO0sREGAeiBwEuKjCuHx nWabCQtqhAglsj0LMKJ4TUaoUn+1eqx/kdVp+TPSr8s1rNauOIH+084SbvzcklVWgEoxsBKlKeFE6 AkaIX9K+pnUyi3yym4K+Y0FDJxT/AMWTADx+G275aVSXsa1zmYfWmBbd6cGSZU+rutKTQsbdSOh+8 Vy0SXyhdD/nv/uVqV9VKBsOYQDzGrkpxz40/A5/7Uet7ZafVIrIqNYsmkRyOxJbeVpxor+bfPQofx us6KNQLc7C+jjEM9GlRQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nPb3L-004zhV-Df; Thu, 03 Mar 2022 02:18:51 +0000 Received: from mail-mw2nam12on2070e.outbound.protection.outlook.com ([2a01:111:f400:fe5a::70e] helo=NAM12-MW2-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nPb3H-004zgm-IJ for linux-arm-kernel@lists.infradead.org; Thu, 03 Mar 2022 02:18:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g+F1N5LMsqvkNo+HMv9iDIZEb+U6U3gRU9b7/wsOIKaixu1gofnKf6zoaxFPLtZSDJmGu0o/dIpyeDwIPPlbnficvYhyxUqehOTNPUMGblUPFx1tItnFtWn6Br4T2zppwQAmofOSznecqCu2Bu7E/+SASF+3kkU3ZUy1n7NNhggFme5AUsxWbZRNcbCQLFmIQOJJ3+rU+GYdq3bO86U8h+1RAp8iIxF/Tpa6R9l4oVaPf21JOVFZ1NhUkoecl10SfTfKF6zf9k5w0WcGG7DshKsWiE/FR+XwC7WLx4+sWX54hqSqBVDYtGJDP2KuvzNjIlFIqSfhGwA/ArTan8WFDA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zwOMs48+WRZjc9i2N1olgQiTe52XZxybh4uFuXarQRM=; b=aDnest2V5R60wXLko2DWM/C9RdVOzjjvHz2g916cUBEhWymNAMIK5+El+8CvQYsUAoVXgsKkL21sqlDcXqO+40R24GxblUSUBFHiAWrupnhVi6LcCg22h9xn0xF11Czs6EbImwqnVX8F4gequiUG2xU/Y2AMulwlZgv/KQFo6gUmWBIskBb9U4aI2FN5Z9ujQWcQuXbRFdD1mQrflhFafFqt15b7Az/vu5WgEk5H9CBszk4C4bsvJQjf8JoBj57tikh4iaPjUXzHyU0Pkf6hzDIHiXeiKqYCBwb6b8PTPIv/rhTJvXiXs9fw6PxEH3yMRUDQfIdkG6hrg+YWoed3bg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=os.amperecomputing.com; dmarc=pass action=none header.from=os.amperecomputing.com; dkim=pass header.d=os.amperecomputing.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=os.amperecomputing.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zwOMs48+WRZjc9i2N1olgQiTe52XZxybh4uFuXarQRM=; b=PB+4NLcz3kTVfcNtaKVwmegSO2qJpeVrOnWiVvfjnV4YGdWlmQQDj0sfZXF2HztZwvR8mnoUG9GsgUKljHefISoZ8z5EgvVpAlF8twwfttLZVcTqODOb80A2awv598JNIF3nhMfto9my5tqnFlp5Q6dwZG3fNbXivLn8dTesUkA= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=os.amperecomputing.com; Received: from SA0PR01MB6329.prod.exchangelabs.com (2603:10b6:806:ee::12) by PH0PR01MB7400.prod.exchangelabs.com (2603:10b6:510:10b::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5038.14; Thu, 3 Mar 2022 02:18:40 +0000 Received: from SA0PR01MB6329.prod.exchangelabs.com ([fe80::7cd6:6594:ce3c:6941]) by SA0PR01MB6329.prod.exchangelabs.com ([fe80::7cd6:6594:ce3c:6941%9]) with mapi id 15.20.5017.027; Thu, 3 Mar 2022 02:18:40 +0000 Date: Wed, 2 Mar 2022 18:18:36 -0800 From: Darren Hart To: Vincent Guittot Cc: LKML , Linux Arm , Catalin Marinas , Will Deacon , Peter Zijlstra , Barry Song , Valentin Schneider , "D . Scott Phillips" , Ilkka Koskinen , stable@vger.kernel.org Subject: Re: [PATCH 1/1] arm64: smp: Skip MC sched domain on SoCs with no LLC Message-ID: References: <84e7cb911936f032318b6b376cf88009c90d93d5.1646094455.git.darren@os.amperecomputing.com> Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: CH0PR04CA0103.namprd04.prod.outlook.com (2603:10b6:610:75::18) To SA0PR01MB6329.prod.exchangelabs.com (2603:10b6:806:ee::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f080d145-63f0-4466-5642-08d9fcbc2183 X-MS-TrafficTypeDiagnostic: PH0PR01MB7400:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qHkhN0TbjuzBAzWXk3+CwbTVmaj0fi6+Q+su2GbOBtXOuAweJMmMkpheI4mtwP8ivKkZkYhPE8g6JQ2EKW6L+ZKcMqKCFte2aJJ8xYmJpDus8IzKck433eJ6aNCYzkLkItP2j2/rJqiOovHeeqcf26ugyg+z2LzF4gpkvsUCpxc/A4Eslw18N+wlpcrMU4+PwRe5B5WBxRrsfzmY2JhA7pNTN5S7BYjtjfb7YqwranYCFki5qD9oidtoGqUhHBAEyDVmr72EkvCXzvH9Q5ES6V44SAursYKo9rRIA3374XKeAX6/pDDeNgNQXERhT/6E30W3BJsphL7Akaxk8lH5BZpT9fJ3UeC4cTGjh+ov698x+kvj+gj8x/8E7gl5XXSPCK9eUCo0tNNohDk5FYIX9zcR0xvhpmhJWn5CnokJkAru7ZP/HHqnoLqDceRBQXJD+gPVnHcP+nDw03DNtDYOzk5c2Wuz2YmSPy1LEIBF3FCb0ttL5OU0jfAjrIqrMm9GFAWPXPcv1GY0FvQ/ek7xAyjHTCjX4Z13gNiYXWZcCy/kbKpvS0mLGuz2r22jetC7mtVd9G8JfNQtb6t/iKBLIfBAL2EBWv3jljHtFi1qMVgU6u33XxzUM2NZC0fR72EY84WEEFHxaRPUcIrgTQH7dcy4wF+Sppyl/ppIQHoHaqtYpzbaDnIYLSBv7KGp6lBx+HgBsxOtm/Qynm9HHs2YDw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SA0PR01MB6329.prod.exchangelabs.com; PTR:; CAT:NONE; SFS:(13230001)(6029001)(7916004)(4636009)(366004)(33716001)(508600001)(6506007)(8936002)(52116002)(5660300002)(86362001)(9686003)(6486002)(6512007)(8676002)(4326008)(38350700002)(38100700002)(66946007)(66476007)(66556008)(186003)(26005)(316002)(6916009)(83380400001)(6666004)(54906003)(2906002); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?MeUcsivSrpbu7cy0OLvIjIOf9VYsjYbJGJ4+2HMHoTv9/3kQ6z1czD2jzyGm?= =?us-ascii?Q?rU6BWs0Ss7SxjOtbkNvOUlFzGVhuOztNh5YGFHkEVuP5qwYiE5aW3CzJp1X9?= =?us-ascii?Q?QMcPQx5tVYq2+H0Rh78M0WWeN5fw4KTIbadRUReaxhkjwaC68nbWeoJUEsf5?= =?us-ascii?Q?o1ULeGHVk7BFe5uwRpLWAkFr7W1N+piBR5P7PHVi0ROJ4V/yj/PaBaaeJWp7?= =?us-ascii?Q?Bmeq6AdMkH4+ZVzK0vYLkuu+2iQggJd7auZXNcwY2DlS60UIFTTKHTMKPSgJ?= =?us-ascii?Q?GQoZZUHX8T7eG+D4IRG8q9+3fzWyu4eXhYQpIC30orzeNTsTudN5DtT8v42Y?= =?us-ascii?Q?5EOLvgthTkV/9sRw6G1V7ImRkhP6Qmt7kSRTF9l3gJShYpdvtUh75Xc6/zzY?= =?us-ascii?Q?HS2sH+seQVJMTC2Ko3dvkeMGl6CRhRDC39q+izvkDANGujT7DhsNeidrYAOi?= =?us-ascii?Q?bp34mHMv50/YMEF/9PHpoiyQwYI8cqaNu9QWwLY4CZ2f5z+DhYnFAfvs5FNc?= =?us-ascii?Q?Cr1I5Ac/PaiQGZV4tzGKxwDM9WnziM+0l25L2a0zoM3E9fkejAmmVFUAX9wq?= =?us-ascii?Q?W5xeDIdkN4ivd89ynq8pX1dIQxiesUyvGZxeiU2xhy+dEAJz76YcKSu3+gqE?= =?us-ascii?Q?DZsHk5duFA/HY8Oj+tpRlSBxZZxsirW165VSRdlOJ/pZ5Vq4Yq4Igb9zmN6a?= =?us-ascii?Q?WX61Ac2slAZjeJfzQFNJVm39gKQ1YSgLiiwNPUz6hGsvLmdl2KAMIsBWGIJb?= =?us-ascii?Q?wdYGT9wkFC76eZoG5IjzivA4PrUNie1WF933J3zplH6twf5rkc3Q1hs1Z4Gm?= =?us-ascii?Q?FqKWHLeXVV+nhlBTlY/x90wN50MaoL4Z7pGW/hSSTYnp3GSJctbWyw3AoCq4?= =?us-ascii?Q?3uSME5YJkN7AZFs/XzDTtaXEDknGbbUqPQFlpOsCf5tXq+LeEOjl6Th1GpOS?= =?us-ascii?Q?N4F/X8LBFXrVM7H3RfyIgYGjnlH2EuHmadelR10FIHWIzc0QsMGUxkOvGemx?= =?us-ascii?Q?CJtMfL4z+JvRhoEiZOymDKnqUuXn+sNxwdQkPDpMAEPdL3m0OX8qKsafxYvb?= =?us-ascii?Q?WhuDHuHbv/alvUrYgupXPuTKbyWWtxTbyPHI5GarSmZ/59De2gzhz6w1OEby?= =?us-ascii?Q?fLTT7U7Tbyl04AdLy1QG2CW9LXsRx9b4hAfXXQO6Rn4S6+LXZgVEl7G6ClyL?= =?us-ascii?Q?nOrb0E4wxFyFw8dsPE6J71ZdCwZ6PgMuPozNYBa71Xim9cSuY8FEQiDUYPGf?= =?us-ascii?Q?M7osIkWfauAzQA9481tqAbDzBqLmkDa/rtVA2LrYnJHgfLaHzhfdCvsHlWh+?= =?us-ascii?Q?a59LxOzH1EwUPdND5BtKPKPj2iLqj6mojkYO85vfbRcZZtJjKHIrNAfDTf/A?= =?us-ascii?Q?0lOjjUEOnNGOtZPpMa1KzC97uN3izOyzDfZ0dQJNm5O+NNpd9TtKnb3gwimk?= =?us-ascii?Q?0ylXEvrLQblTA6TvAWb/J6tyo5zuifbh+MBJ57EMrCqBDgfi5lGYJfJ5A45R?= =?us-ascii?Q?M25ylQA51JOzEH1NawnCWrt0zxq9VLi+WkSGi0M2nP+wR/BsKpm95ausvdaP?= =?us-ascii?Q?DpJtXFhMGCz5103eTqKejhN3ECzdOVJIBhoz9pMnd7GEoftE0NKQeaNO24/M?= =?us-ascii?Q?lAjQxvVmSV5bLxsoX5clG/l4pHe6Hg170Vj2K5XjLiAKVZUJc5fR9EcrLwTP?= =?us-ascii?Q?mtSL3bdkNvcgqFgFizkNw1xBZCwCK+L/2kDNXY/pi+vJLb3VJAQQlNfjqQlL?= =?us-ascii?Q?FI++0/HEfA=3D=3D?= X-OriginatorOrg: os.amperecomputing.com X-MS-Exchange-CrossTenant-Network-Message-Id: f080d145-63f0-4466-5642-08d9fcbc2183 X-MS-Exchange-CrossTenant-AuthSource: SA0PR01MB6329.prod.exchangelabs.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Mar 2022 02:18:40.2792 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3bc2b170-fd94-476d-b0ce-4229bdc904a7 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BZErI/d/x++zCwFlrV6R6GyJc+q5yiKh8kCWpsWG9ojNFMCe+Mw44XLYumklsb048oaFPrgvVnwixwu2WKIEL8DIcmCb5fHbnyA9QbBkm3sWG4TMdASLQw81cdDemFJu X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH0PR01MB7400 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220302_181847_719622_F8F553A3 X-CRM114-Status: GOOD ( 34.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Mar 02, 2022 at 10:32:06AM +0100, Vincent Guittot wrote: > On Tue, 1 Mar 2022 at 01:35, Darren Hart wrote: > > > > Ampere Altra defines CPU clusters in the ACPI PPTT. They share a Snoop > > Control Unit, but have no shared CPU-side last level cache. > > > > cpu_coregroup_mask() will return a cpumask with weight 1, while > > cpu_clustergroup_mask() will return a cpumask with weight 2. > > > > As a result, build_sched_domain() will BUG() once per CPU with: > > > > BUG: arch topology borken > > the CLS domain not a subset of the MC domain > > > > The MC level cpumask is then extended to that of the CLS child, and is > > later removed entirely as redundant. This sched domain topology is an > > improvement over previous topologies, or those built without > > SCHED_CLUSTER, particularly for certain latency sensitive workloads. > > With the current scheduler model and heuristics, this is a desirable > > default topology for Ampere Altra and Altra Max system. > > > > Introduce an alternate sched domain topology for arm64 without the MC > > level and test for llc_sibling weight 1 across all CPUs to enable it. > > > > Do this in arch/arm64/kernel/smp.c (as opposed to > > arch/arm64/kernel/topology.c) as all the CPU sibling maps are now > > populated and we avoid needing to extend the drivers/acpi/pptt.c API to > > detect the cluster level being above the cpu llc level. This is > > consistent with other architectures and provides a readily extensible > > mechanism for other alternate topologies. > > > > The final sched domain topology for a 2 socket Ampere Altra system is > > unchanged with or without CONFIG_SCHED_CLUSTER, and the BUG is avoided: > > > > For CPU0: > > > > CONFIG_SCHED_CLUSTER=y > > CLS [0-1] > > DIE [0-79] > > NUMA [0-159] > > > > CONFIG_SCHED_CLUSTER is not set > > DIE [0-79] > > NUMA [0-159] > > > > Cc: Catalin Marinas > > Cc: Will Deacon > > Cc: Peter Zijlstra > > Cc: Vincent Guittot > > Cc: Barry Song > > Cc: Valentin Schneider > > Cc: D. Scott Phillips > > Cc: Ilkka Koskinen > > Cc: # 5.16.x > > Signed-off-by: Darren Hart > > --- > > arch/arm64/kernel/smp.c | 28 ++++++++++++++++++++++++++++ > > 1 file changed, 28 insertions(+) > > > > diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c > > index 27df5c1e6baa..3597e75645e1 100644 > > --- a/arch/arm64/kernel/smp.c > > +++ b/arch/arm64/kernel/smp.c > > @@ -433,6 +433,33 @@ static void __init hyp_mode_check(void) > > } > > } > > > > +static struct sched_domain_topology_level arm64_no_mc_topology[] = { > > +#ifdef CONFIG_SCHED_SMT > > + { cpu_smt_mask, cpu_smt_flags, SD_INIT_NAME(SMT) }, > > +#endif > > + > > +#ifdef CONFIG_SCHED_CLUSTER > > + { cpu_clustergroup_mask, cpu_cluster_flags, SD_INIT_NAME(CLS) }, > > +#endif > > + > > + { cpu_cpu_mask, SD_INIT_NAME(DIE) }, > > + { NULL, }, > > +}; > > + > > +static void __init update_sched_domain_topology(void) > > +{ > > + int cpu; > > + > > + for_each_possible_cpu(cpu) { > > + if (cpu_topology[cpu].llc_id != -1 && > > Have you tested it with a non-acpi system ? AFAICT, llc_id is only set > by ACPI system and llc_id == -1 for others like DT based system > > > + cpumask_weight(&cpu_topology[cpu].llc_sibling) > 1) > > + return; > > + } Hi Vincent, I did not have a non-acpi system to test, no. You're right of course, llc_id is only set by ACPI systems on arm64. We could wrap this in a CONFIG_ACPI ifdef (or IS_ENABLED), but I think this would be preferable: + for_each_possible_cpu(cpu) { + if (cpu_topology[cpu].llc_id == -1 || + cpumask_weight(&cpu_topology[cpu].llc_sibling) > 1) + return; + } Quickly tested on Altra successfully. Would appreciate anyone with non-acpi arm64 systems who can test and verify this behaves as intended. I will ask around tomorrow as well to see what I may have access to. Thanks, > > + > > + pr_info("No LLC siblings, using No MC sched domains topology\n"); > > + set_sched_topology(arm64_no_mc_topology); > > +} > > + > > void __init smp_cpus_done(unsigned int max_cpus) > > { > > pr_info("SMP: Total of %d processors activated.\n", num_online_cpus()); > > @@ -440,6 +467,7 @@ void __init smp_cpus_done(unsigned int max_cpus) > > hyp_mode_check(); > > apply_alternatives_all(); > > mark_linear_text_alias_ro(); > > + update_sched_domain_topology(); > > } > > > > void __init smp_prepare_boot_cpu(void) > > -- > > 2.31.1 > > -- Darren Hart Ampere Computing / OS and Kernel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel