From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51215C4363A for ; Wed, 21 Oct 2020 22:31:50 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C8A4A241A6 for ; Wed, 21 Oct 2020 22:31:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="PlZdsQBg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C8A4A241A6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:In-reply-to:Subject:To: From:References:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=PALomNOuJ2VG0YPernSCDAyftt5ktW0VxNwYCXW1EQo=; b=PlZdsQBgAYljBdXONfNi1vlrQ QNaCpwFkzTyusJkECyTdWxy34tfoc/FV/XzOe/OL03A3E72IBtyK5u40s7COHJjAVaV+nT7IK9+/4 NX+N3c1l9AYjqxsHmLig8wLvRxvUZ2YgOLbH3YOLCUt40RRuOfQZUtWTSkenPBALltJEdCMWeDx1P IFahCNs0MNlanE5mXLYdipn7Fn++xh0lZCKtiO4GMTMoluUQFiII5SHyhcAktHv/kjdoB7wTRRaGj XysSSBPf+Y5bHTyAo3fZcOM+NFZqxnS0kKeKRLzkVW85T8McsBZEoxsir7fDl2IndDUaWnrP3yuC0 4KkGmRM0A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kVMcD-0006Ve-Bk; Wed, 21 Oct 2020 22:29:53 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kVMcA-0006V9-Ov for linux-arm-kernel@lists.infradead.org; Wed, 21 Oct 2020 22:29:51 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 85DFF31B; Wed, 21 Oct 2020 15:29:47 -0700 (PDT) Received: from e113632-lin (e113632-lin.cambridge.arm.com [10.1.194.46]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9CED33F66B; Wed, 21 Oct 2020 15:29:46 -0700 (PDT) References: <20201020173409.1266576-1-vanshikonda@os.amperecomputing.com> <9e14c0d3-9204-fb32-31db-5b3f98639cba@arm.com> <20201021110201.00002092@Huawei.com> User-agent: mu4e 0.9.17; emacs 26.3 From: Valentin Schneider To: Jonathan Cameron Subject: Re: [PATCH] arm64: NUMA: Kconfig: Increase max number of nodes In-reply-to: <20201021110201.00002092@Huawei.com> Date: Wed, 21 Oct 2020 23:29:41 +0100 Message-ID: MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201021_182950_987787_D2B0C892 X-CRM114-Status: GOOD ( 22.87 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-kernel@vger.kernel.org, patches@amperecomputing.com, Vanshidhar Konda , linux-arm-kernel@lists.infradead.org, Anshuman Khandual Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, On 21/10/20 12:02, Jonathan Cameron wrote: > On Wed, 21 Oct 2020 09:43:21 +0530 > Anshuman Khandual wrote: >> >> Agreed. Do we really need to match X86 right now ? Do we really have >> systems that has 64 nodes ? We should not increase the default node >> value and then try to solve some new problems, when there might not >> be any system which could even use that. I would suggest increase >> NODES_SHIFT value upto as required by a real and available system. > > I'm not going to give precise numbers on near future systems but it is public > that we ship 8 NUMA node ARM64 systems today. Things will get more > interesting as CXL and CCIX enter the market on ARM systems, > given chances are every CXL device will look like another NUMA > node (CXL spec says they should be presented as such) and you > may be able to rack up lots of them. > > So I'd argue minimum that makes sense today is 16 nodes, but looking forward > even a little and 64 is not a great stretch. > I'd make the jump to 64 so we can forget about this again for a year or two. > People will want to run today's distros on these new machines and we'd > rather not have to go around all the distros asking them to carry a patch > increasing this count (I assume they are already carrying such a patch > due to those 8 node systems) > I agree that 4 nodes is somewhat anemic; I've had to bump that just to run some scheduler tests under QEMU. However I still believe we should exercise caution before cranking it too high, especially when seeing things like: ee38d94a0ad8 ("page flags: prioritize kasan bits over last-cpuid") To give some numbers, a defconfig build gives me: SECTIONS_WIDTH=0 ZONES_WIDTH=2 NODES_SHIFT=2 LAST_CPUPID_SHIFT=(8+8) KASAN_TAG_WIDTH=0 BITS_PER_LONG=64 NR_PAGEFLAGS=24 IOW, we need 18 + NODES_SHIFT <= 40 -> NODES_SHIFT <= 22. That looks to be plenty, however this can get cramped fairly easily with any combination of: CONFIG_SPARSEMEM_VMEMMAP=n (-18) CONFIG_IDLE_PAGE_TRACKING=y (-2) CONFIG_KASAN=y + CONFIG_KASAN_SW_TAGS (-8) Taking Arnd's above example, a randconfig build picking !VMEMMAP already limits the NODES_SHIFT to 4 *if* we want to keep the CPUPID thing within the flags (it gets a dedicated field at the tail of struct page otherwise). If that is something we don't care too much about, then consider my concerns taken care of. One more thing though: NR_CPUS can be cranked up to 4096 but we've only set it to 256 IIRC to support the TX2. From that PoV, I'm agreeing with Anshuman in that we should set it to match the max encountered on platforms that are in use right now. > Jonathan > >> >> > >> >> depends on NEED_MULTIPLE_NODES >> >> help >> >> Specify the maximum number of NUMA Nodes available on the target >> > >> >> _______________________________________________ >> linux-arm-kernel mailing list >> linux-arm-kernel@lists.infradead.org >> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel