From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 128A0C4363A for ; Thu, 29 Oct 2020 13:38:33 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 81C9320796 for ; Thu, 29 Oct 2020 13:38:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="0vqlgDol" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 81C9320796 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=wjN2ngqNJygySXGtfo1UB4OZFTmM4XlHJUBJ1PCvc40=; b=0vqlgDolThytkLALKyoacIOk6 mwLfCWSYUBdr4QFo1CyvikDe6aiEX3VESWwwhR12DkDxcXPwJITJzYWvYc447R0yyn8t3hTdbTXEw sCbBn+qI/4FbssRQB5yKOICkcaxS7gznTfaA5APk2v36SftV94gMD3QCZqmmYIRR/Fg6P7HUGiWf9 TPIGJqMahj1qtsYZ6kWUr/NvplnWvm+VThyeK/OsjGO5Fdngo9k1astn3TnTNk3KVQAUYQwkohxJ7 M13rZ4ynjNzmCzE4/PHA+0or8WUEQqv8gVTegYqkUyysKCazpLeYyeskmsrqu2UjoHMd4CzVK7AgR e8WZKvLWg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kY87D-0006aK-0C; Thu, 29 Oct 2020 13:37:19 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kY879-0006ZF-V0 for linux-arm-kernel@lists.infradead.org; Thu, 29 Oct 2020 13:37:16 +0000 Received: from gaia (unknown [95.145.162.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9B9E12076E; Thu, 29 Oct 2020 13:37:12 +0000 (UTC) Date: Thu, 29 Oct 2020 13:37:10 +0000 From: Catalin Marinas To: Valentin Schneider Subject: Re: [PATCH] arm64: NUMA: Kconfig: Increase max number of nodes Message-ID: <20201029133709.GE10776@gaia> References: <20201020173409.1266576-1-vanshikonda@os.amperecomputing.com> <9e14c0d3-9204-fb32-31db-5b3f98639cba@arm.com> <20201021110201.00002092@Huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201029_093716_152983_B1D12E1F X-CRM114-Status: GOOD ( 31.34 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anshuman Khandual , linux-kernel@vger.kernel.org, Jonathan Cameron , patches@amperecomputing.com, Vanshidhar Konda , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Oct 21, 2020 at 11:29:41PM +0100, Valentin Schneider wrote: > On 21/10/20 12:02, Jonathan Cameron wrote: > > On Wed, 21 Oct 2020 09:43:21 +0530 > > Anshuman Khandual wrote: > >> Agreed. Do we really need to match X86 right now ? Do we really have > >> systems that has 64 nodes ? We should not increase the default node > >> value and then try to solve some new problems, when there might not > >> be any system which could even use that. I would suggest increase > >> NODES_SHIFT value upto as required by a real and available system. > > > > I'm not going to give precise numbers on near future systems but it is public > > that we ship 8 NUMA node ARM64 systems today. Things will get more > > interesting as CXL and CCIX enter the market on ARM systems, > > given chances are every CXL device will look like another NUMA > > node (CXL spec says they should be presented as such) and you > > may be able to rack up lots of them. > > > > So I'd argue minimum that makes sense today is 16 nodes, but looking forward > > even a little and 64 is not a great stretch. > > I'd make the jump to 64 so we can forget about this again for a year or two. > > People will want to run today's distros on these new machines and we'd > > rather not have to go around all the distros asking them to carry a patch > > increasing this count (I assume they are already carrying such a patch > > due to those 8 node systems) > > I agree that 4 nodes is somewhat anemic; I've had to bump that just to > run some scheduler tests under QEMU. However I still believe we should > exercise caution before cranking it too high, especially when seeing things > like: > > ee38d94a0ad8 ("page flags: prioritize kasan bits over last-cpuid") > > To give some numbers, a defconfig build gives me: > > SECTIONS_WIDTH=0 ZONES_WIDTH=2 NODES_SHIFT=2 LAST_CPUPID_SHIFT=(8+8) KASAN_TAG_WIDTH=0 > BITS_PER_LONG=64 NR_PAGEFLAGS=24 > > IOW, we need 18 + NODES_SHIFT <= 40 -> NODES_SHIFT <= 22. That looks to be > plenty, however this can get cramped fairly easily with any combination of: > > CONFIG_SPARSEMEM_VMEMMAP=n (-18) > CONFIG_IDLE_PAGE_TRACKING=y (-2) > CONFIG_KASAN=y + CONFIG_KASAN_SW_TAGS (-8) > > Taking Arnd's above example, a randconfig build picking !VMEMMAP already > limits the NODES_SHIFT to 4 *if* we want to keep the CPUPID thing within > the flags (it gets a dedicated field at the tail of struct page > otherwise). If that is something we don't care too much about, then > consider my concerns taken care of. I don't think there's any value in allowing SPARSEMEM_VMEMMAP to be disabled but the option is in the core mm/Kconfig file. We could make NODES_SHIFT depend on SPARSEMEM_VMEMMAP (there's DISCONTIGMEM as well but hopefully that's going away soon). > One more thing though: NR_CPUS can be cranked up to 4096 but we've only set > it to 256 IIRC to support the TX2. From that PoV, I'm agreeing with > Anshuman in that we should set it to match the max encountered on platforms > that are in use right now. I agree. Let's bump NODES_SHIFT to 4 now to cover existing platforms. If distros have a 10-year view, they can always ship a kernel configured to 64 nodes, no need to change Kconfig (distros never ship with defconfig). It may have an impact on more memory constrained platforms but that's not what defconfig is about. It should allow existing hardware to run Linux but not necessarily run it in the most efficient way possible. -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel