From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C7C8C4363A for ; Fri, 30 Oct 2020 10:21:57 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DF8E220825 for ; Fri, 30 Oct 2020 10:21:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="XLVGcjlb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DF8E220825 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Iyk6+m+P+0xZovpbux87/+RdwX+5u4+3V5zNVxhZIm4=; b=XLVGcjlbrLDO62RQ41rtVdTJf V+BWWH8dijXclD8CJPy+g8YUpsH5gay8DaMdc2UcGZLFj40C5KJn9GcE5LUJPjSh0f/mAardzyBIL 1imc4+CqoLKW9KL4f/Qf1peoRms1U0TNhsDFWYfYToCEzSk8RtND3kv24Qxacrri+dDKSfIHZRPxK cX4Ljbf/FeLKNISult3hzHAcR/3BZg0+6zbCWBZcNM8X3uZO5ZyadBlK+I1jJrCgivtWH+CaQI0fw Ql9XM5xGnNgLlXl884iepnKSY0ZIEqdGQFvRZGsYDLHW5I08GS6jGPG5Q31mEEfURzhBr40xTiZnl 5v+edBXwg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kYRX6-0004GI-Fc; Fri, 30 Oct 2020 10:21:20 +0000 Received: from mail.kernel.org ([198.145.29.99]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kYRX0-0004Dq-Lu for linux-arm-kernel@lists.infradead.org; Fri, 30 Oct 2020 10:21:19 +0000 Received: from gaia (unknown [95.145.162.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id CAE0520825; Fri, 30 Oct 2020 10:21:10 +0000 (UTC) Date: Fri, 30 Oct 2020 10:21:08 +0000 From: Catalin Marinas To: Vanshidhar Konda Subject: Re: [PATCH] arm64: NUMA: Kconfig: Increase max number of nodes Message-ID: <20201030102107.GA23196@gaia> References: <20201020173409.1266576-1-vanshikonda@os.amperecomputing.com> <9e14c0d3-9204-fb32-31db-5b3f98639cba@arm.com> <20201021110201.00002092@Huawei.com> <20201029133709.GE10776@gaia> <20201029194850.jon6lxlbhquy7zql@con01sys-r111.scc-lab.amperecomputing.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20201029194850.jon6lxlbhquy7zql@con01sys-r111.scc-lab.amperecomputing.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201030_062114_941681_390D5EC4 X-CRM114-Status: GOOD ( 42.40 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Anshuman Khandual , linux-kernel@vger.kernel.org, Jonathan Cameron , patches@amperecomputing.com, Will Deacon , Valentin Schneider , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Oct 29, 2020 at 12:48:50PM -0700, Vanshidhar Konda wrote: > On Thu, Oct 29, 2020 at 01:37:10PM +0000, Catalin Marinas wrote: > > On Wed, Oct 21, 2020 at 11:29:41PM +0100, Valentin Schneider wrote: > > > On 21/10/20 12:02, Jonathan Cameron wrote: > > > > On Wed, 21 Oct 2020 09:43:21 +0530 > > > > Anshuman Khandual wrote: > > > >> Agreed. Do we really need to match X86 right now ? Do we really have > > > >> systems that has 64 nodes ? We should not increase the default node > > > >> value and then try to solve some new problems, when there might not > > > >> be any system which could even use that. I would suggest increase > > > >> NODES_SHIFT value upto as required by a real and available system. > > > > > > > > I'm not going to give precise numbers on near future systems but it is public > > > > that we ship 8 NUMA node ARM64 systems today. Things will get more > > > > interesting as CXL and CCIX enter the market on ARM systems, > > > > given chances are every CXL device will look like another NUMA > > > > node (CXL spec says they should be presented as such) and you > > > > may be able to rack up lots of them. > > > > > > > > So I'd argue minimum that makes sense today is 16 nodes, but looking forward > > > > even a little and 64 is not a great stretch. > > > > I'd make the jump to 64 so we can forget about this again for a year or two. > > > > People will want to run today's distros on these new machines and we'd > > > > rather not have to go around all the distros asking them to carry a patch > > > > increasing this count (I assume they are already carrying such a patch > > > > due to those 8 node systems) > > > > > > I agree that 4 nodes is somewhat anemic; I've had to bump that just to > > > run some scheduler tests under QEMU. However I still believe we should > > > exercise caution before cranking it too high, especially when seeing things > > > like: > > > > > > ee38d94a0ad8 ("page flags: prioritize kasan bits over last-cpuid") > > > > > > To give some numbers, a defconfig build gives me: > > > > > > SECTIONS_WIDTH=0 ZONES_WIDTH=2 NODES_SHIFT=2 LAST_CPUPID_SHIFT=(8+8) KASAN_TAG_WIDTH=0 > > > BITS_PER_LONG=64 NR_PAGEFLAGS=24 > > > > > > IOW, we need 18 + NODES_SHIFT <= 40 -> NODES_SHIFT <= 22. That looks to be > > > plenty, however this can get cramped fairly easily with any combination of: > > > > > > CONFIG_SPARSEMEM_VMEMMAP=n (-18) > > > CONFIG_IDLE_PAGE_TRACKING=y (-2) > > > CONFIG_KASAN=y + CONFIG_KASAN_SW_TAGS (-8) > > > > > > Taking Arnd's above example, a randconfig build picking !VMEMMAP already > > > limits the NODES_SHIFT to 4 *if* we want to keep the CPUPID thing within > > > the flags (it gets a dedicated field at the tail of struct page > > > otherwise). If that is something we don't care too much about, then > > > consider my concerns taken care of. > > > > I don't think there's any value in allowing SPARSEMEM_VMEMMAP to be > > disabled but the option is in the core mm/Kconfig file. We could make > > NODES_SHIFT depend on SPARSEMEM_VMEMMAP (there's DISCONTIGMEM as well > > but hopefully that's going away soon). > > > > > One more thing though: NR_CPUS can be cranked up to 4096 but we've only set > > > it to 256 IIRC to support the TX2. From that PoV, I'm agreeing with > > > Anshuman in that we should set it to match the max encountered on platforms > > > that are in use right now. > > > > I agree. Let's bump NODES_SHIFT to 4 now to cover existing platforms. If > > distros have a 10-year view, they can always ship a kernel configured to > > 64 nodes, no need to change Kconfig (distros never ship with defconfig). > > > > It may have an impact on more memory constrained platforms but that's > > not what defconfig is about. It should allow existing hardware to run > > Linux but not necessarily run it in the most efficient way possible. > > > > From the discussion it looks like 4 is an acceptable number to support > current hardware. I'll send a patch with NODES_SHIFT set to 4. Is it still > possible to add this change to the 5.10 kernel? I think we can but I'll leave the decision to Will (and don't forget to cc the arm64 maintainers on your next post). -- Catalin _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel