From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E011FC54E49 for ; Thu, 7 Mar 2024 17:49:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Cg1yhMA9emH+adIZ+R3wfh5IiG6qSIJBnYWCtrkogjk=; b=SiM5qMib4+lG0I VdevBy2pK+DYStQtZG611h0umLHK7xzCGn7vNpqDBPtTRWKAAiM0qVCOVTra8Gr/wQbJhD0+wLxDK J+MkvivQfq7hLIN+j79/i9m9xqwg2/ZxLafnkVNYK1Rr61mSjM8rXDkzREAQbQ4H0ihCNWyVuuYWu qN2Bn7i+XyohaEv7rXK2N+dTbQKH/QVhtadajmH/+0qIyFBIwusEMc5aS2xuyq0Jsw2D7Kb5bxi71 fRF93wP5MNgPYRlGUxckmqeNl1WlfHAUqiNlSXHJxGoMIHlPkrx8qU3q0PDQScNpgQw6hYiBhB6iv R4fnZPD2IcQkpVo0msDw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1riHs0-00000005lZJ-2NPD; Thu, 07 Mar 2024 17:49:28 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1riHrw-00000005lXN-0StL for linux-arm-kernel@lists.infradead.org; Thu, 07 Mar 2024 17:49:26 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BCCF11FB; Thu, 7 Mar 2024 09:49:55 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.69.155]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D7D073F762; Thu, 7 Mar 2024 09:49:15 -0800 (PST) Date: Thu, 7 Mar 2024 17:49:12 +0000 From: Mark Rutland To: "Christoph Lameter (Ampere)" , catalin.marinas@arm.com Cc: Will Deacon , Jonathan.Cameron@huawei.com, Matteo.Carlini@arm.com, Valentin.Schneider@arm.com, akpm@linux-foundation.org, anshuman.khandual@arm.com, Eric Mackay , dave.kleikamp@oracle.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux@armlinux.org.uk, robin.murphy@arm.com, vanshikonda@os.amperecomputing.com, yang@os.amperecomputing.com Subject: Re: [PATCH v3] ARM64: Dynamically allocate cpumasks and increase supported CPUs to 512 Message-ID: References: <37099a57-b655-3b3a-56d0-5f7fbd49d7db@gentwo.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <37099a57-b655-3b3a-56d0-5f7fbd49d7db@gentwo.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240307_094924_430357_FBB535F5 X-CRM114-Status: GOOD ( 34.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi Christoph, On Wed, Mar 06, 2024 at 05:45:04PM -0800, Christoph Lameter (Ampere) wrote: > Currently defconfig selects NR_CPUS=256, but some vendors (e.g. Ampere > Computing) are planning to ship systems with 512 CPUs. So that all CPUs on > these systems can be used with defconfig, we'd like to bump NR_CPUS to 512. > Therefore this patch increases the default NR_CPUS from 256 to 512. > > As increasing NR_CPUS will increase the size of cpumasks, there's a fear that > this might have a significant impact on stack usage due to code which places > cpumasks on the stack. To mitigate that concern, we can select > CPUMASK_OFFSTACK. As that doesn't seem to be a problem today with > NR_CPUS=256, we only select this when NR_CPUS > 256. > > CPUMASK_OFFSTACK configures the cpumasks in the kernel to be > dynamically allocated. This was used in the X86 architecture in the > past to enable support for larger CPU configurations up to 8k cpus. > > With that is becomes possible to dynamically size the allocation of > the cpu bitmaps depending on the quantity of processors detected on > bootup. Memory used for cpumasks will increase if the kernel is > run on a machine with more cores. > > Further increases may be needed if ARM processor vendors start > supporting more processors. Given the current inflationary trends > in core counts from multiple processor manufacturers this may occur. > > There are minor regressions for hackbench. The kernel data size > for 512 cpus is smaller with offstack than with onstack. > > Benchmark results using hackbench average over 10 runs of > > hackbench -s 512 -l 2000 -g 15 -f 25 -P > > on Altra 80 Core > > Support for 256 CPUs on stack. Baseline > > 7.8564 sec > > Support for 512 CUs on stack. > > 7.8713 sec + 0.18% > > 512 CPUS offstack > > 7.8916 sec + 0.44% > > Kernel size comparison: > > text data filename Difference to onstack256 baseline > 25755648 9589248 vmlinuz-6.8.0-rc4-onstack256 > 25755648 9607680 vmlinuz-6.8.0-rc4-onstack512 +0.19% > 25755648 9603584 vmlinuz-6.8.0-rc4-offstack512 +0.14% Thanks for this data; I think that's a strong justification that this isn't likely to cause a big problem for us, and so I thing it makes sense to go with this. I have two minor comments below. > Tested-by: Eric Mackay > Reviewed-by: Russell King (Oracle) > Signed-off-by: Christoph Lameter (Ampere) > --- > > > Original post: https://www.spinics.net/lists/linux-mm/msg369701.html > V2: https://lkml.org/lkml/2024/2/7/505 > > > V1->V2 > > - Keep quotation marks > - Remove whiltespace damage > - Add tested by > > V2->V3: > - Add test results > - Rework descriptions > > > arch/arm64/Kconfig | 16 +++++++++++++++- > 1 file changed, 15 insertions(+), 1 deletion(-) > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index aa7c1d435139..4e767dede47d 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -1427,7 +1427,21 @@ config SCHED_SMT > config NR_CPUS > int "Maximum number of CPUs (2-4096)" > range 2 4096 > - default "256" > + default "512" > + > +# > +# Determines the placement of cpumasks. > +# > +# With CPUMASK_OFFSTACK the cpumasks are dynamically allocated. > +# Useful for machines with lots of core because it avoids increasing > +# the size of many of the data structures in the kernel. > +# > +# If this is off then the cpumasks have a static sizes and are > +# embedded within data structures. > +# > + config CPUMASK_OFFSTACK > + def_bool y > + depends on NR_CPUS > 256 As before, can we please delete the comment? That's the general semantic of CPUMASK_OFFSTACK, not why we're selecting it. That aside, this config option is defined in lib/Kconfig, so we should select it rather than redefining it. i.e. this should be: select CPUMASK_OFFSTACK if NR_CPUS > 256 Sorry for not spotting that before. With those changes: Acked-by: Mark Rutland Catalin, are you happy to fix that up when applying? Mark. > > config HOTPLUG_CPU > bool "Support for hot-pluggable CPUs" > -- > 2.39.2 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel