From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_2 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EE19C4363A for ; Wed, 21 Oct 2020 11:06:13 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7DF5C21741 for ; Wed, 21 Oct 2020 11:06:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="VH2HFAir" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7DF5C21741 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=Huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=Jfqi+8QWkBCYNkFgmXIa10UE80ztcMsGr3IBIl1XBlU=; b=VH2HFAirZDfLmto3sGz3VDhex tsi+K1d7T8Y5HhLv2JnwTgm+QIdXQOIYuRulCfIAgVx2N1Qci22T9QJkkNED/JOw8hhjcmtD2S+II 4bt4c2rgVmtZAdSAuq1pyRW/3uRgx43qPrL5hAegXM30t0EO33XRNJzH1VAus/4Mj1NCuwDNUhs3T iNGb6w9GvIl+QFTqoAegKawfurY9VOXcd7/eS2LU3rgVt4qMzoIaITVS+6DPTXAyVBh0uUqvAZCZi SpdluN8hoakHfslluCnAWkOnR5mSbu6DGnJug7Vs6nz50aexPsKObnAu571/+NEOrTGwD5Mbu559u 0Og0OTuDA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kVBug-0002IX-69; Wed, 21 Oct 2020 11:04:14 +0000 Received: from lhrrgout.huawei.com ([185.176.76.210] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kVBuc-0002Hl-LL for linux-arm-kernel@lists.infradead.org; Wed, 21 Oct 2020 11:04:12 +0000 Received: from lhreml710-chm.china.huawei.com (unknown [172.18.7.108]) by Forcepoint Email with ESMTP id 4D152331F30F320DC73; Wed, 21 Oct 2020 12:03:59 +0100 (IST) Received: from localhost (10.52.124.50) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Wed, 21 Oct 2020 12:03:58 +0100 Date: Wed, 21 Oct 2020 12:02:01 +0100 From: Jonathan Cameron To: Anshuman Khandual Subject: Re: [PATCH] arm64: NUMA: Kconfig: Increase max number of nodes Message-ID: <20201021110201.00002092@Huawei.com> In-Reply-To: <9e14c0d3-9204-fb32-31db-5b3f98639cba@arm.com> References: <20201020173409.1266576-1-vanshikonda@os.amperecomputing.com> <9e14c0d3-9204-fb32-31db-5b3f98639cba@arm.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; i686-w64-mingw32) MIME-Version: 1.0 X-Originating-IP: [10.52.124.50] X-ClientProxiedBy: lhreml727-chm.china.huawei.com (10.201.108.78) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201021_070410_796740_A05A485C X-CRM114-Status: GOOD ( 27.94 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-kernel@vger.kernel.org, patches@amperecomputing.com, Valentin Schneider , linux-arm-kernel@lists.infradead.org, Vanshidhar Konda Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCAyMSBPY3QgMjAyMCAwOTo0MzoyMSArMDUzMApBbnNodW1hbiBLaGFuZHVhbCA8YW5z aHVtYW4ua2hhbmR1YWxAYXJtLmNvbT4gd3JvdGU6Cgo+IE9uIDEwLzIwLzIwMjAgMTE6MzkgUE0s IFZhbGVudGluIFNjaG5laWRlciB3cm90ZToKPiA+IAo+ID4gSGksCj4gPiAKPiA+IE5pdCBvbiB0 aGUgc3ViamVjdDogdGhpcyBvbmx5IGluY3JlYXNlcyB0aGUgZGVmYXVsdCwgdGhlIG1heCBpcyBz dGlsbCAywrnigbAuICAKPiAKPiBBZ3JlZWQuCj4gCj4gPiAKPiA+IE9uIDIwLzEwLzIwIDE4OjM0 LCBWYW5zaGlkaGFyIEtvbmRhIHdyb3RlOiAgCj4gPj4gVGhlIGN1cnJlbnQgYXJtNjQgbWF4IE5V TUEgbm9kZXMgZGVmYXVsdCB0byA0LiBUb2RheSdzIGFybTY0IHN5c3RlbXMgY2FuCj4gPj4gcmVh Y2ggb3IgZXhjZWVkIDE2LiBJbmNyZWFzZSB0aGUgbnVtYmVyIHRvIDY0IChtYXRjaGluZyB4ODZf NjQpLgo+ID4+Cj4gPj4gU2lnbmVkLW9mZi1ieTogVmFuc2hpZGhhciBLb25kYSA8dmFuc2hpa29u ZGFAb3MuYW1wZXJlY29tcHV0aW5nLmNvbT4KPiA+PiAtLS0KPiA+PiAgYXJjaC9hcm02NC9LY29u ZmlnIHwgMiArLQo+ID4+ICAxIGZpbGUgY2hhbmdlZCwgMSBpbnNlcnRpb24oKyksIDEgZGVsZXRp b24oLSkKPiA+Pgo+ID4+IGRpZmYgLS1naXQgYS9hcmNoL2FybTY0L0tjb25maWcgYi9hcmNoL2Fy bTY0L0tjb25maWcKPiA+PiBpbmRleCA4OTMxMzBjZTE2MjYuLjNlNjlkM2M5ODFiZSAxMDA2NDQK PiA+PiAtLS0gYS9hcmNoL2FybTY0L0tjb25maWcKPiA+PiArKysgYi9hcmNoL2FybTY0L0tjb25m aWcKPiA+PiBAQCAtOTgwLDcgKzk4MCw3IEBAIGNvbmZpZyBOVU1BCj4gPj4gIGNvbmZpZyBOT0RF U19TSElGVAo+ID4+ICAgICAgIGludCAiTWF4aW11bSBOVU1BIE5vZGVzIChhcyBhIHBvd2VyIG9m IDIpIgo+ID4+ICAgICAgIHJhbmdlIDEgMTAKPiA+PiAtCWRlZmF1bHQgIjIiCj4gPj4gKwlkZWZh dWx0ICI2IiAgCj4gPiAKPiA+IFRoaXMgbGVhZHMgdG8gbW9yZSBzdGF0aWNhbGx5IGFsbG9jYXRl ZCBtZW1vcnkgZm9yIHRoaW5ncyBsaWtlIG5vZGUgdG8gQ1BVCj4gPiBtYXBzIChzZWUgdXNlcyBv ZiBNQVhfTlVNTk9ERVMpLCBidXQgdGhhdCBzaG91bGRuJ3QgYmUgdG9vIG11Y2ggb2YgYW4KPiA+ IGlzc3VlLiAgCj4gCj4gVGhlIHNtYWxsZXIgc3lzdGVtcyBzaG91bGQgbm90IGJlIHJlcXVpcmVk IHRvIHdhc3RlIHRob3NlIG1lbW9yeSBpbgo+IGEgZGVmYXVsdCBjYXNlLCB1bmxlc3MgdGhlcmUg aXMgYSByZWFsIGFuZCBhdmFpbGFibGUgbGFyZ2VyIHN5c3RlbQo+IHdpdGggdGhvc2UgaW5jcmVh c2VkIG5vZGVzLgo+IAo+ID4gCj4gPiBBSVVJIHRoaXMgYWxzbyBkaXJlY3RseSBjb3JyZWxhdGVz IHRvIGhvdyBtYW55IG1vcmUgcGFnZS0+ZmxhZ3MgYml0cyBhcmUKPiA+IHJlcXVpcmVkOiBhcmUg d2Ugc3VyZSB0aGUgbWF4IDEwIHdvcmtzIG9uIGFueSBhYXJjaDY0IHBsYXRmb3JtPyBJJ20gIAo+ IAo+IFdlIHdpbGwgaGF2ZSB0byB0ZXN0IHRoYXQuIEJlc2lkZXMgMjU2ICgyIF4gOCkgaXMgdGhl IGZpcnN0IHRocmVzaG9sZAo+IHRvIGJlIGNyb3NzZWQgaGVyZS4KPiAKPiA+IGdlbnVpbmVseSBh c2tpbmcgaGVyZSwgZ2l2ZW4gdGhhdCBJJ20gbW9zdGx5IGEgc3RyYW5nZXIgdG8gdGhlIG1tCj4g PiB3b3JsZC4gVGhlIGRlZmF1bHQgc2hvdWxkIGJlIHNvbWV0aGluZyB3ZSdyZSBzb21ld2hhdCBj b25maWRlbnQgd29ya3MKPiA+IGV2ZXJ5d2hlcmUuICAKPiAKPiBBZ3JlZWQuIERvIHdlIHJlYWxs eSBuZWVkIHRvIG1hdGNoIFg4NiByaWdodCBub3cgPyBEbyB3ZSByZWFsbHkgaGF2ZQo+IHN5c3Rl bXMgdGhhdCBoYXMgNjQgbm9kZXMgPyBXZSBzaG91bGQgbm90IGluY3JlYXNlIHRoZSBkZWZhdWx0 IG5vZGUKPiB2YWx1ZSBhbmQgdGhlbiB0cnkgdG8gc29sdmUgc29tZSBuZXcgcHJvYmxlbXMsIHdo ZW4gdGhlcmUgbWlnaHQgbm90Cj4gYmUgYW55IHN5c3RlbSB3aGljaCBjb3VsZCBldmVuIHVzZSB0 aGF0LiBJIHdvdWxkIHN1Z2dlc3QgaW5jcmVhc2UKPiBOT0RFU19TSElGVCB2YWx1ZSB1cHRvIGFz IHJlcXVpcmVkIGJ5IGEgcmVhbCBhbmQgYXZhaWxhYmxlIHN5c3RlbS4KCkknbSBub3QgZ29pbmcg dG8gZ2l2ZSBwcmVjaXNlIG51bWJlcnMgb24gbmVhciBmdXR1cmUgc3lzdGVtcyBidXQgaXQgaXMg cHVibGljCnRoYXQgd2Ugc2hpcCA4IE5VTUEgbm9kZSBBUk02NCBzeXN0ZW1zIHRvZGF5LiAgVGhp bmdzIHdpbGwgZ2V0IG1vcmUKaW50ZXJlc3RpbmcgYXMgQ1hMIGFuZCBDQ0lYIGVudGVyIHRoZSBt YXJrZXQgb24gQVJNIHN5c3RlbXMsCmdpdmVuIGNoYW5jZXMgYXJlIGV2ZXJ5IENYTCBkZXZpY2Ug d2lsbCBsb29rIGxpa2UgYW5vdGhlciBOVU1BCm5vZGUgKENYTCBzcGVjIHNheXMgdGhleSBzaG91 bGQgYmUgcHJlc2VudGVkIGFzIHN1Y2gpIGFuZCB5b3UKbWF5IGJlIGFibGUgdG8gcmFjayB1cCBs b3RzIG9mIHRoZW0uCgpTbyBJJ2QgYXJndWUgbWluaW11bSB0aGF0IG1ha2VzIHNlbnNlIHRvZGF5 IGlzIDE2IG5vZGVzLCBidXQgbG9va2luZyBmb3J3YXJkCmV2ZW4gYSBsaXR0bGUgYW5kIDY0IGlz IG5vdCBhIGdyZWF0IHN0cmV0Y2guCkknZCBtYWtlIHRoZSBqdW1wIHRvIDY0IHNvIHdlIGNhbiBm b3JnZXQgYWJvdXQgdGhpcyBhZ2FpbiBmb3IgYSB5ZWFyIG9yIHR3by4KUGVvcGxlIHdpbGwgd2Fu dCB0byBydW4gdG9kYXkncyBkaXN0cm9zIG9uIHRoZXNlIG5ldyBtYWNoaW5lcyBhbmQgd2UnZApy YXRoZXIgbm90IGhhdmUgdG8gZ28gYXJvdW5kIGFsbCB0aGUgZGlzdHJvcyBhc2tpbmcgdGhlbSB0 byBjYXJyeSBhIHBhdGNoCmluY3JlYXNpbmcgdGhpcyBjb3VudCAoSSBhc3N1bWUgdGhleSBhcmUg YWxyZWFkeSBjYXJyeWluZyBzdWNoIGEgcGF0Y2gKZHVlIHRvIHRob3NlIDggbm9kZSBzeXN0ZW1z KQoKSm9uYXRoYW4KCj4gCj4gPiAgIAo+ID4+ICAgICAgIGRlcGVuZHMgb24gTkVFRF9NVUxUSVBM RV9OT0RFUwo+ID4+ICAgICAgIGhlbHAKPiA+PiAgICAgICAgIFNwZWNpZnkgdGhlIG1heGltdW0g bnVtYmVyIG9mIE5VTUEgTm9kZXMgYXZhaWxhYmxlIG9uIHRoZSB0YXJnZXQgIAo+ID4gIAo+IAo+ IF9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCj4gbGludXgt YXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKPiBsaW51eC1hcm0ta2VybmVsQGxpc3RzLmluZnJhZGVh ZC5vcmcKPiBodHRwOi8vbGlzdHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4 LWFybS1rZXJuZWwKCgoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0 cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGlu Zm8vbGludXgtYXJtLWtlcm5lbAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61004C4363A for ; Wed, 21 Oct 2020 11:04:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1248C21741 for ; Wed, 21 Oct 2020 11:04:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2441859AbgJULEB convert rfc822-to-8bit (ORCPT ); Wed, 21 Oct 2020 07:04:01 -0400 Received: from lhrrgout.huawei.com ([185.176.76.210]:2997 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S2439261AbgJULEA (ORCPT ); Wed, 21 Oct 2020 07:04:00 -0400 Received: from lhreml710-chm.china.huawei.com (unknown [172.18.7.108]) by Forcepoint Email with ESMTP id 4D152331F30F320DC73; Wed, 21 Oct 2020 12:03:59 +0100 (IST) Received: from localhost (10.52.124.50) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Wed, 21 Oct 2020 12:03:58 +0100 Date: Wed, 21 Oct 2020 12:02:01 +0100 From: Jonathan Cameron To: Anshuman Khandual CC: Valentin Schneider , Vanshidhar Konda , , , Subject: Re: [PATCH] arm64: NUMA: Kconfig: Increase max number of nodes Message-ID: <20201021110201.00002092@Huawei.com> In-Reply-To: <9e14c0d3-9204-fb32-31db-5b3f98639cba@arm.com> References: <20201020173409.1266576-1-vanshikonda@os.amperecomputing.com> <9e14c0d3-9204-fb32-31db-5b3f98639cba@arm.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; i686-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8BIT X-Originating-IP: [10.52.124.50] X-ClientProxiedBy: lhreml727-chm.china.huawei.com (10.201.108.78) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, 21 Oct 2020 09:43:21 +0530 Anshuman Khandual wrote: > On 10/20/2020 11:39 PM, Valentin Schneider wrote: > > > > Hi, > > > > Nit on the subject: this only increases the default, the max is still 2¹⁰. > > Agreed. > > > > > On 20/10/20 18:34, Vanshidhar Konda wrote: > >> The current arm64 max NUMA nodes default to 4. Today's arm64 systems can > >> reach or exceed 16. Increase the number to 64 (matching x86_64). > >> > >> Signed-off-by: Vanshidhar Konda > >> --- > >> arch/arm64/Kconfig | 2 +- > >> 1 file changed, 1 insertion(+), 1 deletion(-) > >> > >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > >> index 893130ce1626..3e69d3c981be 100644 > >> --- a/arch/arm64/Kconfig > >> +++ b/arch/arm64/Kconfig > >> @@ -980,7 +980,7 @@ config NUMA > >> config NODES_SHIFT > >> int "Maximum NUMA Nodes (as a power of 2)" > >> range 1 10 > >> - default "2" > >> + default "6" > > > > This leads to more statically allocated memory for things like node to CPU > > maps (see uses of MAX_NUMNODES), but that shouldn't be too much of an > > issue. > > The smaller systems should not be required to waste those memory in > a default case, unless there is a real and available larger system > with those increased nodes. > > > > > AIUI this also directly correlates to how many more page->flags bits are > > required: are we sure the max 10 works on any aarch64 platform? I'm > > We will have to test that. Besides 256 (2 ^ 8) is the first threshold > to be crossed here. > > > genuinely asking here, given that I'm mostly a stranger to the mm > > world. The default should be something we're somewhat confident works > > everywhere. > > Agreed. Do we really need to match X86 right now ? Do we really have > systems that has 64 nodes ? We should not increase the default node > value and then try to solve some new problems, when there might not > be any system which could even use that. I would suggest increase > NODES_SHIFT value upto as required by a real and available system. I'm not going to give precise numbers on near future systems but it is public that we ship 8 NUMA node ARM64 systems today. Things will get more interesting as CXL and CCIX enter the market on ARM systems, given chances are every CXL device will look like another NUMA node (CXL spec says they should be presented as such) and you may be able to rack up lots of them. So I'd argue minimum that makes sense today is 16 nodes, but looking forward even a little and 64 is not a great stretch. I'd make the jump to 64 so we can forget about this again for a year or two. People will want to run today's distros on these new machines and we'd rather not have to go around all the distros asking them to carry a patch increasing this count (I assume they are already carrying such a patch due to those 8 node systems) Jonathan > > > > >> depends on NEED_MULTIPLE_NODES > >> help > >> Specify the maximum number of NUMA Nodes available on the target > > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel