From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A218CC433E7 for ; Mon, 19 Oct 2020 10:01:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 617E822263 for ; Mon, 19 Oct 2020 10:01:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728212AbgJSKBg (ORCPT ); Mon, 19 Oct 2020 06:01:36 -0400 Received: from foss.arm.com ([217.140.110.172]:53888 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727192AbgJSKBg (ORCPT ); Mon, 19 Oct 2020 06:01:36 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2BCF030E; Mon, 19 Oct 2020 03:01:36 -0700 (PDT) Received: from bogus (unknown [10.57.13.246]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 412C33F66E; Mon, 19 Oct 2020 03:01:34 -0700 (PDT) Date: Mon, 19 Oct 2020 11:01:00 +0100 From: Sudeep Holla To: Jonathan Cameron Cc: linux-acpi@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, x86@kernel.org, Len Brown , Sudeep Holla , Morten Rasmussen , Greg Kroah-Hartman , guohanjun@huawei.com, Will Deacon , linuxarm@huawei.com, Brice Goglin Subject: Re: [RFC PATCH] topology: Represent clusters of CPUs within a die. Message-ID: <20201019100100.GA12908@bogus> References: <20201016152702.1513592-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20201016152702.1513592-1-Jonathan.Cameron@huawei.com> User-Agent: Mutt/1.9.4 (2018-02-28) Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org +Morten On Fri, Oct 16, 2020 at 11:27:02PM +0800, Jonathan Cameron wrote: > Both ACPI and DT provide the ability to describe additional layers of > topology between that of individual cores and higher level constructs > such as the level at which the last level cache is shared. > In ACPI this can be represented in PPTT as a Processor Hierarchy > Node Structure [1] that is the parent of the CPU cores and in turn > has a parent Processor Hierarchy Nodes Structure representing > a higher level of topology. > > For example Kunpeng 920 has clusters of 4 CPUs. These do not share > any cache resources, but the interconnect topology is such that > the cost to transfer ownership of a cacheline between CPUs within > a cluster is lower than between CPUs in different clusters on the same > die. Hence, it can make sense to deliberately schedule threads > sharing data to a single cluster. > This is very SoC specific and hard to generalise, hence LLC is chosen as one of the main factor to decide. Are there any scheduler topology related changes to achieve the same ? If so, we need their opinion on that. > This patch simply exposes this information to userspace libraries > like hwloc by providing cluster_cpus and related sysfs attributes. > PoC of HWLOC support at [2]. > OK, cluster is too Arm specific with no definition for it whatsoever. How do you plan to support clusters of clusters or higher level of hierarchy present in PPTT. > Note this patch only handle the ACPI case. > If we decide to add this to sysfs, I prefer to keep DT implementation also in sync. The bindings are in sync, just matter of implementation. > Special consideration is needed for SMT processors, where it is > necessary to move 2 levels up the hierarchy from the leaf nodes > (thus skipping the processor core level). > > Currently the ID provided is the offset of the Processor > Hierarchy Nodes Structure within PPTT. Whilst this is unique > it is not terribly elegant so alternative suggestions welcome. > That is firmware choice. May be your firmware just fills in mandatory fields and doesn't care about anything and everything optional. We do check for valid UID fields and if that is not set, offset is used as unique ID in kernel implementation. So if you enhance the firmware, the kernel sysfs will become elegant as you expect 😉 > Note that arm64 / ACPI does not provide any means of identifying > a die level in the topology but that may be unrelate to the cluster > level. > Indeed. It can be cluster of clusters on some platform. If we need that info, we should add it. My assumption was that generally each die forms a NUMA node and hence the information is available there. I may be wrong. > RFC questions: > 1) Naming > 2) Related to naming, do we want to represent all potential levels, > or this enough? On Kunpeng920, the next level up from cluster happens > to be covered by llc cache sharing, but in theory more than one > level of cluster description might be needed by some future system. That is my question above. Can't recall the terminology used in ACPI PPTT, but IIRC "cluster" is not used to keep it generic. May be we need to do the same here as the term "cluster" is ill-defined on Arm and I would avoid using it if possible. > 3) Do we need DT code in place? I'm not sure any DT based ARM64 > systems would have enough complexity for this to be useful. I prefer to keep the implementation in sync > 4) Other architectures? Is this useful on x86 for example? > AMD had multi-die within socket IIRC. IIUC, cpuid will provide the info and nothing more is needed from ACPI ? I may be wrong, just guessing/asking. -- Regards, Sudeep From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D64FC433E7 for ; Mon, 19 Oct 2020 10:02:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9F712076A for ; Mon, 19 Oct 2020 10:02:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="o4pNfeLa" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9F712076A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=UTYlBMdNq0D9ZA2f8bsiL8Wt8pYIB4H4zrSidmOzEPU=; b=o4pNfeLa3T/c7nPPaiOiRqPBn v1HDM4gG3sHb/aNJTSTqqgnGSzMT18IDOBNIN/piGvIRRrEKqJdFaiQxaxzjNaxMBQpe/74iL7EXk s/Y9MaP1uolMD8YvP1X8TEU6EwclAaw0TvzAzQlij32D7idnuHbNlsn+apDZtyGWnVIo++S4tHDiZ hd7/DCgKLfEkKwryusyUAmdSI4OyQYNivmJh67/m3KRyfXBaNGvec8dTkGKAJH8G+nhHkKfK50nSB JPpBnR3xX9z+SdyxpU1D5q4QBJ9yp+3N7jCs4iKIZXyZjMyxMk2kKd8eD6uHhDQXo4B91F5uyt8XU tXnrgkEvA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kURz2-0000Hj-Ke; Mon, 19 Oct 2020 10:01:40 +0000 Received: from foss.arm.com ([217.140.110.172]) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kURz0-0000H7-S5 for linux-arm-kernel@lists.infradead.org; Mon, 19 Oct 2020 10:01:39 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2BCF030E; Mon, 19 Oct 2020 03:01:36 -0700 (PDT) Received: from bogus (unknown [10.57.13.246]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 412C33F66E; Mon, 19 Oct 2020 03:01:34 -0700 (PDT) Date: Mon, 19 Oct 2020 11:01:00 +0100 From: Sudeep Holla To: Jonathan Cameron Subject: Re: [RFC PATCH] topology: Represent clusters of CPUs within a die. Message-ID: <20201019100100.GA12908@bogus> References: <20201016152702.1513592-1-Jonathan.Cameron@huawei.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20201016152702.1513592-1-Jonathan.Cameron@huawei.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201019_060138_992174_718DC4D5 X-CRM114-Status: GOOD ( 26.33 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Len Brown , Greg Kroah-Hartman , x86@kernel.org, guohanjun@huawei.com, linux-kernel@vger.kernel.org, linuxarm@huawei.com, Brice Goglin , linux-acpi@vger.kernel.org, Sudeep Holla , Will Deacon , Morten Rasmussen , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org K01vcnRlbgoKT24gRnJpLCBPY3QgMTYsIDIwMjAgYXQgMTE6Mjc6MDJQTSArMDgwMCwgSm9uYXRo YW4gQ2FtZXJvbiB3cm90ZToKPiBCb3RoIEFDUEkgYW5kIERUIHByb3ZpZGUgdGhlIGFiaWxpdHkg dG8gZGVzY3JpYmUgYWRkaXRpb25hbCBsYXllcnMgb2YKPiB0b3BvbG9neSBiZXR3ZWVuIHRoYXQg b2YgaW5kaXZpZHVhbCBjb3JlcyBhbmQgaGlnaGVyIGxldmVsIGNvbnN0cnVjdHMKPiBzdWNoIGFz IHRoZSBsZXZlbCBhdCB3aGljaCB0aGUgbGFzdCBsZXZlbCBjYWNoZSBpcyBzaGFyZWQuCj4gSW4g QUNQSSB0aGlzIGNhbiBiZSByZXByZXNlbnRlZCBpbiBQUFRUIGFzIGEgUHJvY2Vzc29yIEhpZXJh cmNoeQo+IE5vZGUgU3RydWN0dXJlIFsxXSB0aGF0IGlzIHRoZSBwYXJlbnQgb2YgdGhlIENQVSBj b3JlcyBhbmQgaW4gdHVybgo+IGhhcyBhIHBhcmVudCBQcm9jZXNzb3IgSGllcmFyY2h5IE5vZGVz IFN0cnVjdHVyZSByZXByZXNlbnRpbmcKPiBhIGhpZ2hlciBsZXZlbCBvZiB0b3BvbG9neS4KPgo+ IEZvciBleGFtcGxlIEt1bnBlbmcgOTIwIGhhcyBjbHVzdGVycyBvZiA0IENQVXMuICBUaGVzZSBk byBub3Qgc2hhcmUKPiBhbnkgY2FjaGUgcmVzb3VyY2VzLCBidXQgdGhlIGludGVyY29ubmVjdCB0 b3BvbG9neSBpcyBzdWNoIHRoYXQKPiB0aGUgY29zdCB0byB0cmFuc2ZlciBvd25lcnNoaXAgb2Yg YSBjYWNoZWxpbmUgYmV0d2VlbiBDUFVzIHdpdGhpbgo+IGEgY2x1c3RlciBpcyBsb3dlciB0aGFu IGJldHdlZW4gQ1BVcyBpbiBkaWZmZXJlbnQgY2x1c3RlcnMgb24gdGhlIHNhbWUKPiBkaWUuICAg SGVuY2UsIGl0IGNhbiBtYWtlIHNlbnNlIHRvIGRlbGliZXJhdGVseSBzY2hlZHVsZSB0aHJlYWRz Cj4gc2hhcmluZyBkYXRhIHRvIGEgc2luZ2xlIGNsdXN0ZXIuCj4KClRoaXMgaXMgdmVyeSBTb0Mg c3BlY2lmaWMgYW5kIGhhcmQgdG8gZ2VuZXJhbGlzZSwgaGVuY2UgTExDIGlzIGNob3NlbgphcyBv bmUgb2YgdGhlIG1haW4gZmFjdG9yIHRvIGRlY2lkZS4KCkFyZSB0aGVyZSBhbnkgc2NoZWR1bGVy IHRvcG9sb2d5IHJlbGF0ZWQgY2hhbmdlcyB0byBhY2hpZXZlIHRoZSBzYW1lID8KSWYgc28sIHdl IG5lZWQgdGhlaXIgb3BpbmlvbiBvbiB0aGF0LgoKPiBUaGlzIHBhdGNoIHNpbXBseSBleHBvc2Vz IHRoaXMgaW5mb3JtYXRpb24gdG8gdXNlcnNwYWNlIGxpYnJhcmllcwo+IGxpa2UgaHdsb2MgYnkg cHJvdmlkaW5nIGNsdXN0ZXJfY3B1cyBhbmQgcmVsYXRlZCBzeXNmcyBhdHRyaWJ1dGVzLgo+IFBv QyBvZiBIV0xPQyBzdXBwb3J0IGF0IFsyXS4KPgoKT0ssIGNsdXN0ZXIgaXMgdG9vIEFybSBzcGVj aWZpYyB3aXRoIG5vIGRlZmluaXRpb24gZm9yIGl0IHdoYXRzb2V2ZXIuCkhvdyBkbyB5b3UgcGxh biB0byBzdXBwb3J0IGNsdXN0ZXJzIG9mIGNsdXN0ZXJzIG9yIGhpZ2hlciBsZXZlbCBvZgpoaWVy YXJjaHkgcHJlc2VudCBpbiBQUFRULgoKPiBOb3RlIHRoaXMgcGF0Y2ggb25seSBoYW5kbGUgdGhl IEFDUEkgY2FzZS4KPgoKSWYgd2UgZGVjaWRlIHRvIGFkZCB0aGlzIHRvIHN5c2ZzLCBJIHByZWZl ciB0byBrZWVwIERUIGltcGxlbWVudGF0aW9uCmFsc28gaW4gc3luYy4gVGhlIGJpbmRpbmdzIGFy ZSBpbiBzeW5jLCBqdXN0IG1hdHRlciBvZiBpbXBsZW1lbnRhdGlvbi4KCj4gU3BlY2lhbCBjb25z aWRlcmF0aW9uIGlzIG5lZWRlZCBmb3IgU01UIHByb2Nlc3NvcnMsIHdoZXJlIGl0IGlzCj4gbmVj ZXNzYXJ5IHRvIG1vdmUgMiBsZXZlbHMgdXAgdGhlIGhpZXJhcmNoeSBmcm9tIHRoZSBsZWFmIG5v ZGVzCj4gKHRodXMgc2tpcHBpbmcgdGhlIHByb2Nlc3NvciBjb3JlIGxldmVsKS4KPgo+IEN1cnJl bnRseSB0aGUgSUQgcHJvdmlkZWQgaXMgdGhlIG9mZnNldCBvZiB0aGUgUHJvY2Vzc29yCj4gSGll cmFyY2h5IE5vZGVzIFN0cnVjdHVyZSB3aXRoaW4gUFBUVC4gIFdoaWxzdCB0aGlzIGlzIHVuaXF1 ZQo+IGl0IGlzIG5vdCB0ZXJyaWJseSBlbGVnYW50IHNvIGFsdGVybmF0aXZlIHN1Z2dlc3Rpb25z IHdlbGNvbWUuCj4KClRoYXQgaXMgZmlybXdhcmUgY2hvaWNlLiBNYXkgYmUgeW91ciBmaXJtd2Fy ZSBqdXN0IGZpbGxzIGluIG1hbmRhdG9yeQpmaWVsZHMgYW5kIGRvZXNuJ3QgY2FyZSBhYm91dCBh bnl0aGluZyBhbmQgZXZlcnl0aGluZyBvcHRpb25hbC4gV2UgZG8KY2hlY2sgZm9yIHZhbGlkIFVJ RCBmaWVsZHMgYW5kIGlmIHRoYXQgaXMgbm90IHNldCwgb2Zmc2V0IGlzIHVzZWQgYXMKdW5pcXVl IElEIGluIGtlcm5lbCBpbXBsZW1lbnRhdGlvbi4gU28gaWYgeW91IGVuaGFuY2UgdGhlIGZpcm13 YXJlLCB0aGUKa2VybmVsIHN5c2ZzIHdpbGwgYmVjb21lIGVsZWdhbnQgYXMgeW91IGV4cGVjdCDw n5iJIAoKPiBOb3RlIHRoYXQgYXJtNjQgLyBBQ1BJIGRvZXMgbm90IHByb3ZpZGUgYW55IG1lYW5z IG9mIGlkZW50aWZ5aW5nCj4gYSBkaWUgbGV2ZWwgaW4gdGhlIHRvcG9sb2d5IGJ1dCB0aGF0IG1h eSBiZSB1bnJlbGF0ZSB0byB0aGUgY2x1c3Rlcgo+IGxldmVsLgo+CgpJbmRlZWQuIEl0IGNhbiBi ZSBjbHVzdGVyIG9mIGNsdXN0ZXJzIG9uIHNvbWUgcGxhdGZvcm0uIElmIHdlIG5lZWQgdGhhdApp bmZvLCB3ZSBzaG91bGQgYWRkIGl0LiBNeSBhc3N1bXB0aW9uIHdhcyB0aGF0IGdlbmVyYWxseSBl YWNoIGRpZSBmb3JtcwphIE5VTUEgbm9kZSBhbmQgaGVuY2UgdGhlIGluZm9ybWF0aW9uIGlzIGF2 YWlsYWJsZSB0aGVyZS4gSSBtYXkgYmUgd3JvbmcuCgo+IFJGQyBxdWVzdGlvbnM6Cj4gMSkgTmFt aW5nCj4gMikgUmVsYXRlZCB0byBuYW1pbmcsIGRvIHdlIHdhbnQgdG8gcmVwcmVzZW50IGFsbCBw b3RlbnRpYWwgbGV2ZWxzLAo+ICAgIG9yIHRoaXMgZW5vdWdoPyAgT24gS3VucGVuZzkyMCwgdGhl IG5leHQgbGV2ZWwgdXAgZnJvbSBjbHVzdGVyIGhhcHBlbnMKPiAgICB0byBiZSBjb3ZlcmVkIGJ5 IGxsYyBjYWNoZSBzaGFyaW5nLCBidXQgaW4gdGhlb3J5IG1vcmUgdGhhbiBvbmUKPiAgICBsZXZl bCBvZiBjbHVzdGVyIGRlc2NyaXB0aW9uIG1pZ2h0IGJlIG5lZWRlZCBieSBzb21lIGZ1dHVyZSBz eXN0ZW0uCgpUaGF0IGlzIG15IHF1ZXN0aW9uIGFib3ZlLiBDYW4ndCByZWNhbGwgdGhlIHRlcm1p bm9sb2d5IHVzZWQgaW4gQUNQSQpQUFRULCBidXQgSUlSQyAiY2x1c3RlciIgaXMgbm90IHVzZWQg dG8ga2VlcCBpdCBnZW5lcmljLiBNYXkgYmUgd2UgbmVlZAp0byBkbyB0aGUgc2FtZSBoZXJlIGFz IHRoZSB0ZXJtICJjbHVzdGVyIiBpcyBpbGwtZGVmaW5lZCBvbiBBcm0gYW5kIEkKd291bGQgYXZv aWQgdXNpbmcgaXQgaWYgcG9zc2libGUuCgo+IDMpIERvIHdlIG5lZWQgRFQgY29kZSBpbiBwbGFj ZT8gSSdtIG5vdCBzdXJlIGFueSBEVCBiYXNlZCBBUk02NAo+ICAgIHN5c3RlbXMgd291bGQgaGF2 ZSBlbm91Z2ggY29tcGxleGl0eSBmb3IgdGhpcyB0byBiZSB1c2VmdWwuCgpJIHByZWZlciB0byBr ZWVwIHRoZSBpbXBsZW1lbnRhdGlvbiBpbiBzeW5jCgo+IDQpIE90aGVyIGFyY2hpdGVjdHVyZXM/ ICBJcyB0aGlzIHVzZWZ1bCBvbiB4ODYgZm9yIGV4YW1wbGU/Cj4KCkFNRCBoYWQgbXVsdGktZGll IHdpdGhpbiBzb2NrZXQgSUlSQy4gSUlVQywgY3B1aWQgd2lsbCBwcm92aWRlIHRoZSBpbmZvCmFu ZCBub3RoaW5nIG1vcmUgaXMgbmVlZGVkIGZyb20gQUNQSSA/IEkgbWF5IGJlIHdyb25nLCBqdXN0 IGd1ZXNzaW5nL2Fza2luZy4KCi0tClJlZ2FyZHMsClN1ZGVlcAoKX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxp c3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZy YWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=