From mboxrd@z Thu Jan 1 00:00:00 1970 From: mick@ics.forth.gr (Nick Kossifidis) Date: Wed, 07 Nov 2018 04:31:34 +0200 Subject: [RFC 0/2] Add RISC-V cpu topology In-Reply-To: <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> Message-ID: To: linux-riscv@lists.infradead.org List-Id: linux-riscv.lists.infradead.org ???? 2018-11-06 18:20, Mark Rutland ??????: > On Tue, Nov 06, 2018 at 05:26:01PM +0200, Nick Kossifidis wrote: >> ???? 2018-11-06 16:13, Sudeep Holla ??????: >> > On Fri, Nov 02, 2018 at 08:58:39PM +0200, Nick Kossifidis wrote: >> > > ???? 2018-11-02 01:04, Atish Patra ??????: >> > > > This patch series adds the cpu topology for RISC-V. It contains >> > > > both the DT binding and actual source code. It has been tested on >> > > > QEMU & Unleashed board. >> > > > >> > > > The idea is based on cpu-map in ARM with changes related to how >> > > > we define SMT systems. The reason for adopting a similar approach >> > > > to ARM as I feel it provides a very clear way of defining the >> > > > topology compared to parsing cache nodes to figure out which cpus >> > > > share the same package or core. I am open to any other idea to >> > > > implement cpu-topology as well. >> > > >> > > I was also about to start a discussion about CPU topology on RISC-V >> > > after the last swtools group meeting. The goal is to provide the >> > > scheduler with hints on how to distribute tasks more efficiently >> > > between harts, by populating the scheduling domain topology levels >> > > (https://elixir.bootlin.com/linux/v4.19/ident/sched_domain_topology_level). >> > > What we want to do is define cpu groups and assign them to >> > > scheduling domains with the appropriate SD_ flags >> > > (https://github.com/torvalds/linux/blob/master/include/linux/sched/topology.h#L16). >> > >> > OK are we defining a CPU topology binding for Linux scheduler ? >> > NACK for all the approaches that assumes any knowledge of OS scheduler. >> >> Is there any standard regarding CPU topology on the device tree spec ? >> As far as I know there is none. We are talking about a Linux-specific >> Device Tree binding so I don't see why defining a binding for the >> Linux scheduler is out of scope. > > Speaking as a DT binding maintainer, please avoid OS-specific DT > bindings wherever possible. > > While DT bindings live in the kernel tree, they are not intended to be > Linux-specific, and other OSs (e.g. FreeBSD, zephyr) are aiming to > support the same bindings. > > In general, targeting a specific OS is a bad idea, because the > implementation details of that OS change over time, or the bindings end > up being tailored to one specific use-case. Exposing details to the OS > such that the OS can make decisions at runtime is typically better. > >> Do you have cpu-map on other OSes as well ? > > There is nothing OS-specific about cpu-map, and it may be of use to > other OSs. > >> > > So the cores that belong to a scheduling domain may share: >> > > CPU capacity (SD_SHARE_CPUCAPACITY / SD_ASYM_CPUCAPACITY) >> > > Package resources -e.g. caches, units etc- (SD_SHARE_PKG_RESOURCES) >> > > Power domain (SD_SHARE_POWERDOMAIN) >> > > >> > >> > Too Linux kernel/scheduler specific to be part of $subject >> >> All lists on the cc list are Linux specific, again I don't see your >> point here are we talking about defining a standard CPU topology >> scheme for the device tree spec or a Linux-specific CPU topology >> binding such as cpu-map ? > > The cpu-map binding is not intended to be Linux specific, and avoids > Linux-specific terminology. > > While the cpu-map binding documentation is in the Linux source tree, > the > binding itseld is not intended to be Linux-specific, and it > deliberately > avoids Linux implementation details. > >> Even on this case your point is not valid, the information of two >> harts sharing a common power domain or having the same or not >> capacity/max frequency (or maybe capabilities/extensions in the >> future), is not Linux specific. I just used the Linux specific macros >> used by the Linux scheduler to point out the code path. Even on other >> OSes we still need a way to include this information on the CPU >> topology, and currently cpu-map doesn't. Also the Linux implementation >> of cpu-map ignores multiple levels of shared resources, we only get >> one level for SMT and one level for MC last time I checked. > > Given clusters can be nested, as in the very first example, I don't see > what prevents multiple levels of shared resources. > > Can you please given an example of the topology your considering? Does > that share some resources across clusters at some level? > > We are certainly open to improving the cpu-map binding. > > Thanks, > Mark. Mark and Sundeep thanks a lot for your feedback, I guess you convinced me that having a device tree binding for the scheduler is not a correct approach. It's not a device after all and I agree that the device tree shouldn't become an OS configuration file. Regarding multiple levels of shared resources my point is that since cpu-map doesn't contain any information of what is shared among the cluster/core members it's not easy to do any further translation. Last time I checked the arm code that uses cpu-map, it only defines one domain for SMT, one for MC and then everything else is ignored. No matter how many clusters have been defined, anything above the core level is the same (and then I guess you started talking about adding "packages" on the representation side). The reason I proposed to have a binding for the scheduler directly is not only because it's simpler and closer to what really happens in the code, it also makes more sense to me than the combination of cpu-map with all the related mappings e.g. for numa or caches or power domains etc. However you are right we could definitely augment cpu-map to include support for what I'm saying and clean things up, and since you are open about improving it here is a proposal that I hope you find interesting: At first let's get rid of the nodes, they don't make sense: thread0 { cpu = <&CPU0>; }; A thread node can't have more than one cpu entry and any properties should be on the cpu node itself, so it doesn't / can't add any more information. We could just have an array of cpu nodes on the node, it's much cleaner this way. core0 { members = <&CPU0>, <&CPU1>; }; Then let's allow the cluster and core nodes to accept attributes that are common for the cpus they contain. Right now this is considered invalid. For power domains we have a generic binding described on Documentation/devicetree/bindings/power/power_domain.txt which basically says that we need to put power-domains = attribute on each of the cpu nodes. The same happens with the capacity binding specified for arm on Documentation/devicetree/bindings/arm/cpu-capacity.txt which says we should add the capacity-dmips-mhz on each of the cpu nodes. The same also happens with the generic numa binding on Documentation/devicetree/bindings/numa.txt which says we should add the nuna-node-id on each of the cpu nodes. We could allow for these attributes to exist on cluster and core nodes as well so that we can represent their properties better. It shouldn't be a big deal and it can be done in a backwards-compatible way (if we don't find them on the cpu node, climb up the topology hierarchy until we find them / not find them at all). All I'm saying is that I prefer this: cpus { cpu at 0 { ... }; cpu at 1 { ... }; cpu at 2 { ... }; cpu at 3 { ... }; }; cluster0 { cluster0 { core0 { power-domains = <&pdc 0>; numa-node-id = <0>; capacity-dmips-mhz = <578>; members = <&cpu0>, <&cpu1>; } }; cluster1 { capacity-dmips-mhz = <1024>; core0 { power-domains = <&pdc 1>; numa-node-id = <1>; members = <&cpu2>; }; core1 { power-domains = <&pdc 2>; numa-node-id = <2>; members = <&cpu3>; }; }; } over this: cpus { cpu at 0 { ... power-domains = <&pdc 0>; capacity-dmips-mhz = <578>; numa-node-id = <0>; ... }; cpu at 1 { ... power-domains = <&pdc 0>; capacity-dmips-mhz = <578>; numa-node-id = <0>; ... }; cpu at 2 { ... power-domains = <&pdc 1>; capacity-dmips-mhz = <1024>; numa-node-id = <1>; ... }; cpu at 3 { ... power-domains = <&pdc 2>; capacity-dmips-mhz = <1024>; numa-node-id = <2>; ... }; }; cluster0 { cluster0 { core0 { members = <&cpu0>, <&cpu1>; } }; cluster1 { core0 { members = <&cpu2>; } }; cluster2 { core0 { members = <&cpu3>; } }; } When it comes to shared resources, the standard dt mappings we have are for caches and are on the device spec standard (coming from power pc's ePAPR standard I think). The below comes from HiFive unleashed's device tree (U540Config.dts) that follows the spec: cpus { cpu at 1 { ... next-level-cache = <&L24 &L0>; ... }; cpu at 2 { ... next-level-cache = <&L24 &L0>; ... }; cpu at 3 { ... next-level-cache = <&L24 &L0>; ... }; cpu at 4 { ... next-level-cache = <&L24 &L0>; ... }; }; L2: soc { L0: cache-controller at 2010000 { cache-block-size = <64>; cache-level = <2>; cache-sets = <2048>; cache-size = <2097152>; cache-unified; compatible = "sifive,ccache0", "cache"; ... }; } Note that the cache-controller node that's common between the 4 cores can exist anywhere BUT the cluster node ! However it's a property of the cluster. A quick search through the tree got me r8a77980.dtsi that defines the cache on the cpus node and I'm sure there are other similar cases. Wouldn't this be better ? cluster0 { core0 { cache-controller at 2010000 { cache-block-size = <64>; cache-level = <2>; cache-sets = <2048>; cache-size = <2097152>; cache-unified; compatible = "sifive,ccache0", "cache"; ... }; members = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; }; }; We could even remove next-level-cache from the cpu nodes and infer it from the topology (search the topology upwards until we get a node that's "cache"-compatible), we can again make this backwards-compatible. Finally from the examples above I'd like to stress out that the distinction between a cluster and a core doesn't make much sense and it also makes the representation more complicated. To begin with, how would you call the setup on HiFive Unleashed ? A cluster of 4 cores that share the same L3 cache ? One core with 4 harts that share the same L3 cache ? We could represent it like this instead: cluster0 { cache-controller at 2010000 { cache-block-size = <64>; cache-level = <2>; cache-sets = <2048>; cache-size = <2097152>; cache-unified; compatible = "sifive,ccache0", "cache"; ... }; core0 { members = <&cpu0>; }; core1 { members = <&cpu1>; }; core2 { members = <&cpu2>; }; core3 { members = <&cpu3>; }; }; We could e.g. keep only cluster nodes and allow them to contain either an array of harts or other cluster sub-nodes + optionally a set of attributes, common to the members/sub-nodes of the cluster. This way we'll get in the first example: cluster0 { cluster0 { power-domains = <&pdc 0>; numa-node-id = <0>; capacity-dmips-mhz = <578>; members = <&cpu0>, <&cpu1>; }; cluster1 { capacity-dmips-mhz = <1024>; cluster0 { power-domains = <&pdc 1>; numa-node-id = <1>; members = <&cpu2>; }; cluster1 { power-domains = <&pdc 2>; numa-node-id = <2>; members = <&cpu3>; }; }; } and in the second example: cluster0 { cache-controller at 2010000 { cache-block-size = <64>; cache-level = <2>; cache-sets = <2048>; cache-size = <2097152>; cache-unified; compatible = "sifive,ccache0", "cache"; ... }; members = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; }; Thank you for your time ! Regards, Nick From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.5 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F30FC32789 for ; Wed, 7 Nov 2018 02:33:49 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9C4FD20827 for ; Wed, 7 Nov 2018 02:33:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="YaL5Qnwr"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="ORQiYfNE" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9C4FD20827 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ics.forth.gr Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-ID:References:In-Reply-To:Subject:To:From: Date:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=k+ejrvtIJve2Ht59Ws/CHeWoHyYnRGVpE6Pq76ZS9SQ=; b=YaL5QnwrweJkBzRtUy++Bi1z7 2fBfu0r7WHz0qcl/DEvFg06h8i0rO+eGOMVqtu4PJTeyVRe2oYJ/dLLzch0IQjT6NN1V1KWM7UHK3 PDGw4WONXSsjVaVzc50GowL3l8jU0HpRxU7Jl0nPbaIJ4t/4jUOEi07qSp0vluOjGFZF/HpHrBdak QXZ2HIfIrQE4r4lLQ9QcqsiGgy9X+X6nf61cXOuK58REq/RvjiiU96U7uKwyGNVADbDcL8E4v+KBG ABXhwCD4ysD0UVd+kYAKKkI/GotF0sWYsjNJA3c199GBq7BbmdlT0uR56uiCVJbtzAo3GbjI+6czz KXpPbWKwg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKDf9-0001Rg-K5; Wed, 07 Nov 2018 02:33:47 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKDf8-0001Ra-Dc for linux-riscv@bombadil.infradead.org; Wed, 07 Nov 2018 02:33:46 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Message-ID:References:In-Reply-To: Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version: Sender:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=tX3BdCgixgXhl19KhkYuNA6o2ilLbESFTJIclve3EHA=; b=ORQiYfNEn/ccNLUC9UMCeSh2rC MZvx/wzC4fs1BeLB45cPPoh1AJao93wj6qAAsQvBQOKSEQeMo+jDpyykEAgSPuCnAWd4OwKUExanm GMAo2aRnrO2xzzv9uZqx689rdlxP0pfzWKgpW0LR6Hs7lSdjdajQeeXoJPiksKAFJBZNeelgzznZP fJyl4DOvaLQa+GsqadOUZ+oHlBw5wumNgMh0orwuiVJfkjDgbjEde96PGJSE2RcCX57sDy0t+8DI5 j6xeIXYI7sw7G4u9UKUCuzFSrD2L+SeUNuvI8ietpXLHT/2pbyQZsTYU1nZAysq9nUKMw5B3moQw4 xEE+U/tw==; Received: from mailgate-4.ics.forth.gr ([139.91.1.7]) by casper.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKDf4-0006l0-2L for linux-riscv@lists.infradead.org; Wed, 07 Nov 2018 02:33:44 +0000 Received: from av1.ics.forth.gr (av3in.ics.forth.gr. [139.91.1.77]) by mailgate-4.ics.forth.gr (8.14.5/ICS-FORTH/V10-1.9-GATE-OUT) with ESMTP id wA72VbKa067746; Wed, 7 Nov 2018 04:31:39 +0200 (EET) X-AuditID: 8b5b9d4d-91bff70000000e62-66-5be24e87b883 Received: from enigma.ics.forth.gr (webmail.ics.forth.gr [139.91.1.35]) by av1.ics.forth.gr (SMTP Outbound / FORTH / ICS) with SMTP id DC.89.03682.78E42EB5; Wed, 7 Nov 2018 04:31:35 +0200 (EET) Received: from webmail.ics.forth.gr (localhost [127.0.0.1]) by enigma.ics.forth.gr (8.15.1//ICS-FORTH/V10.5.0C-EXTNULL-SSL-SASL) with ESMTP id wA72VYBk028294; Wed, 7 Nov 2018 04:31:34 +0200 X-ICS-AUTH-INFO: Authenticated user: at ics.forth.gr MIME-Version: 1.0 Date: Wed, 07 Nov 2018 04:31:34 +0200 From: Nick Kossifidis To: Mark Rutland , Sudeep Holla Subject: Re: [RFC 0/2] Add RISC-V cpu topology Organization: FORTH In-Reply-To: <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> Message-ID: X-Sender: mick@mailhost.ics.forth.gr User-Agent: Roundcube Webmail/1.1.2 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrKIsWRmVeSWpSXmKPExsXSHc2orNvh9yjaYM0aSYttS1azWrR8eMdq sWjFdxaL1vZvTBbzj5xjtTg9YRGTxeVdc9gstn1uYbNYev0ik0Xzu3PsFpsnLGC1aN17hN1i +akdLBabN01ltni+spfNgd9jz+lZzB5r5q1h9Jj6+wyLx8NNl5g8Nq/Q8ti0qpPN4925c+we m5fUe1xqvs7u8XmTnEf7gW6mAO4oLpuU1JzMstQifbsEroz1B96wFdyPq9h8dw5zA+Nb9y5G Tg4JAROJJU+msncxcnEICRxhlPg68RIbhHOQUWJt00F2iCpTidl7OxlBbF4BQYmTM5+wgNjM AhYSU6/sZ4Sw5SWat85mBrFZBFQlFsw/ygpiswloSsy/dBCsXkTAR+LA/P+MIAuYBd4xSexa PJUNJCEsoCfRtOEmWAO/gLDEp7sXwWxOAQ+Jw98bWCAumsckcePYZGaIK1wkJvX2sUJcpyLx 4fcDsEtFBZQlXpyYzjqBUWgWkmNnITl2FpJjFzAyr2IUSCwz1stMLtZLyy8qydBLL9rECI7M ub47GM8tsD/EKMDBqMTDe5P9UbQQa2JZcWXuIUYJDmYlEd7Tqx9GC/GmJFZWpRblxxeV5qQW H2KU5mBREuc9/CI8SEggPbEkNTs1tSC1CCbLxMEp1cC4e1fChmPPmM6GPEjne/1237F3Cl+n zL7//oPpDJfKQA/2mYWspU2RHvLvOGU0TlsxPP6zLsbSI3mXkPP0n5lFhy3zLpV++VgjsMRA 4KGP8/49z7/MPZVz/u+bqtyL7Sz5PozPK15Eu8yWVJJ+f77Df6um+bUwOUELIaMClY4JEm/m BfmdV+lWYinOSDTUYi4qTgQAwX39NsgCAAA= X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181107_023342_430871_AA2DF64A X-CRM114-Status: GOOD ( 57.18 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, Damien.LeMoal@wdc.com, alankao@andestech.com, hch@infradead.org, anup@brainfault.org, palmer@sifive.com, linux-kernel@vger.kernel.org, zong@andestech.com, Atish Patra , robh+dt@kernel.org, Nick Kossifidis , linux-riscv@lists.infradead.org, tglx@linutronix.de Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-riscv" Errors-To: linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org Message-ID: <20181107023134.HCv7q130fbrH5ELkybsLh9pM-etzCjA6rUJmNkHILJI@z> zqPPhM65z4IgMjAxOC0xMS0wNiAxODoyMCwgTWFyayBSdXRsYW5kIM6tzrPPgc6xz4jOtToKPiBP biBUdWUsIE5vdiAwNiwgMjAxOCBhdCAwNToyNjowMVBNICswMjAwLCBOaWNrIEtvc3NpZmlkaXMg d3JvdGU6Cj4+IM6jz4TOuc+CIDIwMTgtMTEtMDYgMTY6MTMsIFN1ZGVlcCBIb2xsYSDOrc6zz4HO sc+IzrU6Cj4+ID4gT24gRnJpLCBOb3YgMDIsIDIwMTggYXQgMDg6NTg6MzlQTSArMDIwMCwgTmlj ayBLb3NzaWZpZGlzIHdyb3RlOgo+PiA+ID4gzqPPhM65z4IgMjAxOC0xMS0wMiAwMTowNCwgQXRp c2ggUGF0cmEgzq3Os8+BzrHPiM61Ogo+PiA+ID4gPiBUaGlzIHBhdGNoIHNlcmllcyBhZGRzIHRo ZSBjcHUgdG9wb2xvZ3kgZm9yIFJJU0MtVi4gSXQgY29udGFpbnMKPj4gPiA+ID4gYm90aCB0aGUg RFQgYmluZGluZyBhbmQgYWN0dWFsIHNvdXJjZSBjb2RlLiBJdCBoYXMgYmVlbiB0ZXN0ZWQgb24K Pj4gPiA+ID4gUUVNVSAmIFVubGVhc2hlZCBib2FyZC4KPj4gPiA+ID4KPj4gPiA+ID4gVGhlIGlk ZWEgaXMgYmFzZWQgb24gY3B1LW1hcCBpbiBBUk0gd2l0aCBjaGFuZ2VzIHJlbGF0ZWQgdG8gaG93 Cj4+ID4gPiA+IHdlIGRlZmluZSBTTVQgc3lzdGVtcy4gVGhlIHJlYXNvbiBmb3IgYWRvcHRpbmcg YSBzaW1pbGFyIGFwcHJvYWNoCj4+ID4gPiA+IHRvIEFSTSBhcyBJIGZlZWwgaXQgcHJvdmlkZXMg YSB2ZXJ5IGNsZWFyIHdheSBvZiBkZWZpbmluZyB0aGUKPj4gPiA+ID4gdG9wb2xvZ3kgY29tcGFy ZWQgdG8gcGFyc2luZyBjYWNoZSBub2RlcyB0byBmaWd1cmUgb3V0IHdoaWNoIGNwdXMKPj4gPiA+ ID4gc2hhcmUgdGhlIHNhbWUgcGFja2FnZSBvciBjb3JlLiAgSSBhbSBvcGVuIHRvIGFueSBvdGhl ciBpZGVhIHRvCj4+ID4gPiA+IGltcGxlbWVudCBjcHUtdG9wb2xvZ3kgYXMgd2VsbC4KPj4gPiA+ Cj4+ID4gPiBJIHdhcyBhbHNvIGFib3V0IHRvIHN0YXJ0IGEgZGlzY3Vzc2lvbiBhYm91dCBDUFUg dG9wb2xvZ3kgb24gUklTQy1WCj4+ID4gPiBhZnRlciB0aGUgbGFzdCBzd3Rvb2xzIGdyb3VwIG1l ZXRpbmcuIFRoZSBnb2FsIGlzIHRvIHByb3ZpZGUgdGhlCj4+ID4gPiBzY2hlZHVsZXIgd2l0aCBo aW50cyBvbiBob3cgdG8gZGlzdHJpYnV0ZSB0YXNrcyBtb3JlIGVmZmljaWVudGx5Cj4+ID4gPiBi ZXR3ZWVuIGhhcnRzLCBieSBwb3B1bGF0aW5nIHRoZSBzY2hlZHVsaW5nIGRvbWFpbiB0b3BvbG9n eSBsZXZlbHMKPj4gPiA+IChodHRwczovL2VsaXhpci5ib290bGluLmNvbS9saW51eC92NC4xOS9p ZGVudC9zY2hlZF9kb21haW5fdG9wb2xvZ3lfbGV2ZWwpLgo+PiA+ID4gV2hhdCB3ZSB3YW50IHRv IGRvIGlzIGRlZmluZSBjcHUgZ3JvdXBzIGFuZCBhc3NpZ24gdGhlbSB0bwo+PiA+ID4gc2NoZWR1 bGluZyBkb21haW5zIHdpdGggdGhlIGFwcHJvcHJpYXRlIFNEXyBmbGFncwo+PiA+ID4gKGh0dHBz Oi8vZ2l0aHViLmNvbS90b3J2YWxkcy9saW51eC9ibG9iL21hc3Rlci9pbmNsdWRlL2xpbnV4L3Nj aGVkL3RvcG9sb2d5LmgjTDE2KS4KPj4gPgo+PiA+IE9LIGFyZSB3ZSBkZWZpbmluZyBhIENQVSB0 b3BvbG9neSBiaW5kaW5nIGZvciBMaW51eCBzY2hlZHVsZXIgPwo+PiA+IE5BQ0sgZm9yIGFsbCB0 aGUgYXBwcm9hY2hlcyB0aGF0IGFzc3VtZXMgYW55IGtub3dsZWRnZSBvZiBPUyBzY2hlZHVsZXIu Cj4+IAo+PiBJcyB0aGVyZSBhbnkgc3RhbmRhcmQgcmVnYXJkaW5nIENQVSB0b3BvbG9neSBvbiB0 aGUgZGV2aWNlIHRyZWUgc3BlYyA/Cj4+IEFzIGZhciBhcyBJIGtub3cgdGhlcmUgaXMgbm9uZS4g V2UgYXJlIHRhbGtpbmcgYWJvdXQgYSBMaW51eC1zcGVjaWZpYwo+PiBEZXZpY2UgVHJlZSBiaW5k aW5nIHNvIEkgZG9uJ3Qgc2VlIHdoeSBkZWZpbmluZyBhIGJpbmRpbmcgZm9yIHRoZQo+PiBMaW51 eCBzY2hlZHVsZXIgaXMgb3V0IG9mIHNjb3BlLgo+IAo+IFNwZWFraW5nIGFzIGEgRFQgYmluZGlu ZyBtYWludGFpbmVyLCBwbGVhc2UgYXZvaWQgT1Mtc3BlY2lmaWMgRFQKPiBiaW5kaW5ncyB3aGVy ZXZlciBwb3NzaWJsZS4KPiAKPiBXaGlsZSBEVCBiaW5kaW5ncyBsaXZlIGluIHRoZSBrZXJuZWwg dHJlZSwgdGhleSBhcmUgbm90IGludGVuZGVkIHRvIGJlCj4gTGludXgtc3BlY2lmaWMsIGFuZCBv dGhlciBPU3MgKGUuZy4gRnJlZUJTRCwgemVwaHlyKSBhcmUgYWltaW5nIHRvCj4gc3VwcG9ydCB0 aGUgc2FtZSBiaW5kaW5ncy4KPiAKPiBJbiBnZW5lcmFsLCB0YXJnZXRpbmcgYSBzcGVjaWZpYyBP UyBpcyBhIGJhZCBpZGVhLCBiZWNhdXNlIHRoZQo+IGltcGxlbWVudGF0aW9uIGRldGFpbHMgb2Yg dGhhdCBPUyBjaGFuZ2Ugb3ZlciB0aW1lLCBvciB0aGUgYmluZGluZ3MgZW5kCj4gdXAgYmVpbmcg dGFpbG9yZWQgdG8gb25lIHNwZWNpZmljIHVzZS1jYXNlLiBFeHBvc2luZyBkZXRhaWxzIHRvIHRo ZSBPUwo+IHN1Y2ggdGhhdCB0aGUgT1MgY2FuIG1ha2UgZGVjaXNpb25zIGF0IHJ1bnRpbWUgaXMg dHlwaWNhbGx5IGJldHRlci4KPiAKPj4gRG8geW91IGhhdmUgY3B1LW1hcCBvbiBvdGhlciBPU2Vz IGFzIHdlbGwgPwo+IAo+IFRoZXJlIGlzIG5vdGhpbmcgT1Mtc3BlY2lmaWMgYWJvdXQgY3B1LW1h cCwgYW5kIGl0IG1heSBiZSBvZiB1c2UgdG8KPiBvdGhlciBPU3MuCj4gCj4+ID4gPiBTbyB0aGUg Y29yZXMgdGhhdCBiZWxvbmcgdG8gYSBzY2hlZHVsaW5nIGRvbWFpbiBtYXkgc2hhcmU6Cj4+ID4g PiBDUFUgY2FwYWNpdHkgKFNEX1NIQVJFX0NQVUNBUEFDSVRZIC8gU0RfQVNZTV9DUFVDQVBBQ0lU WSkKPj4gPiA+IFBhY2thZ2UgcmVzb3VyY2VzIC1lLmcuIGNhY2hlcywgdW5pdHMgZXRjLSAoU0Rf U0hBUkVfUEtHX1JFU09VUkNFUykKPj4gPiA+IFBvd2VyIGRvbWFpbiAoU0RfU0hBUkVfUE9XRVJE T01BSU4pCj4+ID4gPgo+PiA+Cj4+ID4gVG9vIExpbnV4IGtlcm5lbC9zY2hlZHVsZXIgc3BlY2lm aWMgdG8gYmUgcGFydCBvZiAkc3ViamVjdAo+PiAKPj4gQWxsIGxpc3RzIG9uIHRoZSBjYyBsaXN0 IGFyZSBMaW51eCBzcGVjaWZpYywgYWdhaW4gSSBkb24ndCBzZWUgeW91cgo+PiBwb2ludCBoZXJl IGFyZSB3ZSB0YWxraW5nIGFib3V0IGRlZmluaW5nIGEgc3RhbmRhcmQgQ1BVIHRvcG9sb2d5Cj4+ IHNjaGVtZSBmb3IgdGhlIGRldmljZSB0cmVlIHNwZWMgb3IgYSBMaW51eC1zcGVjaWZpYyBDUFUg dG9wb2xvZ3kKPj4gYmluZGluZyBzdWNoIGFzIGNwdS1tYXAgPwo+IAo+IFRoZSBjcHUtbWFwIGJp bmRpbmcgaXMgbm90IGludGVuZGVkIHRvIGJlIExpbnV4IHNwZWNpZmljLCBhbmQgYXZvaWRzCj4g TGludXgtc3BlY2lmaWMgdGVybWlub2xvZ3kuCj4gCj4gV2hpbGUgdGhlIGNwdS1tYXAgYmluZGlu ZyBkb2N1bWVudGF0aW9uIGlzIGluIHRoZSBMaW51eCBzb3VyY2UgdHJlZSwgCj4gdGhlCj4gYmlu ZGluZyBpdHNlbGQgaXMgbm90IGludGVuZGVkIHRvIGJlIExpbnV4LXNwZWNpZmljLCBhbmQgaXQg Cj4gZGVsaWJlcmF0ZWx5Cj4gYXZvaWRzIExpbnV4IGltcGxlbWVudGF0aW9uIGRldGFpbHMuCj4g Cj4+IEV2ZW4gb24gdGhpcyBjYXNlIHlvdXIgcG9pbnQgaXMgbm90IHZhbGlkLCB0aGUgaW5mb3Jt YXRpb24gb2YgdHdvCj4+IGhhcnRzIHNoYXJpbmcgYSBjb21tb24gcG93ZXIgZG9tYWluIG9yIGhh dmluZyB0aGUgc2FtZSBvciBub3QKPj4gY2FwYWNpdHkvbWF4IGZyZXF1ZW5jeSAob3IgbWF5YmUg Y2FwYWJpbGl0aWVzL2V4dGVuc2lvbnMgaW4gdGhlCj4+IGZ1dHVyZSksIGlzIG5vdCBMaW51eCBz cGVjaWZpYy4gSSBqdXN0IHVzZWQgdGhlIExpbnV4IHNwZWNpZmljIG1hY3Jvcwo+PiB1c2VkIGJ5 IHRoZSBMaW51eCBzY2hlZHVsZXIgdG8gcG9pbnQgb3V0IHRoZSBjb2RlIHBhdGguICBFdmVuIG9u IG90aGVyCj4+IE9TZXMgd2Ugc3RpbGwgbmVlZCBhIHdheSB0byBpbmNsdWRlIHRoaXMgaW5mb3Jt YXRpb24gb24gdGhlIENQVQo+PiB0b3BvbG9neSwgYW5kIGN1cnJlbnRseSBjcHUtbWFwIGRvZXNu J3QuIEFsc28gdGhlIExpbnV4IGltcGxlbWVudGF0aW9uCj4+IG9mIGNwdS1tYXAgaWdub3JlcyBt dWx0aXBsZSBsZXZlbHMgb2Ygc2hhcmVkIHJlc291cmNlcywgd2Ugb25seSBnZXQKPj4gb25lIGxl dmVsIGZvciBTTVQgYW5kIG9uZSBsZXZlbCBmb3IgTUMgbGFzdCB0aW1lIEkgY2hlY2tlZC4KPiAK PiBHaXZlbiBjbHVzdGVycyBjYW4gYmUgbmVzdGVkLCBhcyBpbiB0aGUgdmVyeSBmaXJzdCBleGFt cGxlLCBJIGRvbid0IHNlZQo+IHdoYXQgcHJldmVudHMgbXVsdGlwbGUgbGV2ZWxzIG9mIHNoYXJl ZCByZXNvdXJjZXMuCj4gCj4gQ2FuIHlvdSBwbGVhc2UgZ2l2ZW4gYW4gZXhhbXBsZSBvZiB0aGUg dG9wb2xvZ3kgeW91ciBjb25zaWRlcmluZz8gRG9lcwo+IHRoYXQgc2hhcmUgc29tZSByZXNvdXJj ZXMgYWNyb3NzIGNsdXN0ZXJzIGF0IHNvbWUgbGV2ZWw/Cj4gCj4gV2UgYXJlIGNlcnRhaW5seSBv cGVuIHRvIGltcHJvdmluZyB0aGUgY3B1LW1hcCBiaW5kaW5nLgo+IAo+IFRoYW5rcywKPiBNYXJr LgoKTWFyayBhbmQgU3VuZGVlcCB0aGFua3MgYSBsb3QgZm9yIHlvdXIgZmVlZGJhY2ssIEkgZ3Vl c3MgeW91IGNvbnZpbmNlZCAKbWUKdGhhdCBoYXZpbmcgYSBkZXZpY2UgdHJlZSBiaW5kaW5nIGZv ciB0aGUgc2NoZWR1bGVyIGlzIG5vdCBhIGNvcnJlY3QgCmFwcHJvYWNoLgpJdCdzIG5vdCBhIGRl dmljZSBhZnRlciBhbGwgYW5kIEkgYWdyZWUgdGhhdCB0aGUgZGV2aWNlIHRyZWUgc2hvdWxkbid0 IApiZWNvbWUKYW4gT1MgY29uZmlndXJhdGlvbiBmaWxlLiBSZWdhcmRpbmcgbXVsdGlwbGUgbGV2 ZWxzIG9mIHNoYXJlZCByZXNvdXJjZXMgCm15IHBvaW50CmlzIHRoYXQgc2luY2UgY3B1LW1hcCBk b2Vzbid0IGNvbnRhaW4gYW55IGluZm9ybWF0aW9uIG9mIHdoYXQgaXMgc2hhcmVkIAphbW9uZwp0 aGUgY2x1c3Rlci9jb3JlIG1lbWJlcnMgaXQncyBub3QgZWFzeSB0byBkbyBhbnkgZnVydGhlciB0 cmFuc2xhdGlvbi4gCkxhc3QgdGltZQpJIGNoZWNrZWQgdGhlIGFybSBjb2RlIHRoYXQgdXNlcyBj cHUtbWFwLCBpdCBvbmx5IGRlZmluZXMgb25lIGRvbWFpbiBmb3IgClNNVCwgb25lCmZvciBNQyBh bmQgdGhlbiBldmVyeXRoaW5nIGVsc2UgaXMgaWdub3JlZC4gTm8gbWF0dGVyIGhvdyBtYW55IGNs dXN0ZXJzIApoYXZlIGJlZW4KZGVmaW5lZCwgYW55dGhpbmcgYWJvdmUgdGhlIGNvcmUgbGV2ZWwg aXMgdGhlIHNhbWUgKGFuZCB0aGVuIEkgZ3Vlc3MgeW91IApzdGFydGVkCnRhbGtpbmcgYWJvdXQg YWRkaW5nICJwYWNrYWdlcyIgb24gdGhlIHJlcHJlc2VudGF0aW9uIHNpZGUpLgoKVGhlIHJlYXNv biBJIHByb3Bvc2VkIHRvIGhhdmUgYSBiaW5kaW5nIGZvciB0aGUgc2NoZWR1bGVyIGRpcmVjdGx5 IGlzIApub3Qgb25seQpiZWNhdXNlIGl0J3Mgc2ltcGxlciBhbmQgY2xvc2VyIHRvIHdoYXQgcmVh bGx5IGhhcHBlbnMgaW4gdGhlIGNvZGUsIGl0IAphbHNvIG1ha2VzCm1vcmUgc2Vuc2UgdG8gbWUg dGhhbiB0aGUgY29tYmluYXRpb24gb2YgY3B1LW1hcCB3aXRoIGFsbCB0aGUgcmVsYXRlZCAKbWFw cGluZ3MgZS5nLgpmb3IgbnVtYSBvciBjYWNoZXMgb3IgcG93ZXIgZG9tYWlucyBldGMuCgpIb3dl dmVyIHlvdSBhcmUgcmlnaHQgd2UgY291bGQgZGVmaW5pdGVseSBhdWdtZW50IGNwdS1tYXAgdG8g aW5jbHVkZSAKc3VwcG9ydCBmb3IKd2hhdCBJJ20gc2F5aW5nIGFuZCBjbGVhbiB0aGluZ3MgdXAs IGFuZCBzaW5jZSB5b3UgYXJlIG9wZW4gYWJvdXQgCmltcHJvdmluZyBpdApoZXJlIGlzIGEgcHJv cG9zYWwgdGhhdCBJIGhvcGUgeW91IGZpbmQgaW50ZXJlc3Rpbmc6CgpBdCBmaXJzdCBsZXQncyBn ZXQgcmlkIG9mIHRoZSA8dGhyZWFkPiBub2RlcywgdGhleSBkb24ndCBtYWtlIHNlbnNlOgoKdGhy ZWFkMCB7CiAgY3B1ID0gPCZDUFUwPjsKfTsKCkEgdGhyZWFkIG5vZGUgY2FuJ3QgaGF2ZSBtb3Jl IHRoYW4gb25lIGNwdSBlbnRyeSBhbmQgYW55IHByb3BlcnRpZXMKc2hvdWxkIGJlIG9uIHRoZSBj cHUgbm9kZSBpdHNlbGYsIHNvIGl0IGRvZXNuJ3QgLyBjYW4ndCBhZGQgYW55Cm1vcmUgaW5mb3Jt YXRpb24uIFdlIGNvdWxkIGp1c3QgaGF2ZSBhbiBhcnJheSBvZiBjcHUgbm9kZXMgb24gdGhlCjxj b3JlPiBub2RlLCBpdCdzIG11Y2ggY2xlYW5lciB0aGlzIHdheS4KCmNvcmUwIHsKICBtZW1iZXJz ID0gPCZDUFUwPiwgPCZDUFUxPjsKfTsKClRoZW4gbGV0J3MgYWxsb3cgdGhlIGNsdXN0ZXIgYW5k IGNvcmUgbm9kZXMgdG8gYWNjZXB0IGF0dHJpYnV0ZXMgdGhhdCAKYXJlCmNvbW1vbiBmb3IgdGhl IGNwdXMgdGhleSBjb250YWluLiBSaWdodCBub3cgdGhpcyBpcyBjb25zaWRlcmVkIGludmFsaWQu CgpGb3IgcG93ZXIgZG9tYWlucyB3ZSBoYXZlIGEgZ2VuZXJpYyBiaW5kaW5nIGRlc2NyaWJlZCBv bgpEb2N1bWVudGF0aW9uL2RldmljZXRyZWUvYmluZGluZ3MvcG93ZXIvcG93ZXJfZG9tYWluLnR4 dAp3aGljaCBiYXNpY2FsbHkgc2F5cyB0aGF0IHdlIG5lZWQgdG8gcHV0IHBvd2VyLWRvbWFpbnMg PSA8cG93ZXIgZG9tYWluIApzcGVjaWZpZXJzPgphdHRyaWJ1dGUgb24gZWFjaCBvZiB0aGUgY3B1 IG5vZGVzLgoKVGhlIHNhbWUgaGFwcGVucyB3aXRoIHRoZSBjYXBhY2l0eSBiaW5kaW5nIHNwZWNp ZmllZCBmb3IgYXJtIG9uCkRvY3VtZW50YXRpb24vZGV2aWNldHJlZS9iaW5kaW5ncy9hcm0vY3B1 LWNhcGFjaXR5LnR4dAp3aGljaCBzYXlzIHdlIHNob3VsZCBhZGQgdGhlIGNhcGFjaXR5LWRtaXBz LW1oeiBvbiBlYWNoIG9mIHRoZSBjcHUgCm5vZGVzLgoKVGhlIHNhbWUgYWxzbyBoYXBwZW5zIHdp dGggdGhlIGdlbmVyaWMgbnVtYSBiaW5kaW5nIG9uCkRvY3VtZW50YXRpb24vZGV2aWNldHJlZS9i aW5kaW5ncy9udW1hLnR4dAp3aGljaCBzYXlzIHdlIHNob3VsZCBhZGQgdGhlIG51bmEtbm9kZS1p ZCBvbiBlYWNoIG9mIHRoZSBjcHUgbm9kZXMuCgpXZSBjb3VsZCBhbGxvdyBmb3IgdGhlc2UgYXR0 cmlidXRlcyB0byBleGlzdCBvbiBjbHVzdGVyIGFuZCBjb3JlIG5vZGVzCmFzIHdlbGwgc28gdGhh dCB3ZSBjYW4gcmVwcmVzZW50IHRoZWlyIHByb3BlcnRpZXMgYmV0dGVyLiBJdCBzaG91bGRuJ3QK YmUgYSBiaWcgZGVhbCBhbmQgaXQgY2FuIGJlIGRvbmUgaW4gYSBiYWNrd2FyZHMtY29tcGF0aWJs ZSB3YXkgKGlmIHdlCmRvbid0IGZpbmQgdGhlbSBvbiB0aGUgY3B1IG5vZGUsIGNsaW1iIHVwIHRo ZSB0b3BvbG9neSBoaWVyYXJjaHkgdW50aWwKd2UgZmluZCB0aGVtIC8gbm90IGZpbmQgdGhlbSBh dCBhbGwpLiBBbGwgSSdtIHNheWluZyBpcyB0aGF0IEkgcHJlZmVyIAp0aGlzOgoKY3B1cyB7CiAg Y3B1QDAgewogICAuLi4KICB9OwogIGNwdUAxIHsKICAgLi4uCiAgfTsKICBjcHVAMiB7CiAgIC4u LgogIH07CiAgY3B1QDMgewogICAuLi4KICB9Owp9OwoKCmNsdXN0ZXIwIHsKICBjbHVzdGVyMCB7 CiAgIGNvcmUwIHsKICAgIHBvd2VyLWRvbWFpbnMgPSA8JnBkYyAwPjsKICAgIG51bWEtbm9kZS1p ZCA9IDwwPjsKICAgIGNhcGFjaXR5LWRtaXBzLW1oeiA9IDw1Nzg+OwogICAgbWVtYmVycyA9IDwm Y3B1MD4sIDwmY3B1MT47CiAgIH0KICB9OwogIGNsdXN0ZXIxIHsKICAgY2FwYWNpdHktZG1pcHMt bWh6ID0gPDEwMjQ+OwogICBjb3JlMCB7CiAgICBwb3dlci1kb21haW5zID0gPCZwZGMgMT47CiAg ICBudW1hLW5vZGUtaWQgPSA8MT47CiAgICBtZW1iZXJzID0gPCZjcHUyPjsKICAgfTsKICAgY29y ZTEgewogICAgcG93ZXItZG9tYWlucyA9IDwmcGRjIDI+OwogICAgbnVtYS1ub2RlLWlkID0gPDI+ OwogICAgbWVtYmVycyA9IDwmY3B1Mz47CiAgIH07CiAgfTsKfQoKb3ZlciB0aGlzOgoKY3B1cyB7 CiAgY3B1QDAgewogICAuLi4KICAgcG93ZXItZG9tYWlucyA9IDwmcGRjIDA+OwogICBjYXBhY2l0 eS1kbWlwcy1taHogPSA8NTc4PjsKICAgbnVtYS1ub2RlLWlkID0gPDA+OwogICAuLi4KICB9Owog IGNwdUAxIHsKICAgLi4uCiAgIHBvd2VyLWRvbWFpbnMgPSA8JnBkYyAwPjsKICAgY2FwYWNpdHkt ZG1pcHMtbWh6ID0gPDU3OD47CiAgIG51bWEtbm9kZS1pZCA9IDwwPjsKICAgLi4uCiAgfTsKICBj cHVAMiB7CiAgIC4uLgogICBwb3dlci1kb21haW5zID0gPCZwZGMgMT47CiAgIGNhcGFjaXR5LWRt aXBzLW1oeiA9IDwxMDI0PjsKICAgbnVtYS1ub2RlLWlkID0gPDE+OwogICAuLi4KICB9OwogIGNw dUAzIHsKICAgLi4uCiAgIHBvd2VyLWRvbWFpbnMgPSA8JnBkYyAyPjsKICAgY2FwYWNpdHktZG1p cHMtbWh6ID0gPDEwMjQ+OwogICBudW1hLW5vZGUtaWQgPSA8Mj47CiAgIC4uLgogIH07Cn07CgoK Y2x1c3RlcjAgewogIGNsdXN0ZXIwIHsKICAgY29yZTAgewogICAgbWVtYmVycyA9IDwmY3B1MD4s IDwmY3B1MT47CiAgIH0KICB9OwogIGNsdXN0ZXIxIHsKICAgY29yZTAgewogICAgbWVtYmVycyA9 IDwmY3B1Mj47CiAgIH0KICB9OwogIGNsdXN0ZXIyIHsKICAgY29yZTAgewogICAgbWVtYmVycyA9 IDwmY3B1Mz47CiAgIH0KICB9Owp9CgoKV2hlbiBpdCBjb21lcyB0byBzaGFyZWQgcmVzb3VyY2Vz LCB0aGUgc3RhbmRhcmQgZHQgbWFwcGluZ3Mgd2UgaGF2ZSBhcmUgCmZvcgpjYWNoZXMgYW5kIGFy ZSBvbiB0aGUgZGV2aWNlIHNwZWMgc3RhbmRhcmQgKGNvbWluZyBmcm9tIHBvd2VyIHBjJ3MgZVBB UFIKc3RhbmRhcmQgSSB0aGluaykuIFRoZSBiZWxvdyBjb21lcyBmcm9tIEhpRml2ZSB1bmxlYXNo ZWQncyBkZXZpY2UgdHJlZQooVTU0MENvbmZpZy5kdHMpIHRoYXQgZm9sbG93cyB0aGUgc3BlYzoK CmNwdXMgewogIGNwdUAxIHsKICAgLi4uCiAgIG5leHQtbGV2ZWwtY2FjaGUgPSA8JkwyNCAmTDA+ OwogICAuLi4KICB9OwogIGNwdUAyIHsKICAgLi4uCiAgIG5leHQtbGV2ZWwtY2FjaGUgPSA8Jkwy NCAmTDA+OwogICAuLi4KICB9OwogIGNwdUAzIHsKICAgLi4uCiAgIG5leHQtbGV2ZWwtY2FjaGUg PSA8JkwyNCAmTDA+OwogICAuLi4KICB9OwogIGNwdUA0IHsKICAgLi4uCiAgIG5leHQtbGV2ZWwt Y2FjaGUgPSA8JkwyNCAmTDA+OwogICAuLi4KICB9Owp9OwoKTDI6IHNvYyB7CiAgTDA6IGNhY2hl LWNvbnRyb2xsZXJAMjAxMDAwMCB7CiAgIGNhY2hlLWJsb2NrLXNpemUgPSA8NjQ+OwogICBjYWNo ZS1sZXZlbCA9IDwyPjsKICAgY2FjaGUtc2V0cyA9IDwyMDQ4PjsKICAgY2FjaGUtc2l6ZSA9IDwy MDk3MTUyPjsKICAgY2FjaGUtdW5pZmllZDsKICAgY29tcGF0aWJsZSA9ICJzaWZpdmUsY2NhY2hl MCIsICJjYWNoZSI7CiAgIC4uLgogIH07Cn0KCk5vdGUgdGhhdCB0aGUgY2FjaGUtY29udHJvbGxl ciBub2RlIHRoYXQncyBjb21tb24gYmV0d2VlbiB0aGUgNCBjb3JlcyAKY2FuCmV4aXN0IGFueXdo ZXJlIEJVVCB0aGUgY2x1c3RlciBub2RlICEgSG93ZXZlciBpdCdzIGEgcHJvcGVydHkgb2YgdGhl IApjbHVzdGVyLgpBIHF1aWNrIHNlYXJjaCB0aHJvdWdoIHRoZSB0cmVlIGdvdCBtZSByOGE3Nzk4 MC5kdHNpIHRoYXQgZGVmaW5lcyB0aGUgCmNhY2hlCm9uIHRoZSBjcHVzIG5vZGUgYW5kIEknbSBz dXJlIHRoZXJlIGFyZSBvdGhlciBzaW1pbGFyIGNhc2VzLiBXb3VsZG4ndCAKdGhpcwpiZSBiZXR0 ZXIgPwoKY2x1c3RlcjAgewogIGNvcmUwIHsKICAgY2FjaGUtY29udHJvbGxlckAyMDEwMDAwIHsK ICAgIGNhY2hlLWJsb2NrLXNpemUgPSA8NjQ+OwogICAgY2FjaGUtbGV2ZWwgPSA8Mj47CiAgICBj YWNoZS1zZXRzID0gPDIwNDg+OwogICAgY2FjaGUtc2l6ZSA9IDwyMDk3MTUyPjsKICAgIGNhY2hl LXVuaWZpZWQ7CiAgICBjb21wYXRpYmxlID0gInNpZml2ZSxjY2FjaGUwIiwgImNhY2hlIjsKICAg IC4uLgogICB9OwogICBtZW1iZXJzID0gPCZjcHUwPiwgPCZjcHUxPiwgPCZjcHUyPiwgPCZjcHUz PjsKICB9Owp9OwoKV2UgY291bGQgZXZlbiByZW1vdmUgbmV4dC1sZXZlbC1jYWNoZSBmcm9tIHRo ZSBjcHUgbm9kZXMgYW5kIGluZmVyIGl0IApmcm9tIHRoZQp0b3BvbG9neSAoc2VhcmNoIHRoZSB0 b3BvbG9neSB1cHdhcmRzIHVudGlsIHdlIGdldCBhIG5vZGUgdGhhdCdzCiJjYWNoZSItY29tcGF0 aWJsZSksIHdlIGNhbiBhZ2FpbiBtYWtlIHRoaXMgYmFja3dhcmRzLWNvbXBhdGlibGUuCgoKRmlu YWxseSBmcm9tIHRoZSBleGFtcGxlcyBhYm92ZSBJJ2QgbGlrZSB0byBzdHJlc3Mgb3V0IHRoYXQg dGhlIApkaXN0aW5jdGlvbgpiZXR3ZWVuIGEgY2x1c3RlciBhbmQgYSBjb3JlIGRvZXNuJ3QgbWFr ZSBtdWNoIHNlbnNlIGFuZCBpdCBhbHNvIG1ha2VzIAp0aGUKcmVwcmVzZW50YXRpb24gbW9yZSBj b21wbGljYXRlZC4gVG8gYmVnaW4gd2l0aCwgaG93IHdvdWxkIHlvdSBjYWxsIHRoZSAKc2V0dXAK b24gSGlGaXZlIFVubGVhc2hlZCA/IEEgY2x1c3RlciBvZiA0IGNvcmVzIHRoYXQgc2hhcmUgdGhl IHNhbWUgTDMgY2FjaGUgCj8KT25lIGNvcmUgd2l0aCA0IGhhcnRzIHRoYXQgc2hhcmUgdGhlIHNh bWUgTDMgY2FjaGUgPyBXZSBjb3VsZCByZXByZXNlbnQgCml0Cmxpa2UgdGhpcyBpbnN0ZWFkOgoK Y2x1c3RlcjAgewogIGNhY2hlLWNvbnRyb2xsZXJAMjAxMDAwMCB7CiAgIGNhY2hlLWJsb2NrLXNp emUgPSA8NjQ+OwogICBjYWNoZS1sZXZlbCA9IDwyPjsKICAgY2FjaGUtc2V0cyA9IDwyMDQ4PjsK ICAgY2FjaGUtc2l6ZSA9IDwyMDk3MTUyPjsKICAgY2FjaGUtdW5pZmllZDsKICAgY29tcGF0aWJs ZSA9ICJzaWZpdmUsY2NhY2hlMCIsICJjYWNoZSI7CiAgIC4uLgogIH07CiAgY29yZTAgewogICBt ZW1iZXJzID0gPCZjcHUwPjsKICB9OwogIGNvcmUxIHsKICAgbWVtYmVycyA9IDwmY3B1MT47CiAg fTsKICBjb3JlMiB7CiAgIG1lbWJlcnMgPSA8JmNwdTI+OwogIH07CiAgY29yZTMgewogICBtZW1i ZXJzID0gPCZjcHUzPjsKICB9Owp9OwoKV2UgY291bGQgZS5nLiBrZWVwIG9ubHkgY2x1c3RlciBu b2RlcyBhbmQgYWxsb3cgdGhlbSB0byBjb250YWluIGVpdGhlciAKYW4gYXJyYXkKb2YgaGFydHMg b3Igb3RoZXIgY2x1c3RlciBzdWItbm9kZXMgKyBvcHRpb25hbGx5IGEgc2V0IG9mIGF0dHJpYnV0 ZXMsIApjb21tb24gdG8KdGhlIG1lbWJlcnMvc3ViLW5vZGVzIG9mIHRoZSBjbHVzdGVyLiBUaGlz IHdheSB3ZSdsbCBnZXQgaW4gdGhlIGZpcnN0IApleGFtcGxlOgoKY2x1c3RlcjAgewogIGNsdXN0 ZXIwIHsKICAgcG93ZXItZG9tYWlucyA9IDwmcGRjIDA+OwogICBudW1hLW5vZGUtaWQgPSA8MD47 CiAgIGNhcGFjaXR5LWRtaXBzLW1oeiA9IDw1Nzg+OwogICBtZW1iZXJzID0gPCZjcHUwPiwgPCZj cHUxPjsKICB9OwogIGNsdXN0ZXIxIHsKICAgY2FwYWNpdHktZG1pcHMtbWh6ID0gPDEwMjQ+Owog ICBjbHVzdGVyMCB7CiAgICBwb3dlci1kb21haW5zID0gPCZwZGMgMT47CiAgICBudW1hLW5vZGUt aWQgPSA8MT47CiAgICBtZW1iZXJzID0gPCZjcHUyPjsKICAgfTsKICAgY2x1c3RlcjEgewogICAg cG93ZXItZG9tYWlucyA9IDwmcGRjIDI+OwogICAgbnVtYS1ub2RlLWlkID0gPDI+OwogICAgbWVt YmVycyA9IDwmY3B1Mz47CiAgIH07CiAgfTsKfQoKYW5kIGluIHRoZSBzZWNvbmQgZXhhbXBsZToK CmNsdXN0ZXIwIHsKICBjYWNoZS1jb250cm9sbGVyQDIwMTAwMDAgewogICBjYWNoZS1ibG9jay1z aXplID0gPDY0PjsKICAgY2FjaGUtbGV2ZWwgPSA8Mj47CiAgIGNhY2hlLXNldHMgPSA8MjA0OD47 CiAgIGNhY2hlLXNpemUgPSA8MjA5NzE1Mj47CiAgIGNhY2hlLXVuaWZpZWQ7CiAgIGNvbXBhdGli bGUgPSAic2lmaXZlLGNjYWNoZTAiLCAiY2FjaGUiOwogICAuLi4KICB9OwogIG1lbWJlcnMgPSA8 JmNwdTA+LCA8JmNwdTE+LCA8JmNwdTI+LCA8JmNwdTM+Owp9OwoKClRoYW5rIHlvdSBmb3IgeW91 ciB0aW1lICEKClJlZ2FyZHMsCk5pY2sKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fCmxpbnV4LXJpc2N2IG1haWxpbmcgbGlzdApsaW51eC1yaXNjdkBsaXN0 cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGlu Zm8vbGludXgtcmlzY3YK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2257C32789 for ; Wed, 7 Nov 2018 02:33:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6DFE920827 for ; Wed, 7 Nov 2018 02:33:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6DFE920827 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ics.forth.gr Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389301AbeKGMCK (ORCPT ); Wed, 7 Nov 2018 07:02:10 -0500 Received: from mailgate-4.ics.forth.gr ([139.91.1.7]:19861 "EHLO mailgate-4.ics.forth.gr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388107AbeKGMCK (ORCPT ); Wed, 7 Nov 2018 07:02:10 -0500 Received: from av1.ics.forth.gr (av3in.ics.forth.gr. [139.91.1.77]) by mailgate-4.ics.forth.gr (8.14.5/ICS-FORTH/V10-1.9-GATE-OUT) with ESMTP id wA72VbKa067746; Wed, 7 Nov 2018 04:31:39 +0200 (EET) X-AuditID: 8b5b9d4d-91bff70000000e62-66-5be24e87b883 Received: from enigma.ics.forth.gr (webmail.ics.forth.gr [139.91.1.35]) by av1.ics.forth.gr (SMTP Outbound / FORTH / ICS) with SMTP id DC.89.03682.78E42EB5; Wed, 7 Nov 2018 04:31:35 +0200 (EET) Received: from webmail.ics.forth.gr (localhost [127.0.0.1]) by enigma.ics.forth.gr (8.15.1//ICS-FORTH/V10.5.0C-EXTNULL-SSL-SASL) with ESMTP id wA72VYBk028294; Wed, 7 Nov 2018 04:31:34 +0200 X-ICS-AUTH-INFO: Authenticated user: at ics.forth.gr MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Wed, 07 Nov 2018 04:31:34 +0200 From: Nick Kossifidis To: Mark Rutland , Sudeep Holla Cc: Nick Kossifidis , Atish Patra , linux-riscv@lists.infradead.org, devicetree@vger.kernel.org, Damien.LeMoal@wdc.com, alankao@andestech.com, zong@andestech.com, anup@brainfault.org, palmer@sifive.com, linux-kernel@vger.kernel.org, hch@infradead.org, robh+dt@kernel.org, tglx@linutronix.de Subject: Re: [RFC 0/2] Add RISC-V cpu topology Organization: FORTH In-Reply-To: <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> Message-ID: X-Sender: mick@mailhost.ics.forth.gr User-Agent: Roundcube Webmail/1.1.2 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrKIsWRmVeSWpSXmKPExsXSHc2orNvh9yjaYM0aSYttS1azWrR8eMdq sWjFdxaL1vZvTBbzj5xjtTg9YRGTxeVdc9gstn1uYbNYev0ik0Xzu3PsFpsnLGC1aN17hN1i +akdLBabN01ltni+spfNgd9jz+lZzB5r5q1h9Jj6+wyLx8NNl5g8Nq/Q8ti0qpPN4925c+we m5fUe1xqvs7u8XmTnEf7gW6mAO4oLpuU1JzMstQifbsEroz1B96wFdyPq9h8dw5zA+Nb9y5G Tg4JAROJJU+msncxcnEICRxhlPg68RIbhHOQUWJt00F2iCpTidl7OxlBbF4BQYmTM5+wgNjM AhYSU6/sZ4Sw5SWat85mBrFZBFQlFsw/ygpiswloSsy/dBCsXkTAR+LA/P+MIAuYBd4xSexa PJUNJCEsoCfRtOEmWAO/gLDEp7sXwWxOAQ+Jw98bWCAumsckcePYZGaIK1wkJvX2sUJcpyLx 4fcDsEtFBZQlXpyYzjqBUWgWkmNnITl2FpJjFzAyr2IUSCwz1stMLtZLyy8qydBLL9rECI7M ub47GM8tsD/EKMDBqMTDe5P9UbQQa2JZcWXuIUYJDmYlEd7Tqx9GC/GmJFZWpRblxxeV5qQW H2KU5mBREuc9/CI8SEggPbEkNTs1tSC1CCbLxMEp1cC4e1fChmPPmM6GPEjne/1237F3Cl+n zL7//oPpDJfKQA/2mYWspU2RHvLvOGU0TlsxPP6zLsbSI3mXkPP0n5lFhy3zLpV++VgjsMRA 4KGP8/49z7/MPZVz/u+bqtyL7Sz5PozPK15Eu8yWVJJ+f77Df6um+bUwOUELIaMClY4JEm/m BfmdV+lWYinOSDTUYi4qTgQAwX39NsgCAAA= Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Στις 2018-11-06 18:20, Mark Rutland έγραψε: > On Tue, Nov 06, 2018 at 05:26:01PM +0200, Nick Kossifidis wrote: >> Στις 2018-11-06 16:13, Sudeep Holla έγραψε: >> > On Fri, Nov 02, 2018 at 08:58:39PM +0200, Nick Kossifidis wrote: >> > > Στις 2018-11-02 01:04, Atish Patra έγραψε: >> > > > This patch series adds the cpu topology for RISC-V. It contains >> > > > both the DT binding and actual source code. It has been tested on >> > > > QEMU & Unleashed board. >> > > > >> > > > The idea is based on cpu-map in ARM with changes related to how >> > > > we define SMT systems. The reason for adopting a similar approach >> > > > to ARM as I feel it provides a very clear way of defining the >> > > > topology compared to parsing cache nodes to figure out which cpus >> > > > share the same package or core. I am open to any other idea to >> > > > implement cpu-topology as well. >> > > >> > > I was also about to start a discussion about CPU topology on RISC-V >> > > after the last swtools group meeting. The goal is to provide the >> > > scheduler with hints on how to distribute tasks more efficiently >> > > between harts, by populating the scheduling domain topology levels >> > > (https://elixir.bootlin.com/linux/v4.19/ident/sched_domain_topology_level). >> > > What we want to do is define cpu groups and assign them to >> > > scheduling domains with the appropriate SD_ flags >> > > (https://github.com/torvalds/linux/blob/master/include/linux/sched/topology.h#L16). >> > >> > OK are we defining a CPU topology binding for Linux scheduler ? >> > NACK for all the approaches that assumes any knowledge of OS scheduler. >> >> Is there any standard regarding CPU topology on the device tree spec ? >> As far as I know there is none. We are talking about a Linux-specific >> Device Tree binding so I don't see why defining a binding for the >> Linux scheduler is out of scope. > > Speaking as a DT binding maintainer, please avoid OS-specific DT > bindings wherever possible. > > While DT bindings live in the kernel tree, they are not intended to be > Linux-specific, and other OSs (e.g. FreeBSD, zephyr) are aiming to > support the same bindings. > > In general, targeting a specific OS is a bad idea, because the > implementation details of that OS change over time, or the bindings end > up being tailored to one specific use-case. Exposing details to the OS > such that the OS can make decisions at runtime is typically better. > >> Do you have cpu-map on other OSes as well ? > > There is nothing OS-specific about cpu-map, and it may be of use to > other OSs. > >> > > So the cores that belong to a scheduling domain may share: >> > > CPU capacity (SD_SHARE_CPUCAPACITY / SD_ASYM_CPUCAPACITY) >> > > Package resources -e.g. caches, units etc- (SD_SHARE_PKG_RESOURCES) >> > > Power domain (SD_SHARE_POWERDOMAIN) >> > > >> > >> > Too Linux kernel/scheduler specific to be part of $subject >> >> All lists on the cc list are Linux specific, again I don't see your >> point here are we talking about defining a standard CPU topology >> scheme for the device tree spec or a Linux-specific CPU topology >> binding such as cpu-map ? > > The cpu-map binding is not intended to be Linux specific, and avoids > Linux-specific terminology. > > While the cpu-map binding documentation is in the Linux source tree, > the > binding itseld is not intended to be Linux-specific, and it > deliberately > avoids Linux implementation details. > >> Even on this case your point is not valid, the information of two >> harts sharing a common power domain or having the same or not >> capacity/max frequency (or maybe capabilities/extensions in the >> future), is not Linux specific. I just used the Linux specific macros >> used by the Linux scheduler to point out the code path. Even on other >> OSes we still need a way to include this information on the CPU >> topology, and currently cpu-map doesn't. Also the Linux implementation >> of cpu-map ignores multiple levels of shared resources, we only get >> one level for SMT and one level for MC last time I checked. > > Given clusters can be nested, as in the very first example, I don't see > what prevents multiple levels of shared resources. > > Can you please given an example of the topology your considering? Does > that share some resources across clusters at some level? > > We are certainly open to improving the cpu-map binding. > > Thanks, > Mark. Mark and Sundeep thanks a lot for your feedback, I guess you convinced me that having a device tree binding for the scheduler is not a correct approach. It's not a device after all and I agree that the device tree shouldn't become an OS configuration file. Regarding multiple levels of shared resources my point is that since cpu-map doesn't contain any information of what is shared among the cluster/core members it's not easy to do any further translation. Last time I checked the arm code that uses cpu-map, it only defines one domain for SMT, one for MC and then everything else is ignored. No matter how many clusters have been defined, anything above the core level is the same (and then I guess you started talking about adding "packages" on the representation side). The reason I proposed to have a binding for the scheduler directly is not only because it's simpler and closer to what really happens in the code, it also makes more sense to me than the combination of cpu-map with all the related mappings e.g. for numa or caches or power domains etc. However you are right we could definitely augment cpu-map to include support for what I'm saying and clean things up, and since you are open about improving it here is a proposal that I hope you find interesting: At first let's get rid of the nodes, they don't make sense: thread0 { cpu = <&CPU0>; }; A thread node can't have more than one cpu entry and any properties should be on the cpu node itself, so it doesn't / can't add any more information. We could just have an array of cpu nodes on the node, it's much cleaner this way. core0 { members = <&CPU0>, <&CPU1>; }; Then let's allow the cluster and core nodes to accept attributes that are common for the cpus they contain. Right now this is considered invalid. For power domains we have a generic binding described on Documentation/devicetree/bindings/power/power_domain.txt which basically says that we need to put power-domains = attribute on each of the cpu nodes. The same happens with the capacity binding specified for arm on Documentation/devicetree/bindings/arm/cpu-capacity.txt which says we should add the capacity-dmips-mhz on each of the cpu nodes. The same also happens with the generic numa binding on Documentation/devicetree/bindings/numa.txt which says we should add the nuna-node-id on each of the cpu nodes. We could allow for these attributes to exist on cluster and core nodes as well so that we can represent their properties better. It shouldn't be a big deal and it can be done in a backwards-compatible way (if we don't find them on the cpu node, climb up the topology hierarchy until we find them / not find them at all). All I'm saying is that I prefer this: cpus { cpu@0 { ... }; cpu@1 { ... }; cpu@2 { ... }; cpu@3 { ... }; }; cluster0 { cluster0 { core0 { power-domains = <&pdc 0>; numa-node-id = <0>; capacity-dmips-mhz = <578>; members = <&cpu0>, <&cpu1>; } }; cluster1 { capacity-dmips-mhz = <1024>; core0 { power-domains = <&pdc 1>; numa-node-id = <1>; members = <&cpu2>; }; core1 { power-domains = <&pdc 2>; numa-node-id = <2>; members = <&cpu3>; }; }; } over this: cpus { cpu@0 { ... power-domains = <&pdc 0>; capacity-dmips-mhz = <578>; numa-node-id = <0>; ... }; cpu@1 { ... power-domains = <&pdc 0>; capacity-dmips-mhz = <578>; numa-node-id = <0>; ... }; cpu@2 { ... power-domains = <&pdc 1>; capacity-dmips-mhz = <1024>; numa-node-id = <1>; ... }; cpu@3 { ... power-domains = <&pdc 2>; capacity-dmips-mhz = <1024>; numa-node-id = <2>; ... }; }; cluster0 { cluster0 { core0 { members = <&cpu0>, <&cpu1>; } }; cluster1 { core0 { members = <&cpu2>; } }; cluster2 { core0 { members = <&cpu3>; } }; } When it comes to shared resources, the standard dt mappings we have are for caches and are on the device spec standard (coming from power pc's ePAPR standard I think). The below comes from HiFive unleashed's device tree (U540Config.dts) that follows the spec: cpus { cpu@1 { ... next-level-cache = <&L24 &L0>; ... }; cpu@2 { ... next-level-cache = <&L24 &L0>; ... }; cpu@3 { ... next-level-cache = <&L24 &L0>; ... }; cpu@4 { ... next-level-cache = <&L24 &L0>; ... }; }; L2: soc { L0: cache-controller@2010000 { cache-block-size = <64>; cache-level = <2>; cache-sets = <2048>; cache-size = <2097152>; cache-unified; compatible = "sifive,ccache0", "cache"; ... }; } Note that the cache-controller node that's common between the 4 cores can exist anywhere BUT the cluster node ! However it's a property of the cluster. A quick search through the tree got me r8a77980.dtsi that defines the cache on the cpus node and I'm sure there are other similar cases. Wouldn't this be better ? cluster0 { core0 { cache-controller@2010000 { cache-block-size = <64>; cache-level = <2>; cache-sets = <2048>; cache-size = <2097152>; cache-unified; compatible = "sifive,ccache0", "cache"; ... }; members = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; }; }; We could even remove next-level-cache from the cpu nodes and infer it from the topology (search the topology upwards until we get a node that's "cache"-compatible), we can again make this backwards-compatible. Finally from the examples above I'd like to stress out that the distinction between a cluster and a core doesn't make much sense and it also makes the representation more complicated. To begin with, how would you call the setup on HiFive Unleashed ? A cluster of 4 cores that share the same L3 cache ? One core with 4 harts that share the same L3 cache ? We could represent it like this instead: cluster0 { cache-controller@2010000 { cache-block-size = <64>; cache-level = <2>; cache-sets = <2048>; cache-size = <2097152>; cache-unified; compatible = "sifive,ccache0", "cache"; ... }; core0 { members = <&cpu0>; }; core1 { members = <&cpu1>; }; core2 { members = <&cpu2>; }; core3 { members = <&cpu3>; }; }; We could e.g. keep only cluster nodes and allow them to contain either an array of harts or other cluster sub-nodes + optionally a set of attributes, common to the members/sub-nodes of the cluster. This way we'll get in the first example: cluster0 { cluster0 { power-domains = <&pdc 0>; numa-node-id = <0>; capacity-dmips-mhz = <578>; members = <&cpu0>, <&cpu1>; }; cluster1 { capacity-dmips-mhz = <1024>; cluster0 { power-domains = <&pdc 1>; numa-node-id = <1>; members = <&cpu2>; }; cluster1 { power-domains = <&pdc 2>; numa-node-id = <2>; members = <&cpu3>; }; }; } and in the second example: cluster0 { cache-controller@2010000 { cache-block-size = <64>; cache-level = <2>; cache-sets = <2048>; cache-size = <2097152>; cache-unified; compatible = "sifive,ccache0", "cache"; ... }; members = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; }; Thank you for your time ! Regards, Nick