From mboxrd@z Thu Jan 1 00:00:00 1970 From: mick@ics.forth.gr (Nick Kossifidis) Date: Thu, 08 Nov 2018 16:52:30 +0200 Subject: [RFC 0/2] Add RISC-V cpu topology In-Reply-To: <20181107122803.GA8433@e107155-lin> References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> <20181107122803.GA8433@e107155-lin> Message-ID: To: linux-riscv@lists.infradead.org List-Id: linux-riscv.lists.infradead.org ???? 2018-11-07 14:28, Sudeep Holla ??????: > On Wed, Nov 07, 2018 at 04:31:34AM +0200, Nick Kossifidis wrote: > [...] > >> >> Mark and Sudeep thanks a lot for your feedback, I guess you convinced >> me >> that having a device tree binding for the scheduler is not a correct >> approach. >> > > Thanks :) > >> It's not a device after all and I agree that the device tree shouldn't >> become an OS configuration file. Regarding multiple levels of shared >> resources my point is that since cpu-map doesn't contain any >> information of >> what is shared among the cluster/core members it's not easy to do any >> further translation. Last time I checked the arm code that uses >> cpu-map, it >> only defines one domain for SMT, one for MC and then everything else >> is >> ignored. No matter how many clusters have been defined, anything above >> the >> core level is the same (and then I guess you started talking about >> adding >> "packages" on the representation side). >> > > Correct. > >> The reason I proposed to have a binding for the scheduler directly is >> not >> only because it's simpler and closer to what really happens in the >> code, it >> also makes more sense to me than the combination of cpu-map with all >> the >> related mappings e.g. for numa or caches or power domains etc. >> > > Again you are just looking at it with Linux kernel perspective. > >> However you are right we could definitely augment cpu-map to include >> support >> for what I'm saying and clean things up, and since you are open about >> improving it here is a proposal that I hope you find interesting: At >> first >> let's get rid of the nodes, they don't make sense: >> >> thread0 { >> cpu = <&CPU0>; >> }; >> > > Do you have any strong reasons to do so ? > Since it's already there for some time, I believe we can't remove it > for > backward compatibility reasons. > >> A thread node can't have more than one cpu entry and any properties >> should be on the cpu node itself, so it doesn't / can't add any >> more information. We could just have an array of cpu nodes on the >> node, it's much cleaner this way. >> >> core0 { >> members = <&CPU0>, <&CPU1>; >> }; >> > > I agree, but we have kernel code using it(arm64/kernel/topology.c). > It's > too late to remove it. But we can always keep to optional if we move > the > ARM64 binding as generic to start with and mandate it for only ARM64. > That's my point as well, if we are going to define something to be used by everybody and in this case, at least for RISC-V, there is no need to carry this from the ARM64 binding. It shouldn't be that hard to fix this in the future for ARM64 as well, we may give the new mapping another name, maybe cpu-map2 or cpu-topology to slowly move to the new one. Changing the dts files shouldn't be this hard, we can provide a script for it. We can even contain some compatibility code that also understands nodes and e.g. merges them together on a core node. >> Then let's allow the cluster and core nodes to accept attributes that >> are >> common for the cpus they contain. Right now this is considered >> invalid. >> > > Yes, we have discussed in the past and decided not to. I am fine if we > need to change it, but assuming the topology implies other information > could be wrong. On ARM platforms we have learnt it, so we kept any > information away from topology. I assume same with RISC-V, different > vendors implement in different ways, so it's better to consider those > factors. > >> For power domains we have a generic binding described on >> Documentation/devicetree/bindings/power/power_domain.txt >> which basically says that we need to put power-domains = > specifiers> attribute on each of the cpu nodes. >> > > OK, but what's wrong with that. I gives full flexibility. > >> The same happens with the capacity binding specified for arm on >> Documentation/devicetree/bindings/arm/cpu-capacity.txt >> which says we should add the capacity-dmips-mhz on each of the cpu >> nodes. >> > > Ditto, we may need this for our single cluster DSU systems. > >> The same also happens with the generic numa binding on >> Documentation/devicetree/bindings/numa.txt >> which says we should add the nuna-node-id on each of the cpu nodes. >> > > Yes, but again what's the problem ? > There is no problem with the above bindings, the problem is that we have to put them on each cpu node which is messy. We could instead put them (optionally) on the various groupings used on cpu-map. This would allow cpu-map to be more specific of what is shared across the members of each group (core/cluster/whatever). As I wrote on my answer to Mark previously, the bindings for infering the cache topology, numa topology, power domain topology etc are already there, they are in the devicet tree spec and provide very specific informations we can use. Much "stronger" hints of what's going on at the hw level. The cpu-map doesn't provide such information, it just provides a view of how the various harts/threads are "packed" on the chip, not what they share inside each level of "packing". It's useful because it saves people from having to define a bunch of cache nodes and describe the cache hierarchy on the device tree using the standard spec. So since cpu-map is there for convenience let's make it more convenient ! What I'm saying is that cpu-map could be a more compact way of using the existing bindings for adding properties on groups of harts instead of putting them on each hart individually. It will simplify the representation and may also optimize the implementation a bit (we may get the information we need faster). I don't see any other reason for using cpu-map on RISC-V or for making it global across archs. >> We could allow for these attributes to exist on cluster and core nodes >> as well so that we can represent their properties better. It shouldn't >> be a big deal and it can be done in a backwards-compatible way (if we >> don't find them on the cpu node, climb up the topology hierarchy until >> we find them / not find them at all). All I'm saying is that I prefer >> this: >> > > [...] > >> >> cluster0 { >> cluster0 { >> core0 { >> power-domains = <&pdc 0>; >> numa-node-id = <0>; >> capacity-dmips-mhz = <578>; >> members = <&cpu0>, <&cpu1>; >> } >> }; >> cluster1 { >> capacity-dmips-mhz = <1024>; >> core0 { >> power-domains = <&pdc 1>; >> numa-node-id = <1>; >> members = <&cpu2>; >> }; >> core1 { >> power-domains = <&pdc 2>; >> numa-node-id = <2>; >> members = <&cpu3>; >> }; >> }; >> } >> > > Why are you so keen on optimising the representation ? > If you are worried about large systems, generate one instead of > handcrafted. > I don't see a reason not to try to optimize it, since we are talking about a binding to be used by RISC-V and potentially everybody, I think it makes sens to improve upon what we already have. >> When it comes to shared resources, the standard dt mappings we have >> are for >> caches and are on the device spec standard (coming from power pc's >> ePAPR >> standard I think). The below comes from HiFive unleashed's device tree >> (U540Config.dts) that follows the spec: >> > > I don't understand what you are trying to explain, ePAPR does specify > per CPU entries. > > [...] > I'm saying we could allow moving these properties on the groupings of cpu-map to simplify the representation and possibly optimize the implementation. This way I believe cpu-map will be much more useful than it is now. >> Note that the cache-controller node that's common between the 4 cores >> can >> exist anywhere BUT the cluster node ! However it's a property of the >> cluster. >> A quick search through the tree got me r8a77980.dtsi that defines the >> cache >> on the cpus node and I'm sure there are other similar cases. Wouldn't >> this >> be better ? >> >> cluster0 { >> core0 { >> cache-controller at 2010000 { >> cache-block-size = <64>; >> cache-level = <2>; >> cache-sets = <2048>; >> cache-size = <2097152>; >> cache-unified; >> compatible = "sifive,ccache0", "cache"; >> ... >> }; >> members = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; > > Not a good idea IMO. > Where in your opinion should this cache node go ? On the first example from a production dts it goes to the SoC node and on another production dts it goes to the cpus/ node. I think it makes more sense to go to the core node or the cluster node, depending on how it's shared. It makes it also easier for the implementation to understand the levels of shared caches and creating cpu masks. >> We could even remove next-level-cache from the cpu nodes and infer it >> from >> the topology (search the topology upwards until we get a node that's >> "cache"-compatible), we can again make this backwards-compatible. >> > > Why are you assuming that they *have* to be so aligned with topology ? > How do you deal with different kind of systems ? > It depends on how you define topology. I believe that the cache hierarchy is a much "stronger" definition of the CPU topology than grouping harts/threads on classes like "core", "socket", "packet" etc that don't provide any information of what's going on inside the hardware. You can think of the question the other way around, why are you assuming that the current topology as specified on cpu-map is aligned with how the harts and cores share their "resources" ? Because that's what's going on at the implementation side of cpu-map. >> >> Finally from the examples above I'd like to stress out that the >> distinction >> between a cluster and a core doesn't make much sense and it also makes >> the >> representation more complicated. To begin with, how would you call the >> setup >> on HiFive Unleashed ? A cluster of 4 cores that share the same L3 >> cache ? >> One core with 4 harts that share the same L3 cache ? We could >> represent it >> like this instead: >> > > Just represent each physical cache and get the list of CPUs sharing it. > Doesn't matter what it is: cluster or cluster of clusters or cluster of > harts, blah, blah. It really doesn't matter. > I totally agree with you, but in this case what's the reason of having cpu-map ? We can infer the topology from what we already have, at least cpu-map adds some convenience (but if we go all the way down the "convinience" path we'll end up on something like my initial approach which was as we both agree now, wrong). >> >> We could e.g. keep only cluster nodes and allow them to contain either >> an >> array of harts or other cluster sub-nodes + optionally a set of >> attributes, >> common to the members/sub-nodes of the cluster. This way we'll get in >> the >> first example: >> > > All these fancy new ideas you are proposing are good if vendors follow > some things religiously, but I really doubt if that's true. So just > have maximum flexibility so that we can represent maximum number of > systems without having to redefine the bindings again and again for the > same thing. > > So instead of finding ways to optimise, you should really come up with > list of shortcomings in the existing bindings so that we cover more > platforms with generic bindings. IMO you are too concerned on > optimisation > of DT representation which may defeat the purpose of generic bindings. > I get your point, but if that's the case, isn't cpu-map an optimization over the generic bindings for cpu nodes anyway ? As for the flexibility, what I'm saying is to allow the flexibility of adding the properties that we would otherwise add to cpu nodes, on the groups we use on cpu-map. Optionaly so that we are also backwards compatible. I see this as more flexible, not less flexible. The same goes for removing the distinction between "core", "cluster" etc, having specific names for each of the topology levels is IMHO less flexible than using abstract names. Also what's the point of defining a binding if vendors don't follow it ? I think it would be easier for the vendors to just use what's there instead of defining a new binding and writing the code for it. On the other hand you are probably more experienced on this than I am, I haven't dealt with various different vendors and their different ways. Regards, Nick From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA793ECDE4C for ; Thu, 8 Nov 2018 14:54:45 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 75A3520827 for ; Thu, 8 Nov 2018 14:54:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ZU0lpE8Z"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Z1tIIIf8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 75A3520827 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ics.forth.gr Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-ID:References:In-Reply-To:Subject:To:From: Date:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=ar9n1TF187hClD8119oeICed2kk6MlJAFcH8+Dd48PU=; b=ZU0lpE8Z+5e/ScW/MrEtxqbc8 L7I+iUxmi2X7vGnjMJ/Bh9WH1N/r/Nw27f6mMgCIDVjpZHLnXAscP89jeDS3xUyg0iC4IhOyfO4W7 SyLrfzB98XNW5dMMSzAnPJhcl0ar55r4NgV8JzbVcMBF3OAQSQ1X9+axSZ7FizKUAITs8Jt3mROGd 4zqGB9xI4qE5HJHuE4s/uHazHm1lBcGK+8NLvS02faJ4pN9MuKCzgjFVP0b6EpMaJ4JMfH7iBDC5b SPQNRqN8NnxQQXnbPeF+29djziesqojRpQxwWuwFfwGVIMP9WbfwR6vszI7/YZmPq2tF4T22/I8cg vIFbWaqUQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKlhi-0003A9-IN; Thu, 08 Nov 2018 14:54:42 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKlhb-0002vt-Od for linux-riscv@bombadil.infradead.org; Thu, 08 Nov 2018 14:54:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Message-ID:References:In-Reply-To: Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version: Sender:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=CsWvRV03WaDwU1P7X/xpmxVPCSRbfySL5IWfo70dMRQ=; b=Z1tIIIf88dFP/2NPvzHb03/HzN f0x2PUasqLRMaXVv1752At59k1GSIjvKlCFi1J0IG7CjdRChOnBVLinDHe3EvlrmfYzyFCSoaW+uU rfcmeMXfeYvFYKKJ+Jad6+31g39dIssaybbU/XAEiNmL/dI6vkdOUgR+dtcFXXAMMH380i+iFyonL FLqrrLymEMVjfweDC+OfTJwFJi527JTbs/ec2lQSphEzAbTDu6zyuNrE8GAhf3oU5KUSQk4A10E9b EH+yFXt6CIXZzi/cZagBIFlUnnuOsVgV2tg3/QeR7VsaE4dKY/R2SE/0BAVEnY8uau90/wR5r0WRp 1cpeFzzA==; Received: from mailgate-4.ics.forth.gr ([139.91.1.7]) by casper.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKlhW-0005YV-Qr for linux-riscv@lists.infradead.org; Thu, 08 Nov 2018 14:54:34 +0000 Received: from av1.ics.forth.gr (av3in.ics.forth.gr. [139.91.1.77]) by mailgate-4.ics.forth.gr (8.14.5/ICS-FORTH/V10-1.9-GATE-OUT) with ESMTP id wA8EqX7T005288; Thu, 8 Nov 2018 16:52:35 +0200 (EET) X-AuditID: 8b5b9d4d-903ff70000000e62-d1-5be44db052f8 Received: from enigma.ics.forth.gr (enigma.ics.forth.gr [139.91.1.35]) by av1.ics.forth.gr (SMTP Outbound / FORTH / ICS) with SMTP id E1.DB.03682.0BD44EB5; Thu, 8 Nov 2018 16:52:32 +0200 (EET) Received: from webmail.ics.forth.gr (localhost [127.0.0.1]) by enigma.ics.forth.gr (8.15.1//ICS-FORTH/V10.5.0C-EXTNULL-SSL-SASL) with ESMTP id wA8EqUhc031162; Thu, 8 Nov 2018 16:52:30 +0200 X-ICS-AUTH-INFO: Authenticated user: at ics.forth.gr MIME-Version: 1.0 Date: Thu, 08 Nov 2018 16:52:30 +0200 From: Nick Kossifidis To: Sudeep Holla Subject: Re: [RFC 0/2] Add RISC-V cpu topology Organization: FORTH In-Reply-To: <20181107122803.GA8433@e107155-lin> References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> <20181107122803.GA8433@e107155-lin> Message-ID: X-Sender: mick@mailhost.ics.forth.gr User-Agent: Roundcube Webmail/1.1.2 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrCIsWRmVeSWpSXmKPExsXSHc2orLvR90m0wfFrbBbblqxmtWj58I7V YtGK7ywWre3fmCzmHznHanF6wiImi8u75gCVfG5hs1h6/SKTRfO7c+wWmycsYLVo3XuE3WL5 qR0sFps3TWW2eL6yl82B32PP6VnMHmvmrWH0mPr7DIvHw02XmDw2r9Dy2LSqk83j3blz7B6b l9R7XGq+zu7xeZOcR/uBbqYA7igum5TUnMyy1CJ9uwSujOsbjzIWHE6umHyitIFxj28XIyeH hICJxKL7z1i7GLk4hAQOM0qcWLiUEcI5yCjx4+0NRogqU4nZezvBbF4BQYmTM5+wgNjMAhYS U6/sZ4Sw5SWat85mBrFZBFQlHjb1sYHYbAKaEvMvHQSrFxFQl1hydgvYAmaBecwS0/dsYQVJ CAvoSTRtuAlm8wsIS3y6exHM5hQwkNjUMZ8Z4qL/TBIN57ayQ1zhInFr9gk2iOtUJD78fgAW FxVQlnhxYjrrBEahWUiOnYXk2FlIjl3AyLyKUSCxzFgvM7lYLy2/qCRDL71oEyM4Luf67mA8 t8D+EKMAB6MSD6+E4uNoIdbEsuLK3EOMEhzMSiK8m3SeRAvxpiRWVqUW5ccXleakFh9ilOZg URLnPfwiPEhIID2xJDU7NbUgtQgmy8TBKdXAuC09rOjfo8Pr9n+TNXQ4+Wiqmvbl/Xs+/62x m7rdccKrw1fPxZdKb1/uKeTLIP3lXtLpkz5vD6h0PF5r5mV/6k54nn723zfOvXtWvJNuZVM2 yV45XbLvrH/f0f59eTtuL1Q7LTL7o1vMeX5x501Nx368s3q04/1Sl4aV2U/uNupNm3DhKN82 +ZVKLMUZiYZazEXFiQAJJshIxwIAAA== X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181108_145431_130091_CD005502 X-CRM114-Status: GOOD ( 72.21 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , devicetree@vger.kernel.org, Damien.LeMoal@wdc.com, alankao@andestech.com, hch@infradead.org, anup@brainfault.org, palmer@sifive.com, linux-kernel@vger.kernel.org, zong@andestech.com, Atish Patra , robh+dt@kernel.org, Nick Kossifidis , linux-riscv@lists.infradead.org, tglx@linutronix.de Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-riscv" Errors-To: linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org Message-ID: <20181108145230.9yVk-k2egFT1e4VVgzSCFvj1d5iVGavDLWGb3Ss4hjA@z> zqPPhM65z4IgMjAxOC0xMS0wNyAxNDoyOCwgU3VkZWVwIEhvbGxhIM6tzrPPgc6xz4jOtToKPiBP biBXZWQsIE5vdiAwNywgMjAxOCBhdCAwNDozMTozNEFNICswMjAwLCBOaWNrIEtvc3NpZmlkaXMg d3JvdGU6Cj4gWy4uLl0KPiAKPj4gCj4+IE1hcmsgYW5kIFN1ZGVlcCB0aGFua3MgYSBsb3QgZm9y IHlvdXIgZmVlZGJhY2ssIEkgZ3Vlc3MgeW91IGNvbnZpbmNlZCAKPj4gbWUKPj4gdGhhdCBoYXZp bmcgYSBkZXZpY2UgdHJlZSBiaW5kaW5nIGZvciB0aGUgc2NoZWR1bGVyIGlzIG5vdCBhIGNvcnJl Y3QKPj4gYXBwcm9hY2guCj4+IAo+IAo+IFRoYW5rcyA6KQo+IAo+PiBJdCdzIG5vdCBhIGRldmlj ZSBhZnRlciBhbGwgYW5kIEkgYWdyZWUgdGhhdCB0aGUgZGV2aWNlIHRyZWUgc2hvdWxkbid0Cj4+ IGJlY29tZSBhbiBPUyBjb25maWd1cmF0aW9uIGZpbGUuIFJlZ2FyZGluZyBtdWx0aXBsZSBsZXZl bHMgb2Ygc2hhcmVkCj4+IHJlc291cmNlcyBteSBwb2ludCBpcyB0aGF0IHNpbmNlIGNwdS1tYXAg ZG9lc24ndCBjb250YWluIGFueSAKPj4gaW5mb3JtYXRpb24gb2YKPj4gd2hhdCBpcyBzaGFyZWQg YW1vbmcgdGhlIGNsdXN0ZXIvY29yZSBtZW1iZXJzIGl0J3Mgbm90IGVhc3kgdG8gZG8gYW55Cj4+ IGZ1cnRoZXIgdHJhbnNsYXRpb24uIExhc3QgdGltZSBJIGNoZWNrZWQgdGhlIGFybSBjb2RlIHRo YXQgdXNlcyAKPj4gY3B1LW1hcCwgaXQKPj4gb25seSBkZWZpbmVzIG9uZSBkb21haW4gZm9yIFNN VCwgb25lIGZvciBNQyBhbmQgdGhlbiBldmVyeXRoaW5nIGVsc2UgCj4+IGlzCj4+IGlnbm9yZWQu IE5vIG1hdHRlciBob3cgbWFueSBjbHVzdGVycyBoYXZlIGJlZW4gZGVmaW5lZCwgYW55dGhpbmcg YWJvdmUgCj4+IHRoZQo+PiBjb3JlIGxldmVsIGlzIHRoZSBzYW1lIChhbmQgdGhlbiBJIGd1ZXNz IHlvdSBzdGFydGVkIHRhbGtpbmcgYWJvdXQgCj4+IGFkZGluZwo+PiAicGFja2FnZXMiIG9uIHRo ZSByZXByZXNlbnRhdGlvbiBzaWRlKS4KPj4gCj4gCj4gQ29ycmVjdC4KPiAKPj4gVGhlIHJlYXNv biBJIHByb3Bvc2VkIHRvIGhhdmUgYSBiaW5kaW5nIGZvciB0aGUgc2NoZWR1bGVyIGRpcmVjdGx5 IGlzIAo+PiBub3QKPj4gb25seSBiZWNhdXNlIGl0J3Mgc2ltcGxlciBhbmQgY2xvc2VyIHRvIHdo YXQgcmVhbGx5IGhhcHBlbnMgaW4gdGhlIAo+PiBjb2RlLCBpdAo+PiBhbHNvIG1ha2VzIG1vcmUg c2Vuc2UgdG8gbWUgdGhhbiB0aGUgY29tYmluYXRpb24gb2YgY3B1LW1hcCB3aXRoIGFsbCAKPj4g dGhlCj4+IHJlbGF0ZWQgbWFwcGluZ3MgZS5nLiBmb3IgbnVtYSBvciBjYWNoZXMgb3IgcG93ZXIg ZG9tYWlucyBldGMuCj4+IAo+IAo+IEFnYWluIHlvdSBhcmUganVzdCBsb29raW5nIGF0IGl0IHdp dGggTGludXgga2VybmVsIHBlcnNwZWN0aXZlLgo+IAo+PiBIb3dldmVyIHlvdSBhcmUgcmlnaHQg d2UgY291bGQgZGVmaW5pdGVseSBhdWdtZW50IGNwdS1tYXAgdG8gaW5jbHVkZSAKPj4gc3VwcG9y dAo+PiBmb3Igd2hhdCBJJ20gc2F5aW5nIGFuZCBjbGVhbiB0aGluZ3MgdXAsIGFuZCBzaW5jZSB5 b3UgYXJlIG9wZW4gYWJvdXQKPj4gaW1wcm92aW5nIGl0IGhlcmUgaXMgYSBwcm9wb3NhbCB0aGF0 IEkgaG9wZSB5b3UgZmluZCBpbnRlcmVzdGluZzogQXQgCj4+IGZpcnN0Cj4+IGxldCdzIGdldCBy aWQgb2YgdGhlIDx0aHJlYWQ+IG5vZGVzLCB0aGV5IGRvbid0IG1ha2Ugc2Vuc2U6Cj4+IAo+PiB0 aHJlYWQwIHsKPj4gIGNwdSA9IDwmQ1BVMD47Cj4+IH07Cj4+IAo+IAo+IERvIHlvdSBoYXZlIGFu eSBzdHJvbmcgcmVhc29ucyB0byBkbyBzbyA/Cj4gU2luY2UgaXQncyBhbHJlYWR5IHRoZXJlIGZv ciBzb21lIHRpbWUsIEkgYmVsaWV2ZSB3ZSBjYW4ndCByZW1vdmUgaXQgCj4gZm9yCj4gYmFja3dh cmQgY29tcGF0aWJpbGl0eSByZWFzb25zLgo+IAo+PiBBIHRocmVhZCBub2RlIGNhbid0IGhhdmUg bW9yZSB0aGFuIG9uZSBjcHUgZW50cnkgYW5kIGFueSBwcm9wZXJ0aWVzCj4+IHNob3VsZCBiZSBv biB0aGUgY3B1IG5vZGUgaXRzZWxmLCBzbyBpdCBkb2Vzbid0IC8gY2FuJ3QgYWRkIGFueQo+PiBt b3JlIGluZm9ybWF0aW9uLiBXZSBjb3VsZCBqdXN0IGhhdmUgYW4gYXJyYXkgb2YgY3B1IG5vZGVz IG9uIHRoZQo+PiA8Y29yZT4gbm9kZSwgaXQncyBtdWNoIGNsZWFuZXIgdGhpcyB3YXkuCj4+IAo+ PiBjb3JlMCB7Cj4+ICBtZW1iZXJzID0gPCZDUFUwPiwgPCZDUFUxPjsKPj4gfTsKPj4gCj4gCj4g SSBhZ3JlZSwgYnV0IHdlIGhhdmUga2VybmVsIGNvZGUgdXNpbmcgaXQoYXJtNjQva2VybmVsL3Rv cG9sb2d5LmMpLiAKPiBJdCdzCj4gdG9vIGxhdGUgdG8gcmVtb3ZlIGl0LiBCdXQgd2UgY2FuIGFs d2F5cyBrZWVwIHRvIG9wdGlvbmFsIGlmIHdlIG1vdmUgCj4gdGhlCj4gQVJNNjQgYmluZGluZyBh cyBnZW5lcmljIHRvIHN0YXJ0IHdpdGggYW5kIG1hbmRhdGUgaXQgZm9yIG9ubHkgQVJNNjQuCj4g CgpUaGF0J3MgbXkgcG9pbnQgYXMgd2VsbCwgaWYgd2UgYXJlIGdvaW5nIHRvIGRlZmluZSBzb21l dGhpbmcgdG8gYmUgdXNlZApieSBldmVyeWJvZHkgYW5kIGluIHRoaXMgY2FzZSwgYXQgbGVhc3Qg Zm9yIFJJU0MtViwgdGhlcmUgaXMgbm8gbmVlZCB0bwpjYXJyeSB0aGlzIGZyb20gdGhlIEFSTTY0 IGJpbmRpbmcuIEl0IHNob3VsZG4ndCBiZSB0aGF0IGhhcmQgdG8gZml4IHRoaXMKaW4gdGhlIGZ1 dHVyZSBmb3IgQVJNNjQgYXMgd2VsbCwgd2UgbWF5IGdpdmUgdGhlIG5ldyBtYXBwaW5nIGFub3Ro ZXIgCm5hbWUsCm1heWJlIGNwdS1tYXAyIG9yIGNwdS10b3BvbG9neSB0byBzbG93bHkgbW92ZSB0 byB0aGUgbmV3IG9uZS4gQ2hhbmdpbmcgCnRoZQpkdHMgZmlsZXMgc2hvdWxkbid0IGJlIHRoaXMg aGFyZCwgd2UgY2FuIHByb3ZpZGUgYSBzY3JpcHQgZm9yIGl0LiBXZSBjYW4KZXZlbiBjb250YWlu IHNvbWUgY29tcGF0aWJpbGl0eSBjb2RlIHRoYXQgYWxzbyB1bmRlcnN0YW5kcyA8dGhyZWFkPiAK bm9kZXMKYW5kIGUuZy4gbWVyZ2VzIHRoZW0gdG9nZXRoZXIgb24gYSBjb3JlIG5vZGUuCgo+PiBU aGVuIGxldCdzIGFsbG93IHRoZSBjbHVzdGVyIGFuZCBjb3JlIG5vZGVzIHRvIGFjY2VwdCBhdHRy aWJ1dGVzIHRoYXQgCj4+IGFyZQo+PiBjb21tb24gZm9yIHRoZSBjcHVzIHRoZXkgY29udGFpbi4g UmlnaHQgbm93IHRoaXMgaXMgY29uc2lkZXJlZCAKPj4gaW52YWxpZC4KPj4gCj4gCj4gWWVzLCB3 ZSBoYXZlIGRpc2N1c3NlZCBpbiB0aGUgcGFzdCBhbmQgZGVjaWRlZCBub3QgdG8uIEkgYW0gZmlu ZSBpZiB3ZQo+IG5lZWQgdG8gY2hhbmdlIGl0LCBidXQgYXNzdW1pbmcgdGhlIHRvcG9sb2d5IGlt cGxpZXMgb3RoZXIgaW5mb3JtYXRpb24KPiBjb3VsZCBiZSB3cm9uZy4gT24gQVJNIHBsYXRmb3Jt cyB3ZSBoYXZlIGxlYXJudCBpdCwgc28gd2Uga2VwdCBhbnkKPiBpbmZvcm1hdGlvbiBhd2F5IGZy b20gdG9wb2xvZ3kuIEkgYXNzdW1lIHNhbWUgd2l0aCBSSVNDLVYsIGRpZmZlcmVudAo+IHZlbmRv cnMgaW1wbGVtZW50IGluIGRpZmZlcmVudCB3YXlzLCBzbyBpdCdzIGJldHRlciB0byBjb25zaWRl ciB0aG9zZQo+IGZhY3RvcnMuCj4gCj4+IEZvciBwb3dlciBkb21haW5zIHdlIGhhdmUgYSBnZW5l cmljIGJpbmRpbmcgZGVzY3JpYmVkIG9uCj4+IERvY3VtZW50YXRpb24vZGV2aWNldHJlZS9iaW5k aW5ncy9wb3dlci9wb3dlcl9kb21haW4udHh0Cj4+IHdoaWNoIGJhc2ljYWxseSBzYXlzIHRoYXQg d2UgbmVlZCB0byBwdXQgcG93ZXItZG9tYWlucyA9IDxwb3dlciBkb21haW4KPj4gc3BlY2lmaWVy cz4gYXR0cmlidXRlIG9uIGVhY2ggb2YgdGhlIGNwdSBub2Rlcy4KPj4gCj4gCj4gT0ssIGJ1dCB3 aGF0J3Mgd3Jvbmcgd2l0aCB0aGF0LiBJIGdpdmVzIGZ1bGwgZmxleGliaWxpdHkuCj4gCj4+IFRo ZSBzYW1lIGhhcHBlbnMgd2l0aCB0aGUgY2FwYWNpdHkgYmluZGluZyBzcGVjaWZpZWQgZm9yIGFy bSBvbgo+PiBEb2N1bWVudGF0aW9uL2RldmljZXRyZWUvYmluZGluZ3MvYXJtL2NwdS1jYXBhY2l0 eS50eHQKPj4gd2hpY2ggc2F5cyB3ZSBzaG91bGQgYWRkIHRoZSBjYXBhY2l0eS1kbWlwcy1taHog b24gZWFjaCBvZiB0aGUgY3B1IAo+PiBub2Rlcy4KPj4gCj4gCj4gRGl0dG8sIHdlIG1heSBuZWVk IHRoaXMgZm9yIG91ciBzaW5nbGUgY2x1c3RlciBEU1Ugc3lzdGVtcy4KPiAKPj4gVGhlIHNhbWUg YWxzbyBoYXBwZW5zIHdpdGggdGhlIGdlbmVyaWMgbnVtYSBiaW5kaW5nIG9uCj4+IERvY3VtZW50 YXRpb24vZGV2aWNldHJlZS9iaW5kaW5ncy9udW1hLnR4dAo+PiB3aGljaCBzYXlzIHdlIHNob3Vs ZCBhZGQgdGhlIG51bmEtbm9kZS1pZCBvbiBlYWNoIG9mIHRoZSBjcHUgbm9kZXMuCj4+IAo+IAo+ IFllcywgYnV0IGFnYWluIHdoYXQncyB0aGUgcHJvYmxlbSA/Cj4gCgpUaGVyZSBpcyBubyBwcm9i bGVtIHdpdGggdGhlIGFib3ZlIGJpbmRpbmdzLCB0aGUgcHJvYmxlbSBpcyB0aGF0IHdlIGhhdmUK dG8gcHV0IHRoZW0gb24gZWFjaCBjcHUgbm9kZSB3aGljaCBpcyBtZXNzeS4gV2UgY291bGQgaW5z dGVhZCBwdXQgdGhlbQoob3B0aW9uYWxseSkgb24gdGhlIHZhcmlvdXMgZ3JvdXBpbmdzIHVzZWQg b24gY3B1LW1hcC4gVGhpcyB3b3VsZCBhbGxvdwpjcHUtbWFwIHRvIGJlIG1vcmUgc3BlY2lmaWMg b2Ygd2hhdCBpcyBzaGFyZWQgYWNyb3NzIHRoZSBtZW1iZXJzIG9mIGVhY2gKZ3JvdXAgKGNvcmUv Y2x1c3Rlci93aGF0ZXZlcikuCgpBcyBJIHdyb3RlIG9uIG15IGFuc3dlciB0byBNYXJrIHByZXZp b3VzbHksIHRoZSBiaW5kaW5ncyBmb3IgaW5mZXJpbmcKdGhlIGNhY2hlIHRvcG9sb2d5LCBudW1h IHRvcG9sb2d5LCBwb3dlciBkb21haW4gdG9wb2xvZ3kgZXRjIGFyZSBhbHJlYWR5CnRoZXJlLCB0 aGV5IGFyZSBpbiB0aGUgZGV2aWNldCB0cmVlIHNwZWMgYW5kIHByb3ZpZGUgdmVyeSBzcGVjaWZp YwppbmZvcm1hdGlvbnMgd2UgY2FuIHVzZS4gTXVjaCAic3Ryb25nZXIiIGhpbnRzIG9mIHdoYXQn cyBnb2luZyBvbiBhdAp0aGUgaHcgbGV2ZWwuIFRoZSBjcHUtbWFwIGRvZXNuJ3QgcHJvdmlkZSBz dWNoIGluZm9ybWF0aW9uLCBpdCBqdXN0CnByb3ZpZGVzIGEgdmlldyBvZiBob3cgdGhlIHZhcmlv dXMgaGFydHMvdGhyZWFkcyBhcmUgInBhY2tlZCIgb24gdGhlIApjaGlwLApub3Qgd2hhdCB0aGV5 IHNoYXJlIGluc2lkZSBlYWNoIGxldmVsIG9mICJwYWNraW5nIi4gSXQncyB1c2VmdWwgYmVjYXVz ZQppdCBzYXZlcyBwZW9wbGUgZnJvbSBoYXZpbmcgdG8gZGVmaW5lIGEgYnVuY2ggb2YgY2FjaGUg bm9kZXMgYW5kIApkZXNjcmliZQp0aGUgY2FjaGUgaGllcmFyY2h5IG9uIHRoZSBkZXZpY2UgdHJl ZSB1c2luZyB0aGUgc3RhbmRhcmQgc3BlYy4KClNvIHNpbmNlIGNwdS1tYXAgaXMgdGhlcmUgZm9y IGNvbnZlbmllbmNlIGxldCdzIG1ha2UgaXQgbW9yZSBjb252ZW5pZW50IAohCldoYXQgSSdtIHNh eWluZyBpcyB0aGF0IGNwdS1tYXAgY291bGQgYmUgYSBtb3JlIGNvbXBhY3Qgd2F5IG9mIHVzaW5n IHRoZQpleGlzdGluZyBiaW5kaW5ncyBmb3IgYWRkaW5nIHByb3BlcnRpZXMgb24gZ3JvdXBzIG9m IGhhcnRzIGluc3RlYWQgb2YKcHV0dGluZyB0aGVtIG9uIGVhY2ggaGFydCBpbmRpdmlkdWFsbHku IEl0IHdpbGwgc2ltcGxpZnkgdGhlIApyZXByZXNlbnRhdGlvbgphbmQgbWF5IGFsc28gb3B0aW1p emUgdGhlIGltcGxlbWVudGF0aW9uIGEgYml0ICh3ZSBtYXkgZ2V0IHRoZSAKaW5mb3JtYXRpb24K d2UgbmVlZCBmYXN0ZXIpLiBJIGRvbid0IHNlZSBhbnkgb3RoZXIgcmVhc29uIGZvciB1c2luZyBj cHUtbWFwIG9uIApSSVNDLVYKb3IgZm9yIG1ha2luZyBpdCBnbG9iYWwgYWNyb3NzIGFyY2hzLgoK Pj4gV2UgY291bGQgYWxsb3cgZm9yIHRoZXNlIGF0dHJpYnV0ZXMgdG8gZXhpc3Qgb24gY2x1c3Rl ciBhbmQgY29yZSBub2Rlcwo+PiBhcyB3ZWxsIHNvIHRoYXQgd2UgY2FuIHJlcHJlc2VudCB0aGVp ciBwcm9wZXJ0aWVzIGJldHRlci4gSXQgc2hvdWxkbid0Cj4+IGJlIGEgYmlnIGRlYWwgYW5kIGl0 IGNhbiBiZSBkb25lIGluIGEgYmFja3dhcmRzLWNvbXBhdGlibGUgd2F5IChpZiB3ZQo+PiBkb24n dCBmaW5kIHRoZW0gb24gdGhlIGNwdSBub2RlLCBjbGltYiB1cCB0aGUgdG9wb2xvZ3kgaGllcmFy Y2h5IHVudGlsCj4+IHdlIGZpbmQgdGhlbSAvIG5vdCBmaW5kIHRoZW0gYXQgYWxsKS4gQWxsIEkn bSBzYXlpbmcgaXMgdGhhdCBJIHByZWZlciAKPj4gdGhpczoKPj4gCj4gCj4gWy4uLl0KPiAKPj4g Cj4+IGNsdXN0ZXIwIHsKPj4gIGNsdXN0ZXIwIHsKPj4gICBjb3JlMCB7Cj4+ICAgIHBvd2VyLWRv bWFpbnMgPSA8JnBkYyAwPjsKPj4gICAgbnVtYS1ub2RlLWlkID0gPDA+Owo+PiAgICBjYXBhY2l0 eS1kbWlwcy1taHogPSA8NTc4PjsKPj4gICAgbWVtYmVycyA9IDwmY3B1MD4sIDwmY3B1MT47Cj4+ ICAgfQo+PiAgfTsKPj4gIGNsdXN0ZXIxIHsKPj4gICBjYXBhY2l0eS1kbWlwcy1taHogPSA8MTAy ND47Cj4+ICAgY29yZTAgewo+PiAgICBwb3dlci1kb21haW5zID0gPCZwZGMgMT47Cj4+ICAgIG51 bWEtbm9kZS1pZCA9IDwxPjsKPj4gICAgbWVtYmVycyA9IDwmY3B1Mj47Cj4+ICAgfTsKPj4gICBj b3JlMSB7Cj4+ICAgIHBvd2VyLWRvbWFpbnMgPSA8JnBkYyAyPjsKPj4gICAgbnVtYS1ub2RlLWlk ID0gPDI+Owo+PiAgICBtZW1iZXJzID0gPCZjcHUzPjsKPj4gICB9Owo+PiAgfTsKPj4gfQo+PiAK PiAKPiBXaHkgYXJlIHlvdSBzbyBrZWVuIG9uIG9wdGltaXNpbmcgdGhlIHJlcHJlc2VudGF0aW9u ID8KPiBJZiB5b3UgYXJlIHdvcnJpZWQgYWJvdXQgbGFyZ2Ugc3lzdGVtcywgZ2VuZXJhdGUgb25l IGluc3RlYWQgb2YKPiBoYW5kY3JhZnRlZC4KPiAKCkkgZG9uJ3Qgc2VlIGEgcmVhc29uIG5vdCB0 byB0cnkgdG8gb3B0aW1pemUgaXQsIHNpbmNlIHdlIGFyZSB0YWxraW5nCmFib3V0IGEgYmluZGlu ZyB0byBiZSB1c2VkIGJ5IFJJU0MtViBhbmQgcG90ZW50aWFsbHkgZXZlcnlib2R5LCBJIHRoaW5r Cml0IG1ha2VzIHNlbnMgdG8gaW1wcm92ZSB1cG9uIHdoYXQgd2UgYWxyZWFkeSBoYXZlLgoKPj4g V2hlbiBpdCBjb21lcyB0byBzaGFyZWQgcmVzb3VyY2VzLCB0aGUgc3RhbmRhcmQgZHQgbWFwcGlu Z3Mgd2UgaGF2ZSAKPj4gYXJlIGZvcgo+PiBjYWNoZXMgYW5kIGFyZSBvbiB0aGUgZGV2aWNlIHNw ZWMgc3RhbmRhcmQgKGNvbWluZyBmcm9tIHBvd2VyIHBjJ3MgCj4+IGVQQVBSCj4+IHN0YW5kYXJk IEkgdGhpbmspLiBUaGUgYmVsb3cgY29tZXMgZnJvbSBIaUZpdmUgdW5sZWFzaGVkJ3MgZGV2aWNl IHRyZWUKPj4gKFU1NDBDb25maWcuZHRzKSB0aGF0IGZvbGxvd3MgdGhlIHNwZWM6Cj4+IAo+IAo+ IEkgZG9uJ3QgdW5kZXJzdGFuZCB3aGF0IHlvdSBhcmUgdHJ5aW5nIHRvIGV4cGxhaW4sIGVQQVBS IGRvZXMgc3BlY2lmeQo+IHBlciBDUFUgZW50cmllcy4KPiAKPiBbLi4uXQo+IAoKSSdtIHNheWlu ZyB3ZSBjb3VsZCBhbGxvdyBtb3ZpbmcgdGhlc2UgcHJvcGVydGllcyBvbiB0aGUgZ3JvdXBpbmdz IG9mIApjcHUtbWFwCnRvIHNpbXBsaWZ5IHRoZSByZXByZXNlbnRhdGlvbiBhbmQgcG9zc2libHkg b3B0aW1pemUgdGhlIGltcGxlbWVudGF0aW9uLiAKVGhpcwp3YXkgSSBiZWxpZXZlIGNwdS1tYXAg d2lsbCBiZSBtdWNoIG1vcmUgdXNlZnVsIHRoYW4gaXQgaXMgbm93LgoKPj4gTm90ZSB0aGF0IHRo ZSBjYWNoZS1jb250cm9sbGVyIG5vZGUgdGhhdCdzIGNvbW1vbiBiZXR3ZWVuIHRoZSA0IGNvcmVz IAo+PiBjYW4KPj4gZXhpc3QgYW55d2hlcmUgQlVUIHRoZSBjbHVzdGVyIG5vZGUgISBIb3dldmVy IGl0J3MgYSBwcm9wZXJ0eSBvZiB0aGUKPj4gY2x1c3Rlci4KPj4gQSBxdWljayBzZWFyY2ggdGhy b3VnaCB0aGUgdHJlZSBnb3QgbWUgcjhhNzc5ODAuZHRzaSB0aGF0IGRlZmluZXMgdGhlIAo+PiBj YWNoZQo+PiBvbiB0aGUgY3B1cyBub2RlIGFuZCBJJ20gc3VyZSB0aGVyZSBhcmUgb3RoZXIgc2lt aWxhciBjYXNlcy4gV291bGRuJ3QgCj4+IHRoaXMKPj4gYmUgYmV0dGVyID8KPj4gCj4+IGNsdXN0 ZXIwIHsKPj4gIGNvcmUwIHsKPj4gICBjYWNoZS1jb250cm9sbGVyQDIwMTAwMDAgewo+PiAgICBj YWNoZS1ibG9jay1zaXplID0gPDY0PjsKPj4gICAgY2FjaGUtbGV2ZWwgPSA8Mj47Cj4+ICAgIGNh Y2hlLXNldHMgPSA8MjA0OD47Cj4+ICAgIGNhY2hlLXNpemUgPSA8MjA5NzE1Mj47Cj4+ICAgIGNh Y2hlLXVuaWZpZWQ7Cj4+ICAgIGNvbXBhdGlibGUgPSAic2lmaXZlLGNjYWNoZTAiLCAiY2FjaGUi Owo+PiAgICAuLi4KPj4gICB9Owo+PiAgIG1lbWJlcnMgPSA8JmNwdTA+LCA8JmNwdTE+LCA8JmNw dTI+LCA8JmNwdTM+Owo+IAo+IE5vdCBhIGdvb2QgaWRlYSBJTU8uCj4gCgpXaGVyZSBpbiB5b3Vy IG9waW5pb24gc2hvdWxkIHRoaXMgY2FjaGUgbm9kZSBnbyA/IE9uIHRoZSBmaXJzdCBleGFtcGxl IApmcm9tCmEgcHJvZHVjdGlvbiBkdHMgaXQgZ29lcyB0byB0aGUgU29DIG5vZGUgYW5kIG9uIGFu b3RoZXIgcHJvZHVjdGlvbiBkdHMgCml0IGdvZXMKdG8gdGhlIGNwdXMvIG5vZGUuIEkgdGhpbmsg aXQgbWFrZXMgbW9yZSBzZW5zZSB0byBnbyB0byB0aGUgY29yZSBub2RlIG9yIAp0aGUKY2x1c3Rl ciBub2RlLCBkZXBlbmRpbmcgb24gaG93IGl0J3Mgc2hhcmVkLiBJdCBtYWtlcyBpdCBhbHNvIGVh c2llcgpmb3IgdGhlIGltcGxlbWVudGF0aW9uIHRvIHVuZGVyc3RhbmQgdGhlIGxldmVscyBvZiBz aGFyZWQgY2FjaGVzIGFuZCAKY3JlYXRpbmcKY3B1IG1hc2tzLgoKPj4gV2UgY291bGQgZXZlbiBy ZW1vdmUgbmV4dC1sZXZlbC1jYWNoZSBmcm9tIHRoZSBjcHUgbm9kZXMgYW5kIGluZmVyIGl0IAo+ PiBmcm9tCj4+IHRoZSAgdG9wb2xvZ3kgKHNlYXJjaCB0aGUgdG9wb2xvZ3kgdXB3YXJkcyB1bnRp bCB3ZSBnZXQgYSBub2RlIHRoYXQncwo+PiAiY2FjaGUiLWNvbXBhdGlibGUpLCB3ZSBjYW4gYWdh aW4gbWFrZSB0aGlzIGJhY2t3YXJkcy1jb21wYXRpYmxlLgo+PiAKPiAKPiBXaHkgYXJlIHlvdSBh c3N1bWluZyB0aGF0IHRoZXkgKmhhdmUqIHRvIGJlIHNvIGFsaWduZWQgd2l0aCB0b3BvbG9neSA/ Cj4gSG93IGRvIHlvdSBkZWFsIHdpdGggZGlmZmVyZW50IGtpbmQgb2Ygc3lzdGVtcyA/Cj4gCgpJ dCBkZXBlbmRzIG9uIGhvdyB5b3UgZGVmaW5lIHRvcG9sb2d5LiBJIGJlbGlldmUgdGhhdCB0aGUg Y2FjaGUgCmhpZXJhcmNoeSBpcwphIG11Y2ggInN0cm9uZ2VyIiBkZWZpbml0aW9uIG9mIHRoZSBD UFUgdG9wb2xvZ3kgdGhhbiBncm91cGluZyAKaGFydHMvdGhyZWFkcwpvbiBjbGFzc2VzIGxpa2Ug ImNvcmUiLCAic29ja2V0IiwgInBhY2tldCIgZXRjIHRoYXQgZG9uJ3QgcHJvdmlkZSBhbnkgCmlu Zm9ybWF0aW9uCm9mIHdoYXQncyBnb2luZyBvbiBpbnNpZGUgdGhlIGhhcmR3YXJlLiBZb3UgY2Fu IHRoaW5rIG9mIHRoZSBxdWVzdGlvbiAKdGhlIG90aGVyCndheSBhcm91bmQsIHdoeSBhcmUgeW91 IGFzc3VtaW5nIHRoYXQgdGhlIGN1cnJlbnQgdG9wb2xvZ3kgYXMgc3BlY2lmaWVkIApvbgpjcHUt bWFwIGlzIGFsaWduZWQgd2l0aCBob3cgdGhlIGhhcnRzIGFuZCBjb3JlcyBzaGFyZSB0aGVpciAi cmVzb3VyY2VzIiAKPwpCZWNhdXNlIHRoYXQncyB3aGF0J3MgZ29pbmcgb24gYXQgdGhlIGltcGxl bWVudGF0aW9uIHNpZGUgb2YgY3B1LW1hcC4KCj4+IAo+PiBGaW5hbGx5IGZyb20gdGhlIGV4YW1w bGVzIGFib3ZlIEknZCBsaWtlIHRvIHN0cmVzcyBvdXQgdGhhdCB0aGUgCj4+IGRpc3RpbmN0aW9u Cj4+IGJldHdlZW4gYSBjbHVzdGVyIGFuZCBhIGNvcmUgZG9lc24ndCBtYWtlIG11Y2ggc2Vuc2Ug YW5kIGl0IGFsc28gbWFrZXMgCj4+IHRoZQo+PiByZXByZXNlbnRhdGlvbiBtb3JlIGNvbXBsaWNh dGVkLiBUbyBiZWdpbiB3aXRoLCBob3cgd291bGQgeW91IGNhbGwgdGhlIAo+PiBzZXR1cAo+PiBv biBIaUZpdmUgVW5sZWFzaGVkID8gQSBjbHVzdGVyIG9mIDQgY29yZXMgdGhhdCBzaGFyZSB0aGUg c2FtZSBMMyAKPj4gY2FjaGUgPwo+PiBPbmUgY29yZSB3aXRoIDQgaGFydHMgdGhhdCBzaGFyZSB0 aGUgc2FtZSBMMyBjYWNoZSA/IFdlIGNvdWxkIAo+PiByZXByZXNlbnQgaXQKPj4gbGlrZSB0aGlz IGluc3RlYWQ6Cj4+IAo+IAo+IEp1c3QgcmVwcmVzZW50IGVhY2ggcGh5c2ljYWwgY2FjaGUgYW5k IGdldCB0aGUgbGlzdCBvZiBDUFVzIHNoYXJpbmcgaXQuCj4gRG9lc24ndCBtYXR0ZXIgd2hhdCBp dCBpczogY2x1c3RlciBvciBjbHVzdGVyIG9mIGNsdXN0ZXJzIG9yIGNsdXN0ZXIgb2YKPiBoYXJ0 cywgYmxhaCwgYmxhaC4gSXQgcmVhbGx5IGRvZXNuJ3QgbWF0dGVyLgo+IAoKSSB0b3RhbGx5IGFn cmVlIHdpdGggeW91LCBidXQgaW4gdGhpcyBjYXNlIHdoYXQncyB0aGUgcmVhc29uIG9mIGhhdmlu ZwpjcHUtbWFwID8gV2UgY2FuIGluZmVyIHRoZSB0b3BvbG9neSBmcm9tIHdoYXQgd2UgYWxyZWFk eSBoYXZlLCBhdCBsZWFzdApjcHUtbWFwIGFkZHMgc29tZSBjb252ZW5pZW5jZSAoYnV0IGlmIHdl IGdvIGFsbCB0aGUgd2F5IGRvd24gdGhlIAoiY29udmluaWVuY2UiCnBhdGggd2UnbGwgZW5kIHVw IG9uIHNvbWV0aGluZyBsaWtlIG15IGluaXRpYWwgYXBwcm9hY2ggd2hpY2ggd2FzIGFzIHdlCmJv dGggYWdyZWUgbm93LCB3cm9uZykuCgo+PiAKPj4gV2UgY291bGQgZS5nLiBrZWVwIG9ubHkgY2x1 c3RlciBub2RlcyBhbmQgYWxsb3cgdGhlbSB0byBjb250YWluIGVpdGhlciAKPj4gYW4KPj4gYXJy YXkgb2YgaGFydHMgb3Igb3RoZXIgY2x1c3RlciBzdWItbm9kZXMgKyBvcHRpb25hbGx5IGEgc2V0 IG9mIAo+PiBhdHRyaWJ1dGVzLAo+PiBjb21tb24gdG8gdGhlIG1lbWJlcnMvc3ViLW5vZGVzIG9m IHRoZSBjbHVzdGVyLiBUaGlzIHdheSB3ZSdsbCBnZXQgaW4gCj4+IHRoZQo+PiBmaXJzdCBleGFt cGxlOgo+PiAKPiAKPiBBbGwgdGhlc2UgZmFuY3kgbmV3IGlkZWFzIHlvdSBhcmUgcHJvcG9zaW5n IGFyZSBnb29kIGlmIHZlbmRvcnMgZm9sbG93Cj4gc29tZSB0aGluZ3MgcmVsaWdpb3VzbHksIGJ1 dCBJIHJlYWxseSBkb3VidCBpZiB0aGF0J3MgdHJ1ZS4gU28ganVzdAo+IGhhdmUgbWF4aW11bSBm bGV4aWJpbGl0eSBzbyB0aGF0IHdlIGNhbiByZXByZXNlbnQgbWF4aW11bSBudW1iZXIgb2YKPiBz eXN0ZW1zIHdpdGhvdXQgaGF2aW5nIHRvIHJlZGVmaW5lIHRoZSBiaW5kaW5ncyBhZ2FpbiBhbmQg YWdhaW4gZm9yIHRoZQo+IHNhbWUgdGhpbmcuCj4gCj4gU28gaW5zdGVhZCBvZiBmaW5kaW5nIHdh eXMgdG8gb3B0aW1pc2UsIHlvdSBzaG91bGQgcmVhbGx5IGNvbWUgdXAgd2l0aAo+IGxpc3Qgb2Yg c2hvcnRjb21pbmdzIGluIHRoZSBleGlzdGluZyBiaW5kaW5ncyBzbyB0aGF0IHdlIGNvdmVyIG1v cmUKPiBwbGF0Zm9ybXMgd2l0aCBnZW5lcmljIGJpbmRpbmdzLiBJTU8geW91IGFyZSB0b28gY29u Y2VybmVkIG9uIAo+IG9wdGltaXNhdGlvbgo+IG9mIERUIHJlcHJlc2VudGF0aW9uIHdoaWNoIG1h eSBkZWZlYXQgdGhlIHB1cnBvc2Ugb2YgZ2VuZXJpYyBiaW5kaW5ncy4KPiAKCkkgZ2V0IHlvdXIg cG9pbnQsIGJ1dCBpZiB0aGF0J3MgdGhlIGNhc2UsIGlzbid0IGNwdS1tYXAgYW4gb3B0aW1pemF0 aW9uIApvdmVyCnRoZSBnZW5lcmljIGJpbmRpbmdzIGZvciBjcHUgbm9kZXMgYW55d2F5ID8gQXMg Zm9yIHRoZSBmbGV4aWJpbGl0eSwgd2hhdCAKSSdtCnNheWluZyBpcyB0byBhbGxvdyB0aGUgZmxl eGliaWxpdHkgb2YgYWRkaW5nIHRoZSBwcm9wZXJ0aWVzIHRoYXQgd2UgCndvdWxkCm90aGVyd2lz ZSBhZGQgdG8gY3B1IG5vZGVzLCBvbiB0aGUgZ3JvdXBzIHdlIHVzZSBvbiBjcHUtbWFwLiBPcHRp b25hbHkgCnNvIHRoYXQKd2UgYXJlIGFsc28gYmFja3dhcmRzIGNvbXBhdGlibGUuIEkgc2VlIHRo aXMgYXMgbW9yZSBmbGV4aWJsZSwgbm90IGxlc3MKZmxleGlibGUuIFRoZSBzYW1lIGdvZXMgZm9y IHJlbW92aW5nIHRoZSBkaXN0aW5jdGlvbiBiZXR3ZWVuICJjb3JlIiwKImNsdXN0ZXIiIGV0Yywg aGF2aW5nIHNwZWNpZmljIG5hbWVzIGZvciBlYWNoIG9mIHRoZSB0b3BvbG9neSBsZXZlbHMgaXMg CklNSE8KbGVzcyBmbGV4aWJsZSB0aGFuIHVzaW5nIGFic3RyYWN0IG5hbWVzLgoKQWxzbyB3aGF0 J3MgdGhlIHBvaW50IG9mIGRlZmluaW5nIGEgYmluZGluZyBpZiB2ZW5kb3JzIGRvbid0IGZvbGxv dyBpdCA/CkkgdGhpbmsgaXQgd291bGQgYmUgZWFzaWVyIGZvciB0aGUgdmVuZG9ycyB0byBqdXN0 IHVzZSB3aGF0J3MgdGhlcmUgCmluc3RlYWQKb2YgZGVmaW5pbmcgYSBuZXcgYmluZGluZyBhbmQg d3JpdGluZyB0aGUgY29kZSBmb3IgaXQuIE9uIHRoZSBvdGhlciBoYW5kCnlvdSBhcmUgcHJvYmFi bHkgbW9yZSBleHBlcmllbmNlZCBvbiB0aGlzIHRoYW4gSSBhbSwgSSBoYXZlbid0IGRlYWx0IAp3 aXRoCnZhcmlvdXMgZGlmZmVyZW50IHZlbmRvcnMgYW5kIHRoZWlyIGRpZmZlcmVudCB3YXlzLgoK UmVnYXJkcywKTmljawoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fCmxpbnV4LXJpc2N2IG1haWxpbmcgbGlzdApsaW51eC1yaXNjdkBsaXN0cy5pbmZyYWRl YWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgt cmlzY3YK From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nick Kossifidis Subject: Re: [RFC 0/2] Add RISC-V cpu topology Date: Thu, 08 Nov 2018 16:52:30 +0200 Message-ID: References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> <20181107122803.GA8433@e107155-lin> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Return-path: In-Reply-To: <20181107122803.GA8433@e107155-lin> Sender: linux-kernel-owner@vger.kernel.org To: Sudeep Holla Cc: Nick Kossifidis , Mark Rutland , Atish Patra , linux-riscv@lists.infradead.org, devicetree@vger.kernel.org, Damien.LeMoal@wdc.com, alankao@andestech.com, zong@andestech.com, anup@brainfault.org, palmer@sifive.com, linux-kernel@vger.kernel.org, hch@infradead.org, robh+dt@kernel.org, tglx@linutronix.de List-Id: devicetree@vger.kernel.org Στις 2018-11-07 14:28, Sudeep Holla έγραψε: > On Wed, Nov 07, 2018 at 04:31:34AM +0200, Nick Kossifidis wrote: > [...] > >> >> Mark and Sudeep thanks a lot for your feedback, I guess you convinced >> me >> that having a device tree binding for the scheduler is not a correct >> approach. >> > > Thanks :) > >> It's not a device after all and I agree that the device tree shouldn't >> become an OS configuration file. Regarding multiple levels of shared >> resources my point is that since cpu-map doesn't contain any >> information of >> what is shared among the cluster/core members it's not easy to do any >> further translation. Last time I checked the arm code that uses >> cpu-map, it >> only defines one domain for SMT, one for MC and then everything else >> is >> ignored. No matter how many clusters have been defined, anything above >> the >> core level is the same (and then I guess you started talking about >> adding >> "packages" on the representation side). >> > > Correct. > >> The reason I proposed to have a binding for the scheduler directly is >> not >> only because it's simpler and closer to what really happens in the >> code, it >> also makes more sense to me than the combination of cpu-map with all >> the >> related mappings e.g. for numa or caches or power domains etc. >> > > Again you are just looking at it with Linux kernel perspective. > >> However you are right we could definitely augment cpu-map to include >> support >> for what I'm saying and clean things up, and since you are open about >> improving it here is a proposal that I hope you find interesting: At >> first >> let's get rid of the nodes, they don't make sense: >> >> thread0 { >> cpu = <&CPU0>; >> }; >> > > Do you have any strong reasons to do so ? > Since it's already there for some time, I believe we can't remove it > for > backward compatibility reasons. > >> A thread node can't have more than one cpu entry and any properties >> should be on the cpu node itself, so it doesn't / can't add any >> more information. We could just have an array of cpu nodes on the >> node, it's much cleaner this way. >> >> core0 { >> members = <&CPU0>, <&CPU1>; >> }; >> > > I agree, but we have kernel code using it(arm64/kernel/topology.c). > It's > too late to remove it. But we can always keep to optional if we move > the > ARM64 binding as generic to start with and mandate it for only ARM64. > That's my point as well, if we are going to define something to be used by everybody and in this case, at least for RISC-V, there is no need to carry this from the ARM64 binding. It shouldn't be that hard to fix this in the future for ARM64 as well, we may give the new mapping another name, maybe cpu-map2 or cpu-topology to slowly move to the new one. Changing the dts files shouldn't be this hard, we can provide a script for it. We can even contain some compatibility code that also understands nodes and e.g. merges them together on a core node. >> Then let's allow the cluster and core nodes to accept attributes that >> are >> common for the cpus they contain. Right now this is considered >> invalid. >> > > Yes, we have discussed in the past and decided not to. I am fine if we > need to change it, but assuming the topology implies other information > could be wrong. On ARM platforms we have learnt it, so we kept any > information away from topology. I assume same with RISC-V, different > vendors implement in different ways, so it's better to consider those > factors. > >> For power domains we have a generic binding described on >> Documentation/devicetree/bindings/power/power_domain.txt >> which basically says that we need to put power-domains = > specifiers> attribute on each of the cpu nodes. >> > > OK, but what's wrong with that. I gives full flexibility. > >> The same happens with the capacity binding specified for arm on >> Documentation/devicetree/bindings/arm/cpu-capacity.txt >> which says we should add the capacity-dmips-mhz on each of the cpu >> nodes. >> > > Ditto, we may need this for our single cluster DSU systems. > >> The same also happens with the generic numa binding on >> Documentation/devicetree/bindings/numa.txt >> which says we should add the nuna-node-id on each of the cpu nodes. >> > > Yes, but again what's the problem ? > There is no problem with the above bindings, the problem is that we have to put them on each cpu node which is messy. We could instead put them (optionally) on the various groupings used on cpu-map. This would allow cpu-map to be more specific of what is shared across the members of each group (core/cluster/whatever). As I wrote on my answer to Mark previously, the bindings for infering the cache topology, numa topology, power domain topology etc are already there, they are in the devicet tree spec and provide very specific informations we can use. Much "stronger" hints of what's going on at the hw level. The cpu-map doesn't provide such information, it just provides a view of how the various harts/threads are "packed" on the chip, not what they share inside each level of "packing". It's useful because it saves people from having to define a bunch of cache nodes and describe the cache hierarchy on the device tree using the standard spec. So since cpu-map is there for convenience let's make it more convenient ! What I'm saying is that cpu-map could be a more compact way of using the existing bindings for adding properties on groups of harts instead of putting them on each hart individually. It will simplify the representation and may also optimize the implementation a bit (we may get the information we need faster). I don't see any other reason for using cpu-map on RISC-V or for making it global across archs. >> We could allow for these attributes to exist on cluster and core nodes >> as well so that we can represent their properties better. It shouldn't >> be a big deal and it can be done in a backwards-compatible way (if we >> don't find them on the cpu node, climb up the topology hierarchy until >> we find them / not find them at all). All I'm saying is that I prefer >> this: >> > > [...] > >> >> cluster0 { >> cluster0 { >> core0 { >> power-domains = <&pdc 0>; >> numa-node-id = <0>; >> capacity-dmips-mhz = <578>; >> members = <&cpu0>, <&cpu1>; >> } >> }; >> cluster1 { >> capacity-dmips-mhz = <1024>; >> core0 { >> power-domains = <&pdc 1>; >> numa-node-id = <1>; >> members = <&cpu2>; >> }; >> core1 { >> power-domains = <&pdc 2>; >> numa-node-id = <2>; >> members = <&cpu3>; >> }; >> }; >> } >> > > Why are you so keen on optimising the representation ? > If you are worried about large systems, generate one instead of > handcrafted. > I don't see a reason not to try to optimize it, since we are talking about a binding to be used by RISC-V and potentially everybody, I think it makes sens to improve upon what we already have. >> When it comes to shared resources, the standard dt mappings we have >> are for >> caches and are on the device spec standard (coming from power pc's >> ePAPR >> standard I think). The below comes from HiFive unleashed's device tree >> (U540Config.dts) that follows the spec: >> > > I don't understand what you are trying to explain, ePAPR does specify > per CPU entries. > > [...] > I'm saying we could allow moving these properties on the groupings of cpu-map to simplify the representation and possibly optimize the implementation. This way I believe cpu-map will be much more useful than it is now. >> Note that the cache-controller node that's common between the 4 cores >> can >> exist anywhere BUT the cluster node ! However it's a property of the >> cluster. >> A quick search through the tree got me r8a77980.dtsi that defines the >> cache >> on the cpus node and I'm sure there are other similar cases. Wouldn't >> this >> be better ? >> >> cluster0 { >> core0 { >> cache-controller@2010000 { >> cache-block-size = <64>; >> cache-level = <2>; >> cache-sets = <2048>; >> cache-size = <2097152>; >> cache-unified; >> compatible = "sifive,ccache0", "cache"; >> ... >> }; >> members = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>; > > Not a good idea IMO. > Where in your opinion should this cache node go ? On the first example from a production dts it goes to the SoC node and on another production dts it goes to the cpus/ node. I think it makes more sense to go to the core node or the cluster node, depending on how it's shared. It makes it also easier for the implementation to understand the levels of shared caches and creating cpu masks. >> We could even remove next-level-cache from the cpu nodes and infer it >> from >> the topology (search the topology upwards until we get a node that's >> "cache"-compatible), we can again make this backwards-compatible. >> > > Why are you assuming that they *have* to be so aligned with topology ? > How do you deal with different kind of systems ? > It depends on how you define topology. I believe that the cache hierarchy is a much "stronger" definition of the CPU topology than grouping harts/threads on classes like "core", "socket", "packet" etc that don't provide any information of what's going on inside the hardware. You can think of the question the other way around, why are you assuming that the current topology as specified on cpu-map is aligned with how the harts and cores share their "resources" ? Because that's what's going on at the implementation side of cpu-map. >> >> Finally from the examples above I'd like to stress out that the >> distinction >> between a cluster and a core doesn't make much sense and it also makes >> the >> representation more complicated. To begin with, how would you call the >> setup >> on HiFive Unleashed ? A cluster of 4 cores that share the same L3 >> cache ? >> One core with 4 harts that share the same L3 cache ? We could >> represent it >> like this instead: >> > > Just represent each physical cache and get the list of CPUs sharing it. > Doesn't matter what it is: cluster or cluster of clusters or cluster of > harts, blah, blah. It really doesn't matter. > I totally agree with you, but in this case what's the reason of having cpu-map ? We can infer the topology from what we already have, at least cpu-map adds some convenience (but if we go all the way down the "convinience" path we'll end up on something like my initial approach which was as we both agree now, wrong). >> >> We could e.g. keep only cluster nodes and allow them to contain either >> an >> array of harts or other cluster sub-nodes + optionally a set of >> attributes, >> common to the members/sub-nodes of the cluster. This way we'll get in >> the >> first example: >> > > All these fancy new ideas you are proposing are good if vendors follow > some things religiously, but I really doubt if that's true. So just > have maximum flexibility so that we can represent maximum number of > systems without having to redefine the bindings again and again for the > same thing. > > So instead of finding ways to optimise, you should really come up with > list of shortcomings in the existing bindings so that we cover more > platforms with generic bindings. IMO you are too concerned on > optimisation > of DT representation which may defeat the purpose of generic bindings. > I get your point, but if that's the case, isn't cpu-map an optimization over the generic bindings for cpu nodes anyway ? As for the flexibility, what I'm saying is to allow the flexibility of adding the properties that we would otherwise add to cpu nodes, on the groups we use on cpu-map. Optionaly so that we are also backwards compatible. I see this as more flexible, not less flexible. The same goes for removing the distinction between "core", "cluster" etc, having specific names for each of the topology levels is IMHO less flexible than using abstract names. Also what's the point of defining a binding if vendors don't follow it ? I think it would be easier for the vendors to just use what's there instead of defining a new binding and writing the code for it. On the other hand you are probably more experienced on this than I am, I haven't dealt with various different vendors and their different ways. Regards, Nick