From mboxrd@z Thu Jan 1 00:00:00 1970 From: mick@ics.forth.gr (Nick Kossifidis) Date: Fri, 09 Nov 2018 05:55:29 +0200 Subject: [RFC 0/2] Add RISC-V cpu topology In-Reply-To: <20181108155404.32ja7bbxu62nyytt@lakrids.cambridge.arm.com> References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> <20181107120645.sc3wjgr2yakvxktl@lakrids.cambridge.arm.com> <20181108155404.32ja7bbxu62nyytt@lakrids.cambridge.arm.com> Message-ID: <0b2ce6ab4fa804e2ce8cea5eca5b02da@mailhost.ics.forth.gr> To: linux-riscv@lists.infradead.org List-Id: linux-riscv.lists.infradead.org ???? 2018-11-08 17:54, Mark Rutland ??????: > On Thu, Nov 08, 2018 at 03:45:36PM +0200, Nick Kossifidis wrote: >> ???? 2018-11-07 14:06, Mark Rutland ??????: >> > On Wed, Nov 07, 2018 at 04:31:34AM +0200, Nick Kossifidis wrote: >> > > Mark and Sundeep thanks a lot for your feedback, I guess you convinced >> > > me that having a device tree binding for the scheduler is not a >> > > correct approach. It's not a device after all and I agree that the >> > > device tree shouldn't become an OS configuration file. >> > >> > Good to hear. >> > >> > > Regarding multiple levels of shared resources my point is that since >> > > cpu-map doesn't contain any information of what is shared among the >> > > cluster/core members it's not easy to do any further translation. Last >> > > time I checked the arm code that uses cpu-map, it only defines one >> > > domain for SMT, one for MC and then everything else is ignored. No >> > > matter how many clusters have been defined, anything above the core >> > > level is the same (and then I guess you started talking about adding >> > > "packages" on the representation side). >> > >> > While cpu-map doesn't contain that information today, we can *add* that >> > information to the cpu-map binding if necessary. >> > >> > > The reason I proposed to have a binding for the scheduler directly is >> > > not only because it's simpler and closer to what really happens in the >> > > code, it also makes more sense to me than the combination of cpu-map >> > > with all the related mappings e.g. for numa or caches or power >> > > domains etc. >> > > >> > > However you are right we could definitely augment cpu-map to include >> > > support for what I'm saying and clean things up, and since you are >> > > open about improving it here is a proposal that I hope you find >> > > interesting: >> > > >> > > At first let's get rid of the nodes, they don't make sense: >> > > >> > > thread0 { >> > > cpu = <&CPU0>; >> > > }; >> > > >> > > A thread node can't have more than one cpu entry and any properties >> > > should be on the cpu node itself, so it doesn't / can't add any >> > > more information. We could just have an array of cpu nodes on the >> > > node, it's much cleaner this way. >> > > >> > > core0 { >> > > members = <&CPU0>, <&CPU1>; >> > > }; >> > >> > Hold on. Rather than reinventing things from first principles, can we >> > please discuss what you want to *achieve*, i.e. what information you >> > need? >> > >> > Having a node is not a significant cost, and there are reasons we may >> > want thread nodes. For example, it means that we can always refer to any >> > level of topology with a phandle, and we might want to describe >> > thread-affine devices in future. >> >> You can use the phandle of the cpu node, the thread node doesn't add >> anything more than complexity to the representation. > > That adds complexity elsewhere, because the phandle of a CPU node is > not > a child of the cpu-map node, and you can't simply walk up the tree > hierarchy to find parent nodes. > > I see no reason to change this part of the binding. Given it's already > out there, with existing parsing code, changing it only serves to add > complexity to any common code which has to parse this. > > Can we please drop the idea of dropping the thread node? > >> > There are a tonne of existing bindings that are ugly, but re-inventing >> > them for taste reasons alone is more costly to the ecosystem than simply >> > using the existing bindings. We avoid re-inventing bindings unless there >> > is a functional problem e.g. cases which they cannot possibly describe. >> >> We are talking about using something for RISC-V and possibly common >> across >> different archs and, I don't see why we should keep the ugliness of a >> binding spec plus in this case the node can't possibly >> describe >> anything else than a cpu= alias, it's redundant. > > While it may be ugly, removing it only serves to add complexity for > common parsing code, and removes flexibility that we may want in > future. > Its presence does not cause any technical problem. > > For better or worse we're all sharing the same codebase here. The > common > case matters, as does the complexity of the codebase as a whole. > > I realise it can be frustrating to have to use something you feel is > sub-optimal, but putting up with a few nodes which you personally feel > are redundant is of much lower cost to the ecosystem than having > incompatible bindings where we could have one. If you put up with that, > you can focus your efforts on more worthwhile technical exercises. > The only reason I mentioned the issue with the node is because you said that you are open for improving cpu-map. I don't think it's such a big deal and even if we decide against it, it's not a big deal to be backwards compatible with what's already there. I fully understand what you say about the impact on the ecosystem and agree with you. >> > > Then let's allow the cluster and core nodes to accept attributes >> > > that are >> > > common for the cpus they contain. Right now this is considered >> > > invalid. >> > > >> > > For power domains we have a generic binding described on >> > > Documentation/devicetree/bindings/power/power_domain.txt >> > > which basically says that we need to put power-domains = > > > specifiers> >> > > attribute on each of the cpu nodes. >> > >> > FWIW, given this is arguably topological, I'm not personally averse to >> > describing this in the cpu-map, if that actually gains us more than the >> > complexity require to support it. >> > >> > Given we don't do this for device power domains, I suspect that it's >> > simpler to stick with the existing binding. >> > >> > > The same happens with the capacity binding specified for arm on >> > > Documentation/devicetree/bindings/arm/cpu-capacity.txt >> > > which says we should add the capacity-dmips-mhz on each of the cpu >> > > nodes. >> > >> > The cpu-map was intended to expose topological dtails, and this isn't >> > really a topological property. For example, Arm DynamIQ systems can have >> > heterogeneous CPUs within clusters. >> > >> > I do not think it's worth moving this, tbh. >> > >> > > The same also happens with the generic numa binding on >> > > Documentation/devicetree/bindings/numa.txt >> > > which says we should add the nuna-node-id on each of the cpu nodes. >> > >> > Is there a strong gain from moving this? >> > >> > [...] >> >> Right now with the device tree spec and the above bindings we can >> use the information from cpu nodes to infer the cache topology, >> the memory topology, the power domain topology etc. >> >> We have of_find_next_cache_node and of_find_last_cache_level for >> example >> that use "next-level-cache" and are used on powerpc (where this spec >> comes >> from), that we can further build on top of them (since this is now >> part >> of the device tree spec anyway), to e.g. populate the levels of >> cache, >> the levels of shared cache and finally create cpu masks for the >> different >> cache sharing levels. > > The cpu-map binding does not describe cache topology. I never suggested > that it did. > Sorry for the misunderstanding Mark ! On ARM the CPU topology is used for configuring the scheduling domain topology (arch/arm/kernel/topology.c) and blindly sets the SD_SHARE_PKG_RESOURCES and SD_SHARE_POWERDOMAIN flags for a core without taking into account the cache hierarchy (hence validating that SD_SHARE_PKG_RESOURCES is correct) or the power domains (but ok, I doubt it's possible to have two threads on the same core that are powered independently). It also supports only two scheduling domains, one for SMT and another one for MC (hence my comments on multiple levels of shared resources not being supported). Since according to the docs cpu-map is a binding common between ARM and ARM64 for describing the CPU topology, I assumed it was a part of that process. Turns out it's only used on ARM64, where set_sched_topology() is not used for configuring the scheduling domain topology (there is a commend there that implies that you pass on the clusters to the scheduler that also led me to believe the issue is also present there). So my assumption was wrong and cpu-map has nothing to do with configuring the scheduler and the issues I mentioned above. >> This is standards-compliant, arch-independent (they are on of/base.c), >> and >> it provides a concrete hint on CPU topology rather grouping harts to >> classes >> like "core", "package", "socket", "cluster" etc that don't mean >> anything >> specific and we assume to map to specific levels of cache sharing. > > Please note that I have never suggested "package", and I'm not sure > what > that's in response to. > https://lkml.org/lkml/2018/11/7/918 [...] >> In a sense it goes against your rule of "re-inventing them for taste >> reasons alone is more costly to the ecosystem than simply using the >> existing bindings", I fail to see anything else than "taste reasons" >> when it comes to cpu-map, since there are existing bindings for >> inferring the CPU topology already, they are just not that convenient. > > If you believe that's the case, then you can perfectly use the existing > cache and NUMA bindings, and not describe the cpu topology at all, no > need to change cpu-map or come up with a new binding. > The problem is that right now nobody is using the generic bindings for configuring the scheduler domain topology. Only the numa bindings are used as expected on ARM64. Since cpu-map is only there for representing the cpu topology/layout, allowing them to exist there is obviously not the right approach. >> I'm proposing to add the option (not enforce) of adding those >> attributes that are meant to be used on cpu nodes, on the various >> groups/classes of the cpu-map, this way cpu-map will provide something >> more meaningful and at least improve the representation side of >> things. On the implementation side it might save us the burden of >> infering the topology from parsing all cpu nodes each time. It's also >> backwards compatible with what you already have, the only thing that's >> not backwards compatible is the removal of the node, which I >> don't think is such a big deal to fix. > > Sorry, but as I have stated repeatedly, this complicates the common > code > for what you admit is a matter of taste. I would rather maintain the > simplicity of this binding and existing parsing code, and not > potentially break other parsers, if the only cost is a few nodes which > you personally consider to be redundant. > [...] >> So we agree, the "core" doesn't say anything useful regarding the >> topology, why keep using this distinction on a binding for RISC-V and >> possibly other archs (since we are also talking on an arch-independent >> approach) ? > > We keep it because the binding has already been finalised and is use in > practice. The existing binding and parsing code is *sufficient* for > your > needs. Personal taste and minor savings in the number of nodes in a DT > are not compelling reasons to change this when the cost is complexity > that we have to maintain *forever*. > Totally fine with that, as I mentioned above I pointed these out since I consider it an improvement. Forgive me if I gave you the impression that was a deal-breaker or something. Regards, Nick From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.5 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 763B0C43441 for ; Fri, 9 Nov 2018 03:58:25 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 15CCB20855 for ; Fri, 9 Nov 2018 03:58:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="fsxToB82"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="y7z8GGL4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 15CCB20855 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ics.forth.gr Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender:Content-Type: Content-Transfer-Encoding:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-ID:References:In-Reply-To:Subject:To:From: Date:MIME-Version:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=oUwZM3iOS6GVb7e/vpKbLT1OyRnyDczo8CSP1IwsCIA=; b=fsxToB82hwICb9ng9iJj3iRPy W5Ej0ODF65xziY564/cQCrKCisbMFxLIrHp7QjcCSIARadkU0JD1MxDmGuZzt7DNMg6s1NT51Fv+o AVP6Odg2pbH+f7/7wdNJPiUM9LpgBOLl5JP4IEapyY/9kmFwca+bD3TrcO8LIKUGsC0sPxPqppMfJ XysNPWWxUgQIKCCav7ZKOwIP5PgKJl2ty3T5mjFQU/Ca+V47yBV4Hdgd1oXYGXXQ4UqwNcpEVI3H8 gSM3RxOzHyfMZao+J3Jgcw/FWdWW9WPWnSv1nZPQ0BSMkahhyBXvLOdOqIzcaF7r97YZSOOQjj8H/ bjvD1AKxA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKxw7-0007eE-Hy; Fri, 09 Nov 2018 03:58:23 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKxw5-0007e4-LA for linux-riscv@bombadil.infradead.org; Fri, 09 Nov 2018 03:58:21 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Message-ID:References:In-Reply-To: Subject:Cc:To:From:Date:Content-Transfer-Encoding:Content-Type:MIME-Version: Sender:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help: List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=VMs5dnvGMjMNc5ovxgxnMjmXFNTs1Tonf9sO9Dolaxk=; b=y7z8GGL4jXc6E8XgKewAFzAcEM Hv5Y7rIMBQRXxBcESCOVC4IT7QQM68Lj7xTe8LCRlJdf+Awm4/kA03IrA+vdeK3uz5zjtcM+Bv6Az pHfc8Bo53t8nS9N3ykKwR1u82KCaOhQBwhehwwE2QxKH8/E9m6V8Lbv/uYyZtDaGsTvqa/ghcJzQ/ EaOv/9AFE3AuvaURBwRdSz0AdqczUuHVO2QvAhpER4bzXXJaTE3mgJG++2PFQtEuOdGBq7409lhO5 +U7rZwn5WPh/91nCtH3lPMWnDtVYMqU7tZn17UOGZ7fc7exi+iTUJiIgx01R4r1Xh7XTsk82tgK/h FGDZxnXg==; Received: from mailgate-4.ics.forth.gr ([139.91.1.7]) by merlin.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1gKxw0-0000f4-IT for linux-riscv@lists.infradead.org; Fri, 09 Nov 2018 03:58:19 +0000 Received: from av1.ics.forth.gr (av3in.ics.forth.gr. [139.91.1.77]) by mailgate-4.ics.forth.gr (8.14.5/ICS-FORTH/V10-1.9-GATE-OUT) with ESMTP id wA93tWJM017440; Fri, 9 Nov 2018 05:55:34 +0200 (EET) X-AuditID: 8b5b9d4d-91bff70000000e62-d6-5be505334562 Received: from enigma.ics.forth.gr (enigma.ics.forth.gr [139.91.1.35]) by av1.ics.forth.gr (SMTP Outbound / FORTH / ICS) with SMTP id 16.2C.03682.33505EB5; Fri, 9 Nov 2018 05:55:31 +0200 (EET) Received: from webmail.ics.forth.gr (localhost [127.0.0.1]) by enigma.ics.forth.gr (8.15.1//ICS-FORTH/V10.5.0C-EXTNULL-SSL-SASL) with ESMTP id wA93tTQB018040; Fri, 9 Nov 2018 05:55:29 +0200 X-ICS-AUTH-INFO: Authenticated user: at ics.forth.gr MIME-Version: 1.0 Date: Fri, 09 Nov 2018 05:55:29 +0200 From: Nick Kossifidis To: Mark Rutland Subject: Re: [RFC 0/2] Add RISC-V cpu topology Organization: FORTH In-Reply-To: <20181108155404.32ja7bbxu62nyytt@lakrids.cambridge.arm.com> References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> <20181107120645.sc3wjgr2yakvxktl@lakrids.cambridge.arm.com> <20181108155404.32ja7bbxu62nyytt@lakrids.cambridge.arm.com> Message-ID: <0b2ce6ab4fa804e2ce8cea5eca5b02da@mailhost.ics.forth.gr> X-Sender: mick@mailhost.ics.forth.gr User-Agent: Roundcube Webmail/1.1.2 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrKIsWRmVeSWpSXmKPExsXSHc2orGvC+jTaYO8RJottS1azWrR8eMdq sWjFdxaL1vZvTBbzj5xjtTg9YRGTxeVdc9gstn1uYbNYev0ik0Xzu3PsFpsnLGC1aN17hN1i +akdLBabN01ltni+spfNgd9jz+lZzB5r5q1h9Jj6+wyLx8NNl5g8Nq/Q8ti0qpPN4925c+we m5fUe1xqvs7u8XmTnEf7gW6mAO4oLpuU1JzMstQifbsErowf96+yFbyOrjg27zFTA+MBjy5G Tg4JAROJ/3uuMHYxcnEICRxmlDjx4Ag7hHOQUWLmvn/MEFWmErP3djKC2LwCghInZz5hAbGZ BSwkpl7Zzwhhy0s0b50NVs8ioCpx5sQaNhCbTUBTYv6lg2D1IgLqEj27vrCALGAWmMcscbJv JViRsICeRNOGm6wgNr+AsMSnuxfBbE4BD4nN344yQ1w0gUWi9eZ+ZogrXCSmTdvHBnGdisSH 3w/YQWxRAWWJFyems05gFJqF5NhZSI6dheTYBYzMqxgFEsuM9TKTi/XS8otKMvTSizYxgiNz ru8OxnML7A8xCnAwKvHwrmh4Ei3EmlhWXJl7iFGCg1lJhHfTP6AQb0piZVVqUX58UWlOavEh RmkOFiVx3sMvwoOEBNITS1KzU1MLUotgskwcnFINjHnNN0ovh0cX6IeFZKntr768LOCCQupd ja/3TROkHe2m82Txf3n55vrph5KFJ8XyOZ6I/p/a+vgs9+0A4xKThCkzw+apbNlzUuq81OZ6 QS7puws3VbhJe+31CU/j+7GaY8EDziRRkzVhbwv/bLd15ZrDdvvQv5MLtly4xv8hLsNF7vpF 4aV7apVYijMSDbWYi4oTATV1jbXIAgAA X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20181108_225817_214152_94BFF1FF X-CRM114-Status: GOOD ( 60.04 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: devicetree@vger.kernel.org, Damien.LeMoal@wdc.com, alankao@andestech.com, hch@infradead.org, anup@brainfault.org, palmer@sifive.com, linux-kernel@vger.kernel.org, zong@andestech.com, Atish Patra , robh+dt@kernel.org, Sudeep Holla , Nick Kossifidis , linux-riscv@lists.infradead.org, tglx@linutronix.de Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-riscv" Errors-To: linux-riscv-bounces+infradead-linux-riscv=archiver.kernel.org@lists.infradead.org Message-ID: <20181109035529.6mBlK6cMEbvhuC5hG_vgDaRLQ-Q2GVroSgjzWehrIbw@z> zqPPhM65z4IgMjAxOC0xMS0wOCAxNzo1NCwgTWFyayBSdXRsYW5kIM6tzrPPgc6xz4jOtToKPiBP biBUaHUsIE5vdiAwOCwgMjAxOCBhdCAwMzo0NTozNlBNICswMjAwLCBOaWNrIEtvc3NpZmlkaXMg d3JvdGU6Cj4+IM6jz4TOuc+CIDIwMTgtMTEtMDcgMTQ6MDYsIE1hcmsgUnV0bGFuZCDOrc6zz4HO sc+IzrU6Cj4+ID4gT24gV2VkLCBOb3YgMDcsIDIwMTggYXQgMDQ6MzE6MzRBTSArMDIwMCwgTmlj ayBLb3NzaWZpZGlzIHdyb3RlOgo+PiA+ID4gTWFyayBhbmQgU3VuZGVlcCB0aGFua3MgYSBsb3Qg Zm9yIHlvdXIgZmVlZGJhY2ssIEkgZ3Vlc3MgeW91IGNvbnZpbmNlZAo+PiA+ID4gbWUgdGhhdCBo YXZpbmcgYSBkZXZpY2UgdHJlZSBiaW5kaW5nIGZvciB0aGUgc2NoZWR1bGVyIGlzIG5vdCBhCj4+ ID4gPiBjb3JyZWN0IGFwcHJvYWNoLiBJdCdzIG5vdCBhIGRldmljZSBhZnRlciBhbGwgYW5kIEkg YWdyZWUgdGhhdCB0aGUKPj4gPiA+IGRldmljZSB0cmVlIHNob3VsZG4ndCBiZWNvbWUgYW4gT1Mg Y29uZmlndXJhdGlvbiBmaWxlLgo+PiA+Cj4+ID4gR29vZCB0byBoZWFyLgo+PiA+Cj4+ID4gPiBS ZWdhcmRpbmcgbXVsdGlwbGUgbGV2ZWxzIG9mIHNoYXJlZCByZXNvdXJjZXMgbXkgcG9pbnQgaXMg dGhhdCBzaW5jZQo+PiA+ID4gY3B1LW1hcCBkb2Vzbid0IGNvbnRhaW4gYW55IGluZm9ybWF0aW9u IG9mIHdoYXQgaXMgc2hhcmVkIGFtb25nIHRoZQo+PiA+ID4gY2x1c3Rlci9jb3JlIG1lbWJlcnMg aXQncyBub3QgZWFzeSB0byBkbyBhbnkgZnVydGhlciB0cmFuc2xhdGlvbi4gTGFzdAo+PiA+ID4g dGltZSBJIGNoZWNrZWQgdGhlIGFybSBjb2RlIHRoYXQgdXNlcyBjcHUtbWFwLCBpdCBvbmx5IGRl ZmluZXMgb25lCj4+ID4gPiBkb21haW4gZm9yIFNNVCwgb25lIGZvciBNQyBhbmQgdGhlbiBldmVy eXRoaW5nIGVsc2UgaXMgaWdub3JlZC4gTm8KPj4gPiA+IG1hdHRlciBob3cgbWFueSBjbHVzdGVy cyBoYXZlIGJlZW4gZGVmaW5lZCwgYW55dGhpbmcgYWJvdmUgdGhlIGNvcmUKPj4gPiA+IGxldmVs IGlzIHRoZSBzYW1lIChhbmQgdGhlbiBJIGd1ZXNzIHlvdSBzdGFydGVkIHRhbGtpbmcgYWJvdXQg YWRkaW5nCj4+ID4gPiAicGFja2FnZXMiIG9uIHRoZSByZXByZXNlbnRhdGlvbiBzaWRlKS4KPj4g Pgo+PiA+IFdoaWxlIGNwdS1tYXAgZG9lc24ndCBjb250YWluIHRoYXQgaW5mb3JtYXRpb24gdG9k YXksIHdlIGNhbiAqYWRkKiB0aGF0Cj4+ID4gaW5mb3JtYXRpb24gdG8gdGhlIGNwdS1tYXAgYmlu ZGluZyBpZiBuZWNlc3NhcnkuCj4+ID4KPj4gPiA+IFRoZSByZWFzb24gSSBwcm9wb3NlZCB0byBo YXZlIGEgYmluZGluZyBmb3IgdGhlIHNjaGVkdWxlciBkaXJlY3RseSBpcwo+PiA+ID4gbm90IG9u bHkgYmVjYXVzZSBpdCdzIHNpbXBsZXIgYW5kIGNsb3NlciB0byB3aGF0IHJlYWxseSBoYXBwZW5z IGluIHRoZQo+PiA+ID4gY29kZSwgaXQgYWxzbyBtYWtlcyBtb3JlIHNlbnNlIHRvIG1lIHRoYW4g dGhlIGNvbWJpbmF0aW9uIG9mIGNwdS1tYXAKPj4gPiA+IHdpdGggYWxsIHRoZSByZWxhdGVkIG1h cHBpbmdzIGUuZy4gIGZvciBudW1hIG9yIGNhY2hlcyBvciBwb3dlcgo+PiA+ID4gZG9tYWlucyBl dGMuCj4+ID4gPgo+PiA+ID4gSG93ZXZlciB5b3UgYXJlIHJpZ2h0IHdlIGNvdWxkIGRlZmluaXRl bHkgYXVnbWVudCBjcHUtbWFwIHRvIGluY2x1ZGUKPj4gPiA+IHN1cHBvcnQgZm9yIHdoYXQgSSdt IHNheWluZyBhbmQgY2xlYW4gdGhpbmdzIHVwLCBhbmQgc2luY2UgeW91IGFyZQo+PiA+ID4gb3Bl biBhYm91dCBpbXByb3ZpbmcgaXQgaGVyZSBpcyBhIHByb3Bvc2FsIHRoYXQgSSBob3BlIHlvdSBm aW5kCj4+ID4gPiBpbnRlcmVzdGluZzoKPj4gPiA+Cj4+ID4gPiBBdCBmaXJzdCBsZXQncyBnZXQg cmlkIG9mIHRoZSA8dGhyZWFkPiBub2RlcywgdGhleSBkb24ndCBtYWtlIHNlbnNlOgo+PiA+ID4K Pj4gPiA+IHRocmVhZDAgewo+PiA+ID4gIGNwdSA9IDwmQ1BVMD47Cj4+ID4gPiB9Owo+PiA+ID4K Pj4gPiA+IEEgdGhyZWFkIG5vZGUgY2FuJ3QgaGF2ZSBtb3JlIHRoYW4gb25lIGNwdSBlbnRyeSBh bmQgYW55IHByb3BlcnRpZXMKPj4gPiA+IHNob3VsZCBiZSBvbiB0aGUgY3B1IG5vZGUgaXRzZWxm LCBzbyBpdCBkb2Vzbid0IC8gY2FuJ3QgYWRkIGFueQo+PiA+ID4gbW9yZSBpbmZvcm1hdGlvbi4g V2UgY291bGQganVzdCBoYXZlIGFuIGFycmF5IG9mIGNwdSBub2RlcyBvbiB0aGUKPj4gPiA+IDxj b3JlPiBub2RlLCBpdCdzIG11Y2ggY2xlYW5lciB0aGlzIHdheS4KPj4gPiA+Cj4+ID4gPiBjb3Jl MCB7Cj4+ID4gPiAgbWVtYmVycyA9IDwmQ1BVMD4sIDwmQ1BVMT47Cj4+ID4gPiB9Owo+PiA+Cj4+ ID4gSG9sZCBvbi4gUmF0aGVyIHRoYW4gcmVpbnZlbnRpbmcgdGhpbmdzIGZyb20gZmlyc3QgcHJp bmNpcGxlcywgY2FuIHdlCj4+ID4gcGxlYXNlIGRpc2N1c3Mgd2hhdCB5b3Ugd2FudCB0byAqYWNo aWV2ZSosIGkuZS4gd2hhdCBpbmZvcm1hdGlvbiB5b3UKPj4gPiBuZWVkPwo+PiA+Cj4+ID4gSGF2 aW5nIGEgbm9kZSBpcyBub3QgYSBzaWduaWZpY2FudCBjb3N0LCBhbmQgdGhlcmUgYXJlIHJlYXNv bnMgd2UgbWF5Cj4+ID4gd2FudCB0aHJlYWQgbm9kZXMuIEZvciBleGFtcGxlLCBpdCBtZWFucyB0 aGF0IHdlIGNhbiBhbHdheXMgcmVmZXIgdG8gYW55Cj4+ID4gbGV2ZWwgb2YgdG9wb2xvZ3kgd2l0 aCBhIHBoYW5kbGUsIGFuZCB3ZSBtaWdodCB3YW50IHRvIGRlc2NyaWJlCj4+ID4gdGhyZWFkLWFm ZmluZSBkZXZpY2VzIGluIGZ1dHVyZS4KPj4gCj4+IFlvdSBjYW4gdXNlIHRoZSBwaGFuZGxlIG9m IHRoZSBjcHUgbm9kZSwgdGhlIHRocmVhZCBub2RlIGRvZXNuJ3QgYWRkCj4+IGFueXRoaW5nIG1v cmUgdGhhbiBjb21wbGV4aXR5IHRvIHRoZSByZXByZXNlbnRhdGlvbi4KPiAKPiBUaGF0IGFkZHMg Y29tcGxleGl0eSBlbHNld2hlcmUsIGJlY2F1c2UgdGhlIHBoYW5kbGUgb2YgYSBDUFUgbm9kZSBp cyAKPiBub3QKPiBhIGNoaWxkIG9mIHRoZSBjcHUtbWFwIG5vZGUsIGFuZCB5b3UgY2FuJ3Qgc2lt cGx5IHdhbGsgdXAgdGhlIHRyZWUKPiBoaWVyYXJjaHkgdG8gZmluZCBwYXJlbnQgbm9kZXMuCj4g Cj4gSSBzZWUgbm8gcmVhc29uIHRvIGNoYW5nZSB0aGlzIHBhcnQgb2YgdGhlIGJpbmRpbmcuIEdp dmVuIGl0J3MgYWxyZWFkeQo+IG91dCB0aGVyZSwgd2l0aCBleGlzdGluZyBwYXJzaW5nIGNvZGUs IGNoYW5naW5nIGl0IG9ubHkgc2VydmVzIHRvIGFkZAo+IGNvbXBsZXhpdHkgdG8gYW55IGNvbW1v biBjb2RlIHdoaWNoIGhhcyB0byBwYXJzZSB0aGlzLgo+IAo+IENhbiB3ZSBwbGVhc2UgZHJvcCB0 aGUgaWRlYSBvZiBkcm9wcGluZyB0aGUgdGhyZWFkIG5vZGU/Cj4gCj4+ID4gVGhlcmUgYXJlIGEg dG9ubmUgb2YgZXhpc3RpbmcgYmluZGluZ3MgdGhhdCBhcmUgdWdseSwgYnV0IHJlLWludmVudGlu Zwo+PiA+IHRoZW0gZm9yIHRhc3RlIHJlYXNvbnMgYWxvbmUgaXMgbW9yZSBjb3N0bHkgdG8gdGhl IGVjb3N5c3RlbSB0aGFuIHNpbXBseQo+PiA+IHVzaW5nIHRoZSBleGlzdGluZyBiaW5kaW5ncy4g V2UgYXZvaWQgcmUtaW52ZW50aW5nIGJpbmRpbmdzIHVubGVzcyB0aGVyZQo+PiA+IGlzIGEgZnVu Y3Rpb25hbCBwcm9ibGVtIGUuZy4gY2FzZXMgd2hpY2ggdGhleSBjYW5ub3QgcG9zc2libHkgZGVz Y3JpYmUuCj4+IAo+PiBXZSBhcmUgdGFsa2luZyBhYm91dCB1c2luZyBzb21ldGhpbmcgZm9yIFJJ U0MtViBhbmQgcG9zc2libHkgY29tbW9uIAo+PiBhY3Jvc3MKPj4gZGlmZmVyZW50IGFyY2hzIGFu ZCwgSSBkb24ndCBzZWUgd2h5IHdlIHNob3VsZCBrZWVwIHRoZSB1Z2xpbmVzcyBvZiBhCj4+IGJp bmRpbmcgc3BlYyBwbHVzIGluIHRoaXMgY2FzZSB0aGUgPHRyaGVhZD4gbm9kZSBjYW4ndCBwb3Nz aWJseSAKPj4gZGVzY3JpYmUKPj4gYW55dGhpbmcgZWxzZSB0aGFuIGEgY3B1PTxub2RlPiBhbGlh cywgaXQncyByZWR1bmRhbnQuCj4gCj4gV2hpbGUgaXQgbWF5IGJlIHVnbHksIHJlbW92aW5nIGl0 IG9ubHkgc2VydmVzIHRvIGFkZCBjb21wbGV4aXR5IGZvcgo+IGNvbW1vbiBwYXJzaW5nIGNvZGUs IGFuZCByZW1vdmVzIGZsZXhpYmlsaXR5IHRoYXQgd2UgbWF5IHdhbnQgaW4gCj4gZnV0dXJlLgo+ IEl0cyBwcmVzZW5jZSBkb2VzIG5vdCBjYXVzZSBhbnkgdGVjaG5pY2FsIHByb2JsZW0uCj4gCj4g Rm9yIGJldHRlciBvciB3b3JzZSB3ZSdyZSBhbGwgc2hhcmluZyB0aGUgc2FtZSBjb2RlYmFzZSBo ZXJlLiBUaGUgCj4gY29tbW9uCj4gY2FzZSBtYXR0ZXJzLCBhcyBkb2VzIHRoZSBjb21wbGV4aXR5 IG9mIHRoZSBjb2RlYmFzZSBhcyBhIHdob2xlLgo+IAo+IEkgcmVhbGlzZSBpdCBjYW4gYmUgZnJ1 c3RyYXRpbmcgdG8gaGF2ZSB0byB1c2Ugc29tZXRoaW5nIHlvdSBmZWVsIGlzCj4gc3ViLW9wdGlt YWwsIGJ1dCBwdXR0aW5nIHVwIHdpdGggYSBmZXcgbm9kZXMgd2hpY2ggeW91IHBlcnNvbmFsbHkg ZmVlbAo+IGFyZSByZWR1bmRhbnQgaXMgb2YgbXVjaCBsb3dlciBjb3N0IHRvIHRoZSBlY29zeXN0 ZW0gdGhhbiBoYXZpbmcKPiBpbmNvbXBhdGlibGUgYmluZGluZ3Mgd2hlcmUgd2UgY291bGQgaGF2 ZSBvbmUuIElmIHlvdSBwdXQgdXAgd2l0aCB0aGF0LAo+IHlvdSBjYW4gZm9jdXMgeW91ciBlZmZv cnRzIG9uIG1vcmUgd29ydGh3aGlsZSB0ZWNobmljYWwgZXhlcmNpc2VzLgo+IAoKVGhlIG9ubHkg cmVhc29uIEkgbWVudGlvbmVkIHRoZSBpc3N1ZSB3aXRoIHRoZSA8dGhyZWFkPiBub2RlIGlzIGJl Y2F1c2UKeW91IHNhaWQgdGhhdCB5b3UgYXJlIG9wZW4gZm9yIGltcHJvdmluZyBjcHUtbWFwLiBJ IGRvbid0IHRoaW5rIGl0J3MKc3VjaCBhIGJpZyBkZWFsIGFuZCBldmVuIGlmIHdlIGRlY2lkZSBh Z2FpbnN0IGl0LCBpdCdzIG5vdCBhIGJpZyBkZWFsCnRvIGJlIGJhY2t3YXJkcyBjb21wYXRpYmxl IHdpdGggd2hhdCdzIGFscmVhZHkgdGhlcmUuIEkgZnVsbHkgdW5kZXJzdGFuZAp3aGF0IHlvdSBz YXkgYWJvdXQgdGhlIGltcGFjdCBvbiB0aGUgZWNvc3lzdGVtIGFuZCBhZ3JlZSB3aXRoIHlvdS4K Cj4+ID4gPiBUaGVuIGxldCdzIGFsbG93IHRoZSBjbHVzdGVyIGFuZCBjb3JlIG5vZGVzIHRvIGFj Y2VwdCBhdHRyaWJ1dGVzCj4+ID4gPiB0aGF0IGFyZQo+PiA+ID4gY29tbW9uIGZvciB0aGUgY3B1 cyB0aGV5IGNvbnRhaW4uIFJpZ2h0IG5vdyB0aGlzIGlzIGNvbnNpZGVyZWQKPj4gPiA+IGludmFs aWQuCj4+ID4gPgo+PiA+ID4gRm9yIHBvd2VyIGRvbWFpbnMgd2UgaGF2ZSBhIGdlbmVyaWMgYmlu ZGluZyBkZXNjcmliZWQgb24KPj4gPiA+IERvY3VtZW50YXRpb24vZGV2aWNldHJlZS9iaW5kaW5n cy9wb3dlci9wb3dlcl9kb21haW4udHh0Cj4+ID4gPiB3aGljaCBiYXNpY2FsbHkgc2F5cyB0aGF0 IHdlIG5lZWQgdG8gcHV0IHBvd2VyLWRvbWFpbnMgPSA8cG93ZXIgZG9tYWluCj4+ID4gPiBzcGVj aWZpZXJzPgo+PiA+ID4gYXR0cmlidXRlIG9uIGVhY2ggb2YgdGhlIGNwdSBub2Rlcy4KPj4gPgo+ PiA+IEZXSVcsIGdpdmVuIHRoaXMgaXMgYXJndWFibHkgdG9wb2xvZ2ljYWwsIEknbSBub3QgcGVy c29uYWxseSBhdmVyc2UgdG8KPj4gPiBkZXNjcmliaW5nIHRoaXMgaW4gdGhlIGNwdS1tYXAsIGlm IHRoYXQgYWN0dWFsbHkgZ2FpbnMgdXMgbW9yZSB0aGFuIHRoZQo+PiA+IGNvbXBsZXhpdHkgcmVx dWlyZSB0byBzdXBwb3J0IGl0Lgo+PiA+Cj4+ID4gR2l2ZW4gd2UgZG9uJ3QgZG8gdGhpcyBmb3Ig ZGV2aWNlIHBvd2VyIGRvbWFpbnMsIEkgc3VzcGVjdCB0aGF0IGl0J3MKPj4gPiBzaW1wbGVyIHRv IHN0aWNrIHdpdGggdGhlIGV4aXN0aW5nIGJpbmRpbmcuCj4+ID4KPj4gPiA+IFRoZSBzYW1lIGhh cHBlbnMgd2l0aCB0aGUgY2FwYWNpdHkgYmluZGluZyBzcGVjaWZpZWQgZm9yIGFybSBvbgo+PiA+ ID4gRG9jdW1lbnRhdGlvbi9kZXZpY2V0cmVlL2JpbmRpbmdzL2FybS9jcHUtY2FwYWNpdHkudHh0 Cj4+ID4gPiB3aGljaCBzYXlzIHdlIHNob3VsZCBhZGQgdGhlIGNhcGFjaXR5LWRtaXBzLW1oeiBv biBlYWNoIG9mIHRoZSBjcHUKPj4gPiA+IG5vZGVzLgo+PiA+Cj4+ID4gVGhlIGNwdS1tYXAgd2Fz IGludGVuZGVkIHRvIGV4cG9zZSB0b3BvbG9naWNhbCBkdGFpbHMsIGFuZCB0aGlzIGlzbid0Cj4+ ID4gcmVhbGx5IGEgdG9wb2xvZ2ljYWwgcHJvcGVydHkuIEZvciBleGFtcGxlLCBBcm0gRHluYW1J USBzeXN0ZW1zIGNhbiBoYXZlCj4+ID4gaGV0ZXJvZ2VuZW91cyBDUFVzIHdpdGhpbiBjbHVzdGVy cy4KPj4gPgo+PiA+IEkgZG8gbm90IHRoaW5rIGl0J3Mgd29ydGggbW92aW5nIHRoaXMsIHRiaC4K Pj4gPgo+PiA+ID4gVGhlIHNhbWUgYWxzbyBoYXBwZW5zIHdpdGggdGhlIGdlbmVyaWMgbnVtYSBi aW5kaW5nIG9uCj4+ID4gPiBEb2N1bWVudGF0aW9uL2RldmljZXRyZWUvYmluZGluZ3MvbnVtYS50 eHQKPj4gPiA+IHdoaWNoIHNheXMgd2Ugc2hvdWxkIGFkZCB0aGUgbnVuYS1ub2RlLWlkIG9uIGVh Y2ggb2YgdGhlIGNwdSBub2Rlcy4KPj4gPgo+PiA+IElzIHRoZXJlIGEgc3Ryb25nIGdhaW4gZnJv bSBtb3ZpbmcgdGhpcz8KPj4gPgo+PiA+IFsuLi5dCj4+IAo+PiBSaWdodCBub3cgd2l0aCB0aGUg ZGV2aWNlIHRyZWUgc3BlYyBhbmQgdGhlIGFib3ZlIGJpbmRpbmdzIHdlIGNhbgo+PiB1c2UgdGhl IGluZm9ybWF0aW9uIGZyb20gY3B1IG5vZGVzIHRvIGluZmVyIHRoZSBjYWNoZSB0b3BvbG9neSwK Pj4gdGhlIG1lbW9yeSB0b3BvbG9neSwgdGhlIHBvd2VyIGRvbWFpbiB0b3BvbG9neSBldGMuCj4+ IAo+PiBXZSBoYXZlIG9mX2ZpbmRfbmV4dF9jYWNoZV9ub2RlIGFuZCBvZl9maW5kX2xhc3RfY2Fj aGVfbGV2ZWwgZm9yIAo+PiBleGFtcGxlCj4+IHRoYXQgdXNlICJuZXh0LWxldmVsLWNhY2hlIiBh bmQgYXJlIHVzZWQgb24gcG93ZXJwYyAod2hlcmUgdGhpcyBzcGVjIAo+PiBjb21lcwo+PiBmcm9t KSwgdGhhdCB3ZSBjYW4gZnVydGhlciBidWlsZCBvbiB0b3Agb2YgdGhlbSAoc2luY2UgdGhpcyBp cyBub3cgCj4+IHBhcnQKPj4gb2YgdGhlICBkZXZpY2UgdHJlZSBzcGVjIGFueXdheSksIHRvIGUu Zy4gcG9wdWxhdGUgdGhlIGxldmVscyBvZiAKPj4gY2FjaGUsCj4+IHRoZSBsZXZlbHMgb2Ygc2hh cmVkIGNhY2hlIGFuZCBmaW5hbGx5IGNyZWF0ZSBjcHUgbWFza3MgZm9yIHRoZSAKPj4gZGlmZmVy ZW50Cj4+IGNhY2hlIHNoYXJpbmcgbGV2ZWxzLgo+IAo+IFRoZSBjcHUtbWFwIGJpbmRpbmcgZG9l cyBub3QgZGVzY3JpYmUgY2FjaGUgdG9wb2xvZ3kuIEkgbmV2ZXIgc3VnZ2VzdGVkCj4gdGhhdCBp dCBkaWQuCj4gCgpTb3JyeSBmb3IgdGhlIG1pc3VuZGVyc3RhbmRpbmcgTWFyayAhCgpPbiBBUk0g dGhlIENQVSB0b3BvbG9neSBpcyB1c2VkIGZvciBjb25maWd1cmluZyB0aGUgc2NoZWR1bGluZyBk b21haW4gCnRvcG9sb2d5CihhcmNoL2FybS9rZXJuZWwvdG9wb2xvZ3kuYykgYW5kIGJsaW5kbHkg c2V0cyB0aGUgU0RfU0hBUkVfUEtHX1JFU09VUkNFUwphbmQgU0RfU0hBUkVfUE9XRVJET01BSU4g ZmxhZ3MgZm9yIGEgY29yZSB3aXRob3V0IHRha2luZyBpbnRvIGFjY291bnQgCnRoZQpjYWNoZSBo aWVyYXJjaHkgKGhlbmNlIHZhbGlkYXRpbmcgdGhhdCBTRF9TSEFSRV9QS0dfUkVTT1VSQ0VTIGlz IApjb3JyZWN0KQpvciB0aGUgcG93ZXIgZG9tYWlucyAoYnV0IG9rLCBJIGRvdWJ0IGl0J3MgcG9z c2libGUgdG8gaGF2ZSB0d28gdGhyZWFkcyAKb24KdGhlIHNhbWUgY29yZSB0aGF0IGFyZSBwb3dl cmVkIGluZGVwZW5kZW50bHkpLiBJdCBhbHNvIHN1cHBvcnRzIG9ubHkgdHdvCnNjaGVkdWxpbmcg ZG9tYWlucywgb25lIGZvciBTTVQgYW5kIGFub3RoZXIgb25lIGZvciBNQyAoaGVuY2UgbXkgCmNv bW1lbnRzCm9uIG11bHRpcGxlIGxldmVscyBvZiBzaGFyZWQgcmVzb3VyY2VzIG5vdCBiZWluZyBz dXBwb3J0ZWQpLgoKU2luY2UgYWNjb3JkaW5nIHRvIHRoZSBkb2NzIGNwdS1tYXAgaXMgYSBiaW5k aW5nIGNvbW1vbiBiZXR3ZWVuIEFSTSBhbmQgCkFSTTY0CmZvciBkZXNjcmliaW5nIHRoZSBDUFUg dG9wb2xvZ3ksIEkgYXNzdW1lZCBpdCB3YXMgYSBwYXJ0IG9mIHRoYXQgCnByb2Nlc3MuCgpUdXJu cyBvdXQgaXQncyBvbmx5IHVzZWQgb24gQVJNNjQsIHdoZXJlIHNldF9zY2hlZF90b3BvbG9neSgp IGlzIG5vdCAKdXNlZCBmb3IKY29uZmlndXJpbmcgdGhlIHNjaGVkdWxpbmcgZG9tYWluIHRvcG9s b2d5ICh0aGVyZSBpcyBhIGNvbW1lbmQgdGhlcmUgCnRoYXQKaW1wbGllcyB0aGF0IHlvdSBwYXNz IG9uIHRoZSBjbHVzdGVycyB0byB0aGUgc2NoZWR1bGVyIHRoYXQgYWxzbyBsZWQgbWUgCnRvCmJl bGlldmUgdGhlIGlzc3VlIGlzIGFsc28gcHJlc2VudCB0aGVyZSkuIFNvIG15IGFzc3VtcHRpb24g d2FzIHdyb25nIGFuZApjcHUtbWFwIGhhcyBub3RoaW5nIHRvIGRvIHdpdGggY29uZmlndXJpbmcg dGhlIHNjaGVkdWxlciBhbmQgdGhlIGlzc3VlcwpJIG1lbnRpb25lZCBhYm92ZS4KCj4+IFRoaXMg aXMgc3RhbmRhcmRzLWNvbXBsaWFudCwgYXJjaC1pbmRlcGVuZGVudCAodGhleSBhcmUgb24gb2Yv YmFzZS5jKSwgCj4+IGFuZAo+PiBpdCBwcm92aWRlcyBhIGNvbmNyZXRlIGhpbnQgb24gQ1BVIHRv cG9sb2d5IHJhdGhlciBncm91cGluZyBoYXJ0cyB0byAKPj4gY2xhc3Nlcwo+PiBsaWtlICJjb3Jl IiwgInBhY2thZ2UiLCAic29ja2V0IiwgImNsdXN0ZXIiIGV0YyB0aGF0IGRvbid0IG1lYW4gCj4+ IGFueXRoaW5nCj4+IHNwZWNpZmljIGFuZCB3ZSBhc3N1bWUgdG8gbWFwIHRvIHNwZWNpZmljIGxl dmVscyBvZiBjYWNoZSBzaGFyaW5nLgo+IAo+IFBsZWFzZSBub3RlIHRoYXQgSSBoYXZlIG5ldmVy IHN1Z2dlc3RlZCAicGFja2FnZSIsIGFuZCBJJ20gbm90IHN1cmUgCj4gd2hhdAo+IHRoYXQncyBp biByZXNwb25zZSB0by4KPiAKCmh0dHBzOi8vbGttbC5vcmcvbGttbC8yMDE4LzExLzcvOTE4Cgpb Li4uXQoKPj4gSW4gYSBzZW5zZSBpdCBnb2VzIGFnYWluc3QgeW91ciBydWxlIG9mICJyZS1pbnZl bnRpbmcgdGhlbSBmb3IgdGFzdGUKPj4gcmVhc29ucyBhbG9uZSBpcyBtb3JlIGNvc3RseSB0byB0 aGUgZWNvc3lzdGVtIHRoYW4gc2ltcGx5IHVzaW5nIHRoZQo+PiBleGlzdGluZyBiaW5kaW5ncyIs IEkgZmFpbCB0byBzZWUgYW55dGhpbmcgZWxzZSB0aGFuICJ0YXN0ZSByZWFzb25zIgo+PiB3aGVu IGl0IGNvbWVzIHRvIGNwdS1tYXAsIHNpbmNlIHRoZXJlIGFyZSBleGlzdGluZyBiaW5kaW5ncyBm b3IKPj4gaW5mZXJyaW5nIHRoZSBDUFUgdG9wb2xvZ3kgYWxyZWFkeSwgdGhleSBhcmUganVzdCBu b3QgdGhhdCBjb252ZW5pZW50Lgo+IAo+IElmIHlvdSBiZWxpZXZlIHRoYXQncyB0aGUgY2FzZSwg dGhlbiB5b3UgY2FuIHBlcmZlY3RseSB1c2UgdGhlIGV4aXN0aW5nCj4gY2FjaGUgYW5kIE5VTUEg YmluZGluZ3MsIGFuZCBub3QgZGVzY3JpYmUgdGhlIGNwdSB0b3BvbG9neSBhdCBhbGwsIG5vCj4g bmVlZCB0byBjaGFuZ2UgY3B1LW1hcCBvciBjb21lIHVwIHdpdGggYSBuZXcgYmluZGluZy4KPiAK ClRoZSBwcm9ibGVtIGlzIHRoYXQgcmlnaHQgbm93IG5vYm9keSBpcyB1c2luZyB0aGUgZ2VuZXJp YyBiaW5kaW5ncwpmb3IgY29uZmlndXJpbmcgdGhlIHNjaGVkdWxlciBkb21haW4gdG9wb2xvZ3ku IE9ubHkgdGhlIG51bWEgYmluZGluZ3MKYXJlIHVzZWQgYXMgZXhwZWN0ZWQgb24gQVJNNjQuIFNp bmNlIGNwdS1tYXAgaXMgb25seSB0aGVyZSBmb3IgCnJlcHJlc2VudGluZwp0aGUgY3B1IHRvcG9s b2d5L2xheW91dCwgYWxsb3dpbmcgdGhlbSB0byBleGlzdCB0aGVyZSBpcyBvYnZpb3VzbHkgbm90 IAp0aGUKcmlnaHQgYXBwcm9hY2guCgo+PiBJJ20gcHJvcG9zaW5nIHRvIGFkZCB0aGUgb3B0aW9u IChub3QgZW5mb3JjZSkgb2YgYWRkaW5nIHRob3NlCj4+IGF0dHJpYnV0ZXMgdGhhdCBhcmUgbWVh bnQgdG8gYmUgdXNlZCBvbiBjcHUgbm9kZXMsIG9uIHRoZSB2YXJpb3VzCj4+IGdyb3Vwcy9jbGFz c2VzIG9mIHRoZSBjcHUtbWFwLCB0aGlzIHdheSBjcHUtbWFwIHdpbGwgcHJvdmlkZSBzb21ldGhp bmcKPj4gbW9yZSBtZWFuaW5nZnVsIGFuZCBhdCBsZWFzdCBpbXByb3ZlIHRoZSByZXByZXNlbnRh dGlvbiBzaWRlIG9mCj4+IHRoaW5ncy4gT24gdGhlIGltcGxlbWVudGF0aW9uIHNpZGUgaXQgbWln aHQgc2F2ZSB1cyB0aGUgYnVyZGVuIG9mCj4+IGluZmVyaW5nIHRoZSB0b3BvbG9neSBmcm9tIHBh cnNpbmcgYWxsIGNwdSBub2RlcyBlYWNoIHRpbWUuIEl0J3MgYWxzbwo+PiBiYWNrd2FyZHMgY29t cGF0aWJsZSB3aXRoIHdoYXQgeW91IGFscmVhZHkgaGF2ZSwgdGhlIG9ubHkgdGhpbmcgdGhhdCdz Cj4+IG5vdCBiYWNrd2FyZHMgY29tcGF0aWJsZSBpcyB0aGUgcmVtb3ZhbCBvZiB0aGUgPHRocmVh ZD4gbm9kZSwgd2hpY2ggSQo+PiBkb24ndCB0aGluayBpcyBzdWNoIGEgYmlnIGRlYWwgdG8gZml4 Lgo+IAo+IFNvcnJ5LCBidXQgYXMgSSBoYXZlIHN0YXRlZCByZXBlYXRlZGx5LCB0aGlzIGNvbXBs aWNhdGVzIHRoZSBjb21tb24gCj4gY29kZQo+IGZvciB3aGF0IHlvdSBhZG1pdCBpcyBhIG1hdHRl ciBvZiB0YXN0ZS4gSSB3b3VsZCByYXRoZXIgbWFpbnRhaW4gdGhlCj4gc2ltcGxpY2l0eSBvZiB0 aGlzIGJpbmRpbmcgYW5kIGV4aXN0aW5nIHBhcnNpbmcgY29kZSwgYW5kIG5vdAo+IHBvdGVudGlh bGx5IGJyZWFrIG90aGVyIHBhcnNlcnMsIGlmIHRoZSBvbmx5IGNvc3QgaXMgYSBmZXcgbm9kZXMg d2hpY2gKPiB5b3UgcGVyc29uYWxseSBjb25zaWRlciB0byBiZSByZWR1bmRhbnQuCj4gCgpbLi4u XQoKPj4gU28gd2UgYWdyZWUsIHRoZSAiY29yZSIgZG9lc24ndCBzYXkgYW55dGhpbmcgdXNlZnVs IHJlZ2FyZGluZyB0aGUKPj4gdG9wb2xvZ3ksIHdoeSBrZWVwIHVzaW5nIHRoaXMgZGlzdGluY3Rp b24gb24gYSBiaW5kaW5nIGZvciBSSVNDLVYgYW5kCj4+IHBvc3NpYmx5IG90aGVyIGFyY2hzIChz aW5jZSB3ZSBhcmUgYWxzbyB0YWxraW5nIG9uIGFuIGFyY2gtaW5kZXBlbmRlbnQKPj4gYXBwcm9h Y2gpID8KPiAKPiBXZSBrZWVwIGl0IGJlY2F1c2UgdGhlIGJpbmRpbmcgaGFzIGFscmVhZHkgYmVl biBmaW5hbGlzZWQgYW5kIGlzIHVzZSBpbgo+IHByYWN0aWNlLiBUaGUgZXhpc3RpbmcgYmluZGlu ZyBhbmQgcGFyc2luZyBjb2RlIGlzICpzdWZmaWNpZW50KiBmb3IgCj4geW91cgo+IG5lZWRzLiBQ ZXJzb25hbCB0YXN0ZSBhbmQgbWlub3Igc2F2aW5ncyBpbiB0aGUgbnVtYmVyIG9mIG5vZGVzIGlu IGEgRFQKPiBhcmUgbm90IGNvbXBlbGxpbmcgcmVhc29ucyB0byBjaGFuZ2UgdGhpcyB3aGVuIHRo ZSBjb3N0IGlzIGNvbXBsZXhpdHkKPiB0aGF0IHdlIGhhdmUgdG8gbWFpbnRhaW4gKmZvcmV2ZXIq Lgo+IAoKVG90YWxseSBmaW5lIHdpdGggdGhhdCwgYXMgSSBtZW50aW9uZWQgYWJvdmUgSSBwb2lu dGVkIHRoZXNlIG91dCBzaW5jZSBJCmNvbnNpZGVyIGl0IGFuIGltcHJvdmVtZW50LiBGb3JnaXZl IG1lIGlmIEkgZ2F2ZSB5b3UgdGhlIGltcHJlc3Npb24gdGhhdAp3YXMgYSBkZWFsLWJyZWFrZXIg b3Igc29tZXRoaW5nLgoKUmVnYXJkcywKTmljawoKX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX18KbGludXgtcmlzY3YgbWFpbGluZyBsaXN0CmxpbnV4LXJpc2N2 QGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9s aXN0aW5mby9saW51eC1yaXNjdgo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13BB1C43610 for ; Fri, 9 Nov 2018 03:58:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C31CE2086C for ; Fri, 9 Nov 2018 03:58:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C31CE2086C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ics.forth.gr Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727718AbeKINhI (ORCPT ); Fri, 9 Nov 2018 08:37:08 -0500 Received: from mailgate-4.ics.forth.gr ([139.91.1.7]:26264 "EHLO mailgate-4.ics.forth.gr" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727238AbeKINhH (ORCPT ); Fri, 9 Nov 2018 08:37:07 -0500 Received: from av1.ics.forth.gr (av3in.ics.forth.gr. [139.91.1.77]) by mailgate-4.ics.forth.gr (8.14.5/ICS-FORTH/V10-1.9-GATE-OUT) with ESMTP id wA93tWJM017440; Fri, 9 Nov 2018 05:55:34 +0200 (EET) X-AuditID: 8b5b9d4d-91bff70000000e62-d6-5be505334562 Received: from enigma.ics.forth.gr (enigma.ics.forth.gr [139.91.1.35]) by av1.ics.forth.gr (SMTP Outbound / FORTH / ICS) with SMTP id 16.2C.03682.33505EB5; Fri, 9 Nov 2018 05:55:31 +0200 (EET) Received: from webmail.ics.forth.gr (localhost [127.0.0.1]) by enigma.ics.forth.gr (8.15.1//ICS-FORTH/V10.5.0C-EXTNULL-SSL-SASL) with ESMTP id wA93tTQB018040; Fri, 9 Nov 2018 05:55:29 +0200 X-ICS-AUTH-INFO: Authenticated user: at ics.forth.gr MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Fri, 09 Nov 2018 05:55:29 +0200 From: Nick Kossifidis To: Mark Rutland Cc: Nick Kossifidis , Sudeep Holla , Atish Patra , linux-riscv@lists.infradead.org, devicetree@vger.kernel.org, Damien.LeMoal@wdc.com, alankao@andestech.com, zong@andestech.com, anup@brainfault.org, palmer@sifive.com, linux-kernel@vger.kernel.org, hch@infradead.org, robh+dt@kernel.org, tglx@linutronix.de Subject: Re: [RFC 0/2] Add RISC-V cpu topology Organization: FORTH In-Reply-To: <20181108155404.32ja7bbxu62nyytt@lakrids.cambridge.arm.com> References: <1541113468-22097-1-git-send-email-atish.patra@wdc.com> <866dedbc78ab4fa0e3b040697e112106@mailhost.ics.forth.gr> <20181106141331.GA28458@e107155-lin> <969fc2a5198984e0dfe8c3f585dc65f9@mailhost.ics.forth.gr> <20181106162051.w7fyweuxrl7ujzuz@lakrids.cambridge.arm.com> <20181107120645.sc3wjgr2yakvxktl@lakrids.cambridge.arm.com> <20181108155404.32ja7bbxu62nyytt@lakrids.cambridge.arm.com> Message-ID: <0b2ce6ab4fa804e2ce8cea5eca5b02da@mailhost.ics.forth.gr> X-Sender: mick@mailhost.ics.forth.gr User-Agent: Roundcube Webmail/1.1.2 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrKIsWRmVeSWpSXmKPExsXSHc2orGvC+jTaYO8RJottS1azWrR8eMdq sWjFdxaL1vZvTBbzj5xjtTg9YRGTxeVdc9gstn1uYbNYev0ik0Xzu3PsFpsnLGC1aN17hN1i +akdLBabN01ltni+spfNgd9jz+lZzB5r5q1h9Jj6+wyLx8NNl5g8Nq/Q8ti0qpPN4925c+we m5fUe1xqvs7u8XmTnEf7gW6mAO4oLpuU1JzMstQifbsErowf96+yFbyOrjg27zFTA+MBjy5G Tg4JAROJ/3uuMHYxcnEICRxmlDjx4Ag7hHOQUWLmvn/MEFWmErP3djKC2LwCghInZz5hAbGZ BSwkpl7Zzwhhy0s0b50NVs8ioCpx5sQaNhCbTUBTYv6lg2D1IgLqEj27vrCALGAWmMcscbJv JViRsICeRNOGm6wgNr+AsMSnuxfBbE4BD4nN344yQ1w0gUWi9eZ+ZogrXCSmTdvHBnGdisSH 3w/YQWxRAWWJFyems05gFJqF5NhZSI6dheTYBYzMqxgFEsuM9TKTi/XS8otKMvTSizYxgiNz ru8OxnML7A8xCnAwKvHwrmh4Ei3EmlhWXJl7iFGCg1lJhHfTP6AQb0piZVVqUX58UWlOavEh RmkOFiVx3sMvwoOEBNITS1KzU1MLUotgskwcnFINjHnNN0ovh0cX6IeFZKntr768LOCCQupd ja/3TROkHe2m82Txf3n55vrph5KFJ8XyOZ6I/p/a+vgs9+0A4xKThCkzw+apbNlzUuq81OZ6 QS7puws3VbhJe+31CU/j+7GaY8EDziRRkzVhbwv/bLd15ZrDdvvQv5MLtly4xv8hLsNF7vpF 4aV7apVYijMSDbWYi4oTATV1jbXIAgAA Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Στις 2018-11-08 17:54, Mark Rutland έγραψε: > On Thu, Nov 08, 2018 at 03:45:36PM +0200, Nick Kossifidis wrote: >> Στις 2018-11-07 14:06, Mark Rutland έγραψε: >> > On Wed, Nov 07, 2018 at 04:31:34AM +0200, Nick Kossifidis wrote: >> > > Mark and Sundeep thanks a lot for your feedback, I guess you convinced >> > > me that having a device tree binding for the scheduler is not a >> > > correct approach. It's not a device after all and I agree that the >> > > device tree shouldn't become an OS configuration file. >> > >> > Good to hear. >> > >> > > Regarding multiple levels of shared resources my point is that since >> > > cpu-map doesn't contain any information of what is shared among the >> > > cluster/core members it's not easy to do any further translation. Last >> > > time I checked the arm code that uses cpu-map, it only defines one >> > > domain for SMT, one for MC and then everything else is ignored. No >> > > matter how many clusters have been defined, anything above the core >> > > level is the same (and then I guess you started talking about adding >> > > "packages" on the representation side). >> > >> > While cpu-map doesn't contain that information today, we can *add* that >> > information to the cpu-map binding if necessary. >> > >> > > The reason I proposed to have a binding for the scheduler directly is >> > > not only because it's simpler and closer to what really happens in the >> > > code, it also makes more sense to me than the combination of cpu-map >> > > with all the related mappings e.g. for numa or caches or power >> > > domains etc. >> > > >> > > However you are right we could definitely augment cpu-map to include >> > > support for what I'm saying and clean things up, and since you are >> > > open about improving it here is a proposal that I hope you find >> > > interesting: >> > > >> > > At first let's get rid of the nodes, they don't make sense: >> > > >> > > thread0 { >> > > cpu = <&CPU0>; >> > > }; >> > > >> > > A thread node can't have more than one cpu entry and any properties >> > > should be on the cpu node itself, so it doesn't / can't add any >> > > more information. We could just have an array of cpu nodes on the >> > > node, it's much cleaner this way. >> > > >> > > core0 { >> > > members = <&CPU0>, <&CPU1>; >> > > }; >> > >> > Hold on. Rather than reinventing things from first principles, can we >> > please discuss what you want to *achieve*, i.e. what information you >> > need? >> > >> > Having a node is not a significant cost, and there are reasons we may >> > want thread nodes. For example, it means that we can always refer to any >> > level of topology with a phandle, and we might want to describe >> > thread-affine devices in future. >> >> You can use the phandle of the cpu node, the thread node doesn't add >> anything more than complexity to the representation. > > That adds complexity elsewhere, because the phandle of a CPU node is > not > a child of the cpu-map node, and you can't simply walk up the tree > hierarchy to find parent nodes. > > I see no reason to change this part of the binding. Given it's already > out there, with existing parsing code, changing it only serves to add > complexity to any common code which has to parse this. > > Can we please drop the idea of dropping the thread node? > >> > There are a tonne of existing bindings that are ugly, but re-inventing >> > them for taste reasons alone is more costly to the ecosystem than simply >> > using the existing bindings. We avoid re-inventing bindings unless there >> > is a functional problem e.g. cases which they cannot possibly describe. >> >> We are talking about using something for RISC-V and possibly common >> across >> different archs and, I don't see why we should keep the ugliness of a >> binding spec plus in this case the node can't possibly >> describe >> anything else than a cpu= alias, it's redundant. > > While it may be ugly, removing it only serves to add complexity for > common parsing code, and removes flexibility that we may want in > future. > Its presence does not cause any technical problem. > > For better or worse we're all sharing the same codebase here. The > common > case matters, as does the complexity of the codebase as a whole. > > I realise it can be frustrating to have to use something you feel is > sub-optimal, but putting up with a few nodes which you personally feel > are redundant is of much lower cost to the ecosystem than having > incompatible bindings where we could have one. If you put up with that, > you can focus your efforts on more worthwhile technical exercises. > The only reason I mentioned the issue with the node is because you said that you are open for improving cpu-map. I don't think it's such a big deal and even if we decide against it, it's not a big deal to be backwards compatible with what's already there. I fully understand what you say about the impact on the ecosystem and agree with you. >> > > Then let's allow the cluster and core nodes to accept attributes >> > > that are >> > > common for the cpus they contain. Right now this is considered >> > > invalid. >> > > >> > > For power domains we have a generic binding described on >> > > Documentation/devicetree/bindings/power/power_domain.txt >> > > which basically says that we need to put power-domains = > > > specifiers> >> > > attribute on each of the cpu nodes. >> > >> > FWIW, given this is arguably topological, I'm not personally averse to >> > describing this in the cpu-map, if that actually gains us more than the >> > complexity require to support it. >> > >> > Given we don't do this for device power domains, I suspect that it's >> > simpler to stick with the existing binding. >> > >> > > The same happens with the capacity binding specified for arm on >> > > Documentation/devicetree/bindings/arm/cpu-capacity.txt >> > > which says we should add the capacity-dmips-mhz on each of the cpu >> > > nodes. >> > >> > The cpu-map was intended to expose topological dtails, and this isn't >> > really a topological property. For example, Arm DynamIQ systems can have >> > heterogeneous CPUs within clusters. >> > >> > I do not think it's worth moving this, tbh. >> > >> > > The same also happens with the generic numa binding on >> > > Documentation/devicetree/bindings/numa.txt >> > > which says we should add the nuna-node-id on each of the cpu nodes. >> > >> > Is there a strong gain from moving this? >> > >> > [...] >> >> Right now with the device tree spec and the above bindings we can >> use the information from cpu nodes to infer the cache topology, >> the memory topology, the power domain topology etc. >> >> We have of_find_next_cache_node and of_find_last_cache_level for >> example >> that use "next-level-cache" and are used on powerpc (where this spec >> comes >> from), that we can further build on top of them (since this is now >> part >> of the device tree spec anyway), to e.g. populate the levels of >> cache, >> the levels of shared cache and finally create cpu masks for the >> different >> cache sharing levels. > > The cpu-map binding does not describe cache topology. I never suggested > that it did. > Sorry for the misunderstanding Mark ! On ARM the CPU topology is used for configuring the scheduling domain topology (arch/arm/kernel/topology.c) and blindly sets the SD_SHARE_PKG_RESOURCES and SD_SHARE_POWERDOMAIN flags for a core without taking into account the cache hierarchy (hence validating that SD_SHARE_PKG_RESOURCES is correct) or the power domains (but ok, I doubt it's possible to have two threads on the same core that are powered independently). It also supports only two scheduling domains, one for SMT and another one for MC (hence my comments on multiple levels of shared resources not being supported). Since according to the docs cpu-map is a binding common between ARM and ARM64 for describing the CPU topology, I assumed it was a part of that process. Turns out it's only used on ARM64, where set_sched_topology() is not used for configuring the scheduling domain topology (there is a commend there that implies that you pass on the clusters to the scheduler that also led me to believe the issue is also present there). So my assumption was wrong and cpu-map has nothing to do with configuring the scheduler and the issues I mentioned above. >> This is standards-compliant, arch-independent (they are on of/base.c), >> and >> it provides a concrete hint on CPU topology rather grouping harts to >> classes >> like "core", "package", "socket", "cluster" etc that don't mean >> anything >> specific and we assume to map to specific levels of cache sharing. > > Please note that I have never suggested "package", and I'm not sure > what > that's in response to. > https://lkml.org/lkml/2018/11/7/918 [...] >> In a sense it goes against your rule of "re-inventing them for taste >> reasons alone is more costly to the ecosystem than simply using the >> existing bindings", I fail to see anything else than "taste reasons" >> when it comes to cpu-map, since there are existing bindings for >> inferring the CPU topology already, they are just not that convenient. > > If you believe that's the case, then you can perfectly use the existing > cache and NUMA bindings, and not describe the cpu topology at all, no > need to change cpu-map or come up with a new binding. > The problem is that right now nobody is using the generic bindings for configuring the scheduler domain topology. Only the numa bindings are used as expected on ARM64. Since cpu-map is only there for representing the cpu topology/layout, allowing them to exist there is obviously not the right approach. >> I'm proposing to add the option (not enforce) of adding those >> attributes that are meant to be used on cpu nodes, on the various >> groups/classes of the cpu-map, this way cpu-map will provide something >> more meaningful and at least improve the representation side of >> things. On the implementation side it might save us the burden of >> infering the topology from parsing all cpu nodes each time. It's also >> backwards compatible with what you already have, the only thing that's >> not backwards compatible is the removal of the node, which I >> don't think is such a big deal to fix. > > Sorry, but as I have stated repeatedly, this complicates the common > code > for what you admit is a matter of taste. I would rather maintain the > simplicity of this binding and existing parsing code, and not > potentially break other parsers, if the only cost is a few nodes which > you personally consider to be redundant. > [...] >> So we agree, the "core" doesn't say anything useful regarding the >> topology, why keep using this distinction on a binding for RISC-V and >> possibly other archs (since we are also talking on an arch-independent >> approach) ? > > We keep it because the binding has already been finalised and is use in > practice. The existing binding and parsing code is *sufficient* for > your > needs. Personal taste and minor savings in the number of nodes in a DT > are not compelling reasons to change this when the cost is complexity > that we have to maintain *forever*. > Totally fine with that, as I mentioned above I pointed these out since I consider it an improvement. Forgive me if I gave you the impression that was a deal-breaker or something. Regards, Nick