From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 703E9C433E7 for ; Mon, 19 Oct 2020 13:16:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0682D222C3 for ; Mon, 19 Oct 2020 13:16:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727691AbgJSNQ0 convert rfc822-to-8bit (ORCPT ); Mon, 19 Oct 2020 09:16:26 -0400 Received: from lhrrgout.huawei.com ([185.176.76.210]:2989 "EHLO huawei.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727297AbgJSNQZ (ORCPT ); Mon, 19 Oct 2020 09:16:25 -0400 Received: from lhreml710-chm.china.huawei.com (unknown [172.18.7.106]) by Forcepoint Email with ESMTP id 1C9D932FDD75741903FF; Mon, 19 Oct 2020 14:16:24 +0100 (IST) Received: from localhost (10.52.126.130) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Mon, 19 Oct 2020 14:16:23 +0100 Date: Mon, 19 Oct 2020 14:14:27 +0100 From: Jonathan Cameron To: Sudeep Holla CC: , , , , Len Brown , Morten Rasmussen , "Greg Kroah-Hartman" , , "Will Deacon" , , Brice Goglin Subject: Re: [RFC PATCH] topology: Represent clusters of CPUs within a die. Message-ID: <20201019131427.00006f40@Huawei.com> In-Reply-To: <20201019100100.GA12908@bogus> References: <20201016152702.1513592-1-Jonathan.Cameron@huawei.com> <20201019100100.GA12908@bogus> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; i686-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8BIT X-Originating-IP: [10.52.126.130] X-ClientProxiedBy: lhreml711-chm.china.huawei.com (10.201.108.62) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-acpi@vger.kernel.org On Mon, 19 Oct 2020 11:01:00 +0100 Sudeep Holla wrote: > +Morten Hi Sudeep, > > On Fri, Oct 16, 2020 at 11:27:02PM +0800, Jonathan Cameron wrote: > > Both ACPI and DT provide the ability to describe additional layers of > > topology between that of individual cores and higher level constructs > > such as the level at which the last level cache is shared. > > In ACPI this can be represented in PPTT as a Processor Hierarchy > > Node Structure [1] that is the parent of the CPU cores and in turn > > has a parent Processor Hierarchy Nodes Structure representing > > a higher level of topology. > > > > For example Kunpeng 920 has clusters of 4 CPUs. These do not share > > any cache resources, but the interconnect topology is such that > > the cost to transfer ownership of a cacheline between CPUs within > > a cluster is lower than between CPUs in different clusters on the same > > die. Hence, it can make sense to deliberately schedule threads > > sharing data to a single cluster. > > > > This is very SoC specific and hard to generalise, hence LLC is chosen > as one of the main factor to decide. I absolutely agree. It is very hard to modify the kernel scheduler to take into account this information. Chances are we'll help some workloads and hurt others. However, it is good to at least expose that it exists. > > Are there any scheduler topology related changes to achieve the same ? > If so, we need their opinion on that. So far no. We have done a few experiments with adding this knowledge into the scheduler but it is still very much a work in progress. However, we do have micro benchmarks showing clearly that can make a substantial difference. Our intent was to first expose this information to user space and get experience of using it in applications via HWLOC before we tried to propose any scheduler changes. > > > This patch simply exposes this information to userspace libraries > > like hwloc by providing cluster_cpus and related sysfs attributes. > > PoC of HWLOC support at [2]. > > > > OK, cluster is too Arm specific with no definition for it whatsoever. It does get used by others (ACPI for example), but I agree there may be a naming issue here. Brice observed that hwloc uses the term group to cover the somewhat obscure drawer and book levels that powerpc uses. Perhaps that is a better name? > How do you plan to support clusters of clusters or higher level of > hierarchy present in PPTT. Right now I don't. I was hoping that question would come up, but didn't want to complicate the first version of this proposal. Guess I got that wrong ;) As Brice suggested, probably adding an index makes sense. I'm not sure if we bother implementing more than one layer for now, but we definitely want a naming scheme that allows for it. If anyone has a CPU that already uses another layer let us know! On another note, we are thinking of an ACPI PPTT addition to add a flag for die, if anyone has another suggestions on how to match the x86 CPUID part for that, it would be great to align. At least die is fairly well defined... Tile and Module not so much... > > > Note this patch only handle the ACPI case. > > > > If we decide to add this to sysfs, I prefer to keep DT implementation > also in sync. The bindings are in sync, just matter of implementation. Ok, happy to look at DT that if we seem to be moving forwards with this in general. > > > Special consideration is needed for SMT processors, where it is > > necessary to move 2 levels up the hierarchy from the leaf nodes > > (thus skipping the processor core level). > > > > Currently the ID provided is the offset of the Processor > > Hierarchy Nodes Structure within PPTT. Whilst this is unique > > it is not terribly elegant so alternative suggestions welcome. > > > > That is firmware choice. May be your firmware just fills in mandatory > fields and doesn't care about anything and everything optional. We do > check for valid UID fields and if that is not set, offset is used as > unique ID in kernel implementation. So if you enhance the firmware, the > kernel sysfs will become elegant as you expect 😉 Fair enough that there is a way to improve it, but it would be nice if the default solution was also elegant! I'll add the processor container DSDT stuff to one or two of my test cases and make sure we can get that working. There are a lot of combinations that need testing and I get bored quickly writing firmware tables by hand. > > > Note that arm64 / ACPI does not provide any means of identifying > > a die level in the topology but that may be unrelate to the cluster > > level. > > > > Indeed. It can be cluster of clusters on some platform. If we need that > info, we should add it. My assumption was that generally each die forms > a NUMA node and hence the information is available there. I may be wrong. There are architectures that have exposed multiple NUMA nodes on one die. Given steady increase in number of cores, I'd be surprised if it didn't happen again. An example would be the Haswell cluster on die. This pages suggests they were exposed as NUMA nodes via SRAT proximity domains. https://www.dell.com/support/article/en-uk/sln315049/intel-cluster-on-die-cod-technology-on-vmware-esxi?lang=en So that isn't sufficient information. If we want to expose it it needs to be explicit. > > > RFC questions: > > 1) Naming > > 2) Related to naming, do we want to represent all potential levels, > > or this enough? On Kunpeng920, the next level up from cluster happens > > to be covered by llc cache sharing, but in theory more than one > > level of cluster description might be needed by some future system. > > That is my question above. Can't recall the terminology used in ACPI > PPTT, but IIRC "cluster" is not used to keep it generic. May be we need > to do the same here as the term "cluster" is ill-defined on Arm and I > would avoid using it if possible. In example text, ACPI uses the term cluster. Of course that may reflect that the author was an ARM person (I have no idea) E.g. The Cache Type Cluster Example Fig 5.13 Various other places in description of power management use cluster to mean similar things. > > > 3) Do we need DT code in place? I'm not sure any DT based ARM64 > > systems would have enough complexity for this to be useful. > > I prefer to keep the implementation in sync Fair enough. Once we are sure this is going somewhere I'll get that implemented (mostly writing test cases *sigh*) > > > 4) Other architectures? Is this useful on x86 for example? > > > > AMD had multi-die within socket IIRC. IIUC, cpuid will provide the info > and nothing more is needed from ACPI ? I may be wrong, just guessing/asking. I'm not sure if any x86 platforms use PPTT. However, the CPUID value, if provided would still need to be hooked up to the sysfs files etc. As mentioned in another branch in this thread my guess is the equivalent is 'tile'. Thanks, Jonathan > > -- > Regards, > Sudeep From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_2 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A25CC433E7 for ; Mon, 19 Oct 2020 13:17:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 14224223FB for ; Mon, 19 Oct 2020 13:17:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="ac1wAegs" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 14224223FB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=Huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID: Subject:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=aHLTfnJB7ygG3oZYI/JGDdv24I3cRD295Swuy/kNvGE=; b=ac1wAegsiEokkN8qfMvvWTWQ4 FvTcbs09FmP/n2KaRdlYsn9EuAnTlCi0tojUPw5+RIBwT2yNe8HKwM4dyheMCyJw7edDA5/sJ/LBa nSc0q61vby0w38/x4o2FluUn8soDqJnPHEW5on/G7kVrqbR2F5IIa+HGaBuD3MHzpn61dzuHzomOa aUDUx8DZk7PWFwl7NCCNzUFFDn/447g52haTsYfN8kdXphxGNYgNQWzFjy1t3L5A0m5VoCM3BsAXu bnNhKeqwZDoH5tWI5KENyErg6OjEhODCBZG4zAXqQKqqgT11ciD5AqOpEOu5dKtU435r3v6O4gxFx rny4/dkAw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kUV1a-000887-In; Mon, 19 Oct 2020 13:16:30 +0000 Received: from lhrrgout.huawei.com ([185.176.76.210] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kUV1W-00086t-ML for linux-arm-kernel@lists.infradead.org; Mon, 19 Oct 2020 13:16:28 +0000 Received: from lhreml710-chm.china.huawei.com (unknown [172.18.7.106]) by Forcepoint Email with ESMTP id 1C9D932FDD75741903FF; Mon, 19 Oct 2020 14:16:24 +0100 (IST) Received: from localhost (10.52.126.130) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1913.5; Mon, 19 Oct 2020 14:16:23 +0100 Date: Mon, 19 Oct 2020 14:14:27 +0100 From: Jonathan Cameron To: Sudeep Holla Subject: Re: [RFC PATCH] topology: Represent clusters of CPUs within a die. Message-ID: <20201019131427.00006f40@Huawei.com> In-Reply-To: <20201019100100.GA12908@bogus> References: <20201016152702.1513592-1-Jonathan.Cameron@huawei.com> <20201019100100.GA12908@bogus> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; i686-w64-mingw32) MIME-Version: 1.0 X-Originating-IP: [10.52.126.130] X-ClientProxiedBy: lhreml711-chm.china.huawei.com (10.201.108.62) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20201019_091626_949430_0ACAB6C7 X-CRM114-Status: GOOD ( 42.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Len Brown , Greg Kroah-Hartman , x86@kernel.org, linux-kernel@vger.kernel.org, linuxarm@huawei.com, Brice Goglin , linux-acpi@vger.kernel.org, guohanjun@huawei.com, Will Deacon , Morten Rasmussen , linux-arm-kernel@lists.infradead.org Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gTW9uLCAxOSBPY3QgMjAyMCAxMTowMTowMCArMDEwMApTdWRlZXAgSG9sbGEgPHN1ZGVlcC5o b2xsYUBhcm0uY29tPiB3cm90ZToKCj4gK01vcnRlbgoKSGkgU3VkZWVwLAo+IAo+IE9uIEZyaSwg T2N0IDE2LCAyMDIwIGF0IDExOjI3OjAyUE0gKzA4MDAsIEpvbmF0aGFuIENhbWVyb24gd3JvdGU6 Cj4gPiBCb3RoIEFDUEkgYW5kIERUIHByb3ZpZGUgdGhlIGFiaWxpdHkgdG8gZGVzY3JpYmUgYWRk aXRpb25hbCBsYXllcnMgb2YKPiA+IHRvcG9sb2d5IGJldHdlZW4gdGhhdCBvZiBpbmRpdmlkdWFs IGNvcmVzIGFuZCBoaWdoZXIgbGV2ZWwgY29uc3RydWN0cwo+ID4gc3VjaCBhcyB0aGUgbGV2ZWwg YXQgd2hpY2ggdGhlIGxhc3QgbGV2ZWwgY2FjaGUgaXMgc2hhcmVkLgo+ID4gSW4gQUNQSSB0aGlz IGNhbiBiZSByZXByZXNlbnRlZCBpbiBQUFRUIGFzIGEgUHJvY2Vzc29yIEhpZXJhcmNoeQo+ID4g Tm9kZSBTdHJ1Y3R1cmUgWzFdIHRoYXQgaXMgdGhlIHBhcmVudCBvZiB0aGUgQ1BVIGNvcmVzIGFu ZCBpbiB0dXJuCj4gPiBoYXMgYSBwYXJlbnQgUHJvY2Vzc29yIEhpZXJhcmNoeSBOb2RlcyBTdHJ1 Y3R1cmUgcmVwcmVzZW50aW5nCj4gPiBhIGhpZ2hlciBsZXZlbCBvZiB0b3BvbG9neS4KPiA+Cj4g PiBGb3IgZXhhbXBsZSBLdW5wZW5nIDkyMCBoYXMgY2x1c3RlcnMgb2YgNCBDUFVzLiAgVGhlc2Ug ZG8gbm90IHNoYXJlCj4gPiBhbnkgY2FjaGUgcmVzb3VyY2VzLCBidXQgdGhlIGludGVyY29ubmVj dCB0b3BvbG9neSBpcyBzdWNoIHRoYXQKPiA+IHRoZSBjb3N0IHRvIHRyYW5zZmVyIG93bmVyc2hp cCBvZiBhIGNhY2hlbGluZSBiZXR3ZWVuIENQVXMgd2l0aGluCj4gPiBhIGNsdXN0ZXIgaXMgbG93 ZXIgdGhhbiBiZXR3ZWVuIENQVXMgaW4gZGlmZmVyZW50IGNsdXN0ZXJzIG9uIHRoZSBzYW1lCj4g PiBkaWUuICAgSGVuY2UsIGl0IGNhbiBtYWtlIHNlbnNlIHRvIGRlbGliZXJhdGVseSBzY2hlZHVs ZSB0aHJlYWRzCj4gPiBzaGFyaW5nIGRhdGEgdG8gYSBzaW5nbGUgY2x1c3Rlci4KPiA+ICAKPiAK PiBUaGlzIGlzIHZlcnkgU29DIHNwZWNpZmljIGFuZCBoYXJkIHRvIGdlbmVyYWxpc2UsIGhlbmNl IExMQyBpcyBjaG9zZW4KPiBhcyBvbmUgb2YgdGhlIG1haW4gZmFjdG9yIHRvIGRlY2lkZS4KCkkg YWJzb2x1dGVseSBhZ3JlZS4gSXQgaXMgdmVyeSBoYXJkIHRvIG1vZGlmeSB0aGUga2VybmVsIHNj aGVkdWxlciB0bwp0YWtlIGludG8gYWNjb3VudCB0aGlzIGluZm9ybWF0aW9uLiAgQ2hhbmNlcyBh cmUgd2UnbGwgaGVscCBzb21lIHdvcmtsb2FkcwphbmQgaHVydCBvdGhlcnMuICBIb3dldmVyLCBp dCBpcyBnb29kIHRvIGF0IGxlYXN0IGV4cG9zZSB0aGF0IGl0IGV4aXN0cy4KCj4gCj4gQXJlIHRo ZXJlIGFueSBzY2hlZHVsZXIgdG9wb2xvZ3kgcmVsYXRlZCBjaGFuZ2VzIHRvIGFjaGlldmUgdGhl IHNhbWUgPwo+IElmIHNvLCB3ZSBuZWVkIHRoZWlyIG9waW5pb24gb24gdGhhdC4KClNvIGZhciBu by4gIFdlIGhhdmUgZG9uZSBhIGZldyBleHBlcmltZW50cyB3aXRoIGFkZGluZyB0aGlzIGtub3ds ZWRnZQppbnRvIHRoZSBzY2hlZHVsZXIgYnV0IGl0IGlzIHN0aWxsIHZlcnkgbXVjaCBhIHdvcmsg aW4gcHJvZ3Jlc3MuCgpIb3dldmVyLCB3ZSBkbyBoYXZlIG1pY3JvIGJlbmNobWFya3Mgc2hvd2lu ZyBjbGVhcmx5IHRoYXQgY2FuIG1ha2UgYQpzdWJzdGFudGlhbCBkaWZmZXJlbmNlLiAgT3VyIGlu dGVudCB3YXMgdG8gZmlyc3QgZXhwb3NlIHRoaXMgaW5mb3JtYXRpb24gdG8KdXNlciBzcGFjZSBh bmQgZ2V0IGV4cGVyaWVuY2Ugb2YgdXNpbmcgaXQgaW4gYXBwbGljYXRpb25zIHZpYSBIV0xPQwpi ZWZvcmUgd2UgdHJpZWQgdG8gcHJvcG9zZSBhbnkgc2NoZWR1bGVyIGNoYW5nZXMuCgo+IAo+ID4g VGhpcyBwYXRjaCBzaW1wbHkgZXhwb3NlcyB0aGlzIGluZm9ybWF0aW9uIHRvIHVzZXJzcGFjZSBs aWJyYXJpZXMKPiA+IGxpa2UgaHdsb2MgYnkgcHJvdmlkaW5nIGNsdXN0ZXJfY3B1cyBhbmQgcmVs YXRlZCBzeXNmcyBhdHRyaWJ1dGVzLgo+ID4gUG9DIG9mIEhXTE9DIHN1cHBvcnQgYXQgWzJdLgo+ ID4gIAo+IAo+IE9LLCBjbHVzdGVyIGlzIHRvbyBBcm0gc3BlY2lmaWMgd2l0aCBubyBkZWZpbml0 aW9uIGZvciBpdCB3aGF0c29ldmVyLgoKSXQgZG9lcyBnZXQgdXNlZCBieSBvdGhlcnMgKEFDUEkg Zm9yIGV4YW1wbGUpLCBidXQgSSBhZ3JlZSB0aGVyZSBtYXkgYmUKYSBuYW1pbmcgaXNzdWUgaGVy ZS4gIEJyaWNlIG9ic2VydmVkIHRoYXQgaHdsb2MgdXNlcyB0aGUgdGVybSBncm91cCB0byBjb3Zl cgp0aGUgc29tZXdoYXQgb2JzY3VyZSBkcmF3ZXIgYW5kIGJvb2sgbGV2ZWxzIHRoYXQgcG93ZXJw YyB1c2VzLgpQZXJoYXBzIHRoYXQgaXMgYSBiZXR0ZXIgbmFtZT8KCj4gSG93IGRvIHlvdSBwbGFu IHRvIHN1cHBvcnQgY2x1c3RlcnMgb2YgY2x1c3RlcnMgb3IgaGlnaGVyIGxldmVsIG9mCj4gaGll cmFyY2h5IHByZXNlbnQgaW4gUFBUVC4KClJpZ2h0IG5vdyBJIGRvbid0LiAgIEkgd2FzIGhvcGlu ZyB0aGF0IHF1ZXN0aW9uIHdvdWxkIGNvbWUgdXAsCmJ1dCBkaWRuJ3Qgd2FudCB0byBjb21wbGlj YXRlIHRoZSBmaXJzdCB2ZXJzaW9uIG9mIHRoaXMgcHJvcG9zYWwuCkd1ZXNzIEkgZ290IHRoYXQg d3JvbmcgOykKCkFzIEJyaWNlIHN1Z2dlc3RlZCwgcHJvYmFibHkgYWRkaW5nIGFuIGluZGV4IG1h a2VzIHNlbnNlLgpJJ20gbm90IHN1cmUgaWYgd2UgYm90aGVyIGltcGxlbWVudGluZyBtb3JlIHRo YW4gb25lIGxheWVyIGZvciBub3csIGJ1dAp3ZSBkZWZpbml0ZWx5IHdhbnQgYSBuYW1pbmcgc2No ZW1lIHRoYXQgYWxsb3dzIGZvciBpdC4gSWYgYW55b25lIGhhcwphIENQVSB0aGF0IGFscmVhZHkg dXNlcyBhbm90aGVyIGxheWVyIGxldCB1cyBrbm93IQoKT24gYW5vdGhlciBub3RlLCB3ZSBhcmUg dGhpbmtpbmcgb2YgYW4gQUNQSSBQUFRUIGFkZGl0aW9uIHRvIGFkZCBhIGZsYWcKZm9yIGRpZSwg aWYgYW55b25lIGhhcyBhbm90aGVyIHN1Z2dlc3Rpb25zIG9uIGhvdyB0byBtYXRjaCB0aGUgeDg2 CkNQVUlEIHBhcnQgZm9yIHRoYXQsIGl0IHdvdWxkIGJlIGdyZWF0IHRvIGFsaWduLiAgQXQgbGVh c3QgZGllIGlzCmZhaXJseSB3ZWxsIGRlZmluZWQuLi4gIFRpbGUgYW5kIE1vZHVsZSBub3Qgc28g bXVjaC4uLgoKPiAKPiA+IE5vdGUgdGhpcyBwYXRjaCBvbmx5IGhhbmRsZSB0aGUgQUNQSSBjYXNl Lgo+ID4gIAo+IAo+IElmIHdlIGRlY2lkZSB0byBhZGQgdGhpcyB0byBzeXNmcywgSSBwcmVmZXIg dG8ga2VlcCBEVCBpbXBsZW1lbnRhdGlvbgo+IGFsc28gaW4gc3luYy4gVGhlIGJpbmRpbmdzIGFy ZSBpbiBzeW5jLCBqdXN0IG1hdHRlciBvZiBpbXBsZW1lbnRhdGlvbi4KCk9rLCBoYXBweSB0byBs b29rIGF0IERUIHRoYXQgaWYgd2Ugc2VlbSB0byBiZSBtb3ZpbmcgZm9yd2FyZHMgd2l0aCB0aGlz CmluIGdlbmVyYWwuCgo+IAo+ID4gU3BlY2lhbCBjb25zaWRlcmF0aW9uIGlzIG5lZWRlZCBmb3Ig U01UIHByb2Nlc3NvcnMsIHdoZXJlIGl0IGlzCj4gPiBuZWNlc3NhcnkgdG8gbW92ZSAyIGxldmVs cyB1cCB0aGUgaGllcmFyY2h5IGZyb20gdGhlIGxlYWYgbm9kZXMKPiA+ICh0aHVzIHNraXBwaW5n IHRoZSBwcm9jZXNzb3IgY29yZSBsZXZlbCkuCj4gPgo+ID4gQ3VycmVudGx5IHRoZSBJRCBwcm92 aWRlZCBpcyB0aGUgb2Zmc2V0IG9mIHRoZSBQcm9jZXNzb3IKPiA+IEhpZXJhcmNoeSBOb2RlcyBT dHJ1Y3R1cmUgd2l0aGluIFBQVFQuICBXaGlsc3QgdGhpcyBpcyB1bmlxdWUKPiA+IGl0IGlzIG5v dCB0ZXJyaWJseSBlbGVnYW50IHNvIGFsdGVybmF0aXZlIHN1Z2dlc3Rpb25zIHdlbGNvbWUuCj4g PiAgCj4gCj4gVGhhdCBpcyBmaXJtd2FyZSBjaG9pY2UuIE1heSBiZSB5b3VyIGZpcm13YXJlIGp1 c3QgZmlsbHMgaW4gbWFuZGF0b3J5Cj4gZmllbGRzIGFuZCBkb2Vzbid0IGNhcmUgYWJvdXQgYW55 dGhpbmcgYW5kIGV2ZXJ5dGhpbmcgb3B0aW9uYWwuIFdlIGRvCj4gY2hlY2sgZm9yIHZhbGlkIFVJ RCBmaWVsZHMgYW5kIGlmIHRoYXQgaXMgbm90IHNldCwgb2Zmc2V0IGlzIHVzZWQgYXMKPiB1bmlx dWUgSUQgaW4ga2VybmVsIGltcGxlbWVudGF0aW9uLiBTbyBpZiB5b3UgZW5oYW5jZSB0aGUgZmly bXdhcmUsIHRoZQo+IGtlcm5lbCBzeXNmcyB3aWxsIGJlY29tZSBlbGVnYW50IGFzIHlvdSBleHBl Y3Qg8J+YiSAKCkZhaXIgZW5vdWdoIHRoYXQgdGhlcmUgaXMgYSB3YXkgdG8gaW1wcm92ZSBpdCwg YnV0IGl0IHdvdWxkIGJlIG5pY2UgaWYKdGhlIGRlZmF1bHQgc29sdXRpb24gd2FzIGFsc28gZWxl Z2FudCEgIEknbGwgYWRkIHRoZSBwcm9jZXNzb3IgY29udGFpbmVyCkRTRFQgc3R1ZmYgdG8gb25l IG9yIHR3byBvZiBteSB0ZXN0IGNhc2VzIGFuZCBtYWtlIHN1cmUgd2UgY2FuIGdldAp0aGF0IHdv cmtpbmcuICBUaGVyZSBhcmUgYSBsb3Qgb2YgY29tYmluYXRpb25zIHRoYXQgbmVlZCB0ZXN0aW5n IGFuZApJIGdldCBib3JlZCBxdWlja2x5IHdyaXRpbmcgZmlybXdhcmUgdGFibGVzIGJ5IGhhbmQu Cgo+IAo+ID4gTm90ZSB0aGF0IGFybTY0IC8gQUNQSSBkb2VzIG5vdCBwcm92aWRlIGFueSBtZWFu cyBvZiBpZGVudGlmeWluZwo+ID4gYSBkaWUgbGV2ZWwgaW4gdGhlIHRvcG9sb2d5IGJ1dCB0aGF0 IG1heSBiZSB1bnJlbGF0ZSB0byB0aGUgY2x1c3Rlcgo+ID4gbGV2ZWwuCj4gPiAgCj4gCj4gSW5k ZWVkLiBJdCBjYW4gYmUgY2x1c3RlciBvZiBjbHVzdGVycyBvbiBzb21lIHBsYXRmb3JtLiBJZiB3 ZSBuZWVkIHRoYXQKPiBpbmZvLCB3ZSBzaG91bGQgYWRkIGl0LiBNeSBhc3N1bXB0aW9uIHdhcyB0 aGF0IGdlbmVyYWxseSBlYWNoIGRpZSBmb3Jtcwo+IGEgTlVNQSBub2RlIGFuZCBoZW5jZSB0aGUg aW5mb3JtYXRpb24gaXMgYXZhaWxhYmxlIHRoZXJlLiBJIG1heSBiZSB3cm9uZy4KClRoZXJlIGFy ZSBhcmNoaXRlY3R1cmVzIHRoYXQgaGF2ZSBleHBvc2VkIG11bHRpcGxlIE5VTUEgbm9kZXMgb24g b25lIGRpZS4KR2l2ZW4gc3RlYWR5IGluY3JlYXNlIGluIG51bWJlciBvZiBjb3JlcywgSSdkIGJl IHN1cnByaXNlZCBpZiBpdCBkaWRuJ3QgaGFwcGVuCmFnYWluLgoKQW4gZXhhbXBsZSB3b3VsZCBi ZSB0aGUgSGFzd2VsbCBjbHVzdGVyIG9uIGRpZS4gVGhpcyBwYWdlcyBzdWdnZXN0cyB0aGV5Cndl cmUgZXhwb3NlZCBhcyBOVU1BIG5vZGVzIHZpYSBTUkFUIHByb3hpbWl0eSBkb21haW5zLgoKaHR0 cHM6Ly93d3cuZGVsbC5jb20vc3VwcG9ydC9hcnRpY2xlL2VuLXVrL3NsbjMxNTA0OS9pbnRlbC1j bHVzdGVyLW9uLWRpZS1jb2QtdGVjaG5vbG9neS1vbi12bXdhcmUtZXN4aT9sYW5nPWVuCgpTbyB0 aGF0IGlzbid0IHN1ZmZpY2llbnQgaW5mb3JtYXRpb24uICBJZiB3ZSB3YW50IHRvIGV4cG9zZSBp dCBpdCBuZWVkcwp0byBiZSBleHBsaWNpdC4KCj4gCj4gPiBSRkMgcXVlc3Rpb25zOgo+ID4gMSkg TmFtaW5nCj4gPiAyKSBSZWxhdGVkIHRvIG5hbWluZywgZG8gd2Ugd2FudCB0byByZXByZXNlbnQg YWxsIHBvdGVudGlhbCBsZXZlbHMsCj4gPiAgICBvciB0aGlzIGVub3VnaD8gIE9uIEt1bnBlbmc5 MjAsIHRoZSBuZXh0IGxldmVsIHVwIGZyb20gY2x1c3RlciBoYXBwZW5zCj4gPiAgICB0byBiZSBj b3ZlcmVkIGJ5IGxsYyBjYWNoZSBzaGFyaW5nLCBidXQgaW4gdGhlb3J5IG1vcmUgdGhhbiBvbmUK PiA+ICAgIGxldmVsIG9mIGNsdXN0ZXIgZGVzY3JpcHRpb24gbWlnaHQgYmUgbmVlZGVkIGJ5IHNv bWUgZnV0dXJlIHN5c3RlbS4gIAo+IAo+IFRoYXQgaXMgbXkgcXVlc3Rpb24gYWJvdmUuIENhbid0 IHJlY2FsbCB0aGUgdGVybWlub2xvZ3kgdXNlZCBpbiBBQ1BJCj4gUFBUVCwgYnV0IElJUkMgImNs dXN0ZXIiIGlzIG5vdCB1c2VkIHRvIGtlZXAgaXQgZ2VuZXJpYy4gTWF5IGJlIHdlIG5lZWQKPiB0 byBkbyB0aGUgc2FtZSBoZXJlIGFzIHRoZSB0ZXJtICJjbHVzdGVyIiBpcyBpbGwtZGVmaW5lZCBv biBBcm0gYW5kIEkKPiB3b3VsZCBhdm9pZCB1c2luZyBpdCBpZiBwb3NzaWJsZS4KCkluIGV4YW1w bGUgdGV4dCwgQUNQSSB1c2VzIHRoZSB0ZXJtIGNsdXN0ZXIuIE9mIGNvdXJzZSB0aGF0IG1heQpy ZWZsZWN0IHRoYXQgdGhlIGF1dGhvciB3YXMgYW4gQVJNIHBlcnNvbiAoSSBoYXZlIG5vIGlkZWEp CkUuZy4gVGhlIENhY2hlIFR5cGUgQ2x1c3RlciBFeGFtcGxlIEZpZyA1LjEzICBWYXJpb3VzIG90 aGVyIHBsYWNlcyBpbgpkZXNjcmlwdGlvbiBvZiBwb3dlciBtYW5hZ2VtZW50IHVzZSBjbHVzdGVy IHRvIG1lYW4gc2ltaWxhciB0aGluZ3MuCgo+IAo+ID4gMykgRG8gd2UgbmVlZCBEVCBjb2RlIGlu IHBsYWNlPyBJJ20gbm90IHN1cmUgYW55IERUIGJhc2VkIEFSTTY0Cj4gPiAgICBzeXN0ZW1zIHdv dWxkIGhhdmUgZW5vdWdoIGNvbXBsZXhpdHkgZm9yIHRoaXMgdG8gYmUgdXNlZnVsLiAgCj4gCj4g SSBwcmVmZXIgdG8ga2VlcCB0aGUgaW1wbGVtZW50YXRpb24gaW4gc3luYwoKRmFpciBlbm91Z2gu ICBPbmNlIHdlIGFyZSBzdXJlIHRoaXMgaXMgZ29pbmcgc29tZXdoZXJlIEknbGwgZ2V0IHRoYXQK aW1wbGVtZW50ZWQgKG1vc3RseSB3cml0aW5nIHRlc3QgY2FzZXMgKnNpZ2gqKQoKPiAKPiA+IDQp IE90aGVyIGFyY2hpdGVjdHVyZXM/ICBJcyB0aGlzIHVzZWZ1bCBvbiB4ODYgZm9yIGV4YW1wbGU/ Cj4gPiAgCj4gCj4gQU1EIGhhZCBtdWx0aS1kaWUgd2l0aGluIHNvY2tldCBJSVJDLiBJSVVDLCBj cHVpZCB3aWxsIHByb3ZpZGUgdGhlIGluZm8KPiBhbmQgbm90aGluZyBtb3JlIGlzIG5lZWRlZCBm cm9tIEFDUEkgPyBJIG1heSBiZSB3cm9uZywganVzdCBndWVzc2luZy9hc2tpbmcuCgpJJ20gbm90 IHN1cmUgaWYgYW55IHg4NiBwbGF0Zm9ybXMgdXNlIFBQVFQuICAgSG93ZXZlciwgdGhlIENQVUlE IHZhbHVlLAppZiBwcm92aWRlZCB3b3VsZCBzdGlsbCBuZWVkIHRvIGJlIGhvb2tlZCB1cCB0byB0 aGUgc3lzZnMgZmlsZXMKZXRjLiAgQXMgbWVudGlvbmVkIGluIGFub3RoZXIgYnJhbmNoIGluIHRo aXMgdGhyZWFkIG15IGd1ZXNzIGlzIHRoZQplcXVpdmFsZW50IGlzICd0aWxlJy4KClRoYW5rcywK CkpvbmF0aGFuCgo+IAo+IC0tCj4gUmVnYXJkcywKPiBTdWRlZXAKCgoKX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5n IGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5p bmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=