From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C1A5EA3 for ; Thu, 6 Apr 2023 08:42:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80BF2C433EF; Thu, 6 Apr 2023 08:42:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680770547; bh=i3Ygol4vLeOR82TBK9DpW4bTLHkspuPPD6gKyABYHMI=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=i5+gAyOI7VCazBa5+/A5mCmEIM4o3q3BpZtKQaaY0H6gzleKFszck/GDN5kMVk1gL AjPmGg8DTt/RJ1pd031YBTz+AlshMAPFf5cCiSJVfNw41eLFseMutPsVF0EhbHGVWk UegmFQ+3591c38spZk/PVV8j1uI9viGaUDyU5YuQr459vXSKZWdPMT708dEkzjsd77 psxxNtfE4TM+UpKND0WlNovrI7SIalxPc3xj6oxd6miAVba2ecpFOM2i5GBnnTUXSN 8mLNKmInlBgwidENNWhlJENR/FmmBBbAdA5qZ9gCWPEXoP/CxpWwzebvQzDtLnv1R2 dTSMkkXDHf6xg== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pkLCL-006FGs-2X; Thu, 06 Apr 2023 09:42:25 +0100 Date: Thu, 06 Apr 2023 09:42:24 +0100 Message-ID: <86o7o1v1u7.wl-maz@kernel.org> From: Marc Zyngier To: Saravana Kannan Cc: David Dai , Oliver Upton , "Rafael J. Wysocki" , Viresh Kumar , Rob Herring , Krzysztof Kozlowski , Paolo Bonzini , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Mark Rutland , Lorenzo Pieralisi , Sudeep Holla , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , kernel-team@android.com, linux-pm@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev Subject: Re: [RFC PATCH 0/6] Improve VM DVFS and task placement behavior In-Reply-To: References: <20230330224348.1006691-1-davidai@google.com> <86sfdfv0e1.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: saravanak@google.com, davidai@google.com, oliver.upton@linux.dev, rafael@kernel.org, viresh.kumar@linaro.org, robh+dt@kernel.org, krzysztof.kozlowski+dt@linaro.org, pbonzini@redhat.com, corbet@lwn.net, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, lpieralisi@kernel.org, sudeep.holla@arm.com, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, kernel-team@android.com, linux-pm@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Wed, 05 Apr 2023 22:00:59 +0100, Saravana Kannan wrote: >=20 > On Tue, Apr 4, 2023 at 1:49=E2=80=AFPM Marc Zyngier wrot= e: > > > > On Tue, 04 Apr 2023 20:43:40 +0100, > > Oliver Upton wrote: > > > > > > Folks, > > > > > > On Thu, Mar 30, 2023 at 03:43:35PM -0700, David Dai wrote: > > > > > > > > > > > > > PCMark > > > > Higher is better > > > > +-------------------+----------+------------+--------+-------+-----= ---+ > > > > | Test Case (score) | Baseline | Hypercall | %delta | MMIO | %del= ta | > > > > +-------------------+----------+------------+--------+-------+-----= ---+ > > > > | Weighted Total | 6136 | 7274 | +19% | 6867 | +1= 2% | > > > > +-------------------+----------+------------+--------+-------+-----= ---+ > > > > | Web Browsing | 5558 | 6273 | +13% | 6035 | += 9% | > > > > +-------------------+----------+------------+--------+-------+-----= ---+ > > > > | Video Editing | 4921 | 5221 | +6% | 5167 | += 5% | > > > > +-------------------+----------+------------+--------+-------+-----= ---+ > > > > | Writing | 6864 | 8825 | +29% | 8529 | +2= 4% | > > > > +-------------------+----------+------------+--------+-------+-----= ---+ > > > > | Photo Editing | 7983 | 11593 | +45% | 10812 | +3= 5% | > > > > +-------------------+----------+------------+--------+-------+-----= ---+ > > > > | Data Manipulation | 5814 | 6081 | +5% | 5327 | -= 8% | > > > > +-------------------+----------+------------+--------+-------+-----= ---+ > > > > > > > > PCMark Performance/mAh > > > > Higher is better > > > > +-----------+----------+-----------+--------+------+--------+ > > > > | | Baseline | Hypercall | %delta | MMIO | %delta | > > > > +-----------+----------+-----------+--------+------+--------+ > > > > | Score/mAh | 79 | 88 | +11% | 83 | +7% | > > > > +-----------+----------+-----------+--------+------+--------+ > > > > > > > > Roblox > > > > Higher is better > > > > +-----+----------+------------+--------+-------+--------+ > > > > | | Baseline | Hypercall | %delta | MMIO | %delta | > > > > +-----+----------+------------+--------+-------+--------+ > > > > | FPS | 18.25 | 28.66 | +57% | 24.06 | +32% | > > > > +-----+----------+------------+--------+-------+--------+ > > > > > > > > Roblox Frames/mAh > > > > Higher is better > > > > +------------+----------+------------+--------+--------+--------+ > > > > | | Baseline | Hypercall | %delta | MMIO | %delta | > > > > +------------+----------+------------+--------+--------+--------+ > > > > | Frames/mAh | 91.25 | 114.64 | +26% | 103.11 | +13% | > > > > +------------+----------+------------+--------+--------+--------+ > > > > > > > > > > > > > Next steps: > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > We are continuing to look into communication mechanisms other than > > > > hypercalls that are just as/more efficient and avoid switching into= the VMM > > > > userspace. Any inputs in this regard are greatly appreciated. >=20 > Hi Oliver and Marc, >=20 > Replying to both of you in this one email. >=20 > > > > > > We're highly unlikely to entertain such an interface in KVM. > > > > > > The entire feature is dependent on pinning vCPUs to physical cores, f= or which > > > userspace is in the driver's seat. That is a well established and doc= umented > > > policy which can be seen in the way we handle heterogeneous systems a= nd > > > vPMU. > > > > > > Additionally, this bloats the KVM PV ABI with highly VMM-dependent in= terfaces > > > that I would not expect to benefit the typical user of KVM. > > > > > > Based on the data above, it would appear that the userspace implement= ation is > > > in the same neighborhood as a KVM-based implementation, which only fu= rther > > > weakens the case for moving this into the kernel. >=20 > Oliver, >=20 > Sorry if the tables/data aren't presented in an intuitive way, but > MMIO vs hypercall is definitely not in the same neighborhood. The > hypercall method often gives close to 2x the improvement that the MMIO > method gives. For example: >=20 > - Roblox FPS: MMIO improves it by 32% vs hypercall improves it by 57%. > - Frames/mAh: MMIO improves it by 13% vs hypercall improves it by 26%. > - PC Mark Data manipulation: MMIO makes it worse by 8% vs hypercall > improves it by 5% >=20 > Hypercall does better for other cases too, just not as good. For example, > - PC Mark Photo editing: Going from MMIO to hypercall gives a 10% improve= ment. >=20 > These are all pretty non-trivial, at least in the mobile world. Heck, > whole teams would spend months for 2% improvement in battery :) > > > > > > > I certainly can appreciate the motivation for the series, but this fe= ature > > > should be in userspace as some form of a virtual device. > > > > +1 on all of the above. >=20 > Marc and Oliver, >=20 > We are not tied to hypercalls. We want to do the right thing here, but > MMIO going all the way to userspace definitely doesn't cut it as is. > This is where we need some guidance. See more below. I don't buy this assertion at all. An MMIO in userspace is already much better than nothing. One of my many objection to the whole series is that it is built as a massively invasive thing that has too many fingers in too many pies, with unsustainable assumptions such as 1:1 mapping between CPU and vCPUs. I'd rather you build something simple first (pure userspace using MMIOs), work out where the bottlenecks are, and work with us to add what is needed to get to something sensible, and only that. I'm not willing to sacrifice maintainability for maximum performance (the whole thing reminds me of the in-kernel http server...). >=20 > > The one thing I'd like to understand that the comment seems to imply > > that there is a significant difference in overhead between a hypercall > > and an MMIO. In my experience, both are pretty similar in cost for a > > handling location (both in userspace or both in the kernel). >=20 > I think the main difference really is that in our hypercall vs MMIO > comparison the hypercall is handled in the kernel vs MMIO goes all the > way to userspace. I agree with you that the difference probably won't > be significant if both of them go to the same "depth" in the privilege > levels. >=20 > > MMIO > > handling is a tiny bit more expensive due to a guaranteed TLB miss > > followed by a walk of the in-kernel device ranges, but that's all. It > > should hardly register. > > > > And if you really want some super-low latency, low overhead > > signalling, maybe an exception is the wrong tool for the job. Shared > > memory communication could be more appropriate. >=20 > Yeah, that's one of our next steps. Ideally, we want to use shared > memory for the host to guest information flow. It's a 32-bit value > representing the current frequency that the host can update whenever > the host CPU frequency changes and the guest can read whenever it > needs it. Why should the guest care? Why can't the guest ask for an arbitrary capacity, and get what it gets? You give no information as to *why* you are doing what you are doing... >=20 > For guest to host information flow, we'll need a kick from guest to > host because we need to take action on the host side when threads > migrate between vCPUs and cause a significant change in vCPU util. > Again it can be just a shared memory and some kick. This is what we > are currently trying to figure out how to do. That kick would have to go to userspace. There is no way I'm willing to introduce scheduling primitives inside KVM (the ones we have are ridiculously bad anyway), and I very much want to avoid extra PV gunk. > If there are APIs to do this, can you point us to those please? We'd > also want the shared memory to be accessible by the VMM (so, shared > between guest kernel, host kernel and VMM). By default, *ALL* the memory is shared. Isn't that wonderful? >=20 > Are the above next steps sane? Or is that a no-go? The main thing we > want to cut out is the need for having to switch to userspace for > every single interaction because, as is, it leaves a lot on the table. Well, for a start, you could disclose how often you hit this DVFS "device", and when are the critical state changes that must happen immediately vs those that can simply be posted without having to take immediate effect. This sort of information would be much more interesting than a bunch of benchmarks I know nothing about. Thanks, M. --=20 Without deviation from the norm, progress is not possible. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 39E32C76196 for ; Thu, 6 Apr 2023 08:43:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TIW4xtKOtvI6W8qftvug7Repc/EI+bTCkk3Mb77VU74=; b=aizE3XE5AEJay2 wOkhGV+ujSaqP86FxVzqKbA0b3//Q69qIMuP6zUdpHD+S7i1rV7Sw9pM2Raxx4s8PonKjiQJM/MhW VXnb+i/NzDSo96iOyLuBi4D0wIayiEO3/IXtd0JTutvikFZsARYEjpAGjxrjn+DxfLyG+hMVqpkUY UvL/JzFPgRVc5updNjxV3TMIsfIGnH3QnGnNJiRlfkBsQu6TBFV40NOkgaxu1zCbUxAWeMI9yE6kB kDgliFXAglpbNZbDSdeN/Dlfo9Yc2/3ApM2l7zhj7RfYp/O35eSuzF28oGdaqlyxIXrQW1/h53fgV 2UuVMN9PZYsPUtVcV/Vg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pkLCR-006n1L-2g; Thu, 06 Apr 2023 08:42:31 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pkLCP-006n0B-0a for linux-arm-kernel@lists.infradead.org; Thu, 06 Apr 2023 08:42:31 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 793B161DDF; Thu, 6 Apr 2023 08:42:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 80BF2C433EF; Thu, 6 Apr 2023 08:42:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1680770547; bh=i3Ygol4vLeOR82TBK9DpW4bTLHkspuPPD6gKyABYHMI=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=i5+gAyOI7VCazBa5+/A5mCmEIM4o3q3BpZtKQaaY0H6gzleKFszck/GDN5kMVk1gL AjPmGg8DTt/RJ1pd031YBTz+AlshMAPFf5cCiSJVfNw41eLFseMutPsVF0EhbHGVWk UegmFQ+3591c38spZk/PVV8j1uI9viGaUDyU5YuQr459vXSKZWdPMT708dEkzjsd77 psxxNtfE4TM+UpKND0WlNovrI7SIalxPc3xj6oxd6miAVba2ecpFOM2i5GBnnTUXSN 8mLNKmInlBgwidENNWhlJENR/FmmBBbAdA5qZ9gCWPEXoP/CxpWwzebvQzDtLnv1R2 dTSMkkXDHf6xg== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pkLCL-006FGs-2X; Thu, 06 Apr 2023 09:42:25 +0100 Date: Thu, 06 Apr 2023 09:42:24 +0100 Message-ID: <86o7o1v1u7.wl-maz@kernel.org> From: Marc Zyngier To: Saravana Kannan Cc: David Dai , Oliver Upton , "Rafael J. Wysocki" , Viresh Kumar , Rob Herring , Krzysztof Kozlowski , Paolo Bonzini , Jonathan Corbet , James Morse , Suzuki K Poulose , Zenghui Yu , Catalin Marinas , Will Deacon , Mark Rutland , Lorenzo Pieralisi , Sudeep Holla , Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , kernel-team@android.com, linux-pm@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev Subject: Re: [RFC PATCH 0/6] Improve VM DVFS and task placement behavior In-Reply-To: References: <20230330224348.1006691-1-davidai@google.com> <86sfdfv0e1.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: saravanak@google.com, davidai@google.com, oliver.upton@linux.dev, rafael@kernel.org, viresh.kumar@linaro.org, robh+dt@kernel.org, krzysztof.kozlowski+dt@linaro.org, pbonzini@redhat.com, corbet@lwn.net, james.morse@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, catalin.marinas@arm.com, will@kernel.org, mark.rutland@arm.com, lpieralisi@kernel.org, sudeep.holla@arm.com, mingo@redhat.com, peterz@infradead.org, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, kernel-team@android.com, linux-pm@vger.kernel.org, devicetree@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230406_014229_301204_8E084E67 X-CRM114-Status: GOOD ( 57.30 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCAwNSBBcHIgMjAyMyAyMjowMDo1OSArMDEwMCwKU2FyYXZhbmEgS2FubmFuIDxzYXJh dmFuYWtAZ29vZ2xlLmNvbT4gd3JvdGU6Cj4gCj4gT24gVHVlLCBBcHIgNCwgMjAyMyBhdCAxOjQ5 4oCvUE0gTWFyYyBaeW5naWVyIDxtYXpAa2VybmVsLm9yZz4gd3JvdGU6Cj4gPgo+ID4gT24gVHVl LCAwNCBBcHIgMjAyMyAyMDo0Mzo0MCArMDEwMCwKPiA+IE9saXZlciBVcHRvbiA8b2xpdmVyLnVw dG9uQGxpbnV4LmRldj4gd3JvdGU6Cj4gPiA+Cj4gPiA+IEZvbGtzLAo+ID4gPgo+ID4gPiBPbiBU aHUsIE1hciAzMCwgMjAyMyBhdCAwMzo0MzozNVBNIC0wNzAwLCBEYXZpZCBEYWkgd3JvdGU6Cj4g PiA+Cj4gPiA+IDxzbmlwPgo+ID4gPgo+ID4gPiA+IFBDTWFyawo+ID4gPiA+IEhpZ2hlciBpcyBi ZXR0ZXIKPiA+ID4gPiArLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLS0tLS0t LSstLS0tLS0tLSstLS0tLS0tKy0tLS0tLS0tKwo+ID4gPiA+IHwgVGVzdCBDYXNlIChzY29yZSkg fCBCYXNlbGluZSB8ICBIeXBlcmNhbGwgfCAlZGVsdGEgfCAgTU1JTyB8ICVkZWx0YSB8Cj4gPiA+ ID4gKy0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0rLS0tLS0tLS0r LS0tLS0tLSstLS0tLS0tLSsKPiA+ID4gPiB8IFdlaWdodGVkIFRvdGFsICAgIHwgICAgIDYxMzYg fCAgICAgICA3Mjc0IHwgICArMTklIHwgIDY4NjcgfCAgICsxMiUgfAo+ID4gPiA+ICstLS0tLS0t LS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tKy0tLS0tLS0tKy0tLS0tLS0rLS0t LS0tLS0rCj4gPiA+ID4gfCBXZWIgQnJvd3NpbmcgICAgICB8ICAgICA1NTU4IHwgICAgICAgNjI3 MyB8ICAgKzEzJSB8ICA2MDM1IHwgICAgKzklIHwKPiA+ID4gPiArLS0tLS0tLS0tLS0tLS0tLS0t LSstLS0tLS0tLS0tKy0tLS0tLS0tLS0tLSstLS0tLS0tLSstLS0tLS0tKy0tLS0tLS0tKwo+ID4g PiA+IHwgVmlkZW8gRWRpdGluZyAgICAgfCAgICAgNDkyMSB8ICAgICAgIDUyMjEgfCAgICArNiUg fCAgNTE2NyB8ICAgICs1JSB8Cj4gPiA+ID4gKy0tLS0tLS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0rLS0tLS0tLS0rLS0tLS0tLSstLS0tLS0tLSsKPiA+ID4gPiB8IFdyaXRp bmcgICAgICAgICAgIHwgICAgIDY4NjQgfCAgICAgICA4ODI1IHwgICArMjklIHwgIDg1MjkgfCAg ICsyNCUgfAo+ID4gPiA+ICstLS0tLS0tLS0tLS0tLS0tLS0tKy0tLS0tLS0tLS0rLS0tLS0tLS0t LS0tKy0tLS0tLS0tKy0tLS0tLS0rLS0tLS0tLS0rCj4gPiA+ID4gfCBQaG90byBFZGl0aW5nICAg ICB8ICAgICA3OTgzIHwgICAgICAxMTU5MyB8ICAgKzQ1JSB8IDEwODEyIHwgICArMzUlIHwKPiA+ ID4gPiArLS0tLS0tLS0tLS0tLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLS0tLS0tLSstLS0tLS0t LSstLS0tLS0tKy0tLS0tLS0tKwo+ID4gPiA+IHwgRGF0YSBNYW5pcHVsYXRpb24gfCAgICAgNTgx NCB8ICAgICAgIDYwODEgfCAgICArNSUgfCAgNTMyNyB8ICAgIC04JSB8Cj4gPiA+ID4gKy0tLS0t LS0tLS0tLS0tLS0tLS0rLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0rLS0tLS0tLS0rLS0tLS0tLSst LS0tLS0tLSsKPiA+ID4gPgo+ID4gPiA+IFBDTWFyayBQZXJmb3JtYW5jZS9tQWgKPiA+ID4gPiBI aWdoZXIgaXMgYmV0dGVyCj4gPiA+ID4gKy0tLS0tLS0tLS0tKy0tLS0tLS0tLS0rLS0tLS0tLS0t LS0rLS0tLS0tLS0rLS0tLS0tKy0tLS0tLS0tKwo+ID4gPiA+IHwgICAgICAgICAgIHwgQmFzZWxp bmUgfCBIeXBlcmNhbGwgfCAlZGVsdGEgfCBNTUlPIHwgJWRlbHRhIHwKPiA+ID4gPiArLS0tLS0t LS0tLS0rLS0tLS0tLS0tLSstLS0tLS0tLS0tLSstLS0tLS0tLSstLS0tLS0rLS0tLS0tLS0rCj4g PiA+ID4gfCBTY29yZS9tQWggfCAgICAgICA3OSB8ICAgICAgICA4OCB8ICAgKzExJSB8ICAgODMg fCAgICArNyUgfAo+ID4gPiA+ICstLS0tLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLS0tLS0tKy0t LS0tLS0tKy0tLS0tLSstLS0tLS0tLSsKPiA+ID4gPgo+ID4gPiA+IFJvYmxveAo+ID4gPiA+IEhp Z2hlciBpcyBiZXR0ZXIKPiA+ID4gPiArLS0tLS0rLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0rLS0t LS0tLS0rLS0tLS0tLSstLS0tLS0tLSsKPiA+ID4gPiB8ICAgICB8IEJhc2VsaW5lIHwgIEh5cGVy Y2FsbCB8ICVkZWx0YSB8ICBNTUlPIHwgJWRlbHRhIHwKPiA+ID4gPiArLS0tLS0rLS0tLS0tLS0t LSstLS0tLS0tLS0tLS0rLS0tLS0tLS0rLS0tLS0tLSstLS0tLS0tLSsKPiA+ID4gPiB8IEZQUyB8 ICAgIDE4LjI1IHwgICAgICAyOC42NiB8ICAgKzU3JSB8IDI0LjA2IHwgICArMzIlIHwKPiA+ID4g PiArLS0tLS0rLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0rLS0tLS0tLS0rLS0tLS0tLSstLS0tLS0t LSsKPiA+ID4gPgo+ID4gPiA+IFJvYmxveCBGcmFtZXMvbUFoCj4gPiA+ID4gSGlnaGVyIGlzIGJl dHRlcgo+ID4gPiA+ICstLS0tLS0tLS0tLS0rLS0tLS0tLS0tLSstLS0tLS0tLS0tLS0rLS0tLS0t LS0rLS0tLS0tLS0rLS0tLS0tLS0rCj4gPiA+ID4gfCAgICAgICAgICAgIHwgQmFzZWxpbmUgfCAg SHlwZXJjYWxsIHwgJWRlbHRhIHwgICBNTUlPIHwgJWRlbHRhIHwKPiA+ID4gPiArLS0tLS0tLS0t LS0tKy0tLS0tLS0tLS0rLS0tLS0tLS0tLS0tKy0tLS0tLS0tKy0tLS0tLS0tKy0tLS0tLS0tKwo+ ID4gPiA+IHwgRnJhbWVzL21BaCB8ICAgIDkxLjI1IHwgICAgIDExNC42NCB8ICAgKzI2JSB8IDEw My4xMSB8ICAgKzEzJSB8Cj4gPiA+ID4gKy0tLS0tLS0tLS0tLSstLS0tLS0tLS0tKy0tLS0tLS0t LS0tLSstLS0tLS0tLSstLS0tLS0tLSstLS0tLS0tLSsKPiA+ID4KPiA+ID4gPC9zbmlwPgo+ID4g Pgo+ID4gPiA+IE5leHQgc3RlcHM6Cj4gPiA+ID4gPT09PT09PT09PT0KPiA+ID4gPiBXZSBhcmUg Y29udGludWluZyB0byBsb29rIGludG8gY29tbXVuaWNhdGlvbiBtZWNoYW5pc21zIG90aGVyIHRo YW4KPiA+ID4gPiBoeXBlcmNhbGxzIHRoYXQgYXJlIGp1c3QgYXMvbW9yZSBlZmZpY2llbnQgYW5k IGF2b2lkIHN3aXRjaGluZyBpbnRvIHRoZSBWTU0KPiA+ID4gPiB1c2Vyc3BhY2UuIEFueSBpbnB1 dHMgaW4gdGhpcyByZWdhcmQgYXJlIGdyZWF0bHkgYXBwcmVjaWF0ZWQuCj4gCj4gSGkgT2xpdmVy IGFuZCBNYXJjLAo+IAo+IFJlcGx5aW5nIHRvIGJvdGggb2YgeW91IGluIHRoaXMgb25lIGVtYWls Lgo+IAo+ID4gPgo+ID4gPiBXZSdyZSBoaWdobHkgdW5saWtlbHkgdG8gZW50ZXJ0YWluIHN1Y2gg YW4gaW50ZXJmYWNlIGluIEtWTS4KPiA+ID4KPiA+ID4gVGhlIGVudGlyZSBmZWF0dXJlIGlzIGRl cGVuZGVudCBvbiBwaW5uaW5nIHZDUFVzIHRvIHBoeXNpY2FsIGNvcmVzLCBmb3Igd2hpY2gKPiA+ ID4gdXNlcnNwYWNlIGlzIGluIHRoZSBkcml2ZXIncyBzZWF0LiBUaGF0IGlzIGEgd2VsbCBlc3Rh Ymxpc2hlZCBhbmQgZG9jdW1lbnRlZAo+ID4gPiBwb2xpY3kgd2hpY2ggY2FuIGJlIHNlZW4gaW4g dGhlIHdheSB3ZSBoYW5kbGUgaGV0ZXJvZ2VuZW91cyBzeXN0ZW1zIGFuZAo+ID4gPiB2UE1VLgo+ ID4gPgo+ID4gPiBBZGRpdGlvbmFsbHksIHRoaXMgYmxvYXRzIHRoZSBLVk0gUFYgQUJJIHdpdGgg aGlnaGx5IFZNTS1kZXBlbmRlbnQgaW50ZXJmYWNlcwo+ID4gPiB0aGF0IEkgd291bGQgbm90IGV4 cGVjdCB0byBiZW5lZml0IHRoZSB0eXBpY2FsIHVzZXIgb2YgS1ZNLgo+ID4gPgo+ID4gPiBCYXNl ZCBvbiB0aGUgZGF0YSBhYm92ZSwgaXQgd291bGQgYXBwZWFyIHRoYXQgdGhlIHVzZXJzcGFjZSBp bXBsZW1lbnRhdGlvbiBpcwo+ID4gPiBpbiB0aGUgc2FtZSBuZWlnaGJvcmhvb2QgYXMgYSBLVk0t YmFzZWQgaW1wbGVtZW50YXRpb24sIHdoaWNoIG9ubHkgZnVydGhlcgo+ID4gPiB3ZWFrZW5zIHRo ZSBjYXNlIGZvciBtb3ZpbmcgdGhpcyBpbnRvIHRoZSBrZXJuZWwuCj4gCj4gT2xpdmVyLAo+IAo+ IFNvcnJ5IGlmIHRoZSB0YWJsZXMvZGF0YSBhcmVuJ3QgcHJlc2VudGVkIGluIGFuIGludHVpdGl2 ZSB3YXksIGJ1dAo+IE1NSU8gdnMgaHlwZXJjYWxsIGlzIGRlZmluaXRlbHkgbm90IGluIHRoZSBz YW1lIG5laWdoYm9yaG9vZC4gVGhlCj4gaHlwZXJjYWxsIG1ldGhvZCBvZnRlbiBnaXZlcyBjbG9z ZSB0byAyeCB0aGUgaW1wcm92ZW1lbnQgdGhhdCB0aGUgTU1JTwo+IG1ldGhvZCBnaXZlcy4gRm9y IGV4YW1wbGU6Cj4gCj4gLSBSb2Jsb3ggRlBTOiBNTUlPIGltcHJvdmVzIGl0IGJ5IDMyJSB2cyBo eXBlcmNhbGwgaW1wcm92ZXMgaXQgYnkgNTclLgo+IC0gRnJhbWVzL21BaDogTU1JTyBpbXByb3Zl cyBpdCBieSAxMyUgdnMgaHlwZXJjYWxsIGltcHJvdmVzIGl0IGJ5IDI2JS4KPiAtIFBDIE1hcmsg RGF0YSBtYW5pcHVsYXRpb246IE1NSU8gbWFrZXMgaXQgd29yc2UgYnkgOCUgdnMgaHlwZXJjYWxs Cj4gaW1wcm92ZXMgaXQgYnkgNSUKPiAKPiBIeXBlcmNhbGwgZG9lcyBiZXR0ZXIgZm9yIG90aGVy IGNhc2VzIHRvbywganVzdCBub3QgYXMgZ29vZC4gRm9yIGV4YW1wbGUsCj4gLSBQQyBNYXJrIFBo b3RvIGVkaXRpbmc6IEdvaW5nIGZyb20gTU1JTyB0byBoeXBlcmNhbGwgZ2l2ZXMgYSAxMCUgaW1w cm92ZW1lbnQuCj4gCj4gVGhlc2UgYXJlIGFsbCBwcmV0dHkgbm9uLXRyaXZpYWwsIGF0IGxlYXN0 IGluIHRoZSBtb2JpbGUgd29ybGQuIEhlY2ssCj4gd2hvbGUgdGVhbXMgd291bGQgc3BlbmQgbW9u dGhzIGZvciAyJSBpbXByb3ZlbWVudCBpbiBiYXR0ZXJ5IDopCj4KPiA+ID4KPiA+ID4gSSBjZXJ0 YWlubHkgY2FuIGFwcHJlY2lhdGUgdGhlIG1vdGl2YXRpb24gZm9yIHRoZSBzZXJpZXMsIGJ1dCB0 aGlzIGZlYXR1cmUKPiA+ID4gc2hvdWxkIGJlIGluIHVzZXJzcGFjZSBhcyBzb21lIGZvcm0gb2Yg YSB2aXJ0dWFsIGRldmljZS4KPiA+Cj4gPiArMSBvbiBhbGwgb2YgdGhlIGFib3ZlLgo+IAo+IE1h cmMgYW5kIE9saXZlciwKPiAKPiBXZSBhcmUgbm90IHRpZWQgdG8gaHlwZXJjYWxscy4gV2Ugd2Fu dCB0byBkbyB0aGUgcmlnaHQgdGhpbmcgaGVyZSwgYnV0Cj4gTU1JTyBnb2luZyBhbGwgdGhlIHdh eSB0byB1c2Vyc3BhY2UgZGVmaW5pdGVseSBkb2Vzbid0IGN1dCBpdCBhcyBpcy4KPiBUaGlzIGlz IHdoZXJlIHdlIG5lZWQgc29tZSBndWlkYW5jZS4gU2VlIG1vcmUgYmVsb3cuCgpJIGRvbid0IGJ1 eSB0aGlzIGFzc2VydGlvbiBhdCBhbGwuIEFuIE1NSU8gaW4gdXNlcnNwYWNlIGlzIGFscmVhZHkK bXVjaCBiZXR0ZXIgdGhhbiBub3RoaW5nLiBPbmUgb2YgbXkgbWFueSBvYmplY3Rpb24gdG8gdGhl IHdob2xlIHNlcmllcwppcyB0aGF0IGl0IGlzIGJ1aWx0IGFzIGEgbWFzc2l2ZWx5IGludmFzaXZl IHRoaW5nIHRoYXQgaGFzIHRvbyBtYW55CmZpbmdlcnMgaW4gdG9vIG1hbnkgcGllcywgd2l0aCB1 bnN1c3RhaW5hYmxlIGFzc3VtcHRpb25zIHN1Y2ggYXMgMToxCm1hcHBpbmcgYmV0d2VlbiBDUFUg YW5kIHZDUFVzLgoKSSdkIHJhdGhlciB5b3UgYnVpbGQgc29tZXRoaW5nIHNpbXBsZSBmaXJzdCAo cHVyZSB1c2Vyc3BhY2UgdXNpbmcKTU1JT3MpLCB3b3JrIG91dCB3aGVyZSB0aGUgYm90dGxlbmVj a3MgYXJlLCBhbmQgd29yayB3aXRoIHVzIHRvIGFkZAp3aGF0IGlzIG5lZWRlZCB0byBnZXQgdG8g c29tZXRoaW5nIHNlbnNpYmxlLCBhbmQgb25seSB0aGF0LiBJJ20gbm90CndpbGxpbmcgdG8gc2Fj cmlmaWNlIG1haW50YWluYWJpbGl0eSBmb3IgbWF4aW11bSBwZXJmb3JtYW5jZSAodGhlCndob2xl IHRoaW5nIHJlbWluZHMgbWUgb2YgdGhlIGluLWtlcm5lbCBodHRwIHNlcnZlci4uLikuCgo+IAo+ ID4gVGhlIG9uZSB0aGluZyBJJ2QgbGlrZSB0byB1bmRlcnN0YW5kIHRoYXQgdGhlIGNvbW1lbnQg c2VlbXMgdG8gaW1wbHkKPiA+IHRoYXQgdGhlcmUgaXMgYSBzaWduaWZpY2FudCBkaWZmZXJlbmNl IGluIG92ZXJoZWFkIGJldHdlZW4gYSBoeXBlcmNhbGwKPiA+IGFuZCBhbiBNTUlPLiBJbiBteSBl eHBlcmllbmNlLCBib3RoIGFyZSBwcmV0dHkgc2ltaWxhciBpbiBjb3N0IGZvciBhCj4gPiBoYW5k bGluZyBsb2NhdGlvbiAoYm90aCBpbiB1c2Vyc3BhY2Ugb3IgYm90aCBpbiB0aGUga2VybmVsKS4K PiAKPiBJIHRoaW5rIHRoZSBtYWluIGRpZmZlcmVuY2UgcmVhbGx5IGlzIHRoYXQgaW4gb3VyIGh5 cGVyY2FsbCB2cyBNTUlPCj4gY29tcGFyaXNvbiB0aGUgaHlwZXJjYWxsIGlzIGhhbmRsZWQgaW4g dGhlIGtlcm5lbCB2cyBNTUlPIGdvZXMgYWxsIHRoZQo+IHdheSB0byB1c2Vyc3BhY2UuIEkgYWdy ZWUgd2l0aCB5b3UgdGhhdCB0aGUgZGlmZmVyZW5jZSBwcm9iYWJseSB3b24ndAo+IGJlIHNpZ25p ZmljYW50IGlmIGJvdGggb2YgdGhlbSBnbyB0byB0aGUgc2FtZSAiZGVwdGgiIGluIHRoZSBwcml2 aWxlZ2UKPiBsZXZlbHMuCj4gCj4gPiBNTUlPCj4gPiBoYW5kbGluZyBpcyBhIHRpbnkgYml0IG1v cmUgZXhwZW5zaXZlIGR1ZSB0byBhIGd1YXJhbnRlZWQgVExCIG1pc3MKPiA+IGZvbGxvd2VkIGJ5 IGEgd2FsayBvZiB0aGUgaW4ta2VybmVsIGRldmljZSByYW5nZXMsIGJ1dCB0aGF0J3MgYWxsLiBJ dAo+ID4gc2hvdWxkIGhhcmRseSByZWdpc3Rlci4KPiA+Cj4gPiBBbmQgaWYgeW91IHJlYWxseSB3 YW50IHNvbWUgc3VwZXItbG93IGxhdGVuY3ksIGxvdyBvdmVyaGVhZAo+ID4gc2lnbmFsbGluZywg bWF5YmUgYW4gZXhjZXB0aW9uIGlzIHRoZSB3cm9uZyB0b29sIGZvciB0aGUgam9iLiBTaGFyZWQK PiA+IG1lbW9yeSBjb21tdW5pY2F0aW9uIGNvdWxkIGJlIG1vcmUgYXBwcm9wcmlhdGUuCj4gCj4g WWVhaCwgdGhhdCdzIG9uZSBvZiBvdXIgbmV4dCBzdGVwcy4gSWRlYWxseSwgd2Ugd2FudCB0byB1 c2Ugc2hhcmVkCj4gbWVtb3J5IGZvciB0aGUgaG9zdCB0byBndWVzdCBpbmZvcm1hdGlvbiBmbG93 LiBJdCdzIGEgMzItYml0IHZhbHVlCj4gcmVwcmVzZW50aW5nIHRoZSBjdXJyZW50IGZyZXF1ZW5j eSB0aGF0IHRoZSBob3N0IGNhbiB1cGRhdGUgd2hlbmV2ZXIKPiB0aGUgaG9zdCBDUFUgZnJlcXVl bmN5IGNoYW5nZXMgYW5kIHRoZSBndWVzdCBjYW4gcmVhZCB3aGVuZXZlciBpdAo+IG5lZWRzIGl0 LgoKV2h5IHNob3VsZCB0aGUgZ3Vlc3QgY2FyZT8gV2h5IGNhbid0IHRoZSBndWVzdCBhc2sgZm9y IGFuIGFyYml0cmFyeQpjYXBhY2l0eSwgYW5kIGdldCB3aGF0IGl0IGdldHM/IFlvdSBnaXZlIG5v IGluZm9ybWF0aW9uIGFzIHRvICp3aHkqCnlvdSBhcmUgZG9pbmcgd2hhdCB5b3UgYXJlIGRvaW5n Li4uCgo+IAo+IEZvciBndWVzdCB0byBob3N0IGluZm9ybWF0aW9uIGZsb3csIHdlJ2xsIG5lZWQg YSBraWNrIGZyb20gZ3Vlc3QgdG8KPiBob3N0IGJlY2F1c2Ugd2UgbmVlZCB0byB0YWtlIGFjdGlv biBvbiB0aGUgaG9zdCBzaWRlIHdoZW4gdGhyZWFkcwo+IG1pZ3JhdGUgYmV0d2VlbiB2Q1BVcyBh bmQgY2F1c2UgYSBzaWduaWZpY2FudCBjaGFuZ2UgaW4gdkNQVSB1dGlsLgo+IEFnYWluIGl0IGNh biBiZSBqdXN0IGEgc2hhcmVkIG1lbW9yeSBhbmQgc29tZSBraWNrLiBUaGlzIGlzIHdoYXQgd2UK PiBhcmUgY3VycmVudGx5IHRyeWluZyB0byBmaWd1cmUgb3V0IGhvdyB0byBkby4KClRoYXQga2lj ayB3b3VsZCBoYXZlIHRvIGdvIHRvIHVzZXJzcGFjZS4gVGhlcmUgaXMgbm8gd2F5IEknbSB3aWxs aW5nCnRvIGludHJvZHVjZSBzY2hlZHVsaW5nIHByaW1pdGl2ZXMgaW5zaWRlIEtWTSAodGhlIG9u ZXMgd2UgaGF2ZSBhcmUKcmlkaWN1bG91c2x5IGJhZCBhbnl3YXkpLCBhbmQgSSB2ZXJ5IG11Y2gg d2FudCB0byBhdm9pZCBleHRyYSBQViBndW5rLgoKPiBJZiB0aGVyZSBhcmUgQVBJcyB0byBkbyB0 aGlzLCBjYW4geW91IHBvaW50IHVzIHRvIHRob3NlIHBsZWFzZT8gV2UnZAo+IGFsc28gd2FudCB0 aGUgc2hhcmVkIG1lbW9yeSB0byBiZSBhY2Nlc3NpYmxlIGJ5IHRoZSBWTU0gKHNvLCBzaGFyZWQK PiBiZXR3ZWVuIGd1ZXN0IGtlcm5lbCwgaG9zdCBrZXJuZWwgYW5kIFZNTSkuCgpCeSBkZWZhdWx0 LCAqQUxMKiB0aGUgbWVtb3J5IGlzIHNoYXJlZC4gSXNuJ3QgdGhhdCB3b25kZXJmdWw/Cgo+IAo+ IEFyZSB0aGUgYWJvdmUgbmV4dCBzdGVwcyBzYW5lPyBPciBpcyB0aGF0IGEgbm8tZ28/IFRoZSBt YWluIHRoaW5nIHdlCj4gd2FudCB0byBjdXQgb3V0IGlzIHRoZSBuZWVkIGZvciBoYXZpbmcgdG8g c3dpdGNoIHRvIHVzZXJzcGFjZSBmb3IKPiBldmVyeSBzaW5nbGUgaW50ZXJhY3Rpb24gYmVjYXVz ZSwgYXMgaXMsIGl0IGxlYXZlcyBhIGxvdCBvbiB0aGUgdGFibGUuCgpXZWxsLCBmb3IgYSBzdGFy dCwgeW91IGNvdWxkIGRpc2Nsb3NlIGhvdyBvZnRlbiB5b3UgaGl0IHRoaXMgRFZGUwoiZGV2aWNl IiwgYW5kIHdoZW4gYXJlIHRoZSBjcml0aWNhbCBzdGF0ZSBjaGFuZ2VzIHRoYXQgbXVzdCBoYXBw ZW4KaW1tZWRpYXRlbHkgdnMgdGhvc2UgdGhhdCBjYW4gc2ltcGx5IGJlIHBvc3RlZCB3aXRob3V0 IGhhdmluZyB0byB0YWtlCmltbWVkaWF0ZSBlZmZlY3QuCgpUaGlzIHNvcnQgb2YgaW5mb3JtYXRp b24gd291bGQgYmUgbXVjaCBtb3JlIGludGVyZXN0aW5nIHRoYW4gYSBidW5jaApvZiBiZW5jaG1h cmtzIEkga25vdyBub3RoaW5nIGFib3V0LgoKVGhhbmtzLAoKCU0uCgotLSAKV2l0aG91dCBkZXZp YXRpb24gZnJvbSB0aGUgbm9ybSwgcHJvZ3Jlc3MgaXMgbm90IHBvc3NpYmxlLgoKX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBt YWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9s aXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=