From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E71F718C16 for ; Tue, 11 Jul 2023 12:30:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C15AC433C9; Tue, 11 Jul 2023 12:30:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1689078624; bh=JAoQImOGHobEUP97NTQ9fB21QVqS1rQvMSn2Qvf/Blk=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=FqsLHE4HlNiVsy3AeTCslZ0MXzU8I+NbNL9dVneGWTvy0RYjmv11hA+18zVF5e5KA EGLaz9msRGRngFnOhWRUmembwHyP8j4V3ePaogTYpe+UxVhs4HYssI4ezgmx88YHl6 GgdGvooQ5MS1AGtxV43hLbrH5bdjpsEWOGi33Y3fnhn/HSkHTe3sGoejVl28kMcpuk V03vDTdobKEjTB9iO1dRRMNOTVukPnM9WPPgsA9KYkXwcvxG4JPpj7YEwVP4iOtBE+ 4HcA6vJZFB8Ffr3WenAdm2onyIGXDQdPF0Ptss0iDUZh7TXdkq+rInAciFEAeoczz+ 0djTss0FyrToA== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1qJCVZ-00C7H6-Se; Tue, 11 Jul 2023 13:30:21 +0100 Date: Tue, 11 Jul 2023 13:30:21 +0100 Message-ID: <86bkgiwrz6.wl-maz@kernel.org> From: Marc Zyngier To: Ganapatrao Kulkarni Cc: Miguel Luis , Eric Auger , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Darren Hart , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: Re: [PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support In-Reply-To: <853a5f76-74fe-a38d-f2cd-785963177c8a@os.amperecomputing.com> References: <20230515173103.1017669-1-maz@kernel.org> <877crmzr5j.wl-maz@kernel.org> <04ec8efb-33ff-153e-3be5-5c84a01bff2a@os.amperecomputing.com> <853a5f76-74fe-a38d-f2cd-785963177c8a@os.amperecomputing.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gankulkarni@os.amperecomputing.com, miguel.luis@oracle.com, eauger@redhat.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, darren@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Tue, 11 Jul 2023 12:56:48 +0100, Ganapatrao Kulkarni wrote: >=20 >=20 >=20 > On 07-07-2023 03:16 pm, Ganapatrao Kulkarni wrote: > >=20 > >=20 > > On 04-07-2023 06:01 pm, Ganapatrao Kulkarni wrote: > >>=20 > >> Hi Marc, > >>=20 > >> On 29-06-2023 12:33 pm, Marc Zyngier wrote: > >>> Hi Ganapatrao, > >>>=20 > >>> On Wed, 28 Jun 2023 07:45:55 +0100, > >>> Ganapatrao Kulkarni wrote: > >>>>=20 > >>>>=20 > >>>> Hi Marc, > >>>>=20 > >>>>=20 > >>>> On 15-05-2023 11:00 pm, Marc Zyngier wrote: > >>>>> This is the 4th drop of NV support on arm64 for this year. > >>>>>=20 > >>>>> For the previous episodes, see [1]. > >>>>>=20 > >>>>> What's changed: > >>>>>=20 > >>>>> - New framework to track system register traps that are reinjected = in > >>>>> =C2=A0=C2=A0=C2=A0 guest EL2. It is expected to replace the discret= e handling we have > >>>>> =C2=A0=C2=A0=C2=A0 enjoyed so far, which didn't scale at all. This = has already > >>>>> fixed a > >>>>> =C2=A0=C2=A0=C2=A0 number of bugs that were hidden (a bunch of trap= s were never > >>>>> =C2=A0=C2=A0=C2=A0 forwarded...). Still a work in progress, but thi= s is going in the > >>>>> =C2=A0=C2=A0=C2=A0 right direction. > >>>>>=20 > >>>>> - Allow the L1 hypervisor to have a S2 that has an input larger than > >>>>> =C2=A0=C2=A0=C2=A0 the L0 IPA space. This fixes a number of subtle = issues, > >>>>> depending on > >>>>> =C2=A0=C2=A0=C2=A0 how the initial guest was created. > >>>>>=20 > >>>>> - Consequently, the patch series has gone longer again. Boo. But > >>>>> =C2=A0=C2=A0=C2=A0 hopefully some of it is easier to review... > >>>>>=20 > >>>>=20 > >>>> I am facing issue in booting NestedVM with V9 as well with 10 patchs= et. > >>>>=20 > >>>> I have tried V9/V10 on Ampere platform using kvmtool and I could boot > >>>> Guest-Hypervisor and then NestedVM without any issue. > >>>> However when I try to boot using QEMU(not using EDK2/EFI), > >>>> Guest-Hypervisor is booted with Fedora 37 using virtio disk. From > >>>> Guest-Hypervisor console(or ssh shell), If I try to boot NestedVM, > >>>> boot hangs very early stage of the boot. > >>>>=20 > >>>> I did some debug using ftrace and it seems the Guest-Hypervisor is > >>>> getting very high rate of arch-timer interrupts, > >>>> due to that all CPU time is going on in serving the Guest-Hypervisor > >>>> and it is never going back to NestedVM. > >>>>=20 > >>>> I am using QEMU vanilla version v7.2.0 with top-up patches for NV [1] > >>>=20 > >>> So I went ahead and gave QEMU a go. On my systems, *nothing* works (I > >>> cannot even boot a L1 with 'virtualization=3Don" (the guest is stuck = at > >>> the point where virtio gets probed and waits for its first interrupt). > >>>=20 > >>> Worse, booting a hVHE guest results in QEMU generating an assert as it > >>> tries to inject an interrupt using the QEMU GICv3 model, something > >>> that should *NEVER* be in use with KVM. > >>>=20 > >>> With help from Eric, I got to a point where the hVHE guest could boot > >>> as long as I kept injecting console interrupts, which is again a > >>> symptom of the vGIC not being used. > >>>=20 > >>> So something is *majorly* wrong with the QEMU patches. I don't know > >>> what makes it possible for you to even boot the L1 - if the GIC is > >>> external, injecting an interrupt in the L2 is simply impossible. > >>>=20 > >>> Miguel, can you please investigate this? > >>>=20 > >>> In the meantime, I'll add some code to the kernel side to refuse the > >>> external interrupt controller configuration with NV. Hopefully that > >>> will lead to some clues about what is going on. > >>=20 > >> Continued debugging of the issue and it seems the endless ptimer > >> interrupts on Ampere platform is due to some mess up of CVAL of > >> ptimer, resulting in interrupt triggered always when it is enabled. > >>=20 > >> I see function "timer_set_offset" called from kvm_arm_timer_set_reg > >> in QEMU case but there is no such calls in kvmtool boot. > >>=20 > >> If I comment the timer_set_offset calls in kvm_arm_timer_set_reg > >> function, then I could boot the Guest-Hypervisor then NestedVM from > >> GH/L1. > >>=20 > >> I also observed in QEMU case, kvm_arm_timer_set_reg is called to > >> set CNT, CVAL and CTL of both vtimer and ptimer. > >> Not sure why QEMU is setting these registers explicitly? need to dig. > >>=20 > >=20 > > I don't see any direct ioctl calls to change any timer > > registers. Looks like it is happening from the emulation > > code(target/arm/helper.c)? >=20 > function "write_list_to_kvmstate(target/arm/kvm.c)" is issuing ioctl > to write timer registers. PTIMER_CNT and TIMER_CNT writes from this > function is resulting in offsets change. Madness. Why is QEMU doing this? It has no business writing to the timer at any stage, at least with KVM. This confirms my suspicions that QEMU is confused about what mode it runs in, and does all sort of funky things it was never expected to do. M. --=20 Without deviation from the norm, progress is not possible. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 79F2CEB64DD for ; Tue, 11 Jul 2023 12:31:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Td1SBVLFeiBAPKvwVRXzumB1N1GFq30H2Hgm/wQomPs=; b=l1yhN7tljWPx6Y cDN2Yp4NHs587TO5YPjEX2YdmNc67La0YQ3SuD05wDFwx5lib2fqgsYZ+5iplQS3PUXqpsx+ylfFQ LRD+PGT76p8R8xj/TYUOYxbHVcZrM3Ot6AX1XUfsVHfU4AYXPMu2V2OGzL/htp/TohMRcaxakoqt8 JR1h+Ba70WBLMMn4Wbb7Vd42c5J/yQkxz/xndykHAGuRFBFEc8KbrVzYCOv4sHWme0kU/jP46Lt4Q UBy4Ju1BuZQ6779mbTnrEd2ZloGY0mmQQ80PIF+XuYERcYqwNNPasqePFI7bcHCtAo+RQ8xN8i55Y /+p8h5TqWTCq9gAWwa/A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1qJCVj-00EsY8-0P; Tue, 11 Jul 2023 12:30:31 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1qJCVd-00EsXI-1t for linux-arm-kernel@lists.infradead.org; Tue, 11 Jul 2023 12:30:29 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id EF5F7614C1; Tue, 11 Jul 2023 12:30:24 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5C15AC433C9; Tue, 11 Jul 2023 12:30:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1689078624; bh=JAoQImOGHobEUP97NTQ9fB21QVqS1rQvMSn2Qvf/Blk=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=FqsLHE4HlNiVsy3AeTCslZ0MXzU8I+NbNL9dVneGWTvy0RYjmv11hA+18zVF5e5KA EGLaz9msRGRngFnOhWRUmembwHyP8j4V3ePaogTYpe+UxVhs4HYssI4ezgmx88YHl6 GgdGvooQ5MS1AGtxV43hLbrH5bdjpsEWOGi33Y3fnhn/HSkHTe3sGoejVl28kMcpuk V03vDTdobKEjTB9iO1dRRMNOTVukPnM9WPPgsA9KYkXwcvxG4JPpj7YEwVP4iOtBE+ 4HcA6vJZFB8Ffr3WenAdm2onyIGXDQdPF0Ptss0iDUZh7TXdkq+rInAciFEAeoczz+ 0djTss0FyrToA== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1qJCVZ-00C7H6-Se; Tue, 11 Jul 2023 13:30:21 +0100 Date: Tue, 11 Jul 2023 13:30:21 +0100 Message-ID: <86bkgiwrz6.wl-maz@kernel.org> From: Marc Zyngier To: Ganapatrao Kulkarni Cc: Miguel Luis , Eric Auger , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Alexandru Elisei , Andre Przywara , Chase Conklin , Christoffer Dall , Darren Hart , Jintack Lim , Russell King , James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu Subject: Re: [PATCH v10 00/59] KVM: arm64: ARMv8.3/8.4 Nested Virtualization support In-Reply-To: <853a5f76-74fe-a38d-f2cd-785963177c8a@os.amperecomputing.com> References: <20230515173103.1017669-1-maz@kernel.org> <877crmzr5j.wl-maz@kernel.org> <04ec8efb-33ff-153e-3be5-5c84a01bff2a@os.amperecomputing.com> <853a5f76-74fe-a38d-f2cd-785963177c8a@os.amperecomputing.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: gankulkarni@os.amperecomputing.com, miguel.luis@oracle.com, eauger@redhat.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, alexandru.elisei@arm.com, andre.przywara@arm.com, chase.conklin@arm.com, christoffer.dall@arm.com, darren@os.amperecomputing.com, jintack@cs.columbia.edu, rmk+kernel@armlinux.org.uk, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230711_053027_258254_B77E178E X-CRM114-Status: GOOD ( 45.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gVHVlLCAxMSBKdWwgMjAyMyAxMjo1Njo0OCArMDEwMCwKR2FuYXBhdHJhbyBLdWxrYXJuaSA8 Z2Fua3Vsa2FybmlAb3MuYW1wZXJlY29tcHV0aW5nLmNvbT4gd3JvdGU6Cj4gCj4gCj4gCj4gT24g MDctMDctMjAyMyAwMzoxNiBwbSwgR2FuYXBhdHJhbyBLdWxrYXJuaSB3cm90ZToKPiA+IAo+ID4g Cj4gPiBPbiAwNC0wNy0yMDIzIDA2OjAxIHBtLCBHYW5hcGF0cmFvIEt1bGthcm5pIHdyb3RlOgo+ ID4+IAo+ID4+IEhpIE1hcmMsCj4gPj4gCj4gPj4gT24gMjktMDYtMjAyMyAxMjozMyBwbSwgTWFy YyBaeW5naWVyIHdyb3RlOgo+ID4+PiBIaSBHYW5hcGF0cmFvLAo+ID4+PiAKPiA+Pj4gT24gV2Vk LCAyOCBKdW4gMjAyMyAwNzo0NTo1NSArMDEwMCwKPiA+Pj4gR2FuYXBhdHJhbyBLdWxrYXJuaSA8 Z2Fua3Vsa2FybmlAb3MuYW1wZXJlY29tcHV0aW5nLmNvbT4gd3JvdGU6Cj4gPj4+PiAKPiA+Pj4+ IAo+ID4+Pj4gSGkgTWFyYywKPiA+Pj4+IAo+ID4+Pj4gCj4gPj4+PiBPbiAxNS0wNS0yMDIzIDEx OjAwIHBtLCBNYXJjIFp5bmdpZXIgd3JvdGU6Cj4gPj4+Pj4gVGhpcyBpcyB0aGUgNHRoIGRyb3Ag b2YgTlYgc3VwcG9ydCBvbiBhcm02NCBmb3IgdGhpcyB5ZWFyLgo+ID4+Pj4+IAo+ID4+Pj4+IEZv ciB0aGUgcHJldmlvdXMgZXBpc29kZXMsIHNlZSBbMV0uCj4gPj4+Pj4gCj4gPj4+Pj4gV2hhdCdz IGNoYW5nZWQ6Cj4gPj4+Pj4gCj4gPj4+Pj4gLSBOZXcgZnJhbWV3b3JrIHRvIHRyYWNrIHN5c3Rl bSByZWdpc3RlciB0cmFwcyB0aGF0IGFyZSByZWluamVjdGVkIGluCj4gPj4+Pj4gwqDCoMKgIGd1 ZXN0IEVMMi4gSXQgaXMgZXhwZWN0ZWQgdG8gcmVwbGFjZSB0aGUgZGlzY3JldGUgaGFuZGxpbmcg d2UgaGF2ZQo+ID4+Pj4+IMKgwqDCoCBlbmpveWVkIHNvIGZhciwgd2hpY2ggZGlkbid0IHNjYWxl IGF0IGFsbC4gVGhpcyBoYXMgYWxyZWFkeQo+ID4+Pj4+IGZpeGVkIGEKPiA+Pj4+PiDCoMKgwqAg bnVtYmVyIG9mIGJ1Z3MgdGhhdCB3ZXJlIGhpZGRlbiAoYSBidW5jaCBvZiB0cmFwcyB3ZXJlIG5l dmVyCj4gPj4+Pj4gwqDCoMKgIGZvcndhcmRlZC4uLikuIFN0aWxsIGEgd29yayBpbiBwcm9ncmVz cywgYnV0IHRoaXMgaXMgZ29pbmcgaW4gdGhlCj4gPj4+Pj4gwqDCoMKgIHJpZ2h0IGRpcmVjdGlv bi4KPiA+Pj4+PiAKPiA+Pj4+PiAtIEFsbG93IHRoZSBMMSBoeXBlcnZpc29yIHRvIGhhdmUgYSBT MiB0aGF0IGhhcyBhbiBpbnB1dCBsYXJnZXIgdGhhbgo+ID4+Pj4+IMKgwqDCoCB0aGUgTDAgSVBB IHNwYWNlLiBUaGlzIGZpeGVzIGEgbnVtYmVyIG9mIHN1YnRsZSBpc3N1ZXMsCj4gPj4+Pj4gZGVw ZW5kaW5nIG9uCj4gPj4+Pj4gwqDCoMKgIGhvdyB0aGUgaW5pdGlhbCBndWVzdCB3YXMgY3JlYXRl ZC4KPiA+Pj4+PiAKPiA+Pj4+PiAtIENvbnNlcXVlbnRseSwgdGhlIHBhdGNoIHNlcmllcyBoYXMg Z29uZSBsb25nZXIgYWdhaW4uIEJvby4gQnV0Cj4gPj4+Pj4gwqDCoMKgIGhvcGVmdWxseSBzb21l IG9mIGl0IGlzIGVhc2llciB0byByZXZpZXcuLi4KPiA+Pj4+PiAKPiA+Pj4+IAo+ID4+Pj4gSSBh bSBmYWNpbmcgaXNzdWUgaW4gYm9vdGluZyBOZXN0ZWRWTSB3aXRoIFY5IGFzIHdlbGwgd2l0aCAx MCBwYXRjaHNldC4KPiA+Pj4+IAo+ID4+Pj4gSSBoYXZlIHRyaWVkIFY5L1YxMCBvbiBBbXBlcmUg cGxhdGZvcm0gdXNpbmcga3ZtdG9vbCBhbmQgSSBjb3VsZCBib290Cj4gPj4+PiBHdWVzdC1IeXBl cnZpc29yIGFuZCB0aGVuIE5lc3RlZFZNIHdpdGhvdXQgYW55IGlzc3VlLgo+ID4+Pj4gSG93ZXZl ciB3aGVuIEkgdHJ5IHRvIGJvb3QgdXNpbmcgUUVNVShub3QgdXNpbmcgRURLMi9FRkkpLAo+ID4+ Pj4gR3Vlc3QtSHlwZXJ2aXNvciBpcyBib290ZWQgd2l0aCBGZWRvcmEgMzcgdXNpbmcgdmlydGlv IGRpc2suIEZyb20KPiA+Pj4+IEd1ZXN0LUh5cGVydmlzb3IgY29uc29sZShvciBzc2ggc2hlbGwp LCBJZiBJIHRyeSB0byBib290IE5lc3RlZFZNLAo+ID4+Pj4gYm9vdCBoYW5ncyB2ZXJ5IGVhcmx5 IHN0YWdlIG9mIHRoZSBib290Lgo+ID4+Pj4gCj4gPj4+PiBJIGRpZCBzb21lIGRlYnVnIHVzaW5n IGZ0cmFjZSBhbmQgaXQgc2VlbXMgdGhlIEd1ZXN0LUh5cGVydmlzb3IgaXMKPiA+Pj4+IGdldHRp bmcgdmVyeSBoaWdoIHJhdGUgb2YgYXJjaC10aW1lciBpbnRlcnJ1cHRzLAo+ID4+Pj4gZHVlIHRv IHRoYXQgYWxsIENQVSB0aW1lIGlzIGdvaW5nIG9uIGluIHNlcnZpbmcgdGhlIEd1ZXN0LUh5cGVy dmlzb3IKPiA+Pj4+IGFuZCBpdCBpcyBuZXZlciBnb2luZyBiYWNrIHRvIE5lc3RlZFZNLgo+ID4+ Pj4gCj4gPj4+PiBJIGFtIHVzaW5nIFFFTVUgdmFuaWxsYSB2ZXJzaW9uIHY3LjIuMCB3aXRoIHRv cC11cCBwYXRjaGVzIGZvciBOViBbMV0KPiA+Pj4gCj4gPj4+IFNvIEkgd2VudCBhaGVhZCBhbmQg Z2F2ZSBRRU1VIGEgZ28uIE9uIG15IHN5c3RlbXMsICpub3RoaW5nKiB3b3JrcyAoSQo+ID4+PiBj YW5ub3QgZXZlbiBib290IGEgTDEgd2l0aCAndmlydHVhbGl6YXRpb249b24iICh0aGUgZ3Vlc3Qg aXMgc3R1Y2sgYXQKPiA+Pj4gdGhlIHBvaW50IHdoZXJlIHZpcnRpbyBnZXRzIHByb2JlZCBhbmQg d2FpdHMgZm9yIGl0cyBmaXJzdCBpbnRlcnJ1cHQpLgo+ID4+PiAKPiA+Pj4gV29yc2UsIGJvb3Rp bmcgYSBoVkhFIGd1ZXN0IHJlc3VsdHMgaW4gUUVNVSBnZW5lcmF0aW5nIGFuIGFzc2VydCBhcyBp dAo+ID4+PiB0cmllcyB0byBpbmplY3QgYW4gaW50ZXJydXB0IHVzaW5nIHRoZSBRRU1VIEdJQ3Yz IG1vZGVsLCBzb21ldGhpbmcKPiA+Pj4gdGhhdCBzaG91bGQgKk5FVkVSKiBiZSBpbiB1c2Ugd2l0 aCBLVk0uCj4gPj4+IAo+ID4+PiBXaXRoIGhlbHAgZnJvbSBFcmljLCBJIGdvdCB0byBhIHBvaW50 IHdoZXJlIHRoZSBoVkhFIGd1ZXN0IGNvdWxkIGJvb3QKPiA+Pj4gYXMgbG9uZyBhcyBJIGtlcHQg aW5qZWN0aW5nIGNvbnNvbGUgaW50ZXJydXB0cywgd2hpY2ggaXMgYWdhaW4gYQo+ID4+PiBzeW1w dG9tIG9mIHRoZSB2R0lDIG5vdCBiZWluZyB1c2VkLgo+ID4+PiAKPiA+Pj4gU28gc29tZXRoaW5n IGlzICptYWpvcmx5KiB3cm9uZyB3aXRoIHRoZSBRRU1VIHBhdGNoZXMuIEkgZG9uJ3Qga25vdwo+ ID4+PiB3aGF0IG1ha2VzIGl0IHBvc3NpYmxlIGZvciB5b3UgdG8gZXZlbiBib290IHRoZSBMMSAt IGlmIHRoZSBHSUMgaXMKPiA+Pj4gZXh0ZXJuYWwsIGluamVjdGluZyBhbiBpbnRlcnJ1cHQgaW4g dGhlIEwyIGlzIHNpbXBseSBpbXBvc3NpYmxlLgo+ID4+PiAKPiA+Pj4gTWlndWVsLCBjYW4geW91 IHBsZWFzZSBpbnZlc3RpZ2F0ZSB0aGlzPwo+ID4+PiAKPiA+Pj4gSW4gdGhlIG1lYW50aW1lLCBJ J2xsIGFkZCBzb21lIGNvZGUgdG8gdGhlIGtlcm5lbCBzaWRlIHRvIHJlZnVzZSB0aGUKPiA+Pj4g ZXh0ZXJuYWwgaW50ZXJydXB0IGNvbnRyb2xsZXIgY29uZmlndXJhdGlvbiB3aXRoIE5WLiBIb3Bl ZnVsbHkgdGhhdAo+ID4+PiB3aWxsIGxlYWQgdG8gc29tZSBjbHVlcyBhYm91dCB3aGF0IGlzIGdv aW5nIG9uLgo+ID4+IAo+ID4+IENvbnRpbnVlZCBkZWJ1Z2dpbmcgb2YgdGhlIGlzc3VlIGFuZCBp dCBzZWVtcyB0aGUgZW5kbGVzcyBwdGltZXIKPiA+PiBpbnRlcnJ1cHRzIG9uIEFtcGVyZSBwbGF0 Zm9ybSBpcyBkdWUgdG8gc29tZSBtZXNzIHVwIG9mIENWQUwgb2YKPiA+PiBwdGltZXIsIHJlc3Vs dGluZyBpbiBpbnRlcnJ1cHQgdHJpZ2dlcmVkIGFsd2F5cyB3aGVuIGl0IGlzIGVuYWJsZWQuCj4g Pj4gCj4gPj4gSSBzZWUgZnVuY3Rpb24gInRpbWVyX3NldF9vZmZzZXQiIGNhbGxlZCBmcm9tIGt2 bV9hcm1fdGltZXJfc2V0X3JlZwo+ID4+IGluIFFFTVUgY2FzZSBidXQgdGhlcmUgaXMgbm8gc3Vj aCBjYWxscyBpbiBrdm10b29sIGJvb3QuCj4gPj4gCj4gPj4gSWYgSSBjb21tZW50IHRoZSB0aW1l cl9zZXRfb2Zmc2V0IGNhbGxzIGluIGt2bV9hcm1fdGltZXJfc2V0X3JlZwo+ID4+IGZ1bmN0aW9u LCB0aGVuIEkgY291bGQgYm9vdCB0aGUgR3Vlc3QtSHlwZXJ2aXNvciB0aGVuIE5lc3RlZFZNIGZy b20KPiA+PiBHSC9MMS4KPiA+PiAKPiA+PiBJIGFsc28gb2JzZXJ2ZWQgaW4gUUVNVSBjYXNlLCBr dm1fYXJtX3RpbWVyX3NldF9yZWcgaXMgY2FsbGVkIHRvCj4gPj4gc2V0IENOVCwgQ1ZBTCBhbmQg Q1RMIG9mIGJvdGggdnRpbWVyIGFuZCBwdGltZXIuCj4gPj4gTm90IHN1cmUgd2h5IFFFTVUgaXMg c2V0dGluZyB0aGVzZSByZWdpc3RlcnMgZXhwbGljaXRseT8gbmVlZCB0byBkaWcuCj4gPj4gCj4g PiAKPiA+IEkgZG9uJ3Qgc2VlIGFueSBkaXJlY3QgaW9jdGwgY2FsbHMgdG8gY2hhbmdlIGFueSB0 aW1lcgo+ID4gcmVnaXN0ZXJzLiBMb29rcyBsaWtlIGl0IGlzIGhhcHBlbmluZyBmcm9tIHRoZSBl bXVsYXRpb24KPiA+IGNvZGUodGFyZ2V0L2FybS9oZWxwZXIuYyk/Cj4gCj4gZnVuY3Rpb24gIndy aXRlX2xpc3RfdG9fa3Ztc3RhdGUodGFyZ2V0L2FybS9rdm0uYykiIGlzIGlzc3VpbmcgaW9jdGwK PiB0byB3cml0ZSB0aW1lciByZWdpc3RlcnMuIFBUSU1FUl9DTlQgYW5kIFRJTUVSX0NOVCB3cml0 ZXMgZnJvbSB0aGlzCj4gZnVuY3Rpb24gaXMgcmVzdWx0aW5nIGluIG9mZnNldHMgY2hhbmdlLgoK TWFkbmVzcy4gV2h5IGlzIFFFTVUgZG9pbmcgdGhpcz8gSXQgaGFzIG5vIGJ1c2luZXNzIHdyaXRp bmcgdG8gdGhlCnRpbWVyIGF0IGFueSBzdGFnZSwgYXQgbGVhc3Qgd2l0aCBLVk0uCgpUaGlzIGNv bmZpcm1zIG15IHN1c3BpY2lvbnMgdGhhdCBRRU1VIGlzIGNvbmZ1c2VkIGFib3V0IHdoYXQgbW9k ZSBpdApydW5zIGluLCBhbmQgZG9lcyBhbGwgc29ydCBvZiBmdW5reSB0aGluZ3MgaXQgd2FzIG5l dmVyIGV4cGVjdGVkIHRvCmRvLgoKCU0uCgotLSAKV2l0aG91dCBkZXZpYXRpb24gZnJvbSB0aGUg bm9ybSwgcHJvZ3Jlc3MgaXMgbm90IHBvc3NpYmxlLgoKX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5lbCBtYWlsaW5nIGxpc3QKbGlu dXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6Ly9saXN0cy5pbmZyYWRlYWQu b3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=