From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; dkim=none Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 3AC6D84; Thu, 30 Nov 2023 03:49:05 -0800 (PST) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 94F3F1042; Thu, 30 Nov 2023 03:49:51 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.43.250]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 128D03F5A1; Thu, 30 Nov 2023 03:49:03 -0800 (PST) Date: Thu, 30 Nov 2023 11:48:54 +0000 From: Mark Rutland To: "Ashley, William" Cc: "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "linux-perf-users@vger.kernel.org" , will@kernel.org Subject: Re: armv8pmu: Pending overflow interrupt is discarded when perf event is disabled Message-ID: References: <950001BD-490C-4BAC-8EEA-CDB9F7C4ADFC@amazon.com> Precedence: bulk X-Mailing-List: linux-perf-users@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Wed, Nov 29, 2023 at 04:35:01PM +0000, Mark Rutland wrote: > Hi William, > > On Mon, Nov 20, 2023 at 10:32:10PM +0000, Ashley, William wrote: > > Adding linux-arm-kernel@lists.infradead.org and linux-kernel@vger.kernel.org, > > sorry for the noise. > > Thanks for this! > > For the benefit of others, the original mail (and attachment) can be found at: > > https://lore.kernel.org/linux-perf-users/950001BD-490C-4BAC-8EEA-CDB9F7C4ADFC@amazon.com/ > > For some reason, links and whitespace seem to have been mangled in the resend; > I'm not sure what happened there. > > I've added Will Deacon, as Will and I co-maintain the ARM PMU drivers. > > > On 11/20/23, 12:36 PM, "Ashley, William" > wrote: > > > > > > An issue [1] was opened in the rr-debugger project reporting occasional missed > > perf event overflow signals on arm64. I've been digging into this and think I > > understand what's happening, but wanted to confirm my understanding. > > > > The attached example application, derived from an rr-debugger test case, reports > > when the value of a counter doesn't increase by the expected period +/- some > > tolerance. When it is ping-ponged between cores (e.g. with taskset) at a high > > frequency, it frequently reports increases of ~2x the expected. I've confirmed > > this same behavior on kernels 5.4, 5.10, 6.1 and 6.5. > > > > > > I found armv8pmu_disable_intens [2] that is called as part of event > > de-scheduling and contains > > /* Clear the overflow flag in case an interrupt is pending. */ > > write_pmovsclr(mask); > > which results in any pending overflow interrupt being dropped. I added some > > debug output here and indeed there is a correlation of this bit being high at > > the point of the above code and the reproducer identifying a missed signal. > > I think you're right that if we had an overflow asserted at this point, we'll > throw away the occurrence of the overflow (and so not call > perf_event_overflow() and generate a sample, etc). > > It looks like we only lose the occurrence of the overflow; the actual counts > will get sampled correctly and when we next reprogram the event, > armpmu_event_set_period() should set up the next overflow period. > > > This behavior does not occur with pseudo-NMIs (irqchip.gicv3_pseudo_nmi=1) > > enabled. > > That's interesting, because it implies that the PMU overflow interrupt is being > recognised by the CPU while regular interrupts are disabled. There are some > narrow races where that could occur (e.g. taking a timer or scheduler IRQ > *just* as an overflow occurs), and some other cases I'd expect RR to avoid by > construction (e.g. if RR isn't using mode exclusion and also counts kernel > events). It's also worth noting that this means there are races even with > pseudo-NMI where overflows could be lost. > > How often do you see the overflow being lost? > > Does RR set any of the perf_event_attr::exclude_* bits? If not, does RR > intentionally count events that occur within the kernel? Looking at the test, I see it sets perf_event_attr::exclude_kernel to 1, but doesn't set perf_event_attr::exclude_host or perf_event_attr::exclude_hv. I think the poorly-defined exclude_* bits are part of the problem here. Using your test as-is on my ThunderX2, I can reproduce the period being longer than expected by concurrently running the following in a shell: while true; do for C in $(seq 0 63); do taskset -c -p $C ${TEST_PID_HERE}; done; done > /dev/null ... resulting in: | [mark@gravadlaks:~]% ./counter-overflow | Pid 20060 running with period 10000 tolerance 1000 | Signal #1: last: 0, cur: 10292, max diff: 0 | Signal #415330: delta of 19999 is outside 10000 +/- 1000 | Signal #415330: last: 4153290187, cur: 4153310186, max diff: 10292 | Signal #489879: delta of 19998 is outside 10000 +/- 1000 | Signal #511842: delta of 20058 is outside 10000 +/- 1000 | Signal #511842: last: 5118430130, cur: 5118450188, max diff: 19999 However, if I modify the test to also set perf_event_attr::exclude_host=1, I do not see any lost overflows after many minutes. On VHE hosts (like the ThunderX2), the host kernel gets counted when perf_event_attr::exclude_host=0, even if perf_event_attr::exclude_kernel=1 (which I agree is surprising), so I think what's happening is the counters are counting in the host kernel, which isn't what RR actually wants regardless. >From looking at: https://github.com/rr-debugger/rr/blob/master/src/PerfCounters.cc ... it looks like RR only sets perf_event_attr::exclude_{kernel,guest}, and due to this I think it's still counting the host kernel (and potentially the hypervisor, depending on that hypervisor's policy for what it filters from the guest). There's still a potential race on the kernel side, but I suspect that if RR were to set the non-user exclude_* bits it could significantly reduce the chance of a lost overflow signal on existing kernels, at least on bare-metal. This is an annoying ABI; the interaction between all the bits is ill-defined, especially host/hv/kernel (since for VHE those are the same thing). Ideally this would have had the opposite polarity and you'd have only needed to set an include_user bit and be done with it. I'll continue to look at what we can do kernel-side, but I reckon it's worth having RR try the other exclude bits regardless, if that's possible? It would be interesting to know whether that helps you under a hypervisor. Thanks, Mark. > > When an event is not being explicitly torn down (e.g. being closed), this seems > > like an undesirable behavior. > > I agree it's undesirable, though my understanding is that for most other users > this isn't any worse than losing samples for other reasons (e.g. the perf ring > buffer being full), and for the general case of sample-based profiling, losing > a sample every so often isn't the end of the world. > > > I haven't attempted to demo it yet, but I suspect > > an application disabling an event temporarily could occasionally see the same > > missed overflow signals. Is my understanding here correct? > > That sounds right to me, though I haven't checked that end-to-end yet. > > > Does anyone have thoughts on how this could be addressed without creating > > other issues? > > We should be able to detect overflow from the counter value alone, so we might > be able to account for that when we actually read the event out, or when we > schedule it back in and reprogram the period. > > I'm not sure if we can reasonably do that when scheduling the event out, since > if we're switching tasks, that'll queue up IRQ work which will be triggered in > the context of the next task. > > We might be able to figure out that we have an overflow when we schedule the > event in under armpmu_start(), but I'll need to go digging to see if there are > any free-running counters as the comment above the call to > armpmu_event_set_period() says, or whether that's a historical artifact. > > I suspect we might need to restructure the code somewhat to be able to catch > overflows more consistently. I'll see if I can come up with something, but we > might not be able to guarantee this in all cases. > > Mark. > > > [1] https://github.com/rr-debugger/rr/issues/3607 > > [2] https://github.com/torvalds/linux/blob/c42d9eeef8e5ba9292eda36fd8e3c11f35ee065c/drivers/perf/arm_pmuv3.c#L652C20-L652C43 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BCD8FC4167B for ; Thu, 30 Nov 2023 11:49:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=jf8P1HQLnRv9WBW6c5w8I7Xz4zVAsYV/5DO+q3pfV/A=; b=eUVNGVMLw8b3/W 7bXJcmr8Xd5rYZTjFWYoXMZMXeQKZmEl9aV/l/kdDHPM4X3DMoDdNhlIGrcsPzw3hkVGjKq6Q4UYr 3CZwC/eqMp/B+unTOdVT01zWzwqEHQFAFr577PjGu8r1XdrzYfu9H9bwZNKgPS04FjGnbXgNUMJYn 6YdMjpToJKWy40t3aYxB3VKLiHRpGogK7JFDdaq8oTxHbNwxRxSA2e0yM5VbOcwHfZx1wAKWGMEWS FUQXumfebXb8bGr/KYbVqhjkrPGWGOTSwh1MW+vAXZ5iEBcmXtDk0yb3tAh8FLQMzmjXfD1/jzjBf pYd1Nf5bgTruqaObkJXg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r8fXa-00AfEZ-2i; Thu, 30 Nov 2023 11:49:10 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1r8fXX-00AfBf-1O for linux-arm-kernel@lists.infradead.org; Thu, 30 Nov 2023 11:49:09 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 94F3F1042; Thu, 30 Nov 2023 03:49:51 -0800 (PST) Received: from FVFF77S0Q05N (unknown [10.57.43.250]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 128D03F5A1; Thu, 30 Nov 2023 03:49:03 -0800 (PST) Date: Thu, 30 Nov 2023 11:48:54 +0000 From: Mark Rutland To: "Ashley, William" Cc: "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "linux-perf-users@vger.kernel.org" , will@kernel.org Subject: Re: armv8pmu: Pending overflow interrupt is discarded when perf event is disabled Message-ID: References: <950001BD-490C-4BAC-8EEA-CDB9F7C4ADFC@amazon.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231130_034907_563078_879FC5B7 X-CRM114-Status: GOOD ( 58.13 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gV2VkLCBOb3YgMjksIDIwMjMgYXQgMDQ6MzU6MDFQTSArMDAwMCwgTWFyayBSdXRsYW5kIHdy b3RlOgo+IEhpIFdpbGxpYW0sCj4gCj4gT24gTW9uLCBOb3YgMjAsIDIwMjMgYXQgMTA6MzI6MTBQ TSArMDAwMCwgQXNobGV5LCBXaWxsaWFtIHdyb3RlOgo+ID4gQWRkaW5nIGxpbnV4LWFybS1rZXJu ZWxAbGlzdHMuaW5mcmFkZWFkLm9yZyBhbmQgbGludXgta2VybmVsQHZnZXIua2VybmVsLm9yZywK PiA+IHNvcnJ5IGZvciB0aGUgbm9pc2UuCj4gCj4gVGhhbmtzIGZvciB0aGlzIQo+IAo+IEZvciB0 aGUgYmVuZWZpdCBvZiBvdGhlcnMsIHRoZSBvcmlnaW5hbCBtYWlsIChhbmQgYXR0YWNobWVudCkg Y2FuIGJlIGZvdW5kIGF0Ogo+IAo+ICAgaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvbGludXgtcGVy Zi11c2Vycy85NTAwMDFCRC00OTBDLTRCQUMtOEVFQS1DREI5RjdDNEFERkNAYW1hem9uLmNvbS8K PiAKPiBGb3Igc29tZSByZWFzb24sIGxpbmtzIGFuZCB3aGl0ZXNwYWNlIHNlZW0gdG8gaGF2ZSBi ZWVuIG1hbmdsZWQgaW4gdGhlIHJlc2VuZDsKPiBJJ20gbm90IHN1cmUgd2hhdCBoYXBwZW5lZCB0 aGVyZS4KPiAKPiBJJ3ZlIGFkZGVkIFdpbGwgRGVhY29uLCBhcyBXaWxsIGFuZCBJIGNvLW1haW50 YWluIHRoZSBBUk0gUE1VIGRyaXZlcnMuCj4gCj4gPiDvu79PbiAxMS8yMC8yMywgMTI6MzYgUE0s ICJBc2hsZXksIFdpbGxpYW0iIDx3YXNoQGFtYXpvbi5jb20gPG1haWx0bzp3YXNoQGFtYXpvbi5j b20+PiB3cm90ZToKPiA+IAo+ID4gCj4gPiBBbiBpc3N1ZSBbMV0gd2FzIG9wZW5lZCBpbiB0aGUg cnItZGVidWdnZXIgcHJvamVjdCByZXBvcnRpbmcgb2NjYXNpb25hbCBtaXNzZWQKPiA+IHBlcmYg ZXZlbnQgb3ZlcmZsb3cgc2lnbmFscyBvbiBhcm02NC4gSSd2ZSBiZWVuIGRpZ2dpbmcgaW50byB0 aGlzIGFuZCB0aGluayBJCj4gPiB1bmRlcnN0YW5kIHdoYXQncyBoYXBwZW5pbmcsIGJ1dCB3YW50 ZWQgdG8gY29uZmlybSBteSB1bmRlcnN0YW5kaW5nLgo+ID4gCj4gPiBUaGUgYXR0YWNoZWQgZXhh bXBsZSBhcHBsaWNhdGlvbiwgZGVyaXZlZCBmcm9tIGFuIHJyLWRlYnVnZ2VyIHRlc3QgY2FzZSwg cmVwb3J0cwo+ID4gd2hlbiB0aGUgdmFsdWUgb2YgYSBjb3VudGVyIGRvZXNuJ3QgaW5jcmVhc2Ug YnkgdGhlIGV4cGVjdGVkIHBlcmlvZCArLy0gc29tZQo+ID4gdG9sZXJhbmNlLiBXaGVuIGl0IGlz IHBpbmctcG9uZ2VkIGJldHdlZW4gY29yZXMgKGUuZy4gd2l0aCB0YXNrc2V0KSBhdCBhIGhpZ2gK PiA+IGZyZXF1ZW5jeSwgaXQgZnJlcXVlbnRseSByZXBvcnRzIGluY3JlYXNlcyBvZiB+MnggdGhl IGV4cGVjdGVkLiBJJ3ZlIGNvbmZpcm1lZAo+ID4gdGhpcyBzYW1lIGJlaGF2aW9yIG9uIGtlcm5l bHMgNS40LCA1LjEwLCA2LjEgYW5kIDYuNS4KPiA+IAo+ID4gCj4gPiBJIGZvdW5kIGFybXY4cG11 X2Rpc2FibGVfaW50ZW5zIFsyXSB0aGF0IGlzIGNhbGxlZCBhcyBwYXJ0IG9mIGV2ZW50Cj4gPiBk ZS1zY2hlZHVsaW5nIGFuZCBjb250YWlucwo+ID4gLyogQ2xlYXIgdGhlIG92ZXJmbG93IGZsYWcg aW4gY2FzZSBhbiBpbnRlcnJ1cHQgaXMgcGVuZGluZy4gKi8KPiA+IHdyaXRlX3Btb3ZzY2xyKG1h c2spOwo+ID4gd2hpY2ggcmVzdWx0cyBpbiBhbnkgcGVuZGluZyBvdmVyZmxvdyBpbnRlcnJ1cHQg YmVpbmcgZHJvcHBlZC4gSSBhZGRlZCBzb21lCj4gPiBkZWJ1ZyBvdXRwdXQgaGVyZSBhbmQgaW5k ZWVkIHRoZXJlIGlzIGEgY29ycmVsYXRpb24gb2YgdGhpcyBiaXQgYmVpbmcgaGlnaCBhdAo+ID4g dGhlIHBvaW50IG9mIHRoZSBhYm92ZSBjb2RlIGFuZCB0aGUgcmVwcm9kdWNlciBpZGVudGlmeWlu ZyBhIG1pc3NlZCBzaWduYWwuCj4gCj4gSSB0aGluayB5b3UncmUgcmlnaHQgdGhhdCBpZiB3ZSBo YWQgYW4gb3ZlcmZsb3cgYXNzZXJ0ZWQgYXQgdGhpcyBwb2ludCwgd2UnbGwKPiB0aHJvdyBhd2F5 IHRoZSBvY2N1cnJlbmNlIG9mIHRoZSBvdmVyZmxvdyAoYW5kIHNvIG5vdCBjYWxsCj4gcGVyZl9l dmVudF9vdmVyZmxvdygpIGFuZCBnZW5lcmF0ZSBhIHNhbXBsZSwgZXRjKS4KPiAKPiBJdCBsb29r cyBsaWtlIHdlIG9ubHkgbG9zZSB0aGUgb2NjdXJyZW5jZSBvZiB0aGUgb3ZlcmZsb3c7IHRoZSBh Y3R1YWwgY291bnRzCj4gd2lsbCBnZXQgc2FtcGxlZCBjb3JyZWN0bHkgYW5kIHdoZW4gd2UgbmV4 dCByZXByb2dyYW0gdGhlIGV2ZW50LAo+IGFybXBtdV9ldmVudF9zZXRfcGVyaW9kKCkgc2hvdWxk IHNldCB1cCB0aGUgbmV4dCBvdmVyZmxvdyBwZXJpb2QuCj4gCj4gPiBUaGlzIGJlaGF2aW9yIGRv ZXMgbm90IG9jY3VyIHdpdGggcHNldWRvLU5NSXMgKGlycWNoaXAuZ2ljdjNfcHNldWRvX25taT0x KQo+ID4gZW5hYmxlZC4KPiAKPiBUaGF0J3MgaW50ZXJlc3RpbmcsIGJlY2F1c2UgaXQgaW1wbGll cyB0aGF0IHRoZSBQTVUgb3ZlcmZsb3cgaW50ZXJydXB0IGlzIGJlaW5nCj4gcmVjb2duaXNlZCBi eSB0aGUgQ1BVIHdoaWxlIHJlZ3VsYXIgaW50ZXJydXB0cyBhcmUgZGlzYWJsZWQuIFRoZXJlIGFy ZSBzb21lCj4gbmFycm93IHJhY2VzIHdoZXJlIHRoYXQgY291bGQgb2NjdXIgKGUuZy4gdGFraW5n IGEgdGltZXIgb3Igc2NoZWR1bGVyIElSUQo+ICpqdXN0KiBhcyBhbiBvdmVyZmxvdyBvY2N1cnMp LCBhbmQgc29tZSBvdGhlciBjYXNlcyBJJ2QgZXhwZWN0IFJSIHRvIGF2b2lkIGJ5Cj4gY29uc3Ry dWN0aW9uIChlLmcuIGlmIFJSIGlzbid0IHVzaW5nIG1vZGUgZXhjbHVzaW9uIGFuZCBhbHNvIGNv dW50cyBrZXJuZWwKPiBldmVudHMpLiBJdCdzIGFsc28gd29ydGggbm90aW5nIHRoYXQgdGhpcyBt ZWFucyB0aGVyZSBhcmUgcmFjZXMgZXZlbiB3aXRoCj4gcHNldWRvLU5NSSB3aGVyZSBvdmVyZmxv d3MgY291bGQgYmUgbG9zdC4KPiAKPiBIb3cgb2Z0ZW4gZG8geW91IHNlZSB0aGUgb3ZlcmZsb3cg YmVpbmcgbG9zdD8KPiAKPiBEb2VzIFJSIHNldCBhbnkgb2YgdGhlIHBlcmZfZXZlbnRfYXR0cjo6 ZXhjbHVkZV8qIGJpdHM/IElmIG5vdCwgZG9lcyBSUgo+IGludGVudGlvbmFsbHkgY291bnQgZXZl bnRzIHRoYXQgb2NjdXIgd2l0aGluIHRoZSBrZXJuZWw/CgpMb29raW5nIGF0IHRoZSB0ZXN0LCBJ IHNlZSBpdCBzZXRzIHBlcmZfZXZlbnRfYXR0cjo6ZXhjbHVkZV9rZXJuZWwgdG8gMSwgYnV0CmRv ZXNuJ3Qgc2V0IHBlcmZfZXZlbnRfYXR0cjo6ZXhjbHVkZV9ob3N0IG9yIHBlcmZfZXZlbnRfYXR0 cjo6ZXhjbHVkZV9odi4gSQp0aGluayB0aGUgcG9vcmx5LWRlZmluZWQgZXhjbHVkZV8qIGJpdHMg YXJlIHBhcnQgb2YgdGhlIHByb2JsZW0gaGVyZS4KClVzaW5nIHlvdXIgdGVzdCBhcy1pcyBvbiBt eSBUaHVuZGVyWDIsIEkgY2FuIHJlcHJvZHVjZSB0aGUgcGVyaW9kIGJlaW5nIGxvbmdlcgp0aGFu IGV4cGVjdGVkIGJ5IGNvbmN1cnJlbnRseSBydW5uaW5nIHRoZSBmb2xsb3dpbmcgaW4gYSBzaGVs bDoKCiAgd2hpbGUgdHJ1ZTsgZG8KICAgIGZvciBDIGluICQoc2VxIDAgNjMpOyBkbwogICAgICB0 YXNrc2V0IC1jIC1wICRDICR7VEVTVF9QSURfSEVSRX07CiAgICBkb25lOyAKICBkb25lID4gL2Rl di9udWxsCgouLi4gcmVzdWx0aW5nIGluOgoKfCBbbWFya0BncmF2YWRsYWtzOn5dJSAuL2NvdW50 ZXItb3ZlcmZsb3cgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCnwgUGlkIDIw MDYwIHJ1bm5pbmcgd2l0aCBwZXJpb2QgMTAwMDAgdG9sZXJhbmNlIDEwMDAKfCBTaWduYWwgIzE6 IGxhc3Q6IDAsIGN1cjogMTAyOTIsIG1heCBkaWZmOiAwCnwgU2lnbmFsICM0MTUzMzA6IGRlbHRh IG9mIDE5OTk5IGlzIG91dHNpZGUgMTAwMDAgKy8tIDEwMDAKfCBTaWduYWwgIzQxNTMzMDogbGFz dDogNDE1MzI5MDE4NywgY3VyOiA0MTUzMzEwMTg2LCBtYXggZGlmZjogMTAyOTIKfCBTaWduYWwg IzQ4OTg3OTogZGVsdGEgb2YgMTk5OTggaXMgb3V0c2lkZSAxMDAwMCArLy0gMTAwMAp8IFNpZ25h bCAjNTExODQyOiBkZWx0YSBvZiAyMDA1OCBpcyBvdXRzaWRlIDEwMDAwICsvLSAxMDAwCnwgU2ln bmFsICM1MTE4NDI6IGxhc3Q6IDUxMTg0MzAxMzAsIGN1cjogNTExODQ1MDE4OCwgbWF4IGRpZmY6 IDE5OTk5CgpIb3dldmVyLCBpZiBJIG1vZGlmeSB0aGUgdGVzdCB0byBhbHNvIHNldCBwZXJmX2V2 ZW50X2F0dHI6OmV4Y2x1ZGVfaG9zdD0xLCBJIGRvCm5vdCBzZWUgYW55IGxvc3Qgb3ZlcmZsb3dz IGFmdGVyIG1hbnkgbWludXRlcy4gT24gVkhFIGhvc3RzIChsaWtlIHRoZQpUaHVuZGVyWDIpLCB0 aGUgaG9zdCBrZXJuZWwgZ2V0cyBjb3VudGVkIHdoZW4gcGVyZl9ldmVudF9hdHRyOjpleGNsdWRl X2hvc3Q9MCwKZXZlbiBpZiBwZXJmX2V2ZW50X2F0dHI6OmV4Y2x1ZGVfa2VybmVsPTEgKHdoaWNo IEkgYWdyZWUgaXMgc3VycHJpc2luZyksIHNvIEkKdGhpbmsgd2hhdCdzIGhhcHBlbmluZyBpcyB0 aGUgY291bnRlcnMgYXJlIGNvdW50aW5nIGluIHRoZSBob3N0IGtlcm5lbCwgd2hpY2gKaXNuJ3Qg d2hhdCBSUiBhY3R1YWxseSB3YW50cyByZWdhcmRsZXNzLgoKRnJvbSBsb29raW5nIGF0OgoKICBo dHRwczovL2dpdGh1Yi5jb20vcnItZGVidWdnZXIvcnIvYmxvYi9tYXN0ZXIvc3JjL1BlcmZDb3Vu dGVycy5jYwoKLi4uIGl0IGxvb2tzIGxpa2UgUlIgb25seSBzZXRzIHBlcmZfZXZlbnRfYXR0cjo6 ZXhjbHVkZV97a2VybmVsLGd1ZXN0fSwgYW5kIGR1ZQp0byB0aGlzIEkgdGhpbmsgaXQncyBzdGls bCBjb3VudGluZyB0aGUgaG9zdCBrZXJuZWwgKGFuZCBwb3RlbnRpYWxseSB0aGUKaHlwZXJ2aXNv ciwgZGVwZW5kaW5nIG9uIHRoYXQgaHlwZXJ2aXNvcidzIHBvbGljeSBmb3Igd2hhdCBpdCBmaWx0 ZXJzIGZyb20gdGhlCmd1ZXN0KS4KClRoZXJlJ3Mgc3RpbGwgYSBwb3RlbnRpYWwgcmFjZSBvbiB0 aGUga2VybmVsIHNpZGUsIGJ1dCBJIHN1c3BlY3QgdGhhdCBpZiBSUgp3ZXJlIHRvIHNldCB0aGUg bm9uLXVzZXIgZXhjbHVkZV8qIGJpdHMgaXQgY291bGQgc2lnbmlmaWNhbnRseSByZWR1Y2UgdGhl CmNoYW5jZSBvZiBhIGxvc3Qgb3ZlcmZsb3cgc2lnbmFsIG9uIGV4aXN0aW5nIGtlcm5lbHMsIGF0 IGxlYXN0IG9uIGJhcmUtbWV0YWwuCgpUaGlzIGlzIGFuIGFubm95aW5nIEFCSTsgdGhlIGludGVy YWN0aW9uIGJldHdlZW4gYWxsIHRoZSBiaXRzIGlzIGlsbC1kZWZpbmVkLAplc3BlY2lhbGx5IGhv c3QvaHYva2VybmVsIChzaW5jZSBmb3IgVkhFIHRob3NlIGFyZSB0aGUgc2FtZSB0aGluZykuIElk ZWFsbHkKdGhpcyB3b3VsZCBoYXZlIGhhZCB0aGUgb3Bwb3NpdGUgcG9sYXJpdHkgYW5kIHlvdSdk IGhhdmUgb25seSBuZWVkZWQgdG8gc2V0IGFuCmluY2x1ZGVfdXNlciBiaXQgYW5kIGJlIGRvbmUg d2l0aCBpdC4KCkknbGwgY29udGludWUgdG8gbG9vayBhdCB3aGF0IHdlIGNhbiBkbyBrZXJuZWwt c2lkZSwgYnV0IEkgcmVja29uIGl0J3Mgd29ydGgKaGF2aW5nIFJSIHRyeSB0aGUgb3RoZXIgZXhj bHVkZSBiaXRzIHJlZ2FyZGxlc3MsIGlmIHRoYXQncyBwb3NzaWJsZT8gSXQgd291bGQKYmUgaW50 ZXJlc3RpbmcgdG8ga25vdyB3aGV0aGVyIHRoYXQgaGVscHMgeW91IHVuZGVyIGEgaHlwZXJ2aXNv ci4KClRoYW5rcywKTWFyay4KCj4gPiBXaGVuIGFuIGV2ZW50IGlzIG5vdCBiZWluZyBleHBsaWNp dGx5IHRvcm4gZG93biAoZS5nLiBiZWluZyBjbG9zZWQpLCB0aGlzIHNlZW1zCj4gPiBsaWtlIGFu IHVuZGVzaXJhYmxlIGJlaGF2aW9yLgo+IAo+IEkgYWdyZWUgaXQncyB1bmRlc2lyYWJsZSwgdGhv dWdoIG15IHVuZGVyc3RhbmRpbmcgaXMgdGhhdCBmb3IgbW9zdCBvdGhlciB1c2Vycwo+IHRoaXMg aXNuJ3QgYW55IHdvcnNlIHRoYW4gbG9zaW5nIHNhbXBsZXMgZm9yIG90aGVyIHJlYXNvbnMgKGUu Zy4gdGhlIHBlcmYgcmluZwo+IGJ1ZmZlciBiZWluZyBmdWxsKSwgYW5kIGZvciB0aGUgZ2VuZXJh bCBjYXNlIG9mIHNhbXBsZS1iYXNlZCBwcm9maWxpbmcsIGxvc2luZwo+IGEgc2FtcGxlIGV2ZXJ5 IHNvIG9mdGVuIGlzbid0IHRoZSBlbmQgb2YgdGhlIHdvcmxkLgo+IAo+ID4gSSBoYXZlbid0IGF0 dGVtcHRlZCB0byBkZW1vIGl0IHlldCwgYnV0IEkgc3VzcGVjdAo+ID4gYW4gYXBwbGljYXRpb24g ZGlzYWJsaW5nIGFuIGV2ZW50IHRlbXBvcmFyaWx5IGNvdWxkIG9jY2FzaW9uYWxseSBzZWUgdGhl IHNhbWUKPiA+IG1pc3NlZCBvdmVyZmxvdyBzaWduYWxzLiBJcyBteSB1bmRlcnN0YW5kaW5nIGhl cmUgY29ycmVjdD8KPiAKPiBUaGF0IHNvdW5kcyByaWdodCB0byBtZSwgdGhvdWdoIEkgaGF2ZW4n dCBjaGVja2VkIHRoYXQgZW5kLXRvLWVuZCB5ZXQuCj4gCj4gPiBEb2VzIGFueW9uZSBoYXZlIHRo b3VnaHRzIG9uIGhvdyB0aGlzIGNvdWxkIGJlIGFkZHJlc3NlZCB3aXRob3V0IGNyZWF0aW5nCj4g PiBvdGhlciBpc3N1ZXM/Cj4gCj4gV2Ugc2hvdWxkIGJlIGFibGUgdG8gZGV0ZWN0IG92ZXJmbG93 IGZyb20gdGhlIGNvdW50ZXIgdmFsdWUgYWxvbmUsIHNvIHdlIG1pZ2h0Cj4gYmUgYWJsZSB0byBh Y2NvdW50IGZvciB0aGF0IHdoZW4gd2UgYWN0dWFsbHkgcmVhZCB0aGUgZXZlbnQgb3V0LCBvciB3 aGVuIHdlCj4gc2NoZWR1bGUgaXQgYmFjayBpbiBhbmQgcmVwcm9ncmFtIHRoZSBwZXJpb2QuCj4g Cj4gSSdtIG5vdCBzdXJlIGlmIHdlIGNhbiByZWFzb25hYmx5IGRvIHRoYXQgd2hlbiBzY2hlZHVs aW5nIHRoZSBldmVudCBvdXQsIHNpbmNlCj4gaWYgd2UncmUgc3dpdGNoaW5nIHRhc2tzLCB0aGF0 J2xsIHF1ZXVlIHVwIElSUSB3b3JrIHdoaWNoIHdpbGwgYmUgdHJpZ2dlcmVkIGluCj4gdGhlIGNv bnRleHQgb2YgdGhlIG5leHQgdGFzay4KPiAKPiBXZSBtaWdodCBiZSBhYmxlIHRvIGZpZ3VyZSBv dXQgdGhhdCB3ZSBoYXZlIGFuIG92ZXJmbG93IHdoZW4gd2Ugc2NoZWR1bGUgdGhlCj4gZXZlbnQg aW4gdW5kZXIgYXJtcG11X3N0YXJ0KCksIGJ1dCBJJ2xsIG5lZWQgdG8gZ28gZGlnZ2luZyB0byBz ZWUgaWYgdGhlcmUgYXJlCj4gYW55IGZyZWUtcnVubmluZyBjb3VudGVycyBhcyB0aGUgY29tbWVu dCBhYm92ZSB0aGUgY2FsbCB0bwo+IGFybXBtdV9ldmVudF9zZXRfcGVyaW9kKCkgc2F5cywgb3Ig d2hldGhlciB0aGF0J3MgYSBoaXN0b3JpY2FsIGFydGlmYWN0Lgo+IAo+IEkgc3VzcGVjdCB3ZSBt aWdodCBuZWVkIHRvIHJlc3RydWN0dXJlIHRoZSBjb2RlIHNvbWV3aGF0IHRvIGJlIGFibGUgdG8g Y2F0Y2gKPiBvdmVyZmxvd3MgbW9yZSBjb25zaXN0ZW50bHkuIEknbGwgc2VlIGlmIEkgY2FuIGNv bWUgdXAgd2l0aCBzb21ldGhpbmcsIGJ1dCB3ZQo+IG1pZ2h0IG5vdCBiZSBhYmxlIHRvIGd1YXJh bnRlZSB0aGlzIGluIGFsbCBjYXNlcy4KPiAKPiBNYXJrLgo+IAo+ID4gWzFdIGh0dHBzOi8vZ2l0 aHViLmNvbS9yci1kZWJ1Z2dlci9yci9pc3N1ZXMvMzYwNyA8aHR0cHM6Ly9naXRodWIuY29tL3Jy LWRlYnVnZ2VyL3JyL2lzc3Vlcy8zNjA3Pgo+ID4gWzJdIGh0dHBzOi8vZ2l0aHViLmNvbS90b3J2 YWxkcy9saW51eC9ibG9iL2M0MmQ5ZWVlZjhlNWJhOTI5MmVkYTM2ZmQ4ZTNjMTFmMzVlZTA2NWMv ZHJpdmVycy9wZXJmL2FybV9wbXV2My5jI0w2NTJDMjAtTDY1MkM0MyA8aHR0cHM6Ly9naXRodWIu Y29tL3RvcnZhbGRzL2xpbnV4L2Jsb2IvYzQyZDllZWVmOGU1YmE5MjkyZWRhMzZmZDhlM2MxMWYz NWVlMDY1Yy9kcml2ZXJzL3BlcmYvYXJtX3BtdXYzLmMjTDY1MkMyMC1MNjUyQzQzPgoKX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGludXgtYXJtLWtlcm5l bCBtYWlsaW5nIGxpc3QKbGludXgtYXJtLWtlcm5lbEBsaXN0cy5pbmZyYWRlYWQub3JnCmh0dHA6 Ly9saXN0cy5pbmZyYWRlYWQub3JnL21haWxtYW4vbGlzdGluZm8vbGludXgtYXJtLWtlcm5lbAo=