From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from canpmsgout05.his.huawei.com (canpmsgout05.his.huawei.com [113.46.200.220]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 032973AFCFC; Wed, 24 Jun 2026 10:00:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=113.46.200.220 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782295258; cv=none; b=qFBI/XMfyURM+pzJf8mSprqnEiqOS6/2cfQguuOOdweJnLcJ1RKra0tQCGzUAhM0ywwkJaownqvl8/kxpOdrhpI87QtomNK7zygfwmE6CB7sQOrrTx1rPOq/S4t8lTE+GMYr0bUFixj9ju0bX1QtjMTPItzNqe8uaB/HX+aUQbI= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1782295258; c=relaxed/simple; bh=dJJUXV6NeuW1qn0h1mROj4BDOaxxeX2+im4Q4TptAlg=; h=Message-ID:Date:MIME-Version:Subject:To:CC:References:From: In-Reply-To:Content-Type; b=IET2TCKMJjqYhOdG+CcHsOaKeu/gNZgHMDJZay9DRdahghGGw/jMtyEiQCvQPucjReYEK8Ww49e33Ba/Md91S3SC1eKFTPbJrpa+qD+36eKqYOG0w8QB8c5bQQN7FP4Aemh9PtlNImJpjCZO7xSq3pZvl/EbS/38e3cM8Qnwe2A= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com; spf=pass smtp.mailfrom=huawei.com; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b=qlRqlki8; arc=none smtp.client-ip=113.46.200.220 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=huawei.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=huawei.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=huawei.com header.i=@huawei.com header.b="qlRqlki8" dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=1395hZTLdzjvGEUA5mY2MXbrSyHdvWfUix1ZT9yBBvU=; b=qlRqlki8Vete58SPLgzXr/AdeIESPRlfvmDp+qvgdcQ9hdmA0UUGgAXDOKcsXvSaxXgIiKOXt 3DjyBcgH9e3sBFpLQEQzJHUd57n5Xz+/Cmhtv/VSkMbJOFp8Ulw3OyOt2bVX4hKemQQiKbc1Qfp 7EHlN4/awC+2PYDruCMPWZw= Received: from mail.maildlp.com (unknown [172.19.162.223]) by canpmsgout05.his.huawei.com (SkyGuard) with ESMTPS id 4glcfW6d8Rz12LCg; Wed, 24 Jun 2026 17:52:03 +0800 (CST) Received: from dggpemf500011.china.huawei.com (unknown [7.185.36.131]) by mail.maildlp.com (Postfix) with ESMTPS id F21A540575; Wed, 24 Jun 2026 18:00:50 +0800 (CST) Received: from [10.67.109.254] (10.67.109.254) by dggpemf500011.china.huawei.com (7.185.36.131) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Wed, 24 Jun 2026 18:00:48 +0800 Message-ID: Date: Wed, 24 Jun 2026 18:00:47 +0800 Precedence: bulk X-Mailing-List: linux-mips@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC 3/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs To: Will Deacon CC: Michael Kelley , "catalin.marinas@arm.com" , "tsbogend@alpha.franken.de" , "pjw@kernel.org" , "palmer@dabbelt.com" , "aou@eecs.berkeley.edu" , "alex@ghiti.fr" , "tglx@kernel.org" , "mingo@redhat.com" , "bp@alien8.de" , "dave.hansen@linux.intel.com" , "hpa@zytor.com" , "peterz@infradead.org" , "kees@kernel.org" , "nathan@kernel.org" , "linusw@kernel.org" , "ojeda@kernel.org" , "david.kaplan@amd.com" , "lukas.bulwahn@redhat.com" , "ryan.roberts@arm.com" , "maz@kernel.org" , "timothy.hayes@arm.com" , "lpieralisi@kernel.org" , "thuth@redhat.com" , "oupton@kernel.org" , "yeoreum.yun@arm.com" , "miko.lenczewski@arm.com" , "broonie@kernel.org" , "kevin.brodsky@arm.com" , "james.clark@linaro.org" , "tabba@google.com" , "mrigendra.chaubey@gmail.com" , "arnd@arndb.de" , "anshuman.khandual@arm.com" , "x86@kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "linux-riscv@lists.infradead.org" References: <20260611133809.3854977-1-ruanjinjie@huawei.com> <20260611133809.3854977-4-ruanjinjie@huawei.com> <2a34ff43-78a0-4d99-885a-0e7f560634ca@huawei.com> From: Jinjie Ruan In-Reply-To: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 8bit X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500011.china.huawei.com (7.185.36.131) On 6/23/2026 10:30 PM, Will Deacon wrote: > On Mon, Jun 22, 2026 at 04:06:38PM +0800, Jinjie Ruan wrote: >> On 6/18/2026 8:21 PM, Will Deacon wrote: >>> On Mon, Jun 15, 2026 at 04:51:48PM +0800, Jinjie Ruan wrote: >>>> On 6/12/2026 11:45 PM, Michael Kelley wrote: >>>>> From: Jinjie Ruan Sent: Thursday, June 11, 2026 6:38 AM >>>>>> >>>>>> Support for parallel secondary CPU bringup is already utilized by x86, >>>>>> MIPS, and RISC-V. This patch brings this capability to the arm64 >>>>>> architecture. >>>>>> >>>>>> Rework the global `secondary_data` accessed during early boot into >>>>>> a per-CPU array. This array maps logical CPU IDs to MPIDR_EL1 values, >>>>>> enabling the early boot code in head.S to resolve each secondary CPU's >>>>>> logical ID concurrently. >>>>>> >>>>>> To fully enable HOTPLUG_PARALLEL, this patch implements: >>>>>> 1) An arm64-specific arch_cpuhp_kick_ap_alive() handler. >>>>>> 2) Callbacks to cpuhp_ap_sync_alive() inside secondary_start_kernel(). >>>>>> >>>>>> Successfully tested on QEMU ARM64 virt machine (KVM on, 128 vCPUs). >>>>>> >>>>>> | test kernel | secondary CPUs boot time | >>>>>> | --------------------- | -------------------- | >>>>>> | Without this patch | 155.672 | >>>>>> | cpuhp.parallel=0 | 62.897 | >>>>>> | cpuhp.parallel=1 | 166.703 | >>>>> >>>>> The last two rows seem mixed up. I would expect parallel=0 to >>>>> result in a longer boot time. >>>> >>>> The results are correct and not mixed up. >>>> >>>> Compared to the original non‑HOTPLUG_PARALLEL approach, the advantage of >>>> cpuhp.parallel=0 lies in its use of cpu_relax(`yield` on arm64) instead >>>> of the wait_for_completion_timeout() mechanism (which may cause sleep >>>> and context switching). This significantly reduces the overhead of VM >>>> exits and context switches in a KVM guest, thereby cutting the secondary >>>> CPU boot time by more than half. >>> >>> I don't think that's a particularly compelling reason to enable this for >>> arm64, in all honesty. The yield instruction typically doesn't do >>> anything on actual arm64 silicon, so this probably means that you're >>> introducing busy-loops which tend to be bad for power and scalability. >> >> After updating the implementation in v2, the performance gains are >> primarily observed on actual hardware. > > ... but that's presumably because the secondary cores are busy-looping. > That's not something we should do during boot. It might be "fast" on > your machine but it will probably be "hot" as well. Hi, Will, I see your point regarding the 'hot boot' issue, which is indeed a valid concern for power-constrained devices, My optimization is tailored for servers and continuously powered single boards, where boot-up speed is much more critical than temporary power usage during the early boot phase. Perhaps we could replace the "yield" instruction with "WFE / SEV" instructions to coordinate the parallel boot of the primary and secondary cores. This approach would allow the secondary cores to enter a low-power standby state rather than busy-looping, effectively preventing the thermal and power issues on battery-constrained machines. > >>> I implemented this a while ago [1] but didn't manage to see much in terms >>> of performance improvement and so I didn't bother to send the patches out >> >> As shown in v2 below, on actual hardware, this results in a 40%–60% >> reduction in boot time. >> >> Bringup Time Comparison (ms, lower is better): >> >> | Platform | Baseline| P=0 | P=1 | Delta(%)| >> | --------------------- | ------- | ------- | ------ | ------- | >> | 64-core ATF QEMU | 2075.8 | 2080.7 | 1653.4 | 20.34% | >> | 192-core server(HIP12)| 14619.2 | 14619.1 | 8589.4 | 41.21% | >> | 32-core board | 2776.5 | 2881.0 | 1045.0 | 62.36% | >> >> Link: >> https://lore.kernel.org/all/20260618092444.1316336-5-ruanjinjie@huawei.com/ > > To be honest, I'm pretty confused with all these numbers. Your first > table above suggests that parallel boot is *slower* but then this table > suggests the opposite. However, it also has a QEMU entry despite being > "on actual hardware". Is that in a VM? Sorry, there is a little confused. 192-core server(HIP12) and 32-core board are tested on real hardware, which has 40%–60% reduction in boot time. > >>> after talking about it at KVM forum [2]. However, as mentioned at the end >>> of that talk, it _is_ still useful for confidential VMs using PSCI so >>> let me dust off my old series and send it out to see what you think. >>> >>> It relies on PSCI v0.2, which means we don't need the NR_CPUS size array >>> for secondary_data and I also have some support for error handling (it >>> doesn't look like you handle __early_cpu_boot_status properly). >> >> I need some time to look closely at your patch. Alternatively, I will >> integrate your changes, re-test everything on actual hardware, and then >> send out a revised version. > > Please just give me a week or so to rebase my changes and send them out > for discussion. It'll be interesting to see what numbers you get. Sounds good! Take your time, and I'm looking forward to your series. In the meantime, I have just sent out v3 of this patch. While working closely with your previous code, I identified a few bugs (including the multi-CPU status trampling issue we discussed) and addressed them in this new version.I wanted to share v3 with you now so you can easily review the fixes and potentially integrate them when you rebase your changes next week. It also includes the updated performance numbers on my setup for your reference. Link: https://lore.kernel.org/all/20260624092537.2916971-1-ruanjinjie@huawei.com/ Looking forward to the discussion! Best regards, Jinjie > >> It seems that the following patch removing >> `rcutree_report_cpu_starting()` will reintroduce the original issue as >> commit ce3d31ad3cac ("arm64/smp: Move >> rcu_cpu_starting() earlier") soloved. >> >> Link: >> https://web.git.kernel.org/pub/scm/linux/kernel/git/will/linux.git/commit/?h=cpu-hotplug&id=bba4b62f45f2614bf6085e6cd3f233528f85bf26 >> >> Indeed, I also noticed that the invocation order of >> rcutree_report_cpu_starting() on arm64 is somewhat suboptimal. It >> hinders the implementation of parallel bringup on arm64 and could >> potentially lead to RCU stalls. >> >> Link: >> https://lore.kernel.org/all/20260618092444.1316336-4-ruanjinjie@huawei.com/ >> >> [ 0.329017] smp: Bringing up secondary CPUs ... >> [ 0.343628] Detected VIPT I-cache on CPU1 >> [ 0.343788] >> [ 0.343806] ============================= >> [ 0.343816] WARNING: suspicious RCU usage >> [ 0.343966] 7.1.0-rc1-g27c1871848a2 #109 Not tainted >> [ 0.344087] ----------------------------- >> [ 0.344098] kernel/locking/lockdep.c:3801 RCU-list traversed in >> non-reader section!! > > Thanks, I'll look into this. > > Will > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E61A1CDB47C for ; Wed, 24 Jun 2026 10:01:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:From:References:CC:To: Subject:MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=c3xMi2zQI0upMFwpE3wf2Mzscnt6aqyyEg5RyTH2Jy8=; b=JPLGTr/HlyCy7o F+epPzVsnc2dQRJnxseEzWAmBirLJRJsQdwzhkIWaE3yTTJx5kV7qPuv1jIgtQm7lJfU3wMRllril D6EmbI7lj2XnPMrvXEcccCZPg2pvgnIKsq/PNlriXoESGcZZ5uMiqMgIYyVXHtHF0Ih5mbzUTFKVn Vnnms4BfiIvnwunk6SvwC6f9YkvF4oL0a4nnjzuTAO8KhlT9gZ9KE7n5NeEfYafok3FozJYxz+P1t ispy458WpfHdVXQIbNtbDFN1yfZUQbjsSIuzTs6o/qjjkrRHrhYFIhILXxV8q3qjAvN+KJY3eVngx jd+T3YAxHoahbX7l/0kQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.99.1 #2 (Red Hat Linux)) id 1wcKPn-00000007YSR-2eZA; Wed, 24 Jun 2026 10:01:03 +0000 Received: from canpmsgout05.his.huawei.com ([113.46.200.220]) by bombadil.infradead.org with esmtps (Exim 4.99.1 #2 (Red Hat Linux)) id 1wcKPh-00000007YQp-2Bma; Wed, 24 Jun 2026 10:01:01 +0000 dkim-signature: v=1; a=rsa-sha256; d=huawei.com; s=dkim; c=relaxed/relaxed; q=dns/txt; h=From; bh=1395hZTLdzjvGEUA5mY2MXbrSyHdvWfUix1ZT9yBBvU=; b=qlRqlki8Vete58SPLgzXr/AdeIESPRlfvmDp+qvgdcQ9hdmA0UUGgAXDOKcsXvSaxXgIiKOXt 3DjyBcgH9e3sBFpLQEQzJHUd57n5Xz+/Cmhtv/VSkMbJOFp8Ulw3OyOt2bVX4hKemQQiKbc1Qfp 7EHlN4/awC+2PYDruCMPWZw= Received: from mail.maildlp.com (unknown [172.19.162.223]) by canpmsgout05.his.huawei.com (SkyGuard) with ESMTPS id 4glcfW6d8Rz12LCg; Wed, 24 Jun 2026 17:52:03 +0800 (CST) Received: from dggpemf500011.china.huawei.com (unknown [7.185.36.131]) by mail.maildlp.com (Postfix) with ESMTPS id F21A540575; Wed, 24 Jun 2026 18:00:50 +0800 (CST) Received: from [10.67.109.254] (10.67.109.254) by dggpemf500011.china.huawei.com (7.185.36.131) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.11; Wed, 24 Jun 2026 18:00:48 +0800 Message-ID: Date: Wed, 24 Jun 2026 18:00:47 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC 3/3] arm64: Add HOTPLUG_PARALLEL support for secondary CPUs To: Will Deacon CC: Michael Kelley , "catalin.marinas@arm.com" , "tsbogend@alpha.franken.de" , "pjw@kernel.org" , "palmer@dabbelt.com" , "aou@eecs.berkeley.edu" , "alex@ghiti.fr" , "tglx@kernel.org" , "mingo@redhat.com" , "bp@alien8.de" , "dave.hansen@linux.intel.com" , "hpa@zytor.com" , "peterz@infradead.org" , "kees@kernel.org" , "nathan@kernel.org" , "linusw@kernel.org" , "ojeda@kernel.org" , "david.kaplan@amd.com" , "lukas.bulwahn@redhat.com" , "ryan.roberts@arm.com" , "maz@kernel.org" , "timothy.hayes@arm.com" , "lpieralisi@kernel.org" , "thuth@redhat.com" , "oupton@kernel.org" , "yeoreum.yun@arm.com" , "miko.lenczewski@arm.com" , "broonie@kernel.org" , "kevin.brodsky@arm.com" , "james.clark@linaro.org" , "tabba@google.com" , "mrigendra.chaubey@gmail.com" , "arnd@arndb.de" , "anshuman.khandual@arm.com" , "x86@kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "linux-riscv@lists.infradead.org" References: <20260611133809.3854977-1-ruanjinjie@huawei.com> <20260611133809.3854977-4-ruanjinjie@huawei.com> <2a34ff43-78a0-4d99-885a-0e7f560634ca@huawei.com> From: Jinjie Ruan In-Reply-To: X-Originating-IP: [10.67.109.254] X-ClientProxiedBy: kwepems500001.china.huawei.com (7.221.188.70) To dggpemf500011.china.huawei.com (7.185.36.131) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.9.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20260624_030058_194726_0BA0FBB8 X-CRM114-Status: GOOD ( 34.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org CgpPbiA2LzIzLzIwMjYgMTA6MzAgUE0sIFdpbGwgRGVhY29uIHdyb3RlOgo+IE9uIE1vbiwgSnVu IDIyLCAyMDI2IGF0IDA0OjA2OjM4UE0gKzA4MDAsIEppbmppZSBSdWFuIHdyb3RlOgo+PiBPbiA2 LzE4LzIwMjYgODoyMSBQTSwgV2lsbCBEZWFjb24gd3JvdGU6Cj4+PiBPbiBNb24sIEp1biAxNSwg MjAyNiBhdCAwNDo1MTo0OFBNICswODAwLCBKaW5qaWUgUnVhbiB3cm90ZToKPj4+PiBPbiA2LzEy LzIwMjYgMTE6NDUgUE0sIE1pY2hhZWwgS2VsbGV5IHdyb3RlOgo+Pj4+PiBGcm9tOiBKaW5qaWUg UnVhbiA8cnVhbmppbmppZUBodWF3ZWkuY29tPiBTZW50OiBUaHVyc2RheSwgSnVuZSAxMSwgMjAy NiA2OjM4IEFNCj4+Pj4+Pgo+Pj4+Pj4gU3VwcG9ydCBmb3IgcGFyYWxsZWwgc2Vjb25kYXJ5IENQ VSBicmluZ3VwIGlzIGFscmVhZHkgdXRpbGl6ZWQgYnkgeDg2LAo+Pj4+Pj4gTUlQUywgYW5kIFJJ U0MtVi4gVGhpcyBwYXRjaCBicmluZ3MgdGhpcyBjYXBhYmlsaXR5IHRvIHRoZSBhcm02NAo+Pj4+ Pj4gYXJjaGl0ZWN0dXJlLgo+Pj4+Pj4KPj4+Pj4+IFJld29yayB0aGUgZ2xvYmFsIGBzZWNvbmRh cnlfZGF0YWAgYWNjZXNzZWQgZHVyaW5nIGVhcmx5IGJvb3QgaW50bwo+Pj4+Pj4gYSBwZXItQ1BV IGFycmF5LiBUaGlzIGFycmF5IG1hcHMgbG9naWNhbCBDUFUgSURzIHRvIE1QSURSX0VMMSB2YWx1 ZXMsCj4+Pj4+PiBlbmFibGluZyB0aGUgZWFybHkgYm9vdCBjb2RlIGluIGhlYWQuUyB0byByZXNv bHZlIGVhY2ggc2Vjb25kYXJ5IENQVSdzCj4+Pj4+PiBsb2dpY2FsIElEIGNvbmN1cnJlbnRseS4K Pj4+Pj4+Cj4+Pj4+PiBUbyBmdWxseSBlbmFibGUgSE9UUExVR19QQVJBTExFTCwgdGhpcyBwYXRj aCBpbXBsZW1lbnRzOgo+Pj4+Pj4gMSkgQW4gYXJtNjQtc3BlY2lmaWMgYXJjaF9jcHVocF9raWNr X2FwX2FsaXZlKCkgaGFuZGxlci4KPj4+Pj4+IDIpIENhbGxiYWNrcyB0byBjcHVocF9hcF9zeW5j X2FsaXZlKCkgaW5zaWRlIHNlY29uZGFyeV9zdGFydF9rZXJuZWwoKS4KPj4+Pj4+Cj4+Pj4+PiBT dWNjZXNzZnVsbHkgdGVzdGVkIG9uIFFFTVUgQVJNNjQgdmlydCBtYWNoaW5lIChLVk0gb24sIDEy OCB2Q1BVcykuCj4+Pj4+Pgo+Pj4+Pj4gfCAgICAgdGVzdCBrZXJuZWwJICAgfCBzZWNvbmRhcnkg Q1BVcyBib290IHRpbWUgfAo+Pj4+Pj4gfCAgLS0tLS0tLS0tLS0tLS0tLS0tLS0tICAgfAktLS0t LS0tLS0tLS0tLS0tLS0tLSAgfAo+Pj4+Pj4gfCAgIFdpdGhvdXQgdGhpcyBwYXRjaCAgICAgfAkJ MTU1LjY3MgkgICAgICB8Cj4+Pj4+PiB8ICAgY3B1aHAucGFyYWxsZWw9MAkgICB8CQk2Mi44OTcJ ICAgICAgfAo+Pj4+Pj4gfCAgIGNwdWhwLnBhcmFsbGVsPTEJICAgfAkJMTY2LjcwMwkgICAgICB8 Cj4+Pj4+Cj4+Pj4+IFRoZSBsYXN0IHR3byByb3dzIHNlZW0gbWl4ZWQgdXAuIEkgd291bGQgZXhw ZWN0IHBhcmFsbGVsPTAgdG8KPj4+Pj4gcmVzdWx0IGluIGEgbG9uZ2VyIGJvb3QgdGltZS4KPj4+ Pgo+Pj4+IFRoZSByZXN1bHRzIGFyZSBjb3JyZWN0IGFuZCBub3QgbWl4ZWQgdXAuCj4+Pj4KPj4+ PiBDb21wYXJlZCB0byB0aGUgb3JpZ2luYWwgbm9u4oCRSE9UUExVR19QQVJBTExFTCBhcHByb2Fj aCwgdGhlIGFkdmFudGFnZSBvZgo+Pj4+IGNwdWhwLnBhcmFsbGVsPTAgbGllcyBpbiBpdHMgdXNl IG9mIGNwdV9yZWxheChgeWllbGRgIG9uIGFybTY0KSBpbnN0ZWFkCj4+Pj4gb2YgdGhlIHdhaXRf Zm9yX2NvbXBsZXRpb25fdGltZW91dCgpIG1lY2hhbmlzbSAod2hpY2ggbWF5IGNhdXNlIHNsZWVw Cj4+Pj4gYW5kIGNvbnRleHQgc3dpdGNoaW5nKS4gVGhpcyBzaWduaWZpY2FudGx5IHJlZHVjZXMg dGhlIG92ZXJoZWFkIG9mIFZNCj4+Pj4gZXhpdHMgYW5kIGNvbnRleHQgc3dpdGNoZXMgaW4gYSBL Vk0gZ3Vlc3QsIHRoZXJlYnkgY3V0dGluZyB0aGUgc2Vjb25kYXJ5Cj4+Pj4gQ1BVIGJvb3QgdGlt ZSBieSBtb3JlIHRoYW4gaGFsZi4KPj4+Cj4+PiBJIGRvbid0IHRoaW5rIHRoYXQncyBhIHBhcnRp Y3VsYXJseSBjb21wZWxsaW5nIHJlYXNvbiB0byBlbmFibGUgdGhpcyBmb3IKPj4+IGFybTY0LCBp biBhbGwgaG9uZXN0eS4gVGhlIHlpZWxkIGluc3RydWN0aW9uIHR5cGljYWxseSBkb2Vzbid0IGRv Cj4+PiBhbnl0aGluZyBvbiBhY3R1YWwgYXJtNjQgc2lsaWNvbiwgc28gdGhpcyBwcm9iYWJseSBt ZWFucyB0aGF0IHlvdSdyZQo+Pj4gaW50cm9kdWNpbmcgYnVzeS1sb29wcyB3aGljaCB0ZW5kIHRv IGJlIGJhZCBmb3IgcG93ZXIgYW5kIHNjYWxhYmlsaXR5Lgo+Pgo+PiBBZnRlciB1cGRhdGluZyB0 aGUgaW1wbGVtZW50YXRpb24gaW4gdjIsIHRoZSBwZXJmb3JtYW5jZSBnYWlucyBhcmUKPj4gcHJp bWFyaWx5IG9ic2VydmVkIG9uIGFjdHVhbCBoYXJkd2FyZS4KPiAKPiAuLi4gYnV0IHRoYXQncyBw cmVzdW1hYmx5IGJlY2F1c2UgdGhlIHNlY29uZGFyeSBjb3JlcyBhcmUgYnVzeS1sb29waW5nLgo+ IFRoYXQncyBub3Qgc29tZXRoaW5nIHdlIHNob3VsZCBkbyBkdXJpbmcgYm9vdC4gSXQgbWlnaHQg YmUgImZhc3QiIG9uCj4geW91ciBtYWNoaW5lIGJ1dCBpdCB3aWxsIHByb2JhYmx5IGJlICJob3Qi IGFzIHdlbGwuCgpIaSwgV2lsbCwKCkkgc2VlIHlvdXIgcG9pbnQgcmVnYXJkaW5nIHRoZSAnaG90 IGJvb3QnIGlzc3VlLCB3aGljaCBpcyBpbmRlZWQgYSB2YWxpZApjb25jZXJuIGZvciBwb3dlci1j b25zdHJhaW5lZCBkZXZpY2VzLAoKTXkgb3B0aW1pemF0aW9uIGlzIHRhaWxvcmVkIGZvciBzZXJ2 ZXJzIGFuZCBjb250aW51b3VzbHkgcG93ZXJlZCBzaW5nbGUKYm9hcmRzLCB3aGVyZSBib290LXVw IHNwZWVkIGlzIG11Y2ggbW9yZSBjcml0aWNhbCB0aGFuIHRlbXBvcmFyeSBwb3dlcgp1c2FnZSBk dXJpbmcgdGhlIGVhcmx5IGJvb3QgcGhhc2UuCgpQZXJoYXBzIHdlIGNvdWxkIHJlcGxhY2UgdGhl ICJ5aWVsZCIgaW5zdHJ1Y3Rpb24gd2l0aCAiV0ZFIC8gU0VWIgppbnN0cnVjdGlvbnMgdG8gY29v cmRpbmF0ZSB0aGUgcGFyYWxsZWwgYm9vdCBvZiB0aGUgcHJpbWFyeSBhbmQKc2Vjb25kYXJ5IGNv cmVzLiBUaGlzIGFwcHJvYWNoIHdvdWxkIGFsbG93IHRoZSBzZWNvbmRhcnkgY29yZXMgdG8gZW50 ZXIKYSBsb3ctcG93ZXIgc3RhbmRieSBzdGF0ZSByYXRoZXIgdGhhbiBidXN5LWxvb3BpbmcsIGVm ZmVjdGl2ZWx5CnByZXZlbnRpbmcgdGhlIHRoZXJtYWwgYW5kIHBvd2VyIGlzc3VlcyBvbiBiYXR0 ZXJ5LWNvbnN0cmFpbmVkIG1hY2hpbmVzLgoKPiAKPj4+IEkgaW1wbGVtZW50ZWQgdGhpcyBhIHdo aWxlIGFnbyBbMV0gYnV0IGRpZG4ndCBtYW5hZ2UgdG8gc2VlIG11Y2ggaW4gdGVybXMKPj4+IG9m IHBlcmZvcm1hbmNlIGltcHJvdmVtZW50IGFuZCBzbyBJIGRpZG4ndCBib3RoZXIgdG8gc2VuZCB0 aGUgcGF0Y2hlcyBvdXQKPj4KPj4gQXMgc2hvd24gaW4gdjIgYmVsb3csIG9uIGFjdHVhbCBoYXJk d2FyZSwgdGhpcyByZXN1bHRzIGluIGEgNDAl4oCTNjAlCj4+IHJlZHVjdGlvbiBpbiBib290IHRp bWUuCj4+Cj4+IEJyaW5ndXAgVGltZSBDb21wYXJpc29uIChtcywgbG93ZXIgaXMgYmV0dGVyKToK Pj4KPj4gfCAgICAgUGxhdGZvcm0JCXwgQmFzZWxpbmV8ICAgUD0wICAgfCAgIFA9MSAgfCBEZWx0 YSglKXwKPj4gfCAtLS0tLS0tLS0tLS0tLS0tLS0tLS0gfCAtLS0tLS0tIHwgLS0tLS0tLSB8IC0t LS0tLSB8IC0tLS0tLS0gfAo+PiB8IDY0LWNvcmUgQVRGIFFFTVUJfCAyMDc1LjggIHwgMjA4MC43 ICB8IDE2NTMuNCB8IDIwLjM0JSAgfAo+PiB8IDE5Mi1jb3JlIHNlcnZlcihISVAxMil8IDE0NjE5 LjIgfCAxNDYxOS4xIHwgODU4OS40IHwgNDEuMjElICB8Cj4+IHwgMzItY29yZSBib2FyZAkgICAg ICAgIHwgMjc3Ni41ICB8IDI4ODEuMCAgfCAxMDQ1LjAgfCA2Mi4zNiUgIHwKPj4KPj4gTGluazoK Pj4gaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvYWxsLzIwMjYwNjE4MDkyNDQ0LjEzMTYzMzYtNS1y dWFuamluamllQGh1YXdlaS5jb20vCj4gCj4gVG8gYmUgaG9uZXN0LCBJJ20gcHJldHR5IGNvbmZ1 c2VkIHdpdGggYWxsIHRoZXNlIG51bWJlcnMuIFlvdXIgZmlyc3QKPiB0YWJsZSBhYm92ZSBzdWdn ZXN0cyB0aGF0IHBhcmFsbGVsIGJvb3QgaXMgKnNsb3dlciogYnV0IHRoZW4gdGhpcyB0YWJsZQo+ IHN1Z2dlc3RzIHRoZSBvcHBvc2l0ZS4gSG93ZXZlciwgaXQgYWxzbyBoYXMgYSBRRU1VIGVudHJ5 IGRlc3BpdGUgYmVpbmcKPiAib24gYWN0dWFsIGhhcmR3YXJlIi4gSXMgdGhhdCBpbiBhIFZNPwoK U29ycnksIHRoZXJlIGlzIGEgbGl0dGxlIGNvbmZ1c2VkLiAxOTItY29yZSBzZXJ2ZXIoSElQMTIp IGFuZCAzMi1jb3JlCmJvYXJkIGFyZSB0ZXN0ZWQgb24gcmVhbCBoYXJkd2FyZSwgd2hpY2ggaGFz IDQwJeKAkzYwJSByZWR1Y3Rpb24gaW4gYm9vdCB0aW1lLgoKPiAKPj4+IGFmdGVyIHRhbGtpbmcg YWJvdXQgaXQgYXQgS1ZNIGZvcnVtIFsyXS4gSG93ZXZlciwgYXMgbWVudGlvbmVkIGF0IHRoZSBl bmQKPj4+IG9mIHRoYXQgdGFsaywgaXQgX2lzXyBzdGlsbCB1c2VmdWwgZm9yIGNvbmZpZGVudGlh bCBWTXMgdXNpbmcgUFNDSSBzbwo+Pj4gbGV0IG1lIGR1c3Qgb2ZmIG15IG9sZCBzZXJpZXMgYW5k IHNlbmQgaXQgb3V0IHRvIHNlZSB3aGF0IHlvdSB0aGluay4KPj4+Cj4+PiBJdCByZWxpZXMgb24g UFNDSSB2MC4yLCB3aGljaCBtZWFucyB3ZSBkb24ndCBuZWVkIHRoZSBOUl9DUFVTIHNpemUgYXJy YXkKPj4+IGZvciBzZWNvbmRhcnlfZGF0YSBhbmQgSSBhbHNvIGhhdmUgc29tZSBzdXBwb3J0IGZv ciBlcnJvciBoYW5kbGluZyAoaXQKPj4+IGRvZXNuJ3QgbG9vayBsaWtlIHlvdSBoYW5kbGUgX19l YXJseV9jcHVfYm9vdF9zdGF0dXMgcHJvcGVybHkpLgo+Pgo+PiBJIG5lZWQgc29tZSB0aW1lIHRv IGxvb2sgY2xvc2VseSBhdCB5b3VyIHBhdGNoLiBBbHRlcm5hdGl2ZWx5LCBJIHdpbGwKPj4gaW50 ZWdyYXRlIHlvdXIgY2hhbmdlcywgcmUtdGVzdCBldmVyeXRoaW5nIG9uIGFjdHVhbCBoYXJkd2Fy ZSwgYW5kIHRoZW4KPj4gc2VuZCBvdXQgYSByZXZpc2VkIHZlcnNpb24uCj4gCj4gUGxlYXNlIGp1 c3QgZ2l2ZSBtZSBhIHdlZWsgb3Igc28gdG8gcmViYXNlIG15IGNoYW5nZXMgYW5kIHNlbmQgdGhl bSBvdXQKPiBmb3IgZGlzY3Vzc2lvbi4gSXQnbGwgYmUgaW50ZXJlc3RpbmcgdG8gc2VlIHdoYXQg bnVtYmVycyB5b3UgZ2V0LgoKU291bmRzIGdvb2QhIFRha2UgeW91ciB0aW1lLCBhbmQgSSdtIGxv b2tpbmcgZm9yd2FyZCB0byB5b3VyIHNlcmllcy4KCkluIHRoZSBtZWFudGltZSwgSSBoYXZlIGp1 c3Qgc2VudCBvdXQgdjMgb2YgdGhpcyBwYXRjaC4gV2hpbGUgd29ya2luZwpjbG9zZWx5IHdpdGgg eW91ciBwcmV2aW91cyBjb2RlLCBJIGlkZW50aWZpZWQgYSBmZXcgYnVncyAoaW5jbHVkaW5nIHRo ZQptdWx0aS1DUFUgc3RhdHVzIHRyYW1wbGluZyBpc3N1ZSB3ZSBkaXNjdXNzZWQpIGFuZCBhZGRy ZXNzZWQgdGhlbSBpbgp0aGlzIG5ldyB2ZXJzaW9uLkkgd2FudGVkIHRvIHNoYXJlIHYzIHdpdGgg eW91IG5vdyBzbyB5b3UgY2FuIGVhc2lseQpyZXZpZXcgdGhlIGZpeGVzIGFuZCBwb3RlbnRpYWxs eSBpbnRlZ3JhdGUgdGhlbSB3aGVuIHlvdSByZWJhc2UgeW91cgpjaGFuZ2VzIG5leHQgd2Vlay4g SXQgYWxzbyBpbmNsdWRlcyB0aGUgdXBkYXRlZCBwZXJmb3JtYW5jZSBudW1iZXJzIG9uCm15IHNl dHVwIGZvciB5b3VyIHJlZmVyZW5jZS4KCkxpbms6Cmh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2Fs bC8yMDI2MDYyNDA5MjUzNy4yOTE2OTcxLTEtcnVhbmppbmppZUBodWF3ZWkuY29tLwoKTG9va2lu ZyBmb3J3YXJkIHRvIHRoZSBkaXNjdXNzaW9uIQoKQmVzdCByZWdhcmRzLApKaW5qaWUKCj4gCj4+ IEl0IHNlZW1zIHRoYXQgdGhlIGZvbGxvd2luZyBwYXRjaCByZW1vdmluZwo+PiBgcmN1dHJlZV9y ZXBvcnRfY3B1X3N0YXJ0aW5nKClgIHdpbGwgcmVpbnRyb2R1Y2UgdGhlIG9yaWdpbmFsIGlzc3Vl IGFzCj4+IGNvbW1pdCBjZTNkMzFhZDNjYWMgKCJhcm02NC9zbXA6IE1vdmUKPj4gcmN1X2NwdV9z dGFydGluZygpIGVhcmxpZXIiKSBzb2xvdmVkLgo+Pgo+PiBMaW5rOgo+PiBodHRwczovL3dlYi5n aXQua2VybmVsLm9yZy9wdWIvc2NtL2xpbnV4L2tlcm5lbC9naXQvd2lsbC9saW51eC5naXQvY29t bWl0Lz9oPWNwdS1ob3RwbHVnJmlkPWJiYTRiNjJmNDVmMjYxNGJmNjA4NWU2Y2QzZjIzMzUyOGY4 NWJmMjYKPj4KPj4gSW5kZWVkLCBJIGFsc28gbm90aWNlZCB0aGF0IHRoZSBpbnZvY2F0aW9uIG9y ZGVyIG9mCj4+IHJjdXRyZWVfcmVwb3J0X2NwdV9zdGFydGluZygpIG9uIGFybTY0IGlzIHNvbWV3 aGF0IHN1Ym9wdGltYWwuIEl0Cj4+IGhpbmRlcnMgdGhlIGltcGxlbWVudGF0aW9uIG9mIHBhcmFs bGVsIGJyaW5ndXAgb24gYXJtNjQgYW5kIGNvdWxkCj4+IHBvdGVudGlhbGx5IGxlYWQgdG8gUkNV IHN0YWxscy4KPj4KPj4gTGluazoKPj4gaHR0cHM6Ly9sb3JlLmtlcm5lbC5vcmcvYWxsLzIwMjYw NjE4MDkyNDQ0LjEzMTYzMzYtNC1ydWFuamluamllQGh1YXdlaS5jb20vCj4+Cj4+IFsgICAgMC4z MjkwMTddIHNtcDogQnJpbmdpbmcgdXAgc2Vjb25kYXJ5IENQVXMgLi4uCj4+IFsgICAgMC4zNDM2 MjhdIERldGVjdGVkIFZJUFQgSS1jYWNoZSBvbiBDUFUxCj4+IFsgICAgMC4zNDM3ODhdCj4+IFsg ICAgMC4zNDM4MDZdID09PT09PT09PT09PT09PT09PT09PT09PT09PT09Cj4+IFsgICAgMC4zNDM4 MTZdIFdBUk5JTkc6IHN1c3BpY2lvdXMgUkNVIHVzYWdlCj4+IFsgICAgMC4zNDM5NjZdIDcuMS4w LXJjMS1nMjdjMTg3MTg0OGEyICMxMDkgTm90IHRhaW50ZWQKPj4gWyAgICAwLjM0NDA4N10gLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KPj4gWyAgICAwLjM0NDA5OF0ga2VybmVsL2xvY2tp bmcvbG9ja2RlcC5jOjM4MDEgUkNVLWxpc3QgdHJhdmVyc2VkIGluCj4+IG5vbi1yZWFkZXIgc2Vj dGlvbiEhCj4gCj4gVGhhbmtzLCBJJ2xsIGxvb2sgaW50byB0aGlzLgo+IAo+IFdpbGwKPiAKCgpf X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpsaW51eC1yaXNj diBtYWlsaW5nIGxpc3QKbGludXgtcmlzY3ZAbGlzdHMuaW5mcmFkZWFkLm9yZwpodHRwOi8vbGlz dHMuaW5mcmFkZWFkLm9yZy9tYWlsbWFuL2xpc3RpbmZvL2xpbnV4LXJpc2N2Cg==