From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6A06C7618F for ; Fri, 19 Jul 2019 00:58:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A2B2420665 for ; Fri, 19 Jul 2019 00:58:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726045AbfGSA67 (ORCPT ); Thu, 18 Jul 2019 20:58:59 -0400 Received: from mail-pg1-f195.google.com ([209.85.215.195]:39867 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726042AbfGSA66 (ORCPT ); Thu, 18 Jul 2019 20:58:58 -0400 Received: by mail-pg1-f195.google.com with SMTP id u17so13655318pgi.6; Thu, 18 Jul 2019 17:58:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=RWWNgv0WcWjfE9wWYHQhoeNKyBfGxLtvDshgZHYc1vE=; b=TI80sLTaRhpFMe8tn7BS5nXuhpKYxs7wmEnANGJMyCFvkvEteuEhqCEtgO0ziIu0rf kEy9Gtv+msFRMIANvUU/oimdps2d2hnYX2IJJFKm/jM1Vfy4c60u8wy2o52mfyyvCmYb dKkQrHZU2RaZQ/41n1L2jj8El/rsasHr2hOnJKG0HOW2gZ9w535y0tv4hV3geHY3dKtn U5ac4XnCiGMY/gTJTTgc/vF2u7pbq5M4ITjv4rPKi+MVyAZfZSeC8FuFkQv8hA2Y5rYw gESZNpyAmd5tyMhxRLQRH7XXgye29x82D4QfoIgonBwQ1oizdLloA1uaSsRVLD6/Vaxz SJZQ== X-Gm-Message-State: APjAAAWj6Tt+Yf5WtByAGrl8JvcjFmbKinrZqT/kY5ux7+vUR6NwdWzT MrcLJln5kPBAKvp4daMHvG8= X-Google-Smtp-Source: APXvYqxZoIKjz9ViGfUIEExjwwIbnjfGtWb7w9xiUnCbHyVRrBrLCxFCFJP5cS9IAYKOxxdelYaABg== X-Received: by 2002:a63:a346:: with SMTP id v6mr4814748pgn.57.1563497937499; Thu, 18 Jul 2019 17:58:57 -0700 (PDT) Received: from htb-2n-eng-dhcp405.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j128sm14025166pfg.28.2019.07.18.17.58.55 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Thu, 18 Jul 2019 17:58:56 -0700 (PDT) From: Nadav Amit To: Andy Lutomirski , Dave Hansen Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Nadav Amit , Borislav Petkov , Boris Ostrovsky , Haiyang Zhang , Josh Poimboeuf , Juergen Gross , "K. Y. Srinivasan" , Paolo Bonzini , Rik van Riel , Sasha Levin , Stephen Hemminger , kvm@vger.kernel.org, linux-hyperv@vger.kernel.org, virtualization@lists.linux-foundation.org, xen-devel@lists.xenproject.org Subject: [PATCH v3 0/9] x86: Concurrent TLB flushes Date: Thu, 18 Jul 2019 17:58:28 -0700 Message-Id: <20190719005837.4150-1-namit@vmware.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-hyperv-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-hyperv@vger.kernel.org [ Cover-letter is identical to v2, including benchmark results, excluding the change log. ] Currently, local and remote TLB flushes are not performed concurrently, which introduces unnecessary overhead - each INVLPG can take 100s of cycles. This patch-set allows TLB flushes to be run concurrently: first request the remote CPUs to initiate the flush, then run it locally, and finally wait for the remote CPUs to finish their work. In addition, there are various small optimizations to avoid unwarranted false-sharing and atomic operations. The proposed changes should also improve the performance of other invocations of on_each_cpu(). Hopefully, no one has relied on this behavior of on_each_cpu() that invoked functions first remotely and only then locally [Peter says he remembers someone might do so, but without further information it is hard to know how to address it]. Running sysbench on dax/ext4 w/emulated-pmem, write-cache disabled on 2-socket, 48-logical-cores (24+SMT) Haswell-X, 5 repetitions: sysbench fileio --file-total-size=3G --file-test-mode=rndwr \ --file-io-mode=mmap --threads=X --file-fsync-mode=fdatasync run Th. tip-jun28 avg (stdev) +patch-set avg (stdev) change --- --------------------- ---------------------- ------ 1 1267765 (14146) 1299253 (5715) +2.4% 2 1734644 (11936) 1799225 (19577) +3.7% 4 2821268 (41184) 2919132 (40149) +3.4% 8 4171652 (31243) 4376925 (65416) +4.9% 16 5590729 (24160) 5829866 (8127) +4.2% 24 6250212 (24481) 6522303 (28044) +4.3% 32 3994314 (26606) 4077543 (10685) +2.0% 48 4345177 (28091) 4417821 (41337) +1.6% (Note that on configurations with up to 24 threads numactl was used to set all threads on socket 1, which explains the drop in performance when going to 32 threads). Running the same benchmark with security mitigations disabled (PTI, Spectre, MDS): Th. tip-jun28 avg (stdev) +patch-set avg (stdev) change --- --------------------- ---------------------- ------ 1 1598896 (5174) 1607903 (4091) +0.5% 2 2109472 (17827) 2224726 (4372) +5.4% 4 3448587 (11952) 3668551 (30219) +6.3% 8 5425778 (29641) 5606266 (33519) +3.3% 16 6931232 (34677) 7054052 (27873) +1.7% 24 7612473 (23482) 7783138 (13871) +2.2% 32 4296274 (18029) 4283279 (32323) -0.3% 48 4770029 (35541) 4764760 (13575) -0.1% Presumably, PTI requires two invalidations of each mapping, which allows to get higher benefits from concurrency when PTI is on. At the same time, when mitigations are on, other overheads reduce the potential speedup. I tried to reduce the size of the code of the main patch, which required restructuring of the series. v2 -> v3: * Open-code the remote/local-flush decision code [Andy] * Fix hyper-v, Xen implementations [Andrew] * Fix redundant TLB flushes. v1 -> v2: * Removing the patches that Thomas took [tglx] * Adding hyper-v, Xen compile-tested implementations [Dave] * Removing UV [Andy] * Adding lazy optimization, removing inline keyword [Dave] * Restructuring patch-set RFCv2 -> v1: * Fix comment on flush_tlb_multi [Juergen] * Removing async invalidation optimizations [Andy] * Adding KVM support [Paolo] Cc: Andy Lutomirski Cc: Borislav Petkov Cc: Boris Ostrovsky Cc: Dave Hansen Cc: Haiyang Zhang Cc: Ingo Molnar Cc: Josh Poimboeuf Cc: Juergen Gross Cc: "K. Y. Srinivasan" Cc: Paolo Bonzini Cc: Peter Zijlstra Cc: Rik van Riel Cc: Sasha Levin Cc: Stephen Hemminger Cc: Thomas Gleixner Cc: kvm@vger.kernel.org Cc: linux-hyperv@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: x86@kernel.org Cc: xen-devel@lists.xenproject.org Nadav Amit (9): smp: Run functions concurrently in smp_call_function_many() x86/mm/tlb: Remove reason as argument for flush_tlb_func_local() x86/mm/tlb: Open-code on_each_cpu_cond_mask() for tlb_is_not_lazy() x86/mm/tlb: Flush remote and local TLBs concurrently x86/mm/tlb: Privatize cpu_tlbstate x86/mm/tlb: Do not make is_lazy dirty for no reason cpumask: Mark functions as pure x86/mm/tlb: Remove UV special case x86/mm/tlb: Remove unnecessary uses of the inline keyword arch/x86/hyperv/mmu.c | 10 +- arch/x86/include/asm/paravirt.h | 6 +- arch/x86/include/asm/paravirt_types.h | 4 +- arch/x86/include/asm/tlbflush.h | 47 ++++----- arch/x86/include/asm/trace/hyperv.h | 2 +- arch/x86/kernel/kvm.c | 11 ++- arch/x86/kernel/paravirt.c | 2 +- arch/x86/mm/init.c | 2 +- arch/x86/mm/tlb.c | 133 ++++++++++++++++---------- arch/x86/xen/mmu_pv.c | 11 +-- include/linux/cpumask.h | 6 +- include/linux/smp.h | 27 ++++-- include/trace/events/xen.h | 2 +- kernel/smp.c | 133 ++++++++++++-------------- 14 files changed, 218 insertions(+), 178 deletions(-) -- 2.20.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43072C76195 for ; Fri, 19 Jul 2019 00:59:30 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 22A362184E for ; Fri, 19 Jul 2019 00:59:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 22A362184E Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=vmware.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hoHEo-0004Hf-GG; Fri, 19 Jul 2019 00:59:06 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hoHEm-0004Ha-Iq for xen-devel@lists.xenproject.org; Fri, 19 Jul 2019 00:59:04 +0000 X-Inumbo-ID: 63b8b4fc-a9c0-11e9-b4aa-07b1dbb27dad Received: from mail-pf1-f195.google.com (unknown [209.85.210.195]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 63b8b4fc-a9c0-11e9-b4aa-07b1dbb27dad; Fri, 19 Jul 2019 00:58:58 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id y15so13391417pfn.5 for ; Thu, 18 Jul 2019 17:58:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=RWWNgv0WcWjfE9wWYHQhoeNKyBfGxLtvDshgZHYc1vE=; b=KRnqL8zV/3jmAFdYj5ALST7u0cIJRM3l7asOiaVGXSP8hpI9I2YvqY2Jzumpp7zFSe BJx19B7foCQ3kJPsZNnGKwFZtB8cnoX/svyh2NjmmcWAJQteEbcXO2bPXv4hO2AF6Fzq 7ScihtpmNSJTeghxs97YBvv+IF+8SuYcthCUWhUenUxVNQG1NbjT45XdRZRDgGRvkGjd XNH/feE0BBF0h8oilcPsZqsVSeYadxU+8qHkyEIMEs9T8xD819ITSLyKSV3HDZCDMjZ1 PdlrZ0zQIBnumshlj7qc1BNC0knTyjndLP/ycKKL7VobXTTfUBSfAXrkx/VJWPEHejH9 yNpg== X-Gm-Message-State: APjAAAUbbpGVRdLzseGL1AW0rhrGDwWO3ISW9gSzXuR7WsYjxxn1ZlrA zftPKYQWLM+b8e5pe3huNBM= X-Google-Smtp-Source: APXvYqxZoIKjz9ViGfUIEExjwwIbnjfGtWb7w9xiUnCbHyVRrBrLCxFCFJP5cS9IAYKOxxdelYaABg== X-Received: by 2002:a63:a346:: with SMTP id v6mr4814748pgn.57.1563497937499; Thu, 18 Jul 2019 17:58:57 -0700 (PDT) Received: from htb-2n-eng-dhcp405.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j128sm14025166pfg.28.2019.07.18.17.58.55 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Thu, 18 Jul 2019 17:58:56 -0700 (PDT) From: Nadav Amit To: Andy Lutomirski , Dave Hansen Date: Thu, 18 Jul 2019 17:58:28 -0700 Message-Id: <20190719005837.4150-1-namit@vmware.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v3 0/9] x86: Concurrent TLB flushes X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , Sasha Levin , linux-hyperv@vger.kernel.org, Stephen Hemminger , kvm@vger.kernel.org, Paolo Bonzini , Rik van Riel , Peter Zijlstra , Haiyang Zhang , x86@kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, xen-devel@lists.xenproject.org, Ingo Molnar , Nadav Amit , Josh Poimboeuf , Borislav Petkov , Thomas Gleixner , "K. Y. Srinivasan" , Boris Ostrovsky Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" WyBDb3Zlci1sZXR0ZXIgaXMgaWRlbnRpY2FsIHRvIHYyLCBpbmNsdWRpbmcgYmVuY2htYXJrIHJl c3VsdHMsCiAgZXhjbHVkaW5nIHRoZSBjaGFuZ2UgbG9nLiBdIAoKQ3VycmVudGx5LCBsb2NhbCBh bmQgcmVtb3RlIFRMQiBmbHVzaGVzIGFyZSBub3QgcGVyZm9ybWVkIGNvbmN1cnJlbnRseSwKd2hp Y2ggaW50cm9kdWNlcyB1bm5lY2Vzc2FyeSBvdmVyaGVhZCAtIGVhY2ggSU5WTFBHIGNhbiB0YWtl IDEwMHMgb2YKY3ljbGVzLiBUaGlzIHBhdGNoLXNldCBhbGxvd3MgVExCIGZsdXNoZXMgdG8gYmUg cnVuIGNvbmN1cnJlbnRseTogZmlyc3QKcmVxdWVzdCB0aGUgcmVtb3RlIENQVXMgdG8gaW5pdGlh dGUgdGhlIGZsdXNoLCB0aGVuIHJ1biBpdCBsb2NhbGx5LCBhbmQKZmluYWxseSB3YWl0IGZvciB0 aGUgcmVtb3RlIENQVXMgdG8gZmluaXNoIHRoZWlyIHdvcmsuCgpJbiBhZGRpdGlvbiwgdGhlcmUg YXJlIHZhcmlvdXMgc21hbGwgb3B0aW1pemF0aW9ucyB0byBhdm9pZCB1bndhcnJhbnRlZApmYWxz ZS1zaGFyaW5nIGFuZCBhdG9taWMgb3BlcmF0aW9ucy4KClRoZSBwcm9wb3NlZCBjaGFuZ2VzIHNo b3VsZCBhbHNvIGltcHJvdmUgdGhlIHBlcmZvcm1hbmNlIG9mIG90aGVyCmludm9jYXRpb25zIG9m IG9uX2VhY2hfY3B1KCkuIEhvcGVmdWxseSwgbm8gb25lIGhhcyByZWxpZWQgb24gdGhpcwpiZWhh dmlvciBvZiBvbl9lYWNoX2NwdSgpIHRoYXQgaW52b2tlZCBmdW5jdGlvbnMgZmlyc3QgcmVtb3Rl bHkgYW5kIG9ubHkKdGhlbiBsb2NhbGx5IFtQZXRlciBzYXlzIGhlIHJlbWVtYmVycyBzb21lb25l IG1pZ2h0IGRvIHNvLCBidXQgd2l0aG91dApmdXJ0aGVyIGluZm9ybWF0aW9uIGl0IGlzIGhhcmQg dG8ga25vdyBob3cgdG8gYWRkcmVzcyBpdF0uCgpSdW5uaW5nIHN5c2JlbmNoIG9uIGRheC9leHQ0 IHcvZW11bGF0ZWQtcG1lbSwgd3JpdGUtY2FjaGUgZGlzYWJsZWQgb24KMi1zb2NrZXQsIDQ4LWxv Z2ljYWwtY29yZXMgKDI0K1NNVCkgSGFzd2VsbC1YLCA1IHJlcGV0aXRpb25zOgoKIHN5c2JlbmNo IGZpbGVpbyAtLWZpbGUtdG90YWwtc2l6ZT0zRyAtLWZpbGUtdGVzdC1tb2RlPXJuZHdyIFwKICAt LWZpbGUtaW8tbW9kZT1tbWFwIC0tdGhyZWFkcz1YIC0tZmlsZS1mc3luYy1tb2RlPWZkYXRhc3lu YyBydW4KCiAgVGguICAgdGlwLWp1bjI4IGF2ZyAoc3RkZXYpICAgK3BhdGNoLXNldCBhdmcgKHN0 ZGV2KSAgY2hhbmdlCiAgLS0tICAgLS0tLS0tLS0tLS0tLS0tLS0tLS0tICAgLS0tLS0tLS0tLS0t LS0tLS0tLS0tLSAgLS0tLS0tCiAgMSAgICAgMTI2Nzc2NSAoMTQxNDYpICAgICAgICAgMTI5OTI1 MyAoNTcxNSkgICAgICAgICAgKzIuNCUKICAyICAgICAxNzM0NjQ0ICgxMTkzNikgICAgICAgICAx Nzk5MjI1ICgxOTU3NykgICAgICAgICArMy43JQogIDQgICAgIDI4MjEyNjggKDQxMTg0KSAgICAg ICAgIDI5MTkxMzIgKDQwMTQ5KSAgICAgICAgICszLjQlCiAgOCAgICAgNDE3MTY1MiAoMzEyNDMp ICAgICAgICAgNDM3NjkyNSAoNjU0MTYpICAgICAgICAgKzQuOSUKICAxNiAgICA1NTkwNzI5ICgy NDE2MCkgICAgICAgICA1ODI5ODY2ICg4MTI3KSAgICAgICAgICArNC4yJQogIDI0ICAgIDYyNTAy MTIgKDI0NDgxKSAgICAgICAgIDY1MjIzMDMgKDI4MDQ0KSAgICAgICAgICs0LjMlCiAgMzIgICAg Mzk5NDMxNCAoMjY2MDYpICAgICAgICAgNDA3NzU0MyAoMTA2ODUpICAgICAgICAgKzIuMCUKICA0 OCAgICA0MzQ1MTc3ICgyODA5MSkgICAgICAgICA0NDE3ODIxICg0MTMzNykgICAgICAgICArMS42 JQoKKE5vdGUgdGhhdCBvbiBjb25maWd1cmF0aW9ucyB3aXRoIHVwIHRvIDI0IHRocmVhZHMgbnVt YWN0bCB3YXMgdXNlZCB0bwpzZXQgYWxsIHRocmVhZHMgb24gc29ja2V0IDEsIHdoaWNoIGV4cGxh aW5zIHRoZSBkcm9wIGluIHBlcmZvcm1hbmNlIHdoZW4KZ29pbmcgdG8gMzIgdGhyZWFkcykuCgpS dW5uaW5nIHRoZSBzYW1lIGJlbmNobWFyayB3aXRoIHNlY3VyaXR5IG1pdGlnYXRpb25zIGRpc2Fi bGVkIChQVEksClNwZWN0cmUsIE1EUyk6CgogIFRoLiAgIHRpcC1qdW4yOCBhdmcgKHN0ZGV2KSAg ICtwYXRjaC1zZXQgYXZnIChzdGRldikgIGNoYW5nZQogIC0tLSAgIC0tLS0tLS0tLS0tLS0tLS0t LS0tLSAgIC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0gIC0tLS0tLQogIDEgICAgIDE1OTg4OTYgKDUx NzQpICAgICAgICAgIDE2MDc5MDMgKDQwOTEpICAgICAgICAgICswLjUlCiAgMiAgICAgMjEwOTQ3 MiAoMTc4MjcpICAgICAgICAgMjIyNDcyNiAoNDM3MikgICAgICAgICAgKzUuNCUKICA0ICAgICAz NDQ4NTg3ICgxMTk1MikgICAgICAgICAzNjY4NTUxICgzMDIxOSkgICAgICAgICArNi4zJQogIDgg ICAgIDU0MjU3NzggKDI5NjQxKSAgICAgICAgIDU2MDYyNjYgKDMzNTE5KSAgICAgICAgICszLjMl CiAgMTYgICAgNjkzMTIzMiAoMzQ2NzcpICAgICAgICAgNzA1NDA1MiAoMjc4NzMpICAgICAgICAg KzEuNyUKICAyNCAgICA3NjEyNDczICgyMzQ4MikgICAgICAgICA3NzgzMTM4ICgxMzg3MSkgICAg ICAgICArMi4yJQogIDMyICAgIDQyOTYyNzQgKDE4MDI5KSAgICAgICAgIDQyODMyNzkgKDMyMzIz KSAgICAgICAgIC0wLjMlCiAgNDggICAgNDc3MDAyOSAoMzU1NDEpICAgICAgICAgNDc2NDc2MCAo MTM1NzUpICAgICAgICAgLTAuMSUKClByZXN1bWFibHksIFBUSSByZXF1aXJlcyB0d28gaW52YWxp ZGF0aW9ucyBvZiBlYWNoIG1hcHBpbmcsIHdoaWNoIGFsbG93cwp0byBnZXQgaGlnaGVyIGJlbmVm aXRzIGZyb20gY29uY3VycmVuY3kgd2hlbiBQVEkgaXMgb24uIEF0IHRoZSBzYW1lCnRpbWUsIHdo ZW4gbWl0aWdhdGlvbnMgYXJlIG9uLCBvdGhlciBvdmVyaGVhZHMgcmVkdWNlIHRoZSBwb3RlbnRp YWwKc3BlZWR1cC4KCkkgdHJpZWQgdG8gcmVkdWNlIHRoZSBzaXplIG9mIHRoZSBjb2RlIG9mIHRo ZSBtYWluIHBhdGNoLCB3aGljaCByZXF1aXJlZApyZXN0cnVjdHVyaW5nIG9mIHRoZSBzZXJpZXMu Cgp2MiAtPiB2MzoKKiBPcGVuLWNvZGUgdGhlIHJlbW90ZS9sb2NhbC1mbHVzaCBkZWNpc2lvbiBj b2RlIFtBbmR5XQoqIEZpeCBoeXBlci12LCBYZW4gaW1wbGVtZW50YXRpb25zIFtBbmRyZXddCiog Rml4IHJlZHVuZGFudCBUTEIgZmx1c2hlcy4KCnYxIC0+IHYyOgoqIFJlbW92aW5nIHRoZSBwYXRj aGVzIHRoYXQgVGhvbWFzIHRvb2sgW3RnbHhdCiogQWRkaW5nIGh5cGVyLXYsIFhlbiBjb21waWxl LXRlc3RlZCBpbXBsZW1lbnRhdGlvbnMgW0RhdmVdCiogUmVtb3ZpbmcgVVYgW0FuZHldCiogQWRk aW5nIGxhenkgb3B0aW1pemF0aW9uLCByZW1vdmluZyBpbmxpbmUga2V5d29yZCBbRGF2ZV0KKiBS ZXN0cnVjdHVyaW5nIHBhdGNoLXNldAoKUkZDdjIgLT4gdjE6CiogRml4IGNvbW1lbnQgb24gZmx1 c2hfdGxiX211bHRpIFtKdWVyZ2VuXQoqIFJlbW92aW5nIGFzeW5jIGludmFsaWRhdGlvbiBvcHRp bWl6YXRpb25zIFtBbmR5XQoqIEFkZGluZyBLVk0gc3VwcG9ydCBbUGFvbG9dCgpDYzogQW5keSBM dXRvbWlyc2tpIDxsdXRvQGtlcm5lbC5vcmc+CkNjOiBCb3Jpc2xhdiBQZXRrb3YgPGJwQGFsaWVu OC5kZT4KQ2M6IEJvcmlzIE9zdHJvdnNreSA8Ym9yaXMub3N0cm92c2t5QG9yYWNsZS5jb20+CkNj OiBEYXZlIEhhbnNlbiA8ZGF2ZS5oYW5zZW5AbGludXguaW50ZWwuY29tPgpDYzogSGFpeWFuZyBa aGFuZyA8aGFpeWFuZ3pAbWljcm9zb2Z0LmNvbT4KQ2M6IEluZ28gTW9sbmFyIDxtaW5nb0ByZWRo YXQuY29tPgpDYzogSm9zaCBQb2ltYm9ldWYgPGpwb2ltYm9lQHJlZGhhdC5jb20+CkNjOiBKdWVy Z2VuIEdyb3NzIDxqZ3Jvc3NAc3VzZS5jb20+CkNjOiAiSy4gWS4gU3Jpbml2YXNhbiIgPGt5c0Bt aWNyb3NvZnQuY29tPgpDYzogUGFvbG8gQm9uemluaSA8cGJvbnppbmlAcmVkaGF0LmNvbT4KQ2M6 IFBldGVyIFppamxzdHJhIDxwZXRlcnpAaW5mcmFkZWFkLm9yZz4KQ2M6IFJpayB2YW4gUmllbCA8 cmllbEBzdXJyaWVsLmNvbT4KQ2M6IFNhc2hhIExldmluIDxzYXNoYWxAa2VybmVsLm9yZz4KQ2M6 IFN0ZXBoZW4gSGVtbWluZ2VyIDxzdGhlbW1pbkBtaWNyb3NvZnQuY29tPgpDYzogVGhvbWFzIEds ZWl4bmVyIDx0Z2x4QGxpbnV0cm9uaXguZGU+CkNjOiBrdm1Admdlci5rZXJuZWwub3JnCkNjOiBs aW51eC1oeXBlcnZAdmdlci5rZXJuZWwub3JnCkNjOiBsaW51eC1rZXJuZWxAdmdlci5rZXJuZWwu b3JnCkNjOiB2aXJ0dWFsaXphdGlvbkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpDYzogeDg2 QGtlcm5lbC5vcmcKQ2M6IHhlbi1kZXZlbEBsaXN0cy54ZW5wcm9qZWN0Lm9yZwoKTmFkYXYgQW1p dCAoOSk6CiAgc21wOiBSdW4gZnVuY3Rpb25zIGNvbmN1cnJlbnRseSBpbiBzbXBfY2FsbF9mdW5j dGlvbl9tYW55KCkKICB4ODYvbW0vdGxiOiBSZW1vdmUgcmVhc29uIGFzIGFyZ3VtZW50IGZvciBm bHVzaF90bGJfZnVuY19sb2NhbCgpCiAgeDg2L21tL3RsYjogT3Blbi1jb2RlIG9uX2VhY2hfY3B1 X2NvbmRfbWFzaygpIGZvciB0bGJfaXNfbm90X2xhenkoKQogIHg4Ni9tbS90bGI6IEZsdXNoIHJl bW90ZSBhbmQgbG9jYWwgVExCcyBjb25jdXJyZW50bHkKICB4ODYvbW0vdGxiOiBQcml2YXRpemUg Y3B1X3RsYnN0YXRlCiAgeDg2L21tL3RsYjogRG8gbm90IG1ha2UgaXNfbGF6eSBkaXJ0eSBmb3Ig bm8gcmVhc29uCiAgY3B1bWFzazogTWFyayBmdW5jdGlvbnMgYXMgcHVyZQogIHg4Ni9tbS90bGI6 IFJlbW92ZSBVViBzcGVjaWFsIGNhc2UKICB4ODYvbW0vdGxiOiBSZW1vdmUgdW5uZWNlc3Nhcnkg dXNlcyBvZiB0aGUgaW5saW5lIGtleXdvcmQKCiBhcmNoL3g4Ni9oeXBlcnYvbW11LmMgICAgICAg ICAgICAgICAgIHwgIDEwICstCiBhcmNoL3g4Ni9pbmNsdWRlL2FzbS9wYXJhdmlydC5oICAgICAg IHwgICA2ICstCiBhcmNoL3g4Ni9pbmNsdWRlL2FzbS9wYXJhdmlydF90eXBlcy5oIHwgICA0ICst CiBhcmNoL3g4Ni9pbmNsdWRlL2FzbS90bGJmbHVzaC5oICAgICAgIHwgIDQ3ICsrKystLS0tLQog YXJjaC94ODYvaW5jbHVkZS9hc20vdHJhY2UvaHlwZXJ2LmggICB8ICAgMiArLQogYXJjaC94ODYv a2VybmVsL2t2bS5jICAgICAgICAgICAgICAgICB8ICAxMSArKy0KIGFyY2gveDg2L2tlcm5lbC9w YXJhdmlydC5jICAgICAgICAgICAgfCAgIDIgKy0KIGFyY2gveDg2L21tL2luaXQuYyAgICAgICAg ICAgICAgICAgICAgfCAgIDIgKy0KIGFyY2gveDg2L21tL3RsYi5jICAgICAgICAgICAgICAgICAg ICAgfCAxMzMgKysrKysrKysrKysrKysrKy0tLS0tLS0tLS0KIGFyY2gveDg2L3hlbi9tbXVfcHYu YyAgICAgICAgICAgICAgICAgfCAgMTEgKy0tCiBpbmNsdWRlL2xpbnV4L2NwdW1hc2suaCAgICAg ICAgICAgICAgIHwgICA2ICstCiBpbmNsdWRlL2xpbnV4L3NtcC5oICAgICAgICAgICAgICAgICAg IHwgIDI3ICsrKystLQogaW5jbHVkZS90cmFjZS9ldmVudHMveGVuLmggICAgICAgICAgICB8ICAg MiArLQoga2VybmVsL3NtcC5jICAgICAgICAgICAgICAgICAgICAgICAgICB8IDEzMyArKysrKysr KysrKystLS0tLS0tLS0tLS0tLQogMTQgZmlsZXMgY2hhbmdlZCwgMjE4IGluc2VydGlvbnMoKyks IDE3OCBkZWxldGlvbnMoLSkKCi0tIAoyLjIwLjEKCgpfX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fXwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBs aXN0cy54ZW5wcm9qZWN0Lm9yZwpodHRwczovL2xpc3RzLnhlbnByb2plY3Qub3JnL21haWxtYW4v bGlzdGluZm8veGVuLWRldmVs