From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4E1DC31E49 for ; Thu, 13 Jun 2019 16:42:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7BA0221721 for ; Thu, 13 Jun 2019 16:42:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730577AbfFMQmG (ORCPT ); Thu, 13 Jun 2019 12:42:06 -0400 Received: from mail-pg1-f193.google.com ([209.85.215.193]:40517 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730523AbfFMGt2 (ORCPT ); Thu, 13 Jun 2019 02:49:28 -0400 Received: by mail-pg1-f193.google.com with SMTP id d30so10353777pgm.7; Wed, 12 Jun 2019 23:49:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EWV8HqUW5nD4lYeuHZCchPJIr3ycVzu67AOhwAlhylY=; b=UZ49QdGgk8Z46G08bZVj8g4zlEmwOTCw5eYH79Um9ramjkPF+VPoyBr1iH01k1nb0P 9vGLR3Pxt5zVkizQfBUja/4BDlYiOjj/sbWqtHrcAR9UubtAEcHHu1qJ6VYcvQ/azQjK SAcfSS4i0rlANIdbDIuxY/A6KmxoWtvdbKBeMJxA2HjKoAkr3C/x7z8LmuhrOxyzyK5w h/QTjA/r0cK/Cje/sp4lXNccpAws5pWQBIdOK6/TW2jKIpiGu938LUnxCBBJqP28dR5a Bs6NMm4XObmoj93mrT8mdGc2v6861OZZjLY/haXHgbAqbW4/n0+F3z+FMgynzFG7+cmP BbPA== X-Gm-Message-State: APjAAAXuK4DaVyTx7ds2LHJk18H5WnKmN9JbNYXIrM0dPu9miBJAn/UO 31wtsU4Ni6YZLmmNOvzzzyw= X-Google-Smtp-Source: APXvYqw+K1atiiGJQgLV9SN1EyEIO2KJbWIsQQ3d5N7HeaIcjGnehk9lu87UMmJtEDHDqZuBy4rhjg== X-Received: by 2002:a17:90a:8a10:: with SMTP id w16mr3546662pjn.133.1560408567383; Wed, 12 Jun 2019 23:49:27 -0700 (PDT) Received: from htb-2n-eng-dhcp405.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id i3sm1559973pfa.175.2019.06.12.23.49.19 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 12 Jun 2019 23:49:26 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Andy Lutomirski Cc: linux-kernel@vger.kernel.org, Ingo Molnar , Borislav Petkov , x86@kernel.org, Thomas Gleixner , Dave Hansen , Nadav Amit , "K. Y. Srinivasan" , Haiyang Zhang , Stephen Hemminger , Sasha Levin , Juergen Gross , Paolo Bonzini , Boris Ostrovsky , linux-hyperv@vger.kernel.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, xen-devel@lists.xenproject.org Subject: [PATCH 4/9] x86/mm/tlb: Flush remote and local TLBs concurrently Date: Wed, 12 Jun 2019 23:48:08 -0700 Message-Id: <20190613064813.8102-5-namit@vmware.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190613064813.8102-1-namit@vmware.com> References: <20190613064813.8102-1-namit@vmware.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-hyperv-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-hyperv@vger.kernel.org To improve TLB shootdown performance, flush the remote and local TLBs concurrently. Introduce flush_tlb_multi() that does so. The current flush_tlb_others() interface is kept, since paravirtual interfaces need to be adapted first before it can be removed. This is left for future work. In such PV environments, TLB flushes are not performed, at this time, concurrently. Add a static key to tell whether this new interface is supported. Cc: "K. Y. Srinivasan" Cc: Haiyang Zhang Cc: Stephen Hemminger Cc: Sasha Levin Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: Juergen Gross Cc: Paolo Bonzini Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: Boris Ostrovsky Cc: linux-hyperv@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: kvm@vger.kernel.org Cc: xen-devel@lists.xenproject.org Signed-off-by: Nadav Amit --- arch/x86/hyperv/mmu.c | 2 + arch/x86/include/asm/paravirt.h | 8 +++ arch/x86/include/asm/paravirt_types.h | 6 +++ arch/x86/include/asm/tlbflush.h | 6 +++ arch/x86/kernel/kvm.c | 1 + arch/x86/kernel/paravirt.c | 3 ++ arch/x86/mm/tlb.c | 71 ++++++++++++++++++++++----- arch/x86/xen/mmu_pv.c | 2 + 8 files changed, 87 insertions(+), 12 deletions(-) diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c index e65d7fe6489f..ca28b400c87c 100644 --- a/arch/x86/hyperv/mmu.c +++ b/arch/x86/hyperv/mmu.c @@ -233,4 +233,6 @@ void hyperv_setup_mmu_ops(void) pr_info("Using hypercall for remote TLB flush\n"); pv_ops.mmu.flush_tlb_others = hyperv_flush_tlb_others; pv_ops.mmu.tlb_remove_table = tlb_remove_table; + + static_key_disable(&flush_tlb_multi_enabled.key); } diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index c25c38a05c1c..192be7254457 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -47,6 +47,8 @@ static inline void slow_down_io(void) #endif } +DECLARE_STATIC_KEY_TRUE(flush_tlb_multi_enabled); + static inline void __flush_tlb(void) { PVOP_VCALL0(mmu.flush_tlb_user); @@ -62,6 +64,12 @@ static inline void __flush_tlb_one_user(unsigned long addr) PVOP_VCALL1(mmu.flush_tlb_one_user, addr); } +static inline void flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info) +{ + PVOP_VCALL2(mmu.flush_tlb_multi, cpumask, info); +} + static inline void flush_tlb_others(const struct cpumask *cpumask, const struct flush_tlb_info *info) { diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h index 946f8f1f1efc..b93b3d90729a 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -211,6 +211,12 @@ struct pv_mmu_ops { void (*flush_tlb_user)(void); void (*flush_tlb_kernel)(void); void (*flush_tlb_one_user)(unsigned long addr); + /* + * flush_tlb_multi() is the preferred interface, which is capable to + * flush both local and remote CPUs. + */ + void (*flush_tlb_multi)(const struct cpumask *cpus, + const struct flush_tlb_info *info); void (*flush_tlb_others)(const struct cpumask *cpus, const struct flush_tlb_info *info); diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index dee375831962..79272938cf79 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -569,6 +569,9 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false); } +void native_flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info); + void native_flush_tlb_others(const struct cpumask *cpumask, const struct flush_tlb_info *info); @@ -593,6 +596,9 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); #ifndef CONFIG_PARAVIRT +#define flush_tlb_multi(mask, info) \ + native_flush_tlb_multi(mask, info) + #define flush_tlb_others(mask, info) \ native_flush_tlb_others(mask, info) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 5169b8cc35bb..00d81e898717 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -630,6 +630,7 @@ static void __init kvm_guest_init(void) kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) { pv_ops.mmu.flush_tlb_others = kvm_flush_tlb_others; pv_ops.mmu.tlb_remove_table = tlb_remove_table; + static_key_disable(&flush_tlb_multi_enabled.key); } if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 98039d7fb998..ac00afed5570 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -159,6 +159,8 @@ unsigned paravirt_patch_insns(void *insn_buff, unsigned len, return insn_len; } +DEFINE_STATIC_KEY_TRUE(flush_tlb_multi_enabled); + static void native_flush_tlb(void) { __native_flush_tlb(); @@ -363,6 +365,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_user = native_flush_tlb, .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, + .mmu.flush_tlb_multi = native_flush_tlb_multi, .mmu.flush_tlb_others = native_flush_tlb_others, .mmu.tlb_remove_table = (void (*)(struct mmu_gather *, void *))tlb_remove_page, diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index c34bcf03f06f..db73d5f1dd43 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -551,7 +551,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, * garbage into our TLB. Since switching to init_mm is barely * slower than a minimal flush, just switch to init_mm. * - * This should be rare, with native_flush_tlb_others skipping + * This should be rare, with native_flush_tlb_multi skipping * IPIs to lazy TLB mode CPUs. */ switch_mm_irqs_off(NULL, &init_mm, NULL); @@ -635,9 +635,12 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, this_cpu_write(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen, mm_tlb_gen); } -static void flush_tlb_func_local(const void *info, enum tlb_flush_reason reason) +static void flush_tlb_func_local(void *info) { const struct flush_tlb_info *f = info; + enum tlb_flush_reason reason; + + reason = (f->mm == NULL) ? TLB_LOCAL_SHOOTDOWN : TLB_LOCAL_MM_SHOOTDOWN; flush_tlb_func_common(f, true, reason); } @@ -655,14 +658,21 @@ static void flush_tlb_func_remote(void *info) flush_tlb_func_common(f, false, TLB_REMOTE_SHOOTDOWN); } -static bool tlb_is_not_lazy(int cpu, void *data) +static inline bool tlb_is_not_lazy(int cpu) { return !per_cpu(cpu_tlbstate.is_lazy, cpu); } -void native_flush_tlb_others(const struct cpumask *cpumask, - const struct flush_tlb_info *info) +static DEFINE_PER_CPU(cpumask_t, flush_tlb_mask); + +void native_flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info) { + /* + * Do accounting and tracing. Note that there are (and have always been) + * cases in which a remote TLB flush will be traced, but eventually + * would not happen. + */ count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); if (info->end == TLB_FLUSH_ALL) trace_tlb_flush(TLB_REMOTE_SEND_IPI, TLB_FLUSH_ALL); @@ -682,10 +692,14 @@ void native_flush_tlb_others(const struct cpumask *cpumask, * means that the percpu tlb_gen variables won't be updated * and we'll do pointless flushes on future context switches. * - * Rather than hooking native_flush_tlb_others() here, I think + * Rather than hooking native_flush_tlb_multi() here, I think * that UV should be updated so that smp_call_function_many(), * etc, are optimal on UV. */ + local_irq_disable(); + flush_tlb_func_local((__force void *)info); + local_irq_enable(); + cpumask = uv_flush_tlb_others(cpumask, info); if (cpumask) smp_call_function_many(cpumask, flush_tlb_func_remote, @@ -704,11 +718,39 @@ void native_flush_tlb_others(const struct cpumask *cpumask, * doing a speculative memory access. */ if (info->freed_tables) - smp_call_function_many(cpumask, flush_tlb_func_remote, - (void *)info, 1); - else - on_each_cpu_cond_mask(tlb_is_not_lazy, flush_tlb_func_remote, - (void *)info, 1, GFP_ATOMIC, cpumask); + __smp_call_function_many(cpumask, flush_tlb_func_remote, + flush_tlb_func_local, (void *)info, 1); + else { + /* + * Although we could have used on_each_cpu_cond_mask(), + * open-coding it has several performance advantages: (1) we can + * use specialized functions for remote and local flushes; (2) + * no need for indirect branch to test if TLB is lazy; (3) we + * can use a designated cpumask for evaluating the condition + * instead of allocating a new one. + * + * This works under the assumption that there are no nested TLB + * flushes, an assumption that is already made in + * flush_tlb_mm_range(). + */ + struct cpumask *cond_cpumask = this_cpu_ptr(&flush_tlb_mask); + int cpu; + + cpumask_clear(cond_cpumask); + + for_each_cpu(cpu, cpumask) { + if (tlb_is_not_lazy(cpu)) + __cpumask_set_cpu(cpu, cond_cpumask); + } + __smp_call_function_many(cond_cpumask, flush_tlb_func_remote, + flush_tlb_func_local, (void *)info, 1); + } +} + +void native_flush_tlb_others(const struct cpumask *cpumask, + const struct flush_tlb_info *info) +{ + native_flush_tlb_multi(cpumask, info); } /* @@ -774,10 +816,15 @@ static void flush_tlb_on_cpus(const cpumask_t *cpumask, { int this_cpu = smp_processor_id(); + if (static_branch_likely(&flush_tlb_multi_enabled)) { + flush_tlb_multi(cpumask, info); + return; + } + if (cpumask_test_cpu(this_cpu, cpumask)) { lockdep_assert_irqs_enabled(); local_irq_disable(); - flush_tlb_func_local(info, TLB_LOCAL_MM_SHOOTDOWN); + flush_tlb_func_local((__force void *)info); local_irq_enable(); } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index beb44e22afdf..0cb277848cb4 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2474,6 +2474,8 @@ void __init xen_init_mmu_ops(void) pv_ops.mmu = xen_mmu_ops; + static_key_disable(&flush_tlb_multi_enabled.key); + memset(dummy_mapping, 0xff, PAGE_SIZE); } -- 2.20.1 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nadav Amit via Virtualization Subject: [PATCH 4/9] x86/mm/tlb: Flush remote and local TLBs concurrently Date: Wed, 12 Jun 2019 23:48:08 -0700 Message-ID: <20190613064813.8102-5-namit@vmware.com> References: <20190613064813.8102-1-namit@vmware.com> Reply-To: Nadav Amit Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20190613064813.8102-1-namit@vmware.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Peter Zijlstra , Andy Lutomirski Cc: Sasha Levin , Juergen Gross , linux-hyperv@vger.kernel.org, Dave Hansen , Stephen Hemminger , xen-devel@lists.xenproject.org, kvm@vger.kernel.org, Haiyang Zhang , x86@kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Ingo Molnar , Borislav Petkov , Paolo Bonzini , Nadav Amit , Thomas Gleixner , Boris Ostrovsky List-Id: virtualization@lists.linuxfoundation.org To improve TLB shootdown performance, flush the remote and local TLBs concurrently. Introduce flush_tlb_multi() that does so. The current flush_tlb_others() interface is kept, since paravirtual interfaces need to be adapted first before it can be removed. This is left for future work. In such PV environments, TLB flushes are not performed, at this time, concurrently. Add a static key to tell whether this new interface is supported. Cc: "K. Y. Srinivasan" Cc: Haiyang Zhang Cc: Stephen Hemminger Cc: Sasha Levin Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: Juergen Gross Cc: Paolo Bonzini Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: Boris Ostrovsky Cc: linux-hyperv@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: kvm@vger.kernel.org Cc: xen-devel@lists.xenproject.org Signed-off-by: Nadav Amit --- arch/x86/hyperv/mmu.c | 2 + arch/x86/include/asm/paravirt.h | 8 +++ arch/x86/include/asm/paravirt_types.h | 6 +++ arch/x86/include/asm/tlbflush.h | 6 +++ arch/x86/kernel/kvm.c | 1 + arch/x86/kernel/paravirt.c | 3 ++ arch/x86/mm/tlb.c | 71 ++++++++++++++++++++++----- arch/x86/xen/mmu_pv.c | 2 + 8 files changed, 87 insertions(+), 12 deletions(-) diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c index e65d7fe6489f..ca28b400c87c 100644 --- a/arch/x86/hyperv/mmu.c +++ b/arch/x86/hyperv/mmu.c @@ -233,4 +233,6 @@ void hyperv_setup_mmu_ops(void) pr_info("Using hypercall for remote TLB flush\n"); pv_ops.mmu.flush_tlb_others = hyperv_flush_tlb_others; pv_ops.mmu.tlb_remove_table = tlb_remove_table; + + static_key_disable(&flush_tlb_multi_enabled.key); } diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index c25c38a05c1c..192be7254457 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -47,6 +47,8 @@ static inline void slow_down_io(void) #endif } +DECLARE_STATIC_KEY_TRUE(flush_tlb_multi_enabled); + static inline void __flush_tlb(void) { PVOP_VCALL0(mmu.flush_tlb_user); @@ -62,6 +64,12 @@ static inline void __flush_tlb_one_user(unsigned long addr) PVOP_VCALL1(mmu.flush_tlb_one_user, addr); } +static inline void flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info) +{ + PVOP_VCALL2(mmu.flush_tlb_multi, cpumask, info); +} + static inline void flush_tlb_others(const struct cpumask *cpumask, const struct flush_tlb_info *info) { diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h index 946f8f1f1efc..b93b3d90729a 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -211,6 +211,12 @@ struct pv_mmu_ops { void (*flush_tlb_user)(void); void (*flush_tlb_kernel)(void); void (*flush_tlb_one_user)(unsigned long addr); + /* + * flush_tlb_multi() is the preferred interface, which is capable to + * flush both local and remote CPUs. + */ + void (*flush_tlb_multi)(const struct cpumask *cpus, + const struct flush_tlb_info *info); void (*flush_tlb_others)(const struct cpumask *cpus, const struct flush_tlb_info *info); diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index dee375831962..79272938cf79 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -569,6 +569,9 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false); } +void native_flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info); + void native_flush_tlb_others(const struct cpumask *cpumask, const struct flush_tlb_info *info); @@ -593,6 +596,9 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); #ifndef CONFIG_PARAVIRT +#define flush_tlb_multi(mask, info) \ + native_flush_tlb_multi(mask, info) + #define flush_tlb_others(mask, info) \ native_flush_tlb_others(mask, info) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 5169b8cc35bb..00d81e898717 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -630,6 +630,7 @@ static void __init kvm_guest_init(void) kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) { pv_ops.mmu.flush_tlb_others = kvm_flush_tlb_others; pv_ops.mmu.tlb_remove_table = tlb_remove_table; + static_key_disable(&flush_tlb_multi_enabled.key); } if (kvm_para_has_feature(KVM_FEATURE_PV_EOI)) diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 98039d7fb998..ac00afed5570 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -159,6 +159,8 @@ unsigned paravirt_patch_insns(void *insn_buff, unsigned len, return insn_len; } +DEFINE_STATIC_KEY_TRUE(flush_tlb_multi_enabled); + static void native_flush_tlb(void) { __native_flush_tlb(); @@ -363,6 +365,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_user = native_flush_tlb, .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, + .mmu.flush_tlb_multi = native_flush_tlb_multi, .mmu.flush_tlb_others = native_flush_tlb_others, .mmu.tlb_remove_table = (void (*)(struct mmu_gather *, void *))tlb_remove_page, diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index c34bcf03f06f..db73d5f1dd43 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -551,7 +551,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, * garbage into our TLB. Since switching to init_mm is barely * slower than a minimal flush, just switch to init_mm. * - * This should be rare, with native_flush_tlb_others skipping + * This should be rare, with native_flush_tlb_multi skipping * IPIs to lazy TLB mode CPUs. */ switch_mm_irqs_off(NULL, &init_mm, NULL); @@ -635,9 +635,12 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, this_cpu_write(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen, mm_tlb_gen); } -static void flush_tlb_func_local(const void *info, enum tlb_flush_reason reason) +static void flush_tlb_func_local(void *info) { const struct flush_tlb_info *f = info; + enum tlb_flush_reason reason; + + reason = (f->mm == NULL) ? TLB_LOCAL_SHOOTDOWN : TLB_LOCAL_MM_SHOOTDOWN; flush_tlb_func_common(f, true, reason); } @@ -655,14 +658,21 @@ static void flush_tlb_func_remote(void *info) flush_tlb_func_common(f, false, TLB_REMOTE_SHOOTDOWN); } -static bool tlb_is_not_lazy(int cpu, void *data) +static inline bool tlb_is_not_lazy(int cpu) { return !per_cpu(cpu_tlbstate.is_lazy, cpu); } -void native_flush_tlb_others(const struct cpumask *cpumask, - const struct flush_tlb_info *info) +static DEFINE_PER_CPU(cpumask_t, flush_tlb_mask); + +void native_flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info) { + /* + * Do accounting and tracing. Note that there are (and have always been) + * cases in which a remote TLB flush will be traced, but eventually + * would not happen. + */ count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); if (info->end == TLB_FLUSH_ALL) trace_tlb_flush(TLB_REMOTE_SEND_IPI, TLB_FLUSH_ALL); @@ -682,10 +692,14 @@ void native_flush_tlb_others(const struct cpumask *cpumask, * means that the percpu tlb_gen variables won't be updated * and we'll do pointless flushes on future context switches. * - * Rather than hooking native_flush_tlb_others() here, I think + * Rather than hooking native_flush_tlb_multi() here, I think * that UV should be updated so that smp_call_function_many(), * etc, are optimal on UV. */ + local_irq_disable(); + flush_tlb_func_local((__force void *)info); + local_irq_enable(); + cpumask = uv_flush_tlb_others(cpumask, info); if (cpumask) smp_call_function_many(cpumask, flush_tlb_func_remote, @@ -704,11 +718,39 @@ void native_flush_tlb_others(const struct cpumask *cpumask, * doing a speculative memory access. */ if (info->freed_tables) - smp_call_function_many(cpumask, flush_tlb_func_remote, - (void *)info, 1); - else - on_each_cpu_cond_mask(tlb_is_not_lazy, flush_tlb_func_remote, - (void *)info, 1, GFP_ATOMIC, cpumask); + __smp_call_function_many(cpumask, flush_tlb_func_remote, + flush_tlb_func_local, (void *)info, 1); + else { + /* + * Although we could have used on_each_cpu_cond_mask(), + * open-coding it has several performance advantages: (1) we can + * use specialized functions for remote and local flushes; (2) + * no need for indirect branch to test if TLB is lazy; (3) we + * can use a designated cpumask for evaluating the condition + * instead of allocating a new one. + * + * This works under the assumption that there are no nested TLB + * flushes, an assumption that is already made in + * flush_tlb_mm_range(). + */ + struct cpumask *cond_cpumask = this_cpu_ptr(&flush_tlb_mask); + int cpu; + + cpumask_clear(cond_cpumask); + + for_each_cpu(cpu, cpumask) { + if (tlb_is_not_lazy(cpu)) + __cpumask_set_cpu(cpu, cond_cpumask); + } + __smp_call_function_many(cond_cpumask, flush_tlb_func_remote, + flush_tlb_func_local, (void *)info, 1); + } +} + +void native_flush_tlb_others(const struct cpumask *cpumask, + const struct flush_tlb_info *info) +{ + native_flush_tlb_multi(cpumask, info); } /* @@ -774,10 +816,15 @@ static void flush_tlb_on_cpus(const cpumask_t *cpumask, { int this_cpu = smp_processor_id(); + if (static_branch_likely(&flush_tlb_multi_enabled)) { + flush_tlb_multi(cpumask, info); + return; + } + if (cpumask_test_cpu(this_cpu, cpumask)) { lockdep_assert_irqs_enabled(); local_irq_disable(); - flush_tlb_func_local(info, TLB_LOCAL_MM_SHOOTDOWN); + flush_tlb_func_local((__force void *)info); local_irq_enable(); } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index beb44e22afdf..0cb277848cb4 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -2474,6 +2474,8 @@ void __init xen_init_mmu_ops(void) pv_ops.mmu = xen_mmu_ops; + static_key_disable(&flush_tlb_multi_enabled.key); + memset(dummy_mapping, 0xff, PAGE_SIZE); } -- 2.20.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C2E3C31E45 for ; Thu, 13 Jun 2019 06:49:59 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E0C50206E0 for ; Thu, 13 Jun 2019 06:49:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E0C50206E0 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=vmware.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hbJYE-0001ok-3L; Thu, 13 Jun 2019 06:49:34 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hbJYD-0001of-De for xen-devel@lists.xenproject.org; Thu, 13 Jun 2019 06:49:33 +0000 X-Inumbo-ID: 63829e74-8da7-11e9-b23c-5b3821526fa9 Received: from mail-pg1-f195.google.com (unknown [209.85.215.195]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 63829e74-8da7-11e9-b23c-5b3821526fa9; Thu, 13 Jun 2019 06:49:28 +0000 (UTC) Received: by mail-pg1-f195.google.com with SMTP id p10so4507559pgn.1 for ; Wed, 12 Jun 2019 23:49:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EWV8HqUW5nD4lYeuHZCchPJIr3ycVzu67AOhwAlhylY=; b=fEy9huXgDg5t3Rto0e0MQ5IGknBcdadYOE9Td/8RNBnNVLgOvypVX0LLZVh0Nxyzo/ +cme6XFc/xR9Fx+J5I/dBY55OfQ0R0ByuQpUfhObXyvQqsgi7eCdPyIVQCmtXN2YDbSi NGF5V/zKvkyI/r3DLiwQxi4R7V4jkCVGNseHMN7a1WzPNHlxOYcGGa04K1R6+aUw9ZWl 9tPlwYf3+YWMCAQKx78OjsI6Pc0QiksDDZ1VPunTJPtS7zrL3+5AZ36WUlIQJTOts6LB J3ZZ8odcvGlFtZnsIeKz9NcHZYhAYNXUA3HaUMds2cz9RjtBfBKNVF1xXFt0gRA02Y97 KKTw== X-Gm-Message-State: APjAAAWdEo4J/5BL7Z31zioqFTeSgGf2NHeUdQspNvCy+KM33dHy7/13 vrQi+l4elaFhhh7n/2YzXFM= X-Google-Smtp-Source: APXvYqw+K1atiiGJQgLV9SN1EyEIO2KJbWIsQQ3d5N7HeaIcjGnehk9lu87UMmJtEDHDqZuBy4rhjg== X-Received: by 2002:a17:90a:8a10:: with SMTP id w16mr3546662pjn.133.1560408567383; Wed, 12 Jun 2019 23:49:27 -0700 (PDT) Received: from htb-2n-eng-dhcp405.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id i3sm1559973pfa.175.2019.06.12.23.49.19 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 12 Jun 2019 23:49:26 -0700 (PDT) From: Nadav Amit To: Peter Zijlstra , Andy Lutomirski Date: Wed, 12 Jun 2019 23:48:08 -0700 Message-Id: <20190613064813.8102-5-namit@vmware.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190613064813.8102-1-namit@vmware.com> References: <20190613064813.8102-1-namit@vmware.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH 4/9] x86/mm/tlb: Flush remote and local TLBs concurrently X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Juergen Gross , linux-hyperv@vger.kernel.org, Dave Hansen , Stephen Hemminger , xen-devel@lists.xenproject.org, kvm@vger.kernel.org, Haiyang Zhang , x86@kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Ingo Molnar , Borislav Petkov , Paolo Bonzini , Nadav Amit , Thomas Gleixner , "K. Y. Srinivasan" , Boris Ostrovsky Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" VG8gaW1wcm92ZSBUTEIgc2hvb3Rkb3duIHBlcmZvcm1hbmNlLCBmbHVzaCB0aGUgcmVtb3RlIGFu ZCBsb2NhbCBUTEJzCmNvbmN1cnJlbnRseS4gSW50cm9kdWNlIGZsdXNoX3RsYl9tdWx0aSgpIHRo YXQgZG9lcyBzby4gVGhlIGN1cnJlbnQKZmx1c2hfdGxiX290aGVycygpIGludGVyZmFjZSBpcyBr ZXB0LCBzaW5jZSBwYXJhdmlydHVhbCBpbnRlcmZhY2VzIG5lZWQKdG8gYmUgYWRhcHRlZCBmaXJz dCBiZWZvcmUgaXQgY2FuIGJlIHJlbW92ZWQuIFRoaXMgaXMgbGVmdCBmb3IgZnV0dXJlCndvcmsu IEluIHN1Y2ggUFYgZW52aXJvbm1lbnRzLCBUTEIgZmx1c2hlcyBhcmUgbm90IHBlcmZvcm1lZCwg YXQgdGhpcwp0aW1lLCBjb25jdXJyZW50bHkuCgpBZGQgYSBzdGF0aWMga2V5IHRvIHRlbGwgd2hl dGhlciB0aGlzIG5ldyBpbnRlcmZhY2UgaXMgc3VwcG9ydGVkLgoKQ2M6ICJLLiBZLiBTcmluaXZh c2FuIiA8a3lzQG1pY3Jvc29mdC5jb20+CkNjOiBIYWl5YW5nIFpoYW5nIDxoYWl5YW5nekBtaWNy b3NvZnQuY29tPgpDYzogU3RlcGhlbiBIZW1taW5nZXIgPHN0aGVtbWluQG1pY3Jvc29mdC5jb20+ CkNjOiBTYXNoYSBMZXZpbiA8c2FzaGFsQGtlcm5lbC5vcmc+CkNjOiBUaG9tYXMgR2xlaXhuZXIg PHRnbHhAbGludXRyb25peC5kZT4KQ2M6IEluZ28gTW9sbmFyIDxtaW5nb0ByZWRoYXQuY29tPgpD YzogQm9yaXNsYXYgUGV0a292IDxicEBhbGllbjguZGU+CkNjOiB4ODZAa2VybmVsLm9yZwpDYzog SnVlcmdlbiBHcm9zcyA8amdyb3NzQHN1c2UuY29tPgpDYzogUGFvbG8gQm9uemluaSA8cGJvbnpp bmlAcmVkaGF0LmNvbT4KQ2M6IERhdmUgSGFuc2VuIDxkYXZlLmhhbnNlbkBsaW51eC5pbnRlbC5j b20+CkNjOiBBbmR5IEx1dG9taXJza2kgPGx1dG9Aa2VybmVsLm9yZz4KQ2M6IFBldGVyIFppamxz dHJhIDxwZXRlcnpAaW5mcmFkZWFkLm9yZz4KQ2M6IEJvcmlzIE9zdHJvdnNreSA8Ym9yaXMub3N0 cm92c2t5QG9yYWNsZS5jb20+CkNjOiBsaW51eC1oeXBlcnZAdmdlci5rZXJuZWwub3JnCkNjOiBs aW51eC1rZXJuZWxAdmdlci5rZXJuZWwub3JnCkNjOiB2aXJ0dWFsaXphdGlvbkBsaXN0cy5saW51 eC1mb3VuZGF0aW9uLm9yZwpDYzoga3ZtQHZnZXIua2VybmVsLm9yZwpDYzogeGVuLWRldmVsQGxp c3RzLnhlbnByb2plY3Qub3JnClNpZ25lZC1vZmYtYnk6IE5hZGF2IEFtaXQgPG5hbWl0QHZtd2Fy ZS5jb20+Ci0tLQogYXJjaC94ODYvaHlwZXJ2L21tdS5jICAgICAgICAgICAgICAgICB8ICAyICsK IGFyY2gveDg2L2luY2x1ZGUvYXNtL3BhcmF2aXJ0LmggICAgICAgfCAgOCArKysKIGFyY2gveDg2 L2luY2x1ZGUvYXNtL3BhcmF2aXJ0X3R5cGVzLmggfCAgNiArKysKIGFyY2gveDg2L2luY2x1ZGUv YXNtL3RsYmZsdXNoLmggICAgICAgfCAgNiArKysKIGFyY2gveDg2L2tlcm5lbC9rdm0uYyAgICAg ICAgICAgICAgICAgfCAgMSArCiBhcmNoL3g4Ni9rZXJuZWwvcGFyYXZpcnQuYyAgICAgICAgICAg IHwgIDMgKysKIGFyY2gveDg2L21tL3RsYi5jICAgICAgICAgICAgICAgICAgICAgfCA3MSArKysr KysrKysrKysrKysrKysrKysrLS0tLS0KIGFyY2gveDg2L3hlbi9tbXVfcHYuYyAgICAgICAgICAg ICAgICAgfCAgMiArCiA4IGZpbGVzIGNoYW5nZWQsIDg3IGluc2VydGlvbnMoKyksIDEyIGRlbGV0 aW9ucygtKQoKZGlmZiAtLWdpdCBhL2FyY2gveDg2L2h5cGVydi9tbXUuYyBiL2FyY2gveDg2L2h5 cGVydi9tbXUuYwppbmRleCBlNjVkN2ZlNjQ4OWYuLmNhMjhiNDAwYzg3YyAxMDA2NDQKLS0tIGEv YXJjaC94ODYvaHlwZXJ2L21tdS5jCisrKyBiL2FyY2gveDg2L2h5cGVydi9tbXUuYwpAQCAtMjMz LDQgKzIzMyw2IEBAIHZvaWQgaHlwZXJ2X3NldHVwX21tdV9vcHModm9pZCkKIAlwcl9pbmZvKCJV c2luZyBoeXBlcmNhbGwgZm9yIHJlbW90ZSBUTEIgZmx1c2hcbiIpOwogCXB2X29wcy5tbXUuZmx1 c2hfdGxiX290aGVycyA9IGh5cGVydl9mbHVzaF90bGJfb3RoZXJzOwogCXB2X29wcy5tbXUudGxi X3JlbW92ZV90YWJsZSA9IHRsYl9yZW1vdmVfdGFibGU7CisKKwlzdGF0aWNfa2V5X2Rpc2FibGUo JmZsdXNoX3RsYl9tdWx0aV9lbmFibGVkLmtleSk7CiB9CmRpZmYgLS1naXQgYS9hcmNoL3g4Ni9p bmNsdWRlL2FzbS9wYXJhdmlydC5oIGIvYXJjaC94ODYvaW5jbHVkZS9hc20vcGFyYXZpcnQuaApp bmRleCBjMjVjMzhhMDVjMWMuLjE5MmJlNzI1NDQ1NyAxMDA2NDQKLS0tIGEvYXJjaC94ODYvaW5j bHVkZS9hc20vcGFyYXZpcnQuaAorKysgYi9hcmNoL3g4Ni9pbmNsdWRlL2FzbS9wYXJhdmlydC5o CkBAIC00Nyw2ICs0Nyw4IEBAIHN0YXRpYyBpbmxpbmUgdm9pZCBzbG93X2Rvd25faW8odm9pZCkK ICNlbmRpZgogfQogCitERUNMQVJFX1NUQVRJQ19LRVlfVFJVRShmbHVzaF90bGJfbXVsdGlfZW5h YmxlZCk7CisKIHN0YXRpYyBpbmxpbmUgdm9pZCBfX2ZsdXNoX3RsYih2b2lkKQogewogCVBWT1Bf VkNBTEwwKG1tdS5mbHVzaF90bGJfdXNlcik7CkBAIC02Miw2ICs2NCwxMiBAQCBzdGF0aWMgaW5s aW5lIHZvaWQgX19mbHVzaF90bGJfb25lX3VzZXIodW5zaWduZWQgbG9uZyBhZGRyKQogCVBWT1Bf VkNBTEwxKG1tdS5mbHVzaF90bGJfb25lX3VzZXIsIGFkZHIpOwogfQogCitzdGF0aWMgaW5saW5l IHZvaWQgZmx1c2hfdGxiX211bHRpKGNvbnN0IHN0cnVjdCBjcHVtYXNrICpjcHVtYXNrLAorCQkJ CSAgIGNvbnN0IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAqaW5mbykKK3sKKwlQVk9QX1ZDQUxMMiht bXUuZmx1c2hfdGxiX211bHRpLCBjcHVtYXNrLCBpbmZvKTsKK30KKwogc3RhdGljIGlubGluZSB2 b2lkIGZsdXNoX3RsYl9vdGhlcnMoY29uc3Qgc3RydWN0IGNwdW1hc2sgKmNwdW1hc2ssCiAJCQkJ ICAgIGNvbnN0IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAqaW5mbykKIHsKZGlmZiAtLWdpdCBhL2Fy Y2gveDg2L2luY2x1ZGUvYXNtL3BhcmF2aXJ0X3R5cGVzLmggYi9hcmNoL3g4Ni9pbmNsdWRlL2Fz bS9wYXJhdmlydF90eXBlcy5oCmluZGV4IDk0NmY4ZjFmMWVmYy4uYjkzYjNkOTA3MjlhIDEwMDY0 NAotLS0gYS9hcmNoL3g4Ni9pbmNsdWRlL2FzbS9wYXJhdmlydF90eXBlcy5oCisrKyBiL2FyY2gv eDg2L2luY2x1ZGUvYXNtL3BhcmF2aXJ0X3R5cGVzLmgKQEAgLTIxMSw2ICsyMTEsMTIgQEAgc3Ry dWN0IHB2X21tdV9vcHMgewogCXZvaWQgKCpmbHVzaF90bGJfdXNlcikodm9pZCk7CiAJdm9pZCAo KmZsdXNoX3RsYl9rZXJuZWwpKHZvaWQpOwogCXZvaWQgKCpmbHVzaF90bGJfb25lX3VzZXIpKHVu c2lnbmVkIGxvbmcgYWRkcik7CisJLyoKKwkgKiBmbHVzaF90bGJfbXVsdGkoKSBpcyB0aGUgcHJl ZmVycmVkIGludGVyZmFjZSwgd2hpY2ggaXMgY2FwYWJsZSB0bworCSAqIGZsdXNoIGJvdGggbG9j YWwgYW5kIHJlbW90ZSBDUFVzLgorCSAqLworCXZvaWQgKCpmbHVzaF90bGJfbXVsdGkpKGNvbnN0 IHN0cnVjdCBjcHVtYXNrICpjcHVzLAorCQkJCWNvbnN0IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAq aW5mbyk7CiAJdm9pZCAoKmZsdXNoX3RsYl9vdGhlcnMpKGNvbnN0IHN0cnVjdCBjcHVtYXNrICpj cHVzLAogCQkJCSBjb25zdCBzdHJ1Y3QgZmx1c2hfdGxiX2luZm8gKmluZm8pOwogCmRpZmYgLS1n aXQgYS9hcmNoL3g4Ni9pbmNsdWRlL2FzbS90bGJmbHVzaC5oIGIvYXJjaC94ODYvaW5jbHVkZS9h c20vdGxiZmx1c2guaAppbmRleCBkZWUzNzU4MzE5NjIuLjc5MjcyOTM4Y2Y3OSAxMDA2NDQKLS0t IGEvYXJjaC94ODYvaW5jbHVkZS9hc20vdGxiZmx1c2guaAorKysgYi9hcmNoL3g4Ni9pbmNsdWRl L2FzbS90bGJmbHVzaC5oCkBAIC01NjksNiArNTY5LDkgQEAgc3RhdGljIGlubGluZSB2b2lkIGZs dXNoX3RsYl9wYWdlKHN0cnVjdCB2bV9hcmVhX3N0cnVjdCAqdm1hLCB1bnNpZ25lZCBsb25nIGEp CiAJZmx1c2hfdGxiX21tX3JhbmdlKHZtYS0+dm1fbW0sIGEsIGEgKyBQQUdFX1NJWkUsIFBBR0Vf U0hJRlQsIGZhbHNlKTsKIH0KIAordm9pZCBuYXRpdmVfZmx1c2hfdGxiX211bHRpKGNvbnN0IHN0 cnVjdCBjcHVtYXNrICpjcHVtYXNrLAorCQkJICAgICBjb25zdCBzdHJ1Y3QgZmx1c2hfdGxiX2lu Zm8gKmluZm8pOworCiB2b2lkIG5hdGl2ZV9mbHVzaF90bGJfb3RoZXJzKGNvbnN0IHN0cnVjdCBj cHVtYXNrICpjcHVtYXNrLAogCQkJICAgICBjb25zdCBzdHJ1Y3QgZmx1c2hfdGxiX2luZm8gKmlu Zm8pOwogCkBAIC01OTMsNiArNTk2LDkgQEAgc3RhdGljIGlubGluZSB2b2lkIGFyY2hfdGxiYmF0 Y2hfYWRkX21tKHN0cnVjdCBhcmNoX3RsYmZsdXNoX3VubWFwX2JhdGNoICpiYXRjaCwKIGV4dGVy biB2b2lkIGFyY2hfdGxiYmF0Y2hfZmx1c2goc3RydWN0IGFyY2hfdGxiZmx1c2hfdW5tYXBfYmF0 Y2ggKmJhdGNoKTsKIAogI2lmbmRlZiBDT05GSUdfUEFSQVZJUlQKKyNkZWZpbmUgZmx1c2hfdGxi X211bHRpKG1hc2ssIGluZm8pCVwKKwluYXRpdmVfZmx1c2hfdGxiX211bHRpKG1hc2ssIGluZm8p CisKICNkZWZpbmUgZmx1c2hfdGxiX290aGVycyhtYXNrLCBpbmZvKQlcCiAJbmF0aXZlX2ZsdXNo X3RsYl9vdGhlcnMobWFzaywgaW5mbykKIApkaWZmIC0tZ2l0IGEvYXJjaC94ODYva2VybmVsL2t2 bS5jIGIvYXJjaC94ODYva2VybmVsL2t2bS5jCmluZGV4IDUxNjliOGNjMzViYi4uMDBkODFlODk4 NzE3IDEwMDY0NAotLS0gYS9hcmNoL3g4Ni9rZXJuZWwva3ZtLmMKKysrIGIvYXJjaC94ODYva2Vy bmVsL2t2bS5jCkBAIC02MzAsNiArNjMwLDcgQEAgc3RhdGljIHZvaWQgX19pbml0IGt2bV9ndWVz dF9pbml0KHZvaWQpCiAJICAgIGt2bV9wYXJhX2hhc19mZWF0dXJlKEtWTV9GRUFUVVJFX1NURUFM X1RJTUUpKSB7CiAJCXB2X29wcy5tbXUuZmx1c2hfdGxiX290aGVycyA9IGt2bV9mbHVzaF90bGJf b3RoZXJzOwogCQlwdl9vcHMubW11LnRsYl9yZW1vdmVfdGFibGUgPSB0bGJfcmVtb3ZlX3RhYmxl OworCQlzdGF0aWNfa2V5X2Rpc2FibGUoJmZsdXNoX3RsYl9tdWx0aV9lbmFibGVkLmtleSk7CiAJ fQogCiAJaWYgKGt2bV9wYXJhX2hhc19mZWF0dXJlKEtWTV9GRUFUVVJFX1BWX0VPSSkpCmRpZmYg LS1naXQgYS9hcmNoL3g4Ni9rZXJuZWwvcGFyYXZpcnQuYyBiL2FyY2gveDg2L2tlcm5lbC9wYXJh dmlydC5jCmluZGV4IDk4MDM5ZDdmYjk5OC4uYWMwMGFmZWQ1NTcwIDEwMDY0NAotLS0gYS9hcmNo L3g4Ni9rZXJuZWwvcGFyYXZpcnQuYworKysgYi9hcmNoL3g4Ni9rZXJuZWwvcGFyYXZpcnQuYwpA QCAtMTU5LDYgKzE1OSw4IEBAIHVuc2lnbmVkIHBhcmF2aXJ0X3BhdGNoX2luc25zKHZvaWQgKmlu c25fYnVmZiwgdW5zaWduZWQgbGVuLAogCXJldHVybiBpbnNuX2xlbjsKIH0KIAorREVGSU5FX1NU QVRJQ19LRVlfVFJVRShmbHVzaF90bGJfbXVsdGlfZW5hYmxlZCk7CisKIHN0YXRpYyB2b2lkIG5h dGl2ZV9mbHVzaF90bGIodm9pZCkKIHsKIAlfX25hdGl2ZV9mbHVzaF90bGIoKTsKQEAgLTM2Myw2 ICszNjUsNyBAQCBzdHJ1Y3QgcGFyYXZpcnRfcGF0Y2hfdGVtcGxhdGUgcHZfb3BzID0gewogCS5t bXUuZmx1c2hfdGxiX3VzZXIJPSBuYXRpdmVfZmx1c2hfdGxiLAogCS5tbXUuZmx1c2hfdGxiX2tl cm5lbAk9IG5hdGl2ZV9mbHVzaF90bGJfZ2xvYmFsLAogCS5tbXUuZmx1c2hfdGxiX29uZV91c2Vy CT0gbmF0aXZlX2ZsdXNoX3RsYl9vbmVfdXNlciwKKwkubW11LmZsdXNoX3RsYl9tdWx0aQk9IG5h dGl2ZV9mbHVzaF90bGJfbXVsdGksCiAJLm1tdS5mbHVzaF90bGJfb3RoZXJzCT0gbmF0aXZlX2Zs dXNoX3RsYl9vdGhlcnMsCiAJLm1tdS50bGJfcmVtb3ZlX3RhYmxlCT0KIAkJCSh2b2lkICgqKShz dHJ1Y3QgbW11X2dhdGhlciAqLCB2b2lkICopKXRsYl9yZW1vdmVfcGFnZSwKZGlmZiAtLWdpdCBh L2FyY2gveDg2L21tL3RsYi5jIGIvYXJjaC94ODYvbW0vdGxiLmMKaW5kZXggYzM0YmNmMDNmMDZm Li5kYjczZDVmMWRkNDMgMTAwNjQ0Ci0tLSBhL2FyY2gveDg2L21tL3RsYi5jCisrKyBiL2FyY2gv eDg2L21tL3RsYi5jCkBAIC01NTEsNyArNTUxLDcgQEAgc3RhdGljIHZvaWQgZmx1c2hfdGxiX2Z1 bmNfY29tbW9uKGNvbnN0IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAqZiwKIAkJICogZ2FyYmFnZSBp bnRvIG91ciBUTEIuICBTaW5jZSBzd2l0Y2hpbmcgdG8gaW5pdF9tbSBpcyBiYXJlbHkKIAkJICog c2xvd2VyIHRoYW4gYSBtaW5pbWFsIGZsdXNoLCBqdXN0IHN3aXRjaCB0byBpbml0X21tLgogCQkg KgotCQkgKiBUaGlzIHNob3VsZCBiZSByYXJlLCB3aXRoIG5hdGl2ZV9mbHVzaF90bGJfb3RoZXJz IHNraXBwaW5nCisJCSAqIFRoaXMgc2hvdWxkIGJlIHJhcmUsIHdpdGggbmF0aXZlX2ZsdXNoX3Rs Yl9tdWx0aSBza2lwcGluZwogCQkgKiBJUElzIHRvIGxhenkgVExCIG1vZGUgQ1BVcy4KIAkJICov CiAJCXN3aXRjaF9tbV9pcnFzX29mZihOVUxMLCAmaW5pdF9tbSwgTlVMTCk7CkBAIC02MzUsOSAr NjM1LDEyIEBAIHN0YXRpYyB2b2lkIGZsdXNoX3RsYl9mdW5jX2NvbW1vbihjb25zdCBzdHJ1Y3Qg Zmx1c2hfdGxiX2luZm8gKmYsCiAJdGhpc19jcHVfd3JpdGUoY3B1X3RsYnN0YXRlLmN0eHNbbG9h ZGVkX21tX2FzaWRdLnRsYl9nZW4sIG1tX3RsYl9nZW4pOwogfQogCi1zdGF0aWMgdm9pZCBmbHVz aF90bGJfZnVuY19sb2NhbChjb25zdCB2b2lkICppbmZvLCBlbnVtIHRsYl9mbHVzaF9yZWFzb24g cmVhc29uKQorc3RhdGljIHZvaWQgZmx1c2hfdGxiX2Z1bmNfbG9jYWwodm9pZCAqaW5mbykKIHsK IAljb25zdCBzdHJ1Y3QgZmx1c2hfdGxiX2luZm8gKmYgPSBpbmZvOworCWVudW0gdGxiX2ZsdXNo X3JlYXNvbiByZWFzb247CisKKwlyZWFzb24gPSAoZi0+bW0gPT0gTlVMTCkgPyBUTEJfTE9DQUxf U0hPT1RET1dOIDogVExCX0xPQ0FMX01NX1NIT09URE9XTjsKIAogCWZsdXNoX3RsYl9mdW5jX2Nv bW1vbihmLCB0cnVlLCByZWFzb24pOwogfQpAQCAtNjU1LDE0ICs2NTgsMjEgQEAgc3RhdGljIHZv aWQgZmx1c2hfdGxiX2Z1bmNfcmVtb3RlKHZvaWQgKmluZm8pCiAJZmx1c2hfdGxiX2Z1bmNfY29t bW9uKGYsIGZhbHNlLCBUTEJfUkVNT1RFX1NIT09URE9XTik7CiB9CiAKLXN0YXRpYyBib29sIHRs Yl9pc19ub3RfbGF6eShpbnQgY3B1LCB2b2lkICpkYXRhKQorc3RhdGljIGlubGluZSBib29sIHRs Yl9pc19ub3RfbGF6eShpbnQgY3B1KQogewogCXJldHVybiAhcGVyX2NwdShjcHVfdGxic3RhdGUu aXNfbGF6eSwgY3B1KTsKIH0KIAotdm9pZCBuYXRpdmVfZmx1c2hfdGxiX290aGVycyhjb25zdCBz dHJ1Y3QgY3B1bWFzayAqY3B1bWFzaywKLQkJCSAgICAgY29uc3Qgc3RydWN0IGZsdXNoX3RsYl9p bmZvICppbmZvKQorc3RhdGljIERFRklORV9QRVJfQ1BVKGNwdW1hc2tfdCwgZmx1c2hfdGxiX21h c2spOworCit2b2lkIG5hdGl2ZV9mbHVzaF90bGJfbXVsdGkoY29uc3Qgc3RydWN0IGNwdW1hc2sg KmNwdW1hc2ssCisJCQkgICAgY29uc3Qgc3RydWN0IGZsdXNoX3RsYl9pbmZvICppbmZvKQogewor CS8qCisJICogRG8gYWNjb3VudGluZyBhbmQgdHJhY2luZy4gTm90ZSB0aGF0IHRoZXJlIGFyZSAo YW5kIGhhdmUgYWx3YXlzIGJlZW4pCisJICogY2FzZXMgaW4gd2hpY2ggYSByZW1vdGUgVExCIGZs dXNoIHdpbGwgYmUgdHJhY2VkLCBidXQgZXZlbnR1YWxseQorCSAqIHdvdWxkIG5vdCBoYXBwZW4u CisJICovCiAJY291bnRfdm1fdGxiX2V2ZW50KE5SX1RMQl9SRU1PVEVfRkxVU0gpOwogCWlmIChp bmZvLT5lbmQgPT0gVExCX0ZMVVNIX0FMTCkKIAkJdHJhY2VfdGxiX2ZsdXNoKFRMQl9SRU1PVEVf U0VORF9JUEksIFRMQl9GTFVTSF9BTEwpOwpAQCAtNjgyLDEwICs2OTIsMTQgQEAgdm9pZCBuYXRp dmVfZmx1c2hfdGxiX290aGVycyhjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1bWFzaywKIAkJICog bWVhbnMgdGhhdCB0aGUgcGVyY3B1IHRsYl9nZW4gdmFyaWFibGVzIHdvbid0IGJlIHVwZGF0ZWQK IAkJICogYW5kIHdlJ2xsIGRvIHBvaW50bGVzcyBmbHVzaGVzIG9uIGZ1dHVyZSBjb250ZXh0IHN3 aXRjaGVzLgogCQkgKgotCQkgKiBSYXRoZXIgdGhhbiBob29raW5nIG5hdGl2ZV9mbHVzaF90bGJf b3RoZXJzKCkgaGVyZSwgSSB0aGluaworCQkgKiBSYXRoZXIgdGhhbiBob29raW5nIG5hdGl2ZV9m bHVzaF90bGJfbXVsdGkoKSBoZXJlLCBJIHRoaW5rCiAJCSAqIHRoYXQgVVYgc2hvdWxkIGJlIHVw ZGF0ZWQgc28gdGhhdCBzbXBfY2FsbF9mdW5jdGlvbl9tYW55KCksCiAJCSAqIGV0YywgYXJlIG9w dGltYWwgb24gVVYuCiAJCSAqLworCQlsb2NhbF9pcnFfZGlzYWJsZSgpOworCQlmbHVzaF90bGJf ZnVuY19sb2NhbCgoX19mb3JjZSB2b2lkICopaW5mbyk7CisJCWxvY2FsX2lycV9lbmFibGUoKTsK KwogCQljcHVtYXNrID0gdXZfZmx1c2hfdGxiX290aGVycyhjcHVtYXNrLCBpbmZvKTsKIAkJaWYg KGNwdW1hc2spCiAJCQlzbXBfY2FsbF9mdW5jdGlvbl9tYW55KGNwdW1hc2ssIGZsdXNoX3RsYl9m dW5jX3JlbW90ZSwKQEAgLTcwNCwxMSArNzE4LDM5IEBAIHZvaWQgbmF0aXZlX2ZsdXNoX3RsYl9v dGhlcnMoY29uc3Qgc3RydWN0IGNwdW1hc2sgKmNwdW1hc2ssCiAJICogZG9pbmcgYSBzcGVjdWxh dGl2ZSBtZW1vcnkgYWNjZXNzLgogCSAqLwogCWlmIChpbmZvLT5mcmVlZF90YWJsZXMpCi0JCXNt cF9jYWxsX2Z1bmN0aW9uX21hbnkoY3B1bWFzaywgZmx1c2hfdGxiX2Z1bmNfcmVtb3RlLAotCQkJ ICAgICAgICh2b2lkICopaW5mbywgMSk7Ci0JZWxzZQotCQlvbl9lYWNoX2NwdV9jb25kX21hc2so dGxiX2lzX25vdF9sYXp5LCBmbHVzaF90bGJfZnVuY19yZW1vdGUsCi0JCQkJKHZvaWQgKilpbmZv LCAxLCBHRlBfQVRPTUlDLCBjcHVtYXNrKTsKKwkJX19zbXBfY2FsbF9mdW5jdGlvbl9tYW55KGNw dW1hc2ssIGZsdXNoX3RsYl9mdW5jX3JlbW90ZSwKKwkJCQkJIGZsdXNoX3RsYl9mdW5jX2xvY2Fs LCAodm9pZCAqKWluZm8sIDEpOworCWVsc2UgeworCQkvKgorCQkgKiBBbHRob3VnaCB3ZSBjb3Vs ZCBoYXZlIHVzZWQgb25fZWFjaF9jcHVfY29uZF9tYXNrKCksCisJCSAqIG9wZW4tY29kaW5nIGl0 IGhhcyBzZXZlcmFsIHBlcmZvcm1hbmNlIGFkdmFudGFnZXM6ICgxKSB3ZSBjYW4KKwkJICogdXNl IHNwZWNpYWxpemVkIGZ1bmN0aW9ucyBmb3IgcmVtb3RlIGFuZCBsb2NhbCBmbHVzaGVzOyAoMikK KwkJICogbm8gbmVlZCBmb3IgaW5kaXJlY3QgYnJhbmNoIHRvIHRlc3QgaWYgVExCIGlzIGxhenk7 ICgzKSB3ZQorCQkgKiBjYW4gdXNlIGEgZGVzaWduYXRlZCBjcHVtYXNrIGZvciBldmFsdWF0aW5n IHRoZSBjb25kaXRpb24KKwkJICogaW5zdGVhZCBvZiBhbGxvY2F0aW5nIGEgbmV3IG9uZS4KKwkJ ICoKKwkJICogVGhpcyB3b3JrcyB1bmRlciB0aGUgYXNzdW1wdGlvbiB0aGF0IHRoZXJlIGFyZSBu byBuZXN0ZWQgVExCCisJCSAqIGZsdXNoZXMsIGFuIGFzc3VtcHRpb24gdGhhdCBpcyBhbHJlYWR5 IG1hZGUgaW4KKwkJICogZmx1c2hfdGxiX21tX3JhbmdlKCkuCisJCSAqLworCQlzdHJ1Y3QgY3B1 bWFzayAqY29uZF9jcHVtYXNrID0gdGhpc19jcHVfcHRyKCZmbHVzaF90bGJfbWFzayk7CisJCWlu dCBjcHU7CisKKwkJY3B1bWFza19jbGVhcihjb25kX2NwdW1hc2spOworCisJCWZvcl9lYWNoX2Nw dShjcHUsIGNwdW1hc2spIHsKKwkJCWlmICh0bGJfaXNfbm90X2xhenkoY3B1KSkKKwkJCQlfX2Nw dW1hc2tfc2V0X2NwdShjcHUsIGNvbmRfY3B1bWFzayk7CisJCX0KKwkJX19zbXBfY2FsbF9mdW5j dGlvbl9tYW55KGNvbmRfY3B1bWFzaywgZmx1c2hfdGxiX2Z1bmNfcmVtb3RlLAorCQkJCQkgZmx1 c2hfdGxiX2Z1bmNfbG9jYWwsICh2b2lkICopaW5mbywgMSk7CisJfQorfQorCit2b2lkIG5hdGl2 ZV9mbHVzaF90bGJfb3RoZXJzKGNvbnN0IHN0cnVjdCBjcHVtYXNrICpjcHVtYXNrLAorCQkJICAg ICBjb25zdCBzdHJ1Y3QgZmx1c2hfdGxiX2luZm8gKmluZm8pCit7CisJbmF0aXZlX2ZsdXNoX3Rs Yl9tdWx0aShjcHVtYXNrLCBpbmZvKTsKIH0KIAogLyoKQEAgLTc3NCwxMCArODE2LDE1IEBAIHN0 YXRpYyB2b2lkIGZsdXNoX3RsYl9vbl9jcHVzKGNvbnN0IGNwdW1hc2tfdCAqY3B1bWFzaywKIHsK IAlpbnQgdGhpc19jcHUgPSBzbXBfcHJvY2Vzc29yX2lkKCk7CiAKKwlpZiAoc3RhdGljX2JyYW5j aF9saWtlbHkoJmZsdXNoX3RsYl9tdWx0aV9lbmFibGVkKSkgeworCQlmbHVzaF90bGJfbXVsdGko Y3B1bWFzaywgaW5mbyk7CisJCXJldHVybjsKKwl9CisKIAlpZiAoY3B1bWFza190ZXN0X2NwdSh0 aGlzX2NwdSwgY3B1bWFzaykpIHsKIAkJbG9ja2RlcF9hc3NlcnRfaXJxc19lbmFibGVkKCk7CiAJ CWxvY2FsX2lycV9kaXNhYmxlKCk7Ci0JCWZsdXNoX3RsYl9mdW5jX2xvY2FsKGluZm8sIFRMQl9M T0NBTF9NTV9TSE9PVERPV04pOworCQlmbHVzaF90bGJfZnVuY19sb2NhbCgoX19mb3JjZSB2b2lk ICopaW5mbyk7CiAJCWxvY2FsX2lycV9lbmFibGUoKTsKIAl9CiAKZGlmZiAtLWdpdCBhL2FyY2gv eDg2L3hlbi9tbXVfcHYuYyBiL2FyY2gveDg2L3hlbi9tbXVfcHYuYwppbmRleCBiZWI0NGUyMmFm ZGYuLjBjYjI3Nzg0OGNiNCAxMDA2NDQKLS0tIGEvYXJjaC94ODYveGVuL21tdV9wdi5jCisrKyBi L2FyY2gveDg2L3hlbi9tbXVfcHYuYwpAQCAtMjQ3NCw2ICsyNDc0LDggQEAgdm9pZCBfX2luaXQg eGVuX2luaXRfbW11X29wcyh2b2lkKQogCiAJcHZfb3BzLm1tdSA9IHhlbl9tbXVfb3BzOwogCisJ c3RhdGljX2tleV9kaXNhYmxlKCZmbHVzaF90bGJfbXVsdGlfZW5hYmxlZC5rZXkpOworCiAJbWVt c2V0KGR1bW15X21hcHBpbmcsIDB4ZmYsIFBBR0VfU0laRSk7CiB9CiAKLS0gCjIuMjAuMQoKCl9f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClhlbi1kZXZlbCBt YWlsaW5nIGxpc3QKWGVuLWRldmVsQGxpc3RzLnhlbnByb2plY3Qub3JnCmh0dHBzOi8vbGlzdHMu eGVucHJvamVjdC5vcmcvbWFpbG1hbi9saXN0aW5mby94ZW4tZGV2ZWw=