From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.9 required=3.0 tests=DATE_IN_PAST_06_12, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AC31C0650E for ; Wed, 3 Jul 2019 07:14:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2ABFF2189F for ; Wed, 3 Jul 2019 07:14:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727264AbfGCHOL (ORCPT ); Wed, 3 Jul 2019 03:14:11 -0400 Received: from mail-pl1-f193.google.com ([209.85.214.193]:38707 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726236AbfGCHOJ (ORCPT ); Wed, 3 Jul 2019 03:14:09 -0400 Received: by mail-pl1-f193.google.com with SMTP id 9so713527ple.5; Wed, 03 Jul 2019 00:14:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ccH1lY4O7n4BXQuRS1GgwOEGuGvqXICf1/+JCvsaWLs=; b=brtAaApk3IjptbtCTY1PFy+nzymP6csVNXYCozRsX3HPpzNN+BAs5Fwg9RiTPW0c+s xe04LZWkGZfRXIEka4s9rEfyduWgy2JHoImF2Rk68YeDN036CGtqYdW/C4QFsBLKZh0b 5M0HTBNXRGzkaZK2IZo9JpUvE6uM1R86tb5LpcAd2NQTr6pVeuQr6vOPI0c8MnF2N1ol cnYc+AAP+6Vfmeab6wXl2Hw4oqJiePCrloB6bFaGJnwPhPC5Fx9LNkD2YLHExe1kLmcu 1vFMrcWXSKYJwLKIP5vY5s/uoDjK/cSL/AIFmj3iRgka5nTTb0LsIzzzZwsvCqJbT1bc UEyg== X-Gm-Message-State: APjAAAVFCpVTNs6ZTeyXHCQubZHguYk+o0q4qD9CC1GccnW8OK62vEqC q0cL+VtI+kUL7tsKArKeAIc= X-Google-Smtp-Source: APXvYqxDiu4XHF/h685l75XKdA+4AtAWMjLUqXNKPxMi634QNbO+Oh5f6iZUYJKpj5KI9u+azIzt2A== X-Received: by 2002:a17:902:2983:: with SMTP id h3mr40803893plb.45.1562138048504; Wed, 03 Jul 2019 00:14:08 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j21sm1256593pfh.86.2019.07.03.00.14.06 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 03 Jul 2019 00:14:07 -0700 (PDT) From: Nadav Amit To: Andy Lutomirski , Dave Hansen Cc: x86@kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Thomas Gleixner , Ingo Molnar , Nadav Amit , "K. Y. Srinivasan" , Haiyang Zhang , Stephen Hemminger , Sasha Levin , Borislav Petkov , Juergen Gross , Paolo Bonzini , Boris Ostrovsky , linux-hyperv@vger.kernel.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, xen-devel@lists.xenproject.org Subject: [PATCH v2 4/9] x86/mm/tlb: Flush remote and local TLBs concurrently Date: Tue, 2 Jul 2019 16:51:46 -0700 Message-Id: <20190702235151.4377-5-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190702235151.4377-1-namit@vmware.com> References: <20190702235151.4377-1-namit@vmware.com> Sender: linux-hyperv-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-hyperv@vger.kernel.org To improve TLB shootdown performance, flush the remote and local TLBs concurrently. Introduce flush_tlb_multi() that does so. Introduce paravirtual versions of flush_tlb_multi() for KVM, Xen and hyper-v (Xen and hyper-v are only compile-tested). While the updated smp infrastructure is capable of running a function on a single local core, it is not optimized for this case. The multiple function calls and the indirect branch introduce some overhead, and might make local TLB flushes slower than they were before the recent changes. Before calling the SMP infrastructure, check if only a local TLB flush is needed to restore the lost performance in this common case. This requires to check mm_cpumask() one more time, but unless this mask is updated very frequently, this should impact performance negatively. Cc: "K. Y. Srinivasan" Cc: Haiyang Zhang Cc: Stephen Hemminger Cc: Sasha Levin Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: Juergen Gross Cc: Paolo Bonzini Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: Boris Ostrovsky Cc: linux-hyperv@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: kvm@vger.kernel.org Cc: xen-devel@lists.xenproject.org Signed-off-by: Nadav Amit --- arch/x86/hyperv/mmu.c | 13 +++--- arch/x86/include/asm/paravirt.h | 6 +-- arch/x86/include/asm/paravirt_types.h | 4 +- arch/x86/include/asm/tlbflush.h | 9 ++-- arch/x86/include/asm/trace/hyperv.h | 2 +- arch/x86/kernel/kvm.c | 11 +++-- arch/x86/kernel/paravirt.c | 2 +- arch/x86/mm/tlb.c | 65 ++++++++++++++++++++------- arch/x86/xen/mmu_pv.c | 20 ++++++--- include/trace/events/xen.h | 2 +- 10 files changed, 91 insertions(+), 43 deletions(-) diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c index e65d7fe6489f..1177f863e4cd 100644 --- a/arch/x86/hyperv/mmu.c +++ b/arch/x86/hyperv/mmu.c @@ -50,8 +50,8 @@ static inline int fill_gva_list(u64 gva_list[], int offset, return gva_n - offset; } -static void hyperv_flush_tlb_others(const struct cpumask *cpus, - const struct flush_tlb_info *info) +static void hyperv_flush_tlb_multi(const struct cpumask *cpus, + const struct flush_tlb_info *info) { int cpu, vcpu, gva_n, max_gvas; struct hv_tlb_flush **flush_pcpu; @@ -59,7 +59,7 @@ static void hyperv_flush_tlb_others(const struct cpumask *cpus, u64 status = U64_MAX; unsigned long flags; - trace_hyperv_mmu_flush_tlb_others(cpus, info); + trace_hyperv_mmu_flush_tlb_multi(cpus, info); if (!hv_hypercall_pg) goto do_native; @@ -69,6 +69,9 @@ static void hyperv_flush_tlb_others(const struct cpumask *cpus, local_irq_save(flags); + if (cpumask_test_cpu(smp_processor_id(), cpus)) + flush_tlb_func_local(info); + flush_pcpu = (struct hv_tlb_flush **) this_cpu_ptr(hyperv_pcpu_input_arg); @@ -156,7 +159,7 @@ static void hyperv_flush_tlb_others(const struct cpumask *cpus, if (!(status & HV_HYPERCALL_RESULT_MASK)) return; do_native: - native_flush_tlb_others(cpus, info); + native_flush_tlb_multi(cpus, info); } static u64 hyperv_flush_tlb_others_ex(const struct cpumask *cpus, @@ -231,6 +234,6 @@ void hyperv_setup_mmu_ops(void) return; pr_info("Using hypercall for remote TLB flush\n"); - pv_ops.mmu.flush_tlb_others = hyperv_flush_tlb_others; + pv_ops.mmu.flush_tlb_multi = hyperv_flush_tlb_multi; pv_ops.mmu.tlb_remove_table = tlb_remove_table; } diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index c25c38a05c1c..316959e89258 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -62,10 +62,10 @@ static inline void __flush_tlb_one_user(unsigned long addr) PVOP_VCALL1(mmu.flush_tlb_one_user, addr); } -static inline void flush_tlb_others(const struct cpumask *cpumask, - const struct flush_tlb_info *info) +static inline void flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info) { - PVOP_VCALL2(mmu.flush_tlb_others, cpumask, info); + PVOP_VCALL2(mmu.flush_tlb_multi, cpumask, info); } static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h index 946f8f1f1efc..54f4c718b5b0 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -211,8 +211,8 @@ struct pv_mmu_ops { void (*flush_tlb_user)(void); void (*flush_tlb_kernel)(void); void (*flush_tlb_one_user)(unsigned long addr); - void (*flush_tlb_others)(const struct cpumask *cpus, - const struct flush_tlb_info *info); + void (*flush_tlb_multi)(const struct cpumask *cpus, + const struct flush_tlb_info *info); void (*tlb_remove_table)(struct mmu_gather *tlb, void *table); diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index dee375831962..36aa2a9b7597 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -517,7 +517,7 @@ static inline void __flush_tlb_one_kernel(unsigned long addr) * - flush_tlb_page(vma, vmaddr) flushes one page * - flush_tlb_range(vma, start, end) flushes a range of pages * - flush_tlb_kernel_range(start, end) flushes a range of kernel pages - * - flush_tlb_others(cpumask, info) flushes TLBs on other cpus + * - flush_tlb_multi(cpumask, info) flushes TLBs on multiple cpus * * ..but the i386 has somewhat limited tlb flushing capabilities, * and page-granular flushes are available only on i486 and up. @@ -563,13 +563,14 @@ extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); +extern void flush_tlb_func_local(const struct flush_tlb_info *info); static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false); } -void native_flush_tlb_others(const struct cpumask *cpumask, +void native_flush_tlb_multi(const struct cpumask *cpumask, const struct flush_tlb_info *info); static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) @@ -593,8 +594,8 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); #ifndef CONFIG_PARAVIRT -#define flush_tlb_others(mask, info) \ - native_flush_tlb_others(mask, info) +#define flush_tlb_multi(mask, info) \ + native_flush_tlb_multi(mask, info) #define paravirt_tlb_remove_table(tlb, page) \ tlb_remove_page(tlb, (void *)(page)) diff --git a/arch/x86/include/asm/trace/hyperv.h b/arch/x86/include/asm/trace/hyperv.h index ace464f09681..85ca8560c7f9 100644 --- a/arch/x86/include/asm/trace/hyperv.h +++ b/arch/x86/include/asm/trace/hyperv.h @@ -8,7 +8,7 @@ #if IS_ENABLED(CONFIG_HYPERV) -TRACE_EVENT(hyperv_mmu_flush_tlb_others, +TRACE_EVENT(hyperv_mmu_flush_tlb_multi, TP_PROTO(const struct cpumask *cpus, const struct flush_tlb_info *info), TP_ARGS(cpus, info), diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 5169b8cc35bb..d00d551d4a2a 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -580,7 +580,7 @@ static void __init kvm_apf_trap_init(void) static DEFINE_PER_CPU(cpumask_var_t, __pv_tlb_mask); -static void kvm_flush_tlb_others(const struct cpumask *cpumask, +static void kvm_flush_tlb_multi(const struct cpumask *cpumask, const struct flush_tlb_info *info) { u8 state; @@ -594,6 +594,11 @@ static void kvm_flush_tlb_others(const struct cpumask *cpumask, * queue flush_on_enter for pre-empted vCPUs */ for_each_cpu(cpu, flushmask) { + /* + * The local vCPU is never preempted, so we do not explicitly + * skip check for local vCPU - it will never be cleared from + * flushmask. + */ src = &per_cpu(steal_time, cpu); state = READ_ONCE(src->preempted); if ((state & KVM_VCPU_PREEMPTED)) { @@ -603,7 +608,7 @@ static void kvm_flush_tlb_others(const struct cpumask *cpumask, } } - native_flush_tlb_others(flushmask, info); + native_flush_tlb_multi(flushmask, info); } static void __init kvm_guest_init(void) @@ -628,7 +633,7 @@ static void __init kvm_guest_init(void) if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH) && !kvm_para_has_hint(KVM_HINTS_REALTIME) && kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) { - pv_ops.mmu.flush_tlb_others = kvm_flush_tlb_others; + pv_ops.mmu.flush_tlb_multi = kvm_flush_tlb_multi; pv_ops.mmu.tlb_remove_table = tlb_remove_table; } diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 98039d7fb998..7cdcffe2a028 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -363,7 +363,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_user = native_flush_tlb, .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, - .mmu.flush_tlb_others = native_flush_tlb_others, + .mmu.flush_tlb_multi = native_flush_tlb_multi, .mmu.tlb_remove_table = (void (*)(struct mmu_gather *, void *))tlb_remove_page, diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 5c9b1607191d..074288a6916e 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -551,7 +551,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, * garbage into our TLB. Since switching to init_mm is barely * slower than a minimal flush, just switch to init_mm. * - * This should be rare, with native_flush_tlb_others skipping + * This should be rare, with native_flush_tlb_multi() skipping * IPIs to lazy TLB mode CPUs. */ switch_mm_irqs_off(NULL, &init_mm, NULL); @@ -635,7 +635,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, this_cpu_write(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen, mm_tlb_gen); } -static void flush_tlb_func_local(void *info) +static void __flush_tlb_func_local(void *info) { const struct flush_tlb_info *f = info; enum tlb_flush_reason reason; @@ -645,6 +645,11 @@ static void flush_tlb_func_local(void *info) flush_tlb_func_common(f, true, reason); } +void flush_tlb_func_local(const struct flush_tlb_info *info) +{ + __flush_tlb_func_local((void *)info); +} + static void flush_tlb_func_remote(void *info) { const struct flush_tlb_info *f = info; @@ -665,9 +670,14 @@ static bool tlb_is_not_lazy(int cpu) static DEFINE_PER_CPU(cpumask_t, flush_tlb_mask); -void native_flush_tlb_others(const struct cpumask *cpumask, - const struct flush_tlb_info *info) +void native_flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info) { + /* + * Do accounting and tracing. Note that there are (and have always been) + * cases in which a remote TLB flush will be traced, but eventually + * would not happen. + */ count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); if (info->end == TLB_FLUSH_ALL) trace_tlb_flush(TLB_REMOTE_SEND_IPI, TLB_FLUSH_ALL); @@ -687,10 +697,12 @@ void native_flush_tlb_others(const struct cpumask *cpumask, * means that the percpu tlb_gen variables won't be updated * and we'll do pointless flushes on future context switches. * - * Rather than hooking native_flush_tlb_others() here, I think + * Rather than hooking native_flush_tlb_multi() here, I think * that UV should be updated so that smp_call_function_many(), * etc, are optimal on UV. */ + flush_tlb_func_local(info); + cpumask = uv_flush_tlb_others(cpumask, info); if (cpumask) smp_call_function_many(cpumask, flush_tlb_func_remote, @@ -709,8 +721,9 @@ void native_flush_tlb_others(const struct cpumask *cpumask, * doing a speculative memory access. */ if (info->freed_tables) { - smp_call_function_many(cpumask, flush_tlb_func_remote, - (void *)info, 1); + __smp_call_function_many(cpumask, flush_tlb_func_remote, + __flush_tlb_func_local, + (void *)info, 1); } else { /* * Although we could have used on_each_cpu_cond_mask(), @@ -737,7 +750,8 @@ void native_flush_tlb_others(const struct cpumask *cpumask, if (tlb_is_not_lazy(cpu)) __cpumask_set_cpu(cpu, cond_cpumask); } - smp_call_function_many(cond_cpumask, flush_tlb_func_remote, + __smp_call_function_many(cond_cpumask, flush_tlb_func_remote, + __flush_tlb_func_local, (void *)info, 1); } } @@ -818,16 +832,29 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, info = get_flush_tlb_info(mm, start, end, stride_shift, freed_tables, new_tlb_gen); - if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) { + /* + * Assert that mm_cpumask() corresponds with the loaded mm. We got one + * exception: for init_mm we do not need to flush anything, and the + * cpumask does not correspond with loaded_mm. + */ + VM_WARN_ON_ONCE(cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)) != + (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) && + mm != &init_mm); + + /* + * flush_tlb_multi() is not optimized for the common case in which only + * a local TLB flush is needed. Optimize this use-case by calling + * flush_tlb_func_local() directly in this case. + */ + if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { + flush_tlb_multi(mm_cpumask(mm), info); + } else { lockdep_assert_irqs_enabled(); local_irq_disable(); flush_tlb_func_local(info); local_irq_enable(); } - if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) - flush_tlb_others(mm_cpumask(mm), info); - put_flush_tlb_info(); put_cpu(); } @@ -890,16 +917,20 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { int cpu = get_cpu(); - if (cpumask_test_cpu(cpu, &batch->cpumask)) { + /* + * flush_tlb_multi() is not optimized for the common case in which only + * a local TLB flush is needed. Optimize this use-case by calling + * flush_tlb_func_local() directly in this case. + */ + if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + flush_tlb_multi(&batch->cpumask, &full_flush_tlb_info); + } else { lockdep_assert_irqs_enabled(); local_irq_disable(); - flush_tlb_func_local((void *)&full_flush_tlb_info); + flush_tlb_func_local(&full_flush_tlb_info); local_irq_enable(); } - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) - flush_tlb_others(&batch->cpumask, &full_flush_tlb_info); - cpumask_clear(&batch->cpumask); put_cpu(); diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index beb44e22afdf..19e481e6e904 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -1355,8 +1355,8 @@ static void xen_flush_tlb_one_user(unsigned long addr) preempt_enable(); } -static void xen_flush_tlb_others(const struct cpumask *cpus, - const struct flush_tlb_info *info) +static void xen_flush_tlb_multi(const struct cpumask *cpus, + const struct flush_tlb_info *info) { struct { struct mmuext_op op; @@ -1366,7 +1366,7 @@ static void xen_flush_tlb_others(const struct cpumask *cpus, const size_t mc_entry_size = sizeof(args->op) + sizeof(args->mask[0]) * BITS_TO_LONGS(num_possible_cpus()); - trace_xen_mmu_flush_tlb_others(cpus, info->mm, info->start, info->end); + trace_xen_mmu_flush_tlb_multi(cpus, info->mm, info->start, info->end); if (cpumask_empty(cpus)) return; /* nothing to do */ @@ -1375,9 +1375,17 @@ static void xen_flush_tlb_others(const struct cpumask *cpus, args = mcs.args; args->op.arg2.vcpumask = to_cpumask(args->mask); - /* Remove us, and any offline CPUS. */ + /* Flush locally if needed and remove us */ + if (cpumask_test_cpu(smp_processor_id(), to_cpumask(args->mask))) { + local_irq_disable(); + flush_tlb_func_local(info); + local_irq_enable(); + + cpumask_clear_cpu(smp_processor_id(), to_cpumask(args->mask)); + } + + /* Remove offline CPUS */ cpumask_and(to_cpumask(args->mask), cpus, cpu_online_mask); - cpumask_clear_cpu(smp_processor_id(), to_cpumask(args->mask)); args->op.cmd = MMUEXT_TLB_FLUSH_MULTI; if (info->end != TLB_FLUSH_ALL && @@ -2406,7 +2414,7 @@ static const struct pv_mmu_ops xen_mmu_ops __initconst = { .flush_tlb_user = xen_flush_tlb, .flush_tlb_kernel = xen_flush_tlb, .flush_tlb_one_user = xen_flush_tlb_one_user, - .flush_tlb_others = xen_flush_tlb_others, + .flush_tlb_multi = xen_flush_tlb_multi, .tlb_remove_table = tlb_remove_table, .pgd_alloc = xen_pgd_alloc, diff --git a/include/trace/events/xen.h b/include/trace/events/xen.h index 9a0e8af21310..546022acf160 100644 --- a/include/trace/events/xen.h +++ b/include/trace/events/xen.h @@ -362,7 +362,7 @@ TRACE_EVENT(xen_mmu_flush_tlb_one_user, TP_printk("addr %lx", __entry->addr) ); -TRACE_EVENT(xen_mmu_flush_tlb_others, +TRACE_EVENT(xen_mmu_flush_tlb_multi, TP_PROTO(const struct cpumask *cpus, struct mm_struct *mm, unsigned long addr, unsigned long end), TP_ARGS(cpus, mm, addr, end), -- 2.17.1 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Nadav Amit via Virtualization Subject: [PATCH v2 4/9] x86/mm/tlb: Flush remote and local TLBs concurrently Date: Tue, 2 Jul 2019 16:51:46 -0700 Message-ID: <20190702235151.4377-5-namit@vmware.com> References: <20190702235151.4377-1-namit@vmware.com> Reply-To: Nadav Amit Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20190702235151.4377-1-namit@vmware.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Andy Lutomirski , Dave Hansen Cc: Sasha Levin , Juergen Gross , linux-hyperv@vger.kernel.org, Stephen Hemminger , xen-devel@lists.xenproject.org, kvm@vger.kernel.org, Peter Zijlstra , Haiyang Zhang , x86@kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Ingo Molnar , Nadav Amit , Paolo Bonzini , Borislav Petkov , Thomas Gleixner , Boris Ostrovsky List-Id: virtualization@lists.linuxfoundation.org To improve TLB shootdown performance, flush the remote and local TLBs concurrently. Introduce flush_tlb_multi() that does so. Introduce paravirtual versions of flush_tlb_multi() for KVM, Xen and hyper-v (Xen and hyper-v are only compile-tested). While the updated smp infrastructure is capable of running a function on a single local core, it is not optimized for this case. The multiple function calls and the indirect branch introduce some overhead, and might make local TLB flushes slower than they were before the recent changes. Before calling the SMP infrastructure, check if only a local TLB flush is needed to restore the lost performance in this common case. This requires to check mm_cpumask() one more time, but unless this mask is updated very frequently, this should impact performance negatively. Cc: "K. Y. Srinivasan" Cc: Haiyang Zhang Cc: Stephen Hemminger Cc: Sasha Levin Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: Juergen Gross Cc: Paolo Bonzini Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: Boris Ostrovsky Cc: linux-hyperv@vger.kernel.org Cc: linux-kernel@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: kvm@vger.kernel.org Cc: xen-devel@lists.xenproject.org Signed-off-by: Nadav Amit --- arch/x86/hyperv/mmu.c | 13 +++--- arch/x86/include/asm/paravirt.h | 6 +-- arch/x86/include/asm/paravirt_types.h | 4 +- arch/x86/include/asm/tlbflush.h | 9 ++-- arch/x86/include/asm/trace/hyperv.h | 2 +- arch/x86/kernel/kvm.c | 11 +++-- arch/x86/kernel/paravirt.c | 2 +- arch/x86/mm/tlb.c | 65 ++++++++++++++++++++------- arch/x86/xen/mmu_pv.c | 20 ++++++--- include/trace/events/xen.h | 2 +- 10 files changed, 91 insertions(+), 43 deletions(-) diff --git a/arch/x86/hyperv/mmu.c b/arch/x86/hyperv/mmu.c index e65d7fe6489f..1177f863e4cd 100644 --- a/arch/x86/hyperv/mmu.c +++ b/arch/x86/hyperv/mmu.c @@ -50,8 +50,8 @@ static inline int fill_gva_list(u64 gva_list[], int offset, return gva_n - offset; } -static void hyperv_flush_tlb_others(const struct cpumask *cpus, - const struct flush_tlb_info *info) +static void hyperv_flush_tlb_multi(const struct cpumask *cpus, + const struct flush_tlb_info *info) { int cpu, vcpu, gva_n, max_gvas; struct hv_tlb_flush **flush_pcpu; @@ -59,7 +59,7 @@ static void hyperv_flush_tlb_others(const struct cpumask *cpus, u64 status = U64_MAX; unsigned long flags; - trace_hyperv_mmu_flush_tlb_others(cpus, info); + trace_hyperv_mmu_flush_tlb_multi(cpus, info); if (!hv_hypercall_pg) goto do_native; @@ -69,6 +69,9 @@ static void hyperv_flush_tlb_others(const struct cpumask *cpus, local_irq_save(flags); + if (cpumask_test_cpu(smp_processor_id(), cpus)) + flush_tlb_func_local(info); + flush_pcpu = (struct hv_tlb_flush **) this_cpu_ptr(hyperv_pcpu_input_arg); @@ -156,7 +159,7 @@ static void hyperv_flush_tlb_others(const struct cpumask *cpus, if (!(status & HV_HYPERCALL_RESULT_MASK)) return; do_native: - native_flush_tlb_others(cpus, info); + native_flush_tlb_multi(cpus, info); } static u64 hyperv_flush_tlb_others_ex(const struct cpumask *cpus, @@ -231,6 +234,6 @@ void hyperv_setup_mmu_ops(void) return; pr_info("Using hypercall for remote TLB flush\n"); - pv_ops.mmu.flush_tlb_others = hyperv_flush_tlb_others; + pv_ops.mmu.flush_tlb_multi = hyperv_flush_tlb_multi; pv_ops.mmu.tlb_remove_table = tlb_remove_table; } diff --git a/arch/x86/include/asm/paravirt.h b/arch/x86/include/asm/paravirt.h index c25c38a05c1c..316959e89258 100644 --- a/arch/x86/include/asm/paravirt.h +++ b/arch/x86/include/asm/paravirt.h @@ -62,10 +62,10 @@ static inline void __flush_tlb_one_user(unsigned long addr) PVOP_VCALL1(mmu.flush_tlb_one_user, addr); } -static inline void flush_tlb_others(const struct cpumask *cpumask, - const struct flush_tlb_info *info) +static inline void flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info) { - PVOP_VCALL2(mmu.flush_tlb_others, cpumask, info); + PVOP_VCALL2(mmu.flush_tlb_multi, cpumask, info); } static inline void paravirt_tlb_remove_table(struct mmu_gather *tlb, void *table) diff --git a/arch/x86/include/asm/paravirt_types.h b/arch/x86/include/asm/paravirt_types.h index 946f8f1f1efc..54f4c718b5b0 100644 --- a/arch/x86/include/asm/paravirt_types.h +++ b/arch/x86/include/asm/paravirt_types.h @@ -211,8 +211,8 @@ struct pv_mmu_ops { void (*flush_tlb_user)(void); void (*flush_tlb_kernel)(void); void (*flush_tlb_one_user)(unsigned long addr); - void (*flush_tlb_others)(const struct cpumask *cpus, - const struct flush_tlb_info *info); + void (*flush_tlb_multi)(const struct cpumask *cpus, + const struct flush_tlb_info *info); void (*tlb_remove_table)(struct mmu_gather *tlb, void *table); diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index dee375831962..36aa2a9b7597 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -517,7 +517,7 @@ static inline void __flush_tlb_one_kernel(unsigned long addr) * - flush_tlb_page(vma, vmaddr) flushes one page * - flush_tlb_range(vma, start, end) flushes a range of pages * - flush_tlb_kernel_range(start, end) flushes a range of kernel pages - * - flush_tlb_others(cpumask, info) flushes TLBs on other cpus + * - flush_tlb_multi(cpumask, info) flushes TLBs on multiple cpus * * ..but the i386 has somewhat limited tlb flushing capabilities, * and page-granular flushes are available only on i486 and up. @@ -563,13 +563,14 @@ extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); +extern void flush_tlb_func_local(const struct flush_tlb_info *info); static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false); } -void native_flush_tlb_others(const struct cpumask *cpumask, +void native_flush_tlb_multi(const struct cpumask *cpumask, const struct flush_tlb_info *info); static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) @@ -593,8 +594,8 @@ static inline void arch_tlbbatch_add_mm(struct arch_tlbflush_unmap_batch *batch, extern void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch); #ifndef CONFIG_PARAVIRT -#define flush_tlb_others(mask, info) \ - native_flush_tlb_others(mask, info) +#define flush_tlb_multi(mask, info) \ + native_flush_tlb_multi(mask, info) #define paravirt_tlb_remove_table(tlb, page) \ tlb_remove_page(tlb, (void *)(page)) diff --git a/arch/x86/include/asm/trace/hyperv.h b/arch/x86/include/asm/trace/hyperv.h index ace464f09681..85ca8560c7f9 100644 --- a/arch/x86/include/asm/trace/hyperv.h +++ b/arch/x86/include/asm/trace/hyperv.h @@ -8,7 +8,7 @@ #if IS_ENABLED(CONFIG_HYPERV) -TRACE_EVENT(hyperv_mmu_flush_tlb_others, +TRACE_EVENT(hyperv_mmu_flush_tlb_multi, TP_PROTO(const struct cpumask *cpus, const struct flush_tlb_info *info), TP_ARGS(cpus, info), diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 5169b8cc35bb..d00d551d4a2a 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -580,7 +580,7 @@ static void __init kvm_apf_trap_init(void) static DEFINE_PER_CPU(cpumask_var_t, __pv_tlb_mask); -static void kvm_flush_tlb_others(const struct cpumask *cpumask, +static void kvm_flush_tlb_multi(const struct cpumask *cpumask, const struct flush_tlb_info *info) { u8 state; @@ -594,6 +594,11 @@ static void kvm_flush_tlb_others(const struct cpumask *cpumask, * queue flush_on_enter for pre-empted vCPUs */ for_each_cpu(cpu, flushmask) { + /* + * The local vCPU is never preempted, so we do not explicitly + * skip check for local vCPU - it will never be cleared from + * flushmask. + */ src = &per_cpu(steal_time, cpu); state = READ_ONCE(src->preempted); if ((state & KVM_VCPU_PREEMPTED)) { @@ -603,7 +608,7 @@ static void kvm_flush_tlb_others(const struct cpumask *cpumask, } } - native_flush_tlb_others(flushmask, info); + native_flush_tlb_multi(flushmask, info); } static void __init kvm_guest_init(void) @@ -628,7 +633,7 @@ static void __init kvm_guest_init(void) if (kvm_para_has_feature(KVM_FEATURE_PV_TLB_FLUSH) && !kvm_para_has_hint(KVM_HINTS_REALTIME) && kvm_para_has_feature(KVM_FEATURE_STEAL_TIME)) { - pv_ops.mmu.flush_tlb_others = kvm_flush_tlb_others; + pv_ops.mmu.flush_tlb_multi = kvm_flush_tlb_multi; pv_ops.mmu.tlb_remove_table = tlb_remove_table; } diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c index 98039d7fb998..7cdcffe2a028 100644 --- a/arch/x86/kernel/paravirt.c +++ b/arch/x86/kernel/paravirt.c @@ -363,7 +363,7 @@ struct paravirt_patch_template pv_ops = { .mmu.flush_tlb_user = native_flush_tlb, .mmu.flush_tlb_kernel = native_flush_tlb_global, .mmu.flush_tlb_one_user = native_flush_tlb_one_user, - .mmu.flush_tlb_others = native_flush_tlb_others, + .mmu.flush_tlb_multi = native_flush_tlb_multi, .mmu.tlb_remove_table = (void (*)(struct mmu_gather *, void *))tlb_remove_page, diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 5c9b1607191d..074288a6916e 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -551,7 +551,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, * garbage into our TLB. Since switching to init_mm is barely * slower than a minimal flush, just switch to init_mm. * - * This should be rare, with native_flush_tlb_others skipping + * This should be rare, with native_flush_tlb_multi() skipping * IPIs to lazy TLB mode CPUs. */ switch_mm_irqs_off(NULL, &init_mm, NULL); @@ -635,7 +635,7 @@ static void flush_tlb_func_common(const struct flush_tlb_info *f, this_cpu_write(cpu_tlbstate.ctxs[loaded_mm_asid].tlb_gen, mm_tlb_gen); } -static void flush_tlb_func_local(void *info) +static void __flush_tlb_func_local(void *info) { const struct flush_tlb_info *f = info; enum tlb_flush_reason reason; @@ -645,6 +645,11 @@ static void flush_tlb_func_local(void *info) flush_tlb_func_common(f, true, reason); } +void flush_tlb_func_local(const struct flush_tlb_info *info) +{ + __flush_tlb_func_local((void *)info); +} + static void flush_tlb_func_remote(void *info) { const struct flush_tlb_info *f = info; @@ -665,9 +670,14 @@ static bool tlb_is_not_lazy(int cpu) static DEFINE_PER_CPU(cpumask_t, flush_tlb_mask); -void native_flush_tlb_others(const struct cpumask *cpumask, - const struct flush_tlb_info *info) +void native_flush_tlb_multi(const struct cpumask *cpumask, + const struct flush_tlb_info *info) { + /* + * Do accounting and tracing. Note that there are (and have always been) + * cases in which a remote TLB flush will be traced, but eventually + * would not happen. + */ count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); if (info->end == TLB_FLUSH_ALL) trace_tlb_flush(TLB_REMOTE_SEND_IPI, TLB_FLUSH_ALL); @@ -687,10 +697,12 @@ void native_flush_tlb_others(const struct cpumask *cpumask, * means that the percpu tlb_gen variables won't be updated * and we'll do pointless flushes on future context switches. * - * Rather than hooking native_flush_tlb_others() here, I think + * Rather than hooking native_flush_tlb_multi() here, I think * that UV should be updated so that smp_call_function_many(), * etc, are optimal on UV. */ + flush_tlb_func_local(info); + cpumask = uv_flush_tlb_others(cpumask, info); if (cpumask) smp_call_function_many(cpumask, flush_tlb_func_remote, @@ -709,8 +721,9 @@ void native_flush_tlb_others(const struct cpumask *cpumask, * doing a speculative memory access. */ if (info->freed_tables) { - smp_call_function_many(cpumask, flush_tlb_func_remote, - (void *)info, 1); + __smp_call_function_many(cpumask, flush_tlb_func_remote, + __flush_tlb_func_local, + (void *)info, 1); } else { /* * Although we could have used on_each_cpu_cond_mask(), @@ -737,7 +750,8 @@ void native_flush_tlb_others(const struct cpumask *cpumask, if (tlb_is_not_lazy(cpu)) __cpumask_set_cpu(cpu, cond_cpumask); } - smp_call_function_many(cond_cpumask, flush_tlb_func_remote, + __smp_call_function_many(cond_cpumask, flush_tlb_func_remote, + __flush_tlb_func_local, (void *)info, 1); } } @@ -818,16 +832,29 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, info = get_flush_tlb_info(mm, start, end, stride_shift, freed_tables, new_tlb_gen); - if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) { + /* + * Assert that mm_cpumask() corresponds with the loaded mm. We got one + * exception: for init_mm we do not need to flush anything, and the + * cpumask does not correspond with loaded_mm. + */ + VM_WARN_ON_ONCE(cpumask_test_cpu(smp_processor_id(), mm_cpumask(mm)) != + (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) && + mm != &init_mm); + + /* + * flush_tlb_multi() is not optimized for the common case in which only + * a local TLB flush is needed. Optimize this use-case by calling + * flush_tlb_func_local() directly in this case. + */ + if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { + flush_tlb_multi(mm_cpumask(mm), info); + } else { lockdep_assert_irqs_enabled(); local_irq_disable(); flush_tlb_func_local(info); local_irq_enable(); } - if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) - flush_tlb_others(mm_cpumask(mm), info); - put_flush_tlb_info(); put_cpu(); } @@ -890,16 +917,20 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { int cpu = get_cpu(); - if (cpumask_test_cpu(cpu, &batch->cpumask)) { + /* + * flush_tlb_multi() is not optimized for the common case in which only + * a local TLB flush is needed. Optimize this use-case by calling + * flush_tlb_func_local() directly in this case. + */ + if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) { + flush_tlb_multi(&batch->cpumask, &full_flush_tlb_info); + } else { lockdep_assert_irqs_enabled(); local_irq_disable(); - flush_tlb_func_local((void *)&full_flush_tlb_info); + flush_tlb_func_local(&full_flush_tlb_info); local_irq_enable(); } - if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids) - flush_tlb_others(&batch->cpumask, &full_flush_tlb_info); - cpumask_clear(&batch->cpumask); put_cpu(); diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index beb44e22afdf..19e481e6e904 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -1355,8 +1355,8 @@ static void xen_flush_tlb_one_user(unsigned long addr) preempt_enable(); } -static void xen_flush_tlb_others(const struct cpumask *cpus, - const struct flush_tlb_info *info) +static void xen_flush_tlb_multi(const struct cpumask *cpus, + const struct flush_tlb_info *info) { struct { struct mmuext_op op; @@ -1366,7 +1366,7 @@ static void xen_flush_tlb_others(const struct cpumask *cpus, const size_t mc_entry_size = sizeof(args->op) + sizeof(args->mask[0]) * BITS_TO_LONGS(num_possible_cpus()); - trace_xen_mmu_flush_tlb_others(cpus, info->mm, info->start, info->end); + trace_xen_mmu_flush_tlb_multi(cpus, info->mm, info->start, info->end); if (cpumask_empty(cpus)) return; /* nothing to do */ @@ -1375,9 +1375,17 @@ static void xen_flush_tlb_others(const struct cpumask *cpus, args = mcs.args; args->op.arg2.vcpumask = to_cpumask(args->mask); - /* Remove us, and any offline CPUS. */ + /* Flush locally if needed and remove us */ + if (cpumask_test_cpu(smp_processor_id(), to_cpumask(args->mask))) { + local_irq_disable(); + flush_tlb_func_local(info); + local_irq_enable(); + + cpumask_clear_cpu(smp_processor_id(), to_cpumask(args->mask)); + } + + /* Remove offline CPUS */ cpumask_and(to_cpumask(args->mask), cpus, cpu_online_mask); - cpumask_clear_cpu(smp_processor_id(), to_cpumask(args->mask)); args->op.cmd = MMUEXT_TLB_FLUSH_MULTI; if (info->end != TLB_FLUSH_ALL && @@ -2406,7 +2414,7 @@ static const struct pv_mmu_ops xen_mmu_ops __initconst = { .flush_tlb_user = xen_flush_tlb, .flush_tlb_kernel = xen_flush_tlb, .flush_tlb_one_user = xen_flush_tlb_one_user, - .flush_tlb_others = xen_flush_tlb_others, + .flush_tlb_multi = xen_flush_tlb_multi, .tlb_remove_table = tlb_remove_table, .pgd_alloc = xen_pgd_alloc, diff --git a/include/trace/events/xen.h b/include/trace/events/xen.h index 9a0e8af21310..546022acf160 100644 --- a/include/trace/events/xen.h +++ b/include/trace/events/xen.h @@ -362,7 +362,7 @@ TRACE_EVENT(xen_mmu_flush_tlb_one_user, TP_printk("addr %lx", __entry->addr) ); -TRACE_EVENT(xen_mmu_flush_tlb_others, +TRACE_EVENT(xen_mmu_flush_tlb_multi, TP_PROTO(const struct cpumask *cpus, struct mm_struct *mm, unsigned long addr, unsigned long end), TP_ARGS(cpus, mm, addr, end), -- 2.17.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.9 required=3.0 tests=DATE_IN_PAST_06_12, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35178C0650E for ; Wed, 3 Jul 2019 07:14:33 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0B5392187F for ; Wed, 3 Jul 2019 07:14:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B5392187F Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=vmware.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hiZT2-0006g2-R2; Wed, 03 Jul 2019 07:14:12 +0000 Received: from us1-rack-dfw2.inumbo.com ([104.130.134.6]) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1hiZT1-0006fg-I5 for xen-devel@lists.xenproject.org; Wed, 03 Jul 2019 07:14:11 +0000 X-Inumbo-ID: 268e2c05-9d62-11e9-8980-bc764e045a96 Received: from mail-pl1-f196.google.com (unknown [209.85.214.196]) by us1-rack-dfw2.inumbo.com (Halon) with ESMTPS id 268e2c05-9d62-11e9-8980-bc764e045a96; Wed, 03 Jul 2019 07:14:09 +0000 (UTC) Received: by mail-pl1-f196.google.com with SMTP id e5so694293pls.13 for ; Wed, 03 Jul 2019 00:14:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ccH1lY4O7n4BXQuRS1GgwOEGuGvqXICf1/+JCvsaWLs=; b=Lv5k44Soq/52cyNVn4DPj8hvMpXRJ5GZzAMCEZbAw57L7GhzZAt3HcB6Rt5mshlLm6 4yFGsyJlWrY8RN5aBJiQncf/jCEfWt0YlgoVWWicrBXeKDZhNunTEa2pChVd8KIw1z+d dUzxtdnI+6NrXQ89Cpq32SzbCWk9nTYN0G0n3p6FzaRFs5fW4qgfxHb5mo4kzrtzCTN4 1sckOxi/kPosJ/lE3B7mUX74tfGfOUGpwvz9FoYzuy/PP15Iof1HEJFh28Q6a115ckFx e+sB3yWwSXwPyHHX43zxVr4dl7uyD+IN3ryfOF4r/F9hVRnlO565wSWJcjXnTUg1Sw3c zr9w== X-Gm-Message-State: APjAAAXpm5WZ9cHpZ2YEMzwsH4TkAMt/JCQmDkjhVe3x9W8LsONBz1lS gzi45Ttv1gkIRqqxVI+tg7g= X-Google-Smtp-Source: APXvYqxDiu4XHF/h685l75XKdA+4AtAWMjLUqXNKPxMi634QNbO+Oh5f6iZUYJKpj5KI9u+azIzt2A== X-Received: by 2002:a17:902:2983:: with SMTP id h3mr40803893plb.45.1562138048504; Wed, 03 Jul 2019 00:14:08 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id j21sm1256593pfh.86.2019.07.03.00.14.06 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 03 Jul 2019 00:14:07 -0700 (PDT) From: Nadav Amit To: Andy Lutomirski , Dave Hansen Date: Tue, 2 Jul 2019 16:51:46 -0700 Message-Id: <20190702235151.4377-5-namit@vmware.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20190702235151.4377-1-namit@vmware.com> References: <20190702235151.4377-1-namit@vmware.com> Subject: [Xen-devel] [PATCH v2 4/9] x86/mm/tlb: Flush remote and local TLBs concurrently X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Sasha Levin , Juergen Gross , linux-hyperv@vger.kernel.org, Stephen Hemminger , xen-devel@lists.xenproject.org, kvm@vger.kernel.org, Peter Zijlstra , Haiyang Zhang , x86@kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, Ingo Molnar , Nadav Amit , Paolo Bonzini , Borislav Petkov , Thomas Gleixner , "K. Y. Srinivasan" , Boris Ostrovsky MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" VG8gaW1wcm92ZSBUTEIgc2hvb3Rkb3duIHBlcmZvcm1hbmNlLCBmbHVzaCB0aGUgcmVtb3RlIGFu ZCBsb2NhbCBUTEJzCmNvbmN1cnJlbnRseS4gSW50cm9kdWNlIGZsdXNoX3RsYl9tdWx0aSgpIHRo YXQgZG9lcyBzby4gSW50cm9kdWNlCnBhcmF2aXJ0dWFsIHZlcnNpb25zIG9mIGZsdXNoX3RsYl9t dWx0aSgpIGZvciBLVk0sIFhlbiBhbmQgaHlwZXItdiAoWGVuCmFuZCBoeXBlci12IGFyZSBvbmx5 IGNvbXBpbGUtdGVzdGVkKS4KCldoaWxlIHRoZSB1cGRhdGVkIHNtcCBpbmZyYXN0cnVjdHVyZSBp cyBjYXBhYmxlIG9mIHJ1bm5pbmcgYSBmdW5jdGlvbiBvbgphIHNpbmdsZSBsb2NhbCBjb3JlLCBp dCBpcyBub3Qgb3B0aW1pemVkIGZvciB0aGlzIGNhc2UuIFRoZSBtdWx0aXBsZQpmdW5jdGlvbiBj YWxscyBhbmQgdGhlIGluZGlyZWN0IGJyYW5jaCBpbnRyb2R1Y2Ugc29tZSBvdmVyaGVhZCwgYW5k Cm1pZ2h0IG1ha2UgbG9jYWwgVExCIGZsdXNoZXMgc2xvd2VyIHRoYW4gdGhleSB3ZXJlIGJlZm9y ZSB0aGUgcmVjZW50CmNoYW5nZXMuCgpCZWZvcmUgY2FsbGluZyB0aGUgU01QIGluZnJhc3RydWN0 dXJlLCBjaGVjayBpZiBvbmx5IGEgbG9jYWwgVExCIGZsdXNoCmlzIG5lZWRlZCB0byByZXN0b3Jl IHRoZSBsb3N0IHBlcmZvcm1hbmNlIGluIHRoaXMgY29tbW9uIGNhc2UuIFRoaXMKcmVxdWlyZXMg dG8gY2hlY2sgbW1fY3B1bWFzaygpIG9uZSBtb3JlIHRpbWUsIGJ1dCB1bmxlc3MgdGhpcyBtYXNr IGlzCnVwZGF0ZWQgdmVyeSBmcmVxdWVudGx5LCB0aGlzIHNob3VsZCBpbXBhY3QgcGVyZm9ybWFu Y2UgbmVnYXRpdmVseS4KCkNjOiAiSy4gWS4gU3Jpbml2YXNhbiIgPGt5c0BtaWNyb3NvZnQuY29t PgpDYzogSGFpeWFuZyBaaGFuZyA8aGFpeWFuZ3pAbWljcm9zb2Z0LmNvbT4KQ2M6IFN0ZXBoZW4g SGVtbWluZ2VyIDxzdGhlbW1pbkBtaWNyb3NvZnQuY29tPgpDYzogU2FzaGEgTGV2aW4gPHNhc2hh bEBrZXJuZWwub3JnPgpDYzogVGhvbWFzIEdsZWl4bmVyIDx0Z2x4QGxpbnV0cm9uaXguZGU+CkNj OiBJbmdvIE1vbG5hciA8bWluZ29AcmVkaGF0LmNvbT4KQ2M6IEJvcmlzbGF2IFBldGtvdiA8YnBA YWxpZW44LmRlPgpDYzogeDg2QGtlcm5lbC5vcmcKQ2M6IEp1ZXJnZW4gR3Jvc3MgPGpncm9zc0Bz dXNlLmNvbT4KQ2M6IFBhb2xvIEJvbnppbmkgPHBib256aW5pQHJlZGhhdC5jb20+CkNjOiBEYXZl IEhhbnNlbiA8ZGF2ZS5oYW5zZW5AbGludXguaW50ZWwuY29tPgpDYzogQW5keSBMdXRvbWlyc2tp IDxsdXRvQGtlcm5lbC5vcmc+CkNjOiBQZXRlciBaaWpsc3RyYSA8cGV0ZXJ6QGluZnJhZGVhZC5v cmc+CkNjOiBCb3JpcyBPc3Ryb3Zza3kgPGJvcmlzLm9zdHJvdnNreUBvcmFjbGUuY29tPgpDYzog bGludXgtaHlwZXJ2QHZnZXIua2VybmVsLm9yZwpDYzogbGludXgta2VybmVsQHZnZXIua2VybmVs Lm9yZwpDYzogdmlydHVhbGl6YXRpb25AbGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcKQ2M6IGt2 bUB2Z2VyLmtlcm5lbC5vcmcKQ2M6IHhlbi1kZXZlbEBsaXN0cy54ZW5wcm9qZWN0Lm9yZwpTaWdu ZWQtb2ZmLWJ5OiBOYWRhdiBBbWl0IDxuYW1pdEB2bXdhcmUuY29tPgotLS0KIGFyY2gveDg2L2h5 cGVydi9tbXUuYyAgICAgICAgICAgICAgICAgfCAxMyArKystLS0KIGFyY2gveDg2L2luY2x1ZGUv YXNtL3BhcmF2aXJ0LmggICAgICAgfCAgNiArLS0KIGFyY2gveDg2L2luY2x1ZGUvYXNtL3BhcmF2 aXJ0X3R5cGVzLmggfCAgNCArLQogYXJjaC94ODYvaW5jbHVkZS9hc20vdGxiZmx1c2guaCAgICAg ICB8ICA5ICsrLS0KIGFyY2gveDg2L2luY2x1ZGUvYXNtL3RyYWNlL2h5cGVydi5oICAgfCAgMiAr LQogYXJjaC94ODYva2VybmVsL2t2bS5jICAgICAgICAgICAgICAgICB8IDExICsrKy0tCiBhcmNo L3g4Ni9rZXJuZWwvcGFyYXZpcnQuYyAgICAgICAgICAgIHwgIDIgKy0KIGFyY2gveDg2L21tL3Rs Yi5jICAgICAgICAgICAgICAgICAgICAgfCA2NSArKysrKysrKysrKysrKysrKysrKy0tLS0tLS0K IGFyY2gveDg2L3hlbi9tbXVfcHYuYyAgICAgICAgICAgICAgICAgfCAyMCArKysrKystLS0KIGlu Y2x1ZGUvdHJhY2UvZXZlbnRzL3hlbi5oICAgICAgICAgICAgfCAgMiArLQogMTAgZmlsZXMgY2hh bmdlZCwgOTEgaW5zZXJ0aW9ucygrKSwgNDMgZGVsZXRpb25zKC0pCgpkaWZmIC0tZ2l0IGEvYXJj aC94ODYvaHlwZXJ2L21tdS5jIGIvYXJjaC94ODYvaHlwZXJ2L21tdS5jCmluZGV4IGU2NWQ3ZmU2 NDg5Zi4uMTE3N2Y4NjNlNGNkIDEwMDY0NAotLS0gYS9hcmNoL3g4Ni9oeXBlcnYvbW11LmMKKysr IGIvYXJjaC94ODYvaHlwZXJ2L21tdS5jCkBAIC01MCw4ICs1MCw4IEBAIHN0YXRpYyBpbmxpbmUg aW50IGZpbGxfZ3ZhX2xpc3QodTY0IGd2YV9saXN0W10sIGludCBvZmZzZXQsCiAJcmV0dXJuIGd2 YV9uIC0gb2Zmc2V0OwogfQogCi1zdGF0aWMgdm9pZCBoeXBlcnZfZmx1c2hfdGxiX290aGVycyhj b25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1cywKLQkJCQkgICAgY29uc3Qgc3RydWN0IGZsdXNoX3Rs Yl9pbmZvICppbmZvKQorc3RhdGljIHZvaWQgaHlwZXJ2X2ZsdXNoX3RsYl9tdWx0aShjb25zdCBz dHJ1Y3QgY3B1bWFzayAqY3B1cywKKwkJCQkgICBjb25zdCBzdHJ1Y3QgZmx1c2hfdGxiX2luZm8g KmluZm8pCiB7CiAJaW50IGNwdSwgdmNwdSwgZ3ZhX24sIG1heF9ndmFzOwogCXN0cnVjdCBodl90 bGJfZmx1c2ggKipmbHVzaF9wY3B1OwpAQCAtNTksNyArNTksNyBAQCBzdGF0aWMgdm9pZCBoeXBl cnZfZmx1c2hfdGxiX290aGVycyhjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1cywKIAl1NjQgc3Rh dHVzID0gVTY0X01BWDsKIAl1bnNpZ25lZCBsb25nIGZsYWdzOwogCi0JdHJhY2VfaHlwZXJ2X21t dV9mbHVzaF90bGJfb3RoZXJzKGNwdXMsIGluZm8pOworCXRyYWNlX2h5cGVydl9tbXVfZmx1c2hf dGxiX211bHRpKGNwdXMsIGluZm8pOwogCiAJaWYgKCFodl9oeXBlcmNhbGxfcGcpCiAJCWdvdG8g ZG9fbmF0aXZlOwpAQCAtNjksNiArNjksOSBAQCBzdGF0aWMgdm9pZCBoeXBlcnZfZmx1c2hfdGxi X290aGVycyhjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1cywKIAogCWxvY2FsX2lycV9zYXZlKGZs YWdzKTsKIAorCWlmIChjcHVtYXNrX3Rlc3RfY3B1KHNtcF9wcm9jZXNzb3JfaWQoKSwgY3B1cykp CisJCWZsdXNoX3RsYl9mdW5jX2xvY2FsKGluZm8pOworCiAJZmx1c2hfcGNwdSA9IChzdHJ1Y3Qg aHZfdGxiX2ZsdXNoICoqKQogCQkgICAgIHRoaXNfY3B1X3B0cihoeXBlcnZfcGNwdV9pbnB1dF9h cmcpOwogCkBAIC0xNTYsNyArMTU5LDcgQEAgc3RhdGljIHZvaWQgaHlwZXJ2X2ZsdXNoX3RsYl9v dGhlcnMoY29uc3Qgc3RydWN0IGNwdW1hc2sgKmNwdXMsCiAJaWYgKCEoc3RhdHVzICYgSFZfSFlQ RVJDQUxMX1JFU1VMVF9NQVNLKSkKIAkJcmV0dXJuOwogZG9fbmF0aXZlOgotCW5hdGl2ZV9mbHVz aF90bGJfb3RoZXJzKGNwdXMsIGluZm8pOworCW5hdGl2ZV9mbHVzaF90bGJfbXVsdGkoY3B1cywg aW5mbyk7CiB9CiAKIHN0YXRpYyB1NjQgaHlwZXJ2X2ZsdXNoX3RsYl9vdGhlcnNfZXgoY29uc3Qg c3RydWN0IGNwdW1hc2sgKmNwdXMsCkBAIC0yMzEsNiArMjM0LDYgQEAgdm9pZCBoeXBlcnZfc2V0 dXBfbW11X29wcyh2b2lkKQogCQlyZXR1cm47CiAKIAlwcl9pbmZvKCJVc2luZyBoeXBlcmNhbGwg Zm9yIHJlbW90ZSBUTEIgZmx1c2hcbiIpOwotCXB2X29wcy5tbXUuZmx1c2hfdGxiX290aGVycyA9 IGh5cGVydl9mbHVzaF90bGJfb3RoZXJzOworCXB2X29wcy5tbXUuZmx1c2hfdGxiX211bHRpID0g aHlwZXJ2X2ZsdXNoX3RsYl9tdWx0aTsKIAlwdl9vcHMubW11LnRsYl9yZW1vdmVfdGFibGUgPSB0 bGJfcmVtb3ZlX3RhYmxlOwogfQpkaWZmIC0tZ2l0IGEvYXJjaC94ODYvaW5jbHVkZS9hc20vcGFy YXZpcnQuaCBiL2FyY2gveDg2L2luY2x1ZGUvYXNtL3BhcmF2aXJ0LmgKaW5kZXggYzI1YzM4YTA1 YzFjLi4zMTY5NTllODkyNTggMTAwNjQ0Ci0tLSBhL2FyY2gveDg2L2luY2x1ZGUvYXNtL3BhcmF2 aXJ0LmgKKysrIGIvYXJjaC94ODYvaW5jbHVkZS9hc20vcGFyYXZpcnQuaApAQCAtNjIsMTAgKzYy LDEwIEBAIHN0YXRpYyBpbmxpbmUgdm9pZCBfX2ZsdXNoX3RsYl9vbmVfdXNlcih1bnNpZ25lZCBs b25nIGFkZHIpCiAJUFZPUF9WQ0FMTDEobW11LmZsdXNoX3RsYl9vbmVfdXNlciwgYWRkcik7CiB9 CiAKLXN0YXRpYyBpbmxpbmUgdm9pZCBmbHVzaF90bGJfb3RoZXJzKGNvbnN0IHN0cnVjdCBjcHVt YXNrICpjcHVtYXNrLAotCQkJCSAgICBjb25zdCBzdHJ1Y3QgZmx1c2hfdGxiX2luZm8gKmluZm8p CitzdGF0aWMgaW5saW5lIHZvaWQgZmx1c2hfdGxiX211bHRpKGNvbnN0IHN0cnVjdCBjcHVtYXNr ICpjcHVtYXNrLAorCQkJCSAgIGNvbnN0IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAqaW5mbykKIHsK LQlQVk9QX1ZDQUxMMihtbXUuZmx1c2hfdGxiX290aGVycywgY3B1bWFzaywgaW5mbyk7CisJUFZP UF9WQ0FMTDIobW11LmZsdXNoX3RsYl9tdWx0aSwgY3B1bWFzaywgaW5mbyk7CiB9CiAKIHN0YXRp YyBpbmxpbmUgdm9pZCBwYXJhdmlydF90bGJfcmVtb3ZlX3RhYmxlKHN0cnVjdCBtbXVfZ2F0aGVy ICp0bGIsIHZvaWQgKnRhYmxlKQpkaWZmIC0tZ2l0IGEvYXJjaC94ODYvaW5jbHVkZS9hc20vcGFy YXZpcnRfdHlwZXMuaCBiL2FyY2gveDg2L2luY2x1ZGUvYXNtL3BhcmF2aXJ0X3R5cGVzLmgKaW5k ZXggOTQ2ZjhmMWYxZWZjLi41NGY0YzcxOGI1YjAgMTAwNjQ0Ci0tLSBhL2FyY2gveDg2L2luY2x1 ZGUvYXNtL3BhcmF2aXJ0X3R5cGVzLmgKKysrIGIvYXJjaC94ODYvaW5jbHVkZS9hc20vcGFyYXZp cnRfdHlwZXMuaApAQCAtMjExLDggKzIxMSw4IEBAIHN0cnVjdCBwdl9tbXVfb3BzIHsKIAl2b2lk ICgqZmx1c2hfdGxiX3VzZXIpKHZvaWQpOwogCXZvaWQgKCpmbHVzaF90bGJfa2VybmVsKSh2b2lk KTsKIAl2b2lkICgqZmx1c2hfdGxiX29uZV91c2VyKSh1bnNpZ25lZCBsb25nIGFkZHIpOwotCXZv aWQgKCpmbHVzaF90bGJfb3RoZXJzKShjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1cywKLQkJCQkg Y29uc3Qgc3RydWN0IGZsdXNoX3RsYl9pbmZvICppbmZvKTsKKwl2b2lkICgqZmx1c2hfdGxiX211 bHRpKShjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1cywKKwkJCQljb25zdCBzdHJ1Y3QgZmx1c2hf dGxiX2luZm8gKmluZm8pOwogCiAJdm9pZCAoKnRsYl9yZW1vdmVfdGFibGUpKHN0cnVjdCBtbXVf Z2F0aGVyICp0bGIsIHZvaWQgKnRhYmxlKTsKIApkaWZmIC0tZ2l0IGEvYXJjaC94ODYvaW5jbHVk ZS9hc20vdGxiZmx1c2guaCBiL2FyY2gveDg2L2luY2x1ZGUvYXNtL3RsYmZsdXNoLmgKaW5kZXgg ZGVlMzc1ODMxOTYyLi4zNmFhMmE5Yjc1OTcgMTAwNjQ0Ci0tLSBhL2FyY2gveDg2L2luY2x1ZGUv YXNtL3RsYmZsdXNoLmgKKysrIGIvYXJjaC94ODYvaW5jbHVkZS9hc20vdGxiZmx1c2guaApAQCAt NTE3LDcgKzUxNyw3IEBAIHN0YXRpYyBpbmxpbmUgdm9pZCBfX2ZsdXNoX3RsYl9vbmVfa2VybmVs KHVuc2lnbmVkIGxvbmcgYWRkcikKICAqICAtIGZsdXNoX3RsYl9wYWdlKHZtYSwgdm1hZGRyKSBm bHVzaGVzIG9uZSBwYWdlCiAgKiAgLSBmbHVzaF90bGJfcmFuZ2Uodm1hLCBzdGFydCwgZW5kKSBm bHVzaGVzIGEgcmFuZ2Ugb2YgcGFnZXMKICAqICAtIGZsdXNoX3RsYl9rZXJuZWxfcmFuZ2Uoc3Rh cnQsIGVuZCkgZmx1c2hlcyBhIHJhbmdlIG9mIGtlcm5lbCBwYWdlcwotICogIC0gZmx1c2hfdGxi X290aGVycyhjcHVtYXNrLCBpbmZvKSBmbHVzaGVzIFRMQnMgb24gb3RoZXIgY3B1cworICogIC0g Zmx1c2hfdGxiX211bHRpKGNwdW1hc2ssIGluZm8pIGZsdXNoZXMgVExCcyBvbiBtdWx0aXBsZSBj cHVzCiAgKgogICogLi5idXQgdGhlIGkzODYgaGFzIHNvbWV3aGF0IGxpbWl0ZWQgdGxiIGZsdXNo aW5nIGNhcGFiaWxpdGllcywKICAqIGFuZCBwYWdlLWdyYW51bGFyIGZsdXNoZXMgYXJlIGF2YWls YWJsZSBvbmx5IG9uIGk0ODYgYW5kIHVwLgpAQCAtNTYzLDEzICs1NjMsMTQgQEAgZXh0ZXJuIHZv aWQgZmx1c2hfdGxiX21tX3JhbmdlKHN0cnVjdCBtbV9zdHJ1Y3QgKm1tLCB1bnNpZ25lZCBsb25n IHN0YXJ0LAogCQkJCXVuc2lnbmVkIGxvbmcgZW5kLCB1bnNpZ25lZCBpbnQgc3RyaWRlX3NoaWZ0 LAogCQkJCWJvb2wgZnJlZWRfdGFibGVzKTsKIGV4dGVybiB2b2lkIGZsdXNoX3RsYl9rZXJuZWxf cmFuZ2UodW5zaWduZWQgbG9uZyBzdGFydCwgdW5zaWduZWQgbG9uZyBlbmQpOworZXh0ZXJuIHZv aWQgZmx1c2hfdGxiX2Z1bmNfbG9jYWwoY29uc3Qgc3RydWN0IGZsdXNoX3RsYl9pbmZvICppbmZv KTsKIAogc3RhdGljIGlubGluZSB2b2lkIGZsdXNoX3RsYl9wYWdlKHN0cnVjdCB2bV9hcmVhX3N0 cnVjdCAqdm1hLCB1bnNpZ25lZCBsb25nIGEpCiB7CiAJZmx1c2hfdGxiX21tX3JhbmdlKHZtYS0+ dm1fbW0sIGEsIGEgKyBQQUdFX1NJWkUsIFBBR0VfU0hJRlQsIGZhbHNlKTsKIH0KIAotdm9pZCBu YXRpdmVfZmx1c2hfdGxiX290aGVycyhjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1bWFzaywKK3Zv aWQgbmF0aXZlX2ZsdXNoX3RsYl9tdWx0aShjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1bWFzaywK IAkJCSAgICAgY29uc3Qgc3RydWN0IGZsdXNoX3RsYl9pbmZvICppbmZvKTsKIAogc3RhdGljIGlu bGluZSB1NjQgaW5jX21tX3RsYl9nZW4oc3RydWN0IG1tX3N0cnVjdCAqbW0pCkBAIC01OTMsOCAr NTk0LDggQEAgc3RhdGljIGlubGluZSB2b2lkIGFyY2hfdGxiYmF0Y2hfYWRkX21tKHN0cnVjdCBh cmNoX3RsYmZsdXNoX3VubWFwX2JhdGNoICpiYXRjaCwKIGV4dGVybiB2b2lkIGFyY2hfdGxiYmF0 Y2hfZmx1c2goc3RydWN0IGFyY2hfdGxiZmx1c2hfdW5tYXBfYmF0Y2ggKmJhdGNoKTsKIAogI2lm bmRlZiBDT05GSUdfUEFSQVZJUlQKLSNkZWZpbmUgZmx1c2hfdGxiX290aGVycyhtYXNrLCBpbmZv KQlcCi0JbmF0aXZlX2ZsdXNoX3RsYl9vdGhlcnMobWFzaywgaW5mbykKKyNkZWZpbmUgZmx1c2hf dGxiX211bHRpKG1hc2ssIGluZm8pCVwKKwluYXRpdmVfZmx1c2hfdGxiX211bHRpKG1hc2ssIGlu Zm8pCiAKICNkZWZpbmUgcGFyYXZpcnRfdGxiX3JlbW92ZV90YWJsZSh0bGIsIHBhZ2UpIFwKIAl0 bGJfcmVtb3ZlX3BhZ2UodGxiLCAodm9pZCAqKShwYWdlKSkKZGlmZiAtLWdpdCBhL2FyY2gveDg2 L2luY2x1ZGUvYXNtL3RyYWNlL2h5cGVydi5oIGIvYXJjaC94ODYvaW5jbHVkZS9hc20vdHJhY2Uv aHlwZXJ2LmgKaW5kZXggYWNlNDY0ZjA5NjgxLi44NWNhODU2MGM3ZjkgMTAwNjQ0Ci0tLSBhL2Fy Y2gveDg2L2luY2x1ZGUvYXNtL3RyYWNlL2h5cGVydi5oCisrKyBiL2FyY2gveDg2L2luY2x1ZGUv YXNtL3RyYWNlL2h5cGVydi5oCkBAIC04LDcgKzgsNyBAQAogCiAjaWYgSVNfRU5BQkxFRChDT05G SUdfSFlQRVJWKQogCi1UUkFDRV9FVkVOVChoeXBlcnZfbW11X2ZsdXNoX3RsYl9vdGhlcnMsCitU UkFDRV9FVkVOVChoeXBlcnZfbW11X2ZsdXNoX3RsYl9tdWx0aSwKIAkgICAgVFBfUFJPVE8oY29u c3Qgc3RydWN0IGNwdW1hc2sgKmNwdXMsCiAJCSAgICAgY29uc3Qgc3RydWN0IGZsdXNoX3RsYl9p bmZvICppbmZvKSwKIAkgICAgVFBfQVJHUyhjcHVzLCBpbmZvKSwKZGlmZiAtLWdpdCBhL2FyY2gv eDg2L2tlcm5lbC9rdm0uYyBiL2FyY2gveDg2L2tlcm5lbC9rdm0uYwppbmRleCA1MTY5YjhjYzM1 YmIuLmQwMGQ1NTFkNGEyYSAxMDA2NDQKLS0tIGEvYXJjaC94ODYva2VybmVsL2t2bS5jCisrKyBi L2FyY2gveDg2L2tlcm5lbC9rdm0uYwpAQCAtNTgwLDcgKzU4MCw3IEBAIHN0YXRpYyB2b2lkIF9f aW5pdCBrdm1fYXBmX3RyYXBfaW5pdCh2b2lkKQogCiBzdGF0aWMgREVGSU5FX1BFUl9DUFUoY3B1 bWFza192YXJfdCwgX19wdl90bGJfbWFzayk7CiAKLXN0YXRpYyB2b2lkIGt2bV9mbHVzaF90bGJf b3RoZXJzKGNvbnN0IHN0cnVjdCBjcHVtYXNrICpjcHVtYXNrLAorc3RhdGljIHZvaWQga3ZtX2Zs dXNoX3RsYl9tdWx0aShjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1bWFzaywKIAkJCWNvbnN0IHN0 cnVjdCBmbHVzaF90bGJfaW5mbyAqaW5mbykKIHsKIAl1OCBzdGF0ZTsKQEAgLTU5NCw2ICs1OTQs MTEgQEAgc3RhdGljIHZvaWQga3ZtX2ZsdXNoX3RsYl9vdGhlcnMoY29uc3Qgc3RydWN0IGNwdW1h c2sgKmNwdW1hc2ssCiAJICogcXVldWUgZmx1c2hfb25fZW50ZXIgZm9yIHByZS1lbXB0ZWQgdkNQ VXMKIAkgKi8KIAlmb3JfZWFjaF9jcHUoY3B1LCBmbHVzaG1hc2spIHsKKwkJLyoKKwkJICogVGhl IGxvY2FsIHZDUFUgaXMgbmV2ZXIgcHJlZW1wdGVkLCBzbyB3ZSBkbyBub3QgZXhwbGljaXRseQor CQkgKiBza2lwIGNoZWNrIGZvciBsb2NhbCB2Q1BVIC0gaXQgd2lsbCBuZXZlciBiZSBjbGVhcmVk IGZyb20KKwkJICogZmx1c2htYXNrLgorCQkgKi8KIAkJc3JjID0gJnBlcl9jcHUoc3RlYWxfdGlt ZSwgY3B1KTsKIAkJc3RhdGUgPSBSRUFEX09OQ0Uoc3JjLT5wcmVlbXB0ZWQpOwogCQlpZiAoKHN0 YXRlICYgS1ZNX1ZDUFVfUFJFRU1QVEVEKSkgewpAQCAtNjAzLDcgKzYwOCw3IEBAIHN0YXRpYyB2 b2lkIGt2bV9mbHVzaF90bGJfb3RoZXJzKGNvbnN0IHN0cnVjdCBjcHVtYXNrICpjcHVtYXNrLAog CQl9CiAJfQogCi0JbmF0aXZlX2ZsdXNoX3RsYl9vdGhlcnMoZmx1c2htYXNrLCBpbmZvKTsKKwlu YXRpdmVfZmx1c2hfdGxiX211bHRpKGZsdXNobWFzaywgaW5mbyk7CiB9CiAKIHN0YXRpYyB2b2lk IF9faW5pdCBrdm1fZ3Vlc3RfaW5pdCh2b2lkKQpAQCAtNjI4LDcgKzYzMyw3IEBAIHN0YXRpYyB2 b2lkIF9faW5pdCBrdm1fZ3Vlc3RfaW5pdCh2b2lkKQogCWlmIChrdm1fcGFyYV9oYXNfZmVhdHVy ZShLVk1fRkVBVFVSRV9QVl9UTEJfRkxVU0gpICYmCiAJICAgICFrdm1fcGFyYV9oYXNfaGludChL Vk1fSElOVFNfUkVBTFRJTUUpICYmCiAJICAgIGt2bV9wYXJhX2hhc19mZWF0dXJlKEtWTV9GRUFU VVJFX1NURUFMX1RJTUUpKSB7Ci0JCXB2X29wcy5tbXUuZmx1c2hfdGxiX290aGVycyA9IGt2bV9m bHVzaF90bGJfb3RoZXJzOworCQlwdl9vcHMubW11LmZsdXNoX3RsYl9tdWx0aSA9IGt2bV9mbHVz aF90bGJfbXVsdGk7CiAJCXB2X29wcy5tbXUudGxiX3JlbW92ZV90YWJsZSA9IHRsYl9yZW1vdmVf dGFibGU7CiAJfQogCmRpZmYgLS1naXQgYS9hcmNoL3g4Ni9rZXJuZWwvcGFyYXZpcnQuYyBiL2Fy Y2gveDg2L2tlcm5lbC9wYXJhdmlydC5jCmluZGV4IDk4MDM5ZDdmYjk5OC4uN2NkY2ZmZTJhMDI4 IDEwMDY0NAotLS0gYS9hcmNoL3g4Ni9rZXJuZWwvcGFyYXZpcnQuYworKysgYi9hcmNoL3g4Ni9r ZXJuZWwvcGFyYXZpcnQuYwpAQCAtMzYzLDcgKzM2Myw3IEBAIHN0cnVjdCBwYXJhdmlydF9wYXRj aF90ZW1wbGF0ZSBwdl9vcHMgPSB7CiAJLm1tdS5mbHVzaF90bGJfdXNlcgk9IG5hdGl2ZV9mbHVz aF90bGIsCiAJLm1tdS5mbHVzaF90bGJfa2VybmVsCT0gbmF0aXZlX2ZsdXNoX3RsYl9nbG9iYWws CiAJLm1tdS5mbHVzaF90bGJfb25lX3VzZXIJPSBuYXRpdmVfZmx1c2hfdGxiX29uZV91c2VyLAot CS5tbXUuZmx1c2hfdGxiX290aGVycwk9IG5hdGl2ZV9mbHVzaF90bGJfb3RoZXJzLAorCS5tbXUu Zmx1c2hfdGxiX211bHRpCT0gbmF0aXZlX2ZsdXNoX3RsYl9tdWx0aSwKIAkubW11LnRsYl9yZW1v dmVfdGFibGUJPQogCQkJKHZvaWQgKCopKHN0cnVjdCBtbXVfZ2F0aGVyICosIHZvaWQgKikpdGxi X3JlbW92ZV9wYWdlLAogCmRpZmYgLS1naXQgYS9hcmNoL3g4Ni9tbS90bGIuYyBiL2FyY2gveDg2 L21tL3RsYi5jCmluZGV4IDVjOWIxNjA3MTkxZC4uMDc0Mjg4YTY5MTZlIDEwMDY0NAotLS0gYS9h cmNoL3g4Ni9tbS90bGIuYworKysgYi9hcmNoL3g4Ni9tbS90bGIuYwpAQCAtNTUxLDcgKzU1MSw3 IEBAIHN0YXRpYyB2b2lkIGZsdXNoX3RsYl9mdW5jX2NvbW1vbihjb25zdCBzdHJ1Y3QgZmx1c2hf dGxiX2luZm8gKmYsCiAJCSAqIGdhcmJhZ2UgaW50byBvdXIgVExCLiAgU2luY2Ugc3dpdGNoaW5n IHRvIGluaXRfbW0gaXMgYmFyZWx5CiAJCSAqIHNsb3dlciB0aGFuIGEgbWluaW1hbCBmbHVzaCwg anVzdCBzd2l0Y2ggdG8gaW5pdF9tbS4KIAkJICoKLQkJICogVGhpcyBzaG91bGQgYmUgcmFyZSwg d2l0aCBuYXRpdmVfZmx1c2hfdGxiX290aGVycyBza2lwcGluZworCQkgKiBUaGlzIHNob3VsZCBi ZSByYXJlLCB3aXRoIG5hdGl2ZV9mbHVzaF90bGJfbXVsdGkoKSBza2lwcGluZwogCQkgKiBJUElz IHRvIGxhenkgVExCIG1vZGUgQ1BVcy4KIAkJICovCiAJCXN3aXRjaF9tbV9pcnFzX29mZihOVUxM LCAmaW5pdF9tbSwgTlVMTCk7CkBAIC02MzUsNyArNjM1LDcgQEAgc3RhdGljIHZvaWQgZmx1c2hf dGxiX2Z1bmNfY29tbW9uKGNvbnN0IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAqZiwKIAl0aGlzX2Nw dV93cml0ZShjcHVfdGxic3RhdGUuY3R4c1tsb2FkZWRfbW1fYXNpZF0udGxiX2dlbiwgbW1fdGxi X2dlbik7CiB9CiAKLXN0YXRpYyB2b2lkIGZsdXNoX3RsYl9mdW5jX2xvY2FsKHZvaWQgKmluZm8p CitzdGF0aWMgdm9pZCBfX2ZsdXNoX3RsYl9mdW5jX2xvY2FsKHZvaWQgKmluZm8pCiB7CiAJY29u c3Qgc3RydWN0IGZsdXNoX3RsYl9pbmZvICpmID0gaW5mbzsKIAllbnVtIHRsYl9mbHVzaF9yZWFz b24gcmVhc29uOwpAQCAtNjQ1LDYgKzY0NSwxMSBAQCBzdGF0aWMgdm9pZCBmbHVzaF90bGJfZnVu Y19sb2NhbCh2b2lkICppbmZvKQogCWZsdXNoX3RsYl9mdW5jX2NvbW1vbihmLCB0cnVlLCByZWFz b24pOwogfQogCit2b2lkIGZsdXNoX3RsYl9mdW5jX2xvY2FsKGNvbnN0IHN0cnVjdCBmbHVzaF90 bGJfaW5mbyAqaW5mbykKK3sKKwlfX2ZsdXNoX3RsYl9mdW5jX2xvY2FsKCh2b2lkICopaW5mbyk7 Cit9CisKIHN0YXRpYyB2b2lkIGZsdXNoX3RsYl9mdW5jX3JlbW90ZSh2b2lkICppbmZvKQogewog CWNvbnN0IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAqZiA9IGluZm87CkBAIC02NjUsOSArNjcwLDE0 IEBAIHN0YXRpYyBib29sIHRsYl9pc19ub3RfbGF6eShpbnQgY3B1KQogCiBzdGF0aWMgREVGSU5F X1BFUl9DUFUoY3B1bWFza190LCBmbHVzaF90bGJfbWFzayk7CiAKLXZvaWQgbmF0aXZlX2ZsdXNo X3RsYl9vdGhlcnMoY29uc3Qgc3RydWN0IGNwdW1hc2sgKmNwdW1hc2ssCi0JCQkgICAgIGNvbnN0 IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAqaW5mbykKK3ZvaWQgbmF0aXZlX2ZsdXNoX3RsYl9tdWx0 aShjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1bWFzaywKKwkJCSAgICBjb25zdCBzdHJ1Y3QgZmx1 c2hfdGxiX2luZm8gKmluZm8pCiB7CisJLyoKKwkgKiBEbyBhY2NvdW50aW5nIGFuZCB0cmFjaW5n LiBOb3RlIHRoYXQgdGhlcmUgYXJlIChhbmQgaGF2ZSBhbHdheXMgYmVlbikKKwkgKiBjYXNlcyBp biB3aGljaCBhIHJlbW90ZSBUTEIgZmx1c2ggd2lsbCBiZSB0cmFjZWQsIGJ1dCBldmVudHVhbGx5 CisJICogd291bGQgbm90IGhhcHBlbi4KKwkgKi8KIAljb3VudF92bV90bGJfZXZlbnQoTlJfVExC X1JFTU9URV9GTFVTSCk7CiAJaWYgKGluZm8tPmVuZCA9PSBUTEJfRkxVU0hfQUxMKQogCQl0cmFj ZV90bGJfZmx1c2goVExCX1JFTU9URV9TRU5EX0lQSSwgVExCX0ZMVVNIX0FMTCk7CkBAIC02ODcs MTAgKzY5NywxMiBAQCB2b2lkIG5hdGl2ZV9mbHVzaF90bGJfb3RoZXJzKGNvbnN0IHN0cnVjdCBj cHVtYXNrICpjcHVtYXNrLAogCQkgKiBtZWFucyB0aGF0IHRoZSBwZXJjcHUgdGxiX2dlbiB2YXJp YWJsZXMgd29uJ3QgYmUgdXBkYXRlZAogCQkgKiBhbmQgd2UnbGwgZG8gcG9pbnRsZXNzIGZsdXNo ZXMgb24gZnV0dXJlIGNvbnRleHQgc3dpdGNoZXMuCiAJCSAqCi0JCSAqIFJhdGhlciB0aGFuIGhv b2tpbmcgbmF0aXZlX2ZsdXNoX3RsYl9vdGhlcnMoKSBoZXJlLCBJIHRoaW5rCisJCSAqIFJhdGhl ciB0aGFuIGhvb2tpbmcgbmF0aXZlX2ZsdXNoX3RsYl9tdWx0aSgpIGhlcmUsIEkgdGhpbmsKIAkJ ICogdGhhdCBVViBzaG91bGQgYmUgdXBkYXRlZCBzbyB0aGF0IHNtcF9jYWxsX2Z1bmN0aW9uX21h bnkoKSwKIAkJICogZXRjLCBhcmUgb3B0aW1hbCBvbiBVVi4KIAkJICovCisJCWZsdXNoX3RsYl9m dW5jX2xvY2FsKGluZm8pOworCiAJCWNwdW1hc2sgPSB1dl9mbHVzaF90bGJfb3RoZXJzKGNwdW1h c2ssIGluZm8pOwogCQlpZiAoY3B1bWFzaykKIAkJCXNtcF9jYWxsX2Z1bmN0aW9uX21hbnkoY3B1 bWFzaywgZmx1c2hfdGxiX2Z1bmNfcmVtb3RlLApAQCAtNzA5LDggKzcyMSw5IEBAIHZvaWQgbmF0 aXZlX2ZsdXNoX3RsYl9vdGhlcnMoY29uc3Qgc3RydWN0IGNwdW1hc2sgKmNwdW1hc2ssCiAJICog ZG9pbmcgYSBzcGVjdWxhdGl2ZSBtZW1vcnkgYWNjZXNzLgogCSAqLwogCWlmIChpbmZvLT5mcmVl ZF90YWJsZXMpIHsKLQkJc21wX2NhbGxfZnVuY3Rpb25fbWFueShjcHVtYXNrLCBmbHVzaF90bGJf ZnVuY19yZW1vdGUsCi0JCQkgICAgICAgKHZvaWQgKilpbmZvLCAxKTsKKwkJX19zbXBfY2FsbF9m dW5jdGlvbl9tYW55KGNwdW1hc2ssIGZsdXNoX3RsYl9mdW5jX3JlbW90ZSwKKwkJCQkJIF9fZmx1 c2hfdGxiX2Z1bmNfbG9jYWwsCisJCQkJCSAodm9pZCAqKWluZm8sIDEpOwogCX0gZWxzZSB7CiAJ CS8qCiAJCSAqIEFsdGhvdWdoIHdlIGNvdWxkIGhhdmUgdXNlZCBvbl9lYWNoX2NwdV9jb25kX21h c2soKSwKQEAgLTczNyw3ICs3NTAsOCBAQCB2b2lkIG5hdGl2ZV9mbHVzaF90bGJfb3RoZXJzKGNv bnN0IHN0cnVjdCBjcHVtYXNrICpjcHVtYXNrLAogCQkJaWYgKHRsYl9pc19ub3RfbGF6eShjcHUp KQogCQkJCV9fY3B1bWFza19zZXRfY3B1KGNwdSwgY29uZF9jcHVtYXNrKTsKIAkJfQotCQlzbXBf Y2FsbF9mdW5jdGlvbl9tYW55KGNvbmRfY3B1bWFzaywgZmx1c2hfdGxiX2Z1bmNfcmVtb3RlLAor CQlfX3NtcF9jYWxsX2Z1bmN0aW9uX21hbnkoY29uZF9jcHVtYXNrLCBmbHVzaF90bGJfZnVuY19y ZW1vdGUsCisJCQkJCSBfX2ZsdXNoX3RsYl9mdW5jX2xvY2FsLAogCQkJCQkgKHZvaWQgKilpbmZv LCAxKTsKIAl9CiB9CkBAIC04MTgsMTYgKzgzMiwyOSBAQCB2b2lkIGZsdXNoX3RsYl9tbV9yYW5n ZShzdHJ1Y3QgbW1fc3RydWN0ICptbSwgdW5zaWduZWQgbG9uZyBzdGFydCwKIAlpbmZvID0gZ2V0 X2ZsdXNoX3RsYl9pbmZvKG1tLCBzdGFydCwgZW5kLCBzdHJpZGVfc2hpZnQsIGZyZWVkX3RhYmxl cywKIAkJCQkgIG5ld190bGJfZ2VuKTsKIAotCWlmIChtbSA9PSB0aGlzX2NwdV9yZWFkKGNwdV90 bGJzdGF0ZS5sb2FkZWRfbW0pKSB7CisJLyoKKwkgKiBBc3NlcnQgdGhhdCBtbV9jcHVtYXNrKCkg Y29ycmVzcG9uZHMgd2l0aCB0aGUgbG9hZGVkIG1tLiBXZSBnb3Qgb25lCisJICogZXhjZXB0aW9u OiBmb3IgaW5pdF9tbSB3ZSBkbyBub3QgbmVlZCB0byBmbHVzaCBhbnl0aGluZywgYW5kIHRoZQor CSAqIGNwdW1hc2sgZG9lcyBub3QgY29ycmVzcG9uZCB3aXRoIGxvYWRlZF9tbS4KKwkgKi8KKwlW TV9XQVJOX09OX09OQ0UoY3B1bWFza190ZXN0X2NwdShzbXBfcHJvY2Vzc29yX2lkKCksIG1tX2Nw dW1hc2sobW0pKSAhPQorCQkJKG1tID09IHRoaXNfY3B1X3JlYWQoY3B1X3RsYnN0YXRlLmxvYWRl ZF9tbSkpICYmCisJCQltbSAhPSAmaW5pdF9tbSk7CisKKwkvKgorCSAqIGZsdXNoX3RsYl9tdWx0 aSgpIGlzIG5vdCBvcHRpbWl6ZWQgZm9yIHRoZSBjb21tb24gY2FzZSBpbiB3aGljaCBvbmx5CisJ ICogYSBsb2NhbCBUTEIgZmx1c2ggaXMgbmVlZGVkLiBPcHRpbWl6ZSB0aGlzIHVzZS1jYXNlIGJ5 IGNhbGxpbmcKKwkgKiBmbHVzaF90bGJfZnVuY19sb2NhbCgpIGRpcmVjdGx5IGluIHRoaXMgY2Fz ZS4KKwkgKi8KKwlpZiAoY3B1bWFza19hbnlfYnV0KG1tX2NwdW1hc2sobW0pLCBjcHUpIDwgbnJf Y3B1X2lkcykgeworCQlmbHVzaF90bGJfbXVsdGkobW1fY3B1bWFzayhtbSksIGluZm8pOworCX0g ZWxzZSB7CiAJCWxvY2tkZXBfYXNzZXJ0X2lycXNfZW5hYmxlZCgpOwogCQlsb2NhbF9pcnFfZGlz YWJsZSgpOwogCQlmbHVzaF90bGJfZnVuY19sb2NhbChpbmZvKTsKIAkJbG9jYWxfaXJxX2VuYWJs ZSgpOwogCX0KIAotCWlmIChjcHVtYXNrX2FueV9idXQobW1fY3B1bWFzayhtbSksIGNwdSkgPCBu cl9jcHVfaWRzKQotCQlmbHVzaF90bGJfb3RoZXJzKG1tX2NwdW1hc2sobW0pLCBpbmZvKTsKLQog CXB1dF9mbHVzaF90bGJfaW5mbygpOwogCXB1dF9jcHUoKTsKIH0KQEAgLTg5MCwxNiArOTE3LDIw IEBAIHZvaWQgYXJjaF90bGJiYXRjaF9mbHVzaChzdHJ1Y3QgYXJjaF90bGJmbHVzaF91bm1hcF9i YXRjaCAqYmF0Y2gpCiB7CiAJaW50IGNwdSA9IGdldF9jcHUoKTsKIAotCWlmIChjcHVtYXNrX3Rl c3RfY3B1KGNwdSwgJmJhdGNoLT5jcHVtYXNrKSkgeworCS8qCisJICogZmx1c2hfdGxiX211bHRp KCkgaXMgbm90IG9wdGltaXplZCBmb3IgdGhlIGNvbW1vbiBjYXNlIGluIHdoaWNoIG9ubHkKKwkg KiBhIGxvY2FsIFRMQiBmbHVzaCBpcyBuZWVkZWQuIE9wdGltaXplIHRoaXMgdXNlLWNhc2UgYnkg Y2FsbGluZworCSAqIGZsdXNoX3RsYl9mdW5jX2xvY2FsKCkgZGlyZWN0bHkgaW4gdGhpcyBjYXNl LgorCSAqLworCWlmIChjcHVtYXNrX2FueV9idXQoJmJhdGNoLT5jcHVtYXNrLCBjcHUpIDwgbnJf Y3B1X2lkcykgeworCQlmbHVzaF90bGJfbXVsdGkoJmJhdGNoLT5jcHVtYXNrLCAmZnVsbF9mbHVz aF90bGJfaW5mbyk7CisJfSBlbHNlIHsKIAkJbG9ja2RlcF9hc3NlcnRfaXJxc19lbmFibGVkKCk7 CiAJCWxvY2FsX2lycV9kaXNhYmxlKCk7Ci0JCWZsdXNoX3RsYl9mdW5jX2xvY2FsKCh2b2lkICop JmZ1bGxfZmx1c2hfdGxiX2luZm8pOworCQlmbHVzaF90bGJfZnVuY19sb2NhbCgmZnVsbF9mbHVz aF90bGJfaW5mbyk7CiAJCWxvY2FsX2lycV9lbmFibGUoKTsKIAl9CiAKLQlpZiAoY3B1bWFza19h bnlfYnV0KCZiYXRjaC0+Y3B1bWFzaywgY3B1KSA8IG5yX2NwdV9pZHMpCi0JCWZsdXNoX3RsYl9v dGhlcnMoJmJhdGNoLT5jcHVtYXNrLCAmZnVsbF9mbHVzaF90bGJfaW5mbyk7Ci0KIAljcHVtYXNr X2NsZWFyKCZiYXRjaC0+Y3B1bWFzayk7CiAKIAlwdXRfY3B1KCk7CmRpZmYgLS1naXQgYS9hcmNo L3g4Ni94ZW4vbW11X3B2LmMgYi9hcmNoL3g4Ni94ZW4vbW11X3B2LmMKaW5kZXggYmViNDRlMjJh ZmRmLi4xOWU0ODFlNmU5MDQgMTAwNjQ0Ci0tLSBhL2FyY2gveDg2L3hlbi9tbXVfcHYuYworKysg Yi9hcmNoL3g4Ni94ZW4vbW11X3B2LmMKQEAgLTEzNTUsOCArMTM1NSw4IEBAIHN0YXRpYyB2b2lk IHhlbl9mbHVzaF90bGJfb25lX3VzZXIodW5zaWduZWQgbG9uZyBhZGRyKQogCXByZWVtcHRfZW5h YmxlKCk7CiB9CiAKLXN0YXRpYyB2b2lkIHhlbl9mbHVzaF90bGJfb3RoZXJzKGNvbnN0IHN0cnVj dCBjcHVtYXNrICpjcHVzLAotCQkJCSBjb25zdCBzdHJ1Y3QgZmx1c2hfdGxiX2luZm8gKmluZm8p CitzdGF0aWMgdm9pZCB4ZW5fZmx1c2hfdGxiX211bHRpKGNvbnN0IHN0cnVjdCBjcHVtYXNrICpj cHVzLAorCQkJCWNvbnN0IHN0cnVjdCBmbHVzaF90bGJfaW5mbyAqaW5mbykKIHsKIAlzdHJ1Y3Qg ewogCQlzdHJ1Y3QgbW11ZXh0X29wIG9wOwpAQCAtMTM2Niw3ICsxMzY2LDcgQEAgc3RhdGljIHZv aWQgeGVuX2ZsdXNoX3RsYl9vdGhlcnMoY29uc3Qgc3RydWN0IGNwdW1hc2sgKmNwdXMsCiAJY29u c3Qgc2l6ZV90IG1jX2VudHJ5X3NpemUgPSBzaXplb2YoYXJncy0+b3ApICsKIAkJc2l6ZW9mKGFy Z3MtPm1hc2tbMF0pICogQklUU19UT19MT05HUyhudW1fcG9zc2libGVfY3B1cygpKTsKIAotCXRy YWNlX3hlbl9tbXVfZmx1c2hfdGxiX290aGVycyhjcHVzLCBpbmZvLT5tbSwgaW5mby0+c3RhcnQs IGluZm8tPmVuZCk7CisJdHJhY2VfeGVuX21tdV9mbHVzaF90bGJfbXVsdGkoY3B1cywgaW5mby0+ bW0sIGluZm8tPnN0YXJ0LCBpbmZvLT5lbmQpOwogCiAJaWYgKGNwdW1hc2tfZW1wdHkoY3B1cykp CiAJCXJldHVybjsJCS8qIG5vdGhpbmcgdG8gZG8gKi8KQEAgLTEzNzUsOSArMTM3NSwxNyBAQCBz dGF0aWMgdm9pZCB4ZW5fZmx1c2hfdGxiX290aGVycyhjb25zdCBzdHJ1Y3QgY3B1bWFzayAqY3B1 cywKIAlhcmdzID0gbWNzLmFyZ3M7CiAJYXJncy0+b3AuYXJnMi52Y3B1bWFzayA9IHRvX2NwdW1h c2soYXJncy0+bWFzayk7CiAKLQkvKiBSZW1vdmUgdXMsIGFuZCBhbnkgb2ZmbGluZSBDUFVTLiAq LworCS8qIEZsdXNoIGxvY2FsbHkgaWYgbmVlZGVkIGFuZCByZW1vdmUgdXMgKi8KKwlpZiAoY3B1 bWFza190ZXN0X2NwdShzbXBfcHJvY2Vzc29yX2lkKCksIHRvX2NwdW1hc2soYXJncy0+bWFzaykp KSB7CisJCWxvY2FsX2lycV9kaXNhYmxlKCk7CisJCWZsdXNoX3RsYl9mdW5jX2xvY2FsKGluZm8p OworCQlsb2NhbF9pcnFfZW5hYmxlKCk7CisKKwkJY3B1bWFza19jbGVhcl9jcHUoc21wX3Byb2Nl c3Nvcl9pZCgpLCB0b19jcHVtYXNrKGFyZ3MtPm1hc2spKTsKKwl9CisKKwkvKiBSZW1vdmUgb2Zm bGluZSBDUFVTICovCiAJY3B1bWFza19hbmQodG9fY3B1bWFzayhhcmdzLT5tYXNrKSwgY3B1cywg Y3B1X29ubGluZV9tYXNrKTsKLQljcHVtYXNrX2NsZWFyX2NwdShzbXBfcHJvY2Vzc29yX2lkKCks IHRvX2NwdW1hc2soYXJncy0+bWFzaykpOwogCiAJYXJncy0+b3AuY21kID0gTU1VRVhUX1RMQl9G TFVTSF9NVUxUSTsKIAlpZiAoaW5mby0+ZW5kICE9IFRMQl9GTFVTSF9BTEwgJiYKQEAgLTI0MDYs NyArMjQxNCw3IEBAIHN0YXRpYyBjb25zdCBzdHJ1Y3QgcHZfbW11X29wcyB4ZW5fbW11X29wcyBf X2luaXRjb25zdCA9IHsKIAkuZmx1c2hfdGxiX3VzZXIgPSB4ZW5fZmx1c2hfdGxiLAogCS5mbHVz aF90bGJfa2VybmVsID0geGVuX2ZsdXNoX3RsYiwKIAkuZmx1c2hfdGxiX29uZV91c2VyID0geGVu X2ZsdXNoX3RsYl9vbmVfdXNlciwKLQkuZmx1c2hfdGxiX290aGVycyA9IHhlbl9mbHVzaF90bGJf b3RoZXJzLAorCS5mbHVzaF90bGJfbXVsdGkgPSB4ZW5fZmx1c2hfdGxiX211bHRpLAogCS50bGJf cmVtb3ZlX3RhYmxlID0gdGxiX3JlbW92ZV90YWJsZSwKIAogCS5wZ2RfYWxsb2MgPSB4ZW5fcGdk X2FsbG9jLApkaWZmIC0tZ2l0IGEvaW5jbHVkZS90cmFjZS9ldmVudHMveGVuLmggYi9pbmNsdWRl L3RyYWNlL2V2ZW50cy94ZW4uaAppbmRleCA5YTBlOGFmMjEzMTAuLjU0NjAyMmFjZjE2MCAxMDA2 NDQKLS0tIGEvaW5jbHVkZS90cmFjZS9ldmVudHMveGVuLmgKKysrIGIvaW5jbHVkZS90cmFjZS9l dmVudHMveGVuLmgKQEAgLTM2Miw3ICszNjIsNyBAQCBUUkFDRV9FVkVOVCh4ZW5fbW11X2ZsdXNo X3RsYl9vbmVfdXNlciwKIAkgICAgVFBfcHJpbnRrKCJhZGRyICVseCIsIF9fZW50cnktPmFkZHIp CiAJKTsKIAotVFJBQ0VfRVZFTlQoeGVuX21tdV9mbHVzaF90bGJfb3RoZXJzLAorVFJBQ0VfRVZF TlQoeGVuX21tdV9mbHVzaF90bGJfbXVsdGksCiAJICAgIFRQX1BST1RPKGNvbnN0IHN0cnVjdCBj cHVtYXNrICpjcHVzLCBzdHJ1Y3QgbW1fc3RydWN0ICptbSwKIAkJICAgICB1bnNpZ25lZCBsb25n IGFkZHIsIHVuc2lnbmVkIGxvbmcgZW5kKSwKIAkgICAgVFBfQVJHUyhjcHVzLCBtbSwgYWRkciwg ZW5kKSwKLS0gCjIuMTcuMQoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fClhlbi1kZXZlbCBtYWlsaW5nIGxpc3QKWGVuLWRldmVsQGxpc3RzLnhlbnByb2pl Y3Qub3JnCmh0dHBzOi8vbGlzdHMueGVucHJvamVjdC5vcmcvbWFpbG1hbi9saXN0aW5mby94ZW4t ZGV2ZWw=