[PATCH 2/2] xen: spin in xen_flush_tlb_others if a target cpu is in gup_fast()

[Jeremy Fitzhardinge <jeremy@goop.org>]

get_user_pages_fast() relies on cross-cpu tlb flushes being a barrier
between clearing and setting a pte, and before freeing a pagetable page.

It normally implements this by disabling interrupts to suspend tlb flush
IPIs.  This doesn't work for us because we don't use kernel-visible
IPIs for tlb flushes, so just spin manually until none of our target
cpus are in gup_fast().

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>

diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index aa16ef4..b0523f8 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -50,6 +50,7 @@
 #include <asm/setup.h>
 #include <asm/paravirt.h>
 #include <asm/linkage.h>
+#include <asm/mmu.h>
 
 #include <asm/xen/hypercall.h>
 #include <asm/xen/hypervisor.h>
@@ -1334,6 +1335,14 @@ static void xen_flush_tlb_others(const struct cpumask *cpus,
 	MULTI_mmuext_op(mcs.mc, &args->op, 1, NULL, DOMID_SELF);
 
 	xen_mc_issue(PARAVIRT_LAZY_MMU);
+
+	/*
+	 * If we're racing with a get_user_pages_fast(), wait here
+	 * until it has finishes so that it can use cross-cpu tlb
+	 * flush as a barrier.
+	 */
+	while (cpus_intersects(*in_gup_cpumask, *cpus))
+		cpu_relax();
 }
 
 static unsigned long xen_read_cr3(void)