Re: [PATCH v5] x86/hvm/viridian: flush remote tlbs by hypercall

[Wei Liu <wei.liu2@citrix.com>]

On Fri, Nov 20, 2015 at 09:55:12AM +0000, Paul Durrant wrote:
> The Microsoft Hypervisor Top Level Functional Spec. (section 3.4) defines
> two bits in CPUID leaf 0x40000004:EAX for the hypervisor to recommend
> whether or not to issue a hypercall for local or remote TLB flush.
> 
> Whilst it's doubtful whether using a hypercall for local TLB flush would
> be any more efficient than a specific INVLPG VMEXIT, a remote TLB flush
> may well be more efficiently done. This is because the alternative
> mechanism is to IPI all the vCPUs in question which (in the absence of
> APIC virtualisation) will require emulation and scheduling of the vCPUs
> only to have them immediately VMEXIT for local TLB flush.
> 
> This patch therefore adds a viridian option which, if selected, enables
> the hypercall for remote TLB flush and implements it using ASID
> invalidation for targetted vCPUs followed by an IPI only to the set of
> CPUs that happened to be running a targetted vCPU (which may be the empty
> set). The flush may be more severe than requested since the hypercall can
> request flush only for a specific address space (CR3) but Xen neither
> keeps a mapping of ASID to guest CR3 nor allows invalidation of a specific
> ASID, but on a host with contended CPUs performance is still likely to
> be better than a more specific flush using IPIs.
> 
> The implementation of the patch introduces per-vCPU viridian_init() and
> viridian_deinit() functions to allow a scratch cpumask to be allocated.
> This avoids needing to put this potentially large data structure on stack
> during hypercall processing. It also modifies the hypercall input and
> output bit-fields to allow a check for the 'fast' calling convention,
> and a white-space fix in the definition of HVMPV_feature_mask (to remove
> hard tabs).
> 
> Signed-off-by: Paul Durrant <paul.durrant@citrix.com>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: Wei Liu <wei.liu2@citrix.com>
> Cc: Keir Fraser <keir@xen.org>
> Reviewed-by: Jan Beulich <jbeulich@suse.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
> 
> v5:
>  - Make sure vcpu_mask is only compared against vcpu_id < 64
> 
> v4:
>  - Remove extraneous blank line in params.h
>  - Use __cpumask_set_cpu() rather than cpumask_set_cpu()
> 
> v3:
>  - Correct use of cpumask_var_t
>  - Extend comment to explain pcpu_mask flush
>  - Other cosmetic changes
> 
> v2:
>  - Re-name viridian_init/deinit() to viridian_vcpu_init/deinit()
>  - Use alloc/free_cpumask_var()
>  - Use hvm_copy_from_guest_phys() to get hypercall arguments
> ---
>  docs/man/xl.cfg.pod.5              |   6 ++
>  tools/libxl/libxl_dom.c            |   3 +
>  tools/libxl/libxl_types.idl        |   1 +
>  xen/arch/x86/hvm/hvm.c             |  12 ++++
>  xen/arch/x86/hvm/viridian.c        | 123 +++++++++++++++++++++++++++++++++----
>  xen/include/asm-x86/hvm/viridian.h |   4 ++
>  xen/include/asm-x86/perfc_defn.h   |   1 +
>  xen/include/public/hvm/params.h    |  13 ++--
>  8 files changed, 146 insertions(+), 17 deletions(-)
> 
> diff --git a/docs/man/xl.cfg.pod.5 b/docs/man/xl.cfg.pod.5
> index b63846a..1a88e36 100644
> --- a/docs/man/xl.cfg.pod.5
> +++ b/docs/man/xl.cfg.pod.5
> @@ -1466,6 +1466,12 @@ This set incorporates the Partition Reference TSC MSR. This
>  enlightenment can improve performance of Windows 7 and Windows
>  Server 2008 R2 onwards.
>  
> +=item B<hcall_remote_tlb_flush>
> +
> +This set incorporates use of hypercalls for remote TLB flushing.
> +This enlightenment may improve performance of Windows guests running
> +on hosts with higher levels of (physical) CPU contention.
> +
>  =item B<defaults>
>  
>  This is a special value that enables the default set of groups, which
> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> index 44d481b..009ca9c 100644
> --- a/tools/libxl/libxl_dom.c
> +++ b/tools/libxl/libxl_dom.c
> @@ -251,6 +251,9 @@ static int hvm_set_viridian_features(libxl__gc *gc, uint32_t domid,
>      if (libxl_bitmap_test(&enlightenments, LIBXL_VIRIDIAN_ENLIGHTENMENT_REFERENCE_TSC))
>          mask |= HVMPV_reference_tsc;
>  
> +    if (libxl_bitmap_test(&enlightenments, LIBXL_VIRIDIAN_ENLIGHTENMENT_HCALL_REMOTE_TLB_FLUSH))
> +        mask |= HVMPV_hcall_remote_tlb_flush;
> +
>      if (mask != 0 &&
>          xc_hvm_param_set(CTX->xch,
>                           domid,
> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> index 4d78f86..0aa5b9d 100644
> --- a/tools/libxl/libxl_types.idl
> +++ b/tools/libxl/libxl_types.idl
> @@ -219,6 +219,7 @@ libxl_viridian_enlightenment = Enumeration("viridian_enlightenment", [
>      (1, "freq"),
>      (2, "time_ref_count"),
>      (3, "reference_tsc"),
> +    (4, "hcall_remote_tlb_flush"),
>      ])
>  

Acked-by: Wei Liu <wei.liu2@citrix.com>