[PATCH v5 0/3] Introduce a flags field to HVM CPU context

[PATCH v5 0/3] Introduce a flags field to HVM CPU context

[Roger Pau Monne]

Hello,

This patch series tries to properly solve the problem seen with the HVMlite 
series, that Xen always assumes the FPU is initialised on CPU context 
restore.

Patch 1/3 is new in this iteration of the series, and aims to unify the way 
FPU area memory is initialised on xsave capable hosts vs non-xsave capable 
ones.

Roger.

[PATCH v5 1/3] xsave: fix initialization of FPU memory area

[Roger Pau Monne]

When the CPU supports XSAVE, the initialization of the FPU memory area is
done during allocation of the XSAVE area. This is different from the way
it's done in non-xsave capable CPUs, so unify both paths and always
initialise the FPU state in vcpu_restore_fpu_lazy.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
 xen/arch/x86/i387.c   | 2 +-
 xen/arch/x86/xstate.c | 8 --------
 2 files changed, 1 insertion(+), 9 deletions(-)

diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c
index b661d39..fcd87a6 100644
--- a/xen/arch/x86/i387.c
+++ b/xen/arch/x86/i387.c
@@ -226,7 +226,7 @@ void vcpu_restore_fpu_lazy(struct vcpu *v)
     if ( v->fpu_dirtied )
         return;
 
-    if ( cpu_has_xsave )
+    if ( cpu_has_xsave && v->fpu_initialised )
         fpu_xrstor(v, XSTATE_LAZY);
     else if ( v->fpu_initialised )
         fpu_fxrstor(v);
diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c
index b65da38..8007342 100644
--- a/xen/arch/x86/xstate.c
+++ b/xen/arch/x86/xstate.c
@@ -415,14 +415,6 @@ int xstate_alloc_save_area(struct vcpu *v)
     if ( save_area == NULL )
         return -ENOMEM;
 
-    /*
-     * Set the memory image to default values, but don't force the context
-     * to be loaded from memory (i.e. keep save_area->xsave_hdr.xstate_bv
-     * clear).
-     */
-    save_area->fpu_sse.fcw = FCW_DEFAULT;
-    save_area->fpu_sse.mxcsr = MXCSR_DEFAULT;
-
     v->arch.xsave_area = save_area;
     v->arch.xcr0 = 0;
     v->arch.xcr0_accum = 0;
-- 
1.9.5 (Apple Git-50.3)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v5 1/3] xsave: fix initialization of FPU memory area

[Andrew Cooper]

On 01/12/15 12:47, Roger Pau Monne wrote:
> When the CPU supports XSAVE, the initialization of the FPU memory area is
> done during allocation of the XSAVE area. This is different from the way
> it's done in non-xsave capable CPUs, so unify both paths and always
> initialise the FPU state in vcpu_restore_fpu_lazy.
>
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
> ---
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
> ---
>  xen/arch/x86/i387.c   | 2 +-
>  xen/arch/x86/xstate.c | 8 --------
>  2 files changed, 1 insertion(+), 9 deletions(-)
>
> diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c
> index b661d39..fcd87a6 100644
> --- a/xen/arch/x86/i387.c
> +++ b/xen/arch/x86/i387.c
> @@ -226,7 +226,7 @@ void vcpu_restore_fpu_lazy(struct vcpu *v)
>      if ( v->fpu_dirtied )
>          return;
>  
> -    if ( cpu_has_xsave )
> +    if ( cpu_has_xsave && v->fpu_initialised )
>          fpu_xrstor(v, XSTATE_LAZY);
>      else if ( v->fpu_initialised )
>          fpu_fxrstor(v);

I feel this would be clearer if it were rearranged as

if ( v->fpu_initialised )
{
    if ( cpu_has_xsave )
        fpu_xrstor(v, XSTATE_LAZY);
    else
    {
        ...
    }
}
else
    fpu_init();


Functionally however, Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

> diff --git a/xen/arch/x86/xstate.c b/xen/arch/x86/xstate.c
> index b65da38..8007342 100644
> --- a/xen/arch/x86/xstate.c
> +++ b/xen/arch/x86/xstate.c
> @@ -415,14 +415,6 @@ int xstate_alloc_save_area(struct vcpu *v)
>      if ( save_area == NULL )
>          return -ENOMEM;
>  
> -    /*
> -     * Set the memory image to default values, but don't force the context
> -     * to be loaded from memory (i.e. keep save_area->xsave_hdr.xstate_bv
> -     * clear).
> -     */
> -    save_area->fpu_sse.fcw = FCW_DEFAULT;
> -    save_area->fpu_sse.mxcsr = MXCSR_DEFAULT;
> -
>      v->arch.xsave_area = save_area;
>      v->arch.xcr0 = 0;
>      v->arch.xcr0_accum = 0;


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v5 1/3] xsave: fix initialization of FPU memory area

[Jan Beulich]

>>> On 01.12.15 at 14:39, <andrew.cooper3@citrix.com> wrote:
> On 01/12/15 12:47, Roger Pau Monne wrote:
>> When the CPU supports XSAVE, the initialization of the FPU memory area is
>> done during allocation of the XSAVE area. This is different from the way
>> it's done in non-xsave capable CPUs, so unify both paths and always
>> initialise the FPU state in vcpu_restore_fpu_lazy.
>>
>> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
>> ---
>> Cc: Jan Beulich <jbeulich@suse.com>
>> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
>> ---
>>  xen/arch/x86/i387.c   | 2 +-
>>  xen/arch/x86/xstate.c | 8 --------
>>  2 files changed, 1 insertion(+), 9 deletions(-)
>>
>> diff --git a/xen/arch/x86/i387.c b/xen/arch/x86/i387.c
>> index b661d39..fcd87a6 100644
>> --- a/xen/arch/x86/i387.c
>> +++ b/xen/arch/x86/i387.c
>> @@ -226,7 +226,7 @@ void vcpu_restore_fpu_lazy(struct vcpu *v)
>>      if ( v->fpu_dirtied )
>>          return;
>>  
>> -    if ( cpu_has_xsave )
>> +    if ( cpu_has_xsave && v->fpu_initialised )
>>          fpu_xrstor(v, XSTATE_LAZY);
>>      else if ( v->fpu_initialised )
>>          fpu_fxrstor(v);
> 
> I feel this would be clearer if it were rearranged as
> 
> if ( v->fpu_initialised )
> {
>     if ( cpu_has_xsave )
>         fpu_xrstor(v, XSTATE_LAZY);
>     else
>     {
>         ...
>     }
> }
> else
>     fpu_init();

I agree, but would want this to then be done as

    if ( !v->fpu_initialised )
        fpu_init();
    else if ( cpu_has_xsave )
        fpu_xrstor(v, XSTATE_LAZY);
    else
        fpu_fxrstor(v);

(perhaps with an unlikely() on the first clause).

However, I'm unconvinced the change as a whole is correct:
XRSTOR initializes more than just FPU and MXCSR.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v5 2/3] xen/hvm: introduce a flags field in the CPU save record

[Roger Pau Monne]

Introduce a new flags field and use bit 0 to signal if the FPU has been
initialised or not. Previously Xen always wrongly assumed the FPU was
initialised on restore.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
Changes since v4:
 - Zero the fpu save area if the FPU is not initialised. This avoids
   leaking hypervisor data.
 - Due to the previous patch in this series, slightly rework the way to load
   the FPU state in a xsave capable host.

Changes since v3:
 - Don't add a comment in the compat structure regaring the fpu_initialised
   field.
 - Rename fpu_initialised to flags and use it as a bit field. Bit 0 will be
   used to signal whether the fpu is initialised.
 - Only save the fpu context if it's initialised.
 - Only restore the fpu context from the save record if the fpu is
   initialised.
 - Check that unused bits in the flags field are 0.

Changes since v1:
 - Don't add yet another compat structure, new fields should always be added
   to the end of the existing structure and offsetof should be used to
   compare sizes.
 - Leave the previous compat structure as-is, since the field was not added
   to the end we cannot remove it and use offsetof in this case.
 - Set xstate_bv based on fpu_initialised value instead of unconditionally
   setting it to XSTATE_FP_SSE.
---
 xen/arch/x86/hvm/hvm.c                 | 42 +++++++++++++++++++++++-----------
 xen/include/public/arch-x86/hvm/save.h | 27 ++++++++++++++++------
 2 files changed, 49 insertions(+), 20 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index db0aeba..ffb3b16 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -1797,9 +1797,15 @@ static int hvm_save_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
         ctxt.ldtr_arbytes = seg.attr.bytes;
 
         if ( v->fpu_initialised )
+        {
             memcpy(ctxt.fpu_regs, v->arch.fpu_ctxt, sizeof(ctxt.fpu_regs));
-        else 
-            memset(ctxt.fpu_regs, 0, sizeof(ctxt.fpu_regs));
+            ctxt.flags = XEN_X86_FPU_INITIALISED;
+        }
+        else
+        {
+             memset(ctxt.fpu_regs, 0, sizeof(ctxt.fpu_regs));
+             ctxt.flags = 0;
+        }
 
         ctxt.rax = v->arch.user_regs.eax;
         ctxt.rbx = v->arch.user_regs.ebx;
@@ -1979,7 +1985,7 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
         return -EINVAL;
     }
 
-    if ( hvm_load_entry(CPU, h, &ctxt) != 0 ) 
+    if ( hvm_load_entry_zeroextend(CPU, h, &ctxt) != 0 )
         return -EINVAL;
 
     /* Sanity check some control registers. */
@@ -2007,6 +2013,13 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
         return -EINVAL;
     }
 
+    if ( (ctxt.flags & ~XEN_X86_FPU_INITIALISED) != 0 )
+    {
+        gprintk(XENLOG_ERR, "bad flags value in CPU context: %#x\n",
+                ctxt.flags);
+        return -EINVAL;
+    }
+
     /* Older Xen versions used to save the segment arbytes directly 
      * from the VMCS on Intel hosts.  Detect this and rearrange them
      * into the struct segment_register format. */
@@ -2085,19 +2098,23 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
     seg.attr.bytes = ctxt.ldtr_arbytes;
     hvm_set_segment_register(v, x86_seg_ldtr, &seg);
 
-    /* In case xsave-absent save file is restored on a xsave-capable host */
-    if ( cpu_has_xsave && !xsave_enabled(v) )
+    v->fpu_initialised = !!(ctxt.flags & XEN_X86_FPU_INITIALISED);
+    if ( v->fpu_initialised )
     {
-        struct xsave_struct *xsave_area = v->arch.xsave_area;
+        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
+        /* In case xsave-absent save file is restored on a xsave-capable host */
+        if ( cpu_has_xsave && !xsave_enabled(v) )
+        {
+            struct xsave_struct *xsave_area = v->arch.xsave_area;
 
-        memcpy(v->arch.xsave_area, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
-        xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
-        if ( cpu_has_xsaves || cpu_has_xsavec )
-            xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
-                                             XSTATE_COMPACTION_ENABLED;
+            xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
+            if ( cpu_has_xsaves || cpu_has_xsavec )
+                xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
+                                                 XSTATE_COMPACTION_ENABLED;
+        }
     }
     else
-        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
+        memset(v->arch.fpu_ctxt, 0, sizeof(ctxt.fpu_regs));
 
     v->arch.user_regs.eax = ctxt.rax;
     v->arch.user_regs.ebx = ctxt.rbx;
@@ -2125,7 +2142,6 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
     v->arch.debugreg[7] = ctxt.dr7;
 
     v->arch.vgc_flags = VGCF_online;
-    v->fpu_initialised = 1;
 
     /* Auxiliary processors should be woken immediately. */
     v->is_initialised = 1;
diff --git a/xen/include/public/arch-x86/hvm/save.h b/xen/include/public/arch-x86/hvm/save.h
index 29d513c..b6b1bf8 100644
--- a/xen/include/public/arch-x86/hvm/save.h
+++ b/xen/include/public/arch-x86/hvm/save.h
@@ -47,7 +47,9 @@ DECLARE_HVM_SAVE_TYPE(HEADER, 1, struct hvm_save_header);
 /*
  * Processor
  *
- * Compat: Pre-3.4 didn't have msr_tsc_aux
+ * Compat:
+ *     - Pre-3.4 didn't have msr_tsc_aux
+ *     - Pre-4.7 didn't have fpu_initialised
  */
 
 struct hvm_hw_cpu {
@@ -157,6 +159,10 @@ struct hvm_hw_cpu {
     };
     /* error code for pending event */
     uint32_t error_code;
+
+#define _XEN_X86_FPU_INITIALISED        0
+#define XEN_X86_FPU_INITIALISED         (1U<<_XEN_X86_FPU_INITIALISED)
+    uint32_t flags;
 };
 
 struct hvm_hw_cpu_compat {
@@ -275,12 +281,19 @@ static inline int _hvm_hw_fix_cpu(void *h, uint32_t size) {
         struct hvm_hw_cpu_compat cmp;
     } *ucpu = (union hvm_hw_cpu_union *)h;
 
-    /* If we copy from the end backwards, we should
-     * be able to do the modification in-place */
-    ucpu->nat.error_code = ucpu->cmp.error_code;
-    ucpu->nat.pending_event = ucpu->cmp.pending_event;
-    ucpu->nat.tsc = ucpu->cmp.tsc;
-    ucpu->nat.msr_tsc_aux = 0;
+    if ( size == sizeof(struct hvm_hw_cpu_compat) )
+    {
+        /*
+         * If we copy from the end backwards, we should
+         * be able to do the modification in-place.
+         */
+        ucpu->nat.error_code = ucpu->cmp.error_code;
+        ucpu->nat.pending_event = ucpu->cmp.pending_event;
+        ucpu->nat.tsc = ucpu->cmp.tsc;
+        ucpu->nat.msr_tsc_aux = 0;
+    }
+    /* Mimic the old behaviour by unconditionally setting fpu_initialised. */
+    ucpu->nat.flags = XEN_X86_FPU_INITIALISED;
 
     return 0;
 }
-- 
1.9.5 (Apple Git-50.3)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v5 2/3] xen/hvm: introduce a flags field in the CPU save record

[Andrew Cooper]

On 01/12/15 12:47, Roger Pau Monne wrote:
> Introduce a new flags field and use bit 0 to signal if the FPU has been
> initialised or not. Previously Xen always wrongly assumed the FPU was
> initialised on restore.
>
> Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v5 3/3] Revert "libxc: create an initial FPU state for HVM guests"

[Roger Pau Monne]

This reverts commit d64dbbcc7c9934a46126c59d78536235908377ad:

Xen always set the FPU as initialized when loading a HVM context, so libxc
has to provide a valid FPU context when setting the CPU registers.

This was a stop-gap measure in order to unblock OSSTest Windows 7 failures
while a proper fix for the HVM CPU save/restore is being worked on.

This can now be reverted because a proper fix is in place and we can signal
in the save record whether the FPU is initialized or not.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
---
 tools/libxc/xc_dom_x86.c | 38 --------------------------------------
 1 file changed, 38 deletions(-)

diff --git a/tools/libxc/xc_dom_x86.c b/tools/libxc/xc_dom_x86.c
index 5ff33ca..50cceee 100644
--- a/tools/libxc/xc_dom_x86.c
+++ b/tools/libxc/xc_dom_x86.c
@@ -910,27 +910,6 @@ static int vcpu_hvm(struct xc_dom_image *dom)
         struct hvm_save_descriptor end_d;
         HVM_SAVE_TYPE(END) end;
     } bsp_ctx;
-    /*
-     * The layout of the fpu context structure is the same for
-     * both 32 and 64 bits.
-     */
-    struct {
-        uint16_t fcw;
-        uint16_t fsw;
-        uint8_t ftw;
-        uint8_t rsvd1;
-        uint16_t fop;
-        union {
-            uint64_t addr;
-            struct {
-                uint32_t offs;
-                uint16_t sel;
-                uint16_t rsvd;
-            };
-        } fip, fdp;
-        uint32_t mxcsr;
-        uint32_t mxcsr_mask;
-    } *fpu_ctxt;
     uint8_t *full_ctx = NULL;
     int rc;
 
@@ -998,23 +977,6 @@ static int vcpu_hvm(struct xc_dom_image *dom)
     /* Set the control registers. */
     bsp_ctx.cpu.cr0 = X86_CR0_PE | X86_CR0_ET;
 
-    /*
-     * XXX: Set initial FPU state.
-     *
-     * This should be removed once Xen is able to know if the
-     * FPU state saved is valid or not, now Xen always sets
-     * fpu_initialised to true regardless of the FPU state.
-     *
-     * The code below mimics the FPU sate after executing
-     * fninit
-     * ldmxcsr 0x1f80
-     */
-    fpu_ctxt = (typeof(fpu_ctxt))bsp_ctx.cpu.fpu_regs;
-
-    fpu_ctxt->fcw = 0x37f;
-    fpu_ctxt->ftw = 0xff;
-    fpu_ctxt->mxcsr = 0x1f80;
-
     /* Set the IP. */
     bsp_ctx.cpu.rip = dom->parms.phys_entry;
 
-- 
1.9.5 (Apple Git-50.3)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel