[PATCH v6 0/2] Introduce a flags field to HVM CPU context

[PATCH v6 0/2] Introduce a flags field to HVM CPU context

[Roger Pau Monne]

Hello,

This patch series tries to properly solve the problem seen with the HVMlite 
series, that Xen always assumes the FPU is initialised on CPU context 
restore.

Roger.

[PATCH v6 1/2] xen/hvm: introduce a flags field in the CPU save record

[Roger Pau Monne]

Introduce a new flags field and use bit 0 to signal if the FPU has been
initialised or not. Previously Xen always wrongly assumed the FPU was
initialised on restore.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
---
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
---
Changes since v5:
 - Adapt the code due to the changes introduced by XSA-165.

Changes since v4:
 - Zero the fpu save area if the FPU is not initialised. This avoids
   leaking hypervisor data.
 - Due to the previous patch in this series, slightly rework the way to load
   the FPU state in a xsave capable host.

Changes since v3:
 - Don't add a comment in the compat structure regaring the fpu_initialised
   field.
 - Rename fpu_initialised to flags and use it as a bit field. Bit 0 will be
   used to signal whether the fpu is initialised.
 - Only save the fpu context if it's initialised.
 - Only restore the fpu context from the save record if the fpu is
   initialised.
 - Check that unused bits in the flags field are 0.

Changes since v1:
 - Don't add yet another compat structure, new fields should always be added
   to the end of the existing structure and offsetof should be used to
   compare sizes.
 - Leave the previous compat structure as-is, since the field was not added
   to the end we cannot remove it and use offsetof in this case.
 - Set xstate_bv based on fpu_initialised value instead of unconditionally
   setting it to XSTATE_FP_SSE.
---
 xen/arch/x86/hvm/hvm.c                 | 42 ++++++++++++++++++++++------------
 xen/include/public/arch-x86/hvm/save.h | 27 ++++++++++++++++------
 2 files changed, 48 insertions(+), 21 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 787b7de..79ec90f 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -1797,9 +1797,15 @@ static int hvm_save_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
         ctxt.ldtr_arbytes = seg.attr.bytes;
 
         if ( v->fpu_initialised )
+        {
             memcpy(ctxt.fpu_regs, v->arch.fpu_ctxt, sizeof(ctxt.fpu_regs));
-        else 
-            memset(ctxt.fpu_regs, 0, sizeof(ctxt.fpu_regs));
+            ctxt.flags = XEN_X86_FPU_INITIALISED;
+        }
+        else
+        {
+             memset(ctxt.fpu_regs, 0, sizeof(ctxt.fpu_regs));
+             ctxt.flags = 0;
+        }
 
         ctxt.rax = v->arch.user_regs.eax;
         ctxt.rbx = v->arch.user_regs.ebx;
@@ -1981,7 +1987,7 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
         return -EINVAL;
     }
 
-    if ( hvm_load_entry(CPU, h, &ctxt) != 0 ) 
+    if ( hvm_load_entry_zeroextend(CPU, h, &ctxt) != 0 )
         return -EINVAL;
 
     /* Sanity check some control registers. */
@@ -2009,6 +2015,13 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
         return -EINVAL;
     }
 
+    if ( (ctxt.flags & ~XEN_X86_FPU_INITIALISED) != 0 )
+    {
+        gprintk(XENLOG_ERR, "bad flags value in CPU context: %#x\n",
+                ctxt.flags);
+        return -EINVAL;
+    }
+
     /* Older Xen versions used to save the segment arbytes directly 
      * from the VMCS on Intel hosts.  Detect this and rearrange them
      * into the struct segment_register format. */
@@ -2087,19 +2100,21 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
     seg.attr.bytes = ctxt.ldtr_arbytes;
     hvm_set_segment_register(v, x86_seg_ldtr, &seg);
 
-    /* In case xsave-absent save file is restored on a xsave-capable host */
-    if ( cpu_has_xsave && !xsave_enabled(v) )
+    v->fpu_initialised = !!(ctxt.flags & XEN_X86_FPU_INITIALISED);
+    if ( v->fpu_initialised )
     {
-        struct xsave_struct *xsave_area = v->arch.xsave_area;
+        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
+        /* In case xsave-absent save file is restored on a xsave-capable host */
+        if ( cpu_has_xsave && !xsave_enabled(v) )
+        {
+            struct xsave_struct *xsave_area = v->arch.xsave_area;
 
-        memcpy(v->arch.xsave_area, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
-        xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
-        if ( cpu_has_xsaves || cpu_has_xsavec )
-            xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
-                                             XSTATE_COMPACTION_ENABLED;
+            xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
+            if ( cpu_has_xsaves || cpu_has_xsavec )
+                xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
+                                                 XSTATE_COMPACTION_ENABLED;
+        }
     }
-    else
-        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
 
     v->arch.user_regs.eax = ctxt.rax;
     v->arch.user_regs.ebx = ctxt.rbx;
@@ -2127,7 +2142,6 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
     v->arch.debugreg[7] = ctxt.dr7;
 
     v->arch.vgc_flags = VGCF_online;
-    v->fpu_initialised = 1;
 
     /* Auxiliary processors should be woken immediately. */
     v->is_initialised = 1;
diff --git a/xen/include/public/arch-x86/hvm/save.h b/xen/include/public/arch-x86/hvm/save.h
index 29d513c..b6b1bf8 100644
--- a/xen/include/public/arch-x86/hvm/save.h
+++ b/xen/include/public/arch-x86/hvm/save.h
@@ -47,7 +47,9 @@ DECLARE_HVM_SAVE_TYPE(HEADER, 1, struct hvm_save_header);
 /*
  * Processor
  *
- * Compat: Pre-3.4 didn't have msr_tsc_aux
+ * Compat:
+ *     - Pre-3.4 didn't have msr_tsc_aux
+ *     - Pre-4.7 didn't have fpu_initialised
  */
 
 struct hvm_hw_cpu {
@@ -157,6 +159,10 @@ struct hvm_hw_cpu {
     };
     /* error code for pending event */
     uint32_t error_code;
+
+#define _XEN_X86_FPU_INITIALISED        0
+#define XEN_X86_FPU_INITIALISED         (1U<<_XEN_X86_FPU_INITIALISED)
+    uint32_t flags;
 };
 
 struct hvm_hw_cpu_compat {
@@ -275,12 +281,19 @@ static inline int _hvm_hw_fix_cpu(void *h, uint32_t size) {
         struct hvm_hw_cpu_compat cmp;
     } *ucpu = (union hvm_hw_cpu_union *)h;
 
-    /* If we copy from the end backwards, we should
-     * be able to do the modification in-place */
-    ucpu->nat.error_code = ucpu->cmp.error_code;
-    ucpu->nat.pending_event = ucpu->cmp.pending_event;
-    ucpu->nat.tsc = ucpu->cmp.tsc;
-    ucpu->nat.msr_tsc_aux = 0;
+    if ( size == sizeof(struct hvm_hw_cpu_compat) )
+    {
+        /*
+         * If we copy from the end backwards, we should
+         * be able to do the modification in-place.
+         */
+        ucpu->nat.error_code = ucpu->cmp.error_code;
+        ucpu->nat.pending_event = ucpu->cmp.pending_event;
+        ucpu->nat.tsc = ucpu->cmp.tsc;
+        ucpu->nat.msr_tsc_aux = 0;
+    }
+    /* Mimic the old behaviour by unconditionally setting fpu_initialised. */
+    ucpu->nat.flags = XEN_X86_FPU_INITIALISED;
 
     return 0;
 }
-- 
1.9.5 (Apple Git-50.3)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[PATCH v6 2/2] Revert "libxc: create an initial FPU state for HVM guests"

[Roger Pau Monne]

This reverts commit d64dbbcc7c9934a46126c59d78536235908377ad:

Xen always set the FPU as initialized when loading a HVM context, so libxc
has to provide a valid FPU context when setting the CPU registers.

This was a stop-gap measure in order to unblock OSSTest Windows 7 failures
while a proper fix for the HVM CPU save/restore is being worked on.

This can now be reverted because a proper fix is in place and we can signal
in the save record whether the FPU is initialized or not.

Signed-off-by: Roger Pau Monné <roger.pau@citrix.com>
Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>
---
 tools/libxc/xc_dom_x86.c | 38 --------------------------------------
 1 file changed, 38 deletions(-)

diff --git a/tools/libxc/xc_dom_x86.c b/tools/libxc/xc_dom_x86.c
index b8d2904..b650762 100644
--- a/tools/libxc/xc_dom_x86.c
+++ b/tools/libxc/xc_dom_x86.c
@@ -987,27 +987,6 @@ static int vcpu_hvm(struct xc_dom_image *dom)
         struct hvm_save_descriptor end_d;
         HVM_SAVE_TYPE(END) end;
     } bsp_ctx;
-    /*
-     * The layout of the fpu context structure is the same for
-     * both 32 and 64 bits.
-     */
-    struct {
-        uint16_t fcw;
-        uint16_t fsw;
-        uint8_t ftw;
-        uint8_t rsvd1;
-        uint16_t fop;
-        union {
-            uint64_t addr;
-            struct {
-                uint32_t offs;
-                uint16_t sel;
-                uint16_t rsvd;
-            };
-        } fip, fdp;
-        uint32_t mxcsr;
-        uint32_t mxcsr_mask;
-    } *fpu_ctxt;
     uint8_t *full_ctx = NULL;
     int rc;
 
@@ -1075,23 +1054,6 @@ static int vcpu_hvm(struct xc_dom_image *dom)
     /* Set the control registers. */
     bsp_ctx.cpu.cr0 = X86_CR0_PE | X86_CR0_ET;
 
-    /*
-     * XXX: Set initial FPU state.
-     *
-     * This should be removed once Xen is able to know if the
-     * FPU state saved is valid or not, now Xen always sets
-     * fpu_initialised to true regardless of the FPU state.
-     *
-     * The code below mimics the FPU sate after executing
-     * fninit
-     * ldmxcsr 0x1f80
-     */
-    fpu_ctxt = (typeof(fpu_ctxt))bsp_ctx.cpu.fpu_regs;
-
-    fpu_ctxt->fcw = 0x37f;
-    fpu_ctxt->ftw = 0xff;
-    fpu_ctxt->mxcsr = 0x1f80;
-
     /* Set the IP. */
     bsp_ctx.cpu.rip = dom->parms.phys_entry;
 
-- 
1.9.5 (Apple Git-50.3)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [PATCH v6 1/2] xen/hvm: introduce a flags field in the CPU save record

[Jan Beulich]

>>> On 12.01.16 at 17:12, <roger.pau@citrix.com> wrote:
> @@ -2087,19 +2100,21 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
>      seg.attr.bytes = ctxt.ldtr_arbytes;
>      hvm_set_segment_register(v, x86_seg_ldtr, &seg);
>  
> -    /* In case xsave-absent save file is restored on a xsave-capable host */
> -    if ( cpu_has_xsave && !xsave_enabled(v) )
> +    v->fpu_initialised = !!(ctxt.flags & XEN_X86_FPU_INITIALISED);
> +    if ( v->fpu_initialised )
>      {
> -        struct xsave_struct *xsave_area = v->arch.xsave_area;
> +        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
> +        /* In case xsave-absent save file is restored on a xsave-capable host */
> +        if ( cpu_has_xsave && !xsave_enabled(v) )
> +        {
> +            struct xsave_struct *xsave_area = v->arch.xsave_area;
>  
> -        memcpy(v->arch.xsave_area, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
> -        xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
> -        if ( cpu_has_xsaves || cpu_has_xsavec )
> -            xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
> -                                             XSTATE_COMPACTION_ENABLED;
> +            xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
> +            if ( cpu_has_xsaves || cpu_has_xsavec )
> +                xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
> +                                                 XSTATE_COMPACTION_ENABLED;
> +        }
>      }
> -    else
> -        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
>  

I would have expected this to simply be re-indentation, yet
you changed from if/else to just if with the else code done
ahead of it. If this really is intended, the commit message should
explain it.

Jan

Re: [PATCH v6 1/2] xen/hvm: introduce a flags field in the CPU save record

[Roger Pau Monné]

El 12/01/16 a les 17.31, Jan Beulich ha escrit:
>>>> On 12.01.16 at 17:12, <roger.pau@citrix.com> wrote:
>> @@ -2087,19 +2100,21 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
>>      seg.attr.bytes = ctxt.ldtr_arbytes;
>>      hvm_set_segment_register(v, x86_seg_ldtr, &seg);
>>  
>> -    /* In case xsave-absent save file is restored on a xsave-capable host */
>> -    if ( cpu_has_xsave && !xsave_enabled(v) )
>> +    v->fpu_initialised = !!(ctxt.flags & XEN_X86_FPU_INITIALISED);
>> +    if ( v->fpu_initialised )
>>      {
>> -        struct xsave_struct *xsave_area = v->arch.xsave_area;
>> +        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
>> +        /* In case xsave-absent save file is restored on a xsave-capable host */
>> +        if ( cpu_has_xsave && !xsave_enabled(v) )
>> +        {
>> +            struct xsave_struct *xsave_area = v->arch.xsave_area;
>>  
>> -        memcpy(v->arch.xsave_area, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
>> -        xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
>> -        if ( cpu_has_xsaves || cpu_has_xsavec )
>> -            xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
>> -                                             XSTATE_COMPACTION_ENABLED;
>> +            xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
>> +            if ( cpu_has_xsaves || cpu_has_xsavec )
>> +                xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
>> +                                                 XSTATE_COMPACTION_ENABLED;
>> +        }
>>      }
>> -    else
>> -        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
>>  
> 
> I would have expected this to simply be re-indentation, yet
> you changed from if/else to just if with the else code done
> ahead of it. If this really is intended, the commit message should
> explain it.

Right, sorry. AFAICT v->arch.fpu_ctxt points to the xsave_area (as set
by vcpu_init_fpu), so I though it was simpler to just do one memcpy for
both cases, since v->arch.fpu_ctxt always points to the right area for
either cases (and I was already modifying the code in question).

I can see that this might be seen as an unrelated change, so if you want
I can split it into a separate patch, or add the following to the commit
message:

"While modifying the FPU restore part of hvm_load_cpu_ctxt remove the
memcpy branching, since v->arch.fpu_ctxt will always point to the right
area for hosts with XSAVE or without it."

Thanks, Roger.

Re: [PATCH v6 1/2] xen/hvm: introduce a flags field in the CPU save record

[Jan Beulich]

>>> On 12.01.16 at 18:49, <roger.pau@citrix.com> wrote:
> El 12/01/16 a les 17.31, Jan Beulich ha escrit:
>>>>> On 12.01.16 at 17:12, <roger.pau@citrix.com> wrote:
>>> @@ -2087,19 +2100,21 @@ static int hvm_load_cpu_ctxt(struct domain *d, hvm_domain_context_t *h)
>>>      seg.attr.bytes = ctxt.ldtr_arbytes;
>>>      hvm_set_segment_register(v, x86_seg_ldtr, &seg);
>>>  
>>> -    /* In case xsave-absent save file is restored on a xsave-capable host */
>>> -    if ( cpu_has_xsave && !xsave_enabled(v) )
>>> +    v->fpu_initialised = !!(ctxt.flags & XEN_X86_FPU_INITIALISED);
>>> +    if ( v->fpu_initialised )
>>>      {
>>> -        struct xsave_struct *xsave_area = v->arch.xsave_area;
>>> +        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
>>> +        /* In case xsave-absent save file is restored on a xsave-capable host */
>>> +        if ( cpu_has_xsave && !xsave_enabled(v) )
>>> +        {
>>> +            struct xsave_struct *xsave_area = v->arch.xsave_area;
>>>  
>>> -        memcpy(v->arch.xsave_area, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
>>> -        xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
>>> -        if ( cpu_has_xsaves || cpu_has_xsavec )
>>> -            xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
>>> -                                             XSTATE_COMPACTION_ENABLED;
>>> +            xsave_area->xsave_hdr.xstate_bv = XSTATE_FP_SSE;
>>> +            if ( cpu_has_xsaves || cpu_has_xsavec )
>>> +                xsave_area->xsave_hdr.xcomp_bv = XSTATE_FP_SSE |
>>> +                                                 XSTATE_COMPACTION_ENABLED;
>>> +        }
>>>      }
>>> -    else
>>> -        memcpy(v->arch.fpu_ctxt, ctxt.fpu_regs, sizeof(ctxt.fpu_regs));
>>>  
>> 
>> I would have expected this to simply be re-indentation, yet
>> you changed from if/else to just if with the else code done
>> ahead of it. If this really is intended, the commit message should
>> explain it.
> 
> Right, sorry. AFAICT v->arch.fpu_ctxt points to the xsave_area (as set
> by vcpu_init_fpu), so I though it was simpler to just do one memcpy for
> both cases, since v->arch.fpu_ctxt always points to the right area for
> either cases (and I was already modifying the code in question).
> 
> I can see that this might be seen as an unrelated change, so if you want
> I can split it into a separate patch, or add the following to the commit
> message:
> 
> "While modifying the FPU restore part of hvm_load_cpu_ctxt remove the
> memcpy branching, since v->arch.fpu_ctxt will always point to the right
> area for hosts with XSAVE or without it."

I'm fine with that commit message extension, and will be happy to
add it while committing unless a need arises for you to re-submit.

Jan