[PATCH XEN v3 0/2] add support for vNVDIMM - Part 1: enable instructions

[PATCH XEN v3 0/2] add support for vNVDIMM - Part 1: enable instructions

[Haozhong Zhang]

I split the previous patch series "[PATCH 0/4] add support for vNVDIMM"
into two parts. This is the first part that enables clflushopt/clwb/pcommit
instructions for HVM guests.

Changes in v3:
 * Remove unnecessary hypervisor code for clflushopt and clwb in patch 1.
 * Move the code refactoring in hvm_cpuid() to patch 2.
 * Replace checking host cpuid by checking VMX capabilities for pcommit.
 * Add a tag 'XEN' in the title to address Ian Campbell's comment in
   the corresponding QEMU patch series.

Changes in v2:
 * Refactor modifications in hvm_cpuid() per Andrew Cooper's comments.

Haozhong Zhang (2):
  x86/hvm: allow guest to use clflushopt and clwb
  x86/hvm: add support for pcommit instruction

 tools/libxc/xc_cpufeature.h        |  4 +++-
 tools/libxc/xc_cpuid_x86.c         |  5 ++++-
 xen/arch/x86/hvm/hvm.c             | 31 +++++++++++++++++++------------
 xen/arch/x86/hvm/vmx/vmcs.c        |  9 ++++++++-
 xen/arch/x86/hvm/vmx/vmx.c         |  1 +
 xen/arch/x86/hvm/vmx/vvmx.c        |  3 +++
 xen/include/asm-x86/cpufeature.h   |  1 +
 xen/include/asm-x86/hvm/vmx/vmcs.h |  4 +++-
 xen/include/asm-x86/hvm/vmx/vmx.h  |  1 +
 9 files changed, 43 insertions(+), 16 deletions(-)

-- 
2.4.8

[PATCH XEN v3 1/2] x86/hvm: allow guest to use clflushopt and clwb

[Haozhong Zhang]

Pass CPU features CLFLUSHOPT and CLWB into HVM domain so that those two
instructions can be used by guest.

The specification of above two instructions can be found in
https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Reviewed-by: Kevin Tian <kevin.tian@intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com> for tools bits
Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
---
 tools/libxc/xc_cpufeature.h | 3 ++-
 tools/libxc/xc_cpuid_x86.c  | 4 +++-
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/tools/libxc/xc_cpufeature.h b/tools/libxc/xc_cpufeature.h
index c3ddc80..5288ac6 100644
--- a/tools/libxc/xc_cpufeature.h
+++ b/tools/libxc/xc_cpufeature.h
@@ -140,6 +140,7 @@
 #define X86_FEATURE_RDSEED      18 /* RDSEED instruction */
 #define X86_FEATURE_ADX         19 /* ADCX, ADOX instructions */
 #define X86_FEATURE_SMAP        20 /* Supervisor Mode Access Protection */
-
+#define X86_FEATURE_CLFLUSHOPT  23 /* CLFLUSHOPT instruction */
+#define X86_FEATURE_CLWB        24 /* CLWB instruction */
 
 #endif /* __LIBXC_CPUFEATURE_H */
diff --git a/tools/libxc/xc_cpuid_x86.c b/tools/libxc/xc_cpuid_x86.c
index 8882c01..fecfd6c 100644
--- a/tools/libxc/xc_cpuid_x86.c
+++ b/tools/libxc/xc_cpuid_x86.c
@@ -426,7 +426,9 @@ static void xc_cpuid_hvm_policy(xc_interface *xch,
                         bitmaskof(X86_FEATURE_RDSEED)  |
                         bitmaskof(X86_FEATURE_ADX)  |
                         bitmaskof(X86_FEATURE_SMAP) |
-                        bitmaskof(X86_FEATURE_FSGSBASE));
+                        bitmaskof(X86_FEATURE_FSGSBASE) |
+                        bitmaskof(X86_FEATURE_CLWB) |
+                        bitmaskof(X86_FEATURE_CLFLUSHOPT));
         } else
             regs[1] = 0;
         regs[0] = regs[2] = regs[3] = 0;
-- 
2.4.8

[PATCH XEN v3 2/2] x86/hvm: add support for pcommit instruction

[Haozhong Zhang]

Pass PCOMMIT CPU feature into HVM domain. Currently, we do not intercept
pcommit instruction for L1 guest, and allow L1 to intercept pcommit
instruction for L2 guest.

The specification of pcommit instruction can be found in
https://software.intel.com/sites/default/files/managed/0d/53/319433-022.pdf

Reviewed-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Kevin Tian <kevin.tian@intel.com>
Acked-by: Wei Liu <wei.liu2@citrix.com> for tools bits
Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
---
 tools/libxc/xc_cpufeature.h        |  1 +
 tools/libxc/xc_cpuid_x86.c         |  1 +
 xen/arch/x86/hvm/hvm.c             | 31 +++++++++++++++++++------------
 xen/arch/x86/hvm/vmx/vmcs.c        |  9 ++++++++-
 xen/arch/x86/hvm/vmx/vmx.c         |  1 +
 xen/arch/x86/hvm/vmx/vvmx.c        |  3 +++
 xen/include/asm-x86/cpufeature.h   |  1 +
 xen/include/asm-x86/hvm/vmx/vmcs.h |  4 +++-
 xen/include/asm-x86/hvm/vmx/vmx.h  |  1 +
 9 files changed, 38 insertions(+), 14 deletions(-)

diff --git a/tools/libxc/xc_cpufeature.h b/tools/libxc/xc_cpufeature.h
index 5288ac6..ee53679 100644
--- a/tools/libxc/xc_cpufeature.h
+++ b/tools/libxc/xc_cpufeature.h
@@ -140,6 +140,7 @@
 #define X86_FEATURE_RDSEED      18 /* RDSEED instruction */
 #define X86_FEATURE_ADX         19 /* ADCX, ADOX instructions */
 #define X86_FEATURE_SMAP        20 /* Supervisor Mode Access Protection */
+#define X86_FEATURE_PCOMMIT     22 /* PCOMMIT instruction */
 #define X86_FEATURE_CLFLUSHOPT  23 /* CLFLUSHOPT instruction */
 #define X86_FEATURE_CLWB        24 /* CLWB instruction */
 
diff --git a/tools/libxc/xc_cpuid_x86.c b/tools/libxc/xc_cpuid_x86.c
index fecfd6c..c142595 100644
--- a/tools/libxc/xc_cpuid_x86.c
+++ b/tools/libxc/xc_cpuid_x86.c
@@ -427,6 +427,7 @@ static void xc_cpuid_hvm_policy(xc_interface *xch,
                         bitmaskof(X86_FEATURE_ADX)  |
                         bitmaskof(X86_FEATURE_SMAP) |
                         bitmaskof(X86_FEATURE_FSGSBASE) |
+                        bitmaskof(X86_FEATURE_PCOMMIT) |
                         bitmaskof(X86_FEATURE_CLWB) |
                         bitmaskof(X86_FEATURE_CLFLUSHOPT));
         } else
diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 21470ec..787b7de 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -4583,21 +4583,28 @@ void hvm_cpuid(unsigned int input, unsigned int *eax, unsigned int *ebx,
             *edx &= ~cpufeat_mask(X86_FEATURE_PSE36);
         break;
     case 0x7:
-        if ( (count == 0) && !cpu_has_smep )
-            *ebx &= ~cpufeat_mask(X86_FEATURE_SMEP);
+        if ( count == 0 )
+        {
+            if ( !cpu_has_smep )
+                *ebx &= ~cpufeat_mask(X86_FEATURE_SMEP);
+
+            if ( !cpu_has_smap )
+                *ebx &= ~cpufeat_mask(X86_FEATURE_SMAP);
 
-        if ( (count == 0) && !cpu_has_smap )
-            *ebx &= ~cpufeat_mask(X86_FEATURE_SMAP);
+            /* Don't expose MPX to hvm when VMX support is not available */
+            if ( !(vmx_vmexit_control & VM_EXIT_CLEAR_BNDCFGS) ||
+                 !(vmx_vmentry_control & VM_ENTRY_LOAD_BNDCFGS) )
+                *ebx &= ~cpufeat_mask(X86_FEATURE_MPX);
 
-        /* Don't expose MPX to hvm when VMX support is not available */
-        if ( (count == 0) &&
-             (!(vmx_vmexit_control & VM_EXIT_CLEAR_BNDCFGS) ||
-              !(vmx_vmentry_control & VM_ENTRY_LOAD_BNDCFGS)) )
-            *ebx &= ~cpufeat_mask(X86_FEATURE_MPX);
+            /* Don't expose INVPCID to non-hap hvm. */
+            if ( !hap_enabled(d) )
+                *ebx &= ~cpufeat_mask(X86_FEATURE_INVPCID);
+
+            /* Don't expose PCOMMIT to hvm when VMX support is not available */
+            if ( !cpu_has_vmx_pcommit )
+                *ebx &= ~cpufeat_mask(X86_FEATURE_PCOMMIT);
+        }
 
-        /* Don't expose INVPCID to non-hap hvm. */
-        if ( (count == 0) && !hap_enabled(d) )
-            *ebx &= ~cpufeat_mask(X86_FEATURE_INVPCID);
         break;
     case 0xb:
         /* Fix the x2APIC identifier. */
diff --git a/xen/arch/x86/hvm/vmx/vmcs.c b/xen/arch/x86/hvm/vmx/vmcs.c
index edd4c8d..5bc3c74 100644
--- a/xen/arch/x86/hvm/vmx/vmcs.c
+++ b/xen/arch/x86/hvm/vmx/vmcs.c
@@ -242,7 +242,8 @@ static int vmx_init_vmcs_config(void)
                SECONDARY_EXEC_ENABLE_INVPCID |
                SECONDARY_EXEC_ENABLE_VM_FUNCTIONS |
                SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS |
-               SECONDARY_EXEC_XSAVES);
+               SECONDARY_EXEC_XSAVES |
+               SECONDARY_EXEC_PCOMMIT);
         rdmsrl(MSR_IA32_VMX_MISC, _vmx_misc_cap);
         if ( _vmx_misc_cap & VMX_MISC_VMWRITE_ALL )
             opt |= SECONDARY_EXEC_ENABLE_VMCS_SHADOWING;
@@ -1075,6 +1076,12 @@ static int construct_vmcs(struct vcpu *v)
         __vmwrite(PLE_WINDOW, ple_window);
     }
 
+    /*
+     * We do not intercept pcommit for L1 guest and allow L1 hypervisor to
+     * intercept pcommit for L2 guest (see nvmx_n2_vmexit_handler()).
+     */
+    v->arch.hvm_vmx.secondary_exec_control &= ~SECONDARY_EXEC_PCOMMIT;
+
     if ( cpu_has_vmx_secondary_exec_control )
         __vmwrite(SECONDARY_VM_EXEC_CONTROL,
                   v->arch.hvm_vmx.secondary_exec_control);
diff --git a/xen/arch/x86/hvm/vmx/vmx.c b/xen/arch/x86/hvm/vmx/vmx.c
index b918b8a..0991cdf 100644
--- a/xen/arch/x86/hvm/vmx/vmx.c
+++ b/xen/arch/x86/hvm/vmx/vmx.c
@@ -3517,6 +3517,7 @@ void vmx_vmexit_handler(struct cpu_user_regs *regs)
     case EXIT_REASON_ACCESS_LDTR_OR_TR:
     case EXIT_REASON_VMX_PREEMPTION_TIMER_EXPIRED:
     case EXIT_REASON_INVPCID:
+    case EXIT_REASON_PCOMMIT:
     /* fall through */
     default:
     exit_and_crash:
diff --git a/xen/arch/x86/hvm/vmx/vvmx.c b/xen/arch/x86/hvm/vmx/vvmx.c
index ea1052e..271ec70 100644
--- a/xen/arch/x86/hvm/vmx/vvmx.c
+++ b/xen/arch/x86/hvm/vmx/vvmx.c
@@ -1950,6 +1950,8 @@ int nvmx_msr_read_intercept(unsigned int msr, u64 *msr_content)
                SECONDARY_EXEC_ENABLE_VPID |
                SECONDARY_EXEC_UNRESTRICTED_GUEST |
                SECONDARY_EXEC_ENABLE_EPT;
+        if ( cpu_has_vmx_pcommit )
+            data |= SECONDARY_EXEC_PCOMMIT;
         data = gen_vmx_msr(data, 0, host_data);
         break;
     case MSR_IA32_VMX_EXIT_CTLS:
@@ -2226,6 +2228,7 @@ int nvmx_n2_vmexit_handler(struct cpu_user_regs *regs,
     case EXIT_REASON_VMXON:
     case EXIT_REASON_INVEPT:
     case EXIT_REASON_XSETBV:
+    case EXIT_REASON_PCOMMIT:
         /* inject to L1 */
         nvcpu->nv_vmexit_pending = 1;
         break;
diff --git a/xen/include/asm-x86/cpufeature.h b/xen/include/asm-x86/cpufeature.h
index ef96514..23f9fb2 100644
--- a/xen/include/asm-x86/cpufeature.h
+++ b/xen/include/asm-x86/cpufeature.h
@@ -162,6 +162,7 @@
 #define X86_FEATURE_RDSEED	(7*32+18) /* RDSEED instruction */
 #define X86_FEATURE_ADX		(7*32+19) /* ADCX, ADOX instructions */
 #define X86_FEATURE_SMAP	(7*32+20) /* Supervisor Mode Access Prevention */
+#define X86_FEATURE_PCOMMIT	(7*32+22) /* PCOMMIT instruction */
 
 /* Intel-defined CPU features, CPUID level 0x00000007:0 (ecx), word 8 */
 #define X86_FEATURE_PKU	(8*32+ 3) /* Protection Keys for Userspace */
diff --git a/xen/include/asm-x86/hvm/vmx/vmcs.h b/xen/include/asm-x86/hvm/vmx/vmcs.h
index d1496b8..a5e7aee 100644
--- a/xen/include/asm-x86/hvm/vmx/vmcs.h
+++ b/xen/include/asm-x86/hvm/vmx/vmcs.h
@@ -236,6 +236,7 @@ extern u32 vmx_vmentry_control;
 #define SECONDARY_EXEC_ENABLE_PML               0x00020000
 #define SECONDARY_EXEC_ENABLE_VIRT_EXCEPTIONS   0x00040000
 #define SECONDARY_EXEC_XSAVES                   0x00100000
+#define SECONDARY_EXEC_PCOMMIT                  0x00200000
 extern u32 vmx_secondary_exec_control;
 
 #define VMX_EPT_EXEC_ONLY_SUPPORTED                         0x00000001
@@ -303,7 +304,8 @@ extern u64 vmx_ept_vpid_cap;
     (vmx_secondary_exec_control & SECONDARY_EXEC_ENABLE_PML)
 #define cpu_has_vmx_xsaves \
     (vmx_secondary_exec_control & SECONDARY_EXEC_XSAVES)
-
+#define cpu_has_vmx_pcommit \
+    (vmx_secondary_exec_control & SECONDARY_EXEC_PCOMMIT)
 #define VMCS_RID_TYPE_MASK              0x80000000
 
 /* GUEST_INTERRUPTIBILITY_INFO flags. */
diff --git a/xen/include/asm-x86/hvm/vmx/vmx.h b/xen/include/asm-x86/hvm/vmx/vmx.h
index 1719965..14f3d32 100644
--- a/xen/include/asm-x86/hvm/vmx/vmx.h
+++ b/xen/include/asm-x86/hvm/vmx/vmx.h
@@ -213,6 +213,7 @@ static inline void pi_clear_sn(struct pi_desc *pi_desc)
 #define EXIT_REASON_PML_FULL            62
 #define EXIT_REASON_XSAVES              63
 #define EXIT_REASON_XRSTORS             64
+#define EXIT_REASON_PCOMMIT             65
 
 /*
  * Interruption-information format
-- 
2.4.8

Re: [PATCH XEN v3 0/2] add support for vNVDIMM - Part 1: enable instructions

[Jan Beulich]

>>> On 08.01.16 at 11:46, <haozhong.zhang@intel.com> wrote:
> I split the previous patch series "[PATCH 0/4] add support for vNVDIMM"
> into two parts. This is the first part that enables clflushopt/clwb/pcommit
> instructions for HVM guests.
> 
> Changes in v3:
>  * Remove unnecessary hypervisor code for clflushopt and clwb in patch 1.

Which should have led to a Cc list reduction.

>  * Move the code refactoring in hvm_cpuid() to patch 2.
>  * Replace checking host cpuid by checking VMX capabilities for pcommit.

Replace?

Jan

Re: [PATCH XEN v3 0/2] add support for vNVDIMM - Part 1: enable instructions

[Haozhong Zhang]

On 01/08/16 04:50, Jan Beulich wrote:
> >>> On 08.01.16 at 11:46, <haozhong.zhang@intel.com> wrote:
> > I split the previous patch series "[PATCH 0/4] add support for vNVDIMM"
> > into two parts. This is the first part that enables clflushopt/clwb/pcommit
> > instructions for HVM guests.
> > 
> > Changes in v3:
> >  * Remove unnecessary hypervisor code for clflushopt and clwb in patch 1.
> 
> Which should have led to a Cc list reduction.
>

I'll take care in the next time.

> >  * Move the code refactoring in hvm_cpuid() to patch 2.
> >  * Replace checking host cpuid by checking VMX capabilities for pcommit.
> 
> Replace?
>

Yes, akin to how MPX is checked in hvm_cpuid(). Or do you mean it's
better to check host cpuid as well?

Haozhong

Re: [PATCH XEN v3 0/2] add support for vNVDIMM - Part 1: enable instructions

[Jan Beulich]

>>> On 08.01.16 at 14:03, <haozhong.zhang@intel.com> wrote:
> On 01/08/16 04:50, Jan Beulich wrote:
>> >>> On 08.01.16 at 11:46, <haozhong.zhang@intel.com> wrote:
>> >  * Move the code refactoring in hvm_cpuid() to patch 2.
>> >  * Replace checking host cpuid by checking VMX capabilities for pcommit.
>> 
>> Replace?
> 
> Yes, akin to how MPX is checked in hvm_cpuid(). Or do you mean it's
> better to check host cpuid as well?

Ah, yes, you're right. Should be fine as is then.

Jan