[Qemu-devel] [PATCH v5 0/4] Add QEMU support for Intel local MCE

[Qemu-devel] [PATCH v5 0/4] Add QEMU support for Intel local MCE

[Haozhong Zhang]

Changes in v5:
 * v4 Patch 1&2 are now merged into v5 Patch 1. (Paolo Bonzini)
 * Add a separate Patch 3 to automatically enable LMCE for "-cpu host"
   if host does support. (Eduardo Habkost)
 * Move the sanity check of misconfiguration of LMCE to a separate
   Patch 4. I'm fine to drop it if it's considered to be
   unnecessary. (Eduardo Habkost)
 * Move check of MCG_LMCE_P capability to the existing code in
   kvm_arch_init_vcpu(). The difference is that if LMCE is not
   supported by host but MCG_LMCE_P is included in env->mcg_cap,
   kvm_arch_init_vcpu() return with error instead of just reporting a
   warning. (Eduardo Habkost)
 * Fix setting has_msr_feature_control in kvm_arch_init_vcpu(). (Paolo
   Bonzini)
 * Fix the type error in fw_cfg file "etc/msr_feature_control".

Changes in v4:
 * Abort starting QEMU if lmce option is present but host does support
   LMCE. (Eduardo Habkost)
 * Remove setting MSR_IA32_FEATURE_CONTROL, which should be left to
   guest. (Radim Krčmá, Paolo Bonzini
 * Adjust error messages in mce_init(). (Boris Petkov)
 * Move adding option 'lmce' to patch 1. (Eduardo Habkost, Paolo Bonzini)
 * Adjust LMCE error message in cpu_post_load(). (Eduardo Habkost)
 * (Patch 3) Add a fw_cfg file 'etc/msr_feature_control' to advise
   bits should be set in MSR_IA32_FEATURE_CONTROL. (Paolo Bonzini)
 * Fix SOB chain in patch 1.

Changes in v3:
 * LMCE can be enabled only for non-intel guests.
 * LMCE is disabled by default and a cpu option 'lmce=on/off' is added
   to explicitly enable/disable LMCE.
 * LMCE is disabled if KVM does not support (even though 'lmce=on').
 * VM on LMCE-enabled QEMU can be only migrated to LMCE-enabled QEMU.
 * MCG_LMCE_P is not included in MCE_CAP_DEF and instead added to
   env->mcg_cap if LMCE is enabled.
 * Code style fix.

This QEMU patch series along with the corresponding KVM patch series
(sent via another email with title "[PATCH v3 0/3] Add KVM support for
Intel local MCE") enables Intel local MCE feature for guest.

Intel Local MCE (LMCE) is a feature on Intel Skylake Server CPU that
can deliver MCE to a single processor thread instead of broadcasting
to all threads, which can reduce software's load when processing MCE
on machines with a large number of processor threads.

The technical details of LMCE can be found in Intel SDM Vol 3, Chapter
"Machine-Check Architecture" (search for 'LMCE'). Basically,
 * The capability of LMCE is indicated by bit 27 (MCG_LMCE_P) of
   MSR_IA32_MCG_CAP.
 * LMCE is enabled by setting bit 20 (MSR_IA32_FEATURE_CONTROL_LMCE)
   of MSR_IA32_FEATURE_CONTROL and bit 0 (MCG_EXT_CTL_LMCE_EN) of
   MSR_IA32_MCG_EXT_CTL.
 * Software can determine if a MCE is local to the current processor
   thread by checking bit 2 (MCG_STATUS_LMCE) of MSR_IA32_MCG_STATUS.

Ashok Raj (1):
  target-i386: KVM: add basic Intel LMCE support

Haozhong Zhang (3):
  i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg
  target-i386: enable LMCE for '-cpu host' if supported by host
  target-i386: abort migration if LMCE config mismatch

 hw/i386/pc.c          | 29 +++++++++++++++++++++++++++++
 target-i386/cpu.c     | 19 ++++++++++++++++++-
 target-i386/cpu.h     | 16 ++++++++++++++++
 target-i386/kvm.c     | 36 +++++++++++++++++++++++++++++++++---
 target-i386/machine.c | 25 +++++++++++++++++++++++++
 5 files changed, 121 insertions(+), 4 deletions(-)

-- 
2.9.0

[Qemu-devel] [PATCH v5 1/4] target-i386: KVM: add basic Intel LMCE support

[Haozhong Zhang]

From: Ashok Raj <ashok.raj@intel.com>

This patch adds the support to inject SRAR and SRAO as LMCE, i.e. they
are injected to only one VCPU rather than broadcast to all VCPUs. As KVM
reports LMCE support on Intel platforms, this features is only available
on Intel platforms.

LMCE is disabled by default and can be enabled/disabled by cpu option
'lmce=on/off'.

Signed-off-by: Ashok Raj <ashok.raj@intel.com>
[Haozhong: Enable LMCE only on Intel platforms
           Disable LMCE by default and add a cpu option 'lmce'
           Handle the error if LMCE is enabled w/o host support
           Remove MCG_LMCE_P from MCE_CAP_DEF
           Add migration support for LMCE
           Minor code style changes]
Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
---
 target-i386/cpu.c     |  4 +++-
 target-i386/cpu.h     | 12 ++++++++++++
 target-i386/kvm.c     | 36 +++++++++++++++++++++++++++++++++---
 target-i386/machine.c | 19 +++++++++++++++++++
 4 files changed, 67 insertions(+), 4 deletions(-)

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index 3bd3cfc..cdab4f1 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -2815,7 +2815,8 @@ static void mce_init(X86CPU *cpu)
     if (((cenv->cpuid_version >> 8) & 0xf) >= 6
         && (cenv->features[FEAT_1_EDX] & (CPUID_MCE | CPUID_MCA)) ==
             (CPUID_MCE | CPUID_MCA)) {
-        cenv->mcg_cap = MCE_CAP_DEF | MCE_BANKS_DEF;
+        cenv->mcg_cap = MCE_CAP_DEF | MCE_BANKS_DEF |
+                        (cpu->enable_lmce ? MCG_LMCE_P : 0);
         cenv->mcg_ctl = ~(uint64_t)0;
         for (bank = 0; bank < MCE_BANKS_DEF; bank++) {
             cenv->mce_banks[bank * 4] = ~(uint64_t)0;
@@ -3262,6 +3263,7 @@ static Property x86_cpu_properties[] = {
     DEFINE_PROP_UINT32("xlevel2", X86CPU, env.cpuid_xlevel2, 0),
     DEFINE_PROP_STRING("hv-vendor-id", X86CPU, hyperv_vendor_id),
     DEFINE_PROP_BOOL("cpuid-0xb", X86CPU, enable_cpuid_0xb, true),
+    DEFINE_PROP_BOOL("lmce", X86CPU, enable_lmce, false),
     DEFINE_PROP_END_OF_LIST()
 };
 
diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index d9ab884..cda3ea1 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -292,6 +292,7 @@
 
 #define MCG_CTL_P       (1ULL<<8)   /* MCG_CAP register available */
 #define MCG_SER_P       (1ULL<<24) /* MCA recovery/new status bits */
+#define MCG_LMCE_P      (1ULL<<27) /* Local Machine Check Supported */
 
 #define MCE_CAP_DEF     (MCG_CTL_P|MCG_SER_P)
 #define MCE_BANKS_DEF   10
@@ -301,6 +302,9 @@
 #define MCG_STATUS_RIPV (1ULL<<0)   /* restart ip valid */
 #define MCG_STATUS_EIPV (1ULL<<1)   /* ip points to correct instruction */
 #define MCG_STATUS_MCIP (1ULL<<2)   /* machine check in progress */
+#define MCG_STATUS_LMCE (1ULL<<3)   /* Local MCE signaled */
+
+#define MCG_EXT_CTL_LMCE_EN (1ULL<<0) /* Local MCE enabled */
 
 #define MCI_STATUS_VAL   (1ULL<<63)  /* valid error */
 #define MCI_STATUS_OVER  (1ULL<<62)  /* previous errors lost */
@@ -343,6 +347,7 @@
 #define MSR_MCG_CAP                     0x179
 #define MSR_MCG_STATUS                  0x17a
 #define MSR_MCG_CTL                     0x17b
+#define MSR_MCG_EXT_CTL                 0x4d0
 
 #define MSR_P6_EVNTSEL0                 0x186
 
@@ -1111,6 +1116,7 @@ typedef struct CPUX86State {
 
     uint64_t mcg_cap;
     uint64_t mcg_ctl;
+    uint64_t mcg_ext_ctl;
     uint64_t mce_banks[MCE_BANKS_DEF*4];
 
     uint64_t tsc_aux;
@@ -1178,6 +1184,12 @@ struct X86CPU {
      */
     bool enable_pmu;
 
+    /* LMCE support can be enabled/disabled via cpu option 'lmce=on/off'. It is
+     * disabled by default to avoid breaking migration between QEMU with
+     * different LMCE configurations.
+     */
+    bool enable_lmce;
+
     /* Compatibility bits for old machine types: */
     bool enable_cpuid_0xb;
 
diff --git a/target-i386/kvm.c b/target-i386/kvm.c
index f3698f1..96c5864 100644
--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -106,6 +106,8 @@ static int has_xsave;
 static int has_xcrs;
 static int has_pit_state2;
 
+static bool has_msr_mcg_ext_ctl;
+
 static struct kvm_cpuid2 *cpuid_cache;
 
 int kvm_has_pit_state2(void)
@@ -382,10 +384,12 @@ static int kvm_get_mce_cap_supported(KVMState *s, uint64_t *mce_cap,
 
 static void kvm_mce_inject(X86CPU *cpu, hwaddr paddr, int code)
 {
+    CPUState *cs = CPU(cpu);
     CPUX86State *env = &cpu->env;
     uint64_t status = MCI_STATUS_VAL | MCI_STATUS_UC | MCI_STATUS_EN |
                       MCI_STATUS_MISCV | MCI_STATUS_ADDRV | MCI_STATUS_S;
     uint64_t mcg_status = MCG_STATUS_MCIP;
+    int flags = 0;
 
     if (code == BUS_MCEERR_AR) {
         status |= MCI_STATUS_AR | 0x134;
@@ -394,10 +398,19 @@ static void kvm_mce_inject(X86CPU *cpu, hwaddr paddr, int code)
         status |= 0xc0;
         mcg_status |= MCG_STATUS_RIPV;
     }
+
+    flags = cpu_x86_support_mca_broadcast(env) ? MCE_INJECT_BROADCAST : 0;
+    /* We need to read back the value of MSR_EXT_MCG_CTL that was set by the
+     * guest kernel back into env->mcg_ext_ctl.
+     */
+    cpu_synchronize_state(cs);
+    if (env->mcg_ext_ctl & MCG_EXT_CTL_LMCE_EN) {
+        mcg_status |= MCG_STATUS_LMCE;
+        flags = 0;
+    }
+
     cpu_x86_inject_mce(NULL, cpu, 9, status, mcg_status, paddr,
-                       (MCM_ADDR_PHYS << 6) | 0xc,
-                       cpu_x86_support_mca_broadcast(env) ?
-                       MCE_INJECT_BROADCAST : 0);
+                       (MCM_ADDR_PHYS << 6) | 0xc, flags);
 }
 
 static void hardware_memory_error(void)
@@ -865,6 +878,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
         unsupported_caps = env->mcg_cap & ~(mcg_cap | MCG_CAP_BANKS_MASK);
         if (unsupported_caps) {
+            if (unsupported_caps & MCG_LMCE_P) {
+                error_report("kvm: LMCE not supported");
+                return -ENOTSUP;
+            }
             error_report("warning: Unsupported MCG_CAP bits: 0x%" PRIx64,
                          unsupported_caps);
         }
@@ -885,6 +902,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
                                   !!(c->ecx & CPUID_EXT_SMX);
     }
 
+    if (env->mcg_cap & MCG_LMCE_P) {
+        has_msr_mcg_ext_ctl = has_msr_feature_control = true;
+    }
+
     c = cpuid_find_entry(&cpuid_data.cpuid, 0x80000007, 0);
     if (c && (c->edx & 1<<8) && invtsc_mig_blocker == NULL) {
         /* for migration */
@@ -1705,6 +1726,9 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
 
         kvm_msr_entry_add(cpu, MSR_MCG_STATUS, env->mcg_status);
         kvm_msr_entry_add(cpu, MSR_MCG_CTL, env->mcg_ctl);
+        if (has_msr_mcg_ext_ctl) {
+            kvm_msr_entry_add(cpu, MSR_MCG_EXT_CTL, env->mcg_ext_ctl);
+        }
         for (i = 0; i < (env->mcg_cap & 0xff) * 4; i++) {
             kvm_msr_entry_add(cpu, MSR_MC0_CTL + i, env->mce_banks[i]);
         }
@@ -2008,6 +2032,9 @@ static int kvm_get_msrs(X86CPU *cpu)
     if (env->mcg_cap) {
         kvm_msr_entry_add(cpu, MSR_MCG_STATUS, 0);
         kvm_msr_entry_add(cpu, MSR_MCG_CTL, 0);
+        if (has_msr_mcg_ext_ctl) {
+            kvm_msr_entry_add(cpu, MSR_MCG_EXT_CTL, 0);
+        }
         for (i = 0; i < (env->mcg_cap & 0xff) * 4; i++) {
             kvm_msr_entry_add(cpu, MSR_MC0_CTL + i, 0);
         }
@@ -2136,6 +2163,9 @@ static int kvm_get_msrs(X86CPU *cpu)
         case MSR_MCG_CTL:
             env->mcg_ctl = msrs[i].data;
             break;
+        case MSR_MCG_EXT_CTL:
+            env->mcg_ext_ctl = msrs[i].data;
+            break;
         case MSR_IA32_MISC_ENABLE:
             env->msr_ia32_misc_enable = msrs[i].data;
             break;
diff --git a/target-i386/machine.c b/target-i386/machine.c
index cb9adf2..71c0e4d 100644
--- a/target-i386/machine.c
+++ b/target-i386/machine.c
@@ -896,6 +896,24 @@ static const VMStateDescription vmstate_tsc_khz = {
     }
 };
 
+static bool mcg_ext_ctl_needed(void *opaque)
+{
+    X86CPU *cpu = opaque;
+    CPUX86State *env = &cpu->env;
+    return cpu->enable_lmce && env->mcg_ext_ctl;
+}
+
+static const VMStateDescription vmstate_mcg_ext_ctl = {
+    .name = "cpu/mcg_ext_ctl",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = mcg_ext_ctl_needed,
+    .fields = (VMStateField[]) {
+        VMSTATE_UINT64(env.mcg_ext_ctl, X86CPU),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 VMStateDescription vmstate_x86_cpu = {
     .name = "cpu",
     .version_id = 12,
@@ -1022,6 +1040,7 @@ VMStateDescription vmstate_x86_cpu = {
 #ifdef TARGET_X86_64
         &vmstate_pkru,
 #endif
+        &vmstate_mcg_ext_ctl,
         NULL
     }
 };
-- 
2.9.0

Re: [Qemu-devel] [PATCH v5 1/4] target-i386: KVM: add basic Intel LMCE support

[Eduardo Habkost]

On Wed, Jun 22, 2016 at 02:56:21PM +0800, Haozhong Zhang wrote:
> From: Ashok Raj <ashok.raj@intel.com>
> 
> This patch adds the support to inject SRAR and SRAO as LMCE, i.e. they
> are injected to only one VCPU rather than broadcast to all VCPUs. As KVM
> reports LMCE support on Intel platforms, this features is only available
> on Intel platforms.
> 
> LMCE is disabled by default and can be enabled/disabled by cpu option
> 'lmce=on/off'.
> 
> Signed-off-by: Ashok Raj <ashok.raj@intel.com>
> [Haozhong: Enable LMCE only on Intel platforms
>            Disable LMCE by default and add a cpu option 'lmce'
>            Handle the error if LMCE is enabled w/o host support
>            Remove MCG_LMCE_P from MCE_CAP_DEF
>            Add migration support for LMCE
>            Minor code style changes]
> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Applied to x86-next. Thanks.

-- 
Eduardo

[Qemu-devel] [PATCH v5 2/4] i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg

[Haozhong Zhang]

It's a prerequisite that certain bits of MSR_IA32_FEATURE_CONTROL should
be set before some features (e.g. VMX and LMCE) can be used, which is
usually done by the firmware. This patch adds a fw_cfg file
"etc/msr_feature_control" which contains the advised value of
MSR_IA32_FEATURE_CONTROL and can be used by guest firmware (e.g. SeaBIOS).

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 hw/i386/pc.c      | 29 +++++++++++++++++++++++++++++
 target-i386/cpu.h |  4 ++++
 2 files changed, 33 insertions(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 7198ed5..2a945fd 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1147,6 +1147,34 @@ void pc_cpus_init(PCMachineState *pcms)
     smbios_set_cpuid(cpu->env.cpuid_version, cpu->env.features[FEAT_1_EDX]);
 }
 
+static void pc_build_feature_control_file(PCMachineState *pcms)
+{
+    X86CPU *cpu = X86_CPU(pcms->possible_cpus->cpus[0].cpu);
+    CPUX86State *env = &cpu->env;
+    uint32_t unused, ecx, edx;
+    uint64_t feature_control_bits = 0;
+    uint64_t *val;
+
+    cpu_x86_cpuid(env, 1, 0, &unused, &unused, &ecx, &edx);
+    if (ecx & CPUID_EXT_VMX) {
+        feature_control_bits |= FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX;
+    }
+
+    if ((edx & (CPUID_EXT2_MCE | CPUID_EXT2_MCA)) ==
+        (CPUID_EXT2_MCE | CPUID_EXT2_MCA) &&
+        (env->mcg_cap & MCG_LMCE_P)) {
+        feature_control_bits |= FEATURE_CONTROL_LMCE;
+    }
+
+    if (!feature_control_bits) {
+        return;
+    }
+
+    val = g_malloc(sizeof(*val));
+    *val = feature_control_bits | FEATURE_CONTROL_LOCKED;
+    fw_cfg_add_file(pcms->fw_cfg, "etc/msr_feature_control", val, sizeof(*val));
+}
+
 static
 void pc_machine_done(Notifier *notifier, void *data)
 {
@@ -1174,6 +1202,7 @@ void pc_machine_done(Notifier *notifier, void *data)
     acpi_setup();
     if (pcms->fw_cfg) {
         pc_build_smbios(pcms->fw_cfg);
+        pc_build_feature_control_file(pcms);
     }
 }
 
diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index cda3ea1..11291b6 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -332,6 +332,10 @@
 #define MSR_TSC_ADJUST                  0x0000003b
 #define MSR_IA32_TSCDEADLINE            0x6e0
 
+#define FEATURE_CONTROL_LOCKED                    (1<<0)
+#define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX (1<<2)
+#define FEATURE_CONTROL_LMCE                      (1<<20)
+
 #define MSR_P6_PERFCTR0                 0xc1
 
 #define MSR_IA32_SMBASE                 0x9e
-- 
2.9.0

Re: [Qemu-devel] [PATCH v5 2/4] i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg

[Paolo Bonzini]

On 22/06/2016 08:56, Haozhong Zhang wrote:
> +
> +    val = g_malloc(sizeof(*val));
> +    *val = feature_control_bits | FEATURE_CONTROL_LOCKED;

As noticed by Laszlo, you need to use cpu_to_le64 here.  The maintainer
can fix it, I think.

Paolo

> +    fw_cfg_add_file(pcms->fw_cfg, "etc/msr_feature_control", val, sizeof(*val));
> +}

Re: [Qemu-devel] [PATCH v5 2/4] i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg

[Haozhong Zhang]

On 06/22/16 19:08, Paolo Bonzini wrote:
> 
> 
> On 22/06/2016 08:56, Haozhong Zhang wrote:
> > +
> > +    val = g_malloc(sizeof(*val));
> > +    *val = feature_control_bits | FEATURE_CONTROL_LOCKED;
> 
> As noticed by Laszlo, you need to use cpu_to_le64 here.  The maintainer
> can fix it, I think.
> 

Thanks for indicating this! I'll resend this one.

Haozhong

[Qemu-devel] [RESEND PATCH v5 2/4] i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg

[Haozhong Zhang]

It's a prerequisite that certain bits of MSR_IA32_FEATURE_CONTROL should
be set before some features (e.g. VMX and LMCE) can be used, which is
usually done by the firmware. This patch adds a fw_cfg file
"etc/msr_feature_control" which contains the advised value of
MSR_IA32_FEATURE_CONTROL and can be used by guest firmware (e.g. SeaBIOS).

Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
Changes in RESEND:
 * Use cpu_to_le64() to ensure file content is in little-endian.
---
 hw/i386/pc.c      | 29 +++++++++++++++++++++++++++++
 target-i386/cpu.h |  4 ++++
 2 files changed, 33 insertions(+)

diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 7198ed5..0c30549 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1147,6 +1147,34 @@ void pc_cpus_init(PCMachineState *pcms)
     smbios_set_cpuid(cpu->env.cpuid_version, cpu->env.features[FEAT_1_EDX]);
 }
 
+static void pc_build_feature_control_file(PCMachineState *pcms)
+{
+    X86CPU *cpu = X86_CPU(pcms->possible_cpus->cpus[0].cpu);
+    CPUX86State *env = &cpu->env;
+    uint32_t unused, ecx, edx;
+    uint64_t feature_control_bits = 0;
+    uint64_t *val;
+
+    cpu_x86_cpuid(env, 1, 0, &unused, &unused, &ecx, &edx);
+    if (ecx & CPUID_EXT_VMX) {
+        feature_control_bits |= FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX;
+    }
+
+    if ((edx & (CPUID_EXT2_MCE | CPUID_EXT2_MCA)) ==
+        (CPUID_EXT2_MCE | CPUID_EXT2_MCA) &&
+        (env->mcg_cap & MCG_LMCE_P)) {
+        feature_control_bits |= FEATURE_CONTROL_LMCE;
+    }
+
+    if (!feature_control_bits) {
+        return;
+    }
+
+    val = g_malloc(sizeof(*val));
+    *val = cpu_to_le64(feature_control_bits | FEATURE_CONTROL_LOCKED);
+    fw_cfg_add_file(pcms->fw_cfg, "etc/msr_feature_control", val, sizeof(*val));
+}
+
 static
 void pc_machine_done(Notifier *notifier, void *data)
 {
@@ -1174,6 +1202,7 @@ void pc_machine_done(Notifier *notifier, void *data)
     acpi_setup();
     if (pcms->fw_cfg) {
         pc_build_smbios(pcms->fw_cfg);
+        pc_build_feature_control_file(pcms);
     }
 }
 
diff --git a/target-i386/cpu.h b/target-i386/cpu.h
index cda3ea1..11291b6 100644
--- a/target-i386/cpu.h
+++ b/target-i386/cpu.h
@@ -332,6 +332,10 @@
 #define MSR_TSC_ADJUST                  0x0000003b
 #define MSR_IA32_TSCDEADLINE            0x6e0
 
+#define FEATURE_CONTROL_LOCKED                    (1<<0)
+#define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX (1<<2)
+#define FEATURE_CONTROL_LMCE                      (1<<20)
+
 #define MSR_P6_PERFCTR0                 0xc1
 
 #define MSR_IA32_SMBASE                 0x9e
-- 
2.9.0

Re: [Qemu-devel] [RESEND PATCH v5 2/4] i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg

[Eduardo Habkost]

On Thu, Jun 23, 2016 at 02:15:43PM +0800, Haozhong Zhang wrote:
> It's a prerequisite that certain bits of MSR_IA32_FEATURE_CONTROL should
> be set before some features (e.g. VMX and LMCE) can be used, which is
> usually done by the firmware. This patch adds a fw_cfg file
> "etc/msr_feature_control" which contains the advised value of
> MSR_IA32_FEATURE_CONTROL and can be used by guest firmware (e.g. SeaBIOS).
> 
> Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Applied to x86-next. Thanks.

-- 
Eduardo

[Qemu-devel] [PATCH v5 3/4] target-i386: enable LMCE for '-cpu host' if supported by host

[Haozhong Zhang]

If -cpu host is used, LMCE will be automatically enabled when it's
supported by host.

Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
---
 target-i386/cpu.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/target-i386/cpu.c b/target-i386/cpu.c
index cdab4f1..610554f 100644
--- a/target-i386/cpu.c
+++ b/target-i386/cpu.c
@@ -1493,6 +1493,17 @@ static uint32_t x86_cpu_get_supported_feature_word(FeatureWord w,
 
 #ifdef CONFIG_KVM
 
+static bool lmce_supported(void)
+{
+    uint64_t mce_cap;
+
+    if (kvm_ioctl(kvm_state, KVM_X86_GET_MCE_CAP_SUPPORTED, &mce_cap) < 0) {
+        return false;
+    }
+
+    return !!(mce_cap & MCG_LMCE_P);
+}
+
 static int cpu_x86_fill_model_id(char *str)
 {
     uint32_t eax = 0, ebx = 0, ecx = 0, edx = 0;
@@ -1565,6 +1576,10 @@ static void host_x86_cpu_initfn(Object *obj)
         env->cpuid_level = kvm_arch_get_supported_cpuid(s, 0x0, 0, R_EAX);
         env->cpuid_xlevel = kvm_arch_get_supported_cpuid(s, 0x80000000, 0, R_EAX);
         env->cpuid_xlevel2 = kvm_arch_get_supported_cpuid(s, 0xC0000000, 0, R_EAX);
+
+        if (lmce_supported()) {
+            object_property_set_bool(OBJECT(cpu), true, "lmce", &error_abort);
+        }
     }
 
     object_property_set_bool(OBJECT(cpu), true, "pmu", &error_abort);
-- 
2.9.0

Re: [Qemu-devel] [PATCH v5 3/4] target-i386: enable LMCE for '-cpu host' if supported by host

[Eduardo Habkost]

On Wed, Jun 22, 2016 at 02:56:23PM +0800, Haozhong Zhang wrote:
> If -cpu host is used, LMCE will be automatically enabled when it's
> supported by host.
> 
> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Applied to x86-next. Thanks.

-- 
Eduardo

[Qemu-devel] [PATCH v5 4/4] target-i386: abort migration if LMCE config mismatch

[Haozhong Zhang]

... to avoid guest errors due to LMCE configurations changes when
migrating from LMCE-enabled QEMU to LMCE-disabled QEMU.

Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
---
 target-i386/machine.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/target-i386/machine.c b/target-i386/machine.c
index 71c0e4d..00375a3 100644
--- a/target-i386/machine.c
+++ b/target-i386/machine.c
@@ -347,6 +347,12 @@ static int cpu_post_load(void *opaque, int version_id)
         return -EINVAL;
     }
 
+    if (!cpu->enable_lmce && (env->mcg_cap & MCG_LMCE_P)) {
+        error_report("Config mismatch: VCPU has LMCE enabled, "
+                     "but \"lmce\" option is disabled");
+        return -EINVAL;
+    }
+
     /*
      * Real mode guest segments register DPL should be zero.
      * Older KVM version were setting it wrongly.
-- 
2.9.0

Re: [Qemu-devel] [PATCH v5 4/4] target-i386: abort migration if LMCE config mismatch

[Eduardo Habkost]

On Wed, Jun 22, 2016 at 02:56:24PM +0800, Haozhong Zhang wrote:
> ... to avoid guest errors due to LMCE configurations changes when
> migrating from LMCE-enabled QEMU to LMCE-disabled QEMU.
> 
> Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>

Paolo doesn't like this change, so I won't apply it.

(Personally, I believe it is nice to have, but not very important.)

-- 
Eduardo

Re: [Qemu-devel] [PATCH v5 4/4] target-i386: abort migration if LMCE config mismatch

[Haozhong Zhang]

On 07/07/16 14:10, Eduardo Habkost wrote:
> On Wed, Jun 22, 2016 at 02:56:24PM +0800, Haozhong Zhang wrote:
> > ... to avoid guest errors due to LMCE configurations changes when
> > migrating from LMCE-enabled QEMU to LMCE-disabled QEMU.
> > 
> > Signed-off-by: Haozhong Zhang <haozhong.zhang@intel.com>
> 
> Paolo doesn't like this change, so I won't apply it.
> 
> (Personally, I believe it is nice to have, but not very important.)
> 

I'm okey with this. Thanks!

Haozhong

Re: [Qemu-devel] [PATCH v5 0/4] Add QEMU support for Intel local MCE

[Laszlo Ersek]

On 06/22/16 08:56, Haozhong Zhang wrote:
> Changes in v5:
>  * v4 Patch 1&2 are now merged into v5 Patch 1. (Paolo Bonzini)
>  * Add a separate Patch 3 to automatically enable LMCE for "-cpu host"
>    if host does support. (Eduardo Habkost)
>  * Move the sanity check of misconfiguration of LMCE to a separate
>    Patch 4. I'm fine to drop it if it's considered to be
>    unnecessary. (Eduardo Habkost)
>  * Move check of MCG_LMCE_P capability to the existing code in
>    kvm_arch_init_vcpu(). The difference is that if LMCE is not
>    supported by host but MCG_LMCE_P is included in env->mcg_cap,
>    kvm_arch_init_vcpu() return with error instead of just reporting a
>    warning. (Eduardo Habkost)
>  * Fix setting has_msr_feature_control in kvm_arch_init_vcpu(). (Paolo
>    Bonzini)
>  * Fix the type error in fw_cfg file "etc/msr_feature_control".
> 
> Changes in v4:
>  * Abort starting QEMU if lmce option is present but host does support
>    LMCE. (Eduardo Habkost)
>  * Remove setting MSR_IA32_FEATURE_CONTROL, which should be left to
>    guest. (Radim Krčmá, Paolo Bonzini
>  * Adjust error messages in mce_init(). (Boris Petkov)
>  * Move adding option 'lmce' to patch 1. (Eduardo Habkost, Paolo Bonzini)
>  * Adjust LMCE error message in cpu_post_load(). (Eduardo Habkost)
>  * (Patch 3) Add a fw_cfg file 'etc/msr_feature_control' to advise
>    bits should be set in MSR_IA32_FEATURE_CONTROL. (Paolo Bonzini)
>  * Fix SOB chain in patch 1.
> 
> Changes in v3:
>  * LMCE can be enabled only for non-intel guests.
>  * LMCE is disabled by default and a cpu option 'lmce=on/off' is added
>    to explicitly enable/disable LMCE.
>  * LMCE is disabled if KVM does not support (even though 'lmce=on').
>  * VM on LMCE-enabled QEMU can be only migrated to LMCE-enabled QEMU.
>  * MCG_LMCE_P is not included in MCE_CAP_DEF and instead added to
>    env->mcg_cap if LMCE is enabled.
>  * Code style fix.
> 
> This QEMU patch series along with the corresponding KVM patch series
> (sent via another email with title "[PATCH v3 0/3] Add KVM support for
> Intel local MCE") enables Intel local MCE feature for guest.
> 
> Intel Local MCE (LMCE) is a feature on Intel Skylake Server CPU that
> can deliver MCE to a single processor thread instead of broadcasting
> to all threads, which can reduce software's load when processing MCE
> on machines with a large number of processor threads.
> 
> The technical details of LMCE can be found in Intel SDM Vol 3, Chapter
> "Machine-Check Architecture" (search for 'LMCE'). Basically,
>  * The capability of LMCE is indicated by bit 27 (MCG_LMCE_P) of
>    MSR_IA32_MCG_CAP.
>  * LMCE is enabled by setting bit 20 (MSR_IA32_FEATURE_CONTROL_LMCE)
>    of MSR_IA32_FEATURE_CONTROL and bit 0 (MCG_EXT_CTL_LMCE_EN) of
>    MSR_IA32_MCG_EXT_CTL.
>  * Software can determine if a MCE is local to the current processor
>    thread by checking bit 2 (MCG_STATUS_LMCE) of MSR_IA32_MCG_STATUS.
> 
> Ashok Raj (1):
>   target-i386: KVM: add basic Intel LMCE support
> 
> Haozhong Zhang (3):
>   i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg
>   target-i386: enable LMCE for '-cpu host' if supported by host
>   target-i386: abort migration if LMCE config mismatch
> 
>  hw/i386/pc.c          | 29 +++++++++++++++++++++++++++++
>  target-i386/cpu.c     | 19 ++++++++++++++++++-
>  target-i386/cpu.h     | 16 ++++++++++++++++
>  target-i386/kvm.c     | 36 +++++++++++++++++++++++++++++++++---
>  target-i386/machine.c | 25 +++++++++++++++++++++++++
>  5 files changed, 121 insertions(+), 4 deletions(-)
> 

How can I test this series?

I'm trying to satisfy the conditions in pc_build_feature_control_file(), so that "etc/msr_feature_control" is created. However, I cannot even trigger

    if (ecx & CPUID_EXT_VMX) {
        feature_control_bits |= FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX;
    }

I tried various -cpu switches (with and without -enable-kvm); nothing seems to work. When I add "+vmx", QEMU always complains that vmx is not available. Any hints?

(My laptop does support VMX, and it has FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX set in the corresponding host MSR.)

Thanks
Laszlo

Re: [Qemu-devel] [PATCH v5 0/4] Add QEMU support for Intel local MCE

[Haozhong Zhang]

On 07/06/16 10:26, Laszlo Ersek wrote:
> On 06/22/16 08:56, Haozhong Zhang wrote:
> > Changes in v5:
> >  * v4 Patch 1&2 are now merged into v5 Patch 1. (Paolo Bonzini)
> >  * Add a separate Patch 3 to automatically enable LMCE for "-cpu host"
> >    if host does support. (Eduardo Habkost)
> >  * Move the sanity check of misconfiguration of LMCE to a separate
> >    Patch 4. I'm fine to drop it if it's considered to be
> >    unnecessary. (Eduardo Habkost)
> >  * Move check of MCG_LMCE_P capability to the existing code in
> >    kvm_arch_init_vcpu(). The difference is that if LMCE is not
> >    supported by host but MCG_LMCE_P is included in env->mcg_cap,
> >    kvm_arch_init_vcpu() return with error instead of just reporting a
> >    warning. (Eduardo Habkost)
> >  * Fix setting has_msr_feature_control in kvm_arch_init_vcpu(). (Paolo
> >    Bonzini)
> >  * Fix the type error in fw_cfg file "etc/msr_feature_control".
> > 
> > Changes in v4:
> >  * Abort starting QEMU if lmce option is present but host does support
> >    LMCE. (Eduardo Habkost)
> >  * Remove setting MSR_IA32_FEATURE_CONTROL, which should be left to
> >    guest. (Radim Krčmá, Paolo Bonzini
> >  * Adjust error messages in mce_init(). (Boris Petkov)
> >  * Move adding option 'lmce' to patch 1. (Eduardo Habkost, Paolo Bonzini)
> >  * Adjust LMCE error message in cpu_post_load(). (Eduardo Habkost)
> >  * (Patch 3) Add a fw_cfg file 'etc/msr_feature_control' to advise
> >    bits should be set in MSR_IA32_FEATURE_CONTROL. (Paolo Bonzini)
> >  * Fix SOB chain in patch 1.
> > 
> > Changes in v3:
> >  * LMCE can be enabled only for non-intel guests.
> >  * LMCE is disabled by default and a cpu option 'lmce=on/off' is added
> >    to explicitly enable/disable LMCE.
> >  * LMCE is disabled if KVM does not support (even though 'lmce=on').
> >  * VM on LMCE-enabled QEMU can be only migrated to LMCE-enabled QEMU.
> >  * MCG_LMCE_P is not included in MCE_CAP_DEF and instead added to
> >    env->mcg_cap if LMCE is enabled.
> >  * Code style fix.
> > 
> > This QEMU patch series along with the corresponding KVM patch series
> > (sent via another email with title "[PATCH v3 0/3] Add KVM support for
> > Intel local MCE") enables Intel local MCE feature for guest.
> > 
> > Intel Local MCE (LMCE) is a feature on Intel Skylake Server CPU that
> > can deliver MCE to a single processor thread instead of broadcasting
> > to all threads, which can reduce software's load when processing MCE
> > on machines with a large number of processor threads.
> > 
> > The technical details of LMCE can be found in Intel SDM Vol 3, Chapter
> > "Machine-Check Architecture" (search for 'LMCE'). Basically,
> >  * The capability of LMCE is indicated by bit 27 (MCG_LMCE_P) of
> >    MSR_IA32_MCG_CAP.
> >  * LMCE is enabled by setting bit 20 (MSR_IA32_FEATURE_CONTROL_LMCE)
> >    of MSR_IA32_FEATURE_CONTROL and bit 0 (MCG_EXT_CTL_LMCE_EN) of
> >    MSR_IA32_MCG_EXT_CTL.
> >  * Software can determine if a MCE is local to the current processor
> >    thread by checking bit 2 (MCG_STATUS_LMCE) of MSR_IA32_MCG_STATUS.
> > 
> > Ashok Raj (1):
> >   target-i386: KVM: add basic Intel LMCE support
> > 
> > Haozhong Zhang (3):
> >   i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg
> >   target-i386: enable LMCE for '-cpu host' if supported by host
> >   target-i386: abort migration if LMCE config mismatch
> > 
> >  hw/i386/pc.c          | 29 +++++++++++++++++++++++++++++
> >  target-i386/cpu.c     | 19 ++++++++++++++++++-
> >  target-i386/cpu.h     | 16 ++++++++++++++++
> >  target-i386/kvm.c     | 36 +++++++++++++++++++++++++++++++++---
> >  target-i386/machine.c | 25 +++++++++++++++++++++++++
> >  5 files changed, 121 insertions(+), 4 deletions(-)
> > 
> 
> How can I test this series?
> 
> I'm trying to satisfy the conditions in pc_build_feature_control_file(), so that "etc/msr_feature_control" is created. However, I cannot even trigger
> 
>     if (ecx & CPUID_EXT_VMX) {
>         feature_control_bits |= FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX;
>     }
> 
> I tried various -cpu switches (with and without -enable-kvm); nothing seems to work. When I add "+vmx", QEMU always complains that vmx is not available. Any hints?
> 
> (My laptop does support VMX, and it has FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX set in the corresponding host MSR.)
> 

You may check if kvm-intel module is loaded with nested support by
'cat /sys/module/kvm_intel/parameters/nested'. If it's 0, you need to
reload kvm-intel with nested parameter: modprobe kvm-intel
nested=1. Moreover, you may also try qemu -cpu host to get the most
your host hardware supports.

Thanks,
Haozhong

Re: [Qemu-devel] [PATCH v5 0/4] Add QEMU support for Intel local MCE

[Laszlo Ersek]

On 07/06/16 10:35, Haozhong Zhang wrote:
> On 07/06/16 10:26, Laszlo Ersek wrote:
>> On 06/22/16 08:56, Haozhong Zhang wrote:
>>> Changes in v5:
>>>  * v4 Patch 1&2 are now merged into v5 Patch 1. (Paolo Bonzini)
>>>  * Add a separate Patch 3 to automatically enable LMCE for "-cpu host"
>>>    if host does support. (Eduardo Habkost)
>>>  * Move the sanity check of misconfiguration of LMCE to a separate
>>>    Patch 4. I'm fine to drop it if it's considered to be
>>>    unnecessary. (Eduardo Habkost)
>>>  * Move check of MCG_LMCE_P capability to the existing code in
>>>    kvm_arch_init_vcpu(). The difference is that if LMCE is not
>>>    supported by host but MCG_LMCE_P is included in env->mcg_cap,
>>>    kvm_arch_init_vcpu() return with error instead of just reporting a
>>>    warning. (Eduardo Habkost)
>>>  * Fix setting has_msr_feature_control in kvm_arch_init_vcpu(). (Paolo
>>>    Bonzini)
>>>  * Fix the type error in fw_cfg file "etc/msr_feature_control".
>>>
>>> Changes in v4:
>>>  * Abort starting QEMU if lmce option is present but host does support
>>>    LMCE. (Eduardo Habkost)
>>>  * Remove setting MSR_IA32_FEATURE_CONTROL, which should be left to
>>>    guest. (Radim Krčmá, Paolo Bonzini
>>>  * Adjust error messages in mce_init(). (Boris Petkov)
>>>  * Move adding option 'lmce' to patch 1. (Eduardo Habkost, Paolo Bonzini)
>>>  * Adjust LMCE error message in cpu_post_load(). (Eduardo Habkost)
>>>  * (Patch 3) Add a fw_cfg file 'etc/msr_feature_control' to advise
>>>    bits should be set in MSR_IA32_FEATURE_CONTROL. (Paolo Bonzini)
>>>  * Fix SOB chain in patch 1.
>>>
>>> Changes in v3:
>>>  * LMCE can be enabled only for non-intel guests.
>>>  * LMCE is disabled by default and a cpu option 'lmce=on/off' is added
>>>    to explicitly enable/disable LMCE.
>>>  * LMCE is disabled if KVM does not support (even though 'lmce=on').
>>>  * VM on LMCE-enabled QEMU can be only migrated to LMCE-enabled QEMU.
>>>  * MCG_LMCE_P is not included in MCE_CAP_DEF and instead added to
>>>    env->mcg_cap if LMCE is enabled.
>>>  * Code style fix.
>>>
>>> This QEMU patch series along with the corresponding KVM patch series
>>> (sent via another email with title "[PATCH v3 0/3] Add KVM support for
>>> Intel local MCE") enables Intel local MCE feature for guest.
>>>
>>> Intel Local MCE (LMCE) is a feature on Intel Skylake Server CPU that
>>> can deliver MCE to a single processor thread instead of broadcasting
>>> to all threads, which can reduce software's load when processing MCE
>>> on machines with a large number of processor threads.
>>>
>>> The technical details of LMCE can be found in Intel SDM Vol 3, Chapter
>>> "Machine-Check Architecture" (search for 'LMCE'). Basically,
>>>  * The capability of LMCE is indicated by bit 27 (MCG_LMCE_P) of
>>>    MSR_IA32_MCG_CAP.
>>>  * LMCE is enabled by setting bit 20 (MSR_IA32_FEATURE_CONTROL_LMCE)
>>>    of MSR_IA32_FEATURE_CONTROL and bit 0 (MCG_EXT_CTL_LMCE_EN) of
>>>    MSR_IA32_MCG_EXT_CTL.
>>>  * Software can determine if a MCE is local to the current processor
>>>    thread by checking bit 2 (MCG_STATUS_LMCE) of MSR_IA32_MCG_STATUS.
>>>
>>> Ashok Raj (1):
>>>   target-i386: KVM: add basic Intel LMCE support
>>>
>>> Haozhong Zhang (3):
>>>   i386: publish advised value of MSR_IA32_FEATURE_CONTROL via fw_cfg
>>>   target-i386: enable LMCE for '-cpu host' if supported by host
>>>   target-i386: abort migration if LMCE config mismatch
>>>
>>>  hw/i386/pc.c          | 29 +++++++++++++++++++++++++++++
>>>  target-i386/cpu.c     | 19 ++++++++++++++++++-
>>>  target-i386/cpu.h     | 16 ++++++++++++++++
>>>  target-i386/kvm.c     | 36 +++++++++++++++++++++++++++++++++---
>>>  target-i386/machine.c | 25 +++++++++++++++++++++++++
>>>  5 files changed, 121 insertions(+), 4 deletions(-)
>>>
>>
>> How can I test this series?
>>
>> I'm trying to satisfy the conditions in pc_build_feature_control_file(), so that "etc/msr_feature_control" is created. However, I cannot even trigger
>>
>>     if (ecx & CPUID_EXT_VMX) {
>>         feature_control_bits |= FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX;
>>     }
>>
>> I tried various -cpu switches (with and without -enable-kvm); nothing seems to work. When I add "+vmx", QEMU always complains that vmx is not available. Any hints?
>>
>> (My laptop does support VMX, and it has FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX set in the corresponding host MSR.)
>>
> 
> You may check if kvm-intel module is loaded with nested support by
> 'cat /sys/module/kvm_intel/parameters/nested'. If it's 0, you need to
> reload kvm-intel with nested parameter: modprobe kvm-intel
> nested=1.

Ah, that's what is missing. I'll look into it, thanks!

> Moreover, you may also try qemu -cpu host to get the most
> your host hardware supports.

Yep, I tried that, but it wasn't enough.

Thanks!
Laszlo