[PATCH v2 00/18] whpx: i386: updates for QEMU 11.2

[PATCH v2 00/18] whpx: i386: updates for QEMU 11.2

[Mohamed Mediouni]

v2:

Interrupt controller rework to use the MSHV implementation.
A lot more register save/restore.
Restoring support for older Windows 10 releases.
CPUID fixes.

v1:
Experimental support for save/restore, a lot more robustness on
reboots are the highlights of this one.

Akshey Taneja (2):
  whpx: i386: do not require ProcessorFeaturesBanks on legacy Windows
  whpx: i386: fall back to xAPIC when the MSR exit bitmap is unavailable

Boden Randin (1):
  whpx: i386: avoid updating hflags for vmexits

Doug Cook (1):
  target/i386/whpx: fix VMX capability MSR values

Doug Cook (WINDOWS) (2):
  target/i386/whpx: whpx_get_xsave_state error handling
  target/i386/whpx: handle processor reset

Kostas Chatzikokolakis (1):
  whpx: i386: forward the dynamic EBX of CPUID[0xD] sub-leaves 0 and 1,
    not 1 and 2

Mohamed Mediouni (10):
  whpx: i386: get APIC state before VM save
  whpx: i386: fix TPR encodings for local APIC
  whpx: i386: save/restore xss
  whpx: i386: make nested virtualisation init failure non-fatal
  whpx: i386: allow migration for restricted scenarios
  target/arm/whpx: handle processor reset
  whpx: apic: migrate APIC implementation to new interface
  whpx: i386: also save debug registers
  whpx: i386: save/restore pending interruption/events
  whpx: docs: update for experimental VM save/restore support

Pierrick Bouvier (1):
  multi-target: restrict whpx-accel-*-cpu to matching target

 accel/whpx/whpx-common.c       |   8 +
 docs/system/whpx.rst           |  12 +
 include/system/whpx-internal.h |   6 +-
 target/arm/whpx/whpx-all.c     |  10 +-
 target/i386/whpx/whpx-all.c    | 436 ++++++++++++++++++++++++++++-----
 target/i386/whpx/whpx-apic.c   | 307 ++++++++++++++++++++++-
 6 files changed, 699 insertions(+), 80 deletions(-)

-- 
2.54.0 (Apple Git-157)

[PATCH v2 01/18] target/i386/whpx: fix VMX capability MSR values

[Mohamed Mediouni]

From: Doug Cook <dcook@microsoft.com>

whpx_get_supported_msr_feature() had two bugs in how it reports the
VMX control MSRs:

- MSR_IA32_VMX_TRUE_PINBASED_CTLS and MSR_IA32_VMX_TRUE_PROCBASED_CTLS
  were queried using the non-TRUE capability codes
  (WHvCapabilityCodeVmxPinbasedCtls and WHvCapabilityCodeVmxProcbasedCtls).

- For PROCBASED_CTLS2 and the four TRUE_* control MSRs, the raw
  allowed-0/allowed-1 pair was returned. The feature-word code expects
  only the bits that may be 1 but are not required to be 1. Apply the
  same transformation as kvm_arch_get_supported_msr_feature();
  make_vmx_msr_value() adds the must-be-one bits back afterwards.

As a result, with +vmx QEMU treated many VMX features the host does
support (secondary controls, EPT, VPID, MSR bitmaps, ...) as missing,
warned about each of them, and dropped them from the CPU model. The
guest itself reads the VMX MSRs from the hypervisor, not from QEMU,
so this did not change what the guest saw.

Signed-off-by: Doug Cook <dcook@microsoft.com>
Reviewed-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 113f79afb9..1386affa9c 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -1452,10 +1452,10 @@ uint64_t whpx_get_supported_msr_feature(uint32_t index)
         cap = WHvCapabilityCodeVmxEptVpidCap;
         break;
     case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
-        cap = WHvCapabilityCodeVmxPinbasedCtls;
+        cap = WHvCapabilityCodeVmxTruePinbasedCtls;
         break;
     case MSR_IA32_VMX_TRUE_PROCBASED_CTLS:
-        cap = WHvCapabilityCodeVmxProcbasedCtls;
+        cap = WHvCapabilityCodeVmxTrueProcbasedCtls;
         break;
     case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
         cap = WHvCapabilityCodeVmxTrueEntryCtls;
@@ -1474,6 +1474,31 @@ uint64_t whpx_get_supported_msr_feature(uint32_t index)
         if (FAILED(hr)) {
             return 0;
         }
+
+        switch (index) {
+        case MSR_IA32_VMX_PROCBASED_CTLS2:
+        case MSR_IA32_VMX_TRUE_PINBASED_CTLS:
+        case MSR_IA32_VMX_TRUE_PROCBASED_CTLS:
+        case MSR_IA32_VMX_TRUE_ENTRY_CTLS:
+        case MSR_IA32_VMX_TRUE_EXIT_CTLS: {
+            /*
+             * These MSRs hold the allowed-0 settings in the low half and the
+             * allowed-1 settings in the high half.  QEMU's feature words want
+             * only the bits that can be one but do not have to be one, so
+             * apply the same transformation as
+             * kvm_arch_get_supported_msr_feature(); make_vmx_msr_value()
+             * restores the must-be-one bits later.
+             */
+            uint32_t must_be_one = (uint32_t)val;
+            uint32_t can_be_one = (uint32_t)(val >> 32);
+
+            val = can_be_one & ~must_be_one;
+            break;
+        }
+        default:
+            break;
+        }
+
         return val;
     }
     return 0;
-- 
2.54.0 (Apple Git-157)

[PATCH v2 02/18] target/i386/whpx: whpx_get_xsave_state error handling

[Mohamed Mediouni]

From: "Doug Cook (WINDOWS)" <dcook@microsoft.com>

Several problems with error handling in whpx_get_xsave_state:

- Does not handle WHV_E_INSUFFICIENT_BUFFER, which occurs frequently
  in practice for XSAVE state.
- Does not free xsavec_buf when an error occurs.
- Error message and return code use errno where they should use hr.

Fixes: cfaa3b6c9597 ("whpx: xsave support")

Signed-off-by: Doug Cook <dcook@microsoft.com>
Reviewed-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 include/system/whpx-internal.h |  5 ++-
 target/i386/whpx/whpx-all.c    | 73 +++++++++++++++++++++++++---------
 2 files changed, 57 insertions(+), 21 deletions(-)

diff --git a/include/system/whpx-internal.h b/include/system/whpx-internal.h
index c295c5a529..b15f3d8faf 100644
--- a/include/system/whpx-internal.h
+++ b/include/system/whpx-internal.h
@@ -58,8 +58,9 @@ void whpx_apic_get(APICCommonState *s);
 
 #define WHV_E_UNKNOWN_CAPABILITY 0x80370300L
 
-/* This should eventually come from the Windows SDK */
-#define WHV_E_UNKNOWN_PROPERTY 0x80370302
+/* These should eventually come from the Windows SDK */
+#define WHV_E_INSUFFICIENT_BUFFER 0x80370301L
+#define WHV_E_UNKNOWN_PROPERTY 0x80370302L
 
 #define LIST_WINHVPLATFORM_FUNCTIONS(X) \
   X(HRESULT, WHvGetCapability, (WHV_CAPABILITY_CODE CapabilityCode, VOID* CapabilityBuffer, UINT32 CapabilityBufferSizeInBytes, UINT32* WrittenSizeInBytes)) \
diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 1386affa9c..68272c5657 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -442,8 +442,8 @@ static int whpx_set_xsave_state(const CPUState *cpu)
 
     qemu_vfree(xsavec_buf);
     if (FAILED(hr)) {
-        error_report("WHPX: Failed to get virtual processor context, hr=%08lx",
-                     hr);
+        error_report("WHPX: Failed to set xsave state, hr=%08lx", hr);
+        return -EIO;
     }
 
     return 0;
@@ -809,9 +809,28 @@ static void whpx_get_legacy_fp_registers(CPUState *cpu, WHPXStateLevel level)
     idx += 1;
 }
 
-static int whpx_get_xsave_state(CPUState *cpu)
+static HRESULT whpx_get_xsave_state_buffer(const CPUState *cpu, void *buf,
+                                           size_t buf_len,
+                                           UINT32 *bytes_written)
 {
     struct whpx_state *whpx = &whpx_global;
+
+    if (!whpx_is_legacy_os()) {
+        return whp_dispatch.WHvGetVirtualProcessorState(
+            whpx->partition, cpu->cpu_index,
+            WHvVirtualProcessorStateTypeXsaveState,
+            buf,
+            buf_len, bytes_written);
+    } else {
+        return whp_dispatch.WHvGetVirtualProcessorXsaveState(
+            whpx->partition, cpu->cpu_index,
+            buf,
+            buf_len, bytes_written);
+    }
+}
+
+static int whpx_get_xsave_state(CPUState *cpu)
+{
     X86CPU *x86cpu = X86_CPU(cpu);
     CPUX86State *env = &x86cpu->env;
     int ret;
@@ -824,27 +843,43 @@ static int whpx_get_xsave_state(CPUState *cpu)
     xsavec_buf = qemu_memalign(page, xsavec_buf_len);
     memset(xsavec_buf, 0, xsavec_buf_len);
 
-    if (!whpx_is_legacy_os()) {
-        hr = whp_dispatch.WHvGetVirtualProcessorState(
-            whpx->partition, cpu->cpu_index,
-            WHvVirtualProcessorStateTypeXsaveState,
-            xsavec_buf,
-            xsavec_buf_len, &bytes_written);
-    } else {
-        hr = whp_dispatch.WHvGetVirtualProcessorXsaveState(
-            whpx->partition, cpu->cpu_index,
-            xsavec_buf,
-            xsavec_buf_len, &bytes_written);
+    bytes_written = 0;
+    hr = whpx_get_xsave_state_buffer(cpu, xsavec_buf, xsavec_buf_len,
+                                     &bytes_written);
+
+    /*
+     * whpx_get_xsave_max_len() returns the size of user state components
+     * enabled in XCR0. The hypervisor returns an XSAVES image, which also
+     * contains the supervisor state, so the whpx_get_xsave_max_len() size
+     * may be too small.
+     */
+    if (hr == WHV_E_INSUFFICIENT_BUFFER && bytes_written > xsavec_buf_len) {
+        qemu_vfree(xsavec_buf);
+        xsavec_buf_len = bytes_written;
+        xsavec_buf = qemu_memalign(page, xsavec_buf_len);
+        memset(xsavec_buf, 0, xsavec_buf_len);
+
+        bytes_written = 0;
+        hr = whpx_get_xsave_state_buffer(cpu, xsavec_buf, xsavec_buf_len,
+                                         &bytes_written);
     }
-    if (FAILED(hr) || bytes_written == 0) {
-        error_report("failed to get xsave state: %s", strerror(errno));
-        return -errno;
+
+    if (FAILED(hr)) {
+        error_report("WHPX: Failed to get xsave state, hr=%08lx", hr);
+        qemu_vfree(xsavec_buf);
+        return -EIO;
+    }
+
+    if (bytes_written == 0) {
+        error_report("WHPX: Failed to get xsave state, no data returned");
+        qemu_vfree(xsavec_buf);
+        return -EIO;
     }
 
-    ret = decompact_xsave_area(xsavec_buf, xsavec_buf_len, env);
+    ret = decompact_xsave_area(xsavec_buf, bytes_written, env);
     qemu_vfree(xsavec_buf);
     if (ret < 0) {
-        error_report("failed to decompact xsave area");
+        error_report("WHPX: Failed to decompact xsave area");
         return ret;
     }
     x86_cpu_xrstor_all_areas(x86cpu, env->xsave_buf, env->xsave_buf_len);
-- 
2.54.0 (Apple Git-157)

[PATCH v2 03/18] whpx: i386: get APIC state before VM save

[Mohamed Mediouni]

Prerequisite for VM save/restore.

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-apic.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/target/i386/whpx/whpx-apic.c b/target/i386/whpx/whpx-apic.c
index cc272f82a5..efdedbcd77 100644
--- a/target/i386/whpx/whpx-apic.c
+++ b/target/i386/whpx/whpx-apic.c
@@ -242,6 +242,11 @@ void whpx_apic_get(APICCommonState *s)
     whpx_get_apic_state(s, &kapic);
 }
 
+static void whpx_apic_pre_save(APICCommonState *s)
+{
+    whpx_apic_get(s);
+}
+
 static void whpx_apic_post_load(APICCommonState *s)
 {
     run_on_cpu(CPU(s->cpu), whpx_apic_put, RUN_ON_CPU_HOST_PTR(s));
@@ -338,6 +343,7 @@ static void whpx_apic_class_init(ObjectClass *klass, const void *data)
     k->set_base = apic_set_base;
     k->set_tpr = whpx_apic_set_tpr;
     k->get_tpr = whpx_apic_get_tpr;
+    k->pre_save = whpx_apic_pre_save;
     k->post_load = whpx_apic_post_load;
     k->vapic_base_update = whpx_apic_vapic_base_update;
     k->external_nmi = whpx_apic_external_nmi;
-- 
2.54.0 (Apple Git-157)

[PATCH v2 04/18] whpx: i386: fix TPR encodings for local APIC

[Mohamed Mediouni]

Convert between the TPR and the APIC priority encodings.

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-apic.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/target/i386/whpx/whpx-apic.c b/target/i386/whpx/whpx-apic.c
index efdedbcd77..2d83291fae 100644
--- a/target/i386/whpx/whpx-apic.c
+++ b/target/i386/whpx/whpx-apic.c
@@ -184,12 +184,12 @@ static void whpx_put_apic_base(CPUState *cpu, uint64_t val)
 
 static void whpx_apic_set_tpr(APICCommonState *s, uint8_t val)
 {
-    s->tpr = val;
+    s->tpr = (val & APIC_PR_SUB_CLASS) << APIC_PR_CLASS_SHIFT;
 }
 
 static uint8_t whpx_apic_get_tpr(APICCommonState *s)
 {
-    return s->tpr;
+    return s->tpr >> APIC_PR_CLASS_SHIFT;
 }
 
 static void whpx_apic_vapic_base_update(APICCommonState *s)
-- 
2.54.0 (Apple Git-157)

[PATCH v2 05/18] whpx: i386: save/restore xss

[Mohamed Mediouni]

More work on VM save/restore

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 68272c5657..408a1667e0 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -346,6 +346,8 @@ static void whpx_set_xcrs(CPUState *cpu)
     struct whpx_state *whpx = &whpx_global;
     WHV_REGISTER_VALUE xcr0;
     WHV_REGISTER_NAME xcr0_name = WHvX64RegisterXCr0;
+    WHV_REGISTER_VALUE xss;
+    WHV_REGISTER_NAME xss_name = WHvX64RegisterXss;
 
     if (!whpx_has_xsave()) {
         return;
@@ -358,6 +360,17 @@ static void whpx_set_xcrs(CPUState *cpu)
     if (FAILED(hr)) {
         error_report("WHPX: Failed to set register xcr0, hr=%08lx", hr);
     }
+
+    if (!whpx_has_xsaves()) {
+        return;
+    }
+
+    xss.Reg64 = cpu_env(cpu)->xss;
+    hr = whp_dispatch.WHvSetVirtualProcessorRegisters(
+        whpx->partition, cpu->cpu_index, &xss_name, 1, &xss);
+    if (FAILED(hr)) {
+        error_report("WHPX: Failed to set register xss, hr=%08lx", hr);
+    }
 }
 
 static int whpx_set_tsc(CPUState *cpu)
@@ -691,6 +704,8 @@ static void whpx_get_xcrs(CPUState *cpu)
     struct whpx_state *whpx = &whpx_global;
     WHV_REGISTER_VALUE xcr0;
     WHV_REGISTER_NAME xcr0_name = WHvX64RegisterXCr0;
+    WHV_REGISTER_VALUE xss;
+    WHV_REGISTER_NAME xss_name = WHvX64RegisterXss;
 
     if (!whpx_has_xsave()) {
         return;
@@ -705,6 +720,19 @@ static void whpx_get_xcrs(CPUState *cpu)
     }
 
     cpu_env(cpu)->xcr0 = xcr0.Reg64;
+
+    if (!whpx_has_xsaves()) {
+        return;
+    }
+
+    hr = whp_dispatch.WHvGetVirtualProcessorRegisters(
+        whpx->partition, cpu->cpu_index, &xss_name, 1, &xss);
+    if (FAILED(hr)) {
+        error_report("WHPX: Failed to get register xss, hr=%08lx", hr);
+        return;
+    }
+
+    cpu_env(cpu)->xss = xss.Reg64;
 }
 
 static void whpx_get_registers_for_vmexit(CPUState *cpu, WHPXStateLevel level)
-- 
2.54.0 (Apple Git-157)

[PATCH v2 06/18] whpx: i386: make nested virtualisation init failure non-fatal

[Mohamed Mediouni]

On some CPUs too old for Windows 11 (Skylake) support but running
it anyway, WHP advertises nested virt despite it
being unavailable.

As such, make failures to initialise nested virt even when advertised
non-fatal.

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 408a1667e0..d13513985c 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -3302,9 +3302,7 @@ int whpx_accel_init(AccelState *as, MachineState *ms)
             &prop,
             sizeof(WHV_PARTITION_PROPERTY));
             if (FAILED(hr)) {
-                error_report("WHPX: Failed to enable nested virtualization, hr=%08lx", hr);
-                ret = -EINVAL;
-                goto error;
+                warn_report("WHPX: Failed to enable nested virtualization, hr=%08lx", hr);
         }
     }
 
-- 
2.54.0 (Apple Git-157)

[PATCH v2 07/18] whpx: i386: allow migration for restricted scenarios

[Mohamed Mediouni]

Currently, nested virtualisation state is not saved/restored and
neither is Hyper-V enlightenment state.

Also, not debug register state nor MTRRs but still usable.

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index d13513985c..5f14564f10 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -2828,6 +2828,22 @@ static void whpx_cpu_update_state(void *opaque, bool running, RunState state)
     }
 }
 
+static bool whpx_is_migration_allowed(CPUState *cpu) {
+    struct whpx_state *whpx = &whpx_global;
+    X86CPU *x86_cpu = X86_CPU(cpu);
+    CPUX86State *env = &x86_cpu->env;
+
+    if (whpx->hyperv_enlightenments_enabled == true) {
+        return false;
+    }
+
+    if (cpu_has_vmx(env) || cpu_has_svm(env)) {
+        return false;
+    }
+
+    return true;
+}
+
 int whpx_init_vcpu(CPUState *cpu)
 {
     HRESULT hr;
@@ -2845,10 +2861,11 @@ int whpx_init_vcpu(CPUState *cpu)
     /* Add migration blockers for all unsupported features of the
      * Windows Hypervisor Platform
      */
-    if (whpx_migration_blocker == NULL) {
+    if (whpx_migration_blocker == NULL
+        && !whpx_is_migration_allowed(cpu)) {
         error_setg(&whpx_migration_blocker,
-               "State blocked due to missing dirty memory tracking support,"
-               "And some system register/state save-restore ");
+               "State blocked due to missing Hyper-V enlightenments,"
+               "save-restore and/or nested virtualisation");
 
         if (migrate_add_blocker(&whpx_migration_blocker, &local_error) < 0) {
             error_report_err(local_error);
-- 
2.54.0 (Apple Git-157)

[PATCH v2 08/18] whpx: i386: avoid updating hflags for vmexits

[Mohamed Mediouni]

From: Boden Randin <bodenrandin1@gmail.com>

whpx_get_registers_for_vmexit only refreshes GPRs, RIP and RFLAGS.

x86_update_hflags() depends on state not refreshed here, while the code
using the fast runtime state does not depend on the hflags it recomputes.

Signed-off-by: Boden Randin <bodenrandin1@gmail.com>
Reviewed-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 2 --
 1 file changed, 2 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 5f14564f10..29a630b03c 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -772,8 +772,6 @@ static void whpx_get_registers_for_vmexit(CPUState *cpu, WHPXStateLevel level)
     rflags_to_lflags(env);
 
     assert(idx == RTL_NUMBER_OF(whpx_register_names_for_vmexit));
-
-    x86_update_hflags(env);
 }
 
 static void whpx_get_legacy_fp_registers(CPUState *cpu, WHPXStateLevel level)
-- 
2.54.0 (Apple Git-157)

[PATCH v2 09/18] target/i386/whpx: handle processor reset

[Mohamed Mediouni]

From: "Doug Cook (WINDOWS)" <dcook@microsoft.com>

WHPX is not properly handling processor reset:

- Does not reset partition, so several registers (e.g. Hypercall,
  GuestOsId) carry values forward as "Locked" into the next session.
- Does not clear hypervisor-managed TSC deadline (live-lock).
- Does not re-arm the wait-for-SIPI trigger, so all CPUs start running
  immediately after reset.

Some firmware may tolerate this, but OVMF does not. Typical result is
that boot hangs or ends in triple-fault, e.g.

  qemu-system-x86_64: WHPX: Unexpected VP exit code 4

After this patch, I am able to boot to Windows desktop, reset, and boot
back to Windows desktop. Tried many times in sequence with no problems.
Used several different reset mechanisms (OS shutdown menu, external
"reset", watchdog reset) with no problems.

Resolves: https://gitlab.com/qemu-project/qemu/-/work_items/2402
Signed-off-by: Doug Cook <dcook@microsoft.com>
Reviewed-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 29 +++++++++++++++++++++++++++++
 1 file changed, 29 insertions(+)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 29a630b03c..6b00724ceb 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -541,6 +541,35 @@ void whpx_set_registers(CPUState *cpu, WHPXStateLevel level)
 
     assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
 
+    /* Assumes that cpu_synchronize_all_post_reset() runs cpu 0 first. */
+    if (level == WHPX_LEVEL_RESET_STATE && cpu->cpu_index == 0) {
+        hr = whp_dispatch.WHvResetPartition(whpx->partition);
+        if (FAILED(hr)) {
+            error_report("WHPX: Failed to reset partition, hr=%08lx", hr);
+        }
+    }
+
+    /* Reset hypervisor-owned TSC deadline. */
+    if (level == WHPX_LEVEL_RESET_STATE) {
+        WHV_REGISTER_VALUE tsc_deadline = {};
+
+        whpx_set_reg(cpu, WHvX64RegisterTscDeadline, tsc_deadline);
+    }
+
+    /*
+     * Re-arm partition's wait-for-SIPI state.
+     * whpx_vcpu_run() ignores cpu->halted, depends on SIPI for this.
+     * This must follow WHvResetPartition.
+     */
+    if (level == WHPX_LEVEL_RESET_STATE && whpx_irqchip_in_kernel()) {
+        WHV_REGISTER_VALUE activity = {};
+
+        whpx_get_reg(cpu, WHvRegisterInternalActivityState, &activity);
+        activity.InternalActivity.StartupSuspend = !cpu_is_bsp(x86_cpu);
+        activity.InternalActivity.HaltSuspend = 0;
+        whpx_set_reg(cpu, WHvRegisterInternalActivityState, activity);
+    }
+
     /*
      * Following MSRs have side effects on the guest or are too heavy for
      * runtime. Limit them to full state update.
-- 
2.54.0 (Apple Git-157)

[PATCH v2 10/18] target/arm/whpx: handle processor reset

[Mohamed Mediouni]

Move WHvResetPartition to where it's effective on multi-core configurations.

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/arm/whpx/whpx-all.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/target/arm/whpx/whpx-all.c b/target/arm/whpx/whpx-all.c
index 46f0b9524e..150a6326d7 100644
--- a/target/arm/whpx/whpx-all.c
+++ b/target/arm/whpx/whpx-all.c
@@ -471,8 +471,6 @@ int whpx_vcpu_run(CPUState *cpu)
             if (arm_cpu->power_state != PSCI_OFF) {
                 whpx_psci_cpu_off(arm_cpu);
             }
-            /* Partition-wide reset, to reset state for reboots to succeed. */
-            whp_dispatch.WHvResetPartition(whpx->partition);
             bql_unlock();
             break;
         case WHvRunVpExitReasonNone:
@@ -573,6 +571,14 @@ void whpx_set_registers(CPUState *cpu, WHPXStateLevel level)
 
     assert(cpu_is_stopped(cpu) || qemu_cpu_is_self(cpu));
 
+    /* Assumes that cpu_synchronize_all_post_reset() runs cpu 0 first. */
+    if (level == WHPX_LEVEL_RESET_STATE && cpu->cpu_index == 0) {
+        hr = whp_dispatch.WHvResetPartition(whpx->partition);
+        if (FAILED(hr)) {
+            error_report("WHPX: Failed to reset partition, hr=%08lx", hr);
+        }
+    }
+
     for (i = 0; i < ARRAY_SIZE(whpx_reg_match); i++) {
         val.Reg64 = *(uint64_t *)((char *)env + whpx_reg_match[i].offset);
         whpx_set_reg(cpu, whpx_reg_match[i].reg, val);
-- 
2.54.0 (Apple Git-157)

[PATCH v2 11/18] whpx: apic: migrate APIC implementation to new interface

[Mohamed Mediouni]

Using the translation logic from the MSHV backend.

The new API is only available on Windows 10 20H2 onwards.
Retain the older interface for backwards compatibility.

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 include/system/whpx-internal.h |   1 +
 target/i386/whpx/whpx-all.c    |   2 +
 target/i386/whpx/whpx-apic.c   | 297 ++++++++++++++++++++++++++++++++-
 3 files changed, 292 insertions(+), 8 deletions(-)

diff --git a/include/system/whpx-internal.h b/include/system/whpx-internal.h
index b15f3d8faf..136ad88872 100644
--- a/include/system/whpx-internal.h
+++ b/include/system/whpx-internal.h
@@ -55,6 +55,7 @@ struct whpx_state {
 
 extern struct whpx_state whpx_global;
 void whpx_apic_get(APICCommonState *s);
+int whpx_apic_init_cpu(CPUState *cpu);
 
 #define WHV_E_UNKNOWN_CAPABILITY 0x80370300L
 
diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 6b00724ceb..4d4ad0919f 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -2969,6 +2969,8 @@ int whpx_init_vcpu(CPUState *cpu)
     header = (X86XSaveHeader *)(env->xsave_buf + sizeof(X86LegacyXSaveArea));
     header->xcomp_bv = header->xstate_bv | (1ULL << 63);
 
+    whpx_apic_init_cpu(cpu);
+
     return 0;
 
 error:
diff --git a/target/i386/whpx/whpx-apic.c b/target/i386/whpx/whpx-apic.c
index 2d83291fae..34d5d57ad2 100644
--- a/target/i386/whpx/whpx-apic.c
+++ b/target/i386/whpx/whpx-apic.c
@@ -13,12 +13,32 @@
 #include "qemu/osdep.h"
 #include "qemu/error-report.h"
 #include "cpu.h"
+#include "migration/vmstate.h"
 #include "hw/i386/apic_internal.h"
 #include "hw/i386/apic-msidef.h"
 #include "hw/pci/msi.h"
 #include "system/hw_accel.h"
 #include "system/whpx.h"
 #include "system/whpx-internal.h"
+#include "target/i386/whpx/whpx-i386.h"
+#include "hw/hyperv/hvgdk_mini.h"
+#include "qemu/memalign.h"
+
+typedef struct hv_local_interrupt_controller_state
+               hv_local_interrupt_controller_state;
+
+struct MshvAPICState {
+    APICCommonState parent_obj;
+
+    uint32_t apic_version;
+    uint32_t apic_lvt_cmci;
+    uint32_t apic_error_status;
+    uint32_t apic_counter_value;
+    uint32_t apic_remote_read;
+};
+
+#define TYPE_MSHV_APIC "whpx-apic"
+OBJECT_DECLARE_SIMPLE_TYPE(MshvAPICState, MSHV_APIC)
 
 struct whpx_lapic_state {
     struct {
@@ -27,7 +47,106 @@ struct whpx_lapic_state {
     } fields[256];
 };
 
-static void whpx_put_apic_state(APICCommonState *s,
+static void populate_hv_lapic_state(hv_local_interrupt_controller_state *hv,
+                                    const CPUState *cpu)
+{
+    uint32_t x2apic_id;
+    X86CPU *x86cpu = X86_CPU(cpu);
+    MshvAPICState *ms = MSHV_APIC(x86cpu->apic_state);
+    APICCommonState *s = &ms->parent_obj;
+    size_t i;
+
+    /*
+     * x2APIC:
+     * - APIC ID is the full 32-bit initial_apic_id
+     * - LDR is read-only, architecturally derived from the ID
+     * - DFR does not exist in x2APIC mode
+     */
+    if (is_x2apic_mode(s)) {
+        x2apic_id = s->initial_apic_id;
+
+        hv->apic_id  = x2apic_id;
+        hv->apic_ldr = ((x2apic_id >> 4) << 16) | (1 << (x2apic_id & 0xf));
+        hv->apic_dfr = 0;
+    } else {
+        hv->apic_id  = s->id << 24;
+        hv->apic_ldr = s->log_dest << 24;
+        hv->apic_dfr = s->dest_mode << 28 | 0x0fffffff;
+    }
+    hv->apic_version = ms->apic_version;
+    hv->apic_spurious = s->spurious_vec;
+    for (i = 0; i < 8; i++) {
+        hv->apic_isr[i] = s->isr[i];
+        hv->apic_tmr[i] = s->tmr[i];
+        hv->apic_irr[i] = s->irr[i];
+    }
+    hv->apic_esr      = s->esr;
+    hv->apic_icr_high = s->icr[1];
+    hv->apic_icr_low  = s->icr[0];
+
+    hv->apic_lvt_timer   = s->lvt[APIC_LVT_TIMER];
+    hv->apic_lvt_thermal = s->lvt[APIC_LVT_THERMAL];
+    hv->apic_lvt_perfmon = s->lvt[APIC_LVT_PERFORM];
+    hv->apic_lvt_lint0   = s->lvt[APIC_LVT_LINT0];
+    hv->apic_lvt_lint1   = s->lvt[APIC_LVT_LINT1];
+    hv->apic_lvt_error   = s->lvt[APIC_LVT_ERROR];
+    hv->apic_lvt_cmci    = ms->apic_lvt_cmci;
+
+    hv->apic_error_status         = ms->apic_error_status;
+    hv->apic_initial_count        = s->initial_count;
+    hv->apic_counter_value        = ms->apic_counter_value;
+    hv->apic_divide_configuration = s->divide_conf;
+    hv->apic_remote_read          = ms->apic_remote_read;
+}
+
+static void populate_apic_state(CPUState *cpu,
+                                const hv_local_interrupt_controller_state *hv)
+{
+    X86CPU *x86cpu = X86_CPU(cpu);
+    MshvAPICState *ms = MSHV_APIC(x86cpu->apic_state);
+    APICCommonState *s = &ms->parent_obj;
+    size_t i;
+
+    /*
+     * x2APIC:
+     * - APIC ID is the full 32-bit initial_apic_id
+     * - LDR is read-only, architecturally derived from the ID
+     * - DFR does not exist in x2APIC mode
+     */
+    if (is_x2apic_mode(s)) {
+        s->initial_apic_id = hv->apic_id;
+    } else {
+        s->id        = hv->apic_id  >> 24;
+        s->log_dest  = hv->apic_ldr >> 24;
+        s->dest_mode = hv->apic_dfr >> 28;
+    }
+    ms->apic_version = hv->apic_version;
+    s->spurious_vec = hv->apic_spurious;
+    for (i = 0; i < 8; i++) {
+        s->isr[i] = hv->apic_isr[i];
+        s->tmr[i] = hv->apic_tmr[i];
+        s->irr[i] = hv->apic_irr[i];
+    }
+    s->esr = hv->apic_esr;
+    s->icr[1] = hv->apic_icr_high;
+    s->icr[0] = hv->apic_icr_low;
+
+    s->lvt[APIC_LVT_TIMER]   = hv->apic_lvt_timer;
+    s->lvt[APIC_LVT_THERMAL] = hv->apic_lvt_thermal;
+    s->lvt[APIC_LVT_PERFORM] = hv->apic_lvt_perfmon;
+    s->lvt[APIC_LVT_LINT0]   = hv->apic_lvt_lint0;
+    s->lvt[APIC_LVT_LINT1]   = hv->apic_lvt_lint1;
+    s->lvt[APIC_LVT_ERROR]   = hv->apic_lvt_error;
+    ms->apic_lvt_cmci        = hv->apic_lvt_cmci;
+
+    ms->apic_error_status  = hv->apic_error_status;
+    s->initial_count       = hv->apic_initial_count;
+    ms->apic_counter_value = hv->apic_counter_value;
+    s->divide_conf         = hv->apic_divide_configuration;
+    ms->apic_remote_read   = hv->apic_remote_read;
+}
+
+static void whpx_put_apic_state_legacy(APICCommonState *s,
                                 struct whpx_lapic_state *kapic)
 {
     int i;
@@ -60,7 +179,7 @@ static void whpx_put_apic_state(APICCommonState *s,
     kapic->fields[0x3e].data = s->divide_conf;
 }
 
-static void whpx_get_apic_state(APICCommonState *s,
+static void whpx_get_apic_state_legacy(APICCommonState *s,
                                 struct whpx_lapic_state *kapic)
 {
     int i, v;
@@ -197,14 +316,14 @@ static void whpx_apic_vapic_base_update(APICCommonState *s)
     /* not implemented yet */
 }
 
-static void whpx_apic_put(CPUState *cs, run_on_cpu_data data)
+static void whpx_apic_put_legacy(CPUState *cs, run_on_cpu_data data)
 {
     APICCommonState *s = data.host_ptr;
     struct whpx_lapic_state kapic;
     HRESULT hr;
 
     whpx_put_apic_base(CPU(s->cpu), s->apicbase);
-    whpx_put_apic_state(s, &kapic);
+    whpx_put_apic_state_legacy(s, &kapic);
 
     hr = whp_dispatch.WHvSetVirtualProcessorInterruptControllerState2(
         whpx_global.partition,
@@ -220,7 +339,7 @@ static void whpx_apic_put(CPUState *cs, run_on_cpu_data data)
     }
 }
 
-void whpx_apic_get(APICCommonState *s)
+static void whpx_apic_get_legacy(APICCommonState *s)
 {
     CPUState *cpu = CPU(s->cpu);
     struct whpx_lapic_state kapic;
@@ -239,7 +358,78 @@ void whpx_apic_get(APICCommonState *s)
         abort();
     }
 
-    whpx_get_apic_state(s, &kapic);
+    whpx_get_apic_state_legacy(s, &kapic);
+}
+
+static void whpx_apic_put(CPUState *cs, run_on_cpu_data data)
+{
+    MshvAPICState *s = data.host_ptr;
+    struct hv_local_interrupt_controller_state kapic;
+    HRESULT hr;
+    /* buffer aligned to 4k, as *state requires that */
+    void *buffer;
+    size_t size = 4096;
+
+    if (!whp_dispatch.WHvSetVirtualProcessorState) {
+        return whpx_apic_put_legacy(cs, data);
+    }
+
+    buffer = qemu_memalign(size, size);
+    whpx_put_apic_base(CPU(s->parent_obj.cpu), s->parent_obj.apicbase);
+    populate_hv_lapic_state(&kapic, cs);
+    memcpy(buffer, &kapic, sizeof(kapic));
+
+    hr = whp_dispatch.WHvSetVirtualProcessorState(
+            whpx_global.partition,
+            cs->cpu_index,
+            WHvVirtualProcessorStateTypeInterruptControllerState2,
+            buffer,
+            size);
+
+    qemu_vfree(buffer);
+    if (FAILED(hr)) {
+        fprintf(stderr,
+            "WHvSetVirtualProcessorState failed: %08lx\n",
+             hr);
+
+        abort();
+    }
+}
+
+void whpx_apic_get(APICCommonState *s)
+{
+    CPUState *cpu = CPU(s->cpu);
+    struct hv_local_interrupt_controller_state kapic;
+    HRESULT hr;
+    /* buffer aligned to 4k, as *state requires that */
+    void *buffer;
+    size_t size = 4096;
+
+    if (!whp_dispatch.WHvSetVirtualProcessorState) {
+        return whpx_apic_get_legacy(s);
+    }
+
+    buffer = qemu_memalign(size, size);
+
+    hr = whp_dispatch.WHvGetVirtualProcessorState(
+        whpx_global.partition,
+        cpu->cpu_index,
+        WHvVirtualProcessorStateTypeInterruptControllerState2,
+        buffer,
+        size,
+        NULL);
+
+    if (FAILED(hr)) {
+        fprintf(stderr,
+            "WHvGetVirtualProcessorState failed: %08lx\n",
+            hr);
+
+        abort();
+    }
+
+    memcpy(&kapic, buffer, sizeof(kapic));
+    qemu_vfree(buffer);
+    populate_apic_state(cpu, &kapic);
 }
 
 static void whpx_apic_pre_save(APICCommonState *s)
@@ -306,6 +496,59 @@ static void whpx_apic_mem_write(void *opaque, hwaddr addr,
     whpx_send_msi(&msg);
 }
 
+static uint32_t set_apic_delivery_mode(uint32_t reg, uint32_t mode)
+{
+    return ((reg) & ~0x700) | ((mode) << 8);
+}
+
+int whpx_apic_init_cpu(CPUState *cpu)
+{
+    uint32_t *lvt_lint0, *lvt_lint1;
+    HRESULT hr;
+    struct hv_local_interrupt_controller_state lapic_state = { 0 };
+    /* buffer aligned to 4k, as *state requires that */
+    void *buffer;
+    size_t size = 4096;
+
+    buffer = qemu_memalign(size, size);
+    hr = whp_dispatch.WHvGetVirtualProcessorState(
+        whpx_global.partition,
+        cpu->cpu_index,
+        WHvVirtualProcessorStateTypeInterruptControllerState2,
+        buffer,
+        size,
+        NULL);
+    if (FAILED(hr)) {
+        return hr;
+    }
+
+    memcpy(&lapic_state, buffer, sizeof(lapic_state));
+
+    lvt_lint0 = &lapic_state.apic_lvt_lint0;
+    *lvt_lint0 = set_apic_delivery_mode(*lvt_lint0, APIC_DM_EXTINT);
+
+    lvt_lint1 = &lapic_state.apic_lvt_lint1;
+    *lvt_lint1 = set_apic_delivery_mode(*lvt_lint1, APIC_DM_NMI);
+
+    /* TODO: should we skip setting lapic if the values are the same? */
+
+    memcpy(buffer, &lapic_state, sizeof(lapic_state));
+    hr = whp_dispatch.WHvSetVirtualProcessorState(
+            whpx_global.partition,
+            cpu->cpu_index,
+            WHvVirtualProcessorStateTypeInterruptControllerState2,
+            buffer,
+            size);
+    if (FAILED(hr)) {
+        return hr;
+    }
+
+    qemu_vfree(buffer);
+    populate_apic_state(cpu, &lapic_state);
+
+    return 0;
+}
+
 static const MemoryRegionOps whpx_apic_io_ops = {
     .read = whpx_apic_mem_read,
     .write = whpx_apic_mem_write,
@@ -324,14 +567,51 @@ static void whpx_apic_reset(APICCommonState *s)
     run_on_cpu(CPU(s->cpu), whpx_apic_put, RUN_ON_CPU_HOST_PTR(s));
 }
 
+static const VMStateDescription vmstate_mshv_apic = {
+    .name = "mshv-apic",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (const VMStateField[]) {
+        VMSTATE_UINT32(apic_version, MshvAPICState),
+        VMSTATE_UINT32(apic_lvt_cmci, MshvAPICState),
+        VMSTATE_UINT32(apic_error_status, MshvAPICState),
+        VMSTATE_UINT32(apic_counter_value, MshvAPICState),
+        VMSTATE_UINT32(apic_remote_read, MshvAPICState),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 static void whpx_apic_realize(DeviceState *dev, Error **errp)
 {
     APICCommonState *s = APIC_COMMON(dev);
+    MshvAPICState *ms = MSHV_APIC(dev);
 
     memory_region_init_io(&s->io_memory, OBJECT(s), &whpx_apic_io_ops, s,
                           "whpx-apic-msi", APIC_SPACE_SIZE);
 
     msi_nonbroken = true;
+
+    /*
+     * We register this state explicity, rather than going via dc->vmsd.
+     * The auto-wiring would register the state with
+     * instance_id == VMSTATE_INSTANCE_ID_ANY, which for the APIC doesn't
+     * work, b/c the ID carries semantic meaning for restoring the state
+     * on the destination (which vcpu it belongs to).
+     */
+    if (!!whp_dispatch.WHvSetVirtualProcessorState) {
+        vmstate_register_with_alias_id(NULL,
+                                   s->initial_apic_id, &vmstate_mshv_apic, ms,
+                                   -1, 0, NULL);
+    }
+}
+
+static void whpx_apic_unrealize(DeviceState *dev)
+{
+    MshvAPICState *ms = MSHV_APIC(dev);
+
+    if (!!whp_dispatch.WHvSetVirtualProcessorState) {
+        vmstate_unregister(NULL, &vmstate_mshv_apic, ms);
+    }
 }
 
 static void whpx_apic_class_init(ObjectClass *klass, const void *data)
@@ -339,6 +619,7 @@ static void whpx_apic_class_init(ObjectClass *klass, const void *data)
     APICCommonClass *k = APIC_COMMON_CLASS(klass);
 
     k->realize = whpx_apic_realize;
+    k->unrealize = whpx_apic_unrealize;
     k->reset = whpx_apic_reset;
     k->set_base = apic_set_base;
     k->set_tpr = whpx_apic_set_tpr;
@@ -351,9 +632,9 @@ static void whpx_apic_class_init(ObjectClass *klass, const void *data)
 }
 
 static const TypeInfo whpx_apic_info = {
-    .name = "whpx-apic",
+    .name = TYPE_MSHV_APIC,
     .parent = TYPE_APIC_COMMON,
-    .instance_size = sizeof(APICCommonState),
+    .instance_size = sizeof(MshvAPICState),
     .class_init = whpx_apic_class_init,
 };
 
-- 
2.54.0 (Apple Git-157)

[PATCH v2 12/18] whpx: i386: forward the dynamic EBX of CPUID[0xD] sub-leaves 0 and 1, not 1 and 2

[Mohamed Mediouni]

From: Kostas Chatzikokolakis <kostas@chatzi.org>

CPUID[0xD,0].EBX is the size of the XSAVE area for the guest's current
XCR0, and CPUID[0xD,1].EBX the size for XCR0 | IA32_XSS: those are the
two values that depend on guest state. For sub-leaf 2 and up, EBX is the
component's offset in the standard format, a constant.

whpx_vcpu_run() forwards Hyper-V's DefaultResultRbx for sub-leaves 1 and
2 instead. Sub-leaf 2 is harmless: QEMU's own offset was read from
Hyper-V at vCPU init. Sub-leaf 0 is not: QEMU computes it from
env->xcr0, which WHPX refreshes only in whpx_get_registers(), never on
a CPUID exit, so after the guest's XSETBV the leaf still reports the
size for the reset XCR0, 576 bytes.

A Linux guest on a host without XSAVES takes that as its kernel XSAVE
size, fails the consistency check against the size it computed from the
component offsets, disables XSAVE and panics on the fallback path:

  XSAVE consistency problem: size 832 != kernel_size 576
  WARNING: CPU: 0 PID: 0 at arch/x86/kernel/fpu/xstate.c:638 fpu__init_system_xstate
  x86/fpu: XSAVE disabled
  BUG: kernel NULL pointer dereference, address: 0000000000000010
  RIP: 0010:fpstate_reset+0x1a/0x90
  Kernel panic - not syncing: Attempted to kill the idle task!

Every CPU model with XSAVE is affected there (-cpu max, Haswell,
SandyBridge); models without it (Westmere, Nehalem, qemu64) boot. A host
with XSAVES boots because Linux takes the compacted size from sub-leaf 1,
which is forwarded; sub-leaf 0 is stale there too.

Reproduced on an Intel Core i5-4440 (Haswell) under Windows 11 Home
build 26300 with QEMU 11.1.2 (the code is unchanged on master), guest
Linux 6.18. With this patch the guest enables xstate 0x7 at 832 bytes
and boots with -cpu max. An AMD Ryzen 9 7900X under build 26100, not
affected before, still boots with the patch, enabling 0xe7 (compacted,
2432 bytes).

Fixes: 79b3cb1f0145 ("whpx: i386: introduce proper cpuid support")
Signed-off-by: Kostas Chatzikokolakis <kostas@chatzi.org>

Reviewed-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 4d4ad0919f..065cfacf3b 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -2733,10 +2733,14 @@ int whpx_vcpu_run(CPUState *cpu)
                 }
             }
 
-            /* CPUID[0xD,{1,2}].EBX are dynamic depending on guest features. */
+            /*
+             * CPUID[0xD,{0,1}].EBX are dynamic: the XSAVE area size for the
+             * guest's current XCR0, resp. XCR0 | IA32_XSS. For sub-leaf >= 2,
+             * EBX is the component's fixed offset, which QEMU already reports.
+             */
             if (vcpu->exit_ctx.CpuidAccess.Rax == 0xd) {
-                if (vcpu->exit_ctx.CpuidAccess.Rcx == 1
-                    || vcpu->exit_ctx.CpuidAccess.Rcx == 2) {
+                if (vcpu->exit_ctx.CpuidAccess.Rcx == 0
+                    || vcpu->exit_ctx.CpuidAccess.Rcx == 1) {
                     reg_values[4].Reg64 = vcpu->exit_ctx.CpuidAccess.DefaultResultRbx;
                 }
             }
-- 
2.54.0 (Apple Git-157)

[PATCH v2 13/18] whpx: i386: do not require ProcessorFeaturesBanks on legacy Windows

[Mohamed Mediouni]

From: Akshey Taneja <aksheynew.tanejanew@gmail.com>

WHvCapabilityCodeProcessorFeaturesBanks and the matching partition
property are implemented starting from Windows 10 version 20H1.

On earlier Windows 10 releases, the capability query returns
WHV_E_UNKNOWN_CAPABILITY (0x80370300) and whpx_accel_init()
aborts with -ENOSPC, reported to the user as "failed to initialize
whpx: No space left on device". This regressed older Windows 10
releases in 8b87283ce4 ("whpx: i386: enable all supported host features").

Fall back to the original single-bank WHvCapabilityCodeProcessorFeatures
on legacy hosts and skip the corresponding SetPartitionProperty, which
restores the pre-8b87283ce4 behaviour there.

Reported-on: Windows 10 Enterprise LTSC 17763, Intel i7-6700
Signed-off-by: Akshey Taneja <aksheynew.tanejanew@gmail.com>

[Editing patch for reflecting the availability of this API, which is
available in Windows 10 version 2004 onwards]
Reviewed-by: Mohamed Mediouni <mohamed@unpredictable.fr>
Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 59 +++++++++++++++++++++++++++----------
 1 file changed, 43 insertions(+), 16 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 065cfacf3b..7552e9f4da 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -3331,13 +3331,33 @@ int whpx_accel_init(AccelState *as, MachineState *ms)
     memset(&processor_features, 0, sizeof(WHV_PROCESSOR_FEATURES_BANKS));
     processor_features.BanksCount = 2;
 
-    hr = whp_dispatch.WHvGetCapability(
-        WHvCapabilityCodeProcessorFeaturesBanks, &processor_features,
-        sizeof(WHV_PROCESSOR_FEATURES_BANKS), &whpx_cap_size);
-    if (FAILED(hr)) {
-        error_report("WHPX: Failed to get processor features, hr=%08lx", hr);
-        ret = -ENOSPC;
-        goto error;
+    if (whpx_is_legacy_os()) {
+        /*
+         * WHvCapabilityCodeProcessorFeaturesBanks is implemented starting
+         * from Windows 10 version 20H2 (19041). Fall back to the single-bank
+         * capability on earlier Windows 10 releases.
+         */
+        WHV_PROCESSOR_FEATURES legacy_features = {0};
+
+        processor_features.BanksCount = 1;
+        hr = whp_dispatch.WHvGetCapability(
+            WHvCapabilityCodeProcessorFeatures, &legacy_features,
+            sizeof(legacy_features), &whpx_cap_size);
+        if (FAILED(hr)) {
+            warn_report("WHPX: Failed to get legacy processor features, "
+                        "hr=%08lx", hr);
+        } else {
+            processor_features.Bank0 = legacy_features;
+        }
+    } else {
+        hr = whp_dispatch.WHvGetCapability(
+            WHvCapabilityCodeProcessorFeaturesBanks, &processor_features,
+            sizeof(WHV_PROCESSOR_FEATURES_BANKS), &whpx_cap_size);
+        if (FAILED(hr)) {
+            error_report("WHPX: Failed to get processor features, hr=%08lx", hr);
+            ret = -ENOSPC;
+            goto error;
+        }
     }
 
     whpx_rdtsc_cap = processor_features.Bank0.RdtscpSupport;
@@ -3382,15 +3402,22 @@ int whpx_accel_init(AccelState *as, MachineState *ms)
         }
     }
 
-    hr = whp_dispatch.WHvSetPartitionProperty(
-            whpx->partition,
-            WHvPartitionPropertyCodeProcessorFeaturesBanks,
-            &processor_features,
-            sizeof(WHV_PROCESSOR_FEATURES_BANKS));
-    if (FAILED(hr)) {
-        error_report("WHPX: Failed to set processor features, hr=%08lx", hr);
-        ret = -EINVAL;
-        goto error;
+    /*
+     * The matching partition property is also Windows Server 2022 and later.
+     * Legacy hosts simply keep the features the hypervisor gave them, which
+     * is how WHPX behaved before the banked feature set was introduced.
+     */
+    if (!whpx_is_legacy_os()) {
+        hr = whp_dispatch.WHvSetPartitionProperty(
+                whpx->partition,
+                WHvPartitionPropertyCodeProcessorFeaturesBanks,
+                &processor_features,
+                sizeof(WHV_PROCESSOR_FEATURES_BANKS));
+        if (FAILED(hr)) {
+            error_report("WHPX: Failed to set processor features, hr=%08lx", hr);
+            ret = -EINVAL;
+            goto error;
+        }
     }
 
 
-- 
2.54.0 (Apple Git-157)

[PATCH v2 14/18] whpx: i386: fall back to xAPIC when the MSR exit bitmap is unavailable

[Mohamed Mediouni]

From: Akshey Taneja <aksheynew.tanejanew@gmail.com>

WHvPartitionPropertyCodeX64MsrExitBitmap is not implemented on Windows
10 version 1809 and earlier, where WHvSetPartitionProperty() returns
WHV_E_UNKNOWN_PROPERTY (0x80370302) and whpx_accel_init() aborts with
-EINVAL. QEMU started requesting it in ba1bda25b7 ("whpx: i386: x2apic emulation").

Treating the failure as non-fatal is not sufficient on its own:
whpx_default_props unconditionally enables x2apic, and x2APIC is reached
through MSRs, so without MSR exits the userspace LAPIC never observes
the guest's APIC accesses and the guest hangs immediately after the
firmware hands over. Observed as a black screen with an Ubuntu 24.04
guest that otherwise boots in about 40 seconds.

Record whether the MSR exit bitmap was accepted and, when it was not,
apply x2apic=off so the guest uses the MMIO based xAPIC, which is still
intercepted normally.

Reported-on: Windows 10 Enterprise LTSC 17763, Intel i7-6700
Signed-off-by: Akshey Taneja <aksheynew.tanejanew@gmail.com>

[Patch edited, version 1903 is the first release of Windows 10 with this API]
Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 30 ++++++++++++++++++++++++++----
 1 file changed, 26 insertions(+), 4 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 7552e9f4da..26ca879c89 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -57,6 +57,7 @@
 #define HV_X64_MSR_GUEST_IDLE           0x400000f0
 
 static bool is_modern_os = true;
+static bool whpx_msr_exits_available = true;
 
 static const WHV_REGISTER_NAME whpx_register_names[] = {
 
@@ -3027,6 +3028,16 @@ static PropValue whpx_default_props[] = {
     { NULL, NULL },
 };
 
+/*
+ * x2APIC is reached through MSRs, so emulating it in userspace requires the
+ * MSR exit bitmap. Hosts without that property must fall back to xAPIC,
+ * which is MMIO based and therefore still intercepted.
+ */
+static PropValue whpx_no_msr_exit_props[] = {
+    { "x2apic", "off" },
+    { NULL, NULL },
+};
+
 
 void whpx_cpu_instance_init(CPUState *cs)
 {
@@ -3034,7 +3045,8 @@ void whpx_cpu_instance_init(CPUState *cs)
     X86CPUClass *xcc = X86_CPU_GET_CLASS(cpu);
 
     host_cpu_instance_init(cpu);
-    x86_cpu_apply_props(cpu, whpx_default_props);
+    x86_cpu_apply_props(cpu, whpx_msr_exits_available ? whpx_default_props
+                                                      : whpx_no_msr_exit_props);
 
     if (xcc->max_features) {
         whpx_cpu_max_instance_init(cpu);
@@ -3478,10 +3490,20 @@ int whpx_accel_init(AccelState *as, MachineState *ms)
             WHvPartitionPropertyCodeX64MsrExitBitmap,
             &prop,
             sizeof(WHV_PARTITION_PROPERTY));
+    /*
+     * Earlier Windows 10 releases not implement this property.
+     * It is implemented starting from Windows 10 version 1903 (18362).
+     */
     if (FAILED(hr)) {
-        error_report("WHPX: Failed to set MSR exit bitmap, hr=%08lx", hr);
-        ret = -EINVAL;
-        goto error;
+        if (hr == WHV_E_UNKNOWN_PROPERTY) {
+            whpx_msr_exits_available = false;
+            warn_report("WHPX: MSR exit bitmap unsupported on this Windows "
+                        "version, falling back to xAPIC");
+        } else {
+            error_report("WHPX: Failed to set MSR exit bitmap, hr=%08lx", hr);
+            ret = -EINVAL;
+            goto error;
+        }
     }
 
     hr = whp_dispatch.WHvSetPartitionProperty(
-- 
2.54.0 (Apple Git-157)

[PATCH v2 15/18] multi-target: restrict whpx-accel-*-cpu to matching target

[Mohamed Mediouni]

From: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>

On windows x64 host:
$ ./build/tests/unit/test-multi-target
Type 'whpx-accel-x86_64-cpu' is missing its parent 'accel-x86_64-cpu'
(same for windows-arm64 hosts and whpx-accel-aarch64-cpu)

Parent type is conditionally registered from current target:
register_accel_target_type uses '"accel-" + target_cpu_type()' as type
name. Since whpx-common.c is shared between arm and x86_64 targets,
only register whpx accelerator cpu type when current target cpu type
matches the one it was compiled for.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@oss.qualcomm.com>
Reviewed-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 accel/whpx/whpx-common.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/accel/whpx/whpx-common.c b/accel/whpx/whpx-common.c
index 247e12db81..b7a3249f33 100644
--- a/accel/whpx/whpx-common.c
+++ b/accel/whpx/whpx-common.c
@@ -20,6 +20,7 @@
 #include "hw/core/boards.h"
 #include "hw/intc/ioapic.h"
 #include "qemu/error-report.h"
+#include "qemu/target-info.h"
 #include "qapi/error.h"
 #include "qapi/qapi-types-common.h"
 #include "qapi/qapi-visit-common.h"
@@ -512,11 +513,18 @@ static void whpx_cpu_accel_class_init(ObjectClass *oc, const void *data)
     acc->cpu_instance_init = whpx_cpu_instance_init;
 }
 
+/* TYPE_ACCEL_CPU is only registered for current target cpu type */
+static bool whpx_cpu_accel_is_available(void)
+{
+    return !strcmp(target_cpu_type(), CPU_RESOLVING_TYPE);
+}
+
 static const TypeInfo whpx_cpu_accel_type = {
     .name = ACCEL_CPU_NAME("whpx"),
 
     .parent = TYPE_ACCEL_CPU,
     .class_init = whpx_cpu_accel_class_init,
+    .is_available = whpx_cpu_accel_is_available,
     .abstract = true,
 };
 
-- 
2.54.0 (Apple Git-157)

[PATCH v2 16/18] whpx: i386: also save debug registers

[Mohamed Mediouni]

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 43 +++++++++++++++++++++++++++----------
 1 file changed, 32 insertions(+), 11 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index 26ca879c89..d2cd8e8001 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -101,16 +101,6 @@ static const WHV_REGISTER_NAME whpx_register_names[] = {
     WHvX64RegisterCr3,
     WHvX64RegisterCr4,
 
-    /* X64 Debug Registers */
-    /*
-     * WHvX64RegisterDr0,
-     * WHvX64RegisterDr1,
-     * WHvX64RegisterDr2,
-     * WHvX64RegisterDr3,
-     * WHvX64RegisterDr6,
-     * WHvX64RegisterDr7,
-     */
-
     /* X64 MSRs */
     WHvX64RegisterEfer,
 #ifdef TARGET_X86_64
@@ -126,7 +116,12 @@ static const WHV_REGISTER_NAME whpx_register_names[] = {
     WHvX64RegisterCstar,
     WHvX64RegisterSfmask,
 #endif
-
+    WHvX64RegisterDr0,
+    WHvX64RegisterDr1,
+    WHvX64RegisterDr2,
+    WHvX64RegisterDr3,
+    WHvX64RegisterDr6,
+    WHvX64RegisterDr7
     /* Interrupt / Event Registers */
     /*
      * WHvRegisterPendingInterruption,
@@ -686,6 +681,19 @@ void whpx_set_registers(CPUState *cpu, WHPXStateLevel level)
         vcxt.values[idx++].Reg64 = env->fmask;
 #endif
 
+        assert(whpx_register_names[idx] == WHvX64RegisterDr0);
+        vcxt.values[idx++].Reg64 = env->dr[0];
+        assert(whpx_register_names[idx] == WHvX64RegisterDr1);
+        vcxt.values[idx++].Reg64 = env->dr[1];
+        assert(whpx_register_names[idx] == WHvX64RegisterDr2);
+        vcxt.values[idx++].Reg64 = env->dr[2];
+        assert(whpx_register_names[idx] == WHvX64RegisterDr3);
+        vcxt.values[idx++].Reg64 = env->dr[3];
+        assert(whpx_register_names[idx] == WHvX64RegisterDr6);
+        vcxt.values[idx++].Reg64 = env->dr[6];
+        assert(whpx_register_names[idx] == WHvX64RegisterDr7);
+        vcxt.values[idx++].Reg64 = env->dr[7];
+
         /* Interrupt / Event Registers - Skipped */
 
         assert(idx == RTL_NUMBER_OF(whpx_register_names));
@@ -1070,6 +1078,19 @@ void whpx_get_registers(CPUState *cpu, WHPXStateLevel level)
     env->fmask = vcxt.values[idx++].Reg64;
 #endif
 
+    assert(whpx_register_names[idx] == WHvX64RegisterDr0);
+    env->dr[0] = vcxt.values[idx++].Reg64;
+    assert(whpx_register_names[idx] == WHvX64RegisterDr1);
+    env->dr[1] = vcxt.values[idx++].Reg64;
+    assert(whpx_register_names[idx] == WHvX64RegisterDr2);
+    env->dr[2] = vcxt.values[idx++].Reg64;
+    assert(whpx_register_names[idx] == WHvX64RegisterDr3);
+    env->dr[3] = vcxt.values[idx++].Reg64;
+    assert(whpx_register_names[idx] == WHvX64RegisterDr6);
+    env->dr[6] = vcxt.values[idx++].Reg64;
+    assert(whpx_register_names[idx] == WHvX64RegisterDr7);
+    env->dr[7] = vcxt.values[idx++].Reg64;
+
     /* Interrupt / Event Registers - Skipped */
 
     assert(idx == RTL_NUMBER_OF(whpx_register_names));
-- 
2.54.0 (Apple Git-157)

[PATCH v2 17/18] whpx: i386: save/restore pending interruption/events

[Mohamed Mediouni]

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 target/i386/whpx/whpx-all.c | 106 ++++++++++++++++++++++++++++++++++--
 1 file changed, 102 insertions(+), 4 deletions(-)

diff --git a/target/i386/whpx/whpx-all.c b/target/i386/whpx/whpx-all.c
index d2cd8e8001..1447c777d3 100644
--- a/target/i386/whpx/whpx-all.c
+++ b/target/i386/whpx/whpx-all.c
@@ -121,10 +121,11 @@ static const WHV_REGISTER_NAME whpx_register_names[] = {
     WHvX64RegisterDr2,
     WHvX64RegisterDr3,
     WHvX64RegisterDr6,
-    WHvX64RegisterDr7
+    WHvX64RegisterDr7,
+    WHvRegisterPendingInterruption,
+    WHvRegisterPendingEvent,
     /* Interrupt / Event Registers */
     /*
-     * WHvRegisterPendingInterruption,
      * WHvRegisterInterruptState,
      * WHvRegisterPendingEvent0,
      * WHvRegisterPendingEvent1
@@ -694,7 +695,47 @@ void whpx_set_registers(CPUState *cpu, WHPXStateLevel level)
         assert(whpx_register_names[idx] == WHvX64RegisterDr7);
         vcxt.values[idx++].Reg64 = env->dr[7];
 
-        /* Interrupt / Event Registers - Skipped */
+        /* Interrupt / Event Registers */
+
+        assert(whpx_register_names[idx] == WHvRegisterPendingInterruption);
+        if (!whpx_irqchip_in_kernel() &&
+            env->exception_pending) {
+            vcxt.values[idx].PendingInterruption.InterruptionPending = 1;
+            vcxt.values[idx].PendingInterruption.InterruptionVector = env->exception_nr;
+            vcxt.values[idx].PendingInterruption.DeliverErrorCode = env->has_error_code;
+            vcxt.values[idx].PendingInterruption.ErrorCode = env->error_code;
+            if (env->exception_is_int) {
+                vcxt.values[idx].PendingInterruption.InterruptionType = WHvX64PendingInterrupt;
+            } else if (env->nmi_pending) {
+                vcxt.values[idx].PendingInterruption.InterruptionType = WHvX64PendingNmi;
+            } else {
+                vcxt.values[idx].PendingInterruption.InterruptionType = WHvX64PendingException;
+            }
+        }
+        idx++;
+
+        assert(whpx_register_names[idx] == WHvRegisterPendingEvent);
+        if (whpx_irqchip_in_kernel() &&
+            env->exception_pending) {
+            if (!env->exception_is_int) {
+                vcxt.values[idx].ExceptionEvent.EventPending = 1;
+                vcxt.values[idx].ExceptionEvent.EventType =
+                    WHvX64PendingEventException;
+                vcxt.values[idx].ExceptionEvent.Vector = env->exception_nr;
+                vcxt.values[idx].ExceptionEvent.DeliverErrorCode
+                    = env->has_error_code;
+                vcxt.values[idx].ExceptionEvent.ErrorCode = env->error_code;
+                vcxt.values[idx].ExceptionEvent.ExceptionParameter =
+                    env->exception_payload;
+            } else if (vcxt.values[idx].ExtIntEvent.EventType
+                == WHvX64PendingEventExtInt) {
+                vcxt.values[idx].ExtIntEvent.EventPending = 1;
+                vcxt.values[idx].ExtIntEvent.Vector = env->exception_nr;
+            }  else {
+                abort();
+            }
+        }
+        idx++;
 
         assert(idx == RTL_NUMBER_OF(whpx_register_names));
     }
@@ -1091,7 +1132,64 @@ void whpx_get_registers(CPUState *cpu, WHPXStateLevel level)
     assert(whpx_register_names[idx] == WHvX64RegisterDr7);
     env->dr[7] = vcxt.values[idx++].Reg64;
 
-    /* Interrupt / Event Registers - Skipped */
+    /* Interrupt / Event Registers */
+
+    assert(whpx_register_names[idx] == WHvRegisterPendingInterruption);
+    if (!whpx_irqchip_in_kernel() &&
+        vcxt.values[idx].PendingInterruption.InterruptionPending) {
+        env->exception_pending = 1;
+        env->exception_nr = 
+            vcxt.values[idx].PendingInterruption.InterruptionVector;
+        env->has_error_code =
+            vcxt.values[idx].PendingInterruption.DeliverErrorCode;
+        env->error_code =
+            vcxt.values[idx].PendingInterruption.ErrorCode;
+        if (vcxt.values[idx].PendingInterruption.InterruptionType
+            == WHvX64PendingInterrupt) {
+            env->exception_is_int = 1;
+            env->nmi_pending = 0;
+        } else if (vcxt.values[idx].PendingInterruption.InterruptionType
+            == WHvX64PendingNmi) {
+            env->exception_is_int = 0;
+            env->nmi_pending = 1;
+        } else if (vcxt.values[idx].PendingInterruption.InterruptionType
+            == WHvX64PendingException) {
+            env->exception_is_int = 0;
+            env->nmi_pending = 0;
+        } else {
+            abort();
+        }
+    }
+    idx++;
+
+    assert(whpx_register_names[idx] == WHvRegisterPendingEvent);
+    if (whpx_irqchip_in_kernel() &&
+        vcxt.values[idx].ExceptionEvent.EventPending) {
+        if (vcxt.values[idx].ExceptionEvent.EventType
+            == WHvX64PendingEventException) {
+            env->exception_pending = 1;
+            env->exception_nr =
+                vcxt.values[idx].ExceptionEvent.Vector;
+            env->has_error_code = 
+                vcxt.values[idx].ExceptionEvent.DeliverErrorCode;
+            env->error_code =
+                vcxt.values[idx].ExceptionEvent.ErrorCode;
+            env->exception_payload =
+                vcxt.values[idx].ExceptionEvent.ExceptionParameter;
+            env->exception_is_int = 0;
+            env->nmi_pending = 0;
+        } else if (vcxt.values[idx].ExtIntEvent.EventType
+            == WHvX64PendingEventExtInt) {
+            env->exception_pending = 1;
+            env->exception_is_int = 1;
+            env->nmi_pending = 0;
+            env->exception_nr = 
+                vcxt.values[idx].ExtIntEvent.Vector;
+        } else {
+            abort();
+        }
+    }
+    idx++;
 
     assert(idx == RTL_NUMBER_OF(whpx_register_names));
 
-- 
2.54.0 (Apple Git-157)

[PATCH v2 18/18] whpx: docs: update for experimental VM save/restore support

[Mohamed Mediouni]

It's there but still highly experimental for the time being.

Signed-off-by: Mohamed Mediouni <mohamed@unpredictable.fr>
---
 docs/system/whpx.rst | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/docs/system/whpx.rst b/docs/system/whpx.rst
index 9909e86831..9ad1448d04 100644
--- a/docs/system/whpx.rst
+++ b/docs/system/whpx.rst
@@ -80,6 +80,8 @@ as in a BTB flush when entering/exiting the guest. This results in a
 significant MMIO performance increase at the detriment of security
 mitigations.
 
+``-cpu host,svm=off,vmx=off`` disables nested virtualisation.
+
 Known issues on x86_64
 ----------------------
 
@@ -114,6 +116,16 @@ disables the PIC. In that configuration, using a UEFI is recommended.
 On this release, ``-M kernel-irqchip=`` is not expected to be manually
 set during normal operation. It remains as a debugging option.
 
+VM save-restore
+^^^^^^^^^^^^^^^
+VM save-restore is currently highly experimental.
+
+It's currently blocked when using Hyper-V enlightenments, which can be disabled with
+``-accel whpx,hyperv=off`` and is blocked when nested virtualisation is enabled.
+
+Dirty page tracking is absent and it is currently best used with ``-M kernel-irqchip=off``
+especially for SMP configurations.
+
 Known issues on Windows 11
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- 
2.54.0 (Apple Git-157)