[PATCH 0/5] i386/cpu: Support APX for KVM

[PATCH 0/5] i386/cpu: Support APX for KVM

[Zhao Liu]

Hi,

This series adds APX (Advanced Performance Extensions) support in QEMU
to enable APX in Guest based on KVM.

This series is based on CET v4:

https://lore.kernel.org/qemu-devel/20251118034231.704240-1-zhao1.liu@intel.com/

And you can also find the code here:

https://gitlab.com/zhao.liu/qemu/-/commits/i386-all-for-dmr-v1.1-11-17-2025

The patches for KVM side can be found at:

https://lore.kernel.org/kvm/20251110180131.28264-1-chang.seok.bae@intel.com/


Thanks for your review!


Overview
========

Intel Advanced Performance Extensions (Intel APX) expands the Intel 64
instruction set architecture with access to more registers (16
additional general-purpose registers (GPRs) R16–R31) and adds various
new features that improve general-purpose performance. The extensions
are designed to provide efficient performance gains across a variety of
workloads without significantly increasing silicon area or power
consumption of the core. 

APX spec link (rev.07) is:
https://cdrdv2.intel.com/v1/dl/getContent/861610

At QEMU side, the enabling work mainly includes two parts:

1. save/restore/migrate the xstate of APX.
   * APX xstate is a user xstate, but it reuses MPX xstate area in
     un-compacted XSAVE buffer.
   * To address this, QEMU will reject both APX and MPX if their CPUID
     feature bits are set at the same (in Patch 1).

2. add related CPUIDs support in feature words.

Thanks and Best Regards,
Zhao
---
Zhao Liu (2):
  i386/cpu: Support APX CPUIDs
  i386/cpu: Mark apx xstate as migratable

Zide Chen (3):
  i386/cpu: Add APX EGPRs into xsave area
  i386/cpu: Cache EGPRs in CPUX86State
  i386/cpu: Add APX migration support

 target/i386/cpu.c          | 68 ++++++++++++++++++++++++++++++++++++--
 target/i386/cpu.h          | 26 +++++++++++++--
 target/i386/machine.c      | 24 ++++++++++++++
 target/i386/xsave_helper.c | 14 ++++++++
 4 files changed, 128 insertions(+), 4 deletions(-)

-- 
2.34.1

[PATCH 1/5] i386/cpu: Add APX EGPRs into xsave area

[Zhao Liu]

From: Zide Chen <zide.chen@intel.com>

APX feature bit is in CPUID_7_1_EDX[21], and APX has EGPR component with
index 19 in xstate area, EGPR component has 16 64bit regs. Add EGRP
component into xstate area.

Note, APX re-uses the 128-byte XSAVE area that had been previously
allocated by MPX which has been deprecated on Intel processors, so check
whether APX and MPX are set at the same for Guest, if this case happens,
mask off them both to avoid conflict for xsave area.

Tested-by: Xudong Hao <xudong.hao@intel.com>
Signed-off-by: Zide Chen <zide.chen@intel.com>
Co-developed-by: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.c | 25 +++++++++++++++++++++++++
 target/i386/cpu.h | 17 +++++++++++++++--
 2 files changed, 40 insertions(+), 2 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 84adfaf99dc8..16bc4b18266c 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -2111,6 +2111,12 @@ ExtSaveArea x86_ext_save_areas[XSAVE_STATE_AREA_COUNT] = {
             { FEAT_7_0_EDX,         CPUID_7_0_EDX_AMX_TILE },
         },
     },
+    [XSTATE_APX_BIT] = {
+        .size = sizeof(XSaveAPX),
+        .features = {
+            { FEAT_7_1_EDX,         CPUID_7_1_EDX_APX },
+        },
+    },
 };
 
 uint32_t xsave_area_size(uint64_t mask, bool compacted)
@@ -9116,6 +9122,25 @@ void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
         env->features[FEAT_KVM] = 0;
     }
 
+    /*
+     * Since Intel MPX had been previously deprecated, APX re-purposes the
+     * 128-byte XSAVE area that had been previously allocated by MPX (state
+     * component indices 3 and 4, making up a 128-byte area located at an
+     * offset of 960 bytes into an un-compacted XSAVE buffer), as a single
+     * state component housing 128-bytes of storage for EGPRs (8-bytes * 16
+     * registers).
+     *
+     * Check the conflict between MPX and APX before initializing xsave
+     * components.
+     */
+    if ((env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_MPX) &&
+        (env->features[FEAT_7_1_EDX] & CPUID_7_1_EDX_APX)) {
+        mark_unavailable_features(cpu, FEAT_7_0_EBX, CPUID_7_0_EBX_MPX,
+            "this feature is conflict with APX");
+        mark_unavailable_features(cpu, FEAT_7_1_EDX, CPUID_7_1_EDX_APX,
+            "this feature is conflict with MPX");
+    }
+
     x86_cpu_enable_xsave_components(cpu);
 
     /* CPUID[EAX=7,ECX=0].EBX always increased level automatically: */
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index d8bdf342f98d..bc7e16d6e6c1 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -604,6 +604,7 @@ typedef enum X86Seg {
 #define XSTATE_ARCH_LBR_BIT             15
 #define XSTATE_XTILE_CFG_BIT            17
 #define XSTATE_XTILE_DATA_BIT           18
+#define XSTATE_APX_BIT                  19
 
 #define XSTATE_FP_MASK                  (1ULL << XSTATE_FP_BIT)
 #define XSTATE_SSE_MASK                 (1ULL << XSTATE_SSE_BIT)
@@ -620,6 +621,7 @@ typedef enum X86Seg {
 #define XSTATE_ARCH_LBR_MASK            (1ULL << XSTATE_ARCH_LBR_BIT)
 #define XSTATE_XTILE_CFG_MASK           (1ULL << XSTATE_XTILE_CFG_BIT)
 #define XSTATE_XTILE_DATA_MASK          (1ULL << XSTATE_XTILE_DATA_BIT)
+#define XSTATE_APX_MASK                 (1ULL << XSTATE_APX_BIT)
 
 #define XSTATE_DYNAMIC_MASK             (XSTATE_XTILE_DATA_MASK)
 
@@ -639,7 +641,8 @@ typedef enum X86Seg {
                                  XSTATE_BNDCSR_MASK | XSTATE_OPMASK_MASK | \
                                  XSTATE_ZMM_Hi256_MASK | \
                                  XSTATE_Hi16_ZMM_MASK | XSTATE_PKRU_MASK | \
-                                 XSTATE_XTILE_CFG_MASK | XSTATE_XTILE_DATA_MASK)
+                                 XSTATE_XTILE_CFG_MASK | \
+                                 XSTATE_XTILE_DATA_MASK | XSTATE_APX_MASK)
 
 /* CPUID feature bits available in XSS */
 #define CPUID_XSTATE_XSS_MASK   (XSTATE_ARCH_LBR_MASK | XSTATE_CET_U_MASK | \
@@ -1042,6 +1045,8 @@ uint64_t x86_cpu_get_supported_feature_word(X86CPU *cpu, FeatureWord w);
 #define CPUID_7_1_EDX_PREFETCHITI       (1U << 14)
 /* Support for Advanced Vector Extensions 10 */
 #define CPUID_7_1_EDX_AVX10             (1U << 19)
+/* Support for Advanced Performance Extensions  */
+#define CPUID_7_1_EDX_APX               (1U << 21)
 
 /* Indicate bit 7 of the IA32_SPEC_CTRL MSR is supported */
 #define CPUID_7_2_EDX_PSFD              (1U << 0)
@@ -1684,6 +1689,8 @@ typedef struct {
 
 #define ARCH_LBR_NR_ENTRIES 32
 
+#define EGPR_NUM  16
+
 /* CPU can't have 0xFFFFFFFF APIC ID, use that value to distinguish
  * that APIC ID hasn't been set yet
  */
@@ -1794,6 +1801,11 @@ typedef struct XSaveXTILEDATA {
     uint8_t xtiledata[8][1024];
 } XSaveXTILEDATA;
 
+/* Ext. save area 19: APX state */
+typedef struct XSaveAPX {
+    uint64_t egprs[EGPR_NUM];
+} XSaveAPX;
+
 QEMU_BUILD_BUG_ON(sizeof(XSaveAVX) != 0x100);
 QEMU_BUILD_BUG_ON(sizeof(XSaveBNDREG) != 0x40);
 QEMU_BUILD_BUG_ON(sizeof(XSaveBNDCSR) != 0x40);
@@ -1806,6 +1818,7 @@ QEMU_BUILD_BUG_ON(sizeof(XSaveCETS) != 0x18);
 QEMU_BUILD_BUG_ON(sizeof(XSaveArchLBR) != 0x328);
 QEMU_BUILD_BUG_ON(sizeof(XSaveXTILECFG) != 0x40);
 QEMU_BUILD_BUG_ON(sizeof(XSaveXTILEDATA) != 0x2000);
+QEMU_BUILD_BUG_ON(sizeof(XSaveAPX) != 0x80);
 
 typedef struct ExtSaveArea {
     uint32_t offset, size;
@@ -1820,7 +1833,7 @@ typedef struct ExtSaveArea {
     const FeatureMask features[2];
 } ExtSaveArea;
 
-#define XSAVE_STATE_AREA_COUNT (XSTATE_XTILE_DATA_BIT + 1)
+#define XSAVE_STATE_AREA_COUNT (XSTATE_APX_BIT + 1)
 
 extern ExtSaveArea x86_ext_save_areas[XSAVE_STATE_AREA_COUNT];
 
-- 
2.34.1

[PATCH 2/5] i386/cpu: Cache EGPRs in CPUX86State

[Zhao Liu]

From: Zide Chen <zide.chen@intel.com>

Cache EGPR[16] in CPUX86State to store APX's EGPR value.

Tested-by: Xudong Hao <xudong.hao@intel.com>
Signed-off-by: Zide Chen <zide.chen@intel.com>
Co-developed-by: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.h          |  1 +
 target/i386/xsave_helper.c | 14 ++++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index bc7e16d6e6c1..48d4d7fcbb9c 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1969,6 +1969,7 @@ typedef struct CPUArchState {
 #ifdef TARGET_X86_64
     uint8_t xtilecfg[64];
     uint8_t xtiledata[8192];
+    uint64_t egprs[EGPR_NUM];
 #endif
 
     /* sysenter registers */
diff --git a/target/i386/xsave_helper.c b/target/i386/xsave_helper.c
index 996e9f3bfef5..2e9265045520 100644
--- a/target/i386/xsave_helper.c
+++ b/target/i386/xsave_helper.c
@@ -140,6 +140,13 @@ void x86_cpu_xsave_all_areas(X86CPU *cpu, void *buf, uint32_t buflen)
 
         memcpy(tiledata, &env->xtiledata, sizeof(env->xtiledata));
     }
+
+    e = &x86_ext_save_areas[XSTATE_APX_BIT];
+    if (e->size && e->offset && buflen) {
+        XSaveAPX *apx = buf + e->offset;
+
+        memcpy(apx, &env->egprs, sizeof(env->egprs));
+    }
 #endif
 }
 
@@ -275,5 +282,12 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen)
 
         memcpy(&env->xtiledata, tiledata, sizeof(env->xtiledata));
     }
+
+    e = &x86_ext_save_areas[XSTATE_APX_BIT];
+    if (e->size && e->offset) {
+        const XSaveAPX *apx = buf + e->offset;
+
+        memcpy(&env->egprs, apx, sizeof(env->egprs));
+    }
 #endif
 }
-- 
2.34.1

Re: [PATCH 2/5] i386/cpu: Cache EGPRs in CPUX86State

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 2706 bytes --]

On Tue, Nov 18, 2025 at 7:43 AM Zhao Liu <zhao1.liu@intel.com> wrote:
>
> From: Zide Chen <zide.chen@intel.com>
>
> Cache EGPR[16] in CPUX86State to store APX's EGPR value.

Please change regs[] to have 32 elements instead; see the attached
patch for a minimal starting point. You can use VMSTATE_SUB_ARRAY to
split their migration data in two parts. You'll have to create a
VMSTATE_UINTTL_SUB_ARRAY similar to VMSTATE_UINT64_SUB_ARRAY.

To support HMP you need to adjust target/i386/monitor.c and
target/i386/cpu-dump.c. Please make x86_cpu_dump_state print R16...R31
only if APX is enabled in CPUID.

Also, it would be best for the series to include gdb support. APX is
supported by gdb as a "coprocessor", the easiest way to do it is to
copy what riscv_cpu_register_gdb_regs_for_features() does for the FPU,
and copy https://github.com/intel/gdb/blob/master/gdb/features/i386/64bit-apx.xml
into QEMU's gdb-xml/ directory.

Paolo

> Tested-by: Xudong Hao <xudong.hao@intel.com>
> Signed-off-by: Zide Chen <zide.chen@intel.com>
> Co-developed-by: Zhao Liu <zhao1.liu@intel.com>
> Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
> ---
>  target/i386/cpu.h          |  1 +
>  target/i386/xsave_helper.c | 14 ++++++++++++++
>  2 files changed, 15 insertions(+)
>
> diff --git a/target/i386/cpu.h b/target/i386/cpu.h
> index bc7e16d6e6c1..48d4d7fcbb9c 100644
> --- a/target/i386/cpu.h
> +++ b/target/i386/cpu.h
> @@ -1969,6 +1969,7 @@ typedef struct CPUArchState {
>  #ifdef TARGET_X86_64
>      uint8_t xtilecfg[64];
>      uint8_t xtiledata[8192];
> +    uint64_t egprs[EGPR_NUM];
>  #endif
>
>      /* sysenter registers */
> diff --git a/target/i386/xsave_helper.c b/target/i386/xsave_helper.c
> index 996e9f3bfef5..2e9265045520 100644
> --- a/target/i386/xsave_helper.c
> +++ b/target/i386/xsave_helper.c
> @@ -140,6 +140,13 @@ void x86_cpu_xsave_all_areas(X86CPU *cpu, void *buf, uint32_t buflen)
>
>          memcpy(tiledata, &env->xtiledata, sizeof(env->xtiledata));
>      }
> +
> +    e = &x86_ext_save_areas[XSTATE_APX_BIT];
> +    if (e->size && e->offset && buflen) {
> +        XSaveAPX *apx = buf + e->offset;
> +
> +        memcpy(apx, &env->egprs, sizeof(env->egprs));
> +    }
>  #endif
>  }
>
> @@ -275,5 +282,12 @@ void x86_cpu_xrstor_all_areas(X86CPU *cpu, const void *buf, uint32_t buflen)
>
>          memcpy(&env->xtiledata, tiledata, sizeof(env->xtiledata));
>      }
> +
> +    e = &x86_ext_save_areas[XSTATE_APX_BIT];
> +    if (e->size && e->offset) {
> +        const XSaveAPX *apx = buf + e->offset;
> +
> +        memcpy(&env->egprs, apx, sizeof(env->egprs));
> +    }
>  #endif
>  }
> --
> 2.34.1
>

[-- Attachment #2: f.patch --]
[-- Type: text/x-patch, Size: 1669 bytes --]

diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index cee1f692a1c..0816f1dd22f 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1638,12 +1638,15 @@ typedef struct {
     uint64_t mask;
 } MTRRVar;
 
+#define CPU_NB_EREGS64 32
 #define CPU_NB_REGS64 16
 #define CPU_NB_REGS32 8
 
 #ifdef TARGET_X86_64
+#define CPU_NB_EREGS CPU_NB_EREGS64
 #define CPU_NB_REGS CPU_NB_REGS64
 #else
+#define CPU_NB_EREGS CPU_NB_REGS32
 #define CPU_NB_REGS CPU_NB_REGS32
 #endif
 
@@ -1845,7 +1848,7 @@ typedef struct CPUCaches {
 
 typedef struct CPUArchState {
     /* standard registers */
-    target_ulong regs[CPU_NB_REGS];
+    target_ulong regs[CPU_NB_EREGS];
     target_ulong eip;
     target_ulong eflags; /* eflags register. During CPU emulation, CC
                         flags and DF are set to zero because they are
@@ -1902,7 +1905,7 @@ typedef struct CPUArchState {
     float_status mmx_status; /* for 3DNow! float ops */
     float_status sse_status;
     uint32_t mxcsr;
-    ZMMReg xmm_regs[CPU_NB_REGS == 8 ? 8 : 32] QEMU_ALIGNED(16);
+    ZMMReg xmm_regs[CPU_NB_EREGS] QEMU_ALIGNED(16);
     ZMMReg xmm_t0 QEMU_ALIGNED(16);
     MMXReg mmx_t0;
 
diff --git a/target/i386/gdbstub.c b/target/i386/gdbstub.c
index 04c49e802d7..17d8e350834 100644
--- a/target/i386/gdbstub.c
+++ b/target/i386/gdbstub.c
@@ -125,6 +125,7 @@ int x86_cpu_gdb_read_register(CPUState *cs, GByteArray *mem_buf, int n)
        of a session. So if we're in 32-bit mode on a 64-bit cpu, still act
        as if we're on a 64-bit cpu. */
 
+    // TODO: APX registers
     if (n < CPU_NB_REGS) {
         if (TARGET_LONG_BITS == 64) {
             if (env->hflags & HF_CS64_MASK) {

Re: [PATCH 2/5] i386/cpu: Cache EGPRs in CPUX86State

[Zhao Liu]

On Tue, Nov 18, 2025 at 09:43:26AM +0100, Paolo Bonzini wrote:
> Date: Tue, 18 Nov 2025 09:43:26 +0100
> From: Paolo Bonzini <pbonzini@redhat.com>
> Subject: Re: [PATCH 2/5] i386/cpu: Cache EGPRs in CPUX86State
> 
> On Tue, Nov 18, 2025 at 7:43 AM Zhao Liu <zhao1.liu@intel.com> wrote:
> >
> > From: Zide Chen <zide.chen@intel.com>
> >
> > Cache EGPR[16] in CPUX86State to store APX's EGPR value.
> 
> Please change regs[] to have 32 elements instead; see the attached
> patch for a minimal starting point. You can use VMSTATE_SUB_ARRAY to
> split their migration data in two parts. You'll have to create a
> VMSTATE_UINTTL_SUB_ARRAY similar to VMSTATE_UINT64_SUB_ARRAY.

Thanks! VMSTATE_UINTTL_SUB_ARRAY is for target_ulong. I'll move EGPRs
to regs[].

> To support HMP you need to adjust target/i386/monitor.c and
> target/i386/cpu-dump.c. Please make x86_cpu_dump_state print R16...R31
> only if APX is enabled in CPUID.
> 
> Also, it would be best for the series to include gdb support. APX is
> supported by gdb as a "coprocessor", the easiest way to do it is to
> copy what riscv_cpu_register_gdb_regs_for_features() does for the FPU,
> and copy https://github.com/intel/gdb/blob/master/gdb/features/i386/64bit-apx.xml
> into QEMU's gdb-xml/ directory.

Good! Thank you for your guidance. I will add GDB support in next
version.

Regards,
Zhao

[PATCH 3/5] i386/cpu: Add APX migration support

[Zhao Liu]

From: Zide Chen <zide.chen@intel.com>

Add a VMStateDescription to migrate APX EGPRs.

Tested-by: Xudong Hao <xudong.hao@intel.com>
Signed-off-by: Zide Chen <zide.chen@intel.com>
Co-developed-by: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/machine.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/target/i386/machine.c b/target/i386/machine.c
index 265388f1fd36..84faa2f8f8d3 100644
--- a/target/i386/machine.c
+++ b/target/i386/machine.c
@@ -1744,6 +1744,27 @@ static const VMStateDescription vmstate_cet = {
     },
 };
 
+#ifdef TARGET_X86_64
+static bool apx_needed(void *opaque)
+{
+    X86CPU *cpu = opaque;
+    CPUX86State *env = &cpu->env;
+
+    return !!(env->features[FEAT_7_1_EDX] & CPUID_7_1_EDX_APX);
+}
+
+static const VMStateDescription vmstate_apx = {
+    .name = "cpu/apx",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = apx_needed,
+    .fields = (VMStateField[]) {
+        VMSTATE_UINT64_ARRAY(env.egprs, X86CPU, EGPR_NUM),
+        VMSTATE_END_OF_LIST()
+    }
+};
+#endif
+
 const VMStateDescription vmstate_x86_cpu = {
     .name = "cpu",
     .version_id = 12,
@@ -1895,6 +1916,9 @@ const VMStateDescription vmstate_x86_cpu = {
         &vmstate_triple_fault,
         &vmstate_pl0_ssp,
         &vmstate_cet,
+#ifdef TARGET_X86_64
+        &vmstate_apx,
+#endif
         NULL
     }
 };
-- 
2.34.1

[PATCH 4/5] i386/cpu: Support APX CPUIDs

[Zhao Liu]

APX is enumerated by CPUID.(EAX=0x7, ECX=1).EDX[21]. And this feature
bit also indicates the existence of dedicated CPUID leaf 0x29, called
the Intel APX Advanced Performance Extensions Leaf.

This new CPUID leaf now is populated with enumerations for a select
set of Intel APX sub-features.

CPUID.(EAX=0x29, ECX=0)
 - EAX
   * Maximum Subleaf CPUID.(EAX=0x29, ECX=0).EAX[31:0] = 0
 - EBX
   * Reserved CPUID.(EAX=0x29, ECX=0).EBX[31:1] = 0
   * APX_NCI_NDD_NF CPUID.(EAX=0x29, ECX=0).EBX[0:0] = 1, which
     enumerates the presence of New Conditional Instructions (NCIs),
     explicit New Data Destination (NDD) controls, and explicit Flags
     Suppression (NF) controls for select sets of EVEX-encoded Intel
     APX instructions (present in EVEX map=4, and EVEX map=2 0x0F38).
 - ECX
   * Reserved CPUID.(EAX=0x29, ECX=0).ECX[31:0] = 0
 - EDX
   * Reserved CPUID.(EAX=0x29, ECX=0).EDX[31:0] = 0

Note, APX_NCI_NDD_NF is documented as always enabled for Intel
processors since APX spec (revision v7.0). Now any Intel processor
that enumerates support for APX_F (CPUID.(EAX=0x7, ECX=1).EDX[21])
will also enumerate support for APX_NCI_NDD_NF.

Tested-by: Xudong Hao <xudong.hao@intel.com>
Co-developed-by: Zide Chen <zide.chen@intel.com>
Signed-off-by: Zide Chen <zide.chen@intel.com>
Co-developed-by: Peter Fang <peter.fang@intel.com>
Signed-off-by: Peter Fang <peter.fang@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.c | 40 +++++++++++++++++++++++++++++++++++++++-
 target/i386/cpu.h |  8 ++++++++
 2 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 16bc4b18266c..9cc553a86442 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1036,6 +1036,7 @@ void x86_cpu_vendor_words2str(char *dst, uint32_t vendor1,
 #define TCG_SGX_12_0_EBX_FEATURES 0
 #define TCG_SGX_12_1_EAX_FEATURES 0
 #define TCG_24_0_EBX_FEATURES 0
+#define TCG_29_0_EBX_FEATURES 0
 
 #if defined CONFIG_USER_ONLY
 #define CPUID_8000_0008_EBX_KERNEL_FEATURES (CPUID_8000_0008_EBX_IBPB | \
@@ -1301,7 +1302,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             "amx-complex", NULL, "avx-vnni-int16", NULL,
             NULL, NULL, "prefetchiti", NULL,
             NULL, NULL, NULL, "avx10",
-            NULL, NULL, NULL, NULL,
+            NULL, "apx", NULL, NULL,
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
         },
@@ -1345,6 +1346,25 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         },
         .tcg_features = TCG_24_0_EBX_FEATURES,
     },
+    [FEAT_29_0_EBX] = {
+        .type = CPUID_FEATURE_WORD,
+        .feat_names = {
+            "apx-nci-ndd-nf", NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, NULL,
+        },
+        .cpuid = {
+            .eax = 0x29,
+            .needs_ecx = true, .ecx = 0,
+            .reg = R_EBX,
+        },
+        .tcg_features = TCG_29_0_EBX_FEATURES,
+    },
     [FEAT_8000_0007_EDX] = {
         .type = CPUID_FEATURE_WORD,
         .feat_names = {
@@ -1996,6 +2016,10 @@ static FeatureDep feature_dependencies[] = {
         .from = { FEAT_7_1_EDX,             CPUID_7_1_EDX_AVX10 },
         .to = { FEAT_24_0_EBX,              ~0ull },
     },
+    {
+        .from = { FEAT_7_1_EDX,             CPUID_7_1_EDX_APX },
+        .to = { FEAT_29_0_EBX,              ~0ull },
+    },
 };
 
 typedef struct X86RegisterInfo32 {
@@ -8411,6 +8435,15 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
         }
         break;
     }
+    case 0x29:
+        *eax = 0;
+        *ebx = 0;
+        *ecx = 0;
+        *edx = 0;
+        if ((env->features[FEAT_7_1_EDX] & CPUID_7_1_EDX_APX) && count == 0) {
+            *ebx = env->features[FEAT_29_0_EBX];
+        }
+        break;
     case 0x40000000:
         /*
          * CPUID code in kvm_arch_init_vcpu() ignores stuff
@@ -9190,6 +9223,11 @@ void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
             x86_cpu_adjust_level(cpu, &env->cpuid_min_level, 0x24);
         }
 
+        /* Advanced Performance Extensions (APX) requires CPUID[0x29] */
+        if (env->features[FEAT_7_1_EDX] & CPUID_7_1_EDX_APX) {
+            x86_cpu_adjust_level(cpu, &env->cpuid_min_level, 0x29);
+        }
+
         /* SVM requires CPUID[0x8000000A] */
         if (env->features[FEAT_8000_0001_ECX] & CPUID_EXT3_SVM) {
             x86_cpu_adjust_level(cpu, &env->cpuid_min_xlevel, 0x8000000A);
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 48d4d7fcbb9c..d15a89f8c72e 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -697,6 +697,7 @@ typedef enum FeatureWord {
     FEAT_7_1_EDX,       /* CPUID[EAX=7,ECX=1].EDX */
     FEAT_7_2_EDX,       /* CPUID[EAX=7,ECX=2].EDX */
     FEAT_24_0_EBX,      /* CPUID[EAX=0x24,ECX=0].EBX */
+    FEAT_29_0_EBX,      /* CPUID[EAX=0x29,ECX=0].EBX */
     FEATURE_WORDS,
 } FeatureWord;
 
@@ -1079,6 +1080,13 @@ uint64_t x86_cpu_get_supported_feature_word(X86CPU *cpu, FeatureWord w);
                                          CPUID_24_0_EBX_AVX10_256 | \
                                          CPUID_24_0_EBX_AVX10_512)
 
+/*
+ * New Conditional Instructions (NCIs), explicit New Data Destination (NDD)
+ * controls, and explicit Flags Suppression (NF) controls for select sets of
+ * EVEX-encoded Intel APX instructions
+ */
+#define CPUID_29_0_EBX_APX_NCI_NDD_NF         (1U << 0)
+
 /* RAS Features */
 #define CPUID_8000_0007_EBX_OVERFLOW_RECOV    (1U << 0)
 #define CPUID_8000_0007_EBX_SUCCOR      (1U << 1)
-- 
2.34.1

Re: [PATCH 4/5] i386/cpu: Support APX CPUIDs

[Paolo Bonzini]

On Tue, Nov 18, 2025 at 7:36 AM Zhao Liu <zhao1.liu@intel.com> wrote:
>
> APX is enumerated by CPUID.(EAX=0x7, ECX=1).EDX[21]. And this feature
> bit also indicates the existence of dedicated CPUID leaf 0x29, called
> the Intel APX Advanced Performance Extensions Leaf.
>
> This new CPUID leaf now is populated with enumerations for a select
> set of Intel APX sub-features.
>
> CPUID.(EAX=0x29, ECX=0)
>  - EAX
>    * Maximum Subleaf CPUID.(EAX=0x29, ECX=0).EAX[31:0] = 0
>  - EBX
>    * Reserved CPUID.(EAX=0x29, ECX=0).EBX[31:1] = 0
>    * APX_NCI_NDD_NF CPUID.(EAX=0x29, ECX=0).EBX[0:0] = 1, which
>      enumerates the presence of New Conditional Instructions (NCIs),
>      explicit New Data Destination (NDD) controls, and explicit Flags
>      Suppression (NF) controls for select sets of EVEX-encoded Intel
>      APX instructions (present in EVEX map=4, and EVEX map=2 0x0F38).
>  - ECX
>    * Reserved CPUID.(EAX=0x29, ECX=0).ECX[31:0] = 0
>  - EDX
>    * Reserved CPUID.(EAX=0x29, ECX=0).EDX[31:0] = 0
>
> Note, APX_NCI_NDD_NF is documented as always enabled for Intel
> processors since APX spec (revision v7.0). Now any Intel processor
> that enumerates support for APX_F (CPUID.(EAX=0x7, ECX=1).EDX[21])
> will also enumerate support for APX_NCI_NDD_NF.

Please just make the new leaf have constant values based on just
APX_F. We'll add the optional NCI/NDD/NF support if needed, i.e.
never. :)

Paolo

Re: [PATCH 4/5] i386/cpu: Support APX CPUIDs

[Zhao Liu]

> Please just make the new leaf have constant values based on just
> APX_F. We'll add the optional NCI/NDD/NF support if needed, i.e.
> never. :)

Maybe not never?

> > Note, APX_NCI_NDD_NF is documented as always enabled for Intel
> > processors since APX spec (revision v7.0). Now any Intel processor
> > that enumerates support for APX_F (CPUID.(EAX=0x7, ECX=1).EDX[21])
> > will also enumerate support for APX_NCI_NDD_NF.

This sentence (from APX spec rev.7) emphasizes the “Intel” vendor,
and its primary goal was to address and explain compatibility concern
for pre-enabling work based on APX spec v6. Prior to v7, APX included
NCI_NDD_NF by default, but this feature has now been separated from
basic APX and requires explicit checking CPUID bit.

x86 ecosystem advisory group has aligned on APX so it may be possible
for other x86 vendors to implement APX without NCI_NDD_NF and this still
match with the APX spec.

If we default to setting this NCI_NDD_NF bit for APX, then in the future
when we run into other vendors that don't support this feature, we'll not
only have to make it optional again, but we'll also need to do fixes
similar to the ARCH_CAPABILITIES situation - checking vendors, fixing
compatibility issues, and all that stuff.

Therefore, compared to default setting to constant, I think the optional
NCI_NDD_NF now not only aligns with arch spec but also prevents future
compatibility issues. :)

Thanks,
Zhao

Re: [PATCH 4/5] i386/cpu: Support APX CPUIDs

[Paolo Bonzini]

[-- Attachment #1: Type: text/plain, Size: 1603 bytes --]

Il mer 19 nov 2025, 08:12 Zhao Liu <zhao1.liu@intel.com> ha scritto:

> > > Note, APX_NCI_NDD_NF is documented as always enabled for Intel
> > > processors since APX spec (revision v7.0). Now any Intel processor
> > > that enumerates support for APX_F (CPUID.(EAX=0x7, ECX=1).EDX[21])
> > > will also enumerate support for APX_NCI_NDD_NF.
>
> This sentence (from APX spec rev.7) emphasizes the “Intel” vendor,
> and its primary goal was to address and explain compatibility concern
> for pre-enabling work based on APX spec v6. Prior to v7, APX included
> NCI_NDD_NF by default, but this feature has now been separated from
> basic APX and requires explicit checking CPUID bit.
>
> x86 ecosystem advisory group has aligned on APX so it may be possible
> for other x86 vendors to implement APX without NCI_NDD_NF and this still
> match with the APX spec.
>

Oh, I was not aware of that. It is really ugly but I guess that's not our
choice. :/ If QEMU ever implements APX emulation it will have NC/NDD/NF
though...

Paolo


> If we default to setting this NCI_NDD_NF bit for APX, then in the future
> when we run into other vendors that don't support this feature, we'll not
> only have to make it optional again, but we'll also need to do fixes
> similar to the ARCH_CAPABILITIES situation - checking vendors, fixing
> compatibility issues, and all that stuff.
>
> Therefore, compared to default setting to constant, I think the optional
> NCI_NDD_NF now not only aligns with arch spec but also prevents future
> compatibility issues. :)
>
> Thanks,
> Zhao
>
>

[-- Attachment #2: Type: text/html, Size: 2336 bytes --]

Re: [PATCH 4/5] i386/cpu: Support APX CPUIDs

[Florian Weimer]

* Zhao Liu:

>> Please just make the new leaf have constant values based on just
>> APX_F. We'll add the optional NCI/NDD/NF support if needed, i.e.
>> never. :)
>
> Maybe not never?
>
>> > Note, APX_NCI_NDD_NF is documented as always enabled for Intel
>> > processors since APX spec (revision v7.0). Now any Intel processor
>> > that enumerates support for APX_F (CPUID.(EAX=0x7, ECX=1).EDX[21])
>> > will also enumerate support for APX_NCI_NDD_NF.
>
> This sentence (from APX spec rev.7) emphasizes the “Intel” vendor,
> and its primary goal was to address and explain compatibility concern
> for pre-enabling work based on APX spec v6. Prior to v7, APX included
> NCI_NDD_NF by default, but this feature has now been separated from
> basic APX and requires explicit checking CPUID bit.
>
> x86 ecosystem advisory group has aligned on APX so it may be possible
> for other x86 vendors to implement APX without NCI_NDD_NF and this still
> match with the APX spec.

Well yes, but I doubt that the ecosystem will produce binaries
specialized for APX *without* NDD.  It's fine to enumerate it
separately, but that doesn't have any immediate consequences.  GCC makes
it rather hard to build for APX without NDD, for example.  At least more
difficult than building for AVX-512F without AVX-512VL.

I just don't think software vendors are enthusiastic about having to
create and support not one, but two builds for APX.  If NDD is optional
in practice, it will not be possible to use it except for run-time
generated code and perhaps very targeted optimizations because that
single extra APX will just not use NDD.

I feel like there has been a misunderstanding somewhere.

(sorry for off-topic)

Thanks,
Florian

Re: [PATCH 4/5] i386/cpu: Support APX CPUIDs

[Paolo Bonzini]

On Wed, Nov 19, 2025 at 7:04 PM Florian Weimer <fweimer@redhat.com> wrote:
> This sentence (from APX spec rev.7) emphasizes the “Intel” vendor,
> > and its primary goal was to address and explain compatibility concern
> > for pre-enabling work based on APX spec v6. Prior to v7, APX included
> > NCI_NDD_NF by default, but this feature has now been separated from
> > basic APX and requires explicit checking CPUID bit.
> >
> > x86 ecosystem advisory group has aligned on APX so it may be possible
> > for other x86 vendors to implement APX without NCI_NDD_NF and this still
> > match with the APX spec.
>
> Well yes, but I doubt that the ecosystem will produce binaries
> specialized for APX *without* NDD.  It's fine to enumerate it
> separately, but that doesn't have any immediate consequences.  GCC makes
> it rather hard to build for APX without NDD, for example.  At least more
> difficult than building for AVX-512F without AVX-512VL.
>
> I just don't think software vendors are enthusiastic about having to
> create and support not one, but two builds for APX.  If NDD is optional
> in practice, it will not be possible to use it except for run-time
> generated code and perhaps very targeted optimizations because that
> single extra APX will just not use NDD.
>
> I feel like there has been a misunderstanding somewhere.

I totally agree and I think this addition to APX was very misguided,
no matter who proposed it.

However, for virtualization we probably should include this code no
matter how much I dislike it, because having to add the bit later
retroactively would be worse.

Paolo

[PATCH 5/5] i386/cpu: Mark apx xstate as migratable

[Zhao Liu]

Apx xstate is user xstate. The related registers are cached in
X86CPUState. And there's a vmsd "vmstate_apx" to migrate these
registers.

Thus, it's safe to mark it as migratable.

Tested-by: Xudong Hao <xudong.hao@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 9cc553a86442..f703b1478d71 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1544,7 +1544,8 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         .migratable_flags = XSTATE_FP_MASK | XSTATE_SSE_MASK |
             XSTATE_YMM_MASK | XSTATE_BNDREGS_MASK | XSTATE_BNDCSR_MASK |
             XSTATE_OPMASK_MASK | XSTATE_ZMM_Hi256_MASK | XSTATE_Hi16_ZMM_MASK |
-            XSTATE_PKRU_MASK | XSTATE_XTILE_CFG_MASK | XSTATE_XTILE_DATA_MASK,
+            XSTATE_PKRU_MASK | XSTATE_XTILE_CFG_MASK | XSTATE_XTILE_DATA_MASK |
+            XSTATE_APX_MASK,
     },
     [FEAT_XSAVE_XCR0_HI] = {
         .type = CPUID_FEATURE_WORD,
-- 
2.34.1

Re: [PATCH 0/5] i386/cpu: Support APX for KVM

[Paolo Bonzini]

On Tue, Nov 18, 2025 at 7:36 AM Zhao Liu <zhao1.liu@intel.com> wrote:
>
> Hi,
>
> This series adds APX (Advanced Performance Extensions) support in QEMU
> to enable APX in Guest based on KVM.

Thanks for sending this out, I left some comments on the patch that
adds EGPRs but otherwise it's pretty simple---good.

Paolo

> This series is based on CET v4:
>
> https://lore.kernel.org/qemu-devel/20251118034231.704240-1-zhao1.liu@intel.com/
>
> And you can also find the code here:
>
> https://gitlab.com/zhao.liu/qemu/-/commits/i386-all-for-dmr-v1.1-11-17-2025
>
> The patches for KVM side can be found at:
>
> https://lore.kernel.org/kvm/20251110180131.28264-1-chang.seok.bae@intel.com/
>
>
> Thanks for your review!
>
>
> Overview
> ========
>
> Intel Advanced Performance Extensions (Intel APX) expands the Intel 64
> instruction set architecture with access to more registers (16
> additional general-purpose registers (GPRs) R16–R31) and adds various
> new features that improve general-purpose performance. The extensions
> are designed to provide efficient performance gains across a variety of
> workloads without significantly increasing silicon area or power
> consumption of the core.
>
> APX spec link (rev.07) is:
> https://cdrdv2.intel.com/v1/dl/getContent/861610
>
> At QEMU side, the enabling work mainly includes two parts:
>
> 1. save/restore/migrate the xstate of APX.
>    * APX xstate is a user xstate, but it reuses MPX xstate area in
>      un-compacted XSAVE buffer.
>    * To address this, QEMU will reject both APX and MPX if their CPUID
>      feature bits are set at the same (in Patch 1).
>
> 2. add related CPUIDs support in feature words.
>
> Thanks and Best Regards,
> Zhao
> ---
> Zhao Liu (2):
>   i386/cpu: Support APX CPUIDs
>   i386/cpu: Mark apx xstate as migratable
>
> Zide Chen (3):
>   i386/cpu: Add APX EGPRs into xsave area
>   i386/cpu: Cache EGPRs in CPUX86State
>   i386/cpu: Add APX migration support
>
>  target/i386/cpu.c          | 68 ++++++++++++++++++++++++++++++++++++--
>  target/i386/cpu.h          | 26 +++++++++++++--
>  target/i386/machine.c      | 24 ++++++++++++++
>  target/i386/xsave_helper.c | 14 ++++++++
>  4 files changed, 128 insertions(+), 4 deletions(-)
>
> --
> 2.34.1
>