[RFC 0/6] Intel Thread Director Virtualization Support in QEMU

[RFC 0/6] Intel Thread Director Virtualization Support in QEMU

[Zhao Liu]

From: Zhao Liu <zhao1.liu@intel.com>

Hi list,

This is our refreshed RFC to support our ITD virtualization patch
series [1] in KVM, and bases on bd2e12310b18 ("Merge tag
'qga-pull-2024-01-30' of https://github.com/kostyanf14/qemu into
staging").

ITD is Intel's client specific feature to optimize scheduling on Intel
hybrid platforms. Though this feature depends on hybrid topology
details, in our parctice on Win11 Guest, ITD works with hyrbid topolohy
and CPU affinity can achieve the most performance improvement in Win11
Guest (for example, on i9-13900K, up to 14%+ improvement on
3DMARK). More data or details, can be found in [1]. Thus, the ITD for
Win11 is also a typical use case of hybrid topology.


Welcome your feedback!


1. Background and Motivation
============================

ITD allows the hardware to provide scheduling hints to the OS to help
optimize scheduling performance, and under the Intel hybrid
architecture, since Core and Atom have different capabilities
(performance, energy effency, etc.),  scheduling based on hardware
hints can take full advantage of this hybrid architecture. This is also
the most ideal scheduling model for intel hybrid architecture.

Therefore, we want to virtualize the ITD feature so that ITD can benefit
performance of the virtual machines on the hybrid machines as well.

Currently, our ITD virtualization is a software virtualization solution.


2. Introduction to HFI and ITD
==============================

Intel provides Hardware Feedback Interface (HFI) feature to allow
hardware to provide guidance to the OS scheduler to perform optimal
workload scheduling through a hardware feedback interface structure in
memory [2]. This hfi structure is called HFI table.

As for now, the guidance includes performance and energy enficency hints,
and it could update via thermal interrupt as the actual operating
conditions of the processor change during run time.

And Intel Thread Director (ITD) feature extends the HFI to provide
performance and energy efficiency data for advanced classes of
instructions.

The virtual HFI table is maintained in KVM, and for QEMU, we just need
to handle HFI/ITD/HRESET (and their dependent features: ACPI, TM and
PTS) related CPUIDs and MSRs.


3. Package level MSRs handling
==============================

PTS, HFI and ITD are all have package level features, such as package
level MSRs and package level HFI tables. But since KVM hasn't
support msr-topology and it just handle these package-level MSRs and
HFI table at VM level, in order to avoid potential contention problems
caused by multiple virtual-packages, we restrict VMs to be able to
enable PTC/HFI/ITD iff there's only 1 package (and only 1 die for
ITD/HFI).


4. HFI/ITD related info in CPUID
================================

KVM provides some basic HFI info in CPUID.0x06 leaf, which is associated
with the virtual HFI table in KVM.

QEMU should configure HFI table index for each vCPU. Here we set the HFI
table index to vCPU index so that different vCPUs have different HFI
entries to avoid unnecessary competition problems.


5. Compatibility issues
=======================

HFI is supported in both server (SPR) and client (ADL/RPL/MTL) platform
products while ITD is the client specific feature.

For client platform, ITD (with HFI) could be enabled in Guest to improve
scheduling, but for server platform, HFI (without ITD) is only useful
on Host and Guest doesn't need it.

To simplify the enabling logic and avoid impacting the common topology
of the Guest, we set PTS, HFI, and ITD as feature bits that are not
automatically enabled.

Only when the user actively specifies these features, QEMU will check
and decide whether to enable them based on the topology constraints and
the ITD constraints.


6. New option "enable-itd"
============================

ITD-related features include PTS, HFI, ITD, and HRESET.

To make it easier for users to enable ITD for Guest without specifying
the above feature bits one by one, we provide a new option "enable-itd"
to set the above feature bits for Guest all at once.

"enable-itd" does not guarantee that ITD will be enabled for Guest.
The success of enabling ITD for guest depends on topology constraints,
platform support, etc., which are checked in QEMU.


7. Patch Summary
================

Patch 1: Add support save/load for ACPI feature related thermal MSRs
         since ACPI feature CPUID has been added in QEMU.
Patch 2: Add support for PTS (package) thermal MSRs and its CPUID
Patch 3: Add support for HFI MSRs and its CPUID
Patch 4: Add support ITD CPUID and MSR_IA32_HW_FEEDBACK_THREAD_CONFIG.
Patch 5: Add support HRESET CPUID and MSR_IA32_HW_HRESET_ENABLE.
Patch 6: Add "enable-itd" to help user set ITD related feature bits.

# 8. References

[1]: KVM RFC: [RFC 00/26] Intel Thread Director Virtualization
     https://lore.kernel.org/kvm/20240203091214.411862-1-zhao1.liu@linux.intel.com/T/#t
[2]: SDM, vol. 3B, section 15.6 HARDWARE FEEDBACK INTERFACE AND INTEL
     THREAD DIRECTOR

Thanks and Best Regards,
Zhao
---
Zhao Liu (2):
  target/i386: Add support for Intel Thread Director feature
  i386: Add a new property to set ITD related feature bits for Guest

Zhuocheng Ding (4):
  target/i386: Add support for save/load of ACPI thermal MSRs
  target/i386: Add support for Package Thermal Management feature
  target/i386: Add support for Hardware Feedback Interface feature
  target/i386: Add support for HRESET feature

 target/i386/cpu.c     | 108 ++++++++++++++++++++++++++++++++++++++++--
 target/i386/cpu.h     |  37 +++++++++++++++
 target/i386/kvm/kvm.c |  84 ++++++++++++++++++++++++++++++++
 3 files changed, 225 insertions(+), 4 deletions(-)

-- 
2.34.1

[RFC 1/6] target/i386: Add support for save/load of ACPI thermal MSRs

[Zhao Liu]

From: Zhuocheng Ding <zhuocheng.ding@intel.com>

The CPUID_ACPI (CPUID.0x01.edx[bit 22]) feature bit has been
introduced as the TCG feature. Currently, based on KVM's ACPI emulation,
add related ACPI support in QEMU.

From SDM [1], ACPI feature means:

"The ACPI flag (bit 22) of the CPUID feature flags indicates the
presence of the IA32_THERM_STATUS, IA32_THERM_INTERRUPT,
IA32_CLOCK_MODULATION MSRs, and the xAPIC thermal LVT entry."

With the emulation of ACPI in KVM, add the support for save/load of ACPI
thermal MSRs: MSR_IA32_THERM_CONTROL, MSR_IA32_THERM_INTERRUPT and
MSR_IA32_THERM_STATUS.

[1]: SDM, vol. 3B, section 15.8.4.1, Detection of Software Controlled
     Clock Modulation Extension.

Tested-by: Yanting Jiang <yanting.jiang@intel.com>
Signed-off-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Co-developed-by: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.h     |  9 +++++++++
 target/i386/kvm/kvm.c | 25 +++++++++++++++++++++++++
 2 files changed, 34 insertions(+)

diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 7f0786e8b98f..e453b3f010e2 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -526,6 +526,10 @@ typedef enum X86Seg {
 #define MSR_IA32_XSS                    0x00000da0
 #define MSR_IA32_UMWAIT_CONTROL         0xe1
 
+#define MSR_IA32_THERM_CONTROL          0x0000019a
+#define MSR_IA32_THERM_INTERRUPT        0x0000019b
+#define MSR_IA32_THERM_STATUS           0x0000019c
+
 #define MSR_IA32_VMX_BASIC              0x00000480
 #define MSR_IA32_VMX_PINBASED_CTLS      0x00000481
 #define MSR_IA32_VMX_PROCBASED_CTLS     0x00000482
@@ -1758,6 +1762,11 @@ typedef struct CPUArchState {
     uint64_t msr_lbr_depth;
     LBREntry lbr_records[ARCH_LBR_NR_ENTRIES];
 
+    /* Per-VCPU thermal MSRs */
+    uint64_t therm_control;
+    uint64_t therm_interrupt;
+    uint64_t therm_status;
+
     /* exception/interrupt handling */
     int error_code;
     int exception_is_int;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 76a66246eb72..3bf57b35bfcd 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -138,6 +138,7 @@ static bool has_msr_ucode_rev;
 static bool has_msr_vmx_procbased_ctls2;
 static bool has_msr_perf_capabs;
 static bool has_msr_pkrs;
+static bool has_msr_therm;
 
 static uint32_t has_architectural_pmu_version;
 static uint32_t num_architectural_pmu_gp_counters;
@@ -2455,6 +2456,11 @@ static int kvm_get_supported_msrs(KVMState *s)
             case MSR_IA32_PKRS:
                 has_msr_pkrs = true;
                 break;
+            case MSR_IA32_THERM_CONTROL:
+            case MSR_IA32_THERM_INTERRUPT:
+            case MSR_IA32_THERM_STATUS:
+                has_msr_therm = true;
+                break;
             }
         }
     }
@@ -3302,6 +3308,11 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
     if (has_msr_virt_ssbd) {
         kvm_msr_entry_add(cpu, MSR_VIRT_SSBD, env->virt_ssbd);
     }
+    if (has_msr_therm) {
+        kvm_msr_entry_add(cpu, MSR_IA32_THERM_CONTROL, env->therm_control);
+        kvm_msr_entry_add(cpu, MSR_IA32_THERM_INTERRUPT, env->therm_interrupt);
+        kvm_msr_entry_add(cpu, MSR_IA32_THERM_STATUS, env->therm_status);
+    }
 
 #ifdef TARGET_X86_64
     if (lm_capable_kernel) {
@@ -3774,6 +3785,11 @@ static int kvm_get_msrs(X86CPU *cpu)
         kvm_msr_entry_add(cpu, MSR_IA32_TSC, 0);
         env->tsc_valid = !runstate_is_running();
     }
+    if (has_msr_therm) {
+        kvm_msr_entry_add(cpu, MSR_IA32_THERM_CONTROL, 0);
+        kvm_msr_entry_add(cpu, MSR_IA32_THERM_INTERRUPT, 0);
+        kvm_msr_entry_add(cpu, MSR_IA32_THERM_STATUS, 0);
+    }
 
 #ifdef TARGET_X86_64
     if (lm_capable_kernel) {
@@ -4255,6 +4271,15 @@ static int kvm_get_msrs(X86CPU *cpu)
         case MSR_ARCH_LBR_INFO_0 ... MSR_ARCH_LBR_INFO_0 + 31:
             env->lbr_records[index - MSR_ARCH_LBR_INFO_0].info = msrs[i].data;
             break;
+        case MSR_IA32_THERM_CONTROL:
+            env->therm_control = msrs[i].data;
+            break;
+        case MSR_IA32_THERM_INTERRUPT:
+            env->therm_interrupt = msrs[i].data;
+            break;
+        case MSR_IA32_THERM_STATUS:
+            env->therm_status = msrs[i].data;
+            break;
         }
     }
 
-- 
2.34.1

[RFC 2/6] target/i386: Add support for Package Thermal Management feature

[Zhao Liu]

From: Zhuocheng Ding <zhuocheng.ding@intel.com>

PTS feature (Package Thermal Management) is a dependency of ITD.

PTS provides 2 package level MSRs: MSR_IA32_PACKAGE_THERM_STATUS and
MSR_IA32_PACKAGE_THERM_INTERRUPT.

They're emulated in KVM, but currently KVM hasn't supported msr
topology.

Thus the emulation of these 2 package-level MSRs are only supported at
the whole VM-level, and all vCPUs share these two MSRs, so that the
emulation of these two MSRs does not distinguish between the different
virtual-packages.

In order to avoid potential contention problems caused by multiple
virtual-packages, add the following restrictions to the PTS feature bit:

1. Mark PTS as no_autoenable_flags and it won't be enabled by default.
2. PTS can't be enabled for the case with multiple packages.
3. PTS can't be enabled if ITD is not set for Guest, since currently PTS
   is only used to help enable ITD in virtualization scenario.

Additionally, add save/load support for 2 PTS related MSRs.

Tested-by: Yanting Jiang <yanting.jiang@intel.com>
Co-developed-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Signed-off-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.c     | 22 +++++++++++++++++++++-
 target/i386/cpu.h     | 13 +++++++++++++
 target/i386/kvm/kvm.c | 24 ++++++++++++++++++++++++
 3 files changed, 58 insertions(+), 1 deletion(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 03822d9ba8ee..e772d35d9403 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1114,7 +1114,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         .type = CPUID_FEATURE_WORD,
         .feat_names = {
             NULL, NULL, "arat", NULL,
-            NULL, NULL, NULL, NULL,
+            NULL, NULL, "pts", NULL,
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
@@ -1124,6 +1124,11 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         },
         .cpuid = { .eax = 6, .reg = R_EAX, },
         .tcg_features = TCG_6_EAX_FEATURES,
+        /*
+         * PTS shouldn't be enabled by default since it has
+         * requirement for cpu topology.
+         */
+        .no_autoenable_flags = CPUID_6_EAX_PTS,
     },
     [FEAT_XSAVE_XCR0_LO] = {
         .type = CPUID_FEATURE_WORD,
@@ -7424,6 +7429,21 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
             goto out;
         }
     }
+
+    if (env->features[FEAT_6_EAX] & CPUID_6_EAX_PTS && ms->smp.sockets > 1) {
+        error_setg(errp,
+                   "PTS currently only supports 1 package, "
+                   "please set by \"-smp ...,sockets=1\"");
+        return;
+    }
+
+    if (env->features[FEAT_6_EAX] & CPUID_6_EAX_PTS &&
+        !(env->features[FEAT_6_EAX] & CPUID_6_EAX_ITD)) {
+        error_setg(errp,
+                   "In the absence of ITD, Guest does "
+                   "not need PTS");
+        return;
+    }
 #endif
 
     mce_init(cpu);
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index e453b3f010e2..a8c247b2ef89 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -530,6 +530,9 @@ typedef enum X86Seg {
 #define MSR_IA32_THERM_INTERRUPT        0x0000019b
 #define MSR_IA32_THERM_STATUS           0x0000019c
 
+#define MSR_IA32_PACKAGE_THERM_STATUS    0x000001b1
+#define MSR_IA32_PACKAGE_THERM_INTERRUPT 0x000001b2
+
 #define MSR_IA32_VMX_BASIC              0x00000480
 #define MSR_IA32_VMX_PINBASED_CTLS      0x00000481
 #define MSR_IA32_VMX_PROCBASED_CTLS     0x00000482
@@ -982,6 +985,8 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,
 #define CPUID_XSAVE_XSAVES     (1U << 3)
 
 #define CPUID_6_EAX_ARAT       (1U << 2)
+#define CPUID_6_EAX_PTS        (1U << 6)
+#define CPUID_6_EAX_ITD        (1U << 23)
 
 /* CPUID[0x80000007].EDX flags: */
 #define CPUID_APM_INVTSC       (1U << 8)
@@ -1767,6 +1772,14 @@ typedef struct CPUArchState {
     uint64_t therm_interrupt;
     uint64_t therm_status;
 
+    /*
+     * Although these are package level MSRs, for the PTS feature, we
+     * temporarily limit it to be enabled for only 1 package, so the value
+     * of each vCPU is same and it's enough to support the save/load.
+     */
+    uint64_t pkg_therm_interrupt;
+    uint64_t pkg_therm_status;
+
     /* exception/interrupt handling */
     int error_code;
     int exception_is_int;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 3bf57b35bfcd..258591535fd5 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -139,6 +139,7 @@ static bool has_msr_vmx_procbased_ctls2;
 static bool has_msr_perf_capabs;
 static bool has_msr_pkrs;
 static bool has_msr_therm;
+static bool has_msr_pkg_therm;
 
 static uint32_t has_architectural_pmu_version;
 static uint32_t num_architectural_pmu_gp_counters;
@@ -2461,6 +2462,10 @@ static int kvm_get_supported_msrs(KVMState *s)
             case MSR_IA32_THERM_STATUS:
                 has_msr_therm = true;
                 break;
+            case MSR_IA32_PACKAGE_THERM_STATUS:
+            case MSR_IA32_PACKAGE_THERM_INTERRUPT:
+                has_msr_pkg_therm = true;
+                break;
             }
         }
     }
@@ -3313,6 +3318,15 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
         kvm_msr_entry_add(cpu, MSR_IA32_THERM_INTERRUPT, env->therm_interrupt);
         kvm_msr_entry_add(cpu, MSR_IA32_THERM_STATUS, env->therm_status);
     }
+    /* Only sync package level MSRs to KVM on the first cpu */
+    if (current_cpu == first_cpu) {
+        if (has_msr_pkg_therm) {
+            kvm_msr_entry_add(cpu, MSR_IA32_PACKAGE_THERM_STATUS,
+                              env->therm_control);
+            kvm_msr_entry_add(cpu, MSR_IA32_PACKAGE_THERM_INTERRUPT,
+                              env->therm_interrupt);
+        }
+    }
 
 #ifdef TARGET_X86_64
     if (lm_capable_kernel) {
@@ -3790,6 +3804,10 @@ static int kvm_get_msrs(X86CPU *cpu)
         kvm_msr_entry_add(cpu, MSR_IA32_THERM_INTERRUPT, 0);
         kvm_msr_entry_add(cpu, MSR_IA32_THERM_STATUS, 0);
     }
+    if (has_msr_pkg_therm) {
+        kvm_msr_entry_add(cpu, MSR_IA32_PACKAGE_THERM_STATUS, 0);
+        kvm_msr_entry_add(cpu, MSR_IA32_PACKAGE_THERM_INTERRUPT, 0);
+    }
 
 #ifdef TARGET_X86_64
     if (lm_capable_kernel) {
@@ -4280,6 +4298,12 @@ static int kvm_get_msrs(X86CPU *cpu)
         case MSR_IA32_THERM_STATUS:
             env->therm_status = msrs[i].data;
             break;
+        case MSR_IA32_PACKAGE_THERM_STATUS:
+            env->pkg_therm_status = msrs[i].data;
+            break;
+        case MSR_IA32_PACKAGE_THERM_INTERRUPT:
+            env->pkg_therm_interrupt = msrs[i].data;
+            break;
         }
     }
 
-- 
2.34.1

[RFC 3/6] target/i386: Add support for Hardware Feedback Interface feature

[Zhao Liu]

From: Zhuocheng Ding <zhuocheng.ding@intel.com>

Hardware Feedback Interface (HFI) is a basic dependency of ITD.

HFI is the feature to allow hardware to provide guidance to the
Operating System (OS) scheduler to perform optimal workload scheduling
through a hardware feedback interface structure in memory [1]. Currently
in virtualization scenario it is used to help enable ITD.

HFI feature provides the basic HFI information in CPUID.0x06:
* 0x06.eax[bit 19]: HFI feature bit
* 0x06.ecx[bits 08-15]: Number of HFI/ITD supported classes
* 0x06.edx[bits 00-07]: Bitmap of supported HFI capabilities
* 0x06.edx[bits 08-11]: Enumerates the size of the HFI table in number
                        of 4 KB pages
* 0x06.edx[bits 16-31]: HFI table index of processor

Here the basic information is generated by KVM based on the virtual HFI
table that can be emulated, and QEMU needs to specify the HFI table
index for each vCPU.

HFI feature also provides 2 package level MSRs:
MSR_IA32_HW_FEEDBACK_CONFIG and MSR_IA32_HW_FEEDBACK_PTR.

They're emulated in KVM, but currently KVM hasn't supported msr
topology.

Thus, like PTS MSRs, the emulation of these 2 package-level HFI MSRs are
only supported at the whole VM-level, and all vCPUs share these two
MSRs, so that the emulation of these two MSRs does not distinguish
between the different virtual-packages.

And HFI driver maintains per die HFI instances, so this can also cause
problems with access to HFI MSRs when multiple virtual-dies exist.

In order to avoid potential contention problems caused by multiple
virtual-packages/dies, add the following restrictions to the HFI feature
bit:

1. Mark HFI as no_autoenable_flags and it won't be enabled by default.
2. HFI can't be enabled for the case with multiple packages/dies.
3. HFI can't be enabled if ITD is not set for Guest, since currently HFI
   is only used to help enable ITD in virtualization scenario.

HFI feature depends on ACPI, TM and PTS, also add their dependencies.

Additionally, add save/load support for 2 HFI related MSRs.

[1]: SDM, vol. 3B, section 15.6 HARDWARE FEEDBACK INTERFACE AND INTEL
     THREAD DIRECTOR

Tested-by: Yanting Jiang <yanting.jiang@intel.com>
Co-developed-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Signed-off-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.c     | 49 ++++++++++++++++++++++++++++++++++++++-----
 target/i386/cpu.h     |  8 ++++++-
 target/i386/kvm/kvm.c | 21 +++++++++++++++++++
 3 files changed, 72 insertions(+), 6 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index e772d35d9403..e3eb361436c9 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1117,7 +1117,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             NULL, NULL, "pts", NULL,
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, "hfi",
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
@@ -1125,10 +1125,10 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
         .cpuid = { .eax = 6, .reg = R_EAX, },
         .tcg_features = TCG_6_EAX_FEATURES,
         /*
-         * PTS shouldn't be enabled by default since it has
+         * PTS and HFI shouldn't be enabled by default since they have
          * requirement for cpu topology.
          */
-        .no_autoenable_flags = CPUID_6_EAX_PTS,
+        .no_autoenable_flags = CPUID_6_EAX_PTS | CPUID_6_EAX_HFI,
     },
     [FEAT_XSAVE_XCR0_LO] = {
         .type = CPUID_FEATURE_WORD,
@@ -1557,6 +1557,18 @@ static FeatureDep feature_dependencies[] = {
         .from = { FEAT_VMX_SECONDARY_CTLS,  VMX_SECONDARY_EXEC_ENABLE_USER_WAIT_PAUSE },
         .to = { FEAT_7_0_ECX,               CPUID_7_0_ECX_WAITPKG },
     },
+    {
+        .from = { FEAT_1_EDX,               CPUID_ACPI },
+        .to = { FEAT_6_EAX,                 CPUID_6_EAX_HFI },
+    },
+    {
+        .from = { FEAT_1_EDX,               CPUID_TM },
+        .to = { FEAT_6_EAX,                 CPUID_6_EAX_HFI },
+    },
+    {
+        .from = { FEAT_6_EAX,               CPUID_6_EAX_PTS },
+        .to = { FEAT_6_EAX,                 CPUID_6_EAX_HFI },
+    },
 };
 
 typedef struct X86RegisterInfo32 {
@@ -6158,6 +6170,25 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
         *ebx = 0;
         *ecx = 0;
         *edx = 0;
+        /*
+         * KVM only supports HFI virtualization with ITD, so
+         * set the HFI information only if the ITD is enabled.
+         */
+        if (*eax & CPUID_6_EAX_ITD) {
+            if (kvm_enabled()) {
+                *ecx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x6,
+                                                    count, R_ECX);
+                /*
+                 * No need to adjust the number of pages since the default
+                 * 1 4KB page is enough to hold the HFI entries of max_cpus
+                 * (1024) supported by i386 machine (q35).
+                 */
+                *edx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x6,
+                                                    count, R_EDX);
+                /* Set HFI table index as CPU index. */
+                *edx |= cs->cpu_index << 16;
+            }
+        }
         break;
     case 7:
         /* Structured Extended Feature Flags Enumeration Leaf */
@@ -7437,11 +7468,19 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
         return;
     }
 
-    if (env->features[FEAT_6_EAX] & CPUID_6_EAX_PTS &&
+    if (env->features[FEAT_6_EAX] & CPUID_6_EAX_HFI &&
+        (ms->smp.dies > 1 || ms->smp.sockets > 1)) {
+        error_setg(errp,
+                   "HFI currently only supports die/package, "
+                   "please set by \"-smp ...,sockets=1,dies=1\"");
+        return;
+    }
+
+    if (env->features[FEAT_6_EAX] & (CPUID_6_EAX_PTS | CPUID_6_EAX_HFI) &&
         !(env->features[FEAT_6_EAX] & CPUID_6_EAX_ITD)) {
         error_setg(errp,
                    "In the absence of ITD, Guest does "
-                   "not need PTS");
+                   "not need PTS/HFI");
         return;
     }
 #endif
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index a8c247b2ef89..b54a2ccd6a6e 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -533,6 +533,9 @@ typedef enum X86Seg {
 #define MSR_IA32_PACKAGE_THERM_STATUS    0x000001b1
 #define MSR_IA32_PACKAGE_THERM_INTERRUPT 0x000001b2
 
+#define MSR_IA32_HW_FEEDBACK_CONFIG     0x000017d0
+#define MSR_IA32_HW_FEEDBACK_PTR        0x000017d1
+
 #define MSR_IA32_VMX_BASIC              0x00000480
 #define MSR_IA32_VMX_PINBASED_CTLS      0x00000481
 #define MSR_IA32_VMX_PROCBASED_CTLS     0x00000482
@@ -986,6 +989,7 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,
 
 #define CPUID_6_EAX_ARAT       (1U << 2)
 #define CPUID_6_EAX_PTS        (1U << 6)
+#define CPUID_6_EAX_HFI        (1U << 19)
 #define CPUID_6_EAX_ITD        (1U << 23)
 
 /* CPUID[0x80000007].EDX flags: */
@@ -1773,12 +1777,14 @@ typedef struct CPUArchState {
     uint64_t therm_status;
 
     /*
-     * Although these are package level MSRs, for the PTS feature, we
+     * Although these are package level MSRs, for the PTS/HFI feature, we
      * temporarily limit it to be enabled for only 1 package, so the value
      * of each vCPU is same and it's enough to support the save/load.
      */
     uint64_t pkg_therm_interrupt;
     uint64_t pkg_therm_status;
+    uint64_t hfi_config;
+    uint64_t hfi_ptr;
 
     /* exception/interrupt handling */
     int error_code;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 258591535fd5..694aa20afc67 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -140,6 +140,7 @@ static bool has_msr_perf_capabs;
 static bool has_msr_pkrs;
 static bool has_msr_therm;
 static bool has_msr_pkg_therm;
+static bool has_msr_hfi;
 
 static uint32_t has_architectural_pmu_version;
 static uint32_t num_architectural_pmu_gp_counters;
@@ -2466,6 +2467,10 @@ static int kvm_get_supported_msrs(KVMState *s)
             case MSR_IA32_PACKAGE_THERM_INTERRUPT:
                 has_msr_pkg_therm = true;
                 break;
+            case MSR_IA32_HW_FEEDBACK_CONFIG:
+            case MSR_IA32_HW_FEEDBACK_PTR:
+                has_msr_hfi = true;
+                break;
             }
         }
     }
@@ -3326,6 +3331,12 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
             kvm_msr_entry_add(cpu, MSR_IA32_PACKAGE_THERM_INTERRUPT,
                               env->therm_interrupt);
         }
+        if (has_msr_hfi) {
+            kvm_msr_entry_add(cpu, MSR_IA32_HW_FEEDBACK_CONFIG,
+                              env->hfi_config);
+            kvm_msr_entry_add(cpu, MSR_IA32_HW_FEEDBACK_PTR,
+                              env->hfi_ptr);
+        }
     }
 
 #ifdef TARGET_X86_64
@@ -3808,6 +3819,10 @@ static int kvm_get_msrs(X86CPU *cpu)
         kvm_msr_entry_add(cpu, MSR_IA32_PACKAGE_THERM_STATUS, 0);
         kvm_msr_entry_add(cpu, MSR_IA32_PACKAGE_THERM_INTERRUPT, 0);
     }
+    if (has_msr_hfi) {
+        kvm_msr_entry_add(cpu, MSR_IA32_HW_FEEDBACK_CONFIG, 0);
+        kvm_msr_entry_add(cpu, MSR_IA32_HW_FEEDBACK_PTR, 0);
+    }
 
 #ifdef TARGET_X86_64
     if (lm_capable_kernel) {
@@ -4304,6 +4319,12 @@ static int kvm_get_msrs(X86CPU *cpu)
         case MSR_IA32_PACKAGE_THERM_INTERRUPT:
             env->pkg_therm_interrupt = msrs[i].data;
             break;
+        case MSR_IA32_HW_FEEDBACK_CONFIG:
+            env->hfi_config = msrs[i].data;
+            break;
+        case MSR_IA32_HW_FEEDBACK_PTR:
+            env->hfi_ptr = msrs[i].data;
+            break;
         }
     }
 
-- 
2.34.1

[RFC 4/6] target/i386: Add support for Intel Thread Director feature

[Zhao Liu]

From: Zhao Liu <zhao1.liu@intel.com>

Intel Thread Director (ITD) is the extension of HFI, and it extends
the HFI to provide performance and energy efficiency data for advanced
classes of instructions [1].

From Alder Lake, Intel's client products support ITD, and this feature
can be used in VM to optimize scheduling on hybrid architectures.

Like HFI, ITD virtualization also has the same topology limitations
(only 1 die and 1 socket) because ITD's virtualization support is based
on HFI.

In order to avoid potential contention problems caused by multiple
virtual-packages/dies, add the following restrictions to the ITD feature
bit:

1. Mark ITD as no_autoenable_flags and it won't be enabled by default.
2. ITD can't be enabled for the case with multiple packages/dies.

ITD feature depends on HFI, so also add its dependency.

[1]: SDM, vol. 3B, section 15.6 HARDWARE FEEDBACK INTERFACE AND INTEL
     THREAD DIRECTOR

Tested-by: Yanting Jiang <yanting.jiang@intel.com>
Co-developed-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Signed-off-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index e3eb361436c9..55287d0a3e73 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1118,17 +1118,18 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, "hfi",
-            NULL, NULL, NULL, NULL,
+            NULL, NULL, NULL, "itd",
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
         },
         .cpuid = { .eax = 6, .reg = R_EAX, },
         .tcg_features = TCG_6_EAX_FEATURES,
         /*
-         * PTS and HFI shouldn't be enabled by default since they have
+         * PTS, HFI and ITD shouldn't be enabled by default since they have
          * requirement for cpu topology.
          */
-        .no_autoenable_flags = CPUID_6_EAX_PTS | CPUID_6_EAX_HFI,
+        .no_autoenable_flags = CPUID_6_EAX_PTS | CPUID_6_EAX_HFI |
+                               CPUID_6_EAX_ITD,
     },
     [FEAT_XSAVE_XCR0_LO] = {
         .type = CPUID_FEATURE_WORD,
@@ -1569,6 +1570,10 @@ static FeatureDep feature_dependencies[] = {
         .from = { FEAT_6_EAX,               CPUID_6_EAX_PTS },
         .to = { FEAT_6_EAX,                 CPUID_6_EAX_HFI },
     },
+    {
+        .from = { FEAT_6_EAX,               CPUID_6_EAX_HFI },
+        .to = { FEAT_6_EAX,                 CPUID_6_EAX_ITD },
+    },
 };
 
 typedef struct X86RegisterInfo32 {
@@ -7468,10 +7473,10 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
         return;
     }
 
-    if (env->features[FEAT_6_EAX] & CPUID_6_EAX_HFI &&
+    if (env->features[FEAT_6_EAX] & (CPUID_6_EAX_HFI | CPUID_6_EAX_ITD) &&
         (ms->smp.dies > 1 || ms->smp.sockets > 1)) {
         error_setg(errp,
-                   "HFI currently only supports die/package, "
+                   "HFI/ITD currently only supports die/package, "
                    "please set by \"-smp ...,sockets=1,dies=1\"");
         return;
     }
-- 
2.34.1

[RFC 5/6] target/i386: Add support for HRESET feature

[Zhao Liu]

From: Zhuocheng Ding <zhuocheng.ding@intel.com>

HRESET provides an HRESET instruction to reset the ITD related history
accumulated on the current logical processor it is executing on [1].

HRESET feature not only needs to have the feature bit of 0x07.0x01.eax
[bit 22] in the CPUID, but also the associated 0x20 leaf, thus, we also
fill HRESET related info provided by KVM into Guest's 0x20 leaf.

Because currently HRESET is only used to reset ITD's history and ITD has
been marked as no_autoenable_flags, mark the HRESET feature bit as
no_autoenable_flags, too.

Additionally, add MSR_IA32_HW_HRESET_ENABLE save/load support since it's
emulated in KVM. This MSR is used to control the enabling of ITD's
history reset.

[1]: SDM, vol. 3B, section 15.6.11 Logical Processor Scope History

Tested-by: Yanting Jiang <yanting.jiang@intel.com>
Co-developed-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Signed-off-by: Zhuocheng Ding <zhuocheng.ding@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.c     | 28 +++++++++++++++++++++++++++-
 target/i386/cpu.h     |  6 ++++++
 target/i386/kvm/kvm.c | 14 ++++++++++++++
 3 files changed, 47 insertions(+), 1 deletion(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 55287d0a3e73..3b26b471b861 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -966,7 +966,7 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             NULL, NULL, "fzrm", "fsrs",
             "fsrc", NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
-            NULL, "amx-fp16", NULL, "avx-ifma",
+            NULL, "amx-fp16", "hreset", "avx-ifma",
             NULL, NULL, NULL, NULL,
             NULL, NULL, NULL, NULL,
         },
@@ -976,6 +976,11 @@ FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
             .reg = R_EAX,
         },
         .tcg_features = TCG_7_1_EAX_FEATURES,
+        /*
+         * Currently HRESET is only used for ITD history reset. ITD is not
+         * autoenable, so also don't enable HRESET by default.
+         */
+        .no_autoenable_flags = CPUID_7_1_EAX_HRESET,
     },
     [FEAT_7_1_EDX] = {
         .type = CPUID_FEATURE_WORD,
@@ -6502,6 +6507,22 @@ void cpu_x86_cpuid(CPUX86State *env, uint32_t index, uint32_t count,
         }
         break;
     }
+    case 0x20: {
+        /* Processor History Reset */
+        if (kvm_enabled() &&
+            env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_HRESET) {
+            *eax = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x20,
+                                                count, R_EAX);
+            *ebx = kvm_arch_get_supported_cpuid(cs->kvm_state, 0x20,
+                                                count, R_EBX);
+        } else {
+            *eax = 0;
+            *ebx = 0;
+        }
+        *ecx = 0;
+        *edx = 0;
+        break;
+    }
     case 0x40000000:
         /*
          * CPUID code in kvm_arch_init_vcpu() ignores stuff
@@ -7147,6 +7168,11 @@ void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
         if (env->features[FEAT_7_0_EBX] & CPUID_7_0_EBX_SGX) {
             x86_cpu_adjust_level(cpu, &env->cpuid_min_level, 0x12);
         }
+
+        /* HRESET requires CPUID[0x20] */
+        if (env->features[FEAT_7_1_EAX] & CPUID_7_1_EAX_HRESET) {
+            x86_cpu_adjust_level(cpu, &env->cpuid_min_level, 0x20);
+        }
     }
 
     /* Set cpuid_*level* based on cpuid_min_*level, if not explicitly set */
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index b54a2ccd6a6e..a68c9d8a8660 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -535,6 +535,7 @@ typedef enum X86Seg {
 
 #define MSR_IA32_HW_FEEDBACK_CONFIG     0x000017d0
 #define MSR_IA32_HW_FEEDBACK_PTR        0x000017d1
+#define MSR_IA32_HW_HRESET_ENABLE       0x000017da
 
 #define MSR_IA32_VMX_BASIC              0x00000480
 #define MSR_IA32_VMX_PINBASED_CTLS      0x00000481
@@ -933,6 +934,8 @@ uint64_t x86_cpu_get_supported_feature_word(FeatureWord w,
 #define CPUID_7_1_EAX_FSRC              (1U << 12)
 /* Support Tile Computational Operations on FP16 Numbers */
 #define CPUID_7_1_EAX_AMX_FP16          (1U << 21)
+/* HISTORY RESET */
+#define CPUID_7_1_EAX_HRESET            (1U << 22)
 /* Support for VPMADD52[H,L]UQ */
 #define CPUID_7_1_EAX_AVX_IFMA          (1U << 23)
 
@@ -1786,6 +1789,9 @@ typedef struct CPUArchState {
     uint64_t hfi_config;
     uint64_t hfi_ptr;
 
+    /* Per-VCPU HRESET MSR */
+    uint64_t hreset_enable;
+
     /* exception/interrupt handling */
     int error_code;
     int exception_is_int;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 694aa20afc67..e490126f23ca 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -141,6 +141,7 @@ static bool has_msr_pkrs;
 static bool has_msr_therm;
 static bool has_msr_pkg_therm;
 static bool has_msr_hfi;
+static bool has_msr_hreset;
 
 static uint32_t has_architectural_pmu_version;
 static uint32_t num_architectural_pmu_gp_counters;
@@ -2471,6 +2472,9 @@ static int kvm_get_supported_msrs(KVMState *s)
             case MSR_IA32_HW_FEEDBACK_PTR:
                 has_msr_hfi = true;
                 break;
+            case MSR_IA32_HW_HRESET_ENABLE:
+                has_msr_hreset = true;
+                break;
             }
         }
     }
@@ -3337,6 +3341,10 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
             kvm_msr_entry_add(cpu, MSR_IA32_HW_FEEDBACK_PTR,
                               env->hfi_ptr);
         }
+        if (has_msr_hreset) {
+            kvm_msr_entry_add(cpu, MSR_IA32_HW_HRESET_ENABLE,
+                              env->hreset_enable);
+        }
     }
 
 #ifdef TARGET_X86_64
@@ -3823,6 +3831,9 @@ static int kvm_get_msrs(X86CPU *cpu)
         kvm_msr_entry_add(cpu, MSR_IA32_HW_FEEDBACK_CONFIG, 0);
         kvm_msr_entry_add(cpu, MSR_IA32_HW_FEEDBACK_PTR, 0);
     }
+    if (has_msr_hreset) {
+        kvm_msr_entry_add(cpu, MSR_IA32_HW_HRESET_ENABLE, 0);
+    }
 
 #ifdef TARGET_X86_64
     if (lm_capable_kernel) {
@@ -4325,6 +4336,9 @@ static int kvm_get_msrs(X86CPU *cpu)
         case MSR_IA32_HW_FEEDBACK_PTR:
             env->hfi_ptr = msrs[i].data;
             break;
+        case MSR_IA32_HW_HRESET_ENABLE:
+            env->hreset_enable = msrs[i].data;
+            break;
         }
     }
 
-- 
2.34.1

[RFC 6/6] i386: Add a new property to set ITD related feature bits for Guest

[Zhao Liu]

From: Zhao Liu <zhao1.liu@intel.com>

The property enable-itd will be used to set ITD related feature bits
for Guest, which includes PTS, HFI, ITD and HRESET.

Now PTS, HFI, ITD and HRESET are marked as no_autoenable_flags, since
PTS, HFI and ITD have additional restrictions on CPU topology, and
HRESET is only used in ITD case. If user wants to enable ITD for Guest,
he need to specify PTS, HFI, ITD and HRESET explicitly in the -cpu
command.

Thus it's necessary to introduce "-cpu enable-itd" to help set these
feature bits.

Tested-by: Yanting Jiang <yanting.jiang@intel.com>
Signed-off-by: Zhao Liu <zhao1.liu@intel.com>
---
 target/i386/cpu.c | 20 +++++++++++++++-----
 target/i386/cpu.h |  3 +++
 2 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 3b26b471b861..070f7ff43a1b 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -7304,6 +7304,12 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
      */
     x86_cpu_hyperv_realize(cpu);
 
+    if (cpu->enable_itd) {
+        env->features[FEAT_6_EAX] |= CPUID_6_EAX_PTS | CPUID_6_EAX_HFI |
+                                     CPUID_6_EAX_ITD;
+        env->features[FEAT_7_1_EAX] |= CPUID_7_1_EAX_HRESET;
+    }
+
     x86_cpu_expand_features(cpu, &local_err);
     if (local_err) {
         goto out;
@@ -7494,22 +7500,25 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp)
 
     if (env->features[FEAT_6_EAX] & CPUID_6_EAX_PTS && ms->smp.sockets > 1) {
         error_setg(errp,
-                   "PTS currently only supports 1 package, "
-                   "please set by \"-smp ...,sockets=1\"");
+                   "%s currently only supports 1 package, "
+                   "please set by \"-smp ...,sockets=1\"",
+                   cpu->enable_itd ? "enable-itd" : "PTS");
         return;
     }
 
     if (env->features[FEAT_6_EAX] & (CPUID_6_EAX_HFI | CPUID_6_EAX_ITD) &&
         (ms->smp.dies > 1 || ms->smp.sockets > 1)) {
         error_setg(errp,
-                   "HFI/ITD currently only supports die/package, "
-                   "please set by \"-smp ...,sockets=1,dies=1\"");
+                   "%s currently only supports 1 die/package, "
+                   "please set by \"-smp ...,sockets=1,dies=1\"",
+                   cpu->enable_itd ? "enable-itd" : "HFI/ITD");
         return;
     }
 
     if (env->features[FEAT_6_EAX] & (CPUID_6_EAX_PTS | CPUID_6_EAX_HFI) &&
         !(env->features[FEAT_6_EAX] & CPUID_6_EAX_ITD)) {
-        error_setg(errp,
+        error_setg(errp, "%s", cpu->enable_itd ?
+                   "Host doesn't support ITD" :
                    "In the absence of ITD, Guest does "
                    "not need PTS/HFI");
         return;
@@ -8003,6 +8012,7 @@ static Property x86_cpu_properties[] = {
                      false),
     DEFINE_PROP_BOOL("x-intel-pt-auto-level", X86CPU, intel_pt_auto_level,
                      true),
+    DEFINE_PROP_BOOL("enable-itd", X86CPU, enable_itd, false),
     DEFINE_PROP_END_OF_LIST()
 };
 
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index a68c9d8a8660..009ec66dead0 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -2071,6 +2071,9 @@ struct ArchCPU {
     int32_t hv_max_vps;
 
     bool xen_vapic;
+
+    /* Set ITD and related feature bits (PTS, HFI and HRESET) for Guest. */
+    bool enable_itd;
 };
 
 typedef struct X86CPUModel X86CPUModel;
-- 
2.34.1

Re: [RFC 0/6] Intel Thread Director Virtualization Support in QEMU

[Zhao Liu]

On Sat, Feb 03, 2024 at 05:30:48PM +0800, Zhao Liu wrote:
> Date: Sat, 3 Feb 2024 17:30:48 +0800
> From: Zhao Liu <zhao1.liu@linux.intel.com>
> Subject: [RFC 0/6] Intel Thread Director Virtualization Support in QEMU
> X-Mailer: git-send-email 2.34.1
> 
> From: Zhao Liu <zhao1.liu@intel.com>
> 
> Hi list,
> 
> This is our refreshed RFC to support our ITD virtualization patch
> series [1] in KVM, and bases on bd2e12310b18 ("Merge tag
> 'qga-pull-2024-01-30' of https://github.com/kostyanf14/qemu into
> staging").
> 
> ITD is Intel's client specific feature to optimize scheduling on Intel
> hybrid platforms. Though this feature depends on hybrid topology

s/depends/doesn't depend/

Regards,
Zhao

> details, in our parctice on Win11 Guest, ITD works with hyrbid topolohy
> and CPU affinity can achieve the most performance improvement in Win11
> Guest (for example, on i9-13900K, up to 14%+ improvement on
> 3DMARK). More data or details, can be found in [1]. Thus, the ITD for
> Win11 is also a typical use case of hybrid topology.
> 
> 
> Welcome your feedback!
> 
> 
> 1. Background and Motivation
> ============================
> 
> ITD allows the hardware to provide scheduling hints to the OS to help
> optimize scheduling performance, and under the Intel hybrid
> architecture, since Core and Atom have different capabilities
> (performance, energy effency, etc.),  scheduling based on hardware
> hints can take full advantage of this hybrid architecture. This is also
> the most ideal scheduling model for intel hybrid architecture.
> 
> Therefore, we want to virtualize the ITD feature so that ITD can benefit
> performance of the virtual machines on the hybrid machines as well.
> 
> Currently, our ITD virtualization is a software virtualization solution.
> 
> 
> 2. Introduction to HFI and ITD
> ==============================
> 
> Intel provides Hardware Feedback Interface (HFI) feature to allow
> hardware to provide guidance to the OS scheduler to perform optimal
> workload scheduling through a hardware feedback interface structure in
> memory [2]. This hfi structure is called HFI table.
> 
> As for now, the guidance includes performance and energy enficency hints,
> and it could update via thermal interrupt as the actual operating
> conditions of the processor change during run time.
> 
> And Intel Thread Director (ITD) feature extends the HFI to provide
> performance and energy efficiency data for advanced classes of
> instructions.
> 
> The virtual HFI table is maintained in KVM, and for QEMU, we just need
> to handle HFI/ITD/HRESET (and their dependent features: ACPI, TM and
> PTS) related CPUIDs and MSRs.
> 
> 
> 3. Package level MSRs handling
> ==============================
> 
> PTS, HFI and ITD are all have package level features, such as package
> level MSRs and package level HFI tables. But since KVM hasn't
> support msr-topology and it just handle these package-level MSRs and
> HFI table at VM level, in order to avoid potential contention problems
> caused by multiple virtual-packages, we restrict VMs to be able to
> enable PTC/HFI/ITD iff there's only 1 package (and only 1 die for
> ITD/HFI).
> 
> 
> 4. HFI/ITD related info in CPUID
> ================================
> 
> KVM provides some basic HFI info in CPUID.0x06 leaf, which is associated
> with the virtual HFI table in KVM.
> 
> QEMU should configure HFI table index for each vCPU. Here we set the HFI
> table index to vCPU index so that different vCPUs have different HFI
> entries to avoid unnecessary competition problems.
> 
> 
> 5. Compatibility issues
> =======================
> 
> HFI is supported in both server (SPR) and client (ADL/RPL/MTL) platform
> products while ITD is the client specific feature.
> 
> For client platform, ITD (with HFI) could be enabled in Guest to improve
> scheduling, but for server platform, HFI (without ITD) is only useful
> on Host and Guest doesn't need it.
> 
> To simplify the enabling logic and avoid impacting the common topology
> of the Guest, we set PTS, HFI, and ITD as feature bits that are not
> automatically enabled.
> 
> Only when the user actively specifies these features, QEMU will check
> and decide whether to enable them based on the topology constraints and
> the ITD constraints.
> 
> 
> 6. New option "enable-itd"
> ============================
> 
> ITD-related features include PTS, HFI, ITD, and HRESET.
> 
> To make it easier for users to enable ITD for Guest without specifying
> the above feature bits one by one, we provide a new option "enable-itd"
> to set the above feature bits for Guest all at once.
> 
> "enable-itd" does not guarantee that ITD will be enabled for Guest.
> The success of enabling ITD for guest depends on topology constraints,
> platform support, etc., which are checked in QEMU.
> 
> 
> 7. Patch Summary
> ================
> 
> Patch 1: Add support save/load for ACPI feature related thermal MSRs
>          since ACPI feature CPUID has been added in QEMU.
> Patch 2: Add support for PTS (package) thermal MSRs and its CPUID
> Patch 3: Add support for HFI MSRs and its CPUID
> Patch 4: Add support ITD CPUID and MSR_IA32_HW_FEEDBACK_THREAD_CONFIG.
> Patch 5: Add support HRESET CPUID and MSR_IA32_HW_HRESET_ENABLE.
> Patch 6: Add "enable-itd" to help user set ITD related feature bits.
> 
> # 8. References
> 
> [1]: KVM RFC: [RFC 00/26] Intel Thread Director Virtualization
>      https://lore.kernel.org/kvm/20240203091214.411862-1-zhao1.liu@linux.intel.com/T/#t
> [2]: SDM, vol. 3B, section 15.6 HARDWARE FEEDBACK INTERFACE AND INTEL
>      THREAD DIRECTOR
> 
> Thanks and Best Regards,
> Zhao
> ---
> Zhao Liu (2):
>   target/i386: Add support for Intel Thread Director feature
>   i386: Add a new property to set ITD related feature bits for Guest
> 
> Zhuocheng Ding (4):
>   target/i386: Add support for save/load of ACPI thermal MSRs
>   target/i386: Add support for Package Thermal Management feature
>   target/i386: Add support for Hardware Feedback Interface feature
>   target/i386: Add support for HRESET feature
> 
>  target/i386/cpu.c     | 108 ++++++++++++++++++++++++++++++++++++++++--
>  target/i386/cpu.h     |  37 +++++++++++++++
>  target/i386/kvm/kvm.c |  84 ++++++++++++++++++++++++++++++++
>  3 files changed, 225 insertions(+), 4 deletions(-)
> 
> -- 
> 2.34.1
>