Re: [RFC 0/6] Intel Thread Director Virtualization Support in QEMU

[Zhao Liu <zhao1.liu@linux.intel.com>]

On Sat, Feb 03, 2024 at 05:30:48PM +0800, Zhao Liu wrote:
> Date: Sat, 3 Feb 2024 17:30:48 +0800
> From: Zhao Liu <zhao1.liu@linux.intel.com>
> Subject: [RFC 0/6] Intel Thread Director Virtualization Support in QEMU
> X-Mailer: git-send-email 2.34.1
> 
> From: Zhao Liu <zhao1.liu@intel.com>
> 
> Hi list,
> 
> This is our refreshed RFC to support our ITD virtualization patch
> series [1] in KVM, and bases on bd2e12310b18 ("Merge tag
> 'qga-pull-2024-01-30' of https://github.com/kostyanf14/qemu into
> staging").
> 
> ITD is Intel's client specific feature to optimize scheduling on Intel
> hybrid platforms. Though this feature depends on hybrid topology

s/depends/doesn't depend/

Regards,
Zhao

> details, in our parctice on Win11 Guest, ITD works with hyrbid topolohy
> and CPU affinity can achieve the most performance improvement in Win11
> Guest (for example, on i9-13900K, up to 14%+ improvement on
> 3DMARK). More data or details, can be found in [1]. Thus, the ITD for
> Win11 is also a typical use case of hybrid topology.
> 
> 
> Welcome your feedback!
> 
> 
> 1. Background and Motivation
> ============================
> 
> ITD allows the hardware to provide scheduling hints to the OS to help
> optimize scheduling performance, and under the Intel hybrid
> architecture, since Core and Atom have different capabilities
> (performance, energy effency, etc.),  scheduling based on hardware
> hints can take full advantage of this hybrid architecture. This is also
> the most ideal scheduling model for intel hybrid architecture.
> 
> Therefore, we want to virtualize the ITD feature so that ITD can benefit
> performance of the virtual machines on the hybrid machines as well.
> 
> Currently, our ITD virtualization is a software virtualization solution.
> 
> 
> 2. Introduction to HFI and ITD
> ==============================
> 
> Intel provides Hardware Feedback Interface (HFI) feature to allow
> hardware to provide guidance to the OS scheduler to perform optimal
> workload scheduling through a hardware feedback interface structure in
> memory [2]. This hfi structure is called HFI table.
> 
> As for now, the guidance includes performance and energy enficency hints,
> and it could update via thermal interrupt as the actual operating
> conditions of the processor change during run time.
> 
> And Intel Thread Director (ITD) feature extends the HFI to provide
> performance and energy efficiency data for advanced classes of
> instructions.
> 
> The virtual HFI table is maintained in KVM, and for QEMU, we just need
> to handle HFI/ITD/HRESET (and their dependent features: ACPI, TM and
> PTS) related CPUIDs and MSRs.
> 
> 
> 3. Package level MSRs handling
> ==============================
> 
> PTS, HFI and ITD are all have package level features, such as package
> level MSRs and package level HFI tables. But since KVM hasn't
> support msr-topology and it just handle these package-level MSRs and
> HFI table at VM level, in order to avoid potential contention problems
> caused by multiple virtual-packages, we restrict VMs to be able to
> enable PTC/HFI/ITD iff there's only 1 package (and only 1 die for
> ITD/HFI).
> 
> 
> 4. HFI/ITD related info in CPUID
> ================================
> 
> KVM provides some basic HFI info in CPUID.0x06 leaf, which is associated
> with the virtual HFI table in KVM.
> 
> QEMU should configure HFI table index for each vCPU. Here we set the HFI
> table index to vCPU index so that different vCPUs have different HFI
> entries to avoid unnecessary competition problems.
> 
> 
> 5. Compatibility issues
> =======================
> 
> HFI is supported in both server (SPR) and client (ADL/RPL/MTL) platform
> products while ITD is the client specific feature.
> 
> For client platform, ITD (with HFI) could be enabled in Guest to improve
> scheduling, but for server platform, HFI (without ITD) is only useful
> on Host and Guest doesn't need it.
> 
> To simplify the enabling logic and avoid impacting the common topology
> of the Guest, we set PTS, HFI, and ITD as feature bits that are not
> automatically enabled.
> 
> Only when the user actively specifies these features, QEMU will check
> and decide whether to enable them based on the topology constraints and
> the ITD constraints.
> 
> 
> 6. New option "enable-itd"
> ============================
> 
> ITD-related features include PTS, HFI, ITD, and HRESET.
> 
> To make it easier for users to enable ITD for Guest without specifying
> the above feature bits one by one, we provide a new option "enable-itd"
> to set the above feature bits for Guest all at once.
> 
> "enable-itd" does not guarantee that ITD will be enabled for Guest.
> The success of enabling ITD for guest depends on topology constraints,
> platform support, etc., which are checked in QEMU.
> 
> 
> 7. Patch Summary
> ================
> 
> Patch 1: Add support save/load for ACPI feature related thermal MSRs
>          since ACPI feature CPUID has been added in QEMU.
> Patch 2: Add support for PTS (package) thermal MSRs and its CPUID
> Patch 3: Add support for HFI MSRs and its CPUID
> Patch 4: Add support ITD CPUID and MSR_IA32_HW_FEEDBACK_THREAD_CONFIG.
> Patch 5: Add support HRESET CPUID and MSR_IA32_HW_HRESET_ENABLE.
> Patch 6: Add "enable-itd" to help user set ITD related feature bits.
> 
> # 8. References
> 
> [1]: KVM RFC: [RFC 00/26] Intel Thread Director Virtualization
>      https://lore.kernel.org/kvm/20240203091214.411862-1-zhao1.liu@linux.intel.com/T/#t
> [2]: SDM, vol. 3B, section 15.6 HARDWARE FEEDBACK INTERFACE AND INTEL
>      THREAD DIRECTOR
> 
> Thanks and Best Regards,
> Zhao
> ---
> Zhao Liu (2):
>   target/i386: Add support for Intel Thread Director feature
>   i386: Add a new property to set ITD related feature bits for Guest
> 
> Zhuocheng Ding (4):
>   target/i386: Add support for save/load of ACPI thermal MSRs
>   target/i386: Add support for Package Thermal Management feature
>   target/i386: Add support for Hardware Feedback Interface feature
>   target/i386: Add support for HRESET feature
> 
>  target/i386/cpu.c     | 108 ++++++++++++++++++++++++++++++++++++++++--
>  target/i386/cpu.h     |  37 +++++++++++++++
>  target/i386/kvm/kvm.c |  84 ++++++++++++++++++++++++++++++++
>  3 files changed, 225 insertions(+), 4 deletions(-)
> 
> -- 
> 2.34.1
>