Re: [PATCH v2 08/10] target/i386/kvm: reset AMD PMU registers during VM reset

[Dongli Zhang <dongli.zhang@oracle.com>]

Hi ewanhai (and FYI Zhao),

Thank you very much for suggestion! Indeed recently I am struggling with QEMU
and Zhaoxin. Please see inline.

On 3/27/25 11:29 PM, ewanhai wrote:
> Hi Zhao,
> 
> Thank you for pointing out the potential impact on Zhaoxin CPUs!
> 
> Hi Dongli,
> 
> Zhaoxin (including vendor "__shanghai__" and "centaurhauls")'s PMU is
> compatible with Intel, so I have some advice for this patch.
> 
> 在 2025/3/3 06:00, Dongli Zhang 写道:
>> [snip]
>> +
>> +static bool is_same_vendor(CPUX86State *env)
>> +{
>> +    static uint32_t host_cpuid_vendor1;
>> +    static uint32_t host_cpuid_vendor2;
>> +    static uint32_t host_cpuid_vendor3;
>> +
>> +    host_cpuid(0x0, 0, NULL, &host_cpuid_vendor1, &host_cpuid_vendor3,
>> +               &host_cpuid_vendor2);
>> +
>> +    return env->cpuid_vendor1 == host_cpuid_vendor1 &&
>> +           env->cpuid_vendor2 == host_cpuid_vendor2 &&
>> +           env->cpuid_vendor3 == host_cpuid_vendor3;
>> +}
> Should we consider a special case, such as emulating Intel CPUs on a
> Zhaoxin platform, or vice versa? If so, maybe we can write a

The vendor and CPU are different. i.e., if we use Zhaoxin CPU without
configuring vendor: "-cpu YongFeng,+pmu \" on Intel KVM.

The CPU is Zhaoxin while vendor is still Intel.

The PMU selection is based on vendor, not CPU.

[    0.321163] smpboot: CPU0: Intel Zhaoxin YongFeng Processor (family: 0x7,
model: 0xb, stepping: 0x3)
[    0.321996] Performance Events: generic architected perfmon, Intel PMU driver.
[    0.322867] ... version:                2
[    0.323738] ... bit width:              48
[    0.323864] ... generic registers:      4
[    0.324776] ... value mask:             0000ffffffffffff
[    0.324864] ... max period:             000000007fffffff
[    0.325864] ... fixed-purpose events:   3
[    0.326749] ... event mask:             000000070000000f

By default, IS_INTEL_CPU() still returns true even we emulate Zhaoxin on Intel KVM.

> 'vendor_compatible()' helper. After all, before this patchset, QEMU
> supported behavior-similar CPU emulation, e.g., emulating an Intel VCPU on
> a Zhaoxin PCPU.

I did many efforts, and I could not use Zhaoxin's PMU on Intel hypervisor.

According to arch/x86/events/zhaoxin/core.c, the Zhaoxin's PMU is working in
limited conditions, especially only when stepping >= 0xe.

switch (boot_cpu_data.x86) {
case 0x06:
    /*
     * Support Zhaoxin CPU from ZXC series, exclude Nano series through FMS.
     * Nano FMS: Family=6, Model=F, Stepping=[0-A][C-D]
     * ZXC FMS: Family=6, Model=F, Stepping=E-F OR Family=6, Model=0x19,
Stepping=0-3
     */
    if ((boot_cpu_data.x86_model == 0x0f && boot_cpu_data.x86_stepping >= 0x0e) ||
            boot_cpu_data.x86_model == 0x19) {


From QEMU, the stepping of YongFeng is always 3.

5502         .name = "YongFeng",
5503         .level = 0x1F,
5504         .vendor = CPUID_VENDOR_ZHAOXIN1,
5505         .family = 7,
5506         .model = 11,
5507         .stepping = 3,

Therefore, I cannot enable Zhaoxin's PMU on Intel KVM.

-cpu YongFeng,vendor="CentaurHauls",+pmu \

[    0.253229] smpboot: CPU0: Centaur Zhaoxin YongFeng Processor (family: 0x7,
model: 0xb, stepping: 0x3)
[    0.254009] Performance Events:
[    0.254009] core: Welcome to zhaoxin pmu!
[    0.254880] core: Version check pass!
[    0.255567] no PMU driver, software events only.


It doesn't work on Intel Icelake hypervisor too, even with "host".

-cpu host,vendor="CentaurHauls",+pmu \

[    0.268434] smpboot: CPU0: Centaur Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz
(family: 0x6, model: 0x6a, stepping: 0x6)
[    0.269237] Performance Events:
[    0.269237] core: Welcome to zhaoxin pmu!
[    0.270112] core: Version check pass!
[    0.270768] no PMU driver, software events only.


The PMU never works, although cpuid returns PMU config.

[root@vm ~]# cpuid -1 -l 0xa
CPU:
   Architecture Performance Monitoring Features (0xa):
      version ID                               = 0x2 (2)
      number of counters per logical processor = 0x8 (8)
      bit width of counter                     = 0x30 (48)
      length of EBX bit vector                 = 0x8 (8)
      core cycle event                         = available
      instruction retired event                = available
      reference cycles event                   = available
      last-level cache ref event               = available
      last-level cache miss event              = available
      branch inst retired event                = available
      branch mispred retired event             = available
      top-down slots event                     = available
... ...
      number of contiguous fixed counters      = 0x3 (3)
      bit width of fixed counters              = 0x30 (48)
      anythread deprecation                    = true


So far I am not able to use Zhaoxin PMU on Intel hypervisor.

Since I don't have Zhaoxin environment, I am not sure about "vice versa".

Unless there is more suggestion from Zhao, I may replace is_same_vendor() with
vendor_compatible().

>> +static void kvm_init_pmu_info(CPUState *cs)
>> +{
>> +    X86CPU *cpu = X86_CPU(cs);
>> +    CPUX86State *env = &cpu->env;
>> +
>> +    /*
>> +     * The PMU virtualization is disabled by kvm.enable_pmu=N.
>> +     */
>> +    if (kvm_pmu_disabled) {
>> +        return;
>> +    }
>> +
>> +    /*
>> +     * It is not supported to virtualize AMD PMU registers on Intel
>> +     * processors, nor to virtualize Intel PMU registers on AMD processors.
>> +     */
>> +    if (!is_same_vendor(env)) {
>> +        return;
>> +    }
> 
> ditto.

Sure. I may replace is_same_vendor() with
vendor_compatible(), unless there is objection from Zhao.

> 
>> [snip]
>> +    /*
>> +     * If KVM_CAP_PMU_CAPABILITY is not supported, there is no way to
>> +     * disable the AMD pmu virtualization.
>> +     *
>> +     * If KVM_CAP_PMU_CAPABILITY is supported !cpu->enable_pmu
>> +     * indicates the KVM has already disabled the PMU virtualization.
>> +     */
>> +    if (has_pmu_cap && !cpu->enable_pmu) {
>> +        return;
>> +    }
>> +
>> +    if (IS_INTEL_CPU(env)) {
>> +        kvm_init_pmu_info_intel(env);
> We can use "if (IS_INTEL_CPU(env) || IS_ZHAOXIN_CPU(env))" instead. This
> helper was introduced to QEMU in commit 5d20aa540b.

Sure.

> 
> The function name kvm_init_pmu_info_"intel"() is acceptable since the
> current Zhaoxin and Intel PMU architectures are compatible. However,
> if Zhaoxin develop any exclusive features in the future, we can always
> implement a separate "zhaoxin" version of the PMU info initialization
> function.
>> +    } else if (IS_AMD_CPU(env)) {
>> +        kvm_init_pmu_info_amd(env);
>> +    }
>> +}
>> +
> [snip]
>>   int kvm_arch_init_vcpu(CPUState *cs)
>>   {
>>       struct {
>> @@ -2288,7 +2376,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>       cpuid_i = kvm_x86_build_cpuid(env, cpuid_data.entries, cpuid_i);
>>       cpuid_data.cpuid.nent = cpuid_i;
>>   -    kvm_init_pmu_info(env);
>> +    kvm_init_pmu_info(cs);
>>         if (((env->cpuid_version >> 8)&0xF) >= 6
>>           && (env->features[FEAT_1_EDX] & (CPUID_MCE | CPUID_MCA)) ==
>> @@ -4064,7 +4152,7 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
>>               kvm_msr_entry_add(cpu, MSR_KVM_POLL_CONTROL, env-
>> >poll_control_msr);
>>           }
>>   -        if (has_pmu_version > 0) {
>> +        if (IS_INTEL_CPU(env) && has_pmu_version > 0) {
> Also use 'if (IS_INTEL_CPU(env) || IS_ZHAOXIN_CPU(env))' instead.
>>               if (has_pmu_version > 1) {
>>                   /* Stop the counter.  */
>>                   kvm_msr_entry_add(cpu, MSR_CORE_PERF_FIXED_CTR_CTRL, 0);
>> @@ -4095,6 +4183,38 @@ static int kvm_put_msrs(X86CPU *cpu, int level)
>>                                     env->msr_global_ctrl);
>>               }
>>           }
>> +
>> +        if (IS_AMD_CPU(env) && has_pmu_version > 0) {
>> +            uint32_t sel_base = MSR_K7_EVNTSEL0;
>> +            uint32_t ctr_base = MSR_K7_PERFCTR0;
>> ...
> [snip]
>> @@ -4542,7 +4662,8 @@ static int kvm_get_msrs(X86CPU *cpu)
>>       if (env->features[FEAT_KVM] & CPUID_KVM_POLL_CONTROL) {
>>           kvm_msr_entry_add(cpu, MSR_KVM_POLL_CONTROL, 1);
>>       }
>> -    if (has_pmu_version > 0) {
>> +
>> +    if (IS_INTEL_CPU(env) && has_pmu_version > 0) {
> Also use 'if (IS_INTEL_CPU(env) || IS_ZHAOXIN_CPU(env))' instead.

Sure.

Thank you very much for suggestion!

Dongli Zhang