Re: [PATCH 2/7] target/i386/kvm: introduce 'pmu-cap-disabled' to set KVM_PMU_CAP_DISABLE

[dongli.zhang@oracle.com]

Hi Dapeng,

On 2/7/25 1:52 AM, Mi, Dapeng wrote:
> 
> On 11/21/2024 6:06 PM, Mi, Dapeng wrote:
>> On 11/8/2024 7:44 AM, dongli.zhang@oracle.com wrote:
>>> Hi Zhao,
>>>
>>>
>>> On 11/6/24 11:52 PM, Zhao Liu wrote:
>>>> (+Dapang & Zide)
>>>>
>>>> Hi Dongli,
>>>>
>>>> On Mon, Nov 04, 2024 at 01:40:17AM -0800, Dongli Zhang wrote:
>>>>> Date: Mon,  4 Nov 2024 01:40:17 -0800
>>>>> From: Dongli Zhang <dongli.zhang@oracle.com>
>>>>> Subject: [PATCH 2/7] target/i386/kvm: introduce 'pmu-cap-disabled' to set
>>>>>  KVM_PMU_CAP_DISABLE
>>>>> X-Mailer: git-send-email 2.43.5
>>>>>
>>>>> The AMD PMU virtualization is not disabled when configuring
>>>>> "-cpu host,-pmu" in the QEMU command line on an AMD server. Neither
>>>>> "-cpu host,-pmu" nor "-cpu EPYC" effectively disables AMD PMU
>>>>> virtualization in such an environment.
>>>>>
>>>>> As a result, VM logs typically show:
>>>>>
>>>>> [    0.510611] Performance Events: Fam17h+ core perfctr, AMD PMU driver.
>>>>>
>>>>> whereas the expected logs should be:
>>>>>
>>>>> [    0.596381] Performance Events: PMU not available due to virtualization, using software events only.
>>>>> [    0.600972] NMI watchdog: Perf NMI watchdog permanently disabled
>>>>>
>>>>> This discrepancy occurs because AMD PMU does not use CPUID to determine
>>>>> whether PMU virtualization is supported.
>>>> Intel platform doesn't have this issue since Linux kernel fails to check
>>>> the CPU family & model when "-cpu *,-pmu" option clears PMU version.
>>>>
>>>> The difference between Intel and AMD platforms, however, is that it seems
>>>> Intel hardly ever reaches the “...due virtualization” message, but
>>>> instead reports an error because it recognizes a mismatched family/model.
>>>>
>>>> This may be a drawback of the PMU driver's print message, but the result
>>>> is the same, it prevents the PMU driver from enabling.
>>>>
>>>> So, please mention that KVM_PMU_CAP_DISABLE doesn't change the PMU
>>>> behavior on Intel platform because current "pmu" property works as
>>>> expected.
>>> Sure. I will mention this in v2.
>>>
>>>>> To address this, we introduce a new property, 'pmu-cap-disabled', for KVM
>>>>> acceleration. This property sets KVM_PMU_CAP_DISABLE if
>>>>> KVM_CAP_PMU_CAPABILITY is supported. Note that this feature currently
>>>>> supports only x86 hosts, as KVM_CAP_PMU_CAPABILITY is used exclusively for
>>>>> x86 systems.
>>>>>
>>>>> Signed-off-by: Dongli Zhang <dongli.zhang@oracle.com>
>>>>> ---
>>>>> Another previous solution to re-use '-cpu host,-pmu':
>>>>> https://urldefense.com/v3/__https://lore.kernel.org/all/20221119122901.2469-1-dongli.zhang@oracle.com/__;!!ACWV5N9M2RV99hQ!Nm8Db-mwBoMIwKkRqzC9kgNi5uZ7SCIf43zUBn92Ar_NEbLXq-ZkrDDvpvDQ4cnS2i4VyKAp6CRVE12bRkMF$ 
>>>> IMO, I prefer the previous version. This VM-level KVM property is
>>>> difficult to integrate with the existing CPU properties. Pls refer later
>>>> comments for reasons.
>>>>
>>>>>  accel/kvm/kvm-all.c        |  1 +
>>>>>  include/sysemu/kvm_int.h   |  1 +
>>>>>  qemu-options.hx            |  9 ++++++-
>>>>>  target/i386/cpu.c          |  2 +-
>>>>>  target/i386/kvm/kvm.c      | 52 ++++++++++++++++++++++++++++++++++++++
>>>>>  target/i386/kvm/kvm_i386.h |  2 ++
>>>>>  6 files changed, 65 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/accel/kvm/kvm-all.c b/accel/kvm/kvm-all.c
>>>>> index 801cff16a5..8b5ba45cf7 100644
>>>>> --- a/accel/kvm/kvm-all.c
>>>>> +++ b/accel/kvm/kvm-all.c
>>>>> @@ -3933,6 +3933,7 @@ static void kvm_accel_instance_init(Object *obj)
>>>>>      s->xen_evtchn_max_pirq = 256;
>>>>>      s->device = NULL;
>>>>>      s->msr_energy.enable = false;
>>>>> +    s->pmu_cap_disabled = false;
>>>>>  }
>>>> The CPU property "pmu" also defaults to "false"...but:
>>>>
>>>>  * max CPU would override this and try to enable PMU by default in
>>>>    max_x86_cpu_initfn().
>>>>
>>>>  * Other named CPU models keep the default setting to avoid affecting
>>>>    the migration.
>>>>
>>>> The pmu_cap_disabled and “pmu” property look unbound and unassociated,
>>>> so this can cause the conflict when they are not synchronized. For
>>>> example,
>>>>
>>>> -cpu host -accel kvm,pmu-cap-disabled=on
>>>>
>>>> The above options will fail to launch a VM (on Intel platform).
>>>>
>>>> Ideally, the “pmu” property and pmu-cap-disabled should be bound to each
>>>> other and be consistent. But it's not easy because:
>>>>  - There is no proper way to have pmu_cap_disabled set different default
>>>>    values (e.g., "false" for max CPU and "true" for named CPU models)
>>>>    based on different CPU models.
>>>>  - And, no proper place to check the consistency of pmu_cap_disabled and
>>>>    enable_pmu.
>>>>
>>>> Therefore, I prefer your previous approach, to reuse current CPU "pmu"
>>>> property.
>>> Thank you very much for the suggestion and reasons.
>>>
>>> I am going to follow your suggestion to switch back to the previous solution in v2.
>> +1.
>>
>>  I also prefer to leverage current exist "+/-pmu" option instead of adding
>> a new option. More options, more complexity. When they are not
>> inconsistent, which has higher priority? all these are issues.
>>
>> Although KVM_CAP_PMU_CAPABILITY is a VM-level PMU capability, but all CPUs
>> in a same VM should always share same PMU configuration (Don't consider
>> hybrid platforms which have many issues need to be handled specifically).
>>
>>
>>>> Further, considering that this is currently the only case that needs to
>>>> to set the VM level's capability in the CPU context, there is no need to
>>>> introduce a new kvm interface (in your previous patch), which can instead
>>>> be set in kvm_cpu_realizefn(), like:
>>>>
>>>>
>>>> diff --git a/target/i386/kvm/kvm-cpu.c b/target/i386/kvm/kvm-cpu.c
>>>> index 99d1941cf51c..05e9c9a1a0cf 100644
>>>> --- a/target/i386/kvm/kvm-cpu.c
>>>> +++ b/target/i386/kvm/kvm-cpu.c
>>>> @@ -42,6 +42,8 @@ static bool kvm_cpu_realizefn(CPUState *cs, Error **errp)
>>>>  {
>>>>      X86CPU *cpu = X86_CPU(cs);
>>>>      CPUX86State *env = &cpu->env;
>>>> +    KVMState *s = kvm_state;
>>>> +    static bool first = true;
>>>>      bool ret;
>>>>
>>>>      /*
>>>> @@ -63,6 +65,29 @@ static bool kvm_cpu_realizefn(CPUState *cs, Error **errp)
>>>>       *   check/update ucode_rev, phys_bits, guest_phys_bits, mwait
>>>>       *   cpu_common_realizefn() (via xcc->parent_realize)
>>>>       */
>>>> +
>>>> +    if (first) {
>>>> +        first = false;
>>>> +
>>>> +        /*
>>>> +         * Since Linux v5.18, KVM provides a VM-level capability to easily
>>>> +         * disable PMUs; however, QEMU has been providing PMU property per
>>>> +         * CPU since v1.6. In order to accommodate both, have to configure
>>>> +         * the VM-level capability here.
>>>> +         */
>>>> +        if (!cpu->enable_pmu &&
>>>> +            kvm_check_extension(s, KVM_CAP_PMU_CAPABILITY)) {
>>>> +            int r = kvm_vm_enable_cap(s, KVM_CAP_PMU_CAPABILITY, 0,
>>>> +                                      KVM_PMU_CAP_DISABLE);
>>>> +
>>>> +            if (r < 0) {
>>>> +                error_setg(errp, "kvm: Failed to disable pmu cap: %s",
>>>> +                           strerror(-r));
>>>> +                return false;
>>>> +            }
>>>> +        }
>> It seems KVM_CAP_PMU_CAPABILITY is called to only disable PMU here. From
>> point view of logic completeness,  KVM_CAP_PMU_CAPABILITY should be called
>> to enabled PMU as well when user wants to enable PMU.
>>
>> I know currently we only need to disable PMU, but we may need to enable PMU
>> via KVM_CAP_PMU_CAPABILITY soon.
>>
>> We are working on the new KVM mediated vPMU framework, Sean suggest to
>> leverage KVM_CAP_PMU_CAPABILITY to enable mediated vPMU dynamically
>> (https://urldefense.com/v3/__https://lore.kernel.org/all/Zz4uhmuPcZl9vJVr@google.com/__;!!ACWV5N9M2RV99hQ!JQx8CdjEI-J6WbzbvB7vHcZ0nJPkzUhvl6TvWvDorAal1XAC17dwpRa6b6Xlva--pK-4ej3Ota0k9SJl3BUWXKTew70$ ). So It would be
>> better if the enable logic can be added here as well.
>>
>> Thanks.
> 
> Hi Dongli,
> 
> May I know if you have plan to continue to update this patch recently? As
> previous comment said, our KVM mediated vPMU solution needs qemu to
> explicitly call KVM_CAP_PMU_CAPABILITY to enable mediated vPMU as well. If
> you have no plan to update this patch recently, would you mind me to pick
> up this patch
> (https://urldefense.com/v3/__https://lore.kernel.org/all/20221119122901.2469-2-dongli.zhang@oracle.com/__;!!ACWV5N9M2RV99hQ!JQx8CdjEI-J6WbzbvB7vHcZ0nJPkzUhvl6TvWvDorAal1XAC17dwpRa6b6Xlva--pK-4ej3Ota0k9SJl3BUWzQmZ_yA$ )
> and post with other our mediated vPMU related patches to enable mediated vPMU?
> 
> Thanks!
> 
> Dapeng Mi


Sorry for the delay — it took some effort to learn about mediated vPMU (as
you suggested) to adapt this patch accordingly.

Yes, feel free to pick up this patch for mediated vPMU, as I don’t want to
block your work, although, I do plan to continue updating it.

I am continuing working on it, but my primary objective is to reset the AMD
PMU during QEMU reset, which depends on KVM_PMU_CAP_DISABLE.

[PATCH 5/7] target/i386/kvm: Reset AMD PMU registers during VM reset
[PATCH 6/7] target/i386/kvm: Support perfmon-v2 for reset

Would you mind keeping me updated on any changes/discussions you make to
QEMU on KVM_PMU_CAP_DISABLE for mediated vPMU? That way, I can adjust my
code accordingly once your QEMU patch for KVM_PMU_CAP_DISABLE is finalized.

In the meantime, I am continuing working on the entire patchset and I can
change the code when you post the relevant QEMU changes on
KVM_PMU_CAP_DISABLE soon.

Would that work for you?

Thank you very much!

Dongli Zhang