* [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1
@ 2022-07-12 13:50 Vitaly Kuznetsov
2022-07-12 14:27 ` Maxim Levitsky
2022-07-12 20:13 ` Dongli Zhang
0 siblings, 2 replies; 6+ messages in thread
From: Vitaly Kuznetsov @ 2022-07-12 13:50 UTC (permalink / raw)
To: kvm, Paolo Bonzini, Sean Christopherson
Cc: Wanpeng Li, Jim Mattson, Maxim Levitsky, linux-kernel
Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to
hang upon boot or shortly after when a non-default TSC frequency was
set for L1. The issue is observed on a host where TSC scaling is
supported. The problem appears to be that Windows doesn't use TSC
frequency for its guests even when the feature is advertised and KVM
filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from
L1's. This leads to L2 running with the default frequency (matching
host's) while L1 is running with an altered one.
Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when
it was set for L1. TSC_MULTIPLIER is already correctly computed and
written by prepare_vmcs02().
Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
arch/x86/kvm/vmx/nested.c | 1 -
1 file changed, 1 deletion(-)
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 778f82015f03..bfa366938c49 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0
SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
SECONDARY_EXEC_APIC_REGISTER_VIRT |
SECONDARY_EXEC_ENABLE_VMFUNC |
- SECONDARY_EXEC_TSC_SCALING |
SECONDARY_EXEC_DESC);
if (nested_cpu_has(vmcs12,
--
2.35.3
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1
2022-07-12 13:50 [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1 Vitaly Kuznetsov
@ 2022-07-12 14:27 ` Maxim Levitsky
2022-07-12 15:33 ` Vitaly Kuznetsov
2022-07-12 20:13 ` Dongli Zhang
1 sibling, 1 reply; 6+ messages in thread
From: Maxim Levitsky @ 2022-07-12 14:27 UTC (permalink / raw)
To: Vitaly Kuznetsov, kvm, Paolo Bonzini, Sean Christopherson
Cc: Wanpeng Li, Jim Mattson, linux-kernel
On Tue, 2022-07-12 at 15:50 +0200, Vitaly Kuznetsov wrote:
> Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to
> hang upon boot or shortly after when a non-default TSC frequency was
> set for L1. The issue is observed on a host where TSC scaling is
> supported. The problem appears to be that Windows doesn't use TSC
> frequency for its guests even when the feature is advertised and KVM
> filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from
> L1's. This leads to L2 running with the default frequency (matching
> host's) while L1 is running with an altered one.
Ouch.
I guess that needs a Fixes tag?
Fixes: d041b5ea93352b ("KVM: nVMX: Enable nested TSC scaling")
Also this is thankfully Intel specific, because in AMD you can't enable
TSC scaling - there is just an MSR with default value of 1.0,
which one can change if TSC scaling is supported in CPUID.
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Best regards,
Maxim Levitsky
>
> Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when
> it was set for L1. TSC_MULTIPLIER is already correctly computed and
> written by prepare_vmcs02().
>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
> arch/x86/kvm/vmx/nested.c | 1 -
> 1 file changed, 1 deletion(-)
>
> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
> index 778f82015f03..bfa366938c49 100644
> --- a/arch/x86/kvm/vmx/nested.c
> +++ b/arch/x86/kvm/vmx/nested.c
> @@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0
> SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
> SECONDARY_EXEC_APIC_REGISTER_VIRT |
> SECONDARY_EXEC_ENABLE_VMFUNC |
> - SECONDARY_EXEC_TSC_SCALING |
> SECONDARY_EXEC_DESC);
>
> if (nested_cpu_has(vmcs12,
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1
2022-07-12 14:27 ` Maxim Levitsky
@ 2022-07-12 15:33 ` Vitaly Kuznetsov
2022-07-12 19:10 ` Sean Christopherson
0 siblings, 1 reply; 6+ messages in thread
From: Vitaly Kuznetsov @ 2022-07-12 15:33 UTC (permalink / raw)
To: Maxim Levitsky, kvm, Paolo Bonzini, Sean Christopherson
Cc: Wanpeng Li, Jim Mattson, linux-kernel
Maxim Levitsky <mlevitsk@redhat.com> writes:
> On Tue, 2022-07-12 at 15:50 +0200, Vitaly Kuznetsov wrote:
>> Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to
>> hang upon boot or shortly after when a non-default TSC frequency was
>> set for L1. The issue is observed on a host where TSC scaling is
>> supported. The problem appears to be that Windows doesn't use TSC
>> frequency
^^^ scaling ^^^
>> for its guests even when the feature is advertised and KVM
>> filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from
>> L1's. This leads to L2 running with the default frequency (matching
>> host's) while L1 is running with an altered one.
>
> Ouch.
>
> I guess that needs a Fixes tag?
>
> Fixes: d041b5ea93352b ("KVM: nVMX: Enable nested TSC scaling")
>
I dismissed that because prior to d041b5ea93352b SECONDARY_EXEC_TSC_SCALING
was filtered out in nested_vmx_setup_ctls_msrs() but now I think I was
wrong, SECONDARY_EXEC_TSC_SCALING was likely kept in VMCS02 regardless
of that. Will add in v2.
> Also this is thankfully Intel specific, because in AMD you can't enable
> TSC scaling - there is just an MSR with default value of 1.0,
> which one can change if TSC scaling is supported in CPUID.
>
> Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Thanks!
> Best regards,
> Maxim Levitsky
>
>
>>
>> Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when
>> it was set for L1. TSC_MULTIPLIER is already correctly computed and
>> written by prepare_vmcs02().
>>
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>> arch/x86/kvm/vmx/nested.c | 1 -
>> 1 file changed, 1 deletion(-)
>>
>> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
>> index 778f82015f03..bfa366938c49 100644
>> --- a/arch/x86/kvm/vmx/nested.c
>> +++ b/arch/x86/kvm/vmx/nested.c
>> @@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0
>> SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
>> SECONDARY_EXEC_APIC_REGISTER_VIRT |
>> SECONDARY_EXEC_ENABLE_VMFUNC |
>> - SECONDARY_EXEC_TSC_SCALING |
>> SECONDARY_EXEC_DESC);
>>
>> if (nested_cpu_has(vmcs12,
>
>
--
Vitaly
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1
2022-07-12 15:33 ` Vitaly Kuznetsov
@ 2022-07-12 19:10 ` Sean Christopherson
0 siblings, 0 replies; 6+ messages in thread
From: Sean Christopherson @ 2022-07-12 19:10 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: Maxim Levitsky, kvm, Paolo Bonzini, Wanpeng Li, Jim Mattson,
linux-kernel
On Tue, Jul 12, 2022, Vitaly Kuznetsov wrote:
> Maxim Levitsky <mlevitsk@redhat.com> writes:
>
> > On Tue, 2022-07-12 at 15:50 +0200, Vitaly Kuznetsov wrote:
> >> Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to
> >> hang upon boot or shortly after when a non-default TSC frequency was
> >> set for L1. The issue is observed on a host where TSC scaling is
> >> supported. The problem appears to be that Windows doesn't use TSC
> >> frequency
>
> ^^^ scaling ^^^
>
> >> for its guests even when the feature is advertised and KVM
> >> filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from
> >> L1's. This leads to L2 running with the default frequency (matching
> >> host's) while L1 is running with an altered one.
> >
> > Ouch.
> >
> > I guess that needs a Fixes tag?
> >
> > Fixes: d041b5ea93352b ("KVM: nVMX: Enable nested TSC scaling")
> >
>
> I dismissed that because prior to d041b5ea93352b SECONDARY_EXEC_TSC_SCALING
> was filtered out in nested_vmx_setup_ctls_msrs() but now I think I was
> wrong, SECONDARY_EXEC_TSC_SCALING was likely kept in VMCS02 regardless
> of that. Will add in v2.
Yep, it would have been kept in vmcs02 even though the feature wasn't advertised
to the L1 VMM. A Cc for stable is warranted as well.
I added this (with the tags and s/frequency/scaling) to the queue of patches for
5.20 I have lined up for Paolo to consume on his return. Paolo and I haven't
hashed out how we'll actually manage anything, i.e. my list is speculative, but
unless you hear otherwise, no need to send a v2.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1
2022-07-12 13:50 [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1 Vitaly Kuznetsov
2022-07-12 14:27 ` Maxim Levitsky
@ 2022-07-12 20:13 ` Dongli Zhang
2022-07-13 7:44 ` Vitaly Kuznetsov
1 sibling, 1 reply; 6+ messages in thread
From: Dongli Zhang @ 2022-07-12 20:13 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: Wanpeng Li, Jim Mattson, Maxim Levitsky, linux-kernel, kvm,
Paolo Bonzini, Sean Christopherson
Hi Vitaly,
On 7/12/22 6:50 AM, Vitaly Kuznetsov wrote:
> Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to
> hang upon boot or shortly after when a non-default TSC frequency was
> set for L1. The issue is observed on a host where TSC scaling is
Would you mind helping clarify if it is L1 or L2 that hangs?
The commit message "Windows 10/11 guests with Hyper-V role (WSL2)" confuses me
if it is L1 or L2 (perhaps due to my lack of knowledge on hyper-v) that hangs.
Thank you very much!
Dongli Zhang
> supported. The problem appears to be that Windows doesn't use TSC
> frequency for its guests even when the feature is advertised and KVM
> filters SECONDARY_EXEC_TSC_SCALING out when creating L2 controls from
> L1's. This leads to L2 running with the default frequency (matching
> host's) while L1 is running with an altered one.
>
> Keep SECONDARY_EXEC_TSC_SCALING in secondary exec controls for L2 when
> it was set for L1. TSC_MULTIPLIER is already correctly computed and
> written by prepare_vmcs02().
>
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
> arch/x86/kvm/vmx/nested.c | 1 -
> 1 file changed, 1 deletion(-)
>
> diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
> index 778f82015f03..bfa366938c49 100644
> --- a/arch/x86/kvm/vmx/nested.c
> +++ b/arch/x86/kvm/vmx/nested.c
> @@ -2284,7 +2284,6 @@ static void prepare_vmcs02_early(struct vcpu_vmx *vmx, struct loaded_vmcs *vmcs0
> SECONDARY_EXEC_VIRTUAL_INTR_DELIVERY |
> SECONDARY_EXEC_APIC_REGISTER_VIRT |
> SECONDARY_EXEC_ENABLE_VMFUNC |
> - SECONDARY_EXEC_TSC_SCALING |
> SECONDARY_EXEC_DESC);
>
> if (nested_cpu_has(vmcs12,
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1
2022-07-12 20:13 ` Dongli Zhang
@ 2022-07-13 7:44 ` Vitaly Kuznetsov
0 siblings, 0 replies; 6+ messages in thread
From: Vitaly Kuznetsov @ 2022-07-13 7:44 UTC (permalink / raw)
To: Dongli Zhang
Cc: Wanpeng Li, Jim Mattson, Maxim Levitsky, linux-kernel, kvm,
Paolo Bonzini, Sean Christopherson
Dongli Zhang <dongli.zhang@oracle.com> writes:
> Hi Vitaly,
>
> On 7/12/22 6:50 AM, Vitaly Kuznetsov wrote:
>> Windows 10/11 guests with Hyper-V role (WSL2) enabled are observed to
>> hang upon boot or shortly after when a non-default TSC frequency was
>> set for L1. The issue is observed on a host where TSC scaling is
>
> Would you mind helping clarify if it is L1 or L2 that hangs?
>
> The commit message "Windows 10/11 guests with Hyper-V role (WSL2)" confuses me
> if it is L1 or L2 (perhaps due to my lack of knowledge on hyper-v) that hangs.
>
I think it's L2 but I'm not sure: there's no easy way to interract with
L1 (Hyper-V) directly, all the interfaces (UI, network,..) are handled
by L2 (Windows). Prior to the observed 'hang' Hyper-V (L1) programms
synthetic timer in KVM too far in the future but my guess is that it's
doing that on Windows' (L2) behalf, basically just relaying the
request. The issue only shows up with 'hv-reenlightenment' +
'hv-frequencies' (in QEMU's terms) features as in this case both Hyper-V
(L1) and Windows (L2) trust the information about TSC frequency from KVM
but the information turns out to be incorrect for Windows (L2).
--
Vitaly
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2022-07-13 7:45 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-07-12 13:50 [PATCH] KVM: nVMX: Always enable TSC scaling for L2 when it was enabled for L1 Vitaly Kuznetsov
2022-07-12 14:27 ` Maxim Levitsky
2022-07-12 15:33 ` Vitaly Kuznetsov
2022-07-12 19:10 ` Sean Christopherson
2022-07-12 20:13 ` Dongli Zhang
2022-07-13 7:44 ` Vitaly Kuznetsov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox