[PATCH] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[PATCH] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Wanpeng Li]

From: Wanpeng Li <wanpengli@tencent.com>

The per-cpu vsyscall pvclock data pointer assigns either an element of the 
static array hv_clock_boot (#vCPU <= 64) or dynamically allocated memory 
hvclock_mem (vCPU > 64), the dynamically memory will not be allocated if 
kvmclock vsyscall is disabled, this can result in cpu hotpluged fails in 
kvmclock_setup_percpu() which returns -ENOMEM. This patch fixes it by not 
assigning vsyscall pvclock data pointer if kvmclock vdso_clock_mode is not 
VDSO_CLOCKMODE_PVCLOCK.

Fixes: 6a1cac56f4 ("x86/kvm: Use __bss_decrypted attribute in shared variables")
Reported-by: Zelin Deng <zelin.deng@linux.alibaba.com>
Tested-by: Haiwei Li <lihaiwei@tencent.com>
Cc: Brijesh Singh <brijesh.singh@amd.com>
Cc: stable@vger.kernel.org#v4.19-rc5+
Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
---
 arch/x86/kernel/kvmclock.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
index aa59374..0624290 100644
--- a/arch/x86/kernel/kvmclock.c
+++ b/arch/x86/kernel/kvmclock.c
@@ -296,7 +296,8 @@ static int kvmclock_setup_percpu(unsigned int cpu)
 	 * pointers. So carefully check. CPU0 has been set up in init
 	 * already.
 	 */
-	if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)))
+	if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)) ||
+	    (kvm_clock.vdso_clock_mode != VDSO_CLOCKMODE_PVCLOCK))
 		return 0;
 
 	/* Use the static page for the first CPUs, allocate otherwise */
-- 
2.7.4

Re: [PATCH] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Vitaly Kuznetsov]

Wanpeng Li <kernellwp@gmail.com> writes:

> From: Wanpeng Li <wanpengli@tencent.com>
>
> The per-cpu vsyscall pvclock data pointer assigns either an element of the 
> static array hv_clock_boot (#vCPU <= 64) or dynamically allocated memory 
> hvclock_mem (vCPU > 64), the dynamically memory will not be allocated if 
> kvmclock vsyscall is disabled, this can result in cpu hotpluged fails in 
> kvmclock_setup_percpu() which returns -ENOMEM. This patch fixes it by not 
> assigning vsyscall pvclock data pointer if kvmclock vdso_clock_mode is not 
> VDSO_CLOCKMODE_PVCLOCK.
>
> Fixes: 6a1cac56f4 ("x86/kvm: Use __bss_decrypted attribute in shared variables")
> Reported-by: Zelin Deng <zelin.deng@linux.alibaba.com>
> Tested-by: Haiwei Li <lihaiwei@tencent.com>
> Cc: Brijesh Singh <brijesh.singh@amd.com>
> Cc: stable@vger.kernel.org#v4.19-rc5+
> Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
> ---
>  arch/x86/kernel/kvmclock.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
> index aa59374..0624290 100644
> --- a/arch/x86/kernel/kvmclock.c
> +++ b/arch/x86/kernel/kvmclock.c
> @@ -296,7 +296,8 @@ static int kvmclock_setup_percpu(unsigned int cpu)
>  	 * pointers. So carefully check. CPU0 has been set up in init
>  	 * already.
>  	 */
> -	if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)))
> +	if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)) ||
> +	    (kvm_clock.vdso_clock_mode != VDSO_CLOCKMODE_PVCLOCK))
>  		return 0;

The comment above should probably be updated as it is not clear why we
check kvm_clock.vdso_clock_mode here. Actually, I would even suggest we
introduce a 'kvmclock_tsc_stable' global instead to avoid this indirect
check.

>  
>  	/* Use the static page for the first CPUs, allocate otherwise */

Also, would it be better if we just avoid cpuhp_setup_state() call in
this case? E.g. both these ideas combined (completely untested):

diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
index aa593743acf6..0827aef3ccb8 100644
--- a/arch/x86/kernel/kvmclock.c
+++ b/arch/x86/kernel/kvmclock.c
@@ -25,6 +25,7 @@
 
 static int kvmclock __initdata = 1;
 static int kvmclock_vsyscall __initdata = 1;
+static bool kvmclock_tsc_stable __ro_after_init = true;
 static int msr_kvm_system_time __ro_after_init = MSR_KVM_SYSTEM_TIME;
 static int msr_kvm_wall_clock __ro_after_init = MSR_KVM_WALL_CLOCK;
 static u64 kvm_sched_clock_offset __ro_after_init;
@@ -275,8 +276,10 @@ static int __init kvm_setup_vsyscall_timeinfo(void)
                return 0;
 
        flags = pvclock_read_flags(&hv_clock_boot[0].pvti);
-       if (!(flags & PVCLOCK_TSC_STABLE_BIT))
+       if (!(flags & PVCLOCK_TSC_STABLE_BIT)) {
+               kvmclock_tsc_stable = false;
                return 0;
+       }
 
        kvm_clock.vdso_clock_mode = VDSO_CLOCKMODE_PVCLOCK;
 #endif
@@ -325,7 +328,8 @@ void __init kvmclock_init(void)
                return;
        }
 
-       if (cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "kvmclock:setup_percpu",
+       if (kvmclock_tsc_stable &&
+           cpuhp_setup_state(CPUHP_BP_PREPARE_DYN, "kvmclock:setup_percpu",
                              kvmclock_setup_percpu, NULL) < 0) {
                return;
        }

-- 
Vitaly

Re: [PATCH] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Wanpeng Li]

On Thu, 14 Jan 2021 at 21:45, Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
> Wanpeng Li <kernellwp@gmail.com> writes:
>
> > From: Wanpeng Li <wanpengli@tencent.com>
> >
> > The per-cpu vsyscall pvclock data pointer assigns either an element of the
> > static array hv_clock_boot (#vCPU <= 64) or dynamically allocated memory
> > hvclock_mem (vCPU > 64), the dynamically memory will not be allocated if
> > kvmclock vsyscall is disabled, this can result in cpu hotpluged fails in
> > kvmclock_setup_percpu() which returns -ENOMEM. This patch fixes it by not
> > assigning vsyscall pvclock data pointer if kvmclock vdso_clock_mode is not
> > VDSO_CLOCKMODE_PVCLOCK.
> >
> > Fixes: 6a1cac56f4 ("x86/kvm: Use __bss_decrypted attribute in shared variables")
> > Reported-by: Zelin Deng <zelin.deng@linux.alibaba.com>
> > Tested-by: Haiwei Li <lihaiwei@tencent.com>
> > Cc: Brijesh Singh <brijesh.singh@amd.com>
> > Cc: stable@vger.kernel.org#v4.19-rc5+
> > Signed-off-by: Wanpeng Li <wanpengli@tencent.com>
> > ---
> >  arch/x86/kernel/kvmclock.c | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/arch/x86/kernel/kvmclock.c b/arch/x86/kernel/kvmclock.c
> > index aa59374..0624290 100644
> > --- a/arch/x86/kernel/kvmclock.c
> > +++ b/arch/x86/kernel/kvmclock.c
> > @@ -296,7 +296,8 @@ static int kvmclock_setup_percpu(unsigned int cpu)
> >        * pointers. So carefully check. CPU0 has been set up in init
> >        * already.
> >        */
> > -     if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)))
> > +     if (!cpu || (p && p != per_cpu(hv_clock_per_cpu, 0)) ||
> > +         (kvm_clock.vdso_clock_mode != VDSO_CLOCKMODE_PVCLOCK))
> >               return 0;
>
> The comment above should probably be updated as it is not clear why we
> check kvm_clock.vdso_clock_mode here. Actually, I would even suggest we
> introduce a 'kvmclock_tsc_stable' global instead to avoid this indirect
> check.

I prefer to update the comment above, assign vsyscall pvclock data
pointer iff kvmclock vsyscall is enabled.

    Wanpeng

Re: [PATCH] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Paolo Bonzini]

On 15/01/21 02:15, Wanpeng Li wrote:
>> The comment above should probably be updated as it is not clear why we
>> check kvm_clock.vdso_clock_mode here. Actually, I would even suggest we
>> introduce a 'kvmclock_tsc_stable' global instead to avoid this indirect
>> check.
> I prefer to update the comment above, assign vsyscall pvclock data
> pointer iff kvmclock vsyscall is enabled.

Are you going to send v2?

Paolo

Re: [PATCH] KVM: kvmclock: Fix vCPUs > 64 can't be online/hotpluged

[Wanpeng Li]

On Tue, 19 Jan 2021 at 02:27, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 15/01/21 02:15, Wanpeng Li wrote:
> >> The comment above should probably be updated as it is not clear why we
> >> check kvm_clock.vdso_clock_mode here. Actually, I would even suggest we
> >> introduce a 'kvmclock_tsc_stable' global instead to avoid this indirect
> >> check.
> > I prefer to update the comment above, assign vsyscall pvclock data
> > pointer iff kvmclock vsyscall is enabled.
>
> Are you going to send v2?

Yes. :) https://lore.kernel.org/kvm/1610960877-3110-1-git-send-email-wanpengli@tencent.com/

    Wanpeng