From mboxrd@z Thu Jan 1 00:00:00 1970 From: Vitaly Kuznetsov Date: Tue, 15 Nov 2022 10:30:14 +0100 Subject: [PATCH 10/44] KVM: VMX: Clean up eVMCS enabling if KVM initialization fails In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-11-seanjc@google.com> <87mt98qfi2.fsf@ovpn-194-252.brq.redhat.com> Message-ID: <87sfikmuop.fsf@redhat.com> List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sean Christopherson writes: > On Thu, Nov 03, 2022, Vitaly Kuznetsov wrote: >> Sean Christopherson writes: >> > + /* >> > + * Reset everything to support using non-enlightened VMCS access later >> > + * (e.g. when we reload the module with enlightened_vmcs=0) >> > + */ >> > + for_each_online_cpu(cpu) { >> > + vp_ap = hv_get_vp_assist_page(cpu); >> > + >> > + if (!vp_ap) >> > + continue; >> > + >> > + vp_ap->nested_control.features.directhypercall = 0; >> > + vp_ap->current_nested_vmcs = 0; >> > + vp_ap->enlighten_vmentry = 0; >> > + } >> >> Unrelated to your patch but while looking at this code I got curious >> about why don't we need a protection against CPU offlining here. Turns >> out that even when we offline a CPU, its VP assist page remains >> allocated (see hv_cpu_die()), we just write '0' to the MSR and thus > > Heh, "die". Hyper-V is quite dramatic. > >> accessing the page is safe. The consequent hv_cpu_init(), however, does >> not restore VP assist page when it's already allocated: >> >> # rdmsr -p 24 0x40000073 >> 10212f001 >> # echo 0 > /sys/devices/system/cpu/cpu24/online >> # echo 1 > /sys/devices/system/cpu/cpu24/online >> # rdmsr -p 24 0x40000073 >> 0 >> >> The culprit is commit e5d9b714fe402 ("x86/hyperv: fix root partition >> faults when writing to VP assist page MSR"). A patch is inbound. >> >> 'hv_root_partition' case is different though. We do memunmap() and reset >> VP assist page to zero so it is theoretically possible we're going to >> clash. Unless I'm missing some obvious reason why module unload can't >> coincide with CPU offlining, we may be better off surrounding this with >> cpus_read_lock()/cpus_read_unlock(). > > I finally see what you're concerned about. If a CPU goes offline and its assist > page is unmapped, zeroing out the nested/eVMCS stuff will fault. > > I think the real problem is that the purging of the eVMCS is in the wrong place. > Move the clearing to vmx_hardware_disable() and then the CPU hotplug bug goes > away once KVM disables hotplug during hardware enabling/disable later in the series. > There's no need to wait until module exit, e.g. it's not like it costs much to > clear a few variables, and IIUC the state is used only when KVM is actively using > VMX/eVMCS. > > However, I believe there's a second bug. KVM's CPU online hook is called before > Hyper-V's online hook (CPUHP_AP_ONLINE_DYN). Before this series, which moves KVM's > hook from STARTING to ONLINE, KVM's hook is waaaay before Hyper-V's. That means > that hv_cpu_init()'s allocation of the VP assist page will come _after_ KVM's > check in vmx_hardware_enable() > > /* > * This can happen if we hot-added a CPU but failed to allocate > * VP assist page for it. > */ > if (static_branch_unlikely(&enable_evmcs) && > !hv_get_vp_assist_page(cpu)) > return -EFAULT; > > I.e. CPU hotplug will never work if KVM is running VMs as a Hyper-V guest. I bet > you can repro by doing a SUSPEND+RESUME. > > Can you try to see if that's actually a bug? If so, the only sane fix seems to > be to add a dedicated ONLINE action for Hyper-V. It seems we can't get away without a dedicated stage for Hyper-V anyway, e.g. see our discussion with Michael: https://lore.kernel.org/linux-hyperv/878rkqr7ku.fsf at ovpn-192-136.brq.redhat.com/ All these issues are more or less "theoretical" as there's no real CPU hotplug on Hyper-V/Azure. Yes, it is possible to trigger problems by doing CPU offline/online but I don't see how this may come handy outside of testing envs. > Per patch > > KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section > > from this series, CPUHP_AP_KVM_ONLINE needs to be before CPUHP_AP_SCHED_WAIT_EMPTY > to ensure there are no tasks, i.e. no vCPUs, running on the to-be-unplugged CPU. > > Back to the original bug, proposed fix is below. The other advantage of moving > the reset to hardware disabling is that the "cleanup" is just disabling the static > key, and at that point can simply be deleted as there's no need to disable the > static key when kvm-intel is unloaded since kvm-intel owns the key. I.e. this > patch (that we're replying to) would get replaced with a patch to delete the > disabling of the static key. > >From a quick glance looks good to me, I'll try to find some time to work on this issue. I will likely end up proposing a dedicated CPU hotplug stage for Hyper-V (which needs to happen before KVM's CPUHP_AP_KVM_ONLINE on CPU hotplug and after on unplug) anyway. Thanks for looking into this! > -- > From: Sean Christopherson > Date: Thu, 10 Nov 2022 17:28:08 -0800 > Subject: [PATCH] KVM: VMX: Reset eVMCS controls in VP assist page during > hardware disabling > > Reset the eVMCS controls in the per-CPU VP assist page during hardware > disabling instead of waiting until kvm-intel's module exit. The controls > are activated if and only if KVM creates a VM, i.e. don't need to be > reset if hardware is never enabled. > > Doing the reset during hardware disabling will naturally fix a potential > NULL pointer deref bug once KVM disables CPU hotplug while enabling and > disabling hardware (which is necessary to fix a variety of bugs). If the > kernel is running as the root partition, the VP assist page is unmapped > during CPU hot unplug, and so KVM's clearing of the eVMCS controls needs > to occur with CPU hot(un)plug disabled, otherwise KVM could attempt to > write to a CPU's VP assist page after it's unmapped. > > Reported-by: Vitaly Kuznetsov > Signed-off-by: Sean Christopherson > --- > arch/x86/kvm/vmx/vmx.c | 50 +++++++++++++++++++++++++----------------- > 1 file changed, 30 insertions(+), 20 deletions(-) > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c > index aca88524fd1e..ae13aa3e8a1d 100644 > --- a/arch/x86/kvm/vmx/vmx.c > +++ b/arch/x86/kvm/vmx/vmx.c > @@ -552,6 +552,33 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) > return 0; > } > > +static void hv_reset_evmcs(void) > +{ > + struct hv_vp_assist_page *vp_ap; > + > + if (!static_branch_unlikely(&enable_evmcs)) > + return; > + > + /* > + * KVM should enable eVMCS if and only if all CPUs have a VP assist > + * page, and should reject CPU onlining if eVMCS is enabled the CPU > + * doesn't have a VP assist page allocated. > + */ > + vp_ap = hv_get_vp_assist_page(smp_processor_id()); > + if (WARN_ON_ONCE(!vp_ap)) > + return; > + > + /* > + * Reset everything to support using non-enlightened VMCS access later > + * (e.g. when we reload the module with enlightened_vmcs=0) > + */ > + vp_ap->nested_control.features.directhypercall = 0; > + vp_ap->current_nested_vmcs = 0; > + vp_ap->enlighten_vmentry = 0; > +} > + > +#else /* IS_ENABLED(CONFIG_HYPERV) */ > +static void hv_reset_evmcs(void) {} > #endif /* IS_ENABLED(CONFIG_HYPERV) */ > > /* > @@ -2497,6 +2524,8 @@ static void vmx_hardware_disable(void) > if (cpu_vmxoff()) > kvm_spurious_fault(); > > + hv_reset_evmcs(); > + > intel_pt_handle_vmx(0); > } > > @@ -8463,27 +8492,8 @@ static void vmx_exit(void) > kvm_exit(); > > #if IS_ENABLED(CONFIG_HYPERV) > - if (static_branch_unlikely(&enable_evmcs)) { > - int cpu; > - struct hv_vp_assist_page *vp_ap; > - /* > - * Reset everything to support using non-enlightened VMCS > - * access later (e.g. when we reload the module with > - * enlightened_vmcs=0) > - */ > - for_each_online_cpu(cpu) { > - vp_ap = hv_get_vp_assist_page(cpu); > - > - if (!vp_ap) > - continue; > - > - vp_ap->nested_control.features.directhypercall = 0; > - vp_ap->current_nested_vmcs = 0; > - vp_ap->enlighten_vmentry = 0; > - } > - > + if (static_branch_unlikely(&enable_evmcs)) > static_branch_disable(&enable_evmcs); > - } > #endif > vmx_cleanup_l1d_flush(); > > > base-commit: 5f47ba6894477dfbdc5416467a25fb7acb47d404 -- Vitaly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E090C4332F for ; Tue, 15 Nov 2022 09:30:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 053344B8CA; Tue, 15 Nov 2022 04:30:28 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@redhat.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OmerKI8XIgWg; Tue, 15 Nov 2022 04:30:26 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 6E6034B8E2; Tue, 15 Nov 2022 04:30:26 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 8D9DB4B8DB for ; Tue, 15 Nov 2022 04:30:24 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GS9AZa6sNHEO for ; Tue, 15 Nov 2022 04:30:22 -0500 (EST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 2F8204B8CA for ; Tue, 15 Nov 2022 04:30:22 -0500 (EST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668504621; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=AcwCInwO9hR30N2K8Kb8YW9jottbyNINOUUeMTmj1TqPCWLZHXLCkK+XYSPZPok2OPGd9d SQgbT8Xu+lpGLwgLbr5aZ9BwgSfE/RT9F8pMaePufjBXkmH7Ez3Hq1SOldLh+YtwHibUgq X40Y+9x4QF9yc2maA6rxNexOQFujzv4= Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com [209.85.218.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-306-MvYcW0AmNtWGhJa7bp3UvQ-1; Tue, 15 Nov 2022 04:30:18 -0500 X-MC-Unique: MvYcW0AmNtWGhJa7bp3UvQ-1 Received: by mail-ej1-f69.google.com with SMTP id hp16-20020a1709073e1000b007adf5a83df7so6913742ejc.1 for ; Tue, 15 Nov 2022 01:30:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=dfogxKytqYJCzDsW0UPW0vOrXz8pcbiKmNAvuLL9MyVVshL02nCyJHX+dUJBWFcJyi OmIQwuMRznTxPOg4Skn0W1IUE9ls7a6Hghozqj4bRZidXnDcS3o4whG7b4KEZ+8qUwJx DHo1Upc4cgToRf+Vlbvf4giwtBfXyAVEnTu7d8lGsLrunjebAOOUNdV8u/WDtVpYyNcA bFDxOpzGeYT7TvWgkaDCOr1FH3ZEtKEKzqRZ/mxemKl/eOE5ZAnsdyaZIKWQdLT6Xa5m pLgwindnlQP3+B48OA86qKDXhPXDbtgNcVMzN5J0G0s00Mf5XqswbsZI0igXtcy0LWQS B1Yg== X-Gm-Message-State: ANoB5pkHVzgbOUg4dX5g0+qM1X3OsbosIVFm5J3rumSlgr4vtVTxh2aR 2WgGxjOvXnUbuMsgaBR287gJNHOGW9Z7WxpGVeAmM/rrWpejbof6dfvE8cPRhCZz802TOCL4Vao l77OiBFEUxnu32mhvcWS64w15 X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489196edt.387.1668504617490; Tue, 15 Nov 2022 01:30:17 -0800 (PST) X-Google-Smtp-Source: AA0mqf5Zsa4CnM7XTxV75wUJGjjLzRgPV7NQ4412B1eIJSCcezAuycjNE+mpvlda5nQMKDxwK/vVkA== X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489153edt.387.1668504617196; Tue, 15 Nov 2022 01:30:17 -0800 (PST) Received: from fedora (nat-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id ew13-20020a056402538d00b004642b35f89esm5950875edb.9.2022.11.15.01.30.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 01:30:16 -0800 (PST) From: Vitaly Kuznetsov To: Sean Christopherson Subject: Re: [PATCH 10/44] KVM: VMX: Clean up eVMCS enabling if KVM initialization fails In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-11-seanjc@google.com> <87mt98qfi2.fsf@ovpn-194-252.brq.redhat.com> Date: Tue, 15 Nov 2022 10:30:14 +0100 Message-ID: <87sfikmuop.fsf@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Cc: kvm@vger.kernel.org, David Hildenbrand , Atish Patra , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Claudio Imbrenda , kvmarm@lists.cs.columbia.edu, linux-s390@vger.kernel.org, Janosch Frank , Michael Ellerman , Huacai Chen , Aleksandar Markovic , Palmer Dabbelt , Christian Borntraeger , Matthew Rosato , Chao Gao , Eric Farman , Albert Ou , Paul Walmsley , Yuan Yao , kvmarm@lists.linux.dev, Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Isaku Yamahata , Fabiano Rosas , linux-mips@vger.kernel.org, kvm-riscv@lists.infradead.org, Marc Zyngier , Paolo Bonzini , linuxppc-dev@lists.ozlabs.org X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu Sean Christopherson writes: > On Thu, Nov 03, 2022, Vitaly Kuznetsov wrote: >> Sean Christopherson writes: >> > + /* >> > + * Reset everything to support using non-enlightened VMCS access later >> > + * (e.g. when we reload the module with enlightened_vmcs=0) >> > + */ >> > + for_each_online_cpu(cpu) { >> > + vp_ap = hv_get_vp_assist_page(cpu); >> > + >> > + if (!vp_ap) >> > + continue; >> > + >> > + vp_ap->nested_control.features.directhypercall = 0; >> > + vp_ap->current_nested_vmcs = 0; >> > + vp_ap->enlighten_vmentry = 0; >> > + } >> >> Unrelated to your patch but while looking at this code I got curious >> about why don't we need a protection against CPU offlining here. Turns >> out that even when we offline a CPU, its VP assist page remains >> allocated (see hv_cpu_die()), we just write '0' to the MSR and thus > > Heh, "die". Hyper-V is quite dramatic. > >> accessing the page is safe. The consequent hv_cpu_init(), however, does >> not restore VP assist page when it's already allocated: >> >> # rdmsr -p 24 0x40000073 >> 10212f001 >> # echo 0 > /sys/devices/system/cpu/cpu24/online >> # echo 1 > /sys/devices/system/cpu/cpu24/online >> # rdmsr -p 24 0x40000073 >> 0 >> >> The culprit is commit e5d9b714fe402 ("x86/hyperv: fix root partition >> faults when writing to VP assist page MSR"). A patch is inbound. >> >> 'hv_root_partition' case is different though. We do memunmap() and reset >> VP assist page to zero so it is theoretically possible we're going to >> clash. Unless I'm missing some obvious reason why module unload can't >> coincide with CPU offlining, we may be better off surrounding this with >> cpus_read_lock()/cpus_read_unlock(). > > I finally see what you're concerned about. If a CPU goes offline and its assist > page is unmapped, zeroing out the nested/eVMCS stuff will fault. > > I think the real problem is that the purging of the eVMCS is in the wrong place. > Move the clearing to vmx_hardware_disable() and then the CPU hotplug bug goes > away once KVM disables hotplug during hardware enabling/disable later in the series. > There's no need to wait until module exit, e.g. it's not like it costs much to > clear a few variables, and IIUC the state is used only when KVM is actively using > VMX/eVMCS. > > However, I believe there's a second bug. KVM's CPU online hook is called before > Hyper-V's online hook (CPUHP_AP_ONLINE_DYN). Before this series, which moves KVM's > hook from STARTING to ONLINE, KVM's hook is waaaay before Hyper-V's. That means > that hv_cpu_init()'s allocation of the VP assist page will come _after_ KVM's > check in vmx_hardware_enable() > > /* > * This can happen if we hot-added a CPU but failed to allocate > * VP assist page for it. > */ > if (static_branch_unlikely(&enable_evmcs) && > !hv_get_vp_assist_page(cpu)) > return -EFAULT; > > I.e. CPU hotplug will never work if KVM is running VMs as a Hyper-V guest. I bet > you can repro by doing a SUSPEND+RESUME. > > Can you try to see if that's actually a bug? If so, the only sane fix seems to > be to add a dedicated ONLINE action for Hyper-V. It seems we can't get away without a dedicated stage for Hyper-V anyway, e.g. see our discussion with Michael: https://lore.kernel.org/linux-hyperv/878rkqr7ku.fsf@ovpn-192-136.brq.redhat.com/ All these issues are more or less "theoretical" as there's no real CPU hotplug on Hyper-V/Azure. Yes, it is possible to trigger problems by doing CPU offline/online but I don't see how this may come handy outside of testing envs. > Per patch > > KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section > > from this series, CPUHP_AP_KVM_ONLINE needs to be before CPUHP_AP_SCHED_WAIT_EMPTY > to ensure there are no tasks, i.e. no vCPUs, running on the to-be-unplugged CPU. > > Back to the original bug, proposed fix is below. The other advantage of moving > the reset to hardware disabling is that the "cleanup" is just disabling the static > key, and at that point can simply be deleted as there's no need to disable the > static key when kvm-intel is unloaded since kvm-intel owns the key. I.e. this > patch (that we're replying to) would get replaced with a patch to delete the > disabling of the static key. > >From a quick glance looks good to me, I'll try to find some time to work on this issue. I will likely end up proposing a dedicated CPU hotplug stage for Hyper-V (which needs to happen before KVM's CPUHP_AP_KVM_ONLINE on CPU hotplug and after on unplug) anyway. Thanks for looking into this! > -- > From: Sean Christopherson > Date: Thu, 10 Nov 2022 17:28:08 -0800 > Subject: [PATCH] KVM: VMX: Reset eVMCS controls in VP assist page during > hardware disabling > > Reset the eVMCS controls in the per-CPU VP assist page during hardware > disabling instead of waiting until kvm-intel's module exit. The controls > are activated if and only if KVM creates a VM, i.e. don't need to be > reset if hardware is never enabled. > > Doing the reset during hardware disabling will naturally fix a potential > NULL pointer deref bug once KVM disables CPU hotplug while enabling and > disabling hardware (which is necessary to fix a variety of bugs). If the > kernel is running as the root partition, the VP assist page is unmapped > during CPU hot unplug, and so KVM's clearing of the eVMCS controls needs > to occur with CPU hot(un)plug disabled, otherwise KVM could attempt to > write to a CPU's VP assist page after it's unmapped. > > Reported-by: Vitaly Kuznetsov > Signed-off-by: Sean Christopherson > --- > arch/x86/kvm/vmx/vmx.c | 50 +++++++++++++++++++++++++----------------- > 1 file changed, 30 insertions(+), 20 deletions(-) > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c > index aca88524fd1e..ae13aa3e8a1d 100644 > --- a/arch/x86/kvm/vmx/vmx.c > +++ b/arch/x86/kvm/vmx/vmx.c > @@ -552,6 +552,33 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) > return 0; > } > > +static void hv_reset_evmcs(void) > +{ > + struct hv_vp_assist_page *vp_ap; > + > + if (!static_branch_unlikely(&enable_evmcs)) > + return; > + > + /* > + * KVM should enable eVMCS if and only if all CPUs have a VP assist > + * page, and should reject CPU onlining if eVMCS is enabled the CPU > + * doesn't have a VP assist page allocated. > + */ > + vp_ap = hv_get_vp_assist_page(smp_processor_id()); > + if (WARN_ON_ONCE(!vp_ap)) > + return; > + > + /* > + * Reset everything to support using non-enlightened VMCS access later > + * (e.g. when we reload the module with enlightened_vmcs=0) > + */ > + vp_ap->nested_control.features.directhypercall = 0; > + vp_ap->current_nested_vmcs = 0; > + vp_ap->enlighten_vmentry = 0; > +} > + > +#else /* IS_ENABLED(CONFIG_HYPERV) */ > +static void hv_reset_evmcs(void) {} > #endif /* IS_ENABLED(CONFIG_HYPERV) */ > > /* > @@ -2497,6 +2524,8 @@ static void vmx_hardware_disable(void) > if (cpu_vmxoff()) > kvm_spurious_fault(); > > + hv_reset_evmcs(); > + > intel_pt_handle_vmx(0); > } > > @@ -8463,27 +8492,8 @@ static void vmx_exit(void) > kvm_exit(); > > #if IS_ENABLED(CONFIG_HYPERV) > - if (static_branch_unlikely(&enable_evmcs)) { > - int cpu; > - struct hv_vp_assist_page *vp_ap; > - /* > - * Reset everything to support using non-enlightened VMCS > - * access later (e.g. when we reload the module with > - * enlightened_vmcs=0) > - */ > - for_each_online_cpu(cpu) { > - vp_ap = hv_get_vp_assist_page(cpu); > - > - if (!vp_ap) > - continue; > - > - vp_ap->nested_control.features.directhypercall = 0; > - vp_ap->current_nested_vmcs = 0; > - vp_ap->enlighten_vmentry = 0; > - } > - > + if (static_branch_unlikely(&enable_evmcs)) > static_branch_disable(&enable_evmcs); > - } > #endif > vmx_cleanup_l1d_flush(); > > > base-commit: 5f47ba6894477dfbdc5416467a25fb7acb47d404 -- Vitaly _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4ACAB28ED for ; Tue, 15 Nov 2022 09:30:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668504621; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=AcwCInwO9hR30N2K8Kb8YW9jottbyNINOUUeMTmj1TqPCWLZHXLCkK+XYSPZPok2OPGd9d SQgbT8Xu+lpGLwgLbr5aZ9BwgSfE/RT9F8pMaePufjBXkmH7Ez3Hq1SOldLh+YtwHibUgq X40Y+9x4QF9yc2maA6rxNexOQFujzv4= Received: from mail-ej1-f70.google.com (mail-ej1-f70.google.com [209.85.218.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-510-54YAh8qqN-CbvkEafEs8NQ-1; Tue, 15 Nov 2022 04:30:18 -0500 X-MC-Unique: 54YAh8qqN-CbvkEafEs8NQ-1 Received: by mail-ej1-f70.google.com with SMTP id sb4-20020a1709076d8400b007ae596eac08so7073164ejc.22 for ; Tue, 15 Nov 2022 01:30:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=yTndR5GyFmHL1ASZrqI3Qi6YMmPJZ+/xGoINFHBy+2hee5WqcmYRc+owo1D6og3ghs D6qXx6PPa8EZrwoeabXR12kAKXFlBQiGo7tUllTXbqriVCWAgdEQ7hJkubg9FaneEhBF 9hnhKkjTmcerEY2s0fR3BhuugDe9JbuPyZ/70LGKcQZcgMjF5woIDHjoiPGDCtA7U2Jp ERhzAeFPLp43aOE4kk7Wpaww/j88vYDPb/rz8a9CsqXAjJDnYRfRH94yt9ALSwVu7c9r csShS6QFr/oVM/ifz4y07f23nR66W3aceA0OJT8lYXTxWeY22Pa3h2MwpHcgzPlDegA9 1pWg== X-Gm-Message-State: ANoB5pksGXXUvjW9csk9t2X5E+I8eXH6UFwApmN5TlK1i4ZFdFPVeKqc pr2O9+J/FwiGEfa/UtZ0WFUcyGi+reenCR/FULCm8WZMa71Qw915XZVMqzMl01PPoNHBGJJ3HNp dQW3KkxoyEJ+ueSUn X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489181edt.387.1668504617481; Tue, 15 Nov 2022 01:30:17 -0800 (PST) X-Google-Smtp-Source: AA0mqf5Zsa4CnM7XTxV75wUJGjjLzRgPV7NQ4412B1eIJSCcezAuycjNE+mpvlda5nQMKDxwK/vVkA== X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489153edt.387.1668504617196; Tue, 15 Nov 2022 01:30:17 -0800 (PST) Received: from fedora (nat-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id ew13-20020a056402538d00b004642b35f89esm5950875edb.9.2022.11.15.01.30.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 01:30:16 -0800 (PST) From: Vitaly Kuznetsov To: Sean Christopherson Cc: James Morse , Alexandru Elisei , Suzuki K Poulose , Oliver Upton , Atish Patra , David Hildenbrand , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, Isaku Yamahata , Fabiano Rosas , Michael Ellerman , Chao Gao , Thomas Gleixner , Yuan Yao , Paolo Bonzini , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Matthew Rosato , Eric Farman Subject: Re: [PATCH 10/44] KVM: VMX: Clean up eVMCS enabling if KVM initialization fails In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-11-seanjc@google.com> <87mt98qfi2.fsf@ovpn-194-252.brq.redhat.com> Date: Tue, 15 Nov 2022 10:30:14 +0100 Message-ID: <87sfikmuop.fsf@redhat.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain Message-ID: <20221115093014.rK587Ma7F3ywHYfmEJld4Qisl-cb63O-frR23zGWWHc@z> Sean Christopherson writes: > On Thu, Nov 03, 2022, Vitaly Kuznetsov wrote: >> Sean Christopherson writes: >> > + /* >> > + * Reset everything to support using non-enlightened VMCS access later >> > + * (e.g. when we reload the module with enlightened_vmcs=0) >> > + */ >> > + for_each_online_cpu(cpu) { >> > + vp_ap = hv_get_vp_assist_page(cpu); >> > + >> > + if (!vp_ap) >> > + continue; >> > + >> > + vp_ap->nested_control.features.directhypercall = 0; >> > + vp_ap->current_nested_vmcs = 0; >> > + vp_ap->enlighten_vmentry = 0; >> > + } >> >> Unrelated to your patch but while looking at this code I got curious >> about why don't we need a protection against CPU offlining here. Turns >> out that even when we offline a CPU, its VP assist page remains >> allocated (see hv_cpu_die()), we just write '0' to the MSR and thus > > Heh, "die". Hyper-V is quite dramatic. > >> accessing the page is safe. The consequent hv_cpu_init(), however, does >> not restore VP assist page when it's already allocated: >> >> # rdmsr -p 24 0x40000073 >> 10212f001 >> # echo 0 > /sys/devices/system/cpu/cpu24/online >> # echo 1 > /sys/devices/system/cpu/cpu24/online >> # rdmsr -p 24 0x40000073 >> 0 >> >> The culprit is commit e5d9b714fe402 ("x86/hyperv: fix root partition >> faults when writing to VP assist page MSR"). A patch is inbound. >> >> 'hv_root_partition' case is different though. We do memunmap() and reset >> VP assist page to zero so it is theoretically possible we're going to >> clash. Unless I'm missing some obvious reason why module unload can't >> coincide with CPU offlining, we may be better off surrounding this with >> cpus_read_lock()/cpus_read_unlock(). > > I finally see what you're concerned about. If a CPU goes offline and its assist > page is unmapped, zeroing out the nested/eVMCS stuff will fault. > > I think the real problem is that the purging of the eVMCS is in the wrong place. > Move the clearing to vmx_hardware_disable() and then the CPU hotplug bug goes > away once KVM disables hotplug during hardware enabling/disable later in the series. > There's no need to wait until module exit, e.g. it's not like it costs much to > clear a few variables, and IIUC the state is used only when KVM is actively using > VMX/eVMCS. > > However, I believe there's a second bug. KVM's CPU online hook is called before > Hyper-V's online hook (CPUHP_AP_ONLINE_DYN). Before this series, which moves KVM's > hook from STARTING to ONLINE, KVM's hook is waaaay before Hyper-V's. That means > that hv_cpu_init()'s allocation of the VP assist page will come _after_ KVM's > check in vmx_hardware_enable() > > /* > * This can happen if we hot-added a CPU but failed to allocate > * VP assist page for it. > */ > if (static_branch_unlikely(&enable_evmcs) && > !hv_get_vp_assist_page(cpu)) > return -EFAULT; > > I.e. CPU hotplug will never work if KVM is running VMs as a Hyper-V guest. I bet > you can repro by doing a SUSPEND+RESUME. > > Can you try to see if that's actually a bug? If so, the only sane fix seems to > be to add a dedicated ONLINE action for Hyper-V. It seems we can't get away without a dedicated stage for Hyper-V anyway, e.g. see our discussion with Michael: https://lore.kernel.org/linux-hyperv/878rkqr7ku.fsf@ovpn-192-136.brq.redhat.com/ All these issues are more or less "theoretical" as there's no real CPU hotplug on Hyper-V/Azure. Yes, it is possible to trigger problems by doing CPU offline/online but I don't see how this may come handy outside of testing envs. > Per patch > > KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section > > from this series, CPUHP_AP_KVM_ONLINE needs to be before CPUHP_AP_SCHED_WAIT_EMPTY > to ensure there are no tasks, i.e. no vCPUs, running on the to-be-unplugged CPU. > > Back to the original bug, proposed fix is below. The other advantage of moving > the reset to hardware disabling is that the "cleanup" is just disabling the static > key, and at that point can simply be deleted as there's no need to disable the > static key when kvm-intel is unloaded since kvm-intel owns the key. I.e. this > patch (that we're replying to) would get replaced with a patch to delete the > disabling of the static key. > >From a quick glance looks good to me, I'll try to find some time to work on this issue. I will likely end up proposing a dedicated CPU hotplug stage for Hyper-V (which needs to happen before KVM's CPUHP_AP_KVM_ONLINE on CPU hotplug and after on unplug) anyway. Thanks for looking into this! > -- > From: Sean Christopherson > Date: Thu, 10 Nov 2022 17:28:08 -0800 > Subject: [PATCH] KVM: VMX: Reset eVMCS controls in VP assist page during > hardware disabling > > Reset the eVMCS controls in the per-CPU VP assist page during hardware > disabling instead of waiting until kvm-intel's module exit. The controls > are activated if and only if KVM creates a VM, i.e. don't need to be > reset if hardware is never enabled. > > Doing the reset during hardware disabling will naturally fix a potential > NULL pointer deref bug once KVM disables CPU hotplug while enabling and > disabling hardware (which is necessary to fix a variety of bugs). If the > kernel is running as the root partition, the VP assist page is unmapped > during CPU hot unplug, and so KVM's clearing of the eVMCS controls needs > to occur with CPU hot(un)plug disabled, otherwise KVM could attempt to > write to a CPU's VP assist page after it's unmapped. > > Reported-by: Vitaly Kuznetsov > Signed-off-by: Sean Christopherson > --- > arch/x86/kvm/vmx/vmx.c | 50 +++++++++++++++++++++++++----------------- > 1 file changed, 30 insertions(+), 20 deletions(-) > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c > index aca88524fd1e..ae13aa3e8a1d 100644 > --- a/arch/x86/kvm/vmx/vmx.c > +++ b/arch/x86/kvm/vmx/vmx.c > @@ -552,6 +552,33 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) > return 0; > } > > +static void hv_reset_evmcs(void) > +{ > + struct hv_vp_assist_page *vp_ap; > + > + if (!static_branch_unlikely(&enable_evmcs)) > + return; > + > + /* > + * KVM should enable eVMCS if and only if all CPUs have a VP assist > + * page, and should reject CPU onlining if eVMCS is enabled the CPU > + * doesn't have a VP assist page allocated. > + */ > + vp_ap = hv_get_vp_assist_page(smp_processor_id()); > + if (WARN_ON_ONCE(!vp_ap)) > + return; > + > + /* > + * Reset everything to support using non-enlightened VMCS access later > + * (e.g. when we reload the module with enlightened_vmcs=0) > + */ > + vp_ap->nested_control.features.directhypercall = 0; > + vp_ap->current_nested_vmcs = 0; > + vp_ap->enlighten_vmentry = 0; > +} > + > +#else /* IS_ENABLED(CONFIG_HYPERV) */ > +static void hv_reset_evmcs(void) {} > #endif /* IS_ENABLED(CONFIG_HYPERV) */ > > /* > @@ -2497,6 +2524,8 @@ static void vmx_hardware_disable(void) > if (cpu_vmxoff()) > kvm_spurious_fault(); > > + hv_reset_evmcs(); > + > intel_pt_handle_vmx(0); > } > > @@ -8463,27 +8492,8 @@ static void vmx_exit(void) > kvm_exit(); > > #if IS_ENABLED(CONFIG_HYPERV) > - if (static_branch_unlikely(&enable_evmcs)) { > - int cpu; > - struct hv_vp_assist_page *vp_ap; > - /* > - * Reset everything to support using non-enlightened VMCS > - * access later (e.g. when we reload the module with > - * enlightened_vmcs=0) > - */ > - for_each_online_cpu(cpu) { > - vp_ap = hv_get_vp_assist_page(cpu); > - > - if (!vp_ap) > - continue; > - > - vp_ap->nested_control.features.directhypercall = 0; > - vp_ap->current_nested_vmcs = 0; > - vp_ap->enlighten_vmentry = 0; > - } > - > + if (static_branch_unlikely(&enable_evmcs)) > static_branch_disable(&enable_evmcs); > - } > #endif > vmx_cleanup_l1d_flush(); > > > base-commit: 5f47ba6894477dfbdc5416467a25fb7acb47d404 -- Vitaly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DCC8CC433FE for ; Tue, 15 Nov 2022 09:32:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dom0bCeRbOZyBv7VJY677ybsjDpH7E+OmHGZQd/ivos=; b=XoIaCzzJoIEBMk KzNKRdd7hbXC6TfAcybMW32cvdy8kaHnaXOw9Xy0NNi7nClDBxREa3utjcLHAx5bc5W02MznyYBQG qWa50waPSRvkv2OHLOIXAFowQVYfJvAF8kaUc7GsYNDxt+/RQhGRJErLABAb2dygUFNM/rSC6Qob2 oVNdlf3arOLb3JfBLqX4SlI8G8DbNFoV3duUuNk9NFzk6qZvMrVwdXT81gzR0fEqDDKHy8A6+Elkz fsYg0Vd7/s8tgEmTtuO5IsT0mj2OlFZ5SpwtSD4/ju47kmF6vzM1TTgGLnkfwTlpQ7GLUa/hrIjB7 YzTs+JvcWD6950PqNJGQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ousIU-009OoJ-1p; Tue, 15 Nov 2022 09:32:02 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ousGs-009O1i-1q for linux-riscv@lists.infradead.org; Tue, 15 Nov 2022 09:30:25 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668504619; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=aa7iI5Lq4Rmc3VrPX4X35FoV8oMPTYrA9nhnHxaPCy0DA9ijQqa0y8mpM4y7h2IBWBYtsU 5tEDotMjAhoi/mMl4mqDqjKwpbzghxt9KKzsTVRX7wZlGOY5P1MvY+hVjcVs9Bv8YzA7Qk R4ZQr1fF44q2/NO45xVwJxBWOZGMPew= Received: from mail-ej1-f70.google.com (mail-ej1-f70.google.com [209.85.218.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-131-Nyw6dj93MP2IAu2hydf_3w-1; Tue, 15 Nov 2022 04:30:18 -0500 X-MC-Unique: Nyw6dj93MP2IAu2hydf_3w-1 Received: by mail-ej1-f70.google.com with SMTP id qf25-20020a1709077f1900b0078c02a23da3so6851789ejc.0 for ; Tue, 15 Nov 2022 01:30:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=AC52nPeZBv8PBFfycx+rhMhZWQPrrje6wfaEJISKNy2MRDHs6y2vtOOnF8OLZOqXlJ fBne1bo4ixzQ8gS8iS8YT1/COABxJmR2iUaPvBLLGRY+NbS8CfjiPHNoPfL318lBUTTM eE7SZrrwfFOItb60IhzaOh2l2LYeFGOqQmTWsUKylcOJksPAby4jgp1CY23db+K6Zl+x R+h+76JPQXpM5quCZmi9xkvlSuRk84a8/4OH2TGzSrVeA/6Auj4CmdNmba7CTzyrvBNg GE/jyHLK8NnmvqNXB3wzKRtnoUXjOjHJ0OceTaIvpoLT+7qKdR5SHka0O02JElXnNW4/ BgnA== X-Gm-Message-State: ANoB5plkMl5HITvwHW1rwvZCDO28wwShgNOSnylobEe9716nJXwaAh2f uE2GT6Wi4hTbouC2tbprxcf8BlDrmfyjXObIrWuoMTtC7bp9c8/BeJT/f+Rq668IYX1c/gu8gkq Z7ZT2uBCnfjgaaJ78Abn1Os6wsphe X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489199edt.387.1668504617517; Tue, 15 Nov 2022 01:30:17 -0800 (PST) X-Google-Smtp-Source: AA0mqf5Zsa4CnM7XTxV75wUJGjjLzRgPV7NQ4412B1eIJSCcezAuycjNE+mpvlda5nQMKDxwK/vVkA== X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489153edt.387.1668504617196; Tue, 15 Nov 2022 01:30:17 -0800 (PST) Received: from fedora (nat-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id ew13-20020a056402538d00b004642b35f89esm5950875edb.9.2022.11.15.01.30.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 01:30:16 -0800 (PST) From: Vitaly Kuznetsov To: Sean Christopherson Cc: James Morse , Alexandru Elisei , Suzuki K Poulose , Oliver Upton , Atish Patra , David Hildenbrand , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, Isaku Yamahata , Fabiano Rosas , Michael Ellerman , Chao Gao , Thomas Gleixner , Yuan Yao , Paolo Bonzini , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Matthew Rosato , Eric Farman Subject: Re: [PATCH 10/44] KVM: VMX: Clean up eVMCS enabling if KVM initialization fails In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-11-seanjc@google.com> <87mt98qfi2.fsf@ovpn-194-252.brq.redhat.com> Date: Tue, 15 Nov 2022 10:30:14 +0100 Message-ID: <87sfikmuop.fsf@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221115_013022_273915_9BB76F23 X-CRM114-Status: GOOD ( 55.38 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Sean Christopherson writes: > On Thu, Nov 03, 2022, Vitaly Kuznetsov wrote: >> Sean Christopherson writes: >> > + /* >> > + * Reset everything to support using non-enlightened VMCS access later >> > + * (e.g. when we reload the module with enlightened_vmcs=0) >> > + */ >> > + for_each_online_cpu(cpu) { >> > + vp_ap = hv_get_vp_assist_page(cpu); >> > + >> > + if (!vp_ap) >> > + continue; >> > + >> > + vp_ap->nested_control.features.directhypercall = 0; >> > + vp_ap->current_nested_vmcs = 0; >> > + vp_ap->enlighten_vmentry = 0; >> > + } >> >> Unrelated to your patch but while looking at this code I got curious >> about why don't we need a protection against CPU offlining here. Turns >> out that even when we offline a CPU, its VP assist page remains >> allocated (see hv_cpu_die()), we just write '0' to the MSR and thus > > Heh, "die". Hyper-V is quite dramatic. > >> accessing the page is safe. The consequent hv_cpu_init(), however, does >> not restore VP assist page when it's already allocated: >> >> # rdmsr -p 24 0x40000073 >> 10212f001 >> # echo 0 > /sys/devices/system/cpu/cpu24/online >> # echo 1 > /sys/devices/system/cpu/cpu24/online >> # rdmsr -p 24 0x40000073 >> 0 >> >> The culprit is commit e5d9b714fe402 ("x86/hyperv: fix root partition >> faults when writing to VP assist page MSR"). A patch is inbound. >> >> 'hv_root_partition' case is different though. We do memunmap() and reset >> VP assist page to zero so it is theoretically possible we're going to >> clash. Unless I'm missing some obvious reason why module unload can't >> coincide with CPU offlining, we may be better off surrounding this with >> cpus_read_lock()/cpus_read_unlock(). > > I finally see what you're concerned about. If a CPU goes offline and its assist > page is unmapped, zeroing out the nested/eVMCS stuff will fault. > > I think the real problem is that the purging of the eVMCS is in the wrong place. > Move the clearing to vmx_hardware_disable() and then the CPU hotplug bug goes > away once KVM disables hotplug during hardware enabling/disable later in the series. > There's no need to wait until module exit, e.g. it's not like it costs much to > clear a few variables, and IIUC the state is used only when KVM is actively using > VMX/eVMCS. > > However, I believe there's a second bug. KVM's CPU online hook is called before > Hyper-V's online hook (CPUHP_AP_ONLINE_DYN). Before this series, which moves KVM's > hook from STARTING to ONLINE, KVM's hook is waaaay before Hyper-V's. That means > that hv_cpu_init()'s allocation of the VP assist page will come _after_ KVM's > check in vmx_hardware_enable() > > /* > * This can happen if we hot-added a CPU but failed to allocate > * VP assist page for it. > */ > if (static_branch_unlikely(&enable_evmcs) && > !hv_get_vp_assist_page(cpu)) > return -EFAULT; > > I.e. CPU hotplug will never work if KVM is running VMs as a Hyper-V guest. I bet > you can repro by doing a SUSPEND+RESUME. > > Can you try to see if that's actually a bug? If so, the only sane fix seems to > be to add a dedicated ONLINE action for Hyper-V. It seems we can't get away without a dedicated stage for Hyper-V anyway, e.g. see our discussion with Michael: https://lore.kernel.org/linux-hyperv/878rkqr7ku.fsf@ovpn-192-136.brq.redhat.com/ All these issues are more or less "theoretical" as there's no real CPU hotplug on Hyper-V/Azure. Yes, it is possible to trigger problems by doing CPU offline/online but I don't see how this may come handy outside of testing envs. > Per patch > > KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section > > from this series, CPUHP_AP_KVM_ONLINE needs to be before CPUHP_AP_SCHED_WAIT_EMPTY > to ensure there are no tasks, i.e. no vCPUs, running on the to-be-unplugged CPU. > > Back to the original bug, proposed fix is below. The other advantage of moving > the reset to hardware disabling is that the "cleanup" is just disabling the static > key, and at that point can simply be deleted as there's no need to disable the > static key when kvm-intel is unloaded since kvm-intel owns the key. I.e. this > patch (that we're replying to) would get replaced with a patch to delete the > disabling of the static key. > >From a quick glance looks good to me, I'll try to find some time to work on this issue. I will likely end up proposing a dedicated CPU hotplug stage for Hyper-V (which needs to happen before KVM's CPUHP_AP_KVM_ONLINE on CPU hotplug and after on unplug) anyway. Thanks for looking into this! > -- > From: Sean Christopherson > Date: Thu, 10 Nov 2022 17:28:08 -0800 > Subject: [PATCH] KVM: VMX: Reset eVMCS controls in VP assist page during > hardware disabling > > Reset the eVMCS controls in the per-CPU VP assist page during hardware > disabling instead of waiting until kvm-intel's module exit. The controls > are activated if and only if KVM creates a VM, i.e. don't need to be > reset if hardware is never enabled. > > Doing the reset during hardware disabling will naturally fix a potential > NULL pointer deref bug once KVM disables CPU hotplug while enabling and > disabling hardware (which is necessary to fix a variety of bugs). If the > kernel is running as the root partition, the VP assist page is unmapped > during CPU hot unplug, and so KVM's clearing of the eVMCS controls needs > to occur with CPU hot(un)plug disabled, otherwise KVM could attempt to > write to a CPU's VP assist page after it's unmapped. > > Reported-by: Vitaly Kuznetsov > Signed-off-by: Sean Christopherson > --- > arch/x86/kvm/vmx/vmx.c | 50 +++++++++++++++++++++++++----------------- > 1 file changed, 30 insertions(+), 20 deletions(-) > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c > index aca88524fd1e..ae13aa3e8a1d 100644 > --- a/arch/x86/kvm/vmx/vmx.c > +++ b/arch/x86/kvm/vmx/vmx.c > @@ -552,6 +552,33 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) > return 0; > } > > +static void hv_reset_evmcs(void) > +{ > + struct hv_vp_assist_page *vp_ap; > + > + if (!static_branch_unlikely(&enable_evmcs)) > + return; > + > + /* > + * KVM should enable eVMCS if and only if all CPUs have a VP assist > + * page, and should reject CPU onlining if eVMCS is enabled the CPU > + * doesn't have a VP assist page allocated. > + */ > + vp_ap = hv_get_vp_assist_page(smp_processor_id()); > + if (WARN_ON_ONCE(!vp_ap)) > + return; > + > + /* > + * Reset everything to support using non-enlightened VMCS access later > + * (e.g. when we reload the module with enlightened_vmcs=0) > + */ > + vp_ap->nested_control.features.directhypercall = 0; > + vp_ap->current_nested_vmcs = 0; > + vp_ap->enlighten_vmentry = 0; > +} > + > +#else /* IS_ENABLED(CONFIG_HYPERV) */ > +static void hv_reset_evmcs(void) {} > #endif /* IS_ENABLED(CONFIG_HYPERV) */ > > /* > @@ -2497,6 +2524,8 @@ static void vmx_hardware_disable(void) > if (cpu_vmxoff()) > kvm_spurious_fault(); > > + hv_reset_evmcs(); > + > intel_pt_handle_vmx(0); > } > > @@ -8463,27 +8492,8 @@ static void vmx_exit(void) > kvm_exit(); > > #if IS_ENABLED(CONFIG_HYPERV) > - if (static_branch_unlikely(&enable_evmcs)) { > - int cpu; > - struct hv_vp_assist_page *vp_ap; > - /* > - * Reset everything to support using non-enlightened VMCS > - * access later (e.g. when we reload the module with > - * enlightened_vmcs=0) > - */ > - for_each_online_cpu(cpu) { > - vp_ap = hv_get_vp_assist_page(cpu); > - > - if (!vp_ap) > - continue; > - > - vp_ap->nested_control.features.directhypercall = 0; > - vp_ap->current_nested_vmcs = 0; > - vp_ap->enlighten_vmentry = 0; > - } > - > + if (static_branch_unlikely(&enable_evmcs)) > static_branch_disable(&enable_evmcs); > - } > #endif > vmx_cleanup_l1d_flush(); > > > base-commit: 5f47ba6894477dfbdc5416467a25fb7acb47d404 -- Vitaly _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CEB36C4332F for ; Tue, 15 Nov 2022 09:31:23 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4NBLXV0DsMz3dvF for ; Tue, 15 Nov 2022 20:31:22 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JkU0WQcm; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JkU0WQcm; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=redhat.com (client-ip=170.10.129.124; helo=us-smtp-delivery-124.mimecast.com; envelope-from=vkuznets@redhat.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JkU0WQcm; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JkU0WQcm; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4NBLWN4YZ3z3bYF for ; Tue, 15 Nov 2022 20:30:22 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668504620; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=JkU0WQcmtoZTK94+GGlj6Z+rGcg0ySjV9BzywnJ6cXqZ0cetxDn08u3ca5TjnZVvoAD1nF 7g/RwCmMCzotuwVvZ73w69wyghXSrilVXQQs64Gv32gkj69cnHEjjs5TIOjDhhrWh6ycy4 wwnIKxyzyK0hTb+eZyF/3gG0OItPCPo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668504620; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=JkU0WQcmtoZTK94+GGlj6Z+rGcg0ySjV9BzywnJ6cXqZ0cetxDn08u3ca5TjnZVvoAD1nF 7g/RwCmMCzotuwVvZ73w69wyghXSrilVXQQs64Gv32gkj69cnHEjjs5TIOjDhhrWh6ycy4 wwnIKxyzyK0hTb+eZyF/3gG0OItPCPo= Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-653-hgcx-XW5NSqlaGqaJGxH4w-1; Tue, 15 Nov 2022 04:30:18 -0500 X-MC-Unique: hgcx-XW5NSqlaGqaJGxH4w-1 Received: by mail-ej1-f71.google.com with SMTP id oz34-20020a1709077da200b007adc8d68e90so6895654ejc.11 for ; Tue, 15 Nov 2022 01:30:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=UxHWU9X8YcEVWLourtLiqS55lp7upOWr7JXYwK5omDImhwZ9tNtmxcVfbGm5LD+nkc 0N1x0npW5y7RIkhZXPE6DFwtByZTQYdvlFbkMNQv/rSM0R77q9W1gSSBtcCfKebURyJW aLSo293MxBtjxVMoqy6ZF1U1Yh7YwBtLfnLKwsMlKg6Gtjf2pWht/lxctN4ThvZ2boIT MBGMS8KxHjem7ijm4FDHUYTCCqe+skdnRxgwg3nzjpAhhBfE/kts9QybFxOG0VtlcC80 FwH3U0WikosS3NlBl3NgrtnZwFpwAkf58c846eqIRA5zdDvh6P0Uz111w6mDsD3HZffR RPLA== X-Gm-Message-State: ANoB5plcy9bj1eCvDNiHfwmntkMejtWCycCWFoNd5afgAokaJpt/H0/J 4PkDcixzQ4CElTFoMZ2XMneyd8ONSVLXnQ3zf31CHSKRtDQfOdgB3r0oaRvRUumTwn07WJt3prL g8JBwnaSrtj7e2y6wijh2zWmp8g== X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489178edt.387.1668504617480; Tue, 15 Nov 2022 01:30:17 -0800 (PST) X-Google-Smtp-Source: AA0mqf5Zsa4CnM7XTxV75wUJGjjLzRgPV7NQ4412B1eIJSCcezAuycjNE+mpvlda5nQMKDxwK/vVkA== X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489153edt.387.1668504617196; Tue, 15 Nov 2022 01:30:17 -0800 (PST) Received: from fedora (nat-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id ew13-20020a056402538d00b004642b35f89esm5950875edb.9.2022.11.15.01.30.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 01:30:16 -0800 (PST) From: Vitaly Kuznetsov To: Sean Christopherson Subject: Re: [PATCH 10/44] KVM: VMX: Clean up eVMCS enabling if KVM initialization fails In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-11-seanjc@google.com> <87mt98qfi2.fsf@ovpn-194-252.brq.redhat.com> Date: Tue, 15 Nov 2022 10:30:14 +0100 Message-ID: <87sfikmuop.fsf@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, David Hildenbrand , Atish Patra , linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Claudio Imbrenda , kvmarm@lists.cs.columbia.edu, linux-s390@vger.kernel.org, Janosch Frank , Huacai Chen , Aleksandar Markovic , Palmer Dabbelt , Christian Borntraeger , Matthew Rosato , Chao Gao , Eric Farman , Albert Ou , Suzuki K Poulose , Paul Walmsley , Yuan Yao , kvmarm@lists.linux.dev, Thomas Gleixner , Alexandru Elisei , linux-arm-kernel@lists.infradead.org, Isaku Yamahata , Fabiano Rosas , Anup Patel , linux-mips@vger.kernel.org, Oliver Upton , James Morse , kvm-riscv@lists.infradead.org, Marc Zyngier , Paolo Bonzini , linuxppc-dev@lists.ozlabs.org Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" Sean Christopherson writes: > On Thu, Nov 03, 2022, Vitaly Kuznetsov wrote: >> Sean Christopherson writes: >> > + /* >> > + * Reset everything to support using non-enlightened VMCS access later >> > + * (e.g. when we reload the module with enlightened_vmcs=0) >> > + */ >> > + for_each_online_cpu(cpu) { >> > + vp_ap = hv_get_vp_assist_page(cpu); >> > + >> > + if (!vp_ap) >> > + continue; >> > + >> > + vp_ap->nested_control.features.directhypercall = 0; >> > + vp_ap->current_nested_vmcs = 0; >> > + vp_ap->enlighten_vmentry = 0; >> > + } >> >> Unrelated to your patch but while looking at this code I got curious >> about why don't we need a protection against CPU offlining here. Turns >> out that even when we offline a CPU, its VP assist page remains >> allocated (see hv_cpu_die()), we just write '0' to the MSR and thus > > Heh, "die". Hyper-V is quite dramatic. > >> accessing the page is safe. The consequent hv_cpu_init(), however, does >> not restore VP assist page when it's already allocated: >> >> # rdmsr -p 24 0x40000073 >> 10212f001 >> # echo 0 > /sys/devices/system/cpu/cpu24/online >> # echo 1 > /sys/devices/system/cpu/cpu24/online >> # rdmsr -p 24 0x40000073 >> 0 >> >> The culprit is commit e5d9b714fe402 ("x86/hyperv: fix root partition >> faults when writing to VP assist page MSR"). A patch is inbound. >> >> 'hv_root_partition' case is different though. We do memunmap() and reset >> VP assist page to zero so it is theoretically possible we're going to >> clash. Unless I'm missing some obvious reason why module unload can't >> coincide with CPU offlining, we may be better off surrounding this with >> cpus_read_lock()/cpus_read_unlock(). > > I finally see what you're concerned about. If a CPU goes offline and its assist > page is unmapped, zeroing out the nested/eVMCS stuff will fault. > > I think the real problem is that the purging of the eVMCS is in the wrong place. > Move the clearing to vmx_hardware_disable() and then the CPU hotplug bug goes > away once KVM disables hotplug during hardware enabling/disable later in the series. > There's no need to wait until module exit, e.g. it's not like it costs much to > clear a few variables, and IIUC the state is used only when KVM is actively using > VMX/eVMCS. > > However, I believe there's a second bug. KVM's CPU online hook is called before > Hyper-V's online hook (CPUHP_AP_ONLINE_DYN). Before this series, which moves KVM's > hook from STARTING to ONLINE, KVM's hook is waaaay before Hyper-V's. That means > that hv_cpu_init()'s allocation of the VP assist page will come _after_ KVM's > check in vmx_hardware_enable() > > /* > * This can happen if we hot-added a CPU but failed to allocate > * VP assist page for it. > */ > if (static_branch_unlikely(&enable_evmcs) && > !hv_get_vp_assist_page(cpu)) > return -EFAULT; > > I.e. CPU hotplug will never work if KVM is running VMs as a Hyper-V guest. I bet > you can repro by doing a SUSPEND+RESUME. > > Can you try to see if that's actually a bug? If so, the only sane fix seems to > be to add a dedicated ONLINE action for Hyper-V. It seems we can't get away without a dedicated stage for Hyper-V anyway, e.g. see our discussion with Michael: https://lore.kernel.org/linux-hyperv/878rkqr7ku.fsf@ovpn-192-136.brq.redhat.com/ All these issues are more or less "theoretical" as there's no real CPU hotplug on Hyper-V/Azure. Yes, it is possible to trigger problems by doing CPU offline/online but I don't see how this may come handy outside of testing envs. > Per patch > > KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section > > from this series, CPUHP_AP_KVM_ONLINE needs to be before CPUHP_AP_SCHED_WAIT_EMPTY > to ensure there are no tasks, i.e. no vCPUs, running on the to-be-unplugged CPU. > > Back to the original bug, proposed fix is below. The other advantage of moving > the reset to hardware disabling is that the "cleanup" is just disabling the static > key, and at that point can simply be deleted as there's no need to disable the > static key when kvm-intel is unloaded since kvm-intel owns the key. I.e. this > patch (that we're replying to) would get replaced with a patch to delete the > disabling of the static key. > >From a quick glance looks good to me, I'll try to find some time to work on this issue. I will likely end up proposing a dedicated CPU hotplug stage for Hyper-V (which needs to happen before KVM's CPUHP_AP_KVM_ONLINE on CPU hotplug and after on unplug) anyway. Thanks for looking into this! > -- > From: Sean Christopherson > Date: Thu, 10 Nov 2022 17:28:08 -0800 > Subject: [PATCH] KVM: VMX: Reset eVMCS controls in VP assist page during > hardware disabling > > Reset the eVMCS controls in the per-CPU VP assist page during hardware > disabling instead of waiting until kvm-intel's module exit. The controls > are activated if and only if KVM creates a VM, i.e. don't need to be > reset if hardware is never enabled. > > Doing the reset during hardware disabling will naturally fix a potential > NULL pointer deref bug once KVM disables CPU hotplug while enabling and > disabling hardware (which is necessary to fix a variety of bugs). If the > kernel is running as the root partition, the VP assist page is unmapped > during CPU hot unplug, and so KVM's clearing of the eVMCS controls needs > to occur with CPU hot(un)plug disabled, otherwise KVM could attempt to > write to a CPU's VP assist page after it's unmapped. > > Reported-by: Vitaly Kuznetsov > Signed-off-by: Sean Christopherson > --- > arch/x86/kvm/vmx/vmx.c | 50 +++++++++++++++++++++++++----------------- > 1 file changed, 30 insertions(+), 20 deletions(-) > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c > index aca88524fd1e..ae13aa3e8a1d 100644 > --- a/arch/x86/kvm/vmx/vmx.c > +++ b/arch/x86/kvm/vmx/vmx.c > @@ -552,6 +552,33 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) > return 0; > } > > +static void hv_reset_evmcs(void) > +{ > + struct hv_vp_assist_page *vp_ap; > + > + if (!static_branch_unlikely(&enable_evmcs)) > + return; > + > + /* > + * KVM should enable eVMCS if and only if all CPUs have a VP assist > + * page, and should reject CPU onlining if eVMCS is enabled the CPU > + * doesn't have a VP assist page allocated. > + */ > + vp_ap = hv_get_vp_assist_page(smp_processor_id()); > + if (WARN_ON_ONCE(!vp_ap)) > + return; > + > + /* > + * Reset everything to support using non-enlightened VMCS access later > + * (e.g. when we reload the module with enlightened_vmcs=0) > + */ > + vp_ap->nested_control.features.directhypercall = 0; > + vp_ap->current_nested_vmcs = 0; > + vp_ap->enlighten_vmentry = 0; > +} > + > +#else /* IS_ENABLED(CONFIG_HYPERV) */ > +static void hv_reset_evmcs(void) {} > #endif /* IS_ENABLED(CONFIG_HYPERV) */ > > /* > @@ -2497,6 +2524,8 @@ static void vmx_hardware_disable(void) > if (cpu_vmxoff()) > kvm_spurious_fault(); > > + hv_reset_evmcs(); > + > intel_pt_handle_vmx(0); > } > > @@ -8463,27 +8492,8 @@ static void vmx_exit(void) > kvm_exit(); > > #if IS_ENABLED(CONFIG_HYPERV) > - if (static_branch_unlikely(&enable_evmcs)) { > - int cpu; > - struct hv_vp_assist_page *vp_ap; > - /* > - * Reset everything to support using non-enlightened VMCS > - * access later (e.g. when we reload the module with > - * enlightened_vmcs=0) > - */ > - for_each_online_cpu(cpu) { > - vp_ap = hv_get_vp_assist_page(cpu); > - > - if (!vp_ap) > - continue; > - > - vp_ap->nested_control.features.directhypercall = 0; > - vp_ap->current_nested_vmcs = 0; > - vp_ap->enlighten_vmentry = 0; > - } > - > + if (static_branch_unlikely(&enable_evmcs)) > static_branch_disable(&enable_evmcs); > - } > #endif > vmx_cleanup_l1d_flush(); > > > base-commit: 5f47ba6894477dfbdc5416467a25fb7acb47d404 -- Vitaly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0193CC433FE for ; Tue, 15 Nov 2022 09:32:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dfIR3s7FK74MhoswTdmhPXActzF3n4HrdibR6HuOnu0=; b=4A4B+XJUAGFktZ NSNtW/Z2hkrI23XHuV1sJq1dOlViEKrKN/IKOs7XGahtDjihCjtTmS6zzRuuOUG+YtG2nkfxe2bJN ev+uSItA7N5azZ1QDG1ySNaAKe0fqPQ8Pl8yWeL8AsM54yggTq+PjVctq8qdmp4WmzyHOTsUzDixG ybEi8l/qul1Mhs4N61fHtjGuO3zPlvsaqJaDidkHQcIzZTqPbOLou7HvmVIw0AxECrdCEkGi2rzeF fm5jjzL5+ow8QRG/WD53c0AX1z4MIW3fotUOMdXeQXAIjtrgRh3UQJsVGdyHoA/Y5TYpGPe0qYXLo +8BQk6h1wi+TAo96J1Cw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ousHz-009OQz-EH; Tue, 15 Nov 2022 09:31:32 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ousGu-009O1j-39 for linux-arm-kernel@lists.infradead.org; Tue, 15 Nov 2022 09:30:26 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668504619; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=aa7iI5Lq4Rmc3VrPX4X35FoV8oMPTYrA9nhnHxaPCy0DA9ijQqa0y8mpM4y7h2IBWBYtsU 5tEDotMjAhoi/mMl4mqDqjKwpbzghxt9KKzsTVRX7wZlGOY5P1MvY+hVjcVs9Bv8YzA7Qk R4ZQr1fF44q2/NO45xVwJxBWOZGMPew= Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-306-3SQ4s4LKMUKmIQ0ZRJCXBg-1; Tue, 15 Nov 2022 04:30:18 -0500 X-MC-Unique: 3SQ4s4LKMUKmIQ0ZRJCXBg-1 Received: by mail-ej1-f72.google.com with SMTP id dn14-20020a17090794ce00b007ae5d040ca8so6962563ejc.17 for ; Tue, 15 Nov 2022 01:30:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PI4QkJgg5xRejZYnaDkrohoS+PcA0MTFv2A75peo2zg=; b=czD+XhhtX18D4yJW4fDYNuOJMF6cfSOZQkB3+k7ASkylVAxT335aMEzO333XTYvzd6 bG++oyoSbBCjZAFmpFO7fVJvoLO65ndtCrAKdZ8taaZp1NUBXpDQDDylOg/59fv7laS0 Y/L595nscu2FkVYPWr/uPra710g06HJWrNE15tgV7tdao7FJIFgrZvjux00+cduW+VgV 58qLRlsTsSCrjxkYzE77zZ0hqHS/Ut3tzAce96EnQF6NqlZ6wRHn/Pk/16kKNQrzlif1 /i3/7CW/Ocugv9irRLvj2EPY4/RMcIy43/bJG4k0JURmGEI8tE016b0LKi2FYdX2Hx6t TA/g== X-Gm-Message-State: ANoB5pkEk6GGYeaB5+zpRzTgmbtyFTiv+neMPZIje2gQQurZnjJjzB2T V6OdI4hLZQH2fH1HI3nbGLIy4aTYgkQQ012tsldA/ffdizGzArT5Z+hqiHCOgPLsMy+7OC9+eZM 8R3yvHcpVncb/pzE1hgzVNWLaXz1cKFlfosQ= X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489187edt.387.1668504617481; Tue, 15 Nov 2022 01:30:17 -0800 (PST) X-Google-Smtp-Source: AA0mqf5Zsa4CnM7XTxV75wUJGjjLzRgPV7NQ4412B1eIJSCcezAuycjNE+mpvlda5nQMKDxwK/vVkA== X-Received: by 2002:a50:fe13:0:b0:461:565e:8779 with SMTP id f19-20020a50fe13000000b00461565e8779mr14489153edt.387.1668504617196; Tue, 15 Nov 2022 01:30:17 -0800 (PST) Received: from fedora (nat-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id ew13-20020a056402538d00b004642b35f89esm5950875edb.9.2022.11.15.01.30.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 01:30:16 -0800 (PST) From: Vitaly Kuznetsov To: Sean Christopherson Cc: James Morse , Alexandru Elisei , Suzuki K Poulose , Oliver Upton , Atish Patra , David Hildenbrand , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, Isaku Yamahata , Fabiano Rosas , Michael Ellerman , Chao Gao , Thomas Gleixner , Yuan Yao , Paolo Bonzini , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Matthew Rosato , Eric Farman Subject: Re: [PATCH 10/44] KVM: VMX: Clean up eVMCS enabling if KVM initialization fails In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-11-seanjc@google.com> <87mt98qfi2.fsf@ovpn-194-252.brq.redhat.com> Date: Tue, 15 Nov 2022 10:30:14 +0100 Message-ID: <87sfikmuop.fsf@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221115_013024_289196_ABE7A6D8 X-CRM114-Status: GOOD ( 57.11 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Sean Christopherson writes: > On Thu, Nov 03, 2022, Vitaly Kuznetsov wrote: >> Sean Christopherson writes: >> > + /* >> > + * Reset everything to support using non-enlightened VMCS access later >> > + * (e.g. when we reload the module with enlightened_vmcs=0) >> > + */ >> > + for_each_online_cpu(cpu) { >> > + vp_ap = hv_get_vp_assist_page(cpu); >> > + >> > + if (!vp_ap) >> > + continue; >> > + >> > + vp_ap->nested_control.features.directhypercall = 0; >> > + vp_ap->current_nested_vmcs = 0; >> > + vp_ap->enlighten_vmentry = 0; >> > + } >> >> Unrelated to your patch but while looking at this code I got curious >> about why don't we need a protection against CPU offlining here. Turns >> out that even when we offline a CPU, its VP assist page remains >> allocated (see hv_cpu_die()), we just write '0' to the MSR and thus > > Heh, "die". Hyper-V is quite dramatic. > >> accessing the page is safe. The consequent hv_cpu_init(), however, does >> not restore VP assist page when it's already allocated: >> >> # rdmsr -p 24 0x40000073 >> 10212f001 >> # echo 0 > /sys/devices/system/cpu/cpu24/online >> # echo 1 > /sys/devices/system/cpu/cpu24/online >> # rdmsr -p 24 0x40000073 >> 0 >> >> The culprit is commit e5d9b714fe402 ("x86/hyperv: fix root partition >> faults when writing to VP assist page MSR"). A patch is inbound. >> >> 'hv_root_partition' case is different though. We do memunmap() and reset >> VP assist page to zero so it is theoretically possible we're going to >> clash. Unless I'm missing some obvious reason why module unload can't >> coincide with CPU offlining, we may be better off surrounding this with >> cpus_read_lock()/cpus_read_unlock(). > > I finally see what you're concerned about. If a CPU goes offline and its assist > page is unmapped, zeroing out the nested/eVMCS stuff will fault. > > I think the real problem is that the purging of the eVMCS is in the wrong place. > Move the clearing to vmx_hardware_disable() and then the CPU hotplug bug goes > away once KVM disables hotplug during hardware enabling/disable later in the series. > There's no need to wait until module exit, e.g. it's not like it costs much to > clear a few variables, and IIUC the state is used only when KVM is actively using > VMX/eVMCS. > > However, I believe there's a second bug. KVM's CPU online hook is called before > Hyper-V's online hook (CPUHP_AP_ONLINE_DYN). Before this series, which moves KVM's > hook from STARTING to ONLINE, KVM's hook is waaaay before Hyper-V's. That means > that hv_cpu_init()'s allocation of the VP assist page will come _after_ KVM's > check in vmx_hardware_enable() > > /* > * This can happen if we hot-added a CPU but failed to allocate > * VP assist page for it. > */ > if (static_branch_unlikely(&enable_evmcs) && > !hv_get_vp_assist_page(cpu)) > return -EFAULT; > > I.e. CPU hotplug will never work if KVM is running VMs as a Hyper-V guest. I bet > you can repro by doing a SUSPEND+RESUME. > > Can you try to see if that's actually a bug? If so, the only sane fix seems to > be to add a dedicated ONLINE action for Hyper-V. It seems we can't get away without a dedicated stage for Hyper-V anyway, e.g. see our discussion with Michael: https://lore.kernel.org/linux-hyperv/878rkqr7ku.fsf@ovpn-192-136.brq.redhat.com/ All these issues are more or less "theoretical" as there's no real CPU hotplug on Hyper-V/Azure. Yes, it is possible to trigger problems by doing CPU offline/online but I don't see how this may come handy outside of testing envs. > Per patch > > KVM: Rename and move CPUHP_AP_KVM_STARTING to ONLINE section > > from this series, CPUHP_AP_KVM_ONLINE needs to be before CPUHP_AP_SCHED_WAIT_EMPTY > to ensure there are no tasks, i.e. no vCPUs, running on the to-be-unplugged CPU. > > Back to the original bug, proposed fix is below. The other advantage of moving > the reset to hardware disabling is that the "cleanup" is just disabling the static > key, and at that point can simply be deleted as there's no need to disable the > static key when kvm-intel is unloaded since kvm-intel owns the key. I.e. this > patch (that we're replying to) would get replaced with a patch to delete the > disabling of the static key. > >From a quick glance looks good to me, I'll try to find some time to work on this issue. I will likely end up proposing a dedicated CPU hotplug stage for Hyper-V (which needs to happen before KVM's CPUHP_AP_KVM_ONLINE on CPU hotplug and after on unplug) anyway. Thanks for looking into this! > -- > From: Sean Christopherson > Date: Thu, 10 Nov 2022 17:28:08 -0800 > Subject: [PATCH] KVM: VMX: Reset eVMCS controls in VP assist page during > hardware disabling > > Reset the eVMCS controls in the per-CPU VP assist page during hardware > disabling instead of waiting until kvm-intel's module exit. The controls > are activated if and only if KVM creates a VM, i.e. don't need to be > reset if hardware is never enabled. > > Doing the reset during hardware disabling will naturally fix a potential > NULL pointer deref bug once KVM disables CPU hotplug while enabling and > disabling hardware (which is necessary to fix a variety of bugs). If the > kernel is running as the root partition, the VP assist page is unmapped > during CPU hot unplug, and so KVM's clearing of the eVMCS controls needs > to occur with CPU hot(un)plug disabled, otherwise KVM could attempt to > write to a CPU's VP assist page after it's unmapped. > > Reported-by: Vitaly Kuznetsov > Signed-off-by: Sean Christopherson > --- > arch/x86/kvm/vmx/vmx.c | 50 +++++++++++++++++++++++++----------------- > 1 file changed, 30 insertions(+), 20 deletions(-) > > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c > index aca88524fd1e..ae13aa3e8a1d 100644 > --- a/arch/x86/kvm/vmx/vmx.c > +++ b/arch/x86/kvm/vmx/vmx.c > @@ -552,6 +552,33 @@ static int hv_enable_direct_tlbflush(struct kvm_vcpu *vcpu) > return 0; > } > > +static void hv_reset_evmcs(void) > +{ > + struct hv_vp_assist_page *vp_ap; > + > + if (!static_branch_unlikely(&enable_evmcs)) > + return; > + > + /* > + * KVM should enable eVMCS if and only if all CPUs have a VP assist > + * page, and should reject CPU onlining if eVMCS is enabled the CPU > + * doesn't have a VP assist page allocated. > + */ > + vp_ap = hv_get_vp_assist_page(smp_processor_id()); > + if (WARN_ON_ONCE(!vp_ap)) > + return; > + > + /* > + * Reset everything to support using non-enlightened VMCS access later > + * (e.g. when we reload the module with enlightened_vmcs=0) > + */ > + vp_ap->nested_control.features.directhypercall = 0; > + vp_ap->current_nested_vmcs = 0; > + vp_ap->enlighten_vmentry = 0; > +} > + > +#else /* IS_ENABLED(CONFIG_HYPERV) */ > +static void hv_reset_evmcs(void) {} > #endif /* IS_ENABLED(CONFIG_HYPERV) */ > > /* > @@ -2497,6 +2524,8 @@ static void vmx_hardware_disable(void) > if (cpu_vmxoff()) > kvm_spurious_fault(); > > + hv_reset_evmcs(); > + > intel_pt_handle_vmx(0); > } > > @@ -8463,27 +8492,8 @@ static void vmx_exit(void) > kvm_exit(); > > #if IS_ENABLED(CONFIG_HYPERV) > - if (static_branch_unlikely(&enable_evmcs)) { > - int cpu; > - struct hv_vp_assist_page *vp_ap; > - /* > - * Reset everything to support using non-enlightened VMCS > - * access later (e.g. when we reload the module with > - * enlightened_vmcs=0) > - */ > - for_each_online_cpu(cpu) { > - vp_ap = hv_get_vp_assist_page(cpu); > - > - if (!vp_ap) > - continue; > - > - vp_ap->nested_control.features.directhypercall = 0; > - vp_ap->current_nested_vmcs = 0; > - vp_ap->enlighten_vmentry = 0; > - } > - > + if (static_branch_unlikely(&enable_evmcs)) > static_branch_disable(&enable_evmcs); > - } > #endif > vmx_cleanup_l1d_flush(); > > > base-commit: 5f47ba6894477dfbdc5416467a25fb7acb47d404 -- Vitaly _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel