From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [PATCH 2/3] kvm: x86: do not use KVM_REQ_EVENT for APICv interrupt injection Date: Wed, 28 Sep 2016 16:46:08 +0300 Message-ID: <20160928164054-mutt-send-email-mst@kernel.org> References: <1475011213-34225-1-git-send-email-pbonzini@redhat.com> <1475011213-34225-3-git-send-email-pbonzini@redhat.com> <20160928020319-mutt-send-email-mst@kernel.org> <286c39fb-6ba3-3267-dff6-b04ee4cbb1c7@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: linux-kernel@vger.kernel.org, kvm@vger.kernel.org, yang.zhang.wz@gmail.com, feng.wu@intel.com, rkrcmar@redhat.com To: Paolo Bonzini Return-path: Received: from mx1.redhat.com ([209.132.183.28]:45980 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932348AbcI1NqL (ORCPT ); Wed, 28 Sep 2016 09:46:11 -0400 Content-Disposition: inline In-Reply-To: <286c39fb-6ba3-3267-dff6-b04ee4cbb1c7@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: On Wed, Sep 28, 2016 at 10:21:41AM +0200, Paolo Bonzini wrote: > > > On 28/09/2016 01:07, Michael S. Tsirkin wrote: > > On Tue, Sep 27, 2016 at 11:20:12PM +0200, Paolo Bonzini wrote: > >> Since bf9f6ac8d749 ("KVM: Update Posted-Interrupts Descriptor when vCPU > >> is blocked", 2015-09-18) the posted interrupt descriptor is checked > >> unconditionally for PIR.ON. Therefore we don't need KVM_REQ_EVENT to > >> trigger the scan and, if NMIs or SMIs are not involved, we can avoid > >> the complicated event injection path. > >> > >> However, there is a race between vmx_deliver_posted_interrupt and > >> vcpu_enter_guest. Fix it by disabling interrupts before vcpu->mode is > >> set to IN_GUEST_MODE. > > > > Could you describe the race a bit more please? > > I'm surprised that locally disabling irqs > > fixes a race with a different CPUs. > > The posted interrupt IPI has a dummy handler in arch/x86/kernel/irq.c > (smp_kvm_posted_intr_ipi). It only does something if it is received > while the guest is running. > > So local_irq_disable has an interesting side effect. Because the > interrupt will not be processed until the guest is entered, > local_irq_disable effectively switches the IRQ handler from the dummy > handler to the processor's posted interrupt handling. > > So you want to do that before setting IN_GUEST_MODE, otherwise the IPI > sent by deliver_posted_interrupt is ignored. > > However, the patch is wrong, because this bit: > > if (kvm_lapic_enabled(vcpu)) { > /* > * Update architecture specific hints for APIC > * virtual interrupt delivery. > */ > if (vcpu->arch.apicv_active) > kvm_x86_ops->hwapic_irr_update(vcpu, > kvm_lapic_find_highest_irr(vcpu)); > } > > also has to be moved after setting IN_GUEST_MODE. Basically the order > for interrupt injection is: > > (1) set PIR > smp_wmb() Empty on x86 btw but good for reasoning about barrier pairing. So - where's the paired smp_rmb? See below. > (2) set ON > smp_mb() This one can be combined to smp_store_mb to save a couple of cycles. > (3) read vcpu->mode > if IN_GUEST_MODE > (4a) send posted interrupt IPI > else > (4b) kick (i.e. cmpxchg vcpu->mode from IN_GUEST_MODE to > EXITING_GUEST_MODE and send reschedule IPI) > > while the order for entering the guest must be the opposite. The > numbers on the left identify the pairing between interrupt injection and > vcpu_entr_guest > > (4a) enable posted interrupt processing (i.e. disable interrupts!) > (3) set vcpu->mode to IN_GUEST_MODE > smp_mb() This one can be combined to smp_store_mb to save a couple of cycles. > (2) read ON > if ON then do we need smp_rmb here? > (1) read PIR > sync PIR to IRR > (4b) read vcpu->mode > if vcpu->mode == EXITING_GUEST_MODE then > cancel vmentry > (3/2/1) # posted interrupts are processed on the next vmentry > > Paolo