Re: [PATCH v2] KVM: x86: Unconditionally set irr_pending when updating APICv state

[Sean Christopherson <seanjc@google.com>]

On Wed, Nov 06, 2024, Chao Gao wrote:
> On Tue, Nov 05, 2024 at 05:51:35PM -0800, Sean Christopherson wrote:
> > arch/x86/kvm/lapic.c | 29 ++++++++++++++++++-----------
> > 1 file changed, 18 insertions(+), 11 deletions(-)
> >
> >diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> >index 65412640cfc7..e470061b744a 100644
> >--- a/arch/x86/kvm/lapic.c
> >+++ b/arch/x86/kvm/lapic.c
> >@@ -2629,19 +2629,26 @@ void kvm_apic_update_apicv(struct kvm_vcpu *vcpu)
> > {
> > 	struct kvm_lapic *apic = vcpu->arch.apic;
> > 
> >-	if (apic->apicv_active) {
> >-		/* irr_pending is always true when apicv is activated. */
> >-		apic->irr_pending = true;
> >+	/*
> >+	 * When APICv is enabled, KVM must always search the IRR for a pending
> >+	 * IRQ, as other vCPUs and devices can set IRR bits even if the vCPU
> >+	 * isn't running.  If APICv is disabled, KVM _should_ search the IRR
> >+	 * for a pending IRQ.  But KVM currently doesn't ensure *all* hardware,
> >+	 * e.g. CPUs and IOMMUs, has seen the change in state, i.e. searching
> >+	 * the IRR at this time could race with IRQ delivery from hardware that
> >+	 * still sees APICv as being enabled.
> >+	 *
> >+	 * FIXME: Ensure other vCPUs and devices observe the change in APICv
> >+	 *        state prior to updating KVM's metadata caches, so that KVM
> >+	 *        can safely search the IRR and set irr_pending accordingly.
> >+	 */
> >+	apic->irr_pending = true;
> 
> Should irr_pending be cleared after the first search of IRR that finds no
> pending IRQ, i.e., in apic_find_highest_irr() when !apic->apicv_active?

Definitely not in apic_find_highest_irr(), because there are ordering issues on
SMP systems.  Huh, and apic_clear_irr() and kvm_lapic_set_irr() are buggy; they
subtly rely on atomic accesses to provide ordering, but really should have memory
barriers of some kinda (off the top of my head, I'm not entirey sure what barrier
is appropriate in apic_clear_irr()).

And definitely not in the context of this bug fix, because that performance flaw
exists in multiple other scenarios.

> Otherwise, irr_pending will be out of sync until the arrival of an interrupt.
> Not sure if we want to avoid the unnecessary performance overhead of repeatedly
> searching IRR.

IMO, it's not worth the risk.  That overhead effectively exists at all times on
modern Intel CPUs, as APICv is highly likely to be enabled, i.e. irr_pending will
always be true.

Hrm, but looking at this again, I'm not sure I like v2.  Ah, it's not the code I
don't like, it's the name of the helper that's flawed.  kvm_apic_update_apicv()
makes it seem like it's only releveant when enable_apicv=true, but that's not the
case.  E.g. doing apic->irr_pending = enable_apicv would be broken.

But that flaws exists even before this patch, because kvm_apic_set_state() and
kvm_lapic_reset() rely on it to update apic->isr_count (which needs to be cleared
on INIT for the latter).

I'll send a (not for 6.12) patch to rename it, e.g. to kvm_apic_update_sw_caches()
or something.