Re: Guest OS migration and lost IPIs

[Paolo Bonzini <pbonzini@redhat.com>]

On 05/08/20 02:07, Paul E. McKenney wrote:
> 
> We are seeing occasional odd hangs, but only in cases where guest OSes
> are being migrated.  Migrating more often makes the hangs happen more
> frequently.
> 
> Added debug showed that the hung CPU is stuck trying to send an IPI (e.g.,
> smp_call_function_single()).  The hung CPU thinks that it has sent the
> IPI, but the destination CPU has interrupts enabled (-not- disabled,
> enabled, as in ready, willing, and able to take interrupts).  In fact,
> the destination CPU usually is going about its business as if nothing
> was wrong, which makes me suspect that the IPI got lost somewhere along
> the way.
> 
> I bumbled a bit through the qemu and KVM source, and didn't find anything
> synchronizing IPIs and migrations, though given that I know pretty much
> nothing about either qemu or KVM, this doesn't count for much.

The code migrating the interrupt controller is in
kvm_x86_ops.sync_pir_to_irr (which calls vmx_sync_pir_to_irr) and
kvm_apic_get_state.  kvm_apic_get_state is called after CPUs are stopped.

It's possible that we're missing a kvm_x86_ops.sync_pir_to_irr call
somewhere.  It would be surprising but it would explain the symptoms
very well.

Paolo