Re: CPU Lockups in KVM with deferred hrtimer rearming

[Sean Christopherson <seanjc@google.com>]

On Wed, Apr 22, 2026, Peter Zijlstra wrote:
> On Wed, Apr 22, 2026 at 09:46:46AM +0200, Peter Zijlstra wrote:
> > +		instrumentation_begin();
> > +		/*
> > +		 * KVM/VMX will dispatch from IRQ-disabled but for a context
> > +		 * that will have IRQs-enabled. This confuses the entry code
> > +		 * and it will not have reprogrammed the timer (or do
> > +		 * preemption). Minimal fixup for now.
> > +		 */
> > +		hrtimer_rearm_deferred();
> > +		instrumentation_end();
> 
> So I've been looking at this preemption thing. After having gotten my
> head in a twist a few times around, I think this is done by
> vcpu_enter_guest() like:
> 
> 	preempt_disable();
> 	local_irq_disable();
> 	...
> 	kvm_x86_call(handle_exit_irqoff)(vcpu); <--- all of this VMX nonsense

To be fair, there's some SVM nonsense too. 

> 	...
> 	local_irq_enable();
> 				<--- WTF goes here :-)

All IRQs on AMD, and tick IRQs on VMX that arrive _just_ after the VM-Exit.

Because IRQ exits on AMD/SVM are purely a notification, KVM needs to enable IRQs
in order to service the exit.  The early enabling exists to get the timeslice
accounting correct.  With the comments...

	/*
	 * Consume any pending interrupts, including the possible source of
	 * VM-Exit on SVM and any ticks that occur between VM-Exit and now.
	 * An instruction is required after local_irq_enable() to fully unblock
	 * interrupts on processors that implement an interrupt shadow, the
	 * stat.exits increment will do nicely.
	 */
	kvm_before_interrupt(vcpu, KVM_HANDLING_IRQ);
	local_irq_enable();
	++vcpu->stat.exits;
	local_irq_disable();
	kvm_after_interrupt(vcpu);

	/*
	 * Wait until after servicing IRQs to account guest time so that any
	 * ticks that occurred while running the guest are properly accounted
	 * to the guest.  Waiting until IRQs are enabled degrades the accuracy
	 * of accounting via context tracking, but the loss of accuracy is
	 * acceptable for all known use cases.
	 */
	guest_timing_exit_irqoff();

When NOT precisely accounting virtual CPU usage, accounting is done by setting
PF_VCPU on current->flags on the way into the guest, and then clearing it on the
way back out.  The latter is done by guest_timing_exit_irqoff().  If KVM waits
to enable IRQs until after guest_timing_exit_irqoff(), then the tick IRQ handler
will account the tick to the host, not that guest, even if 99.99999% of the time
was spent in the guest.

As to why this is in common code, i.e. isn't AMD/SVM specific, if the guest and
host are running with the same tick frequency, it's suprisingly easy to get in
a state where the host tick IRQ almost always arrives just after VM-Exit, before
KVM fully enables IRQs.  Specifically, the guest's programmed tick will trigger
a VMX Preemption Timer VM-Exit at the same frequency the host's tick triggers an
IRQ.

> 	++vcpu->stat.exits;
> 	local_irq_disable();
> 	...
> 	local_irq_enable();
> 	preempt_enable(); <--- here we finally preempt
> 
> This earlier IRQ-enable makes my head hurt and I had to go buy a new
> WTF'o'meter (again!).