[PATCH] KVM: VMX: Clear GUEST_ACTIVITY_STATE when userspace makes a vCPU RUNNABLE

[PATCH] KVM: VMX: Clear GUEST_ACTIVITY_STATE when userspace makes a vCPU RUNNABLE

[Keqiang Duan]

Force a vCPU out of its hardware-tracked halted state when userspace
explicitly declares the vCPU RUNNABLE via KVM_SET_MP_STATE, i.e. clear
VMCS.GUEST_ACTIVITY_STATE if it says the vCPU is halted.  Add an optional
kvm_x86_ops hook to do the clearing, as SVM has no equivalent VMCB field.

When HLT-exiting is disabled for a VM (KVM_CAP_X86_DISABLE_EXITS with
KVM_X86_DISABLE_EXITS_HLT, e.g. QEMU's "-overcommit cpu-pm=on"), a guest
HLT halts the physical CPU instead of exiting to KVM, and hardware saves
GUEST_ACTIVITY_STATE=HLT into the VMCS on the next VM-Exit.  That field is
sticky: it survives VM-Exit/VM-Enter and is only cleared by vmx_clear_hlt()
on event injection, or by vmx_vcpu_reset() on INIT / vCPU creation.

Nothing clears it on a userspace-driven state change.  KVM_SET_REGS only
writes the software register cache and KVM_SET_MP_STATE only writes
vcpu->arch.mp_state; kvm_vcpu_running() likewise consults software state
only.  A VMM that emulates a machine reset therefore ends up with a vCPU
that KVM happily VM-Enters while hardware refuses to fetch instructions.

Reproduce with a Linux guest by triggering a panic/kdump on a non-boot
vCPU: nmi_shootdown_cpus() parks the other vCPUs -- including vCPU0 -- in
crash_nmi_callback(), which does local_irq_disable() followed by a bare
HLT.  The capture kernel then resets the machine via port 0xCF9.  QEMU
rewrites RIP to 0xfff0 and sets mp_state to RUNNABLE, but vCPU0's
GUEST_ACTIVITY_STATE is still HLT, so the BSP never executes the reset
vector, never sends SIPIs, and the entire VM hangs at "reboot: machine
restart" forever.  Only destroying and recreating the VM recovers it.

Clearing the state is always safe: waking from HLT is architecturally
permitted to be spurious, and every HLT in the kernel is inside a loop.
Hook KVM_SET_MP_STATE rather than the VM-Enter path so that the clearing
is driven by an explicit userspace declaration, and so that no work is
added to vmx_vcpu_run().

Note, vmx_clear_hlt() loses its "static" qualifier as the kvm_x86_ops table
now lives in vmx/main.c.  TDX cannot disable HLT-exiting and KVM cannot
access a TD's VMCS, so vt_clear_hlt() short-circuits for TD vCPUs,
following the existing vt_*() wrapper pattern.

Fixes: caa057a2cad6 ("KVM: X86: Provide a capability to disable HLT intercepts")
Cc: stable@vger.kernel.org
Signed-off-by: Keqiang Duan <duankeqiangcym@gmail.com>
---
 arch/x86/include/asm/kvm-x86-ops.h |  1 +
 arch/x86/include/asm/kvm_host.h    |  7 +++++++
 arch/x86/kvm/vmx/main.c            | 13 +++++++++++++
 arch/x86/kvm/vmx/vmx.c             |  2 +-
 arch/x86/kvm/vmx/x86_ops.h         |  1 +
 arch/x86/kvm/x86.c                 | 13 +++++++++++++
 6 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index e213c9ae3e30..dd9026c1071c 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -82,6 +82,7 @@ KVM_X86_OP(interrupt_allowed)
 KVM_X86_OP(nmi_allowed)
 KVM_X86_OP(get_nmi_mask)
 KVM_X86_OP(set_nmi_mask)
+KVM_X86_OP_OPTIONAL(clear_hlt)
 KVM_X86_OP(enable_nmi_window)
 KVM_X86_OP(enable_irq_window)
 KVM_X86_OP_OPTIONAL(update_cr8_intercept)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 283847619ff8..e5a0758536e1 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1607,6 +1607,13 @@ struct kvm_x86_ops {
 	int (*nmi_allowed)(struct kvm_vcpu *vcpu, bool for_injection);
 	bool (*get_nmi_mask)(struct kvm_vcpu *vcpu);
 	void (*set_nmi_mask)(struct kvm_vcpu *vcpu, bool masked);
+	/*
+	 * Force the vCPU out of any hardware-tracked halted/inactive state so
+	 * that it will fetch and execute instructions on the next VM-Enter.
+	 * Only needed by VMX, where VMCS.GUEST_ACTIVITY_STATE persists across
+	 * VM-Exit/VM-Enter; SVM has no equivalent VMCB field.
+	 */
+	void (*clear_hlt)(struct kvm_vcpu *vcpu);
 	/* Whether or not a virtual NMI is pending in hardware. */
 	bool (*is_vnmi_pending)(struct kvm_vcpu *vcpu);
 	/*
diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c
index 0ff3230fd95e..74ce9dd70419 100644
--- a/arch/x86/kvm/vmx/main.c
+++ b/arch/x86/kvm/vmx/main.c
@@ -601,6 +601,18 @@ static void vt_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked)
 	vmx_set_nmi_mask(vcpu, masked);
 }
 
+static void vt_clear_hlt(struct kvm_vcpu *vcpu)
+{
+	/*
+	 * TDX doesn't support disabling HLT-exiting, and KVM can't access a
+	 * TD's VMCS, so there is never any hardware halted state to clear.
+	 */
+	if (is_td_vcpu(vcpu))
+		return;
+
+	vmx_clear_hlt(vcpu);
+}
+
 static void vt_enable_nmi_window(struct kvm_vcpu *vcpu)
 {
 	/* Refer to the comments in tdx_inject_nmi(). */
@@ -964,6 +976,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = {
 	.nmi_allowed = vt_op(nmi_allowed),
 	.get_nmi_mask = vt_op(get_nmi_mask),
 	.set_nmi_mask = vt_op(set_nmi_mask),
+	.clear_hlt = vt_op(clear_hlt),
 	.enable_nmi_window = vt_op(enable_nmi_window),
 	.enable_irq_window = vt_op(enable_irq_window),
 	.update_cr8_intercept = vt_op(update_cr8_intercept),
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index e3bfe6aca1a0..8ad79c60ecc6 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -1909,7 +1909,7 @@ int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu)
 	return skip_emulated_instruction(vcpu);
 }
 
-static void vmx_clear_hlt(struct kvm_vcpu *vcpu)
+void vmx_clear_hlt(struct kvm_vcpu *vcpu)
 {
 	/*
 	 * Ensure that we clear the HLT state in the VMCS.  We don't need to
diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h
index cdb38d940cfb..45e47f502a37 100644
--- a/arch/x86/kvm/vmx/x86_ops.h
+++ b/arch/x86/kvm/vmx/x86_ops.h
@@ -95,6 +95,7 @@ int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection);
 int vmx_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection);
 bool vmx_get_nmi_mask(struct kvm_vcpu *vcpu);
 void vmx_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked);
+void vmx_clear_hlt(struct kvm_vcpu *vcpu);
 void vmx_enable_nmi_window(struct kvm_vcpu *vcpu);
 void vmx_enable_irq_window(struct kvm_vcpu *vcpu);
 void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index d94b59140c45..7777cf88a96b 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -9058,6 +9058,19 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu,
 	}
 
 	kvm_set_mp_state(vcpu, mp_state->mp_state);
+
+	/*
+	 * Force the vCPU out of any hardware-tracked inactive state, e.g. VMX's
+	 * GUEST_ACTIVITY_STATE=HLT.  That state is sticky across VM-Exit and
+	 * VM-Enter and is not touched by any other ioctl, so a vCPU that halted
+	 * with HLT-exiting disabled (KVM_X86_DISABLE_EXITS_HLT) stays wedged
+	 * even after userspace declares it RUNNABLE and rewrites its registers,
+	 * e.g. when a VMM emulates a machine reset.  Waking from HLT is
+	 * architecturally allowed to be spurious, so clearing it is always safe.
+	 */
+	if (vcpu->arch.mp_state == KVM_MP_STATE_RUNNABLE)
+		kvm_x86_call(clear_hlt)(vcpu);
+
 	kvm_make_request(KVM_REQ_EVENT, vcpu);
 
 	ret = 0;

base-commit: 1b731e5ded480bd1e5546aed35584238661ce72e
-- 
2.24.3

Re: [PATCH] KVM: VMX: Clear GUEST_ACTIVITY_STATE when userspace makes a vCPU RUNNABLE

[Chao Gao]

On Wed, Aug 19, 2026 at 11:46:52AM +0800, Keqiang Duan wrote:
>Force a vCPU out of its hardware-tracked halted state when userspace
>explicitly declares the vCPU RUNNABLE via KVM_SET_MP_STATE, i.e. clear
>VMCS.GUEST_ACTIVITY_STATE if it says the vCPU is halted.  Add an optional
>kvm_x86_ops hook to do the clearing, as SVM has no equivalent VMCB field.
>
>When HLT-exiting is disabled for a VM (KVM_CAP_X86_DISABLE_EXITS with
>KVM_X86_DISABLE_EXITS_HLT, e.g. QEMU's "-overcommit cpu-pm=on"), a guest
>HLT halts the physical CPU instead of exiting to KVM, and hardware saves
>GUEST_ACTIVITY_STATE=HLT into the VMCS on the next VM-Exit.  That field is
>sticky: it survives VM-Exit/VM-Enter and is only cleared by vmx_clear_hlt()
>on event injection, or by vmx_vcpu_reset() on INIT / vCPU creation.
>
>Nothing clears it on a userspace-driven state change.  KVM_SET_REGS only
>writes the software register cache and KVM_SET_MP_STATE only writes
>vcpu->arch.mp_state; kvm_vcpu_running() likewise consults software state
>only.  A VMM that emulates a machine reset therefore ends up with a vCPU
>that KVM happily VM-Enters while hardware refuses to fetch instructions.
>
>Reproduce with a Linux guest by triggering a panic/kdump on a non-boot
>vCPU: nmi_shootdown_cpus() parks the other vCPUs -- including vCPU0 -- in
>crash_nmi_callback(), which does local_irq_disable() followed by a bare
>HLT.  The capture kernel then resets the machine via port 0xCF9.  QEMU
>rewrites RIP to 0xfff0 and sets mp_state to RUNNABLE, but vCPU0's
>GUEST_ACTIVITY_STATE is still HLT, so the BSP never executes the reset
>vector, never sends SIPIs, and the entire VM hangs at "reboot: machine
>restart" forever.  Only destroying and recreating the VM recovers it.
>
>Clearing the state is always safe: waking from HLT is architecturally
>permitted to be spurious, and every HLT in the kernel is inside a loop.
>Hook KVM_SET_MP_STATE rather than the VM-Enter path so that the clearing
>is driven by an explicit userspace declaration, and so that no work is
>added to vmx_vcpu_run().
>
>Note, vmx_clear_hlt() loses its "static" qualifier as the kvm_x86_ops table
>now lives in vmx/main.c.  TDX cannot disable HLT-exiting and KVM cannot
>access a TD's VMCS, so vt_clear_hlt() short-circuits for TD vCPUs,
>following the existing vt_*() wrapper pattern.
>
>Fixes: caa057a2cad6 ("KVM: X86: Provide a capability to disable HLT intercepts")
>Cc: stable@vger.kernel.org
>Signed-off-by: Keqiang Duan <duankeqiangcym@gmail.com>

There was an earlier attempt to fix this issue:

https://lore.kernel.org/kvm/20230630072612.1106705-1-aiqi.i7@bytedance.com

Sean suggested doing exactly this there, i.e. clearing the activity state in
kvm_arch_vcpu_ioctl_set_mpstate() via a new kvm_x86_ops hook:

https://lore.kernel.org/kvm/ZMgIQ5m1jMSAogT4@google.com/

Re: [PATCH] KVM: VMX: Clear GUEST_ACTIVITY_STATE when userspace makes a vCPU RUNNABLE

[Sean Christopherson]

On Wed, Aug 19, 2026, Chao Gao wrote:
> On Wed, Aug 19, 2026 at 11:46:52AM +0800, Keqiang Duan wrote:
> >Force a vCPU out of its hardware-tracked halted state when userspace
> >explicitly declares the vCPU RUNNABLE via KVM_SET_MP_STATE, i.e. clear
> >VMCS.GUEST_ACTIVITY_STATE if it says the vCPU is halted.  Add an optional
> >kvm_x86_ops hook to do the clearing, as SVM has no equivalent VMCB field.
> >
> >When HLT-exiting is disabled for a VM (KVM_CAP_X86_DISABLE_EXITS with
> >KVM_X86_DISABLE_EXITS_HLT, e.g. QEMU's "-overcommit cpu-pm=on"), a guest
> >HLT halts the physical CPU instead of exiting to KVM, and hardware saves
> >GUEST_ACTIVITY_STATE=HLT into the VMCS on the next VM-Exit.  That field is
> >sticky: it survives VM-Exit/VM-Enter and is only cleared by vmx_clear_hlt()
> >on event injection, or by vmx_vcpu_reset() on INIT / vCPU creation.
> >
> >Nothing clears it on a userspace-driven state change.  KVM_SET_REGS only
> >writes the software register cache and KVM_SET_MP_STATE only writes
> >vcpu->arch.mp_state; kvm_vcpu_running() likewise consults software state
> >only.  A VMM that emulates a machine reset therefore ends up with a vCPU
> >that KVM happily VM-Enters while hardware refuses to fetch instructions.
> >
> >Reproduce with a Linux guest by triggering a panic/kdump on a non-boot
> >vCPU: nmi_shootdown_cpus() parks the other vCPUs -- including vCPU0 -- in
> >crash_nmi_callback(), which does local_irq_disable() followed by a bare
> >HLT.  The capture kernel then resets the machine via port 0xCF9.  QEMU
> >rewrites RIP to 0xfff0 and sets mp_state to RUNNABLE, but vCPU0's
> >GUEST_ACTIVITY_STATE is still HLT, so the BSP never executes the reset
> >vector, never sends SIPIs, and the entire VM hangs at "reboot: machine
> >restart" forever.  Only destroying and recreating the VM recovers it.
> >
> >Clearing the state is always safe: waking from HLT is architecturally
> >permitted to be spurious, and every HLT in the kernel is inside a loop.
> >Hook KVM_SET_MP_STATE rather than the VM-Enter path so that the clearing
> >is driven by an explicit userspace declaration, and so that no work is
> >added to vmx_vcpu_run().
> >
> >Note, vmx_clear_hlt() loses its "static" qualifier as the kvm_x86_ops table
> >now lives in vmx/main.c.  TDX cannot disable HLT-exiting and KVM cannot
> >access a TD's VMCS, so vt_clear_hlt() short-circuits for TD vCPUs,
> >following the existing vt_*() wrapper pattern.
> >
> >Fixes: caa057a2cad6 ("KVM: X86: Provide a capability to disable HLT intercepts")
> >Cc: stable@vger.kernel.org
> >Signed-off-by: Keqiang Duan <duankeqiangcym@gmail.com>
> 
> There was an earlier attempt to fix this issue:
> 
> https://lore.kernel.org/kvm/20230630072612.1106705-1-aiqi.i7@bytedance.com
> 
> Sean suggested doing exactly this there, i.e. clearing the activity state in
> kvm_arch_vcpu_ioctl_set_mpstate() via a new kvm_x86_ops hook:
> 
> https://lore.kernel.org/kvm/ZMgIQ5m1jMSAogT4@google.com

Thanks Chao, I knew this sounded familiar!

As suggested in the above link, the call into vendor code should be gated on
kvm_hlt_in_guest().  Yes, vmx_clear_hlt() already does that, but it's a cheap
check and makes the common x86 code easier to understand.

The other difference between my suggestion and this patch is clearing HLT if and
only if the vCPU is being made RUNNABLE, versus clearing HLT if the vCPU is being
put into any state other than HALTED.  In practice it probably doesn't matter?

AP_RESET_HOLD is unsupported for VMX, SIPI_RECEIVED is deprecated and is now just
a variant off INIT_RECEIVED, and getting out of INIT_RECIEVED and UNINITIALIZED
requires stuffing RUNNABLE or going through vmx_vcpu_reset(), which explicitly
sets the activity state to GUEST_ACTIVITY_ACTIVE.

But I think my vote would still be to do:

	mp_state->mp_state != KVM_MP_STATE_HALTED

because I can't think of any reason not to?

Side topic, I think we should officially take an erratum in
Documentation/virt/kvm/x86/errata.rst to state that KVM doesn't support save/restore
of HLT state when HLT-exiting is disabled.  AFAIK, there's simply no sane solution
for SVM, and that would justify why the kvm_x86_ops would be clear_hlt() as opposed
to set_mp_state(), i.e. why KVM only handles taking vCPUs out of HLT, not putting
them into HLT.