[PATCH v3] KVM: SVM: Don't skip unrelated instruction if INT3/INTO is replaced

[PATCH v3] KVM: SVM: Don't skip unrelated instruction if INT3/INTO is replaced

[Omar Sandoval]

From: Omar Sandoval <osandov@fb.com>

When re-injecting a soft interrupt from an INT3, INT0, or (select) INTn
instruction, discard the exception and retry the instruction if the code
stream is changed (e.g. by a different vCPU) between when the CPU
executes the instruction and when KVM decodes the instruction to get the
next RIP.

As effectively predicted by commit 6ef88d6e36c2 ("KVM: SVM: Re-inject
INT3/INTO instead of retrying the instruction"), failure to verify that
the correct INTn instruction was decoded can effectively clobber guest
state due to decoding the wrong instruction and thus specifying the
wrong next RIP.

The bug most often manifests as "Oops: int3" panics on static branch
checks in Linux guests.  Enabling or disabling a static branch in Linux
uses the kernel's "text poke" code patching mechanism.  To modify code
while other CPUs may be executing that code, Linux (temporarily)
replaces the first byte of the original instruction with an int3 (opcode
0xcc), then patches in the new code stream except for the first byte,
and finally replaces the int3 with the first byte of the new code
stream.  If a CPU hits the int3, i.e. executes the code while it's being
modified, then the guest kernel must look up the RIP to determine how to
handle the #BP, e.g. by emulating the new instruction.  If the RIP is
incorrect, then this lookup fails and the guest kernel panics.

The bug reproduces almost instantly by hacking the guest kernel to
repeatedly check a static branch[1] while running a drgn script[2] on
the host to constantly swap out the memory containing the guest's TSS.

[1]: https://gist.github.com/osandov/44d17c51c28c0ac998ea0334edf90b5a
[2]: https://gist.github.com/osandov/10e45e45afa29b11e0c7209247afc00b

Fixes: 6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the instruction")
Cc: stable@vger.kernel.org
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Omar Sandoval <osandov@fb.com>
---
Changes from v2 (https://lore.kernel.org/all/6ab4e8e5c5d6ea95f568a1ff8044779137dce428.1761774582.git.osandov@fb.com/):

- Fixed EMULTYPE_SET_SOFT_INT_VECTOR -> EMULTYPE_GET_SOFT_INT_VECTOR
  typo.
- Added explicit u32 cast to EMULTYPE_SET_SOFT_INT_VECTOR to make it
  clear that it won't overflow.
- Rebased on Linus's tree as of c9cfc122f03711a5124b4aafab3211cf4d35a2ac.

 arch/x86/include/asm/kvm_host.h |  9 +++++++++
 arch/x86/kvm/svm/svm.c          | 24 +++++++++++++-----------
 arch/x86/kvm/x86.c              | 21 +++++++++++++++++++++
 3 files changed, 43 insertions(+), 11 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 48598d017d6f..974d64bf0a4d 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -2143,6 +2143,11 @@ u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu);
  *			     the gfn, i.e. retrying the instruction will hit a
  *			     !PRESENT fault, which results in a new shadow page
  *			     and sends KVM back to square one.
+ *
+ * EMULTYPE_SKIP_SOFT_INT - Set in combination with EMULTYPE_SKIP to only skip
+ *                          an instruction if it could generate a given software
+ *                          interrupt, which must be encoded via
+ *                          EMULTYPE_SET_SOFT_INT_VECTOR().
  */
 #define EMULTYPE_NO_DECODE	    (1 << 0)
 #define EMULTYPE_TRAP_UD	    (1 << 1)
@@ -2153,6 +2158,10 @@ u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu);
 #define EMULTYPE_PF		    (1 << 6)
 #define EMULTYPE_COMPLETE_USER_EXIT (1 << 7)
 #define EMULTYPE_WRITE_PF_TO_SP	    (1 << 8)
+#define EMULTYPE_SKIP_SOFT_INT	    (1 << 9)
+
+#define EMULTYPE_SET_SOFT_INT_VECTOR(v)	((u32)((v) & 0xff) << 16)
+#define EMULTYPE_GET_SOFT_INT_VECTOR(e)	(((e) >> 16) & 0xff)
 
 static inline bool kvm_can_emulate_event_vectoring(int emul_type)
 {
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 153c12dbf3eb..a9fb60bbd1c7 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -272,6 +272,7 @@ static void svm_set_interrupt_shadow(struct kvm_vcpu *vcpu, int mask)
 }
 
 static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu,
+					   int emul_type,
 					   bool commit_side_effects)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -293,7 +294,7 @@ static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu,
 		if (unlikely(!commit_side_effects))
 			old_rflags = svm->vmcb->save.rflags;
 
-		if (!kvm_emulate_instruction(vcpu, EMULTYPE_SKIP))
+		if (!kvm_emulate_instruction(vcpu, emul_type))
 			return 0;
 
 		if (unlikely(!commit_side_effects))
@@ -311,11 +312,13 @@ static int __svm_skip_emulated_instruction(struct kvm_vcpu *vcpu,
 
 static int svm_skip_emulated_instruction(struct kvm_vcpu *vcpu)
 {
-	return __svm_skip_emulated_instruction(vcpu, true);
+	return __svm_skip_emulated_instruction(vcpu, EMULTYPE_SKIP, true);
 }
 
-static int svm_update_soft_interrupt_rip(struct kvm_vcpu *vcpu)
+static int svm_update_soft_interrupt_rip(struct kvm_vcpu *vcpu, u8 vector)
 {
+	const int emul_type = EMULTYPE_SKIP | EMULTYPE_SKIP_SOFT_INT |
+			      EMULTYPE_SET_SOFT_INT_VECTOR(vector);
 	unsigned long rip, old_rip = kvm_rip_read(vcpu);
 	struct vcpu_svm *svm = to_svm(vcpu);
 
@@ -331,7 +334,7 @@ static int svm_update_soft_interrupt_rip(struct kvm_vcpu *vcpu)
 	 * in use, the skip must not commit any side effects such as clearing
 	 * the interrupt shadow or RFLAGS.RF.
 	 */
-	if (!__svm_skip_emulated_instruction(vcpu, !nrips))
+	if (!__svm_skip_emulated_instruction(vcpu, emul_type, !nrips))
 		return -EIO;
 
 	rip = kvm_rip_read(vcpu);
@@ -367,7 +370,7 @@ static void svm_inject_exception(struct kvm_vcpu *vcpu)
 	kvm_deliver_exception_payload(vcpu, ex);
 
 	if (kvm_exception_is_soft(ex->vector) &&
-	    svm_update_soft_interrupt_rip(vcpu))
+	    svm_update_soft_interrupt_rip(vcpu, ex->vector))
 		return;
 
 	svm->vmcb->control.event_inj = ex->vector
@@ -3637,11 +3640,12 @@ static bool svm_set_vnmi_pending(struct kvm_vcpu *vcpu)
 
 static void svm_inject_irq(struct kvm_vcpu *vcpu, bool reinjected)
 {
+	struct kvm_queued_interrupt *intr = &vcpu->arch.interrupt;
 	struct vcpu_svm *svm = to_svm(vcpu);
 	u32 type;
 
-	if (vcpu->arch.interrupt.soft) {
-		if (svm_update_soft_interrupt_rip(vcpu))
+	if (intr->soft) {
+		if (svm_update_soft_interrupt_rip(vcpu, intr->nr))
 			return;
 
 		type = SVM_EVTINJ_TYPE_SOFT;
@@ -3649,12 +3653,10 @@ static void svm_inject_irq(struct kvm_vcpu *vcpu, bool reinjected)
 		type = SVM_EVTINJ_TYPE_INTR;
 	}
 
-	trace_kvm_inj_virq(vcpu->arch.interrupt.nr,
-			   vcpu->arch.interrupt.soft, reinjected);
+	trace_kvm_inj_virq(intr->nr, intr->soft, reinjected);
 	++vcpu->stat.irq_injections;
 
-	svm->vmcb->control.event_inj = vcpu->arch.interrupt.nr |
-				       SVM_EVTINJ_VALID | type;
+	svm->vmcb->control.event_inj = intr->nr | SVM_EVTINJ_VALID | type;
 }
 
 void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode,
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index b4b5d2d09634..9dc66cca154d 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -9338,6 +9338,23 @@ static bool is_vmware_backdoor_opcode(struct x86_emulate_ctxt *ctxt)
 	return false;
 }
 
+static bool is_soft_int_instruction(struct x86_emulate_ctxt *ctxt,
+				    int emulation_type)
+{
+	u8 vector = EMULTYPE_GET_SOFT_INT_VECTOR(emulation_type);
+
+	switch (ctxt->b) {
+	case 0xcc:
+		return vector == BP_VECTOR;
+	case 0xcd:
+		return vector == ctxt->src.val;
+	case 0xce:
+		return vector == OF_VECTOR;
+	default:
+		return false;
+	}
+}
+
 /*
  * Decode an instruction for emulation.  The caller is responsible for handling
  * code breakpoints.  Note, manually detecting code breakpoints is unnecessary
@@ -9448,6 +9465,10 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	 * injecting single-step #DBs.
 	 */
 	if (emulation_type & EMULTYPE_SKIP) {
+		if (emulation_type & EMULTYPE_SKIP_SOFT_INT &&
+		    !is_soft_int_instruction(ctxt, emulation_type))
+			return 0;
+
 		if (ctxt->mode != X86EMUL_MODE_PROT64)
 			ctxt->eip = (u32)ctxt->_eip;
 		else
-- 
2.51.1

Re: [PATCH v3] KVM: SVM: Don't skip unrelated instruction if INT3/INTO is replaced

[Omar Sandoval]

On Tue, Nov 04, 2025 at 09:55:26AM -0800, Omar Sandoval wrote:
> From: Omar Sandoval <osandov@fb.com>
> 
> When re-injecting a soft interrupt from an INT3, INT0, or (select) INTn
> instruction, discard the exception and retry the instruction if the code
> stream is changed (e.g. by a different vCPU) between when the CPU
> executes the instruction and when KVM decodes the instruction to get the
> next RIP.
> 
> As effectively predicted by commit 6ef88d6e36c2 ("KVM: SVM: Re-inject
> INT3/INTO instead of retrying the instruction"), failure to verify that
> the correct INTn instruction was decoded can effectively clobber guest
> state due to decoding the wrong instruction and thus specifying the
> wrong next RIP.
> 
> The bug most often manifests as "Oops: int3" panics on static branch
> checks in Linux guests.  Enabling or disabling a static branch in Linux
> uses the kernel's "text poke" code patching mechanism.  To modify code
> while other CPUs may be executing that code, Linux (temporarily)
> replaces the first byte of the original instruction with an int3 (opcode
> 0xcc), then patches in the new code stream except for the first byte,
> and finally replaces the int3 with the first byte of the new code
> stream.  If a CPU hits the int3, i.e. executes the code while it's being
> modified, then the guest kernel must look up the RIP to determine how to
> handle the #BP, e.g. by emulating the new instruction.  If the RIP is
> incorrect, then this lookup fails and the guest kernel panics.
> 
> The bug reproduces almost instantly by hacking the guest kernel to
> repeatedly check a static branch[1] while running a drgn script[2] on
> the host to constantly swap out the memory containing the guest's TSS.
> 
> [1]: https://gist.github.com/osandov/44d17c51c28c0ac998ea0334edf90b5a
> [2]: https://gist.github.com/osandov/10e45e45afa29b11e0c7209247afc00b
> 
> Fixes: 6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the instruction")
> Cc: stable@vger.kernel.org
> Co-developed-by: Sean Christopherson <seanjc@google.com>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> Signed-off-by: Omar Sandoval <osandov@fb.com>
> ---
> Changes from v2 (https://lore.kernel.org/all/6ab4e8e5c5d6ea95f568a1ff8044779137dce428.1761774582.git.osandov@fb.com/):
> 
> - Fixed EMULTYPE_SET_SOFT_INT_VECTOR -> EMULTYPE_GET_SOFT_INT_VECTOR
>   typo.
> - Added explicit u32 cast to EMULTYPE_SET_SOFT_INT_VECTOR to make it
>   clear that it won't overflow.
> - Rebased on Linus's tree as of c9cfc122f03711a5124b4aafab3211cf4d35a2ac.
> 
>  arch/x86/include/asm/kvm_host.h |  9 +++++++++
>  arch/x86/kvm/svm/svm.c          | 24 +++++++++++++-----------
>  arch/x86/kvm/x86.c              | 21 +++++++++++++++++++++
>  3 files changed, 43 insertions(+), 11 deletions(-)

Ping, does this need any more updates?

Thank you,
Omar

Re: [PATCH v3] KVM: SVM: Don't skip unrelated instruction if INT3/INTO is replaced

[Sean Christopherson]

On Tue, Nov 18, 2025, Omar Sandoval wrote:
> On Tue, Nov 04, 2025 at 09:55:26AM -0800, Omar Sandoval wrote:
> > From: Omar Sandoval <osandov@fb.com>
> > 
> > When re-injecting a soft interrupt from an INT3, INT0, or (select) INTn
> > instruction, discard the exception and retry the instruction if the code
> > stream is changed (e.g. by a different vCPU) between when the CPU
> > executes the instruction and when KVM decodes the instruction to get the
> > next RIP.
> > 
> > As effectively predicted by commit 6ef88d6e36c2 ("KVM: SVM: Re-inject
> > INT3/INTO instead of retrying the instruction"), failure to verify that
> > the correct INTn instruction was decoded can effectively clobber guest
> > state due to decoding the wrong instruction and thus specifying the
> > wrong next RIP.
> > 
> > The bug most often manifests as "Oops: int3" panics on static branch
> > checks in Linux guests.  Enabling or disabling a static branch in Linux
> > uses the kernel's "text poke" code patching mechanism.  To modify code
> > while other CPUs may be executing that code, Linux (temporarily)
> > replaces the first byte of the original instruction with an int3 (opcode
> > 0xcc), then patches in the new code stream except for the first byte,
> > and finally replaces the int3 with the first byte of the new code
> > stream.  If a CPU hits the int3, i.e. executes the code while it's being
> > modified, then the guest kernel must look up the RIP to determine how to
> > handle the #BP, e.g. by emulating the new instruction.  If the RIP is
> > incorrect, then this lookup fails and the guest kernel panics.
> > 
> > The bug reproduces almost instantly by hacking the guest kernel to
> > repeatedly check a static branch[1] while running a drgn script[2] on
> > the host to constantly swap out the memory containing the guest's TSS.
> > 
> > [1]: https://gist.github.com/osandov/44d17c51c28c0ac998ea0334edf90b5a
> > [2]: https://gist.github.com/osandov/10e45e45afa29b11e0c7209247afc00b
> > 
> > Fixes: 6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the instruction")
> > Cc: stable@vger.kernel.org
> > Co-developed-by: Sean Christopherson <seanjc@google.com>
> > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > Signed-off-by: Omar Sandoval <osandov@fb.com>
> > ---
> > Changes from v2 (https://lore.kernel.org/all/6ab4e8e5c5d6ea95f568a1ff8044779137dce428.1761774582.git.osandov@fb.com/):
> > 
> > - Fixed EMULTYPE_SET_SOFT_INT_VECTOR -> EMULTYPE_GET_SOFT_INT_VECTOR
> >   typo.
> > - Added explicit u32 cast to EMULTYPE_SET_SOFT_INT_VECTOR to make it
> >   clear that it won't overflow.
> > - Rebased on Linus's tree as of c9cfc122f03711a5124b4aafab3211cf4d35a2ac.
> > 
> >  arch/x86/include/asm/kvm_host.h |  9 +++++++++
> >  arch/x86/kvm/svm/svm.c          | 24 +++++++++++++-----------
> >  arch/x86/kvm/x86.c              | 21 +++++++++++++++++++++
> >  3 files changed, 43 insertions(+), 11 deletions(-)
> 
> Ping, does this need any more updates?

For the record, I had this applied and tested before the ping, I just hadn't sent
the "thank you" yet.  :-D

commit 4da3768e1820cf15cced390242d8789aed34f54d
Author:     Omar Sandoval <osandov@fb.com>
AuthorDate: Tue Nov 4 09:55:26 2025 -0800
Commit:     Sean Christopherson <seanjc@google.com>
CommitDate: Thu Nov 13 13:03:19 2025 -0800  <======= See, I'm not lying!

Re: [PATCH v3] KVM: SVM: Don't skip unrelated instruction if INT3/INTO is replaced

[Omar Sandoval]

On Tue, Nov 18, 2025 at 03:33:30PM -0800, Sean Christopherson wrote:
> On Tue, Nov 18, 2025, Omar Sandoval wrote:
> > On Tue, Nov 04, 2025 at 09:55:26AM -0800, Omar Sandoval wrote:
> > > From: Omar Sandoval <osandov@fb.com>
> > > 
> > > When re-injecting a soft interrupt from an INT3, INT0, or (select) INTn
> > > instruction, discard the exception and retry the instruction if the code
> > > stream is changed (e.g. by a different vCPU) between when the CPU
> > > executes the instruction and when KVM decodes the instruction to get the
> > > next RIP.
> > > 
> > > As effectively predicted by commit 6ef88d6e36c2 ("KVM: SVM: Re-inject
> > > INT3/INTO instead of retrying the instruction"), failure to verify that
> > > the correct INTn instruction was decoded can effectively clobber guest
> > > state due to decoding the wrong instruction and thus specifying the
> > > wrong next RIP.
> > > 
> > > The bug most often manifests as "Oops: int3" panics on static branch
> > > checks in Linux guests.  Enabling or disabling a static branch in Linux
> > > uses the kernel's "text poke" code patching mechanism.  To modify code
> > > while other CPUs may be executing that code, Linux (temporarily)
> > > replaces the first byte of the original instruction with an int3 (opcode
> > > 0xcc), then patches in the new code stream except for the first byte,
> > > and finally replaces the int3 with the first byte of the new code
> > > stream.  If a CPU hits the int3, i.e. executes the code while it's being
> > > modified, then the guest kernel must look up the RIP to determine how to
> > > handle the #BP, e.g. by emulating the new instruction.  If the RIP is
> > > incorrect, then this lookup fails and the guest kernel panics.
> > > 
> > > The bug reproduces almost instantly by hacking the guest kernel to
> > > repeatedly check a static branch[1] while running a drgn script[2] on
> > > the host to constantly swap out the memory containing the guest's TSS.
> > > 
> > > [1]: https://gist.github.com/osandov/44d17c51c28c0ac998ea0334edf90b5a
> > > [2]: https://gist.github.com/osandov/10e45e45afa29b11e0c7209247afc00b
> > > 
> > > Fixes: 6ef88d6e36c2 ("KVM: SVM: Re-inject INT3/INTO instead of retrying the instruction")
> > > Cc: stable@vger.kernel.org
> > > Co-developed-by: Sean Christopherson <seanjc@google.com>
> > > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > > Signed-off-by: Omar Sandoval <osandov@fb.com>
> > > ---
> > > Changes from v2 (https://lore.kernel.org/all/6ab4e8e5c5d6ea95f568a1ff8044779137dce428.1761774582.git.osandov@fb.com/):
> > > 
> > > - Fixed EMULTYPE_SET_SOFT_INT_VECTOR -> EMULTYPE_GET_SOFT_INT_VECTOR
> > >   typo.
> > > - Added explicit u32 cast to EMULTYPE_SET_SOFT_INT_VECTOR to make it
> > >   clear that it won't overflow.
> > > - Rebased on Linus's tree as of c9cfc122f03711a5124b4aafab3211cf4d35a2ac.
> > > 
> > >  arch/x86/include/asm/kvm_host.h |  9 +++++++++
> > >  arch/x86/kvm/svm/svm.c          | 24 +++++++++++++-----------
> > >  arch/x86/kvm/x86.c              | 21 +++++++++++++++++++++
> > >  3 files changed, 43 insertions(+), 11 deletions(-)
> > 
> > Ping, does this need any more updates?
> 
> For the record, I had this applied and tested before the ping, I just hadn't sent
> the "thank you" yet.  :-D
> 
> commit 4da3768e1820cf15cced390242d8789aed34f54d
> Author:     Omar Sandoval <osandov@fb.com>
> AuthorDate: Tue Nov 4 09:55:26 2025 -0800
> Commit:     Sean Christopherson <seanjc@google.com>
> CommitDate: Thu Nov 13 13:03:19 2025 -0800  <======= See, I'm not lying!

No worries, and thanks! I checked
git://git.kernel.org/pub/scm/virt/kvm/kvm.git before pinging, but not
https://github.com/kvm-x86/linux. I appreciate the help with this!

Omar

Re: [PATCH v3] KVM: SVM: Don't skip unrelated instruction if INT3/INTO is replaced

[Sean Christopherson]

On Tue, 04 Nov 2025 09:55:26 -0800, Omar Sandoval wrote:
> When re-injecting a soft interrupt from an INT3, INT0, or (select) INTn
> instruction, discard the exception and retry the instruction if the code
> stream is changed (e.g. by a different vCPU) between when the CPU
> executes the instruction and when KVM decodes the instruction to get the
> next RIP.
> 
> As effectively predicted by commit 6ef88d6e36c2 ("KVM: SVM: Re-inject
> INT3/INTO instead of retrying the instruction"), failure to verify that
> the correct INTn instruction was decoded can effectively clobber guest
> state due to decoding the wrong instruction and thus specifying the
> wrong next RIP.
> 
> [...]

Applied to kvm-x86 svm, thanks!

[1/1] KVM: SVM: Don't skip unrelated instruction if INT3/INTO is replaced
      https://github.com/kvm-x86/linux/commit/4da3768e182

--
https://github.com/kvm-x86/linux/tree/next