[PATCH v3 00/10] KVM: x86: Improve #DB handling in the emulator

[PATCH v3 00/10] KVM: x86: Improve #DB handling in the emulator

[Sean Christopherson]

Hou's series of fixes and cleanups related to #DB handling in the emulator.

v3:
 - Fix the GUEST_PENDING_DBG_EXCEPTIONS.BS consistency check issue by
   stuffing the VMCS during injection, not during emulation.
 - Drop a prep that is no longer necessary.
 - Massage a few shortlogs/changelogs.
 - Fix the selftest that takes an IRQ in the STI-shadow after IRET.
 - Fix an intermediate bug where kvm_queue_exception_e() was being used
   instead of kvm_queue_exception_p().

v2:
 - https://lore.kernel.org/all/cover.1766066076.git.houwenlong.hwl@antgroup.com
 - cleanup in inject_emulated_exception().
 - rename 'set_pending_dbg' callback as 'refresh_pending_dbg_exceptions'.
 - fold refresh_pending_dbg_exceptions() call into
   kvm_vcpu_do_singlestep().
 - Split the change to move up kvm_set_rflags() into a single patch.
 - Move the #DB and IRQ handler registration after guest debug testcases.

v1: https://lore.kernel.org/all/cover.1757416809.git.houwenlong.hwl@antgroup.com

Hou Wenlong (7):
  KVM: x86: Capture "struct x86_exception" in
    inject_emulated_exception()
  KVM: x86: Set guest DR6 by kvm_queue_exception_p() in instruction
    emulation
  KVM: x86: Honor KVM_GUESTDBG_USE_HW_BP when emulating MOV DR (in
    emulator)
  KVM: x86: Honor KVM_GUESTDBG_USE_HW_BP when checking for code
    breakpoints in emulation
  KVM: x86: Move KVM_GUESTDBG_SINGLESTEP handling into
    kvm_inject_emulated_db()
  KVM: selftests: Verify guest debug DR7.GD checking during instruction
    emulation
  KVM: selftests: Verify VMX's GUEST_PENDING_DBG_EXCEPTIONS.BS
    Consistency Check

Sean Christopherson (3):
  KVM: VMX: Refresh GUEST_PENDING_DBG_EXCEPTIONS.BS on all injected #DBs
  KVM: x86: Drop kvm_vcpu_do_singlestep() now that it's been gutted
  KVM: selftests: Add all (known) EFLAGS bit definitions

 arch/x86/kvm/emulate.c                        |  14 +--
 arch/x86/kvm/kvm_emulate.h                    |   7 +-
 arch/x86/kvm/vmx/vmx.c                        |  35 +++---
 arch/x86/kvm/x86.c                            | 111 +++++++++---------
 .../selftests/kvm/include/x86/processor.h     |  19 ++-
 .../testing/selftests/kvm/lib/x86/processor.c |   2 +-
 tools/testing/selftests/kvm/lib/x86/vmx.c     |   2 +-
 tools/testing/selftests/kvm/x86/debug_regs.c  |  83 ++++++++++++-
 8 files changed, 184 insertions(+), 89 deletions(-)


base-commit: b7fbe9a1bf9ee6c967ef77d366ca58c35fcf1887
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 01/10] KVM: VMX: Refresh GUEST_PENDING_DBG_EXCEPTIONS.BS on all injected #DBs

[Sean Christopherson]

Move KVM's stuffing of GUEST_PENDING_DBG_EXCEPTIONS.BS when RFLAGS.TF=1 and
MOV/POP SS or STI blocking is active into the exception injection code so
that KVM fixes up the VMCS for all injected #DBs, not only those that are
reflected back into the guest after #DB interception.  E.g. if KVM queues
a #DB in the emulator, or more importantly if userspace does save/restore
exactly on the #DB+shadow boundary, then KVM needs to massage the VMCS to
avoid the VM-Entry consistency check.

Opportunistically update the wording of the comment to describe the
behavior as a workaround of flawed CPU behavior/architecture, to make it
clear that the *only* thing KVM is doing is fudging around a consistency
check.  Per the SDM:

  There are no pending debug exceptions after VM entry if any of the
  following are true:

    * The VM entry is vectoring with one of the following interruption
      types: external interrupt, non-maskable interrupt (NMI), hardware
      exception, or privileged software exception.

I.e. forcing GUEST_PENDING_DBG_EXCEPTIONS.BS does *not* impact guest-
visible behavior.

Fixes: b9bed78e2fa9 ("KVM: VMX: Set vmcs.PENDING_DBG.BS on #DB in STI/MOVSS blocking shadow")
Cc: stable@vger.kernel.org
Reported-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
Closes: https://lore.kernel.org/all/b1a294bc9ed4dae532474a5dc6c8cb6e5962de7c.1757416809.git.houwenlong.hwl@antgroup.com
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/vmx/vmx.c | 35 ++++++++++++++++++-----------------
 1 file changed, 18 insertions(+), 17 deletions(-)

diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 1701db1b2e18..a0a0ccf342d3 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -1909,6 +1909,24 @@ void vmx_inject_exception(struct kvm_vcpu *vcpu)
 	u32 intr_info = ex->vector | INTR_INFO_VALID_MASK;
 	struct vcpu_vmx *vmx = to_vmx(vcpu);
 
+	/*
+	 * When injecting a #DB, single-stepping is enabled in RFLAGS, and STI
+	 * or MOV-SS blocking is active, set vmcs.PENDING_DBG_EXCEPTIONS.BS to
+	 * prevent a false positive from VM-Entry consistency check.  VM-Entry
+	 * asserts that a single-step #DB _must_ be pending in this scenario,
+	 * as the previous instruction cannot have toggled RFLAGS.TF 0=>1
+	 * (because STI and POP/MOV don't modify RFLAGS), therefore the one
+	 * instruction delay when activating single-step breakpoints must have
+	 * already expired.  However, the CPU isn't smart enough to peek at
+	 * vmcs.VM_ENTRY_INTR_INFO_FIELD and so doesn't realize that yes, there
+	 * is indeed a #DB pending/imminent.
+	 */
+	if (ex->vector == DB_VECTOR &&
+	    (vmx_get_rflags(vcpu) & X86_EFLAGS_TF) &&
+	    vmx_get_interrupt_shadow(vcpu))
+		vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS,
+			    vmcs_readl(GUEST_PENDING_DBG_EXCEPTIONS) | DR6_BS);
+
 	kvm_deliver_exception_payload(vcpu, ex);
 
 	if (ex->has_error_code) {
@@ -5485,26 +5503,9 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu)
 			 * avoid single-step #DB and MTF updates, as ICEBP is
 			 * higher priority.  Note, skipping ICEBP still clears
 			 * STI and MOVSS blocking.
-			 *
-			 * For all other #DBs, set vmcs.PENDING_DBG_EXCEPTIONS.BS
-			 * if single-step is enabled in RFLAGS and STI or MOVSS
-			 * blocking is active, as the CPU doesn't set the bit
-			 * on VM-Exit due to #DB interception.  VM-Entry has a
-			 * consistency check that a single-step #DB is pending
-			 * in this scenario as the previous instruction cannot
-			 * have toggled RFLAGS.TF 0=>1 (because STI and POP/MOV
-			 * don't modify RFLAGS), therefore the one instruction
-			 * delay when activating single-step breakpoints must
-			 * have already expired.  Note, the CPU sets/clears BS
-			 * as appropriate for all other VM-Exits types.
 			 */
 			if (is_icebp(intr_info))
 				WARN_ON(!skip_emulated_instruction(vcpu));
-			else if ((vmx_get_rflags(vcpu) & X86_EFLAGS_TF) &&
-				 (vmcs_read32(GUEST_INTERRUPTIBILITY_INFO) &
-				  (GUEST_INTR_STATE_STI | GUEST_INTR_STATE_MOV_SS)))
-				vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS,
-					    vmcs_readl(GUEST_PENDING_DBG_EXCEPTIONS) | DR6_BS);
 
 			kvm_queue_exception_p(vcpu, DB_VECTOR, dr6);
 			return 1;
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 02/10] KVM: x86: Capture "struct x86_exception" in inject_emulated_exception()

[Sean Christopherson]

From: Hou Wenlong <houwenlong.hwl@antgroup.com>

As all callers in inject_emulated_exception() use "struct x86_exception"
directly, capture it locally instead of using the context.

No functional change intended.

Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/x86.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 48f259015ce4..8ddb878934ed 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8978,15 +8978,14 @@ static void toggle_interruptibility(struct kvm_vcpu *vcpu, u32 mask)
 
 static void inject_emulated_exception(struct kvm_vcpu *vcpu)
 {
-	struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt;
+	struct x86_exception *ex = &vcpu->arch.emulate_ctxt->exception;
 
-	if (ctxt->exception.vector == PF_VECTOR)
-		kvm_inject_emulated_page_fault(vcpu, &ctxt->exception);
-	else if (ctxt->exception.error_code_valid)
-		kvm_queue_exception_e(vcpu, ctxt->exception.vector,
-				      ctxt->exception.error_code);
+	if (ex->vector == PF_VECTOR)
+		kvm_inject_emulated_page_fault(vcpu, ex);
+	else if (ex->error_code_valid)
+		kvm_queue_exception_e(vcpu, ex->vector, ex->error_code);
 	else
-		kvm_queue_exception(vcpu, ctxt->exception.vector);
+		kvm_queue_exception(vcpu, ex->vector);
 }
 
 static struct x86_emulate_ctxt *alloc_emulate_ctxt(struct kvm_vcpu *vcpu)
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 03/10] KVM: x86: Set guest DR6 by kvm_queue_exception_p() in instruction emulation

[Sean Christopherson]

From: Hou Wenlong <houwenlong.hwl@antgroup.com>

Record DR6 in emulate_db() and use kvm_queue_exception_p() to set DR6
instead of directly using kvm_set_dr6() in emulation, which keeps the
handling of DR6 during #DB injection consistent with other code paths.

No functional change intended.

Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
[sean: fix e vs. p goof, add kvm_inject_emulated_db() right away]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/emulate.c     | 14 ++++----------
 arch/x86/kvm/kvm_emulate.h |  6 +++++-
 arch/x86/kvm/x86.c         | 10 +++++++++-
 3 files changed, 18 insertions(+), 12 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index c8c6cc0406d6..510244555a74 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -540,8 +540,9 @@ static int emulate_exception(struct x86_emulate_ctxt *ctxt, int vec,
 	return X86EMUL_PROPAGATE_FAULT;
 }
 
-static int emulate_db(struct x86_emulate_ctxt *ctxt)
+static int emulate_db(struct x86_emulate_ctxt *ctxt, unsigned long dr6)
 {
+	ctxt->exception.dr6 = dr6;
 	return emulate_exception(ctxt, DB_VECTOR, 0, false);
 }
 
@@ -3847,15 +3848,8 @@ static int check_dr_read(struct x86_emulate_ctxt *ctxt)
 	if ((cr4 & X86_CR4_DE) && (dr == 4 || dr == 5))
 		return emulate_ud(ctxt);
 
-	if (ctxt->ops->get_dr(ctxt, 7) & DR7_GD) {
-		ulong dr6;
-
-		dr6 = ctxt->ops->get_dr(ctxt, 6);
-		dr6 &= ~DR_TRAP_BITS;
-		dr6 |= DR6_BD | DR6_ACTIVE_LOW;
-		ctxt->ops->set_dr(ctxt, 6, dr6);
-		return emulate_db(ctxt);
-	}
+	if (ctxt->ops->get_dr(ctxt, 7) & DR7_GD)
+		return emulate_db(ctxt, DR6_BD);
 
 	return X86EMUL_CONTINUE;
 }
diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h
index 0abff36d0994..bb2a2aee0e13 100644
--- a/arch/x86/kvm/kvm_emulate.h
+++ b/arch/x86/kvm/kvm_emulate.h
@@ -24,7 +24,11 @@ struct x86_exception {
 	bool error_code_valid;
 	u16 error_code;
 	bool nested_page_fault;
-	u64 address; /* cr2 or nested page fault gpa */
+	union {
+		u64 address; /* cr2 or nested page fault gpa */
+		unsigned long dr6;
+		u64 payload;
+	};
 	u8 async_page_fault;
 	unsigned long exit_qualification;
 };
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8ddb878934ed..8a862d39302c 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8976,11 +8976,18 @@ static void toggle_interruptibility(struct kvm_vcpu *vcpu, u32 mask)
 	}
 }
 
+static void kvm_inject_emulated_db(struct kvm_vcpu *vcpu, unsigned long dr6)
+{
+	kvm_queue_exception_p(vcpu, DB_VECTOR, dr6);
+}
+
 static void inject_emulated_exception(struct kvm_vcpu *vcpu)
 {
 	struct x86_exception *ex = &vcpu->arch.emulate_ctxt->exception;
 
-	if (ex->vector == PF_VECTOR)
+	if (ex->vector == DB_VECTOR)
+		kvm_inject_emulated_db(vcpu, ex->dr6);
+	else if (ex->vector == PF_VECTOR)
 		kvm_inject_emulated_page_fault(vcpu, ex);
 	else if (ex->error_code_valid)
 		kvm_queue_exception_e(vcpu, ex->vector, ex->error_code);
@@ -9025,6 +9032,7 @@ static void init_emulate_ctxt(struct kvm_vcpu *vcpu)
 	ctxt->interruptibility = 0;
 	ctxt->have_exception = false;
 	ctxt->exception.vector = -1;
+	ctxt->exception.payload = 0;
 	ctxt->perm_ok = false;
 
 	init_decode_cache(ctxt);
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 04/10] KVM: x86: Honor KVM_GUESTDBG_USE_HW_BP when emulating MOV DR (in emulator)

[Sean Christopherson]

From: Hou Wenlong <houwenlong.hwl@antgroup.com>

When emulating a MOV DR instruction, honor KVM_GUESTDBG_USE_HW_BP when
checking DR7.GD, and if there is a general-detect #DB, route it to host
userspace as appropriate.  Consulting only the guest's actual DR7 causes
KVM to fail to report a DR access to userspace (assuming the guest itself
doesn't have DR7.GD=1).

Fixes: ae675ef01cd8 ("KVM: x86: Wire-up hardware breakpoints for guest debugging")
Suggested-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
[sean: only expose effective DR7 to emulator, massage changelog]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/emulate.c     |  2 +-
 arch/x86/kvm/kvm_emulate.h |  1 +
 arch/x86/kvm/x86.c         | 41 ++++++++++++++++++++++++++++++--------
 3 files changed, 35 insertions(+), 9 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 510244555a74..917a521c299f 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -3848,7 +3848,7 @@ static int check_dr_read(struct x86_emulate_ctxt *ctxt)
 	if ((cr4 & X86_CR4_DE) && (dr == 4 || dr == 5))
 		return emulate_ud(ctxt);
 
-	if (ctxt->ops->get_dr(ctxt, 7) & DR7_GD)
+	if (ctxt->ops->get_eff_dr7(ctxt) & DR7_GD)
 		return emulate_db(ctxt, DR6_BD);
 
 	return X86EMUL_CONTINUE;
diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h
index bb2a2aee0e13..33bfc9aa948e 100644
--- a/arch/x86/kvm/kvm_emulate.h
+++ b/arch/x86/kvm/kvm_emulate.h
@@ -215,6 +215,7 @@ struct x86_emulate_ops {
 	ulong (*get_cr)(struct x86_emulate_ctxt *ctxt, int cr);
 	int (*set_cr)(struct x86_emulate_ctxt *ctxt, int cr, ulong val);
 	int (*cpl)(struct x86_emulate_ctxt *ctxt);
+	ulong (*get_eff_dr7)(struct x86_emulate_ctxt *ctxt);
 	ulong (*get_dr)(struct x86_emulate_ctxt *ctxt, int dr);
 	int (*set_dr)(struct x86_emulate_ctxt *ctxt, int dr, ulong value);
 	int (*set_msr_with_filter)(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 data);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8a862d39302c..8b07bd2f8310 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -1601,6 +1601,14 @@ unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr)
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dr);
 
+static unsigned long kvm_get_eff_dr7(struct kvm_vcpu *vcpu)
+{
+	if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)
+		return vcpu->arch.guest_debug_dr7;
+
+	return vcpu->arch.dr7;
+}
+
 int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu)
 {
 	u32 pmc = kvm_rcx_read(vcpu);
@@ -8548,6 +8556,11 @@ static void emulator_wbinvd(struct x86_emulate_ctxt *ctxt)
 	kvm_emulate_wbinvd_noskip(emul_to_vcpu(ctxt));
 }
 
+static unsigned long emulator_get_eff_dr7(struct x86_emulate_ctxt *ctxt)
+{
+	return kvm_get_eff_dr7(emul_to_vcpu(ctxt));
+}
+
 static unsigned long emulator_get_dr(struct x86_emulate_ctxt *ctxt, int dr)
 {
 	return kvm_get_dr(emul_to_vcpu(ctxt), dr);
@@ -8930,6 +8943,7 @@ static const struct x86_emulate_ops emulate_ops = {
 	.get_cr              = emulator_get_cr,
 	.set_cr              = emulator_set_cr,
 	.cpl                 = emulator_get_cpl,
+	.get_eff_dr7         = emulator_get_eff_dr7,
 	.get_dr              = emulator_get_dr,
 	.set_dr              = emulator_set_dr,
 	.set_msr_with_filter = emulator_set_msr_with_filter,
@@ -8976,23 +8990,36 @@ static void toggle_interruptibility(struct kvm_vcpu *vcpu, u32 mask)
 	}
 }
 
-static void kvm_inject_emulated_db(struct kvm_vcpu *vcpu, unsigned long dr6)
+static int kvm_inject_emulated_db(struct kvm_vcpu *vcpu, unsigned long dr6)
 {
+	struct kvm_run *kvm_run = vcpu->run;
+
+	if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) {
+		kvm_run->debug.arch.dr6 = dr6 | DR6_ACTIVE_LOW;
+		kvm_run->debug.arch.pc = kvm_get_linear_rip(vcpu);
+		kvm_run->debug.arch.exception = DB_VECTOR;
+		kvm_run->exit_reason = KVM_EXIT_DEBUG;
+		return 0;
+	}
+
 	kvm_queue_exception_p(vcpu, DB_VECTOR, dr6);
+	return 1;
 }
 
-static void inject_emulated_exception(struct kvm_vcpu *vcpu)
+static int inject_emulated_exception(struct kvm_vcpu *vcpu)
 {
 	struct x86_exception *ex = &vcpu->arch.emulate_ctxt->exception;
 
 	if (ex->vector == DB_VECTOR)
-		kvm_inject_emulated_db(vcpu, ex->dr6);
-	else if (ex->vector == PF_VECTOR)
+		return kvm_inject_emulated_db(vcpu, ex->dr6);
+
+	if (ex->vector == PF_VECTOR)
 		kvm_inject_emulated_page_fault(vcpu, ex);
 	else if (ex->error_code_valid)
 		kvm_queue_exception_e(vcpu, ex->vector, ex->error_code);
 	else
 		kvm_queue_exception(vcpu, ex->vector);
+	return 1;
 }
 
 static struct x86_emulate_ctxt *alloc_emulate_ctxt(struct kvm_vcpu *vcpu)
@@ -9501,8 +9528,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 				 */
 				WARN_ON_ONCE(ctxt->exception.vector == UD_VECTOR ||
 					     exception_type(ctxt->exception.vector) == EXCPT_TRAP);
-				inject_emulated_exception(vcpu);
-				return 1;
+				return inject_emulated_exception(vcpu);
 			}
 			return handle_emulation_failure(vcpu, emulation_type);
 		}
@@ -9597,8 +9623,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 	if (ctxt->have_exception) {
 		WARN_ON_ONCE(vcpu->mmio_needed && !vcpu->mmio_is_write);
 		vcpu->mmio_needed = false;
-		r = 1;
-		inject_emulated_exception(vcpu);
+		r = inject_emulated_exception(vcpu);
 	} else if (vcpu->arch.pio.count) {
 		if (!vcpu->arch.pio.in) {
 			/* FIXME: return into emulator if single-stepping.  */
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 05/10] KVM: x86: Honor KVM_GUESTDBG_USE_HW_BP when checking for code breakpoints in emulation

[Sean Christopherson]

From: Hou Wenlong <houwenlong.hwl@antgroup.com>

When KVM_GUESTDBG_USE_HW_BP is enabled, i.e. userspace is usurping the
guest's hardware debug registers, the guest's effective breakpoints are
controlled by userspace rather than by the guest itself.  Honor the
KVM_GUESTDBG_USE_HW_BP behavior when handling code #DBs in the emulator so
that userspace (and the guest) gets consistent behavior for code #DBs
regardless of whether an instruction is executed natively or emulated by
KVM.

To aid in userspace debug, don't treat code breakpoints as inhibited if
KVM_GUESTDBG_USE_HW_BP is enabled as accurately emulating x86 architecture
is obviously a non-goal of guest-debug.

Fixes: 4a1e10d5b5d8 ("KVM: x86: handle hardware breakpoints during emulation")
Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
[sean: massage changelog]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/x86.c | 35 ++++++++++-------------------------
 1 file changed, 10 insertions(+), 25 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8b07bd2f8310..279e2734e088 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -9319,6 +9319,9 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_skip_emulated_instruction);
 
 static bool kvm_is_code_breakpoint_inhibited(struct kvm_vcpu *vcpu)
 {
+	if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)
+		return false;
+
 	if (kvm_get_rflags(vcpu) & X86_EFLAGS_RF)
 		return true;
 
@@ -9335,6 +9338,8 @@ static bool kvm_is_code_breakpoint_inhibited(struct kvm_vcpu *vcpu)
 static bool kvm_vcpu_check_code_breakpoint(struct kvm_vcpu *vcpu,
 					   int emulation_type, int *r)
 {
+	unsigned long dr7 = kvm_get_eff_dr7(vcpu);
+
 	WARN_ON_ONCE(emulation_type & EMULTYPE_NO_DECODE);
 
 	/*
@@ -9355,34 +9360,14 @@ static bool kvm_vcpu_check_code_breakpoint(struct kvm_vcpu *vcpu,
 			      EMULTYPE_TRAP_UD | EMULTYPE_VMWARE_GP | EMULTYPE_PF))
 		return false;
 
-	if (unlikely(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) &&
-	    (vcpu->arch.guest_debug_dr7 & DR7_BP_EN_MASK)) {
-		struct kvm_run *kvm_run = vcpu->run;
-		unsigned long eip = kvm_get_linear_rip(vcpu);
-		u32 dr6 = kvm_vcpu_check_hw_bp(eip, 0,
-					   vcpu->arch.guest_debug_dr7,
-					   vcpu->arch.eff_db);
-
-		if (dr6 != 0) {
-			kvm_run->debug.arch.dr6 = dr6 | DR6_ACTIVE_LOW;
-			kvm_run->debug.arch.pc = eip;
-			kvm_run->debug.arch.exception = DB_VECTOR;
-			kvm_run->exit_reason = KVM_EXIT_DEBUG;
-			*r = 0;
-			return true;
-		}
-	}
-
-	if (unlikely(vcpu->arch.dr7 & DR7_BP_EN_MASK) &&
+	if (unlikely(dr7 & DR7_BP_EN_MASK) &&
 	    !kvm_is_code_breakpoint_inhibited(vcpu)) {
 		unsigned long eip = kvm_get_linear_rip(vcpu);
-		u32 dr6 = kvm_vcpu_check_hw_bp(eip, 0,
-					   vcpu->arch.dr7,
-					   vcpu->arch.db);
+		u32 dr6 = kvm_vcpu_check_hw_bp(eip, 0, dr7,
+					       vcpu->arch.eff_db);
 
-		if (dr6 != 0) {
-			kvm_queue_exception_p(vcpu, DB_VECTOR, dr6);
-			*r = 1;
+		if (dr6) {
+			*r = kvm_inject_emulated_db(vcpu, dr6);
 			return true;
 		}
 	}
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 06/10] KVM: x86: Move KVM_GUESTDBG_SINGLESTEP handling into kvm_inject_emulated_db()

[Sean Christopherson]

From: Hou Wenlong <houwenlong.hwl@antgroup.com>

Move KVM_GUESTDBG_SINGLESTEP handling from kvm_vcpu_do_singlestep() into
kvm_inject_emulated_db() to dedup the USE_HW_BP vs. SINGLESTEP logic, and
to allow for removing kvm_vcpu_do_singlestep() entirely.

No functional change intended.

Suggested-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
[sean: massage changelog]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/x86.c | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 279e2734e088..ca30a8987f2f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8994,7 +8994,7 @@ static int kvm_inject_emulated_db(struct kvm_vcpu *vcpu, unsigned long dr6)
 {
 	struct kvm_run *kvm_run = vcpu->run;
 
-	if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) {
+	if (vcpu->guest_debug & (KVM_GUESTDBG_USE_HW_BP | KVM_GUESTDBG_SINGLESTEP)) {
 		kvm_run->debug.arch.dr6 = dr6 | DR6_ACTIVE_LOW;
 		kvm_run->debug.arch.pc = kvm_get_linear_rip(vcpu);
 		kvm_run->debug.arch.exception = DB_VECTOR;
@@ -9279,17 +9279,7 @@ static int kvm_vcpu_check_hw_bp(unsigned long addr, u32 type, u32 dr7,
 
 static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu)
 {
-	struct kvm_run *kvm_run = vcpu->run;
-
-	if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) {
-		kvm_run->debug.arch.dr6 = DR6_BS | DR6_ACTIVE_LOW;
-		kvm_run->debug.arch.pc = kvm_get_linear_rip(vcpu);
-		kvm_run->debug.arch.exception = DB_VECTOR;
-		kvm_run->exit_reason = KVM_EXIT_DEBUG;
-		return 0;
-	}
-	kvm_queue_exception_p(vcpu, DB_VECTOR, DR6_BS);
-	return 1;
+	return kvm_inject_emulated_db(vcpu, DR6_BS);
 }
 
 int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu)
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 07/10] KVM: x86: Drop kvm_vcpu_do_singlestep() now that it's been gutted

[Sean Christopherson]

Now that all of kvm_vcpu_do_singlestep()'s previously-unique functionality
has been moved into kvm_inject_emulated_db(), drop the one-line wrapper.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/x86.c | 10 ++--------
 1 file changed, 2 insertions(+), 8 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index ca30a8987f2f..758b99b2fa7f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -133,7 +133,6 @@ static void process_nmi(struct kvm_vcpu *vcpu);
 static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags);
 static void store_regs(struct kvm_vcpu *vcpu);
 static int sync_regs(struct kvm_vcpu *vcpu);
-static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu);
 
 static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);
 static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2);
@@ -9277,11 +9276,6 @@ static int kvm_vcpu_check_hw_bp(unsigned long addr, u32 type, u32 dr7,
 	return dr6;
 }
 
-static int kvm_vcpu_do_singlestep(struct kvm_vcpu *vcpu)
-{
-	return kvm_inject_emulated_db(vcpu, DR6_BS);
-}
-
 int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu)
 {
 	unsigned long rflags = kvm_x86_call(get_rflags)(vcpu);
@@ -9302,7 +9296,7 @@ int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu)
 	 * that sets the TF flag".
 	 */
 	if (unlikely(rflags & X86_EFLAGS_TF))
-		r = kvm_vcpu_do_singlestep(vcpu);
+		r = kvm_inject_emulated_db(vcpu, DR6_BS);
 	return r;
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_skip_emulated_instruction);
@@ -9641,7 +9635,7 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
 				kvm_pmu_branch_retired(vcpu);
 			kvm_rip_write(vcpu, ctxt->eip);
 			if (r && (ctxt->tf || (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)))
-				r = kvm_vcpu_do_singlestep(vcpu);
+				r = kvm_inject_emulated_db(vcpu, DR6_BS);
 			kvm_x86_call(update_emulated_instruction)(vcpu);
 			__kvm_set_rflags(vcpu, ctxt->eflags);
 		}
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 08/10] KVM: selftests: Add all (known) EFLAGS bit definitions

[Sean Christopherson]

Add #defines for all known EFLAGS bit, e.g. so that tests can use things
like EFLAGS.TF to validate single-stepping behavior.  Opportunistically
use X86_EFLAGS_FIXED instead of an open-coded equivalent when stuffing
initial vCPU state.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 .../selftests/kvm/include/x86/processor.h     | 19 ++++++++++++++++++-
 .../testing/selftests/kvm/lib/x86/processor.c |  2 +-
 tools/testing/selftests/kvm/lib/x86/vmx.c     |  2 +-
 3 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index 77f576ee7789..851ffcd3340c 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -38,7 +38,24 @@ extern u64 guest_tsc_khz;
 
 const char *ex_str(int vector);
 
-#define X86_EFLAGS_FIXED	 (1u << 1)
+#define X86_EFLAGS_CF		BIT(0)  /* Carry Flag */
+#define X86_EFLAGS_FIXED	BIT(1)  /* Bit 1 - always on */
+#define X86_EFLAGS_PF		BIT(2)  /* Parity Flag */
+#define X86_EFLAGS_AF		BIT(4)  /* Auxiliary carry Flag */
+#define X86_EFLAGS_ZF		BIT(6)  /* Zero Flag */
+#define X86_EFLAGS_SF		BIT(7)  /* Sign Flag */
+#define X86_EFLAGS_TF		BIT(8)  /* Trap Flag */
+#define X86_EFLAGS_IF		BIT(9)  /* Interrupt Flag */
+#define X86_EFLAGS_DF		BIT(10) /* Direction Flag */
+#define X86_EFLAGS_OF		BIT(11) /* Overflow Flag */
+#define X86_EFLAGS_IOPL		BIT(12) /* I/O Privilege Level (2 bits) */
+#define X86_EFLAGS_NT		BIT(14) /* Nested Task */
+#define X86_EFLAGS_RF		BIT(16) /* Resume Flag */
+#define X86_EFLAGS_VM		BIT(17) /* Virtual Mode */
+#define X86_EFLAGS_AC		BIT(18) /* Alignment Check/Access Control */
+#define X86_EFLAGS_VIF		BIT(19) /* Virtual Interrupt Flag */
+#define X86_EFLAGS_VIP		BIT(20) /* Virtual Interrupt Pending */
+#define X86_EFLAGS_ID		BIT(21) /* CPUID detection */
 
 #define X86_CR4_VME		(1ul << 0)
 #define X86_CR4_PVI		(1ul << 1)
diff --git a/tools/testing/selftests/kvm/lib/x86/processor.c b/tools/testing/selftests/kvm/lib/x86/processor.c
index b51467d70f6e..4ca48de7a926 100644
--- a/tools/testing/selftests/kvm/lib/x86/processor.c
+++ b/tools/testing/selftests/kvm/lib/x86/processor.c
@@ -848,7 +848,7 @@ struct kvm_vcpu *vm_arch_vcpu_add(struct kvm_vm *vm, u32 vcpu_id)
 
 	/* Setup guest general purpose registers */
 	vcpu_regs_get(vcpu, &regs);
-	regs.rflags = regs.rflags | 0x2;
+	regs.rflags = regs.rflags | X86_EFLAGS_FIXED;
 	regs.rsp = stack_gva;
 	vcpu_regs_set(vcpu, &regs);
 
diff --git a/tools/testing/selftests/kvm/lib/x86/vmx.c b/tools/testing/selftests/kvm/lib/x86/vmx.c
index 67642759e4a0..7c10ba6e6fb4 100644
--- a/tools/testing/selftests/kvm/lib/x86/vmx.c
+++ b/tools/testing/selftests/kvm/lib/x86/vmx.c
@@ -360,7 +360,7 @@ static inline void init_vmcs_guest_state(void *rip, void *rsp)
 	vmwrite(GUEST_DR7, 0x400);
 	vmwrite(GUEST_RSP, (u64)rsp);
 	vmwrite(GUEST_RIP, (u64)rip);
-	vmwrite(GUEST_RFLAGS, 2);
+	vmwrite(GUEST_RFLAGS, X86_EFLAGS_FIXED);
 	vmwrite(GUEST_PENDING_DBG_EXCEPTIONS, 0);
 	vmwrite(GUEST_SYSENTER_ESP, vmreadz(HOST_IA32_SYSENTER_ESP));
 	vmwrite(GUEST_SYSENTER_EIP, vmreadz(HOST_IA32_SYSENTER_EIP));
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 09/10] KVM: selftests: Verify guest debug DR7.GD checking during instruction emulation

[Sean Christopherson]

From: Hou Wenlong <houwenlong.hwl@antgroup.com>

Similar to the global disable test case in x86's debug_regs test, use
'KVM_FEP' to trigger instruction emulation in order to verify the guest
debug DR7.GD checking during instruction emulation.

Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 tools/testing/selftests/kvm/x86/debug_regs.c | 23 +++++++++++++++++++-
 1 file changed, 22 insertions(+), 1 deletion(-)

diff --git a/tools/testing/selftests/kvm/x86/debug_regs.c b/tools/testing/selftests/kvm/x86/debug_regs.c
index 0dfaf03cd0a0..ee9d0f3a5807 100644
--- a/tools/testing/selftests/kvm/x86/debug_regs.c
+++ b/tools/testing/selftests/kvm/x86/debug_regs.c
@@ -19,6 +19,7 @@
 u32 guest_value;
 
 extern unsigned char sw_bp, hw_bp, write_data, ss_start, bd_start;
+extern unsigned char fep_bd_start;
 
 static void guest_code(void)
 {
@@ -64,6 +65,10 @@ static void guest_code(void)
 
 	/* DR6.BD test */
 	asm volatile("bd_start: mov %%dr0, %%rax" : : : "rax");
+
+	if (is_forced_emulation_enabled)
+		asm volatile(KVM_FEP "fep_bd_start: mov %%dr0, %%rax" : : : "rax");
+
 	GUEST_DONE();
 }
 
@@ -185,7 +190,7 @@ int main(void)
 			    target_dr6);
 	}
 
-	/* Finally test global disable */
+	/* test global disable */
 	memset(&debug, 0, sizeof(debug));
 	debug.control = KVM_GUESTDBG_ENABLE | KVM_GUESTDBG_USE_HW_BP;
 	debug.arch.debugreg[7] = 0x400 | DR7_GD;
@@ -202,6 +207,22 @@ int main(void)
 			    run->debug.arch.pc, target_rip, run->debug.arch.dr6,
 			    target_dr6);
 
+	/* test global disable in emulation */
+	if (is_forced_emulation_enabled) {
+		/* Skip the 3-bytes "mov dr0" */
+		vcpu_skip_insn(vcpu, 3);
+		vcpu_run(vcpu);
+		TEST_ASSERT(run->exit_reason == KVM_EXIT_DEBUG &&
+			    run->debug.arch.exception == DB_VECTOR &&
+			    run->debug.arch.pc == CAST_TO_RIP(fep_bd_start) &&
+			    run->debug.arch.dr6 == target_dr6,
+			    "DR7.GD: exit %d exception %d rip 0x%llx "
+			    "(should be 0x%llx) dr6 0x%llx (should be 0x%llx)",
+			    run->exit_reason, run->debug.arch.exception,
+			    run->debug.arch.pc, target_rip, run->debug.arch.dr6,
+			    target_dr6);
+	}
+
 	/* Disable all debug controls, run to the end */
 	memset(&debug, 0, sizeof(debug));
 	vcpu_guest_debug_set(vcpu, &debug);
-- 
2.54.0.563.g4f69b47b94-goog

[PATCH v3 10/10] KVM: selftests: Verify VMX's GUEST_PENDING_DBG_EXCEPTIONS.BS Consistency Check

[Sean Christopherson]

From: Hou Wenlong <houwenlong.hwl@antgroup.com>

In x86's debug_regs test, add a test case to cover the scenario where a
single-step #DB occurs in an STI-shadow, in which case KVM needs to stuff
vmcs.GUEST_PENDING_DBG_EXCEPTIONS.BS in order to satisfy a flawed VM-Entry
Consistency Check.

Wire up an IRQ handler to gain a bit of bonus coverage, as the subsequent
IRET from the #DB sets RFLAGS.IF, but *without* STI-blocking, and so the
pending IRQ is expected on the instruction immediately following STI.

Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
[sean: expect the IRQ on the CLI, and explain why]
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 tools/testing/selftests/kvm/x86/debug_regs.c | 64 ++++++++++++++++++--
 1 file changed, 60 insertions(+), 4 deletions(-)

diff --git a/tools/testing/selftests/kvm/x86/debug_regs.c b/tools/testing/selftests/kvm/x86/debug_regs.c
index ee9d0f3a5807..6299e921dc27 100644
--- a/tools/testing/selftests/kvm/x86/debug_regs.c
+++ b/tools/testing/selftests/kvm/x86/debug_regs.c
@@ -15,11 +15,46 @@
 
 #define IRQ_VECTOR 0xAA
 
+#define  CAST_TO_RIP(v)  ((unsigned long long)&(v))
+
 /* For testing data access debug BP */
 u32 guest_value;
 
 extern unsigned char sw_bp, hw_bp, write_data, ss_start, bd_start;
-extern unsigned char fep_bd_start;
+extern unsigned char fep_bd_start, fep_sti_start, fep_sti_end;
+
+static void guest_db_handler(struct ex_regs *regs)
+{
+	static int count;
+	unsigned long target_rips[2] = {
+		CAST_TO_RIP(fep_sti_start),
+		CAST_TO_RIP(fep_sti_end),
+	};
+
+	__GUEST_ASSERT(regs->rip == target_rips[count],
+	               "STI[%u]: unexpected rip 0x%lx (should be 0x%lx)",
+		       count, regs->rip, target_rips[count]);
+	regs->rflags &= ~X86_EFLAGS_TF;
+	count++;
+}
+
+static void guest_irq_handler(struct ex_regs *regs)
+{
+	/*
+	 * The pending IRQ should finally be take when KVM_GUESTDBG_BLOCKIRQ is
+	 * cleared and IRQs are enabled.  Note, the IRQ is expected to arrive
+	 * on the instruction immediately after STI, even though its in an STI
+	 * shadow.  Because the next instruction has a coincident #DB, and #DBs
+	 * are not subject to STI-blocking, the #DB will push RFLAGS.IF=1 on
+	 * the stack, and the eventual IRET will unmask IRQs and obliterate the
+	 * STI shadow in the process.
+	 */
+	unsigned long target_rip = CAST_TO_RIP(fep_sti_start);
+
+	__GUEST_ASSERT(regs->rip == target_rip,
+		       "IRQ: unexpected rip 0x%lx (should be 0x%lx)",
+		       regs->rip, target_rip);
+}
 
 static void guest_code(void)
 {
@@ -66,14 +101,32 @@ static void guest_code(void)
 	/* DR6.BD test */
 	asm volatile("bd_start: mov %%dr0, %%rax" : : : "rax");
 
-	if (is_forced_emulation_enabled)
+	/*
+	 * Note, the IRET from the #DB that occurs in the below STI-shadow will
+	 * unmask IRQs, i.e. the pending interrupt will be delivered after #DB
+	 * handling, on the CLI!
+	 */
+	if (is_forced_emulation_enabled) {
 		asm volatile(KVM_FEP "fep_bd_start: mov %%dr0, %%rax" : : : "rax");
 
+		/* pending debug exceptions for emulation */
+		asm volatile("pushf\n\t"
+			     "orq $" __stringify(X86_EFLAGS_TF) ", (%rsp)\n\t"
+			     "popf\n\t"
+			     "sti\n\t"
+			     "fep_sti_start:"
+			     "cli\n\t"
+			     "pushf\n\t"
+			     "orq $" __stringify(X86_EFLAGS_TF) ", (%rsp)\n\t"
+			     "popf\n\t"
+			     KVM_FEP "sti\n\t"
+			     "fep_sti_end:"
+			     "cli\n\t");
+	}
+
 	GUEST_DONE();
 }
 
-#define  CAST_TO_RIP(v)  ((unsigned long long)&(v))
-
 static void vcpu_skip_insn(struct kvm_vcpu *vcpu, int insn_len)
 {
 	struct kvm_regs regs;
@@ -227,6 +280,9 @@ int main(void)
 	memset(&debug, 0, sizeof(debug));
 	vcpu_guest_debug_set(vcpu, &debug);
 
+	vm_install_exception_handler(vm, DB_VECTOR, guest_db_handler);
+	vm_install_exception_handler(vm, IRQ_VECTOR, guest_irq_handler);
+
 	vcpu_run(vcpu);
 	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
 	cmd = get_ucall(vcpu, &uc);
-- 
2.54.0.563.g4f69b47b94-goog

Re: [PATCH v3 01/10] KVM: VMX: Refresh GUEST_PENDING_DBG_EXCEPTIONS.BS on all injected #DBs

[Hou Wenlong]

On Fri, May 15, 2026 at 03:26:29PM -0700, Sean Christopherson wrote:
> Move KVM's stuffing of GUEST_PENDING_DBG_EXCEPTIONS.BS when RFLAGS.TF=1 and
> MOV/POP SS or STI blocking is active into the exception injection code so
> that KVM fixes up the VMCS for all injected #DBs, not only those that are
> reflected back into the guest after #DB interception.  E.g. if KVM queues
> a #DB in the emulator, or more importantly if userspace does save/restore
> exactly on the #DB+shadow boundary, then KVM needs to massage the VMCS to
> avoid the VM-Entry consistency check.
> 
> Opportunistically update the wording of the comment to describe the
> behavior as a workaround of flawed CPU behavior/architecture, to make it
> clear that the *only* thing KVM is doing is fudging around a consistency
> check.  Per the SDM:
> 
>   There are no pending debug exceptions after VM entry if any of the
>   following are true:
> 
>     * The VM entry is vectoring with one of the following interruption
>       types: external interrupt, non-maskable interrupt (NMI), hardware
>       exception, or privileged software exception.
> 
> I.e. forcing GUEST_PENDING_DBG_EXCEPTIONS.BS does *not* impact guest-
> visible behavior.
> 
> Fixes: b9bed78e2fa9 ("KVM: VMX: Set vmcs.PENDING_DBG.BS on #DB in STI/MOVSS blocking shadow")
> Cc: stable@vger.kernel.org
> Reported-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
> Closes: https://lore.kernel.org/all/b1a294bc9ed4dae532474a5dc6c8cb6e5962de7c.1757416809.git.houwenlong.hwl@antgroup.com
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/kvm/vmx/vmx.c | 35 ++++++++++++++++++-----------------
>  1 file changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index 1701db1b2e18..a0a0ccf342d3 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -1909,6 +1909,24 @@ void vmx_inject_exception(struct kvm_vcpu *vcpu)
>  	u32 intr_info = ex->vector | INTR_INFO_VALID_MASK;
>  	struct vcpu_vmx *vmx = to_vmx(vcpu);
>  
> +	/*
> +	 * When injecting a #DB, single-stepping is enabled in RFLAGS, and STI
> +	 * or MOV-SS blocking is active, set vmcs.PENDING_DBG_EXCEPTIONS.BS to
> +	 * prevent a false positive from VM-Entry consistency check.  VM-Entry
> +	 * asserts that a single-step #DB _must_ be pending in this scenario,
> +	 * as the previous instruction cannot have toggled RFLAGS.TF 0=>1
> +	 * (because STI and POP/MOV don't modify RFLAGS), therefore the one
> +	 * instruction delay when activating single-step breakpoints must have
> +	 * already expired.  However, the CPU isn't smart enough to peek at
> +	 * vmcs.VM_ENTRY_INTR_INFO_FIELD and so doesn't realize that yes, there
> +	 * is indeed a #DB pending/imminent.
> +	 */
> +	if (ex->vector == DB_VECTOR &&
> +	    (vmx_get_rflags(vcpu) & X86_EFLAGS_TF) &&
> +	    vmx_get_interrupt_shadow(vcpu))
> +		vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS,
> +			    vmcs_readl(GUEST_PENDING_DBG_EXCEPTIONS) | DR6_BS);
> +

This matches a version of the modification I had considered earlier.

Reviewed-by: Hou Wenlong <houwenlong.hwl@antgroup.com>

Thanks!

>  	kvm_deliver_exception_payload(vcpu, ex);
>  
>  	if (ex->has_error_code) {
> @@ -5485,26 +5503,9 @@ static int handle_exception_nmi(struct kvm_vcpu *vcpu)
>  			 * avoid single-step #DB and MTF updates, as ICEBP is
>  			 * higher priority.  Note, skipping ICEBP still clears
>  			 * STI and MOVSS blocking.
> -			 *
> -			 * For all other #DBs, set vmcs.PENDING_DBG_EXCEPTIONS.BS
> -			 * if single-step is enabled in RFLAGS and STI or MOVSS
> -			 * blocking is active, as the CPU doesn't set the bit
> -			 * on VM-Exit due to #DB interception.  VM-Entry has a
> -			 * consistency check that a single-step #DB is pending
> -			 * in this scenario as the previous instruction cannot
> -			 * have toggled RFLAGS.TF 0=>1 (because STI and POP/MOV
> -			 * don't modify RFLAGS), therefore the one instruction
> -			 * delay when activating single-step breakpoints must
> -			 * have already expired.  Note, the CPU sets/clears BS
> -			 * as appropriate for all other VM-Exits types.
>  			 */
>  			if (is_icebp(intr_info))
>  				WARN_ON(!skip_emulated_instruction(vcpu));
> -			else if ((vmx_get_rflags(vcpu) & X86_EFLAGS_TF) &&
> -				 (vmcs_read32(GUEST_INTERRUPTIBILITY_INFO) &
> -				  (GUEST_INTR_STATE_STI | GUEST_INTR_STATE_MOV_SS)))
> -				vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS,
> -					    vmcs_readl(GUEST_PENDING_DBG_EXCEPTIONS) | DR6_BS);
>  
>  			kvm_queue_exception_p(vcpu, DB_VECTOR, dr6);
>  			return 1;
> -- 
> 2.54.0.563.g4f69b47b94-goog

Re: [PATCH v3 02/10] KVM: x86: Capture "struct x86_exception" in inject_emulated_exception()

[Yosry Ahmed]

On Fri, May 15, 2026 at 03:26:30PM -0700, Sean Christopherson wrote:
> From: Hou Wenlong <houwenlong.hwl@antgroup.com>
> 
> As all callers in inject_emulated_exception() use "struct x86_exception"
> directly, capture it locally instead of using the context.
> 
> No functional change intended.
> 
> Suggested-by: Sean Christopherson <seanjc@google.com>
> Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry@kernel.org>

Re: [PATCH v3 03/10] KVM: x86: Set guest DR6 by kvm_queue_exception_p() in instruction emulation

[Yosry Ahmed]

On Fri, May 15, 2026 at 03:26:31PM -0700, Sean Christopherson wrote:
> From: Hou Wenlong <houwenlong.hwl@antgroup.com>
> 
> Record DR6 in emulate_db() and use kvm_queue_exception_p() to set DR6
> instead of directly using kvm_set_dr6() in emulation, which keeps the
> handling of DR6 during #DB injection consistent with other code paths.
> 
> No functional change intended.
> 
> Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
> [sean: fix e vs. p goof, add kvm_inject_emulated_db() right away]
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/kvm/emulate.c     | 14 ++++----------
>  arch/x86/kvm/kvm_emulate.h |  6 +++++-
>  arch/x86/kvm/x86.c         | 10 +++++++++-
>  3 files changed, 18 insertions(+), 12 deletions(-)
> 
> diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
> index c8c6cc0406d6..510244555a74 100644
> --- a/arch/x86/kvm/emulate.c
> +++ b/arch/x86/kvm/emulate.c
> @@ -540,8 +540,9 @@ static int emulate_exception(struct x86_emulate_ctxt *ctxt, int vec,
>  	return X86EMUL_PROPAGATE_FAULT;
>  }
>  
> -static int emulate_db(struct x86_emulate_ctxt *ctxt)
> +static int emulate_db(struct x86_emulate_ctxt *ctxt, unsigned long dr6)
>  {
> +	ctxt->exception.dr6 = dr6;
>  	return emulate_exception(ctxt, DB_VECTOR, 0, false);
>  }
>  
> @@ -3847,15 +3848,8 @@ static int check_dr_read(struct x86_emulate_ctxt *ctxt)
>  	if ((cr4 & X86_CR4_DE) && (dr == 4 || dr == 5))
>  		return emulate_ud(ctxt);
>  
> -	if (ctxt->ops->get_dr(ctxt, 7) & DR7_GD) {
> -		ulong dr6;
> -
> -		dr6 = ctxt->ops->get_dr(ctxt, 6);
> -		dr6 &= ~DR_TRAP_BITS;
> -		dr6 |= DR6_BD | DR6_ACTIVE_LOW;
> -		ctxt->ops->set_dr(ctxt, 6, dr6);
> -		return emulate_db(ctxt);
> -	}
> +	if (ctxt->ops->get_dr(ctxt, 7) & DR7_GD)
> +		return emulate_db(ctxt, DR6_BD);

For other readers of this patch, the manipulations done to DR6 here are
no longer needed because kvm_deliver_exception_payload() will apply them
when delivering the exception.

Not including that in the changelog and leaving it as an exercise to the
reader to chase it down is just mean :P

Anyway, it looks correct:
Reviewed-by: Yosry Ahmed <yosry@kernel.org>

Re: [PATCH v3 04/10] KVM: x86: Honor KVM_GUESTDBG_USE_HW_BP when emulating MOV DR (in emulator)

[Yosry Ahmed]

On Fri, May 15, 2026 at 03:26:32PM -0700, Sean Christopherson wrote:
> From: Hou Wenlong <houwenlong.hwl@antgroup.com>
> 
> When emulating a MOV DR instruction, honor KVM_GUESTDBG_USE_HW_BP when
> checking DR7.GD, and if there is a general-detect #DB, route it to host
> userspace as appropriate.  Consulting only the guest's actual DR7 causes
> KVM to fail to report a DR access to userspace (assuming the guest itself
> doesn't have DR7.GD=1).
> 
> Fixes: ae675ef01cd8 ("KVM: x86: Wire-up hardware breakpoints for guest debugging")
> Suggested-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
> Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
> [sean: only expose effective DR7 to emulator, massage changelog]
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/kvm/emulate.c     |  2 +-
>  arch/x86/kvm/kvm_emulate.h |  1 +
>  arch/x86/kvm/x86.c         | 41 ++++++++++++++++++++++++++++++--------
>  3 files changed, 35 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
> index 510244555a74..917a521c299f 100644
> --- a/arch/x86/kvm/emulate.c
> +++ b/arch/x86/kvm/emulate.c
> @@ -3848,7 +3848,7 @@ static int check_dr_read(struct x86_emulate_ctxt *ctxt)
>  	if ((cr4 & X86_CR4_DE) && (dr == 4 || dr == 5))
>  		return emulate_ud(ctxt);
>  
> -	if (ctxt->ops->get_dr(ctxt, 7) & DR7_GD)
> +	if (ctxt->ops->get_eff_dr7(ctxt) & DR7_GD)
>  		return emulate_db(ctxt, DR6_BD);
>  
>  	return X86EMUL_CONTINUE;
> diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h
> index bb2a2aee0e13..33bfc9aa948e 100644
> --- a/arch/x86/kvm/kvm_emulate.h
> +++ b/arch/x86/kvm/kvm_emulate.h
> @@ -215,6 +215,7 @@ struct x86_emulate_ops {
>  	ulong (*get_cr)(struct x86_emulate_ctxt *ctxt, int cr);
>  	int (*set_cr)(struct x86_emulate_ctxt *ctxt, int cr, ulong val);
>  	int (*cpl)(struct x86_emulate_ctxt *ctxt);
> +	ulong (*get_eff_dr7)(struct x86_emulate_ctxt *ctxt);

I would name this get_effective_dr7, and the callback
kvm_get_effective_dr7(). I don't think saving a few characters here
offsets the readability losses.

Re: [PATCH v3 06/10] KVM: x86: Move KVM_GUESTDBG_SINGLESTEP handling into kvm_inject_emulated_db()

[Yosry Ahmed]

On Fri, May 15, 2026 at 03:26:34PM -0700, Sean Christopherson wrote:
> From: Hou Wenlong <houwenlong.hwl@antgroup.com>
> 
> Move KVM_GUESTDBG_SINGLESTEP handling from kvm_vcpu_do_singlestep() into
> kvm_inject_emulated_db() to dedup the USE_HW_BP vs. SINGLESTEP logic, and
> to allow for removing kvm_vcpu_do_singlestep() entirely.
> 
> No functional change intended.
> 
> Suggested-by: Lai Jiangshan <jiangshan.ljs@antgroup.com>
> Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
> [sean: massage changelog]
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry@kernel.org>

Re: [PATCH v3 07/10] KVM: x86: Drop kvm_vcpu_do_singlestep() now that it's been gutted

[Yosry Ahmed]

On Fri, May 15, 2026 at 03:26:35PM -0700, Sean Christopherson wrote:
> Now that all of kvm_vcpu_do_singlestep()'s previously-unique functionality
> has been moved into kvm_inject_emulated_db(), drop the one-line wrapper.
> 
> No functional change intended.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry@kernel.org>

Re: [PATCH v3 01/10] KVM: VMX: Refresh GUEST_PENDING_DBG_EXCEPTIONS.BS on all injected #DBs

[Sean Christopherson]

On Fri, May 15, 2026, Sean Christopherson wrote:
> Move KVM's stuffing of GUEST_PENDING_DBG_EXCEPTIONS.BS when RFLAGS.TF=1 and
> MOV/POP SS or STI blocking is active into the exception injection code so
> that KVM fixes up the VMCS for all injected #DBs, not only those that are
> reflected back into the guest after #DB interception.  E.g. if KVM queues
> a #DB in the emulator, or more importantly if userspace does save/restore
> exactly on the #DB+shadow boundary, then KVM needs to massage the VMCS to
> avoid the VM-Entry consistency check.
> 
> Opportunistically update the wording of the comment to describe the
> behavior as a workaround of flawed CPU behavior/architecture, to make it
> clear that the *only* thing KVM is doing is fudging around a consistency
> check.  Per the SDM:
> 
>   There are no pending debug exceptions after VM entry if any of the
>   following are true:
> 
>     * The VM entry is vectoring with one of the following interruption
>       types: external interrupt, non-maskable interrupt (NMI), hardware
>       exception, or privileged software exception.
> 
> I.e. forcing GUEST_PENDING_DBG_EXCEPTIONS.BS does *not* impact guest-
> visible behavior.
> 
> Fixes: b9bed78e2fa9 ("KVM: VMX: Set vmcs.PENDING_DBG.BS on #DB in STI/MOVSS blocking shadow")
> Cc: stable@vger.kernel.org
> Reported-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
> Closes: https://lore.kernel.org/all/b1a294bc9ed4dae532474a5dc6c8cb6e5962de7c.1757416809.git.houwenlong.hwl@antgroup.com
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/kvm/vmx/vmx.c | 35 ++++++++++++++++++-----------------
>  1 file changed, 18 insertions(+), 17 deletions(-)
> 
> diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> index 1701db1b2e18..a0a0ccf342d3 100644
> --- a/arch/x86/kvm/vmx/vmx.c
> +++ b/arch/x86/kvm/vmx/vmx.c
> @@ -1909,6 +1909,24 @@ void vmx_inject_exception(struct kvm_vcpu *vcpu)
>  	u32 intr_info = ex->vector | INTR_INFO_VALID_MASK;
>  	struct vcpu_vmx *vmx = to_vmx(vcpu);
>  
> +	/*
> +	 * When injecting a #DB, single-stepping is enabled in RFLAGS, and STI
> +	 * or MOV-SS blocking is active, set vmcs.PENDING_DBG_EXCEPTIONS.BS to
> +	 * prevent a false positive from VM-Entry consistency check.  VM-Entry
> +	 * asserts that a single-step #DB _must_ be pending in this scenario,
> +	 * as the previous instruction cannot have toggled RFLAGS.TF 0=>1
> +	 * (because STI and POP/MOV don't modify RFLAGS), therefore the one
> +	 * instruction delay when activating single-step breakpoints must have
> +	 * already expired.  However, the CPU isn't smart enough to peek at
> +	 * vmcs.VM_ENTRY_INTR_INFO_FIELD and so doesn't realize that yes, there
> +	 * is indeed a #DB pending/imminent.
> +	 */
> +	if (ex->vector == DB_VECTOR &&
> +	    (vmx_get_rflags(vcpu) & X86_EFLAGS_TF) &&
> +	    vmx_get_interrupt_shadow(vcpu))
> +		vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS,
> +			    vmcs_readl(GUEST_PENDING_DBG_EXCEPTIONS) | DR6_BS);

Pulling in a Sashiko comment:

 : By restricting this workaround to only when a #DB is injected, does this
 : leave the VM vulnerable to a VM-Entry failure regression after live migration?
 : 
 : KVM does not export GUEST_PENDING_DBG_EXCEPTIONS to userspace via
 : KVM_GET_VCPU_EVENTS. Therefore, upon migration, the destination KVM
 : initializes the VMCS with GUEST_PENDING_DBG_EXCEPTIONS=0.
 : 
 : If a live migration occurs when the guest is in an active interrupt shadow
 : with RFLAGS.TF=1, but a different event is pending (or no event is pending
 : due to a host timer preemption), this DB_VECTOR check is skipped or
 : vmx_inject_exception() is never called.
 : 
 : Consequently, KVM will attempt VM-Entry with TF=1, shadow=1, and BS=0.
 : The Intel SDM mandates that if RFLAGS.TF=1 and STI or MOV SS blocking is
 : active, the VM-Entry consistency check requires
 : GUEST_PENDING_DBG_EXCEPTIONS.BS=1. The hardware VM-Entry will fail due to
 : invalid guest state.
 : 
 : Since vmx_guest_state_valid() does not check the GUEST_PENDING_DBG_EXCEPTIONS
 : field, KVM's emulation_required flag evaluates to false. KVM then falls
 : into the error path in __vmx_handle_exit(), dumping the VMCS and crashing
 : the guest by returning KVM_EXIT_FAIL_ENTRY to userspace.
 : 
 : Does KVM need to handle the BS bit requirement in a broader context to
 : account for live migration when no #DB is being injected?

Yes, but that's a different problem entirely[*], and isn't even solvable on AMD
because SVM lacks an equivalent for GUEST_PENDING_DBG_EXCEPTIONS.  Note, only
MOV/POP-SS blocking matters, because STI blocking doesn't prevent single-step
#DBs, and single-step #DBs have higher priority than IRQs.

[*] https://lore.kernel.org/all/agUgeO5QNenQM9pT@google.com

Re: [PATCH v3 09/10] KVM: selftests: Verify guest debug DR7.GD checking during instruction emulation

[Sean Christopherson]

On Fri, May 15, 2026, Sean Christopherson wrote:
> @@ -202,6 +207,22 @@ int main(void)
>  			    run->debug.arch.pc, target_rip, run->debug.arch.dr6,
>  			    target_dr6);
>  
> +	/* test global disable in emulation */
> +	if (is_forced_emulation_enabled) {
> +		/* Skip the 3-bytes "mov dr0" */
> +		vcpu_skip_insn(vcpu, 3);
> +		vcpu_run(vcpu);
> +		TEST_ASSERT(run->exit_reason == KVM_EXIT_DEBUG &&
> +			    run->debug.arch.exception == DB_VECTOR &&
> +			    run->debug.arch.pc == CAST_TO_RIP(fep_bd_start) &&
> +			    run->debug.arch.dr6 == target_dr6,
> +			    "DR7.GD: exit %d exception %d rip 0x%llx "
> +			    "(should be 0x%llx) dr6 0x%llx (should be 0x%llx)",
> +			    run->exit_reason, run->debug.arch.exception,
> +			    run->debug.arch.pc, target_rip, run->debug.arch.dr6,

Per Sashiko[*], this should print CAST_TO_RIP(fep_bd_start), not target_rip.  I'll
fixup when applying.

[*] https://sashiko.dev/#/patchset/20260515222638.1949982-1-seanjc%40google.com

> +			    target_dr6);
> +	}
> +
>  	/* Disable all debug controls, run to the end */
>  	memset(&debug, 0, sizeof(debug));
>  	vcpu_guest_debug_set(vcpu, &debug);
> -- 
> 2.54.0.563.g4f69b47b94-goog
> 

Re: [PATCH v3 10/10] KVM: selftests: Verify VMX's GUEST_PENDING_DBG_EXCEPTIONS.BS Consistency Check

[Sean Christopherson]

On Fri, May 15, 2026, Sean Christopherson wrote:
> From: Hou Wenlong <houwenlong.hwl@antgroup.com>
> 
> In x86's debug_regs test, add a test case to cover the scenario where a
> single-step #DB occurs in an STI-shadow, in which case KVM needs to stuff
> vmcs.GUEST_PENDING_DBG_EXCEPTIONS.BS in order to satisfy a flawed VM-Entry
> Consistency Check.
> 
> Wire up an IRQ handler to gain a bit of bonus coverage, as the subsequent
> IRET from the #DB sets RFLAGS.IF, but *without* STI-blocking, and so the
> pending IRQ is expected on the instruction immediately following STI.
> 
> Signed-off-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
> [sean: expect the IRQ on the CLI, and explain why]
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  tools/testing/selftests/kvm/x86/debug_regs.c | 64 ++++++++++++++++++--
>  1 file changed, 60 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/testing/selftests/kvm/x86/debug_regs.c b/tools/testing/selftests/kvm/x86/debug_regs.c
> index ee9d0f3a5807..6299e921dc27 100644
> --- a/tools/testing/selftests/kvm/x86/debug_regs.c
> +++ b/tools/testing/selftests/kvm/x86/debug_regs.c
> @@ -15,11 +15,46 @@
>  
>  #define IRQ_VECTOR 0xAA
>  
> +#define  CAST_TO_RIP(v)  ((unsigned long long)&(v))
> +
>  /* For testing data access debug BP */
>  u32 guest_value;
>  
>  extern unsigned char sw_bp, hw_bp, write_data, ss_start, bd_start;
> -extern unsigned char fep_bd_start;
> +extern unsigned char fep_bd_start, fep_sti_start, fep_sti_end;
> +
> +static void guest_db_handler(struct ex_regs *regs)
> +{
> +	static int count;
> +	unsigned long target_rips[2] = {
> +		CAST_TO_RIP(fep_sti_start),
> +		CAST_TO_RIP(fep_sti_end),
> +	};
> +
> +	__GUEST_ASSERT(regs->rip == target_rips[count],
> +	               "STI[%u]: unexpected rip 0x%lx (should be 0x%lx)",
> +		       count, regs->rip, target_rips[count]);
> +	regs->rflags &= ~X86_EFLAGS_TF;
> +	count++;
> +}
> +
> +static void guest_irq_handler(struct ex_regs *regs)
> +{
> +	/*
> +	 * The pending IRQ should finally be take when KVM_GUESTDBG_BLOCKIRQ is
> +	 * cleared and IRQs are enabled.  Note, the IRQ is expected to arrive
> +	 * on the instruction immediately after STI, even though its in an STI
> +	 * shadow.  Because the next instruction has a coincident #DB, and #DBs
> +	 * are not subject to STI-blocking, the #DB will push RFLAGS.IF=1 on
> +	 * the stack, and the eventual IRET will unmask IRQs and obliterate the
> +	 * STI shadow in the process.
> +	 */
> +	unsigned long target_rip = CAST_TO_RIP(fep_sti_start);
> +
> +	__GUEST_ASSERT(regs->rip == target_rip,
> +		       "IRQ: unexpected rip 0x%lx (should be 0x%lx)",
> +		       regs->rip, target_rip);

From Sashiko:

 : Is an End of Interrupt (EOI) acknowledgment required here?
 : 
 : The handler is triggered by an APIC interrupt injected earlier in the test,
 : but does not write to the APIC_EOI register to acknowledge it.
 : 
 : While it might not fail this specific test since it ends immediately after,
 : does failing to send an EOI leave the interrupt permanently marked as
 : in-service in the local APIC state machine?

Yes, this will leave the IRQ dangling in the ISR.  It doesn't really matter, but
I'll throw in a:

	x2apic_write_reg(APIC_EOI, 0);

when applying.

> +}
>  
>  static void guest_code(void)
>  {
> @@ -66,14 +101,32 @@ static void guest_code(void)
>  	/* DR6.BD test */
>  	asm volatile("bd_start: mov %%dr0, %%rax" : : : "rax");
>  
> -	if (is_forced_emulation_enabled)
> +	/*
> +	 * Note, the IRET from the #DB that occurs in the below STI-shadow will
> +	 * unmask IRQs, i.e. the pending interrupt will be delivered after #DB
> +	 * handling, on the CLI!
> +	 */
> +	if (is_forced_emulation_enabled) {
>  		asm volatile(KVM_FEP "fep_bd_start: mov %%dr0, %%rax" : : : "rax");
>  
> +		/* pending debug exceptions for emulation */
> +		asm volatile("pushf\n\t"
> +			     "orq $" __stringify(X86_EFLAGS_TF) ", (%rsp)\n\t"
> +			     "popf\n\t"
> +			     "sti\n\t"
> +			     "fep_sti_start:"
> +			     "cli\n\t"
> +			     "pushf\n\t"
> +			     "orq $" __stringify(X86_EFLAGS_TF) ", (%rsp)\n\t"
> +			     "popf\n\t"
> +			     KVM_FEP "sti\n\t"
> +			     "fep_sti_end:"
> +			     "cli\n\t");
> +	}
> +

Also from Sashiko:

 : Does the test verify that the expected #DB and IRQ events actually occurred?
 : 
 : The assertions are placed inside guest_db_handler() and guest_irq_handler(),
 : but if a bug causes the exceptions to be dropped completely, the guest
 : will simply proceed to execute GUEST_DONE() without error.
 : 
 : Should there be a shared state variable to assert that the handlers fired
 : the expected number of times before concluding the test?

Another "yes".  I'll add an assert that the IRQ actually fired.

Re: [PATCH v3 01/10] KVM: VMX: Refresh GUEST_PENDING_DBG_EXCEPTIONS.BS on all injected #DBs

[Hou Wenlong]

On Wed, May 20, 2026 at 09:11:20AM -0700, Sean Christopherson wrote:
> On Fri, May 15, 2026, Sean Christopherson wrote:
> > Move KVM's stuffing of GUEST_PENDING_DBG_EXCEPTIONS.BS when RFLAGS.TF=1 and
> > MOV/POP SS or STI blocking is active into the exception injection code so
> > that KVM fixes up the VMCS for all injected #DBs, not only those that are
> > reflected back into the guest after #DB interception.  E.g. if KVM queues
> > a #DB in the emulator, or more importantly if userspace does save/restore
> > exactly on the #DB+shadow boundary, then KVM needs to massage the VMCS to
> > avoid the VM-Entry consistency check.
> > 
> > Opportunistically update the wording of the comment to describe the
> > behavior as a workaround of flawed CPU behavior/architecture, to make it
> > clear that the *only* thing KVM is doing is fudging around a consistency
> > check.  Per the SDM:
> > 
> >   There are no pending debug exceptions after VM entry if any of the
> >   following are true:
> > 
> >     * The VM entry is vectoring with one of the following interruption
> >       types: external interrupt, non-maskable interrupt (NMI), hardware
> >       exception, or privileged software exception.
> > 
> > I.e. forcing GUEST_PENDING_DBG_EXCEPTIONS.BS does *not* impact guest-
> > visible behavior.
> > 
> > Fixes: b9bed78e2fa9 ("KVM: VMX: Set vmcs.PENDING_DBG.BS on #DB in STI/MOVSS blocking shadow")
> > Cc: stable@vger.kernel.org
> > Reported-by: Hou Wenlong <houwenlong.hwl@antgroup.com>
> > Closes: https://lore.kernel.org/all/b1a294bc9ed4dae532474a5dc6c8cb6e5962de7c.1757416809.git.houwenlong.hwl@antgroup.com
> > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > ---
> >  arch/x86/kvm/vmx/vmx.c | 35 ++++++++++++++++++-----------------
> >  1 file changed, 18 insertions(+), 17 deletions(-)
> > 
> > diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
> > index 1701db1b2e18..a0a0ccf342d3 100644
> > --- a/arch/x86/kvm/vmx/vmx.c
> > +++ b/arch/x86/kvm/vmx/vmx.c
> > @@ -1909,6 +1909,24 @@ void vmx_inject_exception(struct kvm_vcpu *vcpu)
> >  	u32 intr_info = ex->vector | INTR_INFO_VALID_MASK;
> >  	struct vcpu_vmx *vmx = to_vmx(vcpu);
> >  
> > +	/*
> > +	 * When injecting a #DB, single-stepping is enabled in RFLAGS, and STI
> > +	 * or MOV-SS blocking is active, set vmcs.PENDING_DBG_EXCEPTIONS.BS to
> > +	 * prevent a false positive from VM-Entry consistency check.  VM-Entry
> > +	 * asserts that a single-step #DB _must_ be pending in this scenario,
> > +	 * as the previous instruction cannot have toggled RFLAGS.TF 0=>1
> > +	 * (because STI and POP/MOV don't modify RFLAGS), therefore the one
> > +	 * instruction delay when activating single-step breakpoints must have
> > +	 * already expired.  However, the CPU isn't smart enough to peek at
> > +	 * vmcs.VM_ENTRY_INTR_INFO_FIELD and so doesn't realize that yes, there
> > +	 * is indeed a #DB pending/imminent.
> > +	 */
> > +	if (ex->vector == DB_VECTOR &&
> > +	    (vmx_get_rflags(vcpu) & X86_EFLAGS_TF) &&
> > +	    vmx_get_interrupt_shadow(vcpu))
> > +		vmcs_writel(GUEST_PENDING_DBG_EXCEPTIONS,
> > +			    vmcs_readl(GUEST_PENDING_DBG_EXCEPTIONS) | DR6_BS);
> 
> Pulling in a Sashiko comment:
> 
>  : By restricting this workaround to only when a #DB is injected, does this
>  : leave the VM vulnerable to a VM-Entry failure regression after live migration?
>  : 
>  : KVM does not export GUEST_PENDING_DBG_EXCEPTIONS to userspace via
>  : KVM_GET_VCPU_EVENTS. Therefore, upon migration, the destination KVM
>  : initializes the VMCS with GUEST_PENDING_DBG_EXCEPTIONS=0.
>  : 
>  : If a live migration occurs when the guest is in an active interrupt shadow
>  : with RFLAGS.TF=1, but a different event is pending (or no event is pending
>  : due to a host timer preemption), this DB_VECTOR check is skipped or
>  : vmx_inject_exception() is never called.
>  :

Nice AI review. This reminds me that before writing v1 I was wondering:
if KVM_GUESTDBG_SINGLESTEP is enabled and there is no #DB injection,
then single-stepping over STI should fail as well. Then I noticed the
following code:
```
    /* When single-stepping over STI and MOV SS, we must clear the
     * corresponding interruptibility bits in the guest state. Otherwise
     * vmentry fails as it then expects bit 14 (BS) in pending debug
     * exceptions being set, but that's not correct for the guest
     * debugging
     * case. */
    if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP)
            vmx_set_interrupt_shadow(vcpu, 0);
```

However, at that time I was missing the live-migration case.

>  : Crnsequently, KVM will attempt VM-Entry with TF=1, shadow=1, and BS=0.
>  : The Intel SDM mandates that if RFLAGS.TF=1 and STI or MOV SS blocking is
>  : active, the VM-Entry consistency check requires
>  : GUEST_PENDING_DBG_EXCEPTIONS.BS=1. The hardware VM-Entry will fail due to
>  : invalid guest state.
>  : 
>  : Since vmx_guest_state_valid() does not check the GUEST_PENDING_DBG_EXCEPTIONS
>  : field, KVM's emulation_required flag evaluates to false. KVM then falls
>  : into the error path in __vmx_handle_exit(), dumping the VMCS and crashing
>  : the guest by returning KVM_EXIT_FAIL_ENTRY to userspace.
>  : 
>  : Does KVM need to handle the BS bit requirement in a broader context to
>  : account for live migration when no #DB is being injected?
> 
> Yes, but that's a different problem entirely[*], and isn't even solvable on AMD
> because SVM lacks an equivalent for GUEST_PENDING_DBG_EXCEPTIONS.  Note, only
> MOV/POP-SS blocking matters, because STI blocking doesn't prevent single-step
> #DBs, and single-step #DBs have higher priority than IRQs.
> 

Besides that, if MOV/POP SS is emulated with single-stepping, the
emulator currently injects a #DB. As a result, the guest would observe a
single-step #DB during the MOV/POP SS shadow, as stated in Intel SDM
28.7.1:

"If the VM entry is vectoring, there is no blocking by STI or by MOV SS
following the VM entry, regardless of the contents of the
interruptibility-state field."

If we try to fix this by not injecting #DB, then VM entry fails again.
Since emulating MOV/POP SS only happens with force emulation, perhaps we
can simply document this behavior?

Thanks!

> [*] https://lore.kernel.org/all/agUgeO5QNenQM9pT@google.com

Re: [PATCH v3 00/10] KVM: x86: Improve #DB handling in the emulator

[Sean Christopherson]

On Fri, 15 May 2026 15:26:28 -0700, Sean Christopherson wrote:
> Hou's series of fixes and cleanups related to #DB handling in the emulator.
> 
> v3:
>  - Fix the GUEST_PENDING_DBG_EXCEPTIONS.BS consistency check issue by
>    stuffing the VMCS during injection, not during emulation.
>  - Drop a prep that is no longer necessary.
>  - Massage a few shortlogs/changelogs.
>  - Fix the selftest that takes an IRQ in the STI-shadow after IRET.
>  - Fix an intermediate bug where kvm_queue_exception_e() was being used
>    instead of kvm_queue_exception_p().
> 
> [...]

Applied to kvm-x86 misc, thanks!

[01/10] KVM: VMX: Refresh GUEST_PENDING_DBG_EXCEPTIONS.BS on all injected #DBs
        https://github.com/kvm-x86/linux/commit/c5bad4fa2d5d
[02/10] KVM: x86: Capture "struct x86_exception" in inject_emulated_exception()
        https://github.com/kvm-x86/linux/commit/17afe9b750f2
[03/10] KVM: x86: Set guest DR6 by kvm_queue_exception_p() in instruction emulation
        https://github.com/kvm-x86/linux/commit/dfc1c4687e9c
[04/10] KVM: x86: Honor KVM_GUESTDBG_USE_HW_BP when emulating MOV DR (in emulator)
        https://github.com/kvm-x86/linux/commit/51543660c521
[05/10] KVM: x86: Honor KVM_GUESTDBG_USE_HW_BP when checking for code breakpoints in emulation
        https://github.com/kvm-x86/linux/commit/00b6520669bc
[06/10] KVM: x86: Move KVM_GUESTDBG_SINGLESTEP handling into kvm_inject_emulated_db()
        https://github.com/kvm-x86/linux/commit/6d79302b0834
[07/10] KVM: x86: Drop kvm_vcpu_do_singlestep() now that it's been gutted
        https://github.com/kvm-x86/linux/commit/a1e7aeb1be11
[08/10] KVM: selftests: Add all (known) EFLAGS bit definitions
        https://github.com/kvm-x86/linux/commit/35c08038cbdc
[09/10] KVM: selftests: Verify guest debug DR7.GD checking during instruction emulation
        https://github.com/kvm-x86/linux/commit/063a22451fd9
[10/10] KVM: selftests: Verify VMX's GUEST_PENDING_DBG_EXCEPTIONS.BS Consistency Check
        https://github.com/kvm-x86/linux/commit/b8284279b732

--
https://github.com/kvm-x86/linux/tree/next