[PATCH v4 0/9] KVM: nSVM: Enable DecodeAssists for nested guests

[PATCH v4 0/9] KVM: nSVM: Enable DecodeAssists for nested guests

[Tina Zhang]

The SVM DecodeAssists feature provides decode state for selected
VM-Exits.  KVM currently does not expose this feature to L1.  Some L1
hypervisors may therefore treat the platform's SVM support as
incomplete.

In practice, this was observed with Hyper-V running on top of KVM.
Hyper-V appears to require DecodeAssists before enabling nested SVM for
its guests.  Virtualizing the feature lets users enable Hyper-V
virtualization features inside a Windows VM when needed, e.g. to run
QEMU/KVM in WSL.  Without DecodeAssists, Hyper-V does not enable nested
SVM because DecodeAssists is missing from KVM's virtual SVM model.

Virtualize DecodeAssists for nested SVM.  Propagate instruction bytes
from VMCB02 for hardware-originated data #NPF and intercepted data #PF
VM-Exits.  For KVM-generated exits, synthesize the architectural
EXITINFO state and use matching bytes from the emulator when available,
fetching missing bytes through L2's address translation only as a
fallback.  Do not use the fallback for SEV guests, whose encrypted
memory cannot provide plaintext instruction bytes to KVM.

The selftest coverage in this version is intentionally broad and may be
more extensive than necessary.  Some redundant cases can be removed in
a later revision, but for now the series provides a comprehensive test
set for users to exercise hardware-reflected, KVM-synthesized,
userspace-injected, and boundary cases.

The selftest has been run with kvm.force_emulation_prefix both disabled
and enabled.

Changes since v3:
- Rebase onto kvm-x86/next.
- Register DecodeAssists in the CPUID 0x8000000A SVM capability
  initializer so that common code validates its CPUID word before the
  SVM code enables it for nested guests.
- Make the VMCB02 instruction-byte source const and simplify the
  synthesized-byte copy and fallback-fetch flow.

v3:
https://lore.kernel.org/r/cover.1785411877.git.zhang_wei@open-hieco.net

Changes since v2:
- Rebase onto kvm-x86/next.
- Track hardware-provided instruction bytes independently of the VMCB02
  exit code, and preserve the bytes when L0 handles an intercepted #PF
  before reflecting it to L1.
- Select the instruction-byte source using host-owned VMCB02 state
  instead of control fields in guest-owned VMCB12.
- Record whether a queued #PF VM-Exit has a matching emulator context,
  so userspace-injected #PF exits do not consume stale emulator bytes.
- Stop fallback instruction fetches at noncanonical addresses and at the
  32-bit linear-address boundary.
- Extend the selftest with regression coverage for replacing a hardware
  #NPF with a synthesized #NPF and for userspace-injected #PF during
  emulation, and harden its page layout and ucall handling.

v2:
https://lore.kernel.org/r/cover.1783999988.git.zhang_wei@open-hieco.net

Changes since v1:
- Split the implementation into seven patches to make the individual
  pieces easier to review.
- Add EXITINFO virtualization for emulator-generated MOV CR/DR, INTn,
  INVLPG, and INVLPGA intercepts.
- Limit GuestInstrBytes propagation to data #NPF and intercepted #PF
  exits, and clear the fields for instruction-fetch and unrelated exits.
- Provide GuestInstrBytes for KVM-synthesized data #PF/#NPF exits.  Use
  matching emulator bytes first and fetch missing bytes from L2 RIP as a
  fallback, while avoiding fallback reads for SEV guests.
- Expand the selftest beyond hardware #NPF and stale-state coverage to
  exercise hardware, synthesized, userspace-injected, instruction-fetch,
  page-boundary, and CS-limit cases.

v1:
https://lore.kernel.org/r/20260629125205.52394-1-zhang_wei@open-hieco.net

Tina Zhang (9):
  KVM: x86: Add helper to provide intercept linear addresses
  KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts
  KVM: nSVM: Track hardware-provided instruction bytes
  KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12
  KVM: x86: Track emulator-originated nested #PF VM-Exits
  KVM: nSVM: Use emulator bytes for synthesized nested #NPF/#PF
  KVM: nSVM: Fetch missing DecodeAssist bytes for synthesized #NPF/#PF
  KVM: nSVM: Advertise DecodeAssists to L1
  KVM: selftests: Add nested SVM DecodeAssists test

 arch/x86/include/asm/kvm_host.h               |   1 +
 arch/x86/kvm/cpuid.c                          |   1 +
 arch/x86/kvm/emulate.c                        |  29 +-
 arch/x86/kvm/kvm_emulate.h                    |   1 +
 arch/x86/kvm/svm/nested.c                     | 174 +++-
 arch/x86/kvm/svm/svm.c                        |  72 +-
 arch/x86/kvm/svm/svm.h                        |  20 +-
 arch/x86/kvm/x86.c                            |  33 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../kvm/x86/svm_nested_decode_assists_test.c  | 791 ++++++++++++++++++
 11 files changed, 1094 insertions(+), 30 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c


base-commit: 1b731e5ded480bd1e5546aed35584238661ce72e
-- 
2.43.7

[PATCH v4 1/9] KVM: x86: Add helper to provide intercept linear addresses

[Tina Zhang]

Add get_intercept_linear_addr() to compute the linear address for
intercepts that report one, and pass the result through
x86_instruction_info.

Handle INVLPG as the initial user.  INVLPG's memory operand is decoded
with NoAccess, and thus src_val does not contain the operand address.
Compute the address from the decoded segment base and effective address
in the emulator, where the operand state is available.

This allows intercept handlers to use the linear address directly
without duplicating the emulator's address calculation.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/emulate.c     | 29 +++++++++++++++++++++--------
 arch/x86/kvm/kvm_emulate.h |  1 +
 2 files changed, 22 insertions(+), 8 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 8071b372d233..a0e57b64cadd 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -410,6 +410,26 @@ static int em_salc(struct x86_emulate_ctxt *ctxt)
 	_fault ? X86EMUL_UNHANDLEABLE : X86EMUL_CONTINUE; \
 })
 
+static unsigned long seg_base(struct x86_emulate_ctxt *ctxt, int seg)
+{
+	if (ctxt->mode == X86EMUL_MODE_PROT64 && seg < VCPU_SREG_FS)
+		return 0;
+
+	return ctxt->ops->get_cached_segment_base(ctxt, seg);
+}
+
+static u64 get_intercept_linear_addr(struct x86_emulate_ctxt *ctxt,
+				     enum x86_intercept intercept)
+{
+	u64 la;
+
+	if (intercept != x86_intercept_invlpg)
+		return 0;
+
+	la = seg_base(ctxt, ctxt->src.addr.mem.seg) + ctxt->src.addr.mem.ea;
+	return ctxt->mode == X86EMUL_MODE_PROT64 ? la : (u32)la;
+}
+
 static int emulator_check_intercept(struct x86_emulate_ctxt *ctxt,
 				    enum x86_intercept intercept,
 				    enum x86_intercept_stage stage)
@@ -427,6 +447,7 @@ static int emulator_check_intercept(struct x86_emulate_ctxt *ctxt,
 		.src_type   = ctxt->src.type,
 		.dst_type   = ctxt->dst.type,
 		.ad_bytes   = ctxt->ad_bytes,
+		.intercept_linear_addr = get_intercept_linear_addr(ctxt, intercept),
 		.rip	    = ctxt->eip,
 		.next_rip   = ctxt->_eip,
 	};
@@ -520,14 +541,6 @@ static u32 desc_limit_scaled(struct desc_struct *desc)
 	return desc->g ? (limit << 12) | 0xfff : limit;
 }
 
-static unsigned long seg_base(struct x86_emulate_ctxt *ctxt, int seg)
-{
-	if (ctxt->mode == X86EMUL_MODE_PROT64 && seg < VCPU_SREG_FS)
-		return 0;
-
-	return ctxt->ops->get_cached_segment_base(ctxt, seg);
-}
-
 static int emulate_exception(struct x86_emulate_ctxt *ctxt, int vec,
 			     u32 error, bool valid)
 {
diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h
index 3e375af15c03..9eda648a1314 100644
--- a/arch/x86/kvm/kvm_emulate.h
+++ b/arch/x86/kvm/kvm_emulate.h
@@ -51,6 +51,7 @@ struct x86_instruction_info {
 	u8  src_type;		/* type of source operand		*/
 	u8  dst_type;		/* type of destination operand		*/
 	u8  ad_bytes;           /* size of src/dst address              */
+	u64 intercept_linear_addr;	/* arch-reported linear address, if any */
 	u64 rip;		/* rip of the instruction		*/
 	u64 next_rip;           /* rip following the instruction        */
 };
-- 
2.43.7

[PATCH v4 2/9] KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts

[Tina Zhang]

When the x86 emulator encounters an instruction intercepted by L1,
svm_check_intercept() synthesizes a nested VM-Exit without fresh hardware
DecodeAssist state.  Populate the architectural EXITINFO fields when
DecodeAssists is exposed to L1.

Generate EXITINFO1 with the GPR number for MOV CR/DR, the interrupt vector
for INTn, and the linear address for INVLPG.  Set EXITINFO1 to zero for
CLTS, LMSW, SMSW, and INVLPGA; the INVLPGA address remains in guest rAX.
Clear EXITINFO2 for all covered intercepts and leave unrelated intercepts
unchanged.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/svm.c | 48 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 9d607b98bd06..c7c1f1527c3c 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4810,6 +4810,52 @@ static const struct __x86_intercept {
 #undef POST_EX
 #undef POST_MEM
 
+static void svm_prepare_decode_assist_exit_info(struct kvm_vcpu *vcpu,
+						const struct x86_instruction_info *info)
+{
+	struct vmcb *vmcb = to_svm(vcpu)->vmcb;
+	u64 exit_info_1;
+
+	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
+		return;
+
+	switch (info->intercept) {
+	case x86_intercept_cr_read:
+	case x86_intercept_cr_write:
+		/* MOV CRx: bit 63 set, GPR number in bits 3:0. */
+		exit_info_1 = BIT_ULL(63) | (info->modrm_rm & 0xf);
+		break;
+	case x86_intercept_clts:
+	case x86_intercept_lmsw:
+	case x86_intercept_smsw:
+		/* CLTS/LMSW/SMSW: no decode information, bit 63 clear. */
+		exit_info_1 = 0;
+		break;
+	case x86_intercept_dr_read:
+	case x86_intercept_dr_write:
+		/* MOV DRx: GPR number in bits 3:0. */
+		exit_info_1 = info->modrm_rm & 0xf;
+		break;
+	case x86_intercept_intn:
+		/* INTn: software interrupt number in bits 7:0. */
+		exit_info_1 = info->src_val & 0xff;
+		break;
+	case x86_intercept_invlpg:
+		/* INVLPG: linear address of the target page. */
+		exit_info_1 = info->intercept_linear_addr;
+		break;
+	case x86_intercept_invlpga:
+		/* INVLPGA: the address remains available in guest rAX. */
+		exit_info_1 = 0;
+		break;
+	default:
+		return;
+	}
+
+	vmcb->control.exit_info_1 = exit_info_1;
+	vmcb->control.exit_info_2 = 0;
+}
+
 static int svm_check_intercept(struct kvm_vcpu *vcpu,
 			       struct x86_instruction_info *info,
 			       enum x86_intercept_stage stage,
@@ -4931,6 +4977,8 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
 		break;
 	}
 
+	svm_prepare_decode_assist_exit_info(vcpu, info);
+
 	/* TODO: Advertise NRIPS to guest hypervisor unconditionally */
 	if (static_cpu_has(X86_FEATURE_NRIPS))
 		vmcb->control.next_rip  = info->next_rip;
-- 
2.43.7

[PATCH v4 3/9] KVM: nSVM: Track hardware-provided instruction bytes

[Tina Zhang]

DecodeAssists provides instruction bytes for nested page faults and
intercepted page faults caused by data accesses.  When reflecting such an
exit to L1, KVM needs to distinguish bytes produced by the current hardware
VM-Exit from stale VMCB02 state.

Clear the VMCB02 instruction-byte fields before each nested run and track
whether the nested VM-Exit being reflected originated from hardware.
Preserve that state when a hardware #PF is processed by KVM and then
reflected to L1.

A subsequent change will use this state to propagate hardware-provided
instruction bytes to VMCB12.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/nested.c | 17 +++++++++++++++--
 arch/x86/kvm/svm/svm.c    | 22 ++++++++++++++--------
 arch/x86/kvm/svm/svm.h    |  8 +++++++-
 3 files changed, 36 insertions(+), 11 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 73f37b050d0a..d08c30e9a6f4 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -35,6 +35,13 @@
 
 #define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK
 
+static void nested_svm_clear_insn_bytes(struct vmcb *vmcb)
+{
+	vmcb->control.insn_len = 0;
+	memset(vmcb->control.insn_bytes, 0,
+	       sizeof(vmcb->control.insn_bytes));
+}
+
 static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 				       struct x86_exception *fault,
 				       bool from_hardware)
@@ -68,6 +75,7 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 				    (fault->error_code & ~PFERR_GUEST_FAULT_STAGE_MASK);
 	vmcb->control.exit_info_2 = fault->address;
 
+	svm->nested.vmcb02_insn_bytes_fresh = from_hardware;
 	nested_svm_vmexit(svm);
 }
 
@@ -868,7 +876,10 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 	/*
 	 * Filled at exit: exit_code, exit_info_1, exit_info_2, exit_int_info,
 	 * exit_int_info_err, next_rip, insn_len, insn_bytes.
+	 * Clear stale DecodeAssist data before L2 runs.
 	 */
+	nested_svm_clear_insn_bytes(vmcb02);
+	svm->nested.vmcb02_insn_bytes_fresh = false;
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
 	    (vmcb12_ctrl->int_ctl & V_GIF_ENABLE_MASK))
@@ -1643,14 +1654,16 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
 	return vmexit;
 }
 
-int nested_svm_exit_handled(struct vcpu_svm *svm)
+int nested_svm_exit_handled(struct vcpu_svm *svm, bool vmcb02_insn_bytes_fresh)
 {
 	int vmexit;
 
 	vmexit = nested_svm_intercept(svm);
 
-	if (vmexit == NESTED_EXIT_DONE)
+	if (vmexit == NESTED_EXIT_DONE) {
+		svm->nested.vmcb02_insn_bytes_fresh = vmcb02_insn_bytes_fresh;
 		nested_svm_vmexit(svm);
+	}
 
 	return vmexit;
 }
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index c7c1f1527c3c..b85e43112db9 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -1953,14 +1953,20 @@ static void svm_set_dr7(struct kvm_vcpu *vcpu, unsigned long value)
 static int pf_interception(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
-
 	u64 fault_address = svm->vmcb->control.exit_info_2;
 	u64 error_code = svm->vmcb->control.exit_info_1;
+	int r;
+
+	r = kvm_handle_page_fault(vcpu, error_code, fault_address,
+				  static_cpu_has(X86_FEATURE_DECODEASSISTS) ?
+				  svm->vmcb->control.insn_bytes : NULL,
+				  svm->vmcb->control.insn_len);
 
-	return kvm_handle_page_fault(vcpu, error_code, fault_address,
-			static_cpu_has(X86_FEATURE_DECODEASSISTS) ?
-			svm->vmcb->control.insn_bytes : NULL,
-			svm->vmcb->control.insn_len);
+	if (is_guest_mode(vcpu) && vcpu->arch.exception_vmexit.pending &&
+	    vcpu->arch.exception_vmexit.vector == PF_VECTOR)
+		svm->nested.vmcb02_insn_bytes_fresh = true;
+
+	return r;
 }
 
 static int svm_check_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type,
@@ -2571,7 +2577,7 @@ static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
 
 	if (cr0 ^ val) {
 		svm->vmcb->control.exit_code = SVM_EXIT_CR0_SEL_WRITE;
-		ret = (nested_svm_exit_handled(svm) == NESTED_EXIT_DONE);
+		ret = (nested_svm_exit_handled(svm, false) == NESTED_EXIT_DONE);
 	}
 
 	return ret;
@@ -3723,7 +3729,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 		vmexit = nested_svm_exit_special(svm);
 
 		if (vmexit == NESTED_EXIT_CONTINUE)
-			vmexit = nested_svm_exit_handled(svm);
+			vmexit = nested_svm_exit_handled(svm, true);
 
 		if (vmexit == NESTED_EXIT_DONE)
 			return 1;
@@ -4983,7 +4989,7 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
 	if (static_cpu_has(X86_FEATURE_NRIPS))
 		vmcb->control.next_rip  = info->next_rip;
 	vmcb->control.exit_code = icpt_info.exit_code;
-	vmexit = nested_svm_exit_handled(svm);
+	vmexit = nested_svm_exit_handled(svm, false);
 
 	ret = (vmexit == NESTED_EXIT_DONE) ? X86EMUL_INTERCEPTED
 					   : X86EMUL_CONTINUE;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 66b44b54608e..b201bac227ad 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -242,6 +242,12 @@ struct svm_nested_state {
 	 * on its side.
 	 */
 	bool force_msr_bitmap_recalc;
+
+	/*
+	 * True if VMCB02 contains DecodeAssist instruction bytes from the
+	 * hardware VM-Exit currently being reflected to L1.
+	 */
+	bool vmcb02_insn_bytes_fresh;
 };
 
 struct vcpu_sev_es_state {
@@ -887,7 +893,7 @@ static inline void nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
 	nested_svm_vmexit(svm);
 }
 
-int nested_svm_exit_handled(struct vcpu_svm *svm);
+int nested_svm_exit_handled(struct vcpu_svm *svm, bool vmcb02_insn_bytes_fresh);
 int nested_svm_check_permissions(struct kvm_vcpu *vcpu);
 int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu);
 int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr,
-- 
2.43.7

[PATCH v4 4/9] KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12

[Tina Zhang]

DecodeAssists provides instruction bytes for nested page faults and
intercepted page faults caused by data accesses.  When the feature is
exposed to L1, copy fresh hardware-provided instruction bytes from VMCB02
to VMCB12 for these exits.

Clear the VMCB12 instruction-byte state before rebuilding it, and leave the
length zero for instruction-fetch page faults, unrelated exits, and exits
without fresh hardware bytes.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/nested.c | 49 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index d08c30e9a6f4..d34370b84d5b 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -42,6 +42,53 @@ static void nested_svm_clear_insn_bytes(struct vmcb *vmcb)
 	       sizeof(vmcb->control.insn_bytes));
 }
 
+static void nested_svm_copy_insn_bytes(struct vmcb *to,
+				       const struct vmcb *from)
+{
+	u8 insn_len = from->control.insn_len;
+
+	nested_svm_clear_insn_bytes(to);
+
+	if (WARN_ON_ONCE(insn_len > sizeof(from->control.insn_bytes)))
+		return;
+
+	to->control.insn_len = insn_len;
+	memcpy(to->control.insn_bytes, from->control.insn_bytes, insn_len);
+}
+
+static bool nested_svm_vmexit_supports_insn_bytes(struct kvm_vcpu *vcpu,
+						  const struct vmcb *vmcb02)
+{
+	u64 exit_code = vmcb02->control.exit_code;
+
+	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
+		return false;
+
+	if (exit_code != SVM_EXIT_NPF &&
+	    exit_code != SVM_EXIT_EXCP_BASE + PF_VECTOR)
+		return false;
+
+	return !(vmcb02->control.exit_info_1 & PFERR_FETCH_MASK);
+}
+
+static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
+						struct vmcb *vmcb12,
+						const struct vmcb *vmcb02)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+
+	nested_svm_clear_insn_bytes(vmcb12);
+
+	if (!nested_svm_vmexit_supports_insn_bytes(vcpu, vmcb02))
+		goto out;
+
+	if (svm->nested.vmcb02_insn_bytes_fresh)
+		nested_svm_copy_insn_bytes(vmcb12, vmcb02);
+
+out:
+	svm->nested.vmcb02_insn_bytes_fresh = false;
+}
+
 static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 				       struct x86_exception *fault,
 				       bool from_hardware)
@@ -1308,6 +1355,8 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu)
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
 		vmcb12->control.next_rip  = vmcb02->control.next_rip;
 
+	nested_svm_update_vmcb12_insn_bytes(vcpu, vmcb12, vmcb02);
+
 	if (nested_vmcb12_has_lbrv(vcpu))
 		svm_copy_lbrs(&vmcb12->save, &vmcb02->save);
 
-- 
2.43.7

[PATCH v4 5/9] KVM: x86: Track emulator-originated nested #PF VM-Exits

[Tina Zhang]

Add has_emulator_context to kvm_queued_exception to record whether a
queued exception VM-Exit originated from the current emulator exception.
This provenance allows nested virtualization code to determine whether
the emulator decode state can be used for a nested #PF VM-Exit.

Explicit provenance is needed because a matching emulator RIP does not
prove that the fetch cache describes the faulting instruction. For
example, completing pending emulated I/O can advance RIP while leaving
the fetch cache populated with the completed instruction.

Determine provenance in kvm_inject_page_fault() and carry it through
exception queueing so that it is stored as part of the nested VM-Exit
event. Userspace-injected and other unrelated exceptions leave the flag
clear.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/include/asm/kvm_host.h |  1 +
 arch/x86/kvm/x86.c              | 33 ++++++++++++++++++++++-----------
 2 files changed, 23 insertions(+), 11 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 283847619ff8..eb9f1b9afeb7 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -688,6 +688,7 @@ struct kvm_queued_exception {
 	bool pending;
 	bool injected;
 	bool has_error_code;
+	bool has_emulator_context;
 	u8 vector;
 	u32 error_code;
 	unsigned long payload;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index d94b59140c45..75d00340c71f 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -444,7 +444,8 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_deliver_exception_payload);
 
 static void kvm_queue_exception_vmexit(struct kvm_vcpu *vcpu, unsigned int vector,
 				       bool has_error_code, u32 error_code,
-				       bool has_payload, unsigned long payload)
+				       bool has_payload, unsigned long payload,
+				       bool has_emulator_context)
 {
 	struct kvm_queued_exception *ex = &vcpu->arch.exception_vmexit;
 
@@ -452,6 +453,7 @@ static void kvm_queue_exception_vmexit(struct kvm_vcpu *vcpu, unsigned int vecto
 	ex->injected = false;
 	ex->pending = true;
 	ex->has_error_code = has_error_code;
+	ex->has_emulator_context = has_emulator_context;
 	ex->error_code = error_code;
 	ex->has_payload = has_payload;
 	ex->payload = payload;
@@ -459,7 +461,8 @@ static void kvm_queue_exception_vmexit(struct kvm_vcpu *vcpu, unsigned int vecto
 
 static void kvm_multiple_exception(struct kvm_vcpu *vcpu, unsigned int nr,
 				   bool has_error, u32 error_code,
-				   bool has_payload, unsigned long payload)
+				   bool has_payload, unsigned long payload,
+				   bool has_emulator_context)
 {
 	u32 prev_nr;
 	int class1, class2;
@@ -473,7 +476,8 @@ static void kvm_multiple_exception(struct kvm_vcpu *vcpu, unsigned int nr,
 	if (is_guest_mode(vcpu) &&
 	    kvm_nested_call(is_exception_vmexit)(vcpu, nr, error_code)) {
 		kvm_queue_exception_vmexit(vcpu, nr, has_error, error_code,
-					   has_payload, payload);
+					   has_payload, payload,
+					   has_emulator_context);
 		return;
 	}
 
@@ -519,7 +523,7 @@ static void kvm_multiple_exception(struct kvm_vcpu *vcpu, unsigned int nr,
 
 void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr)
 {
-	kvm_multiple_exception(vcpu, nr, false, 0, false, 0);
+	kvm_multiple_exception(vcpu, nr, false, 0, false, 0, false);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_queue_exception);
 
@@ -527,14 +531,16 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_queue_exception);
 void kvm_queue_exception_p(struct kvm_vcpu *vcpu, unsigned nr,
 			   unsigned long payload)
 {
-	kvm_multiple_exception(vcpu, nr, false, 0, true, payload);
+	kvm_multiple_exception(vcpu, nr, false, 0, true, payload, false);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_queue_exception_p);
 
 static void kvm_queue_exception_e_p(struct kvm_vcpu *vcpu, unsigned nr,
-				    u32 error_code, unsigned long payload)
+				    u32 error_code, unsigned long payload,
+				    bool has_emulator_context)
 {
-	kvm_multiple_exception(vcpu, nr, true, error_code, true, payload);
+	kvm_multiple_exception(vcpu, nr, true, error_code, true, payload,
+			       has_emulator_context);
 }
 
 void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned int nr,
@@ -580,6 +586,9 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_complete_insn_gp);
 void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault,
 			   bool from_hardware)
 {
+	struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt;
+	bool has_emulator_context = ctxt && fault == &ctxt->exception;
+
 	++vcpu->stat.pf_guest;
 
 	/*
@@ -589,10 +598,12 @@ void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault,
 	if (is_guest_mode(vcpu) && fault->async_page_fault)
 		kvm_queue_exception_vmexit(vcpu, PF_VECTOR,
 					   true, fault->error_code,
-					   true, fault->address);
+					   true, fault->address,
+					   has_emulator_context);
 	else
 		kvm_queue_exception_e_p(vcpu, PF_VECTOR, fault->error_code,
-					fault->address);
+					fault->address,
+					has_emulator_context);
 }
 
 void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu,
@@ -627,7 +638,7 @@ void kvm_inject_nmi(struct kvm_vcpu *vcpu)
 
 void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_code)
 {
-	kvm_multiple_exception(vcpu, nr, true, error_code, false, 0);
+	kvm_multiple_exception(vcpu, nr, true, error_code, false, 0, false);
 }
 EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_queue_exception_e);
 
@@ -8957,7 +8968,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
 	    kvm_nested_call(is_exception_vmexit)(vcpu, ex->vector, ex->error_code)) {
 		kvm_queue_exception_vmexit(vcpu, ex->vector,
 					   ex->has_error_code, ex->error_code,
-					   ex->has_payload, ex->payload);
+					   ex->has_payload, ex->payload, false);
 		ex->injected = false;
 		ex->pending = false;
 	}
-- 
2.43.7

[PATCH v4 6/9] KVM: nSVM: Use emulator bytes for synthesized nested #NPF/#PF

[Tina Zhang]

SVM DecodeAssists provides instruction bytes for data-access #NPF and
intercepted #PF exits. Hardware-reflected VM-Exits can use fresh VMCB02
bytes, but KVM-synthesized exits have no hardware byte state to propagate.

For a synthesized nested #NPF, use the emulator fetch cache only when the
fault is the current emulator exception. For a synthesized intercepted #PF,
require the queued exception VM-Exit to carry the same emulator provenance.

Store the bytes in a one-shot buffer that is consumed while constructing
VMCB12 and cleared before the next nested run. This prevents unrelated
nested VM-Exits from reusing stale emulator bytes.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/nested.c | 54 ++++++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/svm/svm.h    | 12 +++++++++
 2 files changed, 65 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index d34370b84d5b..a38f99404943 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -71,22 +71,65 @@ static bool nested_svm_vmexit_supports_insn_bytes(struct kvm_vcpu *vcpu,
 	return !(vmcb02->control.exit_info_1 & PFERR_FETCH_MASK);
 }
 
+static void nested_svm_clear_synthesized_insn_bytes(struct vcpu_svm *svm)
+{
+	svm->nested.synthesized_insn_bytes.prepared = false;
+	svm->nested.synthesized_insn_bytes.insn_len = 0;
+}
+
+static void nested_svm_prepare_synthesized_insn_bytes(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+	struct nested_svm_insn_bytes *synthesized =
+		&svm->nested.synthesized_insn_bytes;
+	struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt;
+
+	static_assert(sizeof(synthesized->insn_bytes) >=
+		      sizeof(ctxt->fetch.data));
+
+	nested_svm_clear_synthesized_insn_bytes(svm);
+
+	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
+		return;
+
+	if (!ctxt || ctxt->eip != kvm_rip_read(vcpu) ||
+	    ctxt->fetch.end < ctxt->fetch.data ||
+	    ctxt->fetch.end > ctxt->fetch.data + sizeof(ctxt->fetch.data))
+		return;
+
+	synthesized->insn_len = ctxt->fetch.end - ctxt->fetch.data;
+	memcpy(synthesized->insn_bytes, ctxt->fetch.data,
+	       synthesized->insn_len);
+	synthesized->prepared = true;
+}
+
 static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
 						struct vmcb *vmcb12,
 						const struct vmcb *vmcb02)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
+	struct nested_svm_insn_bytes *synthesized =
+		&svm->nested.synthesized_insn_bytes;
 
 	nested_svm_clear_insn_bytes(vmcb12);
 
 	if (!nested_svm_vmexit_supports_insn_bytes(vcpu, vmcb02))
 		goto out;
 
-	if (svm->nested.vmcb02_insn_bytes_fresh)
+	if (svm->nested.vmcb02_insn_bytes_fresh) {
 		nested_svm_copy_insn_bytes(vmcb12, vmcb02);
+		goto out;
+	}
+
+	if (synthesized->prepared) {
+		vmcb12->control.insn_len = synthesized->insn_len;
+		memcpy(vmcb12->control.insn_bytes, synthesized->insn_bytes,
+		       vmcb12->control.insn_len);
+	}
 
 out:
 	svm->nested.vmcb02_insn_bytes_fresh = false;
+	nested_svm_clear_synthesized_insn_bytes(svm);
 }
 
 static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
@@ -95,6 +138,8 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct vmcb *vmcb = svm->vmcb;
+	struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt;
+	bool from_emulation = ctxt && fault == &ctxt->exception;
 	u64 fault_stage;
 
 	/*
@@ -123,6 +168,8 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 	vmcb->control.exit_info_2 = fault->address;
 
 	svm->nested.vmcb02_insn_bytes_fresh = from_hardware;
+	if (from_emulation && !(fault->error_code & PFERR_FETCH_MASK))
+		nested_svm_prepare_synthesized_insn_bytes(vcpu);
 	nested_svm_vmexit(svm);
 }
 
@@ -927,6 +974,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 	 */
 	nested_svm_clear_insn_bytes(vmcb02);
 	svm->nested.vmcb02_insn_bytes_fresh = false;
+	nested_svm_clear_synthesized_insn_bytes(svm);
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
 	    (vmcb12_ctrl->int_ctl & V_GIF_ENABLE_MASK))
@@ -1760,6 +1808,10 @@ static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu)
 			vmcb->control.exit_info_2 = ex->payload;
 		else
 			vmcb->control.exit_info_2 = vcpu->arch.cr2;
+
+		if (ex->has_emulator_context &&
+		    (!ex->has_error_code || !(ex->error_code & PFERR_FETCH_MASK)))
+			nested_svm_prepare_synthesized_insn_bytes(vcpu);
 	} else if (ex->vector == DB_VECTOR) {
 		/* See kvm_check_and_inject_events().  */
 		kvm_deliver_exception_payload(vcpu, ex);
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index b201bac227ad..51dc60b9c1d4 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -209,6 +209,12 @@ struct vmcb_ctrl_area_cached {
 	};
 };
 
+struct nested_svm_insn_bytes {
+	bool prepared;
+	u8 insn_len;
+	u8 insn_bytes[X86_MAX_INSTRUCTION_LENGTH];
+};
+
 struct svm_nested_state {
 	struct kvm_vmcb_info vmcb02;
 	u64 hsave_msr;
@@ -248,6 +254,12 @@ struct svm_nested_state {
 	 * hardware VM-Exit currently being reflected to L1.
 	 */
 	bool vmcb02_insn_bytes_fresh;
+
+	/*
+	 * Emulator fetch bytes captured for a synthesized nested #NPF/#PF.  The
+	 * buffer is consumed and cleared when constructing VMCB12.
+	 */
+	struct nested_svm_insn_bytes synthesized_insn_bytes;
 };
 
 struct vcpu_sev_es_state {
-- 
2.43.7

[PATCH v4 7/9] KVM: nSVM: Fetch missing DecodeAssist bytes for synthesized #NPF/#PF

[Tina Zhang]

The emulator fetch cache is not guaranteed to contain the full
architected 15-byte DecodeAssist window, e.g. it may only contain the
bytes needed to decode the instruction.

Keep preparation of synthesized state limited to capturing a matching
emulator fetch cache.  When constructing VMCB12, copy those bytes and
fetch any missing tail through L2 guest page tables, stopping at a
translation fault, read failure, non-canonical address, or CS limit
overrun.  If no matching emulator bytes are available, fetch the full
window from L2 RIP, e.g. for a #PF injected by userspace.

For a non-64-bit L2, truncate each incremented linear address to 32 bits
so that a fetch whose CS.base makes it cross the 4GB boundary wraps as
required.

Do not perform tail or fallback reads for SEV guests.  KVM cannot read
plaintext instruction bytes from encrypted guest memory, and the
existing SEV emulation path treats missing hardware DecodeAssist bytes
as unavailable instead of decoding guest memory.  For nested SEV,
report only matching emulator bytes that are already available,
potentially a zero instruction-byte count.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/nested.c | 58 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index a38f99404943..5901174a8775 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -77,6 +77,54 @@ static void nested_svm_clear_synthesized_insn_bytes(struct vcpu_svm *svm)
 	svm->nested.synthesized_insn_bytes.insn_len = 0;
 }
 
+static u8 nested_svm_fetch_insn_bytes(struct kvm_vcpu *vcpu, u8 *bytes,
+				      u8 count, u8 max_bytes)
+{
+	struct kvm_pagewalk *gva_walk = &vcpu->arch.gva_walk;
+	u64 access = PFERR_FETCH_MASK;
+	gva_t rip = kvm_get_linear_rip(vcpu);
+	struct x86_exception e;
+
+	if (kvm_x86_call(get_cpl)(vcpu) == 3)
+		access |= PFERR_USER_MASK;
+
+	if (!is_64_bit_mode(vcpu)) {
+		u32 eip = kvm_rip_read(vcpu);
+		u32 limit = to_svm(vcpu)->vmcb->save.cs.limit;
+
+		if (eip > limit)
+			return 0;
+		max_bytes = min_t(u64, max_bytes, (u64)limit - eip + 1);
+	}
+
+	count = min(count, max_bytes);
+
+	while (count < max_bytes) {
+		gva_t addr = rip + count;
+		unsigned int chunk;
+		gpa_t gpa;
+
+		if (!is_64_bit_mode(vcpu))
+			addr = (u32)addr;
+		else if (is_noncanonical_address(addr, vcpu, 0))
+			break;
+
+		chunk = min_t(unsigned int, max_bytes - count,
+			      PAGE_SIZE - offset_in_page(addr));
+		gpa = gva_walk->gva_to_gpa(vcpu, gva_walk, addr, access, &e);
+
+		if (gpa == INVALID_GPA ||
+		    kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(gpa),
+					     bytes + count,
+					     offset_in_page(gpa), chunk))
+			break;
+
+		count += chunk;
+	}
+
+	return count;
+}
+
 static void nested_svm_prepare_synthesized_insn_bytes(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -110,6 +158,7 @@ static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct nested_svm_insn_bytes *synthesized =
 		&svm->nested.synthesized_insn_bytes;
+	const u8 max_bytes = sizeof(vmcb12->control.insn_bytes);
 
 	nested_svm_clear_insn_bytes(vmcb12);
 
@@ -124,9 +173,16 @@ static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
 	if (synthesized->prepared) {
 		vmcb12->control.insn_len = synthesized->insn_len;
 		memcpy(vmcb12->control.insn_bytes, synthesized->insn_bytes,
-		       vmcb12->control.insn_len);
+		       synthesized->insn_len);
 	}
 
+	if (!is_sev_guest(vcpu))
+		vmcb12->control.insn_len =
+			nested_svm_fetch_insn_bytes(vcpu,
+						    vmcb12->control.insn_bytes,
+						    vmcb12->control.insn_len,
+						    max_bytes);
+
 out:
 	svm->nested.vmcb02_insn_bytes_fresh = false;
 	nested_svm_clear_synthesized_insn_bytes(svm);
-- 
2.43.7

[PATCH v4 8/9] KVM: nSVM: Advertise DecodeAssists to L1

[Tina Zhang]

Advertise DecodeAssists to L1 now that KVM virtualizes the
guest-visible DecodeAssist state in VMCB12: EXITINFO1 decode data for
MOV CR/DR, INTn, and INVLPG exits, plus instruction bytes for nested
page faults and intercepted #PF exits.  INVLPGA's linear address remains
available directly from the saved guest rAX, as required by the APM.

Expose the feature only when supported by hardware, as KVM still relies
on hardware DecodeAssists for VM-Exits that are reflected directly from
L2.

With DecodeAssists exposed, QEMU configurations that require the feature
(e.g. "-cpu ...,+decodeassists,...,enforce") are no longer rejected.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/cpuid.c   | 1 +
 arch/x86/kvm/svm/svm.c | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index ddb022cb203a..b644e45eac71 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -1208,6 +1208,7 @@ void kvm_initialize_cpu_caps(void)
 		VENDOR_F(NPT),
 		VENDOR_F(VMCBCLEAN),
 		VENDOR_F(FLUSHBYASID),
+		VENDOR_F(DECODEASSISTS),
 		VENDOR_F(NRIPS),
 		VENDOR_F(TSCRATEMSR),
 		VENDOR_F(V_VMSAVE_VMLOAD),
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index b85e43112db9..23e087d65f49 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5566,6 +5566,8 @@ static __init void svm_set_cpu_caps(void)
 		 */
 		kvm_cpu_cap_set(X86_FEATURE_FLUSHBYASID);
 
+		kvm_cpu_cap_check_and_set(X86_FEATURE_DECODEASSISTS);
+
 		if (nrips)
 			kvm_cpu_cap_set(X86_FEATURE_NRIPS);
 
-- 
2.43.7

[PATCH v4 9/9] KVM: selftests: Add nested SVM DecodeAssists test

[Tina Zhang]

Add a nested SVM selftest for DecodeAssists.  Verify that KVM exposes
the feature to L1 and provides architectural exit state for MOV CR/DR,
CLTS, LMSW, SMSW, INTn, INVLPG, and INVLPGA intercepts.

For data #NPF and intercepted #PF exits, cover instruction bytes from
hardware, the emulator fetch cache, and on-demand fetching from L2 RIP.
Verify that instruction-fetch #PF reports no bytes, that fetching stops
at an unreadable page or non-canonical address, and that 32-bit linear
addresses wrap correctly.

Add regressions for a synthesized #NPF that follows a hardware #NPF in
the same emulated instruction, and for a userspace-injected #PF while an
emulated MMIO read is awaiting completion.

The synthesized OUTSB #NPF and userspace-injected #PF paths run by
default.  The remaining synthesized paths are covered when
kvm.force_emulation_prefix=1 is enabled.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../kvm/x86/svm_nested_decode_assists_test.c  | 791 ++++++++++++++++++
 3 files changed, 793 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 00123169a190..41a3db9a386b 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -118,6 +118,7 @@ TEST_GEN_PROGS_x86 += x86/vmx_preemption_timer_test
 TEST_GEN_PROGS_x86 += x86/svm_vmcall_test
 TEST_GEN_PROGS_x86 += x86/svm_int_ctl_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_clear_efer_svme
+TEST_GEN_PROGS_x86 += x86/svm_nested_decode_assists_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_vmcb12_gpa
diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index 6e6f70035508..e9d745a2c131 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -220,6 +220,7 @@ struct kvm_x86_cpu_feature {
 #define	X86_FEATURE_LBRV		KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 1)
 #define	X86_FEATURE_NRIPS		KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 3)
 #define X86_FEATURE_TSCRATEMSR          KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 4)
+#define X86_FEATURE_DECODEASSISTS       KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 7)
 #define X86_FEATURE_PAUSEFILTER         KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 10)
 #define X86_FEATURE_PFTHRESHOLD         KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 12)
 #define	X86_FEATURE_V_VMSAVE_VMLOAD	KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 15)
diff --git a/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c b/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c
new file mode 100644
index 000000000000..13511ca8473b
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c
@@ -0,0 +1,791 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test KVM's virtualization of SVM DecodeAssists for nested guests.
+ */
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "svm_util.h"
+
+#define TEST_INT_VECTOR 0x81
+
+/* Any canonical virtual address that is never mapped by the selftest VM. */
+#define PF_TEST_GVA BIT_ULL(40)
+#define PF_FETCH_TEST_GVA BIT_ULL(41)
+
+#define OUTSB_OPCODE 0x6e
+#define BOUNDARY_OUTSB_CODE_SIZE 15
+#define LINEAR_WRAP_CODE_GVA (BIT_ULL(32) - PAGE_SIZE)
+#define LINEAR_WRAP_OUTSB_OFFSET (PAGE_SIZE - 8)
+#define LINEAR_WRAP_SETUP_SIZE 9
+#define LINEAR_WRAP_SETUP_OFFSET \
+	(LINEAR_WRAP_OUTSB_OFFSET - LINEAR_WRAP_SETUP_SIZE)
+#define LINEAR_WRAP_CS_BASE PAGE_SIZE
+#define TEST_IOPM_SIZE (3 * PAGE_SIZE)
+
+static u8 npf_target[PAGE_SIZE] __aligned(PAGE_SIZE);
+static u8 mmio_source __aligned(PAGE_SIZE);
+static u8 boundary_outsb_code[2 * PAGE_SIZE] __aligned(PAGE_SIZE);
+static const u8 linear_wrap_insn_bytes[15] = {
+	OUTSB_OPCODE, 0x11, 0x22, 0x33, 0x44, 0x55, 0x66, 0x77,
+	0x88, 0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee,
+};
+
+static void l2_read_code(void)
+{
+	asm volatile("mov (%0), %%rax" : : "r"(&npf_target) : "rax", "memory");
+	GUEST_FAIL("L2 read did not cause a nested page fault");
+}
+
+static void l2_outsb_code(void)
+{
+	asm volatile("mov %0, %%rsi\n\t"
+		     "mov $0x80, %%dx\n\t"
+		     "outsb"
+		     : : "r"(&npf_target) : "rsi", "rdx", "memory");
+	GUEST_FAIL("L2 OUTSB did not cause a nested page fault");
+}
+
+static void l2_movsb_code(void)
+{
+	asm volatile("mov %0, %%rsi\n\t"
+		     "mov %1, %%rdi\n\t"
+		     "movsb"
+		     : : "r"(&mmio_source), "r"(&npf_target)
+		     : "rsi", "rdi", "memory");
+	GUEST_FAIL("L2 MOVSB did not cause a nested page fault");
+}
+
+static void l2_pf_code(void)
+{
+	asm volatile("mov (%0), %%rax"
+		     : : "r"(PF_TEST_GVA) : "rax", "memory");
+	GUEST_FAIL("L2 access to an unmapped VA did not #PF");
+}
+
+static void l2_fep_pf_code(void)
+{
+	asm volatile(KVM_FEP "mov (%0), %%rax"
+		     : : "r"(PF_TEST_GVA) : "rax", "memory");
+	GUEST_FAIL("L2 forced-emulated access to an unmapped VA did not #PF");
+}
+
+static void l2_stale_emulator_pf_code(void)
+{
+	asm volatile("movb (%0), %%al\n\t"
+		     "nop"
+		     : : "r"(&mmio_source) : "rax", "memory");
+	GUEST_FAIL("Userspace-injected #PF was not intercepted by L1");
+}
+
+static void l2_mov_from_cr4_code(void)
+{
+	asm volatile("mov %%cr4, %%r10" : : : "r10");
+	GUEST_FAIL("L2 MOV-from-CR4 was not intercepted");
+}
+
+static void l2_fep_mov_from_cr4_code(void)
+{
+	asm volatile(KVM_FEP "mov %%cr4, %%r10" : : : "r10");
+	GUEST_FAIL("L2 forced-emulated MOV-from-CR4 was not intercepted");
+}
+
+static void l2_mov_to_cr4_code(void)
+{
+	asm volatile("mov %%cr4, %%rax\n\t"
+		     "mov %%rax, %%cr4" : : : "rax");
+	GUEST_FAIL("L2 MOV-to-CR4 was not intercepted");
+}
+
+static void l2_fep_mov_to_cr4_code(void)
+{
+	asm volatile("mov %%cr4, %%rax\n\t"
+		     KVM_FEP "mov %%rax, %%cr4" : : : "rax");
+	GUEST_FAIL("L2 forced-emulated MOV-to-CR4 was not intercepted");
+}
+
+static void l2_mov_to_dr7_code(void)
+{
+	asm volatile("mov %%dr7, %%rax\n\t"
+		     "mov %%rax, %%rbx\n\t"
+		     "mov %%rbx, %%dr7" : : : "rax", "rbx");
+	GUEST_FAIL("L2 MOV-to-DR7 was not intercepted");
+}
+
+static void l2_fep_mov_to_dr7_code(void)
+{
+	asm volatile("mov %%dr7, %%rax\n\t"
+		     "mov %%rax, %%rbx\n\t"
+		     KVM_FEP "mov %%rbx, %%dr7" : : : "rax", "rbx");
+	GUEST_FAIL("L2 forced-emulated MOV-to-DR7 was not intercepted");
+}
+
+static void l2_mov_from_dr7_code(void)
+{
+	asm volatile("mov %%dr7, %%r10" : : : "r10");
+	GUEST_FAIL("L2 MOV-from-DR7 was not intercepted");
+}
+
+static void l2_fep_mov_from_dr7_code(void)
+{
+	asm volatile(KVM_FEP "mov %%dr7, %%r10" : : : "r10");
+	GUEST_FAIL("L2 forced-emulated MOV-from-DR7 was not intercepted");
+}
+
+static void l2_clts_code(void)
+{
+	asm volatile("clts" : : : "memory");
+	GUEST_FAIL("L2 CLTS was not intercepted");
+}
+
+static void l2_fep_clts_code(void)
+{
+	asm volatile(KVM_FEP "clts" : : : "memory");
+	GUEST_FAIL("L2 forced-emulated CLTS was not intercepted");
+}
+
+static void l2_lmsw_code(void)
+{
+	asm volatile("smsw %%ax\n\t"
+		     "lmsw %%ax" : : : "rax", "memory");
+	GUEST_FAIL("L2 LMSW was not intercepted");
+}
+
+static void l2_fep_lmsw_code(void)
+{
+	asm volatile("smsw %%ax\n\t"
+		     KVM_FEP "lmsw %%ax" : : : "rax", "memory");
+	GUEST_FAIL("L2 forced-emulated LMSW was not intercepted");
+}
+
+static void l2_smsw_code(void)
+{
+	asm volatile("smsw %%ax" : : : "rax", "memory");
+	GUEST_FAIL("L2 SMSW was not intercepted");
+}
+
+static void l2_fep_smsw_code(void)
+{
+	asm volatile(KVM_FEP "smsw %%ax" : : : "rax", "memory");
+	GUEST_FAIL("L2 forced-emulated SMSW was not intercepted");
+}
+
+static void l2_int_code(void)
+{
+	asm volatile("int %0" : : "i"(TEST_INT_VECTOR));
+	GUEST_FAIL("L2 INTn was not intercepted");
+}
+
+static void l2_fep_int_code(void)
+{
+	asm volatile(KVM_FEP "int %0" : : "i"(TEST_INT_VECTOR));
+	GUEST_FAIL("L2 forced-emulated INTn was not intercepted");
+}
+
+static void l2_invlpg_code(void)
+{
+	asm volatile("invlpg (%0)" : : "r"(&npf_target) : "memory");
+	GUEST_FAIL("L2 INVLPG was not intercepted");
+}
+
+static void l2_fep_invlpg_code(void)
+{
+	asm volatile(KVM_FEP "invlpg (%0)" : : "r"(&npf_target) : "memory");
+	GUEST_FAIL("L2 forced-emulated INVLPG was not intercepted");
+}
+
+static void l2_invlpga_code(void)
+{
+	asm volatile("invlpga" : : "a"(&npf_target), "c"(0) : "memory");
+	GUEST_FAIL("L2 INVLPGA was not intercepted");
+}
+
+static void l2_fep_invlpga_code(void)
+{
+	asm volatile(KVM_FEP "invlpga"
+		     : : "a"(&npf_target), "c"(0) : "memory");
+	GUEST_FAIL("L2 forced-emulated INVLPGA was not intercepted");
+}
+
+struct instruction_intercept_test {
+	const char *name;
+	void (*code)(void);
+	void (*fep_code)(void);
+	u64 intercept;
+	u32 intercept_cr;
+	u32 intercept_dr;
+	u64 exit_code;
+	u64 exit_info_1;
+	u64 exit_info_1_mask;
+	bool check_rax;
+	u64 rax;
+};
+
+static const struct instruction_intercept_test instruction_intercept_tests[] = {
+	{
+		.name = "MOV-to-CR4",
+		.code = l2_mov_to_cr4_code,
+		.fep_code = l2_fep_mov_to_cr4_code,
+		.intercept_cr = BIT(INTERCEPT_CR4_WRITE),
+		.exit_code = SVM_EXIT_WRITE_CR4,
+		.exit_info_1 = BIT_ULL(63),
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "MOV-from-CR4",
+		.code = l2_mov_from_cr4_code,
+		.fep_code = l2_fep_mov_from_cr4_code,
+		.intercept_cr = BIT(INTERCEPT_CR4_READ),
+		.exit_code = SVM_EXIT_READ_CR4,
+		.exit_info_1 = BIT_ULL(63) | 10,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "MOV-to-DR7",
+		.code = l2_mov_to_dr7_code,
+		.fep_code = l2_fep_mov_to_dr7_code,
+		.intercept_dr = BIT(INTERCEPT_DR7_WRITE),
+		.exit_code = SVM_EXIT_WRITE_DR7,
+		.exit_info_1 = 3,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "MOV-from-DR7",
+		.code = l2_mov_from_dr7_code,
+		.fep_code = l2_fep_mov_from_dr7_code,
+		.intercept_dr = BIT(INTERCEPT_DR7_READ),
+		.exit_code = SVM_EXIT_READ_DR7,
+		.exit_info_1 = 10,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "CLTS",
+		.code = l2_clts_code,
+		.fep_code = l2_fep_clts_code,
+		.intercept_cr = BIT(INTERCEPT_CR0_WRITE),
+		.exit_code = SVM_EXIT_WRITE_CR0,
+		.exit_info_1_mask = BIT_ULL(63),
+	}, {
+		.name = "LMSW",
+		.code = l2_lmsw_code,
+		.fep_code = l2_fep_lmsw_code,
+		.intercept_cr = BIT(INTERCEPT_CR0_WRITE),
+		.exit_code = SVM_EXIT_WRITE_CR0,
+		.exit_info_1_mask = BIT_ULL(63),
+	}, {
+		.name = "SMSW",
+		.code = l2_smsw_code,
+		.fep_code = l2_fep_smsw_code,
+		.intercept_cr = BIT(INTERCEPT_CR0_READ),
+		.exit_code = SVM_EXIT_READ_CR0,
+		.exit_info_1_mask = BIT_ULL(63),
+	}, {
+		.name = "INTn",
+		.code = l2_int_code,
+		.fep_code = l2_fep_int_code,
+		.intercept = BIT_ULL(INTERCEPT_INTn),
+		.exit_code = SVM_EXIT_SWINT,
+		.exit_info_1 = TEST_INT_VECTOR,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "INVLPG",
+		.code = l2_invlpg_code,
+		.fep_code = l2_fep_invlpg_code,
+		.intercept = BIT_ULL(INTERCEPT_INVLPG),
+		.exit_code = SVM_EXIT_INVLPG,
+		.exit_info_1 = (u64)&npf_target,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "INVLPGA",
+		.code = l2_invlpga_code,
+		.fep_code = l2_fep_invlpga_code,
+		.intercept = BIT_ULL(INTERCEPT_INVLPGA),
+		.exit_code = SVM_EXIT_INVLPGA,
+		.exit_info_1_mask = ~0ULL,
+		.check_rax = true,
+		.rax = (u64)&npf_target,
+	},
+};
+
+static void assert_decode_assist_insn_bytes(struct vmcb *vmcb)
+{
+	GUEST_ASSERT(vmcb->control.insn_len);
+	GUEST_ASSERT(vmcb->control.insn_len <=
+		     sizeof(vmcb->control.insn_bytes));
+	GUEST_ASSERT(!memcmp(vmcb->control.insn_bytes,
+			     (void *)vmcb->save.rip,
+			     vmcb->control.insn_len));
+}
+
+static void assert_full_decode_assist_insn_bytes(struct vmcb *vmcb)
+{
+	GUEST_ASSERT_EQ(vmcb->control.insn_len,
+			sizeof(vmcb->control.insn_bytes));
+	assert_decode_assist_insn_bytes(vmcb);
+}
+
+static void prepare_l2_for_vmrun(struct svm_test_data *svm, gva_t rip)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	vmcb->save.rip = rip;
+	vmcb->save.rsp = (u64)svm->stack;
+}
+
+static void run_intercept_test(struct svm_test_data *svm,
+			       const struct instruction_intercept_test *test,
+			       bool synthesized)
+{
+	struct vmcb *vmcb = svm->vmcb;
+	struct vmcb_control_area *control = &vmcb->control;
+	const char *source = synthesized ? "synthesized" : "hardware";
+	u64 expected_exit_info_1 = test->exit_info_1 & test->exit_info_1_mask;
+
+	control->intercept |= test->intercept;
+	control->intercept_cr |= test->intercept_cr;
+	control->intercept_dr |= test->intercept_dr;
+
+	if (synthesized) {
+		control->exit_info_1 = ~0ULL;
+		control->exit_info_2 = ~0ULL;
+		prepare_l2_for_vmrun(svm, (u64)test->fep_code);
+	} else {
+		prepare_l2_for_vmrun(svm, (u64)test->code);
+	}
+
+	run_guest(vmcb, svm->vmcb_gpa);
+
+	__GUEST_ASSERT(control->exit_code == test->exit_code,
+		       "%s (%s): expected exit code %#lx, got %#lx",
+		       test->name, source, (unsigned long)test->exit_code,
+		       (unsigned long)control->exit_code);
+	__GUEST_ASSERT((control->exit_info_1 & test->exit_info_1_mask) ==
+		       expected_exit_info_1,
+		       "%s (%s): expected EXITINFO1 %#lx with mask %#lx, got %#lx",
+		       test->name, source, (unsigned long)expected_exit_info_1,
+		       (unsigned long)test->exit_info_1_mask,
+		       (unsigned long)control->exit_info_1);
+	__GUEST_ASSERT(!control->insn_len,
+		       "%s (%s): expected no instruction bytes, got %u",
+		       test->name, source, control->insn_len);
+
+	if (test->check_rax)
+		__GUEST_ASSERT(vmcb->save.rax == test->rax,
+			       "%s (%s): expected rAX %#lx, got %#lx",
+			       test->name, source, (unsigned long)test->rax,
+			       (unsigned long)vmcb->save.rax);
+
+	if (synthesized)
+		__GUEST_ASSERT(!control->exit_info_2,
+			       "%s (%s): expected EXITINFO2 to be clear, got %#lx",
+			       test->name, source,
+			       (unsigned long)control->exit_info_2);
+
+	control->intercept &= ~test->intercept;
+	control->intercept_cr &= ~test->intercept_cr;
+	control->intercept_dr &= ~test->intercept_dr;
+}
+
+static void test_instruction_intercepts(struct svm_test_data *svm)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(instruction_intercept_tests); i++) {
+		run_intercept_test(svm, &instruction_intercept_tests[i], false);
+
+		if (is_forced_emulation_enabled)
+			run_intercept_test(svm, &instruction_intercept_tests[i],
+					   true);
+	}
+}
+
+static void test_hardware_npf(struct svm_test_data *svm, gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(svm, (u64)l2_read_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	assert_decode_assist_insn_bytes(vmcb);
+}
+
+/*
+ * The IOIO intercept causes L0 to emulate OUTSB before accessing its source
+ * operand.  The emulated read then faults on L1's NPT, resulting in a
+ * KVM-synthesized #NPF.
+ */
+static void test_synthesized_npf(struct svm_test_data *svm, gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(svm, (u64)l2_outsb_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	assert_full_decode_assist_insn_bytes(vmcb);
+}
+
+/*
+ * MOVSB first reads from MMIO, causing a hardware #NPF that L0 emulates.
+ * After userspace completes the read, the emulated destination write faults
+ * on L1's NPT.  The new #NPF must not reuse the original hardware exit's GPA.
+ */
+static void test_synthesized_npf_after_hardware_npf(struct svm_test_data *svm,
+						    gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(svm, (u64)l2_movsb_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	assert_full_decode_assist_insn_bytes(vmcb);
+}
+
+/*
+ * OUTSB is the final byte of a mapped code page, and the following page is
+ * not present in L2's page tables.  DecodeAssist byte fetching must stop at
+ * the page boundary and report only the OUTSB opcode.
+ */
+static void test_synthesized_npf_truncated(struct svm_test_data *svm,
+					   gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(svm,
+			     (u64)&boundary_outsb_code[PAGE_SIZE -
+						       BOUNDARY_OUTSB_CODE_SIZE]);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	GUEST_ASSERT_EQ(vmcb->save.rip,
+			(u64)&boundary_outsb_code[PAGE_SIZE - 1]);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len, 1);
+	GUEST_ASSERT_EQ(vmcb->control.insn_bytes[0], OUTSB_OPCODE);
+}
+
+/*
+ * OUTSB is the final byte in the lower canonical address range.  The first
+ * byte after OUTSB is non-canonical, and must terminate DecodeAssist fetching
+ * even though the corresponding high canonical address is mapped.
+ */
+static void test_synthesized_npf_canonical_boundary(struct svm_test_data *svm,
+						    gpa_t npf_gpa,
+						    gva_t code_gva)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(svm, code_gva + PAGE_SIZE -
+				  BOUNDARY_OUTSB_CODE_SIZE);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	GUEST_ASSERT_EQ(vmcb->save.rip, code_gva + PAGE_SIZE - 1);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len, 1);
+	GUEST_ASSERT_EQ(vmcb->control.insn_bytes[0], OUTSB_OPCODE);
+}
+
+/*
+ * A nonzero CS.base makes the DecodeAssist byte window cross the 32-bit
+ * linear-address boundary without crossing the CS limit.
+ */
+static void test_synthesized_npf_linear_wrap(struct svm_test_data *svm,
+					     gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+	u16 cs_attrib = vmcb->save.cs.attrib;
+	u64 cs_base = vmcb->save.cs.base;
+	u32 cs_limit = vmcb->save.cs.limit;
+	u32 outsb_eip = LINEAR_WRAP_CODE_GVA + LINEAR_WRAP_OUTSB_OFFSET -
+			LINEAR_WRAP_CS_BASE;
+	u32 setup_eip = LINEAR_WRAP_CODE_GVA + LINEAR_WRAP_SETUP_OFFSET -
+			LINEAR_WRAP_CS_BASE;
+
+	vmcb->save.cs.attrib &= ~SVM_SELECTOR_L_MASK;
+	vmcb->save.cs.attrib |= SVM_SELECTOR_DB_MASK;
+	vmcb->save.cs.base = LINEAR_WRAP_CS_BASE;
+	vmcb->save.cs.limit = UINT32_MAX;
+	prepare_l2_for_vmrun(svm, setup_eip);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	GUEST_ASSERT_EQ(vmcb->save.rip, outsb_eip);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len,
+			sizeof(linear_wrap_insn_bytes));
+	GUEST_ASSERT(!memcmp(vmcb->control.insn_bytes,
+			     linear_wrap_insn_bytes,
+			     sizeof(linear_wrap_insn_bytes)));
+	vmcb->save.cs.attrib = cs_attrib;
+	vmcb->save.cs.base = cs_base;
+	vmcb->save.cs.limit = cs_limit;
+}
+
+static void test_hardware_intercepted_pf(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(svm, (u64)l2_pf_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_EXCP_BASE + PF_VECTOR);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, PF_TEST_GVA);
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_PRESENT_MASK));
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_FETCH_MASK));
+	assert_decode_assist_insn_bytes(vmcb);
+}
+
+static void test_hardware_intercepted_fetch_pf(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(svm, PF_FETCH_TEST_GVA);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_EXCP_BASE + PF_VECTOR);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, PF_FETCH_TEST_GVA);
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_PRESENT_MASK));
+	GUEST_ASSERT(vmcb->control.exit_info_1 & PFERR_FETCH_MASK);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len, 0);
+}
+
+static void test_synthesized_pf(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	if (!is_forced_emulation_enabled)
+		return;
+
+	prepare_l2_for_vmrun(svm, (u64)l2_fep_pf_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_EXCP_BASE + PF_VECTOR);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, PF_TEST_GVA);
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_PRESENT_MASK));
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_FETCH_MASK));
+	assert_full_decode_assist_insn_bytes(vmcb);
+}
+
+/*
+ * Inject #PF while an emulated MMIO read is awaiting completion.  KVM
+ * completes the instruction before constructing the nested VM-Exit, so the
+ * old emulator fetch cache must not be reported for the following instruction.
+ */
+static void test_userspace_injected_pf_during_emulation(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(svm, (u64)l2_stale_emulator_pf_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_EXCP_BASE + PF_VECTOR);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, PF_TEST_GVA);
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_FETCH_MASK));
+	assert_full_decode_assist_insn_bytes(vmcb);
+}
+
+static void l1_guest_code(struct svm_test_data *svm, gpa_t npf_gpa,
+			  gpa_t iopm_gpa, gva_t canonical_code_gva)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	GUEST_ASSERT(this_cpu_has(X86_FEATURE_DECODEASSISTS));
+
+	generic_svm_setup(svm, l2_read_code);
+	vmcb->control.iopm_base_pa = iopm_gpa;
+
+	vmcb->control.intercept |= BIT_ULL(INTERCEPT_IOIO_PROT);
+	vmcb->control.intercept_exceptions |= 1U << PF_VECTOR;
+
+	test_hardware_npf(svm, npf_gpa);
+	test_synthesized_npf(svm, npf_gpa);
+	test_synthesized_npf_after_hardware_npf(svm, npf_gpa);
+	test_synthesized_npf_truncated(svm, npf_gpa);
+	test_synthesized_npf_canonical_boundary(svm, npf_gpa,
+						canonical_code_gva);
+	test_synthesized_npf_linear_wrap(svm, npf_gpa);
+	test_hardware_intercepted_pf(svm);
+	test_hardware_intercepted_fetch_pf(svm);
+	test_synthesized_pf(svm);
+	test_userspace_injected_pf_during_emulation(svm);
+	test_instruction_intercepts(svm);
+
+	GUEST_DONE();
+}
+
+static void build_boundary_outsb_code(u8 *code)
+{
+	u64 source = (u64)&npf_target;
+
+	/* movabs $npf_target, %rsi */
+	code[0] = 0x48;
+	code[1] = 0xbe;
+	memcpy(&code[2], &source, sizeof(source));
+
+	/* mov $0x80, %dx; outsb */
+	code[10] = 0x66;
+	code[11] = 0xba;
+	code[12] = 0x80;
+	code[13] = 0x00;
+	code[14] = OUTSB_OPCODE;
+}
+
+static void prepare_boundary_outsb_code(struct kvm_vm *vm)
+{
+	gva_t code_gva = (gva_t)&boundary_outsb_code[PAGE_SIZE -
+						      BOUNDARY_OUTSB_CODE_SIZE];
+
+	build_boundary_outsb_code(addr_gva2hva(vm, code_gva));
+}
+
+static gva_t prepare_canonical_boundary_outsb_code(struct kvm_vm *vm)
+{
+	gva_t backing_gva = vm_alloc_pages(vm, 2);
+	u8 *code_page = addr_gva2hva(vm, backing_gva);
+	u8 *alias_page = addr_gva2hva(vm, backing_gva + PAGE_SIZE);
+	gpa_t code_gpa = addr_gva2gpa(vm, backing_gva);
+	gpa_t alias_gpa = addr_gva2gpa(vm, backing_gva + PAGE_SIZE);
+	gva_t code_gva = BIT_ULL(vm->va_bits - 1) - PAGE_SIZE;
+	gva_t alias_gva = ~(BIT_ULL(vm->va_bits - 1) - 1);
+	u8 *code = &code_page[PAGE_SIZE - BOUNDARY_OUTSB_CODE_SIZE];
+
+	build_boundary_outsb_code(code);
+	memset(alias_page, 0xcc, PAGE_SIZE);
+	virt_map(vm, code_gva, code_gpa, 1);
+	virt_map(vm, alias_gva, alias_gpa, 1);
+
+	return code_gva;
+}
+
+static void prepare_linear_wrap_outsb_code(struct kvm_vm *vm)
+{
+	gva_t code_gva = vm_alloc_pages(vm, 2);
+	u8 *high_page = addr_gva2hva(vm, code_gva);
+	u8 *low_page = addr_gva2hva(vm, code_gva + PAGE_SIZE);
+	gpa_t high_gpa = addr_gva2gpa(vm, code_gva);
+	gpa_t low_gpa = addr_gva2gpa(vm, code_gva + PAGE_SIZE);
+	u32 source = (u32)(u64)&npf_target;
+	u8 *setup = &high_page[LINEAR_WRAP_SETUP_OFFSET];
+
+	TEST_ASSERT((u64)&npf_target <= UINT32_MAX,
+		    "npf_target must be addressable from compatibility mode");
+
+	/* mov $npf_target, %esi; mov $0x80, %dx */
+	setup[0] = 0xbe;
+	memcpy(&setup[1], &source, sizeof(source));
+	setup[5] = 0x66;
+	setup[6] = 0xba;
+	setup[7] = 0x80;
+	setup[8] = 0x00;
+
+	memcpy(&high_page[LINEAR_WRAP_OUTSB_OFFSET],
+	       linear_wrap_insn_bytes,
+	       PAGE_SIZE - LINEAR_WRAP_OUTSB_OFFSET);
+	memcpy(low_page,
+	       &linear_wrap_insn_bytes[PAGE_SIZE - LINEAR_WRAP_OUTSB_OFFSET],
+	       sizeof(linear_wrap_insn_bytes) -
+	       (PAGE_SIZE - LINEAR_WRAP_OUTSB_OFFSET));
+
+	virt_map(vm, LINEAR_WRAP_CODE_GVA, high_gpa, 1);
+	virt_map(vm, 0, low_gpa, 1);
+}
+
+static void queue_userspace_pf(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_events events;
+
+	vcpu_events_get(vcpu, &events);
+	TEST_ASSERT(!events.exception.pending && !events.exception.injected,
+		    "Unexpected exception queued before userspace #PF injection");
+	TEST_ASSERT(events.flags & KVM_VCPUEVENT_VALID_PAYLOAD,
+		    "KVM_CAP_EXCEPTION_PAYLOAD was not enabled");
+
+	events.flags |= KVM_VCPUEVENT_VALID_PAYLOAD;
+	events.exception.injected = false;
+	events.exception.pending = true;
+	events.exception.nr = PF_VECTOR;
+	events.exception.has_error_code = true;
+	events.exception.error_code = 0;
+	events.exception_has_payload = true;
+	events.exception_payload = PF_TEST_GVA;
+	vcpu_events_set(vcpu, &events);
+}
+
+static void complete_mmio_read(struct kvm_vcpu *vcpu, gpa_t expected_gpa,
+			       u8 value)
+{
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MMIO);
+	TEST_ASSERT(!vcpu->run->mmio.is_write,
+		    "Expected an MMIO read, got a write");
+	TEST_ASSERT_EQ(vcpu->run->mmio.phys_addr, expected_gpa);
+	TEST_ASSERT_EQ(vcpu->run->mmio.len, 1);
+	vcpu->run->mmio.data[0] = value;
+}
+
+static void assert_ucall_done(struct kvm_vcpu *vcpu)
+{
+	struct ucall uc;
+	u64 actual;
+
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	actual = get_ucall(vcpu, &uc);
+	if (actual == UCALL_ABORT)
+		REPORT_GUEST_ASSERT(uc);
+
+	TEST_ASSERT_EQ(actual, UCALL_DONE);
+}
+
+int main(int argc, char *argv[])
+{
+	gva_t svm_gva, npf_gva, boundary_page_gva, iopm_gva;
+	gva_t canonical_code_gva;
+	gpa_t npf_gpa, mmio_source_gpa, mmio_gpa, iopm_gpa;
+	struct userspace_mem_region *region;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	u64 *pte;
+
+	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM));
+	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_NPT));
+	TEST_REQUIRE(this_cpu_has(X86_FEATURE_DECODEASSISTS));
+	TEST_ASSERT(kvm_cpu_has(X86_FEATURE_DECODEASSISTS),
+		    "KVM failed to expose DecodeAssists");
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_EXCEPTION_PAYLOAD));
+
+	vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
+	vm_enable_cap(vm, KVM_CAP_EXCEPTION_PAYLOAD, 1);
+	prepare_boundary_outsb_code(vm);
+	canonical_code_gva = prepare_canonical_boundary_outsb_code(vm);
+	prepare_linear_wrap_outsb_code(vm);
+	vm_enable_npt(vm);
+	vcpu_alloc_svm(vm, &svm_gva);
+	iopm_gva = vm_alloc_pages(vm, TEST_IOPM_SIZE / PAGE_SIZE);
+	iopm_gpa = addr_gva2gpa(vm, iopm_gva);
+	memset(addr_gva2hva(vm, iopm_gva), 0, TEST_IOPM_SIZE);
+	npf_gva = (gva_t)&npf_target;
+	npf_gpa = addr_gva2gpa(vm, npf_gva);
+
+	tdp_identity_map_default_memslots(vm);
+	pte = tdp_get_pte(vm, npf_gpa);
+	*pte &= ~PTE_PRESENT_MASK(&vm->stage2_mmu);
+	region = memslot2region(vm, 0);
+	mmio_gpa = region->region.guest_phys_addr +
+		   region->region.memory_size + PAGE_SIZE;
+	mmio_source_gpa = addr_gva2gpa(vm, (gva_t)&mmio_source);
+	pte = tdp_get_pte(vm, mmio_source_gpa);
+	*pte = (*pte & ~PHYSICAL_PAGE_MASK) | mmio_gpa;
+
+	boundary_page_gva = (gva_t)&boundary_outsb_code[PAGE_SIZE];
+	pte = vm_get_pte(vm, boundary_page_gva);
+	*pte &= ~PTE_PRESENT_MASK(&vm->mmu);
+
+	vcpu_args_set(vcpu, 4, svm_gva, npf_gpa, iopm_gpa,
+		      canonical_code_gva);
+
+	/* Complete the MMIO source read in the MOVSB #NPF regression test. */
+	vcpu_run(vcpu);
+	complete_mmio_read(vcpu, mmio_gpa, 0xa5);
+
+	/* Leave the second MMIO read pending while injecting #PF. */
+	vcpu_run(vcpu);
+	complete_mmio_read(vcpu, mmio_gpa, 0x5a);
+	queue_userspace_pf(vcpu);
+	vcpu_run(vcpu);
+	assert_ucall_done(vcpu);
+
+	kvm_vm_free(vm);
+	return 0;
+}
-- 
2.43.7

Re: [PATCH v4 3/9] KVM: nSVM: Track hardware-provided instruction bytes

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The `vmcb02_insn_bytes_fresh` flag unconditionally marks DecodeAssist instruction bytes as fresh for all reflected hardware VM-exits, which incorrectly leaks stale VMCB02 bytes to L1 if multiple L0 exits occur during a single nested run.
--

--- Patch [3]: [PATCH v4 3/9] KVM: nSVM: Track hardware-provided instruction bytes ---
commit f8e0d3f413f3670a241d2de3e149f5cbd8843299
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: nSVM: Track hardware-provided instruction bytes

This commit clears the VMCB02 instruction-byte fields before each nested run
and tracks whether a hardware VM-Exit being reflected to L1 includes fresh
instruction bytes. This tracking is intended to distinguish bytes produced by
the current hardware VM-Exit from stale VMCB02 state.

> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index 73f37b050d0a0..d08c30e9a6f44 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c

[ ... ]

> @@ -868,7 +876,10 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
>  	/*
>  	 * Filled at exit: exit_code, exit_info_1, exit_info_2, exit_int_info,
>  	 * exit_int_info_err, next_rip, insn_len, insn_bytes.
> +	 * Clear stale DecodeAssist data before L2 runs.
>  	 */
> +	nested_svm_clear_insn_bytes(vmcb02);
> +	svm->nested.vmcb02_insn_bytes_fresh = false;

[Severity: High]
Does this only clear the instruction bytes during the initial L1 to L2 VMRUN?

If L0 handles an intermediate exit (like a nested page fault) and resumes L2
without re-clearing vmcb02, could the instruction bytes from that intermediate
exit be left behind?

[ ... ]

> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index c7c1f1527c3cf..b85e43112db9e 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c

[ ... ]

> @@ -1953,14 +1953,20 @@ static void svm_set_dr7(struct kvm_vcpu *vcpu, unsigned long value)
>  static int pf_interception(struct kvm_vcpu *vcpu)
>  {
>  	struct vcpu_svm *svm = to_svm(vcpu);
> -
>  	u64 fault_address = svm->vmcb->control.exit_info_2;
>  	u64 error_code = svm->vmcb->control.exit_info_1;
> +	int r;
> +
> +	r = kvm_handle_page_fault(vcpu, error_code, fault_address,
> +				  static_cpu_has(X86_FEATURE_DECODEASSISTS) ?
> +				  svm->vmcb->control.insn_bytes : NULL,
> +				  svm->vmcb->control.insn_len);
>  
> -	return kvm_handle_page_fault(vcpu, error_code, fault_address,
> -			static_cpu_has(X86_FEATURE_DECODEASSISTS) ?
> -			svm->vmcb->control.insn_bytes : NULL,
> -			svm->vmcb->control.insn_len);
> +	if (is_guest_mode(vcpu) && vcpu->arch.exception_vmexit.pending &&
> +	    vcpu->arch.exception_vmexit.vector == PF_VECTOR)
> +		svm->nested.vmcb02_insn_bytes_fresh = true;

[Severity: High]
Does this unconditionally set the flag to true for all #PF exits that are
pending for L1?

Hardware doesn't provide DecodeAssist for instruction fetch faults. If an L2
guest triggers a data NPF followed immediately by a fetch fault, could this
leak stale instruction bytes from the data NPF to L1?

> +
> +	return r;
>  }

[ ... ]

> @@ -3723,7 +3729,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
>  		vmexit = nested_svm_exit_special(svm);
>  
>  		if (vmexit == NESTED_EXIT_CONTINUE)
> -			vmexit = nested_svm_exit_handled(svm);
> +			vmexit = nested_svm_exit_handled(svm, true);

[Severity: High]
Does unconditionally passing true here ignore whether the hardware actually
provided DecodeAssist for this exit?

If the exit didn't populate DecodeAssist, could this pass stale bytes from a
previous exit to L1?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787116250.git.zhang_wei@open-hieco.net?part=3

Re: [PATCH v4 2/9] KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] The population of EXITINFO1 and EXITINFO2 for INVLPG and INVLPGA is incorrectly gated behind the X86_FEATURE_DECODEASSISTS CPU capability.
--

--- Patch [2]: [PATCH v4 2/9] KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts ---
commit 03ef081906dd422c2f595e7c28afbf6acdb25b23
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts

This commit populates the architectural EXITINFO fields during an emulated
L1 intercept when DecodeAssists is exposed to L1. It updates EXITINFO1 and
EXITINFO2 for several instructions including MOV CR/DR, INTn, and INVLPG.

> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -4810,6 +4810,52 @@ static const struct __x86_intercept {
>  #undef POST_EX
>  #undef POST_MEM
>  
> +static void svm_prepare_decode_assist_exit_info(struct kvm_vcpu *vcpu,
> +						const struct x86_instruction_info *info)
> +{
> +	struct vmcb *vmcb = to_svm(vcpu)->vmcb;
> +	u64 exit_info_1;
> +
> +	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
> +		return;

[Severity: High]
Does this early return prevent INVLPG and INVLPGA from updating EXITINFO1 and
EXITINFO2 when the guest lacks the Decode Assists feature?

According to the AMD APM, the INVLPG and INVLPGA intercepts populate these
fields unconditionally. If KVM does not expose Decode Assists to the L1
hypervisor, this return might leave EXITINFO1 uninitialized or stale.

When an L2 guest executes INVLPG under these conditions, could the L1
hypervisor receive an incorrect address and flush the wrong shadow page table
entry?

> +
> +	switch (info->intercept) {
> +	case x86_intercept_cr_read:

[ ... ]

> +	case x86_intercept_intn:
> +		/* INTn: software interrupt number in bits 7:0. */
> +		exit_info_1 = info->src_val & 0xff;
> +		break;
> +	case x86_intercept_invlpg:
> +		/* INVLPG: linear address of the target page. */
> +		exit_info_1 = info->intercept_linear_addr;
> +		break;
> +	case x86_intercept_invlpga:
> +		/* INVLPGA: the address remains available in guest rAX. */
> +		exit_info_1 = 0;
> +		break;
> +	default:
> +		return;
> +	}
> +
> +	vmcb->control.exit_info_1 = exit_info_1;
> +	vmcb->control.exit_info_2 = 0;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787116250.git.zhang_wei@open-hieco.net?part=2

Re: [PATCH v4 9/9] KVM: selftests: Add nested SVM DecodeAssists test

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Test logic codifies a KVM emulator bug regarding linear address wrapping in compatibility mode.
- [Medium] Guest assertion failures are silently masked by MMIO exit assertions.
--

--- Patch [9]: [PATCH v4 9/9] KVM: selftests: Add nested SVM DecodeAssists test ---
commit 05cea14bcd0202443d9a4c1a4e002383cf66eb03
Author: Tina Zhang <zhang_wei@open-hieco.net>

    KVM: selftests: Add nested SVM DecodeAssists test
    
    Add a nested SVM selftest for DecodeAssists.  Verify that KVM exposes
    the feature to L1 and provides architectural exit state for MOV CR/DR,
    CLTS, LMSW, SMSW, INTn, INVLPG, and INVLPGA intercepts.

> diff --git a/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c b/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c
> new file mode 100644
> index 0000000000000..13511ca8473b2
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c
> @@ -0,0 +1,791 @@

[ ... ]

> +static void test_synthesized_npf_linear_wrap(struct svm_test_data *svm,
> +					     gpa_t npf_gpa)
> +{
> +	struct vmcb *vmcb = svm->vmcb;
> +	u16 cs_attrib = vmcb->save.cs.attrib;
> +	u64 cs_base = vmcb->save.cs.base;
> +	u32 cs_limit = vmcb->save.cs.limit;
> +	u32 outsb_eip = LINEAR_WRAP_CODE_GVA + LINEAR_WRAP_OUTSB_OFFSET -
> +			LINEAR_WRAP_CS_BASE;
> +	u32 setup_eip = LINEAR_WRAP_CODE_GVA + LINEAR_WRAP_SETUP_OFFSET -
> +			LINEAR_WRAP_CS_BASE;
> +
> +	vmcb->save.cs.attrib &= ~SVM_SELECTOR_L_MASK;
> +	vmcb->save.cs.attrib |= SVM_SELECTOR_DB_MASK;
> +	vmcb->save.cs.base = LINEAR_WRAP_CS_BASE;
> +	vmcb->save.cs.limit = UINT32_MAX;
> +	prepare_l2_for_vmrun(svm, setup_eip);
> +	run_guest(vmcb, svm->vmcb_gpa);
> +	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
> +	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
> +	GUEST_ASSERT_EQ(vmcb->save.rip, outsb_eip);
> +	GUEST_ASSERT_EQ(vmcb->control.insn_len,
> +			sizeof(linear_wrap_insn_bytes));
> +	GUEST_ASSERT(!memcmp(vmcb->control.insn_bytes,
> +			     linear_wrap_insn_bytes,
> +			     sizeof(linear_wrap_insn_bytes)));

[Severity: Medium]
Will this assertion artificially fail if KVM's address generation is fixed to
comply with the architecture?

In compatibility mode, the 32-bit effective address and 64-bit segment base
are added to form a 64-bit linear address, which does not wrap at 4GB
according to AMD APM Vol 2 and Intel SDM. KVM's emulator currently
incorrectly masks the linear address to 32 bits for instruction fetches
outside of 64-bit mode.

[ ... ]

> +static void prepare_linear_wrap_outsb_code(struct kvm_vm *vm)
> +{
> +	gva_t code_gva = vm_alloc_pages(vm, 2);
> +	u8 *high_page = addr_gva2hva(vm, code_gva);
> +	u8 *low_page = addr_gva2hva(vm, code_gva + PAGE_SIZE);
> +	gpa_t high_gpa = addr_gva2gpa(vm, code_gva);
> +	gpa_t low_gpa = addr_gva2gpa(vm, code_gva + PAGE_SIZE);
> +	u32 source = (u32)(u64)&npf_target;
> +	u8 *setup = &high_page[LINEAR_WRAP_SETUP_OFFSET];
> +
> +	TEST_ASSERT((u64)&npf_target <= UINT32_MAX,
> +		    "npf_target must be addressable from compatibility mode");
> +
> +	/* mov $npf_target, %esi; mov $0x80, %dx */
> +	setup[0] = 0xbe;
> +	memcpy(&setup[1], &source, sizeof(source));
> +	setup[5] = 0x66;
> +	setup[6] = 0xba;
> +	setup[7] = 0x80;
> +	setup[8] = 0x00;
> +
> +	memcpy(&high_page[LINEAR_WRAP_OUTSB_OFFSET],
> +	       linear_wrap_insn_bytes,
> +	       PAGE_SIZE - LINEAR_WRAP_OUTSB_OFFSET);
> +	memcpy(low_page,
> +	       &linear_wrap_insn_bytes[PAGE_SIZE - LINEAR_WRAP_OUTSB_OFFSET],
> +	       sizeof(linear_wrap_insn_bytes) -
> +	       (PAGE_SIZE - LINEAR_WRAP_OUTSB_OFFSET));
> +
> +	virt_map(vm, LINEAR_WRAP_CODE_GVA, high_gpa, 1);
> +	virt_map(vm, 0, low_gpa, 1);

[Severity: Medium]
Does mapping GVA 0 here explicitly rely on the emulator's non-architectural
32-bit wrap to provide the instruction bytes?

If the test is run on a fixed KVM, the true linear address at 0x100000000
will be unmapped.

[ ... ]

> +static void complete_mmio_read(struct kvm_vcpu *vcpu, gpa_t expected_gpa,
> +			       u8 value)
> +{
> +	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_MMIO);

[Severity: Medium]
Does this assert MMIO without checking for a prior guest assertion?

If a test fails and triggers a KVM_EXIT_IO with UCALL_ABORT, we will hit this
assertion and mask the true error payload.

[ ... ]

> +int main(int argc, char *argv[])
> +{
> +	gva_t svm_gva, npf_gva, boundary_page_gva, iopm_gva;
> +	gva_t canonical_code_gva;
> +	gpa_t npf_gpa, mmio_source_gpa, mmio_gpa, iopm_gpa;
> +	struct userspace_mem_region *region;
> +	struct kvm_vcpu *vcpu;
> +	struct kvm_vm *vm;
> +	u64 *pte;
> +
> +	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM));
> +	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_NPT));
> +	TEST_REQUIRE(this_cpu_has(X86_FEATURE_DECODEASSISTS));
> +	TEST_ASSERT(kvm_cpu_has(X86_FEATURE_DECODEASSISTS),
> +		    "KVM failed to expose DecodeAssists");
> +	TEST_REQUIRE(kvm_has_cap(KVM_CAP_EXCEPTION_PAYLOAD));
> +
> +	vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
> +	vm_enable_cap(vm, KVM_CAP_EXCEPTION_PAYLOAD, 1);
> +	prepare_boundary_outsb_code(vm);
> +	canonical_code_gva = prepare_canonical_boundary_outsb_code(vm);
> +	prepare_linear_wrap_outsb_code(vm);
> +	vm_enable_npt(vm);
> +	vcpu_alloc_svm(vm, &svm_gva);
> +	iopm_gva = vm_alloc_pages(vm, TEST_IOPM_SIZE / PAGE_SIZE);
> +	iopm_gpa = addr_gva2gpa(vm, iopm_gva);
> +	memset(addr_gva2hva(vm, iopm_gva), 0, TEST_IOPM_SIZE);
> +	npf_gva = (gva_t)&npf_target;
> +	npf_gpa = addr_gva2gpa(vm, npf_gva);
> +
> +	tdp_identity_map_default_memslots(vm);
> +	pte = tdp_get_pte(vm, npf_gpa);
> +	*pte &= ~PTE_PRESENT_MASK(&vm->stage2_mmu);
> +	region = memslot2region(vm, 0);
> +	mmio_gpa = region->region.guest_phys_addr +
> +		   region->region.memory_size + PAGE_SIZE;
> +	mmio_source_gpa = addr_gva2gpa(vm, (gva_t)&mmio_source);
> +	pte = tdp_get_pte(vm, mmio_source_gpa);
> +	*pte = (*pte & ~PHYSICAL_PAGE_MASK) | mmio_gpa;
> +
> +	boundary_page_gva = (gva_t)&boundary_outsb_code[PAGE_SIZE];
> +	pte = vm_get_pte(vm, boundary_page_gva);
> +	*pte &= ~PTE_PRESENT_MASK(&vm->mmu);
> +
> +	vcpu_args_set(vcpu, 4, svm_gva, npf_gpa, iopm_gpa,
> +		      canonical_code_gva);
> +
> +	/* Complete the MMIO source read in the MOVSB #NPF regression test. */
> +	vcpu_run(vcpu);
> +	complete_mmio_read(vcpu, mmio_gpa, 0xa5);

[Severity: Medium]
If any test preceding this MMIO read (e.g. test_hardware_npf()) fails, the
vcpu will exit early. Will that hit the generic MMIO check in
complete_mmio_read() and lose the REPORT_GUEST_ASSERT output, making it
difficult to debug?

Could the test explicitly check for UCALL_ABORT when encountering KVM_EXIT_IO
before asserting on MMIO?

> +
> +	/* Leave the second MMIO read pending while injecting #PF. */
> +	vcpu_run(vcpu);
> +	complete_mmio_read(vcpu, mmio_gpa, 0x5a);
> +	queue_userspace_pf(vcpu);
> +	vcpu_run(vcpu);
> +	assert_ucall_done(vcpu);
> +
> +	kvm_vm_free(vm);
> +	return 0;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787116250.git.zhang_wei@open-hieco.net?part=9

Re: [PATCH v4 1/9] KVM: x86: Add helper to provide intercept linear addresses

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The newly added `intercept_linear_addr` field is calculated but never consumed by architecture-specific intercept handlers, rendering the logic dead code.
- [Medium] The calculated linear address in `get_intercept_linear_addr()` is not untagged, violating architectural specifications when LAM/UAI is active.
--

--- Patch [1]: [PATCH v4 1/9] KVM: x86: Add helper to provide intercept linear addresses ---
commit bde7560cfe988f666acaf2bb0b4fbe6285414375
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: x86: Add helper to provide intercept linear addresses

This commit adds a helper get_intercept_linear_addr() to compute the linear
address for intercepts like INVLPG, and passes this address through
x86_instruction_info so intercept handlers can use it directly.

> diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
> index 8071b372d233e..a0e57b64cadd4 100644
> --- a/arch/x86/kvm/emulate.c
> +++ b/arch/x86/kvm/emulate.c
[ ... ]
> @@ -410,6 +410,26 @@ static int em_salc(struct x86_emulate_ctxt *ctxt)
[ ... ]
> +static u64 get_intercept_linear_addr(struct x86_emulate_ctxt *ctxt,
> +				     enum x86_intercept intercept)
> +{
> +	u64 la;
> +
> +	if (intercept != x86_intercept_invlpg)
> +		return 0;
> +
> +	la = seg_base(ctxt, ctxt->src.addr.mem.seg) + ctxt->src.addr.mem.ea;
> +	return ctxt->mode == X86EMUL_MODE_PROT64 ? la : (u32)la;

[Severity: Medium]
Should this address be untagged before being returned in 64-bit mode?

According to the Intel SDM and AMD APM, VM exits reporting a linear address
like INVLPG must report the address after masking tag bits if Linear Address
Masking (LAM) or Upper Address Ignore (UAI) is active.

Would it be necessary to pass this through ctxt->ops->get_untagged_addr()
here to ensure the resulting address is architecturally compliant?

> +}
> +
>  static int emulator_check_intercept(struct x86_emulate_ctxt *ctxt,
>  				    enum x86_intercept intercept,
>  				    enum x86_intercept_stage stage)
> @@ -427,6 +447,7 @@ static int emulator_check_intercept(struct x86_emulate_ctxt *ctxt,
>  		.src_type   = ctxt->src.type,
>  		.dst_type   = ctxt->dst.type,
>  		.ad_bytes   = ctxt->ad_bytes,
> +		.intercept_linear_addr = get_intercept_linear_addr(ctxt, intercept),

[Severity: Medium]
Is this new field actually consumed by the architecture-specific handlers?

Looking at svm_check_intercept() and vmx_check_intercept(), it doesn't
appear that either handler reads the intercept_linear_addr field from
x86_instruction_info.

If this field remains unused, does the original issue with nested
hypervisors receiving an incorrect INVLPG address remain unfixed?

>  		.rip	    = ctxt->eip,
>  		.next_rip   = ctxt->_eip,
>  	};

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787116250.git.zhang_wei@open-hieco.net?part=1

Re: [PATCH v4 4/9] KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Missing live migration serialization for DecodeAssist instruction bytes and desynchronization of vmcb02_insn_bytes_fresh flag.
--

--- Patch [4]: [PATCH v4 4/9] KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12 ---
commit 84578efc5f5e5753c950d333f88e1748a68f8d8c
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12

DecodeAssists provides instruction bytes for nested page faults and
intercepted page faults caused by data accesses.  When the feature is
exposed to L1, copy fresh hardware-provided instruction bytes from VMCB02
to VMCB12 for these exits.

> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index d08c30e9a6f44..d34370b84d5b3 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c

[ ... ]

> +static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
> +						struct vmcb *vmcb12,
> +						const struct vmcb *vmcb02)
> +{
> +	struct vcpu_svm *svm = to_svm(vcpu);
> +
> +	nested_svm_clear_insn_bytes(vmcb12);
> +
> +	if (!nested_svm_vmexit_supports_insn_bytes(vcpu, vmcb02))
> +		goto out;
> +
> +	if (svm->nested.vmcb02_insn_bytes_fresh)
> +		nested_svm_copy_insn_bytes(vmcb12, vmcb02);

[Severity: High]
If a live migration occurs while KVM has a pending hardware #PF VM-exit
(where exception_vmexit.pending = true and vmcb02_insn_bytes_fresh = true),
how does the destination hypervisor know to restore this state?

It appears svm_get_nested_state() and svm_set_nested_state() aren't updated
to serialize vmcb02_insn_bytes_fresh or the hardware insn_bytes. Could this
cause the destination hypervisor to inject a #PF into VMCB12 with an
insn_len of 0, effectively losing the instruction bytes?

Additionally, if userspace calls KVM_SET_VCPU_EVENTS to clear a pending
exception:

arch/x86/kvm/x86.c:kvm_vcpu_ioctl_x86_set_vcpu_events() {
    ...
    vcpu->arch.exception_vmexit.pending = false;
    ...
}

The vmcb02_insn_bytes_fresh flag is not cleared alongside it. Could a
subsequent emulator-generated #PF then incorrectly copy these stale
instruction bytes into VMCB12?

> +
> +out:
> +	svm->nested.vmcb02_insn_bytes_fresh = false;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787116250.git.zhang_wei@open-hieco.net?part=4

Re: [PATCH v4 5/9] KVM: x86: Track emulator-originated nested #PF VM-Exits

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] The newly added `has_emulator_context` field is left uninitialized or improperly cleared when queueing non-nested exceptions and restoring state from userspace.
- [Low] The `has_emulator_context` field is populated but never read anywhere in the codebase.
--

--- Patch [5]: [PATCH v4 5/9] KVM: x86: Track emulator-originated nested #PF VM-Exits ---
commit 224b6e38736ada4f6f56c322ba1950bffe0ab244
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: x86: Track emulator-originated nested #PF VM-Exits

Add has_emulator_context to kvm_queued_exception to record whether a
queued exception VM-Exit originated from the current emulator exception.

> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 283847619ff88..eb9f1b9afeb7d 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -688,6 +688,7 @@ struct kvm_queued_exception {
>  	bool pending;
>  	bool injected;
>  	bool has_error_code;
> +	bool has_emulator_context;

[Severity: Low]
Does this code ever read the newly added has_emulator_context field? It
looks like it is populated here and throughout the exception queueing paths,
but is never actually consumed.

>  	u8 vector;
>  	u32 error_code;
>  	unsigned long payload;

> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index d94b59140c452..75d00340c71fa 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c

[ ... ]

> @@ -473,7 +476,8 @@ static void kvm_multiple_exception(struct kvm_vcpu *vcpu, unsigned int nr,
>  	if (is_guest_mode(vcpu) &&
>  	    kvm_nested_call(is_exception_vmexit)(vcpu, nr, error_code)) {
>  		kvm_queue_exception_vmexit(vcpu, nr, has_error, error_code,
> -					   has_payload, payload);
> +					   has_payload, payload,
> +					   has_emulator_context);
>  		return;
>  	}

[Severity: Medium]
Is it possible to leave the has_emulator_context field uninitialized here?

Looking at the rest of kvm_multiple_exception(), the new has_emulator_context
parameter is passed in, but it isn't assigned to the vcpu exception state in
the normal queue path:

    if (!vcpu->arch.exception.pending && !vcpu->arch.exception.injected) {
    queue:
        vcpu->arch.exception.pending = true;
        ...
        vcpu->arch.exception.has_payload = has_payload;
        vcpu->arch.exception.payload = payload;
        return;
    }

Could this result in leaking stale state for has_emulator_context?

Furthermore, does this flag also need to be explicitly cleared in
kvm_vcpu_ioctl_x86_set_vcpu_events() when restoring exception state from
userspace?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787116250.git.zhang_wei@open-hieco.net?part=5