[PATCH v2 0/8] KVM: nSVM: Enable DecodeAssists for nested guests

[PATCH v2 0/8] KVM: nSVM: Enable DecodeAssists for nested guests

[Tina Zhang]

The SVM DecodeAssists feature provides decode state for selected
VM-Exits.  KVM currently does not expose this feature to L1.  Some L1
hypervisors may therefore treat the platform's SVM support as
incomplete.

In practice, this was observed with Hyper-V running on top of KVM.
Hyper-V appears to require DecodeAssists before enabling nested SVM for
its guests.  Virtualizing the feature lets users enable Hyper-V
virtualization features inside a Windows VM when needed, e.g. to run
QEMU/KVM in WSL.  Without DecodeAssists, Hyper-V does not enable nested
SVM because DecodeAssists is missing from KVM's virtual SVM model.

Virtualize DecodeAssists for nested SVM.  Propagate fresh hardware
instruction bytes from VMCB02, synthesize the architectural EXITINFO
state for emulator-generated intercepts, and provide instruction bytes
for KVM-synthesized data #PF/#NPF exits.  Prefer matching bytes already
fetched by the emulator, and fetch missing bytes from L2 RIP only as a
fallback.  Avoid fallback reads for SEV guests, whose encrypted memory
cannot provide plaintext instruction bytes to KVM.

The selftest coverage in this version is intentionally broad and may be
more extensive than necessary.  I kept the full set for now to cover
hardware-originated, KVM-synthesized, userspace-injected, and boundary
cases; it can be trimmed later if some cases prove redundant.

The selftest has been run with kvm.force_emulation_prefix both disabled
and enabled.

---
Changes since v1:
- Split the implementation into seven patches to make the individual
  pieces easier to review.
- Add EXITINFO virtualization for emulator-generated MOV CR/DR, INTn,
  INVLPG, and INVLPGA intercepts.
- Limit GuestInstrBytes propagation to data #NPF and intercepted #PF
  exits, and clear the fields for instruction-fetch and unrelated exits.
- Provide GuestInstrBytes for KVM-synthesized data #PF/#NPF exits.  Use
  matching emulator bytes first and fetch missing bytes from L2 RIP as a
  fallback, while avoiding fallback reads for SEV guests.
- Expand the selftest beyond hardware #NPF and stale-state coverage to
  exercise hardware, synthesized, userspace-injected, instruction-fetch,
  page-boundary, and CS-limit cases.

v1:
https://lore.kernel.org/r/20260629125205.52394-1-zhang_wei@open-hieco.net

Tina Zhang (8):
  KVM: x86: Add intercept_linear_addr to x86_instruction_info
  KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts
  KVM: nSVM: Track fresh VMCB02 DecodeAssist bytes
  KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12
  KVM: nSVM: Use emulator bytes for synthesized nested #NPF/#PF
  KVM: nSVM: Fetch missing DecodeAssist bytes for synthesized #NPF/#PF
  KVM: nSVM: Advertise DecodeAssists to L1
  KVM: selftests: Add nested SVM DecodeAssists test

 arch/x86/kvm/emulate.c                        |  30 +-
 arch/x86/kvm/kvm_emulate.h                    |   1 +
 arch/x86/kvm/svm/nested.c                     | 188 ++++-
 arch/x86/kvm/svm/svm.c                        |  56 +-
 arch/x86/kvm/svm/svm.h                        |  24 +-
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../kvm/x86/svm_nested_decode_assists_test.c  | 694 ++++++++++++++++++
 8 files changed, 980 insertions(+), 15 deletions(-)
 create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c


base-commit: 8cd9520d35a6c38db6567e97dd93b1f11f185dc6
-- 
2.43.7

[PATCH v2 1/8] KVM: x86: Add intercept_linear_addr to x86_instruction_info

[Tina Zhang]

Add intercept_linear_addr to x86_instruction_info so that the emulator
can pass linear addresses needed by intercept handlers.

Populate the field for INVLPG from its decoded memory operand.  INVLPG
is decoded with NoAccess, and therefore src_val does not contain the
operand address.  Calculate the address in the emulator, where the
decoded segment and effective address are available.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/emulate.c     | 30 ++++++++++++++++++++++--------
 arch/x86/kvm/kvm_emulate.h |  1 +
 2 files changed, 23 insertions(+), 8 deletions(-)

diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index 8013dccb3110..03bf95417275 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -410,6 +410,27 @@ static int em_salc(struct x86_emulate_ctxt *ctxt)
 	_fault ? X86EMUL_UNHANDLEABLE : X86EMUL_CONTINUE; \
 })
 
+static unsigned long seg_base(struct x86_emulate_ctxt *ctxt, int seg)
+{
+	if (ctxt->mode == X86EMUL_MODE_PROT64 && seg < VCPU_SREG_FS)
+		return 0;
+
+	return ctxt->ops->get_cached_segment_base(ctxt, seg);
+}
+
+/* Return the linear address for intercepts that report one. */
+static u64 get_intercept_linear_addr(struct x86_emulate_ctxt *ctxt,
+				     enum x86_intercept intercept)
+{
+	u64 la;
+
+	if (intercept != x86_intercept_invlpg)
+		return 0;
+
+	la = seg_base(ctxt, ctxt->src.addr.mem.seg) + ctxt->src.addr.mem.ea;
+	return ctxt->mode == X86EMUL_MODE_PROT64 ? la : (u32)la;
+}
+
 static int emulator_check_intercept(struct x86_emulate_ctxt *ctxt,
 				    enum x86_intercept intercept,
 				    enum x86_intercept_stage stage)
@@ -427,6 +448,7 @@ static int emulator_check_intercept(struct x86_emulate_ctxt *ctxt,
 		.src_type   = ctxt->src.type,
 		.dst_type   = ctxt->dst.type,
 		.ad_bytes   = ctxt->ad_bytes,
+		.intercept_linear_addr = get_intercept_linear_addr(ctxt, intercept),
 		.rip	    = ctxt->eip,
 		.next_rip   = ctxt->_eip,
 	};
@@ -520,14 +542,6 @@ static u32 desc_limit_scaled(struct desc_struct *desc)
 	return desc->g ? (limit << 12) | 0xfff : limit;
 }
 
-static unsigned long seg_base(struct x86_emulate_ctxt *ctxt, int seg)
-{
-	if (ctxt->mode == X86EMUL_MODE_PROT64 && seg < VCPU_SREG_FS)
-		return 0;
-
-	return ctxt->ops->get_cached_segment_base(ctxt, seg);
-}
-
 static int emulate_exception(struct x86_emulate_ctxt *ctxt, int vec,
 			     u32 error, bool valid)
 {
diff --git a/arch/x86/kvm/kvm_emulate.h b/arch/x86/kvm/kvm_emulate.h
index 0abff36d0994..900bd95e9e55 100644
--- a/arch/x86/kvm/kvm_emulate.h
+++ b/arch/x86/kvm/kvm_emulate.h
@@ -47,6 +47,7 @@ struct x86_instruction_info {
 	u8  src_type;		/* type of source operand		*/
 	u8  dst_type;		/* type of destination operand		*/
 	u8  ad_bytes;           /* size of src/dst address              */
+	u64 intercept_linear_addr;	/* arch-reported linear address, if any */
 	u64 rip;		/* rip of the instruction		*/
 	u64 next_rip;           /* rip following the instruction        */
 };
-- 
2.43.7

[PATCH v2 2/8] KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts

[Tina Zhang]

When the x86 emulator encounters an instruction intercepted by L1,
svm_check_intercept() synthesizes a nested VM-Exit without fresh hardware
DecodeAssist state.  Populate the architectural EXITINFO fields when
DecodeAssists is exposed to L1.

Generate EXITINFO1 with the GPR number for MOV CR/DR, the interrupt vector
for INTn, and the linear address for INVLPG.  Set EXITINFO1 to zero for
CLTS, LMSW, SMSW, and INVLPGA; the INVLPGA address remains in guest rAX.
Clear EXITINFO2 for all covered intercepts and leave unrelated intercepts
unchanged.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/svm.c | 48 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 48 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index e02a38da5296..cc35afb59623 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -4754,6 +4754,52 @@ static const struct __x86_intercept {
 #undef POST_EX
 #undef POST_MEM
 
+static void svm_prepare_decode_assist_exit_info(struct kvm_vcpu *vcpu,
+						const struct x86_instruction_info *info)
+{
+	struct vmcb *vmcb = to_svm(vcpu)->vmcb;
+	u64 exit_info_1;
+
+	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
+		return;
+
+	switch (info->intercept) {
+	case x86_intercept_cr_read:
+	case x86_intercept_cr_write:
+		/* MOV CRx: bit 63 set, GPR number in bits 3:0. */
+		exit_info_1 = BIT_ULL(63) | (info->modrm_rm & 0xf);
+		break;
+	case x86_intercept_clts:
+	case x86_intercept_lmsw:
+	case x86_intercept_smsw:
+		/* CLTS/LMSW/SMSW: no decode information, bit 63 clear. */
+		exit_info_1 = 0;
+		break;
+	case x86_intercept_dr_read:
+	case x86_intercept_dr_write:
+		/* MOV DRx: GPR number in bits 3:0. */
+		exit_info_1 = info->modrm_rm & 0xf;
+		break;
+	case x86_intercept_intn:
+		/* INTn: software interrupt number in bits 7:0. */
+		exit_info_1 = info->src_val & 0xff;
+		break;
+	case x86_intercept_invlpg:
+		/* INVLPG: linear address of the target page. */
+		exit_info_1 = info->intercept_linear_addr;
+		break;
+	case x86_intercept_invlpga:
+		/* INVLPGA: the address remains available in guest rAX. */
+		exit_info_1 = 0;
+		break;
+	default:
+		return;
+	}
+
+	vmcb->control.exit_info_1 = exit_info_1;
+	vmcb->control.exit_info_2 = 0;
+}
+
 static int svm_check_intercept(struct kvm_vcpu *vcpu,
 			       struct x86_instruction_info *info,
 			       enum x86_intercept_stage stage,
@@ -4875,6 +4921,8 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
 		break;
 	}
 
+	svm_prepare_decode_assist_exit_info(vcpu, info);
+
 	/* TODO: Advertise NRIPS to guest hypervisor unconditionally */
 	if (static_cpu_has(X86_FEATURE_NRIPS))
 		vmcb->control.next_rip  = info->next_rip;
-- 
2.43.7

[PATCH v2 3/8] KVM: nSVM: Track fresh VMCB02 DecodeAssist bytes

[Tina Zhang]

Only hardware VM-Exits provide fresh DecodeAssist instruction bytes in
VMCB02.  KVM-synthesized nested VM-Exits must not reuse whatever bytes
were left by an earlier hardware exit.

Track whether VMCB02 contains DecodeAssist instruction bytes from the
hardware VM-Exit currently being reflected to L1, and clear the fields
before each nested run so stale bytes cannot be mistaken for fresh
hardware state.

The flag is consumed by a later change that propagates hardware-provided
DecodeAssist bytes to VMCB12.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/nested.c | 20 +++++++++++++++++---
 arch/x86/kvm/svm/svm.c    |  6 +++---
 arch/x86/kvm/svm/svm.h    |  9 ++++++++-
 3 files changed, 28 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index b340dc9991ad..70e2ee3af78b 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -33,13 +33,21 @@
 
 #define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK
 
+static void nested_svm_clear_insn_bytes(struct vmcb *vmcb)
+{
+	vmcb->control.insn_len = 0;
+	memset(vmcb->control.insn_bytes, 0,
+	       sizeof(vmcb->control.insn_bytes));
+}
+
 static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 				       struct x86_exception *fault)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct vmcb *vmcb = svm->vmcb;
+	bool from_hardware = vmcb->control.exit_code == SVM_EXIT_NPF;
 
-	if (vmcb->control.exit_code != SVM_EXIT_NPF) {
+	if (!from_hardware) {
 		/*
 		 * TODO: track the cause of the nested page fault, and
 		 * correctly fill in the high bits of exit_info_1.
@@ -52,6 +60,7 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 	vmcb->control.exit_info_1 &= ~0xffffffffULL;
 	vmcb->control.exit_info_1 |= fault->error_code;
 
+	svm->nested.vmcb02_insn_bytes_fresh = from_hardware;
 	nested_svm_vmexit(svm);
 }
 
@@ -838,7 +847,10 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 	/*
 	 * Filled at exit: exit_code, exit_info_1, exit_info_2, exit_int_info,
 	 * exit_int_info_err, next_rip, insn_len, insn_bytes.
+	 * Clear stale DecodeAssist data before L2 runs.
 	 */
+	nested_svm_clear_insn_bytes(vmcb02);
+	svm->nested.vmcb02_insn_bytes_fresh = false;
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
 	    (vmcb12_ctrl->int_ctl & V_GIF_ENABLE_MASK))
@@ -1587,14 +1599,16 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
 	return vmexit;
 }
 
-int nested_svm_exit_handled(struct vcpu_svm *svm)
+int nested_svm_exit_handled(struct vcpu_svm *svm, bool vmcb02_insn_bytes_fresh)
 {
 	int vmexit;
 
 	vmexit = nested_svm_intercept(svm);
 
-	if (vmexit == NESTED_EXIT_DONE)
+	if (vmexit == NESTED_EXIT_DONE) {
+		svm->nested.vmcb02_insn_bytes_fresh = vmcb02_insn_bytes_fresh;
 		nested_svm_vmexit(svm);
+	}
 
 	return vmexit;
 }
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index cc35afb59623..2c04a8d9da29 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2544,7 +2544,7 @@ static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
 
 	if (cr0 ^ val) {
 		svm->vmcb->control.exit_code = SVM_EXIT_CR0_SEL_WRITE;
-		ret = (nested_svm_exit_handled(svm) == NESTED_EXIT_DONE);
+		ret = (nested_svm_exit_handled(svm, false) == NESTED_EXIT_DONE);
 	}
 
 	return ret;
@@ -3671,7 +3671,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 		vmexit = nested_svm_exit_special(svm);
 
 		if (vmexit == NESTED_EXIT_CONTINUE)
-			vmexit = nested_svm_exit_handled(svm);
+			vmexit = nested_svm_exit_handled(svm, true);
 
 		if (vmexit == NESTED_EXIT_DONE)
 			return 1;
@@ -4927,7 +4927,7 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
 	if (static_cpu_has(X86_FEATURE_NRIPS))
 		vmcb->control.next_rip  = info->next_rip;
 	vmcb->control.exit_code = icpt_info.exit_code;
-	vmexit = nested_svm_exit_handled(svm);
+	vmexit = nested_svm_exit_handled(svm, false);
 
 	ret = (vmexit == NESTED_EXIT_DONE) ? X86EMUL_INTERCEPTED
 					   : X86EMUL_CONTINUE;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 5137416be593..fa65f6a1a59b 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -238,6 +238,13 @@ struct svm_nested_state {
 	 * on its side.
 	 */
 	bool force_msr_bitmap_recalc;
+
+	/*
+	 * True if VMCB02's DecodeAssist instruction bytes belong to the hardware
+	 * VM-Exit currently being reflected to L1.  KVM-synthesized exits leave
+	 * it clear.
+	 */
+	bool vmcb02_insn_bytes_fresh;
 };
 
 struct vcpu_sev_es_state {
@@ -864,7 +871,7 @@ static inline void nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
 	nested_svm_vmexit(svm);
 }
 
-int nested_svm_exit_handled(struct vcpu_svm *svm);
+int nested_svm_exit_handled(struct vcpu_svm *svm, bool vmcb02_insn_bytes_fresh);
 int nested_svm_check_permissions(struct kvm_vcpu *vcpu);
 int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu);
 int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr,
-- 
2.43.7

[PATCH v2 4/8] KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12

[Tina Zhang]

When DecodeAssists is exposed to L1, copy fresh hardware instruction
bytes from VMCB02 to VMCB12 for data #NPF and intercepted #PF VM-Exits.

Clear VMCB12's instruction-byte state for instruction-fetch faults,
unrelated exits, and exits without fresh VMCB02 state so that stale bytes
are not exposed to L1.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/nested.c | 53 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 53 insertions(+)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 70e2ee3af78b..9e12eab7eed9 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -40,6 +40,57 @@ static void nested_svm_clear_insn_bytes(struct vmcb *vmcb)
 	       sizeof(vmcb->control.insn_bytes));
 }
 
+static void nested_svm_copy_insn_bytes(struct vmcb *to,
+				       const struct vmcb *from)
+{
+	u8 insn_len = from->control.insn_len;
+
+	nested_svm_clear_insn_bytes(to);
+
+	if (WARN_ON_ONCE(insn_len > sizeof(from->control.insn_bytes)))
+		return;
+
+	to->control.insn_len = insn_len;
+	memcpy(to->control.insn_bytes, from->control.insn_bytes, insn_len);
+}
+
+static bool nested_svm_vmexit_supports_insn_bytes(struct kvm_vcpu *vcpu,
+						  const struct vmcb *vmcb12)
+{
+	u64 exit_code = vmcb12->control.exit_code;
+
+	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
+		return false;
+
+	if (exit_code != SVM_EXIT_NPF &&
+	    exit_code != SVM_EXIT_EXCP_BASE + PF_VECTOR)
+		return false;
+
+	return !(vmcb12->control.exit_info_1 & PFERR_FETCH_MASK);
+}
+
+/*
+ * Rebuild VMCB12's DecodeAssist bytes for the nested VM-Exit.  Use fresh
+ * hardware VMCB02 state when available; otherwise report a zero byte count.
+ */
+static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
+						struct vmcb *vmcb12,
+						struct vmcb *vmcb02)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+	bool vmcb02_insn_bytes_fresh = svm->nested.vmcb02_insn_bytes_fresh;
+
+	svm->nested.vmcb02_insn_bytes_fresh = false;
+
+	nested_svm_clear_insn_bytes(vmcb12);
+
+	if (!nested_svm_vmexit_supports_insn_bytes(vcpu, vmcb12))
+		return;
+
+	if (vmcb02_insn_bytes_fresh)
+		nested_svm_copy_insn_bytes(vmcb12, vmcb02);
+}
+
 static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 				       struct x86_exception *fault)
 {
@@ -1263,6 +1314,8 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm_vcpu *vcpu)
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
 		vmcb12->control.next_rip  = vmcb02->control.next_rip;
 
+	nested_svm_update_vmcb12_insn_bytes(vcpu, vmcb12, vmcb02);
+
 	if (nested_vmcb12_has_lbrv(vcpu))
 		svm_copy_lbrs(&vmcb12->save, &vmcb02->save);
 
-- 
2.43.7

[PATCH v2 5/8] KVM: nSVM: Use emulator bytes for synthesized nested #NPF/#PF

[Tina Zhang]

SVM DecodeAssists records up to 15 instruction bytes from L2 CS:RIP
for data-access #NPF and intercepted #PF exits.  For nested VM-Exits
reflected directly from hardware, KVM can copy the fresh VMCB02
DecodeAssist bytes into VMCB12.  KVM-synthesized #NPF/#PF exits,
however, do not have fresh hardware DecodeAssist bytes in VMCB02.

When KVM synthesizes a nested #NPF/#PF from emulator context and the
emulator fetch cache still matches L2 RIP, populate a synthesized
instruction-byte buffer from those already-fetched bytes and copy them
into VMCB12 when constructing the nested VM-Exit for L1.

Consume the synthesized buffer exactly once and clear it before each
nested run, so stale emulator bytes cannot leak into unrelated nested
VM-Exits.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/nested.c | 56 +++++++++++++++++++++++++++++++++++++--
 arch/x86/kvm/svm/svm.h    | 15 ++++++++++-
 2 files changed, 68 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 9e12eab7eed9..8cee9aeaf4f2 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -69,26 +69,72 @@ static bool nested_svm_vmexit_supports_insn_bytes(struct kvm_vcpu *vcpu,
 	return !(vmcb12->control.exit_info_1 & PFERR_FETCH_MASK);
 }
 
+static void nested_svm_clear_synthesized_insn_bytes(struct vcpu_svm *svm)
+{
+	svm->nested.synthesized_insn_bytes.prepared = false;
+	svm->nested.synthesized_insn_bytes.insn_len = 0;
+}
+
+static void nested_svm_prepare_synthesized_insn_bytes(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+	struct nested_svm_insn_bytes *insn_bytes =
+		&svm->nested.synthesized_insn_bytes;
+	const u8 max_bytes = sizeof(insn_bytes->insn_bytes);
+	struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt;
+
+	nested_svm_clear_synthesized_insn_bytes(svm);
+
+	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
+		return;
+
+	if (!ctxt || ctxt->eip != kvm_rip_read(vcpu) ||
+	    ctxt->fetch.end < ctxt->fetch.data ||
+	    ctxt->fetch.end > ctxt->fetch.data + max_bytes)
+		return;
+
+	insn_bytes->insn_len = ctxt->fetch.end - ctxt->fetch.data;
+	memcpy(insn_bytes->insn_bytes, ctxt->fetch.data,
+	       insn_bytes->insn_len);
+	insn_bytes->prepared = true;
+}
+
 /*
  * Rebuild VMCB12's DecodeAssist bytes for the nested VM-Exit.  Use fresh
- * hardware VMCB02 state when available; otherwise report a zero byte count.
+ * hardware VMCB02 state when available; otherwise use synthesized bytes from
+ * the emulator fetch cache for KVM-generated #NPF/#PF exits.
  */
 static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
 						struct vmcb *vmcb12,
 						struct vmcb *vmcb02)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
+	struct nested_svm_insn_bytes *insn_bytes =
+		&svm->nested.synthesized_insn_bytes;
+	const u8 max_bytes = sizeof(vmcb12->control.insn_bytes);
 	bool vmcb02_insn_bytes_fresh = svm->nested.vmcb02_insn_bytes_fresh;
+	bool synthesized_prepared = insn_bytes->prepared;
+	u8 synthesized_len = insn_bytes->insn_len;
 
 	svm->nested.vmcb02_insn_bytes_fresh = false;
+	insn_bytes->prepared = false;
 
 	nested_svm_clear_insn_bytes(vmcb12);
 
 	if (!nested_svm_vmexit_supports_insn_bytes(vcpu, vmcb12))
 		return;
 
-	if (vmcb02_insn_bytes_fresh)
+	if (vmcb02_insn_bytes_fresh) {
 		nested_svm_copy_insn_bytes(vmcb12, vmcb02);
+		return;
+	}
+
+	if (synthesized_prepared) {
+		vmcb12->control.insn_len = min(synthesized_len, max_bytes);
+		memcpy(vmcb12->control.insn_bytes, insn_bytes->insn_bytes,
+		       vmcb12->control.insn_len);
+		return;
+	}
 }
 
 static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
@@ -112,6 +158,8 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 	vmcb->control.exit_info_1 |= fault->error_code;
 
 	svm->nested.vmcb02_insn_bytes_fresh = from_hardware;
+	if (!from_hardware && !(fault->error_code & PFERR_FETCH_MASK))
+		nested_svm_prepare_synthesized_insn_bytes(vcpu);
 	nested_svm_vmexit(svm);
 }
 
@@ -902,6 +950,7 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm)
 	 */
 	nested_svm_clear_insn_bytes(vmcb02);
 	svm->nested.vmcb02_insn_bytes_fresh = false;
+	nested_svm_clear_synthesized_insn_bytes(svm);
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
 	    (vmcb12_ctrl->int_ctl & V_GIF_ENABLE_MASK))
@@ -1709,6 +1758,9 @@ static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu)
 			vmcb->control.exit_info_2 = ex->payload;
 		else
 			vmcb->control.exit_info_2 = vcpu->arch.cr2;
+
+		if (!ex->has_error_code || !(ex->error_code & PFERR_FETCH_MASK))
+			nested_svm_prepare_synthesized_insn_bytes(vcpu);
 	} else if (ex->vector == DB_VECTOR) {
 		/* See kvm_check_and_inject_events().  */
 		kvm_deliver_exception_payload(vcpu, ex);
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index fa65f6a1a59b..ef7701ddaea0 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -205,6 +205,12 @@ struct vmcb_ctrl_area_cached {
 	};
 };
 
+struct nested_svm_insn_bytes {
+	bool prepared;
+	u8 insn_len;
+	u8 insn_bytes[15];
+};
+
 struct svm_nested_state {
 	struct kvm_vmcb_info vmcb02;
 	u64 hsave_msr;
@@ -242,9 +248,16 @@ struct svm_nested_state {
 	/*
 	 * True if VMCB02's DecodeAssist instruction bytes belong to the hardware
 	 * VM-Exit currently being reflected to L1.  KVM-synthesized exits leave
-	 * it clear.
+	 * it clear, in which case the #NPF/#PF instruction bytes are synthesized
+	 * from the emulator fetch cache when possible.
 	 */
 	bool vmcb02_insn_bytes_fresh;
+
+	/*
+	 * DecodeAssist instruction bytes for a KVM-synthesized nested #NPF/#PF.
+	 * Populated from the emulator's fetch cache when possible.
+	 */
+	struct nested_svm_insn_bytes synthesized_insn_bytes;
 };
 
 struct vcpu_sev_es_state {
-- 
2.43.7

[PATCH v2 6/8] KVM: nSVM: Fetch missing DecodeAssist bytes for synthesized #NPF/#PF

[Tina Zhang]

The emulator fetch cache is not guaranteed to contain the full
architected 15-byte DecodeAssist window, e.g. it may only contain the
bytes needed to decode the instruction.

Keep preparation of synthesized state limited to capturing a matching
emulator fetch cache.  When constructing VMCB12, copy those bytes and
fetch any missing tail through L2 guest page tables, stopping at the
first translation fault, read failure, or CS limit overrun.  If no
matching emulator bytes are available, fetch the full window from L2
RIP, e.g. for a #PF injected by userspace.

Do not perform tail or fallback reads for SEV guests.  KVM cannot read
plaintext instruction bytes from encrypted guest memory, and the
existing SEV emulation path treats missing hardware DecodeAssist bytes
as unavailable instead of decoding guest memory.  For nested SEV,
report only matching emulator bytes that are already available,
potentially a zero instruction-byte count.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/nested.c | 65 ++++++++++++++++++++++++++++++++++++++-
 arch/x86/kvm/svm/svm.h    |  6 ++--
 2 files changed, 68 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 8cee9aeaf4f2..aaad36c23914 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -75,6 +75,54 @@ static void nested_svm_clear_synthesized_insn_bytes(struct vcpu_svm *svm)
 	svm->nested.synthesized_insn_bytes.insn_len = 0;
 }
 
+/*
+ * Fetch up to 15 bytes at L2's instruction pointer.  Honor the architectural
+ * fetch boundaries by stopping at a translation failure, read failure, or CS
+ * limit.
+ */
+static u8 nested_svm_fetch_insn_bytes(struct kvm_vcpu *vcpu, u8 *bytes,
+				      u8 count, u8 max_bytes)
+{
+	struct kvm_mmu *mmu = vcpu->arch.walk_mmu;
+	u64 access = PFERR_FETCH_MASK;
+	gva_t rip = kvm_get_linear_rip(vcpu);
+	struct x86_exception e;
+
+	if (to_svm(vcpu)->vmcb->save.cpl == 3)
+		access |= PFERR_USER_MASK;
+
+	/*
+	 * Hardware truncates the fetch at the CS limit.  CS.base and CS.limit
+	 * checks do not apply in 64-bit mode.
+	 */
+	if (!is_64_bit_mode(vcpu)) {
+		u32 eip = kvm_rip_read(vcpu);
+		u32 limit = to_svm(vcpu)->vmcb->save.cs.limit;
+
+		if (eip > limit)
+			return 0;
+		max_bytes = min_t(u64, max_bytes, (u64)limit - eip + 1);
+	}
+
+	count = min(count, max_bytes);
+
+	while (count < max_bytes) {
+		unsigned int chunk = min_t(unsigned int, max_bytes - count,
+				PAGE_SIZE - offset_in_page(rip + count));
+		gpa_t gpa = mmu->gva_to_gpa(vcpu, mmu, rip + count, access, &e);
+
+		if (gpa == INVALID_GPA ||
+		    kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(gpa),
+					     bytes + count,
+					     offset_in_page(gpa), chunk))
+			break;
+
+		count += chunk;
+	}
+
+	return count;
+}
+
 static void nested_svm_prepare_synthesized_insn_bytes(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
@@ -102,7 +150,8 @@ static void nested_svm_prepare_synthesized_insn_bytes(struct kvm_vcpu *vcpu)
 /*
  * Rebuild VMCB12's DecodeAssist bytes for the nested VM-Exit.  Use fresh
  * hardware VMCB02 state when available; otherwise use synthesized bytes from
- * the emulator fetch cache for KVM-generated #NPF/#PF exits.
+ * the emulator fetch cache for KVM-generated #NPF/#PF exits, or fetch from
+ * L2 RIP as a last resort.
  */
 static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
 						struct vmcb *vmcb12,
@@ -133,8 +182,22 @@ static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
 		vmcb12->control.insn_len = min(synthesized_len, max_bytes);
 		memcpy(vmcb12->control.insn_bytes, insn_bytes->insn_bytes,
 		       vmcb12->control.insn_len);
+
+		if (!is_sev_guest(vcpu))
+			vmcb12->control.insn_len =
+				nested_svm_fetch_insn_bytes(vcpu,
+							    vmcb12->control.insn_bytes,
+							    vmcb12->control.insn_len,
+							    max_bytes);
 		return;
 	}
+
+	if (is_sev_guest(vcpu))
+		return;
+
+	vmcb12->control.insn_len =
+		nested_svm_fetch_insn_bytes(vcpu, vmcb12->control.insn_bytes,
+					    0, max_bytes);
 }
 
 static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index ef7701ddaea0..7fe5fb9706dd 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -249,13 +249,15 @@ struct svm_nested_state {
 	 * True if VMCB02's DecodeAssist instruction bytes belong to the hardware
 	 * VM-Exit currently being reflected to L1.  KVM-synthesized exits leave
 	 * it clear, in which case the #NPF/#PF instruction bytes are synthesized
-	 * from the emulator fetch cache when possible.
+	 * from the emulator fetch cache and/or refetched through the guest page
+	 * tables.
 	 */
 	bool vmcb02_insn_bytes_fresh;
 
 	/*
 	 * DecodeAssist instruction bytes for a KVM-synthesized nested #NPF/#PF.
-	 * Populated from the emulator's fetch cache when possible.
+	 * Populated from the emulator's fetch cache when it describes the
+	 * instruction at L2's current RIP.
 	 */
 	struct nested_svm_insn_bytes synthesized_insn_bytes;
 };
-- 
2.43.7

[PATCH v2 7/8] KVM: nSVM: Advertise DecodeAssists to L1

[Tina Zhang]

Advertise DecodeAssists to L1 now that KVM virtualizes the
guest-visible DecodeAssist state in VMCB12: EXITINFO1 decode data for
MOV CR/DR, INTn, and INVLPG exits, plus instruction bytes for nested
page faults and intercepted #PF exits.  INVLPGA's linear address remains
available directly from the saved guest rAX, as required by the APM.

Expose the feature only when supported by hardware, as KVM still relies
on hardware DecodeAssists for VM-Exits that are reflected directly from
L2.

With DecodeAssists exposed, QEMU configurations that require the feature
(e.g. "-cpu ...,+decodeassists,...,enforce") are no longer rejected.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 arch/x86/kvm/svm/svm.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 2c04a8d9da29..4af5121070f4 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5518,6 +5518,8 @@ static __init void svm_set_cpu_caps(void)
 		 */
 		kvm_cpu_cap_set(X86_FEATURE_FLUSHBYASID);
 
+		kvm_cpu_cap_check_and_set(X86_FEATURE_DECODEASSISTS);
+
 		if (nrips)
 			kvm_cpu_cap_set(X86_FEATURE_NRIPS);
 
-- 
2.43.7

[PATCH v2 8/8] KVM: selftests: Add nested SVM DecodeAssists test

[Tina Zhang]

Add a nested SVM selftest for DecodeAssists.  Verify that KVM exposes the
feature to L1 and provides the architectural exit state for MOV CR/DR,
CLTS, LMSW, SMSW, INTn, INVLPG, and INVLPGA intercepts.

For data #NPF and intercepted #PF exits, verify instruction bytes from
hardware, the emulator fetch cache, and on-demand fetching from L2 RIP.
Exercise full 15-byte windows and truncation at an unmapped page boundary
and at the CS limit.  Also verify that instruction-fetch faults and
unrelated exits report no instruction bytes.

Run exits back-to-back and prefill exit state to detect stale VMCB data.
The synthesized OUTSB #NPF and userspace-injected #PF paths run by default;
the remaining synthesized paths are covered when
kvm.force_emulation_prefix=1 is enabled.

Signed-off-by: Tina Zhang <zhang_wei@open-hieco.net>
---
 tools/testing/selftests/kvm/Makefile.kvm      |   1 +
 .../selftests/kvm/include/x86/processor.h     |   1 +
 .../kvm/x86/svm_nested_decode_assists_test.c  | 694 ++++++++++++++++++
 3 files changed, 696 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c

diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selftests/kvm/Makefile.kvm
index 9118a5a51b89..23bf51074392 100644
--- a/tools/testing/selftests/kvm/Makefile.kvm
+++ b/tools/testing/selftests/kvm/Makefile.kvm
@@ -114,6 +114,7 @@ TEST_GEN_PROGS_x86 += x86/vmx_preemption_timer_test
 TEST_GEN_PROGS_x86 += x86/svm_vmcall_test
 TEST_GEN_PROGS_x86 += x86/svm_int_ctl_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_clear_efer_svme
+TEST_GEN_PROGS_x86 += x86/svm_nested_decode_assists_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_shutdown_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_soft_inject_test
 TEST_GEN_PROGS_x86 += x86/svm_nested_vmcb12_gpa
diff --git a/tools/testing/selftests/kvm/include/x86/processor.h b/tools/testing/selftests/kvm/include/x86/processor.h
index 77f576ee7789..ee4520ff2f89 100644
--- a/tools/testing/selftests/kvm/include/x86/processor.h
+++ b/tools/testing/selftests/kvm/include/x86/processor.h
@@ -201,6 +201,7 @@ struct kvm_x86_cpu_feature {
 #define	X86_FEATURE_LBRV		KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 1)
 #define	X86_FEATURE_NRIPS		KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 3)
 #define X86_FEATURE_TSCRATEMSR          KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 4)
+#define X86_FEATURE_DECODEASSISTS       KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 7)
 #define X86_FEATURE_PAUSEFILTER         KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 10)
 #define X86_FEATURE_PFTHRESHOLD         KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 12)
 #define	X86_FEATURE_V_VMSAVE_VMLOAD	KVM_X86_CPU_FEATURE(0x8000000A, 0, EDX, 15)
diff --git a/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c b/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c
new file mode 100644
index 000000000000..ac9601dda936
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c
@@ -0,0 +1,694 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Test KVM's virtualization of SVM DecodeAssists for nested guests.
+ */
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "svm_util.h"
+
+#define L2_GUEST_STACK_SIZE 64
+
+#define TEST_INT_VECTOR 0x81
+
+/* Any canonical virtual address that is never mapped by the selftest VM. */
+#define PF_TEST_GVA BIT_ULL(40)
+#define PF_FETCH_TEST_GVA BIT_ULL(41)
+
+#define OUTSB_OPCODE 0x6e
+#define INT3_OPCODE 0xcc
+#define BOUNDARY_OUTSB_CODE_SIZE 15
+#define LIMIT_OUTSB_OPCODE_OFFSET 9
+#define USERSPACE_PF_STAGE 1
+#define TEST_IOPM_SIZE (3 * PAGE_SIZE)
+
+static u64 npf_target __aligned(PAGE_SIZE);
+static u8 boundary_outsb_code[2 * PAGE_SIZE] __aligned(PAGE_SIZE);
+static u8 limit_outsb_code[PAGE_SIZE] __aligned(PAGE_SIZE);
+static unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];
+
+static void l2_read_code(void)
+{
+	asm volatile("mov (%0), %%rax" : : "r"(&npf_target) : "rax", "memory");
+	GUEST_FAIL("L2 read did not cause a nested page fault");
+}
+
+static void l2_outsb_code(void)
+{
+	asm volatile("mov %0, %%rsi\n\t"
+		     "mov $0x80, %%dx\n\t"
+		     "outsb"
+		     : : "r"(&npf_target) : "rsi", "rdx", "memory");
+	GUEST_FAIL("L2 OUTSB did not cause a nested page fault");
+}
+
+static void l2_pf_code(void)
+{
+	asm volatile("mov (%0), %%rax"
+		     : : "r"(PF_TEST_GVA) : "rax", "memory");
+	GUEST_FAIL("L2 access to an unmapped VA did not #PF");
+}
+
+static void l2_fep_pf_code(void)
+{
+	asm volatile(KVM_FEP "mov (%0), %%rax"
+		     : : "r"(PF_TEST_GVA) : "rax", "memory");
+	GUEST_FAIL("L2 forced-emulated access to an unmapped VA did not #PF");
+}
+
+static void l2_userspace_pf_code(void)
+{
+	GUEST_SYNC(USERSPACE_PF_STAGE);
+	GUEST_FAIL("Userspace-injected #PF was not intercepted by L1");
+}
+
+static void l2_mov_from_cr4_code(void)
+{
+	asm volatile("mov %%cr4, %%r10" : : : "r10");
+	GUEST_FAIL("L2 MOV-from-CR4 was not intercepted");
+}
+
+static void l2_fep_mov_from_cr4_code(void)
+{
+	asm volatile(KVM_FEP "mov %%cr4, %%r10" : : : "r10");
+	GUEST_FAIL("L2 forced-emulated MOV-from-CR4 was not intercepted");
+}
+
+static void l2_mov_to_cr4_code(void)
+{
+	asm volatile("mov %%cr4, %%rax\n\t"
+		     "mov %%rax, %%cr4" : : : "rax");
+	GUEST_FAIL("L2 MOV-to-CR4 was not intercepted");
+}
+
+static void l2_fep_mov_to_cr4_code(void)
+{
+	asm volatile("mov %%cr4, %%rax\n\t"
+		     KVM_FEP "mov %%rax, %%cr4" : : : "rax");
+	GUEST_FAIL("L2 forced-emulated MOV-to-CR4 was not intercepted");
+}
+
+static void l2_mov_to_dr7_code(void)
+{
+	asm volatile("mov %%dr7, %%rax\n\t"
+		     "mov %%rax, %%rbx\n\t"
+		     "mov %%rbx, %%dr7" : : : "rax", "rbx");
+	GUEST_FAIL("L2 MOV-to-DR7 was not intercepted");
+}
+
+static void l2_fep_mov_to_dr7_code(void)
+{
+	asm volatile("mov %%dr7, %%rax\n\t"
+		     "mov %%rax, %%rbx\n\t"
+		     KVM_FEP "mov %%rbx, %%dr7" : : : "rax", "rbx");
+	GUEST_FAIL("L2 forced-emulated MOV-to-DR7 was not intercepted");
+}
+
+static void l2_mov_from_dr7_code(void)
+{
+	asm volatile("mov %%dr7, %%r10" : : : "r10");
+	GUEST_FAIL("L2 MOV-from-DR7 was not intercepted");
+}
+
+static void l2_fep_mov_from_dr7_code(void)
+{
+	asm volatile(KVM_FEP "mov %%dr7, %%r10" : : : "r10");
+	GUEST_FAIL("L2 forced-emulated MOV-from-DR7 was not intercepted");
+}
+
+static void l2_clts_code(void)
+{
+	asm volatile("clts" : : : "memory");
+	GUEST_FAIL("L2 CLTS was not intercepted");
+}
+
+static void l2_fep_clts_code(void)
+{
+	asm volatile(KVM_FEP "clts" : : : "memory");
+	GUEST_FAIL("L2 forced-emulated CLTS was not intercepted");
+}
+
+static void l2_lmsw_code(void)
+{
+	asm volatile("smsw %%ax\n\t"
+		     "lmsw %%ax" : : : "rax", "memory");
+	GUEST_FAIL("L2 LMSW was not intercepted");
+}
+
+static void l2_fep_lmsw_code(void)
+{
+	asm volatile("smsw %%ax\n\t"
+		     KVM_FEP "lmsw %%ax" : : : "rax", "memory");
+	GUEST_FAIL("L2 forced-emulated LMSW was not intercepted");
+}
+
+static void l2_smsw_code(void)
+{
+	asm volatile("smsw %%ax" : : : "rax", "memory");
+	GUEST_FAIL("L2 SMSW was not intercepted");
+}
+
+static void l2_fep_smsw_code(void)
+{
+	asm volatile(KVM_FEP "smsw %%ax" : : : "rax", "memory");
+	GUEST_FAIL("L2 forced-emulated SMSW was not intercepted");
+}
+
+static void l2_int_code(void)
+{
+	asm volatile("int %0" : : "i"(TEST_INT_VECTOR));
+	GUEST_FAIL("L2 INTn was not intercepted");
+}
+
+static void l2_fep_int_code(void)
+{
+	asm volatile(KVM_FEP "int %0" : : "i"(TEST_INT_VECTOR));
+	GUEST_FAIL("L2 forced-emulated INTn was not intercepted");
+}
+
+static void l2_invlpg_code(void)
+{
+	asm volatile("invlpg (%0)" : : "r"(&npf_target) : "memory");
+	GUEST_FAIL("L2 INVLPG was not intercepted");
+}
+
+static void l2_fep_invlpg_code(void)
+{
+	asm volatile(KVM_FEP "invlpg (%0)" : : "r"(&npf_target) : "memory");
+	GUEST_FAIL("L2 forced-emulated INVLPG was not intercepted");
+}
+
+static void l2_invlpga_code(void)
+{
+	asm volatile("invlpga" : : "a"(&npf_target), "c"(0) : "memory");
+	GUEST_FAIL("L2 INVLPGA was not intercepted");
+}
+
+static void l2_fep_invlpga_code(void)
+{
+	asm volatile(KVM_FEP "invlpga"
+		     : : "a"(&npf_target), "c"(0) : "memory");
+	GUEST_FAIL("L2 forced-emulated INVLPGA was not intercepted");
+}
+
+static void l2_vmmcall_code(void)
+{
+	vmmcall();
+	GUEST_FAIL("L2 did not exit on VMMCALL");
+}
+
+struct instruction_intercept_test {
+	const char *name;
+	void (*code)(void);
+	void (*fep_code)(void);
+	u64 intercept;
+	u32 intercept_cr;
+	u32 intercept_dr;
+	u64 exit_code;
+	u64 exit_info_1;
+	u64 exit_info_1_mask;
+	bool check_rax;
+	u64 rax;
+};
+
+static const struct instruction_intercept_test instruction_intercept_tests[] = {
+	{
+		.name = "MOV-to-CR4",
+		.code = l2_mov_to_cr4_code,
+		.fep_code = l2_fep_mov_to_cr4_code,
+		.intercept_cr = BIT(INTERCEPT_CR4_WRITE),
+		.exit_code = SVM_EXIT_WRITE_CR4,
+		.exit_info_1 = BIT_ULL(63),
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "MOV-from-CR4",
+		.code = l2_mov_from_cr4_code,
+		.fep_code = l2_fep_mov_from_cr4_code,
+		.intercept_cr = BIT(INTERCEPT_CR4_READ),
+		.exit_code = SVM_EXIT_READ_CR4,
+		.exit_info_1 = BIT_ULL(63) | 10,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "MOV-to-DR7",
+		.code = l2_mov_to_dr7_code,
+		.fep_code = l2_fep_mov_to_dr7_code,
+		.intercept_dr = BIT(INTERCEPT_DR7_WRITE),
+		.exit_code = SVM_EXIT_WRITE_DR7,
+		.exit_info_1 = 3,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "MOV-from-DR7",
+		.code = l2_mov_from_dr7_code,
+		.fep_code = l2_fep_mov_from_dr7_code,
+		.intercept_dr = BIT(INTERCEPT_DR7_READ),
+		.exit_code = SVM_EXIT_READ_DR7,
+		.exit_info_1 = 10,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "CLTS",
+		.code = l2_clts_code,
+		.fep_code = l2_fep_clts_code,
+		.intercept_cr = BIT(INTERCEPT_CR0_WRITE),
+		.exit_code = SVM_EXIT_WRITE_CR0,
+		.exit_info_1_mask = BIT_ULL(63),
+	}, {
+		.name = "LMSW",
+		.code = l2_lmsw_code,
+		.fep_code = l2_fep_lmsw_code,
+		.intercept_cr = BIT(INTERCEPT_CR0_WRITE),
+		.exit_code = SVM_EXIT_WRITE_CR0,
+		.exit_info_1_mask = BIT_ULL(63),
+	}, {
+		.name = "SMSW",
+		.code = l2_smsw_code,
+		.fep_code = l2_fep_smsw_code,
+		.intercept_cr = BIT(INTERCEPT_CR0_READ),
+		.exit_code = SVM_EXIT_READ_CR0,
+		.exit_info_1_mask = BIT_ULL(63),
+	}, {
+		.name = "INTn",
+		.code = l2_int_code,
+		.fep_code = l2_fep_int_code,
+		.intercept = BIT_ULL(INTERCEPT_INTn),
+		.exit_code = SVM_EXIT_SWINT,
+		.exit_info_1 = TEST_INT_VECTOR,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "INVLPG",
+		.code = l2_invlpg_code,
+		.fep_code = l2_fep_invlpg_code,
+		.intercept = BIT_ULL(INTERCEPT_INVLPG),
+		.exit_code = SVM_EXIT_INVLPG,
+		.exit_info_1 = (u64)&npf_target,
+		.exit_info_1_mask = ~0ULL,
+	}, {
+		.name = "INVLPGA",
+		.code = l2_invlpga_code,
+		.fep_code = l2_fep_invlpga_code,
+		.intercept = BIT_ULL(INTERCEPT_INVLPGA),
+		.exit_code = SVM_EXIT_INVLPGA,
+		.exit_info_1_mask = ~0ULL,
+		.check_rax = true,
+		.rax = (u64)&npf_target,
+	},
+};
+
+static void assert_decode_assist_insn_bytes(struct vmcb *vmcb)
+{
+	GUEST_ASSERT(vmcb->control.insn_len);
+	GUEST_ASSERT(vmcb->control.insn_len <=
+		     sizeof(vmcb->control.insn_bytes));
+	GUEST_ASSERT(!memcmp(vmcb->control.insn_bytes,
+			     (void *)vmcb->save.rip,
+			     vmcb->control.insn_len));
+}
+
+static void assert_full_decode_assist_insn_bytes(struct vmcb *vmcb)
+{
+	GUEST_ASSERT_EQ(vmcb->control.insn_len,
+			sizeof(vmcb->control.insn_bytes));
+	assert_decode_assist_insn_bytes(vmcb);
+}
+
+static void prepare_l2_for_vmrun(struct vmcb *vmcb, gva_t rip)
+{
+	vmcb->save.rip = rip;
+	vmcb->save.rsp = (u64)&l2_guest_stack[L2_GUEST_STACK_SIZE];
+}
+
+static void run_intercept_test(struct svm_test_data *svm,
+			       const struct instruction_intercept_test *test,
+			       bool synthesized)
+{
+	struct vmcb *vmcb = svm->vmcb;
+	struct vmcb_control_area *control = &vmcb->control;
+	const char *source = synthesized ? "synthesized" : "hardware";
+	u64 expected_exit_info_1 = test->exit_info_1 & test->exit_info_1_mask;
+
+	control->intercept |= test->intercept;
+	control->intercept_cr |= test->intercept_cr;
+	control->intercept_dr |= test->intercept_dr;
+
+	if (synthesized) {
+		control->exit_info_1 = ~0ULL;
+		control->exit_info_2 = ~0ULL;
+		prepare_l2_for_vmrun(vmcb, (u64)test->fep_code);
+	} else {
+		prepare_l2_for_vmrun(vmcb, (u64)test->code);
+	}
+
+	run_guest(vmcb, svm->vmcb_gpa);
+
+	__GUEST_ASSERT(control->exit_code == test->exit_code,
+		       "%s (%s): expected exit code %#lx, got %#lx",
+		       test->name, source, (unsigned long)test->exit_code,
+		       (unsigned long)control->exit_code);
+	__GUEST_ASSERT((control->exit_info_1 & test->exit_info_1_mask) ==
+		       expected_exit_info_1,
+		       "%s (%s): expected EXITINFO1 %#lx with mask %#lx, got %#lx",
+		       test->name, source, (unsigned long)expected_exit_info_1,
+		       (unsigned long)test->exit_info_1_mask,
+		       (unsigned long)control->exit_info_1);
+	__GUEST_ASSERT(!control->insn_len,
+		       "%s (%s): expected no instruction bytes, got %u",
+		       test->name, source, control->insn_len);
+
+	if (test->check_rax)
+		__GUEST_ASSERT(vmcb->save.rax == test->rax,
+			       "%s (%s): expected rAX %#lx, got %#lx",
+			       test->name, source, (unsigned long)test->rax,
+			       (unsigned long)vmcb->save.rax);
+
+	if (synthesized)
+		__GUEST_ASSERT(!control->exit_info_2,
+			       "%s (%s): expected EXITINFO2 to be clear, got %#lx",
+			       test->name, source,
+			       (unsigned long)control->exit_info_2);
+
+	control->intercept &= ~test->intercept;
+	control->intercept_cr &= ~test->intercept_cr;
+	control->intercept_dr &= ~test->intercept_dr;
+}
+
+static void test_instruction_intercepts(struct svm_test_data *svm)
+{
+	int i;
+
+	for (i = 0; i < ARRAY_SIZE(instruction_intercept_tests); i++) {
+		run_intercept_test(svm, &instruction_intercept_tests[i], false);
+
+		if (is_forced_emulation_enabled)
+			run_intercept_test(svm, &instruction_intercept_tests[i],
+					   true);
+	}
+}
+
+/*
+ * A hardware #NPF (data read of a GPA not mapped in the NPT) must reflect
+ * the instruction bytes recorded by hardware.
+ */
+static void test_hardware_npf(struct svm_test_data *svm, gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(vmcb, (u64)l2_read_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	assert_decode_assist_insn_bytes(vmcb);
+}
+
+/* An instruction-fetch #NPF must not report DecodeAssist instruction bytes. */
+static void test_hardware_fetch_npf(struct svm_test_data *svm,
+				    gva_t fetch_npf_gva,
+				    gpa_t fetch_npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(vmcb, fetch_npf_gva);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, fetch_npf_gpa);
+	GUEST_ASSERT(vmcb->control.exit_info_1 & PFERR_FETCH_MASK);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len, 0);
+}
+
+/*
+ * The IOIO intercept causes L0 to emulate OUTSB before accessing its source
+ * operand.  The emulated read then faults on L1's NPT, resulting in a
+ * KVM-synthesized #NPF.
+ */
+static void test_synthesized_npf(struct svm_test_data *svm, gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(vmcb, (u64)l2_outsb_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	assert_full_decode_assist_insn_bytes(vmcb);
+}
+
+/*
+ * OUTSB is the final byte of a mapped code page, and the following page is
+ * not present in L2's page tables.  DecodeAssist byte fetching must stop at
+ * the page boundary and report only the OUTSB opcode.
+ */
+static void test_synthesized_npf_truncated(struct svm_test_data *svm,
+					   gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(vmcb,
+			     (u64)&boundary_outsb_code[PAGE_SIZE -
+						       BOUNDARY_OUTSB_CODE_SIZE]);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	GUEST_ASSERT_EQ(vmcb->save.rip,
+			(u64)&boundary_outsb_code[PAGE_SIZE - 1]);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len, 1);
+	GUEST_ASSERT_EQ(vmcb->control.insn_bytes[0], OUTSB_OPCODE);
+}
+
+/* A non-64-bit code segment must truncate the byte window at CS.limit. */
+static void test_synthesized_npf_cs_limit(struct svm_test_data *svm,
+					  gpa_t npf_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+	u16 cs_attrib = vmcb->save.cs.attrib;
+	u32 cs_limit = vmcb->save.cs.limit;
+
+	vmcb->save.cs.attrib &= ~SVM_SELECTOR_L_MASK;
+	vmcb->save.cs.attrib |= SVM_SELECTOR_DB_MASK;
+	vmcb->save.cs.limit =
+		(u32)(u64)&limit_outsb_code[LIMIT_OUTSB_OPCODE_OFFSET];
+	prepare_l2_for_vmrun(vmcb, (u64)limit_outsb_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_NPF);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, npf_gpa);
+	GUEST_ASSERT_EQ(vmcb->save.rip,
+			(u64)&limit_outsb_code[LIMIT_OUTSB_OPCODE_OFFSET]);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len, 1);
+	GUEST_ASSERT_EQ(vmcb->control.insn_bytes[0], OUTSB_OPCODE);
+	vmcb->save.cs.attrib = cs_attrib;
+	vmcb->save.cs.limit = cs_limit;
+}
+
+static void test_intercepted_pf(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(vmcb, (u64)l2_pf_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_EXCP_BASE + PF_VECTOR);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, PF_TEST_GVA);
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_PRESENT_MASK));
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_FETCH_MASK));
+	assert_decode_assist_insn_bytes(vmcb);
+}
+
+/* An intercepted instruction-fetch #PF likewise reports no instruction bytes. */
+static void test_intercepted_fetch_pf(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(vmcb, PF_FETCH_TEST_GVA);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_EXCP_BASE + PF_VECTOR);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, PF_FETCH_TEST_GVA);
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_PRESENT_MASK));
+	GUEST_ASSERT(vmcb->control.exit_info_1 & PFERR_FETCH_MASK);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len, 0);
+}
+
+static void test_synthesized_pf(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	if (!is_forced_emulation_enabled)
+		return;
+
+	prepare_l2_for_vmrun(vmcb, (u64)l2_fep_pf_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_EXCP_BASE + PF_VECTOR);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, PF_TEST_GVA);
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_PRESENT_MASK));
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_FETCH_MASK));
+	assert_full_decode_assist_insn_bytes(vmcb);
+}
+
+/* Userspace injects this #PF without a matching emulator fetch cache. */
+static void test_userspace_injected_pf(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(vmcb, (u64)l2_userspace_pf_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_EXCP_BASE + PF_VECTOR);
+	GUEST_ASSERT_EQ(vmcb->control.exit_info_2, PF_TEST_GVA);
+	GUEST_ASSERT(!(vmcb->control.exit_info_1 & PFERR_FETCH_MASK));
+	assert_full_decode_assist_insn_bytes(vmcb);
+}
+
+static void test_vmmcall(struct svm_test_data *svm)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	prepare_l2_for_vmrun(vmcb, (u64)l2_vmmcall_code);
+	run_guest(vmcb, svm->vmcb_gpa);
+	GUEST_ASSERT_EQ(vmcb->control.exit_code, SVM_EXIT_VMMCALL);
+	GUEST_ASSERT_EQ(vmcb->control.insn_len, 0);
+}
+
+static void l1_guest_code(struct svm_test_data *svm, gpa_t npf_gpa,
+			  gva_t fetch_npf_gva,
+			  gpa_t fetch_npf_gpa, gpa_t iopm_gpa)
+{
+	struct vmcb *vmcb = svm->vmcb;
+
+	GUEST_ASSERT(this_cpu_has(X86_FEATURE_DECODEASSISTS));
+
+	generic_svm_setup(svm, l2_read_code,
+			  &l2_guest_stack[L2_GUEST_STACK_SIZE]);
+	vmcb->control.iopm_base_pa = iopm_gpa;
+
+	vmcb->control.intercept |= BIT_ULL(INTERCEPT_IOIO_PROT);
+	vmcb->control.intercept_exceptions |= 1U << PF_VECTOR;
+
+	test_hardware_npf(svm, npf_gpa);
+	test_hardware_fetch_npf(svm, fetch_npf_gva, fetch_npf_gpa);
+	test_synthesized_npf(svm, npf_gpa);
+	test_synthesized_npf_truncated(svm, npf_gpa);
+	test_synthesized_npf_cs_limit(svm, npf_gpa);
+	test_intercepted_pf(svm);
+	test_intercepted_fetch_pf(svm);
+	test_synthesized_pf(svm);
+	test_userspace_injected_pf(svm);
+	test_instruction_intercepts(svm);
+	test_vmmcall(svm);
+
+	GUEST_DONE();
+}
+
+static void prepare_boundary_outsb_code(struct kvm_vm *vm)
+{
+	gva_t code_gva = (gva_t)&boundary_outsb_code[PAGE_SIZE -
+						      BOUNDARY_OUTSB_CODE_SIZE];
+	u8 *code = addr_gva2hva(vm, code_gva);
+	u64 source = (u64)&npf_target;
+
+	/* movabs $npf_target, %rsi */
+	code[0] = 0x48;
+	code[1] = 0xbe;
+	memcpy(&code[2], &source, sizeof(source));
+
+	/* mov $0x80, %dx; outsb */
+	code[10] = 0x66;
+	code[11] = 0xba;
+	code[12] = 0x80;
+	code[13] = 0x00;
+	code[14] = OUTSB_OPCODE;
+}
+
+static void prepare_limit_outsb_code(struct kvm_vm *vm)
+{
+	u8 *code = addr_gva2hva(vm, (gva_t)limit_outsb_code);
+	u32 source = (u32)(u64)&npf_target;
+
+	TEST_ASSERT((u64)&npf_target <= UINT32_MAX,
+		    "npf_target must be addressable from compatibility mode");
+	TEST_ASSERT((u64)&limit_outsb_code[LIMIT_OUTSB_OPCODE_OFFSET] <=
+		    UINT32_MAX,
+		    "limit_outsb_code must be addressable from compatibility mode");
+
+	/* mov $npf_target, %esi; mov $0x80, %dx; outsb */
+	code[0] = 0xbe;
+	memcpy(&code[1], &source, sizeof(source));
+	code[5] = 0x66;
+	code[6] = 0xba;
+	code[7] = 0x80;
+	code[8] = 0x00;
+	code[LIMIT_OUTSB_OPCODE_OFFSET] = OUTSB_OPCODE;
+}
+
+static void queue_userspace_pf(struct kvm_vcpu *vcpu)
+{
+	struct kvm_vcpu_events events;
+
+	vcpu_events_get(vcpu, &events);
+	TEST_ASSERT(!events.exception.pending && !events.exception.injected,
+		    "Unexpected exception queued before userspace #PF injection");
+	TEST_ASSERT(events.flags & KVM_VCPUEVENT_VALID_PAYLOAD,
+		    "KVM_CAP_EXCEPTION_PAYLOAD was not enabled");
+
+	events.flags |= KVM_VCPUEVENT_VALID_PAYLOAD;
+	events.exception.injected = false;
+	events.exception.pending = true;
+	events.exception.nr = PF_VECTOR;
+	events.exception.has_error_code = true;
+	events.exception.error_code = 0;
+	events.exception_has_payload = true;
+	events.exception_payload = PF_TEST_GVA;
+	vcpu_events_set(vcpu, &events);
+}
+
+int main(int argc, char *argv[])
+{
+	gva_t svm_gva, npf_gva, fetch_npf_gva, boundary_page_gva, iopm_gva;
+	gpa_t npf_gpa, fetch_npf_gpa, iopm_gpa;
+	struct kvm_vcpu *vcpu;
+	struct kvm_vm *vm;
+	u64 *pte;
+
+	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_SVM));
+	TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_NPT));
+	TEST_REQUIRE(this_cpu_has(X86_FEATURE_DECODEASSISTS));
+	TEST_ASSERT(kvm_cpu_has(X86_FEATURE_DECODEASSISTS),
+		    "KVM failed to expose DecodeAssists");
+	TEST_REQUIRE(kvm_has_cap(KVM_CAP_EXCEPTION_PAYLOAD));
+
+	vm = vm_create_with_one_vcpu(&vcpu, l1_guest_code);
+	vm_enable_cap(vm, KVM_CAP_EXCEPTION_PAYLOAD, 1);
+	prepare_boundary_outsb_code(vm);
+	prepare_limit_outsb_code(vm);
+	vm_enable_npt(vm);
+	vcpu_alloc_svm(vm, &svm_gva);
+	iopm_gva = vm_alloc_pages(vm, TEST_IOPM_SIZE / PAGE_SIZE);
+	iopm_gpa = addr_gva2gpa(vm, iopm_gva);
+	memset(addr_gva2hva(vm, iopm_gva), 0, TEST_IOPM_SIZE);
+	npf_gva = (gva_t)&npf_target;
+	npf_gpa = addr_gva2gpa(vm, npf_gva);
+	fetch_npf_gva = vm_alloc_page(vm);
+	fetch_npf_gpa = addr_gva2gpa(vm, fetch_npf_gva);
+	*(u8 *)addr_gva2hva(vm, fetch_npf_gva) = INT3_OPCODE;
+
+	tdp_identity_map_default_memslots(vm);
+	pte = tdp_get_pte(vm, npf_gpa);
+	*pte &= ~PTE_PRESENT_MASK(&vm->stage2_mmu);
+	pte = tdp_get_pte(vm, fetch_npf_gpa);
+	*pte &= ~PTE_PRESENT_MASK(&vm->stage2_mmu);
+
+	boundary_page_gva = (gva_t)&boundary_outsb_code[PAGE_SIZE];
+	pte = vm_get_pte(vm, boundary_page_gva);
+	*pte &= ~PTE_PRESENT_MASK(&vm->mmu);
+
+	vcpu_args_set(vcpu, 5, svm_gva, npf_gpa, fetch_npf_gva,
+		      fetch_npf_gpa, iopm_gpa);
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	{
+		struct ucall uc;
+
+		TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC);
+		TEST_ASSERT_EQ(uc.args[1], USERSPACE_PF_STAGE);
+	}
+	queue_userspace_pf(vcpu);
+	vcpu_run(vcpu);
+	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
+	TEST_ASSERT_EQ(get_ucall(vcpu, NULL), UCALL_DONE);
+
+	kvm_vm_free(vm);
+	return 0;
+}
-- 
2.43.7

Re: [PATCH v2 8/8] KVM: selftests: Add nested SVM DecodeAssists test

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Unmapping the page containing `npf_target` may accidentally unmap other variables like `l2_guest_stack`, causing unintended page faults.
- [Medium] Guest assertion failures are obscured because `get_ucall()`'s return value is checked with a simple equality assertion rather than handling `UCALL_ABORT`.
--

--- Patch [8]: [PATCH v2 8/8] KVM: selftests: Add nested SVM DecodeAssists test ---
commit 022271cf32a29cba4b8af8cdce602deac0fd3ec4
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: selftests: Add nested SVM DecodeAssists test

This commit adds a nested SVM selftest for DecodeAssists to verify that KVM
exposes the feature to L1 and provides the correct architectural exit state
for various intercepts.

> diff --git a/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c b/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c
> new file mode 100644
> index 0000000000000..ac9601dda9365
> --- /dev/null
> +++ b/tools/testing/selftests/kvm/x86/svm_nested_decode_assists_test.c
> @@ -0,0 +1,694 @@

[ ... ]

> +#define USERSPACE_PF_STAGE 1
> +#define TEST_IOPM_SIZE (3 * PAGE_SIZE)
> +
> +static u64 npf_target __aligned(PAGE_SIZE);
> +static u8 boundary_outsb_code[2 * PAGE_SIZE] __aligned(PAGE_SIZE);
> +static u8 limit_outsb_code[PAGE_SIZE] __aligned(PAGE_SIZE);
> +static unsigned long l2_guest_stack[L2_GUEST_STACK_SIZE];

[ ... ]

> +	tdp_identity_map_default_memslots(vm);
> +	pte = tdp_get_pte(vm, npf_gpa);
> +	*pte &= ~PTE_PRESENT_MASK(&vm->stage2_mmu);

[Severity: Medium]
Since npf_target is only 8 bytes long but page-aligned, could the linker
pack other variables (like l2_guest_stack) into the remaining 4088 bytes
of the same page?

If so, when this code unmaps the stage-2 PTE for the page containing
npf_target, it might accidentally unmap the guest stack. Would L2
execution then #NPF on its first stack operation and fail the test
misleadingly?

> +	pte = tdp_get_pte(vm, fetch_npf_gpa);
> +	*pte &= ~PTE_PRESENT_MASK(&vm->stage2_mmu);
> +
> +	boundary_page_gva = (gva_t)&boundary_outsb_code[PAGE_SIZE];
> +	pte = vm_get_pte(vm, boundary_page_gva);
> +	*pte &= ~PTE_PRESENT_MASK(&vm->mmu);
> +
> +	vcpu_args_set(vcpu, 5, svm_gva, npf_gpa, fetch_npf_gva,
> +		      fetch_npf_gpa, iopm_gpa);
> +	vcpu_run(vcpu);
> +	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
> +	{
> +		struct ucall uc;
> +
> +		TEST_ASSERT_EQ(get_ucall(vcpu, &uc), UCALL_SYNC);

[Severity: Medium]
If a guest assertion fails, it exits with UCALL_ABORT containing detailed
file, line, and message information.

By using a simple equality assertion here, wouldn't a guest failure result
in a generic host-side failure (e.g., "Expected UCALL_SYNC, got UCALL_ABORT")
and drop the actual guest failure reason, making it harder to debug?

> +		TEST_ASSERT_EQ(uc.args[1], USERSPACE_PF_STAGE);
> +	}
> +	queue_userspace_pf(vcpu);
> +	vcpu_run(vcpu);
> +	TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO);
> +	TEST_ASSERT_EQ(get_ucall(vcpu, NULL), UCALL_DONE);

[Severity: Medium]
Similarly, wouldn't this assertion also silently drop the payload of a
UCALL_ABORT if the guest aborts at the end of the test?

> +
> +	kvm_vm_free(vm);
> +	return 0;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1783999988.git.zhang_wei@open-hieco.net?part=8

Re: [PATCH v2 3/8] KVM: nSVM: Track fresh VMCB02 DecodeAssist bytes

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] The patch fails to set `vmcb02_insn_bytes_fresh` to `true` when reflecting a host-intercepted Guest Page Fault (`#PF`) to L1, discarding hardware-provided DecodeAssist bytes.

Pre-existing issues:
- [High] KVM incorrectly identifies synthesized nested page faults (NPFs) as hardware-generated if they occur during emulation of an original hardware NPF, passing a stale `exit_info_2` (GPA) to L1.
--

--- Patch [3]: [PATCH v2 3/8] KVM: nSVM: Track fresh VMCB02 DecodeAssist bytes ---
commit 1b65a65904c11af4f22d9f6d0ea2c488ec16575d
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: nSVM: Track fresh VMCB02 DecodeAssist bytes

Track whether VMCB02 contains DecodeAssist instruction bytes from the
hardware VM-Exit currently being reflected to L1. Clear the fields before
each nested run so stale bytes cannot be mistaken for fresh hardware state.

> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index b340dc9991adb..70e2ee3af78b2 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -33,13 +33,21 @@
>  
>  #define CC KVM_NESTED_VMENTER_CONSISTENCY_CHECK
>  
> +static void nested_svm_clear_insn_bytes(struct vmcb *vmcb)
> +{
> +	vmcb->control.insn_len = 0;
> +	memset(vmcb->control.insn_bytes, 0,
> +	       sizeof(vmcb->control.insn_bytes));
> +}
> +
>  static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
>  				       struct x86_exception *fault)
>  {
>  	struct vcpu_svm *svm = to_svm(vcpu);
>  	struct vmcb *vmcb = svm->vmcb;
> +	bool from_hardware = vmcb->control.exit_code == SVM_EXIT_NPF;
>  
> -	if (vmcb->control.exit_code != SVM_EXIT_NPF) {
> +	if (!from_hardware) {

[Severity: High]
This isn't a bug introduced by this patch, but does evaluating the original
hardware exit code correctly identify emulator-synthesized nested page faults?

If an L2 guest accesses an unmapped MMIO region, it causes a hardware NPF.
L0 intercepts it and starts emulation. If the emulator then performs a guest
page table walk or instruction fetch which faults in the nested NPT, the
emulator injects a new NPF to L1 by calling nested_svm_inject_npf_exit().

Because the VMCB exit code still contains the original hardware NPF exit code,
this check evaluates to true. This causes KVM to skip updating the faulting
GPA with the new faulting address, misleading the L1 hypervisor with the
original MMIO GPA and incorrectly marking stale instruction bytes as fresh.

[ ... ]
> diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
> index 5137416be593d..fa65f6a1a59ba 100644
> --- a/arch/x86/kvm/svm/svm.h
> +++ b/arch/x86/kvm/svm/svm.h
> @@ -238,6 +238,13 @@ struct svm_nested_state {
>  	 * on its side.
>  	 */
>  	bool force_msr_bitmap_recalc;
> +
> +	/*
> +	 * True if VMCB02's DecodeAssist instruction bytes belong to the hardware
> +	 * VM-Exit currently being reflected to L1.  KVM-synthesized exits leave
> +	 * it clear.
> +	 */
> +	bool vmcb02_insn_bytes_fresh;

[Severity: High]
Does KVM need to set vmcb02_insn_bytes_fresh to true when reflecting a 
host-intercepted Guest Page Fault (#PF) to L1?

If L0 KVM intercepts a #PF from L2 (e.g., for shadow paging or async PF),
it processes the fault and later queues it for reflection to L1 if L1 also
intercepts it. KVM synthesizes the exit via
nested_svm_inject_exception_vmexit():

arch/x86/kvm/svm/nested.c:nested_svm_inject_exception_vmexit() {
	...
	vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + ex->vector;
	...
}

However, this path fails to set vmcb02_insn_bytes_fresh to true. Since the
flag defaults to false when entering L2, it remains false, and any
hardware-provided DecodeAssist bytes for the #PF are discarded. This could
break instruction decoding and emulation logic in the L1 hypervisor.

>  };
>  
>  struct vcpu_sev_es_state {

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1783999988.git.zhang_wei@open-hieco.net?part=3

Re: [PATCH v2 2/8] KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Unconditionally setting bit 63 in EXITINFO1 for CR read intercepts.
--

--- Patch [2]: [PATCH v2 2/8] KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts ---
commit 12fdeb07951db9a20a07afc60e302f9b9e4fcaca
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: nSVM: Synthesize DecodeAssists EXITINFO for emulated intercepts

This commit populates the architectural EXITINFO fields when DecodeAssists is 
exposed to L1 and an emulated intercept occurs. It generates EXITINFO1 
depending on the intercept type, such as CR/DR accesses and INTn.

> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index e02a38da5296e..cc35afb59623f 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
[ ... ]
> @@ -4754,6 +4754,52 @@ static const struct __x86_intercept {
>  #undef POST_EX
>  #undef POST_MEM
>  
> +static void svm_prepare_decode_assist_exit_info(struct kvm_vcpu *vcpu,
> +						const struct x86_instruction_info *info)
> +{
> +	struct vmcb *vmcb = to_svm(vcpu)->vmcb;
> +	u64 exit_info_1;
> +
> +	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
> +		return;
> +
> +	switch (info->intercept) {
> +	case x86_intercept_cr_read:
> +	case x86_intercept_cr_write:
> +		/* MOV CRx: bit 63 set, GPR number in bits 3:0. */
> +		exit_info_1 = BIT_ULL(63) | (info->modrm_rm & 0xf);
                ^^^^^^^^^^^
[Severity: High]
Does this code incorrectly set bit 63 of EXITINFO1 for CR reads?

According to the AMD Architecture Programmer's Manual (APM), bit 63 of
EXITINFO1 for a CR intercept indicates the operation type, where 1 means a
write to the CR and 0 means a read from the CR. By unconditionally setting
BIT_ULL(63) for both x86_intercept_cr_read and x86_intercept_cr_write,
wouldn't KVM report CR reads as CR writes to the L1 hypervisor?

If an L2 guest executes a MOV from CR instruction, this could cause the L1
hypervisor to corrupt its shadow CPU state or misbehave, because it will
be provided a malformed EXITINFO1 and handle the event as a write operation
rather than a read.

> +		break;

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1783999988.git.zhang_wei@open-hieco.net?part=2

Re: [PATCH v2 4/8] KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] Hardware DecodeAssist bytes are not propagated to VMCB12 for intercepted #PF exits when KVM reflects the page fault, breaking the feature.
- [Medium] KVM reads control fields directly from guest-controlled memory (VMCB12) to determine if DecodeAssist bytes should be copied, creating a TOCTOU race condition.
--

--- Patch [4]: [PATCH v2 4/8] KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12 ---
commit 3aed32a695a3f569675b1941e25ebc814c2dc09d
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: nSVM: Propagate hardware DecodeAssist bytes to VMCB12

When DecodeAssists is exposed to L1, copy fresh hardware instruction
bytes from VMCB02 to VMCB12 for data #NPF and intercepted #PF VM-Exits.
Clear VMCB12's instruction-byte state for instruction-fetch faults,
unrelated exits, and exits without fresh VMCB02 state so that stale bytes
are not exposed to L1.

> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index 70e2ee3af78b2..9e12eab7eed97 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
[ ... ]
> +static bool nested_svm_vmexit_supports_insn_bytes(struct kvm_vcpu *vcpu,
> +						  const struct vmcb *vmcb12)
> +{
> +	u64 exit_code = vmcb12->control.exit_code;
> +
> +	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
> +		return false;
> +
> +	if (exit_code != SVM_EXIT_NPF &&
> +	    exit_code != SVM_EXIT_EXCP_BASE + PF_VECTOR)
> +		return false;
> +
> +	return !(vmcb12->control.exit_info_1 & PFERR_FETCH_MASK);

[Severity: Medium]
Could this create a TOCTOU race condition by reading control fields directly
from guest-controlled memory? 

Because vmcb12 is mapped to guest physical memory, a malicious L1 guest on a
sibling vCPU could concurrently modify these fields. For instance, changing
the exit code to SVM_EXIT_NPF after KVM has already populated it.

Would this trick KVM into bypassing the intended security clearing logic,
thereby leaking DecodeAssist bytes for unrelated exits that were supposed to
be scrubbed? Should KVM read these fields from the trusted host-managed
vmcb02 instead?

> +}
> +
> +/*
> + * Rebuild VMCB12's DecodeAssist bytes for the nested VM-Exit.  Use fresh
> + * hardware VMCB02 state when available; otherwise report a zero byte count.
> + */
> +static void nested_svm_update_vmcb12_insn_bytes(struct kvm_vcpu *vcpu,
> +						struct vmcb *vmcb12,
> +						struct vmcb *vmcb02)
> +{
> +	struct vcpu_svm *svm = to_svm(vcpu);
> +	bool vmcb02_insn_bytes_fresh = svm->nested.vmcb02_insn_bytes_fresh;
> +
> +	svm->nested.vmcb02_insn_bytes_fresh = false;
> +
> +	nested_svm_clear_insn_bytes(vmcb12);
> +
> +	if (!nested_svm_vmexit_supports_insn_bytes(vcpu, vmcb12))
> +		return;
> +
> +	if (vmcb02_insn_bytes_fresh)
> +		nested_svm_copy_insn_bytes(vmcb12, vmcb02);

[Severity: High]
Does KVM fail to propagate hardware DecodeAssist bytes for intercepted #PF
exits when it reflects the page fault?

When L0 intercepts a #PF and reflects it to L1 via
nested_svm_inject_exception_vmexit():

nested_svm_inject_exception_vmexit() {
    ...
    vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + ex->vector;
    ...
    nested_svm_vmexit(svm);
}

It seems nested_svm_inject_exception_vmexit() fails to set the
vmcb02_insn_bytes_fresh flag. Because the flag remains false, the valid
hardware instruction bytes are silently dropped here instead of being copied
to VMCB12. 

Will this cause L1 hypervisors relying on DecodeAssist for #PF to receive an
empty instruction buffer and fail to decode the faulting instruction?

> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1783999988.git.zhang_wei@open-hieco.net?part=4

Re: [PATCH v2 5/8] KVM: nSVM: Use emulator bytes for synthesized nested #NPF/#PF

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Flawed heuristic in `nested_svm_prepare_synthesized_insn_bytes` leaks stale instruction bytes to L1 DecodeAssist fields during native #PF/#NPF exits.
--

--- Patch [5]: [PATCH v2 5/8] KVM: nSVM: Use emulator bytes for synthesized nested #NPF/#PF ---
commit d6e2023803dd9b80aeaf2560525bc97569fd2d75
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: nSVM: Use emulator bytes for synthesized nested #NPF/#PF

This commit updates KVM to synthesize DecodeAssist instruction bytes from the
emulator fetch cache for KVM-generated #NPF/#PF exits. It introduces a
heuristic to match the emulator context against the current instruction pointer
to populate the buffer, which is then copied to VMCB12 when constructing the
nested VM-Exit.

> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index 9e12eab7eed97..8cee9aeaf4f27 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -69,26 +69,72 @@ static bool nested_svm_vmexit_supports_insn_bytes(struct kvm_vcpu *vcpu,

[ ... ]

> +static void nested_svm_prepare_synthesized_insn_bytes(struct kvm_vcpu *vcpu)
> +{
> +	struct vcpu_svm *svm = to_svm(vcpu);
> +	struct nested_svm_insn_bytes *insn_bytes =
> +		&svm->nested.synthesized_insn_bytes;
> +	const u8 max_bytes = sizeof(insn_bytes->insn_bytes);
> +	struct x86_emulate_ctxt *ctxt = vcpu->arch.emulate_ctxt;
> +
> +	nested_svm_clear_synthesized_insn_bytes(svm);
> +
> +	if (!guest_cpu_cap_has(vcpu, X86_FEATURE_DECODEASSISTS))
> +		return;
> +
> +	if (!ctxt || ctxt->eip != kvm_rip_read(vcpu) ||

[Severity: High]
Does this heuristic incorrectly validate the emulator cache based solely on RIP
equality?

If an instruction is successfully emulated (such as an MMIO access), the
emulator advances ctxt->eip to the next instruction's RIP. If the guest then
resumes natively and that next instruction triggers an intercepted #PF,
ctxt->eip will exactly match kvm_rip_read(vcpu).

Since ctxt->fetch.data still contains the uncleared bytes of the previously
emulated instruction, won't this cause KVM to incorrectly synthesize and
inject those stale instruction bytes into L1's VMCB12 DecodeAssist fields?

> +	    ctxt->fetch.end < ctxt->fetch.data ||
> +	    ctxt->fetch.end > ctxt->fetch.data + max_bytes)
> +		return;
> +
> +	insn_bytes->insn_len = ctxt->fetch.end - ctxt->fetch.data;
> +	memcpy(insn_bytes->insn_bytes, ctxt->fetch.data,
> +	       insn_bytes->insn_len);
> +	insn_bytes->prepared = true;
> +}

[ ... ]

> @@ -1709,6 +1758,9 @@ static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu)
>  			vmcb->control.exit_info_2 = ex->payload;
>  		else
>  			vmcb->control.exit_info_2 = vcpu->arch.cr2;
> +
> +		if (!ex->has_error_code || !(ex->error_code & PFERR_FETCH_MASK))
> +			nested_svm_prepare_synthesized_insn_bytes(vcpu);

[Severity: High]
Is it safe to call the synthesis function here for #PF exits without knowing
if the queued exception actually originated from the emulator?

Calling this unconditionally allows hardware-generated page faults to trigger
the heuristic above. An unprivileged L2 guest could potentially exploit this
data leak by placing a faulting instruction immediately after an MMIO
instruction to spoof DecodeAssist bytes provided to L1.

>  	} else if (ex->vector == DB_VECTOR) {
>  		/* See kvm_check_and_inject_events().  */
>  		kvm_deliver_exception_payload(vcpu, ex);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1783999988.git.zhang_wei@open-hieco.net?part=5

Re: [PATCH v2 6/8] KVM: nSVM: Fetch missing DecodeAssist bytes for synthesized #NPF/#PF

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing 32-bit linear address truncation when computing `rip + count` for instruction fetch.
--

--- Patch [6]: [PATCH v2 6/8] KVM: nSVM: Fetch missing DecodeAssist bytes for synthesized #NPF/#PF ---
commit 92b0b1d782b47781c3462fcdfb9a37ae5af201c8
Author: Tina Zhang <zhang_wei@open-hieco.net>

KVM: nSVM: Fetch missing DecodeAssist bytes for synthesized #NPF/#PF

This commit updates nested SVM to fetch missing DecodeAssist instruction
bytes directly from the L2 guest page tables when the emulator fetch cache
is incomplete. It honors CS limits and handles page faults, but skips tail
reads for SEV guests.

> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -75,6 +75,54 @@ static void nested_svm_clear_synthesized_insn_bytes(struct vcpu_svm *svm)

[ ... ]

> +static u8 nested_svm_fetch_insn_bytes(struct kvm_vcpu *vcpu, u8 *bytes,
> +				      u8 count, u8 max_bytes)
> +{
> +	struct kvm_mmu *mmu = vcpu->arch.walk_mmu;
> +	u64 access = PFERR_FETCH_MASK;
> +	gva_t rip = kvm_get_linear_rip(vcpu);
> +	struct x86_exception e;
> +
> +	if (to_svm(vcpu)->vmcb->save.cpl == 3)
> +		access |= PFERR_USER_MASK;
> +
> +	/*
> +	 * Hardware truncates the fetch at the CS limit.  CS.base and CS.limit
> +	 * checks do not apply in 64-bit mode.
> +	 */
> +	if (!is_64_bit_mode(vcpu)) {
> +		u32 eip = kvm_rip_read(vcpu);
> +		u32 limit = to_svm(vcpu)->vmcb->save.cs.limit;
> +
> +		if (eip > limit)
> +			return 0;
> +		max_bytes = min_t(u64, max_bytes, (u64)limit - eip + 1);
> +	}
> +
> +	count = min(count, max_bytes);
> +
> +	while (count < max_bytes) {
> +		unsigned int chunk = min_t(unsigned int, max_bytes - count,
> +				PAGE_SIZE - offset_in_page(rip + count));
> +		gpa_t gpa = mmu->gva_to_gpa(vcpu, mmu, rip + count, access, &e);

[Severity: Medium]
Does this properly handle linear address wrap-around for 32-bit guests?

For a 32-bit L2 guest, if CS.base + EIP is close to 0xffffffff, adding
count to rip can cause rip + count to exceed 4GB. Because rip is a
64-bit gva_t, the addition won't automatically truncate to 32 bits,
meaning the address will evaluate as greater than 4GB rather than wrapping
to 0.

Could this cause KVM to fetch instruction bytes from the wrong guest
physical address when the instruction fetch wraps around the 4GB boundary?

> +
> +		if (gpa == INVALID_GPA ||
> +		    kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(gpa),
> +					     bytes + count,
> +					     offset_in_page(gpa), chunk))
> +			break;
> +
> +		count += chunk;
> +	}
> +
> +	return count;
> +}

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1783999988.git.zhang_wei@open-hieco.net?part=6