[PATCH 0/9] KVM: SVM: Fix (hilarious) exit_code bugs

[PATCH 0/9] KVM: SVM: Fix (hilarious) exit_code bugs

[Sean Christopherson]

Hyper-V folks, y'all are getting Cc'd because of a change in
include/hyperv/hvgdk.h to ensure HV_SVM_EXITCODE_ENL is an unsigned value.
AFAICT, only KVM consumes that macro.  That said, any insight you can provide
on relevant Hyper-V behavior would be appreciated :-)


Fix bugs in SVM that mostly impact nested SVM where KVM treats exit codes
as 32-bit values instead of 64-bit values.  I have no idea how KVM ended up
with such an egregious flaw, as the blame trail goes all the way back to
commit 6aa8b732ca01 ("[PATCH] kvm: userspace interface").  Maybe there was
pre-production hardware or something?

I'm also fairly surprised no one has noticed, as at least Xen treats exit
codes as 64-bit values.  Maybe the only people that run hypervisor tests on
top of KVM are also running KVM, or similarly buggy tests?  /shrug

The most dangerous aspect of the mess is that simply fixing KVM would likely
break KVM-on-KVM setups if only L1 is patched.  To try and avoid such
breakage while also fixing KVM, I opted to have KVM retain its checks on
only bits 31:0 if KVM is running as a VM (as detected by
X86_FEATURE_HYPERVISOR).

I stumbled on this when trying to resolve a array_index_nospec() build failure
on 32-bit kernels (array_index_nospec() requires the index to fit in an
"unsigned long").

Oh, and I have KUT changes to detect the nSVM bugs.

Because of the potential for breakage, I tagged only the nSVM fixes for
stable@.  E.g. I almost botched things by sending this as two separate
series, which would have create a window where svm_invoke_exit_handler()
would process a 64-bit code when running KVM-on-KVM and thus break if L0
KVM left gargage in bits 63:32.

Sean Christopherson (9):
  KVM: nSVM: Clear exit_code_hi in VMCB when synthesizing nested
    VM-Exits
  KVM: nSVM: Set exit_code_hi to -1 when synthesizing SVM_EXIT_ERR
    (failed VMRUN)
  KVM: SVM: Add a helper to detect VMRUN failures
  KVM: SVM: Open code handling of unexpected exits in
    svm_invoke_exit_handler()
  KVM: SVM: Check for an unexpected VM-Exit after RETPOLINE "fast"
    handling
  KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on
    bare metal
  KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of
    KVM
  KVM: SVM: Limit incorrect check on SVM_EXIT_ERR to running as a VM
  KVM: SVM: Harden exit_code against being used in Spectre-like attacks

 arch/x86/include/asm/svm.h      |  3 +-
 arch/x86/include/uapi/asm/svm.h | 32 ++++++++++-----------
 arch/x86/kvm/svm/hyperv.c       |  1 -
 arch/x86/kvm/svm/nested.c       | 29 +++++++------------
 arch/x86/kvm/svm/sev.c          | 36 ++++++++----------------
 arch/x86/kvm/svm/svm.c          | 49 +++++++++++++++++++--------------
 arch/x86/kvm/svm/svm.h          | 17 ++++++++----
 arch/x86/kvm/trace.h            |  2 +-
 include/hyperv/hvgdk.h          |  2 +-
 9 files changed, 82 insertions(+), 89 deletions(-)


base-commit: 16ec4fb4ac95d878b879192d280db2baeec43272
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 1/9] KVM: nSVM: Clear exit_code_hi in VMCB when synthesizing nested VM-Exits

[Sean Christopherson]

Explicitly clear exit_code_hi in the VMCB when synthesizing "normal"
nested VM-Exits, as the full exit code is a 64-bit value (spoiler alert),
and all exit codes for non-failing VMRUN use only bits 31:0.

Cc: Jim Mattson <jmattson@google.com>
Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/svm.c | 2 ++
 arch/x86/kvm/svm/svm.h | 7 ++++---
 2 files changed, 6 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index fc42bcdbb520..7ea034ee6b6c 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2433,6 +2433,7 @@ static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
 
 	if (cr0 ^ val) {
 		svm->vmcb->control.exit_code = SVM_EXIT_CR0_SEL_WRITE;
+		svm->vmcb->control.exit_code_hi = 0;
 		ret = (nested_svm_exit_handled(svm) == NESTED_EXIT_DONE);
 	}
 
@@ -4608,6 +4609,7 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
 	if (static_cpu_has(X86_FEATURE_NRIPS))
 		vmcb->control.next_rip  = info->next_rip;
 	vmcb->control.exit_code = icpt_info.exit_code;
+	vmcb->control.exit_code_hi = 0;
 	vmexit = nested_svm_exit_handled(svm);
 
 	ret = (vmexit == NESTED_EXIT_DONE) ? X86EMUL_INTERCEPTED
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index c2acaa49ee1c..253a8dca412c 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -763,9 +763,10 @@ int nested_svm_vmexit(struct vcpu_svm *svm);
 
 static inline int nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
 {
-	svm->vmcb->control.exit_code   = exit_code;
-	svm->vmcb->control.exit_info_1 = 0;
-	svm->vmcb->control.exit_info_2 = 0;
+	svm->vmcb->control.exit_code	= exit_code;
+	svm->vmcb->control.exit_code_hi	= 0;
+	svm->vmcb->control.exit_info_1	= 0;
+	svm->vmcb->control.exit_info_2	= 0;
 	return nested_svm_vmexit(svm);
 }
 
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 2/9] KVM: nSVM: Set exit_code_hi to -1 when synthesizing SVM_EXIT_ERR (failed VMRUN)

[Sean Christopherson]

Set exit_code_hi to -1u as a temporary band-aid to fix a long-standing
(effectively since KVM's inception) bug where KVM treats the exit code as
a 32-bit value, when in reality it's a 64-bit value.  Per the APM, offset
0x70 is a single 64-bit value:

  070h 63:0 EXITCODE

And a sane reading of the error values defined in "Table C-1. SVM Intercept
Codes" is that negative values use the full 64 bits:

  –1 VMEXIT_INVALID Invalid guest state in VMCB.
  –2 VMEXIT_BUSYBUSY bit was set in the VMSA
  –3 VMEXIT_IDLE_REQUIREDThe sibling thread is not in an idle state
  -4 VMEXIT_INVALID_PMC Invalid PMC state

And that interpretation is confirmed by testing on Milan and Turin (by
setting bits in CR0[63:32] to generate VMEXIT_INVALID on VMRUN).

Furthermore, Xen has treated exitcode as a 64-bit value since HVM support
was adding in 2006 (see Xen commit d1bd157fbc ("Big merge the HVM
full-virtualisation abstractions.")).

Cc: Jim Mattson <jmattson@google.com>
Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
Cc: stable@vger.kernel.org
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index c81005b24522..ba0f11c68372 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -985,7 +985,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
 	if (!nested_vmcb_check_save(vcpu) ||
 	    !nested_vmcb_check_controls(vcpu)) {
 		vmcb12->control.exit_code    = SVM_EXIT_ERR;
-		vmcb12->control.exit_code_hi = 0;
+		vmcb12->control.exit_code_hi = -1u;
 		vmcb12->control.exit_info_1  = 0;
 		vmcb12->control.exit_info_2  = 0;
 		goto out;
@@ -1018,7 +1018,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
 	svm->soft_int_injected = false;
 
 	svm->vmcb->control.exit_code    = SVM_EXIT_ERR;
-	svm->vmcb->control.exit_code_hi = 0;
+	svm->vmcb->control.exit_code_hi = -1u;
 	svm->vmcb->control.exit_info_1  = 0;
 	svm->vmcb->control.exit_info_2  = 0;
 
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 3/9] KVM: SVM: Add a helper to detect VMRUN failures

[Sean Christopherson]

Add a helper to detect VMRUN failures so that KVM can guard against its
own long-standing bug, where KVM neglects to set exitcode[63:32] when
synthesizing a nested VMFAIL_INVALID VM-Exit.  This will allow fixing
KVM's mess of treating exitcode as two separate 32-bit values without
breaking KVM-on-KVM when running on an older, unfixed KVM.

Cc: Jim Mattson <jmattson@google.com>
Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/nested.c | 16 +++++++---------
 arch/x86/kvm/svm/svm.c    |  4 ++--
 arch/x86/kvm/svm/svm.h    |  5 +++++
 3 files changed, 14 insertions(+), 11 deletions(-)

diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index ba0f11c68372..8070e20ed5a7 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1134,7 +1134,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 	vmcb12->control.exit_info_1       = vmcb02->control.exit_info_1;
 	vmcb12->control.exit_info_2       = vmcb02->control.exit_info_2;
 
-	if (vmcb12->control.exit_code != SVM_EXIT_ERR)
+	if (svm_is_vmrun_failure(vmcb12->control.exit_code))
 		nested_save_pending_event_to_vmcb12(svm, vmcb12);
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
@@ -1425,6 +1425,9 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
 	u32 exit_code = svm->vmcb->control.exit_code;
 	int vmexit = NESTED_EXIT_HOST;
 
+	if (svm_is_vmrun_failure(exit_code))
+		return NESTED_EXIT_DONE;
+
 	switch (exit_code) {
 	case SVM_EXIT_MSR:
 		vmexit = nested_svm_exit_handled_msr(svm);
@@ -1432,7 +1435,7 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
 	case SVM_EXIT_IOIO:
 		vmexit = nested_svm_intercept_ioio(svm);
 		break;
-	case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f: {
+	case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f:
 		/*
 		 * Host-intercepted exceptions have been checked already in
 		 * nested_svm_exit_special.  There is nothing to do here,
@@ -1440,15 +1443,10 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
 		 */
 		vmexit = NESTED_EXIT_DONE;
 		break;
-	}
-	case SVM_EXIT_ERR: {
-		vmexit = NESTED_EXIT_DONE;
-		break;
-	}
-	default: {
+	default:
 		if (vmcb12_is_intercept(&svm->nested.ctl, exit_code))
 			vmexit = NESTED_EXIT_DONE;
-	}
+		break;
 	}
 
 	return vmexit;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 7ea034ee6b6c..52b759408853 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3530,7 +3530,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 			return 1;
 	}
 
-	if (svm->vmcb->control.exit_code == SVM_EXIT_ERR) {
+	if (svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
 		kvm_run->exit_reason = KVM_EXIT_FAIL_ENTRY;
 		kvm_run->fail_entry.hardware_entry_failure_reason
 			= svm->vmcb->control.exit_code;
@@ -4302,7 +4302,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
 
 		/* Track VMRUNs that have made past consistency checking */
 		if (svm->nested.nested_run_pending &&
-		    svm->vmcb->control.exit_code != SVM_EXIT_ERR)
+		    !svm_is_vmrun_failure(svm->vmcb->control.exit_code))
                         ++vcpu->stat.nested_run;
 
 		svm->nested.nested_run_pending = 0;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 253a8dca412c..6b35925e3a33 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -426,6 +426,11 @@ static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
 	return container_of(vcpu, struct vcpu_svm, vcpu);
 }
 
+static inline bool svm_is_vmrun_failure(u64 exit_code)
+{
+	return (u32)exit_code == (u32)SVM_EXIT_ERR;
+}
+
 /*
  * Only the PDPTRs are loaded on demand into the shadow MMU.  All other
  * fields are synchronized on VM-Exit, because accessing the VMCB is cheap.
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 4/9] KVM: SVM: Open code handling of unexpected exits in svm_invoke_exit_handler()

[Sean Christopherson]

Fold svm_check_exit_valid() and svm_handle_invalid_exit() into their sole
caller, svm_invoke_exit_handler(), as having tiny single-use helpers makes
the code unncessarily difficult to follow.  This will also allow for
additional cleanups in svm_invoke_exit_handler().

No functional change intended.

Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/svm.c | 25 ++++++++++---------------
 1 file changed, 10 insertions(+), 15 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 52b759408853..638a67ef0c37 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3433,23 +3433,13 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
 		sev_free_decrypted_vmsa(vcpu, save);
 }
 
-static bool svm_check_exit_valid(u64 exit_code)
-{
-	return (exit_code < ARRAY_SIZE(svm_exit_handlers) &&
-		svm_exit_handlers[exit_code]);
-}
-
-static int svm_handle_invalid_exit(struct kvm_vcpu *vcpu, u64 exit_code)
-{
-	dump_vmcb(vcpu);
-	kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
-	return 0;
-}
-
 int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
 {
-	if (!svm_check_exit_valid(exit_code))
-		return svm_handle_invalid_exit(vcpu, exit_code);
+	if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
+		goto unexpected_vmexit;
+
+	if (!svm_exit_handlers[exit_code])
+		goto unexpected_vmexit;
 
 #ifdef CONFIG_MITIGATION_RETPOLINE
 	if (exit_code == SVM_EXIT_MSR)
@@ -3468,6 +3458,11 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
 #endif
 #endif
 	return svm_exit_handlers[exit_code](vcpu);
+
+unexpected_vmexit:
+	dump_vmcb(vcpu);
+	kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
+	return 0;
 }
 
 static void svm_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 5/9] KVM: SVM: Check for an unexpected VM-Exit after RETPOLINE "fast" handling

[Sean Christopherson]

Check for an unexpected/unhandled VM-Exit after the manual RETPOLINE=y
handling.  The entire point of the RETPOLINE checks is to optimize for
common VM-Exits, i.e. checking for the rare case of an unsupported
VM-Exit is counter-productive.  This also aligns SVM and VMX exit handling.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/svm.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 638a67ef0c37..202a4d8088a2 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3435,12 +3435,6 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
 
 int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
 {
-	if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
-		goto unexpected_vmexit;
-
-	if (!svm_exit_handlers[exit_code])
-		goto unexpected_vmexit;
-
 #ifdef CONFIG_MITIGATION_RETPOLINE
 	if (exit_code == SVM_EXIT_MSR)
 		return msr_interception(vcpu);
@@ -3457,6 +3451,12 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
 		return sev_handle_vmgexit(vcpu);
 #endif
 #endif
+	if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
+		goto unexpected_vmexit;
+
+	if (!svm_exit_handlers[exit_code])
+		goto unexpected_vmexit;
+
 	return svm_exit_handlers[exit_code](vcpu);
 
 unexpected_vmexit:
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 6/9] KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on bare metal

[Sean Christopherson]

Explicitly filter out 64-bit exit codes when invoking exit handlers, as
svm_exit_handlers[] will never be sized with entries that use bits 63:32.

Processing the non-failing exit code as a 32-bit value will allow tracking
exit_code as a single 64-bit value (which it is, architecturally).  This
will also allow hardening KVM against Spectre-like attacks without needing
to do silly things to avoid build failures on 32-bit kernels
(array_index_nospec() rightly asserts that the index fits in an "unsigned
long").

Omit the check when running as a VM, as KVM has historically failed to set
bits 63:32 appropriately when synthesizing VM-Exits, i.e. KVM could get
false positives when running as a VM on an older, broken KVM/kernel.  From
a functional perspective, omitting the check is "fine", as any unwanted
collision between e.g. VMEXIT_INVALID and a 32-bit exit code will be
fatal to KVM-on-KVM regardless of what KVM-as-L1 does.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/svm.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 202a4d8088a2..3b05476296d0 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3433,8 +3433,22 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
 		sev_free_decrypted_vmsa(vcpu, save);
 }
 
-int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
+int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 __exit_code)
 {
+	u32 exit_code = __exit_code;
+
+	/*
+	 * SVM uses negative values, i.e. 64-bit values, to indicate that VMRUN
+	 * failed.  Report all such errors to userspace (note, VMEXIT_INVALID,
+	 * a.k.a. SVM_EXIT_ERR, is special cased by svm_handle_exit()).  Skip
+	 * the check when running as a VM, as KVM has historically left garbage
+	 * in bits 63:32, i.e. running KVM-on-KVM would hit false positives if
+	 * the underlying kernel is buggy.
+	 */
+	if (!cpu_feature_enabled(X86_FEATURE_HYPERVISOR) &&
+	    (u64)exit_code != __exit_code)
+		goto unexpected_vmexit;
+
 #ifdef CONFIG_MITIGATION_RETPOLINE
 	if (exit_code == SVM_EXIT_MSR)
 		return msr_interception(vcpu);
@@ -3461,7 +3475,7 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
 
 unexpected_vmexit:
 	dump_vmcb(vcpu);
-	kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
+	kvm_prepare_unexpected_reason_exit(vcpu, __exit_code);
 	return 0;
 }
 
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Sean Christopherson]

Fix KVM's long-standing buggy handling of SVM's exit_code as a 32-bit
value.  Per the APM and Xen commit d1bd157fbc ("Big merge the HVM
full-virtualisation abstractions.") (which is arguably more trustworthy
than KVM), offset 0x70 is a single 64-bit value:

  070h 63:0 EXITCODE

Track exit_code as a single u64 to prevent reintroducing bugs where KVM
neglects to correctly set bits 63:32.

Fixes: 6aa8b732ca01 ("[PATCH] kvm: userspace interface")
Cc: Jim Mattson <jmattson@google.com>
Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/svm.h      |  3 +--
 arch/x86/include/uapi/asm/svm.h | 32 ++++++++++++++---------------
 arch/x86/kvm/svm/hyperv.c       |  1 -
 arch/x86/kvm/svm/nested.c       | 13 +++---------
 arch/x86/kvm/svm/sev.c          | 36 +++++++++++----------------------
 arch/x86/kvm/svm/svm.c          |  7 ++-----
 arch/x86/kvm/svm/svm.h          |  4 +---
 arch/x86/kvm/trace.h            |  2 +-
 include/hyperv/hvgdk.h          |  2 +-
 9 files changed, 37 insertions(+), 63 deletions(-)

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index e69b6d0dedcf..66b22cffedfc 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -136,8 +136,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
 	u32 int_vector;
 	u32 int_state;
 	u8 reserved_3[4];
-	u32 exit_code;
-	u32 exit_code_hi;
+	u64 exit_code;
 	u64 exit_info_1;
 	u64 exit_info_2;
 	u32 exit_int_info;
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 650e3256ea7d..010a45c9f614 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -103,38 +103,38 @@
 #define SVM_EXIT_VMGEXIT       0x403
 
 /* SEV-ES software-defined VMGEXIT events */
-#define SVM_VMGEXIT_MMIO_READ			0x80000001
-#define SVM_VMGEXIT_MMIO_WRITE			0x80000002
-#define SVM_VMGEXIT_NMI_COMPLETE		0x80000003
-#define SVM_VMGEXIT_AP_HLT_LOOP			0x80000004
-#define SVM_VMGEXIT_AP_JUMP_TABLE		0x80000005
+#define SVM_VMGEXIT_MMIO_READ			0x80000001ull
+#define SVM_VMGEXIT_MMIO_WRITE			0x80000002ull
+#define SVM_VMGEXIT_NMI_COMPLETE		0x80000003ull
+#define SVM_VMGEXIT_AP_HLT_LOOP			0x80000004ull
+#define SVM_VMGEXIT_AP_JUMP_TABLE		0x80000005ull
 #define SVM_VMGEXIT_SET_AP_JUMP_TABLE		0
 #define SVM_VMGEXIT_GET_AP_JUMP_TABLE		1
-#define SVM_VMGEXIT_PSC				0x80000010
-#define SVM_VMGEXIT_GUEST_REQUEST		0x80000011
-#define SVM_VMGEXIT_EXT_GUEST_REQUEST		0x80000012
-#define SVM_VMGEXIT_AP_CREATION			0x80000013
+#define SVM_VMGEXIT_PSC				0x80000010ull
+#define SVM_VMGEXIT_GUEST_REQUEST		0x80000011ull
+#define SVM_VMGEXIT_EXT_GUEST_REQUEST		0x80000012ull
+#define SVM_VMGEXIT_AP_CREATION			0x80000013ull
 #define SVM_VMGEXIT_AP_CREATE_ON_INIT		0
 #define SVM_VMGEXIT_AP_CREATE			1
 #define SVM_VMGEXIT_AP_DESTROY			2
-#define SVM_VMGEXIT_SNP_RUN_VMPL		0x80000018
-#define SVM_VMGEXIT_SAVIC			0x8000001a
+#define SVM_VMGEXIT_SNP_RUN_VMPL		0x80000018ull
+#define SVM_VMGEXIT_SAVIC			0x8000001aull
 #define SVM_VMGEXIT_SAVIC_REGISTER_GPA		0
 #define SVM_VMGEXIT_SAVIC_UNREGISTER_GPA	1
 #define SVM_VMGEXIT_SAVIC_SELF_GPA		~0ULL
-#define SVM_VMGEXIT_HV_FEATURES			0x8000fffd
-#define SVM_VMGEXIT_TERM_REQUEST		0x8000fffe
+#define SVM_VMGEXIT_HV_FEATURES			0x8000fffdull
+#define SVM_VMGEXIT_TERM_REQUEST		0x8000fffeull
 #define SVM_VMGEXIT_TERM_REASON(reason_set, reason_code)	\
 	/* SW_EXITINFO1[3:0] */					\
 	(((((u64)reason_set) & 0xf)) |				\
 	/* SW_EXITINFO1[11:4] */				\
 	((((u64)reason_code) & 0xff) << 4))
-#define SVM_VMGEXIT_UNSUPPORTED_EVENT		0x8000ffff
+#define SVM_VMGEXIT_UNSUPPORTED_EVENT		0x8000ffffull
 
 /* Exit code reserved for hypervisor/software use */
-#define SVM_EXIT_SW				0xf0000000
+#define SVM_EXIT_SW				0xf0000000ull
 
-#define SVM_EXIT_ERR           -1
+#define SVM_EXIT_ERR           -1ull
 
 #define SVM_EXIT_REASONS \
 	{ SVM_EXIT_READ_CR0,    "read_cr0" }, \
diff --git a/arch/x86/kvm/svm/hyperv.c b/arch/x86/kvm/svm/hyperv.c
index 088f6429b24c..3ec580d687f5 100644
--- a/arch/x86/kvm/svm/hyperv.c
+++ b/arch/x86/kvm/svm/hyperv.c
@@ -11,7 +11,6 @@ void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
 	struct vcpu_svm *svm = to_svm(vcpu);
 
 	svm->vmcb->control.exit_code = HV_SVM_EXITCODE_ENL;
-	svm->vmcb->control.exit_code_hi = 0;
 	svm->vmcb->control.exit_info_1 = HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH;
 	svm->vmcb->control.exit_info_2 = 0;
 	nested_svm_vmexit(svm);
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 8070e20ed5a7..89120245cd22 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -45,7 +45,6 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
 		 * correctly fill in the high bits of exit_info_1.
 		 */
 		vmcb->control.exit_code = SVM_EXIT_NPF;
-		vmcb->control.exit_code_hi = 0;
 		vmcb->control.exit_info_1 = (1ULL << 32);
 		vmcb->control.exit_info_2 = fault->address;
 	}
@@ -421,7 +420,6 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu,
 	to->int_vector          = from->int_vector;
 	to->int_state           = from->int_state;
 	to->exit_code           = from->exit_code;
-	to->exit_code_hi        = from->exit_code_hi;
 	to->exit_info_1         = from->exit_info_1;
 	to->exit_info_2         = from->exit_info_2;
 	to->exit_int_info       = from->exit_int_info;
@@ -727,8 +725,8 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
 	enter_guest_mode(vcpu);
 
 	/*
-	 * Filled at exit: exit_code, exit_code_hi, exit_info_1, exit_info_2,
-	 * exit_int_info, exit_int_info_err, next_rip, insn_len, insn_bytes.
+	 * Filled at exit: exit_code, exit_info_1, exit_info_2, exit_int_info,
+	 * exit_int_info_err, next_rip, insn_len, insn_bytes.
 	 */
 
 	if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
@@ -985,7 +983,6 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
 	if (!nested_vmcb_check_save(vcpu) ||
 	    !nested_vmcb_check_controls(vcpu)) {
 		vmcb12->control.exit_code    = SVM_EXIT_ERR;
-		vmcb12->control.exit_code_hi = -1u;
 		vmcb12->control.exit_info_1  = 0;
 		vmcb12->control.exit_info_2  = 0;
 		goto out;
@@ -1018,7 +1015,6 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
 	svm->soft_int_injected = false;
 
 	svm->vmcb->control.exit_code    = SVM_EXIT_ERR;
-	svm->vmcb->control.exit_code_hi = -1u;
 	svm->vmcb->control.exit_info_1  = 0;
 	svm->vmcb->control.exit_info_2  = 0;
 
@@ -1130,7 +1126,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
 
 	vmcb12->control.int_state         = vmcb02->control.int_state;
 	vmcb12->control.exit_code         = vmcb02->control.exit_code;
-	vmcb12->control.exit_code_hi      = vmcb02->control.exit_code_hi;
 	vmcb12->control.exit_info_1       = vmcb02->control.exit_info_1;
 	vmcb12->control.exit_info_2       = vmcb02->control.exit_info_2;
 
@@ -1422,7 +1417,7 @@ static int nested_svm_intercept_ioio(struct vcpu_svm *svm)
 
 static int nested_svm_intercept(struct vcpu_svm *svm)
 {
-	u32 exit_code = svm->vmcb->control.exit_code;
+	u64 exit_code = svm->vmcb->control.exit_code;
 	int vmexit = NESTED_EXIT_HOST;
 
 	if (svm_is_vmrun_failure(exit_code))
@@ -1494,7 +1489,6 @@ static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu)
 	struct vmcb *vmcb = svm->vmcb;
 
 	vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + ex->vector;
-	vmcb->control.exit_code_hi = 0;
 
 	if (ex->has_error_code)
 		vmcb->control.exit_info_1 = ex->error_code;
@@ -1669,7 +1663,6 @@ static void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,
 	dst->int_vector           = from->int_vector;
 	dst->int_state            = from->int_state;
 	dst->exit_code            = from->exit_code;
-	dst->exit_code_hi         = from->exit_code_hi;
 	dst->exit_info_1          = from->exit_info_1;
 	dst->exit_info_2          = from->exit_info_2;
 	dst->exit_int_info        = from->exit_int_info;
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 0835c664fbfd..5aedd07194aa 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3264,11 +3264,6 @@ void sev_free_vcpu(struct kvm_vcpu *vcpu)
 		kvfree(svm->sev_es.ghcb_sa);
 }
 
-static u64 kvm_get_cached_sw_exit_code(struct vmcb_control_area *control)
-{
-	return (((u64)control->exit_code_hi) << 32) | control->exit_code;
-}
-
 static void dump_ghcb(struct vcpu_svm *svm)
 {
 	struct vmcb_control_area *control = &svm->vmcb->control;
@@ -3290,7 +3285,7 @@ static void dump_ghcb(struct vcpu_svm *svm)
 	 */
 	pr_err("GHCB (GPA=%016llx) snapshot:\n", svm->vmcb->control.ghcb_gpa);
 	pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_code",
-	       kvm_get_cached_sw_exit_code(control), kvm_ghcb_sw_exit_code_is_valid(svm));
+	       control->exit_code, kvm_ghcb_sw_exit_code_is_valid(svm));
 	pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_info_1",
 	       control->exit_info_1, kvm_ghcb_sw_exit_info_1_is_valid(svm));
 	pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_info_2",
@@ -3324,7 +3319,6 @@ static void sev_es_sync_from_ghcb(struct vcpu_svm *svm)
 	struct vmcb_control_area *control = &svm->vmcb->control;
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 	struct ghcb *ghcb = svm->sev_es.ghcb;
-	u64 exit_code;
 
 	/*
 	 * The GHCB protocol so far allows for the following data
@@ -3358,9 +3352,7 @@ static void sev_es_sync_from_ghcb(struct vcpu_svm *svm)
 		__kvm_emulate_msr_write(vcpu, MSR_IA32_XSS, kvm_ghcb_get_xss(svm));
 
 	/* Copy the GHCB exit information into the VMCB fields */
-	exit_code = kvm_ghcb_get_sw_exit_code(svm);
-	control->exit_code = lower_32_bits(exit_code);
-	control->exit_code_hi = upper_32_bits(exit_code);
+	control->exit_code = kvm_ghcb_get_sw_exit_code(svm);
 	control->exit_info_1 = kvm_ghcb_get_sw_exit_info_1(svm);
 	control->exit_info_2 = kvm_ghcb_get_sw_exit_info_2(svm);
 	svm->sev_es.sw_scratch = kvm_ghcb_get_sw_scratch_if_valid(svm);
@@ -3373,15 +3365,8 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
 {
 	struct vmcb_control_area *control = &svm->vmcb->control;
 	struct kvm_vcpu *vcpu = &svm->vcpu;
-	u64 exit_code;
 	u64 reason;
 
-	/*
-	 * Retrieve the exit code now even though it may not be marked valid
-	 * as it could help with debugging.
-	 */
-	exit_code = kvm_get_cached_sw_exit_code(control);
-
 	/* Only GHCB Usage code 0 is supported */
 	if (svm->sev_es.ghcb->ghcb_usage) {
 		reason = GHCB_ERR_INVALID_USAGE;
@@ -3395,7 +3380,7 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
 	    !kvm_ghcb_sw_exit_info_2_is_valid(svm))
 		goto vmgexit_err;
 
-	switch (exit_code) {
+	switch (control->exit_code) {
 	case SVM_EXIT_READ_DR7:
 		break;
 	case SVM_EXIT_WRITE_DR7:
@@ -3496,15 +3481,19 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
 	return 0;
 
 vmgexit_err:
+	/*
+	 * Print the exit code even though it may not be marked valid as it
+	 * could help with debugging.
+	 */
 	if (reason == GHCB_ERR_INVALID_USAGE) {
 		vcpu_unimpl(vcpu, "vmgexit: ghcb usage %#x is not valid\n",
 			    svm->sev_es.ghcb->ghcb_usage);
 	} else if (reason == GHCB_ERR_INVALID_EVENT) {
 		vcpu_unimpl(vcpu, "vmgexit: exit code %#llx is not valid\n",
-			    exit_code);
+			    control->exit_code);
 	} else {
 		vcpu_unimpl(vcpu, "vmgexit: exit code %#llx input is not valid\n",
-			    exit_code);
+			    control->exit_code);
 		dump_ghcb(svm);
 	}
 
@@ -4343,7 +4332,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct vmcb_control_area *control = &svm->vmcb->control;
-	u64 ghcb_gpa, exit_code;
+	u64 ghcb_gpa;
 	int ret;
 
 	/* Validate the GHCB */
@@ -4385,8 +4374,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
 
 	svm_vmgexit_success(svm, 0);
 
-	exit_code = kvm_get_cached_sw_exit_code(control);
-	switch (exit_code) {
+	switch (control->exit_code) {
 	case SVM_VMGEXIT_MMIO_READ:
 		ret = setup_vmgexit_scratch(svm, true, control->exit_info_2);
 		if (ret)
@@ -4478,7 +4466,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
 		ret = -EINVAL;
 		break;
 	default:
-		ret = svm_invoke_exit_handler(vcpu, exit_code);
+		ret = svm_invoke_exit_handler(vcpu, control->exit_code);
 	}
 
 	return ret;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 3b05476296d0..85bc99f93275 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2433,7 +2433,6 @@ static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
 
 	if (cr0 ^ val) {
 		svm->vmcb->control.exit_code = SVM_EXIT_CR0_SEL_WRITE;
-		svm->vmcb->control.exit_code_hi = 0;
 		ret = (nested_svm_exit_handled(svm) == NESTED_EXIT_DONE);
 	}
 
@@ -3265,7 +3264,7 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
 	pr_err("%-20s%08x\n", "int_ctl:", control->int_ctl);
 	pr_err("%-20s%08x\n", "int_vector:", control->int_vector);
 	pr_err("%-20s%08x\n", "int_state:", control->int_state);
-	pr_err("%-20s%08x\n", "exit_code:", control->exit_code);
+	pr_err("%-20s%016llx\n", "exit_code:", control->exit_code);
 	pr_err("%-20s%016llx\n", "exit_info1:", control->exit_info_1);
 	pr_err("%-20s%016llx\n", "exit_info2:", control->exit_info_2);
 	pr_err("%-20s%08x\n", "exit_int_info:", control->exit_int_info);
@@ -3515,7 +3514,6 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_run *kvm_run = vcpu->run;
-	u32 exit_code = svm->vmcb->control.exit_code;
 
 	/* SEV-ES guests must use the CR write traps to track CR registers. */
 	if (!sev_es_guest(vcpu->kvm)) {
@@ -3551,7 +3549,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
 	if (exit_fastpath != EXIT_FASTPATH_NONE)
 		return 1;
 
-	return svm_invoke_exit_handler(vcpu, exit_code);
+	return svm_invoke_exit_handler(vcpu, svm->vmcb->control.exit_code);
 }
 
 static int pre_svm_run(struct kvm_vcpu *vcpu)
@@ -4618,7 +4616,6 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
 	if (static_cpu_has(X86_FEATURE_NRIPS))
 		vmcb->control.next_rip  = info->next_rip;
 	vmcb->control.exit_code = icpt_info.exit_code;
-	vmcb->control.exit_code_hi = 0;
 	vmexit = nested_svm_exit_handled(svm);
 
 	ret = (vmexit == NESTED_EXIT_DONE) ? X86EMUL_INTERCEPTED
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 6b35925e3a33..31ee4f65dcc2 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -162,8 +162,7 @@ struct vmcb_ctrl_area_cached {
 	u32 int_ctl;
 	u32 int_vector;
 	u32 int_state;
-	u32 exit_code;
-	u32 exit_code_hi;
+	u64 exit_code;
 	u64 exit_info_1;
 	u64 exit_info_2;
 	u32 exit_int_info;
@@ -769,7 +768,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm);
 static inline int nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
 {
 	svm->vmcb->control.exit_code	= exit_code;
-	svm->vmcb->control.exit_code_hi	= 0;
 	svm->vmcb->control.exit_info_1	= 0;
 	svm->vmcb->control.exit_info_2	= 0;
 	return nested_svm_vmexit(svm);
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index e79bc9cb7162..4c7a5cd10990 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -781,7 +781,7 @@ TRACE_EVENT_KVM_EXIT(kvm_nested_vmexit);
  * Tracepoint for #VMEXIT reinjected to the guest
  */
 TRACE_EVENT(kvm_nested_vmexit_inject,
-	    TP_PROTO(__u32 exit_code,
+	    TP_PROTO(__u64 exit_code,
 		     __u64 exit_info1, __u64 exit_info2,
 		     __u32 exit_int_info, __u32 exit_int_info_err, __u32 isa),
 	    TP_ARGS(exit_code, exit_info1, exit_info2,
diff --git a/include/hyperv/hvgdk.h b/include/hyperv/hvgdk.h
index dd6d4939ea29..56b695873a72 100644
--- a/include/hyperv/hvgdk.h
+++ b/include/hyperv/hvgdk.h
@@ -281,7 +281,7 @@ struct hv_vmcb_enlightenments {
 #define HV_VMCB_NESTED_ENLIGHTENMENTS		31
 
 /* Synthetic VM-Exit */
-#define HV_SVM_EXITCODE_ENL			0xf0000000
+#define HV_SVM_EXITCODE_ENL			0xf0000000u
 #define HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH	(1)
 
 /* VM_PARTITION_ASSIST_PAGE */
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 8/9] KVM: SVM: Limit incorrect check on SVM_EXIT_ERR to running as a VM

[Sean Christopherson]

Limit KVM's incorrect check for VMXEXIT_INVALID, a.k.a. SVM_EXIT_ERR, to
running as a VM, as detected by X86_FEATURE_HYPERVISOR.  The exit_code and
all failure codes, e.g. VMXEXIT_INVALID, are 64-bit values, and so checking
only bits 31:0 could result in false positives when running on non-broken
hardware, e.g. in the extremely unlikely scenario exit code 0xffffffffull
is ever generated by hardware.

Keep the 32-bit check to play nice with running on broken KVM (for years,
KVM has not set bits 63:32 when synthesizing nested SVM VM-Exits).

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/svm.h | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 31ee4f65dcc2..801dcfc64d0b 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -427,7 +427,10 @@ static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
 
 static inline bool svm_is_vmrun_failure(u64 exit_code)
 {
-	return (u32)exit_code == (u32)SVM_EXIT_ERR;
+	if (cpu_feature_enabled(X86_FEATURE_HYPERVISOR))
+		return (u32)exit_code == (u32)SVM_EXIT_ERR;
+
+	return exit_code == SVM_EXIT_ERR;
 }
 
 /*
-- 
2.52.0.rc1.455.g30608eb744-goog

[PATCH 9/9] KVM: SVM: Harden exit_code against being used in Spectre-like attacks

[Sean Christopherson]

Explicitly clamp the exit code used to index KVM's exit handlers to guard
against Spectre-like attacks, mainly to provide consistency between VMX
and SVM (VMX was given the same treatment by commit c926f2f7230b ("KVM:
x86: Protect exit_reason from being used in Spectre-v1/L1TF attacks").

For normal VMs, it's _extremely_ unlikely the exit code could be used to
exploit a speculation vulnerability, as the exit code is set by hardware
and unexpected/unknown exit codes should be quite well bounded (as is/was
the case with VMX).  But with SEV-ES+, the exit code is guest-controlled
as it comes from the GHCB, not from hardware, i.e. an attack from the
guest is at least somewhat plausible.

Irrespective of SEV-ES+, hardening KVM is easy and inexpensive, and such
an attack is theoretically possible.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/svm.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 85bc99f93275..308c70b6924e 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3467,6 +3467,7 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 __exit_code)
 	if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
 		goto unexpected_vmexit;
 
+	exit_code = array_index_nospec(exit_code, ARRAY_SIZE(svm_exit_handlers));
 	if (!svm_exit_handlers[exit_code])
 		goto unexpected_vmexit;
 
-- 
2.52.0.rc1.455.g30608eb744-goog

Re: [PATCH 1/9] KVM: nSVM: Clear exit_code_hi in VMCB when synthesizing nested VM-Exits

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:13PM -0800, Sean Christopherson wrote:
> Explicitly clear exit_code_hi in the VMCB when synthesizing "normal"
> nested VM-Exits, as the full exit code is a 64-bit value (spoiler alert),
> and all exit codes for non-failing VMRUN use only bits 31:0.
> 
> Cc: Jim Mattson <jmattson@google.com>
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Cc: stable@vger.kernel.org
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/svm.c | 2 ++
>  arch/x86/kvm/svm/svm.h | 7 ++++---
>  2 files changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index fc42bcdbb520..7ea034ee6b6c 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -2433,6 +2433,7 @@ static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
>  
>  	if (cr0 ^ val) {
>  		svm->vmcb->control.exit_code = SVM_EXIT_CR0_SEL_WRITE;
> +		svm->vmcb->control.exit_code_hi = 0;
>  		ret = (nested_svm_exit_handled(svm) == NESTED_EXIT_DONE);
>  	}
>  
> @@ -4608,6 +4609,7 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
>  	if (static_cpu_has(X86_FEATURE_NRIPS))
>  		vmcb->control.next_rip  = info->next_rip;
>  	vmcb->control.exit_code = icpt_info.exit_code;
> +	vmcb->control.exit_code_hi = 0;
>  	vmexit = nested_svm_exit_handled(svm);
>  
>  	ret = (vmexit == NESTED_EXIT_DONE) ? X86EMUL_INTERCEPTED
> diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
> index c2acaa49ee1c..253a8dca412c 100644
> --- a/arch/x86/kvm/svm/svm.h
> +++ b/arch/x86/kvm/svm/svm.h
> @@ -763,9 +763,10 @@ int nested_svm_vmexit(struct vcpu_svm *svm);
>  
>  static inline int nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
>  {
> -	svm->vmcb->control.exit_code   = exit_code;
> -	svm->vmcb->control.exit_info_1 = 0;
> -	svm->vmcb->control.exit_info_2 = 0;
> +	svm->vmcb->control.exit_code	= exit_code;
> +	svm->vmcb->control.exit_code_hi	= 0;
> +	svm->vmcb->control.exit_info_1	= 0;
> +	svm->vmcb->control.exit_info_2	= 0;
>  	return nested_svm_vmexit(svm);
>  }
>  
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

Re: [PATCH 2/9] KVM: nSVM: Set exit_code_hi to -1 when synthesizing SVM_EXIT_ERR (failed VMRUN)

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:14PM -0800, Sean Christopherson wrote:
> Set exit_code_hi to -1u as a temporary band-aid to fix a long-standing
> (effectively since KVM's inception) bug where KVM treats the exit code as
> a 32-bit value, when in reality it's a 64-bit value.  Per the APM, offset
> 0x70 is a single 64-bit value:
> 
>   070h 63:0 EXITCODE
> 
> And a sane reading of the error values defined in "Table C-1. SVM Intercept
> Codes" is that negative values use the full 64 bits:
> 
>   –1 VMEXIT_INVALID Invalid guest state in VMCB.
>   –2 VMEXIT_BUSYBUSY bit was set in the VMSA
>   –3 VMEXIT_IDLE_REQUIREDThe sibling thread is not in an idle state
>   -4 VMEXIT_INVALID_PMC Invalid PMC state
> 
> And that interpretation is confirmed by testing on Milan and Turin (by
> setting bits in CR0[63:32] to generate VMEXIT_INVALID on VMRUN).
> 
> Furthermore, Xen has treated exitcode as a 64-bit value since HVM support
> was adding in 2006 (see Xen commit d1bd157fbc ("Big merge the HVM
> full-virtualisation abstractions.")).
> 
> Cc: Jim Mattson <jmattson@google.com>
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Cc: stable@vger.kernel.org
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/nested.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index c81005b24522..ba0f11c68372 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -985,7 +985,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
>  	if (!nested_vmcb_check_save(vcpu) ||
>  	    !nested_vmcb_check_controls(vcpu)) {
>  		vmcb12->control.exit_code    = SVM_EXIT_ERR;
> -		vmcb12->control.exit_code_hi = 0;
> +		vmcb12->control.exit_code_hi = -1u;
>  		vmcb12->control.exit_info_1  = 0;
>  		vmcb12->control.exit_info_2  = 0;
>  		goto out;
> @@ -1018,7 +1018,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
>  	svm->soft_int_injected = false;
>  
>  	svm->vmcb->control.exit_code    = SVM_EXIT_ERR;
> -	svm->vmcb->control.exit_code_hi = 0;
> +	svm->vmcb->control.exit_code_hi = -1u;
>  	svm->vmcb->control.exit_info_1  = 0;
>  	svm->vmcb->control.exit_info_2  = 0;
>  
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

Re: [PATCH 2/9] KVM: nSVM: Set exit_code_hi to -1 when synthesizing SVM_EXIT_ERR (failed VMRUN)

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:14PM -0800, Sean Christopherson wrote:
> Set exit_code_hi to -1u as a temporary band-aid to fix a long-standing
> (effectively since KVM's inception) bug where KVM treats the exit code as
> a 32-bit value, when in reality it's a 64-bit value.  Per the APM, offset
> 0x70 is a single 64-bit value:
> 
>   070h 63:0 EXITCODE
> 
> And a sane reading of the error values defined in "Table C-1. SVM Intercept
> Codes" is that negative values use the full 64 bits:
> 
>   –1 VMEXIT_INVALID Invalid guest state in VMCB.
>   –2 VMEXIT_BUSYBUSY bit was set in the VMSA
>   –3 VMEXIT_IDLE_REQUIREDThe sibling thread is not in an idle state
>   -4 VMEXIT_INVALID_PMC Invalid PMC state
> 
> And that interpretation is confirmed by testing on Milan and Turin (by
> setting bits in CR0[63:32] to generate VMEXIT_INVALID on VMRUN).
> 
> Furthermore, Xen has treated exitcode as a 64-bit value since HVM support
> was adding in 2006 (see Xen commit d1bd157fbc ("Big merge the HVM
> full-virtualisation abstractions.")).
> 
> Cc: Jim Mattson <jmattson@google.com>
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Cc: stable@vger.kernel.org
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/nested.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index c81005b24522..ba0f11c68372 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -985,7 +985,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
>  	if (!nested_vmcb_check_save(vcpu) ||
>  	    !nested_vmcb_check_controls(vcpu)) {
>  		vmcb12->control.exit_code    = SVM_EXIT_ERR;
> -		vmcb12->control.exit_code_hi = 0;
> +		vmcb12->control.exit_code_hi = -1u;
>  		vmcb12->control.exit_info_1  = 0;
>  		vmcb12->control.exit_info_2  = 0;
>  		goto out;
> @@ -1018,7 +1018,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
>  	svm->soft_int_injected = false;
>  
>  	svm->vmcb->control.exit_code    = SVM_EXIT_ERR;
> -	svm->vmcb->control.exit_code_hi = 0;
> +	svm->vmcb->control.exit_code_hi = -1u;
>  	svm->vmcb->control.exit_info_1  = 0;
>  	svm->vmcb->control.exit_info_2  = 0;
>  
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

Re: [PATCH 3/9] KVM: SVM: Add a helper to detect VMRUN failures

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:15PM -0800, Sean Christopherson wrote:
> Add a helper to detect VMRUN failures so that KVM can guard against its
> own long-standing bug, where KVM neglects to set exitcode[63:32] when
> synthesizing a nested VMFAIL_INVALID VM-Exit.  This will allow fixing
> KVM's mess of treating exitcode as two separate 32-bit values without
> breaking KVM-on-KVM when running on an older, unfixed KVM.
> 
> Cc: Jim Mattson <jmattson@google.com>
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/kvm/svm/nested.c | 16 +++++++---------
>  arch/x86/kvm/svm/svm.c    |  4 ++--
>  arch/x86/kvm/svm/svm.h    |  5 +++++
>  3 files changed, 14 insertions(+), 11 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index ba0f11c68372..8070e20ed5a7 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -1134,7 +1134,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
>  	vmcb12->control.exit_info_1       = vmcb02->control.exit_info_1;
>  	vmcb12->control.exit_info_2       = vmcb02->control.exit_info_2;
>  
> -	if (vmcb12->control.exit_code != SVM_EXIT_ERR)
> +	if (svm_is_vmrun_failure(vmcb12->control.exit_code))

This was flipped, wasn't it?

>  		nested_save_pending_event_to_vmcb12(svm, vmcb12);
>  
>  	if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
> @@ -1425,6 +1425,9 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
>  	u32 exit_code = svm->vmcb->control.exit_code;
>  	int vmexit = NESTED_EXIT_HOST;
>  
> +	if (svm_is_vmrun_failure(exit_code))
> +		return NESTED_EXIT_DONE;
> +
>  	switch (exit_code) {
>  	case SVM_EXIT_MSR:
>  		vmexit = nested_svm_exit_handled_msr(svm);
> @@ -1432,7 +1435,7 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
>  	case SVM_EXIT_IOIO:
>  		vmexit = nested_svm_intercept_ioio(svm);
>  		break;
> -	case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f: {
> +	case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f:
>  		/*
>  		 * Host-intercepted exceptions have been checked already in
>  		 * nested_svm_exit_special.  There is nothing to do here,
> @@ -1440,15 +1443,10 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
>  		 */
>  		vmexit = NESTED_EXIT_DONE;
>  		break;
> -	}
> -	case SVM_EXIT_ERR: {
> -		vmexit = NESTED_EXIT_DONE;
> -		break;
> -	}
> -	default: {
> +	default:
>  		if (vmcb12_is_intercept(&svm->nested.ctl, exit_code))
>  			vmexit = NESTED_EXIT_DONE;
> -	}
> +		break;
>  	}
>  
>  	return vmexit;
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 7ea034ee6b6c..52b759408853 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3530,7 +3530,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
>  			return 1;
>  	}
>  
> -	if (svm->vmcb->control.exit_code == SVM_EXIT_ERR) {
> +	if (svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
>  		kvm_run->exit_reason = KVM_EXIT_FAIL_ENTRY;
>  		kvm_run->fail_entry.hardware_entry_failure_reason
>  			= svm->vmcb->control.exit_code;
> @@ -4302,7 +4302,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
>  
>  		/* Track VMRUNs that have made past consistency checking */
>  		if (svm->nested.nested_run_pending &&
> -		    svm->vmcb->control.exit_code != SVM_EXIT_ERR)
> +		    !svm_is_vmrun_failure(svm->vmcb->control.exit_code))
>                          ++vcpu->stat.nested_run;
>  
>  		svm->nested.nested_run_pending = 0;
> diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
> index 253a8dca412c..6b35925e3a33 100644
> --- a/arch/x86/kvm/svm/svm.h
> +++ b/arch/x86/kvm/svm/svm.h
> @@ -426,6 +426,11 @@ static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
>  	return container_of(vcpu, struct vcpu_svm, vcpu);
>  }
>  
> +static inline bool svm_is_vmrun_failure(u64 exit_code)
> +{
> +	return (u32)exit_code == (u32)SVM_EXIT_ERR;
> +}
> +
>  /*
>   * Only the PDPTRs are loaded on demand into the shadow MMU.  All other
>   * fields are synchronized on VM-Exit, because accessing the VMCB is cheap.
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

Re: [PATCH 4/9] KVM: SVM: Open code handling of unexpected exits in svm_invoke_exit_handler()

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:16PM -0800, Sean Christopherson wrote:
> Fold svm_check_exit_valid() and svm_handle_invalid_exit() into their sole
> caller, svm_invoke_exit_handler(), as having tiny single-use helpers makes
> the code unncessarily difficult to follow.  This will also allow for
> additional cleanups in svm_invoke_exit_handler().
> 
> No functional change intended.
> 
> Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/svm.c | 25 ++++++++++---------------
>  1 file changed, 10 insertions(+), 15 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 52b759408853..638a67ef0c37 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3433,23 +3433,13 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
>  		sev_free_decrypted_vmsa(vcpu, save);
>  }
>  
> -static bool svm_check_exit_valid(u64 exit_code)
> -{
> -	return (exit_code < ARRAY_SIZE(svm_exit_handlers) &&
> -		svm_exit_handlers[exit_code]);
> -}
> -
> -static int svm_handle_invalid_exit(struct kvm_vcpu *vcpu, u64 exit_code)
> -{
> -	dump_vmcb(vcpu);
> -	kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
> -	return 0;
> -}
> -
>  int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
>  {
> -	if (!svm_check_exit_valid(exit_code))
> -		return svm_handle_invalid_exit(vcpu, exit_code);
> +	if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
> +		goto unexpected_vmexit;
> +
> +	if (!svm_exit_handlers[exit_code])
> +		goto unexpected_vmexit;
>  
>  #ifdef CONFIG_MITIGATION_RETPOLINE
>  	if (exit_code == SVM_EXIT_MSR)
> @@ -3468,6 +3458,11 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
>  #endif
>  #endif
>  	return svm_exit_handlers[exit_code](vcpu);
> +
> +unexpected_vmexit:
> +	dump_vmcb(vcpu);
> +	kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
> +	return 0;
>  }
>  
>  static void svm_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

Re: [PATCH 3/9] KVM: SVM: Add a helper to detect VMRUN failures

[Sean Christopherson]

On Thu, Nov 13, 2025, Yosry Ahmed wrote:
> On Thu, Nov 13, 2025 at 02:56:15PM -0800, Sean Christopherson wrote:
> > Add a helper to detect VMRUN failures so that KVM can guard against its
> > own long-standing bug, where KVM neglects to set exitcode[63:32] when
> > synthesizing a nested VMFAIL_INVALID VM-Exit.  This will allow fixing
> > KVM's mess of treating exitcode as two separate 32-bit values without
> > breaking KVM-on-KVM when running on an older, unfixed KVM.
> > 
> > Cc: Jim Mattson <jmattson@google.com>
> > Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > ---
> >  arch/x86/kvm/svm/nested.c | 16 +++++++---------
> >  arch/x86/kvm/svm/svm.c    |  4 ++--
> >  arch/x86/kvm/svm/svm.h    |  5 +++++
> >  3 files changed, 14 insertions(+), 11 deletions(-)
> > 
> > diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> > index ba0f11c68372..8070e20ed5a7 100644
> > --- a/arch/x86/kvm/svm/nested.c
> > +++ b/arch/x86/kvm/svm/nested.c
> > @@ -1134,7 +1134,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
> >  	vmcb12->control.exit_info_1       = vmcb02->control.exit_info_1;
> >  	vmcb12->control.exit_info_2       = vmcb02->control.exit_info_2;
> >  
> > -	if (vmcb12->control.exit_code != SVM_EXIT_ERR)
> > +	if (svm_is_vmrun_failure(vmcb12->control.exit_code))
> 
> This was flipped, wasn't it?

Ugh, yes.  Hrm, I'm surprised this wasn't caught by svm_nested_soft_inject_test.c.

Oof.  We should probably also extend svm_is_vmrun_failure() (in the future) to
detect any failure, e.g. VMEXIT_INVALID_PMC might be relevant soon?

> >  		nested_save_pending_event_to_vmcb12(svm, vmcb12);

Re: [PATCH 5/9] KVM: SVM: Check for an unexpected VM-Exit after RETPOLINE "fast" handling

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:17PM -0800, Sean Christopherson wrote:
> Check for an unexpected/unhandled VM-Exit after the manual RETPOLINE=y
> handling.  The entire point of the RETPOLINE checks is to optimize for
> common VM-Exits, i.e. checking for the rare case of an unsupported
> VM-Exit is counter-productive.  This also aligns SVM and VMX exit handling.
> 
> No functional change intended.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/svm.c | 12 ++++++------
>  1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 638a67ef0c37..202a4d8088a2 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3435,12 +3435,6 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
>  
>  int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
>  {
> -	if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
> -		goto unexpected_vmexit;
> -
> -	if (!svm_exit_handlers[exit_code])
> -		goto unexpected_vmexit;
> -
>  #ifdef CONFIG_MITIGATION_RETPOLINE
>  	if (exit_code == SVM_EXIT_MSR)
>  		return msr_interception(vcpu);
> @@ -3457,6 +3451,12 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
>  		return sev_handle_vmgexit(vcpu);
>  #endif
>  #endif
> +	if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
> +		goto unexpected_vmexit;
> +
> +	if (!svm_exit_handlers[exit_code])
> +		goto unexpected_vmexit;
> +
>  	return svm_exit_handlers[exit_code](vcpu);
>  
>  unexpected_vmexit:
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

Re: [PATCH 6/9] KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on bare metal

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:18PM -0800, Sean Christopherson wrote:
> Explicitly filter out 64-bit exit codes when invoking exit handlers, as
> svm_exit_handlers[] will never be sized with entries that use bits 63:32.
> 
> Processing the non-failing exit code as a 32-bit value will allow tracking
> exit_code as a single 64-bit value (which it is, architecturally).  This
> will also allow hardening KVM against Spectre-like attacks without needing
> to do silly things to avoid build failures on 32-bit kernels
> (array_index_nospec() rightly asserts that the index fits in an "unsigned
> long").
> 
> Omit the check when running as a VM, as KVM has historically failed to set
> bits 63:32 appropriately when synthesizing VM-Exits, i.e. KVM could get
> false positives when running as a VM on an older, broken KVM/kernel.  From
> a functional perspective, omitting the check is "fine", as any unwanted
> collision between e.g. VMEXIT_INVALID and a 32-bit exit code will be
> fatal to KVM-on-KVM regardless of what KVM-as-L1 does.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/svm.c | 18 ++++++++++++++++--
>  1 file changed, 16 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 202a4d8088a2..3b05476296d0 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3433,8 +3433,22 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
>  		sev_free_decrypted_vmsa(vcpu, save);
>  }
>  
> -int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
> +int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 __exit_code)
>  {
> +	u32 exit_code = __exit_code;
> +
> +	/*
> +	 * SVM uses negative values, i.e. 64-bit values, to indicate that VMRUN
> +	 * failed.  Report all such errors to userspace (note, VMEXIT_INVALID,
> +	 * a.k.a. SVM_EXIT_ERR, is special cased by svm_handle_exit()).  Skip
> +	 * the check when running as a VM, as KVM has historically left garbage
> +	 * in bits 63:32, i.e. running KVM-on-KVM would hit false positives if
> +	 * the underlying kernel is buggy.
> +	 */
> +	if (!cpu_feature_enabled(X86_FEATURE_HYPERVISOR) &&
> +	    (u64)exit_code != __exit_code)
> +		goto unexpected_vmexit;
> +
>  #ifdef CONFIG_MITIGATION_RETPOLINE
>  	if (exit_code == SVM_EXIT_MSR)
>  		return msr_interception(vcpu);
> @@ -3461,7 +3475,7 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
>  
>  unexpected_vmexit:
>  	dump_vmcb(vcpu);
> -	kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
> +	kvm_prepare_unexpected_reason_exit(vcpu, __exit_code);
>  	return 0;
>  }
>  
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

Re: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:19PM -0800, Sean Christopherson wrote:
> Fix KVM's long-standing buggy handling of SVM's exit_code as a 32-bit
> value.  Per the APM and Xen commit d1bd157fbc ("Big merge the HVM
> full-virtualisation abstractions.") (which is arguably more trustworthy
> than KVM), offset 0x70 is a single 64-bit value:
> 
>   070h 63:0 EXITCODE
> 
> Track exit_code as a single u64 to prevent reintroducing bugs where KVM
> neglects to correctly set bits 63:32.
> 
> Fixes: 6aa8b732ca01 ("[PATCH] kvm: userspace interface")
> Cc: Jim Mattson <jmattson@google.com>
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/include/asm/svm.h      |  3 +--
>  arch/x86/include/uapi/asm/svm.h | 32 ++++++++++++++---------------
>  arch/x86/kvm/svm/hyperv.c       |  1 -
>  arch/x86/kvm/svm/nested.c       | 13 +++---------
>  arch/x86/kvm/svm/sev.c          | 36 +++++++++++----------------------
>  arch/x86/kvm/svm/svm.c          |  7 ++-----
>  arch/x86/kvm/svm/svm.h          |  4 +---
>  arch/x86/kvm/trace.h            |  2 +-
>  include/hyperv/hvgdk.h          |  2 +-
>  9 files changed, 37 insertions(+), 63 deletions(-)
> 
> diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
> index e69b6d0dedcf..66b22cffedfc 100644
> --- a/arch/x86/include/asm/svm.h
> +++ b/arch/x86/include/asm/svm.h
> @@ -136,8 +136,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
>  	u32 int_vector;
>  	u32 int_state;
>  	u8 reserved_3[4];
> -	u32 exit_code;
> -	u32 exit_code_hi;
> +	u64 exit_code;
>  	u64 exit_info_1;
>  	u64 exit_info_2;
>  	u32 exit_int_info;
> diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
> index 650e3256ea7d..010a45c9f614 100644
> --- a/arch/x86/include/uapi/asm/svm.h
> +++ b/arch/x86/include/uapi/asm/svm.h
> @@ -103,38 +103,38 @@
>  #define SVM_EXIT_VMGEXIT       0x403
>  
>  /* SEV-ES software-defined VMGEXIT events */
> -#define SVM_VMGEXIT_MMIO_READ			0x80000001
> -#define SVM_VMGEXIT_MMIO_WRITE			0x80000002
> -#define SVM_VMGEXIT_NMI_COMPLETE		0x80000003
> -#define SVM_VMGEXIT_AP_HLT_LOOP			0x80000004
> -#define SVM_VMGEXIT_AP_JUMP_TABLE		0x80000005
> +#define SVM_VMGEXIT_MMIO_READ			0x80000001ull
> +#define SVM_VMGEXIT_MMIO_WRITE			0x80000002ull
> +#define SVM_VMGEXIT_NMI_COMPLETE		0x80000003ull
> +#define SVM_VMGEXIT_AP_HLT_LOOP			0x80000004ull
> +#define SVM_VMGEXIT_AP_JUMP_TABLE		0x80000005ull
>  #define SVM_VMGEXIT_SET_AP_JUMP_TABLE		0
>  #define SVM_VMGEXIT_GET_AP_JUMP_TABLE		1
> -#define SVM_VMGEXIT_PSC				0x80000010
> -#define SVM_VMGEXIT_GUEST_REQUEST		0x80000011
> -#define SVM_VMGEXIT_EXT_GUEST_REQUEST		0x80000012
> -#define SVM_VMGEXIT_AP_CREATION			0x80000013
> +#define SVM_VMGEXIT_PSC				0x80000010ull
> +#define SVM_VMGEXIT_GUEST_REQUEST		0x80000011ull
> +#define SVM_VMGEXIT_EXT_GUEST_REQUEST		0x80000012ull
> +#define SVM_VMGEXIT_AP_CREATION			0x80000013ull
>  #define SVM_VMGEXIT_AP_CREATE_ON_INIT		0
>  #define SVM_VMGEXIT_AP_CREATE			1
>  #define SVM_VMGEXIT_AP_DESTROY			2
> -#define SVM_VMGEXIT_SNP_RUN_VMPL		0x80000018
> -#define SVM_VMGEXIT_SAVIC			0x8000001a
> +#define SVM_VMGEXIT_SNP_RUN_VMPL		0x80000018ull
> +#define SVM_VMGEXIT_SAVIC			0x8000001aull
>  #define SVM_VMGEXIT_SAVIC_REGISTER_GPA		0
>  #define SVM_VMGEXIT_SAVIC_UNREGISTER_GPA	1
>  #define SVM_VMGEXIT_SAVIC_SELF_GPA		~0ULL
> -#define SVM_VMGEXIT_HV_FEATURES			0x8000fffd
> -#define SVM_VMGEXIT_TERM_REQUEST		0x8000fffe
> +#define SVM_VMGEXIT_HV_FEATURES			0x8000fffdull
> +#define SVM_VMGEXIT_TERM_REQUEST		0x8000fffeull
>  #define SVM_VMGEXIT_TERM_REASON(reason_set, reason_code)	\
>  	/* SW_EXITINFO1[3:0] */					\
>  	(((((u64)reason_set) & 0xf)) |				\
>  	/* SW_EXITINFO1[11:4] */				\
>  	((((u64)reason_code) & 0xff) << 4))
> -#define SVM_VMGEXIT_UNSUPPORTED_EVENT		0x8000ffff
> +#define SVM_VMGEXIT_UNSUPPORTED_EVENT		0x8000ffffull
>  
>  /* Exit code reserved for hypervisor/software use */
> -#define SVM_EXIT_SW				0xf0000000
> +#define SVM_EXIT_SW				0xf0000000ull
>  
> -#define SVM_EXIT_ERR           -1
> +#define SVM_EXIT_ERR           -1ull
>  
>  #define SVM_EXIT_REASONS \
>  	{ SVM_EXIT_READ_CR0,    "read_cr0" }, \
> diff --git a/arch/x86/kvm/svm/hyperv.c b/arch/x86/kvm/svm/hyperv.c
> index 088f6429b24c..3ec580d687f5 100644
> --- a/arch/x86/kvm/svm/hyperv.c
> +++ b/arch/x86/kvm/svm/hyperv.c
> @@ -11,7 +11,6 @@ void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
>  	struct vcpu_svm *svm = to_svm(vcpu);
>  
>  	svm->vmcb->control.exit_code = HV_SVM_EXITCODE_ENL;
> -	svm->vmcb->control.exit_code_hi = 0;
>  	svm->vmcb->control.exit_info_1 = HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH;
>  	svm->vmcb->control.exit_info_2 = 0;
>  	nested_svm_vmexit(svm);
> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index 8070e20ed5a7..89120245cd22 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -45,7 +45,6 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
>  		 * correctly fill in the high bits of exit_info_1.
>  		 */
>  		vmcb->control.exit_code = SVM_EXIT_NPF;
> -		vmcb->control.exit_code_hi = 0;
>  		vmcb->control.exit_info_1 = (1ULL << 32);
>  		vmcb->control.exit_info_2 = fault->address;
>  	}
> @@ -421,7 +420,6 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu,
>  	to->int_vector          = from->int_vector;
>  	to->int_state           = from->int_state;
>  	to->exit_code           = from->exit_code;
> -	to->exit_code_hi        = from->exit_code_hi;
>  	to->exit_info_1         = from->exit_info_1;
>  	to->exit_info_2         = from->exit_info_2;
>  	to->exit_int_info       = from->exit_int_info;
> @@ -727,8 +725,8 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
>  	enter_guest_mode(vcpu);
>  
>  	/*
> -	 * Filled at exit: exit_code, exit_code_hi, exit_info_1, exit_info_2,
> -	 * exit_int_info, exit_int_info_err, next_rip, insn_len, insn_bytes.
> +	 * Filled at exit: exit_code, exit_info_1, exit_info_2, exit_int_info,
> +	 * exit_int_info_err, next_rip, insn_len, insn_bytes.
>  	 */
>  
>  	if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
> @@ -985,7 +983,6 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
>  	if (!nested_vmcb_check_save(vcpu) ||
>  	    !nested_vmcb_check_controls(vcpu)) {
>  		vmcb12->control.exit_code    = SVM_EXIT_ERR;
> -		vmcb12->control.exit_code_hi = -1u;
>  		vmcb12->control.exit_info_1  = 0;
>  		vmcb12->control.exit_info_2  = 0;
>  		goto out;
> @@ -1018,7 +1015,6 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
>  	svm->soft_int_injected = false;
>  
>  	svm->vmcb->control.exit_code    = SVM_EXIT_ERR;
> -	svm->vmcb->control.exit_code_hi = -1u;
>  	svm->vmcb->control.exit_info_1  = 0;
>  	svm->vmcb->control.exit_info_2  = 0;
>  
> @@ -1130,7 +1126,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
>  
>  	vmcb12->control.int_state         = vmcb02->control.int_state;
>  	vmcb12->control.exit_code         = vmcb02->control.exit_code;
> -	vmcb12->control.exit_code_hi      = vmcb02->control.exit_code_hi;
>  	vmcb12->control.exit_info_1       = vmcb02->control.exit_info_1;
>  	vmcb12->control.exit_info_2       = vmcb02->control.exit_info_2;
>  
> @@ -1422,7 +1417,7 @@ static int nested_svm_intercept_ioio(struct vcpu_svm *svm)
>  
>  static int nested_svm_intercept(struct vcpu_svm *svm)
>  {
> -	u32 exit_code = svm->vmcb->control.exit_code;
> +	u64 exit_code = svm->vmcb->control.exit_code;
>  	int vmexit = NESTED_EXIT_HOST;
>  
>  	if (svm_is_vmrun_failure(exit_code))
> @@ -1494,7 +1489,6 @@ static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu)
>  	struct vmcb *vmcb = svm->vmcb;
>  
>  	vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + ex->vector;
> -	vmcb->control.exit_code_hi = 0;
>  
>  	if (ex->has_error_code)
>  		vmcb->control.exit_info_1 = ex->error_code;
> @@ -1669,7 +1663,6 @@ static void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,
>  	dst->int_vector           = from->int_vector;
>  	dst->int_state            = from->int_state;
>  	dst->exit_code            = from->exit_code;
> -	dst->exit_code_hi         = from->exit_code_hi;
>  	dst->exit_info_1          = from->exit_info_1;
>  	dst->exit_info_2          = from->exit_info_2;
>  	dst->exit_int_info        = from->exit_int_info;
> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index 0835c664fbfd..5aedd07194aa 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c
> @@ -3264,11 +3264,6 @@ void sev_free_vcpu(struct kvm_vcpu *vcpu)
>  		kvfree(svm->sev_es.ghcb_sa);
>  }
>  
> -static u64 kvm_get_cached_sw_exit_code(struct vmcb_control_area *control)
> -{
> -	return (((u64)control->exit_code_hi) << 32) | control->exit_code;
> -}
> -
>  static void dump_ghcb(struct vcpu_svm *svm)
>  {
>  	struct vmcb_control_area *control = &svm->vmcb->control;
> @@ -3290,7 +3285,7 @@ static void dump_ghcb(struct vcpu_svm *svm)
>  	 */
>  	pr_err("GHCB (GPA=%016llx) snapshot:\n", svm->vmcb->control.ghcb_gpa);
>  	pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_code",
> -	       kvm_get_cached_sw_exit_code(control), kvm_ghcb_sw_exit_code_is_valid(svm));
> +	       control->exit_code, kvm_ghcb_sw_exit_code_is_valid(svm));
>  	pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_info_1",
>  	       control->exit_info_1, kvm_ghcb_sw_exit_info_1_is_valid(svm));
>  	pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_info_2",
> @@ -3324,7 +3319,6 @@ static void sev_es_sync_from_ghcb(struct vcpu_svm *svm)
>  	struct vmcb_control_area *control = &svm->vmcb->control;
>  	struct kvm_vcpu *vcpu = &svm->vcpu;
>  	struct ghcb *ghcb = svm->sev_es.ghcb;
> -	u64 exit_code;
>  
>  	/*
>  	 * The GHCB protocol so far allows for the following data
> @@ -3358,9 +3352,7 @@ static void sev_es_sync_from_ghcb(struct vcpu_svm *svm)
>  		__kvm_emulate_msr_write(vcpu, MSR_IA32_XSS, kvm_ghcb_get_xss(svm));
>  
>  	/* Copy the GHCB exit information into the VMCB fields */
> -	exit_code = kvm_ghcb_get_sw_exit_code(svm);
> -	control->exit_code = lower_32_bits(exit_code);
> -	control->exit_code_hi = upper_32_bits(exit_code);
> +	control->exit_code = kvm_ghcb_get_sw_exit_code(svm);
>  	control->exit_info_1 = kvm_ghcb_get_sw_exit_info_1(svm);
>  	control->exit_info_2 = kvm_ghcb_get_sw_exit_info_2(svm);
>  	svm->sev_es.sw_scratch = kvm_ghcb_get_sw_scratch_if_valid(svm);
> @@ -3373,15 +3365,8 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
>  {
>  	struct vmcb_control_area *control = &svm->vmcb->control;
>  	struct kvm_vcpu *vcpu = &svm->vcpu;
> -	u64 exit_code;
>  	u64 reason;
>  
> -	/*
> -	 * Retrieve the exit code now even though it may not be marked valid
> -	 * as it could help with debugging.
> -	 */
> -	exit_code = kvm_get_cached_sw_exit_code(control);
> -
>  	/* Only GHCB Usage code 0 is supported */
>  	if (svm->sev_es.ghcb->ghcb_usage) {
>  		reason = GHCB_ERR_INVALID_USAGE;
> @@ -3395,7 +3380,7 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
>  	    !kvm_ghcb_sw_exit_info_2_is_valid(svm))
>  		goto vmgexit_err;
>  
> -	switch (exit_code) {
> +	switch (control->exit_code) {
>  	case SVM_EXIT_READ_DR7:
>  		break;
>  	case SVM_EXIT_WRITE_DR7:
> @@ -3496,15 +3481,19 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
>  	return 0;
>  
>  vmgexit_err:
> +	/*
> +	 * Print the exit code even though it may not be marked valid as it
> +	 * could help with debugging.
> +	 */
>  	if (reason == GHCB_ERR_INVALID_USAGE) {
>  		vcpu_unimpl(vcpu, "vmgexit: ghcb usage %#x is not valid\n",
>  			    svm->sev_es.ghcb->ghcb_usage);
>  	} else if (reason == GHCB_ERR_INVALID_EVENT) {
>  		vcpu_unimpl(vcpu, "vmgexit: exit code %#llx is not valid\n",
> -			    exit_code);
> +			    control->exit_code);
>  	} else {
>  		vcpu_unimpl(vcpu, "vmgexit: exit code %#llx input is not valid\n",
> -			    exit_code);
> +			    control->exit_code);
>  		dump_ghcb(svm);
>  	}
>  
> @@ -4343,7 +4332,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
>  {
>  	struct vcpu_svm *svm = to_svm(vcpu);
>  	struct vmcb_control_area *control = &svm->vmcb->control;
> -	u64 ghcb_gpa, exit_code;
> +	u64 ghcb_gpa;
>  	int ret;
>  
>  	/* Validate the GHCB */
> @@ -4385,8 +4374,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
>  
>  	svm_vmgexit_success(svm, 0);
>  
> -	exit_code = kvm_get_cached_sw_exit_code(control);
> -	switch (exit_code) {
> +	switch (control->exit_code) {
>  	case SVM_VMGEXIT_MMIO_READ:
>  		ret = setup_vmgexit_scratch(svm, true, control->exit_info_2);
>  		if (ret)
> @@ -4478,7 +4466,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
>  		ret = -EINVAL;
>  		break;
>  	default:
> -		ret = svm_invoke_exit_handler(vcpu, exit_code);
> +		ret = svm_invoke_exit_handler(vcpu, control->exit_code);
>  	}
>  
>  	return ret;
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 3b05476296d0..85bc99f93275 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -2433,7 +2433,6 @@ static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
>  
>  	if (cr0 ^ val) {
>  		svm->vmcb->control.exit_code = SVM_EXIT_CR0_SEL_WRITE;
> -		svm->vmcb->control.exit_code_hi = 0;
>  		ret = (nested_svm_exit_handled(svm) == NESTED_EXIT_DONE);
>  	}
>  
> @@ -3265,7 +3264,7 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
>  	pr_err("%-20s%08x\n", "int_ctl:", control->int_ctl);
>  	pr_err("%-20s%08x\n", "int_vector:", control->int_vector);
>  	pr_err("%-20s%08x\n", "int_state:", control->int_state);
> -	pr_err("%-20s%08x\n", "exit_code:", control->exit_code);
> +	pr_err("%-20s%016llx\n", "exit_code:", control->exit_code);
>  	pr_err("%-20s%016llx\n", "exit_info1:", control->exit_info_1);
>  	pr_err("%-20s%016llx\n", "exit_info2:", control->exit_info_2);
>  	pr_err("%-20s%08x\n", "exit_int_info:", control->exit_int_info);
> @@ -3515,7 +3514,6 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
>  {
>  	struct vcpu_svm *svm = to_svm(vcpu);
>  	struct kvm_run *kvm_run = vcpu->run;
> -	u32 exit_code = svm->vmcb->control.exit_code;
>  
>  	/* SEV-ES guests must use the CR write traps to track CR registers. */
>  	if (!sev_es_guest(vcpu->kvm)) {
> @@ -3551,7 +3549,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
>  	if (exit_fastpath != EXIT_FASTPATH_NONE)
>  		return 1;
>  
> -	return svm_invoke_exit_handler(vcpu, exit_code);
> +	return svm_invoke_exit_handler(vcpu, svm->vmcb->control.exit_code);
>  }
>  
>  static int pre_svm_run(struct kvm_vcpu *vcpu)
> @@ -4618,7 +4616,6 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
>  	if (static_cpu_has(X86_FEATURE_NRIPS))
>  		vmcb->control.next_rip  = info->next_rip;
>  	vmcb->control.exit_code = icpt_info.exit_code;
> -	vmcb->control.exit_code_hi = 0;
>  	vmexit = nested_svm_exit_handled(svm);
>  
>  	ret = (vmexit == NESTED_EXIT_DONE) ? X86EMUL_INTERCEPTED
> diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
> index 6b35925e3a33..31ee4f65dcc2 100644
> --- a/arch/x86/kvm/svm/svm.h
> +++ b/arch/x86/kvm/svm/svm.h
> @@ -162,8 +162,7 @@ struct vmcb_ctrl_area_cached {
>  	u32 int_ctl;
>  	u32 int_vector;
>  	u32 int_state;
> -	u32 exit_code;
> -	u32 exit_code_hi;
> +	u64 exit_code;
>  	u64 exit_info_1;
>  	u64 exit_info_2;
>  	u32 exit_int_info;
> @@ -769,7 +768,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm);
>  static inline int nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
>  {
>  	svm->vmcb->control.exit_code	= exit_code;
> -	svm->vmcb->control.exit_code_hi	= 0;
>  	svm->vmcb->control.exit_info_1	= 0;
>  	svm->vmcb->control.exit_info_2	= 0;
>  	return nested_svm_vmexit(svm);
> diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> index e79bc9cb7162..4c7a5cd10990 100644
> --- a/arch/x86/kvm/trace.h
> +++ b/arch/x86/kvm/trace.h
> @@ -781,7 +781,7 @@ TRACE_EVENT_KVM_EXIT(kvm_nested_vmexit);
>   * Tracepoint for #VMEXIT reinjected to the guest
>   */
>  TRACE_EVENT(kvm_nested_vmexit_inject,
> -	    TP_PROTO(__u32 exit_code,
> +	    TP_PROTO(__u64 exit_code,
>  		     __u64 exit_info1, __u64 exit_info2,
>  		     __u32 exit_int_info, __u32 exit_int_info_err, __u32 isa),
>  	    TP_ARGS(exit_code, exit_info1, exit_info2,
> diff --git a/include/hyperv/hvgdk.h b/include/hyperv/hvgdk.h
> index dd6d4939ea29..56b695873a72 100644
> --- a/include/hyperv/hvgdk.h
> +++ b/include/hyperv/hvgdk.h
> @@ -281,7 +281,7 @@ struct hv_vmcb_enlightenments {
>  #define HV_VMCB_NESTED_ENLIGHTENMENTS		31
>  
>  /* Synthetic VM-Exit */
> -#define HV_SVM_EXITCODE_ENL			0xf0000000
> +#define HV_SVM_EXITCODE_ENL			0xf0000000u
>  #define HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH	(1)
>  
>  /* VM_PARTITION_ASSIST_PAGE */
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

Re: [PATCH 8/9] KVM: SVM: Limit incorrect check on SVM_EXIT_ERR to running as a VM

[Yosry Ahmed]

On Thu, Nov 13, 2025 at 02:56:20PM -0800, Sean Christopherson wrote:
> Limit KVM's incorrect check for VMXEXIT_INVALID, a.k.a. SVM_EXIT_ERR, to
> running as a VM, as detected by X86_FEATURE_HYPERVISOR.  The exit_code and
> all failure codes, e.g. VMXEXIT_INVALID, are 64-bit values, and so checking
> only bits 31:0 could result in false positives when running on non-broken
> hardware, e.g. in the extremely unlikely scenario exit code 0xffffffffull
> is ever generated by hardware.
> 
> Keep the 32-bit check to play nice with running on broken KVM (for years,
> KVM has not set bits 63:32 when synthesizing nested SVM VM-Exits).
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>

> ---
>  arch/x86/kvm/svm/svm.h | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
> index 31ee4f65dcc2..801dcfc64d0b 100644
> --- a/arch/x86/kvm/svm/svm.h
> +++ b/arch/x86/kvm/svm/svm.h
> @@ -427,7 +427,10 @@ static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
>  
>  static inline bool svm_is_vmrun_failure(u64 exit_code)
>  {
> -	return (u32)exit_code == (u32)SVM_EXIT_ERR;
> +	if (cpu_feature_enabled(X86_FEATURE_HYPERVISOR))
> +		return (u32)exit_code == (u32)SVM_EXIT_ERR;
> +
> +	return exit_code == SVM_EXIT_ERR;
>  }
>  
>  /*
> -- 
> 2.52.0.rc1.455.g30608eb744-goog
> 

RE: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Michael Kelley]

From: Sean Christopherson <seanjc@google.com> Sent: Thursday, November 13, 2025 2:56 PM
> 

Adding Microsoft's Nuno Das Neves to the "To:" line since he
originated the work to keep the Linux headers such as hvgdk.h in
sync with the Windows counterparts from which they originate.

> Fix KVM's long-standing buggy handling of SVM's exit_code as a 32-bit
> value.  Per the APM and Xen commit d1bd157fbc ("Big merge the HVM
> full-virtualisation abstractions.") (which is arguably more trustworthy
> than KVM), offset 0x70 is a single 64-bit value:
> 
>   070h 63:0 EXITCODE
> 
> Track exit_code as a single u64 to prevent reintroducing bugs where KVM
> neglects to correctly set bits 63:32.
> 
> Fixes: 6aa8b732ca01 ("[PATCH] kvm: userspace interface")
> Cc: Jim Mattson <jmattson@google.com>
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/include/asm/svm.h      |  3 +--
>  arch/x86/include/uapi/asm/svm.h | 32 ++++++++++++++---------------
>  arch/x86/kvm/svm/hyperv.c       |  1 -
>  arch/x86/kvm/svm/nested.c       | 13 +++---------
>  arch/x86/kvm/svm/sev.c          | 36 +++++++++++----------------------
>  arch/x86/kvm/svm/svm.c          |  7 ++-----
>  arch/x86/kvm/svm/svm.h          |  4 +---
>  arch/x86/kvm/trace.h            |  2 +-
>  include/hyperv/hvgdk.h          |  2 +-
>  9 files changed, 37 insertions(+), 63 deletions(-)
> 
> diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
> index e69b6d0dedcf..66b22cffedfc 100644
> --- a/arch/x86/include/asm/svm.h
> +++ b/arch/x86/include/asm/svm.h
> @@ -136,8 +136,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
>  	u32 int_vector;
>  	u32 int_state;
>  	u8 reserved_3[4];
> -	u32 exit_code;
> -	u32 exit_code_hi;
> +	u64 exit_code;
>  	u64 exit_info_1;
>  	u64 exit_info_2;
>  	u32 exit_int_info;
> diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
> index 650e3256ea7d..010a45c9f614 100644
> --- a/arch/x86/include/uapi/asm/svm.h
> +++ b/arch/x86/include/uapi/asm/svm.h
> @@ -103,38 +103,38 @@
>  #define SVM_EXIT_VMGEXIT       0x403
> 
>  /* SEV-ES software-defined VMGEXIT events */
> -#define SVM_VMGEXIT_MMIO_READ			0x80000001
> -#define SVM_VMGEXIT_MMIO_WRITE			0x80000002
> -#define SVM_VMGEXIT_NMI_COMPLETE		0x80000003
> -#define SVM_VMGEXIT_AP_HLT_LOOP			0x80000004
> -#define SVM_VMGEXIT_AP_JUMP_TABLE		0x80000005
> +#define SVM_VMGEXIT_MMIO_READ			0x80000001ull
> +#define SVM_VMGEXIT_MMIO_WRITE			0x80000002ull
> +#define SVM_VMGEXIT_NMI_COMPLETE		0x80000003ull
> +#define SVM_VMGEXIT_AP_HLT_LOOP			0x80000004ull
> +#define SVM_VMGEXIT_AP_JUMP_TABLE		0x80000005ull
>  #define SVM_VMGEXIT_SET_AP_JUMP_TABLE		0
>  #define SVM_VMGEXIT_GET_AP_JUMP_TABLE		1
> -#define SVM_VMGEXIT_PSC				0x80000010
> -#define SVM_VMGEXIT_GUEST_REQUEST		0x80000011
> -#define SVM_VMGEXIT_EXT_GUEST_REQUEST		0x80000012
> -#define SVM_VMGEXIT_AP_CREATION			0x80000013
> +#define SVM_VMGEXIT_PSC				0x80000010ull
> +#define SVM_VMGEXIT_GUEST_REQUEST		0x80000011ull
> +#define SVM_VMGEXIT_EXT_GUEST_REQUEST		0x80000012ull
> +#define SVM_VMGEXIT_AP_CREATION			0x80000013ull
>  #define SVM_VMGEXIT_AP_CREATE_ON_INIT		0
>  #define SVM_VMGEXIT_AP_CREATE			1
>  #define SVM_VMGEXIT_AP_DESTROY			2
> -#define SVM_VMGEXIT_SNP_RUN_VMPL		0x80000018
> -#define SVM_VMGEXIT_SAVIC			0x8000001a
> +#define SVM_VMGEXIT_SNP_RUN_VMPL		0x80000018ull
> +#define SVM_VMGEXIT_SAVIC			0x8000001aull
>  #define SVM_VMGEXIT_SAVIC_REGISTER_GPA		0
>  #define SVM_VMGEXIT_SAVIC_UNREGISTER_GPA	1
>  #define SVM_VMGEXIT_SAVIC_SELF_GPA		~0ULL
> -#define SVM_VMGEXIT_HV_FEATURES			0x8000fffd
> -#define SVM_VMGEXIT_TERM_REQUEST		0x8000fffe
> +#define SVM_VMGEXIT_HV_FEATURES			0x8000fffdull
> +#define SVM_VMGEXIT_TERM_REQUEST		0x8000fffeull
>  #define SVM_VMGEXIT_TERM_REASON(reason_set, reason_code)	\
>  	/* SW_EXITINFO1[3:0] */					\
>  	(((((u64)reason_set) & 0xf)) |				\
>  	/* SW_EXITINFO1[11:4] */				\
>  	((((u64)reason_code) & 0xff) << 4))
> -#define SVM_VMGEXIT_UNSUPPORTED_EVENT		0x8000ffff
> +#define SVM_VMGEXIT_UNSUPPORTED_EVENT		0x8000ffffull
> 
>  /* Exit code reserved for hypervisor/software use */
> -#define SVM_EXIT_SW				0xf0000000
> +#define SVM_EXIT_SW				0xf0000000ull
> 
> -#define SVM_EXIT_ERR           -1
> +#define SVM_EXIT_ERR           -1ull
> 

[snip]

> diff --git a/include/hyperv/hvgdk.h b/include/hyperv/hvgdk.h
> index dd6d4939ea29..56b695873a72 100644
> --- a/include/hyperv/hvgdk.h
> +++ b/include/hyperv/hvgdk.h
> @@ -281,7 +281,7 @@ struct hv_vmcb_enlightenments {
>  #define HV_VMCB_NESTED_ENLIGHTENMENTS		31
> 
>  /* Synthetic VM-Exit */
> -#define HV_SVM_EXITCODE_ENL			0xf0000000
> +#define HV_SVM_EXITCODE_ENL			0xf0000000u

Is there a reason for making this Hyper-V code just "u", while
making the SVM_VMGEXIT_* values "ull"? I don't think
"u" vs. "ull" shouldn't make any difference when assigning to a
u64, but the inconsistency piqued my interest ....

Michael

Re: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Sean Christopherson]

On Fri, Nov 14, 2025, Michael Kelley wrote:
> From: Sean Christopherson <seanjc@google.com> Sent: Thursday, November 13, 2025 2:56 PM
> > 
> 
> Adding Microsoft's Nuno Das Neves to the "To:" line since he
> originated the work to keep the Linux headers such as hvgdk.h in
> sync with the Windows counterparts from which they originate.

...

> >  /* Exit code reserved for hypervisor/software use */
> > -#define SVM_EXIT_SW				0xf0000000
> > +#define SVM_EXIT_SW				0xf0000000ull
> > 
> > -#define SVM_EXIT_ERR           -1
> > +#define SVM_EXIT_ERR           -1ull
> > 
> 
> [snip]
> 
> > diff --git a/include/hyperv/hvgdk.h b/include/hyperv/hvgdk.h
> > index dd6d4939ea29..56b695873a72 100644
> > --- a/include/hyperv/hvgdk.h
> > +++ b/include/hyperv/hvgdk.h
> > @@ -281,7 +281,7 @@ struct hv_vmcb_enlightenments {
> >  #define HV_VMCB_NESTED_ENLIGHTENMENTS		31
> > 
> >  /* Synthetic VM-Exit */
> > -#define HV_SVM_EXITCODE_ENL			0xf0000000
> > +#define HV_SVM_EXITCODE_ENL			0xf0000000u
> 
> Is there a reason for making this Hyper-V code just "u", while
> making the SVM_VMGEXIT_* values "ull"? I don't think
> "u" vs. "ull" shouldn't make any difference when assigning to a
> u64, but the inconsistency piqued my interest ....

I hedged and went for a more "minimal" change because it isn't KVM code, and at
the time because I thought the value isn't defined by the APM.  Though looking
again at the APM, it does reserve that value for software

  F000_000h    Unused    Reserved for Host.

and I can't find anything in the TLFS.  Ah, my PDF copy is just stale, it's indeed
defined as a synthetic exit.

  https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/nested-virtualization#synthetic-vm-exit

Anyways, I'm in favor of making HV_SVM_EXITCODE_ENL an ull, though part of me
wonders if we should do:

  #define HV_SVM_EXITCODE_ENL	SVM_EXIT_SW

Re: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Sean Christopherson]

On Thu, Nov 13, 2025, Sean Christopherson wrote:
> Fix KVM's long-standing buggy handling of SVM's exit_code as a 32-bit
> value.  Per the APM and Xen commit d1bd157fbc ("Big merge the HVM
> full-virtualisation abstractions.") (which is arguably more trustworthy
> than KVM), offset 0x70 is a single 64-bit value:
> 
>   070h 63:0 EXITCODE
> 
> Track exit_code as a single u64 to prevent reintroducing bugs where KVM
> neglects to correctly set bits 63:32.
> 
> Fixes: 6aa8b732ca01 ("[PATCH] kvm: userspace interface")
> Cc: Jim Mattson <jmattson@google.com>
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---

...

> diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
> index e79bc9cb7162..4c7a5cd10990 100644
> --- a/arch/x86/kvm/trace.h
> +++ b/arch/x86/kvm/trace.h
> @@ -781,7 +781,7 @@ TRACE_EVENT_KVM_EXIT(kvm_nested_vmexit);
>   * Tracepoint for #VMEXIT reinjected to the guest
>   */
>  TRACE_EVENT(kvm_nested_vmexit_inject,
> -	    TP_PROTO(__u32 exit_code,
> +	    TP_PROTO(__u64 exit_code,
>  		     __u64 exit_info1, __u64 exit_info2,
>  		     __u32 exit_int_info, __u32 exit_int_info_err, __u32 isa),
>  	    TP_ARGS(exit_code, exit_info1, exit_info2,

As pointed out by the test bot[*], the trace macro to print exit reasons needs
to use 64-bit variants to play nice with 32-bit builds.

And now I'm questioning all of my testing, because my build setup detects that
as well, _and_ the hyperv_svm_test selftest fails.  *sigh*

diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 4c7a5cd10990..0fd72ce83926 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -383,10 +383,10 @@ TRACE_EVENT(kvm_apic,
 #define kvm_print_exit_reason(exit_reason, isa)                                \
        (isa == KVM_ISA_VMX) ?                                          \
        __print_symbolic(exit_reason & 0xffff, VMX_EXIT_REASONS) :      \
-       __print_symbolic(exit_reason, SVM_EXIT_REASONS),                \
+       __print_symbolic64(exit_reason, SVM_EXIT_REASONS),              \
        (isa == KVM_ISA_VMX && exit_reason & ~0xffff) ? " " : "",       \
        (isa == KVM_ISA_VMX) ?                                          \
-       __print_flags(exit_reason & ~0xffff, " ", VMX_EXIT_REASON_FLAGS) : ""
+       __print_flags64(exit_reason & ~0xffff, " ", VMX_EXIT_REASON_FLAGS) : ""
 
 #define TRACE_EVENT_KVM_EXIT(name)                                          \
 TRACE_EVENT(name,  


[*] https://lore.kernel.org/all/202511141707.t4ad044J-lkp@intel.com

Re: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Sean Christopherson]

On Fri, Nov 14, 2025, Sean Christopherson wrote:
> _and_ the hyperv_svm_test selftest fails.  *sigh*

And the weekend can't come soon enough.  The kernel I'm testing doesn't even have
these patches.  /facepalm

Re: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Wei Liu]

On Fri, Nov 14, 2025 at 07:22:41AM -0800, Sean Christopherson wrote:
> On Fri, Nov 14, 2025, Michael Kelley wrote:
> > From: Sean Christopherson <seanjc@google.com> Sent: Thursday, November 13, 2025 2:56 PM
> > > 
> > 
> > Adding Microsoft's Nuno Das Neves to the "To:" line since he
> > originated the work to keep the Linux headers such as hvgdk.h in
> > sync with the Windows counterparts from which they originate.
> 
> ...
> 
> > >  /* Exit code reserved for hypervisor/software use */
> > > -#define SVM_EXIT_SW				0xf0000000
> > > +#define SVM_EXIT_SW				0xf0000000ull
> > > 
> > > -#define SVM_EXIT_ERR           -1
> > > +#define SVM_EXIT_ERR           -1ull
> > > 
> > 
> > [snip]
> > 
> > > diff --git a/include/hyperv/hvgdk.h b/include/hyperv/hvgdk.h
> > > index dd6d4939ea29..56b695873a72 100644
> > > --- a/include/hyperv/hvgdk.h
> > > +++ b/include/hyperv/hvgdk.h
> > > @@ -281,7 +281,7 @@ struct hv_vmcb_enlightenments {
> > >  #define HV_VMCB_NESTED_ENLIGHTENMENTS		31
> > > 
> > >  /* Synthetic VM-Exit */
> > > -#define HV_SVM_EXITCODE_ENL			0xf0000000
> > > +#define HV_SVM_EXITCODE_ENL			0xf0000000u
> > 
> > Is there a reason for making this Hyper-V code just "u", while
> > making the SVM_VMGEXIT_* values "ull"? I don't think
> > "u" vs. "ull" shouldn't make any difference when assigning to a
> > u64, but the inconsistency piqued my interest ....
> 
> I hedged and went for a more "minimal" change because it isn't KVM code, and at
> the time because I thought the value isn't defined by the APM.  Though looking
> again at the APM, it does reserve that value for software
> 
>   F000_000h    Unused    Reserved for Host.
> 
> and I can't find anything in the TLFS.  Ah, my PDF copy is just stale, it's indeed
> defined as a synthetic exit.
> 
>   https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/nested-virtualization#synthetic-vm-exit
> 
> Anyways, I'm in favor of making HV_SVM_EXITCODE_ENL an ull, though part of me
> wonders if we should do:
> 
>   #define HV_SVM_EXITCODE_ENL	SVM_EXIT_SW

I know this is very tempting, but these headers are supposed to mirror
Microsoft's internal headers, so we would like to keep them
self-contained for ease of tracking.

It should be fine to add the "ull" suffix here. I briefly talked to a
hypervisor developer and they agreed.

Thanks,
Wei

Re: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Sean Christopherson]

On Fri, Nov 14, 2025, Wei Liu wrote:
> On Fri, Nov 14, 2025 at 07:22:41AM -0800, Sean Christopherson wrote:
> > Ah, my PDF copy is just stale, it's indeed
> > defined as a synthetic exit.
> > 
> >   https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/nested-virtualization#synthetic-vm-exit
> > 
> > Anyways, I'm in favor of making HV_SVM_EXITCODE_ENL an ull, though part of me
> > wonders if we should do:
> > 
> >   #define HV_SVM_EXITCODE_ENL	SVM_EXIT_SW
> 
> I know this is very tempting, but these headers are supposed to mirror
> Microsoft's internal headers, so we would like to keep them
> self-contained for ease of tracking.

Ya, no argument from me.  Aha!  Even better, what I can do is have KVM assert
that HV_SVM_EXITCODE_ENL == SVM_EXIT_SW in the KVM Hyper-V code, because what I
really want to do is connect the dots for KVM folks.

> It should be fine to add the "ull" suffix here. I briefly talked to a
> hypervisor developer and they agreed.

Nice, thanks much!

Re: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Wei Liu]

On Fri, Nov 14, 2025 at 10:35:41AM -0800, Sean Christopherson wrote:
> On Fri, Nov 14, 2025, Wei Liu wrote:
> > On Fri, Nov 14, 2025 at 07:22:41AM -0800, Sean Christopherson wrote:
> > > Ah, my PDF copy is just stale, it's indeed
> > > defined as a synthetic exit.
> > > 
> > >   https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/nested-virtualization#synthetic-vm-exit
> > > 
> > > Anyways, I'm in favor of making HV_SVM_EXITCODE_ENL an ull, though part of me
> > > wonders if we should do:
> > > 
> > >   #define HV_SVM_EXITCODE_ENL	SVM_EXIT_SW
> > 
> > I know this is very tempting, but these headers are supposed to mirror
> > Microsoft's internal headers, so we would like to keep them
> > self-contained for ease of tracking.
> 
> Ya, no argument from me.  Aha!  Even better, what I can do is have KVM assert
> that HV_SVM_EXITCODE_ENL == SVM_EXIT_SW in the KVM Hyper-V code, because what I
> really want to do is connect the dots for KVM folks.

This sounds like a good plan.

Wei

Re: [PATCH 6/9] KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on bare metal

[Paolo Bonzini]

On 11/13/25 23:56, Sean Christopherson wrote:
> Explicitly filter out 64-bit exit codes when invoking exit handlers, as
> svm_exit_handlers[] will never be sized with entries that use bits 63:32.
> 
> Processing the non-failing exit code as a 32-bit value will allow tracking
> exit_code as a single 64-bit value (which it is, architecturally).  This
> will also allow hardening KVM against Spectre-like attacks without needing
> to do silly things to avoid build failures on 32-bit kernels
> (array_index_nospec() rightly asserts that the index fits in an "unsigned
> long").
> 
> Omit the check when running as a VM, as KVM has historically failed to set
> bits 63:32 appropriately when synthesizing VM-Exits, i.e. KVM could get
> false positives when running as a VM on an older, broken KVM/kernel.  From
> a functional perspective, omitting the check is "fine", as any unwanted
> collision between e.g. VMEXIT_INVALID and a 32-bit exit code will be
> fatal to KVM-on-KVM regardless of what KVM-as-L1 does.
> 
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>   arch/x86/kvm/svm/svm.c | 18 ++++++++++++++++--
>   1 file changed, 16 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 202a4d8088a2..3b05476296d0 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3433,8 +3433,22 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
>   		sev_free_decrypted_vmsa(vcpu, save);
>   }
>   
> -int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
> +int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 __exit_code)
>   {
> +	u32 exit_code = __exit_code;
> +
> +	/*
> +	 * SVM uses negative values, i.e. 64-bit values, to indicate that VMRUN
> +	 * failed.  Report all such errors to userspace (note, VMEXIT_INVALID,
> +	 * a.k.a. SVM_EXIT_ERR, is special cased by svm_handle_exit()).  Skip
> +	 * the check when running as a VM, as KVM has historically left garbage
> +	 * in bits 63:32, i.e. running KVM-on-KVM would hit false positives if
> +	 * the underlying kernel is buggy.
> +	 */
> +	if (!cpu_feature_enabled(X86_FEATURE_HYPERVISOR) &&
> +	    (u64)exit_code != __exit_code)
> +		goto unexpected_vmexit;

I reviewed the series and it looks good, but with respect to this patch 
and patch 8, is it really worth it?  While there is a possibility that 
code 0x00000000ffffffff is used, or that any high 32-bit values other 
than all-zeros or all-ones are used, they'd be presumably enabled by 
some control bits in the VMCB or some paravirt thing in the hypervisor.

What really matters is that SEV-ES's kvm_get_cached_sw_exit_code() is 
reading the full 64 bits and discarding invalid codes before reaching 
svm_invoke_exit_handler().

I totally agree, of course, with passing __exit_code as u64 and adding a 
comment explaining what's going on with "u32 exit_code == (u32)__exit_code".

Paolo

>   #ifdef CONFIG_MITIGATION_RETPOLINE
>   	if (exit_code == SVM_EXIT_MSR)
>   		return msr_interception(vcpu);
> @@ -3461,7 +3475,7 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
>   
>   unexpected_vmexit:
>   	dump_vmcb(vcpu);
> -	kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
> +	kvm_prepare_unexpected_reason_exit(vcpu, __exit_code);
>   	return 0;
>   }
>   

Re: [PATCH 7/9] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM

[Paolo Bonzini]

On 11/14/25 16:47, Sean Christopherson wrote:
> On Fri, Nov 14, 2025, Sean Christopherson wrote:
>> _and_ the hyperv_svm_test selftest fails.  *sigh*
> 
> And the weekend can't come soon enough.  The kernel I'm testing doesn't even have
> these patches.  /facepalm

Yep, my fault; I did the same mistake (in reverse) when preparing the 
last pull request.  Yosry wrote about this a few days ago.

Paolo

Re: [PATCH 6/9] KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on bare metal

[Sean Christopherson]

On Sat, Nov 15, 2025, Paolo Bonzini wrote:
> On 11/13/25 23:56, Sean Christopherson wrote:
> > Explicitly filter out 64-bit exit codes when invoking exit handlers, as
> > svm_exit_handlers[] will never be sized with entries that use bits 63:32.
> > 
> > Processing the non-failing exit code as a 32-bit value will allow tracking
> > exit_code as a single 64-bit value (which it is, architecturally).  This
> > will also allow hardening KVM against Spectre-like attacks without needing
> > to do silly things to avoid build failures on 32-bit kernels
> > (array_index_nospec() rightly asserts that the index fits in an "unsigned
> > long").
> > 
> > Omit the check when running as a VM, as KVM has historically failed to set
> > bits 63:32 appropriately when synthesizing VM-Exits, i.e. KVM could get
> > false positives when running as a VM on an older, broken KVM/kernel.  From
> > a functional perspective, omitting the check is "fine", as any unwanted
> > collision between e.g. VMEXIT_INVALID and a 32-bit exit code will be
> > fatal to KVM-on-KVM regardless of what KVM-as-L1 does.
> > 
> > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > ---
> >   arch/x86/kvm/svm/svm.c | 18 ++++++++++++++++--
> >   1 file changed, 16 insertions(+), 2 deletions(-)
> > 
> > diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> > index 202a4d8088a2..3b05476296d0 100644
> > --- a/arch/x86/kvm/svm/svm.c
> > +++ b/arch/x86/kvm/svm/svm.c
> > @@ -3433,8 +3433,22 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
> >   		sev_free_decrypted_vmsa(vcpu, save);
> >   }
> > -int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
> > +int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 __exit_code)
> >   {
> > +	u32 exit_code = __exit_code;
> > +
> > +	/*
> > +	 * SVM uses negative values, i.e. 64-bit values, to indicate that VMRUN
> > +	 * failed.  Report all such errors to userspace (note, VMEXIT_INVALID,
> > +	 * a.k.a. SVM_EXIT_ERR, is special cased by svm_handle_exit()).  Skip
> > +	 * the check when running as a VM, as KVM has historically left garbage
> > +	 * in bits 63:32, i.e. running KVM-on-KVM would hit false positives if
> > +	 * the underlying kernel is buggy.
> > +	 */
> > +	if (!cpu_feature_enabled(X86_FEATURE_HYPERVISOR) &&
> > +	    (u64)exit_code != __exit_code)
> > +		goto unexpected_vmexit;
> 
> I reviewed the series and it looks good, but with respect to this patch and
> patch 8, is it really worth it?  While there is a possibility that code
> 0x00000000ffffffff is used, or that any high 32-bit values other than
> all-zeros or all-ones are used, they'd be presumably enabled by some control
> bits in the VMCB or some paravirt thing in the hypervisor.

Maybe.  E.g. TDCALL and SEAMCALL VM-Exits on Intel show up without any enablement
in software (beyond VMXON).  I completely agree that it's extremely unlikely that
AMD will add a on-negative exit code with bits 63:32 != 0, i.e. that we could get
a false positive when truncating exit_code to a u32, but it also seems harmless
to be paranoid.

FWIW, I was assuming VMEXIT_INVALID_PMC (-4) was a generic vPMU thing, but it
looks like that one is also SEV-ES+ specific.

As for e57b84699534 ("KVM: SVM: Limit incorrect check on SVM_EXIT_ERR to running
as a VM"), I agree that being paranoid probably doesn't do anything in practice,
but I like being consistent. :-)

> What really matters is that SEV-ES's kvm_get_cached_sw_exit_code() is
> reading the full 64 bits and discarding invalid codes before reaching
> svm_invoke_exit_handler().

No?  sev_handle_vmgexit() only handles SVM_VMGEXIT_xxx exit codes, everything
else is punted to svm_invoke_exit_handler()

	exit_code = kvm_get_cached_sw_exit_code(control);
	switch (exit_code) {
	case SVM_VMGEXIT_<0>
	...
	case SVM_VMGEXIT_<N>
	default:
		ret = svm_invoke_exit_handler(vcpu, exit_code);
	}

And I don't see anything that filters/modifies exit_code_hi.

Re: [PATCH 0/9] KVM: SVM: Fix (hilarious) exit_code bugs

[Sean Christopherson]

On Thu, 13 Nov 2025 14:56:12 -0800, Sean Christopherson wrote:
> Hyper-V folks, y'all are getting Cc'd because of a change in
> include/hyperv/hvgdk.h to ensure HV_SVM_EXITCODE_ENL is an unsigned value.
> AFAICT, only KVM consumes that macro.  That said, any insight you can provide
> on relevant Hyper-V behavior would be appreciated :-)
> 
> 
> Fix bugs in SVM that mostly impact nested SVM where KVM treats exit codes
> as 32-bit values instead of 64-bit values.  I have no idea how KVM ended up
> with such an egregious flaw, as the blame trail goes all the way back to
> commit 6aa8b732ca01 ("[PATCH] kvm: userspace interface").  Maybe there was
> pre-production hardware or something?
> 
> [...]

Applied 1 and 2 to kvm-x86 fixes.  I'll send v2 for the rest soon-ish.

[1/9] KVM: nSVM: Clear exit_code_hi in VMCB when synthesizing nested VM-Exits
      https://github.com/kvm-x86/linux/commit/da01f64e7470
[2/9] KVM: nSVM: Set exit_code_hi to -1 when synthesizing SVM_EXIT_ERR (failed VMRUN)
      https://github.com/kvm-x86/linux/commit/f402ecd7a8b6

--
https://github.com/kvm-x86/linux/tree/next