* [PATCH v2 1/8] KVM: SVM: Add a helper to detect VMRUN failures
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
@ 2025-12-30 21:13 ` Sean Christopherson
2026-01-02 16:44 ` Yosry Ahmed
2025-12-30 21:13 ` [PATCH v2 2/8] KVM: SVM: Open code handling of unexpected exits in svm_invoke_exit_handler() Sean Christopherson
` (7 subsequent siblings)
8 siblings, 1 reply; 14+ messages in thread
From: Sean Christopherson @ 2025-12-30 21:13 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Add a helper to detect VMRUN failures so that KVM can guard against its
own long-standing bug, where KVM neglects to set exitcode[63:32] when
synthesizing a nested VMFAIL_INVALID VM-Exit. This will allow fixing
KVM's mess of treating exitcode as two separate 32-bit values without
breaking KVM-on-KVM when running on an older, unfixed KVM.
Cc: Jim Mattson <jmattson@google.com>
Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/nested.c | 16 +++++++---------
arch/x86/kvm/svm/svm.c | 4 ++--
arch/x86/kvm/svm/svm.h | 5 +++++
3 files changed, 14 insertions(+), 11 deletions(-)
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index ba0f11c68372..f5bde972a2b1 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -1134,7 +1134,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
vmcb12->control.exit_info_1 = vmcb02->control.exit_info_1;
vmcb12->control.exit_info_2 = vmcb02->control.exit_info_2;
- if (vmcb12->control.exit_code != SVM_EXIT_ERR)
+ if (!svm_is_vmrun_failure(vmcb12->control.exit_code))
nested_save_pending_event_to_vmcb12(svm, vmcb12);
if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
@@ -1425,6 +1425,9 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
u32 exit_code = svm->vmcb->control.exit_code;
int vmexit = NESTED_EXIT_HOST;
+ if (svm_is_vmrun_failure(exit_code))
+ return NESTED_EXIT_DONE;
+
switch (exit_code) {
case SVM_EXIT_MSR:
vmexit = nested_svm_exit_handled_msr(svm);
@@ -1432,7 +1435,7 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
case SVM_EXIT_IOIO:
vmexit = nested_svm_intercept_ioio(svm);
break;
- case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f: {
+ case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f:
/*
* Host-intercepted exceptions have been checked already in
* nested_svm_exit_special. There is nothing to do here,
@@ -1440,15 +1443,10 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
*/
vmexit = NESTED_EXIT_DONE;
break;
- }
- case SVM_EXIT_ERR: {
- vmexit = NESTED_EXIT_DONE;
- break;
- }
- default: {
+ default:
if (vmcb12_is_intercept(&svm->nested.ctl, exit_code))
vmexit = NESTED_EXIT_DONE;
- }
+ break;
}
return vmexit;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 24d59ccfa40d..c2ddf2e0aa1a 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3540,7 +3540,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
return 1;
}
- if (svm->vmcb->control.exit_code == SVM_EXIT_ERR) {
+ if (svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
kvm_run->exit_reason = KVM_EXIT_FAIL_ENTRY;
kvm_run->fail_entry.hardware_entry_failure_reason
= svm->vmcb->control.exit_code;
@@ -4311,7 +4311,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
/* Track VMRUNs that have made past consistency checking */
if (svm->nested.nested_run_pending &&
- svm->vmcb->control.exit_code != SVM_EXIT_ERR)
+ !svm_is_vmrun_failure(svm->vmcb->control.exit_code))
++vcpu->stat.nested_run;
svm->nested.nested_run_pending = 0;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 01be93a53d07..0f006793f973 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -424,6 +424,11 @@ static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
return container_of(vcpu, struct vcpu_svm, vcpu);
}
+static inline bool svm_is_vmrun_failure(u64 exit_code)
+{
+ return (u32)exit_code == (u32)SVM_EXIT_ERR;
+}
+
/*
* Only the PDPTRs are loaded on demand into the shadow MMU. All other
* fields are synchronized on VM-Exit, because accessing the VMCB is cheap.
--
2.52.0.351.gbe84eed79e-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v2 1/8] KVM: SVM: Add a helper to detect VMRUN failures
2025-12-30 21:13 ` [PATCH v2 1/8] KVM: SVM: Add a helper to detect VMRUN failures Sean Christopherson
@ 2026-01-02 16:44 ` Yosry Ahmed
0 siblings, 0 replies; 14+ messages in thread
From: Yosry Ahmed @ 2026-01-02 16:44 UTC (permalink / raw)
To: Sean Christopherson
Cc: Paolo Bonzini, Vitaly Kuznetsov, K. Y. Srinivasan, Haiyang Zhang,
Wei Liu, Dexuan Cui, Long Li, kvm, linux-hyperv, linux-kernel,
Jim Mattson
On Tue, Dec 30, 2025 at 01:13:40PM -0800, Sean Christopherson wrote:
> Add a helper to detect VMRUN failures so that KVM can guard against its
> own long-standing bug, where KVM neglects to set exitcode[63:32] when
> synthesizing a nested VMFAIL_INVALID VM-Exit. This will allow fixing
> KVM's mess of treating exitcode as two separate 32-bit values without
> breaking KVM-on-KVM when running on an older, unfixed KVM.
>
> Cc: Jim Mattson <jmattson@google.com>
> Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>
> ---
> arch/x86/kvm/svm/nested.c | 16 +++++++---------
> arch/x86/kvm/svm/svm.c | 4 ++--
> arch/x86/kvm/svm/svm.h | 5 +++++
> 3 files changed, 14 insertions(+), 11 deletions(-)
>
> diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
> index ba0f11c68372..f5bde972a2b1 100644
> --- a/arch/x86/kvm/svm/nested.c
> +++ b/arch/x86/kvm/svm/nested.c
> @@ -1134,7 +1134,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
> vmcb12->control.exit_info_1 = vmcb02->control.exit_info_1;
> vmcb12->control.exit_info_2 = vmcb02->control.exit_info_2;
>
> - if (vmcb12->control.exit_code != SVM_EXIT_ERR)
> + if (!svm_is_vmrun_failure(vmcb12->control.exit_code))
> nested_save_pending_event_to_vmcb12(svm, vmcb12);
>
> if (guest_cpu_cap_has(vcpu, X86_FEATURE_NRIPS))
> @@ -1425,6 +1425,9 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
> u32 exit_code = svm->vmcb->control.exit_code;
> int vmexit = NESTED_EXIT_HOST;
>
> + if (svm_is_vmrun_failure(exit_code))
> + return NESTED_EXIT_DONE;
> +
> switch (exit_code) {
> case SVM_EXIT_MSR:
> vmexit = nested_svm_exit_handled_msr(svm);
> @@ -1432,7 +1435,7 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
> case SVM_EXIT_IOIO:
> vmexit = nested_svm_intercept_ioio(svm);
> break;
> - case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f: {
> + case SVM_EXIT_EXCP_BASE ... SVM_EXIT_EXCP_BASE + 0x1f:
> /*
> * Host-intercepted exceptions have been checked already in
> * nested_svm_exit_special. There is nothing to do here,
> @@ -1440,15 +1443,10 @@ static int nested_svm_intercept(struct vcpu_svm *svm)
> */
> vmexit = NESTED_EXIT_DONE;
> break;
> - }
> - case SVM_EXIT_ERR: {
> - vmexit = NESTED_EXIT_DONE;
> - break;
> - }
> - default: {
> + default:
> if (vmcb12_is_intercept(&svm->nested.ctl, exit_code))
> vmexit = NESTED_EXIT_DONE;
> - }
> + break;
> }
>
> return vmexit;
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 24d59ccfa40d..c2ddf2e0aa1a 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3540,7 +3540,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
> return 1;
> }
>
> - if (svm->vmcb->control.exit_code == SVM_EXIT_ERR) {
> + if (svm_is_vmrun_failure(svm->vmcb->control.exit_code)) {
> kvm_run->exit_reason = KVM_EXIT_FAIL_ENTRY;
> kvm_run->fail_entry.hardware_entry_failure_reason
> = svm->vmcb->control.exit_code;
> @@ -4311,7 +4311,7 @@ static __no_kcsan fastpath_t svm_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags)
>
> /* Track VMRUNs that have made past consistency checking */
> if (svm->nested.nested_run_pending &&
> - svm->vmcb->control.exit_code != SVM_EXIT_ERR)
> + !svm_is_vmrun_failure(svm->vmcb->control.exit_code))
> ++vcpu->stat.nested_run;
>
> svm->nested.nested_run_pending = 0;
> diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
> index 01be93a53d07..0f006793f973 100644
> --- a/arch/x86/kvm/svm/svm.h
> +++ b/arch/x86/kvm/svm/svm.h
> @@ -424,6 +424,11 @@ static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
> return container_of(vcpu, struct vcpu_svm, vcpu);
> }
>
> +static inline bool svm_is_vmrun_failure(u64 exit_code)
> +{
> + return (u32)exit_code == (u32)SVM_EXIT_ERR;
> +}
> +
> /*
> * Only the PDPTRs are loaded on demand into the shadow MMU. All other
> * fields are synchronized on VM-Exit, because accessing the VMCB is cheap.
> --
> 2.52.0.351.gbe84eed79e-goog
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v2 2/8] KVM: SVM: Open code handling of unexpected exits in svm_invoke_exit_handler()
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
2025-12-30 21:13 ` [PATCH v2 1/8] KVM: SVM: Add a helper to detect VMRUN failures Sean Christopherson
@ 2025-12-30 21:13 ` Sean Christopherson
2026-01-02 11:41 ` Gupta, Pankaj
2025-12-30 21:13 ` [PATCH v2 3/8] KVM: SVM: Check for an unexpected VM-Exit after RETPOLINE "fast" handling Sean Christopherson
` (6 subsequent siblings)
8 siblings, 1 reply; 14+ messages in thread
From: Sean Christopherson @ 2025-12-30 21:13 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Fold svm_check_exit_valid() and svm_handle_invalid_exit() into their sole
caller, svm_invoke_exit_handler(), as having tiny single-use helpers makes
the code unncessarily difficult to follow. This will also allow for
additional cleanups in svm_invoke_exit_handler().
No functional change intended.
Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/svm.c | 25 ++++++++++---------------
1 file changed, 10 insertions(+), 15 deletions(-)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index c2ddf2e0aa1a..a523011f0923 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3443,23 +3443,13 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
sev_free_decrypted_vmsa(vcpu, save);
}
-static bool svm_check_exit_valid(u64 exit_code)
-{
- return (exit_code < ARRAY_SIZE(svm_exit_handlers) &&
- svm_exit_handlers[exit_code]);
-}
-
-static int svm_handle_invalid_exit(struct kvm_vcpu *vcpu, u64 exit_code)
-{
- dump_vmcb(vcpu);
- kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
- return 0;
-}
-
int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
{
- if (!svm_check_exit_valid(exit_code))
- return svm_handle_invalid_exit(vcpu, exit_code);
+ if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
+ goto unexpected_vmexit;
+
+ if (!svm_exit_handlers[exit_code])
+ goto unexpected_vmexit;
#ifdef CONFIG_MITIGATION_RETPOLINE
if (exit_code == SVM_EXIT_MSR)
@@ -3478,6 +3468,11 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
#endif
#endif
return svm_exit_handlers[exit_code](vcpu);
+
+unexpected_vmexit:
+ dump_vmcb(vcpu);
+ kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
+ return 0;
}
static void svm_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,
--
2.52.0.351.gbe84eed79e-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v2 2/8] KVM: SVM: Open code handling of unexpected exits in svm_invoke_exit_handler()
2025-12-30 21:13 ` [PATCH v2 2/8] KVM: SVM: Open code handling of unexpected exits in svm_invoke_exit_handler() Sean Christopherson
@ 2026-01-02 11:41 ` Gupta, Pankaj
0 siblings, 0 replies; 14+ messages in thread
From: Gupta, Pankaj @ 2026-01-02 11:41 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
> Fold svm_check_exit_valid() and svm_handle_invalid_exit() into their sole
> caller, svm_invoke_exit_handler(), as having tiny single-use helpers makes
> the code unncessarily difficult to follow. This will also allow for
> additional cleanups in svm_invoke_exit_handler().
>
> No functional change intended.
>
> Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
> Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Pankaj Gupta <pankaj.gupta@amd.com>
> ---
> arch/x86/kvm/svm/svm.c | 25 ++++++++++---------------
> 1 file changed, 10 insertions(+), 15 deletions(-)
>
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index c2ddf2e0aa1a..a523011f0923 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -3443,23 +3443,13 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
> sev_free_decrypted_vmsa(vcpu, save);
> }
>
> -static bool svm_check_exit_valid(u64 exit_code)
> -{
> - return (exit_code < ARRAY_SIZE(svm_exit_handlers) &&
> - svm_exit_handlers[exit_code]);
> -}
> -
> -static int svm_handle_invalid_exit(struct kvm_vcpu *vcpu, u64 exit_code)
> -{
> - dump_vmcb(vcpu);
> - kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
> - return 0;
> -}
> -
> int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
> {
> - if (!svm_check_exit_valid(exit_code))
> - return svm_handle_invalid_exit(vcpu, exit_code);
> + if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
> + goto unexpected_vmexit;
> +
> + if (!svm_exit_handlers[exit_code])
> + goto unexpected_vmexit;
>
> #ifdef CONFIG_MITIGATION_RETPOLINE
> if (exit_code == SVM_EXIT_MSR)
> @@ -3478,6 +3468,11 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
> #endif
> #endif
> return svm_exit_handlers[exit_code](vcpu);
> +
> +unexpected_vmexit:
> + dump_vmcb(vcpu);
> + kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
> + return 0;
> }
>
> static void svm_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason,
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH v2 3/8] KVM: SVM: Check for an unexpected VM-Exit after RETPOLINE "fast" handling
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
2025-12-30 21:13 ` [PATCH v2 1/8] KVM: SVM: Add a helper to detect VMRUN failures Sean Christopherson
2025-12-30 21:13 ` [PATCH v2 2/8] KVM: SVM: Open code handling of unexpected exits in svm_invoke_exit_handler() Sean Christopherson
@ 2025-12-30 21:13 ` Sean Christopherson
2025-12-30 21:13 ` [PATCH v2 4/8] KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on bare metal Sean Christopherson
` (5 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Sean Christopherson @ 2025-12-30 21:13 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Check for an unexpected/unhandled VM-Exit after the manual RETPOLINE=y
handling. The entire point of the RETPOLINE checks is to optimize for
common VM-Exits, i.e. checking for the rare case of an unsupported
VM-Exit is counter-productive. This also aligns SVM and VMX exit handling.
No functional change intended.
Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/svm.c | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index a523011f0923..e24bedf1fc81 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3445,12 +3445,6 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
{
- if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
- goto unexpected_vmexit;
-
- if (!svm_exit_handlers[exit_code])
- goto unexpected_vmexit;
-
#ifdef CONFIG_MITIGATION_RETPOLINE
if (exit_code == SVM_EXIT_MSR)
return msr_interception(vcpu);
@@ -3467,6 +3461,12 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
return sev_handle_vmgexit(vcpu);
#endif
#endif
+ if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
+ goto unexpected_vmexit;
+
+ if (!svm_exit_handlers[exit_code])
+ goto unexpected_vmexit;
+
return svm_exit_handlers[exit_code](vcpu);
unexpected_vmexit:
--
2.52.0.351.gbe84eed79e-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v2 4/8] KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on bare metal
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
` (2 preceding siblings ...)
2025-12-30 21:13 ` [PATCH v2 3/8] KVM: SVM: Check for an unexpected VM-Exit after RETPOLINE "fast" handling Sean Christopherson
@ 2025-12-30 21:13 ` Sean Christopherson
2025-12-30 21:13 ` [PATCH v2 5/8] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM Sean Christopherson
` (4 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Sean Christopherson @ 2025-12-30 21:13 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Explicitly filter out 64-bit exit codes when invoking exit handlers, as
svm_exit_handlers[] will never be sized with entries that use bits 63:32.
Processing the non-failing exit code as a 32-bit value will allow tracking
exit_code as a single 64-bit value (which it is, architecturally). This
will also allow hardening KVM against Spectre-like attacks without needing
to do silly things to avoid build failures on 32-bit kernels
(array_index_nospec() rightly asserts that the index fits in an "unsigned
long").
Omit the check when running as a VM, as KVM has historically failed to set
bits 63:32 appropriately when synthesizing VM-Exits, i.e. KVM could get
false positives when running as a VM on an older, broken KVM/kernel. From
a functional perspective, omitting the check is "fine", as any unwanted
collision between e.g. VMEXIT_INVALID and a 32-bit exit code will be
fatal to KVM-on-KVM regardless of what KVM-as-L1 does.
Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/svm.c | 18 ++++++++++++++++--
1 file changed, 16 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index e24bedf1fc81..1ffe922e95fd 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3443,8 +3443,22 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
sev_free_decrypted_vmsa(vcpu, save);
}
-int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
+int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 __exit_code)
{
+ u32 exit_code = __exit_code;
+
+ /*
+ * SVM uses negative values, i.e. 64-bit values, to indicate that VMRUN
+ * failed. Report all such errors to userspace (note, VMEXIT_INVALID,
+ * a.k.a. SVM_EXIT_ERR, is special cased by svm_handle_exit()). Skip
+ * the check when running as a VM, as KVM has historically left garbage
+ * in bits 63:32, i.e. running KVM-on-KVM would hit false positives if
+ * the underlying kernel is buggy.
+ */
+ if (!cpu_feature_enabled(X86_FEATURE_HYPERVISOR) &&
+ (u64)exit_code != __exit_code)
+ goto unexpected_vmexit;
+
#ifdef CONFIG_MITIGATION_RETPOLINE
if (exit_code == SVM_EXIT_MSR)
return msr_interception(vcpu);
@@ -3471,7 +3485,7 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code)
unexpected_vmexit:
dump_vmcb(vcpu);
- kvm_prepare_unexpected_reason_exit(vcpu, exit_code);
+ kvm_prepare_unexpected_reason_exit(vcpu, __exit_code);
return 0;
}
--
2.52.0.351.gbe84eed79e-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v2 5/8] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
` (3 preceding siblings ...)
2025-12-30 21:13 ` [PATCH v2 4/8] KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on bare metal Sean Christopherson
@ 2025-12-30 21:13 ` Sean Christopherson
2025-12-30 21:13 ` [PATCH v2 6/8] KVM: SVM: Limit incorrect check on SVM_EXIT_ERR to running as a VM Sean Christopherson
` (3 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Sean Christopherson @ 2025-12-30 21:13 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Fix KVM's long-standing buggy handling of SVM's exit_code as a 32-bit
value. Per the APM and Xen commit d1bd157fbc ("Big merge the HVM
full-virtualisation abstractions.") (which is arguably more trustworthy
than KVM), offset 0x70 is a single 64-bit value:
070h 63:0 EXITCODE
Track exit_code as a single u64 to prevent reintroducing bugs where KVM
neglects to correctly set bits 63:32.
Fixes: 6aa8b732ca01 ("[PATCH] kvm: userspace interface")
Cc: Jim Mattson <jmattson@google.com>
Cc: Yosry Ahmed <yosry.ahmed@linux.dev>
Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/include/asm/svm.h | 3 +-
arch/x86/include/uapi/asm/svm.h | 32 ++++++++---------
arch/x86/kvm/svm/hyperv.c | 1 -
arch/x86/kvm/svm/nested.c | 13 ++-----
arch/x86/kvm/svm/sev.c | 36 +++++++------------
arch/x86/kvm/svm/svm.c | 7 ++--
arch/x86/kvm/svm/svm.h | 4 +--
arch/x86/kvm/trace.h | 6 ++--
include/hyperv/hvgdk.h | 2 +-
tools/testing/selftests/kvm/include/x86/svm.h | 3 +-
.../kvm/x86/svm_nested_soft_inject_test.c | 4 +--
11 files changed, 42 insertions(+), 69 deletions(-)
diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 56aa99503dc4..ba28ff531b60 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -136,8 +136,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
u32 int_vector;
u32 int_state;
u8 reserved_3[4];
- u32 exit_code;
- u32 exit_code_hi;
+ u64 exit_code;
u64 exit_info_1;
u64 exit_info_2;
u32 exit_int_info;
diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 650e3256ea7d..010a45c9f614 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -103,38 +103,38 @@
#define SVM_EXIT_VMGEXIT 0x403
/* SEV-ES software-defined VMGEXIT events */
-#define SVM_VMGEXIT_MMIO_READ 0x80000001
-#define SVM_VMGEXIT_MMIO_WRITE 0x80000002
-#define SVM_VMGEXIT_NMI_COMPLETE 0x80000003
-#define SVM_VMGEXIT_AP_HLT_LOOP 0x80000004
-#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005
+#define SVM_VMGEXIT_MMIO_READ 0x80000001ull
+#define SVM_VMGEXIT_MMIO_WRITE 0x80000002ull
+#define SVM_VMGEXIT_NMI_COMPLETE 0x80000003ull
+#define SVM_VMGEXIT_AP_HLT_LOOP 0x80000004ull
+#define SVM_VMGEXIT_AP_JUMP_TABLE 0x80000005ull
#define SVM_VMGEXIT_SET_AP_JUMP_TABLE 0
#define SVM_VMGEXIT_GET_AP_JUMP_TABLE 1
-#define SVM_VMGEXIT_PSC 0x80000010
-#define SVM_VMGEXIT_GUEST_REQUEST 0x80000011
-#define SVM_VMGEXIT_EXT_GUEST_REQUEST 0x80000012
-#define SVM_VMGEXIT_AP_CREATION 0x80000013
+#define SVM_VMGEXIT_PSC 0x80000010ull
+#define SVM_VMGEXIT_GUEST_REQUEST 0x80000011ull
+#define SVM_VMGEXIT_EXT_GUEST_REQUEST 0x80000012ull
+#define SVM_VMGEXIT_AP_CREATION 0x80000013ull
#define SVM_VMGEXIT_AP_CREATE_ON_INIT 0
#define SVM_VMGEXIT_AP_CREATE 1
#define SVM_VMGEXIT_AP_DESTROY 2
-#define SVM_VMGEXIT_SNP_RUN_VMPL 0x80000018
-#define SVM_VMGEXIT_SAVIC 0x8000001a
+#define SVM_VMGEXIT_SNP_RUN_VMPL 0x80000018ull
+#define SVM_VMGEXIT_SAVIC 0x8000001aull
#define SVM_VMGEXIT_SAVIC_REGISTER_GPA 0
#define SVM_VMGEXIT_SAVIC_UNREGISTER_GPA 1
#define SVM_VMGEXIT_SAVIC_SELF_GPA ~0ULL
-#define SVM_VMGEXIT_HV_FEATURES 0x8000fffd
-#define SVM_VMGEXIT_TERM_REQUEST 0x8000fffe
+#define SVM_VMGEXIT_HV_FEATURES 0x8000fffdull
+#define SVM_VMGEXIT_TERM_REQUEST 0x8000fffeull
#define SVM_VMGEXIT_TERM_REASON(reason_set, reason_code) \
/* SW_EXITINFO1[3:0] */ \
(((((u64)reason_set) & 0xf)) | \
/* SW_EXITINFO1[11:4] */ \
((((u64)reason_code) & 0xff) << 4))
-#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffff
+#define SVM_VMGEXIT_UNSUPPORTED_EVENT 0x8000ffffull
/* Exit code reserved for hypervisor/software use */
-#define SVM_EXIT_SW 0xf0000000
+#define SVM_EXIT_SW 0xf0000000ull
-#define SVM_EXIT_ERR -1
+#define SVM_EXIT_ERR -1ull
#define SVM_EXIT_REASONS \
{ SVM_EXIT_READ_CR0, "read_cr0" }, \
diff --git a/arch/x86/kvm/svm/hyperv.c b/arch/x86/kvm/svm/hyperv.c
index 088f6429b24c..3ec580d687f5 100644
--- a/arch/x86/kvm/svm/hyperv.c
+++ b/arch/x86/kvm/svm/hyperv.c
@@ -11,7 +11,6 @@ void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
struct vcpu_svm *svm = to_svm(vcpu);
svm->vmcb->control.exit_code = HV_SVM_EXITCODE_ENL;
- svm->vmcb->control.exit_code_hi = 0;
svm->vmcb->control.exit_info_1 = HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH;
svm->vmcb->control.exit_info_2 = 0;
nested_svm_vmexit(svm);
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index f5bde972a2b1..18b656508de9 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -45,7 +45,6 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu,
* correctly fill in the high bits of exit_info_1.
*/
vmcb->control.exit_code = SVM_EXIT_NPF;
- vmcb->control.exit_code_hi = 0;
vmcb->control.exit_info_1 = (1ULL << 32);
vmcb->control.exit_info_2 = fault->address;
}
@@ -421,7 +420,6 @@ void __nested_copy_vmcb_control_to_cache(struct kvm_vcpu *vcpu,
to->int_vector = from->int_vector;
to->int_state = from->int_state;
to->exit_code = from->exit_code;
- to->exit_code_hi = from->exit_code_hi;
to->exit_info_1 = from->exit_info_1;
to->exit_info_2 = from->exit_info_2;
to->exit_int_info = from->exit_int_info;
@@ -727,8 +725,8 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
enter_guest_mode(vcpu);
/*
- * Filled at exit: exit_code, exit_code_hi, exit_info_1, exit_info_2,
- * exit_int_info, exit_int_info_err, next_rip, insn_len, insn_bytes.
+ * Filled at exit: exit_code, exit_info_1, exit_info_2, exit_int_info,
+ * exit_int_info_err, next_rip, insn_len, insn_bytes.
*/
if (guest_cpu_cap_has(vcpu, X86_FEATURE_VGIF) &&
@@ -985,7 +983,6 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
if (!nested_vmcb_check_save(vcpu) ||
!nested_vmcb_check_controls(vcpu)) {
vmcb12->control.exit_code = SVM_EXIT_ERR;
- vmcb12->control.exit_code_hi = -1u;
vmcb12->control.exit_info_1 = 0;
vmcb12->control.exit_info_2 = 0;
goto out;
@@ -1018,7 +1015,6 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu)
svm->soft_int_injected = false;
svm->vmcb->control.exit_code = SVM_EXIT_ERR;
- svm->vmcb->control.exit_code_hi = -1u;
svm->vmcb->control.exit_info_1 = 0;
svm->vmcb->control.exit_info_2 = 0;
@@ -1130,7 +1126,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
vmcb12->control.int_state = vmcb02->control.int_state;
vmcb12->control.exit_code = vmcb02->control.exit_code;
- vmcb12->control.exit_code_hi = vmcb02->control.exit_code_hi;
vmcb12->control.exit_info_1 = vmcb02->control.exit_info_1;
vmcb12->control.exit_info_2 = vmcb02->control.exit_info_2;
@@ -1422,7 +1417,7 @@ static int nested_svm_intercept_ioio(struct vcpu_svm *svm)
static int nested_svm_intercept(struct vcpu_svm *svm)
{
- u32 exit_code = svm->vmcb->control.exit_code;
+ u64 exit_code = svm->vmcb->control.exit_code;
int vmexit = NESTED_EXIT_HOST;
if (svm_is_vmrun_failure(exit_code))
@@ -1494,7 +1489,6 @@ static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu)
struct vmcb *vmcb = svm->vmcb;
vmcb->control.exit_code = SVM_EXIT_EXCP_BASE + ex->vector;
- vmcb->control.exit_code_hi = 0;
if (ex->has_error_code)
vmcb->control.exit_info_1 = ex->error_code;
@@ -1669,7 +1663,6 @@ static void nested_copy_vmcb_cache_to_control(struct vmcb_control_area *dst,
dst->int_vector = from->int_vector;
dst->int_state = from->int_state;
dst->exit_code = from->exit_code;
- dst->exit_code_hi = from->exit_code_hi;
dst->exit_info_1 = from->exit_info_1;
dst->exit_info_2 = from->exit_info_2;
dst->exit_int_info = from->exit_int_info;
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index f59c65abe3cf..794640bebb62 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3275,11 +3275,6 @@ void sev_free_vcpu(struct kvm_vcpu *vcpu)
kvfree(svm->sev_es.ghcb_sa);
}
-static u64 kvm_get_cached_sw_exit_code(struct vmcb_control_area *control)
-{
- return (((u64)control->exit_code_hi) << 32) | control->exit_code;
-}
-
static void dump_ghcb(struct vcpu_svm *svm)
{
struct vmcb_control_area *control = &svm->vmcb->control;
@@ -3301,7 +3296,7 @@ static void dump_ghcb(struct vcpu_svm *svm)
*/
pr_err("GHCB (GPA=%016llx) snapshot:\n", svm->vmcb->control.ghcb_gpa);
pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_code",
- kvm_get_cached_sw_exit_code(control), kvm_ghcb_sw_exit_code_is_valid(svm));
+ control->exit_code, kvm_ghcb_sw_exit_code_is_valid(svm));
pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_info_1",
control->exit_info_1, kvm_ghcb_sw_exit_info_1_is_valid(svm));
pr_err("%-20s%016llx is_valid: %u\n", "sw_exit_info_2",
@@ -3335,7 +3330,6 @@ static void sev_es_sync_from_ghcb(struct vcpu_svm *svm)
struct vmcb_control_area *control = &svm->vmcb->control;
struct kvm_vcpu *vcpu = &svm->vcpu;
struct ghcb *ghcb = svm->sev_es.ghcb;
- u64 exit_code;
/*
* The GHCB protocol so far allows for the following data
@@ -3369,9 +3363,7 @@ static void sev_es_sync_from_ghcb(struct vcpu_svm *svm)
__kvm_emulate_msr_write(vcpu, MSR_IA32_XSS, kvm_ghcb_get_xss(svm));
/* Copy the GHCB exit information into the VMCB fields */
- exit_code = kvm_ghcb_get_sw_exit_code(svm);
- control->exit_code = lower_32_bits(exit_code);
- control->exit_code_hi = upper_32_bits(exit_code);
+ control->exit_code = kvm_ghcb_get_sw_exit_code(svm);
control->exit_info_1 = kvm_ghcb_get_sw_exit_info_1(svm);
control->exit_info_2 = kvm_ghcb_get_sw_exit_info_2(svm);
svm->sev_es.sw_scratch = kvm_ghcb_get_sw_scratch_if_valid(svm);
@@ -3384,15 +3376,8 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
{
struct vmcb_control_area *control = &svm->vmcb->control;
struct kvm_vcpu *vcpu = &svm->vcpu;
- u64 exit_code;
u64 reason;
- /*
- * Retrieve the exit code now even though it may not be marked valid
- * as it could help with debugging.
- */
- exit_code = kvm_get_cached_sw_exit_code(control);
-
/* Only GHCB Usage code 0 is supported */
if (svm->sev_es.ghcb->ghcb_usage) {
reason = GHCB_ERR_INVALID_USAGE;
@@ -3406,7 +3391,7 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
!kvm_ghcb_sw_exit_info_2_is_valid(svm))
goto vmgexit_err;
- switch (exit_code) {
+ switch (control->exit_code) {
case SVM_EXIT_READ_DR7:
break;
case SVM_EXIT_WRITE_DR7:
@@ -3507,15 +3492,19 @@ static int sev_es_validate_vmgexit(struct vcpu_svm *svm)
return 0;
vmgexit_err:
+ /*
+ * Print the exit code even though it may not be marked valid as it
+ * could help with debugging.
+ */
if (reason == GHCB_ERR_INVALID_USAGE) {
vcpu_unimpl(vcpu, "vmgexit: ghcb usage %#x is not valid\n",
svm->sev_es.ghcb->ghcb_usage);
} else if (reason == GHCB_ERR_INVALID_EVENT) {
vcpu_unimpl(vcpu, "vmgexit: exit code %#llx is not valid\n",
- exit_code);
+ control->exit_code);
} else {
vcpu_unimpl(vcpu, "vmgexit: exit code %#llx input is not valid\n",
- exit_code);
+ control->exit_code);
dump_ghcb(svm);
}
@@ -4354,7 +4343,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
{
struct vcpu_svm *svm = to_svm(vcpu);
struct vmcb_control_area *control = &svm->vmcb->control;
- u64 ghcb_gpa, exit_code;
+ u64 ghcb_gpa;
int ret;
/* Validate the GHCB */
@@ -4396,8 +4385,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
svm_vmgexit_success(svm, 0);
- exit_code = kvm_get_cached_sw_exit_code(control);
- switch (exit_code) {
+ switch (control->exit_code) {
case SVM_VMGEXIT_MMIO_READ:
ret = setup_vmgexit_scratch(svm, true, control->exit_info_2);
if (ret)
@@ -4489,7 +4477,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
ret = -EINVAL;
break;
default:
- ret = svm_invoke_exit_handler(vcpu, exit_code);
+ ret = svm_invoke_exit_handler(vcpu, control->exit_code);
}
return ret;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 1ffe922e95fd..b97e6763839b 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -2443,7 +2443,6 @@ static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu,
if (cr0 ^ val) {
svm->vmcb->control.exit_code = SVM_EXIT_CR0_SEL_WRITE;
- svm->vmcb->control.exit_code_hi = 0;
ret = (nested_svm_exit_handled(svm) == NESTED_EXIT_DONE);
}
@@ -3275,7 +3274,7 @@ static void dump_vmcb(struct kvm_vcpu *vcpu)
pr_err("%-20s%08x\n", "int_ctl:", control->int_ctl);
pr_err("%-20s%08x\n", "int_vector:", control->int_vector);
pr_err("%-20s%08x\n", "int_state:", control->int_state);
- pr_err("%-20s%08x\n", "exit_code:", control->exit_code);
+ pr_err("%-20s%016llx\n", "exit_code:", control->exit_code);
pr_err("%-20s%016llx\n", "exit_info1:", control->exit_info_1);
pr_err("%-20s%016llx\n", "exit_info2:", control->exit_info_2);
pr_err("%-20s%08x\n", "exit_int_info:", control->exit_int_info);
@@ -3525,7 +3524,6 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
{
struct vcpu_svm *svm = to_svm(vcpu);
struct kvm_run *kvm_run = vcpu->run;
- u32 exit_code = svm->vmcb->control.exit_code;
/* SEV-ES guests must use the CR write traps to track CR registers. */
if (!sev_es_guest(vcpu->kvm)) {
@@ -3561,7 +3559,7 @@ static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath)
if (exit_fastpath != EXIT_FASTPATH_NONE)
return 1;
- return svm_invoke_exit_handler(vcpu, exit_code);
+ return svm_invoke_exit_handler(vcpu, svm->vmcb->control.exit_code);
}
static int pre_svm_run(struct kvm_vcpu *vcpu)
@@ -4627,7 +4625,6 @@ static int svm_check_intercept(struct kvm_vcpu *vcpu,
if (static_cpu_has(X86_FEATURE_NRIPS))
vmcb->control.next_rip = info->next_rip;
vmcb->control.exit_code = icpt_info.exit_code;
- vmcb->control.exit_code_hi = 0;
vmexit = nested_svm_exit_handled(svm);
ret = (vmexit == NESTED_EXIT_DONE) ? X86EMUL_INTERCEPTED
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 0f006793f973..105f1394026e 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -159,8 +159,7 @@ struct vmcb_ctrl_area_cached {
u32 int_ctl;
u32 int_vector;
u32 int_state;
- u32 exit_code;
- u32 exit_code_hi;
+ u64 exit_code;
u64 exit_info_1;
u64 exit_info_2;
u32 exit_int_info;
@@ -767,7 +766,6 @@ int nested_svm_vmexit(struct vcpu_svm *svm);
static inline int nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit_code)
{
svm->vmcb->control.exit_code = exit_code;
- svm->vmcb->control.exit_code_hi = 0;
svm->vmcb->control.exit_info_1 = 0;
svm->vmcb->control.exit_info_2 = 0;
return nested_svm_vmexit(svm);
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index e79bc9cb7162..e7fdbe9efc90 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -383,10 +383,10 @@ TRACE_EVENT(kvm_apic,
#define kvm_print_exit_reason(exit_reason, isa) \
(isa == KVM_ISA_VMX) ? \
__print_symbolic(exit_reason & 0xffff, VMX_EXIT_REASONS) : \
- __print_symbolic(exit_reason, SVM_EXIT_REASONS), \
+ __print_symbolic_u64(exit_reason, SVM_EXIT_REASONS), \
(isa == KVM_ISA_VMX && exit_reason & ~0xffff) ? " " : "", \
(isa == KVM_ISA_VMX) ? \
- __print_flags(exit_reason & ~0xffff, " ", VMX_EXIT_REASON_FLAGS) : ""
+ __print_flags_u64(exit_reason & ~0xffff, " ", VMX_EXIT_REASON_FLAGS) : ""
#define TRACE_EVENT_KVM_EXIT(name) \
TRACE_EVENT(name, \
@@ -781,7 +781,7 @@ TRACE_EVENT_KVM_EXIT(kvm_nested_vmexit);
* Tracepoint for #VMEXIT reinjected to the guest
*/
TRACE_EVENT(kvm_nested_vmexit_inject,
- TP_PROTO(__u32 exit_code,
+ TP_PROTO(__u64 exit_code,
__u64 exit_info1, __u64 exit_info2,
__u32 exit_int_info, __u32 exit_int_info_err, __u32 isa),
TP_ARGS(exit_code, exit_info1, exit_info2,
diff --git a/include/hyperv/hvgdk.h b/include/hyperv/hvgdk.h
index dd6d4939ea29..384c3f3ff4a5 100644
--- a/include/hyperv/hvgdk.h
+++ b/include/hyperv/hvgdk.h
@@ -281,7 +281,7 @@ struct hv_vmcb_enlightenments {
#define HV_VMCB_NESTED_ENLIGHTENMENTS 31
/* Synthetic VM-Exit */
-#define HV_SVM_EXITCODE_ENL 0xf0000000
+#define HV_SVM_EXITCODE_ENL 0xf0000000ull
#define HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH (1)
/* VM_PARTITION_ASSIST_PAGE */
diff --git a/tools/testing/selftests/kvm/include/x86/svm.h b/tools/testing/selftests/kvm/include/x86/svm.h
index 29cffd0a9181..10b30b38bb3f 100644
--- a/tools/testing/selftests/kvm/include/x86/svm.h
+++ b/tools/testing/selftests/kvm/include/x86/svm.h
@@ -92,8 +92,7 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
u32 int_vector;
u32 int_state;
u8 reserved_3[4];
- u32 exit_code;
- u32 exit_code_hi;
+ u64 exit_code;
u64 exit_info_1;
u64 exit_info_2;
u32 exit_int_info;
diff --git a/tools/testing/selftests/kvm/x86/svm_nested_soft_inject_test.c b/tools/testing/selftests/kvm/x86/svm_nested_soft_inject_test.c
index 7b6481d6c0d3..4bd1655f9e6d 100644
--- a/tools/testing/selftests/kvm/x86/svm_nested_soft_inject_test.c
+++ b/tools/testing/selftests/kvm/x86/svm_nested_soft_inject_test.c
@@ -103,7 +103,7 @@ static void l1_guest_code(struct svm_test_data *svm, uint64_t is_nmi, uint64_t i
run_guest(vmcb, svm->vmcb_gpa);
__GUEST_ASSERT(vmcb->control.exit_code == SVM_EXIT_VMMCALL,
- "Expected VMMCAL #VMEXIT, got '0x%x', info1 = '0x%lx, info2 = '0x%lx'",
+ "Expected VMMCAL #VMEXIT, got '0x%lx', info1 = '0x%lx, info2 = '0x%lx'",
vmcb->control.exit_code,
vmcb->control.exit_info_1, vmcb->control.exit_info_2);
@@ -133,7 +133,7 @@ static void l1_guest_code(struct svm_test_data *svm, uint64_t is_nmi, uint64_t i
run_guest(vmcb, svm->vmcb_gpa);
__GUEST_ASSERT(vmcb->control.exit_code == SVM_EXIT_HLT,
- "Expected HLT #VMEXIT, got '0x%x', info1 = '0x%lx, info2 = '0x%lx'",
+ "Expected HLT #VMEXIT, got '0x%lx', info1 = '0x%lx, info2 = '0x%lx'",
vmcb->control.exit_code,
vmcb->control.exit_info_1, vmcb->control.exit_info_2);
--
2.52.0.351.gbe84eed79e-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v2 6/8] KVM: SVM: Limit incorrect check on SVM_EXIT_ERR to running as a VM
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
` (4 preceding siblings ...)
2025-12-30 21:13 ` [PATCH v2 5/8] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM Sean Christopherson
@ 2025-12-30 21:13 ` Sean Christopherson
2025-12-30 21:13 ` [PATCH v2 7/8] KVM: SVM: Harden exit_code against being used in Spectre-like attacks Sean Christopherson
` (2 subsequent siblings)
8 siblings, 0 replies; 14+ messages in thread
From: Sean Christopherson @ 2025-12-30 21:13 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Limit KVM's incorrect check for VMXEXIT_INVALID, a.k.a. SVM_EXIT_ERR, to
running as a VM, as detected by X86_FEATURE_HYPERVISOR. The exit_code and
all failure codes, e.g. VMXEXIT_INVALID, are 64-bit values, and so checking
only bits 31:0 could result in false positives when running on non-broken
hardware, e.g. in the extremely unlikely scenario exit code 0xffffffffull
is ever generated by hardware.
Keep the 32-bit check to play nice with running on broken KVM (for years,
KVM has not set bits 63:32 when synthesizing nested SVM VM-Exits).
Reviewed-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/svm.h | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 105f1394026e..f938679c0231 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -425,7 +425,10 @@ static __always_inline struct vcpu_svm *to_svm(struct kvm_vcpu *vcpu)
static inline bool svm_is_vmrun_failure(u64 exit_code)
{
- return (u32)exit_code == (u32)SVM_EXIT_ERR;
+ if (cpu_feature_enabled(X86_FEATURE_HYPERVISOR))
+ return (u32)exit_code == (u32)SVM_EXIT_ERR;
+
+ return exit_code == SVM_EXIT_ERR;
}
/*
--
2.52.0.351.gbe84eed79e-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v2 7/8] KVM: SVM: Harden exit_code against being used in Spectre-like attacks
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
` (5 preceding siblings ...)
2025-12-30 21:13 ` [PATCH v2 6/8] KVM: SVM: Limit incorrect check on SVM_EXIT_ERR to running as a VM Sean Christopherson
@ 2025-12-30 21:13 ` Sean Christopherson
2025-12-30 21:13 ` [PATCH v2 8/8] KVM: SVM: Assert that Hyper-V's HV_SVM_EXITCODE_ENL == SVM_EXIT_SW Sean Christopherson
2026-01-15 18:03 ` [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
8 siblings, 0 replies; 14+ messages in thread
From: Sean Christopherson @ 2025-12-30 21:13 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Explicitly clamp the exit code used to index KVM's exit handlers to guard
against Spectre-like attacks, mainly to provide consistency between VMX
and SVM (VMX was given the same treatment by commit c926f2f7230b ("KVM:
x86: Protect exit_reason from being used in Spectre-v1/L1TF attacks").
For normal VMs, it's _extremely_ unlikely the exit code could be used to
exploit a speculation vulnerability, as the exit code is set by hardware
and unexpected/unknown exit codes should be quite well bounded (as is/was
the case with VMX). But with SEV-ES+, the exit code is guest-controlled
as it comes from the GHCB, not from hardware, i.e. an attack from the
guest is at least somewhat plausible.
Irrespective of SEV-ES+, hardening KVM is easy and inexpensive, and such
an attack is theoretically possible.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/svm.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index b97e6763839b..a75cd832e194 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3477,6 +3477,7 @@ int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 __exit_code)
if (exit_code >= ARRAY_SIZE(svm_exit_handlers))
goto unexpected_vmexit;
+ exit_code = array_index_nospec(exit_code, ARRAY_SIZE(svm_exit_handlers));
if (!svm_exit_handlers[exit_code])
goto unexpected_vmexit;
--
2.52.0.351.gbe84eed79e-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH v2 8/8] KVM: SVM: Assert that Hyper-V's HV_SVM_EXITCODE_ENL == SVM_EXIT_SW
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
` (6 preceding siblings ...)
2025-12-30 21:13 ` [PATCH v2 7/8] KVM: SVM: Harden exit_code against being used in Spectre-like attacks Sean Christopherson
@ 2025-12-30 21:13 ` Sean Christopherson
2026-01-02 9:58 ` Vitaly Kuznetsov
2026-01-15 18:03 ` [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
8 siblings, 1 reply; 14+ messages in thread
From: Sean Christopherson @ 2025-12-30 21:13 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Add a build-time assertiont that Hyper-V's "enlightened" exit code is that,
same as the AMD-defined "Reserved for Host" exit code, mostly to help
readers connect the dots and understand why synthesizing a software-defined
exit code is safe/ok.
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
arch/x86/kvm/svm/hyperv.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/arch/x86/kvm/svm/hyperv.c b/arch/x86/kvm/svm/hyperv.c
index 3ec580d687f5..4f24dcb45116 100644
--- a/arch/x86/kvm/svm/hyperv.c
+++ b/arch/x86/kvm/svm/hyperv.c
@@ -10,6 +10,12 @@ void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
{
struct vcpu_svm *svm = to_svm(vcpu);
+ /*
+ * The exit code used by Hyper-V for software-defined exits is reserved
+ * by AMD specifically for such use cases.
+ */
+ BUILD_BUG_ON(HV_SVM_EXITCODE_ENL != SVM_EXIT_SW);
+
svm->vmcb->control.exit_code = HV_SVM_EXITCODE_ENL;
svm->vmcb->control.exit_info_1 = HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH;
svm->vmcb->control.exit_info_2 = 0;
--
2.52.0.351.gbe84eed79e-goog
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH v2 8/8] KVM: SVM: Assert that Hyper-V's HV_SVM_EXITCODE_ENL == SVM_EXIT_SW
2025-12-30 21:13 ` [PATCH v2 8/8] KVM: SVM: Assert that Hyper-V's HV_SVM_EXITCODE_ENL == SVM_EXIT_SW Sean Christopherson
@ 2026-01-02 9:58 ` Vitaly Kuznetsov
2026-01-05 15:52 ` Sean Christopherson
0 siblings, 1 reply; 14+ messages in thread
From: Vitaly Kuznetsov @ 2026-01-02 9:58 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, K. Y. Srinivasan,
Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
Sean Christopherson <seanjc@google.com> writes:
> Add a build-time assertiont that Hyper-V's "enlightened" exit code is that,
> same as the AMD-defined "Reserved for Host" exit code, mostly to help
> readers connect the dots and understand why synthesizing a software-defined
> exit code is safe/ok.
>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
> arch/x86/kvm/svm/hyperv.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> diff --git a/arch/x86/kvm/svm/hyperv.c b/arch/x86/kvm/svm/hyperv.c
> index 3ec580d687f5..4f24dcb45116 100644
> --- a/arch/x86/kvm/svm/hyperv.c
> +++ b/arch/x86/kvm/svm/hyperv.c
> @@ -10,6 +10,12 @@ void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
> {
> struct vcpu_svm *svm = to_svm(vcpu);
>
> + /*
> + * The exit code used by Hyper-V for software-defined exits is reserved
> + * by AMD specifically for such use cases.
> + */
> + BUILD_BUG_ON(HV_SVM_EXITCODE_ENL != SVM_EXIT_SW);
> +
> svm->vmcb->control.exit_code = HV_SVM_EXITCODE_ENL;
> svm->vmcb->control.exit_info_1 = HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH;
> svm->vmcb->control.exit_info_2 = 0;
Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
Alternatively (or additionally?) to BUG_ON, I guess we could've
#define HV_SVM_EXITCODE_ENL SVM_EXIT_SW
unless including SVM's headers into include/hyperv/hvgdk.h is too big of
a mess.
--
Vitaly
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH v2 8/8] KVM: SVM: Assert that Hyper-V's HV_SVM_EXITCODE_ENL == SVM_EXIT_SW
2026-01-02 9:58 ` Vitaly Kuznetsov
@ 2026-01-05 15:52 ` Sean Christopherson
0 siblings, 0 replies; 14+ messages in thread
From: Sean Christopherson @ 2026-01-05 15:52 UTC (permalink / raw)
To: Vitaly Kuznetsov
Cc: Paolo Bonzini, K. Y. Srinivasan, Haiyang Zhang, Wei Liu,
Dexuan Cui, Long Li, kvm, linux-hyperv, linux-kernel, Jim Mattson,
Yosry Ahmed
On Fri, Jan 02, 2026, Vitaly Kuznetsov wrote:
> Sean Christopherson <seanjc@google.com> writes:
>
> > Add a build-time assertiont that Hyper-V's "enlightened" exit code is that,
> > same as the AMD-defined "Reserved for Host" exit code, mostly to help
> > readers connect the dots and understand why synthesizing a software-defined
> > exit code is safe/ok.
> >
> > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > ---
> > arch/x86/kvm/svm/hyperv.c | 6 ++++++
> > 1 file changed, 6 insertions(+)
> >
> > diff --git a/arch/x86/kvm/svm/hyperv.c b/arch/x86/kvm/svm/hyperv.c
> > index 3ec580d687f5..4f24dcb45116 100644
> > --- a/arch/x86/kvm/svm/hyperv.c
> > +++ b/arch/x86/kvm/svm/hyperv.c
> > @@ -10,6 +10,12 @@ void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu)
> > {
> > struct vcpu_svm *svm = to_svm(vcpu);
> >
> > + /*
> > + * The exit code used by Hyper-V for software-defined exits is reserved
> > + * by AMD specifically for such use cases.
> > + */
> > + BUILD_BUG_ON(HV_SVM_EXITCODE_ENL != SVM_EXIT_SW);
> > +
> > svm->vmcb->control.exit_code = HV_SVM_EXITCODE_ENL;
> > svm->vmcb->control.exit_info_1 = HV_SVM_ENL_EXITCODE_TRAP_AFTER_FLUSH;
> > svm->vmcb->control.exit_info_2 = 0;
>
> Reviewed-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>
> Alternatively (or additionally?) to BUG_ON, I guess we could've
>
> #define HV_SVM_EXITCODE_ENL SVM_EXIT_SW
>
> unless including SVM's headers into include/hyperv/hvgdk.h is too big of
> a mess.
Heh, I had the same thought[*], but Wei pointed out that the definitions in hvgdk.h
mirror internal Microsoft headers:
On Fri, Nov 14, 2025, Wei Liu wrote:
> On Fri, Nov 14, 2025 at 07:22:41AM -0800, Sean Christopherson wrote:
> > On Fri, Nov 14, 2025, Michael Kelley wrote:
> > > From: Sean Christopherson <seanjc@google.com> Sent: Thursday, November 13, 2025 2:56 PM
> > > > @@ -281,7 +281,7 @@ struct hv_vmcb_enlightenments {
> > > > #define HV_VMCB_NESTED_ENLIGHTENMENTS 31
> > > >
> > > > /* Synthetic VM-Exit */
> > > > -#define HV_SVM_EXITCODE_ENL 0xf0000000
> > > > +#define HV_SVM_EXITCODE_ENL 0xf0000000u
> > >
> > > Is there a reason for making this Hyper-V code just "u", while
> > > making the SVM_VMGEXIT_* values "ull"? I don't think
> > > "u" vs. "ull" shouldn't make any difference when assigning to a
> > > u64, but the inconsistency piqued my interest ....
> >
> > I hedged and went for a more "minimal" change because it isn't KVM code, and at
> > the time because I thought the value isn't defined by the APM. Though looking
> > again at the APM, it does reserve that value for software
> >
> > F000_000h Unused Reserved for Host.
> >
> > and I can't find anything in the TLFS. Ah, my PDF copy is just stale, it's indeed
> > defined as a synthetic exit.
> >
> > https://learn.microsoft.com/en-us/virtualization/hyper-v-on-windows/tlfs/nested-virtualization#synthetic-vm-exit
> >
> > Anyways, I'm in favor of making HV_SVM_EXITCODE_ENL an ull, though part of me
> > wonders if we should do:
> >
> > #define HV_SVM_EXITCODE_ENL SVM_EXIT_SW
>
> I know this is very tempting, but these headers are supposed to mirror
> Microsoft's internal headers, so we would like to keep them
> self-contained for ease of tracking.
>
> It should be fine to add the "ull" suffix here. I briefly talked to a
> hypervisor developer and they agreed.
[*] https://lore.kernel.org/all/aRdJQQ7_j6RcHwjJ@google.com
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs
2025-12-30 21:13 [PATCH v2 0/8] KVM: SVM: Fix exit_code bugs Sean Christopherson
` (7 preceding siblings ...)
2025-12-30 21:13 ` [PATCH v2 8/8] KVM: SVM: Assert that Hyper-V's HV_SVM_EXITCODE_ENL == SVM_EXIT_SW Sean Christopherson
@ 2026-01-15 18:03 ` Sean Christopherson
8 siblings, 0 replies; 14+ messages in thread
From: Sean Christopherson @ 2026-01-15 18:03 UTC (permalink / raw)
To: Sean Christopherson, Paolo Bonzini, Vitaly Kuznetsov,
K. Y. Srinivasan, Haiyang Zhang, Wei Liu, Dexuan Cui, Long Li
Cc: kvm, linux-hyperv, linux-kernel, Jim Mattson, Yosry Ahmed
On Tue, 30 Dec 2025 13:13:39 -0800, Sean Christopherson wrote:
> Fix (mostly benign) bugs in SVM where KVM treats exit codes as 32-bit values
> instead of 64-bit values.
>
> The most dangerous aspect of the mess is that simply fixing KVM would likely
> break KVM-on-KVM setups if only L1 is patched. To try and avoid such
> breakage while also fixing KVM, I opted to have KVM retain its checks on
> only bits 31:0 if KVM is running as a VM (as detected by
> X86_FEATURE_HYPERVISOR).
>
> [...]
Applied to kvm-x86 svm, thanks!
[1/8] KVM: SVM: Add a helper to detect VMRUN failures
https://github.com/kvm-x86/linux/commit/217463aa329e
[2/8] KVM: SVM: Open code handling of unexpected exits in svm_invoke_exit_handler()
https://github.com/kvm-x86/linux/commit/2450c9774510
[3/8] KVM: SVM: Check for an unexpected VM-Exit after RETPOLINE "fast" handling
https://github.com/kvm-x86/linux/commit/194c17bf5eba
[4/8] KVM: SVM: Filter out 64-bit exit codes when invoking exit handlers on bare metal
https://github.com/kvm-x86/linux/commit/405fce694bd1
[5/8] KVM: SVM: Treat exit_code as an unsigned 64-bit value through all of KVM
https://github.com/kvm-x86/linux/commit/d7507a94a072
[6/8] KVM: SVM: Limit incorrect check on SVM_EXIT_ERR to running as a VM
https://github.com/kvm-x86/linux/commit/a08ca6691fd3
[7/8] KVM: SVM: Harden exit_code against being used in Spectre-like attacks
https://github.com/kvm-x86/linux/commit/1e3dddafecee
[8/8] KVM: SVM: Assert that Hyper-V's HV_SVM_EXITCODE_ENL == SVM_EXIT_SW
https://github.com/kvm-x86/linux/commit/d6c20d19f7d3
--
https://github.com/kvm-x86/linux/tree/next
^ permalink raw reply [flat|nested] 14+ messages in thread