From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 640693438A6; Wed, 30 Sep 2026 17:07:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790788073; cv=none; b=BYaYryOkWG7jTOjg6SDXstUXCx4ZeaXsRaEL3UUTfS8ppWuQllbdLNbp5U/1reZ0320oaKR6RYAo5sOXqKVOzEwivki378JIwuRlprhNfgAaouPBYrBXlzOE+FccgOlEa2oC0vPvWcKyjtnW9xlKAAjtkqCWZs04JWK6pb1rlVs= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790788073; c=relaxed/simple; bh=f4p8ewoH6lEqV36MhMjRjvzsn1y4hHoXZY79NySMLCg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qUoGauItvD7W4dD3qRex4n81pBl6veDgdUKzWMtGVlJZRMDVnhhD2C4RlXvD4C5k4k9OWEu8tTKSRb3cXlLOCV0A4wqAbzU5K71j9NRhT8kan4UKmTpCB4ASPljJS4GWiLzza0WooYcW9XTnZrIeosxojDep5vTclU+HO3Iy0Sc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b=HXgsw3kT; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.b="HXgsw3kT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BDA661F000FF; Wed, 30 Sep 2026 17:07:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1790788072; bh=0eHcu1a3Ypho96ZyNHU7GmvkWDsmC+Zw6j7K6ZV+n1E=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=HXgsw3kTMoHS4Gxwg6GTPONo+J403O4MkCwUfwiahs4zcyPvyu07KOJ5Nw7sKDOuf cW2qy5WXEmqUdgt5sIefpTE0G8NIdXElDdCLh/w6x71DuyycxpbYIBAzs5YUE5uCig bmagxbSTdcuKgt7IszvanHhurZIS6ais9gcjtaJI= From: Greg Kroah-Hartman To: stable@vger.kernel.org Cc: Greg Kroah-Hartman , patches@lists.linux.dev, Sean Christopherson , Sasha Levin Subject: [PATCH 7.2 455/457] KVM: x86: Add static calls for nested virtualization ops Date: Wed, 30 Sep 2026 17:29:20 +0200 Message-ID: <20260930152355.827648068@linuxfoundation.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260930152346.024115587@linuxfoundation.org> References: <20260930152346.024115587@linuxfoundation.org> User-Agent: quilt/0.69 X-stable: review X-Patchwork-Hint: ignore Precedence: bulk X-Mailing-List: patches@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit 7.2-stable review patch. If anyone has any objections, please let me know. ------------------ From: Sean Christopherson [ Upstream commit 4b9819a50674dd40d176033f9ea2e23a70889dc6 ] Use static calls to invoke nested virtualization ops, as many of the calls are in relatively hot paths when L2 is active, e.g. checking for events, and because there's no reason not use static calls these days. Opportunistically use a RET0 static call for get_evmcs_version() instead of manually checking for a non-NULL vendor hook. Link: https://patch.msgid.link/20260630202828.440724-3-seanjc@google.com Signed-off-by: Sean Christopherson Backport notes: Preserve the stable tree's kvm_x86_vendor_init()/exit() declarations and ignore_msrs/report_ignored_msrs module parameters at the conflicting insertion points. Initialize nested static calls in the existing kvm_ops_update() instead of adding kvm_nested_ops_update(), preserving the same mandatory, optional, and return-zero hook behavior without adding a function. Keep the get_nested_state_pages call conversion so target 10180a277549339020b08000206092c07e0bff5a applies unchanged. Stable-dep-of: 10180a277549 ("KVM: x86: Re-pend GET_NESTED_STATE_PAGES if getting said pages fails") Signed-off-by: Sasha Levin Signed-off-by: Greg Kroah-Hartman --- arch/x86/include/asm/kvm-x86-nested-ops.h | 36 +++++++++++++++++++++ arch/x86/include/asm/kvm_host.h | 8 ++++ arch/x86/kvm/hyperv.c | 7 +--- arch/x86/kvm/mmu.h | 5 +-- arch/x86/kvm/mmu/paging_tmpl.h | 2 - arch/x86/kvm/x86.c | 50 +++++++++++++++++++----------- arch/x86/kvm/x86.h | 2 - 7 files changed, 83 insertions(+), 27 deletions(-) create mode 100644 arch/x86/include/asm/kvm-x86-nested-ops.h --- /dev/null +++ b/arch/x86/include/asm/kvm-x86-nested-ops.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(KVM_X86_NESTED_OP) || \ + !defined(KVM_X86_NESTED_OP_OPTIONAL) || \ + !defined(KVM_X86_NESTED_OP_OPTIONAL_RET0) +#error Missing one or more KVM_X86_NESTED_OP #defines +#else +/* + * KVM_X86_NESTED_OP() and KVM_X86_NESTED_OP_OPTIONAL() are used to help + * generate both DECLARE/DEFINE_STATIC_CALL() invocations and + * "static_call_update()" calls. + * + * KVM_X86_NESTED_OP_OPTIONAL() can be used for those functions that can have + * a NULL definition. KVM_X86_NESTED_OP_OPTIONAL_RET0() can be used likewise + * to make a definition optional, but in this case the default will + * be __static_call_return0. + */ +KVM_X86_NESTED_OP(leave_nested) +KVM_X86_NESTED_OP(is_exception_vmexit) +KVM_X86_NESTED_OP(check_events) +KVM_X86_NESTED_OP_OPTIONAL_RET0(has_events) +KVM_X86_NESTED_OP(triple_fault) +KVM_X86_NESTED_OP(get_state) +KVM_X86_NESTED_OP(set_state) +KVM_X86_NESTED_OP(get_nested_state_pages) +KVM_X86_NESTED_OP_OPTIONAL_RET0(write_log_dirty) +KVM_X86_NESTED_OP(translate_nested_gpa) +#ifdef CONFIG_KVM_HYPERV +KVM_X86_NESTED_OP_OPTIONAL(enable_evmcs) +KVM_X86_NESTED_OP_OPTIONAL_RET0(get_evmcs_version) +KVM_X86_NESTED_OP(hv_inject_synthetic_vmexit_post_tlb_flush) +#endif +#endif + +#undef KVM_X86_NESTED_OP +#undef KVM_X86_NESTED_OP_OPTIONAL +#undef KVM_X86_NESTED_OP_OPTIONAL_RET0 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2077,6 +2077,14 @@ extern struct kvm_x86_ops kvm_x86_ops; #define KVM_X86_OP_OPTIONAL_RET0 KVM_X86_OP #include +#define kvm_nested_call(func) static_call(kvm_x86_nested_##func) + +#define KVM_X86_NESTED_OP(func) \ + DECLARE_STATIC_CALL(kvm_x86_nested_##func, *(((struct kvm_x86_nested_ops *)0)->func)); +#define KVM_X86_NESTED_OP_OPTIONAL KVM_X86_NESTED_OP +#define KVM_X86_NESTED_OP_OPTIONAL_RET0 KVM_X86_NESTED_OP +#include + int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops); void kvm_x86_vendor_exit(void); --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -2419,7 +2419,7 @@ static int kvm_hv_hypercall_complete(str ret = kvm_skip_emulated_instruction(vcpu); if (tlb_lock_count) - kvm_x86_ops.nested_ops->hv_inject_synthetic_vmexit_post_tlb_flush(vcpu); + kvm_nested_call(hv_inject_synthetic_vmexit_post_tlb_flush)(vcpu); return ret; } @@ -2800,9 +2800,8 @@ int kvm_get_hv_cpuid(struct kvm_vcpu *vc }; int i, nent = ARRAY_SIZE(cpuid_entries); - if (kvm_x86_ops.nested_ops->enabled && - kvm_x86_ops.nested_ops->get_evmcs_version) - evmcs_ver = kvm_x86_ops.nested_ops->get_evmcs_version(vcpu); + if (kvm_x86_ops.nested_ops->enabled) + evmcs_ver = kvm_nested_call(get_evmcs_version)(vcpu); if (cpuid->nent < nent) return -E2BIG; --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -308,9 +308,8 @@ static inline gpa_t kvm_translate_gpa(st { if (mmu != &vcpu->arch.nested_mmu) return gpa; - return kvm_x86_ops.nested_ops->translate_nested_gpa(vcpu, gpa, access, - exception, - pte_access); + return kvm_nested_call(translate_nested_gpa)(vcpu, gpa, access, + exception, pte_access); } static inline bool kvm_has_mirrored_tdp(const struct kvm *kvm) --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -233,7 +233,7 @@ static int FNAME(update_accessed_dirty_b !(pte & PT_GUEST_DIRTY_MASK)) { trace_kvm_mmu_set_dirty_bit(table_gfn, index, sizeof(pte)); #if PTTYPE == PTTYPE_EPT - if (kvm_x86_ops.nested_ops->write_log_dirty(vcpu, addr)) + if (kvm_nested_call(write_log_dirty)(vcpu, addr)) return -EINVAL; #endif pte |= PT_GUEST_DIRTY_MASK; --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -148,6 +148,13 @@ EXPORT_STATIC_CALL_GPL(kvm_x86_get_cs_db EXPORT_STATIC_CALL_GPL(kvm_x86_cache_reg); EXPORT_STATIC_CALL_GPL(kvm_x86_get_cpl); +#define KVM_X86_NESTED_OP(func) \ + DEFINE_STATIC_CALL_NULL(kvm_x86_nested_##func, \ + *(((struct kvm_x86_nested_ops *)0)->func)); +#define KVM_X86_NESTED_OP_OPTIONAL KVM_X86_NESTED_OP +#define KVM_X86_NESTED_OP_OPTIONAL_RET0 KVM_X86_NESTED_OP +#include + static bool __read_mostly ignore_msrs = 0; module_param(ignore_msrs, bool, 0644); @@ -833,7 +840,7 @@ static void kvm_multiple_exception(struc * wants to intercept the exception. */ if (is_guest_mode(vcpu) && - kvm_x86_ops.nested_ops->is_exception_vmexit(vcpu, nr, error_code)) { + kvm_nested_call(is_exception_vmexit)(vcpu, nr, error_code)) { kvm_queue_exception_vmexit(vcpu, nr, has_error, error_code, has_payload, payload); return; @@ -4518,7 +4525,7 @@ int kvm_vm_ioctl_check_extension(struct break; case KVM_CAP_NESTED_STATE: r = kvm_x86_ops.nested_ops->enabled ? - kvm_x86_ops.nested_ops->get_state(NULL, NULL, 0) : 0; + kvm_nested_call(get_state)(NULL, NULL, 0) : 0; break; #ifdef CONFIG_KVM_HYPERV case KVM_CAP_HYPERV_DIRECT_TLBFLUSH: @@ -5571,7 +5578,7 @@ static int kvm_vcpu_ioctl_enable_cap(str if (!kvm_x86_ops.nested_ops->enabled || !kvm_x86_ops.nested_ops->enable_evmcs) return -ENOTTY; - r = kvm_x86_ops.nested_ops->enable_evmcs(vcpu, &vmcs_version); + r = kvm_nested_call(enable_evmcs)(vcpu, &vmcs_version); if (!r) { user_ptr = (void __user *)(uintptr_t)cap->args[0]; if (copy_to_user(user_ptr, &vmcs_version, @@ -6077,8 +6084,7 @@ long kvm_arch_vcpu_ioctl(struct file *fi if (get_user(user_data_size, &user_kvm_nested_state->size)) break; - r = kvm_x86_ops.nested_ops->get_state(vcpu, user_kvm_nested_state, - user_data_size); + r = kvm_nested_call(get_state)(vcpu, user_kvm_nested_state, user_data_size); if (r < 0) break; @@ -6122,7 +6128,7 @@ long kvm_arch_vcpu_ioctl(struct file *fi break; idx = srcu_read_lock(&vcpu->kvm->srcu); - r = kvm_x86_ops.nested_ops->set_state(vcpu, user_kvm_nested_state, &kvm_state); + r = kvm_nested_call(set_state)(vcpu, user_kvm_nested_state, &kvm_state); srcu_read_unlock(&vcpu->kvm->srcu, idx); break; } @@ -9560,6 +9566,8 @@ static void kvm_setup_efer_caps(void) static inline void kvm_ops_update(struct kvm_x86_init_ops *ops) { + const struct kvm_x86_nested_ops *nested_ops = ops->runtime_ops->nested_ops; + memcpy(&kvm_x86_ops, ops->runtime_ops, sizeof(kvm_x86_ops)); #define __KVM_X86_OP(func) \ @@ -9573,6 +9581,17 @@ static inline void kvm_ops_update(struct #include #undef __KVM_X86_OP +#define __KVM_X86_NESTED_OP(func) \ + static_call_update(kvm_x86_nested_##func, nested_ops->func); +#define KVM_X86_NESTED_OP(func) \ + WARN_ON(!nested_ops->func); __KVM_X86_NESTED_OP(func) +#define KVM_X86_NESTED_OP_OPTIONAL __KVM_X86_NESTED_OP +#define KVM_X86_NESTED_OP_OPTIONAL_RET0(func) \ + static_call_update(kvm_x86_nested_##func, (void *)nested_ops->func ? : \ + (void *)__static_call_return0); +#include +#undef __KVM_X86_NESTED_OP + kvm_pmu_ops_update(ops->pmu_ops); } @@ -10109,11 +10128,11 @@ static void post_kvm_run_save(struct kvm int kvm_check_nested_events(struct kvm_vcpu *vcpu) { if (kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu)) { - kvm_x86_ops.nested_ops->triple_fault(vcpu); + kvm_nested_call(triple_fault)(vcpu); return 1; } - return kvm_x86_ops.nested_ops->check_events(vcpu); + return kvm_nested_call(check_events)(vcpu); } static void kvm_inject_exception(struct kvm_vcpu *vcpu) @@ -10351,9 +10370,7 @@ static int kvm_check_and_inject_events(s kvm_x86_call(enable_irq_window)(vcpu); } - if (is_guest_mode(vcpu) && - kvm_x86_ops.nested_ops->has_events && - kvm_x86_ops.nested_ops->has_events(vcpu, true)) + if (is_guest_mode(vcpu) && kvm_nested_call(has_events)(vcpu, true)) *req_immediate_exit = true; /* @@ -10676,7 +10693,7 @@ static int vcpu_enter_guest(struct kvm_v } if (kvm_check_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu)) { - if (unlikely(!kvm_x86_ops.nested_ops->get_nested_state_pages(vcpu))) { + if (unlikely(!kvm_nested_call(get_nested_state_pages)(vcpu))) { r = 0; goto out; } @@ -10728,7 +10745,7 @@ static int vcpu_enter_guest(struct kvm_v } if (kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu)) { if (is_guest_mode(vcpu)) - kvm_x86_ops.nested_ops->triple_fault(vcpu); + kvm_nested_call(triple_fault)(vcpu); if (kvm_check_request(KVM_REQ_TRIPLE_FAULT, vcpu)) { vcpu->run->exit_reason = KVM_EXIT_SHUTDOWN; @@ -11148,9 +11165,7 @@ bool kvm_vcpu_has_events(struct kvm_vcpu if (kvm_hv_has_stimer_pending(vcpu)) return true; - if (is_guest_mode(vcpu) && - kvm_x86_ops.nested_ops->has_events && - kvm_x86_ops.nested_ops->has_events(vcpu, false)) + if (is_guest_mode(vcpu) && kvm_nested_call(has_events)(vcpu, false)) return true; if (kvm_xen_has_pending_events(vcpu)) @@ -11575,8 +11590,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_v * a pending VM-Exit if L1 wants to intercept the exception. */ if (vcpu->arch.exception_from_userspace && is_guest_mode(vcpu) && - kvm_x86_ops.nested_ops->is_exception_vmexit(vcpu, ex->vector, - ex->error_code)) { + kvm_nested_call(is_exception_vmexit)(vcpu, ex->vector, ex->error_code)) { kvm_queue_exception_vmexit(vcpu, ex->vector, ex->has_error_code, ex->error_code, ex->has_payload, ex->payload); --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -151,7 +151,7 @@ int kvm_check_nested_events(struct kvm_v /* Forcibly leave the nested mode in cases like a vCPU reset */ static inline void kvm_leave_nested(struct kvm_vcpu *vcpu) { - kvm_x86_ops.nested_ops->leave_nested(vcpu); + kvm_nested_call(leave_nested)(vcpu); } /*