[RFC PATCH v3 00/27] KVM: SVM: Add support for SEV-SNP Secure AVIC

[RFC PATCH v3 00/27] KVM: SVM: Add support for SEV-SNP Secure AVIC

[Naveen N Rao (AMD)]

This is RFC v3 of the patches adding KVM support for Secure AVIC. RFC v2:
http://lore.kernel.org/r/20250923050317.205482-1-Neeraj.Upadhyay@amd.com

There are significant changes throughout the series since the last RFC, 
with some of the highlights being:
- Integrate with AVIC code since Secure AVIC is APICv, but for protected 
  APIC guests. This aligns with TDX which also requires APICv and 
  enables reuse of APICv code paths in KVM. Patches #8 to #14.
- Mandate use of split irqchip, and drop changes related to the KVM 
  IOAPIC. KVM PIC/PIT can't be supported (described more fully in the 
  commit log of the relevant patches). Patch #24.
- Enable use of SVM_AVIC_INCOMPLETE_IPI VMGEXIT for vCPU notification 
  request from the guest. Patch #22.
- Utilize vNMI injection path for NMI handling. Patch #23.
- Introduce a new kvm_x86_op to distinguish need for injectable 
  interrupts vs. IPI notifications. Patches #18 and #19.
- Retain KVM PV ops, and instead require guests to opt-in/disable.  
  Patch #3.

Patches #1 to #4 are changes to the guest x2APIC Secure AVIC driver.

These patches apply on top of Sean's recent fixes changing AVIC init 
code (since I was carrying a similar patch):
http://lore.kernel.org/r/20260630210156.457151-1-seanjc@google.com

There are still some rough edges to iron out, but I wanted to get some 
feedback on the overall direction. Please let me know if you see any 
major red flags.


Thanks,
- Naveen


Kishon Vijay Abraham I (2):
  KVM: SVM: Set VGIF in VMSA area for Secure AVIC guests
  KVM: SVM: Do not inject exceptions for Secure AVIC

Naveen N Rao (AMD) (20):
  x86/apic: Drop savic_eoi() in favor of native_apic_msr_eoi() for
    Secure AVIC
  x86/kvm: Disable PV_SEND_IPI if Secure AVIC is enabled
  x86/apic: Use AVIC_INCOMPLETE_IPI VMGEXIT for Secure AVIC IPI handling
  KVM: SVM: Add helper to check if Secure AVIC is enabled for a guest
  KVM: SVM: Set guest_apic_protected if Secure AVIC is enabled
  kvm: irqfd: Have kvm_arch_has_irq_bypass() take struct kvm pointer
  KVM: SVM: Disable IRQ bypass for Secure AVIC
  KVM: SVM: Add avic_ipiv_is_soft_disabled() as a wrapper around
    enable_ipiv
  KVM: SVM: Disable IPIv for Secure AVIC
  KVM: SVM: Short-circuit a few AVIC flows for Secure AVIC
  KVM: SVM: Warn if we ever receive AVIC_UNACCELERATED_ACCESS #VMEXIT
  KVM: SVM: Do not inhibit AVIC for SEV-SNP guests if Secure AVIC is
    enabled
  KVM: x86: Add a new kvm_x86_op protected_apic_has_injectable_intr()
  KVM: SVM: Implement kvm_x86_ops->protected_apic_has_injectable_intr()
    for Secure AVIC
  KVM: SVM: Implement kvm_x86_ops->protected_apic_has_interrupt() for
    Secure AVIC
  KVM: SVM: Add interrupt delivery support for Secure AVIC guests
  KVM: SVM: Add support for incomplete IPI handling for Secure AVIC
  KVM: SVM: Add support for injecting NMIs for Secure AVIC guests
  KVM: SVM: Mandate use of split irqchip for Secure AVIC
  KVM: SVM: Advertise Secure AVIC support for SEV-SNP guests

Neeraj Upadhyay (5):
  x86/apic: Propagate APIC_SPIV writes to hv for Secure AVIC
  x86/cpufeatures: Add Secure AVIC CPU feature
  KVM: SVM: Add handler for VMGEXIT Secure AVIC NAE event
  KVM: SVM: Do not intercept SECURE_AVIC_CONTROL MSR for Secure AVIC
    guests
  KVM: SVM: Do not intercept exceptions for Secure AVIC guests

 arch/arm64/include/asm/kvm_host.h   |   2 +-
 arch/x86/include/asm/cpufeatures.h  |   1 +
 arch/x86/include/asm/kvm-x86-ops.h  |   1 +
 arch/x86/include/asm/kvm_host.h     |   9 +-
 arch/x86/include/asm/svm.h          |   9 +-
 arch/x86/include/uapi/asm/svm.h     |   1 +
 arch/x86/kvm/svm/svm.h              |  12 ++
 include/linux/kvm_host.h            |   2 +-
 arch/powerpc/kvm/powerpc.c          |   2 +-
 arch/x86/coco/sev/core.c            |  18 ++-
 arch/x86/kernel/apic/x2apic_savic.c |  36 +-----
 arch/x86/kernel/kvm.c               |   3 +-
 arch/x86/kvm/irq.c                  |   5 +-
 arch/x86/kvm/svm/avic.c             | 105 ++++++++++++++++--
 arch/x86/kvm/svm/sev.c              | 164 +++++++++++++++++++++++++++-
 arch/x86/kvm/svm/svm.c              |  59 ++++++++--
 arch/x86/kvm/vmx/posted_intr.c      |   2 +-
 arch/x86/kvm/x86.c                  |  21 +++-
 virt/kvm/eventfd.c                  |   2 +-
 19 files changed, 382 insertions(+), 72 deletions(-)


base-commit: 50406d35f5635e1cc523e61409d57e851b5f5df8
prerequisite-patch-id: 1428b4903537b426ee3822a74adf29fbbbeb8e02
prerequisite-patch-id: c88b0c2cb26fb85fc7ef07409aeaf643565b8600
prerequisite-patch-id: a3f8fc507b7809645b001e9394a51f8b299aae47
-- 
2.54.0

[RFC PATCH v3 01/27] x86/apic: Propagate APIC_SPIV writes to hv for Secure AVIC

[Naveen N Rao (AMD)]

From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>

Hypervisor needs to be aware of the VAPIC's APIC_SPIV state for
LVT masking/unmasking. So, propagate the guest's APIC_SPIV
writes to the hypervisor.

Fixes: c822f58a4fab ("x86/apic: Populate .read()/.write() callbacks of Secure AVIC driver")
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
---
 arch/x86/kernel/apic/x2apic_savic.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/apic/x2apic_savic.c b/arch/x86/kernel/apic/x2apic_savic.c
index dbc5678bc3b6..cb3b55225ab6 100644
--- a/arch/x86/kernel/apic/x2apic_savic.c
+++ b/arch/x86/kernel/apic/x2apic_savic.c
@@ -214,7 +214,6 @@ static void savic_write(u32 reg, u32 data)
 		break;
 	case APIC_TASKPRI:
 	case APIC_EOI:
-	case APIC_SPIV:
 	case SAVIC_NMI_REQ:
 	case APIC_ESR:
 	case APIC_ECTRL:
@@ -223,6 +222,10 @@ static void savic_write(u32 reg, u32 data)
 	case APIC_EILVTn(0) ... APIC_EILVTn(3):
 		apic_set_reg(ap, reg, data);
 		break;
+	case APIC_SPIV:
+		apic_set_reg(ap, reg, data);
+		savic_ghcb_msr_write(reg, data);
+		break;
 	case APIC_ICR:
 		savic_icr_write(data, 0);
 		break;
-- 
2.54.0

[RFC PATCH v3 02/27] x86/apic: Drop savic_eoi() in favor of native_apic_msr_eoi() for Secure AVIC

[Naveen N Rao (AMD)]

Drop savic_eoi() in favor of using the native helper that writes to the
APIC_EOI MSR. savic_eoi() was added mainly to be able to handle
level-triggered interrupts. However, it relies on APIC_TMR indicating a
vector to be level-triggered, but APIC_TMR can never have a bit set
since it is only updated when the LAPIC accepts a level-triggered
interrupt. In the case of a Secure AVIC SEV-SNP guest, all
level-triggered interrupt sources are in the VMM (emulated IOAPIC
primarily) and KVM accepts them on behalf of the guest resulting in the
APIC_TMR in KVM APIC backing page having a bit set. This is never seen
by the guest, which has its own private APIC backing page. As such, the
savic_eoi() handler is dead code. Remove it.

Fixes: 43b6687ac877 ("x86/apic: Handle EOI writes for Secure AVIC guests")
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kernel/apic/x2apic_savic.c | 31 +----------------------------
 1 file changed, 1 insertion(+), 30 deletions(-)

diff --git a/arch/x86/kernel/apic/x2apic_savic.c b/arch/x86/kernel/apic/x2apic_savic.c
index cb3b55225ab6..f2af149c6005 100644
--- a/arch/x86/kernel/apic/x2apic_savic.c
+++ b/arch/x86/kernel/apic/x2apic_savic.c
@@ -304,35 +304,6 @@ static void savic_update_vector(unsigned int cpu, unsigned int vector, bool set)
 	update_vector(cpu, SAVIC_ALLOWED_IRR, vector, set);
 }
 
-static void savic_eoi(void)
-{
-	unsigned int cpu;
-	int vec;
-
-	cpu = raw_smp_processor_id();
-	vec = apic_find_highest_vector(get_reg_bitmap(cpu, APIC_ISR));
-	if (WARN_ONCE(vec == -1, "EOI write while no active interrupt in APIC_ISR"))
-		return;
-
-	/* Is level-triggered interrupt? */
-	if (apic_test_vector(vec, get_reg_bitmap(cpu, APIC_TMR))) {
-		update_vector(cpu, APIC_ISR, vec, false);
-		/*
-		 * Propagate the EOI write to the hypervisor for level-triggered
-		 * interrupts. Return to the guest from GHCB protocol event takes
-		 * care of re-evaluating interrupt state.
-		 */
-		savic_ghcb_msr_write(APIC_EOI, 0);
-	} else {
-		/*
-		 * Hardware clears APIC_ISR and re-evaluates the interrupt state
-		 * to determine if there is any pending interrupt which can be
-		 * delivered to CPU.
-		 */
-		native_apic_msr_eoi();
-	}
-}
-
 static void savic_teardown(void)
 {
 	/* Disable Secure AVIC */
@@ -421,7 +392,7 @@ static struct apic apic_x2apic_savic __ro_after_init = {
 
 	.read				= savic_read,
 	.write				= savic_write,
-	.eoi				= savic_eoi,
+	.eoi				= native_apic_msr_eoi,
 	.icr_read			= native_x2apic_icr_read,
 	.icr_write			= savic_icr_write,
 
-- 
2.54.0

[RFC PATCH v3 03/27] x86/kvm: Disable PV_SEND_IPI if Secure AVIC is enabled

[Naveen N Rao (AMD)]

In Secure AVIC mode, Linux does not allow IPIs to be injected as the
hypervisor is untrusted. This is achieved by not setting the IPI vector
in SAVIC_ALLOWED_IRR in the Secure AVIC guest APIC Backing page. Due to
this, PV_SEND_IPI cannot work since it needs KVM to be able to inject
IPIs into the guest. Disable the same.

On a related note, PV_EOI does not need to be disabled since the
behavior is exactly the same as AVIC: though it is advertised, KVM never
"enables" it since APICv is always enabled for Secure AVIC SEV-SNP
guests.

Fixes: c4074ab87f34 ("x86/apic: Enable Secure AVIC in the control MSR")
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kernel/kvm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c
index 29226d112029..e2dad507f2a4 100644
--- a/arch/x86/kernel/kvm.c
+++ b/arch/x86/kernel/kvm.c
@@ -495,7 +495,8 @@ static bool pv_tlb_flush_supported(void)
 static bool pv_ipi_supported(void)
 {
 	return (kvm_para_has_feature(KVM_FEATURE_PV_SEND_IPI) &&
-	       (num_possible_cpus() != 1));
+		(num_possible_cpus() != 1) &&
+		!cc_platform_has(CC_ATTR_SNP_SECURE_AVIC));
 }
 
 static bool pv_sched_yield_supported(void)
-- 
2.54.0

[RFC PATCH v3 04/27] x86/apic: Use AVIC_INCOMPLETE_IPI VMGEXIT for Secure AVIC IPI handling

[Naveen N Rao (AMD)]

x2apic_savic driver currently uses SVM_EXIT_MSR to have the hypervisor
emulate ICR writes and to be able to deliver IPIs. However, the driver
also sets APIC_IRR in the APIC backing page of the target vCPU, so the
expectation is only for the hypervisor to notify/wake up the target
vCPU.

This is incorrect since SVM_EXIT_MSR is for requesting full emulation of
a certain MSR access -- and hypervisors expect to be able to _inject_
(set APIC_IRR) *and* deliver the interrupt to the target. This can
result in duplicate interrupts at the guest (assuming the guest is
allowing that vector to be injected by the hypervisor).

Instead, have the driver do AVIC_INCOMPLETE_IPI exit so that it is clear
on what the hypervisor needs to do. This is the same exit used by SVM
AVIC when it has already set APIC_IRR in the target vCPU APIC backing
page to request the hypervisor to send an AVIC doorbell/wake up the
target vCPU. Add the newly added exit ID AVIC_IPI_FAILURE_UNACCELERATED
(ID number 5) and use the same to signal that this is a Secure AVIC
access that is not accelerated by hardware.

Fixes: 2c6978ea1a85 ("x86/apic: Add support to send IPI for Secure AVIC")
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/include/asm/svm.h |  1 +
 arch/x86/coco/sev/core.c   | 18 +++++++++++++++++-
 2 files changed, 18 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index aa63431ba92c..42ececa8963d 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -284,6 +284,7 @@ enum avic_ipi_failure_cause {
 	AVIC_IPI_FAILURE_INVALID_TARGET,
 	AVIC_IPI_FAILURE_INVALID_BACKING_PAGE,
 	AVIC_IPI_FAILURE_INVALID_IPI_VECTOR,
+	AVIC_IPI_FAILURE_UNACCELERATED,
 };
 
 #define AVIC_PHYSICAL_MAX_INDEX_MASK	GENMASK_ULL(11, 0)
diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c
index 7ed3da998489..69c75ea543a3 100644
--- a/arch/x86/coco/sev/core.c
+++ b/arch/x86/coco/sev/core.c
@@ -1014,13 +1014,29 @@ void savic_ghcb_msr_write(u32 reg, u64 value)
 	struct ghcb_state state;
 	enum es_result res;
 	struct ghcb *ghcb;
+	u64 exit_info_2;
 
 	guard(irqsave)();
 
 	ghcb = __sev_get_ghcb(&state);
 	vc_ghcb_invalidate(ghcb);
 
-	res = __vc_handle_msr(ghcb, &ctxt, true);
+	if (reg == APIC_ICR) {
+		/*
+		 * Exit with AVIC_INCOMPLETE_IPI to request hypervisor to notify
+		 * the target vCPU(s). exit_info_1 is just the ICR value.
+		 * exit_info_2 encodes exit id in the upper 32-bits, and icrh
+		 * (IPI dest, since Secure AVIC is x2APIC-only) in the lower 32-bits.
+		 */
+		exit_info_2 = (u64)(AVIC_IPI_FAILURE_UNACCELERATED) << 32;
+		exit_info_2 |= upper_32_bits(value);
+
+		res = sev_es_ghcb_hv_call(ghcb, &ctxt, SVM_EXIT_AVIC_INCOMPLETE_IPI,
+					  value, exit_info_2);
+	} else {
+		res = __vc_handle_msr(ghcb, &ctxt, true);
+	}
+
 	if (res != ES_OK) {
 		pr_err("Secure AVIC MSR (0x%llx) write returned error (%d)\n", msr, res);
 		/* MSR writes should never fail. Any failure is fatal error for SNP guest */
-- 
2.54.0

[RFC PATCH v3 05/27] x86/cpufeatures: Add Secure AVIC CPU feature

[Naveen N Rao (AMD)]

From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>

Add CPU feature detection for Secure AVIC. The Secure AVIC feature
provides hardware acceleration for performance sensitive APIC accesses
and support for managing guest-owned APIC state for the SEV-SNP guests.

Co-developed-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Signed-off-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
---
 arch/x86/include/asm/cpufeatures.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index 35a2a0f9ab32..d538db618f24 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -457,6 +457,7 @@
 #define X86_FEATURE_DEBUG_SWAP		(19*32+14) /* "debug_swap" SEV-ES full debug state swap support */
 #define X86_FEATURE_RMPREAD		(19*32+21) /* RMPREAD instruction */
 #define X86_FEATURE_SEGMENTED_RMP	(19*32+23) /* Segmented RMP support */
+#define X86_FEATURE_SNP_SECURE_AVIC	(19*32+26) /* SEV-SNP Secure AVIC */
 #define X86_FEATURE_ALLOWED_SEV_FEATURES (19*32+27) /* Allowed SEV Features */
 #define X86_FEATURE_SVSM		(19*32+28) /* "svsm" SVSM present */
 #define X86_FEATURE_HV_INUSE_WR_ALLOWED	(19*32+30) /* Allow Write to in-use hypervisor-owned pages */
-- 
2.54.0

[RFC PATCH v3 06/27] KVM: SVM: Add helper to check if Secure AVIC is enabled for a guest

[Naveen N Rao (AMD)]

Add a helper snp_is_secure_avic_enabled() along the lines of the similar
helper for Secure TSC to check if Secure AVIC is enabled in the VMSA SEV
Features for a SEV-SNP guest.

Note: Enabling Secure AVIC in the VMSA SEV Features puts the SEV-SNP
guest in Secure AVIC "mode", and it is up to the guest to then "enable"
Secure AVIC through the Secure AVIC Control MSR. While the use of
"enabled" in the helper might sound like a misnomer, it reflects the
fact that KVM has no visibility into whether the guest has actually
enabled Secure AVIC or not. For all practical purposes, KVM assumes that
the guest has Secure AVIC enabled. Since a Secure AVIC mode guest is
extremely restricted, it is in the best interest of the guest to enable
Secure AVIC at the earliest.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/include/asm/svm.h | 1 +
 arch/x86/kvm/svm/svm.h     | 2 ++
 arch/x86/kvm/svm/sev.c     | 8 ++++++++
 3 files changed, 11 insertions(+)

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 42ececa8963d..5857b942957b 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -311,6 +311,7 @@ static_assert((X2AVIC_4K_MAX_PHYSICAL_ID & AVIC_PHYSICAL_MAX_INDEX_MASK) == X2AV
 #define SVM_SEV_FEAT_ALTERNATE_INJECTION		BIT(4)
 #define SVM_SEV_FEAT_DEBUG_SWAP				BIT(5)
 #define SVM_SEV_FEAT_SECURE_TSC				BIT(9)
+#define SVM_SEV_FEAT_SECURE_AVIC			BIT(16)
 
 #define VMCB_ALLOWED_SEV_FEATURES_VALID			BIT_ULL(63)
 
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 616e45624f4c..1157d022bac1 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1013,6 +1013,7 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
 void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa);
+bool snp_is_secure_avic_enabled(struct kvm *kvm);
 #else
 static inline struct page *snp_safe_alloc_page_node(int node, gfp_t gfp)
 {
@@ -1050,6 +1051,7 @@ static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu)
 	return NULL;
 }
 static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa) {}
+static inline bool snp_is_secure_avic_enabled(struct kvm *kvm) { return false; }
 #endif
 
 /* vmenter.S */
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 74fb15551e83..686227a15328 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -211,6 +211,14 @@ static bool snp_is_secure_tsc_enabled(struct kvm *kvm)
 	       !WARN_ON_ONCE(!sev_snp_guest(kvm));
 }
 
+bool snp_is_secure_avic_enabled(struct kvm *kvm)
+{
+	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
+
+	return (sev->vmsa_features & SVM_SEV_FEAT_SECURE_AVIC) &&
+	       !WARN_ON_ONCE(!sev_snp_guest(kvm));
+}
+
 /* Must be called with the sev_bitmap_lock held */
 static bool __sev_recycle_asids(unsigned int min_asid, unsigned int max_asid)
 {
-- 
2.54.0

[RFC PATCH v3 07/27] KVM: SVM: Set guest_apic_protected if Secure AVIC is enabled

[Naveen N Rao (AMD)]

Mark the guest APIC as protected if Secure AVIC is enabled.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/avic.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index c5b1d294b15a..7340c9e251e0 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -884,6 +884,9 @@ int avic_init_vcpu(struct vcpu_svm *svm)
 	if (!enable_apicv || !irqchip_in_kernel(vcpu->kvm))
 		return 0;
 
+	if (snp_is_secure_avic_enabled(vcpu->kvm))
+		vcpu->arch.apic->guest_apic_protected = true;
+
 	ret = avic_init_backing_page(vcpu);
 	if (ret)
 		return ret;
-- 
2.54.0

[RFC PATCH v3 08/27] kvm: irqfd: Have kvm_arch_has_irq_bypass() take struct kvm pointer

[Naveen N Rao (AMD)]

In preparation for having kvm_arch_has_irq_bypass() take the guest
configuration into account, have the helper take struct kvm pointer as
an argument.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/arm64/include/asm/kvm_host.h | 2 +-
 arch/x86/include/asm/kvm_host.h   | 2 +-
 include/linux/kvm_host.h          | 2 +-
 arch/powerpc/kvm/powerpc.c        | 2 +-
 arch/x86/kvm/irq.c                | 2 +-
 arch/x86/kvm/vmx/posted_intr.c    | 2 +-
 virt/kvm/eventfd.c                | 2 +-
 7 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index cb5ef7e6c2fe..1eee05003ea1 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -1669,7 +1669,7 @@ void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val);
 #define kvm_has_sctlr2(k)				\
 	(kvm_has_feat((k), ID_AA64MMFR3_EL1, SCTLRX, IMP))
 
-static inline bool kvm_arch_has_irq_bypass(void)
+static inline bool kvm_arch_has_irq_bypass(struct kvm *kvm)
 {
 	return true;
 }
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index d8700eb848b4..d81179b43199 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -2064,7 +2064,7 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu)
 	kvm_x86_call(vcpu_unblocking)(vcpu);
 }
 
-static inline bool kvm_arch_has_irq_bypass(void)
+static inline bool kvm_arch_has_irq_bypass(struct kvm *kvm)
 {
 	return enable_device_posted_irqs;
 }
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ab8cfaec82d3..b3b80196f4dc 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2421,7 +2421,7 @@ struct kvm_vcpu * __percpu *kvm_get_running_vcpus(void);
 #if IS_ENABLED(CONFIG_HAVE_KVM_IRQ_BYPASS)
 struct kvm_kernel_irqfd;
 
-bool kvm_arch_has_irq_bypass(void);
+bool kvm_arch_has_irq_bypass(struct kvm *kvm);
 int kvm_arch_irq_bypass_add_producer(struct irq_bypass_consumer *,
 			   struct irq_bypass_producer *);
 void kvm_arch_irq_bypass_del_producer(struct irq_bypass_consumer *,
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 00302399fc37..f536813b1680 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -843,7 +843,7 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu)
  * irq_bypass_stop and irq_bypass_start are not needed and so
  * kvm_ops are not defined for them.
  */
-bool kvm_arch_has_irq_bypass(void)
+bool kvm_arch_has_irq_bypass(struct kvm *kvm)
 {
 	return ((kvmppc_hv_ops && kvmppc_hv_ops->irq_bypass_add_producer) ||
 		(kvmppc_pr_ops && kvmppc_pr_ops->irq_bypass_add_producer));
diff --git a/arch/x86/kvm/irq.c b/arch/x86/kvm/irq.c
index 727245a6ab34..dd776449731f 100644
--- a/arch/x86/kvm/irq.c
+++ b/arch/x86/kvm/irq.c
@@ -439,7 +439,7 @@ static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd,
 	struct kvm_lapic_irq irq;
 	int r;
 
-	if (WARN_ON_ONCE(!irqchip_in_kernel(kvm) || !kvm_arch_has_irq_bypass()))
+	if (WARN_ON_ONCE(!irqchip_in_kernel(kvm) || !kvm_arch_has_irq_bypass(kvm)))
 		return -EINVAL;
 
 	if (entry && entry->type == KVM_IRQ_ROUTING_MSI) {
diff --git a/arch/x86/kvm/vmx/posted_intr.c b/arch/x86/kvm/vmx/posted_intr.c
index 4a6d9a17da23..b0b0b1d5c2cb 100644
--- a/arch/x86/kvm/vmx/posted_intr.c
+++ b/arch/x86/kvm/vmx/posted_intr.c
@@ -151,7 +151,7 @@ static bool vmx_can_use_vtd_pi(struct kvm *kvm)
 	 * bypass IRQ being attached to the VM.  vmx_pi_start_bypass() ensures
 	 * blockng vCPUs will see an elevated count or get KVM_REQ_UNBLOCK.
 	 */
-	return irqchip_in_kernel(kvm) && kvm_arch_has_irq_bypass() &&
+	return irqchip_in_kernel(kvm) && kvm_arch_has_irq_bypass(kvm) &&
 	       READ_ONCE(kvm->arch.nr_possible_bypass_irqs);
 }
 
diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
index 93ad2ebc963f..b170b37fdee2 100644
--- a/virt/kvm/eventfd.c
+++ b/virt/kvm/eventfd.c
@@ -488,7 +488,7 @@ kvm_irqfd_assign(struct kvm *kvm, struct kvm_irqfd *args)
 		schedule_work(&irqfd->inject);
 
 #if IS_ENABLED(CONFIG_HAVE_KVM_IRQ_BYPASS)
-	if (kvm_arch_has_irq_bypass()) {
+	if (kvm_arch_has_irq_bypass(irqfd->kvm)) {
 		irqfd->consumer.add_producer = kvm_arch_irq_bypass_add_producer;
 		irqfd->consumer.del_producer = kvm_arch_irq_bypass_del_producer;
 		irqfd->consumer.stop = kvm_arch_irq_bypass_stop;
-- 
2.54.0

[RFC PATCH v3 09/27] KVM: SVM: Disable IRQ bypass for Secure AVIC

[Naveen N Rao (AMD)]

Secure AVIC does not support accelerating device interrupts, so have
kvm_arch_has_irq_bypass() return false for a Secure AVIC enabled SEV-SNP
guest. Since we now have to consult kvm_arch, move the function to x86.c
and export it for use by KVM submodules.

Due to this, avic_pi_update_irte() should never be called for a Secure
AVIC guest. Add a WARN_ON() so that it is clear something is broken.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/include/asm/kvm_host.h | 8 ++------
 arch/x86/kvm/svm/avic.c         | 7 +++++++
 arch/x86/kvm/svm/sev.c          | 3 +++
 arch/x86/kvm/x86.c              | 7 +++++++
 4 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index d81179b43199..cabb368b7908 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1380,6 +1380,8 @@ struct kvm_arch {
 	bool apic_access_memslot_enabled;
 	bool apic_access_memslot_inhibited;
 
+	bool apicv_has_irq_bypass;
+
 	/*
 	 * Force apicv_update_lock and apicv_nr_irq_window_req to reside in a
 	 * dedicated cacheline.  They are write-mostly, whereas most everything
@@ -2063,10 +2065,4 @@ static inline void kvm_arch_vcpu_unblocking(struct kvm_vcpu *vcpu)
 {
 	kvm_x86_call(vcpu_unblocking)(vcpu);
 }
-
-static inline bool kvm_arch_has_irq_bypass(struct kvm *kvm)
-{
-	return enable_device_posted_irqs;
-}
-
 #endif /* _ASM_X86_KVM_HOST_H */
diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index 7340c9e251e0..9b4d3a10198b 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -919,6 +919,13 @@ int avic_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm *kvm,
 			unsigned int host_irq, uint32_t guest_irq,
 			struct kvm_vcpu *vcpu, u32 vector)
 {
+	/*
+	 * Complain loudly as we should never reach here for Secure AVIC.
+	 * See kvm_arch_has_irq_bypass()
+	 */
+	if (WARN_ON_ONCE(snp_is_secure_avic_enabled(kvm)))
+		return -EINVAL;
+
 	/*
 	 * If the IRQ was affined to a different vCPU, remove the IRTE metadata
 	 * from the *previous* vCPU's list.
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 686227a15328..a47bb417bd98 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -566,6 +566,9 @@ static int __sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp,
 	INIT_LIST_HEAD(&sev->mirror_vms);
 	sev->need_init = false;
 
+	if (snp_is_secure_avic_enabled(kvm))
+		kvm->arch.apicv_has_irq_bypass = false;
+
 	kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_SEV);
 
 	return 0;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8dbc0fa302a8..ed94aa8ebc0d 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7233,6 +7233,7 @@ static void kvm_apicv_init(struct kvm *kvm)
 						       APICV_INHIBIT_REASON_DISABLED;
 
 	set_or_clear_apicv_inhibit(&kvm->arch.apicv_inhibit_reasons, reason, true);
+	kvm->arch.apicv_has_irq_bypass = true;
 
 	init_rwsem(&kvm->arch.apicv_update_lock);
 }
@@ -10572,6 +10573,12 @@ bool kvm_arch_no_poll(struct kvm_vcpu *vcpu)
 	return (vcpu->arch.msr_kvm_poll_control & 1) == 0;
 }
 
+bool kvm_arch_has_irq_bypass(struct kvm *kvm)
+{
+	return enable_device_posted_irqs && kvm->arch.apicv_has_irq_bypass;
+}
+EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_arch_has_irq_bypass);
+
 #ifdef CONFIG_KVM_GUEST_MEMFD
 /*
  * KVM doesn't yet support initializing guest_memfd memory as shared for VMs
-- 
2.54.0

[RFC PATCH v3 10/27] KVM: SVM: Add avic_ipiv_is_soft_disabled() as a wrapper around enable_ipiv

[Naveen N Rao (AMD)]

enable_ipiv is used to disable AVIC IPI virtualization in software to
address hardware errata related to IPI acceleration, or because the user
chose to disable it manually through a module parameter. In either case,
AVIC IPIv itself is available in hardware and AVIC requires setup of
related data structures. To distinguish this from Secure AVIC mode which
completely lacks support for IPIv, add a helper to clarify the true
nature of this knob, which is to soft-disable IPIv.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/avic.c | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index 9b4d3a10198b..c9e375c5a9c1 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -67,6 +67,15 @@ static_assert(__AVIC_GATAG(AVIC_VM_ID_MASK, AVIC_VCPU_IDX_MASK) == -1u);
 
 #define AVIC_AUTO_MODE -1
 
+/*
+ * IPIv is supported, but software disabled due to erratum, or because
+ * the user turned it off through the module parameter.
+ */
+static bool avic_ipiv_is_soft_disabled(void)
+{
+	return !enable_ipiv;
+}
+
 static int avic_param_set(const char *val, const struct kernel_param *kp)
 {
 	if (val && sysfs_streq(val, "auto")) {
@@ -1092,7 +1101,7 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu,
 	 * Keep the APIC ID up-to-date in the entry to minimize the chances of
 	 * things going sideways if hardware peeks at the ID.
 	 */
-	if (!enable_ipiv)
+	if (avic_ipiv_is_soft_disabled())
 		entry &= ~AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK;
 
 	WRITE_ONCE(kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);
@@ -1150,7 +1159,7 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action)
 	 */
 	entry &= ~AVIC_PHYSICAL_ID_ENTRY_IS_RUNNING_MASK;
 
-	if (enable_ipiv)
+	if (!avic_ipiv_is_soft_disabled())
 		WRITE_ONCE(kvm_svm->avic_physical_id_table[vcpu->vcpu_id], entry);
 
 	/*
-- 
2.54.0

[RFC PATCH v3 11/27] KVM: SVM: Disable IPIv for Secure AVIC

[Naveen N Rao (AMD)]

Secure AVIC does not have IPI virtualization (except for Self-IPIs which
is not relevant for KVM), i.e., guest ICR writes are not accelerated by
the hardware but results in a #VC that the guest is expected to handle
with the help of the hypervisor. This also means there are no AVIC
Physical/Logical ID tables to be setup.

Add a helper avic_ipiv_is_supported() to query support for IPIv and use
the same to short-circuit various AVIC flows related to AVIC data
structures such as the Physical and Logical ID tables for Secure AVIC.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/svm.h  |  1 +
 arch/x86/kvm/svm/avic.c | 23 ++++++++++++++++++++++-
 arch/x86/kvm/svm/sev.c  |  4 +++-
 3 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 1157d022bac1..e48744f6d756 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -131,6 +131,7 @@ struct kvm_svm {
 	u32 *avic_logical_id_table;
 	u64 *avic_physical_id_table;
 	struct hlist_node hnode;
+	bool avic_ipiv_is_not_supported;
 
 #ifdef CONFIG_KVM_AMD_SEV
 	struct kvm_sev_info sev_info;
diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index c9e375c5a9c1..0a4e91e15e74 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -76,6 +76,12 @@ static bool avic_ipiv_is_soft_disabled(void)
 	return !enable_ipiv;
 }
 
+/* IPIv is not supported in Secure AVIC mode, no AVIC tables necessary */
+static bool avic_ipiv_is_supported(struct kvm *kvm)
+{
+	return !to_kvm_svm(kvm)->avic_ipiv_is_not_supported;
+}
+
 static int avic_param_set(const char *val, const struct kernel_param *kp)
 {
 	if (val && sysfs_streq(val, "auto")) {
@@ -364,7 +370,7 @@ int avic_vcpu_precreate(struct kvm *kvm)
 {
 	int r;
 
-	if (!irqchip_in_kernel(kvm) || WARN_ON_ONCE(!enable_apicv))
+	if (!irqchip_in_kernel(kvm) || WARN_ON_ONCE(!enable_apicv) || !avic_ipiv_is_supported(kvm))
 		return 0;
 
 	/*
@@ -435,6 +441,9 @@ static int avic_init_backing_page(struct kvm_vcpu *vcpu)
 	u32 id = vcpu->vcpu_id;
 	u64 new_entry;
 
+	if (!avic_ipiv_is_supported(vcpu->kvm))
+		return 0;
+
 	/*
 	 * Inhibit AVIC if the vCPU ID is bigger than what is supported by AVIC
 	 * hardware.  Immediately clear apicv_active, i.e. don't wait until the
@@ -1069,6 +1078,9 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu,
 
 	lockdep_assert_preemption_disabled();
 
+	if (!avic_ipiv_is_supported(vcpu->kvm))
+		return;
+
 	if (WARN_ON(h_physical_id & ~AVIC_PHYSICAL_ID_ENTRY_HOST_PHYSICAL_ID_MASK))
 		return;
 
@@ -1113,6 +1125,9 @@ static void __avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu,
 
 void avic_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
 {
+	if (!avic_ipiv_is_supported(vcpu->kvm))
+		return;
+
 	/*
 	 * No need to update anything if the vCPU is blocking, i.e. if the vCPU
 	 * is being scheduled in after being preempted.  The CPU entries in the
@@ -1135,6 +1150,9 @@ static void __avic_vcpu_put(struct kvm_vcpu *vcpu, enum avic_vcpu_action action)
 
 	lockdep_assert_preemption_disabled();
 
+	if (!avic_ipiv_is_supported(vcpu->kvm))
+		return;
+
 	if (WARN_ON_ONCE(vcpu->vcpu_id * sizeof(entry) >=
 			 PAGE_SIZE << avic_get_physical_id_table_order(vcpu->kvm)))
 		return;
@@ -1185,6 +1203,9 @@ void avic_vcpu_put(struct kvm_vcpu *vcpu)
 	 */
 	u64 entry = to_svm(vcpu)->avic_physical_id_entry;
 
+	if (!avic_ipiv_is_supported(vcpu->kvm))
+		return;
+
 	/*
 	 * Nothing to do if IsRunning == '0' due to vCPU blocking, i.e. if the
 	 * vCPU is preempted while its in the process of blocking.  WARN if the
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index a47bb417bd98..a12e7e9f9fa1 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -566,8 +566,10 @@ static int __sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp,
 	INIT_LIST_HEAD(&sev->mirror_vms);
 	sev->need_init = false;
 
-	if (snp_is_secure_avic_enabled(kvm))
+	if (snp_is_secure_avic_enabled(kvm)) {
 		kvm->arch.apicv_has_irq_bypass = false;
+		to_kvm_svm(kvm)->avic_ipiv_is_not_supported = true;
+	}
 
 	kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_SEV);
 
-- 
2.54.0

[RFC PATCH v3 12/27] KVM: SVM: Short-circuit a few AVIC flows for Secure AVIC

[Naveen N Rao (AMD)]

Short-circuit (and return early from) a few functions in AVIC that are
not relevant for Secure AVIC:
1. In Secure AVIC mode, hardware ignores the MSR permissions bitmap
   w.r.t the APIC MSRs, so do not change MSR intercepts. We still want
   to intercept those MSRs since the guest is free to request emulation
   of those MSRs through GHCB calls and checks require that the
   intercepts be set.
2. In avic_[activate|init]_vmcb(), skip setting up AVIC-related VMCB
   fields since those are not used by Secure AVIC. AVIC/x2AVIC in
   particular should *not* be enabled in the VMCB int_ctl for Secure
   AVIC.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/avic.c | 31 ++++++++++++++++++++++++-------
 1 file changed, 24 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index 0a4e91e15e74..bfd758369b5a 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -141,6 +141,17 @@ static void avic_set_x2apic_msr_interception(struct vcpu_svm *svm,
 	u64 rd_regs;
 	int i;
 
+	/*
+	 * For Secure AVIC, treat all APIC MSRs as intercepted always. Secure AVIC
+	 * hardware controls MSR interception and the MSR permission bitmap is not
+	 * consulted by hardware. However, the guest is free to use GHCB to request
+	 * emulation of APIC MSR reads and writes. In that scenario, we need these
+	 * MSRs to be seen as being intercepted so that sev_es_prevent_msr_access()
+	 * does not reject those MSR accesses.
+	 */
+	if (snp_is_secure_avic_enabled(svm->vcpu.kvm))
+		return;
+
 	if (intercept == svm->x2avic_msrs_intercepted)
 		return;
 
@@ -192,9 +203,6 @@ static void avic_activate_vmcb(struct vcpu_svm *svm)
 	struct kvm_vcpu *vcpu = &svm->vcpu;
 
 	vmcb->control.int_ctl &= ~(AVIC_ENABLE_MASK | X2APIC_MODE_MASK);
-	vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK;
-	vmcb->control.avic_physical_id |= avic_get_max_physical_id(vcpu);
-	vmcb->control.int_ctl |= AVIC_ENABLE_MASK;
 
 	svm_clr_intercept(svm, INTERCEPT_CR8_WRITE);
 
@@ -227,6 +235,13 @@ static void avic_activate_vmcb(struct vcpu_svm *svm)
 	 */
 	kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, &svm->vcpu);
 
+	if (snp_is_secure_avic_enabled(vcpu->kvm))
+		return;
+
+	vmcb->control.avic_physical_id &= ~AVIC_PHYSICAL_MAX_INDEX_MASK;
+	vmcb->control.avic_physical_id |= avic_get_max_physical_id(vcpu);
+	vmcb->control.int_ctl |= AVIC_ENABLE_MASK;
+
 	/*
 	 * Note: KVM supports hybrid-AVIC mode, where KVM emulates x2APIC MSR
 	 * accesses, while interrupt injection to a running vCPU can be
@@ -422,10 +437,12 @@ void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb)
 {
 	struct kvm_svm *kvm_svm = to_kvm_svm(svm->vcpu.kvm);
 
-	vmcb->control.avic_backing_page = avic_get_backing_page_address(svm);
-	vmcb->control.avic_logical_id = __sme_set(__pa(kvm_svm->avic_logical_id_table));
-	vmcb->control.avic_physical_id = __sme_set(__pa(kvm_svm->avic_physical_id_table));
-	vmcb->control.avic_vapic_bar = APIC_DEFAULT_PHYS_BASE;
+	if (!snp_is_secure_avic_enabled(svm->vcpu.kvm)) {
+		vmcb->control.avic_backing_page = avic_get_backing_page_address(svm);
+		vmcb->control.avic_logical_id = __sme_set(__pa(kvm_svm->avic_logical_id_table));
+		vmcb->control.avic_physical_id = __sme_set(__pa(kvm_svm->avic_physical_id_table));
+		vmcb->control.avic_vapic_bar = APIC_DEFAULT_PHYS_BASE;
+	}
 
 	if (kvm_vcpu_apicv_active(&svm->vcpu))
 		avic_activate_vmcb(svm);
-- 
2.54.0

[RFC PATCH v3 13/27] KVM: SVM: Warn if we ever receive AVIC_UNACCELERATED_ACCESS #VMEXIT

[Naveen N Rao (AMD)]

With Secure AVIC, all AVIC-related exits are NAE (Non-Automatic Exits)
and result in a #VC in the guest rather than a #VMEXIT. The guest can
then use GHCB to request specific services from the hypervisor. As such,
we should never see AVIC_UNACCELERATED_ACCESS exits in KVM. Add a
WARN_ON() so that it is clear that something is wrong.

Note that AVIC_INCOMPLETE_IPI is still possible via VMGEXIT.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/avic.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index bfd758369b5a..bf701e604a85 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -896,6 +896,10 @@ int avic_unaccelerated_access_interception(struct kvm_vcpu *vcpu)
 
 	trace_kvm_avic_unaccelerated_access(vcpu->vcpu_id, offset,
 					    trap, write, vector);
+
+	if (WARN_ON_ONCE(snp_is_secure_avic_enabled(vcpu->kvm)))
+		return 1;
+
 	if (trap) {
 		/* Handling Trap */
 		WARN_ONCE(!write, "svm: Handling trap read.\n");
-- 
2.54.0

[RFC PATCH v3 14/27] KVM: SVM: Do not inhibit AVIC for SEV-SNP guests if Secure AVIC is enabled

[Naveen N Rao (AMD)]

Now that Secure AVIC mode is appropriately handled by the AVIC code, do
not inhibit APICv for SEV-SNP guests that have Secure AVIC enabled.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/sev.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index a12e7e9f9fa1..00cb45ce145f 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -569,10 +569,10 @@ static int __sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp,
 	if (snp_is_secure_avic_enabled(kvm)) {
 		kvm->arch.apicv_has_irq_bypass = false;
 		to_kvm_svm(kvm)->avic_ipiv_is_not_supported = true;
+	} else {
+		kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_SEV);
 	}
 
-	kvm_set_apicv_inhibit(kvm, APICV_INHIBIT_REASON_SEV);
-
 	return 0;
 
 e_free:
-- 
2.54.0

[RFC PATCH v3 15/27] KVM: SVM: Set VGIF in VMSA area for Secure AVIC guests

[Naveen N Rao (AMD)]

From: Kishon Vijay Abraham I <kvijayab@amd.com>

In Secure AVIC mode, VGIF is read from the VMSA (vintr_ctrl field), and
the corresponding bit in VMCB control area (int_ctl field) is ignored.
Set this bit in the BSP's VMSA so that interrupts are not masked. It is
up to the guest to set this bit in AP's VMSA that it prepares.

Signed-off-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Co-developed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Co-developed-by: Naveen N Rao (AMD) <naveen@kernel.org>
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/sev.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 00cb45ce145f..f5b9ff69dbc1 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -1031,6 +1031,10 @@ static int sev_es_sync_vmsa(struct vcpu_svm *svm)
 
 	save->sev_features = sev->vmsa_features;
 
+	/* Secure AVIC loads the below from the VMSA, rather than the VMCB */
+	if (snp_is_secure_avic_enabled(vcpu->kvm))
+		save->vintr_ctrl |= V_GIF_MASK;
+
 	/*
 	 * Skip FPU and AVX setup with KVM_SEV_ES_INIT to avoid
 	 * breaking older measurements.
-- 
2.54.0

[RFC PATCH v3 16/27] KVM: SVM: Add handler for VMGEXIT Secure AVIC NAE event

[Naveen N Rao (AMD)]

From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>

[DO NOT MERGE]

VMGEXIT Secure AVIC NAE event is used by the guest for two purposes
determined by VMCB->EXITINFO1:
1. SVM_VMGEXIT_SAVIC_REGISTER_GPA: Used to inform the hypervisor about
   the GPA of the page (RBX) being used as the Secure AVIC backing page.
   RAX indicates APIC ID of the target vCPU (-1 for self)
2. SVM_VMGEXIT_SAVIC_UNREGISTER_GPA: Used to inform the hypervisor that
   the GPA is no longer being used as the backing page for Secure AVIC.
   The previously registered GPA for the Secure AVIC backing page is
   returned by the hypervisor to the guest.

The primary motivation behind these is to ensure that Secure AVIC
hardware accesses to the guest APIC backing page never generate an #NPF,
since Secure AVIC hardware cannot recover from such faults. Quoting the
APM:
  "It is required that the guest APIC backing page for a vCPU is
   pinned in system memory between VMRUN and VMEXIT because some AVIC
   hardware acceleration sequences may not be restartable when secure
   AVIC is enabled. If an access to the guest's own backing page by
   AVIC hardware results in a nested page fault, EXITINFO1 bit 63
   (Not Restartable) is set (this is an Automatic Exit) and the BUSY
   bit in the VMSA is set."

A guest vCPU that has the BUSY bit set in the VMSA cannot be restarted
and the guest will have to be killed.

One of the main reasons why the SPTE for a Secure AVIC backing page may
be invalidated is if it is backed by a huge page in the host, and an
adjacent page changes state forcing the huge page to be split. Currently
though, KVM uses guest_memfd to back SEV-SNP guest private memory, and
those only use 4k pages. As such, this _may_ not be an issue today.

It is possible that KVM may still invalidate an SPTE for other reasons -
those will need to be addressed.

Co-developed-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Signed-off-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Co-developed-by: Naveen N Rao (AMD) <naveen@kernel.org>
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/include/uapi/asm/svm.h |  1 +
 arch/x86/kvm/svm/svm.h          |  2 +
 arch/x86/kvm/svm/sev.c          | 68 +++++++++++++++++++++++++++++++++
 3 files changed, 71 insertions(+)

diff --git a/arch/x86/include/uapi/asm/svm.h b/arch/x86/include/uapi/asm/svm.h
index 010a45c9f614..e8531a9d998d 100644
--- a/arch/x86/include/uapi/asm/svm.h
+++ b/arch/x86/include/uapi/asm/svm.h
@@ -245,6 +245,7 @@
 	{ SVM_VMGEXIT_GUEST_REQUEST,	"vmgexit_guest_request" }, \
 	{ SVM_VMGEXIT_EXT_GUEST_REQUEST, "vmgexit_ext_guest_request" }, \
 	{ SVM_VMGEXIT_AP_CREATION,	"vmgexit_ap_creation" }, \
+	{ SVM_VMGEXIT_SAVIC,		"vmgexit_secure_avic" }, \
 	{ SVM_VMGEXIT_HV_FEATURES,	"vmgexit_hypervisor_feature" }, \
 	{ SVM_EXIT_ERR,         "invalid_guest_state" }
 
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index e48744f6d756..5e9496f8566a 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -367,6 +367,8 @@ struct vcpu_svm {
 
 	/* Guest GIF value, used when vGIF is not enabled */
 	bool guest_gif;
+
+	gpa_t snp_savic_gpa;
 };
 
 struct svm_cpu_data {
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index f5b9ff69dbc1..ca921a185b64 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3474,6 +3474,13 @@ static bool sev_es_are_required_ghcb_fields_valid(struct vcpu_svm *svm)
 	case SVM_VMGEXIT_MMIO_WRITE:
 	case SVM_VMGEXIT_PSC:
 		return kvm_ghcb_sw_scratch_is_valid(svm);
+	case SVM_VMGEXIT_SAVIC:
+		if (!kvm_ghcb_rax_is_valid(svm) ||
+		    (control->exit_info_1 == SVM_VMGEXIT_SAVIC_REGISTER_GPA &&
+		     !kvm_ghcb_rbx_is_valid(svm)))
+			return false;
+
+		return true;
 	default:
 		return true;
 	}
@@ -4420,6 +4427,57 @@ static int sev_handle_vmgexit_msr_protocol(struct vcpu_svm *svm)
 	return 0;
 }
 
+static int sev_handle_savic_vmgexit(struct vcpu_svm *svm)
+{
+	struct kvm_vcpu *target_vcpu;
+	u64 apic_id;
+	gpa_t gpa;
+
+	apic_id = kvm_rax_read_raw(&svm->vcpu);
+	if (apic_id != SVM_VMGEXIT_SAVIC_SELF_GPA && upper_32_bits(apic_id))
+		goto vmgexit_err;
+
+	/* Use invoking vCPU if apic_id is -1 (SVM_VMGEXIT_SAVIC_SELF_GPA) */
+	target_vcpu = &svm->vcpu;
+	if (apic_id != SVM_VMGEXIT_SAVIC_SELF_GPA) {
+		target_vcpu = kvm_get_vcpu_by_id(svm->vcpu.kvm, (int)apic_id);
+		if (!target_vcpu)
+			goto vmgexit_err;
+	}
+
+	switch (svm->vmcb->control.exit_info_1) {
+	case SVM_VMGEXIT_SAVIC_REGISTER_GPA:
+		gpa = kvm_rbx_read_raw(&svm->vcpu);
+		if (!PAGE_ALIGNED(gpa))
+			goto vmgexit_err;
+
+		/*
+		 * TODO: Ensure that guest (Secure AVIC hardware) accesses
+		 * to the guest APIC backing page can never cause an #NPF.
+		 */
+
+		/*
+		 * Don't bother using any synchronization here if updating the
+		 * GPA for a different vCPU. If the guest is invoking this for
+		 * a specific vCPU in parallel, then it gets to keep the pieces.
+		 */
+		to_svm(target_vcpu)->snp_savic_gpa = gpa;
+		break;
+	case SVM_VMGEXIT_SAVIC_UNREGISTER_GPA:
+		kvm_rbx_write_raw(&svm->vcpu, to_svm(target_vcpu)->snp_savic_gpa);
+		to_svm(target_vcpu)->snp_savic_gpa = 0;
+		break;
+	default:
+		goto vmgexit_err;
+	}
+
+	return 1;
+
+vmgexit_err:
+	svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_INPUT);
+	return 1;
+}
+
 static bool is_snp_only_vmgexit(u64 exit_code)
 {
 	switch (exit_code) {
@@ -4427,6 +4485,7 @@ static bool is_snp_only_vmgexit(u64 exit_code)
 	case SVM_VMGEXIT_GUEST_REQUEST:
 	case SVM_VMGEXIT_EXT_GUEST_REQUEST:
 	case SVM_VMGEXIT_PSC:
+	case SVM_VMGEXIT_SAVIC:
 		return true;
 	default:
 		return false;
@@ -4490,6 +4549,13 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
 		return 1;
 	}
 
+	if (control->exit_code == SVM_VMGEXIT_SAVIC && !snp_is_secure_avic_enabled(vcpu->kvm)) {
+		vcpu_unimpl(vcpu, "vmgexit: exit code %#llx is only valid if Secure AVIC is enabled\n",
+			    control->exit_code);
+		svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_EVENT);
+		return 1;
+	}
+
 	if (!sev_es_are_required_ghcb_fields_valid(svm)) {
 		/*
 		 * Print the exit code even though it may not be marked valid
@@ -4610,6 +4676,8 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
 
 		return snp_handle_ext_guest_req(svm, control->exit_info_1,
 						control->exit_info_2);
+	case SVM_VMGEXIT_SAVIC:
+		return sev_handle_savic_vmgexit(svm);
 	case SVM_VMGEXIT_UNSUPPORTED_EVENT:
 		/*
 		 * Note, the _guest_ is reporting an unsupported #VC, i.e. this
-- 
2.54.0

[RFC PATCH v3 17/27] KVM: SVM: Do not intercept SECURE_AVIC_CONTROL MSR for Secure AVIC guests

[Naveen N Rao (AMD)]

From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>

SECURE_AVIC_CONTROL MSR is used by the guest to configure and enable
Secure AVIC. In order for the guest to be able to successfully do this,
the MSR access must not be intercepted.

Co-developed-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Signed-off-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Co-developed-by: Naveen N Rao (AMD) <naveen@kernel.org>
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/avic.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index bf701e604a85..6b3983d4f45e 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -437,7 +437,9 @@ void avic_init_vmcb(struct vcpu_svm *svm, struct vmcb *vmcb)
 {
 	struct kvm_svm *kvm_svm = to_kvm_svm(svm->vcpu.kvm);
 
-	if (!snp_is_secure_avic_enabled(svm->vcpu.kvm)) {
+	if (snp_is_secure_avic_enabled(svm->vcpu.kvm)) {
+		svm_disable_intercept_for_msr(&svm->vcpu, MSR_AMD64_SAVIC_CONTROL, MSR_TYPE_RW);
+	} else {
 		vmcb->control.avic_backing_page = avic_get_backing_page_address(svm);
 		vmcb->control.avic_logical_id = __sme_set(__pa(kvm_svm->avic_logical_id_table));
 		vmcb->control.avic_physical_id = __sme_set(__pa(kvm_svm->avic_physical_id_table));
-- 
2.54.0

[RFC PATCH v3 18/27] KVM: x86: Add a new kvm_x86_op protected_apic_has_injectable_intr()

[Naveen N Rao (AMD)]

Secure AVIC does not support posted interrupts (unlike TDX) but requires
interrupts to be injected through the VMCB. To address this, introduce a
new kvm_x86_op protected_apic_has_injectable_intr() and update
kvm_cpu_has_injectable_intr() to use this if the guest APIC is
protected.

Finally, update kvm_check_and_inject_events() to ignore
kvm_cpu_get_interrupt() for a protected APIC guest. For such guests (and
at least in the case of Secure AVIC), KVM has no visibility into IRQ
windows and all pending vectors can be injected at once.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/include/asm/kvm-x86-ops.h | 1 +
 arch/x86/include/asm/kvm_host.h    | 1 +
 arch/x86/kvm/irq.c                 | 3 +++
 arch/x86/kvm/x86.c                 | 6 +++++-
 4 files changed, 10 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 83dc5086138b..f042af63229b 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -116,6 +116,7 @@ KVM_X86_OP_OPTIONAL(apicv_pre_state_restore)
 KVM_X86_OP_OPTIONAL(apicv_post_state_restore)
 KVM_X86_OP_OPTIONAL_RET0(dy_apicv_has_pending_interrupt)
 KVM_X86_OP_OPTIONAL(protected_apic_has_interrupt)
+KVM_X86_OP_OPTIONAL_RET0(protected_apic_has_injectable_intr)
 KVM_X86_OP_OPTIONAL(set_hv_timer)
 KVM_X86_OP_OPTIONAL(cancel_hv_timer)
 KVM_X86_OP(setup_mce)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index cabb368b7908..31d1fc699e7e 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1861,6 +1861,7 @@ struct kvm_x86_ops {
 	void (*apicv_post_state_restore)(struct kvm_vcpu *vcpu);
 	bool (*dy_apicv_has_pending_interrupt)(struct kvm_vcpu *vcpu);
 	bool (*protected_apic_has_interrupt)(struct kvm_vcpu *vcpu);
+	bool (*protected_apic_has_injectable_intr)(struct kvm_vcpu *vcpu);
 
 	int (*set_hv_timer)(struct kvm_vcpu *vcpu, u64 guest_deadline_tsc,
 			    bool *expired);
diff --git a/arch/x86/kvm/irq.c b/arch/x86/kvm/irq.c
index dd776449731f..3f8c4bf834f0 100644
--- a/arch/x86/kvm/irq.c
+++ b/arch/x86/kvm/irq.c
@@ -98,6 +98,9 @@ int kvm_cpu_has_injectable_intr(struct kvm_vcpu *v)
 	if (kvm_cpu_has_extint(v))
 		return 1;
 
+	if (lapic_in_kernel(v) && v->arch.apic->guest_apic_protected)
+		return kvm_x86_call(protected_apic_has_injectable_intr)(v);
+
 	if (!is_guest_mode(v) && kvm_vcpu_apicv_active(v))
 		return 0;
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index ed94aa8ebc0d..2609a2972526 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7686,7 +7686,11 @@ static int kvm_check_and_inject_events(struct kvm_vcpu *vcpu,
 		if (r) {
 			int irq = kvm_cpu_get_interrupt(vcpu);
 
-			if (!WARN_ON_ONCE(irq == -1)) {
+			/*
+			 * Ignore kvm_cpu_get_interrupt() for a protected APIC guest. We know
+			 * there are injectable interrupts due to the check above.
+			 */
+			if (vcpu->arch.apic->guest_apic_protected || !WARN_ON_ONCE(irq == -1)) {
 				kvm_queue_interrupt(vcpu, irq, false);
 				kvm_x86_call(inject_irq)(vcpu, false);
 				WARN_ON(kvm_x86_call(interrupt_allowed)(vcpu, true) < 0);
-- 
2.54.0

[RFC PATCH v3 19/27] KVM: SVM: Implement kvm_x86_ops->protected_apic_has_injectable_intr() for Secure AVIC

[Naveen N Rao (AMD)]

KVM tracks interrupts to be injected for a Secure AVIC enabled guest in
its copy of the APIC_IRR. Consequently, checking if an interrupt is
pending injection just requires checking KVM's copy of the APIC_IRR. Wire
up protected_apic_has_injectable_intr() to do so.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/svm.h | 2 ++
 arch/x86/kvm/svm/sev.c | 6 ++++++
 arch/x86/kvm/svm/svm.c | 2 ++
 3 files changed, 10 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 5e9496f8566a..af3c84a61c07 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1017,6 +1017,7 @@ int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
 void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa);
 bool snp_is_secure_avic_enabled(struct kvm *kvm);
+bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu);
 #else
 static inline struct page *snp_safe_alloc_page_node(int node, gfp_t gfp)
 {
@@ -1055,6 +1056,7 @@ static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu)
 }
 static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa) {}
 static inline bool snp_is_secure_avic_enabled(struct kvm *kvm) { return false; }
+static inline bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu) { return false; }
 #endif
 
 /* vmenter.S */
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index ca921a185b64..3d36168c3327 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -35,6 +35,7 @@
 #include "svm_ops.h"
 #include "cpuid.h"
 #include "trace.h"
+#include "lapic.h"
 
 #define GHCB_VERSION_MAX	2ULL
 #define GHCB_VERSION_MIN	1ULL
@@ -4941,6 +4942,11 @@ void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, struct sev_es_save_are
 		hostsa->tsc_aux = kvm_get_user_return_msr(tsc_aux_uret_slot);
 }
 
+bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu)
+{
+	return apic_find_highest_vector(vcpu->arch.apic->regs + APIC_IRR) != -1;
+}
+
 void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 0e0dd9618750..441520c1312d 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5398,6 +5398,8 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.apicv_post_state_restore = avic_apicv_post_state_restore,
 	.required_apicv_inhibits = AVIC_REQUIRED_APICV_INHIBITS,
 
+	.protected_apic_has_injectable_intr = snp_protected_apic_has_injectable_intr,
+
 	.get_exit_info = svm_get_exit_info,
 	.get_entry_info = svm_get_entry_info,
 
-- 
2.54.0

[RFC PATCH v3 20/27] KVM: SVM: Implement kvm_x86_ops->protected_apic_has_interrupt() for Secure AVIC

[Naveen N Rao (AMD)]

Since KVM tracks pending interrupts for a Secure AVIC guest in its copy
of the APIC_IRR, wire up protected_apic_has_interrupt() to just use
protected_apic_has_injectable_intr().

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/svm.h | 2 ++
 arch/x86/kvm/svm/sev.c | 5 +++++
 arch/x86/kvm/svm/svm.c | 1 +
 3 files changed, 8 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index af3c84a61c07..f5e270086b51 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1018,6 +1018,7 @@ struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
 void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa);
 bool snp_is_secure_avic_enabled(struct kvm *kvm);
 bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu);
+bool snp_protected_apic_has_interrupt(struct kvm_vcpu *vcpu);
 #else
 static inline struct page *snp_safe_alloc_page_node(int node, gfp_t gfp)
 {
@@ -1057,6 +1058,7 @@ static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu)
 static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa) {}
 static inline bool snp_is_secure_avic_enabled(struct kvm *kvm) { return false; }
 static inline bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu) { return false; }
+static inline bool snp_protected_apic_has_interrupt(struct kvm_vcpu *vcpu) { return false; }
 #endif
 
 /* vmenter.S */
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 3d36168c3327..a0c5271ec4ca 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4947,6 +4947,11 @@ bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu)
 	return apic_find_highest_vector(vcpu->arch.apic->regs + APIC_IRR) != -1;
 }
 
+bool snp_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)
+{
+	return snp_protected_apic_has_injectable_intr(vcpu);
+}
+
 void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 441520c1312d..3f731483f6a2 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5399,6 +5399,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.required_apicv_inhibits = AVIC_REQUIRED_APICV_INHIBITS,
 
 	.protected_apic_has_injectable_intr = snp_protected_apic_has_injectable_intr,
+	.protected_apic_has_interrupt = snp_protected_apic_has_interrupt,
 
 	.get_exit_info = svm_get_exit_info,
 	.get_entry_info = svm_get_entry_info,
-- 
2.54.0

[RFC PATCH v3 21/27] KVM: SVM: Add interrupt delivery support for Secure AVIC guests

[Naveen N Rao (AMD)]

To inject an interrupt into a Secure AVIC enabled guest, KVM needs to:
- set the vector in VMCB->requested_irr (VMCB offset 0x150), which is a
  contiguous 256-bit field representing vectors that the hypervisor
  wants to inject into the guest.
- set VMCB->update_irr (VMCB offset 0x134) so hardware knows to process
  requested_irr from the VMCB, and
- invoke VMRUN.

As the field name suggests, KVM can only "request" for interrupts to be
injected into the guest.  Guest is free to ignore vectors it does not
want to accept from the hypervisor, and it controls which vectors are
actually accepted by setting up allowed_irr field in its private APIC
backing page (8 32-bit APIC registers following APIC_IRR in the backing
page).

On VMRUN, hardware updates APIC_IRR in the private guest APIC backing
page based on update_irr/requested_irr in the VMCB and allowed_irr in
the guest APIC backing page, and then clears VMCB fields update_irr and
requested_irr.

Because hardware clears requested_irr and update_irr in the VMCB on
VMRUN in no particular order, KVM can't stuff incoming vectors directly
into the VMCB fields. Instead, park vectors to be injected into the
guest in KVM's copy of the APIC_IRR (already done in the default flow in
svm_deliver_interrupt()).

Since Secure AVIC requires interrupts to be injected through the VMCB,
do not use AVIC doorbells for interrupt delivery. Instead rely on the
non-AVIC flow in svm_complete_interrupt_delivery() to set KVM_REQ_EVENT
and kick the vCPU.

Before VM entry, KVM invokes svm_inject_irq() (since APIC_IRR has
vectors to be injected, and since kvm_cpu_has_injectable_intr() returns
true), at which point transfer all pending interrupt vectors from KVM's
APIC_IRR to VMCB->requested_irr and set VMCB->update_irr.  Atomically
clear APIC_IRR so that incoming interrupt vectors are not lost. Also
update vCPU irq_injection stats and optionally invoke the corresponding
tracepoint if enabled (its debatable whether kvm_apicv_accept_irq
tracepoint should be invoked instead).

Finally, always return true for interrupt_allowed() since KVM has no
visibility into the guest APIC state and KVM can simply pend interrupts
in requested_irr at any point. Return early from svm_enable_irq_window()
for the same reason -- KVM has no visibility into that.

Co-developed-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Signed-off-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Co-developed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/include/asm/svm.h |  7 +++++--
 arch/x86/kvm/svm/svm.h     |  2 ++
 arch/x86/kvm/svm/sev.c     | 27 +++++++++++++++++++++++++++
 arch/x86/kvm/svm/svm.c     | 25 ++++++++++++++++++++++++-
 4 files changed, 58 insertions(+), 3 deletions(-)

diff --git a/arch/x86/include/asm/svm.h b/arch/x86/include/asm/svm.h
index 5857b942957b..b4a803686e0b 100644
--- a/arch/x86/include/asm/svm.h
+++ b/arch/x86/include/asm/svm.h
@@ -162,10 +162,13 @@ struct __attribute__ ((__packed__)) vmcb_control_area {
 	u64 vmsa_pa;		/* Used for an SEV-ES guest */
 	u8 reserved_8[16];
 	u16 bus_lock_counter;		/* Offset 0x120 */
-	u8 reserved_9[22];
+	u8 reserved_9[18];
+	u32 update_irr;			/* Offset 0x134 */
 	u64 allowed_sev_features;	/* Offset 0x138 */
 	u64 guest_sev_features;		/* Offset 0x140 */
-	u8 reserved_10[664];
+	u8 reserved_10[8];
+	u32 requested_irr[8];           /* Offset 0x150 */
+	u8 reserved_11[624];
 	/*
 	 * Offset 0x3e0, 32 bytes reserved
 	 * for use by hypervisor/software.
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index f5e270086b51..44f1d25a167c 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1019,6 +1019,7 @@ void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa)
 bool snp_is_secure_avic_enabled(struct kvm *kvm);
 bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu);
 bool snp_protected_apic_has_interrupt(struct kvm_vcpu *vcpu);
+void savic_update_requested_irr(struct kvm_vcpu *vcpu);
 #else
 static inline struct page *snp_safe_alloc_page_node(int node, gfp_t gfp)
 {
@@ -1059,6 +1060,7 @@ static inline void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_sa
 static inline bool snp_is_secure_avic_enabled(struct kvm *kvm) { return false; }
 static inline bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu) { return false; }
 static inline bool snp_protected_apic_has_interrupt(struct kvm_vcpu *vcpu) { return false; }
+static inline void savic_update_requested_irr(struct kvm_vcpu *vcpu) {}
 #endif
 
 /* vmenter.S */
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index a0c5271ec4ca..754fe12c2f82 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4428,6 +4428,33 @@ static int sev_handle_vmgexit_msr_protocol(struct vcpu_svm *svm)
 	return 0;
 }
 
+void savic_update_requested_irr(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+	u32 *apic_irr, irr;
+
+	for (int i = 0; i < APIC_ISR_NR; i++) {
+		apic_irr = (u32 *)(vcpu->arch.apic->regs + APIC_IRR + i * 0x10);
+
+		if (!READ_ONCE(*apic_irr))
+			continue;
+
+		irr = xchg(apic_irr, 0);
+		svm->vmcb->control.requested_irr[i] |= irr;
+		vcpu->stat.irq_injections += hweight32(irr);
+
+		if (trace_kvm_inj_virq_enabled()) {
+			unsigned long irr_injected = irr;
+			unsigned int vector;
+
+			for_each_set_bit(vector, &irr_injected, BITS_PER_TYPE(irr))
+				trace_kvm_inj_virq(i * 32 + vector, false, false);
+		}
+	}
+
+	WRITE_ONCE(svm->vmcb->control.update_irr, 1);
+}
+
 static int sev_handle_savic_vmgexit(struct vcpu_svm *svm)
 {
 	struct kvm_vcpu *target_vcpu;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 3f731483f6a2..612cc4ac9bd2 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3824,6 +3824,11 @@ static void svm_inject_irq(struct kvm_vcpu *vcpu, bool reinjected)
 	struct vcpu_svm *svm = to_svm(vcpu);
 	u32 type;
 
+	if (snp_is_secure_avic_enabled(vcpu->kvm)) {
+		savic_update_requested_irr(vcpu);
+		return;
+	}
+
 	if (intr->soft) {
 		if (svm_update_soft_interrupt_rip(vcpu, intr->nr))
 			return;
@@ -3896,7 +3901,7 @@ void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode,
 	bool in_guest_mode = (smp_load_acquire(&vcpu->mode) == IN_GUEST_MODE);
 
 	/* Note, this is called iff the local APIC is in-kernel. */
-	if (!READ_ONCE(vcpu->arch.apic->apicv_active)) {
+	if (!READ_ONCE(vcpu->arch.apic->apicv_active) || snp_is_secure_avic_enabled(vcpu->kvm)) {
 		/* Process the interrupt via kvm_check_and_inject_events(). */
 		kvm_make_request(KVM_REQ_EVENT, vcpu);
 		kvm_vcpu_kick(vcpu);
@@ -4050,6 +4055,14 @@ static int svm_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 
+	/*
+	 * KVM does not have access to the Secure AVIC guest APIC backing page.
+	 * The only thing KVM can do is to queue up interrupts in the VMCB
+	 * (Requested_IRR), which is always allowed.
+	 */
+	if (snp_is_secure_avic_enabled(vcpu->kvm))
+		return 1;
+
 	if (svm_interrupt_blocked(vcpu))
 		return 0;
 
@@ -4070,6 +4083,16 @@ static void svm_enable_irq_window(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 
+	/*
+	 * Secure AVIC does not allow VINTR, so we can't request an interrupt
+	 * window. The only reason we end up here is if an interrupt arrived
+	 * just as we injected pending interrupts (from
+	 * kvm_check_and_inject_events()). Ignore and proceed, any pending
+	 * interrupt(s) will be re-injected on next entry.
+	 */
+	if (snp_is_secure_avic_enabled(vcpu->kvm))
+		return;
+
 	/*
 	 * In case GIF=0 we can't rely on the CPU to tell us when GIF becomes
 	 * 1, because that's a separate STGI/VMRUN intercept.  The next time we
-- 
2.54.0

[RFC PATCH v3 22/27] KVM: SVM: Add support for incomplete IPI handling for Secure AVIC

[Naveen N Rao (AMD)]

Secure AVIC hardware accelerates self IPIs, i.e., on WRMSR to
APIC_SELF_IPI and to APIC_ICR with destination shorthand APIC_DEST_SELF,
hardware updates IRR in the guest APIC backing page of the vCPU,
re-evaluates interrupt state and injects an interrupt if it can be taken
right-away. All other APIC_ICR writes result in a #VC, and the guest
software needs to take care of delivering the IPI to the target vCPU(s).

Guests can choose to handle IPIs in one of two ways:
1. Delegate full IPI handling to the hypervisor by invoking SVM_EXIT_MSR
   VMGEXIT, in which case the guest will need to allow the IPI vector to
   be injected by the hypervisor by updating allowed_irr in the guest
   APIC backing page. Hypervisor follows regular interrupt injection
   flow for delivering the IPIs.
2. Handle updating APIC_IRR in the APIC backing page of the target
   vCPU(s), and rely on the hypervisor only to notify/wake up the target
   vCPU(s). In this scenario, hypervisor can use the AVIC doorbell for
   accelerated IPI delivery. This allows guest to forbid IPI injection
   from the untrusted hypervisor, and this is the model adopted by the
   Linux guest.

Note: for (2), since guest does _not_ set the IPI vector in allowed_irr
in its backing page, even if the hypervisor tries to inject it via
VMCB->requested_irr, hardware does not set these vectors in the guest
APIC_IRR. This ensures that the hypervisor can never inject an IPI into
the guest.

(1) is handled by the stock interrupt delivery flow:
  kvm_lapic_set_irr()
   \-->  KVM_REQ_EVENT, kvm_vcpu_kick()
          \--> VMCB->requested_irr/update_irr
		\--> VMRUN

For (2), guest updates APIC_IRR in the APIC backing page of the target
vCPU(s) and issues AVIC_INCOMPLETE_IPI VMGEXIT requesting KVM to notify
the target vCPU(s). Wire up SEV code to enable and process
AVIC_INCOMPLETE_IPI as a valid exit code for VMGEXIT. Update AVIC
incomplete ipi handling to accept and process unaccelerated Secure AVIC
exit ID similar to SVM AVIC handling for TARGET_NOT_RUNNING.

For AVIC_INCOMPLETE_IPI exits from a Secure AVIC guest, KVM can use an
AVIC doorbell to notify the physical CPU running the target vCPU. This
results in Secure AVIC hardware re-evaluating the private guest APIC
backing page and delivering pending interrupts to the vCPU. This is
distinct from device interrupts being injected by KVM, which require
injection through VMCB. To distinguish the two, update
svm_complete_interrupt_delivery() to only kick the target vCPU if there
is an _injectable_ interrupt pending in KVM's APIC_IRR. Otherwise, take
the SVM AVIC path and send an AVIC doorbell or wake up the vCPU if it
was blocking.

Ensuring IPI Delivery:
=====================
A problem unique to IPI handling described in (2) is KVM's complete lack
of visibility into whether the IPI has been processed by the target vCPU
or not. As an example, if the target vCPU is exiting, the source vCPU
may observe it to still be IN_GUEST_MODE (between #VMEXIT and when KVM
actually updates guest vCPU mode) and choose to send an AVIC doorbell.
However, since the vCPU was in the process of exiting, the doorbell has
no effect and the source vCPU has no feedback on this.  Furthermore,
since KVM has no visibility into the guest private APIC backing page,
there is no way to know if the IPI was processed or not (or if there is
anything pending in APIC_IRR in the guest private APIC backing page). In
the absence of any other event, the target vCPU will exit and block
since protected_apic_has_interrupt() only checks KVM's APIC_IRR which in
this case won't have the IPI vector set.

Since there is no support from hardware to address this, add a flag in
vcpu_svm structure 'snp_savic_has_pending_ipi' to track pending IPIs for
a specific vCPU. Set this flag on the target vCPU in avic_kick_vcpu()
and update snp_protected_apic_has_interrupt() to check this flag for
pending IPIs - this ensures that KVM will never allow a vCPU to block if
it has pending IPIs. Clear this flag unconditionally before VMRUN (in
pre_sev_run()) since VM entry guarantees that the guest private APIC
backing page will be re-evaluated and pending interrupts queued.

Use smp_mb() in avic_kick_vcpu() and snp_protected_apic_has_interrupt()
to order accesses to vcpu->mode and snp_savic_has_pending_ipi
(load-store ordering):

Source vCPU (AVIC_INCOMPLETE_IPI):
  avic_kick_vcpu():
    WRITE_ONCE(snp_savic_has_pending_ipi, 1);
    smp_mb();
    svm_complete_interrupt_delivery():
      READ_ONCE(vcpu->mode);

Target vCPU:
  vcpu_enter_guest():
    vcpu->mode = OUTSIDE_GUEST_MODE;
  ...
  kvm_cpu_has_interrupt():
    snp_protected_apic_has_interrupt():
      smp_mb();
      READ_ONCE(snp_savic_has_pending_ipi);

An unfortunate effect of this is that the guest will see a spurious
wakeup from idle. Irrespective of whether the AVIC doorbell resulted in
the IPI being delivered, KVM now forces an additional VMRUN to clear the
flag snp_savic_has_pending_ipi and this results in a spurious wakeup.
This should not be a functional issue for well-behaved guests.

Co-developed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/svm.h  |  1 +
 arch/x86/kvm/svm/avic.c | 13 +++++++++++++
 arch/x86/kvm/svm/sev.c  | 14 ++++++++++++--
 arch/x86/kvm/svm/svm.c  |  4 +++-
 4 files changed, 29 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 44f1d25a167c..9da6a2da6592 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -368,6 +368,7 @@ struct vcpu_svm {
 	/* Guest GIF value, used when vGIF is not enabled */
 	bool guest_gif;
 
+	bool snp_savic_has_pending_ipi;
 	gpa_t snp_savic_gpa;
 };
 
diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index 6b3983d4f45e..e3758c054783 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -536,6 +536,18 @@ void avic_ring_doorbell(struct kvm_vcpu *vcpu)
 
 static void avic_kick_vcpu(struct kvm_vcpu *vcpu, u32 icrl)
 {
+	if (snp_is_secure_avic_enabled(vcpu->kvm)) {
+		WRITE_ONCE(to_svm(vcpu)->snp_savic_has_pending_ipi, true);
+
+		/*
+		 * Ensure write to snp_savic_has_pending_ipi is visible before the
+		 * subsequent vcpu->mode read in svm_complete_interrupt_delivery().
+		 *
+		 * Pairs with smp_mb() in snp_protected_apic_has_interrupt().
+		 */
+		smp_mb();
+	}
+
 	vcpu->arch.apic->irr_pending = true;
 	svm_complete_interrupt_delivery(vcpu,
 					icrl & APIC_MODE_MASK,
@@ -716,6 +728,7 @@ int avic_incomplete_ipi_interception(struct kvm_vcpu *vcpu)
 			kvm_apic_send_ipi(apic, icrl, icrh);
 		break;
 	case AVIC_IPI_FAILURE_TARGET_NOT_RUNNING:
+	case AVIC_IPI_FAILURE_UNACCELERATED:
 		/*
 		 * At this point, we expect that the AVIC HW has already
 		 * set the appropriate IRR bits on the valid target
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 754fe12c2f82..06555dbec45b 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3580,6 +3580,9 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu)
 	if (!cpumask_test_cpu(cpu, to_kvm_sev_info(kvm)->have_run_cpus))
 		cpumask_set_cpu(cpu, to_kvm_sev_info(kvm)->have_run_cpus);
 
+	if (snp_is_secure_avic_enabled(kvm))
+		WRITE_ONCE(svm->snp_savic_has_pending_ipi, false);
+
 	/* Assign the asid allocated with this SEV guest */
 	svm->asid = asid;
 
@@ -4514,6 +4517,7 @@ static bool is_snp_only_vmgexit(u64 exit_code)
 	case SVM_VMGEXIT_EXT_GUEST_REQUEST:
 	case SVM_VMGEXIT_PSC:
 	case SVM_VMGEXIT_SAVIC:
+	case SVM_EXIT_AVIC_INCOMPLETE_IPI:
 		return true;
 	default:
 		return false;
@@ -4577,7 +4581,9 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
 		return 1;
 	}
 
-	if (control->exit_code == SVM_VMGEXIT_SAVIC && !snp_is_secure_avic_enabled(vcpu->kvm)) {
+	if ((control->exit_code == SVM_VMGEXIT_SAVIC ||
+	     control->exit_code == SVM_EXIT_AVIC_INCOMPLETE_IPI) &&
+	    !snp_is_secure_avic_enabled(vcpu->kvm)) {
 		vcpu_unimpl(vcpu, "vmgexit: exit code %#llx is only valid if Secure AVIC is enabled\n",
 			    control->exit_code);
 		svm_vmgexit_bad_input(svm, GHCB_ERR_INVALID_EVENT);
@@ -4616,6 +4622,7 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
 	case SVM_EXIT_WBINVD:
 	case SVM_EXIT_MONITOR:
 	case SVM_EXIT_MWAIT:
+	case SVM_EXIT_AVIC_INCOMPLETE_IPI:
 		return svm_invoke_exit_handler(vcpu, control->exit_code);
 	case SVM_VMGEXIT_MMIO_READ:
 	case SVM_VMGEXIT_MMIO_WRITE: {
@@ -4976,7 +4983,10 @@ bool snp_protected_apic_has_injectable_intr(struct kvm_vcpu *vcpu)
 
 bool snp_protected_apic_has_interrupt(struct kvm_vcpu *vcpu)
 {
-	return snp_protected_apic_has_injectable_intr(vcpu);
+	/* Pairs with smp_mb() in avic_kick_vcpu() */
+	smp_mb();
+	return snp_protected_apic_has_injectable_intr(vcpu) ||
+	       READ_ONCE(to_svm(vcpu)->snp_savic_has_pending_ipi);
 }
 
 void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 612cc4ac9bd2..63ee36501383 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3901,7 +3901,9 @@ void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_mode,
 	bool in_guest_mode = (smp_load_acquire(&vcpu->mode) == IN_GUEST_MODE);
 
 	/* Note, this is called iff the local APIC is in-kernel. */
-	if (!READ_ONCE(vcpu->arch.apic->apicv_active) || snp_is_secure_avic_enabled(vcpu->kvm)) {
+	if (!READ_ONCE(vcpu->arch.apic->apicv_active) ||
+	    (snp_is_secure_avic_enabled(vcpu->kvm) &&
+	     snp_protected_apic_has_injectable_intr(vcpu))) {
 		/* Process the interrupt via kvm_check_and_inject_events(). */
 		kvm_make_request(KVM_REQ_EVENT, vcpu);
 		kvm_vcpu_kick(vcpu);
-- 
2.54.0

[RFC PATCH v3 23/27] KVM: SVM: Add support for injecting NMIs for Secure AVIC guests

[Naveen N Rao (AMD)]

Secure AVIC relies on NMI virtualization for injecting NMIs into the
guest. Similar to V_GIF, set V_NMI_ENABLE_MASK in the BSP's VMSA as this
is consulted by hardware instead of the corresponding VMCB field.

Since EVENTINJ is not supported for Secure AVIC enabled guests, KVM can
accept/pend at most 1 NMI at any point. Use this as the limit in
process_nmi(). Add a WARN_ON() in svm_inject_nmi() since we should never
hit this path. Always return 1 for nmi_allowed() since KVM can pend an
NMI at any point and does not have visibility into the guest NMI
blocking status.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/sev.c | 2 +-
 arch/x86/kvm/svm/svm.c | 8 ++++++++
 arch/x86/kvm/x86.c     | 8 +++++++-
 3 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 06555dbec45b..707537ad7271 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -1034,7 +1034,7 @@ static int sev_es_sync_vmsa(struct vcpu_svm *svm)
 
 	/* Secure AVIC loads the below from the VMSA, rather than the VMCB */
 	if (snp_is_secure_avic_enabled(vcpu->kvm))
-		save->vintr_ctrl |= V_GIF_MASK;
+		save->vintr_ctrl |= V_GIF_MASK | V_NMI_ENABLE_MASK;
 
 	/*
 	 * Skip FPU and AVX setup with KVM_SEV_ES_INIT to avoid
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 63ee36501383..2cb38953c0cf 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -3768,6 +3768,10 @@ static void svm_inject_nmi(struct kvm_vcpu *vcpu)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 
+	/* We should never reach here for Secure AVIC - see svm_set_vnmi_pending() */
+	if (WARN_ON_ONCE(snp_is_secure_avic_enabled(vcpu->kvm)))
+		return;
+
 	svm->vmcb->control.event_inj = SVM_EVTINJ_VALID | SVM_EVTINJ_TYPE_NMI;
 
 	if (svm->nmi_l1_to_l2)
@@ -4018,6 +4022,10 @@ static int svm_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection)
 	if (vcpu->arch.nested_run_pending)
 		return -EBUSY;
 
+	/* Assume it is always allowed for Secure AVIC (hardware gates delivery) */
+	if (snp_is_secure_avic_enabled(vcpu->kvm))
+		return 1;
+
 	if (svm_nmi_blocked(vcpu))
 		return 0;
 
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 2609a2972526..7f5978cac5f5 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -7742,8 +7742,14 @@ static void process_nmi(struct kvm_vcpu *vcpu)
 	 * waiting for a previous NMI injection to complete (which effectively
 	 * blocks NMIs).  KVM will immediately inject one of the two NMIs, and
 	 * will request an NMI window to handle the second NMI.
+	 *
+	 * For protected APIC guests, NMI window isn't visible to KVM, so the
+	 * only thing KVM can do is to collapse all pending NMIs and inject a
+	 * single NMI. The guest is expected to scan all NMI sources as part
+	 * of handling the NMI.
 	 */
-	if (kvm_x86_call(get_nmi_mask)(vcpu) || vcpu->arch.nmi_injected)
+	if (kvm_x86_call(get_nmi_mask)(vcpu) || vcpu->arch.nmi_injected ||
+	    (lapic_in_kernel(vcpu) && vcpu->arch.apic->guest_apic_protected))
 		limit = 1;
 	else
 		limit = 2;
-- 
2.54.0

[RFC PATCH v3 24/27] KVM: SVM: Mandate use of split irqchip for Secure AVIC

[Naveen N Rao (AMD)]

Do not support in-kernel KVM IOAPIC if Secure AVIC is enabled. There are
two primary reasons (besides aligning with TDX):
1. KVM IOAPIC brings in PIC with no way to disable that. We can't
   support the PIC since there is no way to inject ExtINT.
2. KVM PIT in its default reinject mode can't be supported due to EOI
   acceleration (just like AVIC), but Secure AVIC can't be inhibited.

The latter is not an issue with userspace IOAPIC/PIT (reinject mode is
unique to the KVM PIT). For the former, it is up to the VMM to ensure
PIC is not advertised if Secure AVIC is enabled.

Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/avic.c | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c
index e3758c054783..525fa8834277 100644
--- a/arch/x86/kvm/svm/avic.c
+++ b/arch/x86/kvm/svm/avic.c
@@ -938,8 +938,17 @@ int avic_init_vcpu(struct vcpu_svm *svm)
 	if (!enable_apicv || !irqchip_in_kernel(vcpu->kvm))
 		return 0;
 
-	if (snp_is_secure_avic_enabled(vcpu->kvm))
+	if (snp_is_secure_avic_enabled(vcpu->kvm)) {
+		/*
+		 * Secure AVIC can't work with the legacy PIC since there is no
+		 * way to inject ExtINT. Besides, we can't inhibit Secure AVIC,
+		 * so we can't really support KVM PIT in reinject mode.
+		 */
+		if (!irqchip_split(vcpu->kvm))
+			return -EINVAL;
+
 		vcpu->arch.apic->guest_apic_protected = true;
+	}
 
 	ret = avic_init_backing_page(vcpu);
 	if (ret)
-- 
2.54.0

[RFC PATCH v3 25/27] KVM: SVM: Do not inject exceptions for Secure AVIC

[Naveen N Rao (AMD)]

From: Kishon Vijay Abraham I <kvijayab@amd.com>

In Secure AVIC mode, EVENTINJ in the VMCB is ignored which means
there is no way to inject exceptions from KVM. Return early from
svm_inject_exception() for a Secure AVIC enabled guest.

Note that exceptions generated within the guest themselves are still
handled properly if intercepted (e.g., INTR just before a guest
exception), since those are saved in the VMSA EVENTINJ field and
processed by the hardware.

Signed-off-by: Kishon Vijay Abraham I <kvijayab@amd.com>
Co-developed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Co-developed-by: Naveen N Rao (AMD) <naveen@kernel.org>
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/svm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 2cb38953c0cf..2e32670ff957 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -391,6 +391,10 @@ static void svm_inject_exception(struct kvm_vcpu *vcpu)
 	struct kvm_queued_exception *ex = &vcpu->arch.exception;
 	struct vcpu_svm *svm = to_svm(vcpu);
 
+	/* Secure AVIC does not support EVENTINJ */
+	if (snp_is_secure_avic_enabled(vcpu->kvm))
+		return;
+
 	kvm_deliver_exception_payload(vcpu, ex);
 
 	if (kvm_exception_is_soft(ex->vector) &&
-- 
2.54.0

[RFC PATCH v3 26/27] KVM: SVM: Do not intercept exceptions for Secure AVIC guests

[Naveen N Rao (AMD)]

From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>

In Secure AVIC mode, exceptions cannot be injected by KVM as the VMCB
EVENTINJ field is ignored. As such, do not intercept exceptions since
KVM will be unable to re-inject those back into the guest. The only
exception is #MC since that needs to be handled in the host, so
explicitly force-enable #MC interception.

Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Co-developed-by: Naveen N Rao (AMD) <naveen@kernel.org>
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/sev.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 707537ad7271..7c9c25135df8 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4862,6 +4862,22 @@ static void sev_es_init_vmcb(struct vcpu_svm *svm, bool init_event)
 	/* Can't intercept XSETBV, HV can't modify XCR0 directly */
 	svm_clr_intercept(svm, INTERCEPT_XSETBV);
 
+	if (snp_is_secure_avic_enabled(svm->vcpu.kvm)) {
+		/* Clear all exception intercepts since we can't inject those */
+		for (int i = 0; i < NUM_EXCEPTION_VECTORS; i++)
+			clr_exception_intercept(svm, i);
+
+		/*
+		 * Note that #MC is always intercepted by hardware in Secure
+		 * AVIC mode, so mark #MC as intercepted to stay consistent
+		 * with the hardware behavior. From the APM:
+		 *   "In Secure AVIC mode hardware treats physical INTR, NMI,
+		 *    INIT, and #MC events as intercepted regardless of the
+		 *    corresponding intercept bit values in the VMCB."
+		 */
+		set_exception_intercept(svm, MC_VECTOR);
+	}
+
 	/*
 	 * Set the GHCB MSR value as per the GHCB specification when emulating
 	 * vCPU RESET for an SEV-ES guest.
-- 
2.54.0

[RFC PATCH v3 27/27] KVM: SVM: Advertise Secure AVIC support for SEV-SNP guests

[Naveen N Rao (AMD)]

Add Secure AVIC to sev_supported_vmsa_features allowing VMM to enable
Secure AVIC for SEV-SNP guests. Gate the same on all dependencies for
Secure AVIC being satisfied:
- X86_FEATURE_SNP_SECURE_AVIC itself signaling platform support
- X86_FEATURE_IDLE_HLT, more on this below
- 'enable_apicv' for dependency on AVIC being enabled
- 'vnmi' so that NMIs can be injected. And the implied dependency of
  vnmi on vgif so that VGIF is enabled for a Secure AVIC guest to unmask
  interrupts.

Secure AVIC has a hard dependency on idle hlt intercept since KVM has no
visibility into pending interrupts within the guest. If the guest took a
'#VMEXIT' just before 'sti; hlt' with interrupts disabled, and KVM
injected into the guest IRR (Requested_IRR --> guest vIRR), those won't
be deliverable before 'hlt' due to the interrupt shadow. On hlt
intercept, KVM won't be able to observe interrupts pending in the guest
APIC_IRR, KVM's APIC shadow backing page has nothing pending and so will
block the vCPU. Idle hlt intercept addresses this by preventing the
intercept from firing so that the vCPU proceeds to take those
interrupts.

Because 'enable_apicv' and 'vnmi' are new dependencies, move the call to
sev_hardware_setup() after they have been setup in svm_hardware_setup().

Co-developed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Naveen N Rao (AMD) <naveen@kernel.org>
---
 arch/x86/kvm/svm/sev.c | 13 ++++++++++---
 arch/x86/kvm/svm/svm.c | 17 +++++++++--------
 2 files changed, 19 insertions(+), 11 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 7c9c25135df8..b67cf1399cbf 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -512,7 +512,7 @@ static int __sev_guest_init(struct kvm *kvm, struct kvm_sev_cmd *argp,
 		return -EINVAL;
 
 	if (!snp_active)
-		valid_vmsa_features &= ~SVM_SEV_FEAT_SECURE_TSC;
+		valid_vmsa_features &= ~(SVM_SEV_FEAT_SECURE_TSC | SVM_SEV_FEAT_SECURE_AVIC);
 
 	if (data->vmsa_features & ~valid_vmsa_features)
 		return -EINVAL;
@@ -3234,8 +3234,15 @@ void __init sev_hardware_setup(void)
 	    cpu_feature_enabled(X86_FEATURE_NO_NESTED_DATA_BP))
 		sev_supported_vmsa_features |= SVM_SEV_FEAT_DEBUG_SWAP;
 
-	if (sev_snp_enabled && tsc_khz && cpu_feature_enabled(X86_FEATURE_SNP_SECURE_TSC))
-		sev_supported_vmsa_features |= SVM_SEV_FEAT_SECURE_TSC;
+	if (sev_snp_enabled) {
+		if (tsc_khz && cpu_feature_enabled(X86_FEATURE_SNP_SECURE_TSC))
+			sev_supported_vmsa_features |= SVM_SEV_FEAT_SECURE_TSC;
+
+		if (cpu_feature_enabled(X86_FEATURE_SNP_SECURE_AVIC) &&
+		    cpu_feature_enabled(X86_FEATURE_IDLE_HLT) &&
+		    enable_apicv && vnmi)
+			sev_supported_vmsa_features |= SVM_SEV_FEAT_SECURE_AVIC;
+	}
 }
 
 void sev_hardware_unsetup(void)
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 2e32670ff957..a20f0abe138b 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5731,14 +5731,6 @@ static __init int svm_hardware_setup(void)
 
 	iopm_base = __sme_set(__pa(iopm_va));
 
-	/*
-	 * Note, SEV setup consumes npt_enabled and enable_mmio_caching (which
-	 * may be modified by svm_adjust_mmio_mask()), as well as nrips.
-	 */
-	sev_hardware_setup();
-
-	svm_hv_hardware_setup();
-
 	enable_apicv = avic_hardware_setup();
 	if (!enable_apicv) {
 		enable_ipiv = false;
@@ -5781,6 +5773,15 @@ static __init int svm_hardware_setup(void)
 	if (!enable_pmu)
 		pr_info("PMU virtualization is disabled\n");
 
+	/*
+	 * Note, SEV setup consumes npt_enabled and enable_mmio_caching (which
+	 * may be modified by svm_adjust_mmio_mask()), as well as nrips, vnmi
+	 * and enable_apicv.
+	 */
+	sev_hardware_setup();
+
+	svm_hv_hardware_setup();
+
 	svm_set_cpu_caps();
 
 	kvm_caps.inapplicable_quirks &= ~KVM_X86_QUIRK_CD_NW_CLEARED;
-- 
2.54.0

Re: [RFC PATCH v3 00/27] KVM: SVM: Add support for SEV-SNP Secure AVIC

[Naveen N Rao]

On Wed, Jul 08, 2026 at 12:01:58PM +0530, Naveen N Rao (AMD) wrote:
> This is RFC v3 of the patches adding KVM support for Secure AVIC. RFC v2:
> http://lore.kernel.org/r/20250923050317.205482-1-Neeraj.Upadhyay@amd.com
> 
> There are significant changes throughout the series since the last RFC, 
> with some of the highlights being:
> - Integrate with AVIC code since Secure AVIC is APICv, but for protected 
>   APIC guests. This aligns with TDX which also requires APICv and 
>   enables reuse of APICv code paths in KVM. Patches #8 to #14.
> - Mandate use of split irqchip, and drop changes related to the KVM 
>   IOAPIC. KVM PIC/PIT can't be supported (described more fully in the 
>   commit log of the relevant patches). Patch #24.
> - Enable use of SVM_AVIC_INCOMPLETE_IPI VMGEXIT for vCPU notification 
>   request from the guest. Patch #22.
> - Utilize vNMI injection path for NMI handling. Patch #23.
> - Introduce a new kvm_x86_op to distinguish need for injectable 
>   interrupts vs. IPI notifications. Patches #18 and #19.
> - Retain KVM PV ops, and instead require guests to opt-in/disable.  
>   Patch #3.
> 
> Patches #1 to #4 are changes to the guest x2APIC Secure AVIC driver.

I missed mentioning one of the main TODOs on which I am looking for 
suggestions on: handling SVM_VMGEXIT_SAVIC_REGISTER_GPA and ensuring 
that Secure AVIC backing page accesses by the guest do not cause #NPF.  
Patch #16 has the details.

And it looks like I also goofed up Neeraj's email address - sorry! (Now 
fixed)



- Naveen

Re: [RFC PATCH v3 01/27] x86/apic: Propagate APIC_SPIV writes to hv for Secure AVIC

[Borislav Petkov]

On Wed, Jul 08, 2026 at 12:01:59PM +0530, Naveen N Rao (AMD) wrote:
> From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
> 
> Hypervisor needs to be aware of the VAPIC's APIC_SPIV state for
> LVT masking/unmasking. So, propagate the guest's APIC_SPIV
> writes to the hypervisor.
> 
> Fixes: c822f58a4fab ("x86/apic: Populate .read()/.write() callbacks of Secure AVIC driver")

What is that patch actually fixing?

Does it need to go to stable?

SPIV needs to be masked and unmasked by the guest?

Doesn't seem too earth-shattering to me but what do I know...

> Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>

verify_tags: WARNING: Sender ["Naveen N Rao (AMD)" <naveen@kernel.org>] hasn't signed off on the patch!

When you send someone else's patch, you *must* put you SOB after hers/his.

Thx.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Re: [RFC PATCH v3 01/27] x86/apic: Propagate APIC_SPIV writes to hv for Secure AVIC

[Naveen N Rao]

On Thu, Jul 09, 2026 at 07:03:49PM -0700, Borislav Petkov wrote:
> On Wed, Jul 08, 2026 at 12:01:59PM +0530, Naveen N Rao (AMD) wrote:
> > From: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
> > 
> > Hypervisor needs to be aware of the VAPIC's APIC_SPIV state for
> > LVT masking/unmasking. So, propagate the guest's APIC_SPIV
> > writes to the hypervisor.
> > 
> > Fixes: c822f58a4fab ("x86/apic: Populate .read()/.write() callbacks of Secure AVIC driver")
> 
> What is that patch actually fixing?

Guest doesn't boot without this change.

> 
> Does it need to go to stable?

Yes. I will tag stable on this.

> 
> SPIV needs to be masked and unmasked by the guest?
> 
> Doesn't seem too earth-shattering to me but what do I know...

APIC_SPIV has a bit to (soft-)disable the APIC. It starts off disabled 
and the kernel has to write to it to enable the APIC. If writes from the 
guest are not propagated to KVM, KVM does not deliver interrupts to the 
guest since the APIC is disabled as far as it is concerned.

I will update the commit log with these details.

> 
> > Signed-off-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
> 
> verify_tags: WARNING: Sender ["Naveen N Rao (AMD)" <naveen@kernel.org>] hasn't signed off on the patch!
> 
> When you send someone else's patch, you *must* put you SOB after hers/his.

I was genuinely unaware of this. Just so I get this right:
- Add just the 'Sob:' for patches from others that I post un-modified 
  (without a Co-developed-by: tag)
- Add 'Co-developed-by:' if I make any changes to the patch (commit log 
  included?)


Thanks,
Naveen

Re: [RFC PATCH v3 01/27] x86/apic: Propagate APIC_SPIV writes to hv for Secure AVIC

[Borislav Petkov]

On Fri, Jul 10, 2026 at 08:32:23PM +0530, Naveen N Rao wrote:
> Guest doesn't boot without this change.

Then give the commit message a second try and explain exactly the situation.

> APIC_SPIV has a bit to (soft-)disable the APIC. It starts off disabled 
> and the kernel has to write to it to enable the APIC. If writes from the 
> guest are not propagated to KVM, KVM does not deliver interrupts to the 
> guest since the APIC is disabled as far as it is concerned.
> 
> I will update the commit log with these details.

Oh yeah!

> I was genuinely unaware of this. Just so I get this right:
> - Add just the 'Sob:' for patches from others that I post un-modified 
>   (without a Co-developed-by: tag)

Yes, if you're handling the patch - it passes through you - SOB.

> - Add 'Co-developed-by:' if I make any changes to the patch (commit log 
>   included?)

Co-developed is when you've put a more serious effort into the patch than just
touch ups, I'd say.

But we have it all documented:

https://www.kernel.org/doc/html/latest/process/submitting-patches.html

especially:

https://www.kernel.org/doc/html/latest/process/submitting-patches.html#when-to-use-acked-by-cc-and-co-developed-by

which answers your questions exactly.

-- 
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette