[PATCH v4 00/18] KVM: SEV: Fix RMP #PF due to freeing in-use VMSA

[PATCH v4 00/18] KVM: SEV: Fix RMP #PF due to freeing in-use VMSA

[Sean Christopherson]

Rework KVM's handling of guest-provided (and always guest_memfd-backed) VMSAs
to forcefully reclaim VMSA pages when the pages are being freed from their
backing gmem instance, e.g. in response to PUNCH_HOLE.  In the worst case
scenario, marking the page SHARED in the RMP will fail due to the page being
IN_USE, ultimately leading to RMP #PF violations due to guest_memfd freeing
the memory back to the kernel while it's still assigned to a VM.

Note, the implementation nearly identical to that used by KVM for VMX's APIC
access page (which isn't guest controlled, but is migratable and whose PA is
shoved directly into a vCPU control structure).

On the backend of the series, rework prepare()+invalidate() into a single
convert() hook, to eliminate the misleading use of the term "invalidate", to
(obviously) drop yet another callback, and to prepare for in-place conversion,
e.g. so that KVM ends up with something like:

	if (!to_private)
		kvm_gmem_make_shared(inode, start, end);

instead of a funky:

	if (!to_private)
		kvm_gmem_invalidate(inode, start, end);

v4:
 - Collect reviews. [Mike, Ackerley]
 - Rework prepare()+invalidate() into a common convert() instead of simply
   renaming invalidate() to reclaim_memory(). [Ackerley]
 - Fix several (mostly benign) issues in the "prepare" flow.

v3:
 - https://lore.kernel.org/all/20260630222607.497895-1-seanjc@google.com
 - Ensure disabling quirks while the VM is live won't result in KVM skipping
   the back-half of "zap all fast". [Sashiko]
 - s/gmem_free_folio/gmem_reclaim_memory. [Ackerley]
 - Use {READ,WRITE}_ONCE() when accessing the guest's VMSA GPA outside of the
   per-vCPU mutex, and comment. [Sashiko]

v2:
 - https://lore.kernel.org/all/20260626231416.3943216-1-seanjc@google.com
 - Invalidate VMSAs if the memslot is DELETED or MOVED. [Sashiko]
 - Limit stable@ patches without a Fixes to 6.12+.

v1: https://lore.kernel.org/all/20260625222229.3367197-2-seanjc@google.com

Sean Christopherson (18):
  KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP
    guests
  KVM: SEV: Extract loading of guest-provided VMSA to a separate helper
  KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable
  KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if
    CONFIG_KVM_AMD_SEV=y
  KVM: x86: Serialize writes to disabled_quirks using kvm->lock
  KVM: x86: Ensure runtime reads of disabled_quirks are resolved once
  KVM: x86/mmu: Fold kvm_mmu_zap_memslot() into
    kvm_arch_flush_shadow_memslot()
  KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back"
    halves
  KVM: x86/mmu: Use split "zap all fast" helpers when invalidating
    memslot
  KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is
    zapped
  KVM: SEV: Mark vCPU has having guest-provided VMSA even if its invalid
  KVM: x86: Guard .gmem_prepare() declarations with
    HAVE_KVM_GMEM_PREPARE=y
  KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook
  KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare()
    failure
  KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller
  KVM: guest_memfd: Explicitly pass number of pages to
    kvm_arch_gmem_prepare()
  KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a
    folio
  KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into
    .gmem_convert()

 arch/x86/include/asm/kvm-x86-ops.h |   9 +-
 arch/x86/include/asm/kvm_host.h    |  12 +-
 arch/x86/kvm/Kconfig               |   2 +-
 arch/x86/kvm/mmu/mmu.c             |  74 +++++++-----
 arch/x86/kvm/svm/sev.c             | 186 ++++++++++++++++++++++-------
 arch/x86/kvm/svm/svm.c             |   9 +-
 arch/x86/kvm/svm/svm.h             |  19 +--
 arch/x86/kvm/x86.c                 |  21 ++--
 arch/x86/kvm/x86.h                 |   2 +-
 include/linux/kvm_host.h           |   7 +-
 virt/kvm/Kconfig                   |   2 +-
 virt/kvm/guest_memfd.c             |  53 ++++----
 12 files changed, 256 insertions(+), 140 deletions(-)


base-commit: a204badd8432f93b7e862e7dac6db0fe3d65f370
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 01/18] KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests

[Sean Christopherson]

Track the GPA of the guest-provided VMSA used after AP_CREATION events when
running SNP guests, instead of simply tracking whether or not the vCPU is
using a guest-provided VMSA.  KVM needs to know the GPA of the VMSA that's
actively being used so that it can react to MMU invalidation events, i.e.
so that KVM can drop the VMSA if its backing guest_memfd page is punched
out of existence.

Opportunistically rename snp_vmsa_gpa to clarify that it tracks the pending
VMSA GPA, whereas snp_guest_vmsa_gpa now tracks the in-use VMSA GPA.

Note!  Take care to track the GPA, not the GFN, as VALID_PAGE() won't
behave correctly if an invalid GFN is converted to a GPA for checking.

Note #2!  Keep snp_has_guest_vmsa so that switching to a guest-provided
VMSA is sticky, even if the guest-provided VMSA becomes invalid.

No functional change intended.

Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/sev.c | 14 +++++++++-----
 arch/x86/kvm/svm/svm.h |  3 ++-
 2 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 74fb15551e83..827f5dc06102 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4003,6 +4003,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 
 	/* Clear use of the VMSA */
 	svm->vmcb->control.vmsa_pa = INVALID_PAGE;
+	svm->sev_es.snp_guest_vmsa_gpa = INVALID_PAGE;
 
 	/*
 	 * When replacing the VMSA during SEV-SNP AP creation,
@@ -4010,11 +4011,11 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 	 */
 	vmcb_mark_all_dirty(svm->vmcb);
 
-	if (!VALID_PAGE(svm->sev_es.snp_vmsa_gpa))
+	if (!VALID_PAGE(svm->sev_es.snp_pending_vmsa_gpa))
 		return;
 
-	gfn = gpa_to_gfn(svm->sev_es.snp_vmsa_gpa);
-	svm->sev_es.snp_vmsa_gpa = INVALID_PAGE;
+	gfn = gpa_to_gfn(svm->sev_es.snp_pending_vmsa_gpa);
+	svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
 
 	slot = gfn_to_memslot(vcpu->kvm, gfn);
 	if (!slot)
@@ -4039,6 +4040,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 	svm->sev_es.snp_has_guest_vmsa = true;
 
 	/* Use the new VMSA */
+	svm->sev_es.snp_guest_vmsa_gpa = gfn_to_gpa(gfn);
 	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
 
 	/* Mark the vCPU as runnable */
@@ -4105,10 +4107,10 @@ static int sev_snp_ap_creation(struct vcpu_svm *svm)
 			return -EINVAL;
 		}
 
-		target_svm->sev_es.snp_vmsa_gpa = svm->vmcb->control.exit_info_2;
+		target_svm->sev_es.snp_pending_vmsa_gpa = svm->vmcb->control.exit_info_2;
 		break;
 	case SVM_VMGEXIT_AP_DESTROY:
-		target_svm->sev_es.snp_vmsa_gpa = INVALID_PAGE;
+		target_svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
 		break;
 	default:
 		vcpu_unimpl(vcpu, "vmgexit: invalid AP creation request [%#x] from guest\n",
@@ -4791,6 +4793,8 @@ int sev_vcpu_create(struct kvm_vcpu *vcpu)
 		return -ENOMEM;
 
 	svm->sev_es.vmsa = page_address(vmsa_page);
+	svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
+	svm->sev_es.snp_guest_vmsa_gpa = INVALID_PAGE;
 
 	vcpu->arch.guest_tsc_protected = snp_is_secure_tsc_enabled(vcpu->kvm);
 
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 716be21fba33..d077783c287e 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -271,7 +271,8 @@ struct vcpu_sev_es_state {
 	u64 ghcb_registered_gpa;
 
 	struct mutex snp_vmsa_mutex; /* Used to handle concurrent updates of VMSA. */
-	gpa_t snp_vmsa_gpa;
+	gpa_t snp_pending_vmsa_gpa;
+	gpa_t snp_guest_vmsa_gpa;
 	bool snp_ap_waiting_for_reset;
 	bool snp_has_guest_vmsa;
 };
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 02/18] KVM: SEV: Extract loading of guest-provided VMSA to a separate helper

[Sean Christopherson]

Extract the loading/retrieval of a guest-provided VMSA to a separate helper
so that KVM can reuse the core logic when refreshing the VMSA after an MMU
invalidation from guest_memfd.

No functional change intended.

Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/sev.c | 52 +++++++++++++++++++++++++-----------------
 1 file changed, 31 insertions(+), 21 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 827f5dc06102..d8ed00f76aa3 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3979,29 +3979,17 @@ static int snp_begin_psc(struct vcpu_svm *svm)
 	return snp_do_psc(svm);
 }
 
-/*
- * Invoked as part of svm_vcpu_reset() processing of an init event.
- */
-static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
+static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_memory_slot *slot;
+	gfn_t gfn = gpa_to_gfn(gpa);
 	struct page *page;
 	kvm_pfn_t pfn;
-	gfn_t gfn;
 
-	guard(mutex)(&svm->sev_es.snp_vmsa_mutex);
+	lockdep_assert_held(&svm->sev_es.snp_vmsa_mutex);
 
-	if (!svm->sev_es.snp_ap_waiting_for_reset)
-		return;
-
-	svm->sev_es.snp_ap_waiting_for_reset = false;
-
-	/* Mark the vCPU as offline and not runnable */
-	vcpu->arch.pv.pv_unhalted = false;
-	kvm_set_mp_state(vcpu, KVM_MP_STATE_HALTED);
-
-	/* Clear use of the VMSA */
+	/* Clear use of the VMSA. */
 	svm->vmcb->control.vmsa_pa = INVALID_PAGE;
 	svm->sev_es.snp_guest_vmsa_gpa = INVALID_PAGE;
 
@@ -4011,12 +3999,9 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 	 */
 	vmcb_mark_all_dirty(svm->vmcb);
 
-	if (!VALID_PAGE(svm->sev_es.snp_pending_vmsa_gpa))
+	if (!VALID_PAGE(gpa))
 		return;
 
-	gfn = gpa_to_gfn(svm->sev_es.snp_pending_vmsa_gpa);
-	svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
-
 	slot = gfn_to_memslot(vcpu->kvm, gfn);
 	if (!slot)
 		return;
@@ -4040,7 +4025,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 	svm->sev_es.snp_has_guest_vmsa = true;
 
 	/* Use the new VMSA */
-	svm->sev_es.snp_guest_vmsa_gpa = gfn_to_gpa(gfn);
+	svm->sev_es.snp_guest_vmsa_gpa = gpa;
 	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
 
 	/* Mark the vCPU as runnable */
@@ -4054,6 +4039,31 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 	kvm_release_page_clean(page);
 }
 
+/*
+ * Invoked as part of svm_vcpu_reset() processing of an init event.
+ */
+static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_svm *svm = to_svm(vcpu);
+	gpa_t gpa;
+
+	guard(mutex)(&svm->sev_es.snp_vmsa_mutex);
+
+	if (!svm->sev_es.snp_ap_waiting_for_reset)
+		return;
+
+	svm->sev_es.snp_ap_waiting_for_reset = false;
+
+	/* Mark the vCPU as offline and not runnable */
+	vcpu->arch.pv.pv_unhalted = false;
+	kvm_set_mp_state(vcpu, KVM_MP_STATE_HALTED);
+
+	gpa = svm->sev_es.snp_pending_vmsa_gpa;
+	svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
+
+	sev_snp_reload_vmsa(vcpu, gpa);
+}
+
 static int sev_snp_ap_creation(struct vcpu_svm *svm)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(svm->vcpu.kvm);
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 03/18] KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable

[Sean Christopherson]

Always mark the vCPU as RUNNABLE after responding to AP_CREATE, even if the
guest-specified VMSA is unusable, e.g. isn't backed by a memslot or doesn't
have a backing guest_memfd page.  If the VMSA is unusable, leaving the vCPU
in a non-running state will effectively hang the vCPU instead of reporting
an error to userspace.  This will also allow retrying the VMSA load in the
future, to fix a bug where KVM doesn't honor guest_memfd invalidation
events, e.g. if AP_CREATION races with PUNCH_HOLE.

Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/sev.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index d8ed00f76aa3..30792adcfc8e 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4028,9 +4028,6 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	svm->sev_es.snp_guest_vmsa_gpa = gpa;
 	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
 
-	/* Mark the vCPU as runnable */
-	kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
-
 	/*
 	 * gmem pages aren't currently migratable, but if this ever changes
 	 * then care should be taken to ensure svm->sev_es.vmsa is pinned
@@ -4062,6 +4059,15 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 	svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
 
 	sev_snp_reload_vmsa(vcpu, gpa);
+
+	/*
+	 * Mark the vCPU as runnable for CREATE requests, indicated by a valid
+	 * VMSA GPA, even if installing the VMSA failed, so that KVM_RUN will
+	 * fail instead of blocking indefinitely and hanging the vCPU, e.g. if
+	 * the backing guest_memfd page is unavailable.
+	 */
+	if (VALID_PAGE(gpa))
+		kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
 }
 
 static int sev_snp_ap_creation(struct vcpu_svm *svm)
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 04/18] KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if CONFIG_KVM_AMD_SEV=y

[Sean Christopherson]

Wire up the SEV-SNP guest_memfd kvm_x86_ops hooks if and only if SEV is
actually enabled, and drop the now-unnecessary stubs.  Leaving the hooks
NULL allows the static call infrastructure to elide the CALL+RET, and more
importantly, referencing the hooks if and only if SEV support is enabled
will allow conditionally definining the hooks using their corresponding
HAVE_KVM_ARCH_GMEM_XXX Kconfig.

No functional change intended.

Cc: stable@vger.kernel.org # 6.12.x
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/svm.c |  8 ++++----
 arch/x86/kvm/svm/svm.h | 10 ----------
 2 files changed, 4 insertions(+), 14 deletions(-)

diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index ef69a51ab27f..79c818d91dda 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5448,6 +5448,10 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 
 	.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
 	.vm_move_enc_context_from = sev_vm_move_enc_context_from,
+
+	.gmem_prepare = sev_gmem_prepare,
+	.gmem_invalidate = sev_gmem_invalidate,
+	.gmem_max_mapping_level = sev_gmem_max_mapping_level,
 #endif
 	.check_emulate_instruction = svm_check_emulate_instruction,
 
@@ -5459,10 +5463,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector,
 	.vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons,
 	.alloc_apic_backing_page = svm_alloc_apic_backing_page,
-
-	.gmem_prepare = sev_gmem_prepare,
-	.gmem_invalidate = sev_gmem_invalidate,
-	.gmem_max_mapping_level = sev_gmem_max_mapping_level,
 };
 
 /*
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index d077783c287e..effca6372e15 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1035,16 +1035,6 @@ static inline int sev_cpu_init(struct svm_cpu_data *sd) { return 0; }
 static inline int sev_dev_get_attr(u32 group, u64 attr, u64 *val) { return -ENXIO; }
 #define max_sev_asid 0
 static inline void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) {}
-static inline int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
-{
-	return 0;
-}
-static inline void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end) {}
-static inline int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private)
-{
-	return 0;
-}
-
 static inline struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu)
 {
 	return NULL;
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 05/18] KVM: x86: Serialize writes to disabled_quirks using kvm->lock

[Sean Christopherson]

Protect writes to disabled_quirks with kvm->lock to ensure KVM doesn't
clobber state in the unlikely scenario that userspace disables disparate
quirks from multiple tasks.  More importantly, this will allow wrapping
accesses with {READ,WRITE}_ONCE without "needing" to also guard the writer
with a useless and confusing READ_ONCE (since the RMW wouldn't be atomic
anyways).

Ideally, KVM would disallow disabling quirks once quirks are "live", but
that would be a potentially breaking userspace ABI change, and while all
existing quirks are fully live only after vCPUs have been created, several
MMU-related quirks, IGNORE_GUEST_PAT and SLOT_ZAP_ALL, are partially live
at all times.  Because populating MMUs requires a vCPU, the guest-visible
behavior of IGNORE_GUEST_PAT and SLOT_ZAP_ALL requires a vCPU, but for KVM
itself, processing the quirk (or not) has functional impact, i.e. for all
intents and purposes, KVM can't prevent those quirks from being disabled
after they've been consumed.

Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/x86.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 0626e835e9eb..226c6cfe8062 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -3939,7 +3939,9 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 			break;
 		fallthrough;
 	case KVM_CAP_DISABLE_QUIRKS:
+		mutex_lock(&kvm->lock);
 		kvm->arch.disabled_quirks |= cap->args[0] & kvm_caps.supported_quirks;
+		mutex_unlock(&kvm->lock);
 		r = 0;
 		break;
 	case KVM_CAP_SPLIT_IRQCHIP: {
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 06/18] KVM: x86: Ensure runtime reads of disabled_quirks are resolved once

[Sean Christopherson]

Wrap the sole reader of disabled_quirks with READ_ONCE(), and wrap the
post-VM-creation write to disabled_quirks with WRITE_ONCE(), to ensure
checking the status of a quirk doesn't re-read disabled_quirks *if* the
caller needs such a guarantee.  This will allow splitting the "fast" MMU
zap into front and back halves, without potentially skipping the back
half if SLOT_ZAP_ALL were concurrently disabled (which would be "fine" in
the current code base, but far from ideal).

Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/x86.c | 3 ++-
 arch/x86/kvm/x86.h | 2 +-
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 226c6cfe8062..8abd733d5173 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -3940,7 +3940,8 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		fallthrough;
 	case KVM_CAP_DISABLE_QUIRKS:
 		mutex_lock(&kvm->lock);
-		kvm->arch.disabled_quirks |= cap->args[0] & kvm_caps.supported_quirks;
+		WRITE_ONCE(kvm->arch.disabled_quirks,
+			   kvm->arch.disabled_quirks | (cap->args[0] & kvm_caps.supported_quirks));
 		mutex_unlock(&kvm->lock);
 		r = 0;
 		break;
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index 8ece468087a8..75f13d88db58 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -304,7 +304,7 @@ static inline bool vcpu_match_mmio_gpa(struct kvm_vcpu *vcpu, gpa_t gpa)
 
 static inline bool kvm_check_has_quirk(struct kvm *kvm, u64 quirk)
 {
-	return !(kvm->arch.disabled_quirks & quirk);
+	return !(READ_ONCE(kvm->arch.disabled_quirks) & quirk);
 }
 
 static __always_inline void kvm_request_l1tf_flush_l1d(void)
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 07/18] KVM: x86/mmu: Fold kvm_mmu_zap_memslot() into kvm_arch_flush_shadow_memslot()

[Sean Christopherson]

Fold kvm_mmu_zap_memslot() into its sole caller so that its GFN range
structure can be used to trigger guest_memfd invalidations regardless of
whether KVM will do a partial or full zap of the MMU.

No functional change intended.

Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/mmu/mmu.c | 35 +++++++++++++++--------------------
 1 file changed, 15 insertions(+), 20 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 6c13da942bfc..223d80b12b9b 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -7560,8 +7560,14 @@ static void kvm_mmu_zap_memslot_pages_and_flush(struct kvm *kvm,
 	kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
 }
 
-static void kvm_mmu_zap_memslot(struct kvm *kvm,
-				struct kvm_memory_slot *slot)
+static inline bool kvm_memslot_flush_zap_all(struct kvm *kvm)
+{
+	return kvm->arch.vm_type == KVM_X86_DEFAULT_VM &&
+	       kvm_check_has_quirk(kvm, KVM_X86_QUIRK_SLOT_ZAP_ALL);
+}
+
+void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
+				   struct kvm_memory_slot *slot)
 {
 	struct kvm_gfn_range range = {
 		.slot = slot,
@@ -7572,25 +7578,14 @@ static void kvm_mmu_zap_memslot(struct kvm *kvm,
 	};
 	bool flush;
 
-	write_lock(&kvm->mmu_lock);
-	flush = kvm_unmap_gfn_range(kvm, &range);
-	kvm_mmu_zap_memslot_pages_and_flush(kvm, slot, flush);
-	write_unlock(&kvm->mmu_lock);
-}
-
-static inline bool kvm_memslot_flush_zap_all(struct kvm *kvm)
-{
-	return kvm->arch.vm_type == KVM_X86_DEFAULT_VM &&
-	       kvm_check_has_quirk(kvm, KVM_X86_QUIRK_SLOT_ZAP_ALL);
-}
-
-void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
-				   struct kvm_memory_slot *slot)
-{
-	if (kvm_memslot_flush_zap_all(kvm))
+	if (kvm_memslot_flush_zap_all(kvm)) {
 		kvm_mmu_zap_all_fast(kvm);
-	else
-		kvm_mmu_zap_memslot(kvm, slot);
+	} else {
+		write_lock(&kvm->mmu_lock);
+		flush = kvm_unmap_gfn_range(kvm, &range);
+		kvm_mmu_zap_memslot_pages_and_flush(kvm, slot, flush);
+		write_unlock(&kvm->mmu_lock);
+	}
 }
 
 void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 08/18] KVM: x86/mmu: Split kvm_mmu_zap_all_fast() into "front" and "back" halves

[Sean Christopherson]

Split kvm_mmu_zap_all_fast() into a "front half" and a "back half", where
the front half is everything that runs with mmu_lock held for write, and
the back half is the code that runs outside of mmu_lock.  This will allow
putting more code inside kvm_arch_flush_shadow_memslot()'s critical section
without having to take mmu_lock twice in quick succession.

No functional change intended.

Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/mmu/mmu.c | 37 +++++++++++++++++++++++++------------
 1 file changed, 25 insertions(+), 12 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 223d80b12b9b..a5c2a560a88a 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -6921,20 +6921,11 @@ static void kvm_zap_obsolete_pages(struct kvm *kvm)
 	kvm_mmu_commit_zap_page(kvm, &invalid_list);
 }
 
-/*
- * Fast invalidate all shadow pages and use lock-break technique
- * to zap obsolete pages.
- *
- * It's required when memslot is being deleted or VM is being
- * destroyed, in these cases, we should ensure that KVM MMU does
- * not use any resource of the being-deleted slot or all slots
- * after calling the function.
- */
-static void kvm_mmu_zap_all_fast(struct kvm *kvm)
+static void __kvm_mmu_zap_all_fast_front_half(struct kvm *kvm)
 {
 	lockdep_assert_held(&kvm->slots_lock);
+	lockdep_assert_held_write(&kvm->mmu_lock);
 
-	write_lock(&kvm->mmu_lock);
 	trace_kvm_mmu_zap_all_fast(kvm);
 
 	/*
@@ -6971,8 +6962,12 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 	kvm_make_all_cpus_request(kvm, KVM_REQ_MMU_FREE_OBSOLETE_ROOTS);
 
 	kvm_zap_obsolete_pages(kvm);
+}
 
-	write_unlock(&kvm->mmu_lock);
+static void __kvm_mmu_zap_all_fast_back_half(struct kvm *kvm)
+{
+	lockdep_assert_held(&kvm->slots_lock);
+	lockdep_assert_not_held(&kvm->mmu_lock);
 
 	/*
 	 * Zap the invalidated TDP MMU roots, all SPTEs must be dropped before
@@ -6986,6 +6981,24 @@ static void kvm_mmu_zap_all_fast(struct kvm *kvm)
 		kvm_tdp_mmu_zap_invalidated_roots(kvm, true);
 }
 
+/*
+ * Fast invalidate all shadow pages and use lock-break technique
+ * to zap obsolete pages.
+ *
+ * It's required when memslot is being deleted or VM is being
+ * destroyed, in these cases, we should ensure that KVM MMU does
+ * not use any resource of the being-deleted slot or all slots
+ * after calling the function.
+ */
+static void kvm_mmu_zap_all_fast(struct kvm *kvm)
+{
+	write_lock(&kvm->mmu_lock);
+	__kvm_mmu_zap_all_fast_front_half(kvm);
+	write_unlock(&kvm->mmu_lock);
+
+	__kvm_mmu_zap_all_fast_back_half(kvm);
+}
+
 int kvm_mmu_init_vm(struct kvm *kvm)
 {
 	int r, i;
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 09/18] KVM: x86/mmu: Use split "zap all fast" helpers when invalidating memslot

[Sean Christopherson]

Manually invoke the front half and back half of the "zap all fast" flow
when invalidating a memslot so that mmu_lock is acquired at function scope
in kvm_arch_flush_shadow_memslot().   This will allow putting more code
inside the critical section without having to take mmu_lock twice in quick
succession.

Opportunistically open code checking whether or not to do the fast zap, to
discourage removing the local "zap_all" in a future cleanup, i.e. to ensure
the SLOT_ZAP_ALL quirk is queried exactly once.  Processing the front half
but not the back half of the fast zap (if SLOT_ZAP_ALL were disabled
concurrently) would result in KVM unnecessarily keeping invalid TDP MMU
roots until the VM is destroyed.

No functional change intended.

Cc: stable@vger.kernel.org # 6.12.x
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/mmu/mmu.c | 21 +++++++++++----------
 1 file changed, 11 insertions(+), 10 deletions(-)

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index a5c2a560a88a..3eb1f86593b1 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -7573,12 +7573,6 @@ static void kvm_mmu_zap_memslot_pages_and_flush(struct kvm *kvm,
 	kvm_mmu_remote_flush_or_zap(kvm, &invalid_list, flush);
 }
 
-static inline bool kvm_memslot_flush_zap_all(struct kvm *kvm)
-{
-	return kvm->arch.vm_type == KVM_X86_DEFAULT_VM &&
-	       kvm_check_has_quirk(kvm, KVM_X86_QUIRK_SLOT_ZAP_ALL);
-}
-
 void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 				   struct kvm_memory_slot *slot)
 {
@@ -7589,16 +7583,23 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 		.may_block = true,
 		.attr_filter = KVM_FILTER_PRIVATE | KVM_FILTER_SHARED,
 	};
+	bool zap_all = kvm->arch.vm_type == KVM_X86_DEFAULT_VM &&
+		       kvm_check_has_quirk(kvm, KVM_X86_QUIRK_SLOT_ZAP_ALL);
 	bool flush;
 
-	if (kvm_memslot_flush_zap_all(kvm)) {
-		kvm_mmu_zap_all_fast(kvm);
+	write_lock(&kvm->mmu_lock);
+
+	if (zap_all) {
+		__kvm_mmu_zap_all_fast_front_half(kvm);
 	} else {
-		write_lock(&kvm->mmu_lock);
 		flush = kvm_unmap_gfn_range(kvm, &range);
 		kvm_mmu_zap_memslot_pages_and_flush(kvm, slot, flush);
-		write_unlock(&kvm->mmu_lock);
 	}
+
+	write_unlock(&kvm->mmu_lock);
+
+	if (zap_all)
+		__kvm_mmu_zap_all_fast_back_half(kvm);
 }
 
 void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen)
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 10/18] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped

[Sean Christopherson]

Wire up a gmem_invalidate_range() call for SNP VMs, and use it to force
vCPUs to reload/recheck their guest-provided VMSA if the backing gmem
page is being invalidated, e.g. is being PUNCH_HOLE'd.  Use the same core
logic to handle invalidations as VMX does for the APIC-access page, as the
two concepts are nearly identical: shove the physical address of a page
into the vCPU's control structure:

 1. Snapshot the invalidation sequence counter
 2. Grab the pfn (from guest_memfd in this case)
 3. Acquire mmu_lock for read
 4. Re-request reload if retry is needed, otherwise commit the change.

Note, the re-request action in #4 is necessary as KVM's retry logic is
fuzzy, i.e. can get false positives.  If the guest_memfd page has been
dropped, at some point a subsequent reload will fail to get a PFN from
guest_memfd, and KVM will fail KVM_RUN.  If the retry was due to a false
positive, KVM will retry until there are no relevant MMU notifier events
(and will retry in the "outer" loop, i.e. will drop locks and resched as
needed).

Note #2!  Take care to invalidate the VMSA when a relevant memslot is
DELETED or MOVED, as invalidations in response to PUNCH_HOLE are predicated
on memslot bindings (KVM doesn't know what GFN range(s) to invalidate
without a binding).  And more importantly, the VMSA mapping requires a
memslot, i.e. must be invalidated if its memslots disappears, regardless of
the state of the underlying guest_memfd inode.

Failure to invalidate the vCPU's control.vmsa_pa (which is checked by
pre_sev_run()) can prevent KVM from properly freeing the page as firmware
will reject the RMPUPDATE to reclaim the page with FAIL_INUSE if the vCPU
is actively running, i.e. if VMSA page is in-use.  That in turn leads to an
RMP #PF on the next use, as the page will still be assigned to the SNP VM.

  SEV-SNP: RMPUPDATE failed for PFN 78d198, pg_level: 1, ret: 3
  SEV-SNP: PFN 0x78d198, RMP entry: [0xfff0000000144001 - 0x000000000000000f]
  CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G     U     O
  Tainted: [U]=USER, [O]=OOT_MODULE
  Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
  Call Trace:
   <TASK>
   dump_stack_lvl+0x54/0x70
   rmpupdate+0x12c/0x140
   rmp_make_shared+0x3b/0x60
   sev_gmem_invalidate+0xe0/0x170 [kvm_amd]
   delete_from_page_cache_batch+0x1d8/0x220
   truncate_inode_pages_range+0x120/0x3d0
   kvm_gmem_fallocate+0x19a/0x270 [kvm]
   vfs_fallocate+0x1bc/0x1f0
   __x64_sys_fallocate+0x48/0x70
   do_syscall_64+0x10a/0x480
   entry_SYSCALL_64_after_hwframe+0x4b/0x53
  RIP: 0033:0x496c7e
   </TASK>
  ------------[ cut here ]------------
  SEV: Failed to update RMP entry for PFN 0x78d198 error -14
  WARNING: arch/x86/kvm/svm/sev.c:5160 at sev_gmem_invalidate+0x126/0x170 [kvm_amd], CPU#3: sev_snp_vmsa_pu/31345
  CPU: 3 UID: 0 PID: 31345 Comm: sev_snp_vmsa_pu Tainted: G     U     O
  Tainted: [U]=USER, [O]=OOT_MODULE
  Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
  RIP: 0010:sev_gmem_invalidate+0x12b/0x170 [kvm_amd]
  Call Trace:
   <TASK>
   delete_from_page_cache_batch+0x1d8/0x220
   truncate_inode_pages_range+0x120/0x3d0
   kvm_gmem_fallocate+0x19a/0x270 [kvm]
   vfs_fallocate+0x1bc/0x1f0
   __x64_sys_fallocate+0x48/0x70
   do_syscall_64+0x10a/0x480
   entry_SYSCALL_64_after_hwframe+0x4b/0x53
  RIP: 0033:0x496c7e
   </TASK>
  irq event stamp: 20689
  hardirqs last  enabled at (20699): [<ffffffff8e76092c>] __console_unlock+0x5c/0x60
  hardirqs last disabled at (20708): [<ffffffff8e760911>] __console_unlock+0x41/0x60
  softirqs last  enabled at (20722): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140
  softirqs last disabled at (20717): [<ffffffff8e6cd74e>] __irq_exit_rcu+0x7e/0x140
  ---[ end trace 0000000000000000 ]---
  BUG: unable to handle page fault for address: ffff99a64d198000
  #PF: supervisor write access in kernel mode
  #PF: error_code(0x80000003) - RMP violation
  PGD 13eb001067 P4D 13eb001067 PUD 78d1d1063 PMD 1184e0063 PTE 800000078d198163
  SEV-SNP: PFN 0x78d198, RMP entry: [0x6030000000144001 - 0x000000000000000f]
  Oops: Oops: 0003 [#1] SMP
  CPU: 3 UID: 0 PID: 31407 Comm: highlanderd_hea Tainted: G     U  W  O
  Tainted: [U]=USER, [W]=WARN, [O]=OOT_MODULE
  Hardware name: Google, Inc. Arcadia_IT_80/Arcadia_IT_80, BIOS 34.86.0-102 01/25/2026
  RIP: 0010:prep_new_page+0x67/0x220
  Call Trace:
   <TASK>
   get_page_from_freelist+0x1c40/0x1c70
   __alloc_frozen_pages_noprof+0xca/0x1f0
   alloc_pages_mpol+0x10b/0x1b0
   alloc_pages_noprof+0x81/0x90
   pte_alloc_one+0x1b/0xd0
   do_pte_missing+0xdf/0x1020
   handle_mm_fault+0x7c7/0xb20
   do_user_addr_fault+0x268/0x6b0
   exc_page_fault+0x67/0xa0
   asm_exc_page_fault+0x26/0x30
  RIP: 0033:0x4a6b1e
   </TASK>
  gsmi: Log Shutdown Reason 0x03
  CR2: ffff99a64d198000
  ---[ end trace 0000000000000000 ]---
  RIP: 0010:prep_new_page+0x67/0x220

Drop the pseudo-TODO comment about needing to pin the page if guest_memfd
every supports migration, as integrating with invalidations events means
KVM will Just Work if/when page migration is ever supported (assuming SNP
hardware supports migrating VMSA pages).

Note #3, invalidate() and invalidate_range() have _completely_ different
semantics; the new invalidate_range() is a true invalidation, whereas the
existing invalidate() is really a "make shared" operation.  Ignore the
confusing naming and poor Kconfig bundling for the moment to minimize the
delta for LTS kernels, the mess will be cleaned up shortly.

Reported-by: Hyunwoo Kim <imv4bel@gmail.com>
Closes: https://lore.kernel.org/all/aimMWzAf5b3luM0b@v4bel
Fixes: e366f92ea99e ("KVM: SEV: Support SEV-SNP AP Creation NAE event")
Cc: stable@vger.kernel.org
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Michael Roth <michael.roth@amd.com>
Cc: Jörg Rödel <joro@8bytes.org>
Cc: Fuad Tabba <tabba@google.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/kvm-x86-ops.h |  4 ++
 arch/x86/include/asm/kvm_host.h    |  6 +++
 arch/x86/kvm/mmu/mmu.c             |  5 ++
 arch/x86/kvm/svm/sev.c             | 80 ++++++++++++++++++++++++++----
 arch/x86/kvm/svm/svm.c             |  2 +
 arch/x86/kvm/svm/svm.h             |  2 +
 arch/x86/kvm/x86.c                 |  6 +++
 include/linux/kvm_host.h           |  1 +
 virt/kvm/guest_memfd.c             |  4 ++
 9 files changed, 99 insertions(+), 11 deletions(-)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 83dc5086138b..ccf23b3f0e1c 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -134,6 +134,7 @@ KVM_X86_OP_OPTIONAL(mem_enc_unregister_region)
 KVM_X86_OP_OPTIONAL(vm_copy_enc_context_from)
 KVM_X86_OP_OPTIONAL(vm_move_enc_context_from)
 KVM_X86_OP_OPTIONAL(guest_memory_reclaimed)
+KVM_X86_OP_OPTIONAL(reload_vmsa)
 KVM_X86_OP(get_feature_msr)
 KVM_X86_OP(check_emulate_instruction)
 KVM_X86_OP(apic_init_signal_blocked)
@@ -148,6 +149,9 @@ KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
 KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
 KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level)
 KVM_X86_OP_OPTIONAL(gmem_invalidate)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
+#endif
 #endif
 
 #undef KVM_X86_OP
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index b517257a6315..1c598b40e0c3 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -122,6 +122,8 @@
 	KVM_ARCH_REQ_FLAGS(31, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_HV_TLB_FLUSH \
 	KVM_ARCH_REQ_FLAGS(32, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
+#define KVM_REQ_VMSA_PAGE_RELOAD \
+	KVM_ARCH_REQ_FLAGS(33, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP)
 #define KVM_REQ_UPDATE_PROTECTED_GUEST_STATE \
 	KVM_ARCH_REQ_FLAGS(34, KVM_REQUEST_WAIT)
 
@@ -1878,6 +1880,7 @@ struct kvm_x86_ops {
 	int (*vm_copy_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
 	int (*vm_move_enc_context_from)(struct kvm *kvm, unsigned int source_fd);
 	void (*guest_memory_reclaimed)(struct kvm *kvm);
+	void (*reload_vmsa)(struct kvm_vcpu *vcpu);
 
 	int (*get_feature_msr)(u32 msr, u64 *data);
 
@@ -1902,6 +1905,9 @@ struct kvm_x86_ops {
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
 	int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
 	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+	void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range);
+#endif
 	int (*gmem_max_mapping_level)(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 };
 
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 3eb1f86593b1..e2978e9a1731 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -7589,6 +7589,11 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm,
 
 	write_lock(&kvm->mmu_lock);
 
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+	if (slot->gmem.file)
+		kvm_arch_gmem_invalidate_range(kvm, &range);
+#endif
+
 	if (zap_all) {
 		__kvm_mmu_zap_all_fast_front_half(kvm);
 	} else {
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 30792adcfc8e..62c6126d52c0 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -3979,19 +3979,25 @@ static int snp_begin_psc(struct vcpu_svm *svm)
 	return snp_do_psc(svm);
 }
 
-static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
+static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 {
 	struct vcpu_svm *svm = to_svm(vcpu);
 	struct kvm_memory_slot *slot;
+	struct kvm *kvm = vcpu->kvm;
 	gfn_t gfn = gpa_to_gfn(gpa);
+	unsigned long mmu_seq;
 	struct page *page;
 	kvm_pfn_t pfn;
 
 	lockdep_assert_held(&svm->sev_es.snp_vmsa_mutex);
 
-	/* Clear use of the VMSA. */
+	/*
+	 * Clear use of the VMSA.  Ensure snp_guest_vmsa_gpa is written exactly
+	 * once, as it is read locklessly when responding to gfn invalidations.
+	 * Pairs with the READ_ONCE() in sev_gmem_invalidate_range().
+	 */
 	svm->vmcb->control.vmsa_pa = INVALID_PAGE;
-	svm->sev_es.snp_guest_vmsa_gpa = INVALID_PAGE;
+	WRITE_ONCE(svm->sev_es.snp_guest_vmsa_gpa, INVALID_PAGE);
 
 	/*
 	 * When replacing the VMSA during SEV-SNP AP creation,
@@ -4006,6 +4012,9 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	if (!slot)
 		return;
 
+	mmu_seq = kvm->mmu_invalidate_seq;
+	smp_rmb();
+
 	/*
 	 * The new VMSA will be private memory guest memory, so retrieve the
 	 * PFN from the gmem backend.
@@ -4024,15 +4033,20 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	 */
 	svm->sev_es.snp_has_guest_vmsa = true;
 
-	/* Use the new VMSA */
+	read_lock(&kvm->mmu_lock);
+	/*
+	 * Save the guest-provided GPA.  If retry is needed, then KVM will try
+	 * again with the same GPA.  If the VMSA is usable, then KVM needs to
+	 * track the GPA so that the VMSA can be reloaded if the backing page
+	 * for the GPA is invalidated.
+	 */
 	svm->sev_es.snp_guest_vmsa_gpa = gpa;
-	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
+	if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn))
+		kvm_make_request(KVM_REQ_VMSA_PAGE_RELOAD, vcpu);
+	else
+		svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
+	read_unlock(&kvm->mmu_lock);
 
-	/*
-	 * gmem pages aren't currently migratable, but if this ever changes
-	 * then care should be taken to ensure svm->sev_es.vmsa is pinned
-	 * through some other means.
-	 */
 	kvm_release_page_clean(page);
 }
 
@@ -4058,7 +4072,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 	gpa = svm->sev_es.snp_pending_vmsa_gpa;
 	svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
 
-	sev_snp_reload_vmsa(vcpu, gpa);
+	__sev_snp_reload_vmsa(vcpu, gpa);
 
 	/*
 	 * Mark the vCPU as runnable for CREATE requests, indicated by a valid
@@ -4070,6 +4084,15 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
 		kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
 }
 
+void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu)
+{
+	struct vcpu_sev_es_state *sev_es = &to_svm(vcpu)->sev_es;
+
+	guard(mutex)(&sev_es->snp_vmsa_mutex);
+
+	__sev_snp_reload_vmsa(vcpu, sev_es->snp_guest_vmsa_gpa);
+}
+
 static int sev_snp_ap_creation(struct vcpu_svm *svm)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(svm->vcpu.kvm);
@@ -5199,6 +5222,41 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
 	}
 }
 
+void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+	struct kvm_vcpu *vcpu;
+	unsigned long i;
+
+	lockdep_assert_held_write(&kvm->mmu_lock);
+
+	/*
+	 * An unstable result for "is SNP" is a-ok here, thanks to mmu_lock.
+	 * The vCPU's VMSA GPA is invalidated before the vCPU is made visible
+	 * to other tasks, and can only become valid while holding mmu_lock,
+	 * after the VM is fully committed to being an SNP VM.
+	 */
+	if (!____sev_snp_guest(kvm))
+		return;
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		/*
+		 * Read snp_guest_vmsa_gpa without taking the vCPU's VMSA mutex
+		 * (or its generic mutex) as mmu_lock is held, i.e. this task
+		 * can't sleep.  The VMSA is invalidated outside of mmu_lock,
+		 * but can only become valid inside of mmu_lock, i.e. the below
+		 * can get false positives, but not false negatives.  A false
+		 * positive is benign, as a spurious request simply forces the
+		 * vCPU to re-establish its VMSA.
+		 */
+		gpa_t gpa = READ_ONCE(to_svm(vcpu)->sev_es.snp_guest_vmsa_gpa);
+
+		if (VALID_PAGE(gpa) &&
+		    gpa_to_gfn(gpa) >= range->start &&
+		    gpa_to_gfn(gpa) < range->end)
+			kvm_make_request_and_kick(KVM_REQ_VMSA_PAGE_RELOAD, vcpu);
+	}
+}
+
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private)
 {
 	int level, rc;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 79c818d91dda..dd51df74c2dc 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5445,12 +5445,14 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.mem_enc_register_region = sev_mem_enc_register_region,
 	.mem_enc_unregister_region = sev_mem_enc_unregister_region,
 	.guest_memory_reclaimed = sev_guest_memory_reclaimed,
+	.reload_vmsa = sev_snp_reload_vmsa,
 
 	.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
 	.vm_move_enc_context_from = sev_vm_move_enc_context_from,
 
 	.gmem_prepare = sev_gmem_prepare,
 	.gmem_invalidate = sev_gmem_invalidate,
+	.gmem_invalidate_range = sev_gmem_invalidate_range,
 	.gmem_max_mapping_level = sev_gmem_max_mapping_level,
 #endif
 	.check_emulate_instruction = svm_check_emulate_instruction,
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index effca6372e15..130205defffa 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -996,6 +996,7 @@ static inline struct page *snp_safe_alloc_page(void)
 {
 	return snp_safe_alloc_page_node(numa_node_id(), GFP_KERNEL_ACCOUNT);
 }
+void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu);
 
 int sev_vcpu_create(struct kvm_vcpu *vcpu);
 void sev_free_vcpu(struct kvm_vcpu *vcpu);
@@ -1010,6 +1011,7 @@ extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
 int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
 void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
 void sev_free_decrypted_vmsa(struct kvm_vcpu *vcpu, struct vmcb_save_area *vmsa);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 8abd733d5173..5a5fd6211d23 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -8170,6 +8170,8 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu)
 				goto out;
 			}
 		}
+		if (kvm_check_request(KVM_REQ_VMSA_PAGE_RELOAD, vcpu))
+			kvm_x86_call(reload_vmsa)(vcpu);
 	}
 
 	if (kvm_check_request(KVM_REQ_EVENT, vcpu) || req_int_win ||
@@ -10599,6 +10601,10 @@ void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
 {
 	kvm_x86_call(gmem_invalidate)(start, end);
 }
+void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+	kvm_x86_call(gmem_invalidate_range)(kvm, range);
+}
 #endif
 #endif
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ab8cfaec82d3..c00fc1740ce5 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2608,6 +2608,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
 
 #ifdef CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 86690683b2fe..659b8dbe0b30 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
 		}
 
 		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
+
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+		kvm_arch_gmem_invalidate_range(kvm, &gfn_range);
+#endif
 	}
 
 	if (flush)
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 11/18] KVM: SEV: Mark vCPU has having guest-provided VMSA even if its invalid

[Sean Christopherson]

Track the guest as having a guest-provided VMSA as soon as control.vmsa_pa
is invalidated, instead of waiting to see if the guest-provided VMSA is
usable, so that KVM doesn't switch back to the original VMSA instead of
exiting to userspace (due to an invalid VMSA).  By the time a vCPU tries
to load a guest-provided VMSA, KVM has already communicated "success" for
AP creation, i.e. KVM has committed to using the guest-provided VMSA.

Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/svm/sev.c | 34 +++++++++++++++++-----------------
 1 file changed, 17 insertions(+), 17 deletions(-)

diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 62c6126d52c0..a7584b7ed6dc 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -4005,23 +4005,6 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	 */
 	vmcb_mark_all_dirty(svm->vmcb);
 
-	if (!VALID_PAGE(gpa))
-		return;
-
-	slot = gfn_to_memslot(vcpu->kvm, gfn);
-	if (!slot)
-		return;
-
-	mmu_seq = kvm->mmu_invalidate_seq;
-	smp_rmb();
-
-	/*
-	 * The new VMSA will be private memory guest memory, so retrieve the
-	 * PFN from the gmem backend.
-	 */
-	if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL))
-		return;
-
 	/*
 	 * From this point forward, the VMSA will always be a guest-mapped page
 	 * rather than the initial one allocated by KVM in svm->sev_es.vmsa. In
@@ -4033,6 +4016,23 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
 	 */
 	svm->sev_es.snp_has_guest_vmsa = true;
 
+	if (!VALID_PAGE(gpa))
+		return;
+
+	slot = gfn_to_memslot(vcpu->kvm, gfn);
+	if (!slot)
+		return;
+
+	mmu_seq = kvm->mmu_invalidate_seq;
+	smp_rmb();
+
+	/*
+	 * The new VMSA will be private memory guest memory, so retrieve the
+	 * PFN from the gmem backend.
+	 */
+	if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL))
+		return;
+
 	read_lock(&kvm->mmu_lock);
 	/*
 	 * Save the guest-provided GPA.  If retry is needed, then KVM will try
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 12/18] KVM: x86: Guard .gmem_prepare() declarations with HAVE_KVM_GMEM_PREPARE=y

[Sean Christopherson]

Wrap the .gmem_prepare() declarations with HAVE_KVM_GMEM_PREPARE so that
non-SEV code doesn't try to wire up a callback without doing the necessary
enabling.

No functional change intended.

Fixes: 3bb2531e20bf ("KVM: guest_memfd: Add hook for initializing memory")
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Reviewed-by: Michael Roth <michael.roth@amd.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/kvm-x86-ops.h | 4 +++-
 arch/x86/include/asm/kvm_host.h    | 2 ++
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index ccf23b3f0e1c..736129db272a 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -146,12 +146,14 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL(get_untagged_addr)
 KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
-KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level)
+#endif
 KVM_X86_OP_OPTIONAL(gmem_invalidate)
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
 #endif
+KVM_X86_OP_OPTIONAL_RET0(gmem_max_mapping_level)
 #endif
 
 #undef KVM_X86_OP
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1c598b40e0c3..1f8ef0e1566a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1903,7 +1903,9 @@ struct kvm_x86_ops {
 
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 	int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
+#endif
 	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 	void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range);
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook

[Sean Christopherson]

Pass a GPA instead of a GFN to kvm_arch_gmem_prepare() so that the GFN/GPA
can be made optional in the future, i.e. so that guest_memfd can pass
INVALID_GPA.  This will allow reworking the hook into a general .convert()
hook, i.e. so that it can be used for both "make private" and "make shared"
operations, at which point guest_memfd may not have a corresponding GFN,
e.g. when converting memory back to a shared state when a folio is freed.

Opportunistically swap the ordering of @pfn and @gpa params for
kvm_x86_ops.gmem_prepare() to match that of kvm_arch_gmem_prepare().

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/kvm_host.h | 2 +-
 arch/x86/kvm/svm/sev.c          | 3 ++-
 arch/x86/kvm/svm/svm.h          | 2 +-
 arch/x86/kvm/x86.c              | 4 ++--
 include/linux/kvm_host.h        | 2 +-
 virt/kvm/guest_memfd.c          | 2 +-
 6 files changed, 8 insertions(+), 7 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 1f8ef0e1566a..24a7a9e6910a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1904,7 +1904,7 @@ struct kvm_x86_ops {
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
+	int (*gmem_prepare)(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order);
 #endif
 	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index a7584b7ed6dc..dfe93679a832 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -5112,9 +5112,10 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
 	return false;
 }
 
-int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
+int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
+	gfn_t gfn = gpa_to_gfn(gpa);
 	kvm_pfn_t pfn_aligned;
 	gfn_t gfn_aligned;
 	int level, rc;
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 130205defffa..58390df36af0 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1009,7 +1009,7 @@ int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
-int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
+int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order);
 void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 5a5fd6211d23..cc1404bef8cb 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10590,9 +10590,9 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 }
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order)
+int kvm_arch_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order)
 {
-	return kvm_x86_call(gmem_prepare)(kvm, pfn, gfn, max_order);
+	return kvm_x86_call(gmem_prepare)(kvm, gpa, pfn, max_order);
 }
 #endif
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index c00fc1740ce5..542e5d50849e 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2573,7 +2573,7 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order);
+int kvm_arch_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 659b8dbe0b30..7ad656e30f9f 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -66,7 +66,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 	kvm_pfn_t pfn = folio_file_pfn(folio, index);
 	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-	int rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
+	int rc = kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn), pfn, folio_order(folio));
 	if (rc) {
 		pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
 				    index, gfn, pfn, rc);
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 14/18] KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure

[Sean Christopherson]

Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM
SNP alreads logs more precise messages in all error paths, and whether or
not failure to convert the pfn to private is "unexpected", i.e. warrants
logging, is firmly an architecture specific detail.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/kvm/guest_memfd.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 7ad656e30f9f..f4f8635d1a10 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 	kvm_pfn_t pfn = folio_file_pfn(folio, index);
 	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-	int rc = kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn), pfn, folio_order(folio));
-	if (rc) {
-		pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
-				    index, gfn, pfn, rc);
-		return rc;
-	}
-#endif
 
+	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn), pfn, folio_order(folio));
+#else
 	return 0;
+#endif
 }
 
 /*
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 15/18] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

[Sean Christopherson]

Fold __kvm_gmem_prepare_folio() into kvm_gmem_prepare_folio() and extend
the #ifdef guard to cover all of the kvm_gmem_prepare_folio().  Providing
a single-use, ~2 line tail-call helper adds no value.  In fact, due to
having to reconstitue the gfn, the separate helper is very arguably a net
negative.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/kvm/guest_memfd.c | 21 +++++++--------------
 1 file changed, 7 insertions(+), 14 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index f4f8635d1a10..d5d3688eb1bd 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -60,19 +60,6 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
 	return gfn - slot->base_gfn + slot->gmem.pgoff;
 }
 
-static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
-				    pgoff_t index, struct folio *folio)
-{
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	kvm_pfn_t pfn = folio_file_pfn(folio, index);
-	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-
-	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn), pfn, folio_order(folio));
-#else
-	return 0;
-#endif
-}
-
 /*
  * Process @folio, which contains @gfn, so that the guest can use it.
  * The folio must be locked and the gfn must be contained in @slot.
@@ -82,6 +69,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
 static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 				  gfn_t gfn, struct folio *folio)
 {
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 	pgoff_t index;
 
 	/*
@@ -101,7 +89,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	index = kvm_gmem_get_index(slot, gfn);
 	index = ALIGN_DOWN(index, folio_nr_pages(folio));
 
-	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
+	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn),
+				     folio_file_pfn(folio, index),
+				     folio_order(folio));
+#else
+	return 0;
+#endif
 }
 
 /*
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 16/18] KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()

[Sean Christopherson]

Tweak the guest_memfd prepare() hook to explicitly pass the number of pages
to "prepare", so that the prepare() and invalidate() hooks can be combined
into a single convert() hook, and because the existing code is outright
broken if a guest_memfd folio is comprised of more than one page (which
can't happen, yet).  guest_memfd aligns the pfn down, and the SNP code
*tries* to create a corresponding huge entry, but if the RMP must use 4KiB
entries for whatever reason, KVM will only "prepare" the first pfn, and not
the actual pfn that needs to be converted/prepared.

Alternatively, @max_order could simply be repurposed as _the_ @order, but
that will fall apart when in-place conversion comes along, at which point
KVM will need to deal with conversions that aren't strictly bound to a one
folio.

Note, kvm_gmem_prepare_folio() still mishandles hugepages, as it aligns
only the pfn, i.e. would pass the wrong gfn+pfn pair.  This will be fixed
shortly.

WARN in the SNP code if the number of pages to prepare is anything other
than '1', i.e. if guest_memfd is trying to prepare/convert more than a
single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion
greater than order-0 folios.

Opportunistically convert the existing WARN in the prepare flow to a only
fire once, e.g. so that a guest_memfd bug doesn't unintentionally Dos the
kernel by spamming the log.

Fixes: b85524314a3d ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/kvm_host.h | 3 ++-
 arch/x86/kvm/svm/sev.c          | 6 +++++-
 arch/x86/kvm/svm/svm.h          | 3 ++-
 arch/x86/kvm/x86.c              | 5 +++--
 include/linux/kvm_host.h        | 3 ++-
 virt/kvm/guest_memfd.c          | 7 ++++---
 6 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 24a7a9e6910a..3e34435f7e4e 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1904,7 +1904,8 @@ struct kvm_x86_ops {
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	int (*gmem_prepare)(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order);
+	int (*gmem_prepare)(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order);
 #endif
 	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index dfe93679a832..3e7792afbab4 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -5112,7 +5112,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
 	return false;
 }
 
-int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order)
+int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
 	gfn_t gfn = gpa_to_gfn(gpa);
@@ -5124,6 +5125,9 @@ int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order)
 	if (!sev_snp_guest(kvm))
 		return 0;
 
+	if (WARN_ON_ONCE(nr_pages != 1))
+		return -EIO;
+
 	rc = snp_lookup_rmpentry(pfn, &assigned, &level);
 	if (rc) {
 		pr_err_ratelimited("SEV: Failed to look up RMP entry: GFN %llx PFN %llx error %d\n",
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 58390df36af0..45a9e2d5417c 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1009,7 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
-int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order);
+int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order);
 void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index cc1404bef8cb..a167f42bb1af 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10590,9 +10590,10 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 }
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order)
+int kvm_arch_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order)
 {
-	return kvm_x86_call(gmem_prepare)(kvm, gpa, pfn, max_order);
+	return kvm_x86_call(gmem_prepare)(kvm, gpa, pfn, nr_pages, max_order);
 }
 #endif
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 542e5d50849e..469c22b551ee 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2573,7 +2573,8 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn, int max_order);
+int kvm_arch_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index d5d3688eb1bd..cd7a795834cb 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -70,6 +70,7 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 				  gfn_t gfn, struct folio *folio)
 {
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
+	unsigned long nr_pages = folio_nr_pages(folio);
 	pgoff_t index;
 
 	/*
@@ -85,12 +86,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and
 	 * checked when creating memslots.
 	 */
-	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, nr_pages));
 	index = kvm_gmem_get_index(slot, gfn);
-	index = ALIGN_DOWN(index, folio_nr_pages(folio));
+	index = ALIGN_DOWN(index, nr_pages);
 
 	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn),
-				     folio_file_pfn(folio, index),
+				     folio_file_pfn(folio, index), nr_pages,
 				     folio_order(folio));
 #else
 	return 0;
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio

[Sean Christopherson]

When preparing a guest_memfd folio, align the gfn as well as the pfn, i.e.
preserve the effective gfn+pfn pair provided by the caller.  If the folio
is larger than order-0, aligning the pfn but not the gfn would create a
mismatch set and unintentionally crash the guest due to assigning the pfn
at the wrong gfn.

Fixes: b85524314a3d ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/kvm/guest_memfd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index cd7a795834cb..4082cde3f3fc 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -87,8 +87,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * checked when creating memslots.
 	 */
 	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, nr_pages));
+	gfn = ALIGN_DOWN(gfn, nr_pages);
 	index = kvm_gmem_get_index(slot, gfn);
-	index = ALIGN_DOWN(index, nr_pages);
 
 	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn),
 				     folio_file_pfn(folio, index), nr_pages,
-- 
2.55.0.795.g602f6c329a-goog

[PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

[Sean Christopherson]

Smush guest_memfd's prepare() and invalidate() hooks into a common
convert() flow, as they are effectively two sides of the same coin: they're
invoked when private/shared memory is about to made accessible visible to
the guest/host.  I.e. prepare() is really "make private", and invalidate()
is really "make shared".

Using a single API fixes the wart where invalidate() isn't actually doing
invalidation, avoids adding yet another gmem Kconfig, avoids having to come
up with a name to describe the "make shared" behavior, and will yield more
intuitive code when in-place conversion comes along.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/kvm-x86-ops.h |  5 ++---
 arch/x86/include/asm/kvm_host.h    |  7 +++----
 arch/x86/kvm/Kconfig               |  2 +-
 arch/x86/kvm/svm/sev.c             | 21 ++++++++++++++++++---
 arch/x86/kvm/svm/svm.c             |  3 +--
 arch/x86/kvm/svm/svm.h             |  5 ++---
 arch/x86/kvm/x86.c                 | 15 ++++-----------
 include/linux/kvm_host.h           |  7 +++----
 virt/kvm/Kconfig                   |  2 +-
 virt/kvm/guest_memfd.c             | 25 ++++++++++++-------------
 10 files changed, 47 insertions(+), 45 deletions(-)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 736129db272a..6091987860bd 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -146,10 +146,9 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL(get_untagged_addr)
 KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
 #endif
-KVM_X86_OP_OPTIONAL(gmem_invalidate)
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
 #endif
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 3e34435f7e4e..3b4311048926 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1903,11 +1903,10 @@ struct kvm_x86_ops {
 
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	int (*gmem_prepare)(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
-			    kvm_pfn_t nr_pages, int max_order);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+	int (*gmem_convert)(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
-	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 	void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index 801bf9e520db..52e75d83a4d2 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -160,7 +160,7 @@ config KVM_AMD_SEV
 	depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
 	select ARCH_HAS_CC_PLATFORM
 	select KVM_GENERIC_MEMORY_ATTRIBUTES
-	select HAVE_KVM_ARCH_GMEM_PREPARE
+	select HAVE_KVM_ARCH_GMEM_CONVERT
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
 	select HAVE_KVM_ARCH_GMEM_POPULATE
 	help
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 3e7792afbab4..b7f0d688f699 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -5112,8 +5112,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
 	return false;
 }
 
-int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
-		     kvm_pfn_t nr_pages, int max_order)
+static int sev_gmem_make_private(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+				 kvm_pfn_t nr_pages, int max_order)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
 	gfn_t gfn = gpa_to_gfn(gpa);
@@ -5164,8 +5164,9 @@ int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
 	return 0;
 }
 
-void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+static void sev_gmem_make_shared(kvm_pfn_t start, kvm_pfn_t nr_pages)
 {
+	kvm_pfn_t end = start + nr_pages;
 	kvm_pfn_t pfn;
 
 	if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
@@ -5227,6 +5228,20 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
 	}
 }
 
+int sev_gmem_convert(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int order, bool to_private)
+{
+	if (to_private) {
+		if (WARN_ON_ONCE(!kvm || gpa == INVALID_GPA))
+			return -EIO;
+
+		return sev_gmem_make_private(kvm, gpa, pfn, nr_pages, order);
+	}
+
+	sev_gmem_make_shared(pfn, nr_pages);
+	return 0;
+}
+
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	struct kvm_vcpu *vcpu;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index dd51df74c2dc..f462ac9bbc93 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5450,8 +5450,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
 	.vm_move_enc_context_from = sev_vm_move_enc_context_from,
 
-	.gmem_prepare = sev_gmem_prepare,
-	.gmem_invalidate = sev_gmem_invalidate,
+	.gmem_convert = sev_gmem_convert,
 	.gmem_invalidate_range = sev_gmem_invalidate_range,
 	.gmem_max_mapping_level = sev_gmem_max_mapping_level,
 #endif
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index 45a9e2d5417c..aa12e6a80e83 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1009,9 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
-int sev_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
-		     kvm_pfn_t nr_pages, int max_order);
-void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+int sev_gmem_convert(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order, bool to_private);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index a167f42bb1af..f51303ff3a1e 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10589,18 +10589,11 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return !kvm_arch_has_private_mem(kvm);
 }
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
-			  kvm_pfn_t nr_pages, int max_order)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order, bool to_private)
 {
-	return kvm_x86_call(gmem_prepare)(kvm, gpa, pfn, nr_pages, max_order);
-}
-#endif
-
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
-{
-	kvm_x86_call(gmem_invalidate)(start, end);
+	return kvm_x86_call(gmem_convert)(kvm, gpa, pfn, nr_pages, max_order, to_private);
 }
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 469c22b551ee..eb5a3349381c 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2572,9 +2572,9 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
-			  kvm_pfn_t nr_pages, int max_order);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gpa_t gpa, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
@@ -2608,7 +2608,6 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
 
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index 794976b88c6f..53803c5777a4 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -107,7 +107,7 @@ config KVM_GUEST_MEMFD
        select XARRAY_MULTI
        bool
 
-config HAVE_KVM_ARCH_GMEM_PREPARE
+config HAVE_KVM_ARCH_GMEM_CONVERT
        bool
        depends on KVM_GUEST_MEMFD
 
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 4082cde3f3fc..d780ea0494d2 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -66,10 +66,10 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
  * On successful return the guest sees a zero page so as to avoid
  * leaking host data and the up-to-date flag is set.
  */
-static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
-				  gfn_t gfn, struct folio *folio)
+static int kvm_gmem_make_private(struct kvm *kvm, struct kvm_memory_slot *slot,
+				 gfn_t gfn, struct folio *folio)
 {
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 	unsigned long nr_pages = folio_nr_pages(folio);
 	pgoff_t index;
 
@@ -90,9 +90,9 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	gfn = ALIGN_DOWN(gfn, nr_pages);
 	index = kvm_gmem_get_index(slot, gfn);
 
-	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn),
+	return kvm_arch_gmem_convert(kvm, gfn_to_gpa(gfn),
 				     folio_file_pfn(folio, index), nr_pages,
-				     folio_order(folio));
+				     folio_order(folio), true);
 #else
 	return 0;
 #endif
@@ -517,14 +517,13 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 	return MF_DELAYED;
 }
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 static void kvm_gmem_free_folio(struct folio *folio)
 {
-	struct page *page = folio_page(folio, 0);
-	kvm_pfn_t pfn = page_to_pfn(page);
-	int order = folio_order(folio);
-
-	kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));
+	WARN_ON_ONCE(kvm_arch_gmem_convert(NULL, INVALID_GPA,
+					   folio_file_pfn(folio, 0),
+					   folio_nr_pages(folio),
+					   folio_order(folio), false));
 }
 #endif
 
@@ -532,7 +531,7 @@ static const struct address_space_operations kvm_gmem_aops = {
 	.dirty_folio = noop_dirty_folio,
 	.migrate_folio	= kvm_gmem_migrate_folio,
 	.error_remove_folio = kvm_gmem_error_folio,
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 	.free_folio = kvm_gmem_free_folio,
 #endif
 };
@@ -802,7 +801,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		folio_mark_uptodate(folio);
 	}
 
-	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+	r = kvm_gmem_make_private(kvm, slot, gfn, folio);
 
 	folio_unlock(folio);
 
-- 
2.55.0.795.g602f6c329a-goog

Re: [PATCH v4 02/18] KVM: SEV: Extract loading of guest-provided VMSA to a separate helper

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Missing MMU invalidation sequence check and invalidation hooks when resolving and installing the guest memory PFN into the hardware VMCB (`vmsa_pa`), allowing a guest to trigger a host Use-After-Free via hole-punching.
- [High] Early return on failure to retrieve or load the VMSA PFN leaves the vCPU permanently in the `KVM_MP_STATE_HALTED` state, causing KVM_RUN to hang indefinitely instead of returning an error to userspace.
--

--- Patch [2]: [PATCH v4 02/18] KVM: SEV: Extract loading of guest-provided VMSA to a separate helper ---
commit 77b2286f40978a436ebf37446c67270e552074af
Author: Sean Christopherson <seanjc@google.com>

KVM: SEV: Extract loading of guest-provided VMSA to a separate helper

Extract the loading/retrieval of a guest-provided VMSA to a separate helper
so that KVM can reuse the core logic when refreshing the VMSA after an MMU
invalidation from guest_memfd.

Cc: stable@vger.kernel.org # 6.12.x

> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index 827f5dc061023..d8ed00f76aa31 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c
[ ... ]
> @@ -4011,12 +3999,9 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
>  	 */
>  	vmcb_mark_all_dirty(svm->vmcb);
>  
> -	if (!VALID_PAGE(svm->sev_es.snp_pending_vmsa_gpa))
> +	if (!VALID_PAGE(gpa))
>  		return;
>  
> -	gfn = gpa_to_gfn(svm->sev_es.snp_pending_vmsa_gpa);
> -	svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
> -
>  	slot = gfn_to_memslot(vcpu->kvm, gfn);
>  	if (!slot)
>  		return;

[Severity: High]
This is a pre-existing issue, but if the VMSA loading fails here (e.g., due
to an invalid GPA or missing memslot), does the vCPU remain permanently
locked in KVM_MP_STATE_HALTED?

Since sev_snp_init_protected_guest_state() marks the vCPU as HALTED before
calling this reload helper, returning early bypasses the eventual transition
to KVM_MP_STATE_RUNNABLE. Could this cause KVM_RUN to hang indefinitely
instead of safely returning an error to userspace?

[ ... ]
> @@ -4040,7 +4025,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
>  	svm->sev_es.snp_has_guest_vmsa = true;
>  
>  	/* Use the new VMSA */
> -	svm->sev_es.snp_guest_vmsa_gpa = gfn_to_gpa(gfn);
> +	svm->sev_es.snp_guest_vmsa_gpa = gpa;
>  	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);

[Severity: Critical]
This isn't a bug introduced by this patch, but does resolving and installing
the PFN directly into the hardware VMCB (vmsa_pa) without using the
mmu_invalidate_seq retry protocol allow a guest to trigger a host
use-after-free?

If an untrusted guest triggers AP creation with a guest_memfd-backed VMSA,
and then punches a hole in that guest_memfd, could KVM miss the concurrent
invalidation since this path doesn't check the invalidation sequence under
kvm->mmu_lock? The hardware might continue using and writing to the freed
page.

>  
>  	/* Mark the vCPU as runnable */
> @@ -4054,6 +4039,31 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
>  	kvm_release_page_clean(page);
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260709204948.1988414-1-seanjc@google.com?part=2

Re: [PATCH v4 10/18] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] The guest_memfd invalidation flow fails to synchronously wait for vCPUs to exit guest mode before allowing the host to free the VMSA page, resulting in RMPUPDATE FAIL_INUSE hardware errors and a host kernel panic.
--

--- Patch [10]: [PATCH v4 10/18] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped ---
commit ceb579c141d8889038624fb707ecde51b8427194
Author: Sean Christopherson <seanjc@google.com>

KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped

This commit wires up a gmem_invalidate_range() call for SNP VMs, forcing
vCPUs to reload their VMSA if the backing guest_memfd page is invalidated
(e.g., via PUNCH_HOLE).

Link: https://lore.kernel.org/all/aimMWzAf5b3luM0b@v4bel

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
[ ... ]
> @@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
>  		}
>  
>  		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
> +
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> +		kvm_arch_gmem_invalidate_range(kvm, &gfn_range);
> +#endif
>  	}
>  
>  	if (flush)

[Severity: Critical]
Does this invalidation path need a way to guarantee a synchronous wait before
returning?

Since VMSA pages typically do not have TDP MMU SPTEs,
kvm_mmu_unmap_gfn_range() will return false, leaving flush as false.

Because flush remains false, the subsequent kvm_flush_remote_tlbs(kvm) call
is skipped.

Meanwhile, sev_gmem_invalidate_range() relies on KVM_REQ_VMSA_PAGE_RELOAD
and kvm_make_request_and_kick() to send an asynchronous IPI to the vCPU:

arch/x86/kvm/svm/sev.c:sev_gmem_invalidate_range() {
    ...
    kvm_make_request_and_kick(KVM_REQ_VMSA_PAGE_RELOAD, vcpu);
    ...
}

Because the IPI is asynchronous and the TLB flush (which would otherwise
provide a synchronous wait) is skipped, the host could proceed to free the
VMSA page before the vCPU actually drops it and exits guest mode.

Could this lead to the very RMPUPDATE FAIL_INUSE hardware error and host
kernel panic that the commit is trying to resolve, if a host userspace
process issues fallocate(PUNCH_HOLE) on the guest_memfd region?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260709204948.1988414-1-seanjc@google.com?part=10

Re: [PATCH v4 01/18] KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Missing MMU invalidation retry protocol when resolving and installing the guest_memfd VMSA page, causing a TOCTOU race leading to physical Use-After-Free.
--

--- Patch [1]: [PATCH v4 01/18] KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests ---
commit 78360a5e270c82d6286a0a511ee0ec1f25aec337
Author: Sean Christopherson <seanjc@google.com>

KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests
    
This commit tracks the GPA of the guest-provided VMSA used after AP_CREATION
events for SNP guests. It renames the tracking variables to clarify pending
vs. in-use GPAs to facilitate reacting to MMU invalidation events.

> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index 74fb15551e83f..827f5dc061023 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c
[ ... ]
> @@ -4039,6 +4040,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
>  	svm->sev_es.snp_has_guest_vmsa = true;
>  
>  	/* Use the new VMSA */
> +	svm->sev_es.snp_guest_vmsa_gpa = gfn_to_gpa(gfn);
>  	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);

[Severity: Critical]
This is a pre-existing issue, but does this bypass the MMU invalidation retry
protocol?

Looking at sev_snp_init_protected_guest_state(), KVM resolves the PFN via
kvm_gmem_get_pfn(), directly installs it into the hardware VMCB, and drops
the page reference:

	if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL))
		return;
	...
	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
	...
	kvm_release_page_clean(page);

Can this race with a concurrent guest_memfd hole punch if kvm->mmu_lock isn't
held and the invalidation sequence counter isn't verified?

If the page is freed back to the host allocator, could the hardware continue
to use a freed host physical page for the VMSA, leading to a physical
use-after-free?

>  	/* Mark the vCPU as runnable */
>  	kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260709204948.1988414-1-seanjc@google.com?part=1

Re: [PATCH v4 03/18] KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [High] KVM incorrectly executes VMRUN with a dummy unmeasured host VMSA instead of failing KVM_RUN when a vCPU's initial AP_CREATE VMSA load fails.

Pre-existing issues:
- [Critical] sev_snp_reload_vmsa() bypasses the MMU notifier invalidation retry protocol, leading to a Use-After-Free of a physical page if a concurrent guest_memfd invalidation occurs.
--

--- Patch [3]: [PATCH v4 03/18] KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable ---
commit ca23b8314664c0ed691814678164a510162e2299
Author: Sean Christopherson <seanjc@google.com>

KVM: SEV: Mark vCPU RUNNABLE after AP_CREATE, even if VMSA is unusable

This commit marks a vCPU as RUNNABLE after responding to an AP_CREATE
request, even if the guest-specified VMSA is unusable. This prevents the
vCPU from hanging in a non-running state and instead allows KVM_RUN to
return an error to userspace. It also enables retrying the VMSA load in
the future.

> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index d8ed00f76aa31..30792adcfc8e5 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c
> @@ -4028,9 +4028,6 @@ static void sev_snp_reload_vmsa(struct kvm_vcpu *vcpu, gpa_t gpa)
>  	svm->sev_es.snp_guest_vmsa_gpa = gpa;
>  	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);

[Severity: Critical]
This is a pre-existing issue, but does sev_snp_reload_vmsa() bypass the
MMU notifier invalidation retry protocol here?

If a concurrent PUNCH_HOLE occurs on the guest_memfd while
sev_snp_reload_vmsa() fetches and installs a guest VMSA, this could lead
to a use-after-free of the physical page.

sev_snp_reload_vmsa() resolves the PFN via kvm_gmem_get_pfn() and directly
installs it into svm->vmcb->control.vmsa_pa without holding kvm->mmu_lock
or checking mmu_invalidate_retry_gfn(). It then drops the page reference.

If the page is subsequently freed by guest_memfd, KVM would retain the
stale physical address in vmsa_pa, and the hardware could write guest state
to the freed page during VMRUN.

> -	/* Mark the vCPU as runnable */
> -	kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);
> -

[ ... ]

> @@ -4062,6 +4059,15 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
>  	svm->sev_es.snp_pending_vmsa_gpa = INVALID_PAGE;
>  
>  	sev_snp_reload_vmsa(vcpu, gpa);
> +
> +	/*
> +	 * Mark the vCPU as runnable for CREATE requests, indicated by a valid
> +	 * VMSA GPA, even if installing the VMSA failed, so that KVM_RUN will
> +	 * fail instead of blocking indefinitely and hanging the vCPU, e.g. if
> +	 * the backing guest_memfd page is unavailable.
> +	 */
> +	if (VALID_PAGE(gpa))
> +		kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE);

[Severity: High]
Does this allow KVM to incorrectly execute VMRUN with a dummy unmeasured
host VMSA when the initial AP_CREATE VMSA load fails?

If a guest issues an AP_CREATE request with an invalid VMSA (such as having
no backing guest_memfd page) for a vCPU that has not yet successfully loaded
a guest VMSA, marking it RUNNABLE is intended to make pre_sev_run() fail
KVM_RUN.

However, because snp_has_guest_vmsa remains false on failure,
sev_es_init_vmcb() overwrites the INVALID_PAGE state with the KVM-allocated
fallback VMSA physical address:

	if (!svm->sev_es.snp_has_guest_vmsa) {
		if (svm->sev_es.vmsa)
			svm->vmcb->control.vmsa_pa = __pa(svm->sev_es.vmsa);
		else
			svm->vmcb->control.vmsa_pa = INVALID_PAGE;
	}

Because of this, pre_sev_run() will see a valid vmsa_pa and pass:

	if (is_sev_es_guest(vcpu) && !VALID_PAGE(svm->vmcb->control.vmsa_pa))
		return -EINVAL;

Will this result in a hardware RMP violation or guest state corruption when
KVM attempts VMRUN with the dummy VMSA?

>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260709204948.1988414-1-seanjc@google.com?part=3

Re: [PATCH v4 15/18] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] The patch breaks the GPA alignment for huge folios by passing the original, unaligned faulting `gfn` to `kvm_arch_gmem_prepare()`, resulting in a mismatch between the unaligned GPA and the aligned base PFN of the folio.
--

--- Patch [15]: [PATCH v4 15/18] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller ---
commit 30508b546ec3ffd0655d6bdccee0095661b93a2e
Author: Sean Christopherson <seanjc@google.com>

KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

This commit folds __kvm_gmem_prepare_folio() into kvm_gmem_prepare_folio(),
extending the #ifdef guard to cover the entirety of the function and
eliminating the tail-call helper.

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c

[ ... ]

> @@ -82,6 +69,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
>  static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
>  				  gfn_t gfn, struct folio *folio)
>  {
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
>  	pgoff_t index;
>  
>  	/*
> @@ -101,7 +89,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
>  	index = kvm_gmem_get_index(slot, gfn);
>  	index = ALIGN_DOWN(index, folio_nr_pages(folio));
>  
> -	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
> +	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn),
> +				     folio_file_pfn(folio, index),
> +				     folio_order(folio));

[Severity: Medium]
When folding this helper, did we lose the GFN alignment for huge folios?

In the old __kvm_gmem_prepare_folio() helper, the gfn was recalculated from
the aligned index:

	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;

By passing the original, unaligned faulting gfn to kvm_arch_gmem_prepare()
alongside the aligned base PFN from folio_file_pfn(), does this create a
mismatch between the unaligned GPA and the aligned base PFN for huge folios?

While guest_memfd currently restricts allocations to order-0 folios, this would
pass an offset GPA with a base PFN to the architecture hooks once huge folios
are supported, violating the API contract and potentially corrupting the guest
mapping (e.g., if a backend like SEV-SNP falls back to 4K mappings for a huge
folio).

> +#else
> +	return 0;
> +#endif
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260709204948.1988414-1-seanjc@google.com?part=15

Re: [PATCH v4 16/18] KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:

Pre-existing issues:
- [Critical] Data race between guest memory population and fault-driven zeroing leads to host kernel panic via RMP violation.
--

--- Patch [16]: [PATCH v4 16/18] KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare() ---
commit b12c982da03eae2bbf9f63075885746a3653b16b
Author: Sean Christopherson <seanjc@google.com>

KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()

This commit modifies the guest_memfd prepare() hook to explicitly pass the
number of pages, allowing prepare() and invalidate() hooks to be combined
into a single convert() hook. It also adds a WARN_ON_ONCE to the SNP code if
more than a single 4KiB page is prepared.

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index d5d3688eb1bdf..cd7a795834cb6 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c

[ ... ]

[Severity: Critical]
This is a pre-existing issue, but could a data race between guest memory
population and fault-driven zeroing lead to a host kernel panic?

Looking at __kvm_gmem_populate() in virt/kvm/guest_memfd.c:

	folio_unlock(folio);

	if (!kvm_range_has_memory_attributes(kvm, gfn, gfn + 1,
					     KVM_MEMORY_ATTRIBUTE_PRIVATE,
					     KVM_MEMORY_ATTRIBUTE_PRIVATE)) {
		ret = -EINVAL;
		goto out_put_folio;
	}

	ret = post_populate(kvm, gfn, pfn, src_page, opaque);
	if (!ret)
		folio_mark_uptodate(folio);

The folio is unlocked before the memory is populated or marked up-to-date.
If an unprivileged host userspace process concurrently triggers a page fault
on the same private GFN, could kvm_gmem_get_pfn() fetch this unlocked folio?

kvm_gmem_get_pfn() would observe the !uptodate folio and zero it via the
host direct map:

	if (!folio_test_uptodate(folio)) {
		clear_highpage(folio_page(folio, 0));
		folio_mark_uptodate(folio);
	}

If this host zeroing overlaps with or occurs after rmp_make_private() in the
population callback, wouldn't this host write cause an RMP violation and
crash the host kernel?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260709204948.1988414-1-seanjc@google.com?part=16

Re: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Medium] Aligning GFN down without validating memslot boundaries introduces host out-of-bounds physical memory access when preparing huge folios.

Pre-existing issues:
- [Critical] Concurrent execution of KVM_GMEM_POPULATE and vCPU page faults allows unprivileged userspace to trigger host hardware crashes and guest data corruption.
--

--- Patch [17]: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio ---
commit 5debe95f8712f79807df583e8d5e927a97b4c012
Author: Sean Christopherson <seanjc@google.com>

KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio

When preparing a guest_memfd folio, this patch aligns the gfn as well as
the pfn to preserve the effective gfn+pfn pair. This prevents a mismatch
set and unintentional guest crashes due to assigning the pfn at the wrong
gfn for huge folios.

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index cd7a795834cb6..4082cde3f3fca 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -87,8 +87,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
>  	 * checked when creating memslots.
>  	 */
>  	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, nr_pages));
> +	gfn = ALIGN_DOWN(gfn, nr_pages);
>  	index = kvm_gmem_get_index(slot, gfn);
> -	index = ALIGN_DOWN(index, nr_pages);

[Severity: Medium]
If a userspace caller creates a guest_memfd memslot with a base_gfn that is
not aligned to the huge folio size, will computing the index from the aligned
gfn result in a misaligned index?

When passed to kvm_arch_gmem_prepare(), couldn't this cause the iteration
to go past the end of the huge folio, resulting in an out-of-bounds host
physical memory access?

[Severity: Critical]
This is a pre-existing issue, but I noticed a potential race condition
between KVM_GMEM_POPULATE and vCPU page faults in this same file.

Looking at __kvm_gmem_populate(), the folio lock is dropped before calling
post_populate():

virt/kvm/guest_memfd.c:__kvm_gmem_populate() {
    ...
	folio_unlock(folio);
    ...
	ret = post_populate(kvm, gfn, pfn, src_page, opaque);
    ...
}

Could a concurrent vCPU page fault on the same GFN call kvm_gmem_get_pfn(),
acquire the folio lock, see that it is not uptodate, and clear the page
while the populate thread is executing post-populate updates like
KVM_SEV_SNP_LAUNCH_UPDATE?

virt/kvm/guest_memfd.c:kvm_gmem_get_pfn() {
    ...
	if (!folio_test_uptodate(folio)) {
		clear_highpage(folio_page(folio, 0));
    ...
}

Would this allow unprivileged userspace to trigger host hardware crashes
(Machine Check Exceptions) and guest data corruption simply by running a
vCPU thread concurrently with the population ioctl?

>  
>  	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn),
>  				     folio_file_pfn(folio, index), nr_pages,

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260709204948.1988414-1-seanjc@google.com?part=17

Re: [PATCH v4 04/18] KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if CONFIG_KVM_AMD_SEV=y

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

> Wire up the SEV-SNP guest_memfd kvm_x86_ops hooks if and only if SEV is
> actually enabled, and drop the now-unnecessary stubs.  Leaving the hooks
> NULL allows the static call infrastructure to elide the CALL+RET, and more
> importantly, referencing the hooks if and only if SEV support is enabled
> will allow conditionally definining the hooks using their corresponding
> HAVE_KVM_ARCH_GMEM_XXX Kconfig.
>
> No functional change intended.
>
> Cc: stable@vger.kernel.org # 6.12.x
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/kvm/svm/svm.c |  8 ++++----
>  arch/x86/kvm/svm/svm.h | 10 ----------
>  2 files changed, 4 insertions(+), 14 deletions(-)
>
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index ef69a51ab27f..79c818d91dda 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -5448,6 +5448,10 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
>
>  	.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
>  	.vm_move_enc_context_from = sev_vm_move_enc_context_from,
> +
> +	.gmem_prepare = sev_gmem_prepare,

Here .gmem_prepare assignment is gated on CONFIG_KVM_AMD_SEV but later
the definition of .gmem_prepare is gated on
CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE.

CONFIG_KVM_AMD_SEV selects CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE, but I
wonder if there's a chance due to config disconnects this will cause
compilation issues? Like .gmem_prepare will get assigned but it was not
declared? Or is that actually intentional?

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

> +	.gmem_invalidate = sev_gmem_invalidate,
> +	.gmem_max_mapping_level = sev_gmem_max_mapping_level,
>  #endif
>  	.check_emulate_instruction = svm_check_emulate_instruction,
>
> @@ -5459,10 +5463,6 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
>  	.vcpu_deliver_sipi_vector = svm_vcpu_deliver_sipi_vector,
>  	.vcpu_get_apicv_inhibit_reasons = avic_vcpu_get_apicv_inhibit_reasons,
>  	.alloc_apic_backing_page = svm_alloc_apic_backing_page,
> -
> -	.gmem_prepare = sev_gmem_prepare,
> -	.gmem_invalidate = sev_gmem_invalidate,
> -	.gmem_max_mapping_level = sev_gmem_max_mapping_level,
>  };
>
>  /*
>
> [...snip...]
>

Re: [PATCH v4 14/18] KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

> Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM
> SNP alreads logs more precise messages in all error paths, and whether or
> not failure to convert the pfn to private is "unexpected", i.e. warrants
> logging, is firmly an architecture specific detail.
>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  virt/kvm/guest_memfd.c | 10 +++-------
>  1 file changed, 3 insertions(+), 7 deletions(-)
>
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 7ad656e30f9f..f4f8635d1a10 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
>  	kvm_pfn_t pfn = folio_file_pfn(folio, index);
>  	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
> -	int rc = kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn), pfn, folio_order(folio));
> -	if (rc) {
> -		pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
> -				    index, gfn, pfn, rc);
> -		return rc;
> -	}
> -#endif
>
> +	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn), pfn, folio_order(folio));
> +#else
>  	return 0;
> +#endif
>  }
>
>  /*
> --
> 2.55.0.795.g602f6c329a-goog

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

Re: [PATCH v4 16/18] KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

> Tweak the guest_memfd prepare() hook to explicitly pass the number of pages
> to "prepare", so that the prepare() and invalidate() hooks can be combined
> into a single convert() hook, and because the existing code is outright
> broken if a guest_memfd folio is comprised of more than one page (which
> can't happen, yet).  guest_memfd aligns the pfn down, and the SNP code
> *tries* to create a corresponding huge entry, but if the RMP must use 4KiB
> entries for whatever reason, KVM will only "prepare" the first pfn, and not
> the actual pfn that needs to be converted/prepared.
>
> Alternatively, @max_order could simply be repurposed as _the_ @order, but
> that will fall apart when in-place conversion comes along, at which point
> KVM will need to deal with conversions that aren't strictly bound to a one
> folio.
>
> Note, kvm_gmem_prepare_folio() still mishandles hugepages, as it aligns
> only the pfn, i.e. would pass the wrong gfn+pfn pair.  This will be fixed
> shortly.
>
> WARN in the SNP code if the number of pages to prepare is anything other
> than '1', i.e. if guest_memfd is trying to prepare/convert more than a
> single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion
> greater than order-0 folios.
>
> Opportunistically convert the existing WARN in the prepare flow to a only
> fire once, e.g. so that a guest_memfd bug doesn't unintentionally Dos the
> kernel by spamming the log.
>
> Fixes: b85524314a3d ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")
> Signed-off-by: Sean Christopherson <seanjc@google.com>
>
> [...snip...]
>

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

Re: [PATCH v4 04/18] KVM: SEV: Wire up kvm_x86_ops.gmem_xxx() if and only if CONFIG_KVM_AMD_SEV=y

[Sean Christopherson]

On Thu, Jul 09, 2026, Ackerley Tng wrote:
> Sean Christopherson <seanjc@google.com> writes:
> 
> > Wire up the SEV-SNP guest_memfd kvm_x86_ops hooks if and only if SEV is
> > actually enabled, and drop the now-unnecessary stubs.  Leaving the hooks
> > NULL allows the static call infrastructure to elide the CALL+RET, and more
> > importantly, referencing the hooks if and only if SEV support is enabled
> > will allow conditionally definining the hooks using their corresponding
> > HAVE_KVM_ARCH_GMEM_XXX Kconfig.
> >
> > No functional change intended.
> >
> > Cc: stable@vger.kernel.org # 6.12.x
> > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > ---
> >  arch/x86/kvm/svm/svm.c |  8 ++++----
> >  arch/x86/kvm/svm/svm.h | 10 ----------
> >  2 files changed, 4 insertions(+), 14 deletions(-)
> >
> > diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> > index ef69a51ab27f..79c818d91dda 100644
> > --- a/arch/x86/kvm/svm/svm.c
> > +++ b/arch/x86/kvm/svm/svm.c
> > @@ -5448,6 +5448,10 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
> >
> >  	.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
> >  	.vm_move_enc_context_from = sev_vm_move_enc_context_from,
> > +
> > +	.gmem_prepare = sev_gmem_prepare,
> 
> Here .gmem_prepare assignment is gated on CONFIG_KVM_AMD_SEV but later
> the definition of .gmem_prepare is gated on
> CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE.
> 
> CONFIG_KVM_AMD_SEV selects CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE, but I
> wonder if there's a chance due to config disconnects this will cause
> compilation issues? Like .gmem_prepare will get assigned but it was not
> declared? Or is that actually intentional?

Intentional.  If we end up with KVM_AMD_SEV=y, but HAVE_KVM_ARCH_GMEM_PREPARE=n,
then I definitely want the build to fail.

Re: [PATCH v4 01/18] KVM: SEV: Track the GPA of the guest-controlled VMSA used for SNP guests

[Sean Christopherson]

On Thu, Jul 09, 2026, sashiko-bot@kernel.org wrote:
> > diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> > index 74fb15551e83f..827f5dc061023 100644
> > --- a/arch/x86/kvm/svm/sev.c
> > +++ b/arch/x86/kvm/svm/sev.c
> [ ... ]
> > @@ -4039,6 +4040,7 @@ static void sev_snp_init_protected_guest_state(struct kvm_vcpu *vcpu)
> >  	svm->sev_es.snp_has_guest_vmsa = true;
> >  
> >  	/* Use the new VMSA */
> > +	svm->sev_es.snp_guest_vmsa_gpa = gfn_to_gpa(gfn);
> >  	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
> 
> [Severity: Critical]
> This is a pre-existing issue, but does this bypass the MMU invalidation retry
> protocol?
> 
> Looking at sev_snp_init_protected_guest_state(), KVM resolves the PFN via
> kvm_gmem_get_pfn(), directly installs it into the hardware VMCB, and drops
> the page reference:
> 
> 	if (kvm_gmem_get_pfn(vcpu->kvm, slot, gfn, &pfn, &page, NULL))
> 		return;
> 	...
> 	svm->vmcb->control.vmsa_pa = pfn_to_hpa(pfn);
> 	...
> 	kvm_release_page_clean(page);
> 
> Can this race with a concurrent guest_memfd hole punch if kvm->mmu_lock isn't
> held and the invalidation sequence counter isn't verified?
> 
> If the page is freed back to the host allocator, could the hardware continue
> to use a freed host physical page for the VMSA, leading to a physical
> use-after-free?

Read the cover letter?

Re: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio

[Ackerley Tng]

sashiko-bot@kernel.org writes:

> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
>
> New issues:
> - [Medium] Aligning GFN down without validating memslot boundaries introduces host out-of-bounds physical memory access when preparing huge folios.
>
> Pre-existing issues:
> - [Critical] Concurrent execution of KVM_GMEM_POPULATE and vCPU page faults allows unprivileged userspace to trigger host hardware crashes and guest data corruption.
> --
>
> --- Patch [17]: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio ---
> commit 5debe95f8712f79807df583e8d5e927a97b4c012
> Author: Sean Christopherson <seanjc@google.com>
>
> KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio
>
> When preparing a guest_memfd folio, this patch aligns the gfn as well as
> the pfn to preserve the effective gfn+pfn pair. This prevents a mismatch
> set and unintentional guest crashes due to assigning the pfn at the wrong
> gfn for huge folios.
>
>> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
>> index cd7a795834cb6..4082cde3f3fca 100644
>> --- a/virt/kvm/guest_memfd.c
>> +++ b/virt/kvm/guest_memfd.c
>> @@ -87,8 +87,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
>>  	 * checked when creating memslots.
>>  	 */
>>  	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, nr_pages));
>> +	gfn = ALIGN_DOWN(gfn, nr_pages);
>>  	index = kvm_gmem_get_index(slot, gfn);
>> -	index = ALIGN_DOWN(index, nr_pages);
>
> [Severity: Medium]
> If a userspace caller creates a guest_memfd memslot with a base_gfn that is
> not aligned to the huge folio size, will computing the index from the aligned
> gfn result in a misaligned index?
>
> When passed to kvm_arch_gmem_prepare(), couldn't this cause the iteration
> to go past the end of the huge folio, resulting in an out-of-bounds host
> physical memory access?
>

This sounds like it would be a problem. We could perhaps leave this
until huge pages are supported?

> [Severity: Critical]
> This is a pre-existing issue, but I noticed a potential race condition
> between KVM_GMEM_POPULATE and vCPU page faults in this same file.
>

We're good, vCPU page faults can't happen until the VM is finalized,
populate must be done before finalizing.

>
> [...snip...]
>

Re: [PATCH v4 10/18] KVM: SEV: Forcefully invalidate SNP VMSA if its backing gmem page is zapped

[Sean Christopherson]

On Thu, Jul 09, 2026, sashiko-bot@kernel.org wrote:
> > diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> > --- a/virt/kvm/guest_memfd.c
> > +++ b/virt/kvm/guest_memfd.c
> [ ... ]
> > @@ -185,6 +185,10 @@ static void __kvm_gmem_invalidate_start(struct gmem_file *f, pgoff_t start,
> >  		}
> >  
> >  		flush |= kvm_mmu_unmap_gfn_range(kvm, &gfn_range);
> > +
> > +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> > +		kvm_arch_gmem_invalidate_range(kvm, &gfn_range);
> > +#endif
> >  	}
> >  
> >  	if (flush)
> 
> [Severity: Critical]
> Does this invalidation path need a way to guarantee a synchronous wait before
> returning?
> 
> Since VMSA pages typically do not have TDP MMU SPTEs,
> kvm_mmu_unmap_gfn_range() will return false, leaving flush as false.
> 
> Because flush remains false, the subsequent kvm_flush_remote_tlbs(kvm) call
> is skipped.
> 
> Meanwhile, sev_gmem_invalidate_range() relies on KVM_REQ_VMSA_PAGE_RELOAD
> and kvm_make_request_and_kick() to send an asynchronous IPI to the vCPU:
> 
> arch/x86/kvm/svm/sev.c:sev_gmem_invalidate_range() {
>     ...
>     kvm_make_request_and_kick(KVM_REQ_VMSA_PAGE_RELOAD, vcpu);
>     ...
> }
> 
> Because the IPI is asynchronous and the TLB flush (which would otherwise
> provide a synchronous wait) is skipped, the host could proceed to free the
> VMSA page before the vCPU actually drops it and exits guest mode.

No, KVM_REQ_VMSA_PAGE_RELOAD is declared with KVM_REQUEST_WAIT, i.e. KVM will
ensure the target CPUs have received the IPI before continuing on.  And the
presence of the request will prevent them from re-entering the guest with the
now-defunct VMSA.

Re: [PATCH v4 15/18] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

[Sean Christopherson]

On Thu, Jul 09, 2026, sashiko-bot@kernel.org wrote:
> > diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> > --- a/virt/kvm/guest_memfd.c
> > +++ b/virt/kvm/guest_memfd.c
> 
> [ ... ]
> 
> > @@ -82,6 +69,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
> >  static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
> >  				  gfn_t gfn, struct folio *folio)
> >  {
> > +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
> >  	pgoff_t index;
> >  
> >  	/*
> > @@ -101,7 +89,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
> >  	index = kvm_gmem_get_index(slot, gfn);
> >  	index = ALIGN_DOWN(index, folio_nr_pages(folio));
> >  
> > -	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
> > +	return kvm_arch_gmem_prepare(kvm, gfn_to_gpa(gfn),
> > +				     folio_file_pfn(folio, index),
> > +				     folio_order(folio));
> 
> [Severity: Medium]
> When folding this helper, did we lose the GFN alignment for huge folios?

Ugh, I totally missed that, obviously.  Aligning the index in the caller using
a gfn, and then subtly computing a new gfn is just mean.  I'll fix this by
squashing in "Align the gfn as well as the pfn when "preparing" a folio"

> In the old __kvm_gmem_prepare_folio() helper, the gfn was recalculated from
> the aligned index:
> 
> 	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
> 
> By passing the original, unaligned faulting gfn to kvm_arch_gmem_prepare()
> alongside the aligned base PFN from folio_file_pfn(), does this create a
> mismatch between the unaligned GPA and the aligned base PFN for huge folios?
> 
> While guest_memfd currently restricts allocations to order-0 folios, this would
> pass an offset GPA with a base PFN to the architecture hooks once huge folios
> are supported, violating the API contract and potentially corrupting the guest
> mapping (e.g., if a backend like SEV-SNP falls back to 4K mappings for a huge
> folio).
> 
> > +#else
> > +	return 0;
> > +#endif
> >  }
> 
> -- 
> Sashiko AI review · https://sashiko.dev/#/patchset/20260709204948.1988414-1-seanjc@google.com?part=15

Re: [PATCH v4 17/18] KVM: guest_memfd: Align the gfn as well as the pfn when "preparing" a folio

[Sean Christopherson]

On Thu, Jul 09, 2026, Ackerley Tng wrote:
> sashiko-bot@kernel.org writes:
> >> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> >> index cd7a795834cb6..4082cde3f3fca 100644
> >> --- a/virt/kvm/guest_memfd.c
> >> +++ b/virt/kvm/guest_memfd.c
> >> @@ -87,8 +87,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
> >>  	 * checked when creating memslots.
> >>  	 */
> >>  	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, nr_pages));
> >> +	gfn = ALIGN_DOWN(gfn, nr_pages);
> >>  	index = kvm_gmem_get_index(slot, gfn);
> >> -	index = ALIGN_DOWN(index, nr_pages);
> >
> > [Severity: Medium]
> > If a userspace caller creates a guest_memfd memslot with a base_gfn that is
> > not aligned to the huge folio size, will computing the index from the aligned
> > gfn result in a misaligned index?
> >
> > When passed to kvm_arch_gmem_prepare(), couldn't this cause the iteration
> > to go past the end of the huge folio, resulting in an out-of-bounds host
> > physical memory access?
> >
> 
> This sounds like it would be a problem. We could perhaps leave this
> until huge pages are supported?

Yeah, for sure.

Re: [PATCH v4 18/18] KVM: guest_memfd: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

>
> [...snip...]
>
> @@ -532,7 +531,7 @@ static const struct address_space_operations kvm_gmem_aops = {
>  	.dirty_folio = noop_dirty_folio,
>  	.migrate_folio	= kvm_gmem_migrate_folio,
>  	.error_remove_folio = kvm_gmem_error_folio,
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
>  	.free_folio = kvm_gmem_free_folio,

This part looks odd to have CONVERT gate free_folio, maybe this is where
it's nice to have CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT ifdef the contents
of free_folio above.

Not a huge deal, thanks for this cleanup!

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

>  #endif
>  };
> @@ -802,7 +801,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
>  		folio_mark_uptodate(folio);
>  	}
>
> -	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
> +	r = kvm_gmem_make_private(kvm, slot, gfn, folio);
>
>  	folio_unlock(folio);
>
> --
> 2.55.0.795.g602f6c329a-goog

Re: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

> Pass a GPA instead of a GFN to kvm_arch_gmem_prepare() so that the GFN/GPA
> can be made optional in the future, i.e. so that guest_memfd can pass
> INVALID_GPA.  This will allow reworking the hook into a general .convert()

How about defining a INVALID_GFN instead of using gpa just so
INVALID_GPA can be passed? The SNP side takes a gfn, so it seems like
the 2 PAGE_SHIFTs are extra work.

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

> hook, i.e. so that it can be used for both "make private" and "make shared"
> operations, at which point guest_memfd may not have a corresponding GFN,
> e.g. when converting memory back to a shared state when a folio is freed.
>
> Opportunistically swap the ordering of @pfn and @gpa params for
> kvm_x86_ops.gmem_prepare() to match that of kvm_arch_gmem_prepare().
>
> No functional change intended.
>
>
> [...snip...]
>

Re: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook

[Sean Christopherson]

On Thu, Jul 09, 2026, Ackerley Tng wrote:
> Sean Christopherson <seanjc@google.com> writes:
> 
> > Pass a GPA instead of a GFN to kvm_arch_gmem_prepare() so that the GFN/GPA
> > can be made optional in the future, i.e. so that guest_memfd can pass
> > INVALID_GPA.  This will allow reworking the hook into a general .convert()
> 
> How about defining a INVALID_GFN instead of using gpa just so
> INVALID_GPA can be passed? The SNP side takes a gfn, so it seems like
> the 2 PAGE_SHIFTs are extra work.

They are, but I am very hesitant to define INVALID_GFN as I'm worried about the
potential for subtle bugs if both INVALID_GPA and INVALID_GFN exist.  E.g. I
almost made a goof in this exact series where I was going to do:

	gfn_t gfn = gpa_to_gfn(svm->sev_es.snp_guest_vmsa_gpa);

	...

	if (gfn == INVALID_GFN)
		return;

Where snp_guest_vmsa_gpa is set to INVALID_GPA.

I suppose we could define INVALID_GFN to gpa_to_gfn(INVALID_GPA), but that feels
weird, and I feel more comfortable having magic invalid values in only one domain.

Re: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

> On Thu, Jul 09, 2026, Ackerley Tng wrote:
>> Sean Christopherson <seanjc@google.com> writes:
>>
>> > Pass a GPA instead of a GFN to kvm_arch_gmem_prepare() so that the GFN/GPA
>> > can be made optional in the future, i.e. so that guest_memfd can pass
>> > INVALID_GPA.  This will allow reworking the hook into a general .convert()
>>
>> How about defining a INVALID_GFN instead of using gpa just so
>> INVALID_GPA can be passed? The SNP side takes a gfn, so it seems like
>> the 2 PAGE_SHIFTs are extra work.
>
> They are, but I am very hesitant to define INVALID_GFN as I'm worried about the
> potential for subtle bugs if both INVALID_GPA and INVALID_GFN exist.  E.g. I
> almost made a goof in this exact series where I was going to do:
>
> 	gfn_t gfn = gpa_to_gfn(svm->sev_es.snp_guest_vmsa_gpa);
>
> 	...
>
> 	if (gfn == INVALID_GFN)

When you did this check INVALID_GFN was defined as (~0) too I suppose...

> 		return;
>
> Where snp_guest_vmsa_gpa is set to INVALID_GPA.
>
> I suppose we could define INVALID_GFN to gpa_to_gfn(INVALID_GPA), but that feels
> weird, and I feel more comfortable having magic invalid values in only one domain.

Maybe INVALID_GPA and INVALID_GFN should be defined such that both
gpa_to_gfn and gfn_to_gpa are correct for both. Isn't the relationship
between gpa and gfn always supposed to be always a fixed shift apart?

When a GPA is passed to the arch function, it almost feels like now the
arch should be validating PAGE_ALIGNED(gpa).

Mentioned this off-list, another thing that seems incongruent is that
one parameter is in the address domain (gpa) and the other is in the
page frame domain (pfn).

Re: [PATCH v4 13/18] KVM: guest_memfd: Pass GPA, not GFN, to prepare() hook

[Sean Christopherson]

On Fri, Jul 10, 2026, Ackerley Tng wrote:
> Sean Christopherson <seanjc@google.com> writes:
> 
> > On Thu, Jul 09, 2026, Ackerley Tng wrote:
> >> Sean Christopherson <seanjc@google.com> writes:
> >>
> >> > Pass a GPA instead of a GFN to kvm_arch_gmem_prepare() so that the GFN/GPA
> >> > can be made optional in the future, i.e. so that guest_memfd can pass
> >> > INVALID_GPA.  This will allow reworking the hook into a general .convert()
> >>
> >> How about defining a INVALID_GFN instead of using gpa just so
> >> INVALID_GPA can be passed? The SNP side takes a gfn, so it seems like
> >> the 2 PAGE_SHIFTs are extra work.
> >
> > They are, but I am very hesitant to define INVALID_GFN as I'm worried about the
> > potential for subtle bugs if both INVALID_GPA and INVALID_GFN exist.  E.g. I
> > almost made a goof in this exact series where I was going to do:
> >
> > 	gfn_t gfn = gpa_to_gfn(svm->sev_es.snp_guest_vmsa_gpa);
> >
> > 	...
> >
> > 	if (gfn == INVALID_GFN)
> 
> When you did this check INVALID_GFN was defined as (~0) too I suppose...

Ya.  I didn't actually get that far, because INVALID_GFN didn't exist, but that
was my plan.

> > 		return;
> >
> > Where snp_guest_vmsa_gpa is set to INVALID_GPA.
> >
> > I suppose we could define INVALID_GFN to gpa_to_gfn(INVALID_GPA), but that feels
> > weird, and I feel more comfortable having magic invalid values in only one domain.
> 
> Maybe INVALID_GPA and INVALID_GFN should be defined such that both
> gpa_to_gfn and gfn_to_gpa are correct for both.

Oh, _that_ was actually what I almost broke.  I was going to track the gfn instead
of the gpa, and then do:

	svm->sev_es.snp_guest_vmsa_gfn = INVALID_GFN;

	...

	gpa = gfn_to_gpa(svm->sev_es.snp_guest_vmsa_gfn);

	if (gpa == INVALID_GPA)
		return;

That would have failed miserably, because even if if INVALID_GFN is -1ull, the
lower bits would be zero after the shift, i.e. wouldn't match INVALID_GPA.

I _think_ make that work, by doing something like:

  #define INVALID_GFN	((~(gfn_t)0) >> PAGE_SHIFT)
  #define INVALID_GPA	((gpa_t)INVALID_GFN << PAGE_SHIFT)

Which I actually don't mind conceptually, but I'm still not sure it's a good
idea in practice.  For whatever reason, converting between INVALID_GFN and
INVALID_GPA just feels wrong.

> Isn't the relationship between gpa and gfn always supposed to be always a
> fixed shift apart?

For valid GPAs, yes.  But if a value is completely bogus, is there really any
relationship?

> When a GPA is passed to the arch function, it almost feels like now the
> arch should be validating PAGE_ALIGNED(gpa).

Not sure I follow this.

> Mentioned this off-list, another thing that seems incongruent is that
> one parameter is in the address domain (gpa) and the other is in the
> page frame domain (pfn).

Yeah, I definitely don't love this either.  Hrm.

Aha!  Idea!  What if we just kinda sorta punt on the INVALID_GFN concept for the
moment?  This series doesn't actually _need_ to detect an INVALID_GFN, the only
"usage" is in a sanity check:

       if (to_private) {
               if (WARN_ON_ONCE(!kvm || gpa == INVALID_GPA))
                       return -EIO;

               return sev_gmem_make_private(kvm, gpa, pfn, nr_pages, order);
       }

and that can very easily be this, which is even better from a safety perspective,
and is actually somewhat sensible given that it also ensures KVM isn't trying to
assign a gfn with the C-bit set.  (Untested, i.e. I may botched the math, but the
idea is sound).

	if (to_private) {
		if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SIZE)))
			return -EIO;

		return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order);
	}

Then in kvm_gmem_free_folio(), open code -1ull for the gfn:

	WARN_ON_ONCE(kvm_arch_gmem_convert(NULL, -1ull,
					   folio_file_pfn(folio, 0),
					   folio_nr_pages(folio),
					   folio_order(folio), false));

Which is all a little gross?  But I think it's less gross than passing a gpa, and
we don't risk delaying this and the in-place conversion series trying to find the
perfect answer.