[PATCH v5 0/7] KVM: guest_memfd: reclaim()/convert() cleanups

[PATCH v5 0/7] KVM: guest_memfd: reclaim()/convert() cleanups

[Sean Christopherson]

These are the cleanups from the SNP VMSA series.  I kept it as v5 to maintain
the paper trail.

Rework prepare()+invalidate() into convert()+reclaim(), and have the x86
backend use a single convert() flow for both (only SNP needs the hooks, and
for SNP, reclaim just means "make shared").  The goals are to eliminate the
misleading use of the term "invalidate", to drop a kvm_x86_ops callback, and to
prepare for in-place conversion, e.g. so that KVM ends up with something like:

	if (!to_private)
		kvm_gmem_make_shared(inode, start, end);

instead of a funky:

	if (!to_private)
		kvm_gmem_invalidate(inode, start, end);

v5
 - Drop the VMSA fixes (already applied).
 - Keep a dedicated reclaim() flow. [Fuad, Ackerley]
 - Pass a gfn, not a gpa, to convert() / reclaim(). [Ackerley]
 - Collect reviews. [Ackerley]
 - Align the gfn when folding __kvm_gmem_prepare_folio() into its caller. [Sashiko]
 
v4:
 - https://lore.kernel.org/all/20260709204948.1988414-1-seanjc@google.com

 - Collect reviews. [Mike, Ackerley]
 - Rework prepare()+invalidate() into a common convert() instead of simply
   renaming invalidate() to reclaim_memory(). [Ackerley]
 - Fix several (mostly benign) issues in the "prepare" flow.

v3:
 - https://lore.kernel.org/all/20260630222607.497895-1-seanjc@google.com
 - Ensure disabling quirks while the VM is live won't result in KVM skipping
   the back-half of "zap all fast". [Sashiko]
 - s/gmem_free_folio/gmem_reclaim_memory. [Ackerley]
 - Use {READ,WRITE}_ONCE() when accessing the guest's VMSA GPA outside of the
   per-vCPU mutex, and comment. [Sashiko]

v2:
 - https://lore.kernel.org/all/20260626231416.3943216-1-seanjc@google.com
 - Invalidate VMSAs if the memslot is DELETED or MOVED. [Sashiko]
 - Limit stable@ patches without a Fixes to 6.12+.

v1: https://lore.kernel.org/all/20260625222229.3367197-2-seanjc@google.com

Sean Christopherson (7):
  KVM: guest_memfd: Plumb the number of pages and max order into
    .invalidate()
  KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and
    isolate it
  KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare()
    failure
  KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller
  KVM: guest_memfd: Explicitly pass number of pages to
    kvm_arch_gmem_prepare()
  KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into
    .gmem_convert()
  KVM: guest_memfd: Rework PREPARE config and hook into a more generic
    CONVERT

 arch/x86/include/asm/kvm-x86-ops.h |  5 ++--
 arch/x86/include/asm/kvm_host.h    |  6 ++--
 arch/x86/kvm/Kconfig               |  3 +-
 arch/x86/kvm/svm/sev.c             | 23 +++++++++++++--
 arch/x86/kvm/svm/svm.c             |  3 +-
 arch/x86/kvm/svm/svm.h             |  4 +--
 arch/x86/kvm/x86.c                 | 18 +++++++-----
 include/linux/kvm_host.h           | 10 +++++--
 virt/kvm/Kconfig                   |  6 +++-
 virt/kvm/guest_memfd.c             | 46 +++++++++++-------------------
 10 files changed, 70 insertions(+), 54 deletions(-)


base-commit: 6bc96b971766fbbbbdd9fb2642cedacaf02da957
-- 
2.55.0.141.g00534a21ce-goog

[PATCH v5 1/7] KVM: guest_memfd: Plumb the number of pages and max order into .invalidate()

[Sean Christopherson]

Plumb the number of pages to "invalidate", i.e. reclaim, along with the max
order with which the pages can be mapped.  This will allow x86 to combine
its vendor-specific implementations for conversion and reclaim, a.k.a.
preparation and invalidation, into a single hook.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/x86.c       | 4 ++--
 include/linux/kvm_host.h | 2 +-
 virt/kvm/guest_memfd.c   | 7 ++-----
 3 files changed, 5 insertions(+), 8 deletions(-)

diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 1f5dc685f049..a2cbf6fae118 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10627,9 +10627,9 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
-	kvm_x86_call(gmem_invalidate)(start, end);
+	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
 }
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 9db6eb4023c4..996b39ebdb68 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2614,7 +2614,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
 
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 171a9da1b685..99c070bfd446 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -530,11 +530,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 static void kvm_gmem_free_folio(struct folio *folio)
 {
-	struct page *page = folio_page(folio, 0);
-	kvm_pfn_t pfn = page_to_pfn(page);
-	int order = folio_order(folio);
-
-	kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));
+	kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),
+				 folio_order(folio));
 }
 #endif
 
-- 
2.55.0.141.g00534a21ce-goog

[PATCH v5 2/7] KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and isolate it

[Sean Christopherson]

Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its
own RECLAIM Kconfig, as the hook is called when a folio is freed, which is
far too late and lacks sufficient information for KVM to actually
invalidate its usage of the memory.  E.g. SNP uses the hook to convert
memory back to SHARED so that it can be safely accessed by the host, there
is no invalidation of guest mappings anywhere.  Isolating the hook will
also allow pKVM on arm64 to opt-in to reclaim() without also having to
differentiate between reclaim and conversions to shared for active VMs.

Keep guest_memfd's trampoline, even though it would be trivial to wire up
.free_folio() directly to an arch callback, to avoid bleeding guest_memfd
internals into arch code (specifically, avoid referencing folios in arch
code).

Leave the kvm_x86_ops hook as-is for the moment, as the hook can be dropped
entirely ("reclaim" on SNP is the same as convert-to-shared).

Cc: Fuad Tabba <tabba@google.com>
Cc: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/kvm/Kconfig     | 1 +
 arch/x86/kvm/x86.c       | 7 +++++--
 include/linux/kvm_host.h | 5 ++++-
 virt/kvm/Kconfig         | 4 ++++
 virt/kvm/guest_memfd.c   | 8 ++++----
 5 files changed, 18 insertions(+), 7 deletions(-)

diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index 801bf9e520db..e0e7ad015839 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -161,6 +161,7 @@ config KVM_AMD_SEV
 	select ARCH_HAS_CC_PLATFORM
 	select KVM_GENERIC_MEMORY_ATTRIBUTES
 	select HAVE_KVM_ARCH_GMEM_PREPARE
+	select HAVE_KVM_ARCH_GMEM_RECLAIM
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
 	select HAVE_KVM_ARCH_GMEM_POPULATE
 	help
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index a2cbf6fae118..08d24c6fffa4 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10626,11 +10626,14 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
 }
 #endif
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
 	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
 }
+#endif
+
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	kvm_x86_call(gmem_invalidate_range)(kvm, range);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 996b39ebdb68..ee87ddd3294f 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2613,8 +2613,11 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
 		       kvm_gmem_populate_cb post_populate, void *opaque);
 #endif
 
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
+void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
+#endif
+
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
-void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
 void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
 
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index 794976b88c6f..617876993225 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -111,6 +111,10 @@ config HAVE_KVM_ARCH_GMEM_PREPARE
        bool
        depends on KVM_GUEST_MEMFD
 
+config HAVE_KVM_ARCH_GMEM_RECLAIM
+       bool
+       depends on KVM_GUEST_MEMFD
+
 config HAVE_KVM_ARCH_GMEM_INVALIDATE
        bool
        depends on KVM_GUEST_MEMFD
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 99c070bfd446..d5d227476cda 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -527,11 +527,11 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
 	return MF_DELAYED;
 }
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 static void kvm_gmem_free_folio(struct folio *folio)
 {
-	kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),
-				 folio_order(folio));
+	kvm_arch_gmem_reclaim(folio_file_pfn(folio, 0), folio_nr_pages(folio),
+			      folio_order(folio));
 }
 #endif
 
@@ -539,7 +539,7 @@ static const struct address_space_operations kvm_gmem_aops = {
 	.dirty_folio = noop_dirty_folio,
 	.migrate_folio	= kvm_gmem_migrate_folio,
 	.error_remove_folio = kvm_gmem_error_folio,
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 	.free_folio = kvm_gmem_free_folio,
 #endif
 };
-- 
2.55.0.141.g00534a21ce-goog

[PATCH v5 3/7] KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure

[Sean Christopherson]

Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM
SNP alreads logs more precise messages in all error paths, and whether or
not failure to convert the pfn to private is "unexpected", i.e. warrants
logging, is firmly an architecture specific detail.

Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/kvm/guest_memfd.c | 10 +++-------
 1 file changed, 3 insertions(+), 7 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index d5d227476cda..62c2c157f0c1 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 	kvm_pfn_t pfn = folio_file_pfn(folio, index);
 	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-	int rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
-	if (rc) {
-		pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
-				    index, gfn, pfn, rc);
-		return rc;
-	}
-#endif
 
+	return kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
+#else
 	return 0;
+#endif
 }
 
 /*
-- 
2.55.0.141.g00534a21ce-goog

[PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

[Sean Christopherson]

Fold __kvm_gmem_prepare_folio() into kvm_gmem_prepare_folio() and extend
the #ifdef guard to cover all of the kvm_gmem_prepare_folio().  Take care
to align the gfn _before_ computing the index, otherwise KVM would end up
with a mismatched gfn+pfn pair for folios larger than order-0 (this detail
was subtly handled by recomputing gfn in the helper using the aligned pfn).

Providing a single-use, ~2 line tail-call helper adds no value.  In fact,
due to having to reconstitute the gfn, *and* effectively hiding the
alignment adjustment for hugepages, the separate helper is very arguably a
net negative.

Opportunistically convert the existing WARN in the prepare flow to a only
fire once, e.g. so that a guest_memfd bug doesn't unintentionally Dos the
kernel by spamming the log.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 virt/kvm/guest_memfd.c | 24 ++++++++----------------
 1 file changed, 8 insertions(+), 16 deletions(-)

diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 62c2c157f0c1..f9a1d6d64f6c 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -60,19 +60,6 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
 	return gfn - slot->base_gfn + slot->gmem.pgoff;
 }
 
-static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
-				    pgoff_t index, struct folio *folio)
-{
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	kvm_pfn_t pfn = folio_file_pfn(folio, index);
-	gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
-
-	return kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
-#else
-	return 0;
-#endif
-}
-
 /*
  * Process @folio, which contains @gfn, so that the guest can use it.
  * The folio must be locked and the gfn must be contained in @slot.
@@ -82,6 +69,7 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
 static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 				  gfn_t gfn, struct folio *folio)
 {
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
 	pgoff_t index;
 
 	/*
@@ -97,11 +85,15 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and
 	 * checked when creating memslots.
 	 */
-	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
+	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
 	index = kvm_gmem_get_index(slot, gfn);
-	index = ALIGN_DOWN(index, folio_nr_pages(folio));
 
-	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
+	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
+				     folio_order(folio));
+#else
+	return 0;
+#endif
 }
 
 /*
-- 
2.55.0.141.g00534a21ce-goog

[PATCH v5 5/7] KVM: guest_memfd: Explicitly pass number of pages to kvm_arch_gmem_prepare()

[Sean Christopherson]

Tweak the guest_memfd prepare() hook to explicitly pass the number of pages
to "prepare", so that the prepare() and invalidate() hooks can be combined
into a single convert() hook, and because the existing code is outright
broken if a guest_memfd folio is comprised of more than one page (which
can't happen, yet).  guest_memfd aligns the pfn down, and the SNP code
*tries* to create a corresponding huge entry, but if the RMP must use 4KiB
entries for whatever reason, KVM will only "prepare" the first pfn, and not
the actual pfn that needs to be converted/prepared.

Alternatively, @max_order could simply be repurposed as _the_ @order, but
that will fall apart when in-place conversion comes along, at which point
KVM will need to deal with conversions that aren't strictly bound to a one
folio.

WARN in the SNP code if the number of pages to prepare is anything other
than '1', i.e. if guest_memfd is trying to prepare/convert more than a
single 4KiB page, as sev_gmem_prepare() doesn't actually handle conversion
greater than order-0 folios.

Opportunistically swap the ordering of @pfn and @gfn params for
kvm_x86_ops.gmem_prepare() to match that of kvm_arch_gmem_prepare().

Fixes: b85524314a3d ("KVM: guest_memfd: delay kvm_gmem_prepare_folio() until the memory is passed to the guest")
Reviewed-by: Ackerley Tng <ackerleytng@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/kvm_host.h | 3 ++-
 arch/x86/kvm/svm/sev.c          | 6 +++++-
 arch/x86/kvm/svm/svm.h          | 3 ++-
 arch/x86/kvm/x86.c              | 5 +++--
 include/linux/kvm_host.h        | 3 ++-
 virt/kvm/guest_memfd.c          | 7 ++++---
 6 files changed, 18 insertions(+), 9 deletions(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 58f156ae31e7..e7df537d36d5 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1725,7 +1725,8 @@ struct kvm_x86_ops {
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	int (*gmem_prepare)(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
+	int (*gmem_prepare)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order);
 #endif
 	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 82b983484af2..adc58138fdd0 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -5114,7 +5114,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
 	return false;
 }
 
-int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
+int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
 	kvm_pfn_t pfn_aligned;
@@ -5125,6 +5126,9 @@ int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order)
 	if (!sev_snp_guest(kvm))
 		return 0;
 
+	if (WARN_ON_ONCE(nr_pages != 1))
+		return -EIO;
+
 	rc = snp_lookup_rmpentry(pfn, &assigned, &level);
 	if (rc) {
 		pr_err_ratelimited("SEV: Failed to look up RMP entry: GFN %llx PFN %llx error %d\n",
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index af25e4b56212..e35419456648 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1009,7 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
-int sev_gmem_prepare(struct kvm *kvm, kvm_pfn_t pfn, gfn_t gfn, int max_order);
+int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order);
 void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 08d24c6fffa4..d484d8b509b1 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10620,9 +10620,10 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 }
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order)
+int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order)
 {
-	return kvm_x86_call(gmem_prepare)(kvm, pfn, gfn, max_order);
+	return kvm_x86_call(gmem_prepare)(kvm, gfn, pfn, nr_pages, max_order);
 }
 #endif
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ee87ddd3294f..0a06234a8a8b 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2580,7 +2580,8 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_order);
+int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index f9a1d6d64f6c..0f6745f3b919 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -70,6 +70,7 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 				  gfn_t gfn, struct folio *folio)
 {
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
+	unsigned long nr_pages = folio_nr_pages(folio);
 	pgoff_t index;
 
 	/*
@@ -85,12 +86,12 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	 * The order will be passed when creating the guest_memfd, and
 	 * checked when creating memslots.
 	 */
-	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
-	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
+	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, nr_pages));
+	gfn = ALIGN_DOWN(gfn, nr_pages);
 	index = kvm_gmem_get_index(slot, gfn);
 
 	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
-				     folio_order(folio));
+				     nr_pages, folio_order(folio));
 #else
 	return 0;
 #endif
-- 
2.55.0.141.g00534a21ce-goog

[PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

[Sean Christopherson]

Smush x86's prepare() and invalidate() hooks into a common convert() flow,
as they are effectively two sides of the same coin for SNP: they're invoked
when private/shared memory is about to made accessible/visible to the guest
or host.  I.e. prepare() is really "make private", and invalidate() is
really "make shared".

Using a single hook will yield more intuitive code when in-place conversion
comes along.

For all intents and purposes, no functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/kvm-x86-ops.h |  5 ++---
 arch/x86/include/asm/kvm_host.h    |  7 +++----
 arch/x86/kvm/svm/sev.c             | 21 ++++++++++++++++++---
 arch/x86/kvm/svm/svm.c             |  3 +--
 arch/x86/kvm/svm/svm.h             |  5 ++---
 arch/x86/kvm/x86.c                 |  4 ++--
 6 files changed, 28 insertions(+), 17 deletions(-)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 736129db272a..76ce691de2f8 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -146,10 +146,9 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL(get_untagged_addr)
 KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
 #endif
-KVM_X86_OP_OPTIONAL(gmem_invalidate)
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
 #endif
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index e7df537d36d5..f60fec14c61a 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1724,11 +1724,10 @@ struct kvm_x86_ops {
 
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-	int (*gmem_prepare)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-			    kvm_pfn_t nr_pages, int max_order);
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+	int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			    kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
-	void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
 	void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range);
 #endif
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index adc58138fdd0..b00623b2f26a 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -5114,8 +5114,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
 	return false;
 }
 
-int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-		     kvm_pfn_t nr_pages, int max_order)
+static int sev_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+				 kvm_pfn_t nr_pages, int max_order)
 {
 	struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
 	kvm_pfn_t pfn_aligned;
@@ -5165,8 +5165,9 @@ int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 	return 0;
 }
 
-void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
+static void sev_gmem_make_shared(kvm_pfn_t start, kvm_pfn_t nr_pages)
 {
+	kvm_pfn_t end = start + nr_pages;
 	kvm_pfn_t pfn;
 
 	if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
@@ -5228,6 +5229,20 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
 	}
 }
 
+int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int order, bool to_private)
+{
+	if (to_private) {
+		if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SHIFT)))
+			return -EIO;
+
+		return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order);
+	}
+
+	sev_gmem_make_shared(pfn, nr_pages);
+	return 0;
+}
+
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
 {
 	struct kvm_vcpu *vcpu;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 91286d46d13a..8c5018c65dc5 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -5438,8 +5438,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
 	.vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
 	.vm_move_enc_context_from = sev_vm_move_enc_context_from,
 
-	.gmem_prepare = sev_gmem_prepare,
-	.gmem_invalidate = sev_gmem_invalidate,
+	.gmem_convert = sev_gmem_convert,
 	.gmem_invalidate_range = sev_gmem_invalidate_range,
 	.gmem_max_mapping_level = sev_gmem_max_mapping_level,
 #endif
diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
index e35419456648..57f692b5d709 100644
--- a/arch/x86/kvm/svm/svm.h
+++ b/arch/x86/kvm/svm/svm.h
@@ -1009,9 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);
 int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
 extern unsigned int max_sev_asid;
 void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
-int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-		     kvm_pfn_t nr_pages, int max_order);
-void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
+int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+		     kvm_pfn_t nr_pages, int max_order, bool to_private);
 void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
 int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
 struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index d484d8b509b1..e509d50c247c 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10623,14 +10623,14 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 			  kvm_pfn_t nr_pages, int max_order)
 {
-	return kvm_x86_call(gmem_prepare)(kvm, gfn, pfn, nr_pages, max_order);
+	return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
 }
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
 void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
 {
-	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
+	WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
 }
 #endif
 
-- 
2.55.0.141.g00534a21ce-goog

[PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

[Sean Christopherson]

Rework guest_memfd's "prepare" hook into a more generic "convert" flow in
anticipation of supporting in-place conversion, at which point KVM will use
the hook for both to-private and to-shared conversions, not just to
"prepare" PRIVATE memory.

Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private()
to better reflect its role.

No functional change intended.

Signed-off-by: Sean Christopherson <seanjc@google.com>
---
 arch/x86/include/asm/kvm-x86-ops.h |  2 +-
 arch/x86/include/asm/kvm_host.h    |  2 +-
 arch/x86/kvm/Kconfig               |  2 +-
 arch/x86/kvm/x86.c                 |  8 ++++----
 include/linux/kvm_host.h           |  6 +++---
 virt/kvm/Kconfig                   |  2 +-
 virt/kvm/guest_memfd.c             | 12 ++++++------
 7 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
index 76ce691de2f8..1764bad03b17 100644
--- a/arch/x86/include/asm/kvm-x86-ops.h
+++ b/arch/x86/include/asm/kvm-x86-ops.h
@@ -146,7 +146,7 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
 KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
 KVM_X86_OP_OPTIONAL(get_untagged_addr)
 KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
-#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
 KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
 #endif
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index f60fec14c61a..08c4af487fa1 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1724,7 +1724,7 @@ struct kvm_x86_ops {
 
 	gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
 	void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
-#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
+#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
 	int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
 			    kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
index e0e7ad015839..538ed1e80332 100644
--- a/arch/x86/kvm/Kconfig
+++ b/arch/x86/kvm/Kconfig
@@ -160,7 +160,7 @@ config KVM_AMD_SEV
 	depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
 	select ARCH_HAS_CC_PLATFORM
 	select KVM_GENERIC_MEMORY_ATTRIBUTES
-	select HAVE_KVM_ARCH_GMEM_PREPARE
+	select HAVE_KVM_ARCH_GMEM_CONVERT
 	select HAVE_KVM_ARCH_GMEM_RECLAIM
 	select HAVE_KVM_ARCH_GMEM_INVALIDATE
 	select HAVE_KVM_ARCH_GMEM_POPULATE
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index e509d50c247c..11017f49b94a 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10619,11 +10619,11 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
 	return !kvm_arch_has_private_mem(kvm);
 }
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-			  kvm_pfn_t nr_pages, int max_order)
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order, bool to_private)
 {
-	return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
+	return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, to_private);
 }
 #endif
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 0a06234a8a8b..fdbd697d0337 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -2579,9 +2579,9 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
 }
 #endif /* CONFIG_KVM_GUEST_MEMFD */
 
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
-int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
-			  kvm_pfn_t nr_pages, int max_order);
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
+int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
+			  kvm_pfn_t nr_pages, int max_order, bool to_private);
 #endif
 
 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
index 617876993225..c3c0ee253fc7 100644
--- a/virt/kvm/Kconfig
+++ b/virt/kvm/Kconfig
@@ -107,7 +107,7 @@ config KVM_GUEST_MEMFD
        select XARRAY_MULTI
        bool
 
-config HAVE_KVM_ARCH_GMEM_PREPARE
+config HAVE_KVM_ARCH_GMEM_CONVERT
        bool
        depends on KVM_GUEST_MEMFD
 
diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
index 0f6745f3b919..173a7ed008d1 100644
--- a/virt/kvm/guest_memfd.c
+++ b/virt/kvm/guest_memfd.c
@@ -66,10 +66,10 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
  * On successful return the guest sees a zero page so as to avoid
  * leaking host data and the up-to-date flag is set.
  */
-static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
-				  gfn_t gfn, struct folio *folio)
+static int kvm_gmem_make_private(struct kvm *kvm, struct kvm_memory_slot *slot,
+				 gfn_t gfn, struct folio *folio)
 {
-#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
+#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
 	unsigned long nr_pages = folio_nr_pages(folio);
 	pgoff_t index;
 
@@ -90,8 +90,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
 	gfn = ALIGN_DOWN(gfn, nr_pages);
 	index = kvm_gmem_get_index(slot, gfn);
 
-	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
-				     nr_pages, folio_order(folio));
+	return kvm_arch_gmem_convert(kvm, gfn, folio_file_pfn(folio, index),
+				     nr_pages, folio_order(folio), true);
 #else
 	return 0;
 #endif
@@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
 		folio_mark_uptodate(folio);
 	}
 
-	r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
+	r = kvm_gmem_make_private(kvm, slot, gfn, folio);
 
 	folio_unlock(folio);
 
-- 
2.55.0.141.g00534a21ce-goog

Re: [PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:

New issues:
- [Critical] Refactoring `kvm_gmem_prepare_folio()` and `kvm_gmem_make_private()` to align `gfn` before computing `index` produces a mismatched and unaligned `pfn` if the memslot's `base_gfn` is not aligned to the huge folio size. This causes the architecture-specific prepare function to write beyond the bounds of the folio, leading to host physical memory corruption.

Pre-existing issues:
- [High] In SEV-SNP, if large RMP entry preparation fails for a huge folio, KVM will only prepare the base 4K page, causing an incomplete initialization and an infinite page fault loop for subpage accesses.
--

--- Patch [4]: [PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller ---
commit ef3a2643968d189a00f8b26552b9b814bf1aa548
Author: Sean Christopherson <seanjc@google.com>

KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

This commit folds __kvm_gmem_prepare_folio() into its caller, removing the
single-use tail-call helper. It also adjusts the alignment logic to align the
gfn before computing the index, and converts a WARN to WARN_ON_ONCE.

> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 28cdcf6007453..f065dedfbba8c 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c

[ ... ]

> @@ -97,11 +85,15 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
>  	 * The order will be passed when creating the guest_memfd, and
>  	 * checked when creating memslots.
>  	 */
> -	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
> +	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
> +	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
>  	index = kvm_gmem_get_index(slot, gfn);
> -	index = ALIGN_DOWN(index, folio_nr_pages(folio));
>  
> -	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
> +	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
> +				     folio_order(folio));

[Severity: Critical]
Does aligning gfn before computing index cause an unaligned index when
slot->base_gfn is not huge-page aligned?

If a userspace process maps a guest_memfd using KVM_SET_USER_MEMORY_REGION2
with a base_gfn that is not aligned to the huge page size,
kvm_gmem_get_index() could return an unaligned index.

Because folio_file_pfn() applies the index offset to the base PFN:

virt/kvm/guest_memfd.c:folio_file_pfn() {
    return folio_pfn(folio) + (index & (folio_nr_pages(folio) - 1));
}

an unaligned index causes it to return an offset PFN instead of the
folio's base PFN.

Could this lead to host physical memory corruption? The architecture
backend processes the full page count starting from the offset PFN,
which would write beyond the allocated folio boundaries and overwrite
unrelated host physical memory.

It looks like this same alignment pattern is also present in
kvm_gmem_make_private().


[Severity: High]
This isn't a bug introduced by this patch, but could this generic
function incorrectly assume a successful return means the entire
huge folio was prepared?

Looking at sev_gmem_prepare() in arch/x86/kvm/svm/sev.c, if it falls
back to PG_LEVEL_4K for a huge folio:

    } else {
        level = PG_LEVEL_4K;
        pfn_aligned = pfn;
        gfn_aligned = gfn;
    }

    rc = rmp_make_private(pfn_aligned, gfn_to_gpa(gfn_aligned), level, sev->asid, false);

It appears rmp_make_private() is called only once for the base 4K page,
without looping over the remaining pages in the huge folio.

Since kvm_arch_gmem_prepare() then returns success to this generic code,
are the remaining 511 pages left uninitialized, potentially causing an
infinite page fault loop for subsequent subpage accesses?

> +#else
> +	return 0;
> +#endif
>  }

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260714231015.3337831-1-seanjc@google.com?part=4

Re: [PATCH v5 1/7] KVM: guest_memfd: Plumb the number of pages and max order into .invalidate()

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

>
> [...snip...]
>
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
>  static void kvm_gmem_free_folio(struct folio *folio)
>  {
> -	struct page *page = folio_page(folio, 0);
> -	kvm_pfn_t pfn = page_to_pfn(page);
> -	int order = folio_order(folio);
> -
> -	kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));
> +	kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),

How about just folio_pfn(folio)? Since we're freeing the folio this will
always be 0.

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

> +				 folio_order(folio));
>  }
>  #endif
>

Re: [PATCH v5 2/7] KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and isolate it

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

> Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its
> own RECLAIM Kconfig, as the hook is called when a folio is freed, which is
> far too late and lacks sufficient information for KVM to actually
> invalidate its usage of the memory.  E.g. SNP uses the hook to convert
> memory back to SHARED so that it can be safely accessed by the host, there
> is no invalidation of guest mappings anywhere.  Isolating the hook will
> also allow pKVM on arm64 to opt-in to reclaim() without also having to
> differentiate between reclaim and conversions to shared for active VMs.
>
> Keep guest_memfd's trampoline, even though it would be trivial to wire up
> .free_folio() directly to an arch callback, to avoid bleeding guest_memfd
> internals into arch code (specifically, avoid referencing folios in arch
> code).
>
> Leave the kvm_x86_ops hook as-is for the moment, as the hook can be dropped
> entirely ("reclaim" on SNP is the same as convert-to-shared).
>
>
> [...snip...]
>

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

Re: [PATCH v5 4/7] KVM: guest_memfd: Fold __kvm_gmem_prepare_folio() into its sole caller

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

>
> [...snip...]
>
> @@ -97,11 +85,15 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
>  	 * The order will be passed when creating the guest_memfd, and
>  	 * checked when creating memslots.
>  	 */
> -	WARN_ON(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
> +	WARN_ON_ONCE(!IS_ALIGNED(slot->gmem.pgoff, folio_nr_pages(folio)));
> +	gfn = ALIGN_DOWN(gfn, folio_nr_pages(folio));
>  	index = kvm_gmem_get_index(slot, gfn);
> -	index = ALIGN_DOWN(index, folio_nr_pages(folio));
>
> -	return __kvm_gmem_prepare_folio(kvm, slot, index, folio);
> +	return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),

Could this just be folio_pfn(folio) since this function is
kvm_gmem_prepare_folio() and guest_memfd will always try to prepare the
entire folio?

> +				     folio_order(folio));
> +#else
> +	return 0;
> +#endif
>  }

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

> +int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> +		     kvm_pfn_t nr_pages, int order, bool to_private)
> +{
> +	if (to_private) {
> +		if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SHIFT)))

Omg, maxphyaddr is actually the number of bits in an address and not an
address!

Thanks for explaining on v4.

> +			return -EIO;
> +
> +		return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order);
> +	}
> +
> +	sev_gmem_make_shared(pfn, nr_pages);
> +	return 0;
> +}
> +
>
> [...snip...]
>

>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
>  void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
>  {
> -	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
> +	WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));

This setup means later pKVM would have to define a .gmem_convert but
actually only use it for reclaim, which seems kind of an odd
definition. (We can fix this later.)

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

>  }
>  #endif
>
> --
> 2.55.0.141.g00534a21ce-goog

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

[Ackerley Tng]

Sean Christopherson <seanjc@google.com> writes:

> Rework guest_memfd's "prepare" hook into a more generic "convert" flow in
> anticipation of supporting in-place conversion, at which point KVM will use
> the hook for both to-private and to-shared conversions, not just to
> "prepare" PRIVATE memory.
>
> Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private()
> to better reflect its role.
>
> No functional change intended.
>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Ackerley Tng <ackerleytng@google.com>

Thanks!

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

[Sean Christopherson]

On Tue, Jul 14, 2026, Ackerley Tng wrote:
> Sean Christopherson <seanjc@google.com> writes:
> >  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
> >  void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
> >  {
> > -	kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
> > +	WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
> 
> This setup means later pKVM would have to define a .gmem_convert but
> actually only use it for reclaim, which seems kind of an odd
> definition. (We can fix this later.)

No, pKVM (on arm64) would just implement kvm_arch_gmem_reclaim(), and then do
whatever it wants under the hood.

Re: [PATCH v5 1/7] KVM: guest_memfd: Plumb the number of pages and max order into .invalidate()

[Fuad Tabba]

On Wed, 15 Jul 2026 at 00:10, Sean Christopherson <seanjc@google.com> wrote:
>
> Plumb the number of pages to "invalidate", i.e. reclaim, along with the max
> order with which the pages can be mapped.  This will allow x86 to combine
> its vendor-specific implementations for conversion and reclaim, a.k.a.
> preparation and invalidation, into a single hook.
>
> No functional change intended.
>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  arch/x86/kvm/x86.c       | 4 ++--
>  include/linux/kvm_host.h | 2 +-
>  virt/kvm/guest_memfd.c   | 7 ++-----
>  3 files changed, 5 insertions(+), 8 deletions(-)
>
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index 1f5dc685f049..a2cbf6fae118 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -10627,9 +10627,9 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
>  #endif
>
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> -void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
> +void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
>  {
> -       kvm_x86_call(gmem_invalidate)(start, end);
> +       kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
>  }
>  void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
>  {
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index 9db6eb4023c4..996b39ebdb68 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -2614,7 +2614,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
>  #endif
>
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> -void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
> +void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);

nit: the declaration names the last parameter "order" here, but the
definition in arch/x86/kvm/x86.c calls it "max_order".

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

>  void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
>  #endif
>
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 171a9da1b685..99c070bfd446 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -530,11 +530,8 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
>  static void kvm_gmem_free_folio(struct folio *folio)
>  {
> -       struct page *page = folio_page(folio, 0);
> -       kvm_pfn_t pfn = page_to_pfn(page);
> -       int order = folio_order(folio);
> -
> -       kvm_arch_gmem_invalidate(pfn, pfn + (1ul << order));
> +       kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),
> +                                folio_order(folio));
>  }
>  #endif
>
> --
> 2.55.0.141.g00534a21ce-goog
>

Re: [PATCH v5 2/7] KVM: guest_memfd: Rename invalidate() arch hook to reclaim() and isolate it

[Fuad Tabba]

On Wed, 15 Jul 2026 at 00:10, Sean Christopherson <seanjc@google.com> wrote:

>
> Rename guest_memfd's invalidate() hook to reclaim() and isolate it via its
> own RECLAIM Kconfig, as the hook is called when a folio is freed, which is
> far too late and lacks sufficient information for KVM to actually
> invalidate its usage of the memory.  E.g. SNP uses the hook to convert
> memory back to SHARED so that it can be safely accessed by the host, there
> is no invalidation of guest mappings anywhere.  Isolating the hook will
> also allow pKVM on arm64 to opt-in to reclaim() without also having to
> differentiate between reclaim and conversions to shared for active VMs.
>
> Keep guest_memfd's trampoline, even though it would be trivial to wire up
> .free_folio() directly to an arch callback, to avoid bleeding guest_memfd
> internals into arch code (specifically, avoid referencing folios in arch
> code).
>
> Leave the kvm_x86_ops hook as-is for the moment, as the hook can be dropped
> entirely ("reclaim" on SNP is the same as convert-to-shared).
>
> Cc: Fuad Tabba <tabba@google.com>
> Cc: Ackerley Tng <ackerleytng@google.com>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Thanks Sean, this matches what pKVM needs: we'll select
HAVE_KVM_ARCH_GMEM_RECLAIM and implement kvm_arch_gmem_reclaim() only,
without the convert hook, so teardown (which for pKVM must scrub the
page before it returns to the host) stays distinct from a
private->shared conversion (which must preserve it).

Same nit as patch 1 (order/max_order).

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad


> ---
>  arch/x86/kvm/Kconfig     | 1 +
>  arch/x86/kvm/x86.c       | 7 +++++--
>  include/linux/kvm_host.h | 5 ++++-
>  virt/kvm/Kconfig         | 4 ++++
>  virt/kvm/guest_memfd.c   | 8 ++++----
>  5 files changed, 18 insertions(+), 7 deletions(-)
>
> diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
> index 801bf9e520db..e0e7ad015839 100644
> --- a/arch/x86/kvm/Kconfig
> +++ b/arch/x86/kvm/Kconfig
> @@ -161,6 +161,7 @@ config KVM_AMD_SEV
>         select ARCH_HAS_CC_PLATFORM
>         select KVM_GENERIC_MEMORY_ATTRIBUTES
>         select HAVE_KVM_ARCH_GMEM_PREPARE
> +       select HAVE_KVM_ARCH_GMEM_RECLAIM
>         select HAVE_KVM_ARCH_GMEM_INVALIDATE
>         select HAVE_KVM_ARCH_GMEM_POPULATE
>         help
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index a2cbf6fae118..08d24c6fffa4 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -10626,11 +10626,14 @@ int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, int max_ord
>  }
>  #endif
>
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> -void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
> +void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
>  {
>         kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
>  }
> +#endif
> +
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
>  void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
>  {
>         kvm_x86_call(gmem_invalidate_range)(kvm, range);
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index 996b39ebdb68..ee87ddd3294f 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -2613,8 +2613,11 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src,
>                        kvm_gmem_populate_cb post_populate, void *opaque);
>  #endif
>
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
> +void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
> +#endif
> +
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> -void kvm_arch_gmem_invalidate(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int order);
>  void kvm_arch_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
>  #endif
>
> diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
> index 794976b88c6f..617876993225 100644
> --- a/virt/kvm/Kconfig
> +++ b/virt/kvm/Kconfig
> @@ -111,6 +111,10 @@ config HAVE_KVM_ARCH_GMEM_PREPARE
>         bool
>         depends on KVM_GUEST_MEMFD
>
> +config HAVE_KVM_ARCH_GMEM_RECLAIM
> +       bool
> +       depends on KVM_GUEST_MEMFD
> +
>  config HAVE_KVM_ARCH_GMEM_INVALIDATE
>         bool
>         depends on KVM_GUEST_MEMFD
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 99c070bfd446..d5d227476cda 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -527,11 +527,11 @@ static int kvm_gmem_error_folio(struct address_space *mapping, struct folio *fol
>         return MF_DELAYED;
>  }
>
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
>  static void kvm_gmem_free_folio(struct folio *folio)
>  {
> -       kvm_arch_gmem_invalidate(folio_file_pfn(folio, 0), folio_nr_pages(folio),
> -                                folio_order(folio));
> +       kvm_arch_gmem_reclaim(folio_file_pfn(folio, 0), folio_nr_pages(folio),
> +                             folio_order(folio));
>  }
>  #endif
>
> @@ -539,7 +539,7 @@ static const struct address_space_operations kvm_gmem_aops = {
>         .dirty_folio = noop_dirty_folio,
>         .migrate_folio  = kvm_gmem_migrate_folio,
>         .error_remove_folio = kvm_gmem_error_folio,
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
>         .free_folio = kvm_gmem_free_folio,
>  #endif
>  };
> --
> 2.55.0.141.g00534a21ce-goog
>

Re: [PATCH v5 3/7] KVM: guest_memfd: Drop the redundant printk on arch gmem_prepare() failure

[Fuad Tabba]

On Wed, 15 Jul 2026 at 00:10, Sean Christopherson <seanjc@google.com> wrote:
>
> Drop guest_memfd's ratelimited printk to log "preparation" failures, as KVM
> SNP alreads logs more precise messages in all error paths, and whether or

nit: already

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

> not failure to convert the pfn to private is "unexpected", i.e. warrants
> logging, is firmly an architecture specific detail.
>
> Reviewed-by: Ackerley Tng <ackerleytng@google.com>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
>  virt/kvm/guest_memfd.c | 10 +++-------
>  1 file changed, 3 insertions(+), 7 deletions(-)
>
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index d5d227476cda..62c2c157f0c1 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -66,15 +66,11 @@ static int __kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slo
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
>         kvm_pfn_t pfn = folio_file_pfn(folio, index);
>         gfn_t gfn = slot->base_gfn + index - slot->gmem.pgoff;
> -       int rc = kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
> -       if (rc) {
> -               pr_warn_ratelimited("gmem: Failed to prepare folio for index %lx GFN %llx PFN %llx error %d.\n",
> -                                   index, gfn, pfn, rc);
> -               return rc;
> -       }
> -#endif
>
> +       return kvm_arch_gmem_prepare(kvm, gfn, pfn, folio_order(folio));
> +#else
>         return 0;
> +#endif
>  }
>
>  /*
> --
> 2.55.0.141.g00534a21ce-goog
>

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

[Fuad Tabba]

On Wed, 15 Jul 2026 at 00:10, Sean Christopherson <seanjc@google.com> wrote:
>
> Smush x86's prepare() and invalidate() hooks into a common convert() flow,
> as they are effectively two sides of the same coin for SNP: they're invoked
> when private/shared memory is about to made accessible/visible to the guest
> or host.  I.e. prepare() is really "make private", and invalidate() is
> really "make shared".
>
> Using a single hook will yield more intuitive code when in-place conversion
> comes along.
>
> For all intents and purposes, no functional change intended.
>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad



> ---
>  arch/x86/include/asm/kvm-x86-ops.h |  5 ++---
>  arch/x86/include/asm/kvm_host.h    |  7 +++----
>  arch/x86/kvm/svm/sev.c             | 21 ++++++++++++++++++---
>  arch/x86/kvm/svm/svm.c             |  3 +--
>  arch/x86/kvm/svm/svm.h             |  5 ++---
>  arch/x86/kvm/x86.c                 |  4 ++--
>  6 files changed, 28 insertions(+), 17 deletions(-)
>
> diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
> index 736129db272a..76ce691de2f8 100644
> --- a/arch/x86/include/asm/kvm-x86-ops.h
> +++ b/arch/x86/include/asm/kvm-x86-ops.h
> @@ -146,10 +146,9 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
>  KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
>  KVM_X86_OP_OPTIONAL(get_untagged_addr)
>  KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
> -KVM_X86_OP_OPTIONAL_RET0(gmem_prepare)
> +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
> +KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
>  #endif
> -KVM_X86_OP_OPTIONAL(gmem_invalidate)
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
>  KVM_X86_OP_OPTIONAL(gmem_invalidate_range)
>  #endif
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index e7df537d36d5..f60fec14c61a 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -1724,11 +1724,10 @@ struct kvm_x86_ops {
>
>         gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
>         void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
> -       int (*gmem_prepare)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> -                           kvm_pfn_t nr_pages, int max_order);
> +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
> +       int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> +                           kvm_pfn_t nr_pages, int max_order, bool to_private);
>  #endif
> -       void (*gmem_invalidate)(kvm_pfn_t start, kvm_pfn_t end);
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
>         void (*gmem_invalidate_range)(struct kvm *kvm, struct kvm_gfn_range *range);
>  #endif
> diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
> index adc58138fdd0..b00623b2f26a 100644
> --- a/arch/x86/kvm/svm/sev.c
> +++ b/arch/x86/kvm/svm/sev.c
> @@ -5114,8 +5114,8 @@ static bool is_large_rmp_possible(struct kvm *kvm, kvm_pfn_t pfn, int order)
>         return false;
>  }
>
> -int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> -                    kvm_pfn_t nr_pages, int max_order)
> +static int sev_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> +                                kvm_pfn_t nr_pages, int max_order)
>  {
>         struct kvm_sev_info *sev = to_kvm_sev_info(kvm);
>         kvm_pfn_t pfn_aligned;
> @@ -5165,8 +5165,9 @@ int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
>         return 0;
>  }
>
> -void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
> +static void sev_gmem_make_shared(kvm_pfn_t start, kvm_pfn_t nr_pages)
>  {
> +       kvm_pfn_t end = start + nr_pages;
>         kvm_pfn_t pfn;
>
>         if (!cc_platform_has(CC_ATTR_HOST_SEV_SNP))
> @@ -5228,6 +5229,20 @@ void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end)
>         }
>  }
>
> +int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> +                    kvm_pfn_t nr_pages, int order, bool to_private)
> +{
> +       if (to_private) {
> +               if (WARN_ON_ONCE(!kvm || gfn >> (kvm_host.maxphyaddr - PAGE_SHIFT)))
> +                       return -EIO;
> +
> +               return sev_gmem_make_private(kvm, gfn, pfn, nr_pages, order);
> +       }
> +
> +       sev_gmem_make_shared(pfn, nr_pages);
> +       return 0;
> +}
> +
>  void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range)
>  {
>         struct kvm_vcpu *vcpu;
> diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
> index 91286d46d13a..8c5018c65dc5 100644
> --- a/arch/x86/kvm/svm/svm.c
> +++ b/arch/x86/kvm/svm/svm.c
> @@ -5438,8 +5438,7 @@ struct kvm_x86_ops svm_x86_ops __initdata = {
>         .vm_copy_enc_context_from = sev_vm_copy_enc_context_from,
>         .vm_move_enc_context_from = sev_vm_move_enc_context_from,
>
> -       .gmem_prepare = sev_gmem_prepare,
> -       .gmem_invalidate = sev_gmem_invalidate,
> +       .gmem_convert = sev_gmem_convert,
>         .gmem_invalidate_range = sev_gmem_invalidate_range,
>         .gmem_max_mapping_level = sev_gmem_max_mapping_level,
>  #endif
> diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h
> index e35419456648..57f692b5d709 100644
> --- a/arch/x86/kvm/svm/svm.h
> +++ b/arch/x86/kvm/svm/svm.h
> @@ -1009,9 +1009,8 @@ int sev_cpu_init(struct svm_cpu_data *sd);
>  int sev_dev_get_attr(u32 group, u64 attr, u64 *val);
>  extern unsigned int max_sev_asid;
>  void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code);
> -int sev_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> -                    kvm_pfn_t nr_pages, int max_order);
> -void sev_gmem_invalidate(kvm_pfn_t start, kvm_pfn_t end);
> +int sev_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> +                    kvm_pfn_t nr_pages, int max_order, bool to_private);
>  void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *range);
>  int sev_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, bool is_private);
>  struct vmcb_save_area *sev_decrypt_vmsa(struct kvm_vcpu *vcpu);
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index d484d8b509b1..e509d50c247c 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -10623,14 +10623,14 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
>  int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
>                           kvm_pfn_t nr_pages, int max_order)
>  {
> -       return kvm_x86_call(gmem_prepare)(kvm, gfn, pfn, nr_pages, max_order);
> +       return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
>  }
>  #endif
>
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
>  void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
>  {
> -       kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
> +       WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
>  }
>  #endif
>
> --
> 2.55.0.141.g00534a21ce-goog
>

Re: [PATCH v5 6/7] KVM: x86: Combine .gmem_prepare()+.gmem_invalidate() into .gmem_convert()

[Fuad Tabba]

On Wed, 15 Jul 2026 at 01:11, Sean Christopherson <seanjc@google.com> wrote:
>
> On Tue, Jul 14, 2026, Ackerley Tng wrote:
> > Sean Christopherson <seanjc@google.com> writes:
> > >  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM
> > >  void kvm_arch_gmem_reclaim(kvm_pfn_t pfn, kvm_pfn_t nr_pages, int max_order)
> > >  {
> > > -   kvm_x86_call(gmem_invalidate)(pfn, pfn + nr_pages);
> > > +   WARN_ON_ONCE(kvm_x86_call(gmem_convert)(NULL, -1ull, pfn, nr_pages, max_order, false));
> >
> > This setup means later pKVM would have to define a .gmem_convert but
> > actually only use it for reclaim, which seems kind of an odd
> > definition. (We can fix this later.)
>
> No, pKVM (on arm64) would just implement kvm_arch_gmem_reclaim(), and then do
> whatever it wants under the hood.

Right. pKVM would just select HAVE_KVM_ARCH_GMEM_RECLAIM and implement
kvm_arch_gmem_reclaim() to scrub; the reclaim and convert hooks have
independent Kconfigs.

Thank you Ackerley for checking, and thank you all for the respin.

Cheers,
/fuad

Re: [PATCH v5 7/7] KVM: guest_memfd: Rework PREPARE config and hook into a more generic CONVERT

[Fuad Tabba]

On Wed, 15 Jul 2026 at 00:10, Sean Christopherson <seanjc@google.com> wrote:
>
> Rework guest_memfd's "prepare" hook into a more generic "convert" flow in
> anticipation of supporting in-place conversion, at which point KVM will use
> the hook for both to-private and to-shared conversions, not just to
> "prepare" PRIVATE memory.
>
> Opportunistically rename kvm_gmem_prepare_folio() to kvm_gmem_make_private()
> to better reflect its role.
>
> No functional change intended.
>
> Signed-off-by: Sean Christopherson <seanjc@google.com>

Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>

Cheers,
/fuad

> ---
>  arch/x86/include/asm/kvm-x86-ops.h |  2 +-
>  arch/x86/include/asm/kvm_host.h    |  2 +-
>  arch/x86/kvm/Kconfig               |  2 +-
>  arch/x86/kvm/x86.c                 |  8 ++++----
>  include/linux/kvm_host.h           |  6 +++---
>  virt/kvm/Kconfig                   |  2 +-
>  virt/kvm/guest_memfd.c             | 12 ++++++------
>  7 files changed, 17 insertions(+), 17 deletions(-)
>
> diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h
> index 76ce691de2f8..1764bad03b17 100644
> --- a/arch/x86/include/asm/kvm-x86-ops.h
> +++ b/arch/x86/include/asm/kvm-x86-ops.h
> @@ -146,7 +146,7 @@ KVM_X86_OP(vcpu_deliver_sipi_vector)
>  KVM_X86_OP_OPTIONAL_RET0(vcpu_get_apicv_inhibit_reasons);
>  KVM_X86_OP_OPTIONAL(get_untagged_addr)
>  KVM_X86_OP_OPTIONAL(alloc_apic_backing_page)
> -#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
> +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
>  KVM_X86_OP_OPTIONAL_RET0(gmem_convert)
>  #endif
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index f60fec14c61a..08c4af487fa1 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -1724,7 +1724,7 @@ struct kvm_x86_ops {
>
>         gva_t (*get_untagged_addr)(struct kvm_vcpu *vcpu, gva_t gva, unsigned int flags);
>         void *(*alloc_apic_backing_page)(struct kvm_vcpu *vcpu);
> -#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
> +#if defined(CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT) || defined(CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM)
>         int (*gmem_convert)(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
>                             kvm_pfn_t nr_pages, int max_order, bool to_private);
>  #endif
> diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
> index e0e7ad015839..538ed1e80332 100644
> --- a/arch/x86/kvm/Kconfig
> +++ b/arch/x86/kvm/Kconfig
> @@ -160,7 +160,7 @@ config KVM_AMD_SEV
>         depends on CRYPTO_DEV_SP_PSP && !(KVM_AMD=y && CRYPTO_DEV_CCP_DD=m)
>         select ARCH_HAS_CC_PLATFORM
>         select KVM_GENERIC_MEMORY_ATTRIBUTES
> -       select HAVE_KVM_ARCH_GMEM_PREPARE
> +       select HAVE_KVM_ARCH_GMEM_CONVERT
>         select HAVE_KVM_ARCH_GMEM_RECLAIM
>         select HAVE_KVM_ARCH_GMEM_INVALIDATE
>         select HAVE_KVM_ARCH_GMEM_POPULATE
> diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index e509d50c247c..11017f49b94a 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -10619,11 +10619,11 @@ bool kvm_arch_supports_gmem_init_shared(struct kvm *kvm)
>         return !kvm_arch_has_private_mem(kvm);
>  }
>
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
> -int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> -                         kvm_pfn_t nr_pages, int max_order)
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
> +int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> +                         kvm_pfn_t nr_pages, int max_order, bool to_private)
>  {
> -       return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, true);
> +       return kvm_x86_call(gmem_convert)(kvm, gfn, pfn, nr_pages, max_order, to_private);
>  }
>  #endif
>
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index 0a06234a8a8b..fdbd697d0337 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -2579,9 +2579,9 @@ static inline int kvm_gmem_get_pfn(struct kvm *kvm,
>  }
>  #endif /* CONFIG_KVM_GUEST_MEMFD */
>
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
> -int kvm_arch_gmem_prepare(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> -                         kvm_pfn_t nr_pages, int max_order);
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
> +int kvm_arch_gmem_convert(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn,
> +                         kvm_pfn_t nr_pages, int max_order, bool to_private);
>  #endif
>
>  #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_POPULATE
> diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig
> index 617876993225..c3c0ee253fc7 100644
> --- a/virt/kvm/Kconfig
> +++ b/virt/kvm/Kconfig
> @@ -107,7 +107,7 @@ config KVM_GUEST_MEMFD
>         select XARRAY_MULTI
>         bool
>
> -config HAVE_KVM_ARCH_GMEM_PREPARE
> +config HAVE_KVM_ARCH_GMEM_CONVERT
>         bool
>         depends on KVM_GUEST_MEMFD
>
> diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c
> index 0f6745f3b919..173a7ed008d1 100644
> --- a/virt/kvm/guest_memfd.c
> +++ b/virt/kvm/guest_memfd.c
> @@ -66,10 +66,10 @@ static pgoff_t kvm_gmem_get_index(struct kvm_memory_slot *slot, gfn_t gfn)
>   * On successful return the guest sees a zero page so as to avoid
>   * leaking host data and the up-to-date flag is set.
>   */
> -static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
> -                                 gfn_t gfn, struct folio *folio)
> +static int kvm_gmem_make_private(struct kvm *kvm, struct kvm_memory_slot *slot,
> +                                gfn_t gfn, struct folio *folio)
>  {
> -#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_PREPARE
> +#ifdef CONFIG_HAVE_KVM_ARCH_GMEM_CONVERT
>         unsigned long nr_pages = folio_nr_pages(folio);
>         pgoff_t index;
>
> @@ -90,8 +90,8 @@ static int kvm_gmem_prepare_folio(struct kvm *kvm, struct kvm_memory_slot *slot,
>         gfn = ALIGN_DOWN(gfn, nr_pages);
>         index = kvm_gmem_get_index(slot, gfn);
>
> -       return kvm_arch_gmem_prepare(kvm, gfn, folio_file_pfn(folio, index),
> -                                    nr_pages, folio_order(folio));
> +       return kvm_arch_gmem_convert(kvm, gfn, folio_file_pfn(folio, index),
> +                                    nr_pages, folio_order(folio), true);
>  #else
>         return 0;
>  #endif
> @@ -798,7 +798,7 @@ int kvm_gmem_get_pfn(struct kvm *kvm, struct kvm_memory_slot *slot,
>                 folio_mark_uptodate(folio);
>         }
>
> -       r = kvm_gmem_prepare_folio(kvm, slot, gfn, folio);
> +       r = kvm_gmem_make_private(kvm, slot, gfn, folio);
>
>         folio_unlock(folio);
>
> --
> 2.55.0.141.g00534a21ce-goog
>