[PATCH v1] KVM: arm64: Fast path transparent_hugepage_adjust() for non-PMD folios

[PATCH v1] KVM: arm64: Fast path transparent_hugepage_adjust() for non-PMD folios

[Vincent Donnefort]

transparent_hugepage_adjust() must walk the userspace stage-1 to detect
if a PMD-level mapping is installed. If it is, then a stage-2 block
mapping can be installed. Even though this stage-1 walk is most likely
cache hot (in most cases, the mapping has just been installed), walking
a page-table is still costly. This walks accounts for ~5% of the
user_mem_abort() execution time on an Apple M2 big core. This makes a
fast path interesting.

While we can't be sure by solely looking at the folio if the stage-1 is
a block mapping as userspace has multiple ways to split it, we however
can infer from its size if the stage-1 mapping is __smaller__ than
PMD_SIZE. Anonymous memory or memfd faults will always allocate a folio
big enough for the mapping. The same applies when collapsing into a
block mapping with khugepaged or MADV_COLLAPSE.

Add a fast path based on the folio size. Whenever it is too small for a
PMD_SIZE mapping, bail out early and do not walk the stage-1. This is
interesting for systems where PMD_SIZE THPs are disabled either at
build, or at runtime. Systems under heavy memory pressure will also
benefit from the fast path.

Signed-off-by: Vincent Donnefort <vdonnefort@google.com>
---
 arch/arm64/kvm/mmu.c | 42 ++++++++++++++++++++++++------------------
 1 file changed, 24 insertions(+), 18 deletions(-)

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 1db10d8d6db7..9d81d908e230 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1553,36 +1553,42 @@ static bool fault_supports_stage2_huge_mapping(const struct kvm_s2_fault_desc *s
  * Returns the size of the mapping.
  */
 static long
-transparent_hugepage_adjust(const struct kvm_s2_fault_desc *s2fd, kvm_pfn_t *pfnp, gfn_t *gfnp)
+transparent_hugepage_adjust(const struct kvm_s2_fault_desc *s2fd,
+			    const struct kvm_s2_fault_vma_info *s2vi, kvm_pfn_t *pfnp, gfn_t *gfnp)
 {
 	struct kvm *kvm = s2fd->vcpu->kvm;
-	kvm_pfn_t pfn = *pfnp;
-	gfn_t gfn = *gfnp;
+	int sz;
 
 	/*
 	 * Make sure the adjustment is done only for THP pages. Also make
 	 * sure that the HVA and IPA are sufficiently aligned and that the
 	 * block map is contained within the memslot.
 	 */
-	if (fault_supports_stage2_huge_mapping(s2fd, PMD_SIZE)) {
-		int sz = get_user_mapping_size(kvm, s2fd->hva);
+	if (!fault_supports_stage2_huge_mapping(s2fd, PMD_SIZE))
+		return PAGE_SIZE;
 
-		if (sz < 0)
-			return sz;
+	/*
+	 * If the allocated folio isn't at least PMD_SIZE, there is no way a
+	 * stage-1 PMD level was installed.
+	 */
+	if (s2vi->page && !folio_test_pmd_mappable(page_folio(s2vi->page)))
+		return PAGE_SIZE;
 
-		if (sz < PMD_SIZE)
-			return PAGE_SIZE;
+	/*
+	 * Even though the folio size is big enough, the stage-1 mapping might
+	 * have been split. Verify it.
+	 */
+	sz = get_user_mapping_size(kvm, s2fd->hva);
+	if (sz < 0)
+		return sz;
 
-		gfn &= ~(PTRS_PER_PMD - 1);
-		*gfnp = gfn;
-		pfn &= ~(PTRS_PER_PMD - 1);
-		*pfnp = pfn;
+	if (sz < PMD_SIZE)
+		return PAGE_SIZE;
 
-		return PMD_SIZE;
-	}
+	*gfnp &= ~(PTRS_PER_PMD - 1);
+	*pfnp &= ~(PTRS_PER_PMD - 1);
 
-	/* Use page mapping if we cannot use block mapping. */
-	return PAGE_SIZE;
+	return PMD_SIZE;
 }
 
 static int get_vma_page_shift(struct vm_area_struct *vma)
@@ -2186,7 +2192,7 @@ static int kvm_s2_fault_map(const struct kvm_s2_fault_desc *s2fd,
 		if (perm_fault_granule > PAGE_SIZE) {
 			mapping_size = perm_fault_granule;
 		} else {
-			mapping_size = transparent_hugepage_adjust(s2fd, &pfn, &gfn);
+			mapping_size = transparent_hugepage_adjust(s2fd, s2vi, &pfn, &gfn);
 			if (mapping_size < 0) {
 				ret = mapping_size;
 				goto out_unlock;

base-commit: 6628e7d4c6b25c8d18f1345e3a20d21829aae7e4
-- 
2.56.0.385.gac671c819a-goog

Re: [PATCH v1] KVM: arm64: Fast path transparent_hugepage_adjust() for non-PMD folios

[Marc Zyngier]

On Mon, 02 Nov 2026 21:19:00 +0000,
Vincent Donnefort <vdonnefort@google.com> wrote:
> 
> transparent_hugepage_adjust() must walk the userspace stage-1 to detect
> if a PMD-level mapping is installed. If it is, then a stage-2 block
> mapping can be installed. Even though this stage-1 walk is most likely
> cache hot (in most cases, the mapping has just been installed), walking
> a page-table is still costly. This walks accounts for ~5% of the
> user_mem_abort() execution time on an Apple M2 big core. This makes a
> fast path interesting.

This raises a few questions:

- Isn't this a "fast fail" rather than a fast path? You don't make the
  S1 walk faster, but you instead avoid cases where it would have
  returned an level-3 size.

  If anything, you make transparent_hugepage_adjust() slower for the
  case that succeeds. I don't think this invalidates the approach, but
  it feels that the commit message is just wrong.

- How do you measure this? 5% feels like a huge amount for something
  that should be 3 loads. Isn't this more an indication that the page
  table walker is overly complicated because it was designed for
  something else? In any case, It'd be good to verify your findings,
  so please provide a minimal reproducer.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v1] KVM: arm64: Fast path transparent_hugepage_adjust() for non-PMD folios

[Vincent Donnefort]

On Tue, Nov 03, 2026 at 08:22:47AM +0000, Marc Zyngier wrote:
> On Mon, 02 Nov 2026 21:19:00 +0000,
> Vincent Donnefort <vdonnefort@google.com> wrote:
> > 
> > transparent_hugepage_adjust() must walk the userspace stage-1 to detect
> > if a PMD-level mapping is installed. If it is, then a stage-2 block
> > mapping can be installed. Even though this stage-1 walk is most likely
> > cache hot (in most cases, the mapping has just been installed), walking
> > a page-table is still costly. This walks accounts for ~5% of the
> > user_mem_abort() execution time on an Apple M2 big core. This makes a
> > fast path interesting.
> 
> This raises a few questions:
> 
> - Isn't this a "fast fail" rather than a fast path? You don't make the
>   S1 walk faster, but you instead avoid cases where it would have
>   returned an level-3 size.
> 
>   If anything, you make transparent_hugepage_adjust() slower for the
>   case that succeeds. I don't think this invalidates the approach, but
>   it feels that the commit message is just wrong.

Yes, "fast fail" would be a far better description.

> 
> - How do you measure this? 5% feels like a huge amount for something
>   that should be 3 loads. Isn't this more an indication that the page
>   table walker is overly complicated because it was designed for
>   something else? In any case, It'd be good to verify your findings,
>   so please provide a minimal reproducer.

I have booted with qemu a L1 guest. Pinned to the biggest CPUs of the Mac M2 and
set the performance governor.

Then in the L1 guest, I disabled completely THPs and enabled the tracing
function_graph to measure user_mem_abort() and get_user_mapping_size(), before
booting a L2 guest with a single vCPU.

That gave me something like ...

 0)               |  user_mem_abort() {
 0)   0.208 us    |    get_user_mapping_size();
 0)   5.958 us    |  }

I have skipped the outliers (the function_graph prefix them with '+', '$' ...)
which I believe are most likely related to interrupts.

Then for each pair, I got the percentage get_user_mapping_size() represents in
the user_mem_abort() execution time.

Now looking again at the results I have, I realised I gave the mean (5%) but
perhaps the median which is 2.7% is a better representation of the overall
improvement and is slightly less impressive.

-- 
Vincent

> 
> Thanks,
> 
> 	M.
> 
> -- 
> Without deviation from the norm, progress is not possible.