[RFC PATCH 0/2] KVM: arm64: Support BBM level 3

[RFC PATCH 0/2] KVM: arm64: Support BBM level 3

[Mostafa Saleh]

This patch series adds support for BBM level 3 to KVM pgtable, it
depends on [1]

Motivation
==========
I have been looking into this for the context of:
- Page table sharing between the host CPU stage-2 and the SMMUv3 for
  protected KVM.
- Use the pagtable code to populate SMMUv3 stage-2 shadowed
  page table [2]

However, BBM level 3 is still useful for CPU only operations as it
avoids intermediately breaking translation.

Design
======
Some of the conditions that BBM level 3 will be useful in (RWHZWS):
1) A change to a PTE memory type, shareability, cacheability or OA
2) Changing from block to table
3) Changing from table to block

At the moment in the hyp page table code:
- For #1) stage2_map_walker_try_leaf(): Replaces a leaf with another
  one which does not match the same OA/perms/attrs.

- For #2) Switch block to table is used from:
  - kvm_pgtable_stage2_split(): Explicitly splitting a block (used
    for dirty logging), where a block is replaced by a table with
    the same attributes.
  - stage2_map_walk_leaf(): Updating a mapping that is partially
    part of an existing block.

- #3) Does not exist in the code at the moment as coalescing is not
  supported.

The first patch is a preparation to be able to clean up the old pte
for BBML3, the second patch adds the main logic.

Initially, I encapsulated the full logic of BBM in one function,
which was not readable, due to different ordering and dealing with
CMO, TLBI.

Instead, I kept the logic into 2 functions, where BBML3 is added in
the make step.

One interesting case, as BBML3 will update the PTE atomically, it
can only know it raced with another core at the point of the cmpxchg
failing, unlike the SW implementation which locks the PTE first.
And as we must issue CMOs to the new mapped page before the update,
that means with BBML3 racing cores will issue redundant CMOs, to
improve this:
- We only use BBML3 if the old PTE was live
- To reduce the window of the race an early check is added before
  the CMO to exit early, but that does not eliminate the race.

Testing
=======
This was tested:
- C1-Pro cores, unfortunately the version I have does not run
  upstream, I backported the patches to Android kernel (6.18).

- mainline(7.2-rc3) kernel on a Qualcomm X1 with a hacked cpufeature
  as it does not support BBM, I did not see conflict aborts or TLB
  corruption.

I tested with VHE and protected (hvhe) modes, running VMs
(and protected), and running some selftests, that might exercise and
stress this path tools/testing/selftests/kvm:
- demand_paging_test
- memslot_perf_test
- memslot_modification_stress_test
- dirty_log_test

Future work
===========
Some other changes that would be useful for the SMMUv3:
- Eagerly install table on block split, we can now replace a block
  with a fully populated table atomically when we unmap a partial
  part of the block.

There is more to support page table sharing (such as dealing with
TLB invalidation, coherency…), I submitted a talk to LPC to discuss
this further.

[1] https://lore.kernel.org/linux-arm-kernel/20260715053408.1950475-1-linu.cherian@arm.com/
[2] https://lore.kernel.org/linux-iommu/20260715115906.2664882-1-smostafa@google.com/

Mostafa Saleh (2):
  KVM: arm64: Add stage2_clean_old_pte()
  KVM: arm64: Support BBM level 3

 arch/arm64/kvm/hyp/pgtable.c | 118 ++++++++++++++++++++++++-----------
 1 file changed, 82 insertions(+), 36 deletions(-)

-- 
2.55.0.229.g6434b31f56-goog

[RFC PATCH 1/2] KVM: arm64: Add stage2_clean_old_pte()

[Mostafa Saleh]

At the moment, the pgtable code rely on BBM in SW which looks like:
Break: stage2_try_break_pte()
	1) Break PTE and lock it
	2) TLBI
	3) Put the ref on the old PTE

Make: stage2_make_pte()
	1) Get a ref on the new PTE
	2) Install the live PTE

With BBML3, the sequence will look as
	1) Get ref on the new PTE
	2) Install new PTE
	3) TLBI
	4) Put the ref on the old PTE

Which requires moving step #2 #3 from the break function to the make
function, although it is possible to do that for SW BBM also, that
means the stage2_try_break_pte() did not fully break the PTE as it
is referenced in TLBs, although that works it seems fragile.

Instead, move this logic to a new function stage2_clean_old_pte()
that can be called from BBML3.

Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
 arch/arm64/kvm/hyp/pgtable.c | 67 ++++++++++++++++++++----------------
 1 file changed, 37 insertions(+), 30 deletions(-)

diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 91a7dfad6686..127b7f9541b1 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -810,39 +810,10 @@ static bool stage2_try_set_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_
 	return cmpxchg(ctx->ptep, ctx->old, new) == ctx->old;
 }
 
-/**
- * stage2_try_break_pte() - Invalidates a pte according to the
- *			    'break-before-make' requirements of the
- *			    architecture.
- *
- * @ctx: context of the visited pte.
- * @mmu: stage-2 mmu
- *
- * Returns: true if the pte was successfully broken.
- *
- * If the removed pte was valid, performs the necessary serialization and TLB
- * invalidation for the old value. For counted ptes, drops the reference count
- * on the containing table page.
- */
-static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
+static void stage2_clean_old_pte(const struct kvm_pgtable_visit_ctx *ctx,
 				 struct kvm_s2_mmu *mmu)
 {
 	struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
-	kvm_pte_t locked_pte;
-
-	if (stage2_pte_is_locked(ctx->old)) {
-		/*
-		 * Should never occur if this walker has exclusive access to the
-		 * page tables.
-		 */
-		WARN_ON(!kvm_pgtable_walk_shared(ctx));
-		return false;
-	}
-
-	locked_pte = FIELD_PREP(KVM_INVALID_PTE_TYPE_MASK,
-				KVM_INVALID_PTE_TYPE_LOCKED);
-	if (!stage2_try_set_pte(ctx, locked_pte))
-		return false;
 
 	if (!kvm_pgtable_walk_skip_bbm_tlbi(ctx)) {
 		/*
@@ -862,6 +833,42 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
 
 	if (stage2_pte_is_counted(ctx->old))
 		mm_ops->put_page(ctx->ptep);
+}
+
+/**
+ * stage2_try_break_pte() - Invalidates a pte according to the
+ *			    'break-before-make' requirements of the
+ *			    architecture.
+ *
+ * @ctx: context of the visited pte.
+ * @mmu: stage-2 mmu
+ *
+ * Returns: true if the pte was successfully broken.
+ *
+ * If the removed pte was valid, performs the necessary serialization and TLB
+ * invalidation for the old value. For counted ptes, drops the reference count
+ * on the containing table page.
+ */
+static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
+				 struct kvm_s2_mmu *mmu)
+{
+	kvm_pte_t locked_pte;
+
+	if (stage2_pte_is_locked(ctx->old)) {
+		/*
+		 * Should never occur if this walker has exclusive access to the
+		 * page tables.
+		 */
+		WARN_ON(!kvm_pgtable_walk_shared(ctx));
+		return false;
+	}
+
+	locked_pte = FIELD_PREP(KVM_INVALID_PTE_TYPE_MASK,
+				KVM_INVALID_PTE_TYPE_LOCKED);
+	if (!stage2_try_set_pte(ctx, locked_pte))
+		return false;
+
+	stage2_clean_old_pte(ctx, mmu);
 
 	return true;
 }
-- 
2.55.0.229.g6434b31f56-goog

[RFC PATCH 2/2] KVM: arm64: Support BBM level 3

[Mostafa Saleh]

If the system supports hardware Break-Before-Make (BBM) level 3, use it
to replace stage-2 PTEs directly instead of falling back to the software
break-before-make sequence.

1) Get a reference count on the containing table for the new PTE.
2) Atomically update the PTE with the new valid descriptor.
3) Invalidate the TLB for the old PTE.
4) Drop the reference count holding the old PTE.

One interesting case, as BBML3 will update the PTE atomically, it
can only know it raced with another core at the point of the cmpxchg
failing, unlike the SW implementation which locks the PTE first.
And as we must issue CMOs to the new mapped page before the update,
that means with BBML3 racing cores will issue redundant CMOs,
to improve this:
- We only use BBML3 if the old PTE was live.
- To reduce the window of the race, an early check is added before
  the CMO to exit early, but that does not eliminate the race.

Signed-off-by: Mostafa Saleh <smostafa@google.com>
---
 arch/arm64/kvm/hyp/pgtable.c | 53 +++++++++++++++++++++++++++++++-----
 1 file changed, 46 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 127b7f9541b1..69d52308236f 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -838,7 +838,8 @@ static void stage2_clean_old_pte(const struct kvm_pgtable_visit_ctx *ctx,
 /**
  * stage2_try_break_pte() - Invalidates a pte according to the
  *			    'break-before-make' requirements of the
- *			    architecture.
+ *			    architecture, if BMML3 is supported it
+ *			    will be used, otherwise fallback to SW.
  *
  * @ctx: context of the visited pte.
  * @mmu: stage-2 mmu
@@ -854,6 +855,18 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
 {
 	kvm_pte_t locked_pte;
 
+	if (system_supports_bbml3() && kvm_pte_valid(ctx->old)) {
+		kvm_pte_t curr_pte = READ_ONCE(*ctx->ptep);
+
+		/*
+		 * All handled in stage2_make_pte(). However exit early if we already
+		 * lost the race to avoid extra CMOs.
+		 */
+		 if (curr_pte != ctx->old)
+			return false;
+		 return true;
+	}
+
 	if (stage2_pte_is_locked(ctx->old)) {
 		/*
 		 * Should never occur if this walker has exclusive access to the
@@ -873,16 +886,35 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
 	return true;
 }
 
-static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t new)
+/* Must be paired with stage2_try_break_pte() */
+static bool stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu,
+			    kvm_pte_t new)
 {
 	struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
 
-	WARN_ON(!stage2_pte_is_locked(*ctx->ptep));
-
 	if (stage2_pte_is_counted(new))
 		mm_ops->get_page(ctx->ptep);
 
+	if (system_supports_bbml3() && kvm_pte_valid(ctx->old)) {
+		/*
+		 * Barrier is required because stage2_try_set_pte() uses
+		 * WRITE_ONCE for non-shared walks, lacking release semantics
+		 * used in the software BBM case.
+		 */
+		smp_wmb();
+		if (!stage2_try_set_pte(ctx, new)) {
+			if (stage2_pte_is_counted(new))
+				mm_ops->put_page(ctx->ptep);
+			return false;
+		}
+
+		stage2_clean_old_pte(ctx, mmu);
+		return true;
+	}
+
+	WARN_ON(!stage2_pte_is_locked(*ctx->ptep));
 	smp_store_release(ctx->ptep, new);
+	return true;
 }
 
 static bool stage2_unmap_defer_tlb_flush(struct kvm_pgtable *pgt)
@@ -1014,7 +1046,8 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx,
 	    stage2_pte_executable(new))
 		mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule);
 
-	stage2_make_pte(ctx, new);
+	if (!stage2_make_pte(ctx, data->mmu, new))
+		return -EAGAIN;
 
 	return 0;
 }
@@ -1069,7 +1102,10 @@ static int stage2_map_walk_leaf(const struct kvm_pgtable_visit_ctx *ctx,
 	 * will be mapped lazily.
 	 */
 	new = kvm_init_table_pte(childp, mm_ops);
-	stage2_make_pte(ctx, new);
+	if (!stage2_make_pte(ctx, data->mmu, new)) {
+		mm_ops->put_page(childp);
+		return -EAGAIN;
+	}
 
 	return 0;
 }
@@ -1557,7 +1593,10 @@ static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	 * writes the PTE using smp_store_release().
 	 */
 	new = kvm_init_table_pte(childp, mm_ops);
-	stage2_make_pte(ctx, new);
+	if (!stage2_make_pte(ctx, mmu, new)) {
+		kvm_pgtable_stage2_free_unlinked(mm_ops, childp, level);
+		return -EAGAIN;
+	}
 	return 0;
 }
 
-- 
2.55.0.229.g6434b31f56-goog

Re: [RFC PATCH 2/2] KVM: arm64: Support BBM level 3

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Critical] Replacing a block mapping with an empty table mapping using BBML3 violates architectural requirements and causes TLB conflict aborts.
--

--- Patch [2]: [RFC PATCH 2/2] KVM: arm64: Support BBM level 3 ---
commit ea79cffc563af3beac80c1a139cbab5674ecb7f0
Author: Mostafa Saleh <smostafa@google.com>

KVM: arm64: Support BBM level 3

This patch introduces support for hardware Break-Before-Make (BBM) level 3
to replace stage-2 PTEs directly instead of using the software sequence.
It atomically updates the PTE and relies on hardware features to handle
the transition.

> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c

[ ... ]

> @@ -1069,7 +1102,10 @@ static int stage2_map_walk_leaf(const struct kvm_pgtable_visit_ctx *ctx,
>  	 * will be mapped lazily.
>  	 */
>  	new = kvm_init_table_pte(childp, mm_ops);
> -	stage2_make_pte(ctx, new);
> +	if (!stage2_make_pte(ctx, data->mmu, new)) {
> +		mm_ops->put_page(childp);
> +		return -EAGAIN;
> +	}
>  
>  	return 0;
>  }

[Severity: Critical]
Does using BBML3 to replace a block mapping with an empty table mapping risk
a TLB conflict abort?

In stage2_map_walk_leaf(), the code allocates a zeroed page for the new
table and installs it. With BBML3, stage2_make_pte() atomically overwrites
the valid block PTE with the new empty table PTE, bypassing the software
break step.

Because the new table is unpopulated, if another CPU takes a stage-2 fault
on this range before the TLBI completes, it can populate a 4KB page in the
new table. That CPU will then cache the new 4KB mapping while still holding
the old 2MB block mapping in its TLB. The overlapping entries will trigger
a fatal TLB Conflict Abort, panicking the host.

Would it be necessary to either fall back to the software break-before-make
sequence when splitting a block, or ensure the new table is populated with
equivalent translations before the atomic swap?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20260717130901.2239134-1-smostafa@google.com?part=2

Re: [RFC PATCH 2/2] KVM: arm64: Support BBM level 3

[Mostafa Saleh]

On Fri, Jul 17, 2026 at 01:28:57PM +0000, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
> - [Critical] Replacing a block mapping with an empty table mapping using BBML3 violates architectural requirements and causes TLB conflict aborts.
> --
> 
> --- Patch [2]: [RFC PATCH 2/2] KVM: arm64: Support BBM level 3 ---
> commit ea79cffc563af3beac80c1a139cbab5674ecb7f0
> Author: Mostafa Saleh <smostafa@google.com>
> 
> KVM: arm64: Support BBM level 3
> 
> This patch introduces support for hardware Break-Before-Make (BBM) level 3
> to replace stage-2 PTEs directly instead of using the software sequence.
> It atomically updates the PTE and relies on hardware features to handle
> the transition.
> 
> > --- a/arch/arm64/kvm/hyp/pgtable.c
> > +++ b/arch/arm64/kvm/hyp/pgtable.c
> 
> [ ... ]
> 
> > @@ -1069,7 +1102,10 @@ static int stage2_map_walk_leaf(const struct kvm_pgtable_visit_ctx *ctx,
> >  	 * will be mapped lazily.
> >  	 */
> >  	new = kvm_init_table_pte(childp, mm_ops);
> > -	stage2_make_pte(ctx, new);
> > +	if (!stage2_make_pte(ctx, data->mmu, new)) {
> > +		mm_ops->put_page(childp);
> > +		return -EAGAIN;
> > +	}
> >  
> >  	return 0;
> >  }
> 
> [Severity: Critical]
> Does using BBML3 to replace a block mapping with an empty table mapping risk
> a TLB conflict abort?
> 
> In stage2_map_walk_leaf(), the code allocates a zeroed page for the new
> table and installs it. With BBML3, stage2_make_pte() atomically overwrites
> the valid block PTE with the new empty table PTE, bypassing the software
> break step.
> 
> Because the new table is unpopulated, if another CPU takes a stage-2 fault
> on this range before the TLBI completes, it can populate a 4KB page in the
> new table. That CPU will then cache the new 4KB mapping while still holding
> the old 2MB block mapping in its TLB. The overlapping entries will trigger
> a fatal TLB Conflict Abort, panicking the host.
> 
> Would it be necessary to either fall back to the software break-before-make
> sequence when splitting a block, or ensure the new table is populated with
> equivalent translations before the atomic swap?

The point of BBML3 is not to get conflict aborts, so I do not believe
that can happen.

Thanks,
Mostafa


> 
> -- 
> Sashiko AI review · https://sashiko.dev/#/patchset/20260717130901.2239134-1-smostafa@google.com?part=2

Re: [RFC PATCH 2/2] KVM: arm64: Support BBM level 3

[Oliver Upton]

Hi Mostafa,

On Fri, Jul 17, 2026 at 01:09:00PM +0000, Mostafa Saleh wrote:
> If the system supports hardware Break-Before-Make (BBM) level 3, use it
> to replace stage-2 PTEs directly instead of falling back to the software
> break-before-make sequence.
> 
> 1) Get a reference count on the containing table for the new PTE.
> 2) Atomically update the PTE with the new valid descriptor.
> 3) Invalidate the TLB for the old PTE.
> 4) Drop the reference count holding the old PTE.
> 
> One interesting case, as BBML3 will update the PTE atomically, it
> can only know it raced with another core at the point of the cmpxchg
> failing, unlike the SW implementation which locks the PTE first.
> And as we must issue CMOs to the new mapped page before the update,
> that means with BBML3 racing cores will issue redundant CMOs,

I'd rather we just predicate BBML3-style transformations on an
implementation having FEAT_S2FWB and DIC. You can definitely come along
later and enable it when using a stage-2 in an SMMU makes this
mandatory, possibly at the expense of some extra CMOs.

There's also an extremely subtle detail that BBML3 enablement relies on,
which is that KVM will never change the OA of active translation. IOW,
if the host is moving the PFN we expect an invalidation before
re-mapping it.

I have no issue with relying on that behavior but we should make that
assumption abundantly clear.

One of the things on my wish list for a while has been rebuilding
hugepages after dirty logging is disabled on a memslot. That seems like
like a very good optimization to do when BBML3 is present.

> to improve this:
> - We only use BBML3 if the old PTE was live.
> - To reduce the window of the race, an early check is added before
>   the CMO to exit early, but that does not eliminate the race.
> 
> Signed-off-by: Mostafa Saleh <smostafa@google.com>
> ---
>  arch/arm64/kvm/hyp/pgtable.c | 53 +++++++++++++++++++++++++++++++-----
>  1 file changed, 46 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
> index 127b7f9541b1..69d52308236f 100644
> --- a/arch/arm64/kvm/hyp/pgtable.c
> +++ b/arch/arm64/kvm/hyp/pgtable.c
> @@ -838,7 +838,8 @@ static void stage2_clean_old_pte(const struct kvm_pgtable_visit_ctx *ctx,
>  /**
>   * stage2_try_break_pte() - Invalidates a pte according to the
>   *			    'break-before-make' requirements of the
> - *			    architecture.
> + *			    architecture, if BMML3 is supported it
> + *			    will be used, otherwise fallback to SW.
>   *
>   * @ctx: context of the visited pte.
>   * @mmu: stage-2 mmu
> @@ -854,6 +855,18 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx,
>  {
>  	kvm_pte_t locked_pte;
>  
> +	if (system_supports_bbml3() && kvm_pte_valid(ctx->old)) {
> +		kvm_pte_t curr_pte = READ_ONCE(*ctx->ptep);
> +
> +		/*
> +		 * All handled in stage2_make_pte(). However exit early if we already
> +		 * lost the race to avoid extra CMOs.
> +		 */
> +		 if (curr_pte != ctx->old)
> +			return false;

Does this race detection actually move the needle? We haven't gotten
very far from the read in __kvm_pgtable_visit().

Thanks,
Oliver