[RFC PATCH 0/2] Optimize S2 page splitting

[RFC PATCH 0/2] Optimize S2 page splitting

[Leonardo Bras]

While playing with dirty-bit tracking, I decided to take a look on how page
splitting works. Found out all entries are walked, even though we can infer,
for instance that:
- If a level-3 entry is walked, it means the parent level-2 entry is split
- If a split just succeeded in an table entry, it means all children nodes
  are already split

So I tried to optimize it in a way that it does not break other users.

My main idea is to introduce positive return values that hint to the
pagetable walking mechanism that either siblings or children can be 
skipped. That should be contained to the visitor function, that returns
zero if no error was detected.

Numbers on above optimization are promising:
A 1GB VM, running on the model, splitting all at the beginning 
(no manual protect):
- Memory was already split (4k pages): 	-97.33% runtime (-172ms) - 20 runs
- THP backed memory: 			-19.82% runtime (-153ms) - 10 runs
- 1x1GB hugetlb memory:			-20.65% runtime (-150ms) - 10 runs

This is measured with this snippet[1].
I ran at least 10 times on different 1GB VMs, to make sure the numbers are 
consistent.

Ideas I considered:
- Using a negative return value, using kvm_pgtable_walk_continue to 
  filter it as a non-error, but decided that is kind of counter-intuitive
- Using the introduced return values to hint the split walker to not 
  splitting level-2 blocks (or level-1), if by adding a new parameter in
  kvm_pgtable_stage2_split() and carrying it over to the walker using 
  ctx->arg. (Splitting only up to given hugepage size)
- Looking at other walkers, and trying to think on scenarios to
  optimize them using the new return values.

Do you think it is worth doing this?
Please provide feedback!

Thanks!
Leo


[1]:
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index d089c107d9b7..6424e833b7be 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1272,22 +1273,26 @@ static void kvm_mmu_split_memory_region(struct kvm *kvm, int slot)
        phys_addr_t start, end;
 
        lockdep_assert_held(&kvm->slots_lock);
 
        slots = kvm_memslots(kvm);
        memslot = id_to_memslot(slots, slot);
 
        start = memslot->base_gfn << PAGE_SHIFT;
        end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT;
 
+
        write_lock(&kvm->mmu_lock);
+       u64 sw = ktime_get_real_ns();
        kvm_mmu_split_huge_pages(kvm, start, end);
+       sw = ktime_get_real_ns() - sw;
+       printk("split from %llx to %llx took %llu ns\n", start, end, sw);
        write_unlock(&kvm->mmu_lock);
 }
 

Leonardo Bras (2):
  KVM: arm64: Introduce S2 walker SKIP return options
  KVM: arm64: Improve splitting performance by using SKIP return values

 arch/arm64/kvm/hyp/pgtable.c | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)


base-commit: 5d6919055dec134de3c40167a490f33c74c12581
-- 
2.54.0

[RFC PATCH 1/2] KVM: arm64: Introduce S2 walker SKIP return options

[Leonardo Bras]

Introduce S2 walker return values:
- SKIP_CHILDREN: skip walking the children of the current node
- SKIP_SIBLINGS: skip waling the siblings of the current node

Also, modify __kvm_pgtable_visit() to fulfil the hing on above return
values. Current walkers should not be impacted

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/kvm/hyp/pgtable.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 0c1defa5fb0f..4e43339522bb 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -12,20 +12,26 @@
 #include <asm/stage2_pgtable.h>
 
 struct kvm_pgtable_walk_data {
 	struct kvm_pgtable_walker	*walker;
 
 	const u64			start;
 	u64				addr;
 	const u64			end;
 };
 
+/* Positive walker return values point levels to skip */
+enum walker_return{
+	SKIP_CHILDREN = 1,
+	SKIP_SIBLINGS
+};
+
 static bool kvm_pgtable_walk_skip_bbm_tlbi(const struct kvm_pgtable_visit_ctx *ctx)
 {
 	return unlikely(ctx->flags & KVM_PGTABLE_WALK_SKIP_BBM_TLBI);
 }
 
 static bool kvm_pgtable_walk_skip_cmo(const struct kvm_pgtable_visit_ctx *ctx)
 {
 	return unlikely(ctx->flags & KVM_PGTABLE_WALK_SKIP_CMO);
 }
 
@@ -134,21 +140,21 @@ static bool kvm_pgtable_walk_continue(const struct kvm_pgtable_walker *walker,
 	 * update a PTE. In the context of a fault handler this is interpreted
 	 * as a signal to retry guest execution.
 	 *
 	 * Ignore the return code altogether for walkers outside a fault handler
 	 * (e.g. write protecting a range of memory) and chug along with the
 	 * page table walk.
 	 */
 	if (r == -EAGAIN)
 		return walker->flags & KVM_PGTABLE_WALK_IGNORE_EAGAIN;
 
-	return !r;
+	return r >= 0;
 }
 
 static int __kvm_pgtable_walk(struct kvm_pgtable_walk_data *data,
 			      struct kvm_pgtable_mm_ops *mm_ops, kvm_pteref_t pgtable, s8 level);
 
 static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data,
 				      struct kvm_pgtable_mm_ops *mm_ops,
 				      kvm_pteref_t pteref, s8 level)
 {
 	enum kvm_pgtable_walk_flags flags = data->walker->flags;
@@ -185,23 +191,29 @@ static inline int __kvm_pgtable_visit(struct kvm_pgtable_walk_data *data,
 	 * into a newly installed or replaced table.
 	 */
 	if (reload) {
 		ctx.old = READ_ONCE(*ptep);
 		table = kvm_pte_table(ctx.old, level);
 	}
 
 	if (!kvm_pgtable_walk_continue(data->walker, ret))
 		goto out;
 
-	if (!table) {
-		data->addr = ALIGN_DOWN(data->addr, kvm_granule_size(level));
-		data->addr += kvm_granule_size(level);
+	if (!table || ret >= SKIP_CHILDREN) {
+		u64 size;
+
+		if (ret == SKIP_SIBLINGS)	/* Skip siblings */
+			size = kvm_granule_size(level - 1);
+		else				/* Skip children */
+			size = kvm_granule_size(level);
+
+		data->addr = ALIGN_DOWN(data->addr, size) + size;
 		goto out;
 	}
 
 	childp = (kvm_pteref_t)kvm_pte_follow(ctx.old, mm_ops);
 	ret = __kvm_pgtable_walk(data, mm_ops, childp, level + 1);
 	if (!kvm_pgtable_walk_continue(data->walker, ret))
 		goto out;
 
 	if (ctx.flags & KVM_PGTABLE_WALK_TABLE_POST)
 		ret = kvm_pgtable_visitor_cb(data, &ctx, KVM_PGTABLE_WALK_TABLE_POST);
-- 
2.54.0

Re: [RFC PATCH 1/2] KVM: arm64: Introduce S2 walker SKIP return options

[Oliver Upton]

Hi Leo,

Thanks for having a look at this.

On Fri, May 15, 2026 at 08:59:02PM +0100, Leonardo Bras wrote:
> Introduce S2 walker return values:
> - SKIP_CHILDREN: skip walking the children of the current node
> - SKIP_SIBLINGS: skip waling the siblings of the current node
> 
> Also, modify __kvm_pgtable_visit() to fulfil the hing on above return
> values. Current walkers should not be impacted

I'd rather see something based around new walk flags than introducing an
entirely new mechanic around return values.

e.g. you could split the LEAF flag into separate flags for blocks v.
pages:

	KVM_PGTABLE_WALK_PAGE,
	KVM_PGTABLE_WALK_BLOCK,
	KVM_PGTABLE_WALK_LEAF	= KVM_PGTABLE_WALK_PAGE |
				  KVM_PGTABLE_WALK_BLOCK,

and then let __kvm_pgtable_visit() decide how to steer the walk. You may
need some special handling to get the address arithmetic right when
skipping over a table of page descriptors.

Thanks,
Oliver

Re: [RFC PATCH 1/2] KVM: arm64: Introduce S2 walker SKIP return options

[Will Deacon]

On Mon, May 18, 2026 at 12:22:47AM -0700, Oliver Upton wrote:
> On Fri, May 15, 2026 at 08:59:02PM +0100, Leonardo Bras wrote:
> > Introduce S2 walker return values:
> > - SKIP_CHILDREN: skip walking the children of the current node
> > - SKIP_SIBLINGS: skip waling the siblings of the current node
> > 
> > Also, modify __kvm_pgtable_visit() to fulfil the hing on above return
> > values. Current walkers should not be impacted
> 
> I'd rather see something based around new walk flags than introducing an
> entirely new mechanic around return values.
> 
> e.g. you could split the LEAF flag into separate flags for blocks v.
> pages:
> 
> 	KVM_PGTABLE_WALK_PAGE,
> 	KVM_PGTABLE_WALK_BLOCK,
> 	KVM_PGTABLE_WALK_LEAF	= KVM_PGTABLE_WALK_PAGE |
> 				  KVM_PGTABLE_WALK_BLOCK,
> 
> and then let __kvm_pgtable_visit() decide how to steer the walk. You may
> need some special handling to get the address arithmetic right when
> skipping over a table of page descriptors.

I was wondering along similar lines, but maybe it would be useful just
to pass a maximum level to the walker logic? That feels like the most
general case without complicating the existing logic.

Will

Re: [RFC PATCH 1/2] KVM: arm64: Introduce S2 walker SKIP return options

[Leonardo Bras]

Hello Oliver, Will,
Thanks for reviewing!

On Mon, May 18, 2026 at 09:52:16AM +0100, Will Deacon wrote:
> On Mon, May 18, 2026 at 12:22:47AM -0700, Oliver Upton wrote:
> > On Fri, May 15, 2026 at 08:59:02PM +0100, Leonardo Bras wrote:
> > > Introduce S2 walker return values:
> > > - SKIP_CHILDREN: skip walking the children of the current node
> > > - SKIP_SIBLINGS: skip waling the siblings of the current node
> > > 
> > > Also, modify __kvm_pgtable_visit() to fulfil the hing on above return
> > > values. Current walkers should not be impacted
> > 
> > I'd rather see something based around new walk flags than introducing an
> > entirely new mechanic around return values.
> > 
> > e.g. you could split the LEAF flag into separate flags for blocks v.
> > pages:
> > 
> > 	KVM_PGTABLE_WALK_PAGE,
> > 	KVM_PGTABLE_WALK_BLOCK,
> > 	KVM_PGTABLE_WALK_LEAF	= KVM_PGTABLE_WALK_PAGE |
> > 				  KVM_PGTABLE_WALK_BLOCK,
> > 
> > and then let __kvm_pgtable_visit() decide how to steer the walk. You may
> > need some special handling to get the address arithmetic right when
> > skipping over a table of page descriptors.

I am probably not getting the whole inner workings of this solution, but 
IIUC the idea would be to walk the blocks, but not the pages, right?

Blocks meaning level2- and pages being level3?
 
> I was wondering along similar lines, but maybe it would be useful just
> to pass a maximum level to the walker logic? That feels like the most
> general case without complicating the existing logic.

This proposal seems simpler for me to understand, and indeed looks like a 
better solution than what I have proposed, taking care of  the 
'already split' case with better performance, as it don't even walk a 
single level-3 entry. 

On the 'splitting' case, it also works flawlessly if the memory is given in 
level-2 blocks. There is only one case that I would like to address here:

- Memory given in level-1 blocks (say 1GB)
- Walker flag says 'walk down to level-2 only'
- Split Walker on level-1 will break page down to (up to) level-3 entries.
- Walker will continue to be called on level-2 entries, even though it's 
  not necessary.

To solve this, I would like to suggest a new flag, that skips a table 
that has just been created. This could be easily implemented in 
__kvm_page_visit() on top of the max level flags suggested.

enum kvm_pgtable_walk_flags {
[...]
	KVM_PGTABLE_WALK_SKIP_LEVEL3		= BIT(7),
	KVM_PGTABLE_WALK_SKIP_LEVEL2		= BIT(8),
	KVM_PGTABLE_WALK_SKIP_LEVEL1		= BIT(9),
	KVM_PGTABLE_WALK_SKIP_NEW_TABLE		= BIT(10),
};

How does that sound?

Thanks!
Leo

Re: [RFC PATCH 1/2] KVM: arm64: Introduce S2 walker SKIP return options

[Will Deacon]

On Mon, May 18, 2026 at 02:45:59PM +0100, Leonardo Bras wrote:
> Hello Oliver, Will,
> Thanks for reviewing!
> 
> On Mon, May 18, 2026 at 09:52:16AM +0100, Will Deacon wrote:
> > On Mon, May 18, 2026 at 12:22:47AM -0700, Oliver Upton wrote:
> > > On Fri, May 15, 2026 at 08:59:02PM +0100, Leonardo Bras wrote:
> > > > Introduce S2 walker return values:
> > > > - SKIP_CHILDREN: skip walking the children of the current node
> > > > - SKIP_SIBLINGS: skip waling the siblings of the current node
> > > > 
> > > > Also, modify __kvm_pgtable_visit() to fulfil the hing on above return
> > > > values. Current walkers should not be impacted
> > > 
> > > I'd rather see something based around new walk flags than introducing an
> > > entirely new mechanic around return values.
> > > 
> > > e.g. you could split the LEAF flag into separate flags for blocks v.
> > > pages:
> > > 
> > > 	KVM_PGTABLE_WALK_PAGE,
> > > 	KVM_PGTABLE_WALK_BLOCK,
> > > 	KVM_PGTABLE_WALK_LEAF	= KVM_PGTABLE_WALK_PAGE |
> > > 				  KVM_PGTABLE_WALK_BLOCK,
> > > 
> > > and then let __kvm_pgtable_visit() decide how to steer the walk. You may
> > > need some special handling to get the address arithmetic right when
> > > skipping over a table of page descriptors.
> 
> I am probably not getting the whole inner workings of this solution, but 
> IIUC the idea would be to walk the blocks, but not the pages, right?
> 
> Blocks meaning level2- and pages being level3?
>  
> > I was wondering along similar lines, but maybe it would be useful just
> > to pass a maximum level to the walker logic? That feels like the most
> > general case without complicating the existing logic.
> 
> This proposal seems simpler for me to understand, and indeed looks like a 
> better solution than what I have proposed, taking care of  the 
> 'already split' case with better performance, as it don't even walk a 
> single level-3 entry. 
> 
> On the 'splitting' case, it also works flawlessly if the memory is given in 
> level-2 blocks. There is only one case that I would like to address here:
> 
> - Memory given in level-1 blocks (say 1GB)
> - Walker flag says 'walk down to level-2 only'
> - Split Walker on level-1 will break page down to (up to) level-3 entries.
> - Walker will continue to be called on level-2 entries, even though it's 
>   not necessary.

If you're only visiting leaves, why would it be called on the level-2
table entries?

Will

Re: [RFC PATCH 1/2] KVM: arm64: Introduce S2 walker SKIP return options

[Leonardo Bras]

On Tue, May 19, 2026 at 01:43:37PM +0100, Will Deacon wrote:
> On Mon, May 18, 2026 at 02:45:59PM +0100, Leonardo Bras wrote:
> > Hello Oliver, Will,
> > Thanks for reviewing!
> > 
> > On Mon, May 18, 2026 at 09:52:16AM +0100, Will Deacon wrote:
> > > On Mon, May 18, 2026 at 12:22:47AM -0700, Oliver Upton wrote:
> > > > On Fri, May 15, 2026 at 08:59:02PM +0100, Leonardo Bras wrote:
> > > > > Introduce S2 walker return values:
> > > > > - SKIP_CHILDREN: skip walking the children of the current node
> > > > > - SKIP_SIBLINGS: skip waling the siblings of the current node
> > > > > 
> > > > > Also, modify __kvm_pgtable_visit() to fulfil the hing on above return
> > > > > values. Current walkers should not be impacted
> > > > 
> > > > I'd rather see something based around new walk flags than introducing an
> > > > entirely new mechanic around return values.
> > > > 
> > > > e.g. you could split the LEAF flag into separate flags for blocks v.
> > > > pages:
> > > > 
> > > > 	KVM_PGTABLE_WALK_PAGE,
> > > > 	KVM_PGTABLE_WALK_BLOCK,
> > > > 	KVM_PGTABLE_WALK_LEAF	= KVM_PGTABLE_WALK_PAGE |
> > > > 				  KVM_PGTABLE_WALK_BLOCK,
> > > > 
> > > > and then let __kvm_pgtable_visit() decide how to steer the walk. You may
> > > > need some special handling to get the address arithmetic right when
> > > > skipping over a table of page descriptors.
> > 
> > I am probably not getting the whole inner workings of this solution, but 
> > IIUC the idea would be to walk the blocks, but not the pages, right?
> > 
> > Blocks meaning level2- and pages being level3?
> >  
> > > I was wondering along similar lines, but maybe it would be useful just
> > > to pass a maximum level to the walker logic? That feels like the most
> > > general case without complicating the existing logic.
> > 
> > This proposal seems simpler for me to understand, and indeed looks like a 
> > better solution than what I have proposed, taking care of  the 
> > 'already split' case with better performance, as it don't even walk a 
> > single level-3 entry. 
> > 
> > On the 'splitting' case, it also works flawlessly if the memory is given in 
> > level-2 blocks. There is only one case that I would like to address here:
> > 
> > - Memory given in level-1 blocks (say 1GB)
> > - Walker flag says 'walk down to level-2 only'
> > - Split Walker on level-1 will break page down to (up to) level-3 entries.
> > - Walker will continue to be called on level-2 entries, even though it's 
> >   not necessary.
> 
> If you're only visiting leaves, why would it be called on the level-2
> table entries?
> 

Because once the leaf is turned into a table by the splitting walker, it 
gets reloaded and walked. This is an excerpt of __kvm_pgtable_visit():

---

if (!table && (ctx.flags & KVM_PGTABLE_WALK_LEAF)) {
         ret = kvm_pgtable_visitor_cb(data, &ctx, KVM_PGTABLE_WALK_LEAF);
         reload = true;
 }

 /*
  * Reload the page table after invoking the walker callback for leaf
  * entries or after pre-order traversal, to allow the walker to descend
  * into a newly installed or replaced table.
  */
 if (reload) {
         ctx.old = READ_ONCE(*ptep);
         table = kvm_pte_table(ctx.old, level);
 }

 if (!kvm_pgtable_walk_continue(data->walker, ret))
         goto out;

 if (!table) {
         data->addr = ALIGN_DOWN(data->addr, kvm_granule_size(level));
         data->addr += kvm_granule_size(level);
         goto out;
 }

 childp = (kvm_pteref_t)kvm_pte_follow(ctx.old, mm_ops);
 ret = __kvm_pgtable_walk(data, mm_ops, childp, level + 1);
 if (!kvm_pgtable_walk_continue(data->walker, ret))
         goto out;

---

After the leaf is visited, it makes reload=true, which causes the newly 
created table to be detected as such and waked down. It means even the page 
that just got split will be walked down to the specified level.

Example:
- Split this level-1 leave:
  - Walker creates the whole structure up to given level (currently 3)
  - Walker returns, gets reloaded, table detected, go down on that one
  - Level 2 entries walked (which is unnecessary)

Please let me know if I am misunderstanding something.

Thanks!
Leo

Re: [RFC PATCH 1/2] KVM: arm64: Introduce S2 walker SKIP return options

[Will Deacon]

On Tue, May 19, 2026 at 01:56:48PM +0100, Leonardo Bras wrote:
> On Tue, May 19, 2026 at 01:43:37PM +0100, Will Deacon wrote:
> > > > I was wondering along similar lines, but maybe it would be useful just
> > > > to pass a maximum level to the walker logic? That feels like the most
> > > > general case without complicating the existing logic.
> > > 
> > > This proposal seems simpler for me to understand, and indeed looks like a 
> > > better solution than what I have proposed, taking care of  the 
> > > 'already split' case with better performance, as it don't even walk a 
> > > single level-3 entry. 
> > > 
> > > On the 'splitting' case, it also works flawlessly if the memory is given in 
> > > level-2 blocks. There is only one case that I would like to address here:
> > > 
> > > - Memory given in level-1 blocks (say 1GB)
> > > - Walker flag says 'walk down to level-2 only'
> > > - Split Walker on level-1 will break page down to (up to) level-3 entries.
> > > - Walker will continue to be called on level-2 entries, even though it's 
> > >   not necessary.
> > 
> > If you're only visiting leaves, why would it be called on the level-2
> > table entries?
> > 
> 
> Because once the leaf is turned into a table by the splitting walker, it 
> gets reloaded and walked. This is an excerpt of __kvm_pgtable_visit():

Sorry, I was musing about the semantics after adding something to limit
the maximum level. I don't dispute what the current code would do.

> Example:
> - Split this level-1 leave:
>   - Walker creates the whole structure up to given level (currently 3)
>   - Walker returns, gets reloaded, table detected, go down on that one
>   - Level 2 entries walked (which is unnecessary)
> 
> Please let me know if I am misunderstanding something.

I just don't grok why this would happen if we limited the maximum level
to '2' _and_ said we only wanted to visit the leaf entries. In that
case, I wouldn't expect to descend into any of the L2 table entries
(because that would imply going beyond level 2) and I wouldn't expect to
be called for the table entries either (because we're only interested in
leaves).

Will

Re: [RFC PATCH 1/2] KVM: arm64: Introduce S2 walker SKIP return options

[Leonardo Bras]

On Tue, May 19, 2026 at 02:15:41PM +0100, Will Deacon wrote:
> On Tue, May 19, 2026 at 01:56:48PM +0100, Leonardo Bras wrote:
> > On Tue, May 19, 2026 at 01:43:37PM +0100, Will Deacon wrote:
> > > > > I was wondering along similar lines, but maybe it would be useful just
> > > > > to pass a maximum level to the walker logic? That feels like the most
> > > > > general case without complicating the existing logic.
> > > > 
> > > > This proposal seems simpler for me to understand, and indeed looks like a 
> > > > better solution than what I have proposed, taking care of  the 
> > > > 'already split' case with better performance, as it don't even walk a 
> > > > single level-3 entry. 
> > > > 
> > > > On the 'splitting' case, it also works flawlessly if the memory is given in 
> > > > level-2 blocks. There is only one case that I would like to address here:
> > > > 
> > > > - Memory given in level-1 blocks (say 1GB)
> > > > - Walker flag says 'walk down to level-2 only'
> > > > - Split Walker on level-1 will break page down to (up to) level-3 entries.
> > > > - Walker will continue to be called on level-2 entries, even though it's 
> > > >   not necessary.
> > > 
> > > If you're only visiting leaves, why would it be called on the level-2
> > > table entries?
> > > 
> > 
> > Because once the leaf is turned into a table by the splitting walker, it 
> > gets reloaded and walked. This is an excerpt of __kvm_pgtable_visit():
> 
> Sorry, I was musing about the semantics after adding something to limit
> the maximum level. I don't dispute what the current code would do.
> 
> > Example:
> > - Split this level-1 leave:
> >   - Walker creates the whole structure up to given level (currently 3)
> >   - Walker returns, gets reloaded, table detected, go down on that one
> >   - Level 2 entries walked (which is unnecessary)
> > 
> > Please let me know if I am misunderstanding something.
> 
> I just don't grok why this would happen if we limited the maximum level
> to '2' _and_ said we only wanted to visit the leaf entries. In that
> case, I wouldn't expect to descend into any of the L2 table entries
> (because that would imply going beyond level 2) and I wouldn't expect to
> be called for the table entries either (because we're only interested in
> leaves).

Agree, if we specify to skip level-3 entries, it would only walk up to 
level-2 entries, but take above example in detail:
- Split these level-1 leaves, up to level-3 leaves (regular)
  - INFO: kvm_pgtable_walk will call walker:
    - only up to level-2 entries (skip level-3)
    - only on leaf entries
  - Walk first level-1 leaf, calls walker
    - walker will split the level-1 leaf in level-3 leaves
    - walker return from that first level-1 leaf
  - level-1 leaf is reloaded as a table
    - level-2 entries of that table are also walked (unnecessary)
    - on each of the level-2 table entries, level-3 entries are skipped

To avoid the unecessary walk of the level-2 entries above, we would need to 
specify 'skip level-2' that could be an issue if we have a mix of level-1 
and level-2 leaves, as the level-2 leaves in that case would not be split.

That's why I suggest something like "skip recently created table" as a flag 
as well, so we can guarantee no newly created table gets walked 
unecessarily.

Please help me if I am missing something important.

Thanks!
Leo

[RFC PATCH 2/2] KVM: arm64: Improve splitting performance by using SKIP return values

[Leonardo Bras]

Splitting an S2 pagetable is needed when using dirty-bit tracking.

Currently, when splitting, all the child and sibling nodes will be walked,
with the walker just returning earlier if there is nothing to do. This
means all pagetable entries in the splitting range get a callback from the
walker function, even if it was just split, or it's a level-3 entry.

Optimize splitting in two cases:
- If a level-3 entry is walked, it means the parent level-2 entry is split,
  so avoid walking all level-3 siblings.
- If a split just succeeded in an table entry, it means all children nodes
  are already split, so skip walking this entry's children.

Optimization measured on a 1GB VM, running on the model, splitting all at
the beginning (no manual protect):
- Memory was already split (4k pages): 	-97.33% runtime (-172ms) - 20 runs
- THP backed memory: 			-19.82% runtime (-153ms) - 10 runs
- 1x1GB hugetlb memory:			-20.65% runtime (-150ms) - 10 runs

(Above runtime is measured on kvm_mmu_split_huge_pages(), using
ktime_get_real_ns() before and after function call)

Signed-off-by: Leonardo Bras <leo.bras@arm.com>
---
 arch/arm64/kvm/hyp/pgtable.c | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c
index 4e43339522bb..164c5bcd6026 100644
--- a/arch/arm64/kvm/hyp/pgtable.c
+++ b/arch/arm64/kvm/hyp/pgtable.c
@@ -1502,23 +1502,27 @@ static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx,
 	struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops;
 	struct kvm_mmu_memory_cache *mc = ctx->arg;
 	struct kvm_s2_mmu *mmu;
 	kvm_pte_t pte = ctx->old, new, *childp;
 	enum kvm_pgtable_prot prot;
 	s8 level = ctx->level;
 	bool force_pte;
 	int nr_pages;
 	u64 phys;
 
-	/* No huge-pages exist at the last level */
+	/*
+	 * No huge-pages exist at the last level
+	 * Also, if one PTE exist in the last level, the whole block is already
+	 * split, so skip walking it's siblings.
+	 */
 	if (level == KVM_PGTABLE_LAST_LEVEL)
-		return 0;
+		return SKIP_SIBLINGS;
 
 	/* We only split valid block mappings */
 	if (!kvm_pte_valid(pte))
 		return 0;
 
 	nr_pages = stage2_block_get_nr_page_tables(level);
 	if (nr_pages < 0)
 		return nr_pages;
 
 	if (mc->nobjs >= nr_pages) {
@@ -1554,21 +1558,23 @@ static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx,
 		return -EAGAIN;
 	}
 
 	/*
 	 * Note, the contents of the page table are guaranteed to be made
 	 * visible before the new PTE is assigned because stage2_make_pte()
 	 * writes the PTE using smp_store_release().
 	 */
 	new = kvm_init_table_pte(childp, mm_ops);
 	stage2_make_pte(ctx, new);
-	return 0;
+
+	/* All child entries are already split, so skip walking them */
+	return SKIP_CHILDREN;
 }
 
 int kvm_pgtable_stage2_split(struct kvm_pgtable *pgt, u64 addr, u64 size,
 			     struct kvm_mmu_memory_cache *mc)
 {
 	struct kvm_pgtable_walker walker = {
 		.cb	= stage2_split_walker,
 		.flags	= KVM_PGTABLE_WALK_LEAF,
 		.arg	= mc,
 	};
-- 
2.54.0

Re: [RFC PATCH 0/2] Optimize S2 page splitting

[Marc Zyngier]

On Fri, 15 May 2026 20:59:01 +0100,
Leonardo Bras <leo.bras@arm.com> wrote:
> 
> While playing with dirty-bit tracking, I decided to take a look on how page
> splitting works. Found out all entries are walked, even though we can infer,
> for instance that:
> - If a level-3 entry is walked, it means the parent level-2 entry is split
> - If a split just succeeded in an table entry, it means all children nodes
>   are already split
> 
> So I tried to optimize it in a way that it does not break other users.
> 
> My main idea is to introduce positive return values that hint to the
> pagetable walking mechanism that either siblings or children can be 
> skipped. That should be contained to the visitor function, that returns
> zero if no error was detected.
> 
> Numbers on above optimization are promising:
> A 1GB VM, running on the model, splitting all at the beginning 
> (no manual protect):
> - Memory was already split (4k pages): 	-97.33% runtime (-172ms) - 20 runs
> - THP backed memory: 			-19.82% runtime (-153ms) - 10 runs
> - 1x1GB hugetlb memory:			-20.65% runtime (-150ms) - 10 runs
>

I haven't looked at the changes in details, but the methodology is
quite flawed. For a start, measuring anything on a software model
(QEMU or FVP) doesn't mean anything performance-wise. The trade-offs
are completely different from a HW implementation, and even the notion
of time is pretty inconsistent.

Please run this on actual HW. I'm sure your employer can give you
access to one of these mythical arm64 toys. Measure things from
userspace, not from the kernel, so that you have all the overheads.
Don't add console output, because that will make things far worse.

I'm sure you can hack one of the selftests for this purpose.

Thanks,

	M.

-- 
Jazz isn't dead. It just smells funny.

Re: [RFC PATCH 0/2] Optimize S2 page splitting

[Leonardo Bras]

On Sat, May 16, 2026 at 10:15:36AM +0100, Marc Zyngier wrote:
> On Fri, 15 May 2026 20:59:01 +0100,
> Leonardo Bras <leo.bras@arm.com> wrote:
> > 
> > While playing with dirty-bit tracking, I decided to take a look on how page
> > splitting works. Found out all entries are walked, even though we can infer,
> > for instance that:
> > - If a level-3 entry is walked, it means the parent level-2 entry is split
> > - If a split just succeeded in an table entry, it means all children nodes
> >   are already split
> > 
> > So I tried to optimize it in a way that it does not break other users.
> > 
> > My main idea is to introduce positive return values that hint to the
> > pagetable walking mechanism that either siblings or children can be 
> > skipped. That should be contained to the visitor function, that returns
> > zero if no error was detected.
> > 
> > Numbers on above optimization are promising:
> > A 1GB VM, running on the model, splitting all at the beginning 
> > (no manual protect):
> > - Memory was already split (4k pages): 	-97.33% runtime (-172ms) - 20 runs
> > - THP backed memory: 			-19.82% runtime (-153ms) - 10 runs
> > - 1x1GB hugetlb memory:			-20.65% runtime (-150ms) - 10 runs
> >
> 
> I haven't looked at the changes in details, but the methodology is
> quite flawed. For a start, measuring anything on a software model
> (QEMU or FVP) doesn't mean anything performance-wise. The trade-offs
> are completely different from a HW implementation, and even the notion
> of time is pretty inconsistent.
> 
> Please run this on actual HW. I'm sure your employer can give you
> access to one of these mythical arm64 toys.

Ok, will use real hardware next.

> Measure things from
> userspace, not from the kernel, so that you have all the overheads.
> Don't add console output, because that will make things far worse.
> 
> I'm sure you can hack one of the selftests for this purpose.

I think the dirty_log_perf_test should have an config I can use for that 
without introducing any change. 

Thanks!
Leo