Re: [RFC PATCH 0/2] Optimize S2 page splitting

[Leonardo Bras <leo.bras@arm.com>]

On Sat, May 16, 2026 at 10:15:36AM +0100, Marc Zyngier wrote:
> On Fri, 15 May 2026 20:59:01 +0100,
> Leonardo Bras <leo.bras@arm.com> wrote:
> > 
> > While playing with dirty-bit tracking, I decided to take a look on how page
> > splitting works. Found out all entries are walked, even though we can infer,
> > for instance that:
> > - If a level-3 entry is walked, it means the parent level-2 entry is split
> > - If a split just succeeded in an table entry, it means all children nodes
> >   are already split
> > 
> > So I tried to optimize it in a way that it does not break other users.
> > 
> > My main idea is to introduce positive return values that hint to the
> > pagetable walking mechanism that either siblings or children can be 
> > skipped. That should be contained to the visitor function, that returns
> > zero if no error was detected.
> > 
> > Numbers on above optimization are promising:
> > A 1GB VM, running on the model, splitting all at the beginning 
> > (no manual protect):
> > - Memory was already split (4k pages): 	-97.33% runtime (-172ms) - 20 runs
> > - THP backed memory: 			-19.82% runtime (-153ms) - 10 runs
> > - 1x1GB hugetlb memory:			-20.65% runtime (-150ms) - 10 runs
> >
> 
> I haven't looked at the changes in details, but the methodology is
> quite flawed. For a start, measuring anything on a software model
> (QEMU or FVP) doesn't mean anything performance-wise. The trade-offs
> are completely different from a HW implementation, and even the notion
> of time is pretty inconsistent.
> 
> Please run this on actual HW. I'm sure your employer can give you
> access to one of these mythical arm64 toys. Measure things from
> userspace, not from the kernel, so that you have all the overheads.
> Don't add console output, because that will make things far worse.
> 
> I'm sure you can hack one of the selftests for this purpose.

Hello Marc,

I have ran some tests in real hardware (AmpereOne) and calling from 
userspace as you suggested.

The tests create a 64GB VM, using either regular pages (no split needed), 
THP backed memory or Hugetlb. 

Those tests measured the times for the whole syscall that happens to do
eager page splitting, in this case,
1 - Manual protect and init set: On every KVM_CLEAR_DIRTY_LOG, if the page
    was requested to be cleaned, it is also split.
2 - Otherwise, on dirty log enable (aka KVM_SET_USER_MEMORY_REGION2 with
    KVM_MEM_LOG_DIRTY_PAGES set in flags), for the whole memslot is split
    at once.

Tests reflect new suggestions I got in this patchset: instead of using 
return values, I am using flags to skip whole levels, or skipping children 
nodes. I tested those 2 kernels, on top of vanilla:
a - Skip levels
b - Skip levels, and skip children node

Results averaged across 4k, 16k and 64k pages, a percentage of time 
saved, compared to vanilla kernel. More is better.

For (1), we have
regular pages:	26.7% (a) and 25.3% (b)
THP:		13.2% (a) and 11.9% (b)
Hugetlb:	14.3% (a) and 13.2% (b)

For (2), we have:
regular pages:	33.1% (a) and 35.0% (b)
THP:		11.9% (a) and 10.7% (b)
Hugetlb:	13.4% (a) and 13.2% (b)

On above results, I could notice about 1% overhead showing that skipping 
child nodes (testing flag) ends up being counter-productive compared to 
just skipping levels. The only case this does not happen is (2a), but it's 
not clear on why that happens.

Based on that, I plan to remove the skip_child patch, and send a v2 with a 
flag-based mechanism, which results are shown above.

Please let me know of any thoughts, suggestions or ideas you might have 
about it.

Thanks!
Leo