Re: [RFC PATCH 0/2] Optimize S2 page splitting

[Leonardo Bras <leo.bras@arm.com>]

On Fri, May 29, 2026 at 01:25:10PM +0100, Marc Zyngier wrote:
> On Thu, 28 May 2026 18:00:48 +0100,
> Leonardo Bras <leo.bras@arm.com> wrote:
> > 
> > On Sat, May 16, 2026 at 10:15:36AM +0100, Marc Zyngier wrote:
> > > On Fri, 15 May 2026 20:59:01 +0100,
> > > Leonardo Bras <leo.bras@arm.com> wrote:
> > > > 
> > > > While playing with dirty-bit tracking, I decided to take a look on how page
> > > > splitting works. Found out all entries are walked, even though we can infer,
> > > > for instance that:
> > > > - If a level-3 entry is walked, it means the parent level-2 entry is split
> > > > - If a split just succeeded in an table entry, it means all children nodes
> > > >   are already split
> > > > 
> > > > So I tried to optimize it in a way that it does not break other users.
> > > > 
> > > > My main idea is to introduce positive return values that hint to the
> > > > pagetable walking mechanism that either siblings or children can be 
> > > > skipped. That should be contained to the visitor function, that returns
> > > > zero if no error was detected.
> > > > 
> > > > Numbers on above optimization are promising:
> > > > A 1GB VM, running on the model, splitting all at the beginning 
> > > > (no manual protect):
> > > > - Memory was already split (4k pages): 	-97.33% runtime (-172ms) - 20 runs
> > > > - THP backed memory: 			-19.82% runtime (-153ms) - 10 runs
> > > > - 1x1GB hugetlb memory:			-20.65% runtime (-150ms) - 10 runs
> > > >
> > > 
> > > I haven't looked at the changes in details, but the methodology is
> > > quite flawed. For a start, measuring anything on a software model
> > > (QEMU or FVP) doesn't mean anything performance-wise. The trade-offs
> > > are completely different from a HW implementation, and even the notion
> > > of time is pretty inconsistent.
> > > 
> > > Please run this on actual HW. I'm sure your employer can give you
> > > access to one of these mythical arm64 toys. Measure things from
> > > userspace, not from the kernel, so that you have all the overheads.
> > > Don't add console output, because that will make things far worse.
> > > 
> > > I'm sure you can hack one of the selftests for this purpose.
> > 
> > Hello Marc,
> > 
> > I have ran some tests in real hardware (AmpereOne) and calling from 
> > userspace as you suggested.
> > 
> > The tests create a 64GB VM, using either regular pages (no split needed), 
> > THP backed memory or Hugetlb. 
> > 
> > Those tests measured the times for the whole syscall that happens to do
> > eager page splitting, in this case,
> > 1 - Manual protect and init set: On every KVM_CLEAR_DIRTY_LOG, if the page
> >     was requested to be cleaned, it is also split.
> > 2 - Otherwise, on dirty log enable (aka KVM_SET_USER_MEMORY_REGION2 with
> >     KVM_MEM_LOG_DIRTY_PAGES set in flags), for the whole memslot is split
> >     at once.
> > 
> > Tests reflect new suggestions I got in this patchset: instead of using 
> > return values, I am using flags to skip whole levels, or skipping children 
> > nodes. I tested those 2 kernels, on top of vanilla:
> > a - Skip levels
> > b - Skip levels, and skip children node
> > 
> > Results averaged across 4k, 16k and 64k pages, a percentage of time 
> > saved, compared to vanilla kernel. More is better.
> > 
> > For (1), we have
> > regular pages:	26.7% (a) and 25.3% (b)
> > THP:		13.2% (a) and 11.9% (b)
> > Hugetlb:	14.3% (a) and 13.2% (b)
> > 
> > For (2), we have:
> > regular pages:	33.1% (a) and 35.0% (b)
> > THP:		11.9% (a) and 10.7% (b)
> > Hugetlb:	13.4% (a) and 13.2% (b)
> > 
> > On above results, I could notice about 1% overhead showing that skipping 
> > child nodes (testing flag) ends up being counter-productive compared to 
> > just skipping levels. The only case this does not happen is (2a), but it's 
> > not clear on why that happens.
> > 
> > Based on that, I plan to remove the skip_child patch, and send a v2 with a 
> > flag-based mechanism, which results are shown above.
> > 
> > Please let me know of any thoughts, suggestions or ideas you might have 
> > about it.
> 
> Right, that seems compelling.

Awesome!

> I'd suggest you send a new version with:
> 
> - a pointer to the test and its the exact invocation parameters so
>   that people other than you can reproduce it
> 
> - these numbers with a bit of analysis explaining where the gains are
> 
> With that, we can look at it again and compare results on different
> workloads and HW.
> 

Noted!
Will attach test parameters and full results, describing where the gains 
come from. 

Thanks!
Leo