Re: [PATCH v3 0/3] Optimize S2 page splitting

[Leonardo Bras <leo.bras@arm.com>]

On Wed, Jul 08, 2026 at 02:40:56PM +0100, Leonardo Bras wrote:
> While playing with dirty-bit tracking, I decided to take a look on how page
> splitting works. Found out all entries are walked, even though we can infer,
> for instance that:
> - If a level-3 entry is walked, it means the parent level-2 entry is split
> - If a split just succeeded in an table entry, it means all children nodes
>   are already split
> 
> This patches' idea is to introduce new walking flags to skip pagetable
> levels 0-3.
> 
> The idea of skipping child nodes was also tested, but it was marginally
> slower than just skipping levels, so it was discarted.
> 
> Optimization measured on two scenarios involving eager-splitting on a
> VM with 32 memslot of 2GB (total 64GB), and vcpu per slot:
> - Scenario 1: No manual protect, whole memslot split at dirty-track enable
>   (KVM_SET_USER_MEMORY_REGION2 ioctl with KVM_MEM_LOG_DIRTY_PAGES)
>   - Split happens only once, whole region
>   - Evalutes improved batch performance of splitting
> - Scenario 2: Manual protect, split happens during every dirty-bit clean
>   (KVM_CLEAR_DIRTY_LOG ioctl), average for 2 iterations.
>   - Split called multiple times, for smaller 64-page sections.
>   - Evaluate improved performance for multiple calls
> 
> Scenario 1, improvement on dirty-track enable ioctl for the memslot:
> - Memory was already split (4k pages):  -43.18% runtime (stdev 3.87%)
> - THP backed memory:                    -25.77% runtime (stdev 1.01%)
> - 64x1GB hugetlb memory:                -25.62% runtime (stdev 1.09%)
> 
> Scenario 2, improvement on dirty-log clean ioctl for the memslot:
> - Memory was already split (4k pages):  -39.88% runtime (stdev 2.71%)
> - THP backed memory:                    -25.18% runtime (stdev 0.32%)
> - 64x1GB hugetlb memory:                -49.17% runtime (stdev 1.86%)
> 
> For collecting above numbers, the following script was ran in both vanilla
> and patched kernels, with kernel parameter 'default_hugepagesz=1G', on an
> TX2 with 128GB RAM.
> 
> --- dirty_test.sh
> #!/bin/bash
> filename=$(uname -r |cut -d'-' -f 4-)
> 
> run_test(){
>   base_test="./dirty_log_perf_test -b 2G -v 32 -m 6 -m 8"
> 
>   # Manual cleaning disable
>   ${base_test} -g
>   ${base_test} -g -s anonymous_thp
>   echo 64 > /proc/sys/vm/nr_hugepages  
>   ${base_test} -g -s shared_hugetlb
>   echo 0 > /proc/sys/vm/nr_hugepages
> 
>   # Manual cleaning enable
>   ${base_test}
>   ${base_test} -s anonymous_thp
>   echo 64 > /proc/sys/vm/nr_hugepages  
>   ${base_test} -s shared_hugetlb
>   echo 0 > /proc/sys/vm/nr_hugepages
> }
> 
> run_test 2>&1 | tee ${filename}
> ---
> 
> Above dirty_log_perf_test command is the standard kvm selftest found in the
> kernel tree. It tested the following guest modes:
> Testing guest mode: PA-bits:40,  VA-bits:48,  4K pages
> Testing guest mode: PA-bits:40,  VA-bits:48, 64K pages
> (Modes with PA-bits:36 were discarted in this version, given the amount
> of RAM being used for testing, and the similarity of previous results)
> 
> Performance numbers from above modes were used to calculate average and
> stdev showed in the optimization results.
> 
> Changes since v2:
> - Rebased on top of v7.2-rc1
> - Improved testing, added more memory, re-tested
>   - Now: 32 vcpus @ total of 64G
>   - Before: 1cpu @ 16G
v2 Link: https://lore.kernel.org/all/20260618131447.764085-1-leo.bras@arm.com/

> 
> Changes since v1:
> - Fixed inverted flag verification priority (Sashiko)
> - Fixed incorrectly skipping POST call if level was skipped (Sashiko), and to that
> - New pre-patch that changes goto-out -> return to avoid re-testing walk_continue 
> v1 Link: https://lore.kernel.org/lkml/20260610202112.2695205-2-leo.bras@arm.com/
> 
> Changes since RFC:
> - Changed approach from return value to walk flags (Will Deacon)
> - Discarted skip_child approach (Oliver Upton)
> - Measured in real hardware, and from userspace perspective (Marc Zyngier)
> - Better explanation of what and how numbers were collected
> RFC Link: https://lore.kernel.org/all/20260515195904.2466381-1-leo.bras@arm.com/
> 
> Thanks!
> Leo
> 
> 
> Leonardo Bras (3):
>   KVM: arm64: Avoid re-testing walk_continue
>   KVM: arm64: Introduce KVM_PGTABLE_WALK_SKIP_LEVEL* walk flags
>   KVM: arm64: Make stage2_split_walker() skip unnecessary walks
> 
>  arch/arm64/include/asm/kvm_pgtable.h | 13 +++++++++++++
>  arch/arm64/kvm/hyp/pgtable.c         | 28 +++++++++++++++++++++-------
>  2 files changed, 34 insertions(+), 7 deletions(-)
> 
> 
> base-commit: dc59e4fea9d83f03bad6bddf3fa2e52491777482
> -- 
> 2.55.0
>