Re: [RFC PATCH 5/5] KVM: arm64: Enable HAFDBS for guests not on migration

[Leonardo Bras <leo.bras@arm.com>]

On Thu, Sep 17, 2026 at 02:40:42PM +0100, Leonardo Bras wrote:
> On Wed, Sep 16, 2026 at 04:27:19PM -0700, Oliver Upton wrote:
> > On Wed, Sep 16, 2026 at 03:00:46PM +0100, Leonardo Bras wrote:
> > > On Tue, Sep 15, 2026 at 05:10:02PM -0700, Oliver Upton wrote:
> > > > Hi,
> > > > 
> > > > On Tue, Sep 01, 2026 at 06:15:56PM +0100, Leonardo Bras wrote:
> > > > > When dirty-logging is disabled, even non-write faults make a page dirty,
> > > > > which avoids a second fault when the page is actually written to.
> > > > > 
> > > > > On dirty-logging enable, this approach causes all (writable) pages on the
> > > > > memslot to be marked clean, even if they were not written to, which can
> > > > > take a lot of time, while holding the MMU lock, doing atomic writes to
> > > > > PTEs.
> > > > 
> > > > Do you have any performance numbers for this? Enabling HAFDBS seems a
> > > > bit involved to avoid some stores on the first pass.
> > > 
> > > Not yet, but if the idea does not look too crazy I can find hardware and 
> > > collect some data :)
> > 
> > TBH this looks like a micro-optimization so I'm not expecting the
> > performance gains to justify the behavior change.
> > 
> 
> Okay then, I will get a machine to figure the performance gains :)

Got the performance numbers here. 
I tested both the case where all PTEs are dirty, and the case that 50% of 
the PTEs are dirty.

For that test, actually patch #5 could make kvm_pgtable_stage2_wrprotect() 
slower, which is counter-intuitive, in all platforms tested:

When we walk the S2 pagetable to wrprotect, using the new walker, the 
write to the PTE itself is so cheap* that the cost of the branch 
misprediction (write PTE or not) for the 50/50 case is significantly higher 
than the cost of the 50% extra writes, which makes the 50/50 up to 60% 
slower than the 100% dirty case. It is still slower in the '25% dirty' as 
well, and starts to get even at '10% dirty'. 

* (non-atomic, everything is in order, walker is small, cpu will probably 
bundle it all in a single cacheline write) 

Since the idea of patch #5 was to reduce PTE writes in the first wrprotect 
walk, it becomes faster only if only a small portion (<10%) of the VM was 
write-faulted while the remaining was read-faulted, which seems to be a 
really reduced audience.

Since this comes with increased complexity, and also carry costs of 
having HAFDBS itself enabled outside dirty-tracking, I don't think it's a 
good idea to further pursue patch #5. It's only value is that it served as 
a base for HDBSS implementation.

Thanks for reading!
Leo