Re: [PATCH v2 02/13] KVM: arm64: Enable eager hugepage splitting if HDBSS is available

[Leonardo Bras <leo.bras@arm.com>]

On Mon, Jul 06, 2026 at 03:00:07PM +0100, Leonardo Bras wrote:
> On Wed, Jul 01, 2026 at 11:45:52AM +0100, Leonardo Bras wrote:
> > On Tue, Jun 30, 2026 at 11:43:01AM -0700, Oliver Upton wrote:
> > > On Tue, Jun 30, 2026 at 06:09:56PM +0100, Leonardo Bras wrote:
> > > > On Tue, Jun 30, 2026 at 08:44:20AM -0700, Oliver Upton wrote:
> > > > > On Tue, Jun 30, 2026 at 01:58:48PM +0100, Leonardo Bras wrote:
> > > > > > On Mon, Jun 29, 2026 at 10:06:38AM -0700, Oliver Upton wrote:
> > > > > > > > But this raises a topic I would like to understand:
> > > > > > > > - Do we actually need this to be a block_size to assure correctness? or is 
> > > > > > > >   it just about efficiency?
> > > > > > > 
> > > > > > > What value is there in having a chunk size larger than the largest
> > > > > > > possible block mapping? The whole UAPI is deliberately tied up with page
> > > > > > > table geometry.
> > > > > > 
> > > > > > Not larger, possibly smallerv My concern was the difference in pages to 
> > > > > > split between 4k, 16k and 64k.
> > > > > 
> > > > > Ok, well in any case the upper bound is going to be the largest possible
> > > > > block mapping for a given page granule.
> > > > >
> > > > 
> > > > Sure, we can do this.
> > > 
> > > I'm describing the UAPI behavior of what we already have. Please do not
> > > change anything and continue to leave userspace in charge of eager
> > > splitting / chunk size.
> > > 
> > > > > > > 
> > > > > > > Overall, I'm not buying the argument for changing the behavior of
> > > > > > > KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE. There are very good reasons for
> > > > > > > *not* eagerly splitting the entire address space, especially if you know
> > > > > > > the working set of the VM is small.
> > > > > > > 
> > > > > > > You can still use HDBSS without eagerly splitting, so long as block
> > > > > > > mappings are {DBM, S2AP_W} = {0, 0} and leaf mappings (which have
> > > > > > > a writable PFN) are {1, 0}.
> > > > > > > 
> > > > > > 
> > > > > > Block mappings being read-only, and leaf mappings being writable-clean, 
> > > > > > then? Could you please ellaborate on why does not it need eager-split?
> > > > > 
> > > > > Read-only translations will continue to generate permission faults
> > > > > whereas writable-clean descriptors can be updated by hardware. You get
> > > > > the opportunity to split a block mapping lazily while preserving
> > > > > hardware dirty tracking for page mappings.
> > > > > 
> > > > 
> > > > So you suggest we only enable DBM bit after we split the block, that will 
> > > > happen only after a block is dirtied for the first time after dirty-log 
> > > > starts? 
> > > 
> > > If the VMM wants to do lazy splitting, yes. Otherwise this would happen
> > > as part of eager page splitting for block mappings that are known to be
> > > writable.
> > > 
> > > > > The approach I think we may need is:
> > > > > 
> > > > >  - Use a software bit in the PTE to stash whether or not a PFN is
> > > > >    'software-writable' when constructing the stage-2. By this I mean
> > > > >    we've already faulted it in for write from the primary MMU.
> > > > > 
> > > > >  - At the time of write protection, reap the hardware-writable state
> > > > >    from all PTEs but preserve the software-writable bit.
> > > > > 
> > > > >  - Whenever splitting a block mapping, set the DBM bit in the page-level
> > > > >    PTEs if the block was software-writable and HDBSS is present.
> > > > > 
> > > > > That way you'd have sufficient metadata in the PTE to safely set DBM.
> > > > 
> > > > I remember that, for some reason I can't recall, it would not be great to 
> > > > set DBM during dirty-log start, and instead we should have it since VM 
> > > > creation. Maybe it had to do with part of the pagetable using the old 
> > > > encoding (no DBM), and the other part using the new one.
> > > > 
> > > > IIRC, only blocks that are backed by writable memory (S1) were supposed to 
> > > > receive the DBM bit. We could use that info for deciding what to split, 
> > > > then.
> > > 
> > > That's why I'm saying to use a software bit. We can't leave DBM=1 on
> > > block mappings and set VTCR_EL2.HD=1 for obvious reasons.
> > >
> > 
> > I am failing to see the reason for above.

Ok, I think I got what I was missing, please let me know if I got it right:
- You said that we should still have lazy splitting with HDBSS-enabled 
  system, and for that
- We can't enable DBM=1 + VTCR_EL2.HD=1 at build S2 time

But for that lazy split scenario in which we only enable DBM=1 at first 
dirty of a block after dirty-track enable, we would have to change the 
encoding of the S2 pagetables from old format (ro/rw bit) to new format 
(DBM + clean/dirty bit) while the VM is running, which I remember to have 
some issues related to it. 

Well, let's consider we can handle that issues above, IIUC with your 
suggestion dirty-tracking with HDBSS would be something like:

If Lazy-splitting:
  - Constructing S2 : No change
  - Dirty-log enable
    - Enable HAFDBS + HDBSS (VTCR_EL2.{HD,HDBSS}={1,1})
  - On first dirtying each page:
    - Split block, if needed, into page granularity
      - (Set the Level3 entries as writable-clean? would make sense)
    - Set the dirty page as writable-dirty (DBM=1 + dirty_bit=1)
  - On guest_exit, write all HDBSS entries to dirty bitmap/ring
  - On dirty-clean/get request, use sw routine
    - (HACDBS needs DBM=1 to work)
  - Once the page has DBM=1, tracking happens by HDBSS

If eager-splitting:
  - Constructing S2
    - Set DBM on every writable block, put them in writable-clean
    - Enable VTCR_EL2.HD=1 (HAFDBS will update dirty-state in HW)
  - Dirty-log enable
    - Set VTCR_EL2.HDBSS=1 (any new dirtying gets logged to HDBSS)
    - Split all writable blocks (DBM=1)
  - On guest_exit, write all HDBSS entries to dirty bitmap/ring
  - On page dirtying
    - Tracking happens by hardware
  - On dirty-clean/get request: use HACDBS cleaning
    - (All writable entries have DBM=1 so we should be fine)


Does it match what you suggested?
If not, please help me understand what am I missing.

Thank you for your guidance!
Leo

> > 
> > IIUC, one option would be to:
> 
> (everything -> every block that have DBM=1) 
> 
> > 
> > What am I missing?
> > (above case is considering manual protect = off, but the same idea goes 
> > for the manual_protect_and_init_set, but cleaning is on-demand)
> >  
> 
> IIUC, what we get from your software write bit suggestion is to 
> avoid doing the splitting if we know the page will not be written to, which 
> makes sense.
> Using the DBM bit as suggested above should have the same effect.
> 
> > > > Another option would be to split when we are collecting a dirty-entry from 
> > > > HDBSS, but for live migration that would mean we have to transfer the whole 
> > > > block (possibly a large LEVEL1 block), because we have no idea which part 
> > > > of it got dirty.
> > > 
> > > We need to do dirty tracking at page granularity. The only scenario
> > > where we have writable block mappings while dirty logging is enabled on
> > > a memslot is if the VMM enabled KVM_DIRTY_LOG_INITIALLY_SET.
> > 
> > Agree
> > 
> > Thanks!
> > Leo
>