Re: [PATCH v4 3/6] KVM: arm64: Add auto DBM support for hardware dirty tracking

[Leonardo Bras <leo.bras@arm.com>]

On Mon, Aug 03, 2026 at 05:32:30PM +0100, Leonardo Bras wrote:
> On Mon, Aug 03, 2026 at 09:57:46PM +0800, Tian Zheng wrote:
> > 
> > 
> > On 8/3/2026 6:21 PM, Leonardo Bras wrote:
> > > On Mon, Aug 03, 2026 at 12:04:24PM +0800, Tian Zheng wrote:
> > > > 
> > > > 
> > > > On 8/3/2026 9:33 AM, Tian Zheng wrote:
> > > > > > > > > > > 09, 2026 at 06:40:23PM +0800, Tian Zheng wrote:
> > > > > > > > > > > > -    if (prot & KVM_PGTABLE_PROT_W)
> > > > > > > > > > > > +    if (prot & KVM_PGTABLE_PROT_W) {
> > > > > > > > > > > >              set |= KVM_PTE_LEAF_ATTR_LO_S2_S2AP_W;
> > > > > > > > > > > > 
> > > > > > > > > > > > +        /*
> > > > > > > > > > > > +         * No DEVICE filter needed here:
> > > > > > > > > > > > relax_perms is only called
> > > > > > > > > > > > +         * on FSC_PERM faults. Device pages
> > > > > > > > > > > > always get full RW from
> > > > > > > > > > > > +         * initial mapping and are never write-protected during
> > > > > > > > > > > > +         * migration, so they never trigger a permission fault.
> > > > > > > > > > > > +         */
> > > > > > > > > > > > +        if (pgt->flags & KVM_PGTABLE_S2_DBM)
> > > > > > > > > > > > +            set |= KVM_PTE_LEAF_ATTR_HI_S2_DBM;
> > > > > > > > > > > > +    } else {
> > > > > > > > > > > > +        /*
> > > > > > > > > > > > +         * Clear DBM on W→RO downgrade to prevent hardware from
> > > > > > > > > > > > +         * silently upgrading RO+DBM back to W+dirty, which would
> > > > > > > > > > > > +         * bypass KVM's write tracking and cause data corruption.
> > > > > > > > > > > > +         */
> > > > > > > > > > > > +        clr |= KVM_PTE_LEAF_ATTR_HI_S2_DBM;
> > > > > > > > > > > > +    }
> > > > > > > > > > > > +
> > > > > > > > > > > This block makes it pretty evident that the DBM bit really *is* the
> > > > > > > > > > > write permission bit. I'd much rather we
> > > > > > > > > > > introduce the concept of dirty
> > > > > > > > > > > state to the page table library and migrate the abstract write
> > > > > > > > > > > permission to the DBM field, even if we don't have FEAT_HAFDBS.
> > > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > Ohh, that's an amazing idea!
> > > > > > > > 
> > > > > > > > Thinking about that again...
> > > > > > > > If we adopt the encoding with DBM being the write-permission
> > > > > > > > bit, and all
> > > > > > > > PTEs have it since the start, how can we have lazy-splitting happening?
> > > > > > > > 
> > > > > > > > Only way I think of is removing both DBM and S2_S2AP_W bit
> > > > > > > > from writable
> > > > > > > > PTEs during dirty-track enable, and re-adding them during
> > > > > > > > the first write
> > > > > > > > fault. If we don't remove the DBM bit, systems with HDBSS
> > > > > > > > would just dirty
> > > > > > > > it by hardware, without causing a fault.
> > > > > > > > 
> > > > > > > > DBM=0 would need to happen only in the first write-protect (only on
> > > > > > > > lazy-splitting). All other write-protecting would just clean
> > > > > > > > the S2_S2AP_W
> > > > > > > > bit, as everything is already split.
> > > > > > > > 
> > > > > > > > Is that what was intended?
> > > > > > > > 
> > > > > > > > Thanks!
> > > > > > > > Leo
> > > > > > > > 
> > > > > > > Hi Leo,
> > > > > > > 
> > > > > > > I think the cleanest way to handle this is to simply avoid setting DBM
> > > > > > > on block mappings. If we only set DBM on page-level PTEs, then block
> > > > > > > mappings will naturally stay DBM=0 and trigger a write fault on first
> > > > > > > access — exactly what we need for lazy splitting.
> > > > > > > 
> > > > > > > When the fault occurs, the block gets split into page-level PTEs, and at
> > > > > > > that point we can set DBM=1 on the resulting leaf entries. This way:
> > > > > > > 
> > > > > > > 1. Lazy split works naturally (fault -> split -> set DBM=1)
> > > > > > > 
> > > > > > > 2. No need to clear DBM globally at dirty-track enable
> > > > > > > 
> > > > > > > 3. No special handling for block mappings
> > > > > > > 
> > > > > > > So I think global DBM is still viable — we just need to filter out block
> > > > > > > mappings when setting the DBM bit. That way the lazy split path
> > > > > > > is preserved
> > > > > > > without extra complexity.
> > > > > > 
> > > > > > Hi Tian,
> > > > > > 
> > > > > > Humm, but would not that be contrary to what Oliver suggested:
> > > > > > changing the
> > > > > > encoding from the PTE for all entries?
> > > > > > 
> > > > > > (Like, if the PTE is writable, it has to have DBM set)
> > > > > > 
> > > > > > IIUC what you said, on first faulting of the page in the VM:
> > > > > > - If the entry is a page (level-3 leaf) and writable, add DBM
> > > > > > - If it's a block entry (leaf but not a level-3), don't add DBM
> > > > > > 
> > > > > > So after we enable dirty-logging:
> > > > > > - a level-3 entry would not fault, using HDBSS, and
> > > > > > - a block entry would fault, do the splitting, and add DBM to level-3
> > > > > >     entries during the split.
> > > > > > 
> > > > > > If I got that correct, that would be clean indeed.
> > > > > > 
> > > > > > But then we would have a different encoding for block entries and page
> > > > > > entries. In page entries, DBM could be used to say if the page is
> > > > > > writable,
> > > > > > but on block entries one would have to look at the 'dirty-bit'.
> > > > > > 
> > > > > > Would that be ok?
> > > > > > 
> > > > > > Thanks!
> > > > > > Leo
> > > > > > 
> > > > > Hi Leo,
> > > > > 
> > > > > My initial concern was that clearing all DBM bits at the start of
> > > > > migration would be too expensive, so I thought distinguishing between
> > > > > level-3 entries and block entries would be better.
> > > > > 
> > > 
> > > I think we expect it to be expensive, but since we already clean the
> > > dirty-bit (ro/rw) bit, we can have both happening in the same write :)
> > > 
> > > (since we only mark the DBM bit when we fault the memory on lazy-splitting,
> > > we are expecting to have the same amount of writes to pagetable as we have
> > > before HDBSS, both on faulting and 1st iteration cleaning)
> > > 
> > Hi, Leo
> > 
> > Actually, I have thought about this approach too, but if we clear DBM in
> > kvm_pgtable_stage2_wrprotect(), then during the first round of
> > migration, we will fault and release RO -> W, and then add DBM.
> 
> Yeah, that's only for lazy-splitting, though.
> 
> >
> > But next time, when we migrate the dirty pages in round two, we will run
> > kvm_pgtable_stage2_wrprotect() again, which will clear DBM again. And
> > finally, HDBSS will be useless during migration.
> 
> Right, on lazy splitting, we have to clean the DBM bit on the  
> write-protect only if it's a block entry (hugepage). 
> 
> Once it faults for the first time, it will lazy-split, and we don't need to 
> clean the DBM bit.
> 
> > 
> > > 
> > > > > However, I ran a quick test on a 400GB VM (4 vCPUs), and the overhead
> > > > > turned out to be around 30ns — which I think is acceptable.
> > > > 
> > > > Just a quick correction — I misstated the unit in my previous email. The
> > > > overhead for clearing DBM on the 400GB VM (4 vCPUs) was around 32 µs, not 30
> > > > ns.
> > > > 
> > > 
> > > Oh, that seems more likely :)
> > > 
> > > Question: is tha above amount of memory initially in Level-1 blocks,
> > > level-2 blocks or level-3 pages? (aka: were you using explicit/transparent
> > > hugepages?)
> > > 
> > 
> > I'm using transparent hugepages. However, if we were to use level-3 stage-2
> > pages with -mem-prealloc enabled in QEMU, I believe the time cost would be
> > extremely high — potentially out of our control.
> > 
> 
> Yeah, that's the issue.
> For this not to explode like this, we need to mark as RO only when the 
> entries are blocks AND we are doing lazy splitting.
> 
> We have:
> Mode	DBM	Dirty bit
> RO	0	X
> WC	1	0
> WD	1	1
> 
> On write-protect:
> - Lazy splitting + block entry (hugepage, level 2-) -> RO
> - Otherwise					    -> WC
> 
> On first fault, the block entry will be lazy-splitten, and we can set DBM=1 
> in every new page.

Meaning we set WD on the page that got actually written to, and register it 
in the dirty log (bitmap or ring), and the remaining pages are set to WC.

 
> That way we guarantee that we are not faulting level-3 pages unecessarily, 
> nor need to go through the whole tree setting DBM=1 or DBM=0 on level-3 
> pages.
> 
> How does that sound?
> 
> Thanks!
> Leo