RE: [PATCH 4/5] iommu/arm-smmu-v3: Add set_dirty_tracking() support

[Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>]

> -----Original Message-----
> From: Jason Gunthorpe <jgg@nvidia.com>
> Sent: Thursday, November 30, 2023 12:54 PM
> To: Shameerali Kolothum Thodi <shameerali.kolothum.thodi@huawei.com>
> Cc: iommu@lists.linux.dev; linux-arm-kernel@lists.infradead.org;
> robin.murphy@arm.com; will@kernel.org; joro@8bytes.org;
> kevin.tian@intel.com; nicolinc@nvidia.com; mshavit@google.com;
> eric.auger@redhat.com; joao.m.martins@oracle.com; jiangkunkun
> <jiangkunkun@huawei.com>; zhukeqian <zhukeqian1@huawei.com>;
> Linuxarm <linuxarm@huawei.com>
> Subject: Re: [PATCH 4/5] iommu/arm-smmu-v3: Add set_dirty_tracking()
> support
> 
> On Thu, Nov 30, 2023 at 08:56:32AM +0000, Shameerali Kolothum Thodi
> wrote:
> > > On Tue, Nov 28, 2023 at 09:49:39AM +0000, Shameer Kolothum wrote:
> > > > From: Joao Martins <joao.m.martins@oracle.com>
> > > >
> > > > Dirty tracking will always be enabled with DBM=1 modifier enabled
> > > > by default when HD is supported.
> > >
> > > Is this trying to say that ARM doesn't have a per-table global enable
> > > for dirty tracking but instead pre-sets the DBM bit to avoid the cost?
> > >
> >
> > Yes. SMMUv3 has per-PTE DBM control and I think the initial RFC had
> > it walking the PTEs and setting the DBM on set_dirty_tracking().
> 
> set_dirty_tracking doesn't have access to the necessary locking to
> touch the PTEs.
> 
> > > So on smmuv3 to enable we have to clear everything and disable
> > > continues to pay a penalty since we don't go and mark all things as
> > > dirty again?
> >
> > Yes we clear everything on enable. Sorry I didn't get the second part.
> > We don't mark dirty on disable. How is that different from Intel/AMD?
> 
> Intel/AMD have a global switch so they just turn off the tracking and
> stop paying the cost.
> 
> This approach on ARM means once the tracking is logically turned off
> the HW will continue to generate memory traffic to set dirty bits on
> DMAs. There is no way to back to the at-start state where their is 0
> memory traffic on DMAs. Not sure that it totally matters, but it is
> worth noting someplace.

Ah..you meant the HW penalty of keeping it on always.

> If we do want to solve this then ARM would need iommufd to make a pass
> over the page table to set for disable similar to how we have to clear
> for enable.

I will try to see if there is any overhead in keeping it on. Also we could limit
turning it on for DOMAIN_NESTED ?

Thanks,
Shameer