Re: iommufd dirty page logging overview

[Alex Williamson <alex.williamson@redhat.com>]

On Fri, 18 Mar 2022 09:41:08 -0300
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Fri, Mar 18, 2022 at 09:23:49AM +0000, Tian, Kevin wrote:
> > > From: Jason Gunthorpe <jgg@nvidia.com>
> > > Sent: Thursday, March 17, 2022 7:51 AM
> > >   
> > > > there a rough idea of how the new dirty page logging will look like?
> > > > Is this already explained in the email threads an I missed it?  
> > > 
> > > I'm hoping to get something to show in the next few weeks, but what
> > > I've talked about previously is to have two things:
> > > 
> > > 1) Control and reporting of dirty tracking via the system IOMMU
> > >    through the iommu_domain interface exposed by iommufd
> > > 
> > > 2) Control and reporting of dirty tracking via a VFIO migration
> > >    capable device's internal tracking through a VFIO_DEVICE_FEATURE
> > >    interface similar to the v2 migration interface
> > > 
> > > The two APIs would be semantically very similar but target different
> > > HW blocks. Userspace would be in charge to decide which dirty tracker
> > > to use and how to configure it.
> > >   
> > 
> > for the 2nd option I suppose userspace is expected to retrieve
> > dirty bits via VFIO_DEVICE_FEATURE before every iommufd 
> > unmap operation in precopy phase, just like why we need return
> > the dirty bitmap to userspace in iommufd unmap interface in
> > the 1st option. Correct?  
> 
> It would have to be after unmap, not before
> 
> > Is there any value of having iommufd pull dirty bitmap from
> > vfio driver then the userspace can just stick to a unified
> > iommufd interface for dirty pages no matter they are tracked
> > by system IOMMU or device IP? Sorry if this has been discussed
> > in previous threads which I haven't fully checked.  
> 
> It is something to discuss, this is sort of what the current vfio
> interface imagines

Yes.

> But to do it we need to build a whole bunch of infrastructure to
> register and control these things and add new ioctls to vfio to
> support this. I'm not sure we get a sufficient benifit to be
> worthwhile, infact it is probably a net loss as we loose the ability
> for userspace to pull the dirty bits from multiple device trackers in
> parallel with threading.

It seems like the new ioctls and such would be to configure the 2nd
option, the current assumption is the iommu collects all dirty bits.

There are advantages to each, the 2nd option gives the user more
visibility, more options to thread, but it also possibly duplicates
significant data.

The unmap scenario above is also not quite as cohesive if the user
needs to poll devices for dirty pages in the unmapped range after
performing the unmap.  It might make sense if the iommufd could
generate the merged bitmap on unmap as the threading optimization
probably has less value in that case.  Thanks,

Alex