Re: [PATCH v2 17/20] vfio/common: Support device dirty page tracking with vIOMMU

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 22 Feb 2023 22:08:33 -0400
Jason Gunthorpe <jgg@nvidia.com> wrote:

> On Wed, Feb 22, 2023 at 04:34:39PM -0700, Alex Williamson wrote:
> > > +    /*
> > > +     * With vIOMMU we try to track the entire IOVA space. As the IOVA space can
> > > +     * be rather big, devices might not be able to track it due to HW
> > > +     * limitations. In that case:
> > > +     * (1) Retry tracking a smaller part of the IOVA space.
> > > +     * (2) Retry tracking a range in the size of the physical memory.  
> > 
> > This looks really sketchy, why do we think there's a "good enough"
> > value here?  If we get it wrong, the device potentially has access to
> > IOVA space that we're not tracking, right?  
> 
> The idea was the untracked range becomes permanently dirty, so at
> worst this means the migration never converges.

I didn't spot the mechanics where that's implemented, I'll look again.
 
> #2 is the presumption that the guest is using an identity map.

This is a dangerous assumption.

> > I'd think the only viable fallback if the vIOMMU doesn't report its max
> > IOVA is the full 64-bit address space, otherwise it seems like we need
> > to add a migration blocker.  
> 
> This is basically saying vIOMMU doesn't work with migration, and we've
> heard that this isn't OK. There are cases where vIOMMU is on but the
> guest always uses identity maps. eg for virtual interrupt remapping.

Yes, the vIOMMU can be automatically added to a VM when we exceed 255
vCPUs, but I don't see how we can therefore deduce anything about the
usage mode of the vIOMMU.  Users also make use of vfio with vIOMMU for
nested assignment, ie. userspace drivers running within the guest,
where making assumptions about the IOVA extents of the userspace driver
seems dangerous.

Let's backup though, if a device doesn't support the full address width
of the platform, it's the responsibility of the device driver to
implement a DMA mask such that the device is never asked to DMA outside
of its address space support.  Therefore how could a device ever dirty
pages outside of its own limitations?

Isn't it reasonable to require that a device support dirty tracking for
the entire extent if its DMA address width in order to support this
feature?

If we can make those assumptions, then the vfio driver should happily
accept a range exceeding the device's DMA address width capabilities,
knowing that the device cannot dirty anything beyond its addressable
range.

> We also have future problems that nested translation is incompatible
> with device dirty tracking..

:-\  Thanks,

Alex