Re: iommufd dirty page logging overview

[Joao Martins <joao.m.martins@oracle.com>]

On 3/16/22 23:29, Thanos Makatos wrote:
> We're interested in adopting the new migration v2 interface and the new dirty
> page logging for /dev/iommufd in an out-of-process device emulation protocol
> [1]. Although it's purely userspace, we do want to stay close to the new API(s)
> being proposed for many reasons, mainly to re-use the QEMU implementation. The
> migration-related changes are relatively straightforward, I'm more interested
> in the dirty page logging. I've started reading the relevant email threads and
> my impression so far is that the details are still being decided? I don't see
> any commits related to dirty page logging in Yi's repo
> (https://github.com/luxis1999/iommufd) (at least not in the commit messages). I
> see that Joao has done some work using the existing dirty bitmaps
> (https://github.com/jpemartins/linux/commits/iommufd).

This branch here so far covers current vfio compatibility of iommufd (with an
AMD IOMMU implementation) solely because it was the easiest to start with given
the existing userspace (qemu). There's also a qemu counterpart with the emulated
AMD IOMMU implementation (should you lack the hardware). I'll be updating those branches
as things evolve i.e. once I have an initial version of the iommufd native API and more
IOMMUs support. Now, whether the vfio-compat part remains is TBD.

TBH much how we are discussing on the list -- and that Jason iterated yesterday --
I too don't expect a divergence on the API semantics from current VFIO system IOMMU
tracking. userspace-facing dirty reporting eventually gets a bitmap, with a bit
representing a <page-size>, the bitmap represents a range with a "base" iova and a <size>
(subset of a previously DMA-mapped range) that *matches* the size of the bitmap. And then
you start/stop tracking, read the dirty data and lastly DMA unmaps also fetch the dirty
data (if requested). The device-dirty tracking (via PCI) ought to model the target PF/VF
vendor interface but that is not iommufd.

The new things iommu-wise I expect are more about what the current API doesn't capture,
though, these are somewhat unrelated to the tracking control / reporting itself and more
about the IO page tables mappings i.e. changing domain's <page-size> in-place/dynamically
to increase the granularity of the dirty tracking.

[*] interestingly, arm64 SMMUv3.x seems to have an idea of 'stalling' transactions (not
sure if all kinds are supported) and letting CPU retry them as if the endpoint had just
requested ... without depending on endpoint PRI support.

> Is there a rough idea of
> how the new dirty page logging will look like? Is this already explained in the
> email threads an I missed it?
>
Granted that you came across the repo, I suppose you went through all the threads :)