Re: [Qemu-devel] [PATCH] pcie-root-port: let it has higher migrate priority

[Peter Xu <peterx@redhat.com>]

On Thu, Feb 01, 2018 at 07:51:31PM +0000, Dr. David Alan Gilbert wrote:
> * Peter Xu (peterx@redhat.com) wrote:
> > In the past, we prioritized IOMMU migration so that we have such a
> > priority order:
> > 
> >     IOMMU > PCI Devices
> > 
> > When migrating a guest with both vIOMMU and pcie-root-port, we'll always
> > migrate vIOMMU first, since pcie-root-port will be seen to have the same
> > priority of general PCI devices.
> > 
> > That's problematic.
> > 
> > The thing is that PCI bus number information is stored in the root port,
> > and that is needed by vIOMMU during post_load(), e.g., to figure out
> > context entry for a device.  If we don't have correct bus numbers for
> > devices, we won't be able to recover device state of the DMAR memory
> > regions, and things will be messed up.
> > 
> > So let's boost the PCIe root ports to be even with higher priority:
> > 
> >    PCIe Root Port > IOMMU > PCI Devices
> > 
> > A smoke test shows that this patch fixes bug 1538953.
> 
> Two questions (partially overlapping with what I replied to Michaels):
>   a) What happens with multiple IOMMUs?

If there are more IOMMUs, then the patch will let all the vIOMMUs be
migrated after pcie root ports.

But a more true answer is that: I don't really know. :)

Because I even don't know how multiple vIOMMUs will coop with each
other, especially nested.  In nested case, maybe there will be
dependency between vIOMMUs, but I'll avoid thinking about that until
we support more than one vIOMMUs.

>   b) What happens with multiple root ports?

Same answer as previous one: all of them will be migrated before any
vIOMMUs.

Note that IMHO we don't care which pcie root port is migrated first -
IMHO they should not depend on each other, but Marcel may correct me.

>   c) How correct is this ordering on different implementations 
>     (e.g. ARM/Power/etc)

Currently it won't affect since Intel IOMMU is the only user for
MIG_PRI_IOMMU.  After SMMU is merged it may affect (if it uses this
bit), but IMHO it's fine too as long as pcie root ports won't depend
on anything related to SMMU.

Thanks,

-- 
Peter Xu