Re: [PATCH] xen/mcfg: Call PHYSDEVOP_pci_mmcfg_reserved before PCI enumeration

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Tue, Sep 22, 2015 at 07:57:29AM -0600, Jan Beulich wrote:
> >>> On 22.09.15 at 15:36, <konrad.wilk@oracle.com> wrote:
> > The best I could come up with is to do two loops:
> >  1) for 0:0:0 -> ff:ff:ff call PHYSDEVOP_pci_device_remove
> >     (so blow away what Xen has for its PCI devices.. except for the AMD 
> > IOMMU)
> >  2) list_for_each_pci_device PHYSDEVOP_pci_device_add (or other variants
> >     if VF).
> > 
> > But that is just a hack working around the Linux code.
> 
> The price of being non-intrusive on the Linux side. The above would
> seem okay to me for the (rare?) reassign case. (Not sure why you
> mean to exclude the AMD IOMMU there.)

I think we hide it altogether from the dom0 - so when it does the
bus reassignment it does not see the AMD IOMMU.

My concern was that the bus number of the AMD IOMMU device may
overlap with other bus numbers - which got moved as a result
of the bridge expanding its subordinate bus numbers.

In other words, the AMD IOMMU may have been at bus 0xb, the
bridge in question at 0x01, with an PCI device at 0x02;
some devices between 0x03 down to 0x0a.

The bridge would now cover 0x01->0x04 (with the PCI device
still at 0x02). But the devices at the old 0x03->0x0a have now
been shifted to 0x04->0x0b. And the AMD IOMMU is at 0x0c.

But Xen has no clue about this and "loses" the AMD IOMMU and
starts writting configuration data to whatever poor device used to
be at bus 0x0b.
> 
> > My thinking was that why don't we just make PHYSDEVOP_pci_device_add be
> > capable of dealing with changes like these.
> 
> That would require it to be able to match up devices at their new
> position (bus number) with its original instance (on the old bus).

Exactly!
> 
> > However, you  have also added the code to trap on PCI configuation access
> > we could also do some smarts when PCI_PRIMARY_BUS is modified (see
> > a88b72fddd046a0978242411276861039ec99ad0 x86/PCI: add config space abstract
> > write intercept logic). That would take care of it much easier I think?
> 
> That's a nice idea actually.

Thank you for adding that code.
> 
> Jan
>