Re: RFC: [PATCH 1/3] Enhance platform support for PCI

[Ian Campbell <ian.campbell@citrix.com>]

On Mon, 2015-02-23 at 14:45 +0000, Jan Beulich wrote:
> >>> On 23.02.15 at 15:33, <ian.campbell@citrix.com> wrote:
> > On Mon, 2015-02-23 at 14:07 +0000, Jan Beulich wrote:
> >> >>> On 23.02.15 at 13:45, <ian.campbell@citrix.com> wrote:
> >> > In which case might we be at liberty to specify that on ARM+Device Tree
> >> > systems (i.e. those where the f/w tables don't give an enumeration)
> >> > there is a 1:1 mapping from segments to host bridges?
> >> 
> >> This again can only be answered knowing how bus number
> >> assignment gets done on ARM (see also below). If all bus numbers
> >> are distinct, there's no need for using segments numbers other
> >> then zero. In the end, if you used segment numbers now, you may
> >> end up closing the path to using them for much larger future setups.
> >> But if that's not seen as a problem then yes, I think you could go
> >> that route.
> > 
> > Ultimately we just need to be able to go from the set of input
> > parameters to e.g. PHYSDEVOP_pci_device_add to the associate host
> > bridge.
> > 
> > It seems like the appropriate pair is (segment,bus), which uniquely
> > corresponds to a single host bridge (and many such pairs may do so).
> > 
> > So I think the original question just goes from having to determine a
> > way to map a segment to a host bridge to how to map a (segment,bus)
> > tuple to a host bridge.
> 
> Right. Avoiding (at this point in time) non-zero segments if at all
> possible.

I think it sounds like we are going to leave that up to the dom0 OS and
whatever it does gets registered with Xen. So non-zero segments is no
longer (directly) up to the Xen code. I think that's fine.

> >> >> What I don't get from this is where the BDF is being represented.
> >> > 
> >> > It isn't, since this is representing the host controller not any given
> >> > PCI devices which it contains.
> >> > 
> >> > I thought in general BDFs were probed (or even configured) by the
> >> > firmware and/or OS by walking over the CFG space and so aren't
> >> > necessarily described anywhere in the firmware tables.
> >> 
> >> They're effectively getting assigned by firmware, yes. This mainly
> >> affects bridges, which get stored (in their config space) the
> >> secondary and subordinate bus numbers (bounding the bus range
> >> they're responsible for when it comes to routing requests). If on
> >> ARM firmware doesn't assign bus numbers, is bridge setup then a
> >> job of the OS?
> > 
> > I'm not completely sure I think it depends on the particular firmware
> > (u-boot, EFI etc) but AIUI it can be the case that the OS does the
> > enumeration on at least some ARM platforms (quite how/when it knows to
> > do so I'm not sure).
> 
> In which case the Dom0 OS doing so would need to communicate
> its decisions to the hypervisor, as you suggest further down.

So more concretely something like:
        #define PHYSDEVOP_pci_host_bridge_add <XX>
        struct physdev_pci_host_bridge_add {
            /* IN */
            uint16_t seg;
            uint8_t bus;
            uint64_t address;
        };
        typedef struct physdev_pci_host_bridge_add physdev_pci_host_bridge_add_t;
        DEFINE_XEN_GUEST_HANDLE(physdev_pci_host_bridge_add_t);

Where seg+bus are enumerated/assigned by dom0 and address is some unique
property of the host bridge -- most likely its pci cfg space base
address (which is what physdev_pci_mmcfg_reserved also takes, I think?)

Do you think we would need start_bus + end_bus here? Xen could enumerate
this itself I think, and perhaps should even if dom0 tells us something?

> This
> basically replaces the bus scan (on segment 0) that Xen does on
> x86 (which topology information gets derived from).

Is the reason for the scan being of segment 0 only is that it is the one
which lives at the legacy PCI CFG addresses (or those magic I/O ports)? 

What about other host bridges in segment 0 which aren't at that address?

You could do the others based on MMCFG tables if you wanted, right?

Ian.