Re: RFC: [PATCH 1/3] Enhance platform support for PCI

[Ian Campbell <ian.campbell@citrix.com>]

On Mon, 2015-02-23 at 14:07 +0000, Jan Beulich wrote:
> >>> On 23.02.15 at 13:45, <ian.campbell@citrix.com> wrote:
> > On Mon, 2015-02-23 at 08:43 +0000, Jan Beulich wrote:
> >> >>> On 20.02.15 at 18:33, <ian.campbell@citrix.com> wrote:
> >> > On Fri, 2015-02-20 at 15:15 +0000, Jan Beulich wrote:
> >> >> > That's the issue we are trying to resolve, with device tree there is no
> >> >> > explicit segment ID, so we have an essentially unindexed set of PCI
> >> >> > buses in both Xen and dom0.
> >> >> 
> >> >> How that? What if two bus numbers are equal? There ought to be
> >> >> some kind of topology information. Or if all buses are distinct, then
> >> >> you don't need a segment number.
> >> > 
> >> > It's very possible that I simply don't have the PCI terminology straight
> >> > in my head, leading to me talking nonsense.
> >> > 
> >> > I'll explain how I'm using it and perhaps you can put me straight...
> >> > 
> >> > My understanding was that a PCI segment equates to a PCI host
> >> > controller, i.e. a specific instance of some PCI host IP on an SoC.
> >> 
> >> No - there can be multiple roots (i.e. host bridges)
> > 
> > Where a "host bridge" == what I've been calling "PCI host controller"?
> 
> Some problems here may originate in this naming: I'm not aware of
> anything named "host controller" in PCI. The root of a PCI hierarchy
> (single or multiple buses) connects to the system bus via a host
> bridge. Whether one or more of them sit in a single chip is of no
> interest (on the x86 and ia64 sides at least).

Yes, I think I've just been using the terminology wrongly, I mean "host
bridge" throughout.

There is generally one such bridge per "controller" (i.e. IP block) whic
his what I was trying to get at in the next paragraph. Lets just talk
about host bridges from now on to avoid confusion.

> > I suppose in principal a single controller might expose multiple host
> > bridges, but I think we can logically treat such things as being
> > multiple controllers (e.g. with multiple CFG spaces etc).
> 
> Perhaps.
> 
> > IOW is the mapping from segment->host bridge many->one or
> > many->many?
> 
> Each segment may have multiple host bridges, each host bridge
> connect devices on multiple buses. Any such hierarchy is entirely
> separate from any other such hierarchy (both physically and in
> terms of the SBDFs used to identify them).
> 
> > Maybe what I should read into what you are saying is that segments are
> > purely a software and/or firmware concept with no real basis in the
> > hardware?
> 
> Right - they just represent separation in hardware, but they have
> no real equivalent there.

I think I now understand.

> > In which case might we be at liberty to specify that on ARM+Device Tree
> > systems (i.e. those where the f/w tables don't give an enumeration)
> > there is a 1:1 mapping from segments to host bridges?
> 
> This again can only be answered knowing how bus number
> assignment gets done on ARM (see also below). If all bus numbers
> are distinct, there's no need for using segments numbers other
> then zero. In the end, if you used segment numbers now, you may
> end up closing the path to using them for much larger future setups.
> But if that's not seen as a problem then yes, I think you could go
> that route.

Ultimately we just need to be able to go from the set of input
parameters to e.g. PHYSDEVOP_pci_device_add to the associate host
bridge.

It seems like the appropriate pair is (segment,bus), which uniquely
corresponds to a single host bridge (and many such pairs may do so).

So I think the original question just goes from having to determine a
way to map a segment to a host bridge to how to map a (segment,bus)
tuple to a host bridge.

> >> What I don't get from this is where the BDF is being represented.
> > 
> > It isn't, since this is representing the host controller not any given
> > PCI devices which it contains.
> > 
> > I thought in general BDFs were probed (or even configured) by the
> > firmware and/or OS by walking over the CFG space and so aren't
> > necessarily described anywhere in the firmware tables.
> 
> They're effectively getting assigned by firmware, yes. This mainly
> affects bridges, which get stored (in their config space) the
> secondary and subordinate bus numbers (bounding the bus range
> they're responsible for when it comes to routing requests). If on
> ARM firmware doesn't assign bus numbers, is bridge setup then a
> job of the OS?

I'm not completely sure I think it depends on the particular firmware
(u-boot, EFI etc) but AIUI it can be the case that the OS does the
enumeration on at least some ARM platforms (quite how/when it knows to
do so I'm not sure).

> > FWIW the first 4 bytes in each line of interrupt-map are actually
> > somehow matched against the masked (via interrupt-map-mask) against an
> > encoding of the BDF to give the INTx routing, but BDFs aren't
> > represented in the sense I think you meant in the example above.
> > 
> > There is a capability to have child nodes of this root controller node
> > which describe individual devices, and there is an encoding for the BDF
> > in there, but these are not required. For reference I've pasted a DT
> > snippet from a Nvidia Jetson TK1 (Tegra124) system which has child
> > nodes. I think the BDF is encoded in assigned-addresses somewhere.
> 
> Even if the BDF can be derived out of the addresses there it still
> doesn't make clear to me how a more complex topology (namely
> including bridges) would be represented. As said above - a bridge
> needs to know which buses to route requests for, and hence I
> can't see how you can get away without using bus numbers at all
> in this process; all I can see is that the DT description can get
> away without it, by simply representing the hierarchies that on
> x86 one would discover by scanning the config space for devices.
> I.e. - as indicated above - if bus number assignment is the OSes
> job, then you would want to avoid segments as long as you can
> get away with the 256 buses you have.

That makes sense, although we are then still left with how to go from
(segment,bus) -> host bridge, even if segment is effectively always 0
for now.

I think we will have to add an interface for dom0 to register new host
bridges with Xen such that Xen can then match against future hypercalls
referring to devices.

Ian.