[PATCH 1/4] arm64: gicv3: its: Encode domain number in PCI stream id

[marc.zyngier@arm.com (Marc Zyngier)]

On Fri, 22 May 2015 23:57:40 +0100
"Chalamarla, Tirumalesh" <Tirumalesh.Chalamarla@caviumnetworks.com>
wrote:

Hi Tirumalesh,

> 
> > On May 22, 2015, at 1:26 AM, Marc Zyngier <marc.zyngier@arm.com> wrote:
> > 
> > On 20/05/15 13:48, Robert Richter wrote:
> >> Mark,
> >> 
> >> thanks for review, also of the other patches of this series.
> >> 
> >> See below
> >> 
> >> On 20.05.15 13:11:38, Marc Zyngier wrote:
> >>>> -	dev_alias->dev_id = alias;
> >>>> +	dev_alias->dev_id = (pci_domain_nr(pdev->bus) << 16) | alias;
> >> 
> >>> This feels very scary. We're now assuming that the domain number will
> >>> always be presented to the doorbell. What guarantee do we have that
> >>> this is always the case, irrespective of the platform?
> >>> 
> >>> Also, domains have no PCI reality, they are a Linux thing. And they can
> >>> be "randomly" assigned, unless you force the domain in DT with a
> >>> linux,pci-domain property. This looks even more wrong, specially
> >>> considering ACPI.
> >> 
> >> The main problem here is that device ids (32 bits) are system
> >> specific. Since we have more than one PCI root complex we need the
> >> upper 16 bits in the devid for mapping. Using pci_domain_nr for this
> >> fits our needs for now and shouldn't affect systems with a single RC
> >> only as the domain nr is zero then.
> >> 
> >> The domain number is incremented during initialization beginnig with
> >> zero and the order of it is fixed since it is taken from DT or ACPI
> >> tables. So we have full controll of it. I don't see issues here.
> > 
> > This may match what you have on ThunderX (as long as the kernel doesn't
> > adopt another behaviour when allocating the domain number). But other
> > platforms may have a completely different numbering, which will mess
> > them up entirely.
> > 
> >>> It really feels like we need a way to describe how the BDF numbering is
> >>> augmented. We also need to guarantee that we get the actual bridge
> >>> number, as opposed to the domain number.
> >> 
> >> But true, the obove is just intermediate. In the end we need some sort
> >> of handler that is setup during cpu initialization that registers a
> >> callback for the gic to determine the device id of that paricular
> >> system.
> > 
> > I don't really like the idea of a callback from the GIC - I'd prefer it
> > to be standalone, and rely on the topology information to build the
> > DeviceID. Mark Rutland had some ideas for DT (he posted an RFC a while
> > ago), maybe it would be good to get back to that and find out what we
> > can do. ACPI should also have similar information (IORT?).
> > 
> 
> How can some one pass this from DT, especially in GIC entry. i still
> think it is bus owner responsibility and call back is better idea.
> but if some one has a better idea for DT and ACPI, we are fine as
> long as it works on ThunderX.   

A callback would have to be bus-specific, and depends from the observer
of the access. There is strictly no guarantee that a single write from
the device is performed using the same ID to the IOMMU and to the MSI
doorbell. Actually, they are very likely to be different. A generic
callback would have to know about the point where this access is
observed, and expressing this is a nightmare.

Also, I'm really opposed to having platform-specific code that has for
sole purpose to describe the hardware. This is why we have DT (and to a
lesser extent ACPI). We've been there on 32bit, and learned our lesson.
It doesn't scale, it leads to a bunch of hacks in all corners, and I
don't feel like being on the receiving end of something like this.

I really suggest you look at Mark's suggestion:


http://lists.infradead.org/pipermail/linux-arm-kernel/2015-March/333199.html
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-May/341584.html

because so far, this is the only proposal that makes any sense to me in
the long run. Feel free to comment on it and help us making something
that also work for your favorite SoC.

Thanks,

        M.
-- 
Without deviation from the norm, progress is not possible.