Re: RFC: [PATCH 1/3] Enhance platform support for PCI

[Manish Jaggi <mjaggi@caviumnetworks.com>]

On 24/02/15 3:09 am, Julien Grall wrote:
> On 23/02/2015 17:12, Manish Jaggi wrote:
>>
>> On 23/02/15 8:45 pm, Julien Grall wrote:
>>> On 23/02/15 11:50, Manish Jaggi wrote:
>>>> On 23/02/15 4:44 pm, Julien Grall wrote:
>>>>>
>>>>> On 23/02/2015 10:59, Manish Jaggi wrote:
>>>>>> On 20/02/15 8:09 pm, Ian Campbell wrote:
>>>>>>> On Fri, 2015-02-20 at 19:44 +0530, Manish Jaggi wrote:
>>>>>>>>> Another option might be a new hypercall (assuming one doesn't
>>>>>>>>> already
>>>>>>>>> exist) to register a PCI bus which would take e.g. the PCI CFG 
>>>>>>>>> base
>>>>>>>>> address and return a new u16 segment id to be used for all
>>>>>>>>> subsequent
>>>>>>>>> PCI related calls. This would require the dom0 OS to hook its
>>>>>>>>> pci_bus_add function, which might be doable (more doable than
>>>>>>>>> handling
>>>>>>>>> xen_segment_id DT properties I think).
>>>>>>>> This seems ok, i will try it out.
>>>>>>> I recommend you let this subthread (e.g. the conversation with Jan)
>>>>>>> settle upon a preferred course of action before implementing any 
>>>>>>> one
>>>>>>> suggestion.
>>>>>> Ian we have also to consider for NUMA / multi node where there 
>>>>>> are two
>>>>>> or more its nodes.
>>>>>> pci0{
>>>>>>
>>>>>> msi-parent = <&its0>;
>>>>>> }
>>>>>>
>>>>>> pci1{
>>>>>>
>>>>>> msi-parent = <&its1>;
>>>>>> }
>>>>>>
>>>>>> This requires parsing pci nodes in xen and create a mapping between
>>>>>> pci
>>>>>> nodes and its. Xe would need to be aware of PCI nodes in device tree
>>>>>> prior to dom0 sending a hypercall. Adding a property to pci node in
>>>>>> device tree should be a good approach.
>>>>> Why do you need it early? Wouldn't be sufficient to retrieve those
>>>>> information when the hypercall pci_device_add is called?
>>>>>
>>>> The dom0/U device tree should have one 1 its node, xen should map to
>>>> specific its when trapped.
>>> The DOM0 device tree should expose the same layout as the hardware. By
>>> exposing only one ITS you make your life more complicate.
>> in what way?
>
> Because you have to parse all the device tree to remove the reference 
> to the second ITS. It's pointless and can be difficult to do it.
>
Could you please describe the case where it is difficult
> If you are able to emulate on ITS, you can do it for multiple one.
keeping it simple and similar across dom0/domUs

Consider a case where a domU is assigned two PCI devices which are 
attached to different nodes. (Node is an entity having its own cores are 
host controllers).

Currently for PCI pass-through xen has a virtual PCI bus in domU. In our 
implementation we attach a msi controller which is gic-v3-its to this 
bus in pci-front. Since there is a single bus you cannot attach 2 msi 
controllers. So xen would have to map the commands from virtual ITS to 
different physical ITS based on the deviceID / collection ID.
>
>>> PHYSDEVOP_pci_device_add should be called before any initialization is
>>> done. Therefore ITS should be configured for this PCI after Xen is 
>>> aware
>>> of the PCI.
>> That is for a device, I believe all devices on a host bridge are
>> serviced by a single ITS
>
> Why do you speak about host bridge? Do you need to configure the ITS 
> at boot time for the host bridge?
>
> Do you have any spec stating there is one ITS per host bridge?
>
I was referring to the mapping between ITS and PCI host bridge and not 
between a PCI end point / device which is added by the said hypercall. 
ITS is per node.It is always after but ITS for the PCI node is known to 
Xen before dom0 is started.
>>>
>>> IHMO, any ITS trap before this is wrong.
>> AFAIK guest always sees a virtual ITS, could you please explain what is
>> wrong in trapping.
>
> I never say the trapping is wrong in all case.... The "before" was 
> here for any trap before the PCI has been added to Xen is, IHMO, wrong.
>
There is no trap before.
> Regards,
>