From mboxrd@z Thu Jan 1 00:00:00 1970 From: Manish Jaggi Subject: Re: RFC: [PATCH 1/3] Enhance platform support for PCI Date: Tue, 24 Feb 2015 05:53:26 +0530 Message-ID: <54EBC47E.4040801@caviumnetworks.com> References: <54E71BDE.5020106@caviumnetworks.com> <54E7229C.7000301@linaro.org> <54E72452.3090801@caviumnetworks.com> <54E72688.9010005@linaro.org> <54E729F1.6000804@caviumnetworks.com> <54E73010.2050902@caviumnetworks.com> <1424439941.30924.243.camel@citrix.com> <54E74135.4040302@caviumnetworks.com> <1424443185.30924.268.camel@citrix.com> <54EB0813.20909@caviumnetworks.com> <54EB0B7D.6060909@linaro.org> <54EB1401.2050609@caviumnetworks.com> <54EB440C.9010806@linaro.org> <54EB5F86.40607@caviumnetworks.com> <54EB9E13.7060802@linaro.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <54EB9E13.7060802@linaro.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Julien Grall Cc: prasun.kapoor@cavium.com, Ian Campbell , "Kumar, Vijaya" , xen-devel@lists.xen.org, "Stefano Stabellini (Stefano.Stabellini@citrix.com)" , Jan Beulich List-Id: xen-devel@lists.xenproject.org On 24/02/15 3:09 am, Julien Grall wrote: > On 23/02/2015 17:12, Manish Jaggi wrote: >> >> On 23/02/15 8:45 pm, Julien Grall wrote: >>> On 23/02/15 11:50, Manish Jaggi wrote: >>>> On 23/02/15 4:44 pm, Julien Grall wrote: >>>>> >>>>> On 23/02/2015 10:59, Manish Jaggi wrote: >>>>>> On 20/02/15 8:09 pm, Ian Campbell wrote: >>>>>>> On Fri, 2015-02-20 at 19:44 +0530, Manish Jaggi wrote: >>>>>>>>> Another option might be a new hypercall (assuming one doesn't >>>>>>>>> already >>>>>>>>> exist) to register a PCI bus which would take e.g. the PCI CFG >>>>>>>>> base >>>>>>>>> address and return a new u16 segment id to be used for all >>>>>>>>> subsequent >>>>>>>>> PCI related calls. This would require the dom0 OS to hook its >>>>>>>>> pci_bus_add function, which might be doable (more doable than >>>>>>>>> handling >>>>>>>>> xen_segment_id DT properties I think). >>>>>>>> This seems ok, i will try it out. >>>>>>> I recommend you let this subthread (e.g. the conversation with Jan) >>>>>>> settle upon a preferred course of action before implementing any >>>>>>> one >>>>>>> suggestion. >>>>>> Ian we have also to consider for NUMA / multi node where there >>>>>> are two >>>>>> or more its nodes. >>>>>> pci0{ >>>>>> >>>>>> msi-parent = <&its0>; >>>>>> } >>>>>> >>>>>> pci1{ >>>>>> >>>>>> msi-parent = <&its1>; >>>>>> } >>>>>> >>>>>> This requires parsing pci nodes in xen and create a mapping between >>>>>> pci >>>>>> nodes and its. Xe would need to be aware of PCI nodes in device tree >>>>>> prior to dom0 sending a hypercall. Adding a property to pci node in >>>>>> device tree should be a good approach. >>>>> Why do you need it early? Wouldn't be sufficient to retrieve those >>>>> information when the hypercall pci_device_add is called? >>>>> >>>> The dom0/U device tree should have one 1 its node, xen should map to >>>> specific its when trapped. >>> The DOM0 device tree should expose the same layout as the hardware. By >>> exposing only one ITS you make your life more complicate. >> in what way? > > Because you have to parse all the device tree to remove the reference > to the second ITS. It's pointless and can be difficult to do it. > Could you please describe the case where it is difficult > If you are able to emulate on ITS, you can do it for multiple one. keeping it simple and similar across dom0/domUs Consider a case where a domU is assigned two PCI devices which are attached to different nodes. (Node is an entity having its own cores are host controllers). Currently for PCI pass-through xen has a virtual PCI bus in domU. In our implementation we attach a msi controller which is gic-v3-its to this bus in pci-front. Since there is a single bus you cannot attach 2 msi controllers. So xen would have to map the commands from virtual ITS to different physical ITS based on the deviceID / collection ID. > >>> PHYSDEVOP_pci_device_add should be called before any initialization is >>> done. Therefore ITS should be configured for this PCI after Xen is >>> aware >>> of the PCI. >> That is for a device, I believe all devices on a host bridge are >> serviced by a single ITS > > Why do you speak about host bridge? Do you need to configure the ITS > at boot time for the host bridge? > > Do you have any spec stating there is one ITS per host bridge? > I was referring to the mapping between ITS and PCI host bridge and not between a PCI end point / device which is added by the said hypercall. ITS is per node.It is always after but ITS for the PCI node is known to Xen before dom0 is started. >>> >>> IHMO, any ITS trap before this is wrong. >> AFAIK guest always sees a virtual ITS, could you please explain what is >> wrong in trapping. > > I never say the trapping is wrong in all case.... The "before" was > here for any trap before the PCI has been added to Xen is, IHMO, wrong. > There is no trap before. > Regards, >