Re: RFC: [PATCH 1/3] Enhance platform support for PCI

[Ian Campbell <ian.campbell@citrix.com>]

On Thu, 2015-02-26 at 16:19 +0530, Vijay Kilari wrote:
> On Wed, Feb 25, 2015 at 3:50 PM, Ian Campbell <ian.campbell@citrix.com> wrote:
> > On Wed, 2015-02-25 at 08:03 +0530, Manish Jaggi wrote:
> >> On 24/02/15 7:13 pm, Julien Grall wrote:
> >> > On 24/02/15 00:23, Manish Jaggi wrote:
> >> >>> Because you have to parse all the device tree to remove the reference
> >> >>> to the second ITS. It's pointless and can be difficult to do it.
> >> >>>
> >> >> Could you please describe the case where it is difficult
> >> > You have to parse every node in the device tree and replace the
> >> > msi-parent properties with only one ITS.
> >> Thats the idea.
> >> >
> >> >>> If you are able to emulate on ITS, you can do it for multiple one.
> >> >> keeping it simple and similar across dom0/domUs
> >> >> Consider a case where a domU is assigned two PCI devices which are
> >> >> attached to different nodes. (Node is an entity having its own cores are
> >> >> host controllers).
> >> > The DOM0 view and guest view of the hardware are different.
> >> >
> >> > In the case of DOM0, we want to expose the same hardware layout as the
> >> > host. So if there is 2 ITS then we should expose the 2 ITS.
> >> AFAIK Xen has a microkernel design and timer/mmu/smmu/gic/its are
> >> handled by xen and a virtualized interface is provided to the guest. So
> >> if none of SMMU in the layout of host is presented to dom0 the same can
> >> be valid for multiple ITS.
> >
> > SMMU is one of the things which Xen hides from dom0, for obvious
> > reasons.
> >
> > Interrupts are exposed to dom0 in a 1:1 manner. AFAICT there is no
> > reason for ITS to differ here.
> >
> > Since dom0 needs to be able to cope with being able to see all of the
> > host host I/O devices (in the default no-passthrough case), it is
> > possible, if not likely, that it will need the same amount of ITS
> > resources (i.e. numbers of LPIs) as the host provides.
> >
> >> > In the case of the Guest, we (Xen) controls the memory layout.
> >> For Dom0 as well.
> >
> > Not true.
> >
> > For dom0 the memory layout is determined by the host memory layout. The
> > MMIO regions are mapped through 1:1 and the RAM is a subset of the RAM
> > regions of the host physical address space (often in 1:1, but with
> > sufficient h/w support this need not be the case).
> >
> >> > Therefore
> >> > we can expose only one ITS.
> >> If we follow 2 ITS in dom0 and 1 ITS in domU, how do u expect the Xen
> >> GIC ITS emulation driver to work.
> >> It should check that request came from a dom0 handle it differently. I
> >> think this would be *difficult*.
> >
> > I don't think so. If the vITS is written to handle multiple instances
> > (i.e. in a modular way, as it should be) then it shouldn't matter
> > whether any given domain has 1 or many vITS. The fact that dom0 may have
> > one or more and domU only (currently) has one then becomes largely
> > uninteresting.
> 
> I have few queries
> 
> 1) If Dom0 has 'n' ITS nodes, then how does Xen know which virtual ITS
> command Q is
>     mapped to which Physical ITS command Q.
>     In case of linux, the ITS node is added as msi chip to pci using
> of_pci_msi_chip_add()
>     and from pci_dev structure we can know which ITS to use.
> 
>     But in case of Xen, when ITS command is trapped we have only
> dev_id info from ITS command.

With the proper PCI infrastructure in place we can map the vdev_id to a
pdev_id, and from there to our own struct pci_dev 

The mapping from pdev_id to pci_dev is based on the
PHYSDEVOP_pci_host_bridge_add and PHYSDEVOP_pci_device_add calls I
described just now in my mail to Manish in this thread (specifically
pci_device_add creates and registers struct pci_dev I think).

> 
> 2) If DomU is always given one virtual ITS node. If DomU is assinged
> with two different
>     PCI devices connected to different physical ITS, then Xen vITS
> driver should know how to map
>     PCI device to physical ITS

Correct, I think that all falls out from the proper tracking of the
vdev_id to pdev_id and from vits to pits for a given domain and the
management/tracking of the struct pci_dev.

Ian.

> For the two issues above, Xen should have mapping to pci segment and
> physical ITS node to use
> which can be queried by vITS driver and send command on to correct physical ITS
> 
> Vijay