Re: PCI Passthrough Design - Draft 3

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Aug 12, 2015 at 03:42:10PM +0100, Ian Campbell wrote:
> On Wed, 2015-08-12 at 10:25 -0400, Konrad Rzeszutek Wilk wrote:
> > On Wed, Aug 12, 2015 at 09:56:41AM +0100, Ian Campbell wrote:
> > > On Tue, 2015-08-11 at 16:34 -0400, Konrad Rzeszutek Wilk wrote:
> > > > 
> > > > > 2.2    PHYSDEVOP_pci_host_bridge_add hypercall
> > > > > ----------------------------------------------
> > > > > Xen code accesses PCI configuration space based on the sbdf 
> > > > > received from
> > > > > the
> > > > > guest. The order in which the pci device tree node appear may not 
> > > > > be the
> > > > > same
> > > > > order of device enumeration in dom0. Thus there needs to be a 
> > > > > mechanism to
> > > > > bind
> > > > > the segment number assigned by dom0 to the pci host controller. The
> > > > > hypercall
> > > > > is introduced:
> > > > 
> > > > Why can't we extend the existing hypercall to have the segment value?
> > > > 
> > > > Oh wait, PHYSDEVOP_manage_pci_add_ext does it already!
> > > > 
> > > > And have the hypercall (and Xen) be able to deal with introduction of 
> > > > PCI
> > > > devices that are out of sync?
> > > > 
> > > > Maybe I am confused but aren't PCI host controllers also 'uploaded' 
> > > > to
> > > > Xen?
> > > 
> > > The issue is that Dom0 and Xen need to agree on a common numbering 
> > > space
> > > for the "PCI domain" AKA "segment", which is really just a software 
> > > concept
> > > i.e. on ARM Linux just makes them up (on x86 I believe they come from 
> > > some
> > > firmware table so Xen and Dom0 "agree" to both use that).
> > 
> > Doesn't the PCI domain or segments have an notion of which PCI devices are
> > underneath it? Or vice-verse - PCI devices know what their segment (or domain) is?
> 
> The PCI domain or segment does contain a device, but it is a purely OS
> level concept, it has no real meaning in the hardware. So both Xen and
> Linux are free to fabricate whatever segment naming space they want, but
> obviously they need to agree, hence this hypercall lets Linux tell Xen what
> segment it has associated with a given PCI controller.
> 
> Perhaps an example will help.
> 
> Imagine we have two PCI host bridges, one with CFG space at 0xA0000000 and
> a second with CFG space at 0xB0000000.
> 
> Xen discovers these and assigns segment 0=0xA0000000 and segment
> 1=0xB0000000.
> 
> Dom0 discovers them too but assigns segment 1=0xA0000000 and segment
> 0=0xB0000000 (i.e. the other way).
> 
> Now Dom0 makes a hypercall referring to a device as (segment=1,BDF), i.e.
> the device with BDF behind the root bridge at 0xA0000000. (Perhaps this is
> the PHYSDEVOP_manage_pci_add_ext call).
> 
> But Xen thinks it is talking about the device with BDF behind the root
> bridge at 0xB0000000 because Dom0 and Xen do not agree on what the segments
> mean. Now Xen will use the wrong device ID in the IOMMU (since that is
> associated with the host bridge), or poke the wrong configuration space, or
> whatever.
> 
> Or maybe Xen chose 42=0xB0000000 and 43=0xA0000000 so when Dom0 starts
> talking about segment=0 and =1 it has no idea what is going on.
> 
> PHYSDEVOP_pci_host_bridge_add is intended to allow Dom0 to say "Segment 0
> is the host bridge at 0xB0000000" and "Segment 1 is the host bridge at
> 0xA0000000". With this there is no confusion between Xen and Dom0 because
> Xen isn't picking a segment ID, it is being told what it is by Dom0 which
> has done the picking.
> 
> Does that help?

Yes thank you!

Manish, please include this explanation in the design as it will surely
help other folks in understanding it.
> 
> Ian.
>