Re: PCI Passthrough Design - Draft 3

[Ian Campbell <ian.campbell@citrix.com>]

On Wed, 2015-08-12 at 10:25 -0400, Konrad Rzeszutek Wilk wrote:
> On Wed, Aug 12, 2015 at 09:56:41AM +0100, Ian Campbell wrote:
> > On Tue, 2015-08-11 at 16:34 -0400, Konrad Rzeszutek Wilk wrote:
> > > 
> > > > 2.2    PHYSDEVOP_pci_host_bridge_add hypercall
> > > > ----------------------------------------------
> > > > Xen code accesses PCI configuration space based on the sbdf 
> > > > received from
> > > > the
> > > > guest. The order in which the pci device tree node appear may not 
> > > > be the
> > > > same
> > > > order of device enumeration in dom0. Thus there needs to be a 
> > > > mechanism to
> > > > bind
> > > > the segment number assigned by dom0 to the pci host controller. The
> > > > hypercall
> > > > is introduced:
> > > 
> > > Why can't we extend the existing hypercall to have the segment value?
> > > 
> > > Oh wait, PHYSDEVOP_manage_pci_add_ext does it already!
> > > 
> > > And have the hypercall (and Xen) be able to deal with introduction of 
> > > PCI
> > > devices that are out of sync?
> > > 
> > > Maybe I am confused but aren't PCI host controllers also 'uploaded' 
> > > to
> > > Xen?
> > 
> > The issue is that Dom0 and Xen need to agree on a common numbering 
> > space
> > for the "PCI domain" AKA "segment", which is really just a software 
> > concept
> > i.e. on ARM Linux just makes them up (on x86 I believe they come from 
> > some
> > firmware table so Xen and Dom0 "agree" to both use that).
> 
> Doesn't the PCI domain or segments have an notion of which PCI devices are
> underneath it? Or vice-verse - PCI devices know what their segment (or domain) is?

The PCI domain or segment does contain a device, but it is a purely OS
level concept, it has no real meaning in the hardware. So both Xen and
Linux are free to fabricate whatever segment naming space they want, but
obviously they need to agree, hence this hypercall lets Linux tell Xen what
segment it has associated with a given PCI controller.

Perhaps an example will help.

Imagine we have two PCI host bridges, one with CFG space at 0xA0000000 and
a second with CFG space at 0xB0000000.

Xen discovers these and assigns segment 0=0xA0000000 and segment
1=0xB0000000.

Dom0 discovers them too but assigns segment 1=0xA0000000 and segment
0=0xB0000000 (i.e. the other way).

Now Dom0 makes a hypercall referring to a device as (segment=1,BDF), i.e.
the device with BDF behind the root bridge at 0xA0000000. (Perhaps this is
the PHYSDEVOP_manage_pci_add_ext call).

But Xen thinks it is talking about the device with BDF behind the root
bridge at 0xB0000000 because Dom0 and Xen do not agree on what the segments
mean. Now Xen will use the wrong device ID in the IOMMU (since that is
associated with the host bridge), or poke the wrong configuration space, or
whatever.

Or maybe Xen chose 42=0xB0000000 and 43=0xA0000000 so when Dom0 starts
talking about segment=0 and =1 it has no idea what is going on.

PHYSDEVOP_pci_host_bridge_add is intended to allow Dom0 to say "Segment 0
is the host bridge at 0xB0000000" and "Segment 1 is the host bridge at
0xA0000000". With this there is no confusion between Xen and Dom0 because
Xen isn't picking a segment ID, it is being told what it is by Dom0 which
has done the picking.

Does that help?

Ian.