Re: [Qemu-devel] Attaching PCI devices to the PCIe root complex

["Michael S. Tsirkin" <mst@redhat.com>]

On Wed, Sep 25, 2013 at 05:39:37AM -0400, Laine Stump wrote:
> On 09/25/2013 04:48 AM, Marcel Apfelbaum wrote:
> > On Wed, 2013-09-25 at 10:01 +0300, Michael S. Tsirkin wrote:
> >> On Tue, Sep 24, 2013 at 06:01:02AM -0400, Laine Stump wrote:
> >>> When I added support for the Q35-based machinetypes to libvirt, I
> >>> specifically prohibited attaching any PCI devices (with the exception of
> >>> graphics controllers) to the PCIe root complex,
> >> That's wrong I think. Anything attached to RC is an integrated
> >> endpoint, and these can be PCI devices.
> > I couldn't find on PCIe spec any mention that "Root Complex Integrated EndPoint"
> > must be PCIe. But, from spec 1.3.2.3:
> > - A Root Complex Integrated Endpoint must not require I/O resources claimed through BAR(s).
> > - A Root Complex Integrated Endpoint must not generate I/O Requests.
> > - A Root Complex Integrated Endpoint is required to support MSI or MSI-X or both if an
> > interrupt resource is requested.
> 
> So much easier in the real world, where the rule is "if it fits in the
> socket and the pins match up, then it's okay" :-)
> 

Well real world hardware has even more limitations,
like different PCIe form-factors.

Also I'm not aware of such hardware for PCI/PCIe specifically,
but hardware with support for multiple buses
does exist, e.g. I have a disk with both USB and eSATA
interfaces.
Also, it could be same chip with different interfaces on top.
At some point intel made hardware that looked almost exactly
like a PCI-X part except it had PCI-E interface.

We can stick all this info into hardware type but it's
a really ugly way to do this.

See for example virtio-net-pci which users still can't
wrap their heads around.
They really want to say use virtio net device.


> >> IMO, we really need to grow an interface to query this kind of thing.
> > Basically libvirt needs to know:
> > 1. for (libvirt) controllers: what kind of devices can be plugged in
> 
> The controllers also need a flag indicating if they supporting having
> devices hot-plugged into them. For example, as far as I understand, the
> PCI root complex ("pcie-root" in libvirt) doesn't support hot-plugging
> devices,

Not exactly. It doesn't support native hotplug.

> nor does i82801b11-bridge ("dmi-to-pci-bridge" in libvirt), but
> pci-bridge, ioh3420 ("root-port" in libvirt), and xio3130-downstream
> ("downstream-switch-port" in libvirt) *do* support hot-plugging devices
> (as far as I know, none of these controllers can themselves be
> hot-plugged into another controller, but that could change in the
> future, e.g. I recall someone saying something about  hot-plug of a
> pci-bridge being a future possibility)
> 
> 
> > 2. for devices (controller is also a device)
> >     - to which controllers can it be plugged in
> >     - does it support hot-plug?
> > 3. implicit controllers of the machine types (q35 - "pcie-root", i440fx - "pci-root")
> > All the above must be exported to libvirt
> >
> > Implementation options:
> > 1. Add a compliance field on PCI/PCIe devices and controllers stating if it supports
> >    PCI/PCIe or both (and maybe hot-plug)
> >    - consider plug type + compliance to figure out whether a plug can go into a socket
> >    
> > 2. Use Markus Armbruster idea of introducing a concept of "plug and sockets":
> >    - dividing the devices into adapters and plugs
> >    - adding sockets to bridges(buses?).
> >    In this way it would be clear which devices can connect to bridges
> 
> However it is done, each controller will need to have two sets of flags
> - one for what it can plug into, and one for what can be plugged into it.

Not just into it - what can be plugged into each slot.