Re: [Qemu-devel] Attaching PCI devices to the PCIe root complex

[Marcel Apfelbaum <marcel.apfelbaum@gmail.com>]

On Wed, 2013-09-25 at 10:01 +0300, Michael S. Tsirkin wrote:
> On Tue, Sep 24, 2013 at 06:01:02AM -0400, Laine Stump wrote:
> > When I added support for the Q35-based machinetypes to libvirt, I
> > specifically prohibited attaching any PCI devices (with the exception of
> > graphics controllers) to the PCIe root complex,
> 
> That's wrong I think. Anything attached to RC is an integrated
> endpoint, and these can be PCI devices.
I couldn't find on PCIe spec any mention that "Root Complex Integrated EndPoint"
must be PCIe. But, from spec 1.3.2.3:
- A Root Complex Integrated Endpoint must not require I/O resources claimed through BAR(s).
- A Root Complex Integrated Endpoint must not generate I/O Requests.
- A Root Complex Integrated Endpoint is required to support MSI or MSI-X or both if an
interrupt resource is requested.
I suppose that this restriction can be removed for PCI devices that
1. Actually work when plugged in into RC Integrated EndPoint
2. Respond to the above limitations

> 
> > and had planned to
> > prevent attaching them to PCIe root ports (ioh3420 device) and PCIe
> > downstream switch ports (xio-3130 device) as well. I did this because,
> > even though qemu currently allows attaching a normal PCI device in any
> > of these three places, the restriction exists for real hardware and I
> > didn't see any guarantee that qemu wouldn't add the restriction in the
> > future in order to more closely emulate real hardware.
> > 
> > However, since I did that, I've learned that many of the qemu "pci"
> > devices really should be considered as "pci or pcie". Gerd Hoffman lists
> > some of these cases in a bug he filed against libvirt:
> > 
> >    https://bugzilla.redhat.com/show_bug.cgi?id=1003983
> > 
> > I would like to loosen up the restrictions in libvirt, but want to make
> > sure that I don't allow something that could later be forbidden by qemu
> > (thus creating a compatibility problem during upgrades). Beyond Gerd's
> > specific requests to allow ehci, uhci, and hda controllers to attach to
> > PCIe ports, are there any other devices that I specifically should or
> > shouldn't allow? (I would rather be conservative in what I allow - it's
> > easy to allow more things later, but nearly impossible to revoke
> > permission once it's been allowed).
For the moment I would not remove any restrictions, but only the ones
requested and verified by somebody.

> 
> IMO, we really need to grow an interface to query this kind of thing.
Basically libvirt needs to know:
1. for (libvirt) controllers: what kind of devices can be plugged in
2. for devices (controller is also a device)
    - to which controllers can it be plugged in
    - does it support hot-plug?
3. implicit controllers of the machine types (q35 - "pcie-root", i440fx - "pci-root")
All the above must be exported to libvirt

Implementation options:
1. Add a compliance field on PCI/PCIe devices and controllers stating if it supports
   PCI/PCIe or both (and maybe hot-plug)
   - consider plug type + compliance to figure out whether a plug can go into a socket
   
2. Use Markus Armbruster idea of introducing a concept of "plug and sockets":
   - dividing the devices into adapters and plugs
   - adding sockets to bridges(buses?).
   In this way it would be clear which devices can connect to bridges

Any thoughts?
Thanks,
Marcel


 

>