Re: [Qemu-devel] Attaching PCI devices to the PCIe root complex

["Michael S. Tsirkin" <mst@redhat.com>]

On Wed, Sep 25, 2013 at 11:48:28AM +0300, Marcel Apfelbaum wrote:
> On Wed, 2013-09-25 at 10:01 +0300, Michael S. Tsirkin wrote:
> > On Tue, Sep 24, 2013 at 06:01:02AM -0400, Laine Stump wrote:
> > > When I added support for the Q35-based machinetypes to libvirt, I
> > > specifically prohibited attaching any PCI devices (with the exception of
> > > graphics controllers) to the PCIe root complex,
> > 
> > That's wrong I think. Anything attached to RC is an integrated
> > endpoint, and these can be PCI devices.
> I couldn't find on PCIe spec any mention that "Root Complex Integrated EndPoint"
> must be PCIe. But, from spec 1.3.2.3:
> - A Root Complex Integrated Endpoint must not require I/O resources claimed through BAR(s).
> - A Root Complex Integrated Endpoint must not generate I/O Requests.
> - A Root Complex Integrated Endpoint is required to support MSI or MSI-X or both if an
> interrupt resource is requested.

Heh PCI-SIG keeps fighting against legacy interrupts and IO.
But lots of hardware happily ignores these rules.
And the reason is simple: software does not enforce them.
Here's integrated stuff on my laptop:

00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core
Processor Family Integrated Graphics Controller (rev 09) (prog-if 00
[VGA controller])
        Subsystem: Lenovo Device 21cf
        Flags: bus master, fast devsel, latency 0, IRQ 43
        Memory at f0000000 (64-bit, non-prefetchable) [size=4M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        I/O ports at 5000 [size=64]
        Expansion ROM at <unassigned> [disabled]
        Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [d0] Power Management version 2
        Capabilities: [a4] PCI Advanced Features
        Kernel driver in use: i915

So it has an IO BAR.


00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset
Family USB Enhanced Host Controller #2 (rev 04) (prog-if 20 [EHCI])
        Subsystem: Lenovo Device 21cf
        Flags: bus master, medium devsel, latency 0, IRQ 16
        Memory at f252a000 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [50] Power Management version 2
        Capabilities: [58] Debug port: BAR=1 offset=00a0
        Capabilities: [98] PCI Advanced Features
        Kernel driver in use: ehci-pci

So IRQ but no MSI.


> I suppose that this restriction can be removed for PCI devices that
> 1. Actually work when plugged in into RC Integrated EndPoint
> 2. Respond to the above limitations

These limitations are just guidance for future devices.
They can't change the past, such devices were made.

> > 
> > > and had planned to
> > > prevent attaching them to PCIe root ports (ioh3420 device) and PCIe
> > > downstream switch ports (xio-3130 device) as well. I did this because,
> > > even though qemu currently allows attaching a normal PCI device in any
> > > of these three places, the restriction exists for real hardware and I
> > > didn't see any guarantee that qemu wouldn't add the restriction in the
> > > future in order to more closely emulate real hardware.
> > > 
> > > However, since I did that, I've learned that many of the qemu "pci"
> > > devices really should be considered as "pci or pcie". Gerd Hoffman lists
> > > some of these cases in a bug he filed against libvirt:
> > > 
> > >    https://bugzilla.redhat.com/show_bug.cgi?id=1003983
> > > 
> > > I would like to loosen up the restrictions in libvirt, but want to make
> > > sure that I don't allow something that could later be forbidden by qemu
> > > (thus creating a compatibility problem during upgrades). Beyond Gerd's
> > > specific requests to allow ehci, uhci, and hda controllers to attach to
> > > PCIe ports, are there any other devices that I specifically should or
> > > shouldn't allow? (I would rather be conservative in what I allow - it's
> > > easy to allow more things later, but nearly impossible to revoke
> > > permission once it's been allowed).
> For the moment I would not remove any restrictions, but only the ones
> requested and verified by somebody.
> 
> > 
> > IMO, we really need to grow an interface to query this kind of thing.
> Basically libvirt needs to know:
> 1. for (libvirt) controllers: what kind of devices can be plugged in
> 2. for devices (controller is also a device)
>     - to which controllers can it be plugged in
>     - does it support hot-plug?
> 3. implicit controllers of the machine types (q35 - "pcie-root", i440fx - "pci-root")
> All the above must be exported to libvirt
> 
> Implementation options:
> 1. Add a compliance field on PCI/PCIe devices and controllers stating if it supports
>    PCI/PCIe or both (and maybe hot-plug)
>    - consider plug type + compliance to figure out whether a plug can go into a socket
>    
> 2. Use Markus Armbruster idea of introducing a concept of "plug and sockets":
>    - dividing the devices into adapters and plugs
>    - adding sockets to bridges(buses?).
>    In this way it would be clear which devices can connect to bridges
> 
> Any thoughts?
> Thanks,
> Marcel

It's all not too hard to implement, we just need to know
what kind of interface makes sense for management.

So Cc libvir-list@redhat.com .

> 
> > 
> 
>