[Qemu-devel] Number of usable slots in PCIe Root Port / PCIe Switch Downstream Port

[Qemu-devel] Number of usable slots in PCIe Root Port / PCIe Switch Downstream Port

[Andrea Bolognani]

AFAIK some PCI controllers, particularly:

  * PCIe Root Port (pcie-root-port, ioh3420)
  * PCIe Switch Downstream Port (xio3130-downstream)

only have a single usable slot. libvirt knows about this
fact, and will prevent you from adding more than one
device to the respective bus.

However, as Lukáš recently noticed, QEMU won't complain
if you add more devices:

  $ qemu-system-x86_64 \
    -nodefaults -nographic \
    -M q35 -monitor stdio \
    -device pcie-root-port,id=pci.1 \
    -device virtio-scsi-pci,bus=pci.1 \
    -device virtio-scsi-pci,bus=pci.1
  QEMU 2.9.0 monitor - type 'help' for more information
  (qemu) info qtree
  bus: main-system-bus
    type System
    [...]
    dev: q35-pcihost, id ""
      [...]
      bus: pcie.0
        type PCIE
        dev: pcie-root-port, id "pci.1"
          [...]
          bus: pci.1
            type PCIE
            dev: virtio-scsi-pci, id ""
              [...]
              addr = 01.0
              bus: virtio-bus
                type virtio-pci-bus
                dev: virtio-scsi-device, id ""
                  [...]
                  bus: scsi.1
                    type SCSI
            dev: virtio-scsi-pci, id ""
              [...]
              addr = 00.0
              bus: virtio-bus
                type virtio-pci-bus
                dev: virtio-scsi-device, id ""
                  [...]
                  bus: scsi.0
                    type SCSI
  (qemu)

As you can see, all devices will show up in the qtree;
only the one with addr=00.0, however, will actually be
visible to the guest OS according to my tests.

Is such a configuration considered valid? Should QEMU
complain loudly about it and refuse to start? Or should
libvirt and the guest OS / firmware start allowing it?

-- 
Andrea Bolognani / Red Hat / Virtualization

Re: [Qemu-devel] Number of usable slots in PCIe Root Port / PCIe Switch Downstream Port

[Marcel Apfelbaum]

On 23/08/2017 19:46, Andrea Bolognani wrote:
> AFAIK some PCI controllers, particularly:
> 
>    * PCIe Root Port (pcie-root-port, ioh3420)
>    * PCIe Switch Downstream Port (xio3130-downstream)
> 
> only have a single usable slot. libvirt knows about this
> fact, and will prevent you from adding more than one
> device to the respective bus.
> 
> However, as Lukáš recently noticed, QEMU won't complain
> if you add more devices:
> 
>    $ qemu-system-x86_64 \
>      -nodefaults -nographic \
>      -M q35 -monitor stdio \
>      -device pcie-root-port,id=pci.1 \
>      -device virtio-scsi-pci,bus=pci.1 \
>      -device virtio-scsi-pci,bus=pci.1
>    QEMU 2.9.0 monitor - type 'help' for more information
>    (qemu) info qtree
>    bus: main-system-bus
>      type System
>      [...]
>      dev: q35-pcihost, id ""
>        [...]
>        bus: pcie.0
>          type PCIE
>          dev: pcie-root-port, id "pci.1"
>            [...]
>            bus: pci.1
>              type PCIE
>              dev: virtio-scsi-pci, id ""
>                [...]
>                addr = 01.0
>                bus: virtio-bus
>                  type virtio-pci-bus
>                  dev: virtio-scsi-device, id ""
>                    [...]
>                    bus: scsi.1
>                      type SCSI
>              dev: virtio-scsi-pci, id ""
>                [...]
>                addr = 00.0
>                bus: virtio-bus
>                  type virtio-pci-bus
>                  dev: virtio-scsi-device, id ""
>                    [...]
>                    bus: scsi.0
>                      type SCSI
>    (qemu)
> 

Hi Andrea,

> As you can see, all devices will show up in the qtree;
> only the one with addr=00.0, however, will actually be
> visible to the guest OS according to my tests.
> 
> Is such a configuration considered valid? 

Definitely no.

>Should QEMU
> complain loudly about it and refuse to start? 

Yes, but we need a clean way to do it. Eduardo is working
on a series that would expose to libvirt this info,
Eduardo can you please confirm?

Or should
> libvirt and the guest OS / firmware start allowing it?
> 

I thought libvirt does not allow it anyway and yes, is a bug,
but considered low priority.
I personally hoped Eduardo's work can also help in this direction
(classifying PCI Buses, we can add a 'rule' to allow only one
PCIe device per bus.)

But maybe a simple patch preventing QEMU to start would be enough.

Thanks for bringing the issue to my attention,
Marcel

Re: [Qemu-devel] Number of usable slots in PCIe Root Port / PCIe Switch Downstream Port

[Eduardo Habkost]

On Wed, Aug 23, 2017 at 11:16:27PM +0300, Marcel Apfelbaum wrote:
> On 23/08/2017 19:46, Andrea Bolognani wrote:
> > AFAIK some PCI controllers, particularly:
> > 
> >    * PCIe Root Port (pcie-root-port, ioh3420)
> >    * PCIe Switch Downstream Port (xio3130-downstream)
> > 
> > only have a single usable slot. libvirt knows about this
> > fact, and will prevent you from adding more than one
> > device to the respective bus.
> > 
> > However, as Lukáš recently noticed, QEMU won't complain
> > if you add more devices:
> > 
> >    $ qemu-system-x86_64 \
> >      -nodefaults -nographic \
> >      -M q35 -monitor stdio \
> >      -device pcie-root-port,id=pci.1 \
> >      -device virtio-scsi-pci,bus=pci.1 \
> >      -device virtio-scsi-pci,bus=pci.1
> >    QEMU 2.9.0 monitor - type 'help' for more information
> >    (qemu) info qtree
> >    bus: main-system-bus
> >      type System
> >      [...]
> >      dev: q35-pcihost, id ""
> >        [...]
> >        bus: pcie.0
> >          type PCIE
> >          dev: pcie-root-port, id "pci.1"
> >            [...]
> >            bus: pci.1
> >              type PCIE
> >              dev: virtio-scsi-pci, id ""
> >                [...]
> >                addr = 01.0
> >                bus: virtio-bus
> >                  type virtio-pci-bus
> >                  dev: virtio-scsi-device, id ""
> >                    [...]
> >                    bus: scsi.1
> >                      type SCSI
> >              dev: virtio-scsi-pci, id ""
> >                [...]
> >                addr = 00.0
> >                bus: virtio-bus
> >                  type virtio-pci-bus
> >                  dev: virtio-scsi-device, id ""
> >                    [...]
> >                    bus: scsi.0
> >                      type SCSI
> >    (qemu)
> > 
> 
> Hi Andrea,
> 
> > As you can see, all devices will show up in the qtree;
> > only the one with addr=00.0, however, will actually be
> > visible to the guest OS according to my tests.
> > 
> > Is such a configuration considered valid?
> 
> Definitely no.
> 
> > Should QEMU
> > complain loudly about it and refuse to start?
> 
> Yes, but we need a clean way to do it. Eduardo is working
> on a series that would expose to libvirt this info,
> Eduardo can you please confirm?

The work has two parts: 1) actually preventing invalid
configurations; 2) providing a query interface to let libvirt
know which configurations are valid.

My focus is on (2) (adding a query-device-slots interface), but
we surely need to address bugs related to (1) first, so both the
device-plugging code and slot-querying code are correct and agree
with each other.

-- 
Eduardo

Re: [Qemu-devel] Number of usable slots in PCIe Root Port / PCIe Switch Downstream Port

[Andrea Bolognani]

On Wed, 2017-08-23 at 23:16 +0300, Marcel Apfelbaum wrote:
> > Is such a configuration considered valid?
> 
> Definitely no.
> 
> > Or should
> > libvirt and the guest OS / firmware start allowing it?
> 
> I thought libvirt does not allow it anyway and yes, is a bug,
> but considered low priority.

That's correct, libvirt knows those controllers only have
a single slot and will prevent you from adding more than
one device to them. So no change needed there.

IIUC Lukáš is using QEMU directly in (some parts of)
Avocado, so I imagine emitting an error rather than
silently accepting an invalid configuration would save
him some debugging time :) But he knows not to do that
now, so I guess low priority sounds about right.

Thanks for clarifying!

-- 
Andrea Bolognani / Red Hat / Virtualization