[Qemu-devel] PXB fixes for QEMU, and extra root buses for OVMF

[Laszlo Ersek <lersek@redhat.com>]

Following up on this cross-posted message, I will send two patch sets,
one for QEMU (to qemu-devel) and another for OVMF (to edk2-devel). With
both in place, OVMF supports multiple PCI root buses.

Below I'm writing up the way I tested the feature, plus a few random
notes.

(1) Interrupt line assignments. I patched SeaBIOS (temporarily) to print
    interrupt line assignments, and I also patched OVMF (permanently) to
    print the same. (See both patches in the OVMF patch

      OvmfPkg: PlatformBdsLib: debug log interrupt line assignments

    in one of the followup series; the commit message contains the
    ad-hoc SeaBIOS patch.) The QEMU command line was:

    qemu-system-x86_64 \
      -m 2048 \
      -M pc \
      -enable-kvm \
      -device qxl-vga \
      \
      $FIRMWARE_OPTIONS \
      \
      -drive id=cdrom,if=none,readonly,format=raw,file=$ISO \
      -device virtio-scsi-pci,id=scsi0 \
      -device scsi-cd,bus=scsi0.0,drive=cdrom,bootindex=0 \
      \
      -debugcon file:debug.log \
      -global isa-debugcon.iobase=0x402 \
      \
      -monitor stdio \
      \
      -device pxb,id=bridge1,bus_nr=4 \
      \
      -netdev user,id=netdev0,hostfwd=tcp:127.0.0.1:2222-:22 \
      -device e1000,netdev=netdev0,bus=bridge1,addr=2,romfile= \
      \
      -netdev user,id=netdev1,hostfwd=tcp:127.0.0.1:2223-:22 \
      -device e1000,netdev=netdev1,bus=bridge1,addr=3,romfile= \
      \
      -netdev user,id=netdev2,hostfwd=tcp:127.0.0.1:2224-:22 \
      -device e1000,netdev=netdev2,bus=bridge1,addr=4,romfile= \
      \
      -netdev user,id=netdev3,hostfwd=tcp:127.0.0.1:2225-:22 \
      -device e1000,netdev=netdev3,bus=bridge1,addr=5,romfile=

    The ISO variable pointed to a Fedora 20 LiveCD (although it is not
    relevant for this test). FIRMWARE_OPTIONS were

      -bios .../out/bios.bin

    for the (debug-patched, see above) SeaBIOS test case, and

      -drive if=pflash,readonly,format=raw,file=.../OVMF_CODE.fd \
      -drive if=pflash,format=raw,file=.../COPY_OF_OVMF_VARS.fd

    for the OVMF test case.

    As you can see the QEMU command line places four e1000 NICs on one
    PXB (I used e1000 because that's what Marcel recommended for all
    testing). SeaBIOS then printed

      PCI: init bdf=00:01.3 id=8086:7113
      assigned irq line 10

      PCI: init bdf=00:02.0 id=1b36:0100
      assigned irq line 10

      PCI: init bdf=00:03.0 id=1af4:1004
      assigned irq line 11

      PCI: init bdf=04:00.0 id=1b36:0001
      assigned irq line 11

      PCI: init bdf=05:02.0 id=8086:100e
      assigned irq line 10

      PCI: init bdf=05:03.0 id=8086:100e
      assigned irq line 11

      PCI: init bdf=05:04.0 id=8086:100e
      assigned irq line 11

      PCI: init bdf=05:05.0 id=8086:100e
      assigned irq line 10

    whereas OVMF printed

      SetPciIntLine: PciRoot(0x0)/Pci(0x1,0x3) -> 0x0A
      SetPciIntLine: PciRoot(0x0)/Pci(0x2,0x0) -> 0x0A
      SetPciIntLine: PciRoot(0x0)/Pci(0x3,0x0) -> 0x0B
      SetPciIntLine: PciRoot(0x4)/Pci(0x0,0x0) -> 0x0B
      SetPciIntLine: PciRoot(0x4)/Pci(0x0,0x0)/Pci(0x2,0x0) -> 0x0A
      SetPciIntLine: PciRoot(0x4)/Pci(0x0,0x0)/Pci(0x3,0x0) -> 0x0B
      SetPciIntLine: PciRoot(0x4)/Pci(0x0,0x0)/Pci(0x4,0x0) -> 0x0B
      SetPciIntLine: PciRoot(0x4)/Pci(0x0,0x0)/Pci(0x5,0x0) -> 0x0A

    (Note that the bus number assigned to the devices on the pxb is 5 in
    the OVMF case too. This is not visible from the device paths logged
    above, but it is clear from edk2's PCI bus driver's log, and the
    output of the "PCI" UEFI shell command.)

    Therefore I concluded that the IRQ line assignment logic in OVMF
    that Gabriel had contributed earlier (and that I updated very
    slightly in the OVMF patchset here) continued to work correctly.

(2) Marcel, no new fw_cfg file is necessary for exposing the bus numbers
    of the extra root buses to OVMF. OVMF probes all buses and all
    devices (function 0) to locate the extra root buses. This procedure
    is shortened by adhering to the "etc/extra-pci-roots" fw_cfg file,
    which is already there.

    This recommendation came from both Michael & Marcel, and it was
    "surprisingly easy" to implement with edk2's PciLib.

(3) We discussed earlier the idea to flip the PXB's _HID and _UID
    objects to numeric values in QEMU; I implemented and tested that
    (see more test cases below).

(4) Ping test was successful from the OVMF-booted Fedora 20 LivecD
    environment, using one e1000 NIC on a PXB.

(5) Ping test was successful from the UEFI shell, using Intel's
    proprietary UEFI driver for the e1000 NIC (called PROEFI). The NIC
    was located on the same PXB as in the previous point.

(6) Successfully loaded public web pages with Firefox in an OVMF-booted
    Windows Server 2012 R2 guest. The NIC was located on the same PXB as
    in the previous point.

(7) At least one issue remains to be solved (designed) in QEMU, for both
    SeaBIOS's and OVMF's sake: booting off devices that are located on
    the PXB. The problem is with the "bootorder" fw_cfg file. Consider
    the following example:

      /pci@i0cf8/scsi@3/channel@0/disk@0,0
      /pci/pci-bridge@0/ethernet@2/ethernet-phy@0

    which is generated for the options

      -device virtio-scsi-pci,id=scsi0 \
      -device scsi-cd,bus=scsi0.0,drive=cdrom,bootindex=0 \
      \
      -device pxb,id=bridge1,bus_nr=4 \
      -device e1000,netdev=netdev0,bus=bridge1,addr=2,romfile=,bootindex=1

    While the first entry is recognized by both SeaBIOS and OVMF, the
    second entry (generated for the NIC hanging off the PXB, see above)
    is recognized by neither. (I tested OVMF, and investigated the
    SeaBIOS source, for this claim.)

    For the SeaBIOS explanation, grep the source code for FW_PCI_DOMAIN.

    The OVMF explanation is that OVMF simply rejects the initial
    OpenFirmware device path node "/pci" with a controlled parse error
    (as opposed to the "/pci@i0cf8" node, which it recognizes and
    translates to UEFI in combination with the rest of that OFW device
    path).

    The "/pci" node comes from QEMU's sysbus_get_fw_dev_path() function,
    file "hw/core/sysbus.c", where *neither* of the (s->num_mmio) and
    (s->num_pio) branches apply. (The (s->num_pio) branch applies for
    the first entry, ie. "/pci@i0cf8".)

    Something has to be invented here to clue in both firmwares as to
    the root bus number (here bus_nr=4), in a format that is compliant
    with the "OpenFirmware unit address" concept. (Note that
    "/pci-bridge@0" only gives away the slot number *on* the extra root
    bus, not the number of the root bus itself.) For example:

      /pci@rootbus4/pci-bridge@0/ethernet@2/ethernet-phy@0

    would be acceptable. However, I don't know how to implement this in
    sysbus_get_fw_dev_path().

Thanks
Laszlo