[PATCH v3 0/4] hw/pci-bridge: Configurable apparent bandwidth for the PCIe topology

[PATCH v3 0/4] hw/pci-bridge: Configurable apparent bandwidth for the PCIe topology

[David Vrabel]

When a VM has been assigned a number of PCIe devices behind a common
PCIe switch and the VM intends to use peer-to-peer (P2P) transfers
between the devices, it is useful to be able to more report the
apparent PCIe bandwidth so it reflects the real handwidth of the
underlying hardware. This allows software in the VM (such as NCCL[1])
to make better decisions about whether to use P2P or not.

The existing emulated PCIe switch is a vintage TI XIO3130 and only
support 2.5 GT/s and x1, so we introduces new generic upstream and
downstream port devices (pcie-upstream-port and pcie-downstream-port),
inspired by the existing generic pcie-root-port device.

The existing experimental "x-speed" and "x-width" options are made
non-experimental, allowing the pcie-root-port and pcie-downstream-port
capabilities to be configured to match the hardware.

The pcie-root-port device is updated to default to a Gen6 capable
device.

Finally, where a root or upstream port's bandwidth has been limited,
we also ensure the other end of the link reports the same link status.

Note that the device IDs for the new port devices are currently
placeholders and will need to be allocated.

Changes in v3:
  - Correct PCIe device ID allocation.
  - Compat value for pcie-root-port.speed for QEMU 11.1 and earlier.
  - Trivial style changes.

Changes in v2:
  - Initial public version.

David

[1] https://developer.nvidia.com/nccl

David Vrabel (4):
  hw/pci-bridge: Add generic PCIe switch upstream and downstream ports
  hw/pci-bridge: Make speed and width for PCIe ports non-experimental
  hw/pci-bridge: Default the generic PCIe root port as Gen6 capable
  hw/pci: Ensure PCIe link speed/width is consistent between both ends

 docs/pcie.txt                               |   6 +-
 docs/specs/pci-ids.rst                      |   4 +
 hw/core/machine.c                           |   1 +
 hw/pci-bridge/gen_pcie_downstream_port.c    | 202 ++++++++++++++++++++
 hw/pci-bridge/gen_pcie_root_port.c          |   6 +-
 hw/pci-bridge/gen_pcie_upstream_port.c      | 177 +++++++++++++++++
 hw/pci-bridge/meson.build                   |   3 +-
 hw/pci/pcie.c                               |  26 ++-
 include/hw/pci/pci.h                        |   2 +
 tests/functional/x86_64/meson.build         |   1 +
 tests/functional/x86_64/test_pcie_switch.py |  98 ++++++++++
 11 files changed, 515 insertions(+), 11 deletions(-)
 create mode 100644 hw/pci-bridge/gen_pcie_downstream_port.c
 create mode 100644 hw/pci-bridge/gen_pcie_upstream_port.c
 create mode 100755 tests/functional/x86_64/test_pcie_switch.py

-- 
2.43.0

[PATCH v3 1/4] hw/pci-bridge: Add generic PCIe switch upstream and downstream ports

[David Vrabel]

The TI XIO3130 PCIe switch is a vintage Gen1 device, so we add generic
PCIe upstream and downstream port devices that are Gen6.

In guests with pairs of passthrough PCIe devices (e.g, a GPU+NIC pair)
often need to place the devices behind a common (emulated) PCIe switch
so in-guest software (such as NCCL) can correctly determine that P2P
transactions between the devices is possible.

NCCL also needs to be able to determine the available PCIe
bandwidth. If such a guest sees the TI XIO3130 PCIe switch, then it
incorrectly calculates only 2.5 GT/s and x1 is available, but the real
hardware might be 32 GT/s and x16.

Instead of enhancing the existing switch and claiming it is more
capable than the real hardware we add additional generic PCIe upstream
and downstream port devices, which matches the existing generic PCIe
root port device.

The speed and width of the ports are configurable.

Signed-off-by: David Vrabel <david.vrabel@nutanix.com>
---
Cc: Gerd Hoffmann <kraxel@redhat.com>
---
 docs/pcie.txt                               |   6 +-
 docs/specs/pci-ids.rst                      |   4 +
 hw/pci-bridge/gen_pcie_downstream_port.c    | 202 ++++++++++++++++++++
 hw/pci-bridge/gen_pcie_upstream_port.c      | 177 +++++++++++++++++
 hw/pci-bridge/meson.build                   |   3 +-
 include/hw/pci/pci.h                        |   2 +
 tests/functional/x86_64/meson.build         |   1 +
 tests/functional/x86_64/test_pcie_switch.py |  98 ++++++++++
 8 files changed, 489 insertions(+), 4 deletions(-)
 create mode 100644 hw/pci-bridge/gen_pcie_downstream_port.c
 create mode 100644 hw/pci-bridge/gen_pcie_upstream_port.c
 create mode 100755 tests/functional/x86_64/test_pcie_switch.py

diff --git a/docs/pcie.txt b/docs/pcie.txt
index df49178311..f12a9daaad 100644
--- a/docs/pcie.txt
+++ b/docs/pcie.txt
@@ -85,7 +85,7 @@ number of PCI Express Root Ports per PCI Express Root Bus is 256.
 
 Prefer grouping PCI Express Root Ports into multi-function devices
 to keep a simple flat hierarchy that is enough for most scenarios.
-Only use PCI Express Switches (x3130-upstream, xio3130-downstream)
+Only use PCI Express Switches (pcie-upstream-port, pcie-downstream-port)
 if there is no more room for PCI Express Root Ports.
 Please see section 4. for further justifications.
 
@@ -120,8 +120,8 @@ Plug only PCI Express devices into PCI Express Ports.
       -device pcie-root-port,id=root_port3,chassis=x2,addr=z.2[,slot=y2][,bus=pcie.0] \
 2.2.3 Plugging a PCI Express device into a Switch:
       -device pcie-root-port,id=root_port1,chassis=x,slot=y[,bus=pcie.0][,addr=z]  \
-      -device x3130-upstream,id=upstream_port1,bus=root_port1[,addr=x]          \
-      -device xio3130-downstream,id=downstream_port1,bus=upstream_port1,chassis=x1,slot=y1[,addr=z1]] \
+      -device pcie-upstream-port,id=upstream_port1,bus=root_port1[,addr=x]          \
+      -device pcie-downstream-port,id=downstream_port1,bus=upstream_port1,chassis=x1,slot=y1[,addr=z1]] \
       -device <dev>,bus=downstream_port1
 
 Notes:
diff --git a/docs/specs/pci-ids.rst b/docs/specs/pci-ids.rst
index 261b0f359f..60cb0f7db6 100644
--- a/docs/specs/pci-ids.rst
+++ b/docs/specs/pci-ids.rst
@@ -100,6 +100,10 @@ PCI devices (other than virtio):
   PCI UFS device (``-device ufs``)
 1b36:0014
   PCI RISC-V IOMMU device
+1b36:0015
+  PCIe Upstream Switch Port (``-device pcie-upstream-port``)
+1b36:0016
+  PCIe Downstream Switch Port (``-device pcie-downstream-port``)
 
 All these devices are documented in :doc:`index`.
 
diff --git a/hw/pci-bridge/gen_pcie_downstream_port.c b/hw/pci-bridge/gen_pcie_downstream_port.c
new file mode 100644
index 0000000000..d9c182e881
--- /dev/null
+++ b/hw/pci-bridge/gen_pcie_downstream_port.c
@@ -0,0 +1,202 @@
+/*
+ * Generic PCI Express Downstream Port emulation
+ *
+ * Copyright (C) 2026 Nutanix, Inc.
+ *
+ * Derived from gen_pcie_root_port.c:
+ *   Copyright (C) 2017 Red Hat Inc
+ *
+ * Derived from xio3130_downstream.c:
+ *   Copyright (c) 2010 Isaku Yamahata <yamahata at valinux co jp>
+ *                       VA Linux Systems Japan K.K.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/module.h"
+#include "hw/pci/msix.h"
+#include "hw/pci/pcie.h"
+#include "hw/pci/pcie_port.h"
+#include "hw/core/qdev-properties.h"
+#include "hw/core/qdev-properties-system.h"
+#include "migration/vmstate.h"
+#include "qom/object.h"
+
+#define TYPE_GEN_PCIE_DOWNSTREAM_PORT            "pcie-downstream-port"
+OBJECT_DECLARE_SIMPLE_TYPE(GenPCIEDownstreamPort, GEN_PCIE_DOWNSTREAM_PORT)
+
+struct GenPCIEDownstreamPort {
+    PCIESlot parent_obj;
+};
+
+#define GEN_PCIE_DOWNSTREAM_PORT_AER_OFFSET      0x100
+#define GEN_PCIE_DOWNSTREAM_PORT_ACS_OFFSET \
+    (GEN_PCIE_DOWNSTREAM_PORT_AER_OFFSET + PCI_ERR_SIZEOF)
+#define GEN_PCIE_DOWNSTREAM_PORT_MSIX_NR_VECTOR  1
+
+
+static int gen_downstream_interrupts_init(PCIDevice *d, Error **errp)
+{
+    int rc;
+
+    rc = msix_init_exclusive_bar(d, GEN_PCIE_DOWNSTREAM_PORT_MSIX_NR_VECTOR,
+                                 0, errp);
+    if (rc < 0) {
+        assert(rc == -ENOTSUP);
+    } else {
+        msix_vector_use(d, 0);
+    }
+
+    return rc;
+}
+
+static void gen_downstream_interrupts_uninit(PCIDevice *d)
+{
+    msix_uninit_exclusive_bar(d);
+}
+
+static void gen_downstream_write_config(PCIDevice *d, uint32_t address,
+                                        uint32_t val, int len)
+{
+    uint16_t slt_ctl, slt_sta;
+
+    pcie_cap_slot_get(d, &slt_ctl, &slt_sta);
+    pci_bridge_write_config(d, address, val, len);
+    pcie_cap_flr_write_config(d, address, val, len);
+    pcie_cap_slot_write_config(d, slt_ctl, slt_sta, address, val, len);
+    pcie_aer_write_config(d, address, val, len);
+}
+
+static void gen_downstream_reset(DeviceState *qdev)
+{
+    PCIDevice *d = PCI_DEVICE(qdev);
+
+    pcie_cap_deverr_reset(d);
+    pcie_cap_slot_reset(d);
+    pcie_cap_arifwd_reset(d);
+    pci_bridge_reset(qdev);
+}
+
+static void gen_downstream_realize(PCIDevice *d, Error **errp)
+{
+    PCIEPort *p = PCIE_PORT(d);
+    PCIESlot *s = PCIE_SLOT(d);
+    int rc;
+
+    pci_bridge_initfn(d, TYPE_PCIE_BUS);
+    pcie_port_init_reg(d);
+
+    rc = gen_downstream_interrupts_init(d, errp);
+    if (rc < 0) {
+        goto err_bridge;
+    }
+
+    rc = pcie_cap_init(d, 0x90, PCI_EXP_TYPE_DOWNSTREAM, p->port, errp);
+    if (rc < 0) {
+        goto err_int;
+    }
+    pcie_cap_flr_init(d);
+    pcie_cap_deverr_init(d);
+    pcie_cap_slot_init(d, s);
+    pcie_cap_arifwd_init(d);
+
+    pcie_chassis_create(s->chassis);
+    rc = pcie_chassis_add_slot(s);
+    if (rc < 0) {
+        error_setg(errp, "Can't add chassis slot, error %d", rc);
+        goto err_cap;
+    }
+
+    rc = pcie_aer_init(d, PCI_ERR_VER, GEN_PCIE_DOWNSTREAM_PORT_AER_OFFSET,
+                       PCI_ERR_SIZEOF, errp);
+    if (rc < 0) {
+        goto err_chassis;
+    }
+
+    pcie_acs_init(d, GEN_PCIE_DOWNSTREAM_PORT_ACS_OFFSET);
+
+    return;
+
+err_chassis:
+    pcie_chassis_del_slot(s);
+err_cap:
+    pcie_cap_exit(d);
+err_int:
+    gen_downstream_interrupts_uninit(d);
+err_bridge:
+    pci_bridge_exitfn(d);
+}
+
+static void gen_downstream_exitfn(PCIDevice *d)
+{
+    PCIESlot *s = PCIE_SLOT(d);
+
+    pcie_aer_exit(d);
+    pcie_chassis_del_slot(s);
+    pcie_cap_exit(d);
+    gen_downstream_interrupts_uninit(d);
+    pci_bridge_exitfn(d);
+}
+
+static const VMStateDescription vmstate_gen_pcie_downstream_port = {
+    .name = "pcie-downstream-port",
+    .priority = MIG_PRI_PCI_BUS,
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .post_load = pcie_cap_slot_post_load,
+    .fields = (const VMStateField[]) {
+        VMSTATE_PCI_DEVICE(parent_obj.parent_obj.parent_obj, PCIESlot),
+        VMSTATE_STRUCT(parent_obj.parent_obj.parent_obj.exp.aer_log,
+                       PCIESlot, 0, vmstate_pcie_aer_log, PCIEAERLog),
+        VMSTATE_MSIX(parent_obj.parent_obj.parent_obj.parent_obj,
+                     GenPCIEDownstreamPort),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
+static const Property gen_pcie_downstream_port_props[] = {
+    DEFINE_PROP_BIT(COMPAT_PROP_PCP, PCIDevice, cap_present,
+                    QEMU_PCIE_SLTCAP_PCP_BITNR, true),
+    DEFINE_PROP_PCIE_LINK_SPEED("x-speed", PCIESlot,
+                                speed, PCIE_LINK_SPEED_64),
+    DEFINE_PROP_PCIE_LINK_WIDTH("x-width", PCIESlot,
+                                width, PCIE_LINK_WIDTH_32),
+};
+
+static void gen_pcie_downstream_port_class_init(ObjectClass *klass,
+                                                const void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
+
+    k->config_write = gen_downstream_write_config;
+    k->realize = gen_downstream_realize;
+    k->exit = gen_downstream_exitfn;
+    k->vendor_id = PCI_VENDOR_ID_REDHAT;
+    k->device_id = PCI_DEVICE_ID_REDHAT_PCIE_DSP;
+    set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+    dc->desc = "PCI Express Downstream Port";
+    device_class_set_legacy_reset(dc, gen_downstream_reset);
+    dc->vmsd = &vmstate_gen_pcie_downstream_port;
+    device_class_set_props(dc, gen_pcie_downstream_port_props);
+}
+
+static const TypeInfo gen_pcie_downstream_port_info = {
+    .name          = TYPE_GEN_PCIE_DOWNSTREAM_PORT,
+    .parent        = TYPE_PCIE_SLOT,
+    .instance_size = sizeof(GenPCIEDownstreamPort),
+    .class_init    = gen_pcie_downstream_port_class_init,
+    .interfaces = (const InterfaceInfo[]) {
+        { INTERFACE_PCIE_DEVICE },
+        { }
+    },
+};
+
+static void gen_pcie_downstream_port_register_types(void)
+{
+    type_register_static(&gen_pcie_downstream_port_info);
+}
+
+type_init(gen_pcie_downstream_port_register_types)
diff --git a/hw/pci-bridge/gen_pcie_upstream_port.c b/hw/pci-bridge/gen_pcie_upstream_port.c
new file mode 100644
index 0000000000..e86eff099f
--- /dev/null
+++ b/hw/pci-bridge/gen_pcie_upstream_port.c
@@ -0,0 +1,177 @@
+/*
+ * Generic PCI Express Upstream Port emulation
+ *
+ * Copyright (C) 2026 Nutanix, Inc.
+ *
+ * Derived from gen_pcie_root_port.c:
+ *   Copyright (C) 2017 Red Hat Inc
+ *
+ * Derived from xio3130_upstream.c:
+ *   Copyright (c) 2010 Isaku Yamahata <yamahata at valinux co jp>
+ *                       VA Linux Systems Japan K.K.
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/module.h"
+#include "hw/pci/msix.h"
+#include "hw/pci/pcie.h"
+#include "hw/pci/pcie_port.h"
+#include "hw/core/qdev-properties.h"
+#include "hw/core/qdev-properties-system.h"
+#include "migration/vmstate.h"
+#include "qom/object.h"
+
+#define TYPE_GEN_PCIE_UPSTREAM_PORT              "pcie-upstream-port"
+OBJECT_DECLARE_SIMPLE_TYPE(GenPCIEUpstreamPort, GEN_PCIE_UPSTREAM_PORT)
+
+struct GenPCIEUpstreamPort {
+    PCIEPort parent_obj;
+
+    PCIExpLinkSpeed speed;
+    PCIExpLinkWidth width;
+};
+
+#define GEN_PCIE_UPSTREAM_PORT_AER_OFFSET        0x100
+#define GEN_PCIE_UPSTREAM_PORT_MSIX_NR_VECTOR    1
+
+static int gen_upstream_interrupts_init(PCIDevice *d, Error **errp)
+{
+    int rc;
+
+    rc = msix_init_exclusive_bar(d, GEN_PCIE_UPSTREAM_PORT_MSIX_NR_VECTOR,
+                                 0, errp);
+    if (rc < 0) {
+        assert(rc == -ENOTSUP);
+    } else {
+        msix_vector_use(d, 0);
+    }
+
+    return rc;
+}
+
+static void gen_upstream_interrupts_uninit(PCIDevice *d)
+{
+    msix_uninit_exclusive_bar(d);
+}
+
+static void gen_upstream_write_config(PCIDevice *d, uint32_t address,
+                                      uint32_t val, int len)
+{
+    pci_bridge_write_config(d, address, val, len);
+    pcie_cap_flr_write_config(d, address, val, len);
+    pcie_aer_write_config(d, address, val, len);
+}
+
+static void gen_upstream_reset(DeviceState *qdev)
+{
+    PCIDevice *d = PCI_DEVICE(qdev);
+    GenPCIEUpstreamPort *usp = GEN_PCIE_UPSTREAM_PORT(d);
+
+    pci_bridge_reset(qdev);
+    pcie_cap_deverr_reset(d);
+    pcie_cap_fill_link_ep_usp(d, usp->width, usp->speed, false);
+}
+
+static void gen_upstream_realize(PCIDevice *d, Error **errp)
+{
+    PCIEPort *p = PCIE_PORT(d);
+    int rc;
+
+    pci_bridge_initfn(d, TYPE_PCIE_BUS);
+    pcie_port_init_reg(d);
+
+    rc = gen_upstream_interrupts_init(d, errp);
+    if (rc < 0) {
+        goto err_bridge;
+    }
+
+    rc = pcie_cap_init(d, 0x90, PCI_EXP_TYPE_UPSTREAM, p->port, errp);
+    if (rc < 0) {
+        goto err_int;
+    }
+    pcie_cap_flr_init(d);
+    pcie_cap_deverr_init(d);
+
+    rc = pcie_aer_init(d, PCI_ERR_VER, GEN_PCIE_UPSTREAM_PORT_AER_OFFSET,
+                       PCI_ERR_SIZEOF, errp);
+    if (rc < 0) {
+        goto err_cap;
+    }
+
+    return;
+
+err_cap:
+    pcie_cap_exit(d);
+err_int:
+    gen_upstream_interrupts_uninit(d);
+err_bridge:
+    pci_bridge_exitfn(d);
+}
+
+static void gen_upstream_exitfn(PCIDevice *d)
+{
+    pcie_aer_exit(d);
+    pcie_cap_exit(d);
+    gen_upstream_interrupts_uninit(d);
+    pci_bridge_exitfn(d);
+}
+
+static const VMStateDescription vmstate_gen_pcie_upstream_port = {
+    .name = "pcie-upstream-port",
+    .priority = MIG_PRI_PCI_BUS,
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .fields = (const VMStateField[]) {
+        VMSTATE_PCI_DEVICE(parent_obj.parent_obj, PCIEPort),
+        VMSTATE_STRUCT(parent_obj.parent_obj.exp.aer_log, PCIEPort, 0,
+                       vmstate_pcie_aer_log, PCIEAERLog),
+        VMSTATE_MSIX(parent_obj.parent_obj.parent_obj, GenPCIEUpstreamPort),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
+static const Property gen_pcie_upstream_port_props[] = {
+    DEFINE_PROP_PCIE_LINK_SPEED("x-speed", GenPCIEUpstreamPort,
+                                speed, PCIE_LINK_SPEED_64),
+    DEFINE_PROP_PCIE_LINK_WIDTH("x-width", GenPCIEUpstreamPort,
+                                width, PCIE_LINK_WIDTH_32),
+};
+
+static void gen_pcie_upstream_port_class_init(ObjectClass *klass,
+                                              const void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
+
+    k->config_write = gen_upstream_write_config;
+    k->realize = gen_upstream_realize;
+    k->exit = gen_upstream_exitfn;
+    k->vendor_id = PCI_VENDOR_ID_REDHAT;
+    k->device_id = PCI_DEVICE_ID_REDHAT_PCIE_USP;
+    set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+    dc->desc = "PCI Express Upstream Port";
+    device_class_set_legacy_reset(dc, gen_upstream_reset);
+    dc->vmsd = &vmstate_gen_pcie_upstream_port;
+    device_class_set_props(dc, gen_pcie_upstream_port_props);
+}
+
+static const TypeInfo gen_pcie_upstream_port_info = {
+    .name          = TYPE_GEN_PCIE_UPSTREAM_PORT,
+    .parent        = TYPE_PCIE_PORT,
+    .instance_size = sizeof(GenPCIEUpstreamPort),
+    .class_init    = gen_pcie_upstream_port_class_init,
+    .interfaces = (const InterfaceInfo[]) {
+        { INTERFACE_PCIE_DEVICE },
+        { }
+    },
+};
+
+static void gen_pcie_upstream_port_register_types(void)
+{
+    type_register_static(&gen_pcie_upstream_port_info);
+}
+
+type_init(gen_pcie_upstream_port_register_types)
diff --git a/hw/pci-bridge/meson.build b/hw/pci-bridge/meson.build
index 2e0eb0d233..fd0ad188bc 100644
--- a/hw/pci-bridge/meson.build
+++ b/hw/pci-bridge/meson.build
@@ -2,7 +2,8 @@ pci_ss = ss.source_set()
 pci_ss.add(when: 'CONFIG_PCI_BRIDGE', if_true: files('pci_bridge_dev.c'))
 pci_ss.add(when: 'CONFIG_I82801B11', if_true: files('i82801b11.c'))
 pci_ss.add(when: 'CONFIG_IOH3420', if_true: files('ioh3420.c'))
-pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c'))
+pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c',
+                                                    'gen_pcie_upstream_port.c', 'gen_pcie_downstream_port.c'))
 pci_ss.add(when: 'CONFIG_PCIE_PCI_BRIDGE', if_true: files('pcie_pci_bridge.c'))
 pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c'),
                                if_false: files('pci_expander_bridge_stubs.c'))
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index f2448e941a..6520886855 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -121,6 +121,8 @@ extern bool pci_available;
 #define PCI_DEVICE_ID_REDHAT_ACPI_ERST   0x0012
 #define PCI_DEVICE_ID_REDHAT_UFS         0x0013
 #define PCI_DEVICE_ID_REDHAT_RISCV_IOMMU 0x0014
+#define PCI_DEVICE_ID_REDHAT_PCIE_USP    0x0015
+#define PCI_DEVICE_ID_REDHAT_PCIE_DSP    0x0016
 #define PCI_DEVICE_ID_REDHAT_QXL         0x0100
 
 #define FMT_PCIBUS                      PRIx64
diff --git a/tests/functional/x86_64/meson.build b/tests/functional/x86_64/meson.build
index 0353b2af8e..0ce49764a2 100644
--- a/tests/functional/x86_64/meson.build
+++ b/tests/functional/x86_64/meson.build
@@ -43,6 +43,7 @@ tests_x86_64_system_thorough = [
   'multiprocess',
   'netdev_ethtool',
   'nvme_migration',
+  'pcie_switch',
   'replay',
   'reverse_debug',
   'tuxrun',
diff --git a/tests/functional/x86_64/test_pcie_switch.py b/tests/functional/x86_64/test_pcie_switch.py
new file mode 100755
index 0000000000..7088b85059
--- /dev/null
+++ b/tests/functional/x86_64/test_pcie_switch.py
@@ -0,0 +1,98 @@
+#!/usr/bin/env python3
+#
+# Functional test for generic PCIe switch upstream and downstream ports
+#
+# Copyright (C) 2026 Nutanix, Inc.
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+
+from qemu_test import LinuxKernelTest, Asset, exec_command_and_wait_for_pattern
+
+
+class PCIeSwitchPort(LinuxKernelTest):
+
+    timeout = 120
+
+    ASSET_KERNEL = Asset(
+        ('https://archives.fedoraproject.org/pub/archive/fedora/linux/releases'
+         '/31/Server/x86_64/os/images/pxeboot/vmlinuz'),
+        'd4738d03dbbe083ca610d0821d0a8f1488bebbdccef54ce33e3adb35fda00129')
+
+    ASSET_INITRD = Asset(
+        ('https://archives.fedoraproject.org/pub/archive/fedora/linux/releases'
+         '/31/Server/x86_64/os/images/pxeboot/initrd.img'),
+        '277cd6c7adf77c7e63d73bbb2cded8ef9e2d3a2f100000e92ff1f8396513cd8b')
+
+    USP_VENDOR_ID = '0x1b36'
+    USP_DEVICE_ID = '0x0015'
+    DSP_VENDOR_ID = '0x1b36'
+    DSP_DEVICE_ID = '0x0016'
+    CLASS_BRIDGE_PCI_PCI = '0x060400'
+
+    def command(self, command: str, expected_result: int = 0):
+        command = f'echo __begin ; {command} ; D="done" ; echo "$?/__$D"'
+        output = exec_command_and_wait_for_pattern(self, command, '__done')
+        lines = output.decode().splitlines()
+        for s, line in enumerate(lines):
+            if line == '__begin':
+                break
+        result = int(lines[-1].split("/")[0])
+        assert result == expected_result
+        return "\n".join(lines[s+1:-1])
+
+    def test_pcie_switch_ports(self):
+        """Boot a q35 VM with a PCIe switch (USP + DSP) and verify topology."""
+        self.require_accelerator('kvm')
+        self.set_machine('q35')
+
+        self.vm.add_args('-accel', 'kvm')
+        self.vm.add_args('-m', '1G')
+        self.vm.add_args('-device',
+                         'pcie-root-port,id=rp0,slot=0,chassis=0,bus=pcie.0')
+        self.vm.add_args('-device',
+                         'pcie-upstream-port,id=sw-usp,bus=rp0')
+        self.vm.add_args('-device',
+                         'pcie-downstream-port,id=sw-dsp,bus=sw-usp,'
+                         'chassis=1')
+        self.vm.add_args('-append', 'console=ttyS0 rd.rescue')
+
+        self.launch_kernel(self.ASSET_KERNEL.fetch(),
+                           self.ASSET_INITRD.fetch(),
+                           wait_for='Entering emergency mode.')
+        self.wait_for_console_pattern('# ')
+
+        # Capture lspci output, to diagnose failures.
+        #
+        # [0000:00]-+-00.0  8086:29c0
+        #           +-01.0  8086:10d3
+        #           +-02.0-[01-03]----00.0-[02-03]----00.0-[03]--
+        #           +-1f.0  8086:2918
+        #           +-1f.2  8086:2922
+        #           \-1f.3  8086:2930
+        self.command("lspci -vtn")
+        self.command("lspci -vvv")
+        self.command("ls -R /sys/devices/")
+
+        usp_path = "/sys/devices/pci0000:00/0000:00:02.0/0000:01:00.0"
+        dsp_path = usp_path + "/0000:02:00.0"
+
+        # Upstream port is expected model.
+        vendor_id = self.command(f'cat {usp_path}/vendor')
+        assert vendor_id == self.USP_VENDOR_ID
+
+        device_id = self.command(f'cat {usp_path}/device')
+        assert device_id == self.USP_DEVICE_ID
+
+        class_code = self.command(f'cat {usp_path}/class')
+        assert class_code == self.CLASS_BRIDGE_PCI_PCI
+
+        # Downstream port is expected model.
+        vendor_id = self.command(f'cat {dsp_path}/vendor')
+        assert vendor_id == self.DSP_VENDOR_ID
+
+        device_id = self.command(f'cat {dsp_path}/device')
+        assert device_id == self.DSP_DEVICE_ID
+
+
+if __name__ == '__main__':
+    LinuxKernelTest.main()
-- 
2.43.0

[PATCH v3 2/4] hw/pci-bridge: Make speed and width for PCIe ports non-experimental

[David Vrabel]

For guests with passthrough PCIe devices, in order to make the guest
PCIe topology have the same apparent bandwidth as the underlying
hardware we need to be able to set the capabilities (speed and width)
of the virtual root, upstream, and downstream ports.

Promote the existing experimental `x-speed` and `x-width` options to
non-experimental `speed` and `width`.

Signed-off-by: David Vrabel <david.vrabel@nutanix.com>
---
 hw/pci-bridge/gen_pcie_downstream_port.c | 4 ++--
 hw/pci-bridge/gen_pcie_root_port.c       | 4 ++--
 hw/pci-bridge/gen_pcie_upstream_port.c   | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/hw/pci-bridge/gen_pcie_downstream_port.c b/hw/pci-bridge/gen_pcie_downstream_port.c
index d9c182e881..d00bbd55f0 100644
--- a/hw/pci-bridge/gen_pcie_downstream_port.c
+++ b/hw/pci-bridge/gen_pcie_downstream_port.c
@@ -159,9 +159,9 @@ static const VMStateDescription vmstate_gen_pcie_downstream_port = {
 static const Property gen_pcie_downstream_port_props[] = {
     DEFINE_PROP_BIT(COMPAT_PROP_PCP, PCIDevice, cap_present,
                     QEMU_PCIE_SLTCAP_PCP_BITNR, true),
-    DEFINE_PROP_PCIE_LINK_SPEED("x-speed", PCIESlot,
+    DEFINE_PROP_PCIE_LINK_SPEED("speed", PCIESlot,
                                 speed, PCIE_LINK_SPEED_64),
-    DEFINE_PROP_PCIE_LINK_WIDTH("x-width", PCIESlot,
+    DEFINE_PROP_PCIE_LINK_WIDTH("width", PCIESlot,
                                 width, PCIE_LINK_WIDTH_32),
 };
 
diff --git a/hw/pci-bridge/gen_pcie_root_port.c b/hw/pci-bridge/gen_pcie_root_port.c
index 5434d693d9..d3ace7f0db 100644
--- a/hw/pci-bridge/gen_pcie_root_port.c
+++ b/hw/pci-bridge/gen_pcie_root_port.c
@@ -129,9 +129,9 @@ static const Property gen_rp_props[] = {
                      res_reserve.mem_pref_32, -1),
     DEFINE_PROP_SIZE("pref64-reserve", GenPCIERootPort,
                      res_reserve.mem_pref_64, -1),
-    DEFINE_PROP_PCIE_LINK_SPEED("x-speed", PCIESlot,
+    DEFINE_PROP_PCIE_LINK_SPEED("speed", PCIESlot,
                                 speed, PCIE_LINK_SPEED_16),
-    DEFINE_PROP_PCIE_LINK_WIDTH("x-width", PCIESlot,
+    DEFINE_PROP_PCIE_LINK_WIDTH("width", PCIESlot,
                                 width, PCIE_LINK_WIDTH_32),
 };
 
diff --git a/hw/pci-bridge/gen_pcie_upstream_port.c b/hw/pci-bridge/gen_pcie_upstream_port.c
index e86eff099f..61ae3a294a 100644
--- a/hw/pci-bridge/gen_pcie_upstream_port.c
+++ b/hw/pci-bridge/gen_pcie_upstream_port.c
@@ -134,9 +134,9 @@ static const VMStateDescription vmstate_gen_pcie_upstream_port = {
 };
 
 static const Property gen_pcie_upstream_port_props[] = {
-    DEFINE_PROP_PCIE_LINK_SPEED("x-speed", GenPCIEUpstreamPort,
+    DEFINE_PROP_PCIE_LINK_SPEED("speed", GenPCIEUpstreamPort,
                                 speed, PCIE_LINK_SPEED_64),
-    DEFINE_PROP_PCIE_LINK_WIDTH("x-width", GenPCIEUpstreamPort,
+    DEFINE_PROP_PCIE_LINK_WIDTH("width", GenPCIEUpstreamPort,
                                 width, PCIE_LINK_WIDTH_32),
 };
 
-- 
2.43.0

[PATCH v3 3/4] hw/pci-bridge: Default the generic PCIe root port as Gen6 capable

[David Vrabel]

As part of making virtual PCIe topology having the same apparent
bandwidth as a the underlying hardware, make the generic PCIe root
port support maximum speed so by default it doesn't appear to be a
bottleneck.

This also makes the three generic PCIe ports (root, upstream, and
downstream) have the same link speed/width capabilities.

Signed-off-by: David Vrabel <david.vrabel@nutanix.com>
---
 hw/core/machine.c                  | 1 +
 hw/pci-bridge/gen_pcie_root_port.c | 2 +-
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/hw/core/machine.c b/hw/core/machine.c
index 8939ae1666..c4ced7a8c1 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -43,6 +43,7 @@
 GlobalProperty hw_compat_11_1[] = {
     { "sysbus-ehci-usb", "x-migrate-fetch-addr-64bit", "off" },
     { "pci-ehci-usb", "x-migrate-fetch-addr-64bit", "off" },
+    { "pcie-root-port", "speed", "16" },
 };
 const size_t hw_compat_11_1_len = G_N_ELEMENTS(hw_compat_11_1);
 
diff --git a/hw/pci-bridge/gen_pcie_root_port.c b/hw/pci-bridge/gen_pcie_root_port.c
index d3ace7f0db..8cabeadac8 100644
--- a/hw/pci-bridge/gen_pcie_root_port.c
+++ b/hw/pci-bridge/gen_pcie_root_port.c
@@ -130,7 +130,7 @@ static const Property gen_rp_props[] = {
     DEFINE_PROP_SIZE("pref64-reserve", GenPCIERootPort,
                      res_reserve.mem_pref_64, -1),
     DEFINE_PROP_PCIE_LINK_SPEED("speed", PCIESlot,
-                                speed, PCIE_LINK_SPEED_16),
+                                speed, PCIE_LINK_SPEED_64),
     DEFINE_PROP_PCIE_LINK_WIDTH("width", PCIESlot,
                                 width, PCIE_LINK_WIDTH_32),
 };
-- 
2.43.0

[PATCH v3 4/4] hw/pci: Ensure PCIe link speed/width is consistent between both ends

[David Vrabel]

If an emulated PCIe endpoint has higher link speed or width
capabilities than the upstream port it is connected to, ensure that
the link status on both ends is consistent by using the link status
computed for the upstream port.

If the PCIe endpoint is a passthrough device, the link status
continues to match the real status and consistent values requires the
emulated upstream port to be at least as capable as the endpoint.

Signed-off-by: David Vrabel <david.vrabel@nutanix.com>
---
 hw/pci/pcie.c | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/hw/pci/pcie.c b/hw/pci/pcie.c
index 42bcb9206d..bd00305ecc 100644
--- a/hw/pci/pcie.c
+++ b/hw/pci/pcie.c
@@ -169,18 +169,36 @@ void pcie_cap_fill_link_ep_usp(PCIDevice *dev, PCIExpLinkWidth width,
                                PCIExpLinkSpeed speed, bool flitmode)
 {
     uint8_t *exp_cap = dev->config + dev->exp.exp_cap;
+    PCIDevice *upstream;
 
     /*
-     * For an end point or USP need to set the current status as well
-     * as the capabilities.
+     * For an endpoint or USP, set Link Status as well as capabilities.
+     * Provisionally use this device's width/speed, then adopt the upstream
+     * port's values so both ends of an emulated link agree.  The upstream
+     * LNKSTA is already clamped to the negotiated link capabilities via
+     * pcie_sync_bridge_lnk().
      */
-    pci_long_test_and_clear_mask(exp_cap + PCI_EXP_LNKSTA,
+    pci_word_test_and_clear_mask(exp_cap + PCI_EXP_LNKSTA,
                                  PCI_EXP_LNKSTA_CLS | PCI_EXP_LNKSTA_NLW);
-    pci_long_test_and_set_mask(exp_cap + PCI_EXP_LNKSTA,
+    pci_word_test_and_set_mask(exp_cap + PCI_EXP_LNKSTA,
                                QEMU_PCI_EXP_LNKSTA_NLW(width) |
                                QEMU_PCI_EXP_LNKSTA_CLS(speed));
 
     pcie_cap_fill_lnk(exp_cap, width, speed, flitmode);
+
+    upstream = pci_bridge_get_device(pci_get_bus(dev));
+    if (upstream && pci_is_express_downstream_port(upstream)) {
+        uint8_t *up_exp = upstream->config + upstream->exp.exp_cap;
+        uint16_t lnksta;
+
+        pcie_sync_bridge_lnk(upstream);
+        lnksta = pci_get_word(up_exp + PCI_EXP_LNKSTA);
+        pci_word_test_and_clear_mask(exp_cap + PCI_EXP_LNKSTA,
+                                     PCI_EXP_LNKSTA_CLS | PCI_EXP_LNKSTA_NLW);
+        pci_word_test_and_set_mask(exp_cap + PCI_EXP_LNKSTA,
+                                   lnksta & (PCI_EXP_LNKSTA_CLS |
+                                             PCI_EXP_LNKSTA_NLW));
+    }
 }
 
 static void pcie_cap_fill_slot_lnk(PCIDevice *dev)
-- 
2.43.0

Re: [PATCH v3 0/4] hw/pci-bridge: Configurable apparent bandwidth for the PCIe topology

[David Vrabel]

ping?
On 28/09/2026 16:24, David Vrabel wrote:
> When a VM has been assigned a number of PCIe devices behind a common
> PCIe switch and the VM intends to use peer-to-peer (P2P) transfers
> between the devices, it is useful to be able to more report the
> apparent PCIe bandwidth so it reflects the real handwidth of the
> underlying hardware. This allows software in the VM (such as NCCL[1])
> to make better decisions about whether to use P2P or not.
> 
> The existing emulated PCIe switch is a vintage TI XIO3130 and only
> support 2.5 GT/s and x1, so we introduces new generic upstream and
> downstream port devices (pcie-upstream-port and pcie-downstream-port),
> inspired by the existing generic pcie-root-port device.
> 
> The existing experimental "x-speed" and "x-width" options are made
> non-experimental, allowing the pcie-root-port and pcie-downstream-port
> capabilities to be configured to match the hardware.
> 
> The pcie-root-port device is updated to default to a Gen6 capable
> device.
> 
> Finally, where a root or upstream port's bandwidth has been limited,
> we also ensure the other end of the link reports the same link status.

Re: [PATCH v3 0/4] hw/pci-bridge: Configurable apparent bandwidth for the PCIe topology

[Michael S. Tsirkin]

On Thu, Oct 15, 2026 at 02:41:55PM +0100, David Vrabel wrote:
> 
> ping?
> On 28/09/2026 16:24, David Vrabel wrote:
> > When a VM has been assigned a number of PCIe devices behind a common
> > PCIe switch and the VM intends to use peer-to-peer (P2P) transfers
> > between the devices, it is useful to be able to more report the
> > apparent PCIe bandwidth so it reflects the real handwidth of the
> > underlying hardware. This allows software in the VM (such as NCCL[1])
> > to make better decisions about whether to use P2P or not.
> > 
> > The existing emulated PCIe switch is a vintage TI XIO3130 and only
> > support 2.5 GT/s and x1, so we introduces new generic upstream and
> > downstream port devices (pcie-upstream-port and pcie-downstream-port),
> > inspired by the existing generic pcie-root-port device.
> > 
> > The existing experimental "x-speed" and "x-width" options are made
> > non-experimental, allowing the pcie-root-port and pcie-downstream-port
> > capabilities to be configured to match the hardware.
> > 
> > The pcie-root-port device is updated to default to a Gen6 capable
> > device.
> > 
> > Finally, where a root or upstream port's bandwidth has been limited,
> > we also ensure the other end of the link reports the same link status.


will merge, thanks!