[RFC QEMU PATCH 00/10] Initial Support for VCS Switching

[RFC QEMU PATCH 00/10] Initial Support for VCS Switching

[Joshua Lant]

Hi,

(All references to the CXL specification here are made to v3.2.)

VCS= Virtual CXL Switch
PPB= PCI-PCI bridge
vPPB= virtual PCI-PCI bridge
SLD= Single Logical Device
FM= Fabric Manager
USP/DSP=Upstream/Downstream Port

This patchset provides basic functionality for emulation of an FM-owned,
multi-VCS CXL switch (7.1.3). Primarily it adds a new object defined
in hw/cxl/cxl-vcs-switch.c, and supports the FMAPI commands vppb get
info/bind/unbind 0x5200/0x5201/0x5202 (7.6.7.2.1-7.6.7.2.3) to be used on SLD’s.

It is posted as an RFC since:

1). There are several limitations with this work and I would like to discuss
how to proceed.
2). There has been prior discussion on potential restructuring of the CXL
cci/switch code[1], which I suppose will dictate how to move forward with parts
of this series…

-------------------
VCS Emulation

The VCS command set should ultimately allow for multi-host sharing of a MLD/DCD
device through the same switched fabric, with a single Fabric Manager/CCI-mailbox
endpoint for configuration/management. The work here is a stepping stone toward
that, allowing for SLD’s to be bound/unbound to a local QEMU instance by the FM.

The cxl-vcs-switch is comprised of one or more VCSs, and one or more hidden
endpoint devices. A single VCS is defined as a single cxl-upstream-port
(meaning a 1:1 mapping between physical upstream PPBs and number of VCS’s),
and one or more cxl-downstream-ports (these form the vPPBs of each VCS). The
number of vcs-attached endpoints defined on the CLI forms the number of
downstream PPBs the switch would have. These endpoints are hidden on boot, and
connect to one of the vPPBs upon bind. This means that there is in effect
no real downstream PPB in QEMU. The vPPB device effectively becomes the
downstream PPB following bind.

When the topology is initialised:
- Upstream/downstream ports instantiated as part of a VCS switch are realized
normally, but additionally register with the cxl-vcs-switch object, which are
then referenced by the bind/unbind FMAPI commands etc.
- The endpoint devices which connect to the downstream PPBs use the
DeviceListener functionality for hiding devices (see the reference in the
commit message of patch 2). The QDict from the CLI is stored in the VCS structs,
which are then realized/unrealized on bind/unbind commands from the FM.

All this means that on boot the guest sees its upstream port and all downstream
ports (vppbs) enumerated (as is described in section 7.1.4 and 7.2.1.3), but
none of the endpoints are seen.

The cxl-vcs-switch itself is implemented as a user creatable class, since it
does not fit the single inheritance device model of QEMU, which would force
association with a single PCIe bus, which will not work for multiple USPs.

The cxl-vcs-switch uses local-fm=true/false CLI option to dictate whether the
object will have a CCI mailbox attached. VCS state information will be held in
the local-fm=true instance, and the FM will communicate directly with this
instance only. IPC will be used (in multi-USP, multi-QEMU process environments)
in order to maintain correct state information in the local-fm, and to
bind/unbind devices in remote QEMU processes.

Currently only the “managed hot-remove” flow is complete (Table 7-34) for the
unbind operation, notifying the guest of removal and awaiting signal from
the OS for unbind completion from the unrealize DeviceListener function.

This is tested with the topology below and some additional libcxlmi test
programs[2]. I am able to bind and unbind correctly to multiple VCSs, see the
updated switch state, and see the delay in the unrealize listener callback from
the delays in OS notification.

-------------------
Limitations and open questions

1. Unbound, but fully realized devices (allowing proper MLDs/DCDs)

The current method of hiding the endpoint device and storing the QDicts works
for simple devices only. But ultimately the FM should be able to tunnel commands
through the switch to communicate with the device, whether a guest is bound to
it or not... We need a method of fully realizing the device, but on some sort of
dummy bus that is not seen by the guest. I am unsure how to do this currently,
since AFAICT it goes against the qdev model of device realization, being
inherently associated with attaching to the guest’s bus (please correct me if
I’m wrong on this). It will require a way to properly realize the devices, but
bypass automatic association with the guest’s QOM tree? I don’t know if there
is any precedent for behaviour like this in QEMU?

2. Integration with Physical Switch Command set.

Currently the physical switch command set in cci-mailbox-utils.c has not been
modified to account for a VCS target (i.e. using those commands with a VCS
target currently breaks things). This is where the discussion in [1] comes in.

> - Move the call that caches state to the cxl_upstream_port reset
>  to ensure downstream ports are in place before it is called.
> Also will make it available from whatever CCI.  If we ever support
> multiple VCS switches this will need to move an appropriate structure
> representing whole switch information. With only one USP that is
> a reasonable place to put full switch info.

Since in my implementation the CXLPhyPortInfo will end up being associated with
the VCS object and not with the USP, further refactoring will be required for
this patch series to generalise the physical switch command set functions.

3. Distributed VCS control

Since the ultimate aim of this is to give the FM control of multiple VCSs in
multiple QEMU instances, some method of sending FM commands between QEMU
processes is needed, since the switch state will be distributed over these
processes (with status kept in the local-fm=true instance). Does QEMU have a
standard way for implementing such IPC or shall I just add some simple sockets
communication into the cxl-vcs-switch.c?

4. Unimplemented commands from virtual switch command set.


The actual bind process described in 7.6.6.7 shows how event records must be
used, and the FMAPI command should return success without waiting on binding
completion. Further work is needed to completely emulate the flow as described
in the specification, and implement the remaining FMAPI commands.

5. Tiered switching.

Currently the downstream PPBs are nothing more than a struct, and only endpoint
devices can be hidden. There should be some way to implement another complete
switch below the downstream PPB of the first switch, as described in
section 9.12.2.

-------------------
Testing

topology:

-device usb-ehci,id=ehci \
-object memory-backend-file,id=cxl-mem1,share=on,mem-path=/$LOG_DIR/t3_cxl1.raw,size=8G \
-object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/$LOG_DIR/t3_lsa1.raw,size=1M \
-object memory-backend-file,id=cxl-mem2,share=on,mem-path=/$LOG_DIR/t3_cxl2.raw,size=8G \
-object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/$LOG_DIR/t3_lsa2.raw,size=1M \
-object memory-backend-file,id=cxl-mem3,share=on,mem-path=/$LOG_DIR/t3_cxl3.raw,size=8G \
-object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/$LOG_DIR/t3_lsa3.raw,size=1M \
-object memory-backend-file,id=cxl-mem4,share=on,mem-path=/$LOG_DIR/t3_cxl4.raw,size=8G \
-object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/$LOG_DIR/t3_lsa4.raw,size=1M \
-object cxl-vcs-switch,id=vcs0,usp-ppbs=2,dsp-ppbs=4,local-fm=true \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.0,hdm_for_passthrough=true \
-device cxl-rp,port=0,bus=cxl.0,id=root_port1,chassis=0,slot=1 \
-device pxb-cxl,bus_nr=22,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
-device cxl-rp,port=0,bus=cxl.1,id=root_port2,chassis=1,slot=1 \
-device cxl-upstream,port=0,sn=1234,bus=root_port1,id=us0,addr=0.0,multifunction=on,vcs=vcs0,usppb=0 \
-device cxl-upstream,port=0,sn=5678,bus=root_port2,id=us1,addr=0.0,multifunction=on,vcs=vcs0,usppb=1 \
-device cxl-switch-mailbox-cci,bus=root_port1,addr=0.3,target=vcs0 \
-device usb-cxl-mctp,bus=ehci.0,id=usb0,target=vcs0 \
-device cxl-downstream,port=0,bus=us0,id=swport0,slot=3 \
-device cxl-downstream,port=1,bus=us0,id=swport1,slot=4 \
-device cxl-downstream,port=2,bus=us0,id=swport2,slot=5 \
-device cxl-downstream,port=3,bus=us0,id=swport3,slot=6 \
-device cxl-downstream,port=0,bus=us1,id=swport4,slot=7 \
-device cxl-downstream,port=1,bus=us1,id=swport5,slot=8 \
-device cxl-downstream,port=2,bus=us1,id=swport6,slot=9 \
-device cxl-downstream,port=3,bus=us1,id=swport7,slot=10 \
-device cxl-type3,persistent-memdev=cxl-mem1,id=cxl-ep1,lsa=cxl-lsa1,sn=99,vcs=vcs0,dsppb=0 \
-device cxl-type3,persistent-memdev=cxl-mem2,id=cxl-ep2,lsa=cxl-lsa2,sn=100,vcs=vcs0,dsppb=1 \
-device cxl-type3,volatile-dc-memdev=cxl-mem3,id=cxl-dcd1,lsa=cxl-lsa3,num-dc-regions=8,sn=101,vcs=vcs0,dsppb=2 \
-device cxl-type3,volatile-dc-memdev=cxl-mem4,id=cxl-dcd2,lsa=cxl-lsa4,num-dc-regions=8,sn=102,vcs=vcs0,dsppb=3 \
-machine cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=8G,cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=8G

libcxlmi:

See [2]…

1. Setup MCTP communication: e.g.
mctp link set mctpusb0 up;
mctp addr add 8 dev mctpusb0;
mctp link set mctpusb0 net 1;
systemctl restart mctpd.service
busctl call au.com.codeconstruct.MCTP1 /au/com/codeconstruct/mctp1/interfaces/mctpusb0 au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0

2. Build, then run libcxlmi commands to bind vcs0:
./build/examples/vcs-bind-mctp 1 9 0 0 0
./build/examples/vcs-get-virtual-switch-info-mctp 1 9

3. Demonstrate the CXL device is usable:
cxl create-region -m -t pmem -d decoder0.0 -w 1 -g 1024 -s 256M mem0
ndctl create-namespace --region region0 --mode fsdax --size 256M
echo "HELLO WORLD..." > /dev/pmem0
cat /dev/pmem0

4. Teardown the device, and rebind to vcs1, and check that it maps
correctly:
ndctl disable-namespace namespace0.0
ndctl destroy-namespace namespace0.0
cxl disable-region region0
cxl destroy-region region0
cxl disable-memdev mem0
./build/examples/vcs-unbind-mctp 1 9 0 0 1
# wait 5s for the hp notification... see lspci/dmesg change
# bind the same device to the other VCS.
./build/examples/vcs-bind-mctp 1 9 1 0 0
cxl create-region -m -t pmem -d decoder0.1 -w 1 -g 1024 -s 256M mem0
ndctl create-namespace --region region1 --mode fsdax --size 256M
cat /dev/pmem1
# See the hello world originally written by vcs0!

-------------------
Build

The patches are applied on the upstream qemu 10.2 release, on top of the
following patchsets from various branches of Jonathan’s fork:

1: [PATCH qemu v5 0/5] cxl: r3.2 specification event updates.
https://lore.kernel.org/linux-cxl/20260205112350.60681-1-Jonathan.Cameron@huawei.com/

2: [PATCH qemu for 10.2 0/3] cxl: Additional RAS features support.
https://lore.kernel.org/linux-cxl/20250917143330.294698-1-Jonathan.Cameron@huawei.com/

3: [PATCH qemu 0/2] hw/cxl: Two media operations related fixes.
https://lore.kernel.org/linux-cxl/20260102154731.474859-1-Jonathan.Cameron@huawei.com/

4: [PATCH qemu v7 0/7] hw/cxl: Support Back-Invalidate (+ PCIe Flit mode)
https://lore.kernel.org/linux-cxl/20260204170936.43959-1-Jonathan.Cameron@huawei.com/

5: [PATCH qemu v5 0/3] hw/cxl: FM-API Physical Switch Command Set Support.
https://lore.kernel.org/linux-cxl/20260204173223.44122-1-Jonathan.Cameron@huawei.com/

6: [RFC PATCH qemu 0/5] hw/cxl/mctp/i2c/usb: MCTP for OoB control of CXL devices.
https://lore.kernel.org/linux-cxl/20250609163334.922346-1-Jonathan.Cameron@huawei.com/

-------------------
References

[1] https://lore.kernel.org/linux-cxl/20260127152350.00006447@huawei.com/

[2] https://github.com/joshualant/libcxlmi/tree/vcs-testing


Many thanks,

Josh

Joshua Lant (10):
  docs: Add documentation for cxl-vcs-switch
  qdev/qbus: Allow hidden devices to be busless on QEMU startup
  cxl-type3: Properly unmap the memory-backend on device exit
  cxl_downstream: enable power controller present capability.
  cxl-vcs-switch: Initial support for CXL VCS.
  cxl-upstream-port: Add support for targeting a VCS switch
  cxl-downstream-port: Add support for VCS switching
  cxl-cci-mailbox: Add support for targeting a VCS switch
  cxl-mailbox-utils: Add support for VCS bind/unbind commands.
  cxl-mailbox-utils: Add support for VCS Get Virtual CXL Switch Info
    command.

 docs/system/devices/cxl.rst               |  90 +++-
 hw/cxl/cxl-mailbox-utils.c                | 207 ++++++++-
 hw/cxl/cxl-vcs-switch.c                   | 524 ++++++++++++++++++++++
 hw/cxl/meson.build                        |   1 +
 hw/cxl/switch-mailbox-cci.c               |  33 +-
 hw/mem/cxl_type3.c                        |   3 +
 hw/pci-bridge/cxl_downstream.c            |  13 +
 hw/pci-bridge/cxl_upstream.c              |  20 +
 hw/usb/dev-mctp.c                         |  23 +-
 include/hw/cxl/cxl_device.h               |  10 +-
 include/hw/cxl/cxl_vcs_switch.h           | 134 ++++++
 include/hw/pci-bridge/cxl_upstream_port.h |   2 +
 qapi/qom.json                             |  19 +
 system/qdev-monitor.c                     |  10 +-
 14 files changed, 1055 insertions(+), 34 deletions(-)
 create mode 100644 hw/cxl/cxl-vcs-switch.c
 create mode 100644 include/hw/cxl/cxl_vcs_switch.h

-- 
2.47.3

[RFC QEMU PATCH 01/10] docs: Add documentation for cxl-vcs-switch

[Joshua Lant]

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 docs/system/devices/cxl.rst | 90 ++++++++++++++++++++++++++++++++++---
 1 file changed, 85 insertions(+), 5 deletions(-)

diff --git a/docs/system/devices/cxl.rst b/docs/system/devices/cxl.rst
index 32b1b5d773..9e8452e576 100644
--- a/docs/system/devices/cxl.rst
+++ b/docs/system/devices/cxl.rst
@@ -119,11 +119,11 @@ and associated component register access via PCI bars.
 CXL Switch
 ~~~~~~~~~~
 Here we consider a simple CXL switch with only a single
-virtual hierarchy. Whilst more complex devices exist, their
-visibility to a particular host is generally the same as for
-a simple switch design. Hosts often have no awareness
-of complex rerouting and device pooling, they simply see
-devices being hot added or hot removed.
+virtual hierarchy. Whilst more complex devices exist (see VCS
+Switching below), their visibility to a particular host is
+generally the same as for a simple switch design. Hosts often
+have no awareness of complex rerouting and device pooling,
+they simply see devices being hot added or hot removed.
 
 A CXL switch has a similar architecture to those in PCIe,
 with a single upstream port, internal PCI bus and multiple
@@ -467,6 +467,86 @@ Example configuration:
 Guest OS communication with the MCTP CCI can then be established using standard
 MCTP configuration tools.
 
+CXL Multi-VCS Switching
+-----------------------
+
+The cxl-vcs-switch object allows for a Fabric Manager to dynamically reconfigure
+the switching within a multi-upstream port CXL/PCIe topology, This moves beyond
+the static switching configuration described above. The use of vcs=X on an
+endpoint device indicates that it should be hidden from guests at boot. Each
+upstream port with vcs=X set will conceptually become an upstream PPB. Any
+downstream port that is connected to an upstream port with vcs=X set will
+automatically become a vPPB for that VCS. The overall cxl-virtual-switch has a
+single CCI mailbox used for config/status of all ports within the switch.
+Setting local-fm=true indicates that this QEMU instance has the CCI mailbox
+attached. Setting it false will create listeners for commands from a remote
+QEMU process (yet to be implemented).
+
+An example of how the topology is described on the CLI is shown below:
+
+  -object cxl-vcs-switch,id=vcs0,usp-ppbs=2,dsp-ppbs=4,local-fm=true \
+  -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.0,hdm_for_passthrough=true \
+  -device cxl-rp,port=0,bus=cxl.0,id=root_port1,chassis=0,slot=1 \
+  -device pxb-cxl,bus_nr=22,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
+  -device cxl-rp,port=0,bus=cxl.1,id=root_port2,chassis=1,slot=1 \
+  -device cxl-upstream,port=0,sn=1234,bus=root_port1,id=us0,addr=0.0,multifunction=on,vcs=vcs0,usppb=0 \
+  -device cxl-upstream,port=0,sn=5678,bus=root_port2,id=us1,addr=0.0,multifunction=on,vcs=vcs0,usppb=1 \
+  -device cxl-switch-mailbox-cci,bus=root_port1,addr=0.3,target=vcs0 \
+  -device usb-cxl-mctp,bus=ehci.0,id=usb0,target=vcs0 \
+  -device cxl-downstream,port=0,bus=us0,id=dsp0,slot=3 \
+  -device cxl-downstream,port=1,bus=us0,id=dsp1,slot=4 \
+  -device cxl-downstream,port=0,bus=us1,id=dsp2,slot=7 \
+  -device cxl-downstream,port=1,bus=us1,id=dsp3,slot=8 \
+  -device cxl-type3,persistent-memdev=cxl-mem1,id=cxl-ep1,lsa=cxl-lsa1,sn=99,vcs=vcs0,dsppb=0 \
+  -device cxl-type3,persistent-memdev=cxl-mem2,id=cxl-ep2,lsa=cxl-lsa2,sn=100,vcs=vcs0,dsppb=1 \
+  -device cxl-type3,persistent-memdev=cxl-mem3,id=cxl-ep3,lsa=cxl-lsa3,sn=101,vcs=vcs0,dsppb=2 \
+  -device cxl-type3,persistent-memdev=cxl-mem4,id=cxl-ep4,lsa=cxl-lsa4,sn=102,vcs=vcs0,dsppb=3 \
+  -machine cxl-fmw.0.targets.0=cxl.0,cxl-fmw.0.size=8G,cxl-fmw.1.targets.0=cxl.1,cxl-fmw.1.size=8G
+
+Example topology involving VCS switching::
+
+               +--------------------+  +--------------------+
+               |   Host Bridge 0    |  |   Host Bridge 1    |
+               +----------+---------+  +----------+---------+
+    +-------+             |                       |
+    | MCTP  |             |                       |
+    | USB/  |  +----------+---------+  +----------+---------+
+    | I2C   |  |    Root Port 0     |  |    Root Port 1     |
+    +-----+-+  +----------+---------+  +----------+---------+
+          |               |                       |
+          |               |                       |
+   +------|---------------+-----------------------+-----------------------+
+   |    +-+--------+      |  cxl-vcs-switch (vcs0)|                       |
+   | +--| CCI MBOX |---*  |                       |                       |
+   | |  +----------+      |                       |                       |
+   | |  +-----------------+--------+      +-------+------------------+    |
+   | +--+                 |   VCS0 |  *---+       |             VCS1 |    |
+   |    | +---------------+------+ |      | +-----+----------------+ |    |
+   |    | |                      | |      | |                      | |    |
+   |    | |        USP 0         | |      | |        USP 1         | |    |
+   |    | |                      | |      | |                      | |    |
+   |    | +----+------------+----+ |      | +----+------------+----+ |    |
+   |    |      |            |      |      |      |            |      |    |
+   |    | +----+----+  +----+----+ |      | +----+----+  +----+----+ |    |
+   |    | |  DSP 0  |  |  DSP 1  | |      | |  DSP 2  |  |  DSP 3  | |    |
+   |    | |(vPPB 0) |  |(vPPB 1) | |      | |(vPPB 0) |  |(vPPB 1) | |    |
+   |    | |         |  |         | |      | |         |  |         | |    |
+   |    | +---------+  +---------+ |      | +---------+  +----+----+ |    |
+   |    +--------------------------+      +-------------------+------+    |
+   |                                                          |           |
+   |           +----------------------------------------------+           |
+   |           |                                                          |
+   |           |            -                    -            -           |
+   +-----------|------------|--------------------|------------|-----------+
+               |            |                    |            |
+          +---------+  +---------+          +---------+  +---------+
+          |CXL/PCIe |  |CXL/PCIe |          |CXL/PCIe |  |CXL/PCIe |
+          |  EP 0   |  |  EP 1   |          |  EP 2   |  |  EP 3   |
+          | (PPB0)  |  | (PPB1)  |          | (PPB2)  |  | (PPB3)  |
+          +---------+  +---------+          +---------+  +---------+
+                 PPB0 Bound to VCS1, vPPB1. Others unbound...
+
+
 References
 ----------
 
-- 
2.47.3

[RFC QEMU PATCH 02/10] qdev/qbus: Allow hidden devices to be busless on QEMU startup

[Joshua Lant]

Extend the capability for hiding devices, introduced for virtio-net
device in:

commit f3a8505656935cde32e28c1c6317f725084da1e0
Author: Jens Freimann <jfreimann@redhat.com>
Date:   Tue Oct 29 12:48:55 2019 +0100
qdev/qbus: add hidden device support

Currently only endpoint devices can be hidden with a primary device
and failover (known static configuration). However, looking at future
composable systems, we see a need for hidden devices which have no associated
bus upon boot. Move the check for hidden devices to before the bus
search, and if it is hidden ignore the case where the device was
described on the CLI without the "bus=" field.

This is motivated by a specific use-case: implementing the VCS
command set, part of the CXL specification (CXL r3.2 Section 7.1.3). In
this scenario a switch controlled by a Fabric Manager is able to change the
virtual hierarchy of devices seen by a guest within a fixed physical system
topology. The connecting bus is not known until runtime when
a bind command is issued by the Fabric Manager.

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 system/qdev-monitor.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/system/qdev-monitor.c b/system/qdev-monitor.c
index f2aa400a77..b51dfe0645 100644
--- a/system/qdev-monitor.c
+++ b/system/qdev-monitor.c
@@ -650,6 +650,7 @@ DeviceState *qdev_device_add_from_qdict(const QDict *opts,
     DeviceState *dev;
     BusState *bus = NULL;
     QDict *properties;
+    bool hide_device;
 
     driver = qdict_get_try_str(opts, "driver");
     if (!driver) {
@@ -663,6 +664,11 @@ DeviceState *qdev_device_add_from_qdict(const QDict *opts,
         return NULL;
     }
 
+    /* Is the device hidden from the guest?
+     * If yes, no need to find a default bus if none given...
+     * Bus could be provided at runtime (i.e. in a switch)*/
+    hide_device = qdev_should_hide_device(opts, from_json, errp);
+
     /* find bus */
     path = qdict_get_try_str(opts, "bus");
     if (path != NULL) {
@@ -675,14 +681,14 @@ DeviceState *qdev_device_add_from_qdict(const QDict *opts,
                        driver, object_get_typename(OBJECT(bus)));
             return NULL;
         }
-    } else if (dc->bus_type != NULL) {
+    } else if (dc->bus_type != NULL && !hide_device) {
         bus = qdev_find_default_bus(dc, errp);
         if (!bus) {
             return NULL;
         }
     }
 
-    if (qdev_should_hide_device(opts, from_json, errp)) {
+    if (hide_device) {
         if (bus && !qbus_is_hotpluggable(bus)) {
             error_setg(errp, "Bus '%s' does not support hotplugging",
                        bus->name);
-- 
2.47.3

[RFC QEMU PATCH 03/10] cxl-type3: Properly unmap the memory-backend on device exit

[Joshua Lant]

Currently the backend remains mapped, meaning that if the device owning
the backend is hot-removed, it cannot be readded in the same QEMU
instance.

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 hw/mem/cxl_type3.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 68cd04b7d9..414c776028 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1072,12 +1072,15 @@ static void ct3_exit(PCIDevice *pci_dev)
     cxl_destroy_cci(&ct3d->cci);
     if (ct3d->dc.host_dc) {
         cxl_destroy_dc_regions(ct3d);
+        host_memory_backend_set_mapped(ct3d->dc.host_dc, false);
         address_space_destroy(&ct3d->dc.host_dc_as);
     }
     if (ct3d->hostpmem) {
+        host_memory_backend_set_mapped(ct3d->hostpmem, false);
         address_space_destroy(&ct3d->hostpmem_as);
     }
     if (ct3d->hostvmem) {
+        host_memory_backend_set_mapped(ct3d->hostvmem, false);
         address_space_destroy(&ct3d->hostvmem_as);
     }
 }
-- 
2.47.3

[RFC QEMU PATCH 04/10] cxl_downstream: enable power controller present capability.

[Joshua Lant]

PCI_EXP_SLTCAP_PCP (Power Controller Present) must be set in the DSP's
slot capabilities for PCIe managed hot-remove to complete. Without this
notification from the guest of removal cannot be sent back to the
device, so device listeners for unrealizing will not fire.

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 hw/pci-bridge/cxl_downstream.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/hw/pci-bridge/cxl_downstream.c b/hw/pci-bridge/cxl_downstream.c
index 91c5e6a605..5322c46900 100644
--- a/hw/pci-bridge/cxl_downstream.c
+++ b/hw/pci-bridge/cxl_downstream.c
@@ -228,6 +228,8 @@ static const Property cxl_dsp_props[] = {
     DEFINE_PROP_PCIE_LINK_WIDTH("x-width", PCIESlot,
                                 width, PCIE_LINK_WIDTH_16),
     DEFINE_PROP_BOOL("x-256b-flit", PCIESlot, flitmode, true),
+    DEFINE_PROP_BIT(COMPAT_PROP_PCP, PCIDevice, cap_present,
+                    QEMU_PCIE_SLTCAP_PCP_BITNR, true),
 };
 
 static void cxl_dsp_class_init(ObjectClass *oc, const void *data)
-- 
2.47.3

[RFC QEMU PATCH 05/10] cxl-vcs-switch: Initial support for CXL VCS.

[Joshua Lant]

Add initial support for VCS (multi-USP/multi-logic-device) switch
emulation in CXL. Currently the object only supports the identify/bind/unbind
commands (0x5200/0x5201/0x5202), as well as support for device hiding
listeners. This enables preliminary testing of the VCS capability,
but will not allow for complete emulated flow as described in
spec (CXL v3.2, 7.2.3). Nor will it allow for multiple USP's to be
disributed over multiple QEMU instances.

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 hw/cxl/cxl-vcs-switch.c         | 524 ++++++++++++++++++++++++++++++++
 hw/cxl/meson.build              |   1 +
 include/hw/cxl/cxl_vcs_switch.h | 134 ++++++++
 qapi/qom.json                   |  19 ++
 4 files changed, 678 insertions(+)
 create mode 100644 hw/cxl/cxl-vcs-switch.c
 create mode 100644 include/hw/cxl/cxl_vcs_switch.h

diff --git a/hw/cxl/cxl-vcs-switch.c b/hw/cxl/cxl-vcs-switch.c
new file mode 100644
index 0000000000..9a492330cc
--- /dev/null
+++ b/hw/cxl/cxl-vcs-switch.c
@@ -0,0 +1,524 @@
+/*
+ * CXL VCS Capable Switch Object
+ *
+ * Copyright(C) 2026 University of Manchester.
+ * Author: Joshua Lant <joshualant@gmail.com>.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ *
+ * SPDX-License-Identifier: GPL-v2-only
+ */
+#include "hw/cxl/cxl_vcs_switch.h"
+#include "qobject/qdict.h"
+#include "monitor/qdev.h"
+
+/* Primary FM here is used when the FM is local to the QEMU instance. Setting
+ * local-fm=false in cli results in instantiation of remote listener for FM
+ * commands (TODO: to be implemented) */
+static bool cxl_vcs_get_local_fm(Object *obj, Error **errp)
+{
+    CXLVCSSwitch *vcs = CXL_VCS_SWITCH(obj);
+    return vcs->local_fm;
+}
+static void cxl_vcs_set_local_fm(Object *obj, bool value, Error **errp)
+{
+    CXLVCSSwitch *vcs = CXL_VCS_SWITCH(obj);
+    vcs->local_fm = value;
+}
+
+/* Binds an already realized MLD to a VPPB in the switch */
+static CXLRetCode cxl_vcs_bind_realized_vppb(CXLVCSSwitch *sw, uint8_t vcs_id,
+        uint8_t vppb_id, uint8_t dsp_ppb_id, uint8_t ld_id)
+{
+    /* TODO: Implement... Required for handling MLD devices... */
+    return CXL_MBOX_UNSUPPORTED;
+}
+
+/* Binds a hidden device (simple SLD) to a VPPB in the switch */
+static CXLRetCode cxl_vcs_bind_qdict_vppb(CXLVCSSwitch *sw, uint8_t vcs_id,
+        uint8_t vppb_id, uint8_t dsp_ppb_id, uint8_t ld_id)
+{
+    Error *local_err = NULL;
+    CXLVPPBInfo *vppb;
+    CXLDownstreamPPB *dspppb;
+    QDict *bind_opts;
+    DeviceState *dev;
+
+    // Upstream to bind
+    if(!sw->usp_ppbs[vcs_id]) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+    vppb = sw->usp_ppbs[vcs_id]->info->vppbs[vppb_id];
+    if(!vppb) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+    PCIDevice *vppb_dev = PCI_DEVICE(vppb->dsp);
+
+    //Downstream to bind
+    dspppb = sw->dsp_ppbs[dsp_ppb_id];
+    if(!dspppb) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+
+    // Get DSP bus id from qdev
+    bind_opts = qdict_clone_shallow(dspppb->opts);
+    qdict_put_str(bind_opts, "bus", vppb_dev->qdev.id);
+    qdict_del(bind_opts, "vcs");
+    qdict_del(bind_opts, "dsppb");
+
+    uint8_t prev_binding_status = vppb->binding_status;
+    vppb->binding_status = CXL_VPPB_BINDING_STATUS_IN_PROGRESS;
+    dev = qdev_device_add_from_qdict(bind_opts, dspppb->from_json, &local_err);
+    if(!dev || local_err) {
+        vppb->binding_status = prev_binding_status;
+        return CXL_MBOX_INTERNAL_ERROR;
+    }
+
+    vppb->bound_port_id = dsp_ppb_id;
+    vppb->bound_ld_id = CXL_INVALID_BOUND_LD_ID; // TODO: support MLDs.
+    vppb->binding_status = CXL_VPPB_BINDING_STATUS_BOUND_PORT;
+    dspppb->dev = dev;
+    dspppb->is_bound = true;
+    dspppb->bound_vcs_id = vcs_id;
+    dspppb->bound_vppb_id = vppb_id;
+    qobject_unref(bind_opts);
+
+    return CXL_MBOX_SUCCESS;
+}
+
+CXLRetCode cxl_vcs_bind_vppb(CXLVCSSwitch *sw, uint8_t vcs_id, uint8_t vppb_id,
+        uint8_t dsp_ppb_id, uint16_t ld_id)
+{
+    if(cxl_vcs_get_local_fm(OBJECT(sw), NULL)) {
+        if(ld_id == CXL_UNSUPPORTED_LD_ID) {
+            return cxl_vcs_bind_qdict_vppb(sw, vcs_id, vppb_id,
+                    dsp_ppb_id, ld_id);
+        } else {
+            // TODO: Connect up already realized device
+            // (needed for MLD support).
+            return cxl_vcs_bind_realized_vppb(sw, vcs_id, vppb_id,
+                    dsp_ppb_id, ld_id);
+        }
+    }
+    else {
+        // TODO: Send bind command to remote QEMU process...
+        return CXL_MBOX_UNSUPPORTED;
+    }
+}
+
+static CXLRetCode cxl_vcs_unbind_qdict_vppb(CXLVCSSwitch *sw, uint8_t vcs_id,
+                                       uint8_t vppb_id, uint8_t options)
+{
+    Error *local_err = NULL;
+    CXLVPPBInfo *vppb_info;
+    CXLDownstreamPPB *dspppb = NULL;
+
+    if(!sw->usp_ppbs[vcs_id]) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+
+    vppb_info = sw->usp_ppbs[vcs_id]->info->vppbs[vppb_id];
+    if (!vppb_info ||
+        (vppb_info->binding_status == CXL_VPPB_BINDING_STATUS_UNBOUND) ||
+        (vppb_info->binding_status == CXL_VPPB_BINDING_STATUS_IN_PROGRESS)) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+
+    for (int i = 0; i < sw->num_dsp_ppbs; i++) {
+        if (sw->dsp_ppbs[i] &&
+                sw->dsp_ppbs[i]->is_bound &&
+                sw->dsp_ppbs[i]->bound_vcs_id == vcs_id &&
+                sw->dsp_ppbs[i]->bound_vppb_id == vppb_id) {
+            dspppb = sw->dsp_ppbs[i];
+            break;
+        }
+    }
+
+    if (!dspppb) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+
+    if (!dspppb->dev) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+
+    /* Options from cxl v3.2 (Table 7-34), Bits[3:0].
+      — 0h = Wait for port Link Down before unbinding
+      — 1h = Simulate Managed Hot-Remove
+      — 2h = Simulate Surprise Hot-Remove */
+    options = options & 0x0F;
+    if (options == CXL_VPPB_UNBIND_WAIT_FOR_LINK_DOWN) {
+        // TODO: implement
+        return CXL_MBOX_INVALID_INPUT;
+    }
+    else if(options == CXL_VPPB_UNBIND_MANAGED_HOT_REMOVE) {
+        // unrealize listener will fire once guest notifies the port...
+        qdev_unplug(dspppb->dev, &local_err);
+        vppb_info->binding_status = CXL_VPPB_BINDING_STATUS_IN_PROGRESS;
+        if (local_err) {
+            return CXL_MBOX_INTERNAL_ERROR;
+        }
+        return CXL_MBOX_SUCCESS;
+    }
+    else if (options == CXL_VPPB_UNBIND_SURPRISE_HOT_REMOVE) {
+        // TODO: implement, This isn't a true surprise removal...
+        qdev_unplug(dspppb->dev, &local_err);
+        if (local_err) {
+            return CXL_MBOX_INTERNAL_ERROR;
+        }
+        object_unparent(OBJECT(dspppb->dev));
+        dspppb->is_bound      = false;
+        dspppb->bound_vcs_id  = 0;
+        dspppb->bound_vppb_id = 0;
+
+        /* Clear VPPB binding state so Get Virtual Switch Info reflects unbound */
+        vppb_info->binding_status = CXL_VPPB_BINDING_STATUS_UNBOUND;
+        vppb_info->bound_port_id  = 0;
+        vppb_info->bound_ld_id    = CXL_INVALID_BOUND_LD_ID;
+        dspppb->dev = NULL;
+        /* TODO: Generate Virtual CXL Switch Event Record per CXL spec
+         * section 7.6.6.6  */
+        return CXL_MBOX_SUCCESS;
+    }
+    else {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+
+}
+
+CXLRetCode cxl_vcs_unbind_vppb(CXLVCSSwitch *sw, uint8_t vcs_id,
+        uint8_t vppb_id, uint16_t option)
+{
+    /*TODO: Only currently unbinding an unrealizing whole device.
+     * Implement for MLD partial unbinding of single extents etc.
+     * */
+    return cxl_vcs_unbind_qdict_vppb(sw, vcs_id, vppb_id, option);
+}
+
+void cxl_vcs_identify_switch_device(CXLVCSSwitch *sw)
+{
+    //TODO: implement
+}
+void cxl_vcs_get_physical_port_state(CXLVCSSwitch *sw)
+{
+    //TODO: implement
+}
+void cxl_vcs_physical_port_control(CXLVCSSwitch *sw)
+{
+    //TODO: implement
+}
+void cxl_vcs_get_virtual_switch_info(CXLVCSSwitch *sw)
+{
+    //TODO: implement
+}
+
+void cxl_vcs_register_usp(CXLVCSSwitch *sw, CXLUpstreamPort *usp,
+        Error **errp)
+{
+    uint8_t ppb = usp->ppb;
+
+    if(strcmp(object_get_canonical_path_component(OBJECT(sw)), usp->vcs_name)) {
+        error_setg(errp, "VCS id for USP and switch do not match...\n");
+    }
+
+    if (ppb >= sw->num_usp_ppbs) {
+        error_setg(errp, "vcs '%s': ppb %u >= usp-ppbs %u",
+                   object_get_canonical_path_component(OBJECT(sw)),
+                   ppb, sw->num_usp_ppbs);
+        return;
+    }
+    if (sw->usp_ppbs[ppb]) {
+        error_setg(errp, "vcs '%s': USP slot %u already registered",
+                   object_get_canonical_path_component(OBJECT(sw)), ppb);
+        return;
+    }
+
+    sw->usp_ppbs[ppb] = g_new0(CXLUpstreamPPB, 1);
+    sw->usp_ppbs[ppb]->usp = usp;
+    sw->usp_ppbs[ppb]->info = g_new0(CXLVCSInfoBlock, 1);
+    sw->usp_ppbs[ppb]->info->vcs_id = ppb;
+    sw->usp_ppbs[ppb]->info->usp_id = ppb;
+    sw->usp_ppbs[ppb]->info->num_vppbs = 0;
+    sw->usp_ppbs[ppb]->info->vcs_state = CXL_VCS_STATE_ENABLED;
+    usp->swcci.vcs = sw;
+}
+
+void cxl_vcs_register_vppb(CXLVCSSwitch *sw, CXLUpstreamPort *usp,
+        CXLDownstreamPort *dsp, Error **errp)
+{
+    CXLUpstreamPPB *vcs;
+
+    for(int i = 0; i < CXL_MAX_VCS_PORTS; i++) {
+        if(sw->usp_ppbs[i]) {
+            if(usp == sw->usp_ppbs[i]->usp) {
+                vcs = sw->usp_ppbs[i];
+                break;
+            }
+        } else {
+            error_setg(errp, "The USP was not found in the VCS list...");
+            return;
+        }
+    }
+
+    for(int i = 0; i < CXL_MAX_VPPB_PER_VCS; i++) {
+        if(!vcs->info->vppbs[i]) {
+            // Free vppb slot.. lets allocate and populate it...
+            CXLVPPBInfo *vppb = g_new0(CXLVPPBInfo, 1);
+            vppb->dsp = PCI_DEVICE(dsp);
+            vppb->binding_status = CXL_VPPB_BINDING_STATUS_UNBOUND;
+            vcs->info->vppbs[i] = vppb;
+            vcs->info->num_vppbs = vcs->info->num_vppbs + 1;
+            return;
+        }
+    }
+
+    error_setg(errp, "No free VPPB slots in the VCS...");
+    return;
+}
+
+void cxl_vcs_register_qdict_dsppb(CXLVCSSwitch *sw, const QDict *opts,
+        bool from_json, Error **errp)
+{
+    QDict *dev_opts;
+    int ppb;
+    const char *ppb_str = qdict_get_try_str(opts, "dsppb");
+    ppb = atoi(ppb_str);
+    if(ppb == -1) {
+        error_setg(errp, "No ppb id given in cli.");
+        return;
+    }
+
+    if (ppb >= sw->num_dsp_ppbs) {
+        error_setg(errp, "vcs '%s': dsppb %u >= dsp-ppbs %u",
+                   object_get_canonical_path_component(OBJECT(sw)),
+                   ppb, sw->num_dsp_ppbs);
+        return;
+    }
+    if (sw->dsp_ppbs[ppb]) {
+        error_setg(errp, "vcs '%s': DSP PPB slot %u already occupied",
+                   object_get_canonical_path_component(OBJECT(sw)), ppb);
+        return;
+    }
+
+    dev_opts = qdict_clone_shallow(opts);
+    qdict_del(dev_opts, "vcs");
+    qdict_del(dev_opts, "dsppb");
+    qdict_del(dev_opts, "bus");
+
+    sw->dsp_ppbs[ppb] = g_new0(CXLDownstreamPPB, 1);
+    sw->dsp_ppbs[ppb]->opts      = dev_opts;
+    sw->dsp_ppbs[ppb]->from_json = from_json;
+    sw->dsp_ppbs[ppb]->is_bound  = false;
+}
+
+void cxl_vcs_register_dsppb(CXLVCSSwitch *sw, const QDict *opts,
+        bool from_json, Error **errp)
+{
+
+    DeviceState *dev = qdev_new(qdict_get_str(opts, "driver"));
+    qdev_set_id(dev, g_strdup(qdict_get_try_str(opts, "id")), errp);
+    int ppb;
+    const char *ppb_str = qdict_get_try_str(opts, "dsppb");
+    ppb = atoi(ppb_str);
+    if(ppb == -1) {
+        error_setg(errp, "No ppb id given in cli.");
+        return;
+    }
+
+    QDict *dev_opts = qdict_clone_shallow(opts);
+    qdict_del(dev_opts, "driver");
+    qdict_del(dev_opts, "bus");
+    qdict_del(dev_opts, "id");
+    qdict_del(dev_opts, "vcs");
+    qdict_del(dev_opts, "dsppb");
+    object_set_properties_from_keyval(OBJECT(dev), dev_opts, from_json, errp);
+    qobject_unref(dev_opts);
+    // store, don't realize
+    sw->dsp_ppbs[ppb]->dev = dev;
+}
+
+static bool cxl_vcs_switch_can_be_deleted(UserCreatable *uc)
+{
+    return false;
+}
+
+/* When a device is instantiated downstream of a VCS's PPB, we
+ * store the qdicts from the CLI, to realize the device at a
+ * future time.
+ */
+bool cxl_vcs_hide_device_listener(DeviceListener *listener, const QDict *opts,
+                        bool from_json, Error **errp)
+{
+    CXLVCSSwitch *vcs = container_of(listener, CXLVCSSwitch, listener);
+    const char *vcs_id_str   = qdict_get_try_str(opts, "vcs");
+    const char *ppb_str = qdict_get_try_str(opts, "dsppb");
+    int ppb;
+
+    /* Not our device — don't claim it */
+    if (!vcs_id_str || !ppb_str) {
+        return false;
+    }
+    if(vcs_id_str) {
+        if(strcmp(vcs_id_str,
+                    object_get_canonical_path_component(OBJECT(vcs)))) {
+            return false;
+        }
+    }
+    if(ppb_str) {
+        ppb = atoi(ppb_str);
+        if(vcs->dsp_ppbs[ppb]) {
+            error_setg(errp,
+                "The ppb %s is already populated by another device.", ppb_str);
+            return false;
+        }
+    }
+
+    cxl_vcs_register_qdict_dsppb(vcs, opts, from_json, errp);
+
+    return true;
+}
+
+/*
+ * Listener is added for unbinding. When a device is unbound using the
+ * Fabric Manager with 'managed' hot-remove option, a notificaiton is
+ * sent to the guest, which should perform a graceful teardown and notify
+ * the port of completion. At this point this listener unrealizes the device.
+ */
+static void cxl_vcs_ppb_unrealize_listener(DeviceListener *listener,
+        DeviceState *dev)
+{
+    CXLVCSSwitch *sw = container_of(listener, CXLVCSSwitch, listener);
+    CXLDownstreamPPB *dspppb;
+    CXLVPPBInfo *vppb_info;
+
+    for (int i = 0; i < sw->num_dsp_ppbs; i++) {
+        if (sw->dsp_ppbs[i] && (sw->dsp_ppbs[i]->dev == dev)) {
+            dspppb = sw->dsp_ppbs[i];
+            break;
+        }
+    }
+    if(!dspppb) {
+        return;
+    }
+
+    vppb_info =
+        sw->usp_ppbs[dspppb->bound_vcs_id]->info->vppbs[dspppb->bound_vppb_id];
+    object_unparent(OBJECT(dspppb->dev));
+    dspppb->dev = NULL;
+    dspppb->is_bound      = false;
+    dspppb->bound_vcs_id  = 0;
+    dspppb->bound_vppb_id = 0;
+    vppb_info->binding_status = CXL_VPPB_BINDING_STATUS_UNBOUND;
+    vppb_info->bound_port_id  = 0;
+    vppb_info->bound_ld_id    = CXL_INVALID_BOUND_LD_ID;
+    /* TODO: generate Virtual CXL Switch Event Record */
+}
+
+static void cxl_vcs_switch_complete(UserCreatable *uc, Error **errp)
+{
+    CXLVCSSwitch *sw = CXL_VCS_SWITCH(uc);
+    sw->listener.hide_device = cxl_vcs_hide_device_listener;
+    sw->listener.unrealize = cxl_vcs_ppb_unrealize_listener;
+    device_listener_register(&sw->listener);
+}
+
+static void vcs_get_usp_ppbs(Object *obj, Visitor *v, const char *name,
+                              void *opaque, Error **errp)
+{
+    CXLVCSSwitch *vcs = CXL_VCS_SWITCH(obj);
+    uint8_t val = vcs->num_usp_ppbs;
+    visit_type_uint8(v, name, &val, errp);
+}
+
+static void vcs_set_usp_ppbs(Object *obj, Visitor *v, const char *name,
+                              void *opaque, Error **errp)
+{
+    CXLVCSSwitch *vcs = CXL_VCS_SWITCH(obj);
+    uint8_t val;
+    if (!visit_type_uint8(v, name, &val, errp)) {
+        return;
+    }
+    vcs->num_usp_ppbs = val;
+}
+static void vcs_get_dsp_ppbs(Object *obj, Visitor *v, const char *name,
+                              void *opaque, Error **errp)
+{
+    CXLVCSSwitch *vcs = CXL_VCS_SWITCH(obj);
+    uint8_t val = vcs->num_dsp_ppbs;
+    visit_type_uint8(v, name, &val, errp);
+}
+
+static void vcs_set_dsp_ppbs(Object *obj, Visitor *v, const char *name,
+                              void *opaque, Error **errp)
+{
+    CXLVCSSwitch *vcs = CXL_VCS_SWITCH(obj);
+    uint8_t val;
+    if (!visit_type_uint8(v, name, &val, errp)) {
+        return;
+    }
+    vcs->num_dsp_ppbs = val;
+}
+
+static void cxl_vcs_class_init(ObjectClass *oc, const void *data)
+{
+    CXLVCSSwitchClass *cc = CXL_VCS_SWITCH_CLASS(oc);
+    UserCreatableClass *ucc = USER_CREATABLE_CLASS(oc);
+    ucc->complete = cxl_vcs_switch_complete;
+    ucc->can_be_deleted = cxl_vcs_switch_can_be_deleted;
+    cc->identify_switch_device = cxl_vcs_identify_switch_device;
+    cc->get_physical_port_state = cxl_vcs_get_physical_port_state;
+    cc->physical_port_control = cxl_vcs_physical_port_control;
+    cc->get_virtual_switch_info = cxl_vcs_get_virtual_switch_info;
+    cc->bind_vppb = cxl_vcs_bind_vppb;
+    cc->unbind_vppb = cxl_vcs_unbind_vppb;
+
+    object_class_property_add_bool(oc, "local-fm",
+                              cxl_vcs_get_local_fm, cxl_vcs_set_local_fm);
+    object_class_property_set_description(oc, "local-fm",
+        "true = FM authority (mctp connected guest), \
+        false = slave listener (IPC)");
+    object_class_property_add(oc, "usp-ppbs", "uint8",
+                          vcs_get_usp_ppbs, vcs_set_usp_ppbs,
+                          NULL, NULL);
+    object_class_property_set_description(oc, "usp-ppbs",
+        "Number of upstream ports in the switch");
+    object_class_property_add(oc, "dsp-ppbs", "uint8",
+                          vcs_get_dsp_ppbs, vcs_set_dsp_ppbs,
+                          NULL, NULL);
+    object_class_property_set_description(oc, "dsp-ppbs",
+        "Number of downstream ports in the switch");
+}
+
+static void cxl_vcs_instance_init(Object *obj)
+{
+    // TODO: Nothing here yet...
+}
+
+static void cxl_vcs_instance_finalize(Object *obj)
+{
+    // TODO: Nothing here yet...
+}
+
+static const InterfaceInfo cxl_vcs_interfaces[] = {
+    { TYPE_USER_CREATABLE },
+    { }
+};
+
+static const TypeInfo cxl_vcs_info = {
+    .name = TYPE_CXL_VCS_SWITCH,
+    .parent = TYPE_OBJECT,
+    .instance_size = sizeof(CXLVCSSwitch),
+    .class_size = sizeof(CXLVCSSwitchClass),
+    .class_init = cxl_vcs_class_init,
+    .instance_init = cxl_vcs_instance_init,
+    .instance_finalize = cxl_vcs_instance_finalize,
+    .interfaces = cxl_vcs_interfaces,
+};
+
+static void cxl_vcs_register(void)
+{
+    type_register_static(&cxl_vcs_info);
+}
+
+type_init(cxl_vcs_register)
diff --git a/hw/cxl/meson.build b/hw/cxl/meson.build
index ccad565c5c..c37563cd05 100644
--- a/hw/cxl/meson.build
+++ b/hw/cxl/meson.build
@@ -7,6 +7,7 @@ system_ss.add(when: 'CONFIG_CXL',
                    'cxl-cdat.c',
                    'cxl-events.c',
                    'switch-mailbox-cci.c',
+                   'cxl-vcs-switch.c',
                ),
                if_false: files(
                    'cxl-host-stubs.c',
diff --git a/include/hw/cxl/cxl_vcs_switch.h b/include/hw/cxl/cxl_vcs_switch.h
new file mode 100644
index 0000000000..6576870bf3
--- /dev/null
+++ b/include/hw/cxl/cxl_vcs_switch.h
@@ -0,0 +1,134 @@
+/*
+ * CXL VCS Capable Switch Object Header
+ *
+ * Copyright(C) 2026 University of Manchester
+ * Author: Joshua Lant <joshualant@gmail.com>.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ *
+ * SPDX-License-Identifier: GPL-v2-only
+ */
+
+#ifndef CXL_VCS_SWITCH_H
+#define CXL_VCS_SWITCH_H
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qom/object.h"
+#include "qom/object_interfaces.h"
+#include "hw/pci-bridge/cxl_upstream_port.h"
+#include "hw/pci-bridge/cxl_downstream_port.h"
+#include "include/hw/cxl/cxl_device.h"
+
+#define TYPE_CXL_VCS_SWITCH "cxl-vcs-switch"
+OBJECT_DECLARE_TYPE(CXLVCSSwitch, CXLVCSSwitchClass, CXL_VCS_SWITCH);
+
+#define CXL_VPPB_BINDING_STATUS_UNBOUND        0x00
+#define CXL_VPPB_BINDING_STATUS_IN_PROGRESS    0x01
+#define CXL_VPPB_BINDING_STATUS_BOUND_PORT     0x02
+#define CXL_VPPB_BINDING_STATUS_BOUND_LD       0x03
+#define CXL_VPPB_BINDING_STATUS_BOUND_PID      0x04
+
+#define CXL_VPPB_UNBIND_WAIT_FOR_LINK_DOWN      0x0
+#define CXL_VPPB_UNBIND_MANAGED_HOT_REMOVE      0x1
+#define CXL_VPPB_UNBIND_SURPRISE_HOT_REMOVE     0x2
+
+#define CXL_UNSUPPORTED_LD_ID 0xFFFF
+#define CXL_INVALID_BOUND_LD_ID 0xFF
+
+#define CXL_VCS_STATE_INVALID_ID        0xFF
+#define CXL_VCS_STATE_DISABLED 0x0
+#define CXL_VCS_STATE_ENABLED 0x1
+
+#define CXL_MAX_VCS_PORTS 0x8
+#define CXL_MAX_VPPB_PER_VCS 0x8
+#define CXL_MAX_USP_PPBS CXL_MAX_VCS_PORTS
+#define CXL_MAX_DSP_PPBS CXL_MAX_VPPB_PER_VCS
+
+#define CXL_MAX_PPBS (CXL_MAX_USP_PPBS + CXL_MAX_DSP_PPBS)
+#define VPPB_LIST_LIMIT 8
+
+struct CXLVCSSwitchClass {
+    ObjectClass parent_class;
+    void (*identify_switch_device)(CXLVCSSwitch *sw);
+    void (*get_physical_port_state)(CXLVCSSwitch *sw);
+    void (*physical_port_control)(CXLVCSSwitch *sw);
+    void (*get_virtual_switch_info)(CXLVCSSwitch *sw);
+    CXLRetCode (*bind_vppb)(CXLVCSSwitch *sw, uint8_t vcs_id, uint8_t vppb_id,
+            uint8_t dsppb_id, uint16_t ld_id);
+    CXLRetCode (*unbind_vppb)(CXLVCSSwitch *sw, uint8_t vcs_id, uint8_t vppb_id,
+            uint16_t option);
+};
+
+/* CXL r3.2 Table 7-32: Get Virtual CXL Switch Info VCS Info Block Format */
+typedef struct CXLVPPBInfo {
+        PCIDevice *dsp;
+        uint8_t binding_status;
+        uint8_t bound_port_id;
+        uint8_t bound_ld_id;
+        uint8_t rsv1;
+} CXLVPPBInfo;
+
+typedef struct CXLVCSInfoBlock {
+        uint8_t vcs_id;
+        uint8_t vcs_state;
+        uint8_t usp_id;
+        uint8_t num_vppbs;
+        struct CXLVPPBInfo *vppbs[CXL_MAX_VPPB_PER_VCS];
+} CXLVCSInfoBlock;
+
+/* Physical Upstream and Downstream PCI-PCI Bridge (PPBs) structs */
+typedef struct CXLUpstreamPPB {
+    CXLUpstreamPort   *usp;
+    struct CXLVCSInfoBlock *info;
+} CXLUpstreamPPB;
+
+typedef struct CXLDownstreamPPB {
+    DeviceState *dev;
+    PCIDevice   *pdev;
+    // store opts and json in case of simple device hiding.
+    QDict       *opts;
+    bool        from_json;
+    bool        is_bound;
+    uint8_t     bound_vcs_id;
+    uint8_t     bound_vppb_id;
+} CXLDownstreamPPB;
+
+typedef struct CXLVCSSwitch {
+    Object parent;
+    uint8_t num_usp_ppbs;
+    uint8_t num_dsp_ppbs;
+    uint8_t num_vppbs_per_vcs;
+    CXLCCI swcci;
+    CXLCCI mctpcci;
+    DeviceListener listener;
+    CXLUpstreamPPB *usp_ppbs[CXL_MAX_USP_PPBS];
+    CXLDownstreamPPB *dsp_ppbs[CXL_MAX_DSP_PPBS];
+    // true if FM is in this guest, false if remote FM.
+    bool local_fm;
+} CXLVCSSwitch;
+
+// Called by USP/DSP/EP CXL devices to build the VCS structures.
+void cxl_vcs_register_usp(CXLVCSSwitch *sw, CXLUpstreamPort *usp, Error **errp);
+void cxl_vcs_register_vppb(CXLVCSSwitch *sw, CXLUpstreamPort *usp,
+        CXLDownstreamPort *dsp, Error **errp);
+void cxl_vcs_register_dsppb(CXLVCSSwitch *sw, const QDict *opts,
+        bool from_json, Error **errp);
+void cxl_vcs_register_qdict_dsppb(CXLVCSSwitch *sw, const QDict *opts,
+        bool from_json, Error **errp);
+bool cxl_vcs_hide_device_listener(DeviceListener *listener,
+        const QDict *device_opts, bool from_json, Error **errp);
+
+
+// Called by the CCI mailbox utils for FMAPI control of VCS.
+CXLRetCode cxl_vcs_bind_vppb(CXLVCSSwitch *sw, uint8_t vcs_id, uint8_t vppb_id,
+        uint8_t dsppb_id, uint16_t ld_id);
+CXLRetCode cxl_vcs_unbind_vppb(CXLVCSSwitch *sw, uint8_t vcs_id,
+        uint8_t vppb_id, uint16_t option);
+void cxl_vcs_identify_switch_device(CXLVCSSwitch *sw);
+void cxl_vcs_get_physical_port_state(CXLVCSSwitch *sw);
+void cxl_vcs_physical_port_control(CXLVCSSwitch *sw);
+void cxl_vcs_get_virtual_switch_info(CXLVCSSwitch *sw);
+
+#endif /* CXL_VCS_SWITCH_H */
diff --git a/qapi/qom.json b/qapi/qom.json
index 6f5c9de0f0..f66ef6b68b 100644
--- a/qapi/qom.json
+++ b/qapi/qom.json
@@ -357,6 +357,23 @@
   'data': { 'chardev': 'str' },
   'if': 'CONFIG_VHOST_CRYPTO' }
 
+##
+# @CXLVCSSwitchProperties:
+#
+# Properties for cxl-vcs-switch objects.
+#
+# @usp-ppbs: number of physical upstream PPBs in the switch
+# @dsp-ppbs: number of physical downstream PPBs in the switch
+# @local-fm: true if this instance of qemu contains the MCTP device
+# (FM capabilities), false if vcs will listen for incoming traffic
+# from the remote FM.
+##
+{ 'struct': 'CXLVCSSwitchProperties',
+  'data': {
+    '*usp-ppbs': 'uint8',
+    '*dsp-ppbs': 'uint8',
+    '*local-fm':   'bool'} }
+
 ##
 # @DBusVMStateProperties:
 #
@@ -1194,6 +1211,7 @@
     'cryptodev-backend-lkcf',
     { 'name': 'cryptodev-vhost-user',
       'if': 'CONFIG_VHOST_CRYPTO' },
+    'cxl-vcs-switch',
     'dbus-vmstate',
     'filter-buffer',
     'filter-dump',
@@ -1272,6 +1290,7 @@
       'cryptodev-backend-lkcf':     'CryptodevBackendProperties',
       'cryptodev-vhost-user':       { 'type': 'CryptodevVhostUserProperties',
                                       'if': 'CONFIG_VHOST_CRYPTO' },
+      'cxl-vcs-switch':             'CXLVCSSwitchProperties',
       'dbus-vmstate':               'DBusVMStateProperties',
       'filter-buffer':              'FilterBufferProperties',
       'filter-dump':                'FilterDumpProperties',
-- 
2.47.3

[RFC QEMU PATCH 06/10] cxl-upstream-port: Add support for targeting a VCS switch

[Joshua Lant]

Allows an upstream port to be registered on the CLI as a VCS within a
cxl-vcs-switch. A VCS is defined as an upstream pci-pci bridge, and a set of
downstream virtual pci-pci bridges VPPBs, which form a single Virtual
Hierarchy within the switch.

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 hw/pci-bridge/cxl_upstream.c              | 20 ++++++++++++++++++++
 include/hw/pci-bridge/cxl_upstream_port.h |  2 ++
 2 files changed, 22 insertions(+)

diff --git a/hw/pci-bridge/cxl_upstream.c b/hw/pci-bridge/cxl_upstream.c
index 3fcfc2cde6..a91f721c9e 100644
--- a/hw/pci-bridge/cxl_upstream.c
+++ b/hw/pci-bridge/cxl_upstream.c
@@ -16,6 +16,8 @@
 #include "hw/pci/pcie.h"
 #include "hw/pci/pcie_port.h"
 #include "hw/pci-bridge/cxl_upstream_port.h"
+#include "hw/cxl/cxl_vcs_switch.h"
+
 /*
  * Null value of all Fs suggested by IEEE RA guidelines for use of
  * EU, OUI and CID
@@ -346,6 +348,22 @@ static void cxl_usp_realize(PCIDevice *d, Error **errp)
         goto err_cap;
     }
 
+    if (usp->vcs_name && usp->ppb != UINT8_MAX) {
+        Object *obj = object_resolve_path_component(
+                          object_get_objects_root(), usp->vcs_name);
+        if (!obj) {
+            error_setg(errp, "vcs '%s' not found", usp->vcs_name);
+            goto err_cap;
+        }
+        if (!object_dynamic_cast(obj, TYPE_CXL_VCS_SWITCH)) {
+            error_setg(errp, "'%s' is not a cxl-vcs-switch", usp->vcs_name);
+            goto err_cap;
+        }
+        cxl_vcs_register_usp(CXL_VCS_SWITCH(obj), usp, errp);
+        if (*errp) 
+            goto err_cap;
+    }
+
     return;
 
 err_cap:
@@ -372,6 +390,8 @@ static const Property cxl_upstream_props[] = {
     DEFINE_PROP_PCIE_LINK_WIDTH("x-width", CXLUpstreamPort,
                                 width, PCIE_LINK_WIDTH_16),
     DEFINE_PROP_BOOL("x-256b-flit", CXLUpstreamPort, flitmode, false),
+    DEFINE_PROP_STRING("vcs", CXLUpstreamPort, vcs_name),
+    DEFINE_PROP_UINT8("usppb", CXLUpstreamPort, ppb, UINT8_MAX),
 };
 
 static void cxl_upstream_class_init(ObjectClass *oc, const void *data)
diff --git a/include/hw/pci-bridge/cxl_upstream_port.h b/include/hw/pci-bridge/cxl_upstream_port.h
index ce248f3dca..05632286db 100644
--- a/include/hw/pci-bridge/cxl_upstream_port.h
+++ b/include/hw/pci-bridge/cxl_upstream_port.h
@@ -22,6 +22,8 @@ typedef struct CXLUpstreamPort {
 
     DOECap doe_cdat;
     uint64_t sn;
+    char *vcs_name;
+    uint8_t ppb;
 } CXLUpstreamPort;
 
 #endif /* CXL_SUP_H */
-- 
2.47.3

[RFC QEMU PATCH 07/10] cxl-downstream-port: Add support for VCS switching

[Joshua Lant]

When a downstream port is connnected to an upstream port which is
defined as a VCS within a cxl-vcs-switch, the port will automatically be
instantiated as a VPPB within that VCS. This downstream port will then
appear normally to the guest, but can be connected to a real downstream
PPB at runtime when the switch is issued a bind command by the Fabric
Manager.

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 hw/pci-bridge/cxl_downstream.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/hw/pci-bridge/cxl_downstream.c b/hw/pci-bridge/cxl_downstream.c
index 5322c46900..5e7ce6410d 100644
--- a/hw/pci-bridge/cxl_downstream.c
+++ b/hw/pci-bridge/cxl_downstream.c
@@ -14,11 +14,14 @@
 #include "hw/pci/pcie.h"
 #include "hw/pci/pcie_port.h"
 #include "hw/pci-bridge/cxl_downstream_port.h"
+#include "hw/pci-bridge/cxl_upstream_port.h"
 #include "hw/qdev-properties.h"
 #include "hw/qdev-properties-system.h"
 #include "hw/cxl/cxl.h"
 #include "hw/cxl/cxl_port.h"
 #include "qapi/error.h"
+#include "hw/cxl/cxl_vcs_switch.h"
+
 
 typedef struct CXLDownstreamPort {
     /*< private >*/
@@ -199,6 +202,14 @@ static void cxl_dsp_realize(PCIDevice *d, Error **errp)
                          PCI_BASE_ADDRESS_MEM_TYPE_64,
                      component_bar);
 
+    PCIDevice *parent = pci_bridge_get_device(pci_get_bus(PCI_DEVICE(d)));
+    if (parent && object_dynamic_cast(OBJECT(parent), TYPE_CXL_USP)) {
+        CXLUpstreamPort *usp = CXL_USP(parent);
+        if (usp->vcs_name) {
+            cxl_vcs_register_vppb(usp->swcci.vcs, usp, dsp, errp);
+        }
+    }
+
     return;
 
  err_chassis:
-- 
2.47.3

[RFC QEMU PATCH 08/10] cxl-cci-mailbox: Add support for targeting a VCS switch

[Joshua Lant]

Allows a cci mailbox and mctp device to target a cxl-vcs-switch. The
target needs to be changed from a PCIDevice to a more generic Object,
since the target can now also be a cxl-vcs-switch, as well as USP or
type3 device. The VCS is an object which is composed of many
PCI devices, but cannot be classed as one itself (since it is busless).

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 hw/cxl/cxl-mailbox-utils.c  | 44 +++++++++++++++++++++++--------------
 hw/cxl/switch-mailbox-cci.c | 33 ++++++++++++++++++++++------
 hw/usb/dev-mctp.c           | 23 ++++++++++++++++---
 include/hw/cxl/cxl_device.h | 10 ++++++++-
 4 files changed, 83 insertions(+), 27 deletions(-)

diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index f12e42a648..779138cdc8 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -4600,22 +4600,24 @@ int cxl_process_cci_message(CXLCCI *cci, uint8_t set, uint8_t cmd,
     }
 
     /* forbid any selected commands while the media is disabled */
-    if (object_dynamic_cast(OBJECT(cci->d), TYPE_CXL_TYPE3)) {
-        cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate;
-
-        if (cxl_dev_media_disabled(cxl_dstate)) {
-            if (h == cmd_events_get_records ||
-                h == cmd_ccls_get_partition_info ||
-                h == cmd_ccls_set_lsa ||
-                h == cmd_ccls_get_lsa ||
-                h == cmd_logs_get_log ||
-                h == cmd_media_get_poison_list ||
-                h == cmd_media_inject_poison ||
-                h == cmd_media_clear_poison ||
-                h == cmd_sanitize_overwrite ||
-                h == cmd_firmware_update_transfer ||
-                h == cmd_firmware_update_activate) {
-                return CXL_MBOX_MEDIA_DISABLED;
+    if(cci->d) {
+        if (object_dynamic_cast(OBJECT(cci->d), TYPE_CXL_TYPE3)) {
+            cxl_dstate = &CXL_TYPE3(cci->d)->cxl_dstate;
+
+            if (cxl_dev_media_disabled(cxl_dstate)) {
+                if (h == cmd_events_get_records ||
+                    h == cmd_ccls_get_partition_info ||
+                    h == cmd_ccls_set_lsa ||
+                    h == cmd_ccls_get_lsa ||
+                    h == cmd_logs_get_log ||
+                    h == cmd_media_get_poison_list ||
+                    h == cmd_media_inject_poison ||
+                    h == cmd_media_clear_poison ||
+                    h == cmd_sanitize_overwrite ||
+                    h == cmd_firmware_update_transfer ||
+                    h == cmd_firmware_update_activate) {
+                    return CXL_MBOX_MEDIA_DISABLED;
+                }
             }
         }
     }
@@ -4909,3 +4911,13 @@ void cxl_initialize_usp_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf,
     cci->intf = intf;
     cxl_init_cci(cci, payload_max);
 }
+
+void cxl_initialize_vcs_mctpcci(CXLCCI *cci, CXLVCSSwitch *vcs, DeviceState *d,
+        DeviceState *intf, size_t payload_max)
+{
+    cxl_copy_cci_commands(cci, cxl_cmd_set_usp_mctp);
+    cci->vcs = vcs;
+    cci->d = d;
+    cci->intf = intf;
+    cxl_init_cci(cci, payload_max);
+}
diff --git a/hw/cxl/switch-mailbox-cci.c b/hw/cxl/switch-mailbox-cci.c
index 223f220433..73f76484cc 100644
--- a/hw/cxl/switch-mailbox-cci.c
+++ b/hw/cxl/switch-mailbox-cci.c
@@ -16,6 +16,7 @@
 #include "qemu/module.h"
 #include "hw/qdev-properties.h"
 #include "hw/cxl/cxl.h"
+#include "include/hw/cxl/cxl_vcs_switch.h"
 
 #define CXL_SWCCI_MSIX_MBOX 3
 
@@ -32,17 +33,26 @@ static void cswbcci_realize(PCIDevice *pci_dev, Error **errp)
     CXLDeviceState *cxl_dstate = &cswmb->cxl_dstate;
     CXLDVSECRegisterLocator *regloc_dvsec;
     CXLUpstreamPort *usp;
+    CXLVCSSwitch *vcs = 0;
 
     if (!cswmb->target) {
-        error_setg(errp, "Target not set");
+        error_setg(errp, "No target or vcs has been set...");
         return;
     }
-    usp = CXL_USP(cswmb->target);
 
     pcie_endpoint_cap_init(pci_dev, 0x80);
     cxl_cstate->dvsec_offset = 0x100;
     cxl_cstate->pdev = pci_dev;
-    cswmb->cci = &usp->swcci;
+    if (object_dynamic_cast(cswmb->target, TYPE_CXL_VCS_SWITCH)) {
+	vcs = CXL_VCS_SWITCH(cswmb->target);
+	cswmb->cci = &vcs->swcci;
+    } else if (object_dynamic_cast(cswmb->target, TYPE_CXL_USP)) {
+	usp = CXL_USP(cswmb->target);
+	cswmb->cci = &usp->swcci;
+    } else {
+	error_setg(errp, "Target must be a VCS switch or USP");
+	return;
+    }
     cxl_device_register_block_init(OBJECT(pci_dev), cxl_dstate, cswmb->cci);
     pci_register_bar(pci_dev, 0,
                      PCI_BASE_ADDRESS_SPACE_MEMORY |
@@ -57,9 +67,18 @@ static void cswbcci_realize(PCIDevice *pci_dev, Error **errp)
                                REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
                                REG_LOC_DVSEC_REVID, (uint8_t *)regloc_dvsec);
 
-    cxl_initialize_mailbox_swcci(cswmb->cci, DEVICE(pci_dev),
-                                 DEVICE(cswmb->target),
-                                 CXL_MAILBOX_MAX_PAYLOAD_SIZE);
+    if(vcs) {
+	if(vcs->usp_ppbs[0]) {
+	    cxl_initialize_vcs_mctpcci(cswmb->cci,
+		vcs, DEVICE(vcs->usp_ppbs[0]->usp), DEVICE(pci_dev), CXL_MAILBOX_MAX_PAYLOAD_SIZE);
+	} else {
+	    error_setg(errp, "The VCS requires a master USP...");
+	}
+    } else {
+	cxl_initialize_mailbox_swcci(cswmb->cci, DEVICE(pci_dev),
+				     DEVICE(CXL_USP(cswmb->target)),
+				     CXL_MAILBOX_MAX_PAYLOAD_SIZE);
+    }
 }
 
 static void cswmbcci_exit(PCIDevice *pci_dev)
@@ -69,7 +88,7 @@ static void cswmbcci_exit(PCIDevice *pci_dev)
 
 static const Property cxl_switch_cci_props[] = {
     DEFINE_PROP_LINK("target", CSWMBCCIDev,
-                     target, TYPE_CXL_USP, PCIDevice *),
+                     target, TYPE_OBJECT, Object *),
 };
 
 static void cswmbcci_class_init(ObjectClass *oc, const void *data)
diff --git a/hw/usb/dev-mctp.c b/hw/usb/dev-mctp.c
index aafb9e7e96..792a375245 100644
--- a/hw/usb/dev-mctp.c
+++ b/hw/usb/dev-mctp.c
@@ -25,6 +25,7 @@
 #include "hw/usb.h"
 #include "hw/usb/desc.h"
 #include "net/mctp.h"
+#include "include/hw/cxl/cxl_vcs_switch.h"
 
 /* TODO: Move to header */
 
@@ -77,7 +78,7 @@ enum cxl_dev_type {
 
 typedef struct USBCXLMCTPState {
     USBDevice dev;
-    PCIDevice *target;
+    Object *target;
     CXLCCI *cci;
     enum cxl_dev_type type;
     USBPacket *cached_tohost;
@@ -600,12 +601,28 @@ static void usb_cxl_mctp_realize(USBDevice *dev, Error **errp)
         return;
     }
 
+    if (object_dynamic_cast(OBJECT(s->target), TYPE_CXL_VCS_SWITCH)) {
+        CXLVCSSwitch *sw = CXL_VCS_SWITCH(s->target);
+
+        s->type = cxl_switch;
+        s->cci = &sw->mctpcci;
+	s->cci->vcs = sw;
+
+	if(sw->usp_ppbs[0]) {
+	    cxl_initialize_vcs_mctpcci(s->cci, CXL_VCS_SWITCH(s->target),
+		DEVICE(sw->usp_ppbs[0]->usp), DEVICE(dev), MCTP_CXL_MAILBOX_BYTES);
+	} else {
+	    error_setg(errp, "No master USP associated with the VCS");
+	}
+        return;
+    }
+
     error_setg(errp, "Unhandled target type for CXL MCTP EP");
 }
 
 static const Property usb_cxl_mctp_properties[] = {
-    DEFINE_PROP_LINK("target", USBCXLMCTPState, target, TYPE_PCI_DEVICE,
-                     PCIDevice *),
+    DEFINE_PROP_LINK("target", USBCXLMCTPState, target, TYPE_OBJECT,
+                     Object *),
 };
 
 static void usb_cxl_mctp_class_initfn(ObjectClass *klass, const void *data)
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 6158560e6c..5d9f537a03 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -143,6 +143,7 @@ typedef enum {
 } CXLDSMASFlags;
 
 typedef struct CXLCCI CXLCCI;
+typedef struct CXLVCSSwitch CXLVCSSwitch;
 typedef struct cxl_device_state CXLDeviceState;
 struct cxl_cmd;
 typedef CXLRetCode (*opcode_handler)(const struct cxl_cmd *cmd,
@@ -212,6 +213,8 @@ typedef struct CXLCCI {
     DeviceState *d;
     /* Pointer to the device hosting the protocol conversion */
     DeviceState *intf;
+    /* Pointer to the vcs hosting the CCI (if used) */
+    CXLVCSSwitch *vcs;
     bool initialized;
 } CXLCCI;
 
@@ -349,6 +352,10 @@ void cxl_initialize_t3_ld_cci(CXLCCI *cci, DeviceState *d,
 void cxl_initialize_usp_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf,
                                 size_t payload_max);
 
+void cxl_initialize_vcs_mctpcci(CXLCCI *cci, CXLVCSSwitch *vcs,
+	DeviceState *d, DeviceState *intf, size_t payload_max);
+
+
 #define cxl_device_cap_init(dstate, reg, cap_id, ver)                      \
     do {                                                                   \
         uint32_t *cap_hdrs = dstate->caps_reg_state32;                     \
@@ -820,9 +827,10 @@ struct CXLType3Class {
                           uint8_t *data);
 };
 
+
 struct CSWMBCCIDev {
     PCIDevice parent_obj;
-    PCIDevice *target;
+    Object *target;
     CXLComponentState cxl_cstate;
     CXLDeviceState cxl_dstate;
     CXLCCI *cci;
-- 
2.47.3

[RFC QEMU PATCH 09/10] cxl-mailbox-utils: Add support for VCS bind/unbind commands.

[Joshua Lant]

Adds support for FMAPI commands 0x5201 and 0x5202, for more
information see CXL Specification v3.2 (7.6.7.2.2 and 7.6.7.2.3).

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 hw/cxl/cxl-mailbox-utils.c | 81 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 80 insertions(+), 1 deletion(-)

diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index 779138cdc8..cf6cb65eb6 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -27,6 +27,7 @@
 #include "system/hostmem.h"
 #include "qemu/range.h"
 #include "qapi/qapi-types-cxl.h"
+#include "hw/cxl/cxl_vcs_switch.h"
 
 #define CXL_CAPACITY_MULTIPLIER   (256 * MiB)
 #define CXL_DC_EVENT_LOG_SIZE 8
@@ -121,6 +122,9 @@ enum {
         #define IDENTIFY_SWITCH_DEVICE      0x0
         #define GET_PHYSICAL_PORT_STATE     0x1
         #define PHYSICAL_PORT_CONTROL       0x2
+    VIRTUAL_SWITCH = 0x52,
+        #define BIND_VPPB     0x1
+        #define UNBIND_VPPB       0x2
     TUNNEL = 0x53,
         #define MANAGEMENT_COMMAND     0x0
     FMAPI_DCD_MGMT = 0x56,
@@ -821,6 +825,51 @@ static CXLRetCode cmd_physical_port_control(const struct cxl_cmd *cmd,
     }
 }
 
+/* CXL r3.2 Section 7.6.7.2.2: Bind vPPB (Opcode 5201h) */
+static CXLRetCode cmd_bind_vppb(const struct cxl_cmd *cmd,
+                                            uint8_t *payload_in,
+                                            size_t len_in,
+                                            uint8_t *payload_out,
+                                            size_t *len_out,
+                                            CXLCCI *cci)
+{
+    struct cxl_fmapi_bind_vppb_req_pl {
+        uint8_t vcs_id;
+        uint8_t vppb_id;
+        uint8_t physp_id;
+        uint8_t rsvd;
+        uint16_t ld_id;
+    } QEMU_PACKED *in = (void *)payload_in;
+    if(!cci->vcs) {
+        return CXL_MBOX_UNSUPPORTED;
+    }
+    // Multi-logical device currently not supported (force 0xFFFF).
+    if(in->ld_id != CXL_UNSUPPORTED_LD_ID) {
+        return CXL_MBOX_INVALID_INPUT;
+    }
+    return cxl_vcs_bind_vppb(cci->vcs, in->vcs_id, in->vppb_id,
+            in->physp_id, CXL_UNSUPPORTED_LD_ID);
+}
+
+/* CXL r3.2 Section 7.6.7.2.3: Unbind vPPB (Opcode 5202h) */
+static CXLRetCode cmd_unbind_vppb(const struct cxl_cmd *cmd,
+                                            uint8_t *payload_in,
+                                            size_t len_in,
+                                            uint8_t *payload_out,
+                                            size_t *len_out,
+                                            CXLCCI *cci)
+{
+    struct cxl_fmapi_bind_vppb_req_pl {
+        uint8_t vcs_id;
+        uint8_t vppb_id;
+        uint16_t option;
+    } QEMU_PACKED *in = (void *)payload_in;
+    if(!cci->vcs) {
+        return CXL_MBOX_UNSUPPORTED;
+    }
+    return cxl_vcs_unbind_vppb(cci->vcs, in->vcs_id, in->vppb_id, in->option);
+}
+
 /* CXL r3.1 Section 8.2.9.1.2: Background Operation Status (Opcode 0002h) */
 static CXLRetCode cmd_infostat_bg_op_sts(const struct cxl_cmd *cmd,
                                          uint8_t *payload_in,
@@ -4903,6 +4952,36 @@ static const struct cxl_cmd cxl_cmd_set_usp_mctp[256][256] = {
                                      cmd_tunnel_management_cmd, ~0, 0 },
 };
 
+static const struct cxl_cmd cxl_cmd_set_vcs_mctp[256][256] = {
+    [INFOSTAT][IS_IDENTIFY] = { "IDENTIFY", cmd_infostat_identify, 0, 0 },
+    [INFOSTAT][GET_RESPONSE_MSG_LIMIT] = { "GET_RESPONSE_MSG_LIMIT",
+                                           cmd_get_response_msg_limit, 0, 0 },
+    [INFOSTAT][SET_RESPONSE_MSG_LIMIT] = { "SET_RESPONSE_MSG_LIMIT",
+                                           cmd_set_response_msg_limit, 1, 0 },
+    [INFOSTAT][BACKGROUND_OPERATION_STATUS] = { "BACKGROUND_OPERATION_STATUS",
+        cmd_infostat_bg_op_sts, 0, 0 },
+    [INFOSTAT][BACKGROUND_OPERATION_ABORT] = { "BACKGROUND_OPERATION_ABORT",
+        cmd_infostat_bg_op_abort, 0, 0 },
+    [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 },
+    [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8,
+                         CXL_MBOX_IMMEDIATE_POLICY_CHANGE },
+    [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported,
+                              0, 0 },
+    [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 },
+    [PHYSICAL_SWITCH][IDENTIFY_SWITCH_DEVICE] = { "IDENTIFY_SWITCH_DEVICE",
+        cmd_identify_switch_device, 0, 0 },
+    [PHYSICAL_SWITCH][GET_PHYSICAL_PORT_STATE] = { "SWITCH_PHYSICAL_PORT_STATS",
+        cmd_get_physical_port_state, ~0, 0 },
+    [PHYSICAL_SWITCH][PHYSICAL_PORT_CONTROL] = { "SWITCH_PHYSICAL_PORT_CONTROL",
+        cmd_physical_port_control, 2, 0 },
+    [VIRTUAL_SWITCH][BIND_VPPB] = { "BIND_VPPB",
+        cmd_bind_vppb, ~0, 0 },
+    [VIRTUAL_SWITCH][UNBIND_VPPB] = { "UNBIND_VPPB",
+        cmd_unbind_vppb, ~0, 0 },
+    [TUNNEL][MANAGEMENT_COMMAND] = { "TUNNEL_MANAGEMENT_COMMAND",
+                                     cmd_tunnel_management_cmd, ~0, 0 },
+};
+
 void cxl_initialize_usp_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf,
                                 size_t payload_max)
 {
@@ -4915,7 +4994,7 @@ void cxl_initialize_usp_mctpcci(CXLCCI *cci, DeviceState *d, DeviceState *intf,
 void cxl_initialize_vcs_mctpcci(CXLCCI *cci, CXLVCSSwitch *vcs, DeviceState *d,
         DeviceState *intf, size_t payload_max)
 {
-    cxl_copy_cci_commands(cci, cxl_cmd_set_usp_mctp);
+    cxl_copy_cci_commands(cci, cxl_cmd_set_vcs_mctp);
     cci->vcs = vcs;
     cci->d = d;
     cci->intf = intf;
-- 
2.47.3

[RFC QEMU PATCH 10/10] cxl-mailbox-utils: Add support for VCS Get Virtual CXL Switch Info command.

[Joshua Lant]

Adds support for FMAPI command 0x5200, for more
information see CXL Specification v3.2 (7.6.7.2.1).

Signed-off-by: Joshua Lant <joshualant@gmail.com>
---
 hw/cxl/cxl-mailbox-utils.c | 84 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index cf6cb65eb6..825ef35179 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -123,6 +123,7 @@ enum {
         #define GET_PHYSICAL_PORT_STATE     0x1
         #define PHYSICAL_PORT_CONTROL       0x2
     VIRTUAL_SWITCH = 0x52,
+        #define GET_VIRTUAL_SWITCH_INFO      0x0
         #define BIND_VPPB     0x1
         #define UNBIND_VPPB       0x2
     TUNNEL = 0x53,
@@ -825,6 +826,87 @@ static CXLRetCode cmd_physical_port_control(const struct cxl_cmd *cmd,
     }
 }
 
+/* CXL r3.2 Section 7.6.7.2.1: Get Virtual CXL Switch Info (Opcode 5200h) */
+static CXLRetCode cmd_get_virtual_switch_info(const struct cxl_cmd *cmd,
+                                            uint8_t *payload_in,
+                                            size_t len_in,
+                                            uint8_t *payload_out,
+                                            size_t *len_out,
+                                            CXLCCI *cci)
+{
+    struct cxl_fmapi_get_virtual_switch_info_req_pl {
+        uint8_t start_vppb;
+        uint8_t vppb_list_limit;
+        uint8_t number_of_vcs;
+        uint8_t vcs_id_list[];
+    } QEMU_PACKED *in = (void *)payload_in;
+    struct vppb_info {
+            uint8_t binding_status;
+            uint8_t bound_port_id;
+            uint8_t bound_ld_id;
+            uint8_t rsv1;
+    } QEMU_PACKED;
+    struct vcs_info_block {
+            uint8_t vcs_id;
+            uint8_t vcs_state;
+            uint8_t usp_id;
+            uint8_t num_vppbs;
+            struct vppb_info vppbs[];
+    } QEMU_PACKED;
+    struct cxl_fmapi_get_virtual_switch_info_rsp_pl {
+            uint8_t num_vcs;
+            uint8_t rsv1[3];
+            struct vcs_info_block vcs_info_list[];
+    } QEMU_PACKED *out = (void *)payload_out;
+
+    uint8_t num_vcs_ret = cci->vcs->num_usp_ppbs;
+    uint8_t num_vppbs_ret;
+    ssize_t len_out_tmp = *len_out;
+    len_out_tmp = sizeof(struct cxl_fmapi_get_virtual_switch_info_rsp_pl) +
+        (num_vcs_ret * sizeof(*out->vcs_info_list));
+    for(int i = 0; i < num_vcs_ret; i++) {
+        //num_vppbs_ret = get_num_vppbs_in_vcs(cci->d, i);
+        num_vppbs_ret = cci->vcs->usp_ppbs[i]->info->num_vppbs;
+        //vPPB List Entry Count=min(vPPB List Limit, Number of vPPBs).
+        len_out_tmp = (num_vppbs_ret <= VPPB_LIST_LIMIT) ?
+                len_out_tmp + (num_vppbs_ret * sizeof(struct vppb_info)) :
+                len_out_tmp + (VPPB_LIST_LIMIT * sizeof(struct vppb_info));
+    }
+    *len_out = len_out_tmp;
+
+    out->num_vcs = num_vcs_ret;
+    out->rsv1[0] = 0;
+    out->rsv1[1] = 0;
+    out->rsv1[2] = 0;
+
+    uint8_t *ptr = (uint8_t *)out->vcs_info_list;
+
+    for (int i = 0; i < num_vcs_ret; i++) {
+        CXLVCSInfoBlock *info = cci->vcs->usp_ppbs[i]->info;
+        uint8_t num_vppbs = info->num_vppbs;
+        uint8_t vppb_count = MIN(num_vppbs - in->start_vppb,
+                                 in->vppb_list_limit);
+
+        /* write fixed vcs_info_block header (4 bytes) */
+        *ptr++ = info->vcs_id;
+        *ptr++ = info->vcs_state;
+        *ptr++ = info->usp_id;
+        *ptr++ = num_vppbs;
+
+        /* write vppb entries for current vcs */
+        for (int j = in->start_vppb; j < in->start_vppb + vppb_count; j++) {
+            struct vppb_info *v = (struct vppb_info *)ptr;
+            v->binding_status = info->vppbs[j]->binding_status;
+            v->bound_port_id  = info->vppbs[j]->bound_port_id;
+            v->bound_ld_id    = info->vppbs[j]->bound_ld_id;
+            v->rsv1           = 0;
+            ptr += sizeof(struct vppb_info);
+        }
+    }
+
+    return CXL_MBOX_SUCCESS;
+}
+
 /* CXL r3.2 Section 7.6.7.2.2: Bind vPPB (Opcode 5201h) */
 static CXLRetCode cmd_bind_vppb(const struct cxl_cmd *cmd,
                                             uint8_t *payload_in,
@@ -4974,6 +5056,8 @@ static const struct cxl_cmd cxl_cmd_set_vcs_mctp[256][256] = {
         cmd_get_physical_port_state, ~0, 0 },
     [PHYSICAL_SWITCH][PHYSICAL_PORT_CONTROL] = { "SWITCH_PHYSICAL_PORT_CONTROL",
         cmd_physical_port_control, 2, 0 },
+    [VIRTUAL_SWITCH][GET_VIRTUAL_SWITCH_INFO] = { "GET_VIRTUAL_SWITCH_INFO",
+        cmd_get_virtual_switch_info, ~0, 0 },
     [VIRTUAL_SWITCH][BIND_VPPB] = { "BIND_VPPB",
         cmd_bind_vppb, ~0, 0 },
     [VIRTUAL_SWITCH][UNBIND_VPPB] = { "UNBIND_VPPB",
-- 
2.47.3