Re: [PATCH v1 0/4] virtio-msg transport layer

["Edgar E. Iglesias" <edgar.iglesias@amd.com>]

On Wed, Feb 25, 2026 at 02:53:58PM +0000, Bertrand Marquis wrote:
> Hi,
> 
> > On 25 Feb 2026, at 15:49, Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Feb 25, 2026 at 02:45:35PM +0000, Parav Pandit wrote:
> >>
> >>> From: Bertrand Marquis <Bertrand.Marquis@arm.com>
> >>> Sent: 25 February 2026 04:06 PM
> >>>
> >>> Hi Parav,
> >>>
> >>>> On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote:
> >>>>
> >>>>>
> >>>>> From: Manivannan Sadhasivam <mani@kernel.org>
> >>>>> Sent: 25 February 2026 03:37 PM
> >>>>>
> >>>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote:
> >>>>>> Hi Manivannan,
> >>>>>>
> >>>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote:
> >>>>>>>
> >>>>>>> Hi Bertrand,
> >>>>>>>
> >>>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote:
> >>>>>>>> Hi Parav,
> >>>>>>>>
> >>>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>>>>>>> Sent: 20 February 2026 05:25 AM
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote:
> >>>>>>>>>>> Hi Bill,
> >>>>>>>>>>>
> >>>>>>>>>>>> From: Bill Mills <bill.mills@linaro.org>
> >>>>>>>>>>>> Sent: 26 January 2026 10:02 PM
> >>>>>>>>>>>>
> >>>>>>>>>>>> This series adds the virtio-msg transport layer.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in
> >>>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one
> >>>>>>>>>>>> more transport that performs its transport layer operations by sending and
> >>>>>>>>>>>> receiving messages.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers:
> >>>>>>>>>>>> * common / device level
> >>>>>>>>>>>> * bus level
> >>>>>>>>>>>>
> >>>>>>>>>>>> The common / device level defines the messages exchanged between the driver
> >>>>>>>>>>>> and a device. This common part should lead to a common driver holding most
> >>>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations.
> >>>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport
> >>>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device
> >>>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the
> >>>>>>>>>>>> main focus of this version of the patch series.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines
> >>>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and
> >>>>>>>>>>>> receive operations. A number of bus implementations are envisioned,
> >>>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations
> >>>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card
> >>>>>>>>>>>> and its driver.
> >>>>>>>>>>>>
> >>>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations
> >>>>>>>>>>>> doing the same thing in different ways for no good reason. However the
> >>>>>>>>>>>> different environments will require different things. Instead of trying to
> >>>>>>>>>>>> anticipate all needs and provide something very abstract, we think
> >>>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time,
> >>>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to
> >>>>>>>>>>>> standardize a bus level message for that.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I would review more, had first round of sparse review.
> >>>>>>>>>>> Please find few comments/questions below.
> >>>>>>>>>>
> >>>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and
> >>>>>>>>>> then add performance features on top as appropriate.
> >>>>>>>>> Sounds good. Simple but complete is needed.
> >>>>>>>>
> >>>>>>>> Agree.
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> So one way to address some of these comments is to show how
> >>>>>>>>>> they can be addressed with a feature bit down the road.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header.
> >>>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices.
> >>>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field.
> >>>>>>>>>>>
> >>>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer.
> >>>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data.
> >>>>>>>>>>> Needs 32-bits.
> >>>>>>>>>>>
> >>>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY)
> >>>>>>>>>>> But more below.
> >>>>>>>>>>>
> >>>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to
> >>>>> operate
> >>>>>>>>>> the virtqueues.
> >>>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response?
> >>>>>>>>>>
> >>>>>>>>>> surely this can be an optional transport feature bit.
> >>>>>>>>>>
> >>>>>>>>> How is this optional?
> >>>>>>>>
> >>>>>>>> As said in a previous mail, we have messages already for that.
> >>>>>>>> Please confirm if that answer your question.
> >>>>>>>>
> >>>>>>>>> How can one implement a transport without defining the basic data transfer semantics?
> >>>>>>>>
> >>>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O.
> >>>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue.
> >>>>>>>>
> >>>>>>>
> >>>>>>> I'd love to have this transport over PCI because it addresses the shortcomings
> >>>>>>> of the existing PCI transport which just assumes that every config space access\
> >>>>>>> is trap and emulate.
> >>>>>>
> >>>>>> Agree and AMD did exactly that in their demonstrator.
> >>>>>> I will give you answers here as i know them but Edgar will probably give you more
> >>>>>> details (and probably fix my mistakes).
> >>>>>>
> >>>>>>>
> >>>>>>> But that being said, I somewhat agree with Parav that we should define the bus
> >>>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For
> >>>>>>> instance, if we try to use this transport over PCI, we've got questions like:
> >>>>>>>
> >>>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with
> >>>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs?
> >>>>>>
> >>>>>> One bus is appearing as one pci device with its own Vendor ID,
> >>>>>>
> >>>>>
> >>>>> What should be the 'own Vendor ID' here?
> >>>>>
> >>>>> The existing virtio-pci driver binds to all devices with the Vendor ID of
> >>>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own
> >>>>> VID for exposing the Virtio devices? That would mean, the drivers on the host
> >>>>> need update as well, which will not scale.
> >>>>>
> >>>>> It would be good if the existing virtio-pci devices can use this new transport
> >>>>> with only device side modifications.
> >>>>>
> >>>>>>>
> >>>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config
> >>>>>>> space or through some other means?
> >>>>>>
> >>>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI
> >>>>>> device (ending up being memory shared between both sides)
> >>>>>>
> >>>>>
> >>>>> What should be the BAR number and size?
> >>>>>
> >>>>>>>
> >>>>>>> 3. How the notification be delivered from the device to the host? Through
> >>>>>>> INT-X/MSI/MSI-X or even polling?
> >>>>>>
> >>>>>> Notifications are delivered through MSI.
> >>>>>>
> >>>>>
> >>>>> So no INT-X or MSI-X? Why so?
> >>>>>
> >>>>> Anyhow, my objective is not to get answers for my above questions here in this
> >>>>> thread, but to state the reality that it would be hard for us to make use of
> >>>>> this new transport without defining the bus implementation.
> >>>>>
> >>>> +1 to most of the points that Manivannan explained.
> >>>>
> >>>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet
> >>> another interface for _Everything_ that already exists.
> >>>> and device is still have to implement all the existing things because the device does not know which driver will operate.
> >>>>
> >>>> And that too some register based inefficient interface.
> >>>> Just to reset the device one needs to fully setup the new message interface but device still have to be working.
> >>>> That defeats the whole purpose of reset_1 and reset_2 in the device.
> >>>>
> >>>> This does not bring anything better for the PCI devices at all.
> >>>>
> >>>> A transport binding should be defined for the bus binding.
> >>>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are.
> >>>>
> >>>> If we are creating something brand-new, for PCI the only thing needed is:
> >>>> 1. Reset the device
> >>>> 2. Create an admin virtqueue
> >>>> 3. Transport everything needed through this virtqueue including features, configs, control.
> >>>>
> >>>> And this will work for any other bus or msg based too given only contract needed is to creating the aq.
> >>>
> >>> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain.
> >>>
> >>> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI".
> >>>
> >>> When the virtio-msg bus over PCI it will communicate through this device with an external
> >>> system connected through the PCI bus.
> >>> The driver will enumerate virtio devices available behind this bus and register them so that
> >>> the corresponding virtio drivers are probed for them.
> >>> All virtio-msg messages required to communicate with those devices will be transferred through
> >>> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the
> >>> virtqueues with all the devices on the bus.
> >>>
> >>> So the PCI device is not one virtio device but one bus behind which there can be many devices.
> >>>
> >>> Is this making the concept a bit clearer ?
> >>>
> >> Yes. This makes a lot of sense now.
> >>
> >> This is a virtio-msg-transport device that needs its own device id in the table.
> >> And its binding to the PCI transport.
> >
> > ok. how about an rfc of that idea on the list?
> 
> 
> I will let Edgar answer on this.


Yes, I agree it makes sense to document the virtio-msg PCI bus.

We've been looking at a virtio-msg PCI bus for several reasons. One is to
share devices between two SoCs connected over PCI, e.g., an x86 host and an
ARM endpoint.

To enable virtio-msg, we need shared memory for FIFOs plus notifications in
both directions. We also need DMA for virtqueue access.

In the simple case where the endpoint shares devices to the host, we have a
BAR0 with notification registers (host to EP) and a prefetchable BAR1 with RAM
for FIFOs. We implement software FIFOs in RAM and move virtio-msg messages
over these. When the host needs to notify the EP, it writes to the
notification registers in BAR0. When the EP needs to notify the host, it
raises an interrupt (we're using MSI-X now, but it could be MSI or INTX).

When the device model running on the EP needs to access the host's VQs, it
uses ordinary PCI DMA.

When the host wants to share a device with the EP, we need an additional
large BAR2 (e.g., 4 GB) to expose EP memory to the host for DMA. The virtio-msg
queues work the same way as in the first scenario, but DMA to the VQs is done
over BAR2. This is still a work in progress, but we have a setup that
demonstrates EP-to-host sharing.

In our setup, we're using virtio-msg to share devices between host kernels,
e.g., dom0 on the x86 and dom0 on the ARM side. To share with guests we're
currently re-sharing using virtio-pci to support unmodified
non-virtio-msg-capable guests. If we wanted to do virtio-msg end-to-end
between guests, we'd need to multiply the PCI functions to address the IOMMU
issue that Parav mentioned, either by using multiple functions or multiple
VFs (SR-IOV) (one per guest). This is something we've talked about but not
yet tried.

A second reason for looking at PCI was to provide a hypervisor-neutral and
easy way to access and try virtio-msg. So we created a QEMU model of the
EP-to-host sharing mechanisms described above. This is what we've submitted
to QEMU as the virtio-msg AMP PCI device.

https://lore.kernel.org/qemu-devel/20260224155721.612314-1-edgar.iglesias@gmail.com/