Re: [PATCH v1 0/4] virtio-msg transport layer

["Michael S. Tsirkin" <mst@redhat.com>]

On Wed, Feb 25, 2026 at 10:55:05AM +0000, Bertrand Marquis wrote:
> Hi
> 
> > On 25 Feb 2026, at 11:52, Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Feb 25, 2026 at 10:35:41AM +0000, Bertrand Marquis wrote:
> >> Hi Parav,
> >>
> >>> On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote:
> >>>
> >>>>
> >>>> From: Manivannan Sadhasivam <mani@kernel.org>
> >>>> Sent: 25 February 2026 03:37 PM
> >>>>
> >>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote:
> >>>>> Hi Manivannan,
> >>>>>
> >>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote:
> >>>>>>
> >>>>>> Hi Bertrand,
> >>>>>>
> >>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote:
> >>>>>>> Hi Parav,
> >>>>>>>
> >>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>>>>>> Sent: 20 February 2026 05:25 AM
> >>>>>>>>>
> >>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote:
> >>>>>>>>>> Hi Bill,
> >>>>>>>>>>
> >>>>>>>>>>> From: Bill Mills <bill.mills@linaro.org>
> >>>>>>>>>>> Sent: 26 January 2026 10:02 PM
> >>>>>>>>>>>
> >>>>>>>>>>> This series adds the virtio-msg transport layer.
> >>>>>>>>>>>
> >>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in
> >>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one
> >>>>>>>>>>> more transport that performs its transport layer operations by sending and
> >>>>>>>>>>> receiving messages.
> >>>>>>>>>>>
> >>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers:
> >>>>>>>>>>> * common / device level
> >>>>>>>>>>> * bus level
> >>>>>>>>>>>
> >>>>>>>>>>> The common / device level defines the messages exchanged between the driver
> >>>>>>>>>>> and a device. This common part should lead to a common driver holding most
> >>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations.
> >>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport
> >>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device
> >>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the
> >>>>>>>>>>> main focus of this version of the patch series.
> >>>>>>>>>>>
> >>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines
> >>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and
> >>>>>>>>>>> receive operations. A number of bus implementations are envisioned,
> >>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations
> >>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card
> >>>>>>>>>>> and its driver.
> >>>>>>>>>>>
> >>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations
> >>>>>>>>>>> doing the same thing in different ways for no good reason. However the
> >>>>>>>>>>> different environments will require different things. Instead of trying to
> >>>>>>>>>>> anticipate all needs and provide something very abstract, we think
> >>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time,
> >>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to
> >>>>>>>>>>> standardize a bus level message for that.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> I would review more, had first round of sparse review.
> >>>>>>>>>> Please find few comments/questions below.
> >>>>>>>>>
> >>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and
> >>>>>>>>> then add performance features on top as appropriate.
> >>>>>>>> Sounds good. Simple but complete is needed.
> >>>>>>>
> >>>>>>> Agree.
> >>>>>>>
> >>>>>>>>
> >>>>>>>>> So one way to address some of these comments is to show how
> >>>>>>>>> they can be addressed with a feature bit down the road.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header.
> >>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices.
> >>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field.
> >>>>>>>>>>
> >>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer.
> >>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data.
> >>>>>>>>>> Needs 32-bits.
> >>>>>>>>>>
> >>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY)
> >>>>>>>>>> But more below.
> >>>>>>>>>>
> >>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to
> >>>> operate
> >>>>>>>>> the virtqueues.
> >>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response?
> >>>>>>>>>
> >>>>>>>>> surely this can be an optional transport feature bit.
> >>>>>>>>>
> >>>>>>>> How is this optional?
> >>>>>>>
> >>>>>>> As said in a previous mail, we have messages already for that.
> >>>>>>> Please confirm if that answer your question.
> >>>>>>>
> >>>>>>>> How can one implement a transport without defining the basic data transfer semantics?
> >>>>>>>
> >>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O.
> >>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue.
> >>>>>>>
> >>>>>>
> >>>>>> I'd love to have this transport over PCI because it addresses the shortcomings
> >>>>>> of the existing PCI transport which just assumes that every config space access\
> >>>>>> is trap and emulate.
> >>>>>
> >>>>> Agree and AMD did exactly that in their demonstrator.
> >>>>> I will give you answers here as i know them but Edgar will probably give you more
> >>>>> details (and probably fix my mistakes).
> >>>>>
> >>>>>>
> >>>>>> But that being said, I somewhat agree with Parav that we should define the bus
> >>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For
> >>>>>> instance, if we try to use this transport over PCI, we've got questions like:
> >>>>>>
> >>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with
> >>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs?
> >>>>>
> >>>>> One bus is appearing as one pci device with its own Vendor ID,
> >>>>>
> >>>>
> >>>> What should be the 'own Vendor ID' here?
> >>>>
> >>>> The existing virtio-pci driver binds to all devices with the Vendor ID of
> >>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own
> >>>> VID for exposing the Virtio devices? That would mean, the drivers on the host
> >>>> need update as well, which will not scale.
> >>>>
> >>>> It would be good if the existing virtio-pci devices can use this new transport
> >>>> with only device side modifications.
> >>>>
> >>>>>>
> >>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config
> >>>>>> space or through some other means?
> >>>>>
> >>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI
> >>>>> device (ending up being memory shared between both sides)
> >>>>>
> >>>>
> >>>> What should be the BAR number and size?
> >>>>
> >>>>>>
> >>>>>> 3. How the notification be delivered from the device to the host? Through
> >>>>>> INT-X/MSI/MSI-X or even polling?
> >>>>>
> >>>>> Notifications are delivered through MSI.
> >>>>>
> >>>>
> >>>> So no INT-X or MSI-X? Why so?
> >>>>
> >>>> Anyhow, my objective is not to get answers for my above questions here in this
> >>>> thread, but to state the reality that it would be hard for us to make use of
> >>>> this new transport without defining the bus implementation.
> >>>>
> >>> +1 to most of the points that Manivannan explained.
> >>>
> >>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet another interface for _Everything_ that already exists.
> >>> and device is still have to implement all the existing things because the device does not know which driver will operate.
> >>>
> >>> And that too some register based inefficient interface.
> >>> Just to reset the device one needs to fully setup the new message interface but device still have to be working.
> >>> That defeats the whole purpose of reset_1 and reset_2 in the device.
> >>>
> >>> This does not bring anything better for the PCI devices at all.
> >>>
> >>> A transport binding should be defined for the bus binding.
> >>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are.
> >>>
> >>> If we are creating something brand-new, for PCI the only thing needed is:
> >>> 1. Reset the device
> >>> 2. Create an admin virtqueue
> >>> 3. Transport everything needed through this virtqueue including features, configs, control.
> >>>
> >>> And this will work for any other bus or msg based too given only contract needed is to creating the aq.
> >>
> >> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain.
> >>
> >> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI".
> >>
> >> When the virtio-msg bus over PCI it will communicate through this device with an external
> >> system connected through the PCI bus.
> >> The driver will enumerate virtio devices available behind this bus and register them so that
> >> the corresponding virtio drivers are probed for them.
> >> All virtio-msg messages required to communicate with those devices will be transferred through
> >> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the
> >> virtqueues with all the devices on the bus.
> >>
> >> So the PCI device is not one virtio device but one bus behind which there can be many devices.
> >>
> >> Is this making the concept a bit clearer ?
> >>
> >> Cheers
> >> Bertrand
> >>
> >
> > So this is fundamentally pci over virtio-msg?
> 
> we are not doing anything PCI, we use PCI bars and device as a way to implement a virtio-msg
> bus. The devices on top are virtio devices not PCI devices and they use a virtio-msg transport
> not PCI.
> 
> PCI here is only used as a communication system between 2 heterogenous systems connected
> by PCI where one system is seen by the other as a PCI device.
> 
> pci over virtio-msg is making me think we provide some pci functionality over virtio-msg.
> 
> Cheers
> Bertrand
> 

it might make sense to write this up a bit more formally then.