Re: [PATCH v1 0/4] virtio-msg transport layer

["Edgar E. Iglesias" <edgar.iglesias@amd.com>]

On Thu, Feb 26, 2026 at 02:20:01PM -0500, Demi Marie Obenour wrote:
> On 2/26/26 00:36, Manivannan Sadhasivam wrote:
> > On Wed, Feb 25, 2026 at 03:15:39PM +0000, Parav Pandit wrote:
> >>
> >>> From: Manivannan Sadhasivam <mani@kernel.org>
> >>> Sent: 25 February 2026 08:41 PM
> >>>
> >>> On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote:
> >>>
> >>> [...]
> >>>
> >>>>>>> Yes. This makes a lot of sense now.
> >>>>>>>
> >>>>>>> This is a virtio-msg-transport device that needs its own device id in the table.
> >>>>>>> And its binding to the PCI transport.
> >>>>>>
> >>>>>> ok. how about an rfc of that idea on the list?
> >>>>>
> >>>>>
> >>>>> I will let Edgar answer on this.
> >>>>>
> >>>> Sure.
> >>>> Also please explain how is this different than SR-IOV devices?
> >>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport.
> >>>> And all of them are going to share same PCI BDF.
> >>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either.
> >>>> So what would this virtio-msg-transport device achieve?
> >>>>
> >>>
> >>> IDK why you are comparing this transport with SR-IOV, which altogether serves a
> >>> different purpose.
> >>>
> >> In the example showed there are multiple devices behind one PCI device.
> >> Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices.
> >>
> > 
> > That could be one usecase, but not the actual design.
> > 
> >>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story.
> >>>> But underlying transport being PCI, its back to square one for the originally described motivation.
> >>>>
> >>>
> >>> Even with PCI as the physical bus, we don't need any trap and emulate assumption
> >>> with this transport, because the communication is message based, not writing and
> >>> reading registers directly in the BAR memory.
> >>>
> >> That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport.
> >> So I do not know why t+e was mentioned.
> >> I don’t have the link to that email at the moment.
> >>
> > 
> > I mentioned that because to run the existing Virtio PCI one would need an
> > emulated PCI device. I think it is also possible to expose custom designed HW as
> > a Virtio PCI device, but that's not the majority.
> 
> I suspect that HW virtio-PCI implementations do exist.
> That's the purpose of ACCESS_PLATFORM if I recall correctly.
> 
> > One primary example is how the virtqueue configuration happens. The driver
> > selects the queue by writing to the 'queue_select' and immediately it expects
> > the device to select that virtqueue and give response in other fields such as
> > 'queue_size'. This only happens in an emulated PCI bus. But in a real PCI bus,
> > the device on the other end cannot synchronize itself with the host. And this is
> > what being fixed by the message based communication.
> > 
> > We tried using the existing Virtio PCI transport in a design where two SoCs are
> > connected using PCIe bus, both running Linux. One will act as Virtio Frontend
> > and another as backend. Then we ran into all sort issues due to the above
> > mentioned assumption/expectation by the transport.
> My understanding is that PCIe guarantees TLP ordering and that reads
> are implemented as messages that require a response.  If so, the
> write and read TLPs will appear in the appropriate order and could
> be handled correctly.
> 
> However, this would require the PCIe endpoint support software-defined
> MMIO reads.  Is this something that is supported, or is it not an
> option due to latency requirements, at least without wasting a CPU
> core on polling?

Right, I think it would be hard to get something like that to perform well.
We would also need the HW logic to hand-over memory accesses to SW and
back..

There are several benefits with a message-based transport compared to
trap+emulate, for example:

1. Trap and emulation is not really available in AMP scenarios. It could
be worked around by trapping locally and converting each memory access
into some kind of message (like Xen IOREQs) but this adds complexity and
latency. With an end-to-end message based transport we can natively speak
across system boundaries.

2. Real-time. Imagine a guest with a single core running some
kind of real-time sensitive workload. Whenever guests access a register,
traps and emulates, the guest "stalls" for the time it takes to respond
to the trapped access. This can lead to high response times which can be
hard to quantify depending on the device implementation.

With a message based transport, guests crafts messages, send them
over and can do other work (e.g take interrupts) while waiting for
responses.

Best regards,
Edgar