Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer

[Demi Marie Obenour <demiobenour@gmail.com>]

[-- Attachment #1.1.1: Type: text/plain, Size: 3527 bytes --]

On 2/24/26 12:20, Bertrand Marquis wrote:
> Hi Demi,
> 
>> On 24 Feb 2026, at 18:14, Demi Marie Obenour <demiobenour@gmail.com> wrote:
>>
>> On 1/26/26 11:32, Bill Mills wrote:
>>> Add a new transport layer that is based on messages.
>>>
>>> This transport layer still uses virtqueues as the other transport layers do
>>> but implements transport layer operations by sending and receiving messages
>>> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci.
>>>
>>> This transport is useful when the device and driver are both implemented in
>>> software but the trap and emulate operations of virtio-mmio and virtio-pci
>>> can not be used.
>>>
>>> This transport is intended to be used in many situations, including:
>>> * between a host processor and its co-processors
>>> * between two different systems (not SMP) connected via PCIe
>>> * between normal and secure worlds
>>> * host to vm
>>> * vm to vm
>>
>> (snip)
>>
>>> +As described in Sections
>>> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications}
>>> +and
>>> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications},
>>> +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and
>>> +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band
>>> +mechanisms.
>>
>> Sending these notifications out of band should almost always be more
>> efficient.  Under Xen, one should use per-virtqueue, per-direction
>> event channels.  KVM has irqfd and ioeventfd, and the loopback
>> transport can use per-virtqueue eventfds or io_uring queues.  I would
>> be quite surprised if FF-A and PCI did not have similar mechanisms
>> that could be used.
>>
>> I recommend removing these from the spec and expecting buses to
>> provide their own notification mechanism.
> 
> Busses can have more efficient way and can map that other real interrupts or xen events
> but the bus has to create a fake message back to the transport to keep the interface
> coherent.
> 
> In ffa case we do use notifications but we still use messages as those can be more
> efficient in some cases (indirect messages) and we have a limited number of notifications.
> In ffa we have a also a transfer system using a FIFO which can hold several events at a time
> preventing several ping/pong.
> 
> We do let the door open so that a bus can choose how to do this but we keep the message
> (even fake) between the bus and the transport to have a transport independent and handling
> the OS specific way of signaling the drivers.
> 
> Does that answer your question ?

I expect that most implementations will have a single queue in each
direction for bus messages.  This means that sending bus messages will
require taking a lock.  Sending virtqueue notifications out of band,
without a corresponding bus message, will avoid taking the lock and
therefore significantly improve scalability on multi-core systems.
For instance, a Xen transport can use per-virtqueue event channels
and a KVM or MSHV transport can use per-virtqueue irqfds/ioeventfds.

I strongly recommend language like the following:

EVENT_CONFIG, EVENT_USED, and EVENT_AVAIL MAY be sent in-band or
out-of-band.  The specification of a given bus MUST specify which
is used.  If a bus specifies that out-of-band notifications are to
be used, it MUST document how to send them.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 7253 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]