* [PATCH v1 0/4] virtio-msg transport layer
@ 2026-01-26 16:32 Bill Mills
2026-01-26 16:32 ` [PATCH v1 1/4] virtio-msg: add new command for bus normative Bill Mills
` (9 more replies)
0 siblings, 10 replies; 105+ messages in thread
From: Bill Mills @ 2026-01-26 16:32 UTC (permalink / raw)
To: virtio-comment
Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen,
Viresh Kumar, Alex Bennee, Armelle Laine, Bill Mills
This series adds the virtio-msg transport layer.
The individuals and organizations involved in this effort have had difficulty in
using the existing virtio-transports in various situations and desire to add one
more transport that performs its transport layer operations by sending and
receiving messages.
Implementations of virtio-msg will normally be done in multiple layers:
* common / device level
* bus level
The common / device level defines the messages exchanged between the driver
and a device. This common part should lead to a common driver holding most
of the virtio specifics and can be shared by all virtio-msg bus implementations.
The kernel implementation in [3] shows this separation. As with other transport
layers, virtio-msg should not require modifications to existing virtio device
implementations (virtio-net, virtio-blk etc). The common / device level is the
main focus of this version of the patch series.
The virtio-msg bus level implements the normal things a bus defines
(enumeration, dma operations, etc) but also implements the message send and
receive operations. A number of bus implementations are envisioned,
some of which will be reusable and general purpose. Other bus implementations
might be unique to a given situation, for example only used by a PCIe card
and its driver.
The standard bus messages are an effort to avoid different bus implementations
doing the same thing in different ways for no good reason. However the
different environments will require different things. Instead of trying to
anticipate all needs and provide something very abstract, we think
implementation specific messages will be needed at the bus level. Over time,
if we see similar messages across multiple bus implementations, we will move to
standardize a bus level message for that.
We are working on a few reusable bus implementations:
* virtio-msg-ffa based on Arm FF-A interface for use between:
* normal world and secure world
* host and VM or VM to VM
* Can be used w/ or with out a hypervisor
* Any Hypervisor that implements FF-A can be used
* We have this working with pKVM and Xen
* We have this working with Trusty and OP-TEE
* virtio-msg-amp for use between heterogenous systems
* The main processors and its co-processors on an AMP SOC
* Two or more systems connected via PCIe
* Minimal requirements: bi-directional interrupts and
at least one shared memory area
* hvac-demo has 2 demos of this
* This is working on two hardware platforms
* virtio-msg-loopback for userspace implemented devices
* Allows user space to provide devices to its own kernel
* This is similar to fuse, cuse or loopback block devices but for virtio
* hvac-demo has a demo
We also anticipate a few more:
* virtio-msg-xen specific to Xen
* Usable on any Xen system (including x86 where FF-A does not exist)
* Using Xen events and page grants
* virtio-msg over admin virtqueues
* This allows any virtio-pci device that supports admin virtqueues to also
support a virtio-msg bus that supports sub devices
* [We are looking for collaborators for this work]
Changes since RFC2:
Spec Functional:
* Made the common message header 8 bytes and added a token for optional use
when parallel outstanding requests are possible
* Made the 8 byte fields align to 8 byte offsets.
This effects the {SET,GET}_VQUEUE messages
* Many conformance cases have been tightened.
Spec Editorial:
* Major restructure to better align with virtio spec
* Conformance model now matches other transports
* Use C structures to define messages instead of tables
* Added a section to describe how responses are matched to requests
This includes the use of the new token field
* Redefined / better described error handling between transport and bus
layers to eliminate the need for the bus to generate fake response messages
* Included editorial feedback from RFC2
Eco-system:
* Added virtio-msg-loopback demo
* Arm has published the first draft of the virtio-msg over FFA spec [6]
* virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE
secure world
* LKML RFC has been sent [7]
* QEMU RFC has been sent [8]
This is the first non-RFC patch series. The known short comings have been
addressed. We ask for review in earnest on this series and thank you for
any feedback you can provide.
Background info and work in progress implementations:
* HVAC project page with intro slides [1]
* HVAC demo repo w/ instructions in README.md [2]
* Kernel w/ virtio-msg common level and ffa support [3]
* QEMU w/ support for one form of virtio-msg-amp [4]
* Portable RTOS library w/ one form of virtio-msg-amp [5]
In addition to the QEMU system based demos in the hvac-demo repo, we also have
two hardware systems running:
* AMD x86 + AMD Arm Versal connected via PCIe
* ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr
Please note that although the demos work, a few have not been aligned
with this version of the spec.
[1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview
[2] https://github.com/wmamills/hvac-demo
[3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=virtio/msg
[4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/
[5] https://github.com/arnopo/open-amp/commits/virtio-msg/
[6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c
[7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/
[8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html
Bertrand Marquis (2):
virtio-msg: add new command for bus normative
virtio-msg: add conformance entries in conformance chapter
Bill Mills (2):
virtio-msg: Add virtio-msg, a message based virtio transport layer
virtio-msg: link virtio-msg content
commands.tex | 3 +-
conformance.tex | 105 ++-
content.tex | 1 +
transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++
4 files changed, 1746 insertions(+), 3 deletions(-)
create mode 100644 transport-msg.tex
--
2.34.1
^ permalink raw reply [flat|nested] 105+ messages in thread* [PATCH v1 1/4] virtio-msg: add new command for bus normative 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills @ 2026-01-26 16:32 ` Bill Mills 2026-02-03 19:42 ` Edgar E. Iglesias 2026-01-26 16:32 ` [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer Bill Mills ` (8 subsequent siblings) 9 siblings, 1 reply; 105+ messages in thread From: Bill Mills @ 2026-01-26 16:32 UTC (permalink / raw) To: virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine, Bill Mills From: Bertrand Marquis <bertrand.marquis@arm.com> Add a new command to have a standard way to define virtio-msg bus normative statements. Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> Signed-off-by: Bill Mills <bill.mills@linaro.org> --- commands.tex | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/commands.tex b/commands.tex index 25ea8ee..122a97f 100644 --- a/commands.tex +++ b/commands.tex @@ -7,9 +7,10 @@ % How we format a field name \newcommand{\field}[1]{\emph{#1}} -% Mark a normative section (driver or device) +% Mark a normative section (driver, device, or bus) \newcommand{\drivernormative}[3]{#1{Driver Requirements: #2}\label{drivernormative:#3}} \newcommand{\devicenormative}[3]{#1{Device Requirements: #2}\label{devicenormative:#3}} +\newcommand{\busnormative}[3]{#1{Bus Requirements: #2}\label{busnormative:#3}} \newcounter{clausecounter} \newcommand{\conformance}[2]{ \stepcounter{clausecounter} -- 2.34.1 ^ permalink raw reply related [flat|nested] 105+ messages in thread
* Re: [PATCH v1 1/4] virtio-msg: add new command for bus normative 2026-01-26 16:32 ` [PATCH v1 1/4] virtio-msg: add new command for bus normative Bill Mills @ 2026-02-03 19:42 ` Edgar E. Iglesias 0 siblings, 0 replies; 105+ messages in thread From: Edgar E. Iglesias @ 2026-02-03 19:42 UTC (permalink / raw) To: Bill Mills Cc: virtio-comment, Bertrand Marquis, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Mon, Jan 26, 2026 at 11:32:27AM -0500, Bill Mills wrote: > From: Bertrand Marquis <bertrand.marquis@arm.com> > > Add a new command to have a standard way to define virtio-msg bus > normative statements. > > Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> > Signed-off-by: Bill Mills <bill.mills@linaro.org> Acked-by: Edgar E. Iglesias <edgar.iglesias@amd.com> > --- > commands.tex | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/commands.tex b/commands.tex > index 25ea8ee..122a97f 100644 > --- a/commands.tex > +++ b/commands.tex > @@ -7,9 +7,10 @@ > % How we format a field name > \newcommand{\field}[1]{\emph{#1}} > > -% Mark a normative section (driver or device) > +% Mark a normative section (driver, device, or bus) > \newcommand{\drivernormative}[3]{#1{Driver Requirements: #2}\label{drivernormative:#3}} > \newcommand{\devicenormative}[3]{#1{Device Requirements: #2}\label{devicenormative:#3}} > +\newcommand{\busnormative}[3]{#1{Bus Requirements: #2}\label{busnormative:#3}} > \newcounter{clausecounter} > \newcommand{\conformance}[2]{ > \stepcounter{clausecounter} > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 105+ messages in thread
* [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills 2026-01-26 16:32 ` [PATCH v1 1/4] virtio-msg: add new command for bus normative Bill Mills @ 2026-01-26 16:32 ` Bill Mills 2026-02-06 16:28 ` Peter Hilber ` (4 more replies) 2026-01-26 16:32 ` [PATCH v1 3/4] virtio-msg: link virtio-msg content Bill Mills ` (7 subsequent siblings) 9 siblings, 5 replies; 105+ messages in thread From: Bill Mills @ 2026-01-26 16:32 UTC (permalink / raw) To: virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine, Bill Mills Add a new transport layer that is based on messages. This transport layer still uses virtqueues as the other transport layers do but implements transport layer operations by sending and receiving messages instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. This transport is useful when the device and driver are both implemented in software but the trap and emulate operations of virtio-mmio and virtio-pci can not be used. This transport is intended to be used in many situations, including: * between a host processor and its co-processors * between two different systems (not SMP) connected via PCIe * between normal and secure worlds * host to vm * vm to vm Signed-off-by: Bill Mills <bill.mills@linaro.org> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com> --- transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 1640 insertions(+) create mode 100644 transport-msg.tex diff --git a/transport-msg.tex b/transport-msg.tex new file mode 100644 index 0000000..d4e31d7 --- /dev/null +++ b/transport-msg.tex @@ -0,0 +1,1640 @@ +\section{Virtio Over Messages}\label{sec:Virtio Transport Options / Virtio Over Messages} + +\newcommand{\conceptref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / #1]{#1}} +\newcommand{\msgref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1]{VIRTIO_MSG_#1}} +\newcommand{\busref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1]{BUS_MSG_#1}} +\newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1}} +\newcommand{\busdef}[1]{\subsubsection{BUS_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1}} + +This section defines \textbf{virtio-msg}, a transport mechanism that encapsulates +virtio operations as discrete message exchanges rather than relying on PCI or +memory-mapped I/O regions. It separates bus-level functionality (e.g., device +enumeration, hotplug events) from device-specific operations (e.g., feature +negotiation, virtqueue setup), ensuring that a single, generic transport layer +can be reused across multiple bus implementations. + +virtio-msg addresses several key objectives: + +\begin{itemize} + \item \textbf{Support multiple bus implementations:} + Systems can rely on various communication methods such as hypercalls, local + IPC, network channels, or device trees for enumerating devices. virtio-msg + defines a common transport interface suitable for any of these mechanisms. + + \item \textbf{Reduce per-bus complexity:} + Buses can implement a fully message-based workflow (including optional + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE}) + or they can discover and manage devices through + alternative means such as platform firmware data. In either case, they + forward transport messages to and from each device. + + \item \textbf{Preserve virtio semantics:} + The transport leverages standard virtio concepts (features, configuration + space, virtqueues), so existing virtio drivers and device implementations can + integrate smoothly once a device is discovered and registered. +\end{itemize} + +\subsection{Basic Concepts} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts} + +The virtio-msg transport relies on a set of foundational concepts to ensure +reusability across different bus implementations and flexibility in device +capabilities. This section defines those concepts and clarifies how they apply, +regardless of whether the bus leverages message-based enumeration or platform +data for device discovery. + +\subsubsection{High-Level Architecture} + +virtio-msg operates around two layers: + +\begin{enumerate} + \item \textbf{Bus Layer}: A \emph{bus instance} exposes zero or more virtio + devices. It discovers available devices through mechanisms such as: + \begin{itemize} + \item Optional message-based queries (\busref{GET_DEVICES}), + \item External data sources (e.g., device tree, ACPI tables, hypervisor + firmware calls), + \item Dynamic hotplug notifications (optionally via \busref{EVENT_DEVICE}). + \end{itemize} + Once a device is identified, regardless of discovery method, the bus + uses \msgref{GET_DEVICE_INFO} to read its device ID and vendor ID, then + \emph{registers} that device with the host OS so the usual virtio driver + probe can occur. + + \item \textbf{Transport Layer}: After the bus knows about a device, the + virtio-msg transport handles all device-specific operations: + \begin{itemize} + \item Retrieving and setting features (\msgref{GET_DEVICE_FEATURES}, + \msgref{SET_DRIVER_FEATURES}), + \item Accessing the device configuration space (\msgref{GET_CONFIG}, + \msgref{SET_CONFIG}), + \item Setting up virtqueues (\msgref{SET_VQUEUE}, \msgref{RESET_VQUEUE}), + \item Managing status and runtime notifications (\msgref{SET_DEVICE_STATUS}, + \msgref{EVENT_USED}, etc.). + \end{itemize} + These transport messages remain the same across different bus instances, + allowing a single virtio-msg driver component to function in multiple + environments. +\end{enumerate} + +\subsubsection{Relationship Between Bus and Transport} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport} + +This subsubsection explains the division of responsibilities: the bus layer is +the mandatory carrier that moves messages between driver and device endpoints +(e.g., over IPC, shared memory with signalling or hardware messaging), while +the virtio-msg transport defines the semantics of those messages. + +virtio-msg groups messages into two categories: + +\begin{description} + \item[\textbf{Bus Messages}:] + Intended for global bus operations such as enumerating device numbers + (\busref{GET_DEVICES}), managing device hotplug events (\busref{EVENT_DEVICE}) + or assessing bus-wide health (\busref{PING}). + These messages are \emph{optional} in environments where + device discovery or state changes occur through other means (e.g., device + tree). However, if a bus chooses to handle those tasks via messages, + it implements the appropriate bus message definitions described in this + section. + + \item[\textbf{Transport Messages}:] + Used for device-specific operations, such as: + \begin{itemize} + \item Retrieving or setting features (\msgref{GET_DEVICE_FEATURES}, + \msgref{SET_DRIVER_FEATURES}), + \item Accessing device configuration (\msgref{GET_CONFIG}, + \msgref{SET_CONFIG}), + \item Managing virtqueues (\msgref{SET_VQUEUE}, \msgref{RESET_VQUEUE}), + \item Handling device status and notifications (\msgref{SET_DEVICE_STATUS}, + \msgref{EVENT_USED}, etc.). + \end{itemize} +\end{description} + +This separation lets a bus remain minimal if it obtains device information from +firmware tables, while still supporting fully message-based enumeration and +hotplug when desired. + +\busnormative{\paragraph}{Bus Message Implementation}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport / Bus Messages} +\begin{itemize} + \item A bus implementation that provides enumeration, hotplug, or bus + health handling via bus messages SHOULD implement the corresponding Bus + Message definitions described in this section. +\end{itemize} + +\busnormative{\paragraph}{Transport Message Forwarding}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport / Transport Message Forwarding} +\begin{itemize} + \item A bus implementation MUST relay each transport message to the device + number identified in the message header, regardless of how it + discovered or enumerated that device. + \item A bus implementation SHOULD treat transport messages as opaque apart + from enforcing generic transport limits, such as the advertised maximum + message size, and SHOULD NOT modify the transport payload. +\end{itemize} + +\subsubsection{System Topology} + +A virtio-msg system contains the following elements: + +\begin{itemize} + \item \textbf{Bus Instances and Devices}: Each bus instance advertises its + capabilities (e.g., transport revision, maximum message size) and + discovers devices via message-based queries or external data sources. + Every discovered device has a unique \emph{device number}. + \item \textbf{Driver}: Communicates with the bus to learn about + available devices. Once a device is recognized, the driver uses the + common transport messages to perform feature negotiation, configuration, + and virtqueue setup. + \item \textbf{Device}: Implement virtio device functionality and + respond to the transport messages. The bus forwards these messages to + the correct device instance based on device number. +\end{itemize} + +\subsubsection{Transport Revisions and Maximum Message Size} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions} + +Each \textbf{virtio-msg bus instance} advertises the following to the transport +layer: +\begin{itemize} + \item \textbf{Transport revision}: the protocol version supported by the bus + instance (independent of the overall Virtio specification version). + \item \textbf{Maximum message size}: the largest payload (in bytes) that can + be carried per request or response, including the common header. + \item \textbf{Transport feature bits}: revision-specific optional features + implemented by the bus instance. +\end{itemize} + +The mechanism for obtaining these parameters is implementation-defined and can +vary between bus instances. Common approaches include: +\begin{itemize} + \item Reading firmware or device-tree data that describes each bus instance, + \item Performing a message exchange during bus setup to retrieve the values, + \item Relying on per-bus configuration structures or driver-specific defaults. +\end{itemize} + +\busnormative{\paragraph}{Advertising Transport Parameters}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Advertising Transport Parameters} +\begin{itemize} + \item Each bus instance MUST make its transport revision, maximum message + size, and transport feature bits available to the virtio-msg transport + before any transport messages are exchanged for that device. + \item A bus instance MUST apply the same limits to both driver-originated and + device-originated transport messages; if the values change, the bus + MUST inform the transport layer before accepting additional messages. +\end{itemize} + +\drivernormative{\paragraph}{Respecting Bus Limits}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Driver Limits} +\begin{itemize} + \item A driver MUST NOT send a transport message whose total size exceeds the + maximum message size advertised for the target bus instance. + \item A driver MUST NOT rely on transport features or messages that require a + higher transport revision than the bus instance reports. +\end{itemize} + +\devicenormative{\paragraph}{Respecting Bus Limits}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Device Limits} +\begin{itemize} + \item A device MUST ensure its responses and device-originated messages do + not exceed the maximum message size advertised by the bus instance that + relays them. + \item A device MUST NOT require transport features or messages beyond the + transport revision reported by the bus instance, and MUST respond with + an error or ignore requests for unsupported features. +\end{itemize} + +\paragraph{virtio-msg revisions} + +The following table lists the currently defined virtio-msg revisions: + +\begin{tabular}{ |l|l|l|l| } +\hline +\field{revision} & \field{maximum size} & \field{features} & remarks \\ +\hline \hline +1 & 44-65536 & <empty> & Virtio Message Revision 1 \\ +\hline +\end{tabular} + +Note that a change in the virtio standard does not necessarily +correspond to a change in the virtio-msg revision. + +The maximum message size is specified from the transport-layer point of view +and includes the 8-byte common header plus payload. Any extra encapsulation +imposed by the underlying bus (for example, a framing header) does not count +against this limit. Today the largest practical transport payload is 256 bytes: +messages such as GET_CONFIG and SET_CONFIG can carry up to 256 bytes of device +configuration in one transfer, and larger configuration regions can be accessed +through multiple exchanges without requiring a larger per-message limit. + +\busnormative{\paragraph}{Message Size Bounds}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Message Size} +\begin{itemize} + \item A bus implementation MUST advertise a maximum message size of at least + 44 bytes. + \item A bus implementation SHOULD NOT advertise a maximum message size that + exceeds 264 bytes (256-byte payload plus the common header). +\end{itemize} + +\paragraph{Versioning and Forward Compatibility} + +A higher transport revision or additional transport feature bits extend the +protocol with new messages or capabilities. Implementations are expected to +remain interoperable across revisions: devices and drivers designed for a newer +revision still implement the mandatory messages and semantics defined in prior +revisions. + +\drivernormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Driver} +\begin{itemize} + \item A driver that negotiates transport revision $N$ MUST implement all + mandatory driver behavior defined for revisions $1$ through $N$. + \item If a driver receives a device-originated message or feature indication + that requires an unsupported revision or transport feature, it MUST + ignore the message (or treat the request as failed) and MUST NOT act on + partially understood data. +\end{itemize} + +\devicenormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Device} +\begin{itemize} + \item A device that advertises transport revision $N$ MUST implement all + mandatory device behavior defined for revisions $1$ through $N$. + \item If a device receives a driver request that relies on an unsupported + revision or transport feature, it MUST reject the request using the + message-specific error mechanism (if any) or silently ignore it. +\end{itemize} + +\busnormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Bus} +\begin{itemize} + \item A bus instance that advertises transport revision $N$ MUST satisfy every + bus requirement defined for revisions $1$ through $N$. + \item If a bus instance cannot forward a message because it requires an + unsupported revision or transport feature, it MUST surface a transport + error or drop the message without forwarding it. +\end{itemize} + +\subsubsection{Device Numbers and Enumeration} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / +Device Numbers} + +Each virtio-msg bus instance contains one or more \emph{devices}, identified +by a 16-bit \textbf{device number}. Buses discover these device numbers through +mechanisms such as: +\begin{itemize} + \item \textbf{Message-Based Enumeration}: Using \busref{GET_DEVICES} to query + which numbers exist (optional). + \item \textbf{Platform Data}: A device tree, ACPI tables, or hypervisor calls + might inform the bus of available device numbers and their properties. +\end{itemize} + +Once a bus confirms that a device number is valid—regardless of the discovery +method—it normally issues \msgref{GET_DEVICE_INFO} to retrieve the device and +vendor IDs before registering the device with the host OS so the usual Virtio +driver binding process can begin. + +\busnormative{\paragraph}{Device Number Assignment}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Device Numbers / Assignment} +\begin{itemize} + \item A bus implementation MUST assign a unique device number to every + device on a given bus instance and MUST NOT forward transport messages + for a device number that has not been validated. + \item A bus implementation SHOULD provide the driver with sufficient + information—either via \busref{GET_DEVICES} or equivalent platform + data—to discover each valid device number. +\end{itemize} + +\subsubsection{Configuration Generation Count} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count} + +Each device maintains a \textbf{Configuration Generation Count} to prevent +inconsistent updates. This count is incremented at least once by the device for +every driver-visible change it makes to its configuration data. The current +count is exposed in \msgref{EVENT_CONFIG} and \msgref{GET_CONFIG} responses so +the driver can determine whether its view of the configuration is current. The +count does not necessarily start at zero and is not automatically reset when +the device resets. + +\devicenormative{\paragraph}{Configuration Generation Count}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Device} +\begin{itemize} + \item A device MUST increment the generation count before it makes a change + that is visible to the driver and MUST ensure that each + \msgref{EVENT_CONFIG} carrying configuration data uses a unique + generation count. + \item If updated configuration data cannot fit in a single + \msgref{EVENT_CONFIG}, the device SHOULD send an \msgref{EVENT_CONFIG} + with zero data length to advertise the new generation count and MUST + make the updated data available via \msgref{GET_CONFIG}. + \item If a \msgref{SET_CONFIG} request includes a generation count that does + not match the device's current count, the device MUST reject the + request. +\end{itemize} + +\drivernormative{\paragraph}{Configuration Generation Count}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Driver} +\begin{itemize} + \item A driver MUST track the most recent generation count observed (via + \msgref{EVENT_CONFIG} or \msgref{GET_CONFIG}) and include it in every + \msgref{SET_CONFIG} request. + \item If a \msgref{SET_CONFIG} request is rejected due to a mismatched + generation count, the driver SHOULD issue \msgref{GET_CONFIG} to obtain + the latest configuration data and generation count before retrying. +\end{itemize} + +This mechanism ensures updates are not lost or overwritten due to stale +information. + +\subsubsection{Feature Negotiation Blocks} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Feature Blocks} + +The virtio-msg transport organizes feature bits into 32-bit blocks accessed by +\msgref{GET_DEVICE_FEATURES} and \msgref{SET_DRIVER_FEATURES}. Each block +represents up to 32 feature bits: + +\begin{itemize} + \item \textbf{Block Index}: The starting block (e.g., block 0 for + features 0--31, block 1 for features 32--63, etc.). + \item \textbf{Number of Blocks}: How many blocks the driver wishes to retrieve + or modify in a single message. + \item \textbf{Feature Data}: The 32-bit values representing the supported (or + driver-requested) feature bits in the selected blocks. +\end{itemize} + +\devicenormative{\paragraph}{Feature Blocks}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Feature Blocks / Device} +\begin{itemize} + \item When \msgref{GET_DEVICE_FEATURES} covers blocks that extend beyond the + features a device implements, the device MUST return zero for the + feature data in those positions. +\end{itemize} + +\subsubsection{Error Signaling} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling} + +Transport errors can arise from malformed messages, routing failures inside a +bus implementation, or device-side faults while processing a valid request. +Implementations should handle such faults locally where possible, but a bus may +surface an error to the virtio-msg transport if it cannot deliver a request or +obtain a response within its policy. + +\busnormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus} +\begin{itemize} + \item A bus implementation MAY report a transport-visible failure (for + example, after exhausting a bounded retry policy) when it cannot deliver + a request or obtain a response. + \item A bus implementation MUST treat malformed headers or unsupported + \field{msg_id} values as invalid, MUST discard them without generating + additional protocol traffic, and MAY log the condition for diagnostics. + \item A bus implementation MUST NOT generate error responses to event + (one-way) messages. +\end{itemize} + +\devicenormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device} +\begin{itemize} + \item A device receiving a malformed or unsupported transport message MUST + discard it without producing further protocol traffic. + \item Recovery actions taken in response to an error (such as retries, + selective resets, or device removal) MUST follow the normative reset and + status semantics defined in + \ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}. +\end{itemize} + +This specification does not define a dedicated error-reporting message; it only +permits implementations to surface failures when silent recovery is not +feasible. + +\subsubsection{Endianness} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness} + +Unless otherwise stated, all numeric fields defined for virtio-msg messages use +little-endian encoding. + +\drivernormative{\paragraph}{Endianness}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Driver} +\begin{itemize} + \item A driver MUST encode and decode the common header and payload fields + defined by this transport using little-endian byte order. +\end{itemize} + +\devicenormative{\paragraph}{Endianness}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Device} +\begin{itemize} + \item A device MUST emit and consume the common header and payload fields + defined by this transport using little-endian byte order. +\end{itemize} + +\subsubsection{Common Message Format} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format} + +All virtio-msg exchanges, whether \emph{bus messages} or \emph{transport +messages}, begin with an 8-byte header followed by an optional payload. The +fields below describe the wire format for that header. + +The header layout is: +\begin{lstlisting} +struct virtio_msg_header { + u8 type; /* request/response + bus/transport */ + u8 msg_id; /* message id */ + le16 dev_num; /* device number (0 for bus messages) */ + le16 token; /* bus-managed correlation identifier */ + le16 msg_size; /* total size: header (8) + payload */ + u8 payload[]; +}; +\end{lstlisting} + +Field semantics: +\begin{itemize} + \item \field{type}: + \begin{itemize} + \item Bit[0]: 0=request, 1=response. + \item Bit[1]: 0=transport message, 1=bus message. + \item Bits[2..7]: reserved for future use. + \end{itemize} + \item \field{msg_id}: Message ID identifying the message definition. Ranges + are defined in + \ref{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} + and + \ref{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages}. + \item \field{dev_num}: For transport messages, the device number that should + receive the message. Bus messages operate on device number 0. + \item \field{token}: Correlation identifier managed by the bus. Drivers and + devices treat this field as opaque; the bus MAY overwrite it to track + outstanding requests or maintain ordering. + \item \field{msg_size}: Total size in bytes of the complete message (header + plus payload). + \item \field{payload}: Operation-specific data. If a bus introduces extra + padding bytes, those bytes are not part of the payload semantics. +\end{itemize} + +\drivernormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Driver} +\begin{itemize} + \item A driver MUST set bits 2..7 of \field{type} to zero and MUST treat them + as reserved when parsing received headers. + \item A driver MUST ensure \field{msg_size} reflects the total message length + (header plus payload) and MUST NOT exceed the maximum message size + advertised by the bus instance. + \item When sending a transport message, a driver MUST set \field{dev_num} to + the intended device number. + \item If a driver introduces padding bytes that become part of the transport + payload, it MUST set those bytes to zero. + \item A driver MUST treat the \field{token} field as opaque and MUST NOT rely + on its value when processing received messages. +\end{itemize} + +\devicenormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Device} +\begin{itemize} + \item A device MUST set bits 2..7 of \field{type} to zero in transmitted + messages and MUST ignore those bits on receive. + \item A device MUST ensure \field{msg_size} reflects the total message length + (header plus payload) and does not exceed the bus's advertised maximum. + \item When sending a transport message, a device MUST set \field{dev_num} to + its own device number. + \item A device MUST ignore padding bytes that are documented as bus-specific + and MUST zero any such bytes it introduces into the transport payload. + \item A device MUST treat the \field{token} field as opaque and MUST NOT rely + on its value when processing received messages. +\end{itemize} + +\busnormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Bus} +\begin{itemize} + \item A bus implementation MUST deliver bus messages with \field{dev_num}=0 + and MUST NOT alter \field{dev_num} for transport messages beyond the + routing needed to reach the addressed device. + \item A bus implementation MUST set bits 2..7 of \field{type} to zero when + generating bus messages and MUST ignore those bits when forwarding + transport messages. + \item If the bus adds framing or padding bytes around the common header or + payload, it MUST set those bytes to zero before delivering the message + to the opposite side and MUST present the same zero padding when the + opposite side reads a message. + \item A bus implementation owns the \field{token} field; it MAY insert or + overwrite values for correlation purposes and MUST ensure that any such + use is transparent to drivers and devices. +\end{itemize} + +Reserved header bits and unspecified header values MUST be transmitted as zero +and ignored on receive to preserve forward compatibility. + +\subsubsection{Message Ordering} +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering} + +Transport messages fall into two classes: requests (which expect responses) and +events (which are one-way). Drivers and devices rely on the bus to preserve the +relative ordering of request/response pairs for each device number; they do not +interpret the \field{token} field directly. + +\busnormative{\paragraph}{Message Ordering}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} +\begin{itemize} + \item For each device number, a bus implementation MUST deliver responses to + the driver in the same order that it forwarded the corresponding + requests to the device. + \item A bus implementation MUST ensure that every request forwarded to a + device results in exactly one response delivered to the driver (unless + the request is defined as an event). +\end{itemize} + +\subsection{Device Discovery}\label{sec:Virtio Transport Options / Virtio Over Messages / Device Discovery} + +A virtio-msg implementation can learn about devices via bus-level enumeration +(e.g., \busref{GET_DEVICES}), platform data (e.g., device tree, ACPI), or +hypervisor/firmware interfaces. The following informative text describes +typical flows; the normative obligations follow. + +Bus implementations discover their devices through a mix of platform data, +hypervisor-provided enumeration, or message-based queries such as +\busref{GET_DEVICES}. This specification defines \busref{GET_DEVICES} for +environments that prefer message-driven enumeration, but it does not require +its use when equivalent information is already known out-of-band. + +\busnormative{\paragraph}{Device Discovery}{Virtio Transport Options / Virtio Over Messages / Device Discovery / Bus} +\begin{itemize} + \item A bus implementation MUST ensure that every device number it exposes to + the driver has been validated via either platform data or a successful + query such as \busref{GET_DEVICES}. + \item Once a device number is deemed present, the bus implementation SHOULD + invoke \msgref{GET_DEVICE_INFO} so it can register the device with the + host OS and allow the appropriate virtio driver to bind. + \item If a bus implementation provides an alternative enumeration mechanism + (e.g., ACPI, device tree), it MAY omit \busref{GET_DEVICES} provided the + alternative delivers equivalent information to the driver. +\end{itemize} + +\subsection{Device Initialization} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization} + +After a bus has identified a virtio-msg device (whether by message-based +enumeration or platform-specific discovery), the driver undertakes a series of +steps to configure and ready the device for normal operation. This section +details the recommended order of operations and the associated messages. + +\subsubsection{Initialization Flow Overview} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview} + +A typical device initialization flow includes: +\begin{enumerate} + \item \textbf{Obtain Device Information:} + Query static parameters with \msgref{GET_DEVICE_INFO}. + \item \textbf{Negotiate Features:} + Read feature blocks via \msgref{GET_DEVICE_FEATURES}, decide which to + enable, and write them back with \msgref{SET_DRIVER_FEATURES} before + updating the device status. + \item \textbf{Initialize Configuration Space:} + Read or write configuration data with + \msgref{GET_CONFIG}/\msgref{SET_CONFIG}, guarding updates with the + configuration generation count. + \item \textbf{Set Up Virtqueues:} + Configure each virtqueue via \msgref{SET_VQUEUE} and verify the settings + with \msgref{GET_VQUEUE}. + \item \textbf{Set Device Status:} + Use \msgref{SET_DEVICE_STATUS} to drive the device through the standard + virtio status states. +\end{enumerate} + +The exact order may vary slightly across implementations, but these steps form +the baseline for virtio-msg initialization. + +\drivernormative{\paragraph}{Initialization Flow}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview / Driver} +\begin{itemize} + \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature + negotiation or queue setup. + \item A driver MUST complete feature negotiation via + \msgref{GET_DEVICE_FEATURES}/\msgref{SET_DRIVER_FEATURES} + and confirm the FEATURES\_OK state via \msgref{SET_DEVICE_STATUS} before + enabling virtqueues. + \item A driver MUST configure each virtqueue via \msgref{SET_VQUEUE} and + confirm its parameters (e.g., with \msgref{GET_VQUEUE}) before marking + the queue ready for I/O. + \item A driver MUST drive the device status transitions using + \msgref{SET_DEVICE_STATUS}, ensuring the device reaches DRIVER\_OK + before issuing normal I/O. +\end{itemize} + +\subsubsection{Device Information} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Information} + +Once a device number is confirmed, the driver uses \msgref{GET_DEVICE_INFO} to +retrieve static identification data (device ID, vendor ID), the number of +feature bits, configuration space size, maximum virtqueues, shared memory +segments, and any admin virtqueue details. This information determines which +virtio driver should bind to the device, how many feature blocks to query, and +how much configuration space is valid. + +\drivernormative{\paragraph}{Device Information}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Information / Driver} +\begin{itemize} + \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature + negotiation or queue setup so it can discover the device ID, vendor ID, + feature count, configuration size, and virtqueue limits. + \item A driver MUST use the configuration size reported via + \msgref{GET_DEVICE_INFO} to bound any offsets and lengths supplied in + \msgref{GET_CONFIG} and \msgref{SET_CONFIG} requests. +\end{itemize} + +\subsubsection{Feature Negotiation} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features} + +Drivers read available features in 32-bit blocks using +\msgref{GET_DEVICE_FEATURES}; the device returns the requested bitfields, +padding with zeros for any out-of-range blocks. To enable selected features, the +driver writes them back with \msgref{SET_DRIVER_FEATURES} and then updates the +device status via \msgref{SET_DEVICE_STATUS}, checking whether the FEATURES\_OK +bit remains set. + +\drivernormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Driver} +\begin{itemize} + \item A driver MUST use \msgref{GET_DEVICE_FEATURES} to discover the feature + bits offered by the device and MUST write back only the features it + intends to enable via \msgref{SET_DRIVER_FEATURES}. + \item After writing the desired features, the driver MUST attempt to set the + FEATURES\_OK bit using \msgref{SET_DEVICE_STATUS} and MUST check the + returned status to ensure the device accepted the set. +\end{itemize} + +\devicenormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} +\begin{itemize} + \item When handling \msgref{GET_DEVICE_FEATURES}, a device MUST return zero + for any requested bits that fall outside the number of feature bits it + implements. + \item After receiving \msgref{SET_DRIVER_FEATURES}, a device MUST update its + internal feature mask to match the acknowledged set and MUST reflect + acceptance or rejection by leaving the FEATURES\_OK bit set or clearing + it in the status returned by \msgref{SET_DEVICE_STATUS}. +\end{itemize} + +\subsubsection{Device Configuration} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration} + +Drivers use \msgref{GET_CONFIG} to read portions of the configuration space by +supplying an offset and length; the device returns the requested data plus the +current configuration generation count. Writing is performed via +\msgref{SET_CONFIG}, which carries the same offset/length along with the +driver's notion of the generation count and the new data. + +\drivernormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Driver} +\begin{itemize} + \item A driver MUST ensure that the offset and length in each + \msgref{GET_CONFIG} or \msgref{SET_CONFIG} request stay within the + configuration size reported by \msgref{GET_DEVICE_INFO}. + \item A driver MUST include its most recently observed configuration + generation count in a \msgref{SET_CONFIG} request and SHOULD re-read the + configuration (via \msgref{GET_CONFIG}) if the request is rejected for a + generation mismatch. +\end{itemize} + +\devicenormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Device} +\begin{itemize} + \item A device MUST reject a \msgref{SET_CONFIG} request whose generation + count does not match its current value and MUST indicate the rejection + in the response. + \item A device MUST return the current configuration generation count + alongside any data returned via \msgref{GET_CONFIG}. +\end{itemize} + +\subsubsection{Virtqueue Configuration} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration} + +Drivers query virtqueue parameters with \msgref{GET_VQUEUE}, configure them via +\msgref{SET_VQUEUE}, and optionally reset them using \msgref{RESET_VQUEUE} (if +VIRTIO\_F\_RING\_RESET is negotiated). Each queue is typically configured by +reading its maximum size, provisioning descriptor/available/used buffers, and +then calling \msgref{SET_VQUEUE} with the chosen size and guest-physical +addresses. + +\drivernormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Driver} +\begin{itemize} + \item A driver MUST use \msgref{GET_VQUEUE} to determine the maximum queue + size and confirm that a queue is inactive before programming it. + \item A driver MUST ensure the queue size provided in \msgref{SET_VQUEUE} does + not exceed the maximum reported by the device and MUST supply valid + descriptor/driver/device addresses before enabling the queue. + \item If VIRTIO\_F\_RING\_RESET has been negotiated and a queue requires + reinitialization, the driver SHOULD use \msgref{RESET_VQUEUE} before + reprogramming it. +\end{itemize} + +\devicenormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} +\begin{itemize} + \item A device MUST report accurate maximum queue sizes in \msgref{GET_VQUEUE} + and MUST persist the parameters supplied via \msgref{SET_VQUEUE} (size, + descriptor, driver, and device addresses). + \item When \msgref{RESET_VQUEUE} is issued (and VIRTIO\_F\_RING\_RESET is + negotiated), the device MUST quiesce the queue, release any resources + associated with it, and allow the driver to reconfigure it. +\end{itemize} + +\subsubsection{Status Information} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information} + +Drivers query the device status via \msgref{GET_DEVICE_STATUS} to observe +progress or detect errors, and they drive the Virtio status transitions via +\msgref{SET_DEVICE_STATUS}. Writing zero to the status field resets the device, +invalidating any configuration or virtqueue state. + +\drivernormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} +\begin{itemize} + \item A driver SHOULD read the device status via \msgref{GET_DEVICE_STATUS} + when diagnosing errors or determining whether the device is ready to + move to the next initialization phase. + \item A driver MUST use \msgref{SET_DEVICE_STATUS} to drive the device through + the virtio-defined status states and MUST write 0 to request a device + reset when needed. +\end{itemize} + +\devicenormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} +\begin{itemize} + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST + reset its internal state, invalidate existing configuration and + virtqueue settings, and present the status field as 0. + \item A device MUST report its current status accurately via + \msgref{GET_DEVICE_STATUS}, including whether the FEATURES\_OK bit has + been accepted or cleared. +\end{itemize} + +\subsubsection{Finalizing Initialization} + +After configuring virtqueues and agreeing on features, the driver transitions +the device to DRIVER\_OK (and any other required status bits) via +\msgref{SET_DEVICE_STATUS}. At that point, the device is considered ready for +normal virtqueue I/O. + +\drivernormative{\paragraph}{Final Status}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Driver} +\begin{itemize} + \item A driver MUST set DRIVER\_OK via \msgref{SET_DEVICE_STATUS} only after + it has completed feature negotiation and initialized all required + virtqueues. + \item A driver MUST NOT queue normal I/O until the device reports a status + that includes DRIVER\_OK. + \item A driver MUST NOT supply buffers or send driver notifications for a + virtqueue until that queue has been configured via \msgref{SET_VQUEUE}. +\end{itemize} + +\devicenormative{\paragraph}{Final Status}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Device} +\begin{itemize} + \item A device MUST NOT process normal virtqueue I/O until the driver has set + DRIVER\_OK. + \item Once DRIVER\_OK is set, the device MUST begin processing virtqueue + requests subject to the negotiated features and queue configurations. +\end{itemize} + +\subsection{Device Operation} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation} + +Once a virtio-msg device is fully initialized (see +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization}), +the driver and device exchange messages to perform I/O and respond to +configuration changes. This section details the primary messaging paths for +\emph{driver notifications}, \emph{device notifications}, and the handling of +runtime resets or shutdown sequences. + +\subsubsection{Driver Notifications} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} + +After buffers are made available to a virtqueue, the driver issues +\msgref{EVENT_AVAIL} to notify the device. A bus implementation may translate +these notifications into an out-of-band mechanism or deliver them in-band. + +\drivernormative{\paragraph}{Driver Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Driver} +\begin{itemize} + \item A driver MUST send \msgref{EVENT_AVAIL} when it places new buffers on a + virtqueue. + \item Each \msgref{EVENT_AVAIL} MUST identify the target virtqueue index. + \item If VIRTIO\_F\_NOTIFICATION\_DATA has been negotiated, the driver MUST + populate the “next offset and wrap” fields so the device can locate the + head of the updated ring; otherwise those fields MUST be zero. +\end{itemize} + +\busnormative{\paragraph}{Driver Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Bus} +\begin{itemize} + \item A bus implementation MAY convert \msgref{EVENT_AVAIL} into an + out-of-band notification but, if it does not, it MUST relay the message + over the virtio-msg channel. + \item A bus implementation MUST NOT drop \msgref{EVENT_AVAIL} unless it has + arranged for the device to detect new buffers through polling or an + equivalent mechanism. +\end{itemize} + +\subsubsection{Device Notifications} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications} + +\msgref{EVENT_CONFIG} and \msgref{EVENT_USED} provide asynchronous notifications +from the device (or device-side bus) to the driver. The bus may forward these +messages in-band or synthesize them based on other signals such as interrupts +or polling. + +\devicenormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} +\begin{itemize} + \item A device (or device-side bus) MUST send \msgref{EVENT_CONFIG} whenever + it makes a configuration change or status update that becomes visible to + the driver. The message MUST include the new configuration generation + count and MAY include the updated configuration data. + \item If the configuration data is omitted from \msgref{EVENT_CONFIG}, the + device SHOULD include the relevant offsets/lengths so the driver can + re-fetch the data via \msgref{GET_CONFIG}. + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when + buffers on a virtqueue have been consumed, unless the device relies on + an alternative, agreed-upon completion mechanism. +\end{itemize} + +\drivernormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Driver} +\begin{itemize} + \item Upon receiving \msgref{EVENT_CONFIG}, a driver SHOULD update its view of + the configuration using the provided data (or by issuing + \msgref{GET_CONFIG} if necessary) before resuming normal I/O. + \item Upon receiving \msgref{EVENT_USED}, a driver MUST examine the indicated + virtqueue (for example, by reading the used ring) to reclaim completed + buffers. +\end{itemize} + +\busnormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Bus} +\begin{itemize} + \item A bus implementation MUST forward \msgref{EVENT_CONFIG} and + \msgref{EVENT_USED} notifications (or their equivalents) to the driver, + either in-band or by synthesizing the appropriate message. + \item If a bus implementation relies on polling or other mechanisms instead of + direct notifications, it SHOULD limit that mode to scenarios where no + other notification method is available. +\end{itemize} + +\subsubsection{Configuration Changes During Operation} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation} + +Drivers may update configuration fields at runtime using \msgref{SET_CONFIG} +when features such as device modes or limits need to change. Devices can also +update configuration data autonomously but must signal those changes via +\msgref{EVENT_CONFIG}. + +\drivernormative{\paragraph}{Runtime Configuration}{Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Driver} +\begin{itemize} + \item A driver MAY issue \msgref{SET_CONFIG} after initialization, provided it + includes the current configuration generation count and follows the same + validation rules as during setup. + \item Upon receiving an \msgref{EVENT_CONFIG}, the driver SHOULD refresh its + view of the configuration (via the data provided or by reissuing + \msgref{GET_CONFIG}) before relying on the updated values. +\end{itemize} + +\devicenormative{\paragraph}{Runtime Configuration}{Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Device} +\begin{itemize} + \item If a device updates its configuration after initialization, it MUST send + \msgref{EVENT_CONFIG} to inform the driver of the change and provide the + new configuration generation count. +\end{itemize} + +\subsubsection{Virtqueue Changes During Operation} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation} + +Drivers may provision unused virtqueues later in the device lifetime by issuing +\msgref{SET_VQUEUE}, and they may reconfigure existing queues if the +VIRTIO\_F\_RING\_RESET feature has been negotiated. + +\drivernormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Driver} +\begin{itemize} + \item A driver MAY configure additional virtqueues after initialization using + \msgref{SET_VQUEUE}, provided it follows the same validation steps + (e.g., checking the maximum queue size). + \item If VIRTIO\_F\_RING\_RESET is negotiated, the driver SHOULD use + \msgref{RESET_VQUEUE} before reprogramming a queue to avoid races with + the device. +\end{itemize} + +\devicenormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Device} +\begin{itemize} + \item A device MUST honor \msgref{SET_VQUEUE} requests issued after + initialization and update the queue parameters accordingly. + \item When \msgref{RESET_VQUEUE} is received (and VIRTIO\_F\_RING\_RESET is + negotiated), the device MUST quiesce the queue and allow the driver to + reconfigure it without processing stale data. +\end{itemize} + +\subsubsection{Device Reset and Shutdown} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Reset} + +Drivers reset a device by writing 0 to the status field via \msgref{SET_DEVICE_STATUS}, +which forces the device to discard pending operations and return to the initial state. +A device may also signal that a reset is required by sending \msgref{EVENT_CONFIG} +with the DEVICE\_NEEDS\_RESET bit set. + +\drivernormative{\paragraph}{Reset and Shutdown}{Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Driver} +\begin{itemize} + \item A driver MAY request a device reset at any time by writing 0 through + \msgref{SET_DEVICE_STATUS} and MAY reinitialize the device afterwards if + it intends to resume operation. + \item If a device signals DEVICE\_NEEDS\_RESET (for example, via + \msgref{EVENT_CONFIG}), the driver SHOULD write 0 to + \msgref{SET_DEVICE_STATUS} and reinitialize the device before resuming + I/O. +\end{itemize} + +\devicenormative{\paragraph}{Reset and Shutdown}{Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Device} +\begin{itemize} + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST + reset its internal state, discard pending virtqueue operations, and + present the status field as 0. + \item If the device encounters an unrecoverable error that requires driver + intervention, it SHOULD signal DEVICE\_NEEDS\_RESET (such as via + \msgref{EVENT_CONFIG}) so the driver can initiate a reset. +\end{itemize} + +\subsubsection{Hotplug and Removal} +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Hotplug and Removal} + +If the bus supports dynamic addition or removal of devices, it can advertise +those changes with \busref{EVENT_DEVICE} (READY or REMOVED). Some platforms may +instead rely on external signals such as ACPI, device tree updates, or +hypervisor events; virtio-msg does not mandate a specific mechanism. + +\busnormative{\paragraph}{Hotplug and Removal}{Virtio Transport Options / Virtio Over Messages / Device Operation / Hotplug and Removal / Bus} +\begin{itemize} + \item A bus implementation that uses \busref{EVENT_DEVICE} MUST send the READY + event when a device becomes accessible and the REMOVED event when it is + no longer available. + \item Regardless of how hotplug information is delivered, once a new device is + reported the bus SHOULD query it (e.g., via \msgref{GET_DEVICE_INFO}) + and register it with the host OS. When a device is removed, the bus + SHOULD prompt the OS to quiesce and release the associated driver. +\end{itemize} + +\subsection{Transport Messages}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} + +Transport messages configure and operate virtio devices once they have been +discovered. Unlike bus messages (which cover enumeration or hotplug), transport +messages focus on feature negotiation, configuration access, virtqueue setup, +and runtime notifications. The subsections below describe each message and how +it is used in the virtio-msg transport. + +\subsubsection{Overview} +\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / Overview} + +Drivers send transport messages to a specific device number, and the virtio-msg +device replies or emits events accordingly. The bus simply forwards these +messages after enforcing generic checks such as maximum size or verifying that +the target device exists. Most transport messages are request/response pairs, +with events reserved for asynchronous notifications. + +\paragraph{Messages IDs and issuers} + +\begin{tabular}{|l|l|l|} +\hline +Name & ID & Sender \\ +\hline +\hline +Reserved & 0x0 & \\ +\hline +Reserved & 0x1 & \\ +\hline +\msgref{GET_DEVICE_INFO} & 0x2 & Driver \\ +\hline +\msgref{GET_DEVICE_FEATURES} & 0x3 & Driver \\ +\hline +\msgref{SET_DRIVER_FEATURES} & 0x4 & Driver \\ +\hline +\msgref{GET_CONFIG} & 0x5 & Driver \\ +\hline +\msgref{SET_CONFIG} & 0x6 & Driver \\ +\hline +\msgref{GET_DEVICE_STATUS} & 0x7 & Driver \\ +\hline +\msgref{SET_DEVICE_STATUS} & 0x8 & Driver \\ +\hline +\msgref{GET_VQUEUE} & 0x9 & Driver \\ +\hline +\msgref{SET_VQUEUE} & 0xA & Driver \\ +\hline +\msgref{RESET_VQUEUE} & 0xB & Driver \\ +\hline +\msgref{GET_SHM} & 0xC & Driver \\ +\hline +\msgref{EVENT_CONFIG} & 0x40 & Device \\ +\hline +\msgref{EVENT_AVAIL} & 0x41 & Driver \\ +\hline +\msgref{EVENT_USED} & 0x42 & Device \\ +\hline +\end{tabular} + +IDs are grouped as follows; bit 6 distinguishes requests (0) from events (1): + +\begin{tabular}{|l|l|p{7cm}|} +\hline +Range & Usage & Notes \\ +\hline +0x00-0x3F & Request/Response & Standard transport requests that expect responses. \\ +0x40-0x7F & Events & Standard transport events (no response). \\ +0x80-0xBF & Implementation-defined Requests & Optional, transport-specific request extensions. \\ +0xC0-0xFF & Implementation-defined Events & Optional, transport-specific event extensions. \\ +\hline +\end{tabular} + +\devicenormative{\paragraph}{Core Messages}{Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Device} +\begin{itemize} + \item A device MUST implement \msgref{GET_DEVICE_INFO}, + \msgref{GET_DEVICE_FEATURES}, \msgref{SET_DRIVER_FEATURES}, + \msgref{GET_CONFIG}, \msgref{SET_CONFIG}, \msgref{GET_DEVICE_STATUS}, + \msgref{SET_DEVICE_STATUS}, \msgref{GET_VQUEUE}, \msgref{SET_VQUEUE}, + and \msgref{RESET_VQUEUE}. +\end{itemize} + +\drivernormative{\paragraph}{Core Messages}{Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Driver} +\begin{itemize} + \item A driver MUST support sending and interpreting the messages listed + above in order to initialize and operate a virtio-msg device. +\end{itemize} + +As described in Sections +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} +and +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications}, +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band +mechanisms. + +\msgdef{GET_DEVICE_INFO} + +The driver issues \msgref{GET_DEVICE_INFO} to discover static identification +data and limits for a device. It is the only transport message permitted before +the device transitions out of the inactive state. + +\begin{lstlisting} +struct virtio_msg_get_device_info_req { + /* no payload */ +}; + +struct virtio_msg_get_device_info_resp { + le32 device_id; /* virtio device type */ + le32 vendor_id; /* implementation-defined vendor ID */ + le32 num_feature_bits; /* total feature bits (multiples of 32) */ + le32 config_size; /* bytes in the device configuration space */ + le32 max_virtqueues; /* maximum virtqueues supported */ + le16 admin_vq_start; /* index of the first admin virtqueue */ + le16 admin_vq_count; /* number of admin virtqueues */ +}; +\end{lstlisting} + +\devicenormative{\paragraph}{GET\_DEVICE\_INFO}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Device} +\begin{itemize} + \item A device MUST respond to \msgref{GET_DEVICE_INFO} while inactive and + MUST supply consistent values for device ID, vendor ID, feature count (a + multiple of 32 bits), configuration size, and virtqueue limits. +\end{itemize} + +\drivernormative{\paragraph}{GET\_DEVICE\_INFO}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Driver} +\begin{itemize} + \item A driver SHOULD treat the response as the authoritative source for + feature count, configuration size, and virtqueue limits before + proceeding with initialization. +\end{itemize} + +\msgdef{GET_DEVICE_FEATURES} + +Drivers retrieve device feature bits in 32-bit blocks via +\msgref{GET_DEVICE_FEATURES}; the response echoes the requested block index and +returns one or more 32-bit values with the feature bits in that range. + +\begin{lstlisting} +struct virtio_msg_get_device_features_req { + le32 block_index; /* starting block (0 == bits 0-31) */ + le32 num_blocks; /* number of 32-bit blocks requested */ +}; + +struct virtio_msg_get_device_features_resp { + le32 block_index; /* echoed starting block */ + le32 num_blocks; /* echoed number of blocks */ + le32 features[]; /* num_blocks entries, zero-padded if needed */ +}; +\end{lstlisting} + +\devicenormative{\paragraph}{GET\_DEVICE\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Device} +\begin{itemize} + \item A device MUST return zero for any feature bits beyond those it + implements when responding to \msgref{GET_DEVICE_FEATURES}. +\end{itemize} + +\drivernormative{\paragraph}{GET\_DEVICE\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Driver} +\begin{itemize} + \item A driver MUST treat the returned data as 32-bit feature blocks starting + at the requested index, padding with zeros as needed, before deciding + which features to enable. +\end{itemize} + +\msgdef{SET_DRIVER_FEATURES} + +Drivers use \msgref{SET_DRIVER_FEATURES} to acknowledge the subset of feature +bits they wish to enable. The payload mirrors \msgref{GET_DEVICE_FEATURES}, +supplying an index, count, and 32-bit feature data for the selected blocks. + +\begin{lstlisting} +struct virtio_msg_set_driver_features_req { + le32 block_index; /* starting block being acknowledged */ + le32 num_blocks; /* number of blocks provided */ + le32 features[]; /* num_blocks entries with desired bits */ +}; + +struct virtio_msg_set_driver_features_resp { + /* no payload */ +}; +\end{lstlisting} + +\drivernormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Driver} +\begin{itemize} + \item A driver MUST set only the feature bits it intends to enable in the + blocks supplied to \msgref{SET_DRIVER_FEATURES}. +\end{itemize} + +\devicenormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Device} +\begin{itemize} + \item After processing \msgref{SET_DRIVER_FEATURES}, a device MUST update its + acknowledged feature set to match the data supplied and MUST report that + set consistently in subsequent \msgref{GET_DEVICE_FEATURES} responses. + If it cannot support the requested set, it SHOULD clear the FEATURES\_OK + bit in the device status. +\end{itemize} + +\msgdef{GET_CONFIG} + +\msgref{GET_CONFIG} reads a portion of the configuration space. The request +specifies the byte offset and length; the response echoes those values, returns +the data, and includes the current configuration generation count. + +\begin{lstlisting} +struct virtio_msg_get_config_req { + le32 offset; /* byte offset into configuration space */ + le32 length; /* number of bytes requested */ +}; + +struct virtio_msg_get_config_resp { + le32 generation; /* current configuration generation count */ + le32 offset; /* echoed offset */ + le32 length; /* echoed length */ + u8 data[]; /* configuration payload */ +}; +\end{lstlisting} + +\drivernormative{\paragraph}{GET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Driver} +\begin{itemize} + \item A driver MUST ensure the requested offset and length in \msgref{GET_CONFIG} + stay within the configuration size reported by \msgref{GET_DEVICE_INFO}. +\end{itemize} + +\devicenormative{\paragraph}{GET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Device} +\begin{itemize} + \item A device MUST return the current configuration generation count with + every \msgref{GET_CONFIG} response. +\end{itemize} + +\msgdef{SET_CONFIG} + +\msgref{SET_CONFIG} writes a portion of configuration space, supplying the +offset, length, the driver's view of the generation count, and the new data. +The device echoes the offset/length, returns the resulting generation count, +and may mirror the applied data or set the length to zero if the write was +rejected. + +\begin{lstlisting} +struct virtio_msg_set_config_req { + le32 generation; /* driver's view of generation count */ + le32 offset; /* byte offset being written */ + le32 length; /* number of bytes written */ + u8 data[]; /* configuration payload */ +}; + +struct virtio_msg_set_config_resp { + le32 generation; /* resulting generation count */ + le32 offset; /* echoed offset */ + le32 length; /* bytes applied (0 if rejected) */ + u8 data[]; /* optional echoed data */ +}; +\end{lstlisting} + +\drivernormative{\paragraph}{SET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Driver} +\begin{itemize} + \item A driver MUST include its most recent configuration generation count in + each \msgref{SET_CONFIG} request and MUST keep the offset and length + within the reported configuration size. +\end{itemize} + +\devicenormative{\paragraph}{SET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Device} +\begin{itemize} + \item A device MUST reject a \msgref{SET_CONFIG} request if the supplied + generation count does not match its current value and MUST report that + rejection in the response. +\end{itemize} + +\msgdef{GET_DEVICE_STATUS} + +\msgref{GET_DEVICE_STATUS} reads the current device status (e.g., ACKNOWLEDGE, +DRIVER, DRIVER\_OK, FEATURES\_OK, or DEVICE\_NEEDS\_RESET bits). The request has +no payload; the response returns the 32-bit status value. + +\begin{lstlisting} +struct virtio_msg_get_device_status_req { + /* no payload */ +}; + +struct virtio_msg_get_device_status_resp { + le32 status; /* current device status bits */ +}; +\end{lstlisting} + +\devicenormative{\paragraph}{GET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_STATUS / Device} +\begin{itemize} + \item A device MUST report its current status accurately via + \msgref{GET_DEVICE_STATUS}, including whether FEATURES\_OK or + DEVICE\_NEEDS\_RESET are set. +\end{itemize} + +\msgdef{SET_DEVICE_STATUS} + +\msgref{SET_DEVICE_STATUS} writes a new device status value. Drivers use it to +progress through the virtio-defined states or to request a reset by writing 0. +The device responds with its resulting status, which may differ (for example, +if it refuses FEATURES\_OK or sets DEVICE\_NEEDS\_RESET). + +\begin{lstlisting} +struct virtio_msg_set_device_status_req { + le32 status; /* desired device status value */ +}; + +struct virtio_msg_set_device_status_resp { + le32 status; /* resulting device status */ +}; +\end{lstlisting} + +\drivernormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver} +\begin{itemize} + \item A driver MUST write 0 via \msgref{SET_DEVICE_STATUS} to request a device + reset and MUST re-read the status (e.g., via \msgref{GET_DEVICE_STATUS}) + if it needs to confirm acceptance. +\end{itemize} + +\devicenormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Device} +\begin{itemize} + \item A device MAY clear FEATURES\_OK or set DEVICE\_NEEDS\_RESET in its + response if it cannot accept the requested status, but it MUST report + the resulting status accurately. +\end{itemize} + +\msgdef{GET_VQUEUE} + +\msgref{GET_VQUEUE} returns information about a specific virtqueue, including +its maximum size, current size, and, if already configured, the descriptor, +driver, and device area addresses. + +\begin{lstlisting} +struct virtio_msg_get_vqueue_req { + le32 index; /* virtqueue index */ +}; + +struct virtio_msg_get_vqueue_resp { + le32 index; /* echoed virtqueue index */ + le32 max_size; /* maximum queue size */ + le32 cur_size; /* current size (0 if unconfigured) */ + le32 reserved; /* must be zero */ + le64 desc_addr; /* descriptor area address */ + le64 driver_addr; /* driver area address */ + le64 device_addr; /* device area address */ +}; +\end{lstlisting} + +\devicenormative{\paragraph}{GET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} +\begin{itemize} + \item A device MUST report accurate maxima and current queue sizes for each + virtqueue and MUST return zero as the current size if the queue has not + yet been configured. +\end{itemize} + +\msgdef{SET_VQUEUE} + +\msgref{SET_VQUEUE} programs a virtqueue's size and buffer addresses. The driver +selects a queue index, supplies the desired size (not exceeding the maximum +reported via \msgref{GET_VQUEUE}), and provides guest-physical addresses for the +descriptor, driver, and device areas. + +\begin{lstlisting} +struct virtio_msg_set_vqueue_req { + le32 index; /* virtqueue index */ + le32 reserved0; /* must be zero */ + le32 size; /* number of descriptors */ + le32 reserved1; /* must be zero */ + le64 desc_addr; /* descriptor area address */ + le64 driver_addr; /* driver area address */ + le64 device_addr; /* device area address */ +}; + +struct virtio_msg_set_vqueue_resp { + /* no payload */ +}; +\end{lstlisting} + +\drivernormative{\paragraph}{SET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Driver} +\begin{itemize} + \item A driver MUST set the virtqueue size field to a value not exceeding the + maximum reported by \msgref{GET_VQUEUE} and MUST ensure the supplied + addresses point to properly aligned memory. +\end{itemize} + +\devicenormative{\paragraph}{SET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Device} +\begin{itemize} + \item A device MUST configure the queue using the parameters provided via + \msgref{SET_VQUEUE} and treat the queue as inactive until + \msgref{SET_VQUEUE} has successfully completed. +\end{itemize} + +\msgdef{RESET_VQUEUE} + +\msgref{RESET_VQUEUE} resets an individual virtqueue (if VIRTIO\_F\_RING\_RESET +has been negotiated). It requires only the queue index; the device responds +after the queue has been quiesced. + +\begin{lstlisting} +struct virtio_msg_reset_vqueue_req { + le32 index; /* virtqueue index */ +}; + +struct virtio_msg_reset_vqueue_resp { + /* no payload */ +}; +\end{lstlisting} + +\drivernormative{\paragraph}{RESET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Driver} +\begin{itemize} + \item A driver SHOULD issue \msgref{RESET_VQUEUE} only if + VIRTIO\_F\_RING\_RESET has been negotiated and it needs to reconfigure + or recover a specific queue. +\end{itemize} + +\devicenormative{\paragraph}{RESET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Device} +\begin{itemize} + \item Upon receiving \msgref{RESET_VQUEUE}, a device MUST stop processing the + indicated queue, reset its internal state for that queue, and allow the + driver to reconfigure it via \msgref{SET_VQUEUE}. +\end{itemize} + +\msgdef{GET_SHM} + +\msgref{GET_SHM} returns the location and size of a device-owned shared memory +region. The request carries the region index; the response echoes the index and +provides the base address and length. A length of zero indicates that the +specified region does not exist. + +\begin{lstlisting} +struct virtio_msg_get_shm_req { + le32 index; /* shared memory region index */ +}; + +struct virtio_msg_get_shm_resp { + le32 index; /* echoed index */ + le32 length; /* region length (0 if nonexistent) */ + le32 address;/* region address */ +}; +\end{lstlisting} + +\devicenormative{\paragraph}{GET\_SHM}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_SHM / Device} +\begin{itemize} + \item A device MUST return zero length if the requested shared memory region + does not exist and MUST report accurate address/length information for + regions it supports. +\end{itemize} + +\msgdef{EVENT_CONFIG} + +\msgref{EVENT_CONFIG} notifies the driver about configuration or status changes. +The payload includes the current device status, configuration generation count, +and optionally the updated configuration data (offset plus length). A length of +zero indicates that the driver should re-fetch the affected range via +\msgref{GET_CONFIG}. + +\begin{lstlisting} +struct virtio_msg_event_config { + le32 device_status; /* new device status value */ + le32 generation; /* configuration generation count */ + le32 offset; /* configuration offset */ + le32 length; /* number of bytes (may be zero) */ + u8 data[]; /* optional configuration data */ +}; +\end{lstlisting} + +\drivernormative{\paragraph}{EVENT\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Driver} +\begin{itemize} + \item When \msgref{EVENT_CONFIG} provides configuration data, the driver + SHOULD apply it; otherwise it SHOULD re-read the affected range via + \msgref{GET_CONFIG} before proceeding. +\end{itemize} + +\devicenormative{\paragraph}{EVENT\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Device} +\begin{itemize} + \item A device MUST send \msgref{EVENT_CONFIG} whenever it changes + configuration data or status in a way that is visible to the driver, + including the new configuration generation count. +\end{itemize} + +\msgdef{EVENT_AVAIL} + +\msgref{EVENT_AVAIL} notifies the device that a virtqueue has new buffers. The +message includes the queue index and, if VIRTIO\_F\_NOTIFICATION\_DATA is +negotiated, a combined next-offset/next-wrap field so the device can jump +directly to the updated position. Otherwise that field is zero. + +\begin{lstlisting} +struct virtio_msg_event_avail { + le32 vq_index; /* virtqueue index */ + le32 next_offset; /* bits[30:0] offset, bit[31] wrap; zero if unused */ +}; +\end{lstlisting} + +\drivernormative{\paragraph}{EVENT\_AVAIL}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Driver} +\begin{itemize} + \item If VIRTIO\_F\_NOTIFICATION\_DATA has not been negotiated, a driver MUST + set the next offset/wrap field to zero in \msgref{EVENT_AVAIL} messages. + If it has been negotiated, the driver MUST fill in the field with the + current available ring position. +\end{itemize} + +\busnormative{\paragraph}{EVENT\_AVAIL}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Bus} +\begin{itemize} + \item A bus implementation that uses out-of-band notifications SHOULD prevent + negotiation of VIRTIO\_F\_NOTIFICATION\_DATA so that devices do not + expect in-band offset/wrap values. +\end{itemize} + +\msgdef{EVENT_USED} + +\msgref{EVENT_USED} notifies the driver that buffers on a particular virtqueue +have been consumed. The payload carries only the queue index; the driver uses +its used ring to determine which descriptors completed. + +\begin{lstlisting} +struct virtio_msg_event_used { + le32 vq_index; /* virtqueue index */ +}; +\end{lstlisting} + +\devicenormative{\paragraph}{EVENT\_USED}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Device} +\begin{itemize} + \item A device SHOULD send \msgref{EVENT_USED} (or an equivalent notification) + when it processes buffers on a virtqueue unless the driver has + explicitly disabled such notifications. +\end{itemize} + +\drivernormative{\paragraph}{EVENT\_USED}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Driver} +\begin{itemize} + \item Upon receiving \msgref{EVENT_USED}, a driver MUST examine the indicated + virtqueue (for example, by reading the used ring) to reclaim completed + buffers. +\end{itemize} + +\subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages} + +Bus messages cover device discovery, hotplug notifications, and bus-level +liveness checks. Implementations that already provide equivalent functionality +through platform mechanisms (e.g., ACPI, device tree, hypervisor events) may +reuse that infrastructure instead of the messages defined here. The following +subsections describe standardized bus messages that virtio-msg supports today. + +\subsubsection{Overview} +\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / Overview} + +Each bus implementation chooses which of these messages to support. Some may +implement all of them for a fully message-based workflow, while others may rely +on out-of-band enumeration and use only the subset that fits their needs. + +\paragraph{Messages IDs and issuers} + +\begin{tabular}{|l|l|l|} +\hline +Name & ID & Sender \\ +\hline +\hline +Reserved & 0x0 & \\ +\hline +Reserved & 0x1 & \\ +\hline +\busref{GET_DEVICES} & 0x2 & Driver \\ +\hline +\busref{PING} & 0x3 & Any \\ +\hline +\busref{EVENT_DEVICE} & 0x40 & Device \\ +\hline +\end{tabular} + +Bus message IDs are grouped as follows: + +\begin{tabular}{|l|l|p{7cm}|} +\hline +Range & Usage & Notes \\ +\hline +0x00-0x3F & Request/Response & Standardized bus request messages. \\ +0x40-0x7F & Events & Standardized bus event messages. \\ +0x80-0xBF & Implementation-defined Requests & Optional, bus-specific request messages. \\ +0xC0-0xFF & Implementation-defined Events & Optional, bus-specific event messages. \\ +\hline +\end{tabular} + +\paragraph{Bus Specific Messages} +\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Operation / Bus Specific Messages} + +A range of message IDs are reserved for use by the specific bus +implementation. These messages can be used for any implementation specific +usage. Example usage could include: + +\begin{itemize} + \item Configuration of out-of-band notification methods + \item Setup shared memory regions to be used for buffers or virtqueues + \item Declare bus specific error conditions + \item Provide extra debug or logging information +\end{itemize} + +\busdef{GET_DEVICES} + +The driver-side bus uses this request to enumerate device numbers on the +device-side bus. The response describes a “window” of device numbers and signals +which entries are present. + +\begin{lstlisting} +struct virtio_bus_msg_get_devices_req { + le16 offset; /* starting device number */ + le16 count; /* number of device-number slots (multiple of 8) */ +}; + +struct virtio_bus_msg_get_devices_resp { + le16 offset; /* echoed starting device number */ + le16 count; /* window length actually returned */ + le16 next_offset; /* 0 or suggested start for next query */ + u8 bitmap[]; /* count/8 bytes, LSB-first packing */ +}; +\end{lstlisting} + +The \textbf{(offset, count)} tuple defines a window of \textbf{count} +consecutive device numbers beginning at \textbf{offset}. The number of present +devices equals the number of 1-bits in the bitmap. Responders SHOULD return the +requested \textbf{count} unless constrained (e.g., by maximum message size); +otherwise they MAY return a smaller value, in which case the bitmap covers the +reduced window. + +Example: a request with \textbf{offset}=0, \textbf{count}=16 might produce +\textbf{bitmap}=0b00100101 00000000 and \textbf{next_offset}=16. That indicates +devices 0, 2, and 5 are present within the 0–15 window and suggests continuing +at device number 16. + +\paragraph{Intended usage} +Drivers MAY use this message to enumerate device numbers. Treat each window as +a snapshot, advance using \textbf{next_offset}, and confirm candidates via +\msgref{GET_DEVICE_INFO} before issuing other transport messages. + +\busnormative{\paragraph}{GET\_DEVICES}{Virtio Transport Options / Virtio Over Messages / Bus Messages / GET_DEVICES / Bus} +\begin{itemize} + \item The bus-side responder MUST enforce that \textbf{offset} and + \textbf{count} are multiples of 8, and it MUST return a bitmap of length + \textbf{count/8} packed least-significant-bit first. + \item \textbf{next\_offset} MUST be 0 (indicating no further windows) or an + aligned value $\ge \textbf{offset} + \textbf{count}$ recommending where + the driver should query next. + \item Responders SHOULD return the requested \textbf{count} unless constrained + (e.g., by maximum message size); if a smaller count is returned, the + bitmap MUST still represent the window defined by the echoed + \textbf{offset} and \textbf{count}. +\end{itemize} + +\busdef{PING} + +\busref{PING} is a simple liveness check that can be sent by either side of the +bus; the response echoes the 32-bit data value from the request. + +\begin{lstlisting} +struct virtio_bus_msg_ping_req { + le32 data; /* arbitrary value chosen by the sender */ +}; + +struct virtio_bus_msg_ping_resp { + le32 data; /* echoed request data */ +}; +\end{lstlisting} + +\paragraph{Intended usage} +Drivers or bus implementations MAY use \busref{PING} as a keepalive. If a reply +is missing or delayed beyond policy, the initiator SHOULD verify device status +(e.g., via \msgref{GET_DEVICE_STATUS}) and take recovery actions as needed. + +\busnormative{\paragraph}{PING}{Virtio Transport Options / Virtio Over Messages / Bus Messages / PING / Bus} +\begin{itemize} + \item The bus-side responder MUST echo the 32-bit data field from the request + exactly in the \busref{PING} response. +\end{itemize} + +\busdef{EVENT_DEVICE} + +\busref{EVENT_DEVICE} signals device-level hotplug changes. The driver-side bus +receives these indications from the device-side bus; the table below lists the +defined device bus states. + +\begin{lstlisting} +struct virtio_bus_msg_event_device { + le16 device_number; + le16 device_bus_state; /* see table below */ +}; +\end{lstlisting} + +\begin{tabular}{|l|l|p{7cm}|} +\hline +Constant & Value & Meaning \\ +\hline +DEVICE\_BUS\_STATE\_READY & 0x0001 & Device is present and ready. \\ +DEVICE\_BUS\_STATE\_REMOVED & 0x0002 & Device is no longer present. \\ +Reserved & 0x0003–0x7FFF & Reserved for standard use. \\ +Implementation-defined & 0x8000–0xFFFF & MAY be used by the bus implementation. \\ +\hline +\end{tabular} + +\paragraph{Intended usage} +Upon receiving DEVICE\_BUS\_STATE\_READY, drivers SHOULD probe the device via +\msgref{GET_DEVICE_INFO} and proceed with initialization if successful. Upon +DEVICE\_BUS\_STATE\_REMOVED, drivers MUST stop queueing new work, quiesce and +reset as applicable, and release resources. Drivers SHOULD tolerate duplicate or +out-of-order events and MAY rely on additional bus-level monitoring (e.g., +\busref{PING}) if needed. -- 2.34.1 ^ permalink raw reply related [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-01-26 16:32 ` [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer Bill Mills @ 2026-02-06 16:28 ` Peter Hilber 2026-02-10 9:39 ` Bertrand Marquis 2026-02-13 19:09 ` Demi Marie Obenour ` (3 subsequent siblings) 4 siblings, 1 reply; 105+ messages in thread From: Peter Hilber @ 2026-02-06 16:28 UTC (permalink / raw) To: Bill Mills Cc: virtio-comment, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Mon, Jan 26, 2026 at 11:32:28AM -0500, Bill Mills wrote: > Add a new transport layer that is based on messages. > > This transport layer still uses virtqueues as the other transport layers do > but implements transport layer operations by sending and receiving messages > instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. > > This transport is useful when the device and driver are both implemented in > software but the trap and emulate operations of virtio-mmio and virtio-pci > can not be used. > > This transport is intended to be used in many situations, including: > * between a host processor and its co-processors > * between two different systems (not SMP) connected via PCIe > * between normal and secure worlds > * host to vm > * vm to vm Hi, please find some remarks below. As I wrote before, I am not a transport expert, so please take the following comments with a grain of salt. > > Signed-off-by: Bill Mills <bill.mills@linaro.org> > Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> > Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> > Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com> > --- > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 1640 insertions(+) > create mode 100644 transport-msg.tex > > diff --git a/transport-msg.tex b/transport-msg.tex > new file mode 100644 > index 0000000..d4e31d7 > --- /dev/null > +++ b/transport-msg.tex > @@ -0,0 +1,1640 @@ > +\section{Virtio Over Messages}\label{sec:Virtio Transport Options / Virtio Over Messages} > + > +\newcommand{\conceptref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / #1]{#1}} > +\newcommand{\msgref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1]{VIRTIO_MSG_#1}} > +\newcommand{\busref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1]{BUS_MSG_#1}} > +\newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1}} > +\newcommand{\busdef}[1]{\subsubsection{BUS_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1}} > + > +This section defines \textbf{virtio-msg}, a transport mechanism that encapsulates > +virtio operations as discrete message exchanges rather than relying on PCI or > +memory-mapped I/O regions. It separates bus-level functionality (e.g., device > +enumeration, hotplug events) from device-specific operations (e.g., feature > +negotiation, virtqueue setup), ensuring that a single, generic transport layer > +can be reused across multiple bus implementations. > + > +virtio-msg addresses several key objectives: > + > +\begin{itemize} > + \item \textbf{Support multiple bus implementations:} > + Systems can rely on various communication methods such as hypercalls, local > + IPC, network channels, or device trees for enumerating devices. virtio-msg > + defines a common transport interface suitable for any of these mechanisms. > + > + \item \textbf{Reduce per-bus complexity:} > + Buses can implement a fully message-based workflow (including optional > + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE}) > + or they can discover and manage devices through > + alternative means such as platform firmware data. In either case, they > + forward transport messages to and from each device. > + > + \item \textbf{Preserve virtio semantics:} > + The transport leverages standard virtio concepts (features, configuration > + space, virtqueues), so existing virtio drivers and device implementations can > + integrate smoothly once a device is discovered and registered. > +\end{itemize} > + > +\subsection{Basic Concepts} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts} > + > +The virtio-msg transport relies on a set of foundational concepts to ensure > +reusability across different bus implementations and flexibility in device > +capabilities. This section defines those concepts and clarifies how they apply, > +regardless of whether the bus leverages message-based enumeration or platform > +data for device discovery. > + > +\subsubsection{High-Level Architecture} > + > +virtio-msg operates around two layers: > + > +\begin{enumerate} > + \item \textbf{Bus Layer}: A \emph{bus instance} exposes zero or more virtio > + devices. It discovers available devices through mechanisms such as: > + \begin{itemize} > + \item Optional message-based queries (\busref{GET_DEVICES}), > + \item External data sources (e.g., device tree, ACPI tables, hypervisor > + firmware calls), > + \item Dynamic hotplug notifications (optionally via \busref{EVENT_DEVICE}). > + \end{itemize} > + Once a device is identified, regardless of discovery method, the bus > + uses \msgref{GET_DEVICE_INFO} to read its device ID and vendor ID, then > + \emph{registers} that device with the host OS so the usual virtio driver > + probe can occur. > + > + \item \textbf{Transport Layer}: After the bus knows about a device, the > + virtio-msg transport handles all device-specific operations: > + \begin{itemize} > + \item Retrieving and setting features (\msgref{GET_DEVICE_FEATURES}, > + \msgref{SET_DRIVER_FEATURES}), > + \item Accessing the device configuration space (\msgref{GET_CONFIG}, > + \msgref{SET_CONFIG}), > + \item Setting up virtqueues (\msgref{SET_VQUEUE}, \msgref{RESET_VQUEUE}), > + \item Managing status and runtime notifications (\msgref{SET_DEVICE_STATUS}, > + \msgref{EVENT_USED}, etc.). > + \end{itemize} > + These transport messages remain the same across different bus instances, > + allowing a single virtio-msg driver component to function in multiple > + environments. > +\end{enumerate} > + > +\subsubsection{Relationship Between Bus and Transport} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport} > + > +This subsubsection explains the division of responsibilities: the bus layer is > +the mandatory carrier that moves messages between driver and device endpoints > +(e.g., over IPC, shared memory with signalling or hardware messaging), while > +the virtio-msg transport defines the semantics of those messages. > + > +virtio-msg groups messages into two categories: > + > +\begin{description} > + \item[\textbf{Bus Messages}:] > + Intended for global bus operations such as enumerating device numbers > + (\busref{GET_DEVICES}), managing device hotplug events (\busref{EVENT_DEVICE}) > + or assessing bus-wide health (\busref{PING}). > + These messages are \emph{optional} in environments where > + device discovery or state changes occur through other means (e.g., device > + tree). However, if a bus chooses to handle those tasks via messages, > + it implements the appropriate bus message definitions described in this > + section. > + > + \item[\textbf{Transport Messages}:] > + Used for device-specific operations, such as: > + \begin{itemize} > + \item Retrieving or setting features (\msgref{GET_DEVICE_FEATURES}, > + \msgref{SET_DRIVER_FEATURES}), > + \item Accessing device configuration (\msgref{GET_CONFIG}, > + \msgref{SET_CONFIG}), > + \item Managing virtqueues (\msgref{SET_VQUEUE}, \msgref{RESET_VQUEUE}), > + \item Handling device status and notifications (\msgref{SET_DEVICE_STATUS}, > + \msgref{EVENT_USED}, etc.). > + \end{itemize} > +\end{description} > + > +This separation lets a bus remain minimal if it obtains device information from > +firmware tables, while still supporting fully message-based enumeration and > +hotplug when desired. > + +\busnormative{\paragraph}{Bus Message Implementation}{Virtio > Transport Options / Virtio Over Messages / Basic Concepts / > Relationship between bus and transport / Bus Messages} > +\begin{itemize} > + \item A bus implementation that provides enumeration, hotplug, or bus > + health handling via bus messages SHOULD implement the corresponding Bus > + Message definitions described in this section. > +\end{itemize} > + > +\busnormative{\paragraph}{Transport Message Forwarding}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport / Transport Message Forwarding} > +\begin{itemize} > + \item A bus implementation MUST relay each transport message to the device > + number identified in the message header, regardless of how it > + discovered or enumerated that device. > + \item A bus implementation SHOULD treat transport messages as opaque apart > + from enforcing generic transport limits, such as the advertised maximum > + message size, and SHOULD NOT modify the transport payload. > +\end{itemize} > + > +\subsubsection{System Topology} > + > +A virtio-msg system contains the following elements: > + > +\begin{itemize} > + \item \textbf{Bus Instances and Devices}: Each bus instance advertises its > + capabilities (e.g., transport revision, maximum message size) and > + discovers devices via message-based queries or external data sources. > + Every discovered device has a unique \emph{device number}. "Device number" is also used as a synonym for "device ID" in a few places in the existing spec. Maybe these places could be changed to "device ID" for clarity? > + \item \textbf{Driver}: Communicates with the bus to learn about > + available devices. Once a device is recognized, the driver uses the > + common transport messages to perform feature negotiation, configuration, > + and virtqueue setup. > + \item \textbf{Device}: Implement virtio device functionality and > + respond to the transport messages. The bus forwards these messages to Minor editorial: Implement -> Implements respond -> responds > + the correct device instance based on device number. > +\end{itemize} > + > +\subsubsection{Transport Revisions and Maximum Message Size} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions} > + > +Each \textbf{virtio-msg bus instance} advertises the following to the transport > +layer: > +\begin{itemize} > + \item \textbf{Transport revision}: the protocol version supported by the bus > + instance (independent of the overall Virtio specification version). > + \item \textbf{Maximum message size}: the largest payload (in bytes) that can > + be carried per request or response, including the common header. > + \item \textbf{Transport feature bits}: revision-specific optional features > + implemented by the bus instance. > +\end{itemize} > + > +The mechanism for obtaining these parameters is implementation-defined and can > +vary between bus instances. Common approaches include: > +\begin{itemize} > + \item Reading firmware or device-tree data that describes each bus instance, > + \item Performing a message exchange during bus setup to retrieve the values, > + \item Relying on per-bus configuration structures or driver-specific defaults. > +\end{itemize} > + > +\busnormative{\paragraph}{Advertising Transport Parameters}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Advertising Transport Parameters} > +\begin{itemize} > + \item Each bus instance MUST make its transport revision, maximum message > + size, and transport feature bits available to the virtio-msg transport > + before any transport messages are exchanged for that device. > + \item A bus instance MUST apply the same limits to both driver-originated and > + device-originated transport messages; if the values change, the bus > + MUST inform the transport layer before accepting additional messages. > +\end{itemize} > + > +\drivernormative{\paragraph}{Respecting Bus Limits}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Driver Limits} > +\begin{itemize} > + \item A driver MUST NOT send a transport message whose total size exceeds the > + maximum message size advertised for the target bus instance. > + \item A driver MUST NOT rely on transport features or messages that require a > + higher transport revision than the bus instance reports. > +\end{itemize} > + > +\devicenormative{\paragraph}{Respecting Bus Limits}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Device Limits} > +\begin{itemize} > + \item A device MUST ensure its responses and device-originated messages do > + not exceed the maximum message size advertised by the bus instance that > + relays them. > + \item A device MUST NOT require transport features or messages beyond the > + transport revision reported by the bus instance, and MUST respond with > + an error or ignore requests for unsupported features. > +\end{itemize} > + > +\paragraph{virtio-msg revisions} IIUC, this is called "transport revision" elsewhere. I suggest to unify the naming. > + > +The following table lists the currently defined virtio-msg revisions: > + > +\begin{tabular}{ |l|l|l|l| } > +\hline > +\field{revision} & \field{maximum size} & \field{features} & remarks \\ > +\hline \hline > +1 & 44-65536 & <empty> & Virtio Message Revision 1 \\ > +\hline > +\end{tabular} > + > +Note that a change in the virtio standard does not necessarily > +correspond to a change in the virtio-msg revision. > + > +The maximum message size is specified from the transport-layer point of view > +and includes the 8-byte common header plus payload. Any extra encapsulation > +imposed by the underlying bus (for example, a framing header) does not count > +against this limit. Today the largest practical transport payload is 256 bytes: > +messages such as GET_CONFIG and SET_CONFIG can carry up to 256 bytes of device > +configuration in one transfer, and larger configuration regions can be accessed > +through multiple exchanges without requiring a larger per-message limit. > + > +\busnormative{\paragraph}{Message Size Bounds}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Message Size} > +\begin{itemize} > + \item A bus implementation MUST advertise a maximum message size of at least > + 44 bytes. > + \item A bus implementation SHOULD NOT advertise a maximum message size that > + exceeds 264 bytes (256-byte payload plus the common header). > +\end{itemize} > + > +\paragraph{Versioning and Forward Compatibility} > + > +A higher transport revision or additional transport feature bits extend the > +protocol with new messages or capabilities. Implementations are expected to > +remain interoperable across revisions: devices and drivers designed for a newer > +revision still implement the mandatory messages and semantics defined in prior > +revisions. > + > +\drivernormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Driver} > +\begin{itemize} > + \item A driver that negotiates transport revision $N$ MUST implement all > + mandatory driver behavior defined for revisions $1$ through $N$. > + \item If a driver receives a device-originated message or feature indication > + that requires an unsupported revision or transport feature, it MUST > + ignore the message (or treat the request as failed) and MUST NOT act on > + partially understood data. > +\end{itemize} > + > +\devicenormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Device} > +\begin{itemize} > + \item A device that advertises transport revision $N$ MUST implement all > + mandatory device behavior defined for revisions $1$ through $N$. > + \item If a device receives a driver request that relies on an unsupported > + revision or transport feature, it MUST reject the request using the > + message-specific error mechanism (if any) or silently ignore it. > +\end{itemize} > + > +\busnormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Bus} > +\begin{itemize} > + \item A bus instance that advertises transport revision $N$ MUST satisfy every > + bus requirement defined for revisions $1$ through $N$. Are transport feature bits supposed to be negotiated? I think this could make sense so that behavior can change without affecting interoperability between implementations of different revisions of the Virtio standard. > + \item If a bus instance cannot forward a message because it requires an > + unsupported revision or transport feature, it MUST surface a transport > + error or drop the message without forwarding it. > +\end{itemize} > + > +\subsubsection{Device Numbers and Enumeration} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / > +Device Numbers} > + > +Each virtio-msg bus instance contains one or more \emph{devices}, identified > +by a 16-bit \textbf{device number}. Buses discover these device numbers through > +mechanisms such as: > +\begin{itemize} > + \item \textbf{Message-Based Enumeration}: Using \busref{GET_DEVICES} to query > + which numbers exist (optional). > + \item \textbf{Platform Data}: A device tree, ACPI tables, or hypervisor calls > + might inform the bus of available device numbers and their properties. > +\end{itemize} > + > +Once a bus confirms that a device number is valid—regardless of the discovery > +method—it normally issues \msgref{GET_DEVICE_INFO} to retrieve the device and > +vendor IDs before registering the device with the host OS so the usual Virtio > +driver binding process can begin. > + > +\busnormative{\paragraph}{Device Number Assignment}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Device Numbers / Assignment} > +\begin{itemize} > + \item A bus implementation MUST assign a unique device number to every > + device on a given bus instance and MUST NOT forward transport messages > + for a device number that has not been validated. > + \item A bus implementation SHOULD provide the driver with sufficient > + information—either via \busref{GET_DEVICES} or equivalent platform > + data—to discover each valid device number. > +\end{itemize} > + > +\subsubsection{Configuration Generation Count} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count} > + > +Each device maintains a \textbf{Configuration Generation Count} to prevent > +inconsistent updates. This count is incremented at least once by the device for > +every driver-visible change it makes to its configuration data. The current > +count is exposed in \msgref{EVENT_CONFIG} and \msgref{GET_CONFIG} responses so > +the driver can determine whether its view of the configuration is current. The > +count does not necessarily start at zero and is not automatically reset when > +the device resets. Could the spec clarify whether the device MUST NOT, or MAY reset it? > + > +\devicenormative{\paragraph}{Configuration Generation Count}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Device} > +\begin{itemize} > + \item A device MUST increment the generation count before it makes a change > + that is visible to the driver and MUST ensure that each > + \msgref{EVENT_CONFIG} carrying configuration data uses a unique > + generation count. > + \item If updated configuration data cannot fit in a single > + \msgref{EVENT_CONFIG}, the device SHOULD send an \msgref{EVENT_CONFIG} > + with zero data length to advertise the new generation count and MUST > + make the updated data available via \msgref{GET_CONFIG}. > + \item If a \msgref{SET_CONFIG} request includes a generation count that does > + not match the device's current count, the device MUST reject the > + request. > +\end{itemize} > + > +\drivernormative{\paragraph}{Configuration Generation Count}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Driver} > +\begin{itemize} > + \item A driver MUST track the most recent generation count observed (via > + \msgref{EVENT_CONFIG} or \msgref{GET_CONFIG}) and include it in every > + \msgref{SET_CONFIG} request. > + \item If a \msgref{SET_CONFIG} request is rejected due to a mismatched > + generation count, the driver SHOULD issue \msgref{GET_CONFIG} to obtain > + the latest configuration data and generation count before retrying. In my understanding this only helps if the driver for the specific device type reacts to this rejection. Currently, other standardized transports - and consequently Linux kernel drivers - assume that configuration writes will not be rejected like this. It might be preferable to make the generation count in SET_CONFIG optional, or to drop it. IIUC the Linux kernel driver mentioned in the cover letter currently does not retry and is therefore susceptible to races, so that some configuration writes might not take effect. > +\end{itemize} > + > +This mechanism ensures updates are not lost or overwritten due to stale > +information. > + > +\subsubsection{Feature Negotiation Blocks} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Feature Blocks} > + > +The virtio-msg transport organizes feature bits into 32-bit blocks accessed by > +\msgref{GET_DEVICE_FEATURES} and \msgref{SET_DRIVER_FEATURES}. Each block > +represents up to 32 feature bits: > + > +\begin{itemize} > + \item \textbf{Block Index}: The starting block (e.g., block 0 for > + features 0--31, block 1 for features 32--63, etc.). > + \item \textbf{Number of Blocks}: How many blocks the driver wishes to retrieve > + or modify in a single message. > + \item \textbf{Feature Data}: The 32-bit values representing the supported (or > + driver-requested) feature bits in the selected blocks. > +\end{itemize} > + > +\devicenormative{\paragraph}{Feature Blocks}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Feature Blocks / Device} > +\begin{itemize} > + \item When \msgref{GET_DEVICE_FEATURES} covers blocks that extend beyond the > + features a device implements, the device MUST return zero for the > + feature data in those positions. > +\end{itemize} > + > +\subsubsection{Error Signaling} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling} > + > +Transport errors can arise from malformed messages, routing failures inside a > +bus implementation, or device-side faults while processing a valid request. > +Implementations should handle such faults locally where possible, but a bus may > +surface an error to the virtio-msg transport if it cannot deliver a request or > +obtain a response within its policy. > + > +\busnormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus} > +\begin{itemize} > + \item A bus implementation MAY report a transport-visible failure (for > + example, after exhausting a bounded retry policy) when it cannot deliver > + a request or obtain a response. > + \item A bus implementation MUST treat malformed headers or unsupported > + \field{msg_id} values as invalid, MUST discard them without generating > + additional protocol traffic, and MAY log the condition for diagnostics. > + \item A bus implementation MUST NOT generate error responses to event > + (one-way) messages. > +\end{itemize} > + > +\devicenormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device} > +\begin{itemize} > + \item A device receiving a malformed or unsupported transport message MUST > + discard it without producing further protocol traffic. > + \item Recovery actions taken in response to an error (such as retries, > + selective resets, or device removal) MUST follow the normative reset and > + status semantics defined in > + \ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}. > +\end{itemize} > + > +This specification does not define a dedicated error-reporting message; it only > +permits implementations to surface failures when silent recovery is not > +feasible. > + > +\subsubsection{Endianness} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness} > + > +Unless otherwise stated, all numeric fields defined for virtio-msg messages use > +little-endian encoding. > + > +\drivernormative{\paragraph}{Endianness}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Driver} > +\begin{itemize} > + \item A driver MUST encode and decode the common header and payload fields > + defined by this transport using little-endian byte order. > +\end{itemize} > + > +\devicenormative{\paragraph}{Endianness}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Device} > +\begin{itemize} > + \item A device MUST emit and consume the common header and payload fields > + defined by this transport using little-endian byte order. > +\end{itemize} > + > +\subsubsection{Common Message Format} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format} > + > +All virtio-msg exchanges, whether \emph{bus messages} or \emph{transport > +messages}, begin with an 8-byte header followed by an optional payload. The > +fields below describe the wire format for that header. > + > +The header layout is: > +\begin{lstlisting} > +struct virtio_msg_header { > + u8 type; /* request/response + bus/transport */ > + u8 msg_id; /* message id */ > + le16 dev_num; /* device number (0 for bus messages) */ > + le16 token; /* bus-managed correlation identifier */ > + le16 msg_size; /* total size: header (8) + payload */ > + u8 payload[]; > +}; > +\end{lstlisting} > + > +Field semantics: > +\begin{itemize} > + \item \field{type}: > + \begin{itemize} > + \item Bit[0]: 0=request, 1=response. > + \item Bit[1]: 0=transport message, 1=bus message. > + \item Bits[2..7]: reserved for future use. > + \end{itemize} > + \item \field{msg_id}: Message ID identifying the message definition. Ranges > + are defined in > + \ref{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} > + and > + \ref{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages}. > + \item \field{dev_num}: For transport messages, the device number that should > + receive the message. Bus messages operate on device number 0. > + \item \field{token}: Correlation identifier managed by the bus. Drivers and > + devices treat this field as opaque; the bus MAY overwrite it to track > + outstanding requests or maintain ordering. > + \item \field{msg_size}: Total size in bytes of the complete message (header > + plus payload). > + \item \field{payload}: Operation-specific data. If a bus introduces extra > + padding bytes, those bytes are not part of the payload semantics. > +\end{itemize} > + > +\drivernormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Driver} > +\begin{itemize} > + \item A driver MUST set bits 2..7 of \field{type} to zero and MUST treat them > + as reserved when parsing received headers. > + \item A driver MUST ensure \field{msg_size} reflects the total message length > + (header plus payload) and MUST NOT exceed the maximum message size > + advertised by the bus instance. > + \item When sending a transport message, a driver MUST set \field{dev_num} to > + the intended device number. > + \item If a driver introduces padding bytes that become part of the transport > + payload, it MUST set those bytes to zero. > + \item A driver MUST treat the \field{token} field as opaque and MUST NOT rely > + on its value when processing received messages. > +\end{itemize} > + > +\devicenormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Device} > +\begin{itemize} > + \item A device MUST set bits 2..7 of \field{type} to zero in transmitted > + messages and MUST ignore those bits on receive. > + \item A device MUST ensure \field{msg_size} reflects the total message length > + (header plus payload) and does not exceed the bus's advertised maximum. > + \item When sending a transport message, a device MUST set \field{dev_num} to > + its own device number. > + \item A device MUST ignore padding bytes that are documented as bus-specific > + and MUST zero any such bytes it introduces into the transport payload. > + \item A device MUST treat the \field{token} field as opaque and MUST NOT rely > + on its value when processing received messages. > +\end{itemize} > + > +\busnormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Bus} > +\begin{itemize} > + \item A bus implementation MUST deliver bus messages with \field{dev_num}=0 > + and MUST NOT alter \field{dev_num} for transport messages beyond the > + routing needed to reach the addressed device. > + \item A bus implementation MUST set bits 2..7 of \field{type} to zero when > + generating bus messages and MUST ignore those bits when forwarding > + transport messages. > + \item If the bus adds framing or padding bytes around the common header or > + payload, it MUST set those bytes to zero before delivering the message > + to the opposite side and MUST present the same zero padding when the > + opposite side reads a message. > + \item A bus implementation owns the \field{token} field; it MAY insert or > + overwrite values for correlation purposes and MUST ensure that any such > + use is transparent to drivers and devices. > +\end{itemize} > + > +Reserved header bits and unspecified header values MUST be transmitted as zero > +and ignored on receive to preserve forward compatibility. This should be phrased in active tense: -> The bus implementation MUST transmit... > + > +\subsubsection{Message Ordering} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering} > + > +Transport messages fall into two classes: requests (which expect responses) and > +events (which are one-way). Drivers and devices rely on the bus to preserve the > +relative ordering of request/response pairs for each device number; they do not > +interpret the \field{token} field directly. > + > +\busnormative{\paragraph}{Message Ordering}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} > +\begin{itemize} > + \item For each device number, a bus implementation MUST deliver responses to > + the driver in the same order that it forwarded the corresponding > + requests to the device. > + \item A bus implementation MUST ensure that every request forwarded to a > + device results in exactly one response delivered to the driver (unless > + the request is defined as an event). > +\end{itemize} > + > +\subsection{Device Discovery}\label{sec:Virtio Transport Options / Virtio Over Messages / Device Discovery} > + > +A virtio-msg implementation can learn about devices via bus-level enumeration > +(e.g., \busref{GET_DEVICES}), platform data (e.g., device tree, ACPI), or > +hypervisor/firmware interfaces. The following informative text describes > +typical flows; the normative obligations follow. > + > +Bus implementations discover their devices through a mix of platform data, > +hypervisor-provided enumeration, or message-based queries such as > +\busref{GET_DEVICES}. This specification defines \busref{GET_DEVICES} for > +environments that prefer message-driven enumeration, but it does not require > +its use when equivalent information is already known out-of-band. > + > +\busnormative{\paragraph}{Device Discovery}{Virtio Transport Options / Virtio Over Messages / Device Discovery / Bus} > +\begin{itemize} > + \item A bus implementation MUST ensure that every device number it exposes to > + the driver has been validated via either platform data or a successful > + query such as \busref{GET_DEVICES}. > + \item Once a device number is deemed present, the bus implementation SHOULD > + invoke \msgref{GET_DEVICE_INFO} so it can register the device with the > + host OS and allow the appropriate virtio driver to bind. > + \item If a bus implementation provides an alternative enumeration mechanism > + (e.g., ACPI, device tree), it MAY omit \busref{GET_DEVICES} provided the > + alternative delivers equivalent information to the driver. > +\end{itemize} > + > +\subsection{Device Initialization} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization} > + > +After a bus has identified a virtio-msg device (whether by message-based > +enumeration or platform-specific discovery), the driver undertakes a series of > +steps to configure and ready the device for normal operation. This section > +details the recommended order of operations and the associated messages. > + > +\subsubsection{Initialization Flow Overview} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview} > + > +A typical device initialization flow includes: > +\begin{enumerate} > + \item \textbf{Obtain Device Information:} > + Query static parameters with \msgref{GET_DEVICE_INFO}. > + \item \textbf{Negotiate Features:} > + Read feature blocks via \msgref{GET_DEVICE_FEATURES}, decide which to > + enable, and write them back with \msgref{SET_DRIVER_FEATURES} before > + updating the device status. > + \item \textbf{Initialize Configuration Space:} > + Read or write configuration data with > + \msgref{GET_CONFIG}/\msgref{SET_CONFIG}, guarding updates with the > + configuration generation count. > + \item \textbf{Set Up Virtqueues:} > + Configure each virtqueue via \msgref{SET_VQUEUE} and verify the settings > + with \msgref{GET_VQUEUE}. > + \item \textbf{Set Device Status:} > + Use \msgref{SET_DEVICE_STATUS} to drive the device through the standard > + virtio status states. > +\end{enumerate} > + > +The exact order may vary slightly across implementations, but these steps form > +the baseline for virtio-msg initialization. > + > +\drivernormative{\paragraph}{Initialization Flow}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview / Driver} > +\begin{itemize} > + \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature > + negotiation or queue setup. > + \item A driver MUST complete feature negotiation via > + \msgref{GET_DEVICE_FEATURES}/\msgref{SET_DRIVER_FEATURES} > + and confirm the FEATURES\_OK state via \msgref{SET_DEVICE_STATUS} before > + enabling virtqueues. > + \item A driver MUST configure each virtqueue via \msgref{SET_VQUEUE} and > + confirm its parameters (e.g., with \msgref{GET_VQUEUE}) before marking > + the queue ready for I/O. > + \item A driver MUST drive the device status transitions using > + \msgref{SET_DEVICE_STATUS}, ensuring the device reaches DRIVER\_OK > + before issuing normal I/O. > +\end{itemize} > + > +\subsubsection{Device Information} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Information} > + > +Once a device number is confirmed, the driver uses \msgref{GET_DEVICE_INFO} to > +retrieve static identification data (device ID, vendor ID), the number of > +feature bits, configuration space size, maximum virtqueues, shared memory > +segments, and any admin virtqueue details. This information determines which > +virtio driver should bind to the device, how many feature blocks to query, and > +how much configuration space is valid. > + > +\drivernormative{\paragraph}{Device Information}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Information / Driver} > +\begin{itemize} > + \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature > + negotiation or queue setup so it can discover the device ID, vendor ID, > + feature count, configuration size, and virtqueue limits. "A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature negotiation or queue setup" is already stipulated in the previous paragraph "Initialization Flow". > + \item A driver MUST use the configuration size reported via > + \msgref{GET_DEVICE_INFO} to bound any offsets and lengths supplied in > + \msgref{GET_CONFIG} and \msgref{SET_CONFIG} requests. > +\end{itemize} > + > +\subsubsection{Feature Negotiation} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features} > + > +Drivers read available features in 32-bit blocks using > +\msgref{GET_DEVICE_FEATURES}; the device returns the requested bitfields, > +padding with zeros for any out-of-range blocks. To enable selected features, the > +driver writes them back with \msgref{SET_DRIVER_FEATURES} and then updates the > +device status via \msgref{SET_DEVICE_STATUS}, checking whether the FEATURES\_OK > +bit remains set. > + > +\drivernormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Driver} > +\begin{itemize} > + \item A driver MUST use \msgref{GET_DEVICE_FEATURES} to discover the feature > + bits offered by the device and MUST write back only the features it > + intends to enable via \msgref{SET_DRIVER_FEATURES}. > + \item After writing the desired features, the driver MUST attempt to set the > + FEATURES\_OK bit using \msgref{SET_DEVICE_STATUS} and MUST check the > + returned status to ensure the device accepted the set. > +\end{itemize} > + > +\devicenormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} > +\begin{itemize} > + \item When handling \msgref{GET_DEVICE_FEATURES}, a device MUST return zero > + for any requested bits that fall outside the number of feature bits it > + implements. > + \item After receiving \msgref{SET_DRIVER_FEATURES}, a device MUST update its > + internal feature mask to match the acknowledged set and MUST reflect > + acceptance or rejection by leaving the FEATURES\_OK bit set or clearing > + it in the status returned by \msgref{SET_DEVICE_STATUS}. > +\end{itemize} > + > +\subsubsection{Device Configuration} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration} > + > +Drivers use \msgref{GET_CONFIG} to read portions of the configuration space by > +supplying an offset and length; the device returns the requested data plus the > +current configuration generation count. Writing is performed via > +\msgref{SET_CONFIG}, which carries the same offset/length along with the > +driver's notion of the generation count and the new data. > + > +\drivernormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Driver} > +\begin{itemize} > + \item A driver MUST ensure that the offset and length in each > + \msgref{GET_CONFIG} or \msgref{SET_CONFIG} request stay within the > + configuration size reported by \msgref{GET_DEVICE_INFO}. > + \item A driver MUST include its most recently observed configuration > + generation count in a \msgref{SET_CONFIG} request and SHOULD re-read the > + configuration (via \msgref{GET_CONFIG}) if the request is rejected for a > + generation mismatch. > +\end{itemize} > + > +\devicenormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Device} > +\begin{itemize} > + \item A device MUST reject a \msgref{SET_CONFIG} request whose generation > + count does not match its current value and MUST indicate the rejection > + in the response. > + \item A device MUST return the current configuration generation count > + alongside any data returned via \msgref{GET_CONFIG}. > +\end{itemize} > + > +\subsubsection{Virtqueue Configuration} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration} > + > +Drivers query virtqueue parameters with \msgref{GET_VQUEUE}, configure them via > +\msgref{SET_VQUEUE}, and optionally reset them using \msgref{RESET_VQUEUE} (if > +VIRTIO\_F\_RING\_RESET is negotiated). Each queue is typically configured by > +reading its maximum size, provisioning descriptor/available/used buffers, and > +then calling \msgref{SET_VQUEUE} with the chosen size and guest-physical > +addresses. > + > +\drivernormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Driver} > +\begin{itemize} > + \item A driver MUST use \msgref{GET_VQUEUE} to determine the maximum queue > + size and confirm that a queue is inactive before programming it. > + \item A driver MUST ensure the queue size provided in \msgref{SET_VQUEUE} does > + not exceed the maximum reported by the device and MUST supply valid > + descriptor/driver/device addresses before enabling the queue. > + \item If VIRTIO\_F\_RING\_RESET has been negotiated and a queue requires > + reinitialization, the driver SHOULD use \msgref{RESET_VQUEUE} before > + reprogramming it. > +\end{itemize} > + > +\devicenormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} > +\begin{itemize} > + \item A device MUST report accurate maximum queue sizes in \msgref{GET_VQUEUE} > + and MUST persist the parameters supplied via \msgref{SET_VQUEUE} (size, > + descriptor, driver, and device addresses). > + \item When \msgref{RESET_VQUEUE} is issued (and VIRTIO\_F\_RING\_RESET is > + negotiated), the device MUST quiesce the queue, release any resources > + associated with it, and allow the driver to reconfigure it. > +\end{itemize} > + > +\subsubsection{Status Information} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information} > + > +Drivers query the device status via \msgref{GET_DEVICE_STATUS} to observe > +progress or detect errors, and they drive the Virtio status transitions via > +\msgref{SET_DEVICE_STATUS}. Writing zero to the status field resets the device, > +invalidating any configuration or virtqueue state. > + > +\drivernormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} > +\begin{itemize} > + \item A driver SHOULD read the device status via \msgref{GET_DEVICE_STATUS} > + when diagnosing errors or determining whether the device is ready to > + move to the next initialization phase. > + \item A driver MUST use \msgref{SET_DEVICE_STATUS} to drive the device through > + the virtio-defined status states and MUST write 0 to request a device > + reset when needed. > +\end{itemize} > + > +\devicenormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} > +\begin{itemize} > + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST > + reset its internal state, invalidate existing configuration and > + virtqueue settings, and present the status field as 0. > + \item A device MUST report its current status accurately via > + \msgref{GET_DEVICE_STATUS}, including whether the FEATURES\_OK bit has > + been accepted or cleared. > +\end{itemize} > + > +\subsubsection{Finalizing Initialization} > + > +After configuring virtqueues and agreeing on features, the driver transitions > +the device to DRIVER\_OK (and any other required status bits) via > +\msgref{SET_DEVICE_STATUS}. At that point, the device is considered ready for > +normal virtqueue I/O. > + > +\drivernormative{\paragraph}{Final Status}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Driver} > +\begin{itemize} > + \item A driver MUST set DRIVER\_OK via \msgref{SET_DEVICE_STATUS} only after > + it has completed feature negotiation and initialized all required > + virtqueues. > + \item A driver MUST NOT queue normal I/O until the device reports a status > + that includes DRIVER\_OK. > + \item A driver MUST NOT supply buffers or send driver notifications for a > + virtqueue until that queue has been configured via \msgref{SET_VQUEUE}. > +\end{itemize} > + > +\devicenormative{\paragraph}{Final Status}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Device} > +\begin{itemize} > + \item A device MUST NOT process normal virtqueue I/O until the driver has set > + DRIVER\_OK. > + \item Once DRIVER\_OK is set, the device MUST begin processing virtqueue > + requests subject to the negotiated features and queue configurations. > +\end{itemize} > + > +\subsection{Device Operation} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation} > + > +Once a virtio-msg device is fully initialized (see > +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization}), > +the driver and device exchange messages to perform I/O and respond to > +configuration changes. This section details the primary messaging paths for > +\emph{driver notifications}, \emph{device notifications}, and the handling of > +runtime resets or shutdown sequences. > + > +\subsubsection{Driver Notifications} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} > + > +After buffers are made available to a virtqueue, the driver issues > +\msgref{EVENT_AVAIL} to notify the device. A bus implementation may translate > +these notifications into an out-of-band mechanism or deliver them in-band. > + > +\drivernormative{\paragraph}{Driver Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Driver} > +\begin{itemize} > + \item A driver MUST send \msgref{EVENT_AVAIL} when it places new buffers on a > + virtqueue. I suggest not defining when exactly the driver/device should send notifications (cf. "Driver and Device Event Suppression", which is not addressed here). It should suffice to state that the driver gives available buffer notifications through EVENT_AVAIL, and to add similar statements for configuration change and used buffer notifications. > + \item Each \msgref{EVENT_AVAIL} MUST identify the target virtqueue index. > + \item If VIRTIO\_F\_NOTIFICATION\_DATA has been negotiated, the driver MUST > + populate the “next offset and wrap” fields so the device can locate the > + head of the updated ring; otherwise those fields MUST be zero. > +\end{itemize} > + > +\busnormative{\paragraph}{Driver Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Bus} > +\begin{itemize} > + \item A bus implementation MAY convert \msgref{EVENT_AVAIL} into an > + out-of-band notification but, if it does not, it MUST relay the message > + over the virtio-msg channel. > + \item A bus implementation MUST NOT drop \msgref{EVENT_AVAIL} unless it has > + arranged for the device to detect new buffers through polling or an > + equivalent mechanism. > +\end{itemize} > + > +\subsubsection{Device Notifications} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications} > + > +\msgref{EVENT_CONFIG} and \msgref{EVENT_USED} provide asynchronous notifications > +from the device (or device-side bus) to the driver. The bus may forward these > +messages in-band or synthesize them based on other signals such as interrupts > +or polling. > + > +\devicenormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} > +\begin{itemize} > + \item A device (or device-side bus) MUST send \msgref{EVENT_CONFIG} whenever > + it makes a configuration change or status update that becomes visible to > + the driver. The message MUST include the new configuration generation > + count and MAY include the updated configuration data. > + \item If the configuration data is omitted from \msgref{EVENT_CONFIG}, the > + device SHOULD include the relevant offsets/lengths so the driver can > + re-fetch the data via \msgref{GET_CONFIG}. > + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when > + buffers on a virtqueue have been consumed, unless the device relies on > + an alternative, agreed-upon completion mechanism. Should this not be a MUST requirement? The driver might hang or need to poll otherwise. > +\end{itemize} > + > +\drivernormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Driver} > +\begin{itemize} > + \item Upon receiving \msgref{EVENT_CONFIG}, a driver SHOULD update its view of > + the configuration using the provided data (or by issuing > + \msgref{GET_CONFIG} if necessary) before resuming normal I/O. > + \item Upon receiving \msgref{EVENT_USED}, a driver MUST examine the indicated > + virtqueue (for example, by reading the used ring) to reclaim completed > + buffers. > +\end{itemize} > + > +\busnormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Bus} > +\begin{itemize} > + \item A bus implementation MUST forward \msgref{EVENT_CONFIG} and > + \msgref{EVENT_USED} notifications (or their equivalents) to the driver, > + either in-band or by synthesizing the appropriate message. > + \item If a bus implementation relies on polling or other mechanisms instead of > + direct notifications, it SHOULD limit that mode to scenarios where no > + other notification method is available. > +\end{itemize} > + > +\subsubsection{Configuration Changes During Operation} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation} > + > +Drivers may update configuration fields at runtime using \msgref{SET_CONFIG} > +when features such as device modes or limits need to change. Devices can also > +update configuration data autonomously but must signal those changes via > +\msgref{EVENT_CONFIG}. > + > +\drivernormative{\paragraph}{Runtime Configuration}{Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Driver} > +\begin{itemize} > + \item A driver MAY issue \msgref{SET_CONFIG} after initialization, provided it > + includes the current configuration generation count and follows the same > + validation rules as during setup. > + \item Upon receiving an \msgref{EVENT_CONFIG}, the driver SHOULD refresh its > + view of the configuration (via the data provided or by reissuing > + \msgref{GET_CONFIG}) before relying on the updated values. > +\end{itemize} > + > +\devicenormative{\paragraph}{Runtime Configuration}{Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Device} > +\begin{itemize} > + \item If a device updates its configuration after initialization, it MUST send > + \msgref{EVENT_CONFIG} to inform the driver of the change and provide the > + new configuration generation count. > +\end{itemize} > + > +\subsubsection{Virtqueue Changes During Operation} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation} > + > +Drivers may provision unused virtqueues later in the device lifetime by issuing > +\msgref{SET_VQUEUE}, and they may reconfigure existing queues if the > +VIRTIO\_F\_RING\_RESET feature has been negotiated. > + > +\drivernormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Driver} > +\begin{itemize} > + \item A driver MAY configure additional virtqueues after initialization using > + \msgref{SET_VQUEUE}, provided it follows the same validation steps > + (e.g., checking the maximum queue size). > + \item If VIRTIO\_F\_RING\_RESET is negotiated, the driver SHOULD use > + \msgref{RESET_VQUEUE} before reprogramming a queue to avoid races with > + the device. I do not understand how this avoids races which could not be prevented through a single message. > +\end{itemize} > + > +\devicenormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Device} > +\begin{itemize} > + \item A device MUST honor \msgref{SET_VQUEUE} requests issued after > + initialization and update the queue parameters accordingly. > + \item When \msgref{RESET_VQUEUE} is received (and VIRTIO\_F\_RING\_RESET is > + negotiated), the device MUST quiesce the queue and allow the driver to > + reconfigure it without processing stale data. > +\end{itemize} > + > +\subsubsection{Device Reset and Shutdown} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Reset} > + > +Drivers reset a device by writing 0 to the status field via \msgref{SET_DEVICE_STATUS}, > +which forces the device to discard pending operations and return to the initial state. > +A device may also signal that a reset is required by sending \msgref{EVENT_CONFIG} > +with the DEVICE\_NEEDS\_RESET bit set. > + > +\drivernormative{\paragraph}{Reset and Shutdown}{Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Driver} > +\begin{itemize} > + \item A driver MAY request a device reset at any time by writing 0 through > + \msgref{SET_DEVICE_STATUS} and MAY reinitialize the device afterwards if > + it intends to resume operation. > + \item If a device signals DEVICE\_NEEDS\_RESET (for example, via > + \msgref{EVENT_CONFIG}), the driver SHOULD write 0 to > + \msgref{SET_DEVICE_STATUS} and reinitialize the device before resuming > + I/O. > +\end{itemize} > + > +\devicenormative{\paragraph}{Reset and Shutdown}{Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Device} > +\begin{itemize} > + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST > + reset its internal state, discard pending virtqueue operations, and > + present the status field as 0. > + \item If the device encounters an unrecoverable error that requires driver > + intervention, it SHOULD signal DEVICE\_NEEDS\_RESET (such as via > + \msgref{EVENT_CONFIG}) so the driver can initiate a reset. > +\end{itemize} > + > +\subsubsection{Hotplug and Removal} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Hotplug and Removal} > + > +If the bus supports dynamic addition or removal of devices, it can advertise > +those changes with \busref{EVENT_DEVICE} (READY or REMOVED). Some platforms may > +instead rely on external signals such as ACPI, device tree updates, or > +hypervisor events; virtio-msg does not mandate a specific mechanism. > + > +\busnormative{\paragraph}{Hotplug and Removal}{Virtio Transport Options / Virtio Over Messages / Device Operation / Hotplug and Removal / Bus} > +\begin{itemize} > + \item A bus implementation that uses \busref{EVENT_DEVICE} MUST send the READY > + event when a device becomes accessible and the REMOVED event when it is > + no longer available. > + \item Regardless of how hotplug information is delivered, once a new device is > + reported the bus SHOULD query it (e.g., via \msgref{GET_DEVICE_INFO}) > + and register it with the host OS. When a device is removed, the bus > + SHOULD prompt the OS to quiesce and release the associated driver. > +\end{itemize} > + > +\subsection{Transport Messages}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} > + > +Transport messages configure and operate virtio devices once they have been > +discovered. Unlike bus messages (which cover enumeration or hotplug), transport > +messages focus on feature negotiation, configuration access, virtqueue setup, > +and runtime notifications. The subsections below describe each message and how > +it is used in the virtio-msg transport. > + > +\subsubsection{Overview} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / Overview} > + > +Drivers send transport messages to a specific device number, and the virtio-msg > +device replies or emits events accordingly. The bus simply forwards these > +messages after enforcing generic checks such as maximum size or verifying that > +the target device exists. Most transport messages are request/response pairs, > +with events reserved for asynchronous notifications. > + > +\paragraph{Messages IDs and issuers} > + > +\begin{tabular}{|l|l|l|} > +\hline > +Name & ID & Sender \\ > +\hline > +\hline > +Reserved & 0x0 & \\ > +\hline > +Reserved & 0x1 & \\ > +\hline > +\msgref{GET_DEVICE_INFO} & 0x2 & Driver \\ > +\hline > +\msgref{GET_DEVICE_FEATURES} & 0x3 & Driver \\ > +\hline > +\msgref{SET_DRIVER_FEATURES} & 0x4 & Driver \\ > +\hline > +\msgref{GET_CONFIG} & 0x5 & Driver \\ > +\hline > +\msgref{SET_CONFIG} & 0x6 & Driver \\ > +\hline > +\msgref{GET_DEVICE_STATUS} & 0x7 & Driver \\ > +\hline > +\msgref{SET_DEVICE_STATUS} & 0x8 & Driver \\ > +\hline > +\msgref{GET_VQUEUE} & 0x9 & Driver \\ > +\hline > +\msgref{SET_VQUEUE} & 0xA & Driver \\ > +\hline > +\msgref{RESET_VQUEUE} & 0xB & Driver \\ > +\hline > +\msgref{GET_SHM} & 0xC & Driver \\ > +\hline > +\msgref{EVENT_CONFIG} & 0x40 & Device \\ > +\hline > +\msgref{EVENT_AVAIL} & 0x41 & Driver \\ > +\hline > +\msgref{EVENT_USED} & 0x42 & Device \\ > +\hline > +\end{tabular} > + > +IDs are grouped as follows; bit 6 distinguishes requests (0) from events (1): > + > +\begin{tabular}{|l|l|p{7cm}|} > +\hline > +Range & Usage & Notes \\ > +\hline > +0x00-0x3F & Request/Response & Standard transport requests that expect responses. \\ > +0x40-0x7F & Events & Standard transport events (no response). \\ > +0x80-0xBF & Implementation-defined Requests & Optional, transport-specific request extensions. \\ > +0xC0-0xFF & Implementation-defined Events & Optional, transport-specific event extensions. \\ > +\hline > +\end{tabular} > + > +\devicenormative{\paragraph}{Core Messages}{Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Device} > +\begin{itemize} > + \item A device MUST implement \msgref{GET_DEVICE_INFO}, > + \msgref{GET_DEVICE_FEATURES}, \msgref{SET_DRIVER_FEATURES}, > + \msgref{GET_CONFIG}, \msgref{SET_CONFIG}, \msgref{GET_DEVICE_STATUS}, > + \msgref{SET_DEVICE_STATUS}, \msgref{GET_VQUEUE}, \msgref{SET_VQUEUE}, > + and \msgref{RESET_VQUEUE}. > +\end{itemize} > + > +\drivernormative{\paragraph}{Core Messages}{Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Driver} > +\begin{itemize} > + \item A driver MUST support sending and interpreting the messages listed > + above in order to initialize and operate a virtio-msg device. > +\end{itemize} > + > +As described in Sections > +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} > +and > +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications}, > +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and > +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band > +mechanisms. > + > +\msgdef{GET_DEVICE_INFO} > + > +The driver issues \msgref{GET_DEVICE_INFO} to discover static identification > +data and limits for a device. It is the only transport message permitted before > +the device transitions out of the inactive state. > + > +\begin{lstlisting} > +struct virtio_msg_get_device_info_req { > + /* no payload */ > +}; > + > +struct virtio_msg_get_device_info_resp { > + le32 device_id; /* virtio device type */ > + le32 vendor_id; /* implementation-defined vendor ID */ > + le32 num_feature_bits; /* total feature bits (multiples of 32) */ > + le32 config_size; /* bytes in the device configuration space */ > + le32 max_virtqueues; /* maximum virtqueues supported */ > + le16 admin_vq_start; /* index of the first admin virtqueue */ > + le16 admin_vq_count; /* number of admin virtqueues */ Having le32 for max_virtqueues but le16 for admin vqs located at the end seems inconsistent. > +}; > +\end{lstlisting} > + > +\devicenormative{\paragraph}{GET\_DEVICE\_INFO}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Device} > +\begin{itemize} > + \item A device MUST respond to \msgref{GET_DEVICE_INFO} while inactive and > + MUST supply consistent values for device ID, vendor ID, feature count (a > + multiple of 32 bits), configuration size, and virtqueue limits. > +\end{itemize} > + > +\drivernormative{\paragraph}{GET\_DEVICE\_INFO}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Driver} > +\begin{itemize} > + \item A driver SHOULD treat the response as the authoritative source for > + feature count, configuration size, and virtqueue limits before > + proceeding with initialization. > +\end{itemize} > + > +\msgdef{GET_DEVICE_FEATURES} > + > +Drivers retrieve device feature bits in 32-bit blocks via > +\msgref{GET_DEVICE_FEATURES}; the response echoes the requested block index and > +returns one or more 32-bit values with the feature bits in that range. I do not understand the purpose of echoing request values. Looking at virtqueue messages, the driver keeps track of its outstanding requests internally and looks them up upon receiving the response. Doesn't a virtio-msg transport also keep track internally? What advantage has echoing? > + > +\begin{lstlisting} > +struct virtio_msg_get_device_features_req { > + le32 block_index; /* starting block (0 == bits 0-31) */ > + le32 num_blocks; /* number of 32-bit blocks requested */ > +}; > + > +struct virtio_msg_get_device_features_resp { > + le32 block_index; /* echoed starting block */ > + le32 num_blocks; /* echoed number of blocks */ > + le32 features[]; /* num_blocks entries, zero-padded if needed */ > +}; > +\end{lstlisting} > + > +\devicenormative{\paragraph}{GET\_DEVICE\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Device} > +\begin{itemize} > + \item A device MUST return zero for any feature bits beyond those it > + implements when responding to \msgref{GET_DEVICE_FEATURES}. > +\end{itemize} > + > +\drivernormative{\paragraph}{GET\_DEVICE\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Driver} > +\begin{itemize} > + \item A driver MUST treat the returned data as 32-bit feature blocks starting > + at the requested index, padding with zeros as needed, before deciding > + which features to enable. > +\end{itemize} > + > +\msgdef{SET_DRIVER_FEATURES} > + > +Drivers use \msgref{SET_DRIVER_FEATURES} to acknowledge the subset of feature > +bits they wish to enable. The payload mirrors \msgref{GET_DEVICE_FEATURES}, > +supplying an index, count, and 32-bit feature data for the selected blocks. > + > +\begin{lstlisting} > +struct virtio_msg_set_driver_features_req { > + le32 block_index; /* starting block being acknowledged */ > + le32 num_blocks; /* number of blocks provided */ > + le32 features[]; /* num_blocks entries with desired bits */ > +}; > + > +struct virtio_msg_set_driver_features_resp { > + /* no payload */ > +}; > +\end{lstlisting} > + > +\drivernormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Driver} > +\begin{itemize} > + \item A driver MUST set only the feature bits it intends to enable in the > + blocks supplied to \msgref{SET_DRIVER_FEATURES}. > +\end{itemize} > + > +\devicenormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Device} > +\begin{itemize} > + \item After processing \msgref{SET_DRIVER_FEATURES}, a device MUST update its > + acknowledged feature set to match the data supplied and MUST report that > + set consistently in subsequent \msgref{GET_DEVICE_FEATURES} responses. > + If it cannot support the requested set, it SHOULD clear the FEATURES\_OK > + bit in the device status. W.r.t. device status I think this rewording would be clearer: it SHOULD clear -> it MUST NOT set > +\end{itemize} > + > +\msgdef{GET_CONFIG} > + > +\msgref{GET_CONFIG} reads a portion of the configuration space. The request > +specifies the byte offset and length; the response echoes those values, returns > +the data, and includes the current configuration generation count. > + > +\begin{lstlisting} > +struct virtio_msg_get_config_req { > + le32 offset; /* byte offset into configuration space */ > + le32 length; /* number of bytes requested */ > +}; > + > +struct virtio_msg_get_config_resp { > + le32 generation; /* current configuration generation count */ > + le32 offset; /* echoed offset */ > + le32 length; /* echoed length */ > + u8 data[]; /* configuration payload */ > +}; > +\end{lstlisting} > + > +\drivernormative{\paragraph}{GET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Driver} > +\begin{itemize} > + \item A driver MUST ensure the requested offset and length in \msgref{GET_CONFIG} > + stay within the configuration size reported by \msgref{GET_DEVICE_INFO}. > +\end{itemize} > + > +\devicenormative{\paragraph}{GET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Device} > +\begin{itemize} > + \item A device MUST return the current configuration generation count with > + every \msgref{GET_CONFIG} response. > +\end{itemize} > + > +\msgdef{SET_CONFIG} > + > +\msgref{SET_CONFIG} writes a portion of configuration space, supplying the > +offset, length, the driver's view of the generation count, and the new data. > +The device echoes the offset/length, returns the resulting generation count, > +and may mirror the applied data or set the length to zero if the write was > +rejected. > + > +\begin{lstlisting} > +struct virtio_msg_set_config_req { > + le32 generation; /* driver's view of generation count */ > + le32 offset; /* byte offset being written */ > + le32 length; /* number of bytes written */ > + u8 data[]; /* configuration payload */ > +}; > + > +struct virtio_msg_set_config_resp { > + le32 generation; /* resulting generation count */ > + le32 offset; /* echoed offset */ > + le32 length; /* bytes applied (0 if rejected) */ > + u8 data[]; /* optional echoed data */ > +}; > +\end{lstlisting} > + > +\drivernormative{\paragraph}{SET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Driver} > +\begin{itemize} > + \item A driver MUST include its most recent configuration generation count in > + each \msgref{SET_CONFIG} request and MUST keep the offset and length > + within the reported configuration size. > +\end{itemize} > + > +\devicenormative{\paragraph}{SET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Device} > +\begin{itemize} > + \item A device MUST reject a \msgref{SET_CONFIG} request if the supplied > + generation count does not match its current value and MUST report that > + rejection in the response. > +\end{itemize} > + > +\msgdef{GET_DEVICE_STATUS} > + > +\msgref{GET_DEVICE_STATUS} reads the current device status (e.g., ACKNOWLEDGE, > +DRIVER, DRIVER\_OK, FEATURES\_OK, or DEVICE\_NEEDS\_RESET bits). The request has > +no payload; the response returns the 32-bit status value. > + > +\begin{lstlisting} > +struct virtio_msg_get_device_status_req { > + /* no payload */ > +}; > + > +struct virtio_msg_get_device_status_resp { > + le32 status; /* current device status bits */ > +}; > +\end{lstlisting} > + > +\devicenormative{\paragraph}{GET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_STATUS / Device} > +\begin{itemize} > + \item A device MUST report its current status accurately via > + \msgref{GET_DEVICE_STATUS}, including whether FEATURES\_OK or > + DEVICE\_NEEDS\_RESET are set. > +\end{itemize} > + > +\msgdef{SET_DEVICE_STATUS} > + > +\msgref{SET_DEVICE_STATUS} writes a new device status value. Drivers use it to > +progress through the virtio-defined states or to request a reset by writing 0. > +The device responds with its resulting status, which may differ (for example, > +if it refuses FEATURES\_OK or sets DEVICE\_NEEDS\_RESET). > + > +\begin{lstlisting} > +struct virtio_msg_set_device_status_req { > + le32 status; /* desired device status value */ > +}; > + > +struct virtio_msg_set_device_status_resp { > + le32 status; /* resulting device status */ > +}; > +\end{lstlisting} > + > +\drivernormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver} > +\begin{itemize} > + \item A driver MUST write 0 via \msgref{SET_DEVICE_STATUS} to request a device > + reset and MUST re-read the status (e.g., via \msgref{GET_DEVICE_STATUS}) > + if it needs to confirm acceptance. > +\end{itemize} > + > +\devicenormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Device} > +\begin{itemize} > + \item A device MAY clear FEATURES\_OK or set DEVICE\_NEEDS\_RESET in its > + response if it cannot accept the requested status, but it MUST report > + the resulting status accurately. > +\end{itemize} > + > +\msgdef{GET_VQUEUE} > + > +\msgref{GET_VQUEUE} returns information about a specific virtqueue, including > +its maximum size, current size, and, if already configured, the descriptor, > +driver, and device area addresses. Maybe GET_VQUEUE and the SET_VQUEUE response could also return whether the vqueue is enabled? > + > +\begin{lstlisting} > +struct virtio_msg_get_vqueue_req { > + le32 index; /* virtqueue index */ > +}; > + > +struct virtio_msg_get_vqueue_resp { > + le32 index; /* echoed virtqueue index */ > + le32 max_size; /* maximum queue size */ > + le32 cur_size; /* current size (0 if unconfigured) */ > + le32 reserved; /* must be zero */ > + le64 desc_addr; /* descriptor area address */ > + le64 driver_addr; /* driver area address */ > + le64 device_addr; /* device area address */ > +}; > +\end{lstlisting} > + > +\devicenormative{\paragraph}{GET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} > +\begin{itemize} > + \item A device MUST report accurate maxima and current queue sizes for each > + virtqueue and MUST return zero as the current size if the queue has not > + yet been configured. > +\end{itemize} > + > +\msgdef{SET_VQUEUE} > + > +\msgref{SET_VQUEUE} programs a virtqueue's size and buffer addresses. The driver > +selects a queue index, supplies the desired size (not exceeding the maximum > +reported via \msgref{GET_VQUEUE}), and provides guest-physical addresses for the > +descriptor, driver, and device areas. > + > +\begin{lstlisting} > +struct virtio_msg_set_vqueue_req { > + le32 index; /* virtqueue index */ > + le32 reserved0; /* must be zero */ > + le32 size; /* number of descriptors */ > + le32 reserved1; /* must be zero */ > + le64 desc_addr; /* descriptor area address */ > + le64 driver_addr; /* driver area address */ > + le64 device_addr; /* device area address */ > +}; > + > +struct virtio_msg_set_vqueue_resp { > + /* no payload */ > +}; > +\end{lstlisting} > + > +\drivernormative{\paragraph}{SET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Driver} > +\begin{itemize} > + \item A driver MUST set the virtqueue size field to a value not exceeding the > + maximum reported by \msgref{GET_VQUEUE} and MUST ensure the supplied > + addresses point to properly aligned memory. > +\end{itemize} > + > +\devicenormative{\paragraph}{SET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Device} > +\begin{itemize} > + \item A device MUST configure the queue using the parameters provided via > + \msgref{SET_VQUEUE} and treat the queue as inactive until > + \msgref{SET_VQUEUE} has successfully completed. > +\end{itemize} > + > +\msgdef{RESET_VQUEUE} > + > +\msgref{RESET_VQUEUE} resets an individual virtqueue (if VIRTIO\_F\_RING\_RESET > +has been negotiated). It requires only the queue index; the device responds > +after the queue has been quiesced. > + > +\begin{lstlisting} > +struct virtio_msg_reset_vqueue_req { > + le32 index; /* virtqueue index */ > +}; > + > +struct virtio_msg_reset_vqueue_resp { > + /* no payload */ > +}; > +\end{lstlisting} > + > +\drivernormative{\paragraph}{RESET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Driver} > +\begin{itemize} > + \item A driver SHOULD issue \msgref{RESET_VQUEUE} only if > + VIRTIO\_F\_RING\_RESET has been negotiated and it needs to reconfigure > + or recover a specific queue. > +\end{itemize} > + > +\devicenormative{\paragraph}{RESET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Device} > +\begin{itemize} > + \item Upon receiving \msgref{RESET_VQUEUE}, a device MUST stop processing the > + indicated queue, reset its internal state for that queue, and allow the > + driver to reconfigure it via \msgref{SET_VQUEUE}. > +\end{itemize} > + > +\msgdef{GET_SHM} > + > +\msgref{GET_SHM} returns the location and size of a device-owned shared memory > +region. The request carries the region index; the response echoes the index and > +provides the base address and length. A length of zero indicates that the > +specified region does not exist. > + > +\begin{lstlisting} > +struct virtio_msg_get_shm_req { > + le32 index; /* shared memory region index */ > +}; > + > +struct virtio_msg_get_shm_resp { > + le32 index; /* echoed index */ > + le32 length; /* region length (0 if nonexistent) */ > + le32 address;/* region address */ length and address should be le64. > +}; > +\end{lstlisting} > + > +\devicenormative{\paragraph}{GET\_SHM}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_SHM / Device} > +\begin{itemize} > + \item A device MUST return zero length if the requested shared memory region > + does not exist and MUST report accurate address/length information for > + regions it supports. > +\end{itemize} > + > +\msgdef{EVENT_CONFIG} > + > +\msgref{EVENT_CONFIG} notifies the driver about configuration or status changes. > +The payload includes the current device status, configuration generation count, > +and optionally the updated configuration data (offset plus length). A length of > +zero indicates that the driver should re-fetch the affected range via > +\msgref{GET_CONFIG}. IIUC the affected range should still be communicated through virtio_msg_event_config.length. Maybe the above should be rephrased to say that .data can be empty. > + > +\begin{lstlisting} > +struct virtio_msg_event_config { > + le32 device_status; /* new device status value */ > + le32 generation; /* configuration generation count */ > + le32 offset; /* configuration offset */ > + le32 length; /* number of bytes (may be zero) */ > + u8 data[]; /* optional configuration data */ > +}; > +\end{lstlisting} > + > +\drivernormative{\paragraph}{EVENT\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Driver} > +\begin{itemize} > + \item When \msgref{EVENT_CONFIG} provides configuration data, the driver > + SHOULD apply it; otherwise it SHOULD re-read the affected range via > + \msgref{GET_CONFIG} before proceeding. > +\end{itemize} > + > +\devicenormative{\paragraph}{EVENT\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Device} > +\begin{itemize} > + \item A device MUST send \msgref{EVENT_CONFIG} whenever it changes > + configuration data or status in a way that is visible to the driver, > + including the new configuration generation count. Isn't it enough to specify to send EVENT_CONFIG for every configuration change notification? ("Device Requirements: Device Status Field" defines that a device sends a configuration change notification when setting DEVICE_NEEDS_RESET.) > +\end{itemize} > + > +\msgdef{EVENT_AVAIL} > + > +\msgref{EVENT_AVAIL} notifies the device that a virtqueue has new buffers. The > +message includes the queue index and, if VIRTIO\_F\_NOTIFICATION\_DATA is > +negotiated, a combined next-offset/next-wrap field so the device can jump > +directly to the updated position. Otherwise that field is zero. > + > +\begin{lstlisting} > +struct virtio_msg_event_avail { > + le32 vq_index; /* virtqueue index */ > + le32 next_offset; /* bits[30:0] offset, bit[31] wrap; zero if unused */ > +}; > +\end{lstlisting} > + > +\drivernormative{\paragraph}{EVENT\_AVAIL}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Driver} > +\begin{itemize} > + \item If VIRTIO\_F\_NOTIFICATION\_DATA has not been negotiated, a driver MUST > + set the next offset/wrap field to zero in \msgref{EVENT_AVAIL} messages. > + If it has been negotiated, the driver MUST fill in the field with the > + current available ring position. > +\end{itemize} > + > +\busnormative{\paragraph}{EVENT\_AVAIL}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Bus} > +\begin{itemize} > + \item A bus implementation that uses out-of-band notifications SHOULD prevent > + negotiation of VIRTIO\_F\_NOTIFICATION\_DATA so that devices do not > + expect in-band offset/wrap values. > +\end{itemize} > + > +\msgdef{EVENT_USED} > + > +\msgref{EVENT_USED} notifies the driver that buffers on a particular virtqueue > +have been consumed. The payload carries only the queue index; the driver uses > +its used ring to determine which descriptors completed. > + > +\begin{lstlisting} > +struct virtio_msg_event_used { > + le32 vq_index; /* virtqueue index */ > +}; > +\end{lstlisting} > + > +\devicenormative{\paragraph}{EVENT\_USED}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Device} > +\begin{itemize} > + \item A device SHOULD send \msgref{EVENT_USED} (or an equivalent notification) > + when it processes buffers on a virtqueue unless the driver has > + explicitly disabled such notifications. > +\end{itemize} > + > +\drivernormative{\paragraph}{EVENT\_USED}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Driver} > +\begin{itemize} > + \item Upon receiving \msgref{EVENT_USED}, a driver MUST examine the indicated > + virtqueue (for example, by reading the used ring) to reclaim completed > + buffers. > +\end{itemize} > + > +\subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages} > + > +Bus messages cover device discovery, hotplug notifications, and bus-level > +liveness checks. Implementations that already provide equivalent functionality > +through platform mechanisms (e.g., ACPI, device tree, hypervisor events) may > +reuse that infrastructure instead of the messages defined here. The following > +subsections describe standardized bus messages that virtio-msg supports today. > + > +\subsubsection{Overview} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / Overview} > + > +Each bus implementation chooses which of these messages to support. Some may > +implement all of them for a fully message-based workflow, while others may rely > +on out-of-band enumeration and use only the subset that fits their needs. > + > +\paragraph{Messages IDs and issuers} > + > +\begin{tabular}{|l|l|l|} > +\hline > +Name & ID & Sender \\ > +\hline > +\hline > +Reserved & 0x0 & \\ > +\hline > +Reserved & 0x1 & \\ > +\hline > +\busref{GET_DEVICES} & 0x2 & Driver \\ > +\hline > +\busref{PING} & 0x3 & Any \\ > +\hline > +\busref{EVENT_DEVICE} & 0x40 & Device \\ > +\hline > +\end{tabular} > + > +Bus message IDs are grouped as follows: > + > +\begin{tabular}{|l|l|p{7cm}|} > +\hline > +Range & Usage & Notes \\ > +\hline > +0x00-0x3F & Request/Response & Standardized bus request messages. \\ > +0x40-0x7F & Events & Standardized bus event messages. \\ > +0x80-0xBF & Implementation-defined Requests & Optional, bus-specific request messages. \\ > +0xC0-0xFF & Implementation-defined Events & Optional, bus-specific event messages. \\ > +\hline > +\end{tabular} > + > +\paragraph{Bus Specific Messages} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Operation / Bus Specific Messages} > + > +A range of message IDs are reserved for use by the specific bus > +implementation. These messages can be used for any implementation specific > +usage. Example usage could include: > + > +\begin{itemize} > + \item Configuration of out-of-band notification methods > + \item Setup shared memory regions to be used for buffers or virtqueues > + \item Declare bus specific error conditions > + \item Provide extra debug or logging information > +\end{itemize} > + > +\busdef{GET_DEVICES} > + > +The driver-side bus uses this request to enumerate device numbers on the > +device-side bus. The response describes a “window” of device numbers and signals > +which entries are present. > + > +\begin{lstlisting} > +struct virtio_bus_msg_get_devices_req { > + le16 offset; /* starting device number */ > + le16 count; /* number of device-number slots (multiple of 8) */ > +}; > + > +struct virtio_bus_msg_get_devices_resp { > + le16 offset; /* echoed starting device number */ > + le16 count; /* window length actually returned */ > + le16 next_offset; /* 0 or suggested start for next query */ > + u8 bitmap[]; /* count/8 bytes, LSB-first packing */ > +}; > +\end{lstlisting} > + > +The \textbf{(offset, count)} tuple defines a window of \textbf{count} > +consecutive device numbers beginning at \textbf{offset}. The number of present > +devices equals the number of 1-bits in the bitmap. Responders SHOULD return the > +requested \textbf{count} unless constrained (e.g., by maximum message size); > +otherwise they MAY return a smaller value, in which case the bitmap covers the > +reduced window. > + > +Example: a request with \textbf{offset}=0, \textbf{count}=16 might produce > +\textbf{bitmap}=0b00100101 00000000 and \textbf{next_offset}=16. That indicates > +devices 0, 2, and 5 are present within the 0–15 window and suggests continuing > +at device number 16. > + > +\paragraph{Intended usage} > +Drivers MAY use this message to enumerate device numbers. Treat each window as > +a snapshot, advance using \textbf{next_offset}, and confirm candidates via > +\msgref{GET_DEVICE_INFO} before issuing other transport messages. > + > +\busnormative{\paragraph}{GET\_DEVICES}{Virtio Transport Options / Virtio Over Messages / Bus Messages / GET_DEVICES / Bus} > +\begin{itemize} > + \item The bus-side responder MUST enforce that \textbf{offset} and > + \textbf{count} are multiples of 8, and it MUST return a bitmap of length > + \textbf{count/8} packed least-significant-bit first. > + \item \textbf{next\_offset} MUST be 0 (indicating no further windows) or an > + aligned value $\ge \textbf{offset} + \textbf{count}$ recommending where > + the driver should query next. > + \item Responders SHOULD return the requested \textbf{count} unless constrained > + (e.g., by maximum message size); if a smaller count is returned, the > + bitmap MUST still represent the window defined by the echoed > + \textbf{offset} and \textbf{count}. > +\end{itemize} > + > +\busdef{PING} > + > +\busref{PING} is a simple liveness check that can be sent by either side of the > +bus; the response echoes the 32-bit data value from the request. > + > +\begin{lstlisting} > +struct virtio_bus_msg_ping_req { > + le32 data; /* arbitrary value chosen by the sender */ > +}; > + > +struct virtio_bus_msg_ping_resp { > + le32 data; /* echoed request data */ > +}; > +\end{lstlisting} > + > +\paragraph{Intended usage} > +Drivers or bus implementations MAY use \busref{PING} as a keepalive. If a reply > +is missing or delayed beyond policy, the initiator SHOULD verify device status > +(e.g., via \msgref{GET_DEVICE_STATUS}) and take recovery actions as needed. > + > +\busnormative{\paragraph}{PING}{Virtio Transport Options / Virtio Over Messages / Bus Messages / PING / Bus} > +\begin{itemize} > + \item The bus-side responder MUST echo the 32-bit data field from the request > + exactly in the \busref{PING} response. > +\end{itemize} > + > +\busdef{EVENT_DEVICE} > + > +\busref{EVENT_DEVICE} signals device-level hotplug changes. The driver-side bus > +receives these indications from the device-side bus; the table below lists the > +defined device bus states. > + > +\begin{lstlisting} > +struct virtio_bus_msg_event_device { > + le16 device_number; > + le16 device_bus_state; /* see table below */ > +}; > +\end{lstlisting} > + > +\begin{tabular}{|l|l|p{7cm}|} > +\hline > +Constant & Value & Meaning \\ > +\hline > +DEVICE\_BUS\_STATE\_READY & 0x0001 & Device is present and ready. \\ > +DEVICE\_BUS\_STATE\_REMOVED & 0x0002 & Device is no longer present. \\ > +Reserved & 0x0003–0x7FFF & Reserved for standard use. \\ > +Implementation-defined & 0x8000–0xFFFF & MAY be used by the bus implementation. \\ > +\hline > +\end{tabular} > + > +\paragraph{Intended usage} > +Upon receiving DEVICE\_BUS\_STATE\_READY, drivers SHOULD probe the device via > +\msgref{GET_DEVICE_INFO} and proceed with initialization if successful. Upon > +DEVICE\_BUS\_STATE\_REMOVED, drivers MUST stop queueing new work, quiesce and > +reset as applicable, and release resources. Drivers SHOULD tolerate duplicate or What does "reset" mean here? AFAIU the device is no longer present, so cannot be reset any more? Best regards, Peter > +out-of-order events and MAY rely on additional bus-level monitoring (e.g., > +\busref{PING}) if needed. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-06 16:28 ` Peter Hilber @ 2026-02-10 9:39 ` Bertrand Marquis 2026-02-12 11:16 ` Peter Hilber 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-10 9:39 UTC (permalink / raw) To: Peter Hilber Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Peter, > On 6 Feb 2026, at 17:28, Peter Hilber <peter.hilber@oss.qualcomm.com> wrote: > > On Mon, Jan 26, 2026 at 11:32:28AM -0500, Bill Mills wrote: >> Add a new transport layer that is based on messages. >> >> This transport layer still uses virtqueues as the other transport layers do >> but implements transport layer operations by sending and receiving messages >> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. >> >> This transport is useful when the device and driver are both implemented in >> software but the trap and emulate operations of virtio-mmio and virtio-pci >> can not be used. >> >> This transport is intended to be used in many situations, including: >> * between a host processor and its co-processors >> * between two different systems (not SMP) connected via PCIe >> * between normal and secure worlds >> * host to vm >> * vm to vm > > Hi, > > please find some remarks below. As I wrote before, I am not a transport > expert, so please take the following comments with a grain of salt. Thanks a lot for the review, please find my answers inline after. > >> >> Signed-off-by: Bill Mills <bill.mills@linaro.org> >> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> >> Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> >> Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com> >> --- >> transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 1640 insertions(+) >> create mode 100644 transport-msg.tex >> >> diff --git a/transport-msg.tex b/transport-msg.tex >> new file mode 100644 >> index 0000000..d4e31d7 >> --- /dev/null >> +++ b/transport-msg.tex >> @@ -0,0 +1,1640 @@ >> +\section{Virtio Over Messages}\label{sec:Virtio Transport Options / Virtio Over Messages} >> + >> +\newcommand{\conceptref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / #1]{#1}} >> +\newcommand{\msgref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1]{VIRTIO_MSG_#1}} >> +\newcommand{\busref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1]{BUS_MSG_#1}} >> +\newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1}} >> +\newcommand{\busdef}[1]{\subsubsection{BUS_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1}} >> + >> +This section defines \textbf{virtio-msg}, a transport mechanism that encapsulates >> +virtio operations as discrete message exchanges rather than relying on PCI or >> +memory-mapped I/O regions. It separates bus-level functionality (e.g., device >> +enumeration, hotplug events) from device-specific operations (e.g., feature >> +negotiation, virtqueue setup), ensuring that a single, generic transport layer >> +can be reused across multiple bus implementations. >> + >> +virtio-msg addresses several key objectives: >> + >> +\begin{itemize} >> + \item \textbf{Support multiple bus implementations:} >> + Systems can rely on various communication methods such as hypercalls, local >> + IPC, network channels, or device trees for enumerating devices. virtio-msg >> + defines a common transport interface suitable for any of these mechanisms. >> + >> + \item \textbf{Reduce per-bus complexity:} >> + Buses can implement a fully message-based workflow (including optional >> + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE}) >> + or they can discover and manage devices through >> + alternative means such as platform firmware data. In either case, they >> + forward transport messages to and from each device. >> + >> + \item \textbf{Preserve virtio semantics:} >> + The transport leverages standard virtio concepts (features, configuration >> + space, virtqueues), so existing virtio drivers and device implementations can >> + integrate smoothly once a device is discovered and registered. >> +\end{itemize} >> + >> +\subsection{Basic Concepts} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts} >> + >> +The virtio-msg transport relies on a set of foundational concepts to ensure >> +reusability across different bus implementations and flexibility in device >> +capabilities. This section defines those concepts and clarifies how they apply, >> +regardless of whether the bus leverages message-based enumeration or platform >> +data for device discovery. >> + >> +\subsubsection{High-Level Architecture} >> + >> +virtio-msg operates around two layers: >> + >> +\begin{enumerate} >> + \item \textbf{Bus Layer}: A \emph{bus instance} exposes zero or more virtio >> + devices. It discovers available devices through mechanisms such as: >> + \begin{itemize} >> + \item Optional message-based queries (\busref{GET_DEVICES}), >> + \item External data sources (e.g., device tree, ACPI tables, hypervisor >> + firmware calls), >> + \item Dynamic hotplug notifications (optionally via \busref{EVENT_DEVICE}). >> + \end{itemize} >> + Once a device is identified, regardless of discovery method, the bus >> + uses \msgref{GET_DEVICE_INFO} to read its device ID and vendor ID, then >> + \emph{registers} that device with the host OS so the usual virtio driver >> + probe can occur. >> + >> + \item \textbf{Transport Layer}: After the bus knows about a device, the >> + virtio-msg transport handles all device-specific operations: >> + \begin{itemize} >> + \item Retrieving and setting features (\msgref{GET_DEVICE_FEATURES}, >> + \msgref{SET_DRIVER_FEATURES}), >> + \item Accessing the device configuration space (\msgref{GET_CONFIG}, >> + \msgref{SET_CONFIG}), >> + \item Setting up virtqueues (\msgref{SET_VQUEUE}, \msgref{RESET_VQUEUE}), >> + \item Managing status and runtime notifications (\msgref{SET_DEVICE_STATUS}, >> + \msgref{EVENT_USED}, etc.). >> + \end{itemize} >> + These transport messages remain the same across different bus instances, >> + allowing a single virtio-msg driver component to function in multiple >> + environments. >> +\end{enumerate} >> + >> +\subsubsection{Relationship Between Bus and Transport} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport} >> + >> +This subsubsection explains the division of responsibilities: the bus layer is >> +the mandatory carrier that moves messages between driver and device endpoints >> +(e.g., over IPC, shared memory with signalling or hardware messaging), while >> +the virtio-msg transport defines the semantics of those messages. >> + >> +virtio-msg groups messages into two categories: >> + >> +\begin{description} >> + \item[\textbf{Bus Messages}:] >> + Intended for global bus operations such as enumerating device numbers >> + (\busref{GET_DEVICES}), managing device hotplug events (\busref{EVENT_DEVICE}) >> + or assessing bus-wide health (\busref{PING}). >> + These messages are \emph{optional} in environments where >> + device discovery or state changes occur through other means (e.g., device >> + tree). However, if a bus chooses to handle those tasks via messages, >> + it implements the appropriate bus message definitions described in this >> + section. >> + >> + \item[\textbf{Transport Messages}:] >> + Used for device-specific operations, such as: >> + \begin{itemize} >> + \item Retrieving or setting features (\msgref{GET_DEVICE_FEATURES}, >> + \msgref{SET_DRIVER_FEATURES}), >> + \item Accessing device configuration (\msgref{GET_CONFIG}, >> + \msgref{SET_CONFIG}), >> + \item Managing virtqueues (\msgref{SET_VQUEUE}, \msgref{RESET_VQUEUE}), >> + \item Handling device status and notifications (\msgref{SET_DEVICE_STATUS}, >> + \msgref{EVENT_USED}, etc.). >> + \end{itemize} >> +\end{description} >> + >> +This separation lets a bus remain minimal if it obtains device information from >> +firmware tables, while still supporting fully message-based enumeration and >> +hotplug when desired. >> + +\busnormative{\paragraph}{Bus Message Implementation}{Virtio >> Transport Options / Virtio Over Messages / Basic Concepts / >> Relationship between bus and transport / Bus Messages} >> +\begin{itemize} >> + \item A bus implementation that provides enumeration, hotplug, or bus >> + health handling via bus messages SHOULD implement the corresponding Bus >> + Message definitions described in this section. >> +\end{itemize} >> + >> +\busnormative{\paragraph}{Transport Message Forwarding}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport / Transport Message Forwarding} >> +\begin{itemize} >> + \item A bus implementation MUST relay each transport message to the device >> + number identified in the message header, regardless of how it >> + discovered or enumerated that device. >> + \item A bus implementation SHOULD treat transport messages as opaque apart >> + from enforcing generic transport limits, such as the advertised maximum >> + message size, and SHOULD NOT modify the transport payload. >> +\end{itemize} >> + >> +\subsubsection{System Topology} >> + >> +A virtio-msg system contains the following elements: >> + >> +\begin{itemize} >> + \item \textbf{Bus Instances and Devices}: Each bus instance advertises its >> + capabilities (e.g., transport revision, maximum message size) and >> + discovers devices via message-based queries or external data sources. >> + Every discovered device has a unique \emph{device number}. > > "Device number" is also used as a synonym for "device ID" in a few > places in the existing spec. Maybe these places could be changed to > "device ID" for clarity? Agree, we will use coherently Device ID in v2. > >> + \item \textbf{Driver}: Communicates with the bus to learn about >> + available devices. Once a device is recognized, the driver uses the >> + common transport messages to perform feature negotiation, configuration, >> + and virtqueue setup. >> + \item \textbf{Device}: Implement virtio device functionality and >> + respond to the transport messages. The bus forwards these messages to > > Minor editorial: > > Implement -> Implements > > respond -> responds Ack > >> + the correct device instance based on device number. >> +\end{itemize} >> + >> +\subsubsection{Transport Revisions and Maximum Message Size} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions} >> + >> +Each \textbf{virtio-msg bus instance} advertises the following to the transport >> +layer: >> +\begin{itemize} >> + \item \textbf{Transport revision}: the protocol version supported by the bus >> + instance (independent of the overall Virtio specification version). >> + \item \textbf{Maximum message size}: the largest payload (in bytes) that can >> + be carried per request or response, including the common header. >> + \item \textbf{Transport feature bits}: revision-specific optional features >> + implemented by the bus instance. >> +\end{itemize} >> + >> +The mechanism for obtaining these parameters is implementation-defined and can >> +vary between bus instances. Common approaches include: >> +\begin{itemize} >> + \item Reading firmware or device-tree data that describes each bus instance, >> + \item Performing a message exchange during bus setup to retrieve the values, >> + \item Relying on per-bus configuration structures or driver-specific defaults. >> +\end{itemize} >> + >> +\busnormative{\paragraph}{Advertising Transport Parameters}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Advertising Transport Parameters} >> +\begin{itemize} >> + \item Each bus instance MUST make its transport revision, maximum message >> + size, and transport feature bits available to the virtio-msg transport >> + before any transport messages are exchanged for that device. >> + \item A bus instance MUST apply the same limits to both driver-originated and >> + device-originated transport messages; if the values change, the bus >> + MUST inform the transport layer before accepting additional messages. >> +\end{itemize} >> + >> +\drivernormative{\paragraph}{Respecting Bus Limits}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Driver Limits} >> +\begin{itemize} >> + \item A driver MUST NOT send a transport message whose total size exceeds the >> + maximum message size advertised for the target bus instance. >> + \item A driver MUST NOT rely on transport features or messages that require a >> + higher transport revision than the bus instance reports. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Respecting Bus Limits}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Device Limits} >> +\begin{itemize} >> + \item A device MUST ensure its responses and device-originated messages do >> + not exceed the maximum message size advertised by the bus instance that >> + relays them. >> + \item A device MUST NOT require transport features or messages beyond the >> + transport revision reported by the bus instance, and MUST respond with >> + an error or ignore requests for unsupported features. >> +\end{itemize} >> + >> +\paragraph{virtio-msg revisions} > > IIUC, this is called "transport revision" elsewhere. I suggest to unify > the naming. Agree, we will unify in v2. > >> + >> +The following table lists the currently defined virtio-msg revisions: >> + >> +\begin{tabular}{ |l|l|l|l| } >> +\hline >> +\field{revision} & \field{maximum size} & \field{features} & remarks \\ >> +\hline \hline >> +1 & 44-65536 & <empty> & Virtio Message Revision 1 \\ >> +\hline >> +\end{tabular} >> + >> +Note that a change in the virtio standard does not necessarily >> +correspond to a change in the virtio-msg revision. >> + >> +The maximum message size is specified from the transport-layer point of view >> +and includes the 8-byte common header plus payload. Any extra encapsulation >> +imposed by the underlying bus (for example, a framing header) does not count >> +against this limit. Today the largest practical transport payload is 256 bytes: >> +messages such as GET_CONFIG and SET_CONFIG can carry up to 256 bytes of device >> +configuration in one transfer, and larger configuration regions can be accessed >> +through multiple exchanges without requiring a larger per-message limit. >> + >> +\busnormative{\paragraph}{Message Size Bounds}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Message Size} >> +\begin{itemize} >> + \item A bus implementation MUST advertise a maximum message size of at least >> + 44 bytes. >> + \item A bus implementation SHOULD NOT advertise a maximum message size that >> + exceeds 264 bytes (256-byte payload plus the common header). >> +\end{itemize} >> + >> +\paragraph{Versioning and Forward Compatibility} >> + >> +A higher transport revision or additional transport feature bits extend the >> +protocol with new messages or capabilities. Implementations are expected to >> +remain interoperable across revisions: devices and drivers designed for a newer >> +revision still implement the mandatory messages and semantics defined in prior >> +revisions. >> + >> +\drivernormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Driver} >> +\begin{itemize} >> + \item A driver that negotiates transport revision $N$ MUST implement all >> + mandatory driver behavior defined for revisions $1$ through $N$. >> + \item If a driver receives a device-originated message or feature indication >> + that requires an unsupported revision or transport feature, it MUST >> + ignore the message (or treat the request as failed) and MUST NOT act on >> + partially understood data. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Device} >> +\begin{itemize} >> + \item A device that advertises transport revision $N$ MUST implement all >> + mandatory device behavior defined for revisions $1$ through $N$. >> + \item If a device receives a driver request that relies on an unsupported >> + revision or transport feature, it MUST reject the request using the >> + message-specific error mechanism (if any) or silently ignore it. >> +\end{itemize} >> + >> +\busnormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Bus} >> +\begin{itemize} >> + \item A bus instance that advertises transport revision $N$ MUST satisfy every >> + bus requirement defined for revisions $1$ through $N$. > > Are transport feature bits supposed to be negotiated? I think this could > make sense so that behavior can change without affecting > interoperability between implementations of different revisions of the > Virtio standard. Yes transport feature bits are supposed to be negotiated. It is expected that the bus will determine what is the common set of features supported and will only declare those to the transport. A simple bus implementation could just get from the device side the mask of bits supported and mask it with the list of features bits supported on the driver side and give back the common denominator to the transport so that only features supported by both sides are advertised and used. We will make that clearer in v2 if you agree with that principle. > >> + \item If a bus instance cannot forward a message because it requires an >> + unsupported revision or transport feature, it MUST surface a transport >> + error or drop the message without forwarding it. >> +\end{itemize} >> + >> +\subsubsection{Device Numbers and Enumeration} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / >> +Device Numbers} >> + >> +Each virtio-msg bus instance contains one or more \emph{devices}, identified >> +by a 16-bit \textbf{device number}. Buses discover these device numbers through >> +mechanisms such as: >> +\begin{itemize} >> + \item \textbf{Message-Based Enumeration}: Using \busref{GET_DEVICES} to query >> + which numbers exist (optional). >> + \item \textbf{Platform Data}: A device tree, ACPI tables, or hypervisor calls >> + might inform the bus of available device numbers and their properties. >> +\end{itemize} >> + >> +Once a bus confirms that a device number is valid—regardless of the discovery >> +method—it normally issues \msgref{GET_DEVICE_INFO} to retrieve the device and >> +vendor IDs before registering the device with the host OS so the usual Virtio >> +driver binding process can begin. >> + >> +\busnormative{\paragraph}{Device Number Assignment}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Device Numbers / Assignment} >> +\begin{itemize} >> + \item A bus implementation MUST assign a unique device number to every >> + device on a given bus instance and MUST NOT forward transport messages >> + for a device number that has not been validated. >> + \item A bus implementation SHOULD provide the driver with sufficient >> + information—either via \busref{GET_DEVICES} or equivalent platform >> + data—to discover each valid device number. >> +\end{itemize} >> + >> +\subsubsection{Configuration Generation Count} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count} >> + >> +Each device maintains a \textbf{Configuration Generation Count} to prevent >> +inconsistent updates. This count is incremented at least once by the device for >> +every driver-visible change it makes to its configuration data. The current >> +count is exposed in \msgref{EVENT_CONFIG} and \msgref{GET_CONFIG} responses so >> +the driver can determine whether its view of the configuration is current. The >> +count does not necessarily start at zero and is not automatically reset when >> +the device resets. > > Could the spec clarify whether the device MUST NOT, or MAY reset it? Yes this is a MAY. What we want to pass through here is the fact that the driver side should not make any assumption on the generation count value or evolution on changes. We will make this clearer on v2. > >> + >> +\devicenormative{\paragraph}{Configuration Generation Count}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Device} >> +\begin{itemize} >> + \item A device MUST increment the generation count before it makes a change >> + that is visible to the driver and MUST ensure that each >> + \msgref{EVENT_CONFIG} carrying configuration data uses a unique >> + generation count. >> + \item If updated configuration data cannot fit in a single >> + \msgref{EVENT_CONFIG}, the device SHOULD send an \msgref{EVENT_CONFIG} >> + with zero data length to advertise the new generation count and MUST >> + make the updated data available via \msgref{GET_CONFIG}. >> + \item If a \msgref{SET_CONFIG} request includes a generation count that does >> + not match the device's current count, the device MUST reject the >> + request. >> +\end{itemize} >> + >> +\drivernormative{\paragraph}{Configuration Generation Count}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Driver} >> +\begin{itemize} >> + \item A driver MUST track the most recent generation count observed (via >> + \msgref{EVENT_CONFIG} or \msgref{GET_CONFIG}) and include it in every >> + \msgref{SET_CONFIG} request. >> + \item If a \msgref{SET_CONFIG} request is rejected due to a mismatched >> + generation count, the driver SHOULD issue \msgref{GET_CONFIG} to obtain >> + the latest configuration data and generation count before retrying. > > In my understanding this only helps if the driver for the specific > device type reacts to this rejection. Currently, other standardized > transports - and consequently Linux kernel drivers - assume that > configuration writes will not be rejected like this. Yes this is true and I would expect this part to be handled at the transport level in transparent way for drivers. The main idea here was to have a way for the transport (if it wants) to cache the configuration data so that i could prevent exchanges when not required > > It might be preferable to make the generation count in SET_CONFIG > optional, or to drop it. That is a good idea, we will investigate dropping or making this optional in v2. > > IIUC the Linux kernel driver mentioned in the cover letter currently > does not retry and is therefore susceptible to races, so that some > configuration writes might not take effect. Agree. > >> +\end{itemize} >> + >> +This mechanism ensures updates are not lost or overwritten due to stale >> +information. >> + >> +\subsubsection{Feature Negotiation Blocks} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Feature Blocks} >> + >> +The virtio-msg transport organizes feature bits into 32-bit blocks accessed by >> +\msgref{GET_DEVICE_FEATURES} and \msgref{SET_DRIVER_FEATURES}. Each block >> +represents up to 32 feature bits: >> + >> +\begin{itemize} >> + \item \textbf{Block Index}: The starting block (e.g., block 0 for >> + features 0--31, block 1 for features 32--63, etc.). >> + \item \textbf{Number of Blocks}: How many blocks the driver wishes to retrieve >> + or modify in a single message. >> + \item \textbf{Feature Data}: The 32-bit values representing the supported (or >> + driver-requested) feature bits in the selected blocks. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Feature Blocks}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Feature Blocks / Device} >> +\begin{itemize} >> + \item When \msgref{GET_DEVICE_FEATURES} covers blocks that extend beyond the >> + features a device implements, the device MUST return zero for the >> + feature data in those positions. >> +\end{itemize} >> + >> +\subsubsection{Error Signaling} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling} >> + >> +Transport errors can arise from malformed messages, routing failures inside a >> +bus implementation, or device-side faults while processing a valid request. >> +Implementations should handle such faults locally where possible, but a bus may >> +surface an error to the virtio-msg transport if it cannot deliver a request or >> +obtain a response within its policy. >> + >> +\busnormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus} >> +\begin{itemize} >> + \item A bus implementation MAY report a transport-visible failure (for >> + example, after exhausting a bounded retry policy) when it cannot deliver >> + a request or obtain a response. >> + \item A bus implementation MUST treat malformed headers or unsupported >> + \field{msg_id} values as invalid, MUST discard them without generating >> + additional protocol traffic, and MAY log the condition for diagnostics. >> + \item A bus implementation MUST NOT generate error responses to event >> + (one-way) messages. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device} >> +\begin{itemize} >> + \item A device receiving a malformed or unsupported transport message MUST >> + discard it without producing further protocol traffic. >> + \item Recovery actions taken in response to an error (such as retries, >> + selective resets, or device removal) MUST follow the normative reset and >> + status semantics defined in >> + \ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}. >> +\end{itemize} >> + >> +This specification does not define a dedicated error-reporting message; it only >> +permits implementations to surface failures when silent recovery is not >> +feasible. >> + >> +\subsubsection{Endianness} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness} >> + >> +Unless otherwise stated, all numeric fields defined for virtio-msg messages use >> +little-endian encoding. >> + >> +\drivernormative{\paragraph}{Endianness}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Driver} >> +\begin{itemize} >> + \item A driver MUST encode and decode the common header and payload fields >> + defined by this transport using little-endian byte order. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Endianness}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Device} >> +\begin{itemize} >> + \item A device MUST emit and consume the common header and payload fields >> + defined by this transport using little-endian byte order. >> +\end{itemize} >> + >> +\subsubsection{Common Message Format} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format} >> + >> +All virtio-msg exchanges, whether \emph{bus messages} or \emph{transport >> +messages}, begin with an 8-byte header followed by an optional payload. The >> +fields below describe the wire format for that header. >> + >> +The header layout is: >> +\begin{lstlisting} >> +struct virtio_msg_header { >> + u8 type; /* request/response + bus/transport */ >> + u8 msg_id; /* message id */ >> + le16 dev_num; /* device number (0 for bus messages) */ >> + le16 token; /* bus-managed correlation identifier */ >> + le16 msg_size; /* total size: header (8) + payload */ >> + u8 payload[]; >> +}; >> +\end{lstlisting} >> + >> +Field semantics: >> +\begin{itemize} >> + \item \field{type}: >> + \begin{itemize} >> + \item Bit[0]: 0=request, 1=response. >> + \item Bit[1]: 0=transport message, 1=bus message. >> + \item Bits[2..7]: reserved for future use. >> + \end{itemize} >> + \item \field{msg_id}: Message ID identifying the message definition. Ranges >> + are defined in >> + \ref{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} >> + and >> + \ref{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages}. >> + \item \field{dev_num}: For transport messages, the device number that should >> + receive the message. Bus messages operate on device number 0. >> + \item \field{token}: Correlation identifier managed by the bus. Drivers and >> + devices treat this field as opaque; the bus MAY overwrite it to track >> + outstanding requests or maintain ordering. >> + \item \field{msg_size}: Total size in bytes of the complete message (header >> + plus payload). >> + \item \field{payload}: Operation-specific data. If a bus introduces extra >> + padding bytes, those bytes are not part of the payload semantics. >> +\end{itemize} >> + >> +\drivernormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Driver} >> +\begin{itemize} >> + \item A driver MUST set bits 2..7 of \field{type} to zero and MUST treat them >> + as reserved when parsing received headers. >> + \item A driver MUST ensure \field{msg_size} reflects the total message length >> + (header plus payload) and MUST NOT exceed the maximum message size >> + advertised by the bus instance. >> + \item When sending a transport message, a driver MUST set \field{dev_num} to >> + the intended device number. >> + \item If a driver introduces padding bytes that become part of the transport >> + payload, it MUST set those bytes to zero. >> + \item A driver MUST treat the \field{token} field as opaque and MUST NOT rely >> + on its value when processing received messages. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Device} >> +\begin{itemize} >> + \item A device MUST set bits 2..7 of \field{type} to zero in transmitted >> + messages and MUST ignore those bits on receive. >> + \item A device MUST ensure \field{msg_size} reflects the total message length >> + (header plus payload) and does not exceed the bus's advertised maximum. >> + \item When sending a transport message, a device MUST set \field{dev_num} to >> + its own device number. >> + \item A device MUST ignore padding bytes that are documented as bus-specific >> + and MUST zero any such bytes it introduces into the transport payload. >> + \item A device MUST treat the \field{token} field as opaque and MUST NOT rely >> + on its value when processing received messages. >> +\end{itemize} >> + >> +\busnormative{\paragraph}{Common Header}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Bus} >> +\begin{itemize} >> + \item A bus implementation MUST deliver bus messages with \field{dev_num}=0 >> + and MUST NOT alter \field{dev_num} for transport messages beyond the >> + routing needed to reach the addressed device. >> + \item A bus implementation MUST set bits 2..7 of \field{type} to zero when >> + generating bus messages and MUST ignore those bits when forwarding >> + transport messages. >> + \item If the bus adds framing or padding bytes around the common header or >> + payload, it MUST set those bytes to zero before delivering the message >> + to the opposite side and MUST present the same zero padding when the >> + opposite side reads a message. >> + \item A bus implementation owns the \field{token} field; it MAY insert or >> + overwrite values for correlation purposes and MUST ensure that any such >> + use is transparent to drivers and devices. >> +\end{itemize} >> + >> +Reserved header bits and unspecified header values MUST be transmitted as zero >> +and ignored on receive to preserve forward compatibility. > > This should be phrased in active tense: > > -> The bus implementation MUST transmit... Ack. > >> + >> +\subsubsection{Message Ordering} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering} >> + >> +Transport messages fall into two classes: requests (which expect responses) and >> +events (which are one-way). Drivers and devices rely on the bus to preserve the >> +relative ordering of request/response pairs for each device number; they do not >> +interpret the \field{token} field directly. >> + >> +\busnormative{\paragraph}{Message Ordering}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} >> +\begin{itemize} >> + \item For each device number, a bus implementation MUST deliver responses to >> + the driver in the same order that it forwarded the corresponding >> + requests to the device. >> + \item A bus implementation MUST ensure that every request forwarded to a >> + device results in exactly one response delivered to the driver (unless >> + the request is defined as an event). >> +\end{itemize} >> + >> +\subsection{Device Discovery}\label{sec:Virtio Transport Options / Virtio Over Messages / Device Discovery} >> + >> +A virtio-msg implementation can learn about devices via bus-level enumeration >> +(e.g., \busref{GET_DEVICES}), platform data (e.g., device tree, ACPI), or >> +hypervisor/firmware interfaces. The following informative text describes >> +typical flows; the normative obligations follow. >> + >> +Bus implementations discover their devices through a mix of platform data, >> +hypervisor-provided enumeration, or message-based queries such as >> +\busref{GET_DEVICES}. This specification defines \busref{GET_DEVICES} for >> +environments that prefer message-driven enumeration, but it does not require >> +its use when equivalent information is already known out-of-band. >> + >> +\busnormative{\paragraph}{Device Discovery}{Virtio Transport Options / Virtio Over Messages / Device Discovery / Bus} >> +\begin{itemize} >> + \item A bus implementation MUST ensure that every device number it exposes to >> + the driver has been validated via either platform data or a successful >> + query such as \busref{GET_DEVICES}. >> + \item Once a device number is deemed present, the bus implementation SHOULD >> + invoke \msgref{GET_DEVICE_INFO} so it can register the device with the >> + host OS and allow the appropriate virtio driver to bind. >> + \item If a bus implementation provides an alternative enumeration mechanism >> + (e.g., ACPI, device tree), it MAY omit \busref{GET_DEVICES} provided the >> + alternative delivers equivalent information to the driver. >> +\end{itemize} >> + >> +\subsection{Device Initialization} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization} >> + >> +After a bus has identified a virtio-msg device (whether by message-based >> +enumeration or platform-specific discovery), the driver undertakes a series of >> +steps to configure and ready the device for normal operation. This section >> +details the recommended order of operations and the associated messages. >> + >> +\subsubsection{Initialization Flow Overview} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview} >> + >> +A typical device initialization flow includes: >> +\begin{enumerate} >> + \item \textbf{Obtain Device Information:} >> + Query static parameters with \msgref{GET_DEVICE_INFO}. >> + \item \textbf{Negotiate Features:} >> + Read feature blocks via \msgref{GET_DEVICE_FEATURES}, decide which to >> + enable, and write them back with \msgref{SET_DRIVER_FEATURES} before >> + updating the device status. >> + \item \textbf{Initialize Configuration Space:} >> + Read or write configuration data with >> + \msgref{GET_CONFIG}/\msgref{SET_CONFIG}, guarding updates with the >> + configuration generation count. >> + \item \textbf{Set Up Virtqueues:} >> + Configure each virtqueue via \msgref{SET_VQUEUE} and verify the settings >> + with \msgref{GET_VQUEUE}. >> + \item \textbf{Set Device Status:} >> + Use \msgref{SET_DEVICE_STATUS} to drive the device through the standard >> + virtio status states. >> +\end{enumerate} >> + >> +The exact order may vary slightly across implementations, but these steps form >> +the baseline for virtio-msg initialization. >> + >> +\drivernormative{\paragraph}{Initialization Flow}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview / Driver} >> +\begin{itemize} >> + \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature >> + negotiation or queue setup. >> + \item A driver MUST complete feature negotiation via >> + \msgref{GET_DEVICE_FEATURES}/\msgref{SET_DRIVER_FEATURES} >> + and confirm the FEATURES\_OK state via \msgref{SET_DEVICE_STATUS} before >> + enabling virtqueues. >> + \item A driver MUST configure each virtqueue via \msgref{SET_VQUEUE} and >> + confirm its parameters (e.g., with \msgref{GET_VQUEUE}) before marking >> + the queue ready for I/O. >> + \item A driver MUST drive the device status transitions using >> + \msgref{SET_DEVICE_STATUS}, ensuring the device reaches DRIVER\_OK >> + before issuing normal I/O. >> +\end{itemize} >> + >> +\subsubsection{Device Information} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Information} >> + >> +Once a device number is confirmed, the driver uses \msgref{GET_DEVICE_INFO} to >> +retrieve static identification data (device ID, vendor ID), the number of >> +feature bits, configuration space size, maximum virtqueues, shared memory >> +segments, and any admin virtqueue details. This information determines which >> +virtio driver should bind to the device, how many feature blocks to query, and >> +how much configuration space is valid. >> + >> +\drivernormative{\paragraph}{Device Information}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Information / Driver} >> +\begin{itemize} >> + \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature >> + negotiation or queue setup so it can discover the device ID, vendor ID, >> + feature count, configuration size, and virtqueue limits. > > "A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature > negotiation or queue setup" is already stipulated in the previous > paragraph "Initialization Flow". True we will prevent repeating this part here in v2. > > >> + \item A driver MUST use the configuration size reported via >> + \msgref{GET_DEVICE_INFO} to bound any offsets and lengths supplied in >> + \msgref{GET_CONFIG} and \msgref{SET_CONFIG} requests. >> +\end{itemize} >> + >> +\subsubsection{Feature Negotiation} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features} >> + >> +Drivers read available features in 32-bit blocks using >> +\msgref{GET_DEVICE_FEATURES}; the device returns the requested bitfields, >> +padding with zeros for any out-of-range blocks. To enable selected features, the >> +driver writes them back with \msgref{SET_DRIVER_FEATURES} and then updates the >> +device status via \msgref{SET_DEVICE_STATUS}, checking whether the FEATURES\_OK >> +bit remains set. >> + >> +\drivernormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Driver} >> +\begin{itemize} >> + \item A driver MUST use \msgref{GET_DEVICE_FEATURES} to discover the feature >> + bits offered by the device and MUST write back only the features it >> + intends to enable via \msgref{SET_DRIVER_FEATURES}. >> + \item After writing the desired features, the driver MUST attempt to set the >> + FEATURES\_OK bit using \msgref{SET_DEVICE_STATUS} and MUST check the >> + returned status to ensure the device accepted the set. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} >> +\begin{itemize} >> + \item When handling \msgref{GET_DEVICE_FEATURES}, a device MUST return zero >> + for any requested bits that fall outside the number of feature bits it >> + implements. >> + \item After receiving \msgref{SET_DRIVER_FEATURES}, a device MUST update its >> + internal feature mask to match the acknowledged set and MUST reflect >> + acceptance or rejection by leaving the FEATURES\_OK bit set or clearing >> + it in the status returned by \msgref{SET_DEVICE_STATUS}. >> +\end{itemize} >> + >> +\subsubsection{Device Configuration} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration} >> + >> +Drivers use \msgref{GET_CONFIG} to read portions of the configuration space by >> +supplying an offset and length; the device returns the requested data plus the >> +current configuration generation count. Writing is performed via >> +\msgref{SET_CONFIG}, which carries the same offset/length along with the >> +driver's notion of the generation count and the new data. >> + >> +\drivernormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Driver} >> +\begin{itemize} >> + \item A driver MUST ensure that the offset and length in each >> + \msgref{GET_CONFIG} or \msgref{SET_CONFIG} request stay within the >> + configuration size reported by \msgref{GET_DEVICE_INFO}. >> + \item A driver MUST include its most recently observed configuration >> + generation count in a \msgref{SET_CONFIG} request and SHOULD re-read the >> + configuration (via \msgref{GET_CONFIG}) if the request is rejected for a >> + generation mismatch. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Device} >> +\begin{itemize} >> + \item A device MUST reject a \msgref{SET_CONFIG} request whose generation >> + count does not match its current value and MUST indicate the rejection >> + in the response. >> + \item A device MUST return the current configuration generation count >> + alongside any data returned via \msgref{GET_CONFIG}. >> +\end{itemize} >> + >> +\subsubsection{Virtqueue Configuration} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration} >> + >> +Drivers query virtqueue parameters with \msgref{GET_VQUEUE}, configure them via >> +\msgref{SET_VQUEUE}, and optionally reset them using \msgref{RESET_VQUEUE} (if >> +VIRTIO\_F\_RING\_RESET is negotiated). Each queue is typically configured by >> +reading its maximum size, provisioning descriptor/available/used buffers, and >> +then calling \msgref{SET_VQUEUE} with the chosen size and guest-physical >> +addresses. >> + >> +\drivernormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Driver} >> +\begin{itemize} >> + \item A driver MUST use \msgref{GET_VQUEUE} to determine the maximum queue >> + size and confirm that a queue is inactive before programming it. >> + \item A driver MUST ensure the queue size provided in \msgref{SET_VQUEUE} does >> + not exceed the maximum reported by the device and MUST supply valid >> + descriptor/driver/device addresses before enabling the queue. >> + \item If VIRTIO\_F\_RING\_RESET has been negotiated and a queue requires >> + reinitialization, the driver SHOULD use \msgref{RESET_VQUEUE} before >> + reprogramming it. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} >> +\begin{itemize} >> + \item A device MUST report accurate maximum queue sizes in \msgref{GET_VQUEUE} >> + and MUST persist the parameters supplied via \msgref{SET_VQUEUE} (size, >> + descriptor, driver, and device addresses). >> + \item When \msgref{RESET_VQUEUE} is issued (and VIRTIO\_F\_RING\_RESET is >> + negotiated), the device MUST quiesce the queue, release any resources >> + associated with it, and allow the driver to reconfigure it. >> +\end{itemize} >> + >> +\subsubsection{Status Information} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information} >> + >> +Drivers query the device status via \msgref{GET_DEVICE_STATUS} to observe >> +progress or detect errors, and they drive the Virtio status transitions via >> +\msgref{SET_DEVICE_STATUS}. Writing zero to the status field resets the device, >> +invalidating any configuration or virtqueue state. >> + >> +\drivernormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} >> +\begin{itemize} >> + \item A driver SHOULD read the device status via \msgref{GET_DEVICE_STATUS} >> + when diagnosing errors or determining whether the device is ready to >> + move to the next initialization phase. >> + \item A driver MUST use \msgref{SET_DEVICE_STATUS} to drive the device through >> + the virtio-defined status states and MUST write 0 to request a device >> + reset when needed. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} >> +\begin{itemize} >> + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST >> + reset its internal state, invalidate existing configuration and >> + virtqueue settings, and present the status field as 0. >> + \item A device MUST report its current status accurately via >> + \msgref{GET_DEVICE_STATUS}, including whether the FEATURES\_OK bit has >> + been accepted or cleared. >> +\end{itemize} >> + >> +\subsubsection{Finalizing Initialization} >> + >> +After configuring virtqueues and agreeing on features, the driver transitions >> +the device to DRIVER\_OK (and any other required status bits) via >> +\msgref{SET_DEVICE_STATUS}. At that point, the device is considered ready for >> +normal virtqueue I/O. >> + >> +\drivernormative{\paragraph}{Final Status}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Driver} >> +\begin{itemize} >> + \item A driver MUST set DRIVER\_OK via \msgref{SET_DEVICE_STATUS} only after >> + it has completed feature negotiation and initialized all required >> + virtqueues. >> + \item A driver MUST NOT queue normal I/O until the device reports a status >> + that includes DRIVER\_OK. >> + \item A driver MUST NOT supply buffers or send driver notifications for a >> + virtqueue until that queue has been configured via \msgref{SET_VQUEUE}. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Final Status}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Device} >> +\begin{itemize} >> + \item A device MUST NOT process normal virtqueue I/O until the driver has set >> + DRIVER\_OK. >> + \item Once DRIVER\_OK is set, the device MUST begin processing virtqueue >> + requests subject to the negotiated features and queue configurations. >> +\end{itemize} >> + >> +\subsection{Device Operation} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation} >> + >> +Once a virtio-msg device is fully initialized (see >> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization}), >> +the driver and device exchange messages to perform I/O and respond to >> +configuration changes. This section details the primary messaging paths for >> +\emph{driver notifications}, \emph{device notifications}, and the handling of >> +runtime resets or shutdown sequences. >> + >> +\subsubsection{Driver Notifications} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} >> + >> +After buffers are made available to a virtqueue, the driver issues >> +\msgref{EVENT_AVAIL} to notify the device. A bus implementation may translate >> +these notifications into an out-of-band mechanism or deliver them in-band. >> + >> +\drivernormative{\paragraph}{Driver Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Driver} >> +\begin{itemize} >> + \item A driver MUST send \msgref{EVENT_AVAIL} when it places new buffers on a >> + virtqueue. > > I suggest not defining when exactly the driver/device should send > notifications (cf. "Driver and Device Event Suppression", which is not > addressed here). It should suffice to state that the driver gives > available buffer notifications through EVENT_AVAIL, and to add similar > statements for configuration change and used buffer notifications. Agree. > >> + \item Each \msgref{EVENT_AVAIL} MUST identify the target virtqueue index. >> + \item If VIRTIO\_F\_NOTIFICATION\_DATA has been negotiated, the driver MUST >> + populate the “next offset and wrap” fields so the device can locate the >> + head of the updated ring; otherwise those fields MUST be zero. >> +\end{itemize} >> + >> +\busnormative{\paragraph}{Driver Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Bus} >> +\begin{itemize} >> + \item A bus implementation MAY convert \msgref{EVENT_AVAIL} into an >> + out-of-band notification but, if it does not, it MUST relay the message >> + over the virtio-msg channel. >> + \item A bus implementation MUST NOT drop \msgref{EVENT_AVAIL} unless it has >> + arranged for the device to detect new buffers through polling or an >> + equivalent mechanism. >> +\end{itemize} >> + >> +\subsubsection{Device Notifications} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications} >> + >> +\msgref{EVENT_CONFIG} and \msgref{EVENT_USED} provide asynchronous notifications >> +from the device (or device-side bus) to the driver. The bus may forward these >> +messages in-band or synthesize them based on other signals such as interrupts >> +or polling. >> + >> +\devicenormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} >> +\begin{itemize} >> + \item A device (or device-side bus) MUST send \msgref{EVENT_CONFIG} whenever >> + it makes a configuration change or status update that becomes visible to >> + the driver. The message MUST include the new configuration generation >> + count and MAY include the updated configuration data. >> + \item If the configuration data is omitted from \msgref{EVENT_CONFIG}, the >> + device SHOULD include the relevant offsets/lengths so the driver can >> + re-fetch the data via \msgref{GET_CONFIG}. >> + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when >> + buffers on a virtqueue have been consumed, unless the device relies on >> + an alternative, agreed-upon completion mechanism. > > Should this not be a MUST requirement? The driver might hang or need to > poll otherwise. The idea here was to let this open so that implementation relying only on poll are still compliant hence the SHOULD. Maybe we can try to rephrase this in a better way. > >> +\end{itemize} >> + >> +\drivernormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Driver} >> +\begin{itemize} >> + \item Upon receiving \msgref{EVENT_CONFIG}, a driver SHOULD update its view of >> + the configuration using the provided data (or by issuing >> + \msgref{GET_CONFIG} if necessary) before resuming normal I/O. >> + \item Upon receiving \msgref{EVENT_USED}, a driver MUST examine the indicated >> + virtqueue (for example, by reading the used ring) to reclaim completed >> + buffers. >> +\end{itemize} >> + >> +\busnormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Bus} >> +\begin{itemize} >> + \item A bus implementation MUST forward \msgref{EVENT_CONFIG} and >> + \msgref{EVENT_USED} notifications (or their equivalents) to the driver, >> + either in-band or by synthesizing the appropriate message. >> + \item If a bus implementation relies on polling or other mechanisms instead of >> + direct notifications, it SHOULD limit that mode to scenarios where no >> + other notification method is available. >> +\end{itemize} >> + >> +\subsubsection{Configuration Changes During Operation} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation} >> + >> +Drivers may update configuration fields at runtime using \msgref{SET_CONFIG} >> +when features such as device modes or limits need to change. Devices can also >> +update configuration data autonomously but must signal those changes via >> +\msgref{EVENT_CONFIG}. >> + >> +\drivernormative{\paragraph}{Runtime Configuration}{Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Driver} >> +\begin{itemize} >> + \item A driver MAY issue \msgref{SET_CONFIG} after initialization, provided it >> + includes the current configuration generation count and follows the same >> + validation rules as during setup. >> + \item Upon receiving an \msgref{EVENT_CONFIG}, the driver SHOULD refresh its >> + view of the configuration (via the data provided or by reissuing >> + \msgref{GET_CONFIG}) before relying on the updated values. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Runtime Configuration}{Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Device} >> +\begin{itemize} >> + \item If a device updates its configuration after initialization, it MUST send >> + \msgref{EVENT_CONFIG} to inform the driver of the change and provide the >> + new configuration generation count. >> +\end{itemize} >> + >> +\subsubsection{Virtqueue Changes During Operation} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation} >> + >> +Drivers may provision unused virtqueues later in the device lifetime by issuing >> +\msgref{SET_VQUEUE}, and they may reconfigure existing queues if the >> +VIRTIO\_F\_RING\_RESET feature has been negotiated. >> + >> +\drivernormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Driver} >> +\begin{itemize} >> + \item A driver MAY configure additional virtqueues after initialization using >> + \msgref{SET_VQUEUE}, provided it follows the same validation steps >> + (e.g., checking the maximum queue size). >> + \item If VIRTIO\_F\_RING\_RESET is negotiated, the driver SHOULD use >> + \msgref{RESET_VQUEUE} before reprogramming a queue to avoid races with >> + the device. > > I do not understand how this avoids races which could not be prevented > through a single message. The idea here is to say that an active queue should first be stopped/reseted before reconfiguring it so that there could not any ongoing usage of the queue when it being modified. At the end i guess this is not specific to that transport, so we might simply remove this. > >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Device} >> +\begin{itemize} >> + \item A device MUST honor \msgref{SET_VQUEUE} requests issued after >> + initialization and update the queue parameters accordingly. >> + \item When \msgref{RESET_VQUEUE} is received (and VIRTIO\_F\_RING\_RESET is >> + negotiated), the device MUST quiesce the queue and allow the driver to >> + reconfigure it without processing stale data. >> +\end{itemize} >> + >> +\subsubsection{Device Reset and Shutdown} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Reset} >> + >> +Drivers reset a device by writing 0 to the status field via \msgref{SET_DEVICE_STATUS}, >> +which forces the device to discard pending operations and return to the initial state. >> +A device may also signal that a reset is required by sending \msgref{EVENT_CONFIG} >> +with the DEVICE\_NEEDS\_RESET bit set. >> + >> +\drivernormative{\paragraph}{Reset and Shutdown}{Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Driver} >> +\begin{itemize} >> + \item A driver MAY request a device reset at any time by writing 0 through >> + \msgref{SET_DEVICE_STATUS} and MAY reinitialize the device afterwards if >> + it intends to resume operation. >> + \item If a device signals DEVICE\_NEEDS\_RESET (for example, via >> + \msgref{EVENT_CONFIG}), the driver SHOULD write 0 to >> + \msgref{SET_DEVICE_STATUS} and reinitialize the device before resuming >> + I/O. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Reset and Shutdown}{Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Device} >> +\begin{itemize} >> + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST >> + reset its internal state, discard pending virtqueue operations, and >> + present the status field as 0. >> + \item If the device encounters an unrecoverable error that requires driver >> + intervention, it SHOULD signal DEVICE\_NEEDS\_RESET (such as via >> + \msgref{EVENT_CONFIG}) so the driver can initiate a reset. >> +\end{itemize} >> + >> +\subsubsection{Hotplug and Removal} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Hotplug and Removal} >> + >> +If the bus supports dynamic addition or removal of devices, it can advertise >> +those changes with \busref{EVENT_DEVICE} (READY or REMOVED). Some platforms may >> +instead rely on external signals such as ACPI, device tree updates, or >> +hypervisor events; virtio-msg does not mandate a specific mechanism. >> + >> +\busnormative{\paragraph}{Hotplug and Removal}{Virtio Transport Options / Virtio Over Messages / Device Operation / Hotplug and Removal / Bus} >> +\begin{itemize} >> + \item A bus implementation that uses \busref{EVENT_DEVICE} MUST send the READY >> + event when a device becomes accessible and the REMOVED event when it is >> + no longer available. >> + \item Regardless of how hotplug information is delivered, once a new device is >> + reported the bus SHOULD query it (e.g., via \msgref{GET_DEVICE_INFO}) >> + and register it with the host OS. When a device is removed, the bus >> + SHOULD prompt the OS to quiesce and release the associated driver. >> +\end{itemize} >> + >> +\subsection{Transport Messages}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages} >> + >> +Transport messages configure and operate virtio devices once they have been >> +discovered. Unlike bus messages (which cover enumeration or hotplug), transport >> +messages focus on feature negotiation, configuration access, virtqueue setup, >> +and runtime notifications. The subsections below describe each message and how >> +it is used in the virtio-msg transport. >> + >> +\subsubsection{Overview} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / Overview} >> + >> +Drivers send transport messages to a specific device number, and the virtio-msg >> +device replies or emits events accordingly. The bus simply forwards these >> +messages after enforcing generic checks such as maximum size or verifying that >> +the target device exists. Most transport messages are request/response pairs, >> +with events reserved for asynchronous notifications. >> + >> +\paragraph{Messages IDs and issuers} >> + >> +\begin{tabular}{|l|l|l|} >> +\hline >> +Name & ID & Sender \\ >> +\hline >> +\hline >> +Reserved & 0x0 & \\ >> +\hline >> +Reserved & 0x1 & \\ >> +\hline >> +\msgref{GET_DEVICE_INFO} & 0x2 & Driver \\ >> +\hline >> +\msgref{GET_DEVICE_FEATURES} & 0x3 & Driver \\ >> +\hline >> +\msgref{SET_DRIVER_FEATURES} & 0x4 & Driver \\ >> +\hline >> +\msgref{GET_CONFIG} & 0x5 & Driver \\ >> +\hline >> +\msgref{SET_CONFIG} & 0x6 & Driver \\ >> +\hline >> +\msgref{GET_DEVICE_STATUS} & 0x7 & Driver \\ >> +\hline >> +\msgref{SET_DEVICE_STATUS} & 0x8 & Driver \\ >> +\hline >> +\msgref{GET_VQUEUE} & 0x9 & Driver \\ >> +\hline >> +\msgref{SET_VQUEUE} & 0xA & Driver \\ >> +\hline >> +\msgref{RESET_VQUEUE} & 0xB & Driver \\ >> +\hline >> +\msgref{GET_SHM} & 0xC & Driver \\ >> +\hline >> +\msgref{EVENT_CONFIG} & 0x40 & Device \\ >> +\hline >> +\msgref{EVENT_AVAIL} & 0x41 & Driver \\ >> +\hline >> +\msgref{EVENT_USED} & 0x42 & Device \\ >> +\hline >> +\end{tabular} >> + >> +IDs are grouped as follows; bit 6 distinguishes requests (0) from events (1): >> + >> +\begin{tabular}{|l|l|p{7cm}|} >> +\hline >> +Range & Usage & Notes \\ >> +\hline >> +0x00-0x3F & Request/Response & Standard transport requests that expect responses. \\ >> +0x40-0x7F & Events & Standard transport events (no response). \\ >> +0x80-0xBF & Implementation-defined Requests & Optional, transport-specific request extensions. \\ >> +0xC0-0xFF & Implementation-defined Events & Optional, transport-specific event extensions. \\ >> +\hline >> +\end{tabular} >> + >> +\devicenormative{\paragraph}{Core Messages}{Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Device} >> +\begin{itemize} >> + \item A device MUST implement \msgref{GET_DEVICE_INFO}, >> + \msgref{GET_DEVICE_FEATURES}, \msgref{SET_DRIVER_FEATURES}, >> + \msgref{GET_CONFIG}, \msgref{SET_CONFIG}, \msgref{GET_DEVICE_STATUS}, >> + \msgref{SET_DEVICE_STATUS}, \msgref{GET_VQUEUE}, \msgref{SET_VQUEUE}, >> + and \msgref{RESET_VQUEUE}. >> +\end{itemize} >> + >> +\drivernormative{\paragraph}{Core Messages}{Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Driver} >> +\begin{itemize} >> + \item A driver MUST support sending and interpreting the messages listed >> + above in order to initialize and operate a virtio-msg device. >> +\end{itemize} >> + >> +As described in Sections >> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} >> +and >> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications}, >> +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and >> +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band >> +mechanisms. >> + >> +\msgdef{GET_DEVICE_INFO} >> + >> +The driver issues \msgref{GET_DEVICE_INFO} to discover static identification >> +data and limits for a device. It is the only transport message permitted before >> +the device transitions out of the inactive state. >> + >> +\begin{lstlisting} >> +struct virtio_msg_get_device_info_req { >> + /* no payload */ >> +}; >> + >> +struct virtio_msg_get_device_info_resp { >> + le32 device_id; /* virtio device type */ >> + le32 vendor_id; /* implementation-defined vendor ID */ >> + le32 num_feature_bits; /* total feature bits (multiples of 32) */ >> + le32 config_size; /* bytes in the device configuration space */ >> + le32 max_virtqueues; /* maximum virtqueues supported */ >> + le16 admin_vq_start; /* index of the first admin virtqueue */ >> + le16 admin_vq_count; /* number of admin virtqueues */ > > Having le32 for max_virtqueues but le16 for admin vqs located at the end > seems inconsistent. Definitely, we will fix that in v2. > >> +}; >> +\end{lstlisting} >> + >> +\devicenormative{\paragraph}{GET\_DEVICE\_INFO}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Device} >> +\begin{itemize} >> + \item A device MUST respond to \msgref{GET_DEVICE_INFO} while inactive and >> + MUST supply consistent values for device ID, vendor ID, feature count (a >> + multiple of 32 bits), configuration size, and virtqueue limits. >> +\end{itemize} >> + >> +\drivernormative{\paragraph}{GET\_DEVICE\_INFO}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Driver} >> +\begin{itemize} >> + \item A driver SHOULD treat the response as the authoritative source for >> + feature count, configuration size, and virtqueue limits before >> + proceeding with initialization. >> +\end{itemize} >> + >> +\msgdef{GET_DEVICE_FEATURES} >> + >> +Drivers retrieve device feature bits in 32-bit blocks via >> +\msgref{GET_DEVICE_FEATURES}; the response echoes the requested block index and >> +returns one or more 32-bit values with the feature bits in that range. > > I do not understand the purpose of echoing request values. Looking at > virtqueue messages, the driver keeps track of its outstanding requests > internally and looks them up upon receiving the response. Doesn't a > virtio-msg transport also keep track internally? What advantage has > echoing? We only echo the index in the response (there is no data in the request). The idea here is to allow asynchronous requests/response and provide an easy way to identify the content of the data in the response. > >> + >> +\begin{lstlisting} >> +struct virtio_msg_get_device_features_req { >> + le32 block_index; /* starting block (0 == bits 0-31) */ >> + le32 num_blocks; /* number of 32-bit blocks requested */ >> +}; >> + >> +struct virtio_msg_get_device_features_resp { >> + le32 block_index; /* echoed starting block */ >> + le32 num_blocks; /* echoed number of blocks */ >> + le32 features[]; /* num_blocks entries, zero-padded if needed */ >> +}; >> +\end{lstlisting} >> + >> +\devicenormative{\paragraph}{GET\_DEVICE\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Device} >> +\begin{itemize} >> + \item A device MUST return zero for any feature bits beyond those it >> + implements when responding to \msgref{GET_DEVICE_FEATURES}. >> +\end{itemize} >> + >> +\drivernormative{\paragraph}{GET\_DEVICE\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Driver} >> +\begin{itemize} >> + \item A driver MUST treat the returned data as 32-bit feature blocks starting >> + at the requested index, padding with zeros as needed, before deciding >> + which features to enable. >> +\end{itemize} >> + >> +\msgdef{SET_DRIVER_FEATURES} >> + >> +Drivers use \msgref{SET_DRIVER_FEATURES} to acknowledge the subset of feature >> +bits they wish to enable. The payload mirrors \msgref{GET_DEVICE_FEATURES}, >> +supplying an index, count, and 32-bit feature data for the selected blocks. >> + >> +\begin{lstlisting} >> +struct virtio_msg_set_driver_features_req { >> + le32 block_index; /* starting block being acknowledged */ >> + le32 num_blocks; /* number of blocks provided */ >> + le32 features[]; /* num_blocks entries with desired bits */ >> +}; >> + >> +struct virtio_msg_set_driver_features_resp { >> + /* no payload */ >> +}; >> +\end{lstlisting} >> + >> +\drivernormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Driver} >> +\begin{itemize} >> + \item A driver MUST set only the feature bits it intends to enable in the >> + blocks supplied to \msgref{SET_DRIVER_FEATURES}. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Device} >> +\begin{itemize} >> + \item After processing \msgref{SET_DRIVER_FEATURES}, a device MUST update its >> + acknowledged feature set to match the data supplied and MUST report that >> + set consistently in subsequent \msgref{GET_DEVICE_FEATURES} responses. >> + If it cannot support the requested set, it SHOULD clear the FEATURES\_OK >> + bit in the device status. > > W.r.t. device status I think this rewording would be clearer: > > it SHOULD clear -> it MUST NOT set How about: It MUST clear the FEATURES_OK bit in the device status if it cannot support the requested set ? MUST NOT set implies that it was necessarily unset before which be a bit less precise. > >> +\end{itemize} >> + >> +\msgdef{GET_CONFIG} >> + >> +\msgref{GET_CONFIG} reads a portion of the configuration space. The request >> +specifies the byte offset and length; the response echoes those values, returns >> +the data, and includes the current configuration generation count. >> + >> +\begin{lstlisting} >> +struct virtio_msg_get_config_req { >> + le32 offset; /* byte offset into configuration space */ >> + le32 length; /* number of bytes requested */ >> +}; >> + >> +struct virtio_msg_get_config_resp { >> + le32 generation; /* current configuration generation count */ >> + le32 offset; /* echoed offset */ >> + le32 length; /* echoed length */ >> + u8 data[]; /* configuration payload */ >> +}; >> +\end{lstlisting} >> + >> +\drivernormative{\paragraph}{GET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Driver} >> +\begin{itemize} >> + \item A driver MUST ensure the requested offset and length in \msgref{GET_CONFIG} >> + stay within the configuration size reported by \msgref{GET_DEVICE_INFO}. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{GET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Device} >> +\begin{itemize} >> + \item A device MUST return the current configuration generation count with >> + every \msgref{GET_CONFIG} response. >> +\end{itemize} >> + >> +\msgdef{SET_CONFIG} >> + >> +\msgref{SET_CONFIG} writes a portion of configuration space, supplying the >> +offset, length, the driver's view of the generation count, and the new data. >> +The device echoes the offset/length, returns the resulting generation count, >> +and may mirror the applied data or set the length to zero if the write was >> +rejected. >> + >> +\begin{lstlisting} >> +struct virtio_msg_set_config_req { >> + le32 generation; /* driver's view of generation count */ >> + le32 offset; /* byte offset being written */ >> + le32 length; /* number of bytes written */ >> + u8 data[]; /* configuration payload */ >> +}; >> + >> +struct virtio_msg_set_config_resp { >> + le32 generation; /* resulting generation count */ >> + le32 offset; /* echoed offset */ >> + le32 length; /* bytes applied (0 if rejected) */ >> + u8 data[]; /* optional echoed data */ >> +}; >> +\end{lstlisting} >> + >> +\drivernormative{\paragraph}{SET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Driver} >> +\begin{itemize} >> + \item A driver MUST include its most recent configuration generation count in >> + each \msgref{SET_CONFIG} request and MUST keep the offset and length >> + within the reported configuration size. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{SET\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Device} >> +\begin{itemize} >> + \item A device MUST reject a \msgref{SET_CONFIG} request if the supplied >> + generation count does not match its current value and MUST report that >> + rejection in the response. >> +\end{itemize} >> + >> +\msgdef{GET_DEVICE_STATUS} >> + >> +\msgref{GET_DEVICE_STATUS} reads the current device status (e.g., ACKNOWLEDGE, >> +DRIVER, DRIVER\_OK, FEATURES\_OK, or DEVICE\_NEEDS\_RESET bits). The request has >> +no payload; the response returns the 32-bit status value. >> + >> +\begin{lstlisting} >> +struct virtio_msg_get_device_status_req { >> + /* no payload */ >> +}; >> + >> +struct virtio_msg_get_device_status_resp { >> + le32 status; /* current device status bits */ >> +}; >> +\end{lstlisting} >> + >> +\devicenormative{\paragraph}{GET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_STATUS / Device} >> +\begin{itemize} >> + \item A device MUST report its current status accurately via >> + \msgref{GET_DEVICE_STATUS}, including whether FEATURES\_OK or >> + DEVICE\_NEEDS\_RESET are set. >> +\end{itemize} >> + >> +\msgdef{SET_DEVICE_STATUS} >> + >> +\msgref{SET_DEVICE_STATUS} writes a new device status value. Drivers use it to >> +progress through the virtio-defined states or to request a reset by writing 0. >> +The device responds with its resulting status, which may differ (for example, >> +if it refuses FEATURES\_OK or sets DEVICE\_NEEDS\_RESET). >> + >> +\begin{lstlisting} >> +struct virtio_msg_set_device_status_req { >> + le32 status; /* desired device status value */ >> +}; >> + >> +struct virtio_msg_set_device_status_resp { >> + le32 status; /* resulting device status */ >> +}; >> +\end{lstlisting} >> + >> +\drivernormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver} >> +\begin{itemize} >> + \item A driver MUST write 0 via \msgref{SET_DEVICE_STATUS} to request a device >> + reset and MUST re-read the status (e.g., via \msgref{GET_DEVICE_STATUS}) >> + if it needs to confirm acceptance. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Device} >> +\begin{itemize} >> + \item A device MAY clear FEATURES\_OK or set DEVICE\_NEEDS\_RESET in its >> + response if it cannot accept the requested status, but it MUST report >> + the resulting status accurately. >> +\end{itemize} >> + >> +\msgdef{GET_VQUEUE} >> + >> +\msgref{GET_VQUEUE} returns information about a specific virtqueue, including >> +its maximum size, current size, and, if already configured, the descriptor, >> +driver, and device area addresses. > > Maybe GET_VQUEUE and the SET_VQUEUE response could also return whether > the vqueue is enabled? So getting a status back in get and having the possibility to set and enable in one call. Yes definitely possible. We will check to do that in v2. > >> + >> +\begin{lstlisting} >> +struct virtio_msg_get_vqueue_req { >> + le32 index; /* virtqueue index */ >> +}; >> + >> +struct virtio_msg_get_vqueue_resp { >> + le32 index; /* echoed virtqueue index */ >> + le32 max_size; /* maximum queue size */ >> + le32 cur_size; /* current size (0 if unconfigured) */ >> + le32 reserved; /* must be zero */ >> + le64 desc_addr; /* descriptor area address */ >> + le64 driver_addr; /* driver area address */ >> + le64 device_addr; /* device area address */ >> +}; >> +\end{lstlisting} >> + >> +\devicenormative{\paragraph}{GET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} >> +\begin{itemize} >> + \item A device MUST report accurate maxima and current queue sizes for each >> + virtqueue and MUST return zero as the current size if the queue has not >> + yet been configured. >> +\end{itemize} >> + >> +\msgdef{SET_VQUEUE} >> + >> +\msgref{SET_VQUEUE} programs a virtqueue's size and buffer addresses. The driver >> +selects a queue index, supplies the desired size (not exceeding the maximum >> +reported via \msgref{GET_VQUEUE}), and provides guest-physical addresses for the >> +descriptor, driver, and device areas. >> + >> +\begin{lstlisting} >> +struct virtio_msg_set_vqueue_req { >> + le32 index; /* virtqueue index */ >> + le32 reserved0; /* must be zero */ >> + le32 size; /* number of descriptors */ >> + le32 reserved1; /* must be zero */ >> + le64 desc_addr; /* descriptor area address */ >> + le64 driver_addr; /* driver area address */ >> + le64 device_addr; /* device area address */ >> +}; >> + >> +struct virtio_msg_set_vqueue_resp { >> + /* no payload */ >> +}; >> +\end{lstlisting} >> + >> +\drivernormative{\paragraph}{SET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Driver} >> +\begin{itemize} >> + \item A driver MUST set the virtqueue size field to a value not exceeding the >> + maximum reported by \msgref{GET_VQUEUE} and MUST ensure the supplied >> + addresses point to properly aligned memory. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{SET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Device} >> +\begin{itemize} >> + \item A device MUST configure the queue using the parameters provided via >> + \msgref{SET_VQUEUE} and treat the queue as inactive until >> + \msgref{SET_VQUEUE} has successfully completed. >> +\end{itemize} >> + >> +\msgdef{RESET_VQUEUE} >> + >> +\msgref{RESET_VQUEUE} resets an individual virtqueue (if VIRTIO\_F\_RING\_RESET >> +has been negotiated). It requires only the queue index; the device responds >> +after the queue has been quiesced. >> + >> +\begin{lstlisting} >> +struct virtio_msg_reset_vqueue_req { >> + le32 index; /* virtqueue index */ >> +}; >> + >> +struct virtio_msg_reset_vqueue_resp { >> + /* no payload */ >> +}; >> +\end{lstlisting} >> + >> +\drivernormative{\paragraph}{RESET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Driver} >> +\begin{itemize} >> + \item A driver SHOULD issue \msgref{RESET_VQUEUE} only if >> + VIRTIO\_F\_RING\_RESET has been negotiated and it needs to reconfigure >> + or recover a specific queue. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{RESET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Device} >> +\begin{itemize} >> + \item Upon receiving \msgref{RESET_VQUEUE}, a device MUST stop processing the >> + indicated queue, reset its internal state for that queue, and allow the >> + driver to reconfigure it via \msgref{SET_VQUEUE}. >> +\end{itemize} >> + >> +\msgdef{GET_SHM} >> + >> +\msgref{GET_SHM} returns the location and size of a device-owned shared memory >> +region. The request carries the region index; the response echoes the index and >> +provides the base address and length. A length of zero indicates that the >> +specified region does not exist. >> + >> +\begin{lstlisting} >> +struct virtio_msg_get_shm_req { >> + le32 index; /* shared memory region index */ >> +}; >> + >> +struct virtio_msg_get_shm_resp { >> + le32 index; /* echoed index */ >> + le32 length; /* region length (0 if nonexistent) */ >> + le32 address;/* region address */ > > length and address should be le64. Ack, we will fix that in v2. > >> +}; >> +\end{lstlisting} >> + >> +\devicenormative{\paragraph}{GET\_SHM}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_SHM / Device} >> +\begin{itemize} >> + \item A device MUST return zero length if the requested shared memory region >> + does not exist and MUST report accurate address/length information for >> + regions it supports. >> +\end{itemize} >> + >> +\msgdef{EVENT_CONFIG} >> + >> +\msgref{EVENT_CONFIG} notifies the driver about configuration or status changes. >> +The payload includes the current device status, configuration generation count, >> +and optionally the updated configuration data (offset plus length). A length of >> +zero indicates that the driver should re-fetch the affected range via >> +\msgref{GET_CONFIG}. > > IIUC the affected range should still be communicated through > virtio_msg_event_config.length. Maybe the above should be rephrased to > say that .data can be empty. Yes that would be clearer. We will update that. > >> + >> +\begin{lstlisting} >> +struct virtio_msg_event_config { >> + le32 device_status; /* new device status value */ >> + le32 generation; /* configuration generation count */ >> + le32 offset; /* configuration offset */ >> + le32 length; /* number of bytes (may be zero) */ >> + u8 data[]; /* optional configuration data */ >> +}; >> +\end{lstlisting} >> + >> +\drivernormative{\paragraph}{EVENT\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Driver} >> +\begin{itemize} >> + \item When \msgref{EVENT_CONFIG} provides configuration data, the driver >> + SHOULD apply it; otherwise it SHOULD re-read the affected range via >> + \msgref{GET_CONFIG} before proceeding. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{EVENT\_CONFIG}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Device} >> +\begin{itemize} >> + \item A device MUST send \msgref{EVENT_CONFIG} whenever it changes >> + configuration data or status in a way that is visible to the driver, >> + including the new configuration generation count. > > Isn't it enough to specify to send EVENT_CONFIG for every configuration > change notification? ("Device Requirements: Device Status Field" defines > that a device sends a configuration change notification when setting > DEVICE_NEEDS_RESET.) Yes this might be overly complex for no reason. We will rephrase. > >> +\end{itemize} >> + >> +\msgdef{EVENT_AVAIL} >> + >> +\msgref{EVENT_AVAIL} notifies the device that a virtqueue has new buffers. The >> +message includes the queue index and, if VIRTIO\_F\_NOTIFICATION\_DATA is >> +negotiated, a combined next-offset/next-wrap field so the device can jump >> +directly to the updated position. Otherwise that field is zero. >> + >> +\begin{lstlisting} >> +struct virtio_msg_event_avail { >> + le32 vq_index; /* virtqueue index */ >> + le32 next_offset; /* bits[30:0] offset, bit[31] wrap; zero if unused */ >> +}; >> +\end{lstlisting} >> + >> +\drivernormative{\paragraph}{EVENT\_AVAIL}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Driver} >> +\begin{itemize} >> + \item If VIRTIO\_F\_NOTIFICATION\_DATA has not been negotiated, a driver MUST >> + set the next offset/wrap field to zero in \msgref{EVENT_AVAIL} messages. >> + If it has been negotiated, the driver MUST fill in the field with the >> + current available ring position. >> +\end{itemize} >> + >> +\busnormative{\paragraph}{EVENT\_AVAIL}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Bus} >> +\begin{itemize} >> + \item A bus implementation that uses out-of-band notifications SHOULD prevent >> + negotiation of VIRTIO\_F\_NOTIFICATION\_DATA so that devices do not >> + expect in-band offset/wrap values. >> +\end{itemize} >> + >> +\msgdef{EVENT_USED} >> + >> +\msgref{EVENT_USED} notifies the driver that buffers on a particular virtqueue >> +have been consumed. The payload carries only the queue index; the driver uses >> +its used ring to determine which descriptors completed. >> + >> +\begin{lstlisting} >> +struct virtio_msg_event_used { >> + le32 vq_index; /* virtqueue index */ >> +}; >> +\end{lstlisting} >> + >> +\devicenormative{\paragraph}{EVENT\_USED}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Device} >> +\begin{itemize} >> + \item A device SHOULD send \msgref{EVENT_USED} (or an equivalent notification) >> + when it processes buffers on a virtqueue unless the driver has >> + explicitly disabled such notifications. >> +\end{itemize} >> + >> +\drivernormative{\paragraph}{EVENT\_USED}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Driver} >> +\begin{itemize} >> + \item Upon receiving \msgref{EVENT_USED}, a driver MUST examine the indicated >> + virtqueue (for example, by reading the used ring) to reclaim completed >> + buffers. >> +\end{itemize} >> + >> +\subsection{Bus Messages}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages} >> + >> +Bus messages cover device discovery, hotplug notifications, and bus-level >> +liveness checks. Implementations that already provide equivalent functionality >> +through platform mechanisms (e.g., ACPI, device tree, hypervisor events) may >> +reuse that infrastructure instead of the messages defined here. The following >> +subsections describe standardized bus messages that virtio-msg supports today. >> + >> +\subsubsection{Overview} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / Overview} >> + >> +Each bus implementation chooses which of these messages to support. Some may >> +implement all of them for a fully message-based workflow, while others may rely >> +on out-of-band enumeration and use only the subset that fits their needs. >> + >> +\paragraph{Messages IDs and issuers} >> + >> +\begin{tabular}{|l|l|l|} >> +\hline >> +Name & ID & Sender \\ >> +\hline >> +\hline >> +Reserved & 0x0 & \\ >> +\hline >> +Reserved & 0x1 & \\ >> +\hline >> +\busref{GET_DEVICES} & 0x2 & Driver \\ >> +\hline >> +\busref{PING} & 0x3 & Any \\ >> +\hline >> +\busref{EVENT_DEVICE} & 0x40 & Device \\ >> +\hline >> +\end{tabular} >> + >> +Bus message IDs are grouped as follows: >> + >> +\begin{tabular}{|l|l|p{7cm}|} >> +\hline >> +Range & Usage & Notes \\ >> +\hline >> +0x00-0x3F & Request/Response & Standardized bus request messages. \\ >> +0x40-0x7F & Events & Standardized bus event messages. \\ >> +0x80-0xBF & Implementation-defined Requests & Optional, bus-specific request messages. \\ >> +0xC0-0xFF & Implementation-defined Events & Optional, bus-specific event messages. \\ >> +\hline >> +\end{tabular} >> + >> +\paragraph{Bus Specific Messages} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Operation / Bus Specific Messages} >> + >> +A range of message IDs are reserved for use by the specific bus >> +implementation. These messages can be used for any implementation specific >> +usage. Example usage could include: >> + >> +\begin{itemize} >> + \item Configuration of out-of-band notification methods >> + \item Setup shared memory regions to be used for buffers or virtqueues >> + \item Declare bus specific error conditions >> + \item Provide extra debug or logging information >> +\end{itemize} >> + >> +\busdef{GET_DEVICES} >> + >> +The driver-side bus uses this request to enumerate device numbers on the >> +device-side bus. The response describes a “window” of device numbers and signals >> +which entries are present. >> + >> +\begin{lstlisting} >> +struct virtio_bus_msg_get_devices_req { >> + le16 offset; /* starting device number */ >> + le16 count; /* number of device-number slots (multiple of 8) */ >> +}; >> + >> +struct virtio_bus_msg_get_devices_resp { >> + le16 offset; /* echoed starting device number */ >> + le16 count; /* window length actually returned */ >> + le16 next_offset; /* 0 or suggested start for next query */ >> + u8 bitmap[]; /* count/8 bytes, LSB-first packing */ >> +}; >> +\end{lstlisting} >> + >> +The \textbf{(offset, count)} tuple defines a window of \textbf{count} >> +consecutive device numbers beginning at \textbf{offset}. The number of present >> +devices equals the number of 1-bits in the bitmap. Responders SHOULD return the >> +requested \textbf{count} unless constrained (e.g., by maximum message size); >> +otherwise they MAY return a smaller value, in which case the bitmap covers the >> +reduced window. >> + >> +Example: a request with \textbf{offset}=0, \textbf{count}=16 might produce >> +\textbf{bitmap}=0b00100101 00000000 and \textbf{next_offset}=16. That indicates >> +devices 0, 2, and 5 are present within the 0–15 window and suggests continuing >> +at device number 16. >> + >> +\paragraph{Intended usage} >> +Drivers MAY use this message to enumerate device numbers. Treat each window as >> +a snapshot, advance using \textbf{next_offset}, and confirm candidates via >> +\msgref{GET_DEVICE_INFO} before issuing other transport messages. >> + >> +\busnormative{\paragraph}{GET\_DEVICES}{Virtio Transport Options / Virtio Over Messages / Bus Messages / GET_DEVICES / Bus} >> +\begin{itemize} >> + \item The bus-side responder MUST enforce that \textbf{offset} and >> + \textbf{count} are multiples of 8, and it MUST return a bitmap of length >> + \textbf{count/8} packed least-significant-bit first. >> + \item \textbf{next\_offset} MUST be 0 (indicating no further windows) or an >> + aligned value $\ge \textbf{offset} + \textbf{count}$ recommending where >> + the driver should query next. >> + \item Responders SHOULD return the requested \textbf{count} unless constrained >> + (e.g., by maximum message size); if a smaller count is returned, the >> + bitmap MUST still represent the window defined by the echoed >> + \textbf{offset} and \textbf{count}. >> +\end{itemize} >> + >> +\busdef{PING} >> + >> +\busref{PING} is a simple liveness check that can be sent by either side of the >> +bus; the response echoes the 32-bit data value from the request. >> + >> +\begin{lstlisting} >> +struct virtio_bus_msg_ping_req { >> + le32 data; /* arbitrary value chosen by the sender */ >> +}; >> + >> +struct virtio_bus_msg_ping_resp { >> + le32 data; /* echoed request data */ >> +}; >> +\end{lstlisting} >> + >> +\paragraph{Intended usage} >> +Drivers or bus implementations MAY use \busref{PING} as a keepalive. If a reply >> +is missing or delayed beyond policy, the initiator SHOULD verify device status >> +(e.g., via \msgref{GET_DEVICE_STATUS}) and take recovery actions as needed. >> + >> +\busnormative{\paragraph}{PING}{Virtio Transport Options / Virtio Over Messages / Bus Messages / PING / Bus} >> +\begin{itemize} >> + \item The bus-side responder MUST echo the 32-bit data field from the request >> + exactly in the \busref{PING} response. >> +\end{itemize} >> + >> +\busdef{EVENT_DEVICE} >> + >> +\busref{EVENT_DEVICE} signals device-level hotplug changes. The driver-side bus >> +receives these indications from the device-side bus; the table below lists the >> +defined device bus states. >> + >> +\begin{lstlisting} >> +struct virtio_bus_msg_event_device { >> + le16 device_number; >> + le16 device_bus_state; /* see table below */ >> +}; >> +\end{lstlisting} >> + >> +\begin{tabular}{|l|l|p{7cm}|} >> +\hline >> +Constant & Value & Meaning \\ >> +\hline >> +DEVICE\_BUS\_STATE\_READY & 0x0001 & Device is present and ready. \\ >> +DEVICE\_BUS\_STATE\_REMOVED & 0x0002 & Device is no longer present. \\ >> +Reserved & 0x0003–0x7FFF & Reserved for standard use. \\ >> +Implementation-defined & 0x8000–0xFFFF & MAY be used by the bus implementation. \\ >> +\hline >> +\end{tabular} >> + >> +\paragraph{Intended usage} >> +Upon receiving DEVICE\_BUS\_STATE\_READY, drivers SHOULD probe the device via >> +\msgref{GET_DEVICE_INFO} and proceed with initialization if successful. Upon >> +DEVICE\_BUS\_STATE\_REMOVED, drivers MUST stop queueing new work, quiesce and >> +reset as applicable, and release resources. Drivers SHOULD tolerate duplicate or > > What does "reset" mean here? AFAIU the device is no longer present, so > cannot be reset any more? This was meant for reseting the driver side (calling remove or something like that). We will rephrase that. Thanks a lot for the very useful review. Tell me if you agree with my answers or if you have extra comments. Regards Bertrand IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-10 9:39 ` Bertrand Marquis @ 2026-02-12 11:16 ` Peter Hilber 2026-02-20 8:23 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Peter Hilber @ 2026-02-12 11:16 UTC (permalink / raw) To: Bertrand Marquis Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Tue, Feb 10, 2026 at 09:39:09AM +0000, Bertrand Marquis wrote: > Hi Peter, > Hi Bertrand, thank you for the fast response! I added replies where I think they are relevant. I also added a few new comments. Best regards, Peter > > On 6 Feb 2026, at 17:28, Peter Hilber <peter.hilber@oss.qualcomm.com> wrote: > > > > On Mon, Jan 26, 2026 at 11:32:28AM -0500, Bill Mills wrote: [...] > >> +\busnormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Bus} > >> +\begin{itemize} > >> + \item A bus instance that advertises transport revision $N$ MUST satisfy every > >> + bus requirement defined for revisions $1$ through $N$. > > > > Are transport feature bits supposed to be negotiated? I think this could > > make sense so that behavior can change without affecting > > interoperability between implementations of different revisions of the > > Virtio standard. > > Yes transport feature bits are supposed to be negotiated. > It is expected that the bus will determine what is the common set > of features supported and will only declare those to the transport. > > A simple bus implementation could just get from the device side the > mask of bits supported and mask it with the list of features bits supported > on the driver side and give back the common denominator to the transport > so that only features supported by both sides are advertised and used. > > We will make that clearer in v2 if you agree with that principle. > Ack. > > > >> + \item If a bus instance cannot forward a message because it requires an > >> + unsupported revision or transport feature, it MUST surface a transport > >> + error or drop the message without forwarding it. > >> +\end{itemize} > >> + > >> +\subsubsection{Device Numbers and Enumeration} > >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / > >> +Device Numbers} > >> + > >> +Each virtio-msg bus instance contains one or more \emph{devices}, identified According to a statement above, an instance can also have zero devices. > >> +by a 16-bit \textbf{device number}. Buses discover these device numbers through > >> +mechanisms such as: > >> +\begin{itemize} > >> + \item \textbf{Message-Based Enumeration}: Using \busref{GET_DEVICES} to query > >> + which numbers exist (optional). > >> + \item \textbf{Platform Data}: A device tree, ACPI tables, or hypervisor calls > >> + might inform the bus of available device numbers and their properties. > >> +\end{itemize} > >> + > >> +Once a bus confirms that a device number is valid—regardless of the discovery > >> +method—it normally issues \msgref{GET_DEVICE_INFO} to retrieve the device and > >> +vendor IDs before registering the device with the host OS so the usual Virtio > >> +driver binding process can begin. > >> + > >> +\busnormative{\paragraph}{Device Number Assignment}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Device Numbers / Assignment} > >> +\begin{itemize} > >> + \item A bus implementation MUST assign a unique device number to every > >> + device on a given bus instance and MUST NOT forward transport messages > >> + for a device number that has not been validated. An additional requirement could be added to limit the reuse of a device number after the device has disappeared, in order to avoid race conditions. > >> + \item A bus implementation SHOULD provide the driver with sufficient > >> + information—either via \busref{GET_DEVICES} or equivalent platform > >> + data—to discover each valid device number. > >> +\end{itemize} > >> + [...] > >> +\busnormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus} > >> +\begin{itemize} > >> + \item A bus implementation MAY report a transport-visible failure (for > >> + example, after exhausting a bounded retry policy) when it cannot deliver > >> + a request or obtain a response. > >> + \item A bus implementation MUST treat malformed headers or unsupported > >> + \field{msg_id} values as invalid, MUST discard them without generating > >> + additional protocol traffic, and MAY log the condition for diagnostics. The bus does not have to understand transport messages, so does it have to recognize all unsupported msg_id values? > >> + \item A bus implementation MUST NOT generate error responses to event > >> + (one-way) messages. > >> +\end{itemize} > >> + > >> +\devicenormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device} > >> +\begin{itemize} > >> + \item A device receiving a malformed or unsupported transport message MUST > >> + discard it without producing further protocol traffic. > >> + \item Recovery actions taken in response to an error (such as retries, > >> + selective resets, or device removal) MUST follow the normative reset and > >> + status semantics defined in > >> + \ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}. > >> +\end{itemize} > >> + > >> +This specification does not define a dedicated error-reporting message; it only > >> +permits implementations to surface failures when silent recovery is not > >> +feasible. > >> + [...] > >> +\drivernormative{\paragraph}{Initialization Flow}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview / Driver} > >> +\begin{itemize} > >> + \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature > >> + negotiation or queue setup. > >> + \item A driver MUST complete feature negotiation via > >> + \msgref{GET_DEVICE_FEATURES}/\msgref{SET_DRIVER_FEATURES} > >> + and confirm the FEATURES\_OK state via \msgref{SET_DEVICE_STATUS} before > >> + enabling virtqueues. > >> + \item A driver MUST configure each virtqueue via \msgref{SET_VQUEUE} and > >> + confirm its parameters (e.g., with \msgref{GET_VQUEUE}) before marking > >> + the queue ready for I/O. The Linux kernel driver mentioned in the cover letter currently doesn't confirm using GET_VQUEUE. But if SET_VQUEUE would return whether the queue is enabled, as was discussed below, the driver would only need to check whether the queue was successfully enabled. > >> + \item A driver MUST drive the device status transitions using > >> + \msgref{SET_DEVICE_STATUS}, ensuring the device reaches DRIVER\_OK > >> + before issuing normal I/O. > >> +\end{itemize} > >> + [...] > >> +\devicenormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} > >> +\begin{itemize} > >> + \item A device (or device-side bus) MUST send \msgref{EVENT_CONFIG} whenever > >> + it makes a configuration change or status update that becomes visible to > >> + the driver. The message MUST include the new configuration generation > >> + count and MAY include the updated configuration data. > >> + \item If the configuration data is omitted from \msgref{EVENT_CONFIG}, the > >> + device SHOULD include the relevant offsets/lengths so the driver can > >> + re-fetch the data via \msgref{GET_CONFIG}. > >> + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when > >> + buffers on a virtqueue have been consumed, unless the device relies on > >> + an alternative, agreed-upon completion mechanism. > > > > Should this not be a MUST requirement? The driver might hang or need to > > poll otherwise. > > The idea here was to let this open so that implementation relying only on poll are > still compliant hence the SHOULD. > > Maybe we can try to rephrase this in a better way. In my understanding rephrasing as MUST would be OK, since that only applies "unless the device relies on an alternative [...]". > > > > >> +\end{itemize} > >> + [...] > >> +\drivernormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Driver} > >> +\begin{itemize} > >> + \item A driver MAY configure additional virtqueues after initialization using > >> + \msgref{SET_VQUEUE}, provided it follows the same validation steps > >> + (e.g., checking the maximum queue size). > >> + \item If VIRTIO\_F\_RING\_RESET is negotiated, the driver SHOULD use > >> + \msgref{RESET_VQUEUE} before reprogramming a queue to avoid races with > >> + the device. > > > > I do not understand how this avoids races which could not be prevented > > through a single message. > > The idea here is to say that an active queue should first be stopped/reseted before > reconfiguring it so that there could not any ongoing usage of the queue when it being > modified. > > At the end i guess this is not specific to that transport, so we might simply remove this. I think it should be fine to remove the requirement. [...] > >> +\msgdef{GET_DEVICE_FEATURES} > >> + > >> +Drivers retrieve device feature bits in 32-bit blocks via > >> +\msgref{GET_DEVICE_FEATURES}; the response echoes the requested block index and > >> +returns one or more 32-bit values with the feature bits in that range. > > > > I do not understand the purpose of echoing request values. Looking at > > virtqueue messages, the driver keeps track of its outstanding requests > > internally and looks them up upon receiving the response. Doesn't a > > virtio-msg transport also keep track internally? What advantage has > > echoing? > > We only echo the index in the response (there is no data in the request). > The idea here is to allow asynchronous requests/response and provide > an easy way to identify the content of the data in the response. I would have thought that it is simpler and more robust if implementations would refer to the token to identify which request is being responded to (instead of possibly taking the response echoes at face value). Implementations might even have at most one outstanding request at a time. > > > > >> + > >> +\begin{lstlisting} > >> +struct virtio_msg_get_device_features_req { > >> + le32 block_index; /* starting block (0 == bits 0-31) */ > >> + le32 num_blocks; /* number of 32-bit blocks requested */ > >> +}; > >> + > >> +struct virtio_msg_get_device_features_resp { > >> + le32 block_index; /* echoed starting block */ > >> + le32 num_blocks; /* echoed number of blocks */ > >> + le32 features[]; /* num_blocks entries, zero-padded if needed */ > >> +}; > >> +\end{lstlisting} > >> + [...] > >> +\drivernormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Driver} > >> +\begin{itemize} > >> + \item A driver MUST set only the feature bits it intends to enable in the > >> + blocks supplied to \msgref{SET_DRIVER_FEATURES}. > >> +\end{itemize} > >> + > >> +\devicenormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Device} > >> +\begin{itemize} > >> + \item After processing \msgref{SET_DRIVER_FEATURES}, a device MUST update its > >> + acknowledged feature set to match the data supplied and MUST report that > >> + set consistently in subsequent \msgref{GET_DEVICE_FEATURES} responses. > >> + If it cannot support the requested set, it SHOULD clear the FEATURES\_OK > >> + bit in the device status. > > > > W.r.t. device status I think this rewording would be clearer: > > > > it SHOULD clear -> it MUST NOT set > > How about: It MUST clear the FEATURES_OK bit in the device status if it cannot support the requested set ? > > MUST NOT set implies that it was necessarily unset before which be a bit less precise. My understanding: "Device Requirements: Feature Bits" mandates that The device SHOULD accept any valid subset of features the driver accepts, otherwise it MUST fail to set the FEATURES_OK device status bit when the driver writes it. According to "Driver Requirements: Device Initialization", the FEATURES_OK bit in the device status is reset in step 1. It is only set in step 5 if the device accepts the subset accepted by the driver. Therefore, the FEATURES_OK bit in the device status is not set in the scenario outlined in the last sentence. But the last sentence also seems unnecessary as a transport requirement, so could just be removed. > > > > >> +\end{itemize} > >> + [...] > >> +\msgdef{SET_DEVICE_STATUS} > >> + > >> +\msgref{SET_DEVICE_STATUS} writes a new device status value. Drivers use it to > >> +progress through the virtio-defined states or to request a reset by writing 0. > >> +The device responds with its resulting status, which may differ (for example, > >> +if it refuses FEATURES\_OK or sets DEVICE\_NEEDS\_RESET). > >> + > >> +\begin{lstlisting} > >> +struct virtio_msg_set_device_status_req { > >> + le32 status; /* desired device status value */ > >> +}; > >> + > >> +struct virtio_msg_set_device_status_resp { > >> + le32 status; /* resulting device status */ > >> +}; > >> +\end{lstlisting} > >> + > >> +\drivernormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver} > >> +\begin{itemize} > >> + \item A driver MUST write 0 via \msgref{SET_DEVICE_STATUS} to request a device > >> + reset and MUST re-read the status (e.g., via \msgref{GET_DEVICE_STATUS}) > >> + if it needs to confirm acceptance. SET_DEVICE_STATUS already responds with the resulting device status. Why would re-reading the status be necessary? > >> +\end{itemize} > >> + > >> +\devicenormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Device} > >> +\begin{itemize} > >> + \item A device MAY clear FEATURES\_OK or set DEVICE\_NEEDS\_RESET in its > >> + response if it cannot accept the requested status, but it MUST report > >> + the resulting status accurately. > >> +\end{itemize} > >> + ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-12 11:16 ` Peter Hilber @ 2026-02-20 8:23 ` Bertrand Marquis 2026-02-26 13:53 ` Peter Hilber 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-20 8:23 UTC (permalink / raw) To: Peter Hilber Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Peter, > On 12 Feb 2026, at 12:16, Peter Hilber <peter.hilber@oss.qualcomm.com> wrote: > > On Tue, Feb 10, 2026 at 09:39:09AM +0000, Bertrand Marquis wrote: >> Hi Peter, >> > > Hi Bertrand, > > thank you for the fast response! I added replies where I think they are > relevant. I also added a few new comments. > > Best regards, > > Peter > >>> On 6 Feb 2026, at 17:28, Peter Hilber <peter.hilber@oss.qualcomm.com> wrote: >>> >>> On Mon, Jan 26, 2026 at 11:32:28AM -0500, Bill Mills wrote: > > [...] > >>>> +\busnormative{\paragraph}{Revision Compatibility}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Bus} >>>> +\begin{itemize} >>>> + \item A bus instance that advertises transport revision $N$ MUST satisfy every >>>> + bus requirement defined for revisions $1$ through $N$. >>> >>> Are transport feature bits supposed to be negotiated? I think this could >>> make sense so that behavior can change without affecting >>> interoperability between implementations of different revisions of the >>> Virtio standard. >> >> Yes transport feature bits are supposed to be negotiated. >> It is expected that the bus will determine what is the common set >> of features supported and will only declare those to the transport. >> >> A simple bus implementation could just get from the device side the >> mask of bits supported and mask it with the list of features bits supported >> on the driver side and give back the common denominator to the transport >> so that only features supported by both sides are advertised and used. >> >> We will make that clearer in v2 if you agree with that principle. >> > > Ack. > >>> >>>> + \item If a bus instance cannot forward a message because it requires an >>>> + unsupported revision or transport feature, it MUST surface a transport >>>> + error or drop the message without forwarding it. >>>> +\end{itemize} >>>> + >>>> +\subsubsection{Device Numbers and Enumeration} >>>> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / >>>> +Device Numbers} >>>> + >>>> +Each virtio-msg bus instance contains one or more \emph{devices}, identified > > According to a statement above, an instance can also have zero devices. Right I need to fix that to be coherent. Point of allowing 0 devices is to allow the case where devices are hotplugged. > >>>> +by a 16-bit \textbf{device number}. Buses discover these device numbers through >>>> +mechanisms such as: >>>> +\begin{itemize} >>>> + \item \textbf{Message-Based Enumeration}: Using \busref{GET_DEVICES} to query >>>> + which numbers exist (optional). >>>> + \item \textbf{Platform Data}: A device tree, ACPI tables, or hypervisor calls >>>> + might inform the bus of available device numbers and their properties. >>>> +\end{itemize} >>>> + >>>> +Once a bus confirms that a device number is valid—regardless of the discovery >>>> +method—it normally issues \msgref{GET_DEVICE_INFO} to retrieve the device and >>>> +vendor IDs before registering the device with the host OS so the usual Virtio >>>> +driver binding process can begin. >>>> + >>>> +\busnormative{\paragraph}{Device Number Assignment}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Device Numbers / Assignment} >>>> +\begin{itemize} >>>> + \item A bus implementation MUST assign a unique device number to every >>>> + device on a given bus instance and MUST NOT forward transport messages >>>> + for a device number that has not been validated. > > An additional requirement could be added to limit the reuse of a device > number after the device has disappeared, in order to avoid race > conditions. That is an interesting point that would require to increase the number of device IDs as Demi pointed out. Point noted and we will investigate (i will answer the increase of device IDs directly to Demi's mail). > >>>> + \item A bus implementation SHOULD provide the driver with sufficient >>>> + information—either via \busref{GET_DEVICES} or equivalent platform >>>> + data—to discover each valid device number. >>>> +\end{itemize} >>>> + > > [...] > >>>> +\busnormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus} >>>> +\begin{itemize} >>>> + \item A bus implementation MAY report a transport-visible failure (for >>>> + example, after exhausting a bounded retry policy) when it cannot deliver >>>> + a request or obtain a response. >>>> + \item A bus implementation MUST treat malformed headers or unsupported >>>> + \field{msg_id} values as invalid, MUST discard them without generating >>>> + additional protocol traffic, and MAY log the condition for diagnostics. > > The bus does not have to understand transport messages, so does it have > to recognize all unsupported msg_id values? Very true, i will need to fix the wording to state invalid bus msg ids, transport should discard invalid transport message IDs. > >>>> + \item A bus implementation MUST NOT generate error responses to event >>>> + (one-way) messages. >>>> +\end{itemize} >>>> + >>>> +\devicenormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device} >>>> +\begin{itemize} >>>> + \item A device receiving a malformed or unsupported transport message MUST >>>> + discard it without producing further protocol traffic. >>>> + \item Recovery actions taken in response to an error (such as retries, >>>> + selective resets, or device removal) MUST follow the normative reset and >>>> + status semantics defined in >>>> + \ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}. >>>> +\end{itemize} >>>> + >>>> +This specification does not define a dedicated error-reporting message; it only >>>> +permits implementations to surface failures when silent recovery is not >>>> +feasible. >>>> + > > [...] > >>>> +\drivernormative{\paragraph}{Initialization Flow}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview / Driver} >>>> +\begin{itemize} >>>> + \item A driver MUST issue \msgref{GET_DEVICE_INFO} before attempting feature >>>> + negotiation or queue setup. >>>> + \item A driver MUST complete feature negotiation via >>>> + \msgref{GET_DEVICE_FEATURES}/\msgref{SET_DRIVER_FEATURES} >>>> + and confirm the FEATURES\_OK state via \msgref{SET_DEVICE_STATUS} before >>>> + enabling virtqueues. >>>> + \item A driver MUST configure each virtqueue via \msgref{SET_VQUEUE} and >>>> + confirm its parameters (e.g., with \msgref{GET_VQUEUE}) before marking >>>> + the queue ready for I/O. > > The Linux kernel driver mentioned in the cover letter currently doesn't > confirm using GET_VQUEUE. But if SET_VQUEUE would return whether the > queue is enabled, as was discussed below, the driver would only need to > check whether the queue was successfully enabled. GET_VQUEUE would still be need to have functionalities equivalent to PCI or MMIO. But agree that it will not be needed to be done anymore there in the driver if we have the enabled status in the answer to SET. > >>>> + \item A driver MUST drive the device status transitions using >>>> + \msgref{SET_DEVICE_STATUS}, ensuring the device reaches DRIVER\_OK >>>> + before issuing normal I/O. >>>> +\end{itemize} >>>> + > > [...] > >>>> +\devicenormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} >>>> +\begin{itemize} >>>> + \item A device (or device-side bus) MUST send \msgref{EVENT_CONFIG} whenever >>>> + it makes a configuration change or status update that becomes visible to >>>> + the driver. The message MUST include the new configuration generation >>>> + count and MAY include the updated configuration data. >>>> + \item If the configuration data is omitted from \msgref{EVENT_CONFIG}, the >>>> + device SHOULD include the relevant offsets/lengths so the driver can >>>> + re-fetch the data via \msgref{GET_CONFIG}. >>>> + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when >>>> + buffers on a virtqueue have been consumed, unless the device relies on >>>> + an alternative, agreed-upon completion mechanism. >>> >>> Should this not be a MUST requirement? The driver might hang or need to >>> poll otherwise. >> >> The idea here was to let this open so that implementation relying only on poll are >> still compliant hence the SHOULD. >> >> Maybe we can try to rephrase this in a better way. > > In my understanding rephrasing as MUST would be OK, since that only > applies "unless the device relies on an alternative [...]". Ack > >> >>> >>>> +\end{itemize} >>>> + > > [...] > >>>> +\drivernormative{\paragraph}{Runtime Virtqueue Changes}{Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Driver} >>>> +\begin{itemize} >>>> + \item A driver MAY configure additional virtqueues after initialization using >>>> + \msgref{SET_VQUEUE}, provided it follows the same validation steps >>>> + (e.g., checking the maximum queue size). >>>> + \item If VIRTIO\_F\_RING\_RESET is negotiated, the driver SHOULD use >>>> + \msgref{RESET_VQUEUE} before reprogramming a queue to avoid races with >>>> + the device. >>> >>> I do not understand how this avoids races which could not be prevented >>> through a single message. >> >> The idea here is to say that an active queue should first be stopped/reseted before >> reconfiguring it so that there could not any ongoing usage of the queue when it being >> modified. >> >> At the end i guess this is not specific to that transport, so we might simply remove this. > > I think it should be fine to remove the requirement. Ack. > > [...] > >>>> +\msgdef{GET_DEVICE_FEATURES} >>>> + >>>> +Drivers retrieve device feature bits in 32-bit blocks via >>>> +\msgref{GET_DEVICE_FEATURES}; the response echoes the requested block index and >>>> +returns one or more 32-bit values with the feature bits in that range. >>> >>> I do not understand the purpose of echoing request values. Looking at >>> virtqueue messages, the driver keeps track of its outstanding requests >>> internally and looks them up upon receiving the response. Doesn't a >>> virtio-msg transport also keep track internally? What advantage has >>> echoing? >> >> We only echo the index in the response (there is no data in the request). >> The idea here is to allow asynchronous requests/response and provide >> an easy way to identify the content of the data in the response. > > I would have thought that it is simpler and more robust if > implementations would refer to the token to identify which request is > being responded to (instead of possibly taking the response echoes at > face value). Implementations might even have at most one outstanding > request at a time. We want to be future proof so that this is possible in the future. If tokens are purely handled by the bus, having all information might be easier for the transport. > >> >>> >>>> + >>>> +\begin{lstlisting} >>>> +struct virtio_msg_get_device_features_req { >>>> + le32 block_index; /* starting block (0 == bits 0-31) */ >>>> + le32 num_blocks; /* number of 32-bit blocks requested */ >>>> +}; >>>> + >>>> +struct virtio_msg_get_device_features_resp { >>>> + le32 block_index; /* echoed starting block */ >>>> + le32 num_blocks; /* echoed number of blocks */ >>>> + le32 features[]; /* num_blocks entries, zero-padded if needed */ >>>> +}; >>>> +\end{lstlisting} >>>> + > > [...] > >>>> +\drivernormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Driver} >>>> +\begin{itemize} >>>> + \item A driver MUST set only the feature bits it intends to enable in the >>>> + blocks supplied to \msgref{SET_DRIVER_FEATURES}. >>>> +\end{itemize} >>>> + >>>> +\devicenormative{\paragraph}{SET\_DRIVER\_FEATURES}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Device} >>>> +\begin{itemize} >>>> + \item After processing \msgref{SET_DRIVER_FEATURES}, a device MUST update its >>>> + acknowledged feature set to match the data supplied and MUST report that >>>> + set consistently in subsequent \msgref{GET_DEVICE_FEATURES} responses. >>>> + If it cannot support the requested set, it SHOULD clear the FEATURES\_OK >>>> + bit in the device status. >>> >>> W.r.t. device status I think this rewording would be clearer: >>> >>> it SHOULD clear -> it MUST NOT set >> >> How about: It MUST clear the FEATURES_OK bit in the device status if it cannot support the requested set ? >> >> MUST NOT set implies that it was necessarily unset before which be a bit less precise. > > My understanding: > > "Device Requirements: Feature Bits" mandates that > > The device SHOULD accept any valid subset of features the driver > accepts, otherwise it MUST fail to set the FEATURES_OK device > status bit when the driver writes it. > > According to "Driver Requirements: Device Initialization", the > FEATURES_OK bit in the device status is reset in step 1. It is only set > in step 5 if the device accepts the subset accepted by the driver. > > Therefore, the FEATURES_OK bit in the device status is not set in the > scenario outlined in the last sentence. But the last sentence also seems > unnecessary as a transport requirement, so could just be removed. Agree > >> >>> >>>> +\end{itemize} >>>> + > > [...] > >>>> +\msgdef{SET_DEVICE_STATUS} >>>> + >>>> +\msgref{SET_DEVICE_STATUS} writes a new device status value. Drivers use it to >>>> +progress through the virtio-defined states or to request a reset by writing 0. >>>> +The device responds with its resulting status, which may differ (for example, >>>> +if it refuses FEATURES\_OK or sets DEVICE\_NEEDS\_RESET). >>>> + >>>> +\begin{lstlisting} >>>> +struct virtio_msg_set_device_status_req { >>>> + le32 status; /* desired device status value */ >>>> +}; >>>> + >>>> +struct virtio_msg_set_device_status_resp { >>>> + le32 status; /* resulting device status */ >>>> +}; >>>> +\end{lstlisting} >>>> + >>>> +\drivernormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver} >>>> +\begin{itemize} >>>> + \item A driver MUST write 0 via \msgref{SET_DEVICE_STATUS} to request a device >>>> + reset and MUST re-read the status (e.g., via \msgref{GET_DEVICE_STATUS}) >>>> + if it needs to confirm acceptance. > > SET_DEVICE_STATUS already responds with the resulting device status. > Why would re-reading the status be necessary? We identified some cases in which a reset cannot be handled atomically hence the need to have a way to confirm or wait until reset is finished. > >>>> +\end{itemize} >>>> + >>>> +\devicenormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Device} >>>> +\begin{itemize} >>>> + \item A device MAY clear FEATURES\_OK or set DEVICE\_NEEDS\_RESET in its >>>> + response if it cannot accept the requested status, but it MUST report >>>> + the resulting status accurately. >>>> +\end{itemize} >>>> + Cheers Bertrand IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-20 8:23 ` Bertrand Marquis @ 2026-02-26 13:53 ` Peter Hilber 0 siblings, 0 replies; 105+ messages in thread From: Peter Hilber @ 2026-02-26 13:53 UTC (permalink / raw) To: Bertrand Marquis Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Fri, Feb 20, 2026 at 08:23:26AM +0000, Bertrand Marquis wrote: > Hi Peter, > > > On 12 Feb 2026, at 12:16, Peter Hilber > > <peter.hilber@oss.qualcomm.com> wrote: > > > > On Tue, Feb 10, 2026 at 09:39:09AM +0000, Bertrand Marquis wrote: > >> Hi Peter, > >> > > > > Hi Bertrand, > > > > thank you for the fast response! I added replies where I think they are > > relevant. I also added a few new comments. > > > > Best regards, > > > > Peter > > > >>> On 6 Feb 2026, at 17:28, Peter Hilber <peter.hilber@oss.qualcomm.com> wrote: > >>> > >>> On Mon, Jan 26, 2026 at 11:32:28AM -0500, Bill Mills wrote: [...] > >>>> +\busnormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus} > >>>> +\begin{itemize} > >>>> + \item A bus implementation MAY report a transport-visible failure (for > >>>> + example, after exhausting a bounded retry policy) when it cannot deliver > >>>> + a request or obtain a response. > >>>> + \item A bus implementation MUST treat malformed headers or unsupported > >>>> + \field{msg_id} values as invalid, MUST discard them without generating > >>>> + additional protocol traffic, and MAY log the condition for diagnostics. > > > > The bus does not have to understand transport messages, so does it have > > to recognize all unsupported msg_id values? > > Very true, i will need to fix the wording to state invalid bus msg ids, transport should > discard invalid transport message IDs. > I am also wondering why any "additional protocol traffic" is forbidden for unsupported (bus or transport) requests. Couldn't it be reasonable to indicate an error so that the remote endpoint doesn't need to wait for a response timeout? > > > >>>> + \item A bus implementation MUST NOT generate error responses to event > >>>> + (one-way) messages. > >>>> +\end{itemize} > >>>> + > >>>> +\devicenormative{\paragraph}{Error Handling}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device} > >>>> +\begin{itemize} > >>>> + \item A device receiving a malformed or unsupported transport message MUST > >>>> + discard it without producing further protocol traffic. > >>>> + \item Recovery actions taken in response to an error (such as retries, > >>>> + selective resets, or device removal) MUST follow the normative reset and > >>>> + status semantics defined in > >>>> + \ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation}. > >>>> +\end{itemize} > >>>> + > >>>> +This specification does not define a dedicated error-reporting message; it only > >>>> +permits implementations to surface failures when silent recovery is not > >>>> +feasible. > >>>> + [...] > >>>> +\msgdef{GET_DEVICE_FEATURES} > >>>> + > >>>> +Drivers retrieve device feature bits in 32-bit blocks via > >>>> +\msgref{GET_DEVICE_FEATURES}; the response echoes the requested block index and > >>>> +returns one or more 32-bit values with the feature bits in that range. > >>> > >>> I do not understand the purpose of echoing request values. Looking at > >>> virtqueue messages, the driver keeps track of its outstanding requests > >>> internally and looks them up upon receiving the response. Doesn't a > >>> virtio-msg transport also keep track internally? What advantage has > >>> echoing? > >> > >> We only echo the index in the response (there is no data in the request). > >> The idea here is to allow asynchronous requests/response and provide > >> an easy way to identify the content of the data in the response. > > > > I would have thought that it is simpler and more robust if > > implementations would refer to the token to identify which request is > > being responded to (instead of possibly taking the response echoes at > > face value). Implementations might even have at most one outstanding > > request at a time. > > We want to be future proof so that this is possible in the future. > If tokens are purely handled by the bus, having all information might be > easier for the transport. > I still feel uneasy, since I think the context of a response would be available locally easily and in a more robust manner (e.g. if the same request is issued repeatedly). The bus preserves the (per-device) request-response order anyway. So couldn't the bus layer point out the association of responses to requests to the transport (as needed, e.g. through a pointer submitted by the transport alongside the request)? Also, omitting the echoes would shrink the response size (with message sizes being a concern AFAIU). > >>>> +\msgdef{SET_DEVICE_STATUS} > >>>> + > >>>> +\msgref{SET_DEVICE_STATUS} writes a new device status value. Drivers use it to > >>>> +progress through the virtio-defined states or to request a reset by writing 0. > >>>> +The device responds with its resulting status, which may differ (for example, > >>>> +if it refuses FEATURES\_OK or sets DEVICE\_NEEDS\_RESET). > >>>> + > >>>> +\begin{lstlisting} > >>>> +struct virtio_msg_set_device_status_req { > >>>> + le32 status; /* desired device status value */ > >>>> +}; > >>>> + > >>>> +struct virtio_msg_set_device_status_resp { > >>>> + le32 status; /* resulting device status */ > >>>> +}; > >>>> +\end{lstlisting} > >>>> + > >>>> +\drivernormative{\paragraph}{SET\_DEVICE\_STATUS}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver} > >>>> +\begin{itemize} > >>>> + \item A driver MUST write 0 via \msgref{SET_DEVICE_STATUS} to request a device > >>>> + reset and MUST re-read the status (e.g., via \msgref{GET_DEVICE_STATUS}) > >>>> + if it needs to confirm acceptance. > > > > SET_DEVICE_STATUS already responds with the resulting device status. > > Why would re-reading the status be necessary? > > We identified some cases in which a reset cannot be handled atomically hence the need to have a way to confirm or wait until reset is finished. I think this should be rephrased to explicitly indicate that the driver must wait until it observes the device status change. As of now, it could be interpreted to mean that the driver just needs to confirm that a status change did not fail by re-reading once. Best regards, Peter ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-01-26 16:32 ` [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer Bill Mills 2026-02-06 16:28 ` Peter Hilber @ 2026-02-13 19:09 ` Demi Marie Obenour 2026-02-20 8:52 ` Bertrand Marquis 2026-02-24 15:41 ` Demi Marie Obenour ` (2 subsequent siblings) 4 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-13 19:09 UTC (permalink / raw) To: Bill Mills, virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 12672 bytes --] On 1/26/26 11:32, Bill Mills wrote: > Add a new transport layer that is based on messages. > > This transport layer still uses virtqueues as the other transport layers do > but implements transport layer operations by sending and receiving messages > instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. > > This transport is useful when the device and driver are both implemented in > software but the trap and emulate operations of virtio-mmio and virtio-pci > can not be used. (snip) > +\subsubsection{Message Ordering} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering} > + > +Transport messages fall into two classes: requests (which expect responses) and > +events (which are one-way). Drivers and devices rely on the bus to preserve the > +relative ordering of request/response pairs for each device number; they do not > +interpret the \field{token} field directly. I expect that this requires all messages for a given device to be processed sequentially, which is not going to be fast. At the very least, messages on different virtqueues should be able to be processed out of order. I would have some sort of stream identifier that is made visible to the bus layer. This would map to a QUIC stream, a single ring buffer, or something else that ensures in-order delivery. Messages with different stream identifiers may be processed out of order. > +\busnormative{\paragraph}{Message Ordering}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} > +\begin{itemize} > + \item For each device number, a bus implementation MUST deliver responses to > + the driver in the same order that it forwarded the corresponding > + requests to the device. > + \item A bus implementation MUST ensure that every request forwarded to a > + device results in exactly one response delivered to the driver (unless > + the request is defined as an event). > +\end{itemize} What is the reason for the device/driver distinction here? Intuitively, I expect both requests and responses to just be messages at the bus layer. (snip) > +\devicenormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} > +\begin{itemize} > + \item When handling \msgref{GET_DEVICE_FEATURES}, a device MUST return zero > + for any requested bits that fall outside the number of feature bits it > + implements. > + \item After receiving \msgref{SET_DRIVER_FEATURES}, a device MUST update its > + internal feature mask to match the acknowledged set and MUST reflect > + acceptance or rejection by leaving the FEATURES\_OK bit set or clearing > + it in the status returned by \msgref{SET_DEVICE_STATUS}. > +\end{itemize} What should the device do if it doesn't support the feature? > +\subsubsection{Device Configuration} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration} > + > +Drivers use \msgref{GET_CONFIG} to read portions of the configuration space by > +supplying an offset and length; the device returns the requested data plus the > +current configuration generation count. Writing is performed via > +\msgref{SET_CONFIG}, which carries the same offset/length along with the > +driver's notion of the generation count and the new data. I think this should come with a (non-normative) note that the use of configuration space is discouraged if the data can instead be sent over a virtqueue. For MMIO and PCI transports, the configuration space is slightly more convenient as it is accessed synchronously, but for the message transport, reads are also asynchronous unless the driver spins or sleeps until getting a reply. > +\drivernormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Driver} > +\begin{itemize} > + \item A driver MUST ensure that the offset and length in each > + \msgref{GET_CONFIG} or \msgref{SET_CONFIG} request stay within the > + configuration size reported by \msgref{GET_DEVICE_INFO}. > + \item A driver MUST include its most recently observed configuration > + generation count in a \msgref{SET_CONFIG} request and SHOULD re-read the > + configuration (via \msgref{GET_CONFIG}) if the request is rejected for a > + generation mismatch. > +\end{itemize} > + > +\devicenormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Device} > +\begin{itemize} > + \item A device MUST reject a \msgref{SET_CONFIG} request whose generation > + count does not match its current value and MUST indicate the rejection > + in the response. > + \item A device MUST return the current configuration generation count > + alongside any data returned via \msgref{GET_CONFIG}. > +\end{itemize} > + > +\subsubsection{Virtqueue Configuration} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration} > + > +Drivers query virtqueue parameters with \msgref{GET_VQUEUE}, configure them via > +\msgref{SET_VQUEUE}, and optionally reset them using \msgref{RESET_VQUEUE} (if > +VIRTIO\_F\_RING\_RESET is negotiated). Each queue is typically configured by > +reading its maximum size, provisioning descriptor/available/used buffers, and > +then calling \msgref{SET_VQUEUE} with the chosen size and guest-physical > +addresses. > + > +\drivernormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Driver} > +\begin{itemize} > + \item A driver MUST use \msgref{GET_VQUEUE} to determine the maximum queue > + size and confirm that a queue is inactive before programming it. > + \item A driver MUST ensure the queue size provided in \msgref{SET_VQUEUE} does > + not exceed the maximum reported by the device and MUST supply valid > + descriptor/driver/device addresses before enabling the queue. > + \item If VIRTIO\_F\_RING\_RESET has been negotiated and a queue requires > + reinitialization, the driver SHOULD use \msgref{RESET_VQUEUE} before > + reprogramming it. > +\end{itemize} Why SHOULD and not MUST? > +\devicenormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} > +\begin{itemize} > + \item A device MUST report accurate maximum queue sizes in \msgref{GET_VQUEUE} > + and MUST persist the parameters supplied via \msgref{SET_VQUEUE} (size, > + descriptor, driver, and device addresses). > + \item When \msgref{RESET_VQUEUE} is issued (and VIRTIO\_F\_RING\_RESET is > + negotiated), the device MUST quiesce the queue, release any resources > + associated with it, and allow the driver to reconfigure it. > +\end{itemize} What should a device do if it gets a request that a driver is forbidden from making? Untested corner-cases tend to be a good source of security vulnerabilities, so defining behavior in all cases seems better. > +\subsubsection{Status Information} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information} > + > +Drivers query the device status via \msgref{GET_DEVICE_STATUS} to observe > +progress or detect errors, and they drive the Virtio status transitions via > +\msgref{SET_DEVICE_STATUS}. Writing zero to the status field resets the device, > +invalidating any configuration or virtqueue state. > + > +\drivernormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} > +\begin{itemize} > + \item A driver SHOULD read the device status via \msgref{GET_DEVICE_STATUS} > + when diagnosing errors or determining whether the device is ready to > + move to the next initialization phase. > + \item A driver MUST use \msgref{SET_DEVICE_STATUS} to drive the device through > + the virtio-defined status states and MUST write 0 to request a device > + reset when needed. > +\end{itemize} > + > +\devicenormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} > +\begin{itemize} > + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST > + reset its internal state, invalidate existing configuration and > + virtqueue settings, and present the status field as 0. > + \item A device MUST report its current status accurately via > + \msgref{GET_DEVICE_STATUS}, including whether the FEATURES\_OK bit has > + been accepted or cleared. > +\end{itemize} This is fine if all messages are processed in-order, but that is very bad for performance (see above). As soon as out-of-order message handling becomes possible, a race condition will arise: replies and notifications from before the reset can arrive after the reset. I thknk solving this requires either having a generation or stream number in each message, or delegating reset to the bus layer. (snip) > +\subsubsection{Device Notifications} > +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications} > + > +\msgref{EVENT_CONFIG} and \msgref{EVENT_USED} provide asynchronous notifications > +from the device (or device-side bus) to the driver. The bus may forward these > +messages in-band or synthesize them based on other signals such as interrupts > +or polling. > + > +\devicenormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} > +\begin{itemize} > + \item A device (or device-side bus) MUST send \msgref{EVENT_CONFIG} whenever > + it makes a configuration change or status update that becomes visible to > + the driver. The message MUST include the new configuration generation > + count and MAY include the updated configuration data. I would change the second line to a MUST, to avoid a round trip. > + \item If the configuration data is omitted from \msgref{EVENT_CONFIG}, the > + device SHOULD include the relevant offsets/lengths so the driver can > + re-fetch the data via \msgref{GET_CONFIG}. This makes this clause unnecessary. > + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when > + buffers on a virtqueue have been consumed, unless the device relies on > + an alternative, agreed-upon completion mechanism. Why SHOULD and not MUST? > +\end{itemize} (snip) > +\msgdef{GET_VQUEUE} > + > +\msgref{GET_VQUEUE} returns information about a specific virtqueue, including > +its maximum size, current size, and, if already configured, the descriptor, > +driver, and device area addresses. > + > +\begin{lstlisting} > +struct virtio_msg_get_vqueue_req { > + le32 index; /* virtqueue index */ > +}; > + > +struct virtio_msg_get_vqueue_resp { > + le32 index; /* echoed virtqueue index */ > + le32 max_size; /* maximum queue size */ > + le32 cur_size; /* current size (0 if unconfigured) */ > + le32 reserved; /* must be zero */ > + le64 desc_addr; /* descriptor area address */ > + le64 driver_addr; /* driver area address */ > + le64 device_addr; /* device area address */ > +}; > +\end{lstlisting} > + > +\devicenormative{\paragraph}{GET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} > +\begin{itemize} > + \item A device MUST report accurate maxima and current queue sizes for each > + virtqueue and MUST return zero as the current size if the queue has not > + yet been configured. > +\end{itemize} > + > +\msgdef{SET_VQUEUE} > + > +\msgref{SET_VQUEUE} programs a virtqueue's size and buffer addresses. The driver > +selects a queue index, supplies the desired size (not exceeding the maximum > +reported via \msgref{GET_VQUEUE}), and provides guest-physical addresses for the > +descriptor, driver, and device areas. Is the intention to still require shared memory? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-13 19:09 ` Demi Marie Obenour @ 2026-02-20 8:52 ` Bertrand Marquis 2026-02-21 2:04 ` Demi Marie Obenour 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-20 8:52 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 13 Feb 2026, at 20:09, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 1/26/26 11:32, Bill Mills wrote: >> Add a new transport layer that is based on messages. >> >> This transport layer still uses virtqueues as the other transport layers do >> but implements transport layer operations by sending and receiving messages >> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. >> >> This transport is useful when the device and driver are both implemented in >> software but the trap and emulate operations of virtio-mmio and virtio-pci >> can not be used. > > (snip) > >> +\subsubsection{Message Ordering} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering} >> + >> +Transport messages fall into two classes: requests (which expect responses) and >> +events (which are one-way). Drivers and devices rely on the bus to preserve the >> +relative ordering of request/response pairs for each device number; they do not >> +interpret the \field{token} field directly. > > I expect that this requires all messages for a given device to be > processed sequentially, which is not going to be fast. At the very > least, messages on different virtqueues should be able to be processed > out of order. > > I would have some sort of stream identifier that is made visible to > the bus layer. This would map to a QUIC stream, a single ring buffer, > or something else that ensures in-order delivery. Messages with > different stream identifiers may be processed out of order. We have the token meant to be used to reorder messages or make them correspond to a request. Now we choosed on the first version to enforce the order to prevent having something that would not work in the existing virtio implementation where drivers are not designed to handle this (configuration space access is done in order and is not expected to get answers coming back randomly when reading it). Now I do agree that this could be a place for performance optimization and it would be possible to (as pointed out by Mickael) introduce this as a feature bit so that implementation could support that or not and disable it. This could be handled at bus level or directly on the device side. > >> +\busnormative{\paragraph}{Message Ordering}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} >> +\begin{itemize} >> + \item For each device number, a bus implementation MUST deliver responses to >> + the driver in the same order that it forwarded the corresponding >> + requests to the device. >> + \item A bus implementation MUST ensure that every request forwarded to a >> + device results in exactly one response delivered to the driver (unless >> + the request is defined as an event). >> +\end{itemize} > What is the reason for the device/driver distinction here? > Intuitively, I expect both requests and responses to just be messages > at the bus layer. At the bus layer there are only requests and responses but at the end on the device side you only get requests that you have to answer or event that you send. There is no case of request send by the device side which would require an answer from the driver side. This why we have this here but we could relax it if you think this could be useful. > > (snip) > >> +\devicenormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} >> +\begin{itemize} >> + \item When handling \msgref{GET_DEVICE_FEATURES}, a device MUST return zero >> + for any requested bits that fall outside the number of feature bits it >> + implements. >> + \item After receiving \msgref{SET_DRIVER_FEATURES}, a device MUST update its >> + internal feature mask to match the acknowledged set and MUST reflect >> + acceptance or rejection by leaving the FEATURES\_OK bit set or clearing >> + it in the status returned by \msgref{SET_DEVICE_STATUS}. >> +\end{itemize} > > What should the device do if it doesn't support the feature? This is what we meant to say with acceptance/rejection. If something is not accepted the FEATURES_OK bit will not be set in the message returned. Maybe we need rephrasing to make that clearer ? > >> +\subsubsection{Device Configuration} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration} >> + >> +Drivers use \msgref{GET_CONFIG} to read portions of the configuration space by >> +supplying an offset and length; the device returns the requested data plus the >> +current configuration generation count. Writing is performed via >> +\msgref{SET_CONFIG}, which carries the same offset/length along with the >> +driver's notion of the generation count and the new data. > > I think this should come with a (non-normative) note that the use of > configuration space is discouraged if the data can instead be sent > over a virtqueue. For MMIO and PCI transports, the configuration > space is slightly more convenient as it is accessed synchronously, > but for the message transport, reads are also asynchronous unless > the driver spins or sleeps until getting a reply. Good point, we will add something in v2. > >> +\drivernormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Driver} >> +\begin{itemize} >> + \item A driver MUST ensure that the offset and length in each >> + \msgref{GET_CONFIG} or \msgref{SET_CONFIG} request stay within the >> + configuration size reported by \msgref{GET_DEVICE_INFO}. >> + \item A driver MUST include its most recently observed configuration >> + generation count in a \msgref{SET_CONFIG} request and SHOULD re-read the >> + configuration (via \msgref{GET_CONFIG}) if the request is rejected for a >> + generation mismatch. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Device Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Device} >> +\begin{itemize} >> + \item A device MUST reject a \msgref{SET_CONFIG} request whose generation >> + count does not match its current value and MUST indicate the rejection >> + in the response. >> + \item A device MUST return the current configuration generation count >> + alongside any data returned via \msgref{GET_CONFIG}. >> +\end{itemize} >> + >> +\subsubsection{Virtqueue Configuration} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration} >> + >> +Drivers query virtqueue parameters with \msgref{GET_VQUEUE}, configure them via >> +\msgref{SET_VQUEUE}, and optionally reset them using \msgref{RESET_VQUEUE} (if >> +VIRTIO\_F\_RING\_RESET is negotiated). Each queue is typically configured by >> +reading its maximum size, provisioning descriptor/available/used buffers, and >> +then calling \msgref{SET_VQUEUE} with the chosen size and guest-physical >> +addresses. >> + >> +\drivernormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Driver} >> +\begin{itemize} >> + \item A driver MUST use \msgref{GET_VQUEUE} to determine the maximum queue >> + size and confirm that a queue is inactive before programming it. >> + \item A driver MUST ensure the queue size provided in \msgref{SET_VQUEUE} does >> + not exceed the maximum reported by the device and MUST supply valid >> + descriptor/driver/device addresses before enabling the queue. >> + \item If VIRTIO\_F\_RING\_RESET has been negotiated and a queue requires >> + reinitialization, the driver SHOULD use \msgref{RESET_VQUEUE} before >> + reprogramming it. >> +\end{itemize} > > Why SHOULD and not MUST? Ack > >> +\devicenormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} >> +\begin{itemize} >> + \item A device MUST report accurate maximum queue sizes in \msgref{GET_VQUEUE} >> + and MUST persist the parameters supplied via \msgref{SET_VQUEUE} (size, >> + descriptor, driver, and device addresses). >> + \item When \msgref{RESET_VQUEUE} is issued (and VIRTIO\_F\_RING\_RESET is >> + negotiated), the device MUST quiesce the queue, release any resources >> + associated with it, and allow the driver to reconfigure it. >> +\end{itemize} > > What should a device do if it gets a request that a driver is > forbidden from making? Untested corner-cases tend to be a good > source of security vulnerabilities, so defining behavior in all cases > seems better. This is to be handled at the bus level which should return an error to the transport. FF-A bus defines an error message so that the sender bus when receiving it back for a request can give an error back to the transport. Maybe we should describe this model in non normative way in the spec ? > >> +\subsubsection{Status Information} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information} >> + >> +Drivers query the device status via \msgref{GET_DEVICE_STATUS} to observe >> +progress or detect errors, and they drive the Virtio status transitions via >> +\msgref{SET_DEVICE_STATUS}. Writing zero to the status field resets the device, >> +invalidating any configuration or virtqueue state. >> + >> +\drivernormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} >> +\begin{itemize} >> + \item A driver SHOULD read the device status via \msgref{GET_DEVICE_STATUS} >> + when diagnosing errors or determining whether the device is ready to >> + move to the next initialization phase. >> + \item A driver MUST use \msgref{SET_DEVICE_STATUS} to drive the device through >> + the virtio-defined status states and MUST write 0 to request a device >> + reset when needed. >> +\end{itemize} >> + >> +\devicenormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} >> +\begin{itemize} >> + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST >> + reset its internal state, invalidate existing configuration and >> + virtqueue settings, and present the status field as 0. >> + \item A device MUST report its current status accurately via >> + \msgref{GET_DEVICE_STATUS}, including whether the FEATURES\_OK bit has >> + been accepted or cleared. >> +\end{itemize} > > This is fine if all messages are processed in-order, but that is very > bad for performance (see above). As soon as out-of-order message > handling becomes possible, a race condition will arise: replies and > notifications from before the reset can arrive after the reset. > > I thknk solving this requires either having a generation or stream > number in each message, or delegating reset to the bus layer. The token is meant for that and as said in other mails, we will investigate the solution to introduce a feature bit to enable/disable out-of-order to leave a door open to such optimizations. > > (snip) > >> +\subsubsection{Device Notifications} >> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications} >> + >> +\msgref{EVENT_CONFIG} and \msgref{EVENT_USED} provide asynchronous notifications >> +from the device (or device-side bus) to the driver. The bus may forward these >> +messages in-band or synthesize them based on other signals such as interrupts >> +or polling. >> + >> +\devicenormative{\paragraph}{Device Notifications}{Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} >> +\begin{itemize} >> + \item A device (or device-side bus) MUST send \msgref{EVENT_CONFIG} whenever >> + it makes a configuration change or status update that becomes visible to >> + the driver. The message MUST include the new configuration generation >> + count and MAY include the updated configuration data. > > I would change the second line to a MUST, to avoid a round trip. Ack. > >> + \item If the configuration data is omitted from \msgref{EVENT_CONFIG}, the >> + device SHOULD include the relevant offsets/lengths so the driver can >> + re-fetch the data via \msgref{GET_CONFIG}. > > This makes this clause unnecessary. Ack. > >> + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when >> + buffers on a virtqueue have been consumed, unless the device relies on >> + an alternative, agreed-upon completion mechanism. > > Why SHOULD and not MUST? If polling is used, there might be cases where this will never be sent. Maybe this should be a MUST with something like "unless polling is used" or something like that. > >> +\end{itemize} > > (snip) > >> +\msgdef{GET_VQUEUE} >> + >> +\msgref{GET_VQUEUE} returns information about a specific virtqueue, including >> +its maximum size, current size, and, if already configured, the descriptor, >> +driver, and device area addresses. >> + >> +\begin{lstlisting} >> +struct virtio_msg_get_vqueue_req { >> + le32 index; /* virtqueue index */ >> +}; >> + >> +struct virtio_msg_get_vqueue_resp { >> + le32 index; /* echoed virtqueue index */ >> + le32 max_size; /* maximum queue size */ >> + le32 cur_size; /* current size (0 if unconfigured) */ >> + le32 reserved; /* must be zero */ >> + le64 desc_addr; /* descriptor area address */ >> + le64 driver_addr; /* driver area address */ >> + le64 device_addr; /* device area address */ >> +}; >> +\end{lstlisting} >> + >> +\devicenormative{\paragraph}{GET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} >> +\begin{itemize} >> + \item A device MUST report accurate maxima and current queue sizes for each >> + virtqueue and MUST return zero as the current size if the queue has not >> + yet been configured. >> +\end{itemize} >> + >> +\msgdef{SET_VQUEUE} >> + >> +\msgref{SET_VQUEUE} programs a virtqueue's size and buffer addresses. The driver >> +selects a queue index, supplies the desired size (not exceeding the maximum >> +reported via \msgref{GET_VQUEUE}), and provides guest-physical addresses for the >> +descriptor, driver, and device areas. > > Is the intention to still require shared memory? Shared memory can be used and we have a message for that. To be able to use virtqueues and dma between device and driver memory must be made accessible to the other side as it is the case for other transports. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-20 8:52 ` Bertrand Marquis @ 2026-02-21 2:04 ` Demi Marie Obenour 2026-02-23 7:44 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-21 2:04 UTC (permalink / raw) To: Bertrand Marquis Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 11533 bytes --] On 2/20/26 03:52, Bertrand Marquis wrote: > Hi Demi, > >> On 13 Feb 2026, at 20:09, Demi Marie Obenour <demiobenour@gmail.com> wrote: >> >> On 1/26/26 11:32, Bill Mills wrote: >>> Add a new transport layer that is based on messages. >>> >>> This transport layer still uses virtqueues as the other transport layers do >>> but implements transport layer operations by sending and receiving messages >>> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. >>> >>> This transport is useful when the device and driver are both implemented in >>> software but the trap and emulate operations of virtio-mmio and virtio-pci >>> can not be used. >> >> (snip) >> >>> +\subsubsection{Message Ordering} >>> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering} >>> + >>> +Transport messages fall into two classes: requests (which expect responses) and >>> +events (which are one-way). Drivers and devices rely on the bus to preserve the >>> +relative ordering of request/response pairs for each device number; they do not >>> +interpret the \field{token} field directly. >> >> I expect that this requires all messages for a given device to be >> processed sequentially, which is not going to be fast. At the very >> least, messages on different virtqueues should be able to be processed >> out of order. >> >> I would have some sort of stream identifier that is made visible to >> the bus layer. This would map to a QUIC stream, a single ring buffer, >> or something else that ensures in-order delivery. Messages with >> different stream identifiers may be processed out of order. > > We have the token meant to be used to reorder messages or make them correspond to a request. > > Now we choosed on the first version to enforce the order to prevent having something that would not > work in the existing virtio implementation where drivers are not designed to handle this (configuration > space access is done in order and is not expected to get answers coming back randomly when reading it). > > Now I do agree that this could be a place for performance optimization and it would be possible to (as pointed > out by Mickael) introduce this as a feature bit so that implementation could support that or not and disable it. > This could be handled at bus level or directly on the device side. This was made under the (mistaken) assumption that virtqueue data was also sent over messages. An ordering requirement for that _would_ be a performance problem. I do believe that virtqueue notifications (data available/data consumed) should be allowed to be received out of order. There is no requirement that these be _sent_ in any particular order, so further reordering by the bus isn't going to make things worse. >>> +\busnormative{\paragraph}{Message Ordering}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} >>> +\begin{itemize} >>> + \item For each device number, a bus implementation MUST deliver responses to >>> + the driver in the same order that it forwarded the corresponding >>> + requests to the device. >>> + \item A bus implementation MUST ensure that every request forwarded to a >>> + device results in exactly one response delivered to the driver (unless >>> + the request is defined as an event). >>> +\end{itemize} >> What is the reason for the device/driver distinction here? >> Intuitively, I expect both requests and responses to just be messages >> at the bus layer. > > At the bus layer there are only requests and responses but at the end on the device > side you only get requests that you have to answer or event that you send. There is > no case of request send by the device side which would require an answer from the > driver side. > This why we have this here but we could relax it if you think this could be useful. I think this would be better written as a constraint on devices, rather than as a constraint on buses. Right now, it reads as if the bus is responsible for ensuring messages are replied to. >>> +\devicenormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} >>> +\begin{itemize} >>> + \item When handling \msgref{GET_DEVICE_FEATURES}, a device MUST return zero >>> + for any requested bits that fall outside the number of feature bits it >>> + implements. >>> + \item After receiving \msgref{SET_DRIVER_FEATURES}, a device MUST update its >>> + internal feature mask to match the acknowledged set and MUST reflect >>> + acceptance or rejection by leaving the FEATURES\_OK bit set or clearing >>> + it in the status returned by \msgref{SET_DEVICE_STATUS}. >>> +\end{itemize} >> >> What should the device do if it doesn't support the feature? > > This is what we meant to say with acceptance/rejection. If something is not accepted > the FEATURES_OK bit will not be set in the message returned. > Maybe we need rephrasing to make that clearer ? I agree. >>> +\devicenormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} >>> +\begin{itemize} >>> + \item A device MUST report accurate maximum queue sizes in \msgref{GET_VQUEUE} >>> + and MUST persist the parameters supplied via \msgref{SET_VQUEUE} (size, >>> + descriptor, driver, and device addresses). >>> + \item When \msgref{RESET_VQUEUE} is issued (and VIRTIO\_F\_RING\_RESET is >>> + negotiated), the device MUST quiesce the queue, release any resources >>> + associated with it, and allow the driver to reconfigure it. >>> +\end{itemize} >> >> What should a device do if it gets a request that a driver is >> forbidden from making? Untested corner-cases tend to be a good >> source of security vulnerabilities, so defining behavior in all cases >> seems better. > > This is to be handled at the bus level which should return an error to the transport. > FF-A bus defines an error message so that the sender bus when receiving it back > for a request can give an error back to the transport. > > Maybe we should describe this model in non normative way in the spec ? > > >> >>> +\subsubsection{Status Information} >>> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information} >>> + >>> +Drivers query the device status via \msgref{GET_DEVICE_STATUS} to observe >>> +progress or detect errors, and they drive the Virtio status transitions via >>> +\msgref{SET_DEVICE_STATUS}. Writing zero to the status field resets the device, >>> +invalidating any configuration or virtqueue state. >>> + >>> +\drivernormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} >>> +\begin{itemize} >>> + \item A driver SHOULD read the device status via \msgref{GET_DEVICE_STATUS} >>> + when diagnosing errors or determining whether the device is ready to >>> + move to the next initialization phase. >>> + \item A driver MUST use \msgref{SET_DEVICE_STATUS} to drive the device through >>> + the virtio-defined status states and MUST write 0 to request a device >>> + reset when needed. >>> +\end{itemize} >>> + >>> +\devicenormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} >>> +\begin{itemize} >>> + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST >>> + reset its internal state, invalidate existing configuration and >>> + virtqueue settings, and present the status field as 0. >>> + \item A device MUST report its current status accurately via >>> + \msgref{GET_DEVICE_STATUS}, including whether the FEATURES\_OK bit has >>> + been accepted or cleared. >>> +\end{itemize} >> >> This is fine if all messages are processed in-order, but that is very >> bad for performance (see above). As soon as out-of-order message >> handling becomes possible, a race condition will arise: replies and >> notifications from before the reset can arrive after the reset. >> >> I thknk solving this requires either having a generation or stream >> number in each message, or delegating reset to the bus layer. > > The token is meant for that and as said in other mails, we will investigate the solution to > introduce a feature bit to enable/disable out-of-order to leave a door open to such optimizations. See above. (snip) >>> + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when >>> + buffers on a virtqueue have been consumed, unless the device relies on >>> + an alternative, agreed-upon completion mechanism. >> >> Why SHOULD and not MUST? > > If polling is used, there might be cases where this will never be sent. > Maybe this should be a MUST with something like "unless polling is used" or something > like that. What about this? + \item A device MUST send \msgref{EVENT_USED} to inform the driver when + buffers on a virtqueue have been consumed, unless the device relies on + an alternative, agreed-upon completion mechanism such as polling. >>> +\end{itemize} >> >> (snip) >> >>> +\msgdef{GET_VQUEUE} >>> + >>> +\msgref{GET_VQUEUE} returns information about a specific virtqueue, including >>> +its maximum size, current size, and, if already configured, the descriptor, >>> +driver, and device area addresses. >>> + >>> +\begin{lstlisting} >>> +struct virtio_msg_get_vqueue_req { >>> + le32 index; /* virtqueue index */ >>> +}; >>> + >>> +struct virtio_msg_get_vqueue_resp { >>> + le32 index; /* echoed virtqueue index */ >>> + le32 max_size; /* maximum queue size */ >>> + le32 cur_size; /* current size (0 if unconfigured) */ >>> + le32 reserved; /* must be zero */ >>> + le64 desc_addr; /* descriptor area address */ >>> + le64 driver_addr; /* driver area address */ >>> + le64 device_addr; /* device area address */ >>> +}; >>> +\end{lstlisting} >>> + >>> +\devicenormative{\paragraph}{GET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} >>> +\begin{itemize} >>> + \item A device MUST report accurate maxima and current queue sizes for each >>> + virtqueue and MUST return zero as the current size if the queue has not >>> + yet been configured. >>> +\end{itemize} >>> + >>> +\msgdef{SET_VQUEUE} >>> + >>> +\msgref{SET_VQUEUE} programs a virtqueue's size and buffer addresses. The driver >>> +selects a queue index, supplies the desired size (not exceeding the maximum >>> +reported via \msgref{GET_VQUEUE}), and provides guest-physical addresses for the >>> +descriptor, driver, and device areas. >> >> Is the intention to still require shared memory? > > Shared memory can be used and we have a message for that. > > To be able to use virtqueues and dma between device and driver memory must be made accessible > to the other side as it is the case for other transports. Should these be I/O virtual addresses instead of guest physical addresses? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-21 2:04 ` Demi Marie Obenour @ 2026-02-23 7:44 ` Bertrand Marquis 0 siblings, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-23 7:44 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 21 Feb 2026, at 03:04, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 2/20/26 03:52, Bertrand Marquis wrote: >> Hi Demi, >> >>> On 13 Feb 2026, at 20:09, Demi Marie Obenour <demiobenour@gmail.com> wrote: >>> >>> On 1/26/26 11:32, Bill Mills wrote: >>>> Add a new transport layer that is based on messages. >>>> >>>> This transport layer still uses virtqueues as the other transport layers do >>>> but implements transport layer operations by sending and receiving messages >>>> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. >>>> >>>> This transport is useful when the device and driver are both implemented in >>>> software but the trap and emulate operations of virtio-mmio and virtio-pci >>>> can not be used. >>> >>> (snip) >>> >>>> +\subsubsection{Message Ordering} >>>> +\label{sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering} >>>> + >>>> +Transport messages fall into two classes: requests (which expect responses) and >>>> +events (which are one-way). Drivers and devices rely on the bus to preserve the >>>> +relative ordering of request/response pairs for each device number; they do not >>>> +interpret the \field{token} field directly. >>> >>> I expect that this requires all messages for a given device to be >>> processed sequentially, which is not going to be fast. At the very >>> least, messages on different virtqueues should be able to be processed >>> out of order. >>> >>> I would have some sort of stream identifier that is made visible to >>> the bus layer. This would map to a QUIC stream, a single ring buffer, >>> or something else that ensures in-order delivery. Messages with >>> different stream identifiers may be processed out of order. >> >> We have the token meant to be used to reorder messages or make them correspond to a request. >> >> Now we choosed on the first version to enforce the order to prevent having something that would not >> work in the existing virtio implementation where drivers are not designed to handle this (configuration >> space access is done in order and is not expected to get answers coming back randomly when reading it). >> >> Now I do agree that this could be a place for performance optimization and it would be possible to (as pointed >> out by Mickael) introduce this as a feature bit so that implementation could support that or not and disable it. >> This could be handled at bus level or directly on the device side. > > This was made under the (mistaken) assumption that virtqueue data was > also sent over messages. An ordering requirement for that _would_ > be a performance problem. > > I do believe that virtqueue notifications (data available/data > consumed) should be allowed to be received out of order. There is no > requirement that these be _sent_ in any particular order, so further > reordering by the bus isn't going to make things worse. Agree. I will also find a place in the transport to clear up that virtqueue data is not transmitted through messages but normal DMA/shared memory to make things clear for the reader. > >>>> +\busnormative{\paragraph}{Message Ordering}{Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} >>>> +\begin{itemize} >>>> + \item For each device number, a bus implementation MUST deliver responses to >>>> + the driver in the same order that it forwarded the corresponding >>>> + requests to the device. >>>> + \item A bus implementation MUST ensure that every request forwarded to a >>>> + device results in exactly one response delivered to the driver (unless >>>> + the request is defined as an event). >>>> +\end{itemize} >>> What is the reason for the device/driver distinction here? >>> Intuitively, I expect both requests and responses to just be messages >>> at the bus layer. >> >> At the bus layer there are only requests and responses but at the end on the device >> side you only get requests that you have to answer or event that you send. There is >> no case of request send by the device side which would require an answer from the >> driver side. >> This why we have this here but we could relax it if you think this could be useful. > > I think this would be better written as a constraint on devices, rather than as > a constraint on buses. Right now, it reads as if the bus is responsible for > ensuring messages are replied to. Makes sense, I will rephrase from device/driver point of view in the next version. > >>>> +\devicenormative{\paragraph}{Feature Negotiation}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} >>>> +\begin{itemize} >>>> + \item When handling \msgref{GET_DEVICE_FEATURES}, a device MUST return zero >>>> + for any requested bits that fall outside the number of feature bits it >>>> + implements. >>>> + \item After receiving \msgref{SET_DRIVER_FEATURES}, a device MUST update its >>>> + internal feature mask to match the acknowledged set and MUST reflect >>>> + acceptance or rejection by leaving the FEATURES\_OK bit set or clearing >>>> + it in the status returned by \msgref{SET_DEVICE_STATUS}. >>>> +\end{itemize} >>> >>> What should the device do if it doesn't support the feature? >> >> This is what we meant to say with acceptance/rejection. If something is not accepted >> the FEATURES_OK bit will not be set in the message returned. >> Maybe we need rephrasing to make that clearer ? > > I agree. > >>>> +\devicenormative{\paragraph}{Virtqueue Configuration}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} >>>> +\begin{itemize} >>>> + \item A device MUST report accurate maximum queue sizes in \msgref{GET_VQUEUE} >>>> + and MUST persist the parameters supplied via \msgref{SET_VQUEUE} (size, >>>> + descriptor, driver, and device addresses). >>>> + \item When \msgref{RESET_VQUEUE} is issued (and VIRTIO\_F\_RING\_RESET is >>>> + negotiated), the device MUST quiesce the queue, release any resources >>>> + associated with it, and allow the driver to reconfigure it. >>>> +\end{itemize} >>> >>> What should a device do if it gets a request that a driver is >>> forbidden from making? Untested corner-cases tend to be a good >>> source of security vulnerabilities, so defining behavior in all cases >>> seems better. >> >> This is to be handled at the bus level which should return an error to the transport. >> FF-A bus defines an error message so that the sender bus when receiving it back >> for a request can give an error back to the transport. >> >> Maybe we should describe this model in non normative way in the spec ? >> >> >>> >>>> +\subsubsection{Status Information} >>>> +\label{sec:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information} >>>> + >>>> +Drivers query the device status via \msgref{GET_DEVICE_STATUS} to observe >>>> +progress or detect errors, and they drive the Virtio status transitions via >>>> +\msgref{SET_DEVICE_STATUS}. Writing zero to the status field resets the device, >>>> +invalidating any configuration or virtqueue state. >>>> + >>>> +\drivernormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} >>>> +\begin{itemize} >>>> + \item A driver SHOULD read the device status via \msgref{GET_DEVICE_STATUS} >>>> + when diagnosing errors or determining whether the device is ready to >>>> + move to the next initialization phase. >>>> + \item A driver MUST use \msgref{SET_DEVICE_STATUS} to drive the device through >>>> + the virtio-defined status states and MUST write 0 to request a device >>>> + reset when needed. >>>> +\end{itemize} >>>> + >>>> +\devicenormative{\paragraph}{Status Handling}{Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} >>>> +\begin{itemize} >>>> + \item Upon receiving a \msgref{SET_DEVICE_STATUS} write of 0, a device MUST >>>> + reset its internal state, invalidate existing configuration and >>>> + virtqueue settings, and present the status field as 0. >>>> + \item A device MUST report its current status accurately via >>>> + \msgref{GET_DEVICE_STATUS}, including whether the FEATURES\_OK bit has >>>> + been accepted or cleared. >>>> +\end{itemize} >>> >>> This is fine if all messages are processed in-order, but that is very >>> bad for performance (see above). As soon as out-of-order message >>> handling becomes possible, a race condition will arise: replies and >>> notifications from before the reset can arrive after the reset. >>> >>> I thknk solving this requires either having a generation or stream >>> number in each message, or delegating reset to the bus layer. >> >> The token is meant for that and as said in other mails, we will investigate the solution to >> introduce a feature bit to enable/disable out-of-order to leave a door open to such optimizations. > > See above. Having a feature bit for this might be a good way to be future proof anyway even though i am not really sure the performance improvement would be noticeable when virtqueues are used as the main data patch. I will investigate that as in my tests there could be corner cases which are tricky, for example: - activate something using a config bit/value - event resulting from this activation coming back before the write to config is acknowledged This might be a case that a driver has to handle anyway as it could happen in mmio or pci but this could be more common as messages are asynchronous so an event could come before the write is acknowledged. > > (snip) > >>>> + \item A device SHOULD send \msgref{EVENT_USED} to inform the driver when >>>> + buffers on a virtqueue have been consumed, unless the device relies on >>>> + an alternative, agreed-upon completion mechanism. >>> >>> Why SHOULD and not MUST? >> >> If polling is used, there might be cases where this will never be sent. >> Maybe this should be a MUST with something like "unless polling is used" or something >> like that. > > What about this? > > + \item A device MUST send \msgref{EVENT_USED} to inform the driver when > + buffers on a virtqueue have been consumed, unless the device relies on > + an alternative, agreed-upon completion mechanism such as polling. Sound good, will use that. > >>>> +\end{itemize} >>> >>> (snip) >>> >>>> +\msgdef{GET_VQUEUE} >>>> + >>>> +\msgref{GET_VQUEUE} returns information about a specific virtqueue, including >>>> +its maximum size, current size, and, if already configured, the descriptor, >>>> +driver, and device area addresses. >>>> + >>>> +\begin{lstlisting} >>>> +struct virtio_msg_get_vqueue_req { >>>> + le32 index; /* virtqueue index */ >>>> +}; >>>> + >>>> +struct virtio_msg_get_vqueue_resp { >>>> + le32 index; /* echoed virtqueue index */ >>>> + le32 max_size; /* maximum queue size */ >>>> + le32 cur_size; /* current size (0 if unconfigured) */ >>>> + le32 reserved; /* must be zero */ >>>> + le64 desc_addr; /* descriptor area address */ >>>> + le64 driver_addr; /* driver area address */ >>>> + le64 device_addr; /* device area address */ >>>> +}; >>>> +\end{lstlisting} >>>> + >>>> +\devicenormative{\paragraph}{GET\_VQUEUE}{Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} >>>> +\begin{itemize} >>>> + \item A device MUST report accurate maxima and current queue sizes for each >>>> + virtqueue and MUST return zero as the current size if the queue has not >>>> + yet been configured. >>>> +\end{itemize} >>>> + >>>> +\msgdef{SET_VQUEUE} >>>> + >>>> +\msgref{SET_VQUEUE} programs a virtqueue's size and buffer addresses. The driver >>>> +selects a queue index, supplies the desired size (not exceeding the maximum >>>> +reported via \msgref{GET_VQUEUE}), and provides guest-physical addresses for the >>>> +descriptor, driver, and device areas. >>> >>> Is the intention to still require shared memory? >> >> Shared memory can be used and we have a message for that. >> >> To be able to use virtqueues and dma between device and driver memory must be made accessible >> to the other side as it is the case for other transports. > > Should these be I/O virtual addresses instead of guest physical addresses? MMIO is using the term "physical address" and we should keep using the same terminology as I/O virtual might be wrongly interpreted. I will remove the guest part because this one is misleading. I have been using term "bus address" in the ffa bus specification to improve the abstraction as in it the "physical address" has nothing to do with any physical address but is a common way to refer to some memory so that it can be mapped on both side and virtual address be retrieved. But here i think being coherent with other transports might be a better idea. What do you think ? Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-01-26 16:32 ` [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer Bill Mills 2026-02-06 16:28 ` Peter Hilber 2026-02-13 19:09 ` Demi Marie Obenour @ 2026-02-24 15:41 ` Demi Marie Obenour 2026-02-24 16:14 ` Bertrand Marquis 2026-02-24 17:36 ` Edgar E. Iglesias 2026-02-24 17:14 ` Demi Marie Obenour 2026-02-25 12:36 ` Demi Marie Obenour 4 siblings, 2 replies; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-24 15:41 UTC (permalink / raw) To: Bill Mills, virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine, Alyssa Ross [-- Attachment #1.1.1: Type: text/plain, Size: 5160 bytes --] On 1/26/26 11:32, Bill Mills wrote: > Add a new transport layer that is based on messages. > > This transport layer still uses virtqueues as the other transport layers do > but implements transport layer operations by sending and receiving messages > instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. > > This transport is useful when the device and driver are both implemented in > software but the trap and emulate operations of virtio-mmio and virtio-pci > can not be used. Is it possible to map a virtio-mmio or virtio-pci device to virtio-msg? For instance, can a VMM present a virtio-mmio or virtio-pci device to its VM, while using virtio-msg to communicate with its backend? > This transport is intended to be used in many situations, including: > * between a host processor and its co-processors > * between two different systems (not SMP) connected via PCIe > * between normal and secure worlds > * host to vm How does this map to the vhost-user protocol [1]? Vhost-user is a very common way to implement devices and it would be nice for virtio-msg to cleanly map to it. Requiring a separate AF_UNIX transport could be quite disruptive. [1]: https://www.qemu.org/docs/master/interop/vhost-user.html > * vm to vm This requires a method for VMs to implement devices, not just drivers. For Xen and FF-A this is easy, provided that there is an out-of-band means to enumerate which devices should be implemented. However, these mechanisms are not hypervisor-agnostic. It would be much better to have a standard virtio device that backends could use to implement virtio-msg devices. There have been multiple attempts at a virtio vhost-user device backend [2], but it was never widely deployed. [2]: https://stefanha.github.io/virtio/vhost-user-slave.html > Signed-off-by: Bill Mills <bill.mills@linaro.org> > Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> > Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> > Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com> > --- > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 1640 insertions(+) > create mode 100644 transport-msg.tex > > diff --git a/transport-msg.tex b/transport-msg.tex > new file mode 100644 > index 0000000..d4e31d7 > --- /dev/null > +++ b/transport-msg.tex > @@ -0,0 +1,1640 @@ > +\section{Virtio Over Messages}\label{sec:Virtio Transport Options / Virtio Over Messages} > + > +\newcommand{\conceptref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / #1]{#1}} > +\newcommand{\msgref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1]{VIRTIO_MSG_#1}} > +\newcommand{\busref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1]{BUS_MSG_#1}} > +\newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1}} > +\newcommand{\busdef}[1]{\subsubsection{BUS_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1}} > + > +This section defines \textbf{virtio-msg}, a transport mechanism that encapsulates > +virtio operations as discrete message exchanges rather than relying on PCI or > +memory-mapped I/O regions. It separates bus-level functionality (e.g., device > +enumeration, hotplug events) from device-specific operations (e.g., feature > +negotiation, virtqueue setup), ensuring that a single, generic transport layer > +can be reused across multiple bus implementations. > + > +virtio-msg addresses several key objectives: > + > +\begin{itemize} > + \item \textbf{Support multiple bus implementations:} > + Systems can rely on various communication methods such as hypercalls, local > + IPC, network channels, or device trees for enumerating devices. virtio-msg > + defines a common transport interface suitable for any of these mechanisms. > + > + \item \textbf{Reduce per-bus complexity:} > + Buses can implement a fully message-based workflow (including optional > + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE}) > + or they can discover and manage devices through > + alternative means such as platform firmware data. In either case, they > + forward transport messages to and from each device. > + > + \item \textbf{Preserve virtio semantics:} > + The transport leverages standard virtio concepts (features, configuration > + space, virtqueues), so existing virtio drivers and device implementations can > + integrate smoothly once a device is discovered and registered. > +\end{itemize} I see that there is support for device enumeration by drivers. What about driver enumeration by devices? An FF-A endpoint or Xen VM needs to know which devices it should implement. In embedded systems, this can be obtained from the device tree or ACPI. However, dynamic systems need to create and destroy devices at runtime. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-24 15:41 ` Demi Marie Obenour @ 2026-02-24 16:14 ` Bertrand Marquis 2026-02-24 17:36 ` Edgar E. Iglesias 1 sibling, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-24 16:14 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine, Alyssa Ross Hi Demi, > On 24 Feb 2026, at 16:41, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 1/26/26 11:32, Bill Mills wrote: >> Add a new transport layer that is based on messages. >> >> This transport layer still uses virtqueues as the other transport layers do >> but implements transport layer operations by sending and receiving messages >> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. >> >> This transport is useful when the device and driver are both implemented in >> software but the trap and emulate operations of virtio-mmio and virtio-pci >> can not be used. > > Is it possible to map a virtio-mmio or virtio-pci device to virtio-msg? > For instance, can a VMM present a virtio-mmio or virtio-pci device > to its VM, while using virtio-msg to communicate with its backend? I am not sure to understand what you mean here. Right now a VMM would present a virtio device which will behave the same on mmio, pci or virtio-msg. In our transport implementation in Linux, we did not change anything in the virtio drivers. So the transport used is transparent even to the virtio driver on top. > >> This transport is intended to be used in many situations, including: >> * between a host processor and its co-processors >> * between two different systems (not SMP) connected via PCIe >> * between normal and secure worlds >> * host to vm > > How does this map to the vhost-user protocol [1]? Vhost-user is a very > common way to implement devices and it would be nice for virtio-msg > to cleanly map to it. Requiring a separate AF_UNIX transport could > be quite disruptive. > > [1]: https://www.qemu.org/docs/master/interop/vhost-user.html This is something i will need to look at to provide an answer but vsock works for sure so this should to but I will have to confirm. > >> * vm to vm > > This requires a method for VMs to implement devices, not just drivers. > For Xen and FF-A this is easy, provided that there is an out-of-band > means to enumerate which devices should be implemented. However, > these mechanisms are not hypervisor-agnostic. For FF-A the way to enumerate devices is hypervisor agnostic as we have extra messages in the bus to do so. I started to research how to do a xen implementation (using only grant tables and xen events) which will use a FIFO in one or two pages shared using grant tables to transfer the messages and enumeration should work the same way as with FF-A. > > It would be much better to have a standard virtio device that backends > could use to implement virtio-msg devices. There have been multiple > attempts at a virtio vhost-user device backend [2], but it was never > widely deployed. What I am looking at right now is the following in linux: - virtio-msg bus backend implemented in the kernel with a standard virtio-msg bus interface to user - virtio-msg transport backend implemented in Qemu (or any other VMM) communicating with bus using a kernel interface This makes Qemu bus agnostic and allows to have more implementations of different busses using kernel drivers without changing Qemu. Having something of a vhost-user backend seems feasible but would need some work definitely. > > [2]: https://uk01.z.antigena.com/l/RPtFHkpliiy7NRgNtyaO30yBQpiAnIuLVyzAK2QZ0fGEIjSJVGx5Mu3MwXXt-LgQn2ZVpDwK1wfUHdgKxzrauRK4sFfiVoW3xPL5PablM52jV-ngF4DwusMzPUNSF6WQ1Ec3WMxworPUgzNgc1gDTMsD1H17lnI6v1cOcD1m5vAuK4eS7Ma8l5f5HhciCHG0H4ibv8wtWquW2fvs6m > >> Signed-off-by: Bill Mills <bill.mills@linaro.org> >> Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> >> Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> >> Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com> >> --- >> transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 1640 insertions(+) >> create mode 100644 transport-msg.tex >> >> diff --git a/transport-msg.tex b/transport-msg.tex >> new file mode 100644 >> index 0000000..d4e31d7 >> --- /dev/null >> +++ b/transport-msg.tex >> @@ -0,0 +1,1640 @@ >> +\section{Virtio Over Messages}\label{sec:Virtio Transport Options / Virtio Over Messages} >> + >> +\newcommand{\conceptref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / #1]{#1}} >> +\newcommand{\msgref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1]{VIRTIO_MSG_#1}} >> +\newcommand{\busref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1]{BUS_MSG_#1}} >> +\newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1}} >> +\newcommand{\busdef}[1]{\subsubsection{BUS_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1}} >> + >> +This section defines \textbf{virtio-msg}, a transport mechanism that encapsulates >> +virtio operations as discrete message exchanges rather than relying on PCI or >> +memory-mapped I/O regions. It separates bus-level functionality (e.g., device >> +enumeration, hotplug events) from device-specific operations (e.g., feature >> +negotiation, virtqueue setup), ensuring that a single, generic transport layer >> +can be reused across multiple bus implementations. >> + >> +virtio-msg addresses several key objectives: >> + >> +\begin{itemize} >> + \item \textbf{Support multiple bus implementations:} >> + Systems can rely on various communication methods such as hypercalls, local >> + IPC, network channels, or device trees for enumerating devices. virtio-msg >> + defines a common transport interface suitable for any of these mechanisms. >> + >> + \item \textbf{Reduce per-bus complexity:} >> + Buses can implement a fully message-based workflow (including optional >> + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE}) >> + or they can discover and manage devices through >> + alternative means such as platform firmware data. In either case, they >> + forward transport messages to and from each device. >> + >> + \item \textbf{Preserve virtio semantics:} >> + The transport leverages standard virtio concepts (features, configuration >> + space, virtqueues), so existing virtio drivers and device implementations can >> + integrate smoothly once a device is discovered and registered. >> +\end{itemize} > > I see that there is support for device enumeration by drivers. > What about driver enumeration by devices? An FF-A endpoint or Xen VM > needs to know which devices it should implement. In embedded systems, > this can be obtained from the device tree or ACPI. However, dynamic > systems need to create and destroy devices at runtime. For now on this would work as it works right now for xen pv devices or virtio devices, you will start a qemu telling it which devices to create and those will be detected using the normal device enumeration message (or hotplugged if created after boot). This is why we add the message to signal that a device was added or removed. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers) > <OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-24 15:41 ` Demi Marie Obenour 2026-02-24 16:14 ` Bertrand Marquis @ 2026-02-24 17:36 ` Edgar E. Iglesias 1 sibling, 0 replies; 105+ messages in thread From: Edgar E. Iglesias @ 2026-02-24 17:36 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bill Mills, virtio-comment, Bertrand Marquis, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine, Alyssa Ross On Tue, Feb 24, 2026 at 10:41:59AM -0500, Demi Marie Obenour wrote: > On 1/26/26 11:32, Bill Mills wrote: > > Add a new transport layer that is based on messages. > > > > This transport layer still uses virtqueues as the other transport layers do > > but implements transport layer operations by sending and receiving messages > > instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. > > > > This transport is useful when the device and driver are both implemented in > > software but the trap and emulate operations of virtio-mmio and virtio-pci > > can not be used. > > Is it possible to map a virtio-mmio or virtio-pci device to virtio-msg? > For instance, can a VMM present a virtio-mmio or virtio-pci device > to its VM, while using virtio-msg to communicate with its backend? Hi Demi, Yes, that is possible. One of the first implementations of virtio-msg was using a generic proxy in QEMU that translated virtio-mmio and virtio-pci to virtio-msg, at the time there was no virtio-msg Linux driver so this served as a stepping stone. I haven't kept the proxy up to date though. https://github.com/edgarigl/qemu/blob/edgar/virtio-msg/hw/virtio/virtio-msg-proxy-driver.c We also have a proxy for Xen that translates a slightly modified version of virtio-mmio to virtio-msg. https://github.com/Xilinx/xen/tree/xlnx_rebase_4.20/xen/common/virtio Best regards, Edgar > > > This transport is intended to be used in many situations, including: > > * between a host processor and its co-processors > > * between two different systems (not SMP) connected via PCIe > > * between normal and secure worlds > > * host to vm > > How does this map to the vhost-user protocol [1]? Vhost-user is a very > common way to implement devices and it would be nice for virtio-msg > to cleanly map to it. Requiring a separate AF_UNIX transport could > be quite disruptive. > > [1]: https://www.qemu.org/docs/master/interop/vhost-user.html > > > * vm to vm > > This requires a method for VMs to implement devices, not just drivers. > For Xen and FF-A this is easy, provided that there is an out-of-band > means to enumerate which devices should be implemented. However, > these mechanisms are not hypervisor-agnostic. > > It would be much better to have a standard virtio device that backends > could use to implement virtio-msg devices. There have been multiple > attempts at a virtio vhost-user device backend [2], but it was never > widely deployed. > > [2]: https://stefanha.github.io/virtio/vhost-user-slave.html > > > Signed-off-by: Bill Mills <bill.mills@linaro.org> > > Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> > > Signed-off-by: Edgar E. Iglesias <edgar.iglesias@amd.com> > > Signed-off-by: Arnaud Pouliquen <arnaud.pouliquen@foss.st.com> > > --- > > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 1640 insertions(+) > > create mode 100644 transport-msg.tex > > > > diff --git a/transport-msg.tex b/transport-msg.tex > > new file mode 100644 > > index 0000000..d4e31d7 > > --- /dev/null > > +++ b/transport-msg.tex > > @@ -0,0 +1,1640 @@ > > +\section{Virtio Over Messages}\label{sec:Virtio Transport Options / Virtio Over Messages} > > + > > +\newcommand{\conceptref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Basic Concepts / #1]{#1}} > > +\newcommand{\msgref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1]{VIRTIO_MSG_#1}} > > +\newcommand{\busref}[1]{\hyperref[sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1]{BUS_MSG_#1}} > > +\newcommand{\msgdef}[1]{\subsubsection{VIRTIO_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_#1}} > > +\newcommand{\busdef}[1]{\subsubsection{BUS_MSG_#1}\label{sec:Virtio Transport Options / Virtio Over Messages / Bus Messages / BUS_MSG_#1}} > > + > > +This section defines \textbf{virtio-msg}, a transport mechanism that encapsulates > > +virtio operations as discrete message exchanges rather than relying on PCI or > > +memory-mapped I/O regions. It separates bus-level functionality (e.g., device > > +enumeration, hotplug events) from device-specific operations (e.g., feature > > +negotiation, virtqueue setup), ensuring that a single, generic transport layer > > +can be reused across multiple bus implementations. > > + > > +virtio-msg addresses several key objectives: > > + > > +\begin{itemize} > > + \item \textbf{Support multiple bus implementations:} > > + Systems can rely on various communication methods such as hypercalls, local > > + IPC, network channels, or device trees for enumerating devices. virtio-msg > > + defines a common transport interface suitable for any of these mechanisms. > > + > > + \item \textbf{Reduce per-bus complexity:} > > + Buses can implement a fully message-based workflow (including optional > > + enumeration via \busref{GET_DEVICES} and hotplug via \busref{EVENT_DEVICE}) > > + or they can discover and manage devices through > > + alternative means such as platform firmware data. In either case, they > > + forward transport messages to and from each device. > > + > > + \item \textbf{Preserve virtio semantics:} > > + The transport leverages standard virtio concepts (features, configuration > > + space, virtqueues), so existing virtio drivers and device implementations can > > + integrate smoothly once a device is discovered and registered. > > +\end{itemize} > > I see that there is support for device enumeration by drivers. > What about driver enumeration by devices? An FF-A endpoint or Xen VM > needs to know which devices it should implement. In embedded systems, > this can be obtained from the device tree or ACPI. However, dynamic > systems need to create and destroy devices at runtime. > -- > Sincerely, > Demi Marie Obenour (she/her/hers) ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-01-26 16:32 ` [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer Bill Mills ` (2 preceding siblings ...) 2026-02-24 15:41 ` Demi Marie Obenour @ 2026-02-24 17:14 ` Demi Marie Obenour 2026-02-24 17:20 ` Bertrand Marquis 2026-02-25 12:36 ` Demi Marie Obenour 4 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-24 17:14 UTC (permalink / raw) To: Bill Mills, virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 1727 bytes --] On 1/26/26 11:32, Bill Mills wrote: > Add a new transport layer that is based on messages. > > This transport layer still uses virtqueues as the other transport layers do > but implements transport layer operations by sending and receiving messages > instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. > > This transport is useful when the device and driver are both implemented in > software but the trap and emulate operations of virtio-mmio and virtio-pci > can not be used. > > This transport is intended to be used in many situations, including: > * between a host processor and its co-processors > * between two different systems (not SMP) connected via PCIe > * between normal and secure worlds > * host to vm > * vm to vm (snip) > +As described in Sections > +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} > +and > +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications}, > +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and > +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band > +mechanisms. Sending these notifications out of band should almost always be more efficient. Under Xen, one should use per-virtqueue, per-direction event channels. KVM has irqfd and ioeventfd, and the loopback transport can use per-virtqueue eventfds or io_uring queues. I would be quite surprised if FF-A and PCI did not have similar mechanisms that could be used. I recommend removing these from the spec and expecting buses to provide their own notification mechanism. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-24 17:14 ` Demi Marie Obenour @ 2026-02-24 17:20 ` Bertrand Marquis 2026-02-24 17:46 ` Demi Marie Obenour 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-24 17:20 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 24 Feb 2026, at 18:14, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 1/26/26 11:32, Bill Mills wrote: >> Add a new transport layer that is based on messages. >> >> This transport layer still uses virtqueues as the other transport layers do >> but implements transport layer operations by sending and receiving messages >> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. >> >> This transport is useful when the device and driver are both implemented in >> software but the trap and emulate operations of virtio-mmio and virtio-pci >> can not be used. >> >> This transport is intended to be used in many situations, including: >> * between a host processor and its co-processors >> * between two different systems (not SMP) connected via PCIe >> * between normal and secure worlds >> * host to vm >> * vm to vm > > (snip) > >> +As described in Sections >> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} >> +and >> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications}, >> +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and >> +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band >> +mechanisms. > > Sending these notifications out of band should almost always be more > efficient. Under Xen, one should use per-virtqueue, per-direction > event channels. KVM has irqfd and ioeventfd, and the loopback > transport can use per-virtqueue eventfds or io_uring queues. I would > be quite surprised if FF-A and PCI did not have similar mechanisms > that could be used. > > I recommend removing these from the spec and expecting buses to > provide their own notification mechanism. Busses can have more efficient way and can map that other real interrupts or xen events but the bus has to create a fake message back to the transport to keep the interface coherent. In ffa case we do use notifications but we still use messages as those can be more efficient in some cases (indirect messages) and we have a limited number of notifications. In ffa we have a also a transfer system using a FIFO which can hold several events at a time preventing several ping/pong. We do let the door open so that a bus can choose how to do this but we keep the message (even fake) between the bus and the transport to have a transport independent and handling the OS specific way of signaling the drivers. Does that answer your question ? Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-24 17:20 ` Bertrand Marquis @ 2026-02-24 17:46 ` Demi Marie Obenour 2026-02-25 7:26 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-24 17:46 UTC (permalink / raw) To: Bertrand Marquis Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 3527 bytes --] On 2/24/26 12:20, Bertrand Marquis wrote: > Hi Demi, > >> On 24 Feb 2026, at 18:14, Demi Marie Obenour <demiobenour@gmail.com> wrote: >> >> On 1/26/26 11:32, Bill Mills wrote: >>> Add a new transport layer that is based on messages. >>> >>> This transport layer still uses virtqueues as the other transport layers do >>> but implements transport layer operations by sending and receiving messages >>> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. >>> >>> This transport is useful when the device and driver are both implemented in >>> software but the trap and emulate operations of virtio-mmio and virtio-pci >>> can not be used. >>> >>> This transport is intended to be used in many situations, including: >>> * between a host processor and its co-processors >>> * between two different systems (not SMP) connected via PCIe >>> * between normal and secure worlds >>> * host to vm >>> * vm to vm >> >> (snip) >> >>> +As described in Sections >>> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} >>> +and >>> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications}, >>> +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and >>> +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band >>> +mechanisms. >> >> Sending these notifications out of band should almost always be more >> efficient. Under Xen, one should use per-virtqueue, per-direction >> event channels. KVM has irqfd and ioeventfd, and the loopback >> transport can use per-virtqueue eventfds or io_uring queues. I would >> be quite surprised if FF-A and PCI did not have similar mechanisms >> that could be used. >> >> I recommend removing these from the spec and expecting buses to >> provide their own notification mechanism. > > Busses can have more efficient way and can map that other real interrupts or xen events > but the bus has to create a fake message back to the transport to keep the interface > coherent. > > In ffa case we do use notifications but we still use messages as those can be more > efficient in some cases (indirect messages) and we have a limited number of notifications. > In ffa we have a also a transfer system using a FIFO which can hold several events at a time > preventing several ping/pong. > > We do let the door open so that a bus can choose how to do this but we keep the message > (even fake) between the bus and the transport to have a transport independent and handling > the OS specific way of signaling the drivers. > > Does that answer your question ? I expect that most implementations will have a single queue in each direction for bus messages. This means that sending bus messages will require taking a lock. Sending virtqueue notifications out of band, without a corresponding bus message, will avoid taking the lock and therefore significantly improve scalability on multi-core systems. For instance, a Xen transport can use per-virtqueue event channels and a KVM or MSHV transport can use per-virtqueue irqfds/ioeventfds. I strongly recommend language like the following: EVENT_CONFIG, EVENT_USED, and EVENT_AVAIL MAY be sent in-band or out-of-band. The specification of a given bus MUST specify which is used. If a bus specifies that out-of-band notifications are to be used, it MUST document how to send them. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-24 17:46 ` Demi Marie Obenour @ 2026-02-25 7:26 ` Bertrand Marquis 0 siblings, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 7:26 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 24 Feb 2026, at 18:46, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 2/24/26 12:20, Bertrand Marquis wrote: >> Hi Demi, >> >>> On 24 Feb 2026, at 18:14, Demi Marie Obenour <demiobenour@gmail.com> wrote: >>> >>> On 1/26/26 11:32, Bill Mills wrote: >>>> Add a new transport layer that is based on messages. >>>> >>>> This transport layer still uses virtqueues as the other transport layers do >>>> but implements transport layer operations by sending and receiving messages >>>> instead of the "MMR" reads and writes used in virtio-mmio and virtio-pci. >>>> >>>> This transport is useful when the device and driver are both implemented in >>>> software but the trap and emulate operations of virtio-mmio and virtio-pci >>>> can not be used. >>>> >>>> This transport is intended to be used in many situations, including: >>>> * between a host processor and its co-processors >>>> * between two different systems (not SMP) connected via PCIe >>>> * between normal and secure worlds >>>> * host to vm >>>> * vm to vm >>> >>> (snip) >>> >>>> +As described in Sections >>>> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications} >>>> +and >>>> +\ref{sec:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications}, >>>> +runtime notifications (\msgref{EVENT_AVAIL}, \msgref{EVENT_USED}, and >>>> +\msgref{EVENT_CONFIG}) may be delivered in-band or via equivalent out-of-band >>>> +mechanisms. >>> >>> Sending these notifications out of band should almost always be more >>> efficient. Under Xen, one should use per-virtqueue, per-direction >>> event channels. KVM has irqfd and ioeventfd, and the loopback >>> transport can use per-virtqueue eventfds or io_uring queues. I would >>> be quite surprised if FF-A and PCI did not have similar mechanisms >>> that could be used. >>> >>> I recommend removing these from the spec and expecting buses to >>> provide their own notification mechanism. >> >> Busses can have more efficient way and can map that other real interrupts or xen events >> but the bus has to create a fake message back to the transport to keep the interface >> coherent. >> >> In ffa case we do use notifications but we still use messages as those can be more >> efficient in some cases (indirect messages) and we have a limited number of notifications. >> In ffa we have a also a transfer system using a FIFO which can hold several events at a time >> preventing several ping/pong. >> >> We do let the door open so that a bus can choose how to do this but we keep the message >> (even fake) between the bus and the transport to have a transport independent and handling >> the OS specific way of signaling the drivers. >> >> Does that answer your question ? > > I expect that most implementations will have a single queue in each > direction for bus messages. This means that sending bus messages will > require taking a lock. Sending virtqueue notifications out of band, > without a corresponding bus message, will avoid taking the lock and > therefore significantly improve scalability on multi-core systems. FF-A implementation using FIFOs for transfer has been designed to use 2 lock-less queues (one in each direction) which should prevent those kind of issues. In any case the transport does not limit what you can do in any way. In his design for system to system, Bill has designed something where messages are transfered using queues but events are transferred using out of band notifications and some extra info stored in a shared memory. > For instance, a Xen transport can use per-virtqueue event channels > and a KVM or MSHV transport can use per-virtqueue irqfds/ioeventfds. It is definitely possible and the transport is not enforcing or denying any way to do and is mentioning out of band: - in Driver notifications: "the bus may translate EVENT_AVAIL into out-of-band or deliver in-band; it MUST relay in-band if it does not translate; it MUST NOT drop EVENT_AVAIL unless polling/equivalent is arranged." - in Device notifications: "the bus may forward in-band or synthesize from other signals; it MUST forward (or equivalent) to the driver." We even have a bus normative to say that NOTIFICATION_DATA should not be negotiated if out of band notification is used. But I agree we could benefit from a more explicit mention ... > > I strongly recommend language like the following: > > EVENT_CONFIG, EVENT_USED, and EVENT_AVAIL MAY be sent in-band or > out-of-band. The specification of a given bus MUST specify which > is used. If a bus specifies that out-of-band notifications are to > be used, it MUST document how to send them. so I will include something like that in the v2 of the spec. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-01-26 16:32 ` [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer Bill Mills ` (3 preceding siblings ...) 2026-02-24 17:14 ` Demi Marie Obenour @ 2026-02-25 12:36 ` Demi Marie Obenour 2026-02-25 12:46 ` Bertrand Marquis 4 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 12:36 UTC (permalink / raw) To: Bill Mills, virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 871 bytes --] On 1/26/26 11:32, Bill Mills wrote: (snip) > +\msgdef{GET_SHM} > + > +\msgref{GET_SHM} returns the location and size of a device-owned shared memory > +region. The request carries the region index; the response echoes the index and > +provides the base address and length. A length of zero indicates that the > +specified region does not exist. > + > +\begin{lstlisting} > +struct virtio_msg_get_shm_req { > + le32 index; /* shared memory region index */ > +}; > + > +struct virtio_msg_get_shm_resp { > + le32 index; /* echoed index */ > + le32 length; /* region length (0 if nonexistent) */ > + le32 address;/* region address */ > +}; > +\end{lstlisting} Why only le32? Some shared memory regions, such as mapped GPU buffers, can be much larger than 2^32 bytes. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer 2026-02-25 12:36 ` Demi Marie Obenour @ 2026-02-25 12:46 ` Bertrand Marquis 0 siblings, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 12:46 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 25 Feb 2026, at 13:36, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 1/26/26 11:32, Bill Mills wrote: > > (snip) > >> +\msgdef{GET_SHM} >> + >> +\msgref{GET_SHM} returns the location and size of a device-owned shared memory >> +region. The request carries the region index; the response echoes the index and >> +provides the base address and length. A length of zero indicates that the >> +specified region does not exist. >> + >> +\begin{lstlisting} >> +struct virtio_msg_get_shm_req { >> + le32 index; /* shared memory region index */ >> +}; >> + >> +struct virtio_msg_get_shm_resp { >> + le32 index; /* echoed index */ >> + le32 length; /* region length (0 if nonexistent) */ >> + le32 address;/* region address */ >> +}; >> +\end{lstlisting} > > Why only le32? Some shared memory regions, such as mapped GPU buffers, > can be much larger than 2^32 bytes. Very right, i will update this to hold 64bit addresses and length. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* [PATCH v1 3/4] virtio-msg: link virtio-msg content 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills 2026-01-26 16:32 ` [PATCH v1 1/4] virtio-msg: add new command for bus normative Bill Mills 2026-01-26 16:32 ` [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer Bill Mills @ 2026-01-26 16:32 ` Bill Mills 2026-02-03 19:43 ` Edgar E. Iglesias 2026-01-26 16:32 ` [PATCH v1 4/4] virtio-msg: add conformance entries in conformance chapter Bill Mills ` (6 subsequent siblings) 9 siblings, 1 reply; 105+ messages in thread From: Bill Mills @ 2026-01-26 16:32 UTC (permalink / raw) To: virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine, Bill Mills Add the new transport layer into the content table Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> Signed-off-by: Bill Mills <bill.mills@linaro.org> --- content.tex | 1 + 1 file changed, 1 insertion(+) diff --git a/content.tex b/content.tex index 5de811f..c785c5c 100644 --- a/content.tex +++ b/content.tex @@ -693,6 +693,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} \input{transport-pci.tex} \input{transport-mmio.tex} \input{transport-ccw.tex} +\input{transport-msg.tex} \chapter{Device Types}\label{sec:Device Types} -- 2.34.1 ^ permalink raw reply related [flat|nested] 105+ messages in thread
* Re: [PATCH v1 3/4] virtio-msg: link virtio-msg content 2026-01-26 16:32 ` [PATCH v1 3/4] virtio-msg: link virtio-msg content Bill Mills @ 2026-02-03 19:43 ` Edgar E. Iglesias 0 siblings, 0 replies; 105+ messages in thread From: Edgar E. Iglesias @ 2026-02-03 19:43 UTC (permalink / raw) To: Bill Mills Cc: virtio-comment, Bertrand Marquis, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Mon, Jan 26, 2026 at 11:32:29AM -0500, Bill Mills wrote: > Add the new transport layer into the content table > > Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> > Signed-off-by: Bill Mills <bill.mills@linaro.org> Acked-by: Edgar E. Iglesias <edgar.iglesias@amd.com> > --- > content.tex | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/content.tex b/content.tex > index 5de811f..c785c5c 100644 > --- a/content.tex > +++ b/content.tex > @@ -693,6 +693,7 @@ \chapter{Virtio Transport Options}\label{sec:Virtio Transport Options} > \input{transport-pci.tex} > \input{transport-mmio.tex} > \input{transport-ccw.tex} > +\input{transport-msg.tex} > > \chapter{Device Types}\label{sec:Device Types} > > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 105+ messages in thread
* [PATCH v1 4/4] virtio-msg: add conformance entries in conformance chapter 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills ` (2 preceding siblings ...) 2026-01-26 16:32 ` [PATCH v1 3/4] virtio-msg: link virtio-msg content Bill Mills @ 2026-01-26 16:32 ` Bill Mills 2026-02-03 19:43 ` Edgar E. Iglesias 2026-01-26 21:47 ` [PATCH v1 0/4] virtio-msg transport layer Demi Marie Obenour ` (5 subsequent siblings) 9 siblings, 1 reply; 105+ messages in thread From: Bill Mills @ 2026-01-26 16:32 UTC (permalink / raw) To: virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine, Bill Mills From: Bertrand Marquis <bertrand.marquis@arm.com> Add entries for conformance of Virtio over Message for Driver, Device and bus. Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> Signed-off-by: Bill Mills <bill.mills@linaro.org> --- conformance.tex | 105 +++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 103 insertions(+), 2 deletions(-) diff --git a/conformance.tex b/conformance.tex index 9af31e2..7d5d00f 100644 --- a/conformance.tex +++ b/conformance.tex @@ -14,7 +14,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item[Driver] A driver MUST conform to four conformance clauses: \begin{itemize} \item Clause \ref{sec:Conformance / Driver Conformance}. - \item One of clauses \ref{sec:Conformance / Driver Conformance / PCI Driver Conformance}, \ref{sec:Conformance / Driver Conformance / MMIO Driver Conformance} or \ref{sec:Conformance / Driver Conformance / Channel I/O Driver Conformance}. + \item One of clauses \ref{sec:Conformance / Driver Conformance / PCI Driver Conformance}, \ref{sec:Conformance / Driver Conformance / MMIO Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Channel I/O Driver Conformance} or \ref{sec:Conformance / Driver Conformance / Virtio Over Messages Driver Conformance}. \item One of clauses \ref{sec:Conformance / Driver Conformance / Network Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Block Driver Conformance}, @@ -45,7 +45,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item[Device] A device MUST conform to four conformance clauses: \begin{itemize} \item Clause \ref{sec:Conformance / Device Conformance}. - \item One of clauses \ref{sec:Conformance / Device Conformance / PCI Device Conformance}, \ref{sec:Conformance / Device Conformance / MMIO Device Conformance} or \ref{sec:Conformance / Device Conformance / Channel I/O Device Conformance}. + \item One of clauses \ref{sec:Conformance / Device Conformance / PCI Device Conformance}, \ref{sec:Conformance / Device Conformance / MMIO Device Conformance}, \ref{sec:Conformance / Device Conformance / Channel I/O Device Conformance} or \ref{sec:Conformance / Device Conformance / Virtio Over Messages Device Conformance}. \item One of clauses \ref{sec:Conformance / Device Conformance / Network Device Conformance}, \ref{sec:Conformance / Device Conformance / Block Device Conformance}, @@ -73,6 +73,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize} +\item[Bus] A bus implementation MUST conform to clause \ref{sec:Conformance / Bus Conformance / Virtio Over Messages Bus Conformance} when providing Virtio Over Messages. \end{description} \conformance{\section}{Driver Conformance}\label{sec:Conformance / Driver Conformance} @@ -147,6 +148,42 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{drivernormative:Virtio Transport Options / Virtio over channel I/O / Device Operation / Resetting Devices} \end{itemize} +\conformance{\subsection}{Virtio Over Messages Driver Conformance}\label{sec:Conformance / Driver Conformance / Virtio Over Messages Driver Conformance} + +A Virtio Over Messages driver MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Driver Limits} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Information / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Driver} +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Driver} +\end{itemize} + \input{device-types/net/driver-conformance.tex} \input{device-types/blk/driver-conformance.tex} \input{device-types/console/driver-conformance.tex} @@ -241,6 +278,43 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \item \ref{devicenormative:Virtio Transport Options / Virtio over channel I/O / Device Operation / Resetting Devices} \end{itemize} +\conformance{\subsection}{Virtio Over Messages Device Conformance}\label{sec:Conformance / Device Conformance / Virtio Over Messages Device Conformance} + +A Virtio Over Messages device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Device Limits} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Feature Blocks / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_STATUS / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_SHM / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Device} +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Device} +\end{itemize} + \input{device-types/net/device-conformance.tex} \input{device-types/blk/device-conformance.tex} \input{device-types/console/device-conformance.tex} @@ -265,6 +339,33 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/media/device-conformance.tex} \input{device-types/rtc/device-conformance.tex} +\conformance{\section}{Bus Conformance}\label{sec:Conformance / Bus Conformance} + +A bus implementation MUST conform to the relevant bus normative statements. + +\conformance{\subsection}{Virtio Over Messages Bus Conformance}\label{sec:Conformance / Bus Conformance / Virtio Over Messages Bus Conformance} + +A Virtio Over Messages bus implementation MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport / Bus Messages} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport / Transport Message Forwarding} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Advertising Transport Parameters} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Message Size} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Device Numbers / Assignment} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Device Discovery / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Hotplug and Removal / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Bus Messages / GET_DEVICES / Bus} +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Bus Messages / PING / Bus} +\end{itemize} + \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional or non-transitional, see \ref{intro:Legacy -- 2.34.1 ^ permalink raw reply related [flat|nested] 105+ messages in thread
* Re: [PATCH v1 4/4] virtio-msg: add conformance entries in conformance chapter 2026-01-26 16:32 ` [PATCH v1 4/4] virtio-msg: add conformance entries in conformance chapter Bill Mills @ 2026-02-03 19:43 ` Edgar E. Iglesias 0 siblings, 0 replies; 105+ messages in thread From: Edgar E. Iglesias @ 2026-02-03 19:43 UTC (permalink / raw) To: Bill Mills Cc: virtio-comment, Bertrand Marquis, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Mon, Jan 26, 2026 at 11:32:30AM -0500, Bill Mills wrote: > From: Bertrand Marquis <bertrand.marquis@arm.com> > > Add entries for conformance of Virtio over Message for Driver, Device > and bus. > > Signed-off-by: Bertrand Marquis <bertrand.marquis@arm.com> > Signed-off-by: Bill Mills <bill.mills@linaro.org> Acked-by: Edgar E. Iglesias <edgar.iglesias@amd.com> > --- > conformance.tex | 105 +++++++++++++++++++++++++++++++++++++++++++++++- > 1 file changed, 103 insertions(+), 2 deletions(-) > > diff --git a/conformance.tex b/conformance.tex > index 9af31e2..7d5d00f 100644 > --- a/conformance.tex > +++ b/conformance.tex > @@ -14,7 +14,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \item[Driver] A driver MUST conform to four conformance clauses: > \begin{itemize} > \item Clause \ref{sec:Conformance / Driver Conformance}. > - \item One of clauses \ref{sec:Conformance / Driver Conformance / PCI Driver Conformance}, \ref{sec:Conformance / Driver Conformance / MMIO Driver Conformance} or \ref{sec:Conformance / Driver Conformance / Channel I/O Driver Conformance}. > + \item One of clauses \ref{sec:Conformance / Driver Conformance / PCI Driver Conformance}, \ref{sec:Conformance / Driver Conformance / MMIO Driver Conformance}, \ref{sec:Conformance / Driver Conformance / Channel I/O Driver Conformance} or \ref{sec:Conformance / Driver Conformance / Virtio Over Messages Driver Conformance}. > \item One of clauses > \ref{sec:Conformance / Driver Conformance / Network Driver Conformance}, > \ref{sec:Conformance / Driver Conformance / Block Driver Conformance}, > @@ -45,7 +45,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \item[Device] A device MUST conform to four conformance clauses: > \begin{itemize} > \item Clause \ref{sec:Conformance / Device Conformance}. > - \item One of clauses \ref{sec:Conformance / Device Conformance / PCI Device Conformance}, \ref{sec:Conformance / Device Conformance / MMIO Device Conformance} or \ref{sec:Conformance / Device Conformance / Channel I/O Device Conformance}. > + \item One of clauses \ref{sec:Conformance / Device Conformance / PCI Device Conformance}, \ref{sec:Conformance / Device Conformance / MMIO Device Conformance}, \ref{sec:Conformance / Device Conformance / Channel I/O Device Conformance} or \ref{sec:Conformance / Device Conformance / Virtio Over Messages Device Conformance}. > \item One of clauses > \ref{sec:Conformance / Device Conformance / Network Device Conformance}, > \ref{sec:Conformance / Device Conformance / Block Device Conformance}, > @@ -73,6 +73,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > > \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. > \end{itemize} > +\item[Bus] A bus implementation MUST conform to clause \ref{sec:Conformance / Bus Conformance / Virtio Over Messages Bus Conformance} when providing Virtio Over Messages. > \end{description} > > \conformance{\section}{Driver Conformance}\label{sec:Conformance / Driver Conformance} > @@ -147,6 +148,42 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \item \ref{drivernormative:Virtio Transport Options / Virtio over channel I/O / Device Operation / Resetting Devices} > \end{itemize} > > +\conformance{\subsection}{Virtio Over Messages Driver Conformance}\label{sec:Conformance / Driver Conformance / Virtio Over Messages Driver Conformance} > + > +A Virtio Over Messages driver MUST conform to the following normative statements: > + > +\begin{itemize} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Driver Limits} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Overview / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Information / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Driver} > +\item \ref{drivernormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Driver} > +\end{itemize} > + > \input{device-types/net/driver-conformance.tex} > \input{device-types/blk/driver-conformance.tex} > \input{device-types/console/driver-conformance.tex} > @@ -241,6 +278,43 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \item \ref{devicenormative:Virtio Transport Options / Virtio over channel I/O / Device Operation / Resetting Devices} > \end{itemize} > > +\conformance{\subsection}{Virtio Over Messages Device Conformance}\label{sec:Conformance / Device Conformance / Virtio Over Messages Device Conformance} > + > +A Virtio Over Messages device MUST conform to the following normative statements: > + > +\begin{itemize} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Device Limits} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Configuration Generation Count / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Feature Blocks / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Endianness / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Features / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Device Configuration / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Virtqueue Configuration / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Status Information / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Initialization / Finalizing Initialization / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Configuration Changes During Operation / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Virtqueue Changes During Operation / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Reset / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / Mandatory / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_INFO / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_FEATURES / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DRIVER_FEATURES / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_CONFIG / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_CONFIG / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_DEVICE_STATUS / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_DEVICE_STATUS / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_VQUEUE / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_SET_VQUEUE / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_RESET_VQUEUE / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_GET_SHM / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_CONFIG / Device} > +\item \ref{devicenormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_USED / Device} > +\end{itemize} > + > \input{device-types/net/device-conformance.tex} > \input{device-types/blk/device-conformance.tex} > \input{device-types/console/device-conformance.tex} > @@ -265,6 +339,33 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \input{device-types/media/device-conformance.tex} > \input{device-types/rtc/device-conformance.tex} > > +\conformance{\section}{Bus Conformance}\label{sec:Conformance / Bus Conformance} > + > +A bus implementation MUST conform to the relevant bus normative statements. > + > +\conformance{\subsection}{Virtio Over Messages Bus Conformance}\label{sec:Conformance / Bus Conformance / Virtio Over Messages Bus Conformance} > + > +A Virtio Over Messages bus implementation MUST conform to the following normative statements: > + > +\begin{itemize} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport / Bus Messages} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Relationship between bus and transport / Transport Message Forwarding} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Advertising Transport Parameters} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Revisions / Message Size} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Versioning and Forward Compatibility / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Device Numbers / Assignment} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Error Signaling / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Common Message Format / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Basic Concepts / Ordering / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Device Discovery / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Driver Notifications / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Device Notifications / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Device Operation / Hotplug and Removal / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Transport Messages / VIRTIO_MSG_EVENT_AVAIL / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Bus Messages / GET_DEVICES / Bus} > +\item \ref{busnormative:Virtio Transport Options / Virtio Over Messages / Bus Messages / PING / Bus} > +\end{itemize} > + > \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} > A conformant implementation MUST be either transitional or > non-transitional, see \ref{intro:Legacy > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills ` (3 preceding siblings ...) 2026-01-26 16:32 ` [PATCH v1 4/4] virtio-msg: add conformance entries in conformance chapter Bill Mills @ 2026-01-26 21:47 ` Demi Marie Obenour 2026-02-03 13:21 ` Michael S. Tsirkin ` (4 subsequent siblings) 9 siblings, 0 replies; 105+ messages in thread From: Demi Marie Obenour @ 2026-01-26 21:47 UTC (permalink / raw) To: Bill Mills, virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 2470 bytes --] On 1/26/26 11:32, Bill Mills wrote: > This series adds the virtio-msg transport layer. > > The individuals and organizations involved in this effort have had difficulty in > using the existing virtio-transports in various situations and desire to add one > more transport that performs its transport layer operations by sending and > receiving messages. > > Implementations of virtio-msg will normally be done in multiple layers: > * common / device level > * bus level > > The common / device level defines the messages exchanged between the driver > and a device. This common part should lead to a common driver holding most > of the virtio specifics and can be shared by all virtio-msg bus implementations. > The kernel implementation in [3] shows this separation. As with other transport > layers, virtio-msg should not require modifications to existing virtio device > implementations (virtio-net, virtio-blk etc). The common / device level is the > main focus of this version of the patch series. > > The virtio-msg bus level implements the normal things a bus defines > (enumeration, dma operations, etc) but also implements the message send and > receive operations. A number of bus implementations are envisioned, > some of which will be reusable and general purpose. Other bus implementations > might be unique to a given situation, for example only used by a PCIe card > and its driver. > > The standard bus messages are an effort to avoid different bus implementations > doing the same thing in different ways for no good reason. However the > different environments will require different things. Instead of trying to > anticipate all needs and provide something very abstract, we think > implementation specific messages will be needed at the bus level. Over time, > if we see similar messages across multiple bus implementations, we will move to > standardize a bus level message for that. > > We are working on a few reusable bus implementations: > > * virtio-msg-ffa based on Arm FF-A interface for use between: > * normal world and secure world > * host and VM or VM to VM > * Can be used w/ or with out a hypervisor > * Any Hypervisor that implements FF-A can be used > * We have this working with pKVM and Xen FYI, FF-A is explicitly not security supported by Xen. The virtio-msg-xen option might be a better alternative, even on Arm. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills ` (4 preceding siblings ...) 2026-01-26 21:47 ` [PATCH v1 0/4] virtio-msg transport layer Demi Marie Obenour @ 2026-02-03 13:21 ` Michael S. Tsirkin 2026-02-03 19:48 ` Edgar E. Iglesias ` (2 more replies) 2026-02-04 3:29 ` Viresh Kumar ` (3 subsequent siblings) 9 siblings, 3 replies; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-03 13:21 UTC (permalink / raw) To: Bill Mills Cc: virtio-comment, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Mon, Jan 26, 2026 at 11:32:26AM -0500, Bill Mills wrote: > This series adds the virtio-msg transport layer. > > The individuals and organizations involved in this effort have had difficulty in > using the existing virtio-transports in various situations and desire to add one > more transport that performs its transport layer operations by sending and > receiving messages. Is it arm and linaro, or more? If more, could we get some acks on list from the people involved? > Implementations of virtio-msg will normally be done in multiple layers: > * common / device level > * bus level > > The common / device level defines the messages exchanged between the driver > and a device. This common part should lead to a common driver holding most > of the virtio specifics and can be shared by all virtio-msg bus implementations. > The kernel implementation in [3] shows this separation. As with other transport > layers, virtio-msg should not require modifications to existing virtio device > implementations (virtio-net, virtio-blk etc). The common / device level is the > main focus of this version of the patch series. > > The virtio-msg bus level implements the normal things a bus defines > (enumeration, dma operations, etc) but also implements the message send and > receive operations. A number of bus implementations are envisioned, > some of which will be reusable and general purpose. Other bus implementations > might be unique to a given situation, for example only used by a PCIe card > and its driver. > > The standard bus messages are an effort to avoid different bus implementations > doing the same thing in different ways for no good reason. However the > different environments will require different things. Instead of trying to > anticipate all needs and provide something very abstract, we think > implementation specific messages will be needed at the bus level. Over time, > if we see similar messages across multiple bus implementations, we will move to > standardize a bus level message for that. > > We are working on a few reusable bus implementations: > > * virtio-msg-ffa based on Arm FF-A interface for use between: > * normal world and secure world > * host and VM or VM to VM > * Can be used w/ or with out a hypervisor > * Any Hypervisor that implements FF-A can be used > * We have this working with pKVM and Xen > * We have this working with Trusty and OP-TEE > > * virtio-msg-amp for use between heterogenous systems > * The main processors and its co-processors on an AMP SOC > * Two or more systems connected via PCIe > * Minimal requirements: bi-directional interrupts and > at least one shared memory area > * hvac-demo has 2 demos of this > * This is working on two hardware platforms > > * virtio-msg-loopback for userspace implemented devices > * Allows user space to provide devices to its own kernel > * This is similar to fuse, cuse or loopback block devices but for virtio > * hvac-demo has a demo > > We also anticipate a few more: > > * virtio-msg-xen specific to Xen > * Usable on any Xen system (including x86 where FF-A does not exist) > * Using Xen events and page grants > > * virtio-msg over admin virtqueues > * This allows any virtio-pci device that supports admin virtqueues to also > support a virtio-msg bus that supports sub devices > * [We are looking for collaborators for this work] > > Changes since RFC2: > > Spec Functional: > * Made the common message header 8 bytes and added a token for optional use > when parallel outstanding requests are possible > * Made the 8 byte fields align to 8 byte offsets. > This effects the {SET,GET}_VQUEUE messages > * Many conformance cases have been tightened. > > Spec Editorial: > * Major restructure to better align with virtio spec > * Conformance model now matches other transports > * Use C structures to define messages instead of tables > * Added a section to describe how responses are matched to requests > This includes the use of the new token field > * Redefined / better described error handling between transport and bus > layers to eliminate the need for the bus to generate fake response messages > * Included editorial feedback from RFC2 > > Eco-system: > * Added virtio-msg-loopback demo > * Arm has published the first draft of the virtio-msg over FFA spec [6] > * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE > secure world > * LKML RFC has been sent [7] > * QEMU RFC has been sent [8] > > This is the first non-RFC patch series. The known short comings have been > addressed. We ask for review in earnest on this series and thank you for > any feedback you can provide. > > Background info and work in progress implementations: > * HVAC project page with intro slides [1] > * HVAC demo repo w/ instructions in README.md [2] > * Kernel w/ virtio-msg common level and ffa support [3] > * QEMU w/ support for one form of virtio-msg-amp [4] > * Portable RTOS library w/ one form of virtio-msg-amp [5] > > In addition to the QEMU system based demos in the hvac-demo repo, we also have > two hardware systems running: > * AMD x86 + AMD Arm Versal connected via PCIe > * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr > > Please note that although the demos work, a few have not been aligned > with this version of the spec. > > [1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview > [2] https://github.com/wmamills/hvac-demo > [3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=virtio/msg > [4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/ > [5] https://github.com/arnopo/open-amp/commits/virtio-msg/ > [6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c > [7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/ > [8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html > > Bertrand Marquis (2): > virtio-msg: add new command for bus normative > virtio-msg: add conformance entries in conformance chapter > > Bill Mills (2): > virtio-msg: Add virtio-msg, a message based virtio transport layer > virtio-msg: link virtio-msg content > > commands.tex | 3 +- > conformance.tex | 105 ++- > content.tex | 1 + > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > 4 files changed, 1746 insertions(+), 3 deletions(-) > create mode 100644 transport-msg.tex > > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-03 13:21 ` Michael S. Tsirkin @ 2026-02-03 19:48 ` Edgar E. Iglesias 2026-02-03 19:55 ` Michael S. Tsirkin 2026-02-04 8:33 ` Bertrand Marquis 2026-02-04 13:50 ` Arnaud POULIQUEN 2 siblings, 1 reply; 105+ messages in thread From: Edgar E. Iglesias @ 2026-02-03 19:48 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Bill Mills, virtio-comment, Bertrand Marquis, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Tue, Feb 03, 2026 at 08:21:01AM -0500, Michael S. Tsirkin wrote: > On Mon, Jan 26, 2026 at 11:32:26AM -0500, Bill Mills wrote: > > This series adds the virtio-msg transport layer. > > > > The individuals and organizations involved in this effort have had difficulty in > > using the existing virtio-transports in various situations and desire to add one > > more transport that performs its transport layer operations by sending and > > receiving messages. > > Is it arm and linaro, or more? > If more, could we get some acks on list from the people involved? > > AMD is also involved, I've acked patches 1, 3, and 4 (patch 2 has my Signed-off-by). Cheers, Edgar > > Implementations of virtio-msg will normally be done in multiple layers: > > * common / device level > > * bus level > > > > The common / device level defines the messages exchanged between the driver > > and a device. This common part should lead to a common driver holding most > > of the virtio specifics and can be shared by all virtio-msg bus implementations. > > The kernel implementation in [3] shows this separation. As with other transport > > layers, virtio-msg should not require modifications to existing virtio device > > implementations (virtio-net, virtio-blk etc). The common / device level is the > > main focus of this version of the patch series. > > > > The virtio-msg bus level implements the normal things a bus defines > > (enumeration, dma operations, etc) but also implements the message send and > > receive operations. A number of bus implementations are envisioned, > > some of which will be reusable and general purpose. Other bus implementations > > might be unique to a given situation, for example only used by a PCIe card > > and its driver. > > > > The standard bus messages are an effort to avoid different bus implementations > > doing the same thing in different ways for no good reason. However the > > different environments will require different things. Instead of trying to > > anticipate all needs and provide something very abstract, we think > > implementation specific messages will be needed at the bus level. Over time, > > if we see similar messages across multiple bus implementations, we will move to > > standardize a bus level message for that. > > > > We are working on a few reusable bus implementations: > > > > * virtio-msg-ffa based on Arm FF-A interface for use between: > > * normal world and secure world > > * host and VM or VM to VM > > * Can be used w/ or with out a hypervisor > > * Any Hypervisor that implements FF-A can be used > > * We have this working with pKVM and Xen > > * We have this working with Trusty and OP-TEE > > > > * virtio-msg-amp for use between heterogenous systems > > * The main processors and its co-processors on an AMP SOC > > * Two or more systems connected via PCIe > > * Minimal requirements: bi-directional interrupts and > > at least one shared memory area > > * hvac-demo has 2 demos of this > > * This is working on two hardware platforms > > > > * virtio-msg-loopback for userspace implemented devices > > * Allows user space to provide devices to its own kernel > > * This is similar to fuse, cuse or loopback block devices but for virtio > > * hvac-demo has a demo > > > > We also anticipate a few more: > > > > * virtio-msg-xen specific to Xen > > * Usable on any Xen system (including x86 where FF-A does not exist) > > * Using Xen events and page grants > > > > * virtio-msg over admin virtqueues > > * This allows any virtio-pci device that supports admin virtqueues to also > > support a virtio-msg bus that supports sub devices > > * [We are looking for collaborators for this work] > > > > Changes since RFC2: > > > > Spec Functional: > > * Made the common message header 8 bytes and added a token for optional use > > when parallel outstanding requests are possible > > * Made the 8 byte fields align to 8 byte offsets. > > This effects the {SET,GET}_VQUEUE messages > > * Many conformance cases have been tightened. > > > > Spec Editorial: > > * Major restructure to better align with virtio spec > > * Conformance model now matches other transports > > * Use C structures to define messages instead of tables > > * Added a section to describe how responses are matched to requests > > This includes the use of the new token field > > * Redefined / better described error handling between transport and bus > > layers to eliminate the need for the bus to generate fake response messages > > * Included editorial feedback from RFC2 > > > > Eco-system: > > * Added virtio-msg-loopback demo > > * Arm has published the first draft of the virtio-msg over FFA spec [6] > > * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE > > secure world > > * LKML RFC has been sent [7] > > * QEMU RFC has been sent [8] > > > > This is the first non-RFC patch series. The known short comings have been > > addressed. We ask for review in earnest on this series and thank you for > > any feedback you can provide. > > > > Background info and work in progress implementations: > > * HVAC project page with intro slides [1] > > * HVAC demo repo w/ instructions in README.md [2] > > * Kernel w/ virtio-msg common level and ffa support [3] > > * QEMU w/ support for one form of virtio-msg-amp [4] > > * Portable RTOS library w/ one form of virtio-msg-amp [5] > > > > In addition to the QEMU system based demos in the hvac-demo repo, we also have > > two hardware systems running: > > * AMD x86 + AMD Arm Versal connected via PCIe > > * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr > > > > Please note that although the demos work, a few have not been aligned > > with this version of the spec. > > > > [1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview > > [2] https://github.com/wmamills/hvac-demo > > [3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=virtio/msg > > [4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/ > > [5] https://github.com/arnopo/open-amp/commits/virtio-msg/ > > [6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c > > [7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/ > > [8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html > > > > Bertrand Marquis (2): > > virtio-msg: add new command for bus normative > > virtio-msg: add conformance entries in conformance chapter > > > > Bill Mills (2): > > virtio-msg: Add virtio-msg, a message based virtio transport layer > > virtio-msg: link virtio-msg content > > > > commands.tex | 3 +- > > conformance.tex | 105 ++- > > content.tex | 1 + > > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > > 4 files changed, 1746 insertions(+), 3 deletions(-) > > create mode 100644 transport-msg.tex > > > > -- > > 2.34.1 > > > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-03 19:48 ` Edgar E. Iglesias @ 2026-02-03 19:55 ` Michael S. Tsirkin 0 siblings, 0 replies; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-03 19:55 UTC (permalink / raw) To: Edgar E. Iglesias Cc: Bill Mills, virtio-comment, Bertrand Marquis, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Tue, Feb 03, 2026 at 08:48:00PM +0100, Edgar E. Iglesias wrote: > On Tue, Feb 03, 2026 at 08:21:01AM -0500, Michael S. Tsirkin wrote: > > On Mon, Jan 26, 2026 at 11:32:26AM -0500, Bill Mills wrote: > > > This series adds the virtio-msg transport layer. > > > > > > The individuals and organizations involved in this effort have had difficulty in > > > using the existing virtio-transports in various situations and desire to add one > > > more transport that performs its transport layer operations by sending and > > > receiving messages. > > > > Is it arm and linaro, or more? > > If more, could we get some acks on list from the people involved? > > > > > > AMD is also involved, I've acked patches 1, 3, and 4 (patch 2 has my Signed-off-by). > > Cheers, > Edgar > Thank you > > > > Implementations of virtio-msg will normally be done in multiple layers: > > > * common / device level > > > * bus level > > > > > > The common / device level defines the messages exchanged between the driver > > > and a device. This common part should lead to a common driver holding most > > > of the virtio specifics and can be shared by all virtio-msg bus implementations. > > > The kernel implementation in [3] shows this separation. As with other transport > > > layers, virtio-msg should not require modifications to existing virtio device > > > implementations (virtio-net, virtio-blk etc). The common / device level is the > > > main focus of this version of the patch series. > > > > > > The virtio-msg bus level implements the normal things a bus defines > > > (enumeration, dma operations, etc) but also implements the message send and > > > receive operations. A number of bus implementations are envisioned, > > > some of which will be reusable and general purpose. Other bus implementations > > > might be unique to a given situation, for example only used by a PCIe card > > > and its driver. > > > > > > The standard bus messages are an effort to avoid different bus implementations > > > doing the same thing in different ways for no good reason. However the > > > different environments will require different things. Instead of trying to > > > anticipate all needs and provide something very abstract, we think > > > implementation specific messages will be needed at the bus level. Over time, > > > if we see similar messages across multiple bus implementations, we will move to > > > standardize a bus level message for that. > > > > > > We are working on a few reusable bus implementations: > > > > > > * virtio-msg-ffa based on Arm FF-A interface for use between: > > > * normal world and secure world > > > * host and VM or VM to VM > > > * Can be used w/ or with out a hypervisor > > > * Any Hypervisor that implements FF-A can be used > > > * We have this working with pKVM and Xen > > > * We have this working with Trusty and OP-TEE > > > > > > * virtio-msg-amp for use between heterogenous systems > > > * The main processors and its co-processors on an AMP SOC > > > * Two or more systems connected via PCIe > > > * Minimal requirements: bi-directional interrupts and > > > at least one shared memory area > > > * hvac-demo has 2 demos of this > > > * This is working on two hardware platforms > > > > > > * virtio-msg-loopback for userspace implemented devices > > > * Allows user space to provide devices to its own kernel > > > * This is similar to fuse, cuse or loopback block devices but for virtio > > > * hvac-demo has a demo > > > > > > We also anticipate a few more: > > > > > > * virtio-msg-xen specific to Xen > > > * Usable on any Xen system (including x86 where FF-A does not exist) > > > * Using Xen events and page grants > > > > > > * virtio-msg over admin virtqueues > > > * This allows any virtio-pci device that supports admin virtqueues to also > > > support a virtio-msg bus that supports sub devices > > > * [We are looking for collaborators for this work] > > > > > > Changes since RFC2: > > > > > > Spec Functional: > > > * Made the common message header 8 bytes and added a token for optional use > > > when parallel outstanding requests are possible > > > * Made the 8 byte fields align to 8 byte offsets. > > > This effects the {SET,GET}_VQUEUE messages > > > * Many conformance cases have been tightened. > > > > > > Spec Editorial: > > > * Major restructure to better align with virtio spec > > > * Conformance model now matches other transports > > > * Use C structures to define messages instead of tables > > > * Added a section to describe how responses are matched to requests > > > This includes the use of the new token field > > > * Redefined / better described error handling between transport and bus > > > layers to eliminate the need for the bus to generate fake response messages > > > * Included editorial feedback from RFC2 > > > > > > Eco-system: > > > * Added virtio-msg-loopback demo > > > * Arm has published the first draft of the virtio-msg over FFA spec [6] > > > * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE > > > secure world > > > * LKML RFC has been sent [7] > > > * QEMU RFC has been sent [8] > > > > > > This is the first non-RFC patch series. The known short comings have been > > > addressed. We ask for review in earnest on this series and thank you for > > > any feedback you can provide. > > > > > > Background info and work in progress implementations: > > > * HVAC project page with intro slides [1] > > > * HVAC demo repo w/ instructions in README.md [2] > > > * Kernel w/ virtio-msg common level and ffa support [3] > > > * QEMU w/ support for one form of virtio-msg-amp [4] > > > * Portable RTOS library w/ one form of virtio-msg-amp [5] > > > > > > In addition to the QEMU system based demos in the hvac-demo repo, we also have > > > two hardware systems running: > > > * AMD x86 + AMD Arm Versal connected via PCIe > > > * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr > > > > > > Please note that although the demos work, a few have not been aligned > > > with this version of the spec. > > > > > > [1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview > > > [2] https://github.com/wmamills/hvac-demo > > > [3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=virtio/msg > > > [4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/ > > > [5] https://github.com/arnopo/open-amp/commits/virtio-msg/ > > > [6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c > > > [7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/ > > > [8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html > > > > > > Bertrand Marquis (2): > > > virtio-msg: add new command for bus normative > > > virtio-msg: add conformance entries in conformance chapter > > > > > > Bill Mills (2): > > > virtio-msg: Add virtio-msg, a message based virtio transport layer > > > virtio-msg: link virtio-msg content > > > > > > commands.tex | 3 +- > > > conformance.tex | 105 ++- > > > content.tex | 1 + > > > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > > > 4 files changed, 1746 insertions(+), 3 deletions(-) > > > create mode 100644 transport-msg.tex > > > > > > -- > > > 2.34.1 > > > > > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-03 13:21 ` Michael S. Tsirkin 2026-02-03 19:48 ` Edgar E. Iglesias @ 2026-02-04 8:33 ` Bertrand Marquis 2026-02-04 13:50 ` Arnaud POULIQUEN 2 siblings, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-04 8:33 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi, > On 3 Feb 2026, at 14:21, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Mon, Jan 26, 2026 at 11:32:26AM -0500, Bill Mills wrote: >> This series adds the virtio-msg transport layer. >> >> The individuals and organizations involved in this effort have had difficulty in >> using the existing virtio-transports in various situations and desire to add one >> more transport that performs its transport layer operations by sending and >> receiving messages. > > Is it arm and linaro, or more? > If more, could we get some acks on list from the people involved? Arm is also interested in virtio message for Various use cases including the Android one and has been deeply involved in designing and writing the specification (hence my signed-off in the patches of this serie). Acked-by: Bertrand Marquis <bertrand.marquis@arm.com> Cheers Bertrand IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-03 13:21 ` Michael S. Tsirkin 2026-02-03 19:48 ` Edgar E. Iglesias 2026-02-04 8:33 ` Bertrand Marquis @ 2026-02-04 13:50 ` Arnaud POULIQUEN 2 siblings, 0 replies; 105+ messages in thread From: Arnaud POULIQUEN @ 2026-02-04 13:50 UTC (permalink / raw) To: Michael S. Tsirkin, Bill Mills Cc: virtio-comment, Bertrand Marquis, Edgar E . Iglesias, Viresh Kumar, Alex Bennee, Armelle Laine Hello, On 2/3/26 14:21, Michael S. Tsirkin wrote: > On Mon, Jan 26, 2026 at 11:32:26AM -0500, Bill Mills wrote: >> This series adds the virtio-msg transport layer. >> >> The individuals and organizations involved in this effort have had difficulty in >> using the existing virtio-transports in various situations and desire to add one >> more transport that performs its transport layer operations by sending and >> receiving messages. > > Is it arm and linaro, or more? > If more, could we get some acks on list from the people involved? STMicroelectronics is also interested in this transport. I have already my signed off on patch 2/4. For the rest of the series: Acked-by: Arnaud Pouliquen arnaud.pouliquen@foss.st.com Thanks, Arnaud > > >> Implementations of virtio-msg will normally be done in multiple layers: >> * common / device level >> * bus level >> >> The common / device level defines the messages exchanged between the driver >> and a device. This common part should lead to a common driver holding most >> of the virtio specifics and can be shared by all virtio-msg bus implementations. >> The kernel implementation in [3] shows this separation. As with other transport >> layers, virtio-msg should not require modifications to existing virtio device >> implementations (virtio-net, virtio-blk etc). The common / device level is the >> main focus of this version of the patch series. >> >> The virtio-msg bus level implements the normal things a bus defines >> (enumeration, dma operations, etc) but also implements the message send and >> receive operations. A number of bus implementations are envisioned, >> some of which will be reusable and general purpose. Other bus implementations >> might be unique to a given situation, for example only used by a PCIe card >> and its driver. >> >> The standard bus messages are an effort to avoid different bus implementations >> doing the same thing in different ways for no good reason. However the >> different environments will require different things. Instead of trying to >> anticipate all needs and provide something very abstract, we think >> implementation specific messages will be needed at the bus level. Over time, >> if we see similar messages across multiple bus implementations, we will move to >> standardize a bus level message for that. >> >> We are working on a few reusable bus implementations: >> >> * virtio-msg-ffa based on Arm FF-A interface for use between: >> * normal world and secure world >> * host and VM or VM to VM >> * Can be used w/ or with out a hypervisor >> * Any Hypervisor that implements FF-A can be used >> * We have this working with pKVM and Xen >> * We have this working with Trusty and OP-TEE >> >> * virtio-msg-amp for use between heterogenous systems >> * The main processors and its co-processors on an AMP SOC >> * Two or more systems connected via PCIe >> * Minimal requirements: bi-directional interrupts and >> at least one shared memory area >> * hvac-demo has 2 demos of this >> * This is working on two hardware platforms >> >> * virtio-msg-loopback for userspace implemented devices >> * Allows user space to provide devices to its own kernel >> * This is similar to fuse, cuse or loopback block devices but for virtio >> * hvac-demo has a demo >> >> We also anticipate a few more: >> >> * virtio-msg-xen specific to Xen >> * Usable on any Xen system (including x86 where FF-A does not exist) >> * Using Xen events and page grants >> >> * virtio-msg over admin virtqueues >> * This allows any virtio-pci device that supports admin virtqueues to also >> support a virtio-msg bus that supports sub devices >> * [We are looking for collaborators for this work] >> >> Changes since RFC2: >> >> Spec Functional: >> * Made the common message header 8 bytes and added a token for optional use >> when parallel outstanding requests are possible >> * Made the 8 byte fields align to 8 byte offsets. >> This effects the {SET,GET}_VQUEUE messages >> * Many conformance cases have been tightened. >> >> Spec Editorial: >> * Major restructure to better align with virtio spec >> * Conformance model now matches other transports >> * Use C structures to define messages instead of tables >> * Added a section to describe how responses are matched to requests >> This includes the use of the new token field >> * Redefined / better described error handling between transport and bus >> layers to eliminate the need for the bus to generate fake response messages >> * Included editorial feedback from RFC2 >> >> Eco-system: >> * Added virtio-msg-loopback demo >> * Arm has published the first draft of the virtio-msg over FFA spec [6] >> * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE >> secure world >> * LKML RFC has been sent [7] >> * QEMU RFC has been sent [8] >> >> This is the first non-RFC patch series. The known short comings have been >> addressed. We ask for review in earnest on this series and thank you for >> any feedback you can provide. >> >> Background info and work in progress implementations: >> * HVAC project page with intro slides [1] >> * HVAC demo repo w/ instructions in README.md [2] >> * Kernel w/ virtio-msg common level and ffa support [3] >> * QEMU w/ support for one form of virtio-msg-amp [4] >> * Portable RTOS library w/ one form of virtio-msg-amp [5] >> >> In addition to the QEMU system based demos in the hvac-demo repo, we also have >> two hardware systems running: >> * AMD x86 + AMD Arm Versal connected via PCIe >> * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr >> >> Please note that although the demos work, a few have not been aligned >> with this version of the spec. >> >> [1] https://linaro.atlassian.net/wiki/spaces/HVAC/overview >> [2] https://github.com/wmamills/hvac-demo >> [3] https://git.kernel.org/pub/scm/linux/kernel/git/vireshk/linux.git/log/?h=virtio/msg >> [4] https://github.com/edgarigl/qemu/commits/edgar/virtio-msg-rfc/ >> [5] https://github.com/arnopo/open-amp/commits/virtio-msg/ >> [6] https://documentation-service.arm.com/static/68f647791134f773ab3f0a7c >> [7] https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linaro.org/ >> [8] https://mail.gnu.org/archive/html/qemu-devel/2025-10/msg07438.html >> >> Bertrand Marquis (2): >> virtio-msg: add new command for bus normative >> virtio-msg: add conformance entries in conformance chapter >> >> Bill Mills (2): >> virtio-msg: Add virtio-msg, a message based virtio transport layer >> virtio-msg: link virtio-msg content >> >> commands.tex | 3 +- >> conformance.tex | 105 ++- >> content.tex | 1 + >> transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ >> 4 files changed, 1746 insertions(+), 3 deletions(-) >> create mode 100644 transport-msg.tex >> >> -- >> 2.34.1 >> > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills ` (5 preceding siblings ...) 2026-02-03 13:21 ` Michael S. Tsirkin @ 2026-02-04 3:29 ` Viresh Kumar 2026-02-04 5:34 ` Manivannan Sadhasivam ` (2 subsequent siblings) 9 siblings, 0 replies; 105+ messages in thread From: Viresh Kumar @ 2026-02-04 3:29 UTC (permalink / raw) To: Bill Mills Cc: virtio-comment, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Alex Bennee, Armelle Laine On 26-01-26, 11:32, Bill Mills wrote: > This series adds the virtio-msg transport layer. > > The individuals and organizations involved in this effort have had difficulty in > using the existing virtio-transports in various situations and desire to add one > more transport that performs its transport layer operations by sending and > receiving messages. > > Implementations of virtio-msg will normally be done in multiple layers: > * common / device level > * bus level > > The common / device level defines the messages exchanged between the driver > and a device. This common part should lead to a common driver holding most > of the virtio specifics and can be shared by all virtio-msg bus implementations. > The kernel implementation in [3] shows this separation. As with other transport > layers, virtio-msg should not require modifications to existing virtio device > implementations (virtio-net, virtio-blk etc). The common / device level is the > main focus of this version of the patch series. > > The virtio-msg bus level implements the normal things a bus defines > (enumeration, dma operations, etc) but also implements the message send and > receive operations. A number of bus implementations are envisioned, > some of which will be reusable and general purpose. Other bus implementations > might be unique to a given situation, for example only used by a PCIe card > and its driver. > > The standard bus messages are an effort to avoid different bus implementations > doing the same thing in different ways for no good reason. However the > different environments will require different things. Instead of trying to > anticipate all needs and provide something very abstract, we think > implementation specific messages will be needed at the bus level. Over time, > if we see similar messages across multiple bus implementations, we will move to > standardize a bus level message for that. > > We are working on a few reusable bus implementations: > Bertrand Marquis (2): > virtio-msg: add new command for bus normative > virtio-msg: add conformance entries in conformance chapter > > Bill Mills (2): > virtio-msg: Add virtio-msg, a message based virtio transport layer > virtio-msg: link virtio-msg content > > commands.tex | 3 +- > conformance.tex | 105 ++- > content.tex | 1 + > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > 4 files changed, 1746 insertions(+), 3 deletions(-) > create mode 100644 transport-msg.tex Acked-by: Viresh Kumar <viresh.kumar@linaro.org> -- viresh ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills ` (6 preceding siblings ...) 2026-02-04 3:29 ` Viresh Kumar @ 2026-02-04 5:34 ` Manivannan Sadhasivam 2026-02-13 13:52 ` Parav Pandit 2026-02-24 17:57 ` Demi Marie Obenour 9 siblings, 0 replies; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-04 5:34 UTC (permalink / raw) To: Bill Mills Cc: virtio-comment, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Mon, Jan 26, 2026 at 11:32:26AM -0500, Bill Mills wrote: > This series adds the virtio-msg transport layer. > > The individuals and organizations involved in this effort have had difficulty in > using the existing virtio-transports in various situations and desire to add one > more transport that performs its transport layer operations by sending and > receiving messages. > > Implementations of virtio-msg will normally be done in multiple layers: > * common / device level > * bus level > > The common / device level defines the messages exchanged between the driver > and a device. This common part should lead to a common driver holding most > of the virtio specifics and can be shared by all virtio-msg bus implementations. > The kernel implementation in [3] shows this separation. As with other transport > layers, virtio-msg should not require modifications to existing virtio device > implementations (virtio-net, virtio-blk etc). The common / device level is the > main focus of this version of the patch series. > > The virtio-msg bus level implements the normal things a bus defines > (enumeration, dma operations, etc) but also implements the message send and > receive operations. A number of bus implementations are envisioned, > some of which will be reusable and general purpose. Other bus implementations > might be unique to a given situation, for example only used by a PCIe card > and its driver. > > The standard bus messages are an effort to avoid different bus implementations > doing the same thing in different ways for no good reason. However the > different environments will require different things. Instead of trying to > anticipate all needs and provide something very abstract, we think > implementation specific messages will be needed at the bus level. Over time, > if we see similar messages across multiple bus implementations, we will move to > standardize a bus level message for that. > > We are working on a few reusable bus implementations: > > * virtio-msg-ffa based on Arm FF-A interface for use between: > * normal world and secure world > * host and VM or VM to VM > * Can be used w/ or with out a hypervisor > * Any Hypervisor that implements FF-A can be used > * We have this working with pKVM and Xen > * We have this working with Trusty and OP-TEE > > * virtio-msg-amp for use between heterogenous systems > * The main processors and its co-processors on an AMP SOC > * Two or more systems connected via PCIe > * Minimal requirements: bi-directional interrupts and > at least one shared memory area > * hvac-demo has 2 demos of this > * This is working on two hardware platforms > > * virtio-msg-loopback for userspace implemented devices > * Allows user space to provide devices to its own kernel > * This is similar to fuse, cuse or loopback block devices but for virtio > * hvac-demo has a demo > > We also anticipate a few more: > > * virtio-msg-xen specific to Xen > * Usable on any Xen system (including x86 where FF-A does not exist) > * Using Xen events and page grants > > * virtio-msg over admin virtqueues > * This allows any virtio-pci device that supports admin virtqueues to also > support a virtio-msg bus that supports sub devices > * [We are looking for collaborators for this work] > > Changes since RFC2: > > Spec Functional: > * Made the common message header 8 bytes and added a token for optional use > when parallel outstanding requests are possible > * Made the 8 byte fields align to 8 byte offsets. > This effects the {SET,GET}_VQUEUE messages > * Many conformance cases have been tightened. > > Spec Editorial: > * Major restructure to better align with virtio spec > * Conformance model now matches other transports > * Use C structures to define messages instead of tables > * Added a section to describe how responses are matched to requests > This includes the use of the new token field > * Redefined / better described error handling between transport and bus > layers to eliminate the need for the bus to generate fake response messages > * Included editorial feedback from RFC2 > > Eco-system: > * Added virtio-msg-loopback demo > * Arm has published the first draft of the virtio-msg over FFA spec [6] > * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE > secure world > * LKML RFC has been sent [7] > * QEMU RFC has been sent [8] > > This is the first non-RFC patch series. The known short comings have been > addressed. We ask for review in earnest on this series and thank you for > any feedback you can provide. > > Background info and work in progress implementations: > * HVAC project page with intro slides [1] > * HVAC demo repo w/ instructions in README.md [2] > * Kernel w/ virtio-msg common level and ffa support [3] > * QEMU w/ support for one form of virtio-msg-amp [4] > * Portable RTOS library w/ one form of virtio-msg-amp [5] > > In addition to the QEMU system based demos in the hvac-demo repo, we also have > two hardware systems running: > * AMD x86 + AMD Arm Versal connected via PCIe > * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr > > Please note that although the demos work, a few have not been aligned > with this version of the spec. > FWIW, Acked-by: Manivannan Sadhasivam <manivannan.sadhasivam@oss.qualcomm.com> - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills ` (7 preceding siblings ...) 2026-02-04 5:34 ` Manivannan Sadhasivam @ 2026-02-13 13:52 ` Parav Pandit 2026-02-13 19:45 ` Demi Marie Obenour ` (2 more replies) 2026-02-24 17:57 ` Demi Marie Obenour 9 siblings, 3 replies; 105+ messages in thread From: Parav Pandit @ 2026-02-13 13:52 UTC (permalink / raw) To: Bill Mills, virtio-comment@lists.linux.dev Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Bill, > From: Bill Mills <bill.mills@linaro.org> > Sent: 26 January 2026 10:02 PM > > This series adds the virtio-msg transport layer. > > The individuals and organizations involved in this effort have had difficulty in > using the existing virtio-transports in various situations and desire to add one > more transport that performs its transport layer operations by sending and > receiving messages. > > Implementations of virtio-msg will normally be done in multiple layers: > * common / device level > * bus level > > The common / device level defines the messages exchanged between the driver > and a device. This common part should lead to a common driver holding most > of the virtio specifics and can be shared by all virtio-msg bus implementations. > The kernel implementation in [3] shows this separation. As with other transport > layers, virtio-msg should not require modifications to existing virtio device > implementations (virtio-net, virtio-blk etc). The common / device level is the > main focus of this version of the patch series. > > The virtio-msg bus level implements the normal things a bus defines > (enumeration, dma operations, etc) but also implements the message send and > receive operations. A number of bus implementations are envisioned, > some of which will be reusable and general purpose. Other bus implementations > might be unique to a given situation, for example only used by a PCIe card > and its driver. > > The standard bus messages are an effort to avoid different bus implementations > doing the same thing in different ways for no good reason. However the > different environments will require different things. Instead of trying to > anticipate all needs and provide something very abstract, we think > implementation specific messages will be needed at the bus level. Over time, > if we see similar messages across multiple bus implementations, we will move to > standardize a bus level message for that. > I would review more, had first round of sparse review. Please find few comments/questions below. 1. device number should be 32-bit in struct virtio_msg_header. From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. Also mapping PCI BDF wont be enough in 16-bits considering domain field. 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. Needs 32-bits. 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) But more below. 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate the virtqueues. Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? Also, the queue notification message is missing at bus level. But I dont think queue notification via message good idea anyway. We rather need a more higher-level message for virtqueues. Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. Without this msg transport is incomplete. 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request for reads. 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. This requires caching all the values on the driver side before sending the transport request. I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. It may require more plumbing, but it is cleaner interface when a new transport binding is created. 8. We should have the message response header for req-resp method with status code. (even though they are plain MMIO writes for pci). As opposed to that, proposed messages are doing composite tasks, hence we need the status response. 9. virtio stack cannot handle hotplug devices today, at least Linux. So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. A graceful method is needed. > We are working on a few reusable bus implementations: > > * virtio-msg-ffa based on Arm FF-A interface for use between: > * normal world and secure world > * host and VM or VM to VM > * Can be used w/ or with out a hypervisor > * Any Hypervisor that implements FF-A can be used > * We have this working with pKVM and Xen > * We have this working with Trusty and OP-TEE > > * virtio-msg-amp for use between heterogenous systems > * The main processors and its co-processors on an AMP SOC > * Two or more systems connected via PCIe > * Minimal requirements: bi-directional interrupts and > at least one shared memory area > * hvac-demo has 2 demos of this > * This is working on two hardware platforms > > * virtio-msg-loopback for userspace implemented devices > * Allows user space to provide devices to its own kernel > * This is similar to fuse, cuse or loopback block devices but for virtio > * hvac-demo has a demo > > We also anticipate a few more: > > * virtio-msg-xen specific to Xen > * Usable on any Xen system (including x86 where FF-A does not exist) > * Using Xen events and page grants > > * virtio-msg over admin virtqueues > * This allows any virtio-pci device that supports admin virtqueues to also > support a virtio-msg bus that supports sub devices > * [We are looking for collaborators for this work] > > Changes since RFC2: > > Spec Functional: > * Made the common message header 8 bytes and added a token for optional use > when parallel outstanding requests are possible > * Made the 8 byte fields align to 8 byte offsets. > This effects the {SET,GET}_VQUEUE messages > * Many conformance cases have been tightened. > > Spec Editorial: > * Major restructure to better align with virtio spec > * Conformance model now matches other transports > * Use C structures to define messages instead of tables > * Added a section to describe how responses are matched to requests > This includes the use of the new token field > * Redefined / better described error handling between transport and bus > layers to eliminate the need for the bus to generate fake response messages > * Included editorial feedback from RFC2 > > Eco-system: > * Added virtio-msg-loopback demo > * Arm has published the first draft of the virtio-msg over FFA spec [6] > * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE > secure world > * LKML RFC has been sent [7] > * QEMU RFC has been sent [8] > > This is the first non-RFC patch series. The known short comings have been > addressed. We ask for review in earnest on this series and thank you for > any feedback you can provide. > > Background info and work in progress implementations: > * HVAC project page with intro slides [1] > * HVAC demo repo w/ instructions in README.md [2] > * Kernel w/ virtio-msg common level and ffa support [3] > * QEMU w/ support for one form of virtio-msg-amp [4] > * Portable RTOS library w/ one form of virtio-msg-amp [5] > > In addition to the QEMU system based demos in the hvac-demo repo, we also have > two hardware systems running: > * AMD x86 + AMD Arm Versal connected via PCIe > * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr > > Please note that although the demos work, a few have not been aligned > with this version of the spec. > > [1] > https://linaro.atlassian.net/wiki/spaces/HVAC/overview > 05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6390 > 50419790817373%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIs > IldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JnCO8QgAIiyGAprBLkprUd8A3162wCFpmGic3JKoHXo%3D&reserved=0 > [2] https://github.com/wmamills/hvac- > demo&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7 > C0%7C639050419790868944%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj > oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XeBuS8Hr%2FDETKfSyidKKGoGq6OTUcOYoeSGDouUMcf4%3D&reserved=0 > [3] > https://git.kernel.org/pub/scm/linux/kernel/git/vireshk%252 > Flinux.git%2Flog%2F%3Fh%3Dvirtio%2Fmsg&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d1 > 5727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419790913867%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PuLq68Y0ZNCYbfrgDZlipfm44xHZp1%2F%2 > BLu%2FaKQBBB%2Fo%3D&reserved=0 > [4] https://github.com/edgarigl/qemu/commits/edgar/virtio- > msg- > rfc%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0% > 7C0%7C639050419790960971%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFO > IjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xyowKD9KG3IasQTiYCzfCRGqCLSF%2Fa1g4%2F2%2ByKb1OoA%3D&reserved=0 > [5] https://github.com/arnopo/open-amp/commits/virtio- > msg%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0 > %7C0%7C639050419791007707%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF > OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=69OwO2wXi0%2FEhDDSR2uLG2iis5RYzI%2FLnlfkcRYao9o%3D&reserved=0 > [6] https://documentation-/ > service.arm.com%2Fstatic%2F68f647791134f773ab3f0a7c&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d9 > 5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419791052848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy > dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cD5VqKnTuIFqlN2XV3j%2FHQ > Wvsz8UVe5PM0ZjMU3o0sM%3D&reserved=0 > [7] > https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linar > o.org%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C > 0%7C0%7C639050419791093596%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NmhmL%2BuyVqnpHT5r9EfI2uzPf9LSXLga7bUw3FO5x4w%3D&reserved=0 > [8] https://mail.gnu.org/archive/html/qemu-devel/2025- > 10%2Fmsg07438.html&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd > 9ccc17a%7C0%7C0%7C639050419791139232%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJ > XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=osI1FttOO9%2B01e%2B5EtwnYe%2Fl1%2BIvFjH%2Blyz2ipM3RM > Q%3D&reserved=0 > > Bertrand Marquis (2): > virtio-msg: add new command for bus normative > virtio-msg: add conformance entries in conformance chapter > > Bill Mills (2): > virtio-msg: Add virtio-msg, a message based virtio transport layer > virtio-msg: link virtio-msg content > > commands.tex | 3 +- > conformance.tex | 105 ++- > content.tex | 1 + > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > 4 files changed, 1746 insertions(+), 3 deletions(-) > create mode 100644 transport-msg.tex > > -- > 2.34.1 > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-13 13:52 ` Parav Pandit @ 2026-02-13 19:45 ` Demi Marie Obenour 2026-02-19 17:31 ` Armelle Laine 2026-02-20 8:55 ` Bertrand Marquis 2026-02-19 23:54 ` Michael S. Tsirkin 2026-02-20 8:40 ` Bertrand Marquis 2 siblings, 2 replies; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-13 19:45 UTC (permalink / raw) To: Parav Pandit, Bill Mills, virtio-comment@lists.linux.dev Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 2668 bytes --] On 2/13/26 08:52, Parav Pandit wrote: > Hi Bill, > >> From: Bill Mills <bill.mills@linaro.org> >> Sent: 26 January 2026 10:02 PM >> >> This series adds the virtio-msg transport layer. >> >> The individuals and organizations involved in this effort have had difficulty in >> using the existing virtio-transports in various situations and desire to add one >> more transport that performs its transport layer operations by sending and >> receiving messages. >> >> Implementations of virtio-msg will normally be done in multiple layers: >> * common / device level >> * bus level >> >> The common / device level defines the messages exchanged between the driver >> and a device. This common part should lead to a common driver holding most >> of the virtio specifics and can be shared by all virtio-msg bus implementations. >> The kernel implementation in [3] shows this separation. As with other transport >> layers, virtio-msg should not require modifications to existing virtio device >> implementations (virtio-net, virtio-blk etc). The common / device level is the >> main focus of this version of the patch series. >> >> The virtio-msg bus level implements the normal things a bus defines >> (enumeration, dma operations, etc) but also implements the message send and >> receive operations. A number of bus implementations are envisioned, >> some of which will be reusable and general purpose. Other bus implementations >> might be unique to a given situation, for example only used by a PCIe card >> and its driver. >> >> The standard bus messages are an effort to avoid different bus implementations >> doing the same thing in different ways for no good reason. However the >> different environments will require different things. Instead of trying to >> anticipate all needs and provide something very abstract, we think >> implementation specific messages will be needed at the bus level. Over time, >> if we see similar messages across multiple bus implementations, we will move to >> standardize a bus level message for that. >> > > I would review more, had first round of sparse review. > Please find few comments/questions below. > > 1. device number should be 32-bit in struct virtio_msg_header. > From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > Also mapping PCI BDF wont be enough in 16-bits considering domain field. I recommend using a 64-bit or even 128-bit number. This allows ensuring that device IDs are unique for the life of a VM or even globally, making reasoning about device identity simpler. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-13 19:45 ` Demi Marie Obenour @ 2026-02-19 17:31 ` Armelle Laine 2026-02-20 8:55 ` Bertrand Marquis 1 sibling, 0 replies; 105+ messages in thread From: Armelle Laine @ 2026-02-19 17:31 UTC (permalink / raw) To: Demi Marie Obenour Cc: Parav Pandit, Bill Mills, virtio-comment@lists.linux.dev, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee [-- Attachment #1: Type: text/plain, Size: 3302 bytes --] Hi everyone, We are planning to release virtio-msg over FFA for Android Virtualization Framework (AVF) pVM to TrustZone communication in the next Android release. The Guest OS is Trusty OS, and the virtio-vsock over virtio-msg over FFA driver/device support will be upstreamed to AOSP at the time of the release. Acked-by: Armelle Laine armellel@google.com Thanks, Armelle On Fri, Feb 13, 2026 at 11:45 AM Demi Marie Obenour <demiobenour@gmail.com> wrote: > On 2/13/26 08:52, Parav Pandit wrote: > > Hi Bill, > > > >> From: Bill Mills <bill.mills@linaro.org> > >> Sent: 26 January 2026 10:02 PM > >> > >> This series adds the virtio-msg transport layer. > >> > >> The individuals and organizations involved in this effort have had > difficulty in > >> using the existing virtio-transports in various situations and desire > to add one > >> more transport that performs its transport layer operations by sending > and > >> receiving messages. > >> > >> Implementations of virtio-msg will normally be done in multiple layers: > >> * common / device level > >> * bus level > >> > >> The common / device level defines the messages exchanged between the > driver > >> and a device. This common part should lead to a common driver holding > most > >> of the virtio specifics and can be shared by all virtio-msg bus > implementations. > >> The kernel implementation in [3] shows this separation. As with other > transport > >> layers, virtio-msg should not require modifications to existing virtio > device > >> implementations (virtio-net, virtio-blk etc). The common / device level > is the > >> main focus of this version of the patch series. > >> > >> The virtio-msg bus level implements the normal things a bus defines > >> (enumeration, dma operations, etc) but also implements the message send > and > >> receive operations. A number of bus implementations are envisioned, > >> some of which will be reusable and general purpose. Other bus > implementations > >> might be unique to a given situation, for example only used by a PCIe > card > >> and its driver. > >> > >> The standard bus messages are an effort to avoid different bus > implementations > >> doing the same thing in different ways for no good reason. However the > >> different environments will require different things. Instead of trying > to > >> anticipate all needs and provide something very abstract, we think > >> implementation specific messages will be needed at the bus level. Over > time, > >> if we see similar messages across multiple bus implementations, we will > move to > >> standardize a bus level message for that. > >> > > > > I would review more, had first round of sparse review. > > Please find few comments/questions below. > > > > 1. device number should be 32-bit in struct virtio_msg_header. > > From SIOV_R2 experiences, we learnt that some uses have use case for > more than 64k devices. > > Also mapping PCI BDF wont be enough in 16-bits considering domain field. > > I recommend using a 64-bit or even 128-bit number. This allows > ensuring that device IDs are unique for the life of a VM or even > globally, making reasoning about device identity simpler. > -- > Sincerely, > Demi Marie Obenour (she/her/hers) [-- Attachment #2: Type: text/html, Size: 4095 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-13 19:45 ` Demi Marie Obenour 2026-02-19 17:31 ` Armelle Laine @ 2026-02-20 8:55 ` Bertrand Marquis 1 sibling, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-20 8:55 UTC (permalink / raw) To: Demi Marie Obenour Cc: Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 13 Feb 2026, at 20:45, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 2/13/26 08:52, Parav Pandit wrote: >> Hi Bill, >> >>> From: Bill Mills <bill.mills@linaro.org> >>> Sent: 26 January 2026 10:02 PM >>> >>> This series adds the virtio-msg transport layer. >>> >>> The individuals and organizations involved in this effort have had difficulty in >>> using the existing virtio-transports in various situations and desire to add one >>> more transport that performs its transport layer operations by sending and >>> receiving messages. >>> >>> Implementations of virtio-msg will normally be done in multiple layers: >>> * common / device level >>> * bus level >>> >>> The common / device level defines the messages exchanged between the driver >>> and a device. This common part should lead to a common driver holding most >>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>> The kernel implementation in [3] shows this separation. As with other transport >>> layers, virtio-msg should not require modifications to existing virtio device >>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>> main focus of this version of the patch series. >>> >>> The virtio-msg bus level implements the normal things a bus defines >>> (enumeration, dma operations, etc) but also implements the message send and >>> receive operations. A number of bus implementations are envisioned, >>> some of which will be reusable and general purpose. Other bus implementations >>> might be unique to a given situation, for example only used by a PCIe card >>> and its driver. >>> >>> The standard bus messages are an effort to avoid different bus implementations >>> doing the same thing in different ways for no good reason. However the >>> different environments will require different things. Instead of trying to >>> anticipate all needs and provide something very abstract, we think >>> implementation specific messages will be needed at the bus level. Over time, >>> if we see similar messages across multiple bus implementations, we will move to >>> standardize a bus level message for that. >>> >> >> I would review more, had first round of sparse review. >> Please find few comments/questions below. >> >> 1. device number should be 32-bit in struct virtio_msg_header. >> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > > I recommend using a 64-bit or even 128-bit number. This allows > ensuring that device IDs are unique for the life of a VM or even > globally, making reasoning about device identity simpler. I will look into that but have a 128-bit number might introduce an header to big for some environments where the maximum message size is constrained (FF-A with direct message is constrained to 120 bytes, use cases using an hardware messaging system might also be constrained) so i will dig a bit to see the consequences. Now preventing to reuse the device ID could definitely solve some potential issues and apparently there are also use cases with a great number of devices that we need to cover. 64-bit should be possible and i will check if we could have some provision for 128-bit. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-13 13:52 ` Parav Pandit 2026-02-13 19:45 ` Demi Marie Obenour @ 2026-02-19 23:54 ` Michael S. Tsirkin 2026-02-20 6:13 ` Parav Pandit 2026-02-20 8:58 ` Bertrand Marquis 2026-02-20 8:40 ` Bertrand Marquis 2 siblings, 2 replies; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-19 23:54 UTC (permalink / raw) To: Parav Pandit Cc: Bill Mills, virtio-comment@lists.linux.dev, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > Hi Bill, > > > From: Bill Mills <bill.mills@linaro.org> > > Sent: 26 January 2026 10:02 PM > > > > This series adds the virtio-msg transport layer. > > > > The individuals and organizations involved in this effort have had difficulty in > > using the existing virtio-transports in various situations and desire to add one > > more transport that performs its transport layer operations by sending and > > receiving messages. > > > > Implementations of virtio-msg will normally be done in multiple layers: > > * common / device level > > * bus level > > > > The common / device level defines the messages exchanged between the driver > > and a device. This common part should lead to a common driver holding most > > of the virtio specifics and can be shared by all virtio-msg bus implementations. > > The kernel implementation in [3] shows this separation. As with other transport > > layers, virtio-msg should not require modifications to existing virtio device > > implementations (virtio-net, virtio-blk etc). The common / device level is the > > main focus of this version of the patch series. > > > > The virtio-msg bus level implements the normal things a bus defines > > (enumeration, dma operations, etc) but also implements the message send and > > receive operations. A number of bus implementations are envisioned, > > some of which will be reusable and general purpose. Other bus implementations > > might be unique to a given situation, for example only used by a PCIe card > > and its driver. > > > > The standard bus messages are an effort to avoid different bus implementations > > doing the same thing in different ways for no good reason. However the > > different environments will require different things. Instead of trying to > > anticipate all needs and provide something very abstract, we think > > implementation specific messages will be needed at the bus level. Over time, > > if we see similar messages across multiple bus implementations, we will move to > > standardize a bus level message for that. > > > > I would review more, had first round of sparse review. > Please find few comments/questions below. I'd like to comment that I think it makes sense to have a basic simple transport and then add performance features on top as appropriate. So one way to address some of these comments is to show how they can be addressed with a feature bit down the road. > 1. device number should be 32-bit in struct virtio_msg_header. > >From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > Also mapping PCI BDF wont be enough in 16-bits considering domain field. > > 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > Needs 32-bits. > > 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > But more below. > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate the virtqueues. > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? surely this can be an optional transport feature bit. > Also, the queue notification message is missing at bus level. > But I dont think queue notification via message good idea anyway. > We rather need a more higher-level message for virtqueues. > Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. > Without this msg transport is incomplete. this too can be a basic method and upgrade to a different one with a feature negotiation > 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in > > 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request for reads. here too, relaxed ordering could be a feature bit. > 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. > This requires caching all the values on the driver side before sending the transport request. > I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. > and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. > It may require more plumbing, but it is cleaner interface when a new transport binding is created. > > 8. We should have the message response header for req-resp method with status code. > (even though they are plain MMIO writes for pci). > As opposed to that, proposed messages are doing composite tasks, hence we need the status response. > > 9. virtio stack cannot handle hotplug devices today, at least Linux. > So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. > A graceful method is needed. > > > We are working on a few reusable bus implementations: > > > > * virtio-msg-ffa based on Arm FF-A interface for use between: > > * normal world and secure world > > * host and VM or VM to VM > > * Can be used w/ or with out a hypervisor > > * Any Hypervisor that implements FF-A can be used > > * We have this working with pKVM and Xen > > * We have this working with Trusty and OP-TEE > > > > * virtio-msg-amp for use between heterogenous systems > > * The main processors and its co-processors on an AMP SOC > > * Two or more systems connected via PCIe > > * Minimal requirements: bi-directional interrupts and > > at least one shared memory area > > * hvac-demo has 2 demos of this > > * This is working on two hardware platforms > > > > * virtio-msg-loopback for userspace implemented devices > > * Allows user space to provide devices to its own kernel > > * This is similar to fuse, cuse or loopback block devices but for virtio > > * hvac-demo has a demo > > > > We also anticipate a few more: > > > > * virtio-msg-xen specific to Xen > > * Usable on any Xen system (including x86 where FF-A does not exist) > > * Using Xen events and page grants > > > > * virtio-msg over admin virtqueues > > * This allows any virtio-pci device that supports admin virtqueues to also > > support a virtio-msg bus that supports sub devices > > * [We are looking for collaborators for this work] > > > > Changes since RFC2: > > > > Spec Functional: > > * Made the common message header 8 bytes and added a token for optional use > > when parallel outstanding requests are possible > > * Made the 8 byte fields align to 8 byte offsets. > > This effects the {SET,GET}_VQUEUE messages > > * Many conformance cases have been tightened. > > > > Spec Editorial: > > * Major restructure to better align with virtio spec > > * Conformance model now matches other transports > > * Use C structures to define messages instead of tables > > * Added a section to describe how responses are matched to requests > > This includes the use of the new token field > > * Redefined / better described error handling between transport and bus > > layers to eliminate the need for the bus to generate fake response messages > > * Included editorial feedback from RFC2 > > > > Eco-system: > > * Added virtio-msg-loopback demo > > * Arm has published the first draft of the virtio-msg over FFA spec [6] > > * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE > > secure world > > * LKML RFC has been sent [7] > > * QEMU RFC has been sent [8] > > > > This is the first non-RFC patch series. The known short comings have been > > addressed. We ask for review in earnest on this series and thank you for > > any feedback you can provide. > > > > Background info and work in progress implementations: > > * HVAC project page with intro slides [1] > > * HVAC demo repo w/ instructions in README.md [2] > > * Kernel w/ virtio-msg common level and ffa support [3] > > * QEMU w/ support for one form of virtio-msg-amp [4] > > * Portable RTOS library w/ one form of virtio-msg-amp [5] > > > > In addition to the QEMU system based demos in the hvac-demo repo, we also have > > two hardware systems running: > > * AMD x86 + AMD Arm Versal connected via PCIe > > * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr > > > > Please note that although the demos work, a few have not been aligned > > with this version of the spec. > > > > [1] > > https://linaro.atlassian.net/wiki/spaces/HVAC/overview > > 05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6390 > > 50419790817373%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIs > > IldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JnCO8QgAIiyGAprBLkprUd8A3162wCFpmGic3JKoHXo%3D&reserved=0 > > [2] https://github.com/wmamills/hvac- > > demo&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7 > > C0%7C639050419790868944%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj > > oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XeBuS8Hr%2FDETKfSyidKKGoGq6OTUcOYoeSGDouUMcf4%3D&reserved=0 > > [3] > > https://git.kernel.org/pub/scm/linux/kernel/git/vireshk%252 > > Flinux.git%2Flog%2F%3Fh%3Dvirtio%2Fmsg&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d1 > > 5727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419790913867%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw > > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PuLq68Y0ZNCYbfrgDZlipfm44xHZp1%2F%2 > > BLu%2FaKQBBB%2Fo%3D&reserved=0 > > [4] https://github.com/edgarigl/qemu/commits/edgar/virtio- > > msg- > > rfc%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0% > > 7C0%7C639050419790960971%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFO > > IjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xyowKD9KG3IasQTiYCzfCRGqCLSF%2Fa1g4%2F2%2ByKb1OoA%3D&reserved=0 > > [5] https://github.com/arnopo/open-amp/commits/virtio- > > msg%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0 > > %7C0%7C639050419791007707%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF > > OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=69OwO2wXi0%2FEhDDSR2uLG2iis5RYzI%2FLnlfkcRYao9o%3D&reserved=0 > > [6] https://documentation-/ > > service.arm.com%2Fstatic%2F68f647791134f773ab3f0a7c&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d9 > > 5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419791052848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy > > dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cD5VqKnTuIFqlN2XV3j%2FHQ > > Wvsz8UVe5PM0ZjMU3o0sM%3D&reserved=0 > > [7] > > https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linar > > o.org%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C > > 0%7C0%7C639050419791093596%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NmhmL%2BuyVqnpHT5r9EfI2uzPf9LSXLga7bUw3FO5x4w%3D&reserved=0 > > [8] https://mail.gnu.org/archive/html/qemu-devel/2025- > > 10%2Fmsg07438.html&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd > > 9ccc17a%7C0%7C0%7C639050419791139232%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJ > > XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=osI1FttOO9%2B01e%2B5EtwnYe%2Fl1%2BIvFjH%2Blyz2ipM3RM > > Q%3D&reserved=0 > > > > Bertrand Marquis (2): > > virtio-msg: add new command for bus normative > > virtio-msg: add conformance entries in conformance chapter > > > > Bill Mills (2): > > virtio-msg: Add virtio-msg, a message based virtio transport layer > > virtio-msg: link virtio-msg content > > > > commands.tex | 3 +- > > conformance.tex | 105 ++- > > content.tex | 1 + > > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > > 4 files changed, 1746 insertions(+), 3 deletions(-) > > create mode 100644 transport-msg.tex > > > > -- > > 2.34.1 > > > ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-19 23:54 ` Michael S. Tsirkin @ 2026-02-20 6:13 ` Parav Pandit 2026-02-20 9:02 ` Bertrand Marquis 2026-02-20 10:03 ` Michael S. Tsirkin 2026-02-20 8:58 ` Bertrand Marquis 1 sibling, 2 replies; 105+ messages in thread From: Parav Pandit @ 2026-02-20 6:13 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Bill Mills, virtio-comment@lists.linux.dev, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Michael S. Tsirkin <mst@redhat.com> > Sent: 20 February 2026 05:25 AM > > On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > > Hi Bill, > > > > > From: Bill Mills <bill.mills@linaro.org> > > > Sent: 26 January 2026 10:02 PM > > > > > > This series adds the virtio-msg transport layer. > > > > > > The individuals and organizations involved in this effort have had difficulty in > > > using the existing virtio-transports in various situations and desire to add one > > > more transport that performs its transport layer operations by sending and > > > receiving messages. > > > > > > Implementations of virtio-msg will normally be done in multiple layers: > > > * common / device level > > > * bus level > > > > > > The common / device level defines the messages exchanged between the driver > > > and a device. This common part should lead to a common driver holding most > > > of the virtio specifics and can be shared by all virtio-msg bus implementations. > > > The kernel implementation in [3] shows this separation. As with other transport > > > layers, virtio-msg should not require modifications to existing virtio device > > > implementations (virtio-net, virtio-blk etc). The common / device level is the > > > main focus of this version of the patch series. > > > > > > The virtio-msg bus level implements the normal things a bus defines > > > (enumeration, dma operations, etc) but also implements the message send and > > > receive operations. A number of bus implementations are envisioned, > > > some of which will be reusable and general purpose. Other bus implementations > > > might be unique to a given situation, for example only used by a PCIe card > > > and its driver. > > > > > > The standard bus messages are an effort to avoid different bus implementations > > > doing the same thing in different ways for no good reason. However the > > > different environments will require different things. Instead of trying to > > > anticipate all needs and provide something very abstract, we think > > > implementation specific messages will be needed at the bus level. Over time, > > > if we see similar messages across multiple bus implementations, we will move to > > > standardize a bus level message for that. > > > > > > > I would review more, had first round of sparse review. > > Please find few comments/questions below. > > I'd like to comment that I think it makes sense to have a basic simple transport and > then add performance features on top as appropriate. Sounds good. Simple but complete is needed. > So one way to address some of these comments is to show how > they can be addressed with a feature bit down the road. > > > > 1. device number should be 32-bit in struct virtio_msg_header. > > >From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > > Also mapping PCI BDF wont be enough in 16-bits considering domain field. > > > > 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > > For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > > Needs 32-bits. > > > > 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > > But more below. > > > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate > the virtqueues. > > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > surely this can be an optional transport feature bit. > How is this optional? How can one implement a transport without defining the basic data transfer semantics? For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. How does it work? > > Also, the queue notification message is missing at bus level. > > But I dont think queue notification via message good idea anyway. > > We rather need a more higher-level message for virtqueues. > > Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. > > Without this msg transport is incomplete. > > this too can be a basic method and upgrade to a different one > with a feature negotiation > > > > 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in > > > > 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request > for reads. > > here too, relaxed ordering could be a feature bit. > Well the newly defined transport scheme is not even able to fit for the existing transport of PCI. The BAR for new transport definition can't be this low.. But before we go there, I would like to see the bus binding details for the data transfer semantics. Without this simple thing is incomplete. Will wait for Bill to respond. > > 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. > > This requires caching all the values on the driver side before sending the transport request. > > I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. > > and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. > > It may require more plumbing, but it is cleaner interface when a new transport binding is created. > > > > 8. We should have the message response header for req-resp method with status code. > > (even though they are plain MMIO writes for pci). > > As opposed to that, proposed messages are doing composite tasks, hence we need the status response. > > > > 9. virtio stack cannot handle hotplug devices today, at least Linux. > > So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. > > A graceful method is needed. > > > > > We are working on a few reusable bus implementations: > > > > > > * virtio-msg-ffa based on Arm FF-A interface for use between: > > > * normal world and secure world > > > * host and VM or VM to VM > > > * Can be used w/ or with out a hypervisor > > > * Any Hypervisor that implements FF-A can be used > > > * We have this working with pKVM and Xen > > > * We have this working with Trusty and OP-TEE > > > > > > * virtio-msg-amp for use between heterogenous systems > > > * The main processors and its co-processors on an AMP SOC > > > * Two or more systems connected via PCIe > > > * Minimal requirements: bi-directional interrupts and > > > at least one shared memory area > > > * hvac-demo has 2 demos of this > > > * This is working on two hardware platforms > > > > > > * virtio-msg-loopback for userspace implemented devices > > > * Allows user space to provide devices to its own kernel > > > * This is similar to fuse, cuse or loopback block devices but for virtio > > > * hvac-demo has a demo > > > > > > We also anticipate a few more: > > > > > > * virtio-msg-xen specific to Xen > > > * Usable on any Xen system (including x86 where FF-A does not exist) > > > * Using Xen events and page grants > > > > > > * virtio-msg over admin virtqueues > > > * This allows any virtio-pci device that supports admin virtqueues to also > > > support a virtio-msg bus that supports sub devices > > > * [We are looking for collaborators for this work] > > > > > > Changes since RFC2: > > > > > > Spec Functional: > > > * Made the common message header 8 bytes and added a token for optional use > > > when parallel outstanding requests are possible > > > * Made the 8 byte fields align to 8 byte offsets. > > > This effects the {SET,GET}_VQUEUE messages > > > * Many conformance cases have been tightened. > > > > > > Spec Editorial: > > > * Major restructure to better align with virtio spec > > > * Conformance model now matches other transports > > > * Use C structures to define messages instead of tables > > > * Added a section to describe how responses are matched to requests > > > This includes the use of the new token field > > > * Redefined / better described error handling between transport and bus > > > layers to eliminate the need for the bus to generate fake response messages > > > * Included editorial feedback from RFC2 > > > > > > Eco-system: > > > * Added virtio-msg-loopback demo > > > * Arm has published the first draft of the virtio-msg over FFA spec [6] > > > * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE > > > secure world > > > * LKML RFC has been sent [7] > > > * QEMU RFC has been sent [8] > > > > > > This is the first non-RFC patch series. The known short comings have been > > > addressed. We ask for review in earnest on this series and thank you for > > > any feedback you can provide. > > > > > > Background info and work in progress implementations: > > > * HVAC project page with intro slides [1] > > > * HVAC demo repo w/ instructions in README.md [2] > > > * Kernel w/ virtio-msg common level and ffa support [3] > > > * QEMU w/ support for one form of virtio-msg-amp [4] > > > * Portable RTOS library w/ one form of virtio-msg-amp [5] > > > > > > In addition to the QEMU system based demos in the hvac-demo repo, we also have > > > two hardware systems running: > > > * AMD x86 + AMD Arm Versal connected via PCIe > > > * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr > > > > > > Please note that although the demos work, a few have not been aligned > > > with this version of the spec. > > > > > > [1] > > > https://linaro.atlassian.net/wiki/spaces/HVAC/overview > > > > 05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6390 > > > > 50419790817373%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIs > > > IldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JnCO8QgAIiyGAprBLkprUd8A3162wCFpmGic3JKoHXo%3D&reserved=0 > > > [2] https://github.com/wmamills/hvac- > > > > demo&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7 > > > > C0%7C639050419790868944%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj > > > oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XeBuS8Hr%2FDETKfSyidKKGoGq6OTUcOYoeSGDouUMcf4%3D&reserved=0 > > > [3] > > > https://git.kernel.org/pub/scm/linux/kernel/git/vireshk%252 > > > > Flinux.git%2Flog%2F%3Fh%3Dvirtio%2Fmsg&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d1 > > > > 5727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419790913867%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw > > > > LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PuLq68Y0ZNCYbfrgDZlipfm44xHZp1%2F%2 > > > BLu%2FaKQBBB%2Fo%3D&reserved=0 > > > [4] https://github.com/edgarigl/qemu/commits/edgar/virtio- > > > msg- > > > > rfc%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0% > > > > 7C0%7C639050419790960971%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFO > > > > IjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xyowKD9KG3IasQTiYCzfCRGqCLSF%2Fa1g4%2F2%2ByKb1OoA%3D&reserved=0 > > > [5] https://github.com/arnopo/open-amp/commits/virtio- > > > > msg%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0 > > > > %7C0%7C639050419791007707%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF > > > OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=69OwO2wXi0%2FEhDDSR2uLG2iis5RYzI%2FLnlfkcRYao9o%3D&reserved=0 > > > [6] https://documentation-/ > > > > service.arm.com%2Fstatic%2F68f647791134f773ab3f0a7c&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d9 > > > > 5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419791052848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy > > > > dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cD5VqKnTuIFqlN2XV3j%2FHQ > > > Wvsz8UVe5PM0ZjMU3o0sM%3D&reserved=0 > > > [7] > > > https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linar > > > > o.org%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C > > > > 0%7C0%7C639050419791093596%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI > > > > kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NmhmL%2BuyVqnpHT5r9EfI2uzPf9LSXLga7bUw3FO5x4w%3D&reserved=0 > > > [8] https://mail.gnu.org/archive/html/qemu-devel/2025- > > > > 10%2Fmsg07438.html&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd > > > > 9ccc17a%7C0%7C0%7C639050419791139232%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJ > > > > XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=osI1FttOO9%2B01e%2B5EtwnYe%2Fl1%2BIvFjH%2Blyz2ipM3RM > > > Q%3D&reserved=0 > > > > > > Bertrand Marquis (2): > > > virtio-msg: add new command for bus normative > > > virtio-msg: add conformance entries in conformance chapter > > > > > > Bill Mills (2): > > > virtio-msg: Add virtio-msg, a message based virtio transport layer > > > virtio-msg: link virtio-msg content > > > > > > commands.tex | 3 +- > > > conformance.tex | 105 ++- > > > content.tex | 1 + > > > transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ > > > 4 files changed, 1746 insertions(+), 3 deletions(-) > > > create mode 100644 transport-msg.tex > > > > > > -- > > > 2.34.1 > > > > > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-20 6:13 ` Parav Pandit @ 2026-02-20 9:02 ` Bertrand Marquis 2026-02-25 7:45 ` Manivannan Sadhasivam 2026-02-20 10:03 ` Michael S. Tsirkin 1 sibling, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-20 9:02 UTC (permalink / raw) To: Parav Pandit Cc: Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Parav, > On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > > > >> From: Michael S. Tsirkin <mst@redhat.com> >> Sent: 20 February 2026 05:25 AM >> >> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: >>> Hi Bill, >>> >>>> From: Bill Mills <bill.mills@linaro.org> >>>> Sent: 26 January 2026 10:02 PM >>>> >>>> This series adds the virtio-msg transport layer. >>>> >>>> The individuals and organizations involved in this effort have had difficulty in >>>> using the existing virtio-transports in various situations and desire to add one >>>> more transport that performs its transport layer operations by sending and >>>> receiving messages. >>>> >>>> Implementations of virtio-msg will normally be done in multiple layers: >>>> * common / device level >>>> * bus level >>>> >>>> The common / device level defines the messages exchanged between the driver >>>> and a device. This common part should lead to a common driver holding most >>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>> The kernel implementation in [3] shows this separation. As with other transport >>>> layers, virtio-msg should not require modifications to existing virtio device >>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>> main focus of this version of the patch series. >>>> >>>> The virtio-msg bus level implements the normal things a bus defines >>>> (enumeration, dma operations, etc) but also implements the message send and >>>> receive operations. A number of bus implementations are envisioned, >>>> some of which will be reusable and general purpose. Other bus implementations >>>> might be unique to a given situation, for example only used by a PCIe card >>>> and its driver. >>>> >>>> The standard bus messages are an effort to avoid different bus implementations >>>> doing the same thing in different ways for no good reason. However the >>>> different environments will require different things. Instead of trying to >>>> anticipate all needs and provide something very abstract, we think >>>> implementation specific messages will be needed at the bus level. Over time, >>>> if we see similar messages across multiple bus implementations, we will move to >>>> standardize a bus level message for that. >>>> >>> >>> I would review more, had first round of sparse review. >>> Please find few comments/questions below. >> >> I'd like to comment that I think it makes sense to have a basic simple transport and >> then add performance features on top as appropriate. > Sounds good. Simple but complete is needed. Agree. > >> So one way to address some of these comments is to show how >> they can be addressed with a feature bit down the road. >> >> >>> 1. device number should be 32-bit in struct virtio_msg_header. >>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>> >>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>> Needs 32-bits. >>> >>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>> But more below. >>> >>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate >> the virtqueues. >>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >> >> surely this can be an optional transport feature bit. >> > How is this optional? As said in a previous mail, we have messages already for that. Please confirm if that answer your question. > How can one implement a transport without defining the basic data transfer semantics? We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. If anything is missing, we are more than happy to discuss it and solve the issue. > For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. > How does it work? As said, this is a use case I am expecting is working using virtqueue, not MMIO access and this is how virtio-net or vsock are working. Maybe you are speaking of a use case we missed, could you elaborate ? > >>> Also, the queue notification message is missing at bus level. >>> But I dont think queue notification via message good idea anyway. >>> We rather need a more higher-level message for virtqueues. >>> Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. >>> Without this msg transport is incomplete. >> >> this too can be a basic method and upgrade to a different one >> with a feature negotiation >> >> >>> 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in >>> >>> 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request >> for reads. >> >> here too, relaxed ordering could be a feature bit. >> > Well the newly defined transport scheme is not even able to fit for the existing transport of PCI. > The BAR for new transport definition can't be this low.. We think the new transport does bring features equivalent to PCI or MMIO but if we missed anything we are more than happy to handle it. > > But before we go there, I would like to see the bus binding details for the data transfer semantics. > Without this simple thing is incomplete. I tried to answer all your questions, if I missed something please tell me. Regards Bertrand > > Will wait for Bill to respond. > >>> 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. >>> This requires caching all the values on the driver side before sending the transport request. >>> I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. >>> and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. >>> It may require more plumbing, but it is cleaner interface when a new transport binding is created. >>> >>> 8. We should have the message response header for req-resp method with status code. >>> (even though they are plain MMIO writes for pci). >>> As opposed to that, proposed messages are doing composite tasks, hence we need the status response. >>> >>> 9. virtio stack cannot handle hotplug devices today, at least Linux. >>> So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. >>> A graceful method is needed. >>> >>>> We are working on a few reusable bus implementations: >>>> >>>> * virtio-msg-ffa based on Arm FF-A interface for use between: >>>> * normal world and secure world >>>> * host and VM or VM to VM >>>> * Can be used w/ or with out a hypervisor >>>> * Any Hypervisor that implements FF-A can be used >>>> * We have this working with pKVM and Xen >>>> * We have this working with Trusty and OP-TEE >>>> >>>> * virtio-msg-amp for use between heterogenous systems >>>> * The main processors and its co-processors on an AMP SOC >>>> * Two or more systems connected via PCIe >>>> * Minimal requirements: bi-directional interrupts and >>>> at least one shared memory area >>>> * hvac-demo has 2 demos of this >>>> * This is working on two hardware platforms >>>> >>>> * virtio-msg-loopback for userspace implemented devices >>>> * Allows user space to provide devices to its own kernel >>>> * This is similar to fuse, cuse or loopback block devices but for virtio >>>> * hvac-demo has a demo >>>> >>>> We also anticipate a few more: >>>> >>>> * virtio-msg-xen specific to Xen >>>> * Usable on any Xen system (including x86 where FF-A does not exist) >>>> * Using Xen events and page grants >>>> >>>> * virtio-msg over admin virtqueues >>>> * This allows any virtio-pci device that supports admin virtqueues to also >>>> support a virtio-msg bus that supports sub devices >>>> * [We are looking for collaborators for this work] >>>> >>>> Changes since RFC2: >>>> >>>> Spec Functional: >>>> * Made the common message header 8 bytes and added a token for optional use >>>> when parallel outstanding requests are possible >>>> * Made the 8 byte fields align to 8 byte offsets. >>>> This effects the {SET,GET}_VQUEUE messages >>>> * Many conformance cases have been tightened. >>>> >>>> Spec Editorial: >>>> * Major restructure to better align with virtio spec >>>> * Conformance model now matches other transports >>>> * Use C structures to define messages instead of tables >>>> * Added a section to describe how responses are matched to requests >>>> This includes the use of the new token field >>>> * Redefined / better described error handling between transport and bus >>>> layers to eliminate the need for the bus to generate fake response messages >>>> * Included editorial feedback from RFC2 >>>> >>>> Eco-system: >>>> * Added virtio-msg-loopback demo >>>> * Arm has published the first draft of the virtio-msg over FFA spec [6] >>>> * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE >>>> secure world >>>> * LKML RFC has been sent [7] >>>> * QEMU RFC has been sent [8] >>>> >>>> This is the first non-RFC patch series. The known short comings have been >>>> addressed. We ask for review in earnest on this series and thank you for >>>> any feedback you can provide. >>>> >>>> Background info and work in progress implementations: >>>> * HVAC project page with intro slides [1] >>>> * HVAC demo repo w/ instructions in README.md [2] >>>> * Kernel w/ virtio-msg common level and ffa support [3] >>>> * QEMU w/ support for one form of virtio-msg-amp [4] >>>> * Portable RTOS library w/ one form of virtio-msg-amp [5] >>>> >>>> In addition to the QEMU system based demos in the hvac-demo repo, we also have >>>> two hardware systems running: >>>> * AMD x86 + AMD Arm Versal connected via PCIe >>>> * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr >>>> >>>> Please note that although the demos work, a few have not been aligned >>>> with this version of the spec. >>>> >>>> [1] >>>> https://linaro.atlassian.net/wiki/spaces/HVAC/overview >>>> >> 05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6390 >>>> >> 50419790817373%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIs >>>> IldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JnCO8QgAIiyGAprBLkprUd8A3162wCFpmGic3JKoHXo%3D&reserved=0 >>>> [2] https://github.com/wmamills/hvac- >>>> >> demo&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7 >>>> >> C0%7C639050419790868944%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj >>>> oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XeBuS8Hr%2FDETKfSyidKKGoGq6OTUcOYoeSGDouUMcf4%3D&reserved=0 >>>> [3] >>>> https://git.kernel.org/pub/scm/linux/kernel/git/vireshk%252 >>>> >> Flinux.git%2Flog%2F%3Fh%3Dvirtio%2Fmsg&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d1 >>>> >> 5727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419790913867%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw >>>> >> LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PuLq68Y0ZNCYbfrgDZlipfm44xHZp1%2F%2 >>>> BLu%2FaKQBBB%2Fo%3D&reserved=0 >>>> [4] https://github.com/edgarigl/qemu/commits/edgar/virtio- >>>> msg- >>>> >> rfc%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0% >>>> >> 7C0%7C639050419790960971%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFO >>>> >> IjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xyowKD9KG3IasQTiYCzfCRGqCLSF%2Fa1g4%2F2%2ByKb1OoA%3D&reserved=0 >>>> [5] https://github.com/arnopo/open-amp/commits/virtio- >>>> >> msg%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0 >>>> >> %7C0%7C639050419791007707%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF >>>> OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=69OwO2wXi0%2FEhDDSR2uLG2iis5RYzI%2FLnlfkcRYao9o%3D&reserved=0 >>>> [6] https://documentation-/ >>>> >> service.arm.com%2Fstatic%2F68f647791134f773ab3f0a7c&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d9 >>>> >> 5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419791052848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy >>>> >> dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cD5VqKnTuIFqlN2XV3j%2FHQ >>>> Wvsz8UVe5PM0ZjMU3o0sM%3D&reserved=0 >>>> [7] >>>> https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linar >>>> >> o.org%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C >>>> >> 0%7C0%7C639050419791093596%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI >>>> >> kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NmhmL%2BuyVqnpHT5r9EfI2uzPf9LSXLga7bUw3FO5x4w%3D&reserved=0 >>>> [8] https://mail.gnu.org/archive/html/qemu-devel/2025- >>>> >> 10%2Fmsg07438.html&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd >>>> >> 9ccc17a%7C0%7C0%7C639050419791139232%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJ >>>> >> XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=osI1FttOO9%2B01e%2B5EtwnYe%2Fl1%2BIvFjH%2Blyz2ipM3RM >>>> Q%3D&reserved=0 >>>> >>>> Bertrand Marquis (2): >>>> virtio-msg: add new command for bus normative >>>> virtio-msg: add conformance entries in conformance chapter >>>> >>>> Bill Mills (2): >>>> virtio-msg: Add virtio-msg, a message based virtio transport layer >>>> virtio-msg: link virtio-msg content >>>> >>>> commands.tex | 3 +- >>>> conformance.tex | 105 ++- >>>> content.tex | 1 + >>>> transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ >>>> 4 files changed, 1746 insertions(+), 3 deletions(-) >>>> create mode 100644 transport-msg.tex >>>> >>>> -- >>>> 2.34.1 >>>> >>> > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-20 9:02 ` Bertrand Marquis @ 2026-02-25 7:45 ` Manivannan Sadhasivam 2026-02-25 8:03 ` Bertrand Marquis 2026-02-25 12:53 ` Demi Marie Obenour 0 siblings, 2 replies; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-25 7:45 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Bertrand, On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > Hi Parav, > > > On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > > > > > > > >> From: Michael S. Tsirkin <mst@redhat.com> > >> Sent: 20 February 2026 05:25 AM > >> > >> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > >>> Hi Bill, > >>> > >>>> From: Bill Mills <bill.mills@linaro.org> > >>>> Sent: 26 January 2026 10:02 PM > >>>> > >>>> This series adds the virtio-msg transport layer. > >>>> > >>>> The individuals and organizations involved in this effort have had difficulty in > >>>> using the existing virtio-transports in various situations and desire to add one > >>>> more transport that performs its transport layer operations by sending and > >>>> receiving messages. > >>>> > >>>> Implementations of virtio-msg will normally be done in multiple layers: > >>>> * common / device level > >>>> * bus level > >>>> > >>>> The common / device level defines the messages exchanged between the driver > >>>> and a device. This common part should lead to a common driver holding most > >>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >>>> The kernel implementation in [3] shows this separation. As with other transport > >>>> layers, virtio-msg should not require modifications to existing virtio device > >>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > >>>> main focus of this version of the patch series. > >>>> > >>>> The virtio-msg bus level implements the normal things a bus defines > >>>> (enumeration, dma operations, etc) but also implements the message send and > >>>> receive operations. A number of bus implementations are envisioned, > >>>> some of which will be reusable and general purpose. Other bus implementations > >>>> might be unique to a given situation, for example only used by a PCIe card > >>>> and its driver. > >>>> > >>>> The standard bus messages are an effort to avoid different bus implementations > >>>> doing the same thing in different ways for no good reason. However the > >>>> different environments will require different things. Instead of trying to > >>>> anticipate all needs and provide something very abstract, we think > >>>> implementation specific messages will be needed at the bus level. Over time, > >>>> if we see similar messages across multiple bus implementations, we will move to > >>>> standardize a bus level message for that. > >>>> > >>> > >>> I would review more, had first round of sparse review. > >>> Please find few comments/questions below. > >> > >> I'd like to comment that I think it makes sense to have a basic simple transport and > >> then add performance features on top as appropriate. > > Sounds good. Simple but complete is needed. > > Agree. > > > > >> So one way to address some of these comments is to show how > >> they can be addressed with a feature bit down the road. > >> > >> > >>> 1. device number should be 32-bit in struct virtio_msg_header. > >>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > >>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > >>> > >>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > >>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > >>> Needs 32-bits. > >>> > >>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > >>> But more below. > >>> > >>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate > >> the virtqueues. > >>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >> > >> surely this can be an optional transport feature bit. > >> > > How is this optional? > > As said in a previous mail, we have messages already for that. > Please confirm if that answer your question. > > > How can one implement a transport without defining the basic data transfer semantics? > > We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > If anything is missing, we are more than happy to discuss it and solve the issue. > I'd love to have this transport over PCI because it addresses the shortcomings of the existing PCI transport which just assumes that every config space access\ is trap and emulate. But that being said, I somewhat agree with Parav that we should define the bus implementations in the spec to avoid fixing the ABI in the implementations. For instance, if we try to use this transport over PCI, we've got questions like: 1. How the device should be bind to the virtio-msg-pci bus driver and not with the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? 2. How the Virtio messages should be transferred? Is it through endpoint config space or through some other means? 3. How the notification be delivered from the device to the host? Through INT-X/MSI/MSI-X or even polling? And these are just a few questions that comes to the top of my head. There could be plenty more. How can we expect all the virtio-msg bus implementations to adhere to the same format so that the interoperability offered by the Virtio spec is guaranteed? - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 7:45 ` Manivannan Sadhasivam @ 2026-02-25 8:03 ` Bertrand Marquis 2026-02-25 8:12 ` Michael S. Tsirkin 2026-02-25 10:06 ` Manivannan Sadhasivam 2026-02-25 12:53 ` Demi Marie Obenour 1 sibling, 2 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 8:03 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Manivannan, > On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > > Hi Bertrand, > > On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >> Hi Parav, >> >>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: >>> >>> >>> >>>> From: Michael S. Tsirkin <mst@redhat.com> >>>> Sent: 20 February 2026 05:25 AM >>>> >>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: >>>>> Hi Bill, >>>>> >>>>>> From: Bill Mills <bill.mills@linaro.org> >>>>>> Sent: 26 January 2026 10:02 PM >>>>>> >>>>>> This series adds the virtio-msg transport layer. >>>>>> >>>>>> The individuals and organizations involved in this effort have had difficulty in >>>>>> using the existing virtio-transports in various situations and desire to add one >>>>>> more transport that performs its transport layer operations by sending and >>>>>> receiving messages. >>>>>> >>>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>>> * common / device level >>>>>> * bus level >>>>>> >>>>>> The common / device level defines the messages exchanged between the driver >>>>>> and a device. This common part should lead to a common driver holding most >>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>>> main focus of this version of the patch series. >>>>>> >>>>>> The virtio-msg bus level implements the normal things a bus defines >>>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>>> receive operations. A number of bus implementations are envisioned, >>>>>> some of which will be reusable and general purpose. Other bus implementations >>>>>> might be unique to a given situation, for example only used by a PCIe card >>>>>> and its driver. >>>>>> >>>>>> The standard bus messages are an effort to avoid different bus implementations >>>>>> doing the same thing in different ways for no good reason. However the >>>>>> different environments will require different things. Instead of trying to >>>>>> anticipate all needs and provide something very abstract, we think >>>>>> implementation specific messages will be needed at the bus level. Over time, >>>>>> if we see similar messages across multiple bus implementations, we will move to >>>>>> standardize a bus level message for that. >>>>>> >>>>> >>>>> I would review more, had first round of sparse review. >>>>> Please find few comments/questions below. >>>> >>>> I'd like to comment that I think it makes sense to have a basic simple transport and >>>> then add performance features on top as appropriate. >>> Sounds good. Simple but complete is needed. >> >> Agree. >> >>> >>>> So one way to address some of these comments is to show how >>>> they can be addressed with a feature bit down the road. >>>> >>>> >>>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>>>> >>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>>>> Needs 32-bits. >>>>> >>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>>>> But more below. >>>>> >>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate >>>> the virtqueues. >>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >>>> >>>> surely this can be an optional transport feature bit. >>>> >>> How is this optional? >> >> As said in a previous mail, we have messages already for that. >> Please confirm if that answer your question. >> >>> How can one implement a transport without defining the basic data transfer semantics? >> >> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >> If anything is missing, we are more than happy to discuss it and solve the issue. >> > > I'd love to have this transport over PCI because it addresses the shortcomings > of the existing PCI transport which just assumes that every config space access\ > is trap and emulate. Agree and AMD did exactly that in their demonstrator. I will give you answers here as i know them but Edgar will probably give you more details (and probably fix my mistakes). > > But that being said, I somewhat agree with Parav that we should define the bus > implementations in the spec to avoid fixing the ABI in the implementations. For > instance, if we try to use this transport over PCI, we've got questions like: > > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? One bus is appearing as one pci device with its own Vendor ID, > > 2. How the Virtio messages should be transferred? Is it through endpoint config > space or through some other means? The virtio messages are transfered using FIFOs stored in the BAR of the PCI device (ending up being memory shared between both sides) > > 3. How the notification be delivered from the device to the host? Through > INT-X/MSI/MSI-X or even polling? Notifications are delivered through MSI. > > And these are just a few questions that comes to the top of my head. There could > be plenty more. > > How can we expect all the virtio-msg bus implementations to adhere to the same > format so that the interoperability offered by the Virtio spec is guaranteed? We spent a lot of time thinking on that (this started around 2 years ago) and we discussed several use cases and did some PoC to try to have everything covered (secure to non secure and vm to vm using ffa, system to system over PCI or hardware messaging system, PCI, Xen specific implementation) to check the needs and try to cover as much as we can. Now there might be cases we missed but we think that having a purely message based interface between the bus and the transport and split responsibilities the way we did is allowing lots of different bus implementations without affecting the transport and driver/device implementations on top. We identified that a common use case will be for the bus to transfer messages using FIFOs to optimize speed (at the end you need to have a way to share memory between both sides so why not using a part of it to transfer the messages to and reduce the number of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in practice (so we might standardize the FIFO format in the future to allow even more code reuse between busses). If you have any questions or doubts, or if you have a use case that should be investigated please tell us. Cheers Bertrand > > - Mani > > -- > மணிவண்ணன் சதாசிவம் IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 8:03 ` Bertrand Marquis @ 2026-02-25 8:12 ` Michael S. Tsirkin 2026-02-25 10:06 ` Manivannan Sadhasivam 1 sibling, 0 replies; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 8:12 UTC (permalink / raw) To: Bertrand Marquis Cc: Manivannan Sadhasivam, Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > Hi Manivannan, > > > On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > > > > Hi Bertrand, > > > > On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > >> Hi Parav, > >> > >>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > >>> > >>> > >>> > >>>> From: Michael S. Tsirkin <mst@redhat.com> > >>>> Sent: 20 February 2026 05:25 AM > >>>> > >>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > >>>>> Hi Bill, > >>>>> > >>>>>> From: Bill Mills <bill.mills@linaro.org> > >>>>>> Sent: 26 January 2026 10:02 PM > >>>>>> > >>>>>> This series adds the virtio-msg transport layer. > >>>>>> > >>>>>> The individuals and organizations involved in this effort have had difficulty in > >>>>>> using the existing virtio-transports in various situations and desire to add one > >>>>>> more transport that performs its transport layer operations by sending and > >>>>>> receiving messages. > >>>>>> > >>>>>> Implementations of virtio-msg will normally be done in multiple layers: > >>>>>> * common / device level > >>>>>> * bus level > >>>>>> > >>>>>> The common / device level defines the messages exchanged between the driver > >>>>>> and a device. This common part should lead to a common driver holding most > >>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >>>>>> The kernel implementation in [3] shows this separation. As with other transport > >>>>>> layers, virtio-msg should not require modifications to existing virtio device > >>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > >>>>>> main focus of this version of the patch series. > >>>>>> > >>>>>> The virtio-msg bus level implements the normal things a bus defines > >>>>>> (enumeration, dma operations, etc) but also implements the message send and > >>>>>> receive operations. A number of bus implementations are envisioned, > >>>>>> some of which will be reusable and general purpose. Other bus implementations > >>>>>> might be unique to a given situation, for example only used by a PCIe card > >>>>>> and its driver. > >>>>>> > >>>>>> The standard bus messages are an effort to avoid different bus implementations > >>>>>> doing the same thing in different ways for no good reason. However the > >>>>>> different environments will require different things. Instead of trying to > >>>>>> anticipate all needs and provide something very abstract, we think > >>>>>> implementation specific messages will be needed at the bus level. Over time, > >>>>>> if we see similar messages across multiple bus implementations, we will move to > >>>>>> standardize a bus level message for that. > >>>>>> > >>>>> > >>>>> I would review more, had first round of sparse review. > >>>>> Please find few comments/questions below. > >>>> > >>>> I'd like to comment that I think it makes sense to have a basic simple transport and > >>>> then add performance features on top as appropriate. > >>> Sounds good. Simple but complete is needed. > >> > >> Agree. > >> > >>> > >>>> So one way to address some of these comments is to show how > >>>> they can be addressed with a feature bit down the road. > >>>> > >>>> > >>>>> 1. device number should be 32-bit in struct virtio_msg_header. > >>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > >>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > >>>>> > >>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > >>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > >>>>> Needs 32-bits. > >>>>> > >>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > >>>>> But more below. > >>>>> > >>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate > >>>> the virtqueues. > >>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >>>> > >>>> surely this can be an optional transport feature bit. > >>>> > >>> How is this optional? > >> > >> As said in a previous mail, we have messages already for that. > >> Please confirm if that answer your question. > >> > >>> How can one implement a transport without defining the basic data transfer semantics? > >> > >> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > >> If anything is missing, we are more than happy to discuss it and solve the issue. > >> > > > > I'd love to have this transport over PCI because it addresses the shortcomings > > of the existing PCI transport which just assumes that every config space access\ > > is trap and emulate. > > Agree and AMD did exactly that in their demonstrator. > I will give you answers here as i know them but Edgar will probably give you more > details (and probably fix my mistakes). > > > > > But that being said, I somewhat agree with Parav that we should define the bus > > implementations in the spec to avoid fixing the ABI in the implementations. For > > instance, if we try to use this transport over PCI, we've got questions like: > > > > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > One bus is appearing as one pci device with its own Vendor ID, It might be pretty handy to include, as a separate RFC, a quick description of that binding. > > > > 2. How the Virtio messages should be transferred? Is it through endpoint config > > space or through some other means? > > The virtio messages are transfered using FIFOs stored in the BAR of the PCI > device (ending up being memory shared between both sides) > > > > > 3. How the notification be delivered from the device to the host? Through > > INT-X/MSI/MSI-X or even polling? > > Notifications are delivered through MSI. > > > > > And these are just a few questions that comes to the top of my head. There could > > be plenty more. > > > > How can we expect all the virtio-msg bus implementations to adhere to the same > > format so that the interoperability offered by the Virtio spec is guaranteed? > > We spent a lot of time thinking on that (this started around 2 years ago) and we > discussed several use cases and did some PoC to try to have everything covered > (secure to non secure and vm to vm using ffa, system to system over PCI or hardware > messaging system, PCI, Xen specific implementation) to check the needs and try to > cover as much as we can. > > Now there might be cases we missed but we think that having a purely message based > interface between the bus and the transport and split responsibilities the way we did > is allowing lots of different bus implementations without affecting the transport and > driver/device implementations on top. > > We identified that a common use case will be for the bus to transfer messages using > FIFOs to optimize speed (at the end you need to have a way to share memory between > both sides so why not using a part of it to transfer the messages to and reduce the number > of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in > practice (so we might standardize the FIFO format in the future to allow even more code > reuse between busses). > > If you have any questions or doubts, or if you have a use case that should be investigated > please tell us. > > Cheers > Bertrand > > > > > > - Mani > > > > -- > > மணிவண்ணன் சதாசிவம் > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 8:03 ` Bertrand Marquis 2026-02-25 8:12 ` Michael S. Tsirkin @ 2026-02-25 10:06 ` Manivannan Sadhasivam 2026-02-25 10:10 ` Michael S. Tsirkin ` (3 more replies) 1 sibling, 4 replies; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-25 10:06 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > Hi Manivannan, > > > On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > > > > Hi Bertrand, > > > > On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > >> Hi Parav, > >> > >>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > >>> > >>> > >>> > >>>> From: Michael S. Tsirkin <mst@redhat.com> > >>>> Sent: 20 February 2026 05:25 AM > >>>> > >>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > >>>>> Hi Bill, > >>>>> > >>>>>> From: Bill Mills <bill.mills@linaro.org> > >>>>>> Sent: 26 January 2026 10:02 PM > >>>>>> > >>>>>> This series adds the virtio-msg transport layer. > >>>>>> > >>>>>> The individuals and organizations involved in this effort have had difficulty in > >>>>>> using the existing virtio-transports in various situations and desire to add one > >>>>>> more transport that performs its transport layer operations by sending and > >>>>>> receiving messages. > >>>>>> > >>>>>> Implementations of virtio-msg will normally be done in multiple layers: > >>>>>> * common / device level > >>>>>> * bus level > >>>>>> > >>>>>> The common / device level defines the messages exchanged between the driver > >>>>>> and a device. This common part should lead to a common driver holding most > >>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >>>>>> The kernel implementation in [3] shows this separation. As with other transport > >>>>>> layers, virtio-msg should not require modifications to existing virtio device > >>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > >>>>>> main focus of this version of the patch series. > >>>>>> > >>>>>> The virtio-msg bus level implements the normal things a bus defines > >>>>>> (enumeration, dma operations, etc) but also implements the message send and > >>>>>> receive operations. A number of bus implementations are envisioned, > >>>>>> some of which will be reusable and general purpose. Other bus implementations > >>>>>> might be unique to a given situation, for example only used by a PCIe card > >>>>>> and its driver. > >>>>>> > >>>>>> The standard bus messages are an effort to avoid different bus implementations > >>>>>> doing the same thing in different ways for no good reason. However the > >>>>>> different environments will require different things. Instead of trying to > >>>>>> anticipate all needs and provide something very abstract, we think > >>>>>> implementation specific messages will be needed at the bus level. Over time, > >>>>>> if we see similar messages across multiple bus implementations, we will move to > >>>>>> standardize a bus level message for that. > >>>>>> > >>>>> > >>>>> I would review more, had first round of sparse review. > >>>>> Please find few comments/questions below. > >>>> > >>>> I'd like to comment that I think it makes sense to have a basic simple transport and > >>>> then add performance features on top as appropriate. > >>> Sounds good. Simple but complete is needed. > >> > >> Agree. > >> > >>> > >>>> So one way to address some of these comments is to show how > >>>> they can be addressed with a feature bit down the road. > >>>> > >>>> > >>>>> 1. device number should be 32-bit in struct virtio_msg_header. > >>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > >>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > >>>>> > >>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > >>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > >>>>> Needs 32-bits. > >>>>> > >>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > >>>>> But more below. > >>>>> > >>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate > >>>> the virtqueues. > >>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >>>> > >>>> surely this can be an optional transport feature bit. > >>>> > >>> How is this optional? > >> > >> As said in a previous mail, we have messages already for that. > >> Please confirm if that answer your question. > >> > >>> How can one implement a transport without defining the basic data transfer semantics? > >> > >> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > >> If anything is missing, we are more than happy to discuss it and solve the issue. > >> > > > > I'd love to have this transport over PCI because it addresses the shortcomings > > of the existing PCI transport which just assumes that every config space access\ > > is trap and emulate. > > Agree and AMD did exactly that in their demonstrator. > I will give you answers here as i know them but Edgar will probably give you more > details (and probably fix my mistakes). > > > > > But that being said, I somewhat agree with Parav that we should define the bus > > implementations in the spec to avoid fixing the ABI in the implementations. For > > instance, if we try to use this transport over PCI, we've got questions like: > > > > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > One bus is appearing as one pci device with its own Vendor ID, > What should be the 'own Vendor ID' here? The existing virtio-pci driver binds to all devices with the Vendor ID of PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own VID for exposing the Virtio devices? That would mean, the drivers on the host need update as well, which will not scale. It would be good if the existing virtio-pci devices can use this new transport with only device side modifications. > > > > 2. How the Virtio messages should be transferred? Is it through endpoint config > > space or through some other means? > > The virtio messages are transfered using FIFOs stored in the BAR of the PCI > device (ending up being memory shared between both sides) > What should be the BAR number and size? > > > > 3. How the notification be delivered from the device to the host? Through > > INT-X/MSI/MSI-X or even polling? > > Notifications are delivered through MSI. > So no INT-X or MSI-X? Why so? Anyhow, my objective is not to get answers for my above questions here in this thread, but to state the reality that it would be hard for us to make use of this new transport without defining the bus implementation. > > > > And these are just a few questions that comes to the top of my head. There could > > be plenty more. > > > > How can we expect all the virtio-msg bus implementations to adhere to the same > > format so that the interoperability offered by the Virtio spec is guaranteed? > > We spent a lot of time thinking on that (this started around 2 years ago) and we > discussed several use cases and did some PoC to try to have everything covered > (secure to non secure and vm to vm using ffa, system to system over PCI or hardware > messaging system, PCI, Xen specific implementation) to check the needs and try to > cover as much as we can. > > Now there might be cases we missed but we think that having a purely message based > interface between the bus and the transport and split responsibilities the way we did > is allowing lots of different bus implementations without affecting the transport and > driver/device implementations on top. > > We identified that a common use case will be for the bus to transfer messages using > FIFOs to optimize speed (at the end you need to have a way to share memory between > both sides so why not using a part of it to transfer the messages to and reduce the number > of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in > practice (so we might standardize the FIFO format in the future to allow even more code > reuse between busses). > Not just the FIFO format, but how that FIFO gets shared between the device and the host also needs to be documented. Maybe for this initial transport version, you can start with defining the FF-A bus implementation? - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:06 ` Manivannan Sadhasivam @ 2026-02-25 10:10 ` Michael S. Tsirkin 2026-02-25 10:14 ` Bertrand Marquis ` (2 subsequent siblings) 3 siblings, 0 replies; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 10:10 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Bertrand Marquis, Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 03:36:36PM +0530, Manivannan Sadhasivam wrote: > On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > > Hi Manivannan, > > > > > On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > > > > > > Hi Bertrand, > > > > > > On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > > >> Hi Parav, > > >> > > >>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > > >>> > > >>> > > >>> > > >>>> From: Michael S. Tsirkin <mst@redhat.com> > > >>>> Sent: 20 February 2026 05:25 AM > > >>>> > > >>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > > >>>>> Hi Bill, > > >>>>> > > >>>>>> From: Bill Mills <bill.mills@linaro.org> > > >>>>>> Sent: 26 January 2026 10:02 PM > > >>>>>> > > >>>>>> This series adds the virtio-msg transport layer. > > >>>>>> > > >>>>>> The individuals and organizations involved in this effort have had difficulty in > > >>>>>> using the existing virtio-transports in various situations and desire to add one > > >>>>>> more transport that performs its transport layer operations by sending and > > >>>>>> receiving messages. > > >>>>>> > > >>>>>> Implementations of virtio-msg will normally be done in multiple layers: > > >>>>>> * common / device level > > >>>>>> * bus level > > >>>>>> > > >>>>>> The common / device level defines the messages exchanged between the driver > > >>>>>> and a device. This common part should lead to a common driver holding most > > >>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > > >>>>>> The kernel implementation in [3] shows this separation. As with other transport > > >>>>>> layers, virtio-msg should not require modifications to existing virtio device > > >>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > > >>>>>> main focus of this version of the patch series. > > >>>>>> > > >>>>>> The virtio-msg bus level implements the normal things a bus defines > > >>>>>> (enumeration, dma operations, etc) but also implements the message send and > > >>>>>> receive operations. A number of bus implementations are envisioned, > > >>>>>> some of which will be reusable and general purpose. Other bus implementations > > >>>>>> might be unique to a given situation, for example only used by a PCIe card > > >>>>>> and its driver. > > >>>>>> > > >>>>>> The standard bus messages are an effort to avoid different bus implementations > > >>>>>> doing the same thing in different ways for no good reason. However the > > >>>>>> different environments will require different things. Instead of trying to > > >>>>>> anticipate all needs and provide something very abstract, we think > > >>>>>> implementation specific messages will be needed at the bus level. Over time, > > >>>>>> if we see similar messages across multiple bus implementations, we will move to > > >>>>>> standardize a bus level message for that. > > >>>>>> > > >>>>> > > >>>>> I would review more, had first round of sparse review. > > >>>>> Please find few comments/questions below. > > >>>> > > >>>> I'd like to comment that I think it makes sense to have a basic simple transport and > > >>>> then add performance features on top as appropriate. > > >>> Sounds good. Simple but complete is needed. > > >> > > >> Agree. > > >> > > >>> > > >>>> So one way to address some of these comments is to show how > > >>>> they can be addressed with a feature bit down the road. > > >>>> > > >>>> > > >>>>> 1. device number should be 32-bit in struct virtio_msg_header. > > >>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > > >>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > > >>>>> > > >>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > > >>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > > >>>>> Needs 32-bits. > > >>>>> > > >>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > > >>>>> But more below. > > >>>>> > > >>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate > > >>>> the virtqueues. > > >>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > >>>> > > >>>> surely this can be an optional transport feature bit. > > >>>> > > >>> How is this optional? > > >> > > >> As said in a previous mail, we have messages already for that. > > >> Please confirm if that answer your question. > > >> > > >>> How can one implement a transport without defining the basic data transfer semantics? > > >> > > >> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > > >> If anything is missing, we are more than happy to discuss it and solve the issue. > > >> > > > > > > I'd love to have this transport over PCI because it addresses the shortcomings > > > of the existing PCI transport which just assumes that every config space access\ > > > is trap and emulate. > > > > Agree and AMD did exactly that in their demonstrator. > > I will give you answers here as i know them but Edgar will probably give you more > > details (and probably fix my mistakes). > > > > > > > > But that being said, I somewhat agree with Parav that we should define the bus > > > implementations in the spec to avoid fixing the ABI in the implementations. For > > > instance, if we try to use this transport over PCI, we've got questions like: > > > > > > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > > > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > > > One bus is appearing as one pci device with its own Vendor ID, > > > > What should be the 'own Vendor ID' here? > > The existing virtio-pci driver binds to all devices with the Vendor ID of > PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > VID for exposing the Virtio devices? That would mean, the drivers on the host > need update as well, which will not scale. > > It would be good if the existing virtio-pci devices can use this new transport > with only device side modifications. Oh another transitional device hack( You guys should think hard whether you really want that. > > > > > > 2. How the Virtio messages should be transferred? Is it through endpoint config > > > space or through some other means? > > > > The virtio messages are transfered using FIFOs stored in the BAR of the PCI > > device (ending up being memory shared between both sides) > > > > What should be the BAR number and size? > > > > > > > 3. How the notification be delivered from the device to the host? Through > > > INT-X/MSI/MSI-X or even polling? > > > > Notifications are delivered through MSI. > > > > So no INT-X or MSI-X? Why so? > > Anyhow, my objective is not to get answers for my above questions here in this > thread, but to state the reality that it would be hard for us to make use of > this new transport without defining the bus implementation. > > > > > > > And these are just a few questions that comes to the top of my head. There could > > > be plenty more. > > > > > > How can we expect all the virtio-msg bus implementations to adhere to the same > > > format so that the interoperability offered by the Virtio spec is guaranteed? > > > > We spent a lot of time thinking on that (this started around 2 years ago) and we > > discussed several use cases and did some PoC to try to have everything covered > > (secure to non secure and vm to vm using ffa, system to system over PCI or hardware > > messaging system, PCI, Xen specific implementation) to check the needs and try to > > cover as much as we can. > > > > Now there might be cases we missed but we think that having a purely message based > > interface between the bus and the transport and split responsibilities the way we did > > is allowing lots of different bus implementations without affecting the transport and > > driver/device implementations on top. > > > > We identified that a common use case will be for the bus to transfer messages using > > FIFOs to optimize speed (at the end you need to have a way to share memory between > > both sides so why not using a part of it to transfer the messages to and reduce the number > > of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in > > practice (so we might standardize the FIFO format in the future to allow even more code > > reuse between busses). > > > > Not just the FIFO format, but how that FIFO gets shared between the device and > the host also needs to be documented. Maybe for this initial transport version, > you can start with defining the FF-A bus implementation? > > - Mani > > -- > மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:06 ` Manivannan Sadhasivam 2026-02-25 10:10 ` Michael S. Tsirkin @ 2026-02-25 10:14 ` Bertrand Marquis 2026-02-25 10:22 ` Michael S. Tsirkin 2026-02-25 10:24 ` Parav Pandit 3 siblings, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 10:14 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Manivannan, I will let Edgar respond on thing related to virtio-msg bus over PCI as he is the expert here. Cheers Bertrand > On 25 Feb 2026, at 11:06, Manivannan Sadhasivam <mani@kernel.org> wrote: > > On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: >> Hi Manivannan, >> >>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: >>> >>> Hi Bertrand, >>> >>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>> Hi Parav, >>>> >>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: >>>>> >>>>> >>>>> >>>>>> From: Michael S. Tsirkin <mst@redhat.com> >>>>>> Sent: 20 February 2026 05:25 AM >>>>>> >>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: >>>>>>> Hi Bill, >>>>>>> >>>>>>>> From: Bill Mills <bill.mills@linaro.org> >>>>>>>> Sent: 26 January 2026 10:02 PM >>>>>>>> >>>>>>>> This series adds the virtio-msg transport layer. >>>>>>>> >>>>>>>> The individuals and organizations involved in this effort have had difficulty in >>>>>>>> using the existing virtio-transports in various situations and desire to add one >>>>>>>> more transport that performs its transport layer operations by sending and >>>>>>>> receiving messages. >>>>>>>> >>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>>>>> * common / device level >>>>>>>> * bus level >>>>>>>> >>>>>>>> The common / device level defines the messages exchanged between the driver >>>>>>>> and a device. This common part should lead to a common driver holding most >>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>>>>> main focus of this version of the patch series. >>>>>>>> >>>>>>>> The virtio-msg bus level implements the normal things a bus defines >>>>>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>>>>> receive operations. A number of bus implementations are envisioned, >>>>>>>> some of which will be reusable and general purpose. Other bus implementations >>>>>>>> might be unique to a given situation, for example only used by a PCIe card >>>>>>>> and its driver. >>>>>>>> >>>>>>>> The standard bus messages are an effort to avoid different bus implementations >>>>>>>> doing the same thing in different ways for no good reason. However the >>>>>>>> different environments will require different things. Instead of trying to >>>>>>>> anticipate all needs and provide something very abstract, we think >>>>>>>> implementation specific messages will be needed at the bus level. Over time, >>>>>>>> if we see similar messages across multiple bus implementations, we will move to >>>>>>>> standardize a bus level message for that. >>>>>>>> >>>>>>> >>>>>>> I would review more, had first round of sparse review. >>>>>>> Please find few comments/questions below. >>>>>> >>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and >>>>>> then add performance features on top as appropriate. >>>>> Sounds good. Simple but complete is needed. >>>> >>>> Agree. >>>> >>>>> >>>>>> So one way to address some of these comments is to show how >>>>>> they can be addressed with a feature bit down the road. >>>>>> >>>>>> >>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>>>>>> >>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>>>>>> Needs 32-bits. >>>>>>> >>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>>>>>> But more below. >>>>>>> >>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate >>>>>> the virtqueues. >>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >>>>>> >>>>>> surely this can be an optional transport feature bit. >>>>>> >>>>> How is this optional? >>>> >>>> As said in a previous mail, we have messages already for that. >>>> Please confirm if that answer your question. >>>> >>>>> How can one implement a transport without defining the basic data transfer semantics? >>>> >>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>> >>> >>> I'd love to have this transport over PCI because it addresses the shortcomings >>> of the existing PCI transport which just assumes that every config space access\ >>> is trap and emulate. >> >> Agree and AMD did exactly that in their demonstrator. >> I will give you answers here as i know them but Edgar will probably give you more >> details (and probably fix my mistakes). >> >>> >>> But that being said, I somewhat agree with Parav that we should define the bus >>> implementations in the spec to avoid fixing the ABI in the implementations. For >>> instance, if we try to use this transport over PCI, we've got questions like: >>> >>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with >>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? >> >> One bus is appearing as one pci device with its own Vendor ID, >> > > What should be the 'own Vendor ID' here? > > The existing virtio-pci driver binds to all devices with the Vendor ID of > PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > VID for exposing the Virtio devices? That would mean, the drivers on the host > need update as well, which will not scale. > > It would be good if the existing virtio-pci devices can use this new transport > with only device side modifications. > >>> >>> 2. How the Virtio messages should be transferred? Is it through endpoint config >>> space or through some other means? >> >> The virtio messages are transfered using FIFOs stored in the BAR of the PCI >> device (ending up being memory shared between both sides) >> > > What should be the BAR number and size? > >>> >>> 3. How the notification be delivered from the device to the host? Through >>> INT-X/MSI/MSI-X or even polling? >> >> Notifications are delivered through MSI. >> > > So no INT-X or MSI-X? Why so? > > Anyhow, my objective is not to get answers for my above questions here in this > thread, but to state the reality that it would be hard for us to make use of > this new transport without defining the bus implementation. > >>> >>> And these are just a few questions that comes to the top of my head. There could >>> be plenty more. >>> >>> How can we expect all the virtio-msg bus implementations to adhere to the same >>> format so that the interoperability offered by the Virtio spec is guaranteed? >> >> We spent a lot of time thinking on that (this started around 2 years ago) and we >> discussed several use cases and did some PoC to try to have everything covered >> (secure to non secure and vm to vm using ffa, system to system over PCI or hardware >> messaging system, PCI, Xen specific implementation) to check the needs and try to >> cover as much as we can. >> >> Now there might be cases we missed but we think that having a purely message based >> interface between the bus and the transport and split responsibilities the way we did >> is allowing lots of different bus implementations without affecting the transport and >> driver/device implementations on top. >> >> We identified that a common use case will be for the bus to transfer messages using >> FIFOs to optimize speed (at the end you need to have a way to share memory between >> both sides so why not using a part of it to transfer the messages to and reduce the number >> of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in >> practice (so we might standardize the FIFO format in the future to allow even more code >> reuse between busses). >> > > Not just the FIFO format, but how that FIFO gets shared between the device and > the host also needs to be documented. Maybe for this initial transport version, > you can start with defining the FF-A bus implementation? > > - Mani > > -- > மணிவண்ணன் சதாசிவம் IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:06 ` Manivannan Sadhasivam 2026-02-25 10:10 ` Michael S. Tsirkin 2026-02-25 10:14 ` Bertrand Marquis @ 2026-02-25 10:22 ` Michael S. Tsirkin 2026-02-25 10:53 ` Manivannan Sadhasivam 2026-02-25 10:24 ` Parav Pandit 3 siblings, 1 reply; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 10:22 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Bertrand Marquis, Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 03:36:36PM +0530, Manivannan Sadhasivam wrote: > > > But that being said, I somewhat agree with Parav that we should define the bus > > > implementations in the spec to avoid fixing the ABI in the implementations. For > > > instance, if we try to use this transport over PCI, we've got questions like: > > > > > > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > > > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > > > One bus is appearing as one pci device with its own Vendor ID, > > > > What should be the 'own Vendor ID' here? > > The existing virtio-pci driver binds to all devices with the Vendor ID of > PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > VID for exposing the Virtio devices? That would mean, the drivers on the host > need update as well, which will not scale. > > It would be good if the existing virtio-pci devices can use this new transport > with only device side modifications. ah maybe i misuderstand. you do not mean do not change drivers. you mean a generic driver for everyone? there are several ways to address this in pci - use a shared vendor/device id: worked so far reasonably well for virtio. we can allocate a new range of device ids or a new vendor id. - class / prog ifc it seems like a cleaner solution but will preclude declaring e.g. a virtio net interface as a network interface. -- MST ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:22 ` Michael S. Tsirkin @ 2026-02-25 10:53 ` Manivannan Sadhasivam 0 siblings, 0 replies; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-25 10:53 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Bertrand Marquis, Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 05:22:32AM -0500, Michael S. Tsirkin wrote: > On Wed, Feb 25, 2026 at 03:36:36PM +0530, Manivannan Sadhasivam wrote: > > > > But that being said, I somewhat agree with Parav that we should define the bus > > > > implementations in the spec to avoid fixing the ABI in the implementations. For > > > > instance, if we try to use this transport over PCI, we've got questions like: > > > > > > > > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > > > > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > > > > > One bus is appearing as one pci device with its own Vendor ID, > > > > > > > What should be the 'own Vendor ID' here? > > > > The existing virtio-pci driver binds to all devices with the Vendor ID of > > PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > > VID for exposing the Virtio devices? That would mean, the drivers on the host > > need update as well, which will not scale. > > > > It would be good if the existing virtio-pci devices can use this new transport > > with only device side modifications. > > ah maybe i misuderstand. you do not mean do not change drivers. > you mean a generic driver for everyone? > I actually meant to say that any device that supports virtio-msg-pci transport should just work without modifying the virtio-msg-pci driver on the host. > there are several ways to address this in pci > - use a shared vendor/device id: worked so far reasonably well for virtio. > we can allocate a new range of device ids or a new vendor id. > This sounds good to me. - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:06 ` Manivannan Sadhasivam ` (2 preceding siblings ...) 2026-02-25 10:22 ` Michael S. Tsirkin @ 2026-02-25 10:24 ` Parav Pandit 2026-02-25 10:35 ` Bertrand Marquis 3 siblings, 1 reply; 105+ messages in thread From: Parav Pandit @ 2026-02-25 10:24 UTC (permalink / raw) To: Manivannan Sadhasivam, Bertrand Marquis Cc: Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Manivannan Sadhasivam <mani@kernel.org> > Sent: 25 February 2026 03:37 PM > > On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > > Hi Manivannan, > > > > > On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > > > > > > Hi Bertrand, > > > > > > On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > > >> Hi Parav, > > >> > > >>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > > >>> > > >>> > > >>> > > >>>> From: Michael S. Tsirkin <mst@redhat.com> > > >>>> Sent: 20 February 2026 05:25 AM > > >>>> > > >>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > > >>>>> Hi Bill, > > >>>>> > > >>>>>> From: Bill Mills <bill.mills@linaro.org> > > >>>>>> Sent: 26 January 2026 10:02 PM > > >>>>>> > > >>>>>> This series adds the virtio-msg transport layer. > > >>>>>> > > >>>>>> The individuals and organizations involved in this effort have had difficulty in > > >>>>>> using the existing virtio-transports in various situations and desire to add one > > >>>>>> more transport that performs its transport layer operations by sending and > > >>>>>> receiving messages. > > >>>>>> > > >>>>>> Implementations of virtio-msg will normally be done in multiple layers: > > >>>>>> * common / device level > > >>>>>> * bus level > > >>>>>> > > >>>>>> The common / device level defines the messages exchanged between the driver > > >>>>>> and a device. This common part should lead to a common driver holding most > > >>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > > >>>>>> The kernel implementation in [3] shows this separation. As with other transport > > >>>>>> layers, virtio-msg should not require modifications to existing virtio device > > >>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > > >>>>>> main focus of this version of the patch series. > > >>>>>> > > >>>>>> The virtio-msg bus level implements the normal things a bus defines > > >>>>>> (enumeration, dma operations, etc) but also implements the message send and > > >>>>>> receive operations. A number of bus implementations are envisioned, > > >>>>>> some of which will be reusable and general purpose. Other bus implementations > > >>>>>> might be unique to a given situation, for example only used by a PCIe card > > >>>>>> and its driver. > > >>>>>> > > >>>>>> The standard bus messages are an effort to avoid different bus implementations > > >>>>>> doing the same thing in different ways for no good reason. However the > > >>>>>> different environments will require different things. Instead of trying to > > >>>>>> anticipate all needs and provide something very abstract, we think > > >>>>>> implementation specific messages will be needed at the bus level. Over time, > > >>>>>> if we see similar messages across multiple bus implementations, we will move to > > >>>>>> standardize a bus level message for that. > > >>>>>> > > >>>>> > > >>>>> I would review more, had first round of sparse review. > > >>>>> Please find few comments/questions below. > > >>>> > > >>>> I'd like to comment that I think it makes sense to have a basic simple transport and > > >>>> then add performance features on top as appropriate. > > >>> Sounds good. Simple but complete is needed. > > >> > > >> Agree. > > >> > > >>> > > >>>> So one way to address some of these comments is to show how > > >>>> they can be addressed with a feature bit down the road. > > >>>> > > >>>> > > >>>>> 1. device number should be 32-bit in struct virtio_msg_header. > > >>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > > >>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > > >>>>> > > >>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > > >>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > > >>>>> Needs 32-bits. > > >>>>> > > >>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > > >>>>> But more below. > > >>>>> > > >>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > operate > > >>>> the virtqueues. > > >>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > >>>> > > >>>> surely this can be an optional transport feature bit. > > >>>> > > >>> How is this optional? > > >> > > >> As said in a previous mail, we have messages already for that. > > >> Please confirm if that answer your question. > > >> > > >>> How can one implement a transport without defining the basic data transfer semantics? > > >> > > >> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > > >> If anything is missing, we are more than happy to discuss it and solve the issue. > > >> > > > > > > I'd love to have this transport over PCI because it addresses the shortcomings > > > of the existing PCI transport which just assumes that every config space access\ > > > is trap and emulate. > > > > Agree and AMD did exactly that in their demonstrator. > > I will give you answers here as i know them but Edgar will probably give you more > > details (and probably fix my mistakes). > > > > > > > > But that being said, I somewhat agree with Parav that we should define the bus > > > implementations in the spec to avoid fixing the ABI in the implementations. For > > > instance, if we try to use this transport over PCI, we've got questions like: > > > > > > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > > > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > > > One bus is appearing as one pci device with its own Vendor ID, > > > > What should be the 'own Vendor ID' here? > > The existing virtio-pci driver binds to all devices with the Vendor ID of > PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > VID for exposing the Virtio devices? That would mean, the drivers on the host > need update as well, which will not scale. > > It would be good if the existing virtio-pci devices can use this new transport > with only device side modifications. > > > > > > > 2. How the Virtio messages should be transferred? Is it through endpoint config > > > space or through some other means? > > > > The virtio messages are transfered using FIFOs stored in the BAR of the PCI > > device (ending up being memory shared between both sides) > > > > What should be the BAR number and size? > > > > > > > 3. How the notification be delivered from the device to the host? Through > > > INT-X/MSI/MSI-X or even polling? > > > > Notifications are delivered through MSI. > > > > So no INT-X or MSI-X? Why so? > > Anyhow, my objective is not to get answers for my above questions here in this > thread, but to state the reality that it would be hard for us to make use of > this new transport without defining the bus implementation. > +1 to most of the points that Manivannan explained. The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet another interface for _Everything_ that already exists. and device is still have to implement all the existing things because the device does not know which driver will operate. And that too some register based inefficient interface. Just to reset the device one needs to fully setup the new message interface but device still have to be working. That defeats the whole purpose of reset_1 and reset_2 in the device. This does not bring anything better for the PCI devices at all. A transport binding should be defined for the bus binding. A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. If we are creating something brand-new, for PCI the only thing needed is: 1. Reset the device 2. Create an admin virtqueue 3. Transport everything needed through this virtqueue including features, configs, control. And this will work for any other bus or msg based too given only contract needed is to creating the aq. > > > > > > And these are just a few questions that comes to the top of my head. There could > > > be plenty more. > > > > > > How can we expect all the virtio-msg bus implementations to adhere to the same > > > format so that the interoperability offered by the Virtio spec is guaranteed? > > > > We spent a lot of time thinking on that (this started around 2 years ago) and we > > discussed several use cases and did some PoC to try to have everything covered > > (secure to non secure and vm to vm using ffa, system to system over PCI or hardware > > messaging system, PCI, Xen specific implementation) to check the needs and try to > > cover as much as we can. > > > > Now there might be cases we missed but we think that having a purely message based > > interface between the bus and the transport and split responsibilities the way we did > > is allowing lots of different bus implementations without affecting the transport and > > driver/device implementations on top. > > > > We identified that a common use case will be for the bus to transfer messages using > > FIFOs to optimize speed (at the end you need to have a way to share memory between > > both sides so why not using a part of it to transfer the messages to and reduce the number > > of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in > > practice (so we might standardize the FIFO format in the future to allow even more code > > reuse between busses). > > > > Not just the FIFO format, but how that FIFO gets shared between the device and > the host also needs to be documented. Maybe for this initial transport version, > you can start with defining the FF-A bus implementation? > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:24 ` Parav Pandit @ 2026-02-25 10:35 ` Bertrand Marquis 2026-02-25 10:52 ` Michael S. Tsirkin 2026-02-25 14:45 ` Parav Pandit 0 siblings, 2 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 10:35 UTC (permalink / raw) To: Parav Pandit Cc: Manivannan Sadhasivam, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Parav, > On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: > >> >> From: Manivannan Sadhasivam <mani@kernel.org> >> Sent: 25 February 2026 03:37 PM >> >> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: >>> Hi Manivannan, >>> >>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: >>>> >>>> Hi Bertrand, >>>> >>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>>> Hi Parav, >>>>> >>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>>> From: Michael S. Tsirkin <mst@redhat.com> >>>>>>> Sent: 20 February 2026 05:25 AM >>>>>>> >>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: >>>>>>>> Hi Bill, >>>>>>>> >>>>>>>>> From: Bill Mills <bill.mills@linaro.org> >>>>>>>>> Sent: 26 January 2026 10:02 PM >>>>>>>>> >>>>>>>>> This series adds the virtio-msg transport layer. >>>>>>>>> >>>>>>>>> The individuals and organizations involved in this effort have had difficulty in >>>>>>>>> using the existing virtio-transports in various situations and desire to add one >>>>>>>>> more transport that performs its transport layer operations by sending and >>>>>>>>> receiving messages. >>>>>>>>> >>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>>>>>> * common / device level >>>>>>>>> * bus level >>>>>>>>> >>>>>>>>> The common / device level defines the messages exchanged between the driver >>>>>>>>> and a device. This common part should lead to a common driver holding most >>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>>>>>> main focus of this version of the patch series. >>>>>>>>> >>>>>>>>> The virtio-msg bus level implements the normal things a bus defines >>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>>>>>> receive operations. A number of bus implementations are envisioned, >>>>>>>>> some of which will be reusable and general purpose. Other bus implementations >>>>>>>>> might be unique to a given situation, for example only used by a PCIe card >>>>>>>>> and its driver. >>>>>>>>> >>>>>>>>> The standard bus messages are an effort to avoid different bus implementations >>>>>>>>> doing the same thing in different ways for no good reason. However the >>>>>>>>> different environments will require different things. Instead of trying to >>>>>>>>> anticipate all needs and provide something very abstract, we think >>>>>>>>> implementation specific messages will be needed at the bus level. Over time, >>>>>>>>> if we see similar messages across multiple bus implementations, we will move to >>>>>>>>> standardize a bus level message for that. >>>>>>>>> >>>>>>>> >>>>>>>> I would review more, had first round of sparse review. >>>>>>>> Please find few comments/questions below. >>>>>>> >>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and >>>>>>> then add performance features on top as appropriate. >>>>>> Sounds good. Simple but complete is needed. >>>>> >>>>> Agree. >>>>> >>>>>> >>>>>>> So one way to address some of these comments is to show how >>>>>>> they can be addressed with a feature bit down the road. >>>>>>> >>>>>>> >>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>>>>>>> >>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>>>>>>> Needs 32-bits. >>>>>>>> >>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>>>>>>> But more below. >>>>>>>> >>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to >> operate >>>>>>> the virtqueues. >>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >>>>>>> >>>>>>> surely this can be an optional transport feature bit. >>>>>>> >>>>>> How is this optional? >>>>> >>>>> As said in a previous mail, we have messages already for that. >>>>> Please confirm if that answer your question. >>>>> >>>>>> How can one implement a transport without defining the basic data transfer semantics? >>>>> >>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>>> >>>> >>>> I'd love to have this transport over PCI because it addresses the shortcomings >>>> of the existing PCI transport which just assumes that every config space access\ >>>> is trap and emulate. >>> >>> Agree and AMD did exactly that in their demonstrator. >>> I will give you answers here as i know them but Edgar will probably give you more >>> details (and probably fix my mistakes). >>> >>>> >>>> But that being said, I somewhat agree with Parav that we should define the bus >>>> implementations in the spec to avoid fixing the ABI in the implementations. For >>>> instance, if we try to use this transport over PCI, we've got questions like: >>>> >>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with >>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? >>> >>> One bus is appearing as one pci device with its own Vendor ID, >>> >> >> What should be the 'own Vendor ID' here? >> >> The existing virtio-pci driver binds to all devices with the Vendor ID of >> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own >> VID for exposing the Virtio devices? That would mean, the drivers on the host >> need update as well, which will not scale. >> >> It would be good if the existing virtio-pci devices can use this new transport >> with only device side modifications. >> >>>> >>>> 2. How the Virtio messages should be transferred? Is it through endpoint config >>>> space or through some other means? >>> >>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI >>> device (ending up being memory shared between both sides) >>> >> >> What should be the BAR number and size? >> >>>> >>>> 3. How the notification be delivered from the device to the host? Through >>>> INT-X/MSI/MSI-X or even polling? >>> >>> Notifications are delivered through MSI. >>> >> >> So no INT-X or MSI-X? Why so? >> >> Anyhow, my objective is not to get answers for my above questions here in this >> thread, but to state the reality that it would be hard for us to make use of >> this new transport without defining the bus implementation. >> > +1 to most of the points that Manivannan explained. > > The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet another interface for _Everything_ that already exists. > and device is still have to implement all the existing things because the device does not know which driver will operate. > > And that too some register based inefficient interface. > Just to reset the device one needs to fully setup the new message interface but device still have to be working. > That defeats the whole purpose of reset_1 and reset_2 in the device. > > This does not bring anything better for the PCI devices at all. > > A transport binding should be defined for the bus binding. > A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. > > If we are creating something brand-new, for PCI the only thing needed is: > 1. Reset the device > 2. Create an admin virtqueue > 3. Transport everything needed through this virtqueue including features, configs, control. > > And this will work for any other bus or msg based too given only contract needed is to creating the aq. I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". When the virtio-msg bus over PCI it will communicate through this device with an external system connected through the PCI bus. The driver will enumerate virtio devices available behind this bus and register them so that the corresponding virtio drivers are probed for them. All virtio-msg messages required to communicate with those devices will be transferred through a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the virtqueues with all the devices on the bus. So the PCI device is not one virtio device but one bus behind which there can be many devices. Is this making the concept a bit clearer ? Cheers Bertrand > >>>> >>>> And these are just a few questions that comes to the top of my head. There could >>>> be plenty more. >>>> >>>> How can we expect all the virtio-msg bus implementations to adhere to the same >>>> format so that the interoperability offered by the Virtio spec is guaranteed? >>> >>> We spent a lot of time thinking on that (this started around 2 years ago) and we >>> discussed several use cases and did some PoC to try to have everything covered >>> (secure to non secure and vm to vm using ffa, system to system over PCI or hardware >>> messaging system, PCI, Xen specific implementation) to check the needs and try to >>> cover as much as we can. >>> >>> Now there might be cases we missed but we think that having a purely message based >>> interface between the bus and the transport and split responsibilities the way we did >>> is allowing lots of different bus implementations without affecting the transport and >>> driver/device implementations on top. >>> >>> We identified that a common use case will be for the bus to transfer messages using >>> FIFOs to optimize speed (at the end you need to have a way to share memory between >>> both sides so why not using a part of it to transfer the messages to and reduce the number >>> of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in >>> practice (so we might standardize the FIFO format in the future to allow even more code >>> reuse between busses). >>> >> >> Not just the FIFO format, but how that FIFO gets shared between the device and >> the host also needs to be documented. Maybe for this initial transport version, >> you can start with defining the FF-A bus implementation? IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:35 ` Bertrand Marquis @ 2026-02-25 10:52 ` Michael S. Tsirkin 2026-02-25 10:55 ` Bertrand Marquis 2026-02-25 14:45 ` Parav Pandit 1 sibling, 1 reply; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 10:52 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 10:35:41AM +0000, Bertrand Marquis wrote: > Hi Parav, > > > On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: > > > >> > >> From: Manivannan Sadhasivam <mani@kernel.org> > >> Sent: 25 February 2026 03:37 PM > >> > >> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > >>> Hi Manivannan, > >>> > >>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > >>>> > >>>> Hi Bertrand, > >>>> > >>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > >>>>> Hi Parav, > >>>>> > >>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>>> From: Michael S. Tsirkin <mst@redhat.com> > >>>>>>> Sent: 20 February 2026 05:25 AM > >>>>>>> > >>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > >>>>>>>> Hi Bill, > >>>>>>>> > >>>>>>>>> From: Bill Mills <bill.mills@linaro.org> > >>>>>>>>> Sent: 26 January 2026 10:02 PM > >>>>>>>>> > >>>>>>>>> This series adds the virtio-msg transport layer. > >>>>>>>>> > >>>>>>>>> The individuals and organizations involved in this effort have had difficulty in > >>>>>>>>> using the existing virtio-transports in various situations and desire to add one > >>>>>>>>> more transport that performs its transport layer operations by sending and > >>>>>>>>> receiving messages. > >>>>>>>>> > >>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: > >>>>>>>>> * common / device level > >>>>>>>>> * bus level > >>>>>>>>> > >>>>>>>>> The common / device level defines the messages exchanged between the driver > >>>>>>>>> and a device. This common part should lead to a common driver holding most > >>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport > >>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device > >>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > >>>>>>>>> main focus of this version of the patch series. > >>>>>>>>> > >>>>>>>>> The virtio-msg bus level implements the normal things a bus defines > >>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and > >>>>>>>>> receive operations. A number of bus implementations are envisioned, > >>>>>>>>> some of which will be reusable and general purpose. Other bus implementations > >>>>>>>>> might be unique to a given situation, for example only used by a PCIe card > >>>>>>>>> and its driver. > >>>>>>>>> > >>>>>>>>> The standard bus messages are an effort to avoid different bus implementations > >>>>>>>>> doing the same thing in different ways for no good reason. However the > >>>>>>>>> different environments will require different things. Instead of trying to > >>>>>>>>> anticipate all needs and provide something very abstract, we think > >>>>>>>>> implementation specific messages will be needed at the bus level. Over time, > >>>>>>>>> if we see similar messages across multiple bus implementations, we will move to > >>>>>>>>> standardize a bus level message for that. > >>>>>>>>> > >>>>>>>> > >>>>>>>> I would review more, had first round of sparse review. > >>>>>>>> Please find few comments/questions below. > >>>>>>> > >>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and > >>>>>>> then add performance features on top as appropriate. > >>>>>> Sounds good. Simple but complete is needed. > >>>>> > >>>>> Agree. > >>>>> > >>>>>> > >>>>>>> So one way to address some of these comments is to show how > >>>>>>> they can be addressed with a feature bit down the road. > >>>>>>> > >>>>>>> > >>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. > >>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > >>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > >>>>>>>> > >>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > >>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > >>>>>>>> Needs 32-bits. > >>>>>>>> > >>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > >>>>>>>> But more below. > >>>>>>>> > >>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > >> operate > >>>>>>> the virtqueues. > >>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >>>>>>> > >>>>>>> surely this can be an optional transport feature bit. > >>>>>>> > >>>>>> How is this optional? > >>>>> > >>>>> As said in a previous mail, we have messages already for that. > >>>>> Please confirm if that answer your question. > >>>>> > >>>>>> How can one implement a transport without defining the basic data transfer semantics? > >>>>> > >>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > >>>>> If anything is missing, we are more than happy to discuss it and solve the issue. > >>>>> > >>>> > >>>> I'd love to have this transport over PCI because it addresses the shortcomings > >>>> of the existing PCI transport which just assumes that every config space access\ > >>>> is trap and emulate. > >>> > >>> Agree and AMD did exactly that in their demonstrator. > >>> I will give you answers here as i know them but Edgar will probably give you more > >>> details (and probably fix my mistakes). > >>> > >>>> > >>>> But that being said, I somewhat agree with Parav that we should define the bus > >>>> implementations in the spec to avoid fixing the ABI in the implementations. For > >>>> instance, if we try to use this transport over PCI, we've got questions like: > >>>> > >>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with > >>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > >>> > >>> One bus is appearing as one pci device with its own Vendor ID, > >>> > >> > >> What should be the 'own Vendor ID' here? > >> > >> The existing virtio-pci driver binds to all devices with the Vendor ID of > >> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > >> VID for exposing the Virtio devices? That would mean, the drivers on the host > >> need update as well, which will not scale. > >> > >> It would be good if the existing virtio-pci devices can use this new transport > >> with only device side modifications. > >> > >>>> > >>>> 2. How the Virtio messages should be transferred? Is it through endpoint config > >>>> space or through some other means? > >>> > >>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI > >>> device (ending up being memory shared between both sides) > >>> > >> > >> What should be the BAR number and size? > >> > >>>> > >>>> 3. How the notification be delivered from the device to the host? Through > >>>> INT-X/MSI/MSI-X or even polling? > >>> > >>> Notifications are delivered through MSI. > >>> > >> > >> So no INT-X or MSI-X? Why so? > >> > >> Anyhow, my objective is not to get answers for my above questions here in this > >> thread, but to state the reality that it would be hard for us to make use of > >> this new transport without defining the bus implementation. > >> > > +1 to most of the points that Manivannan explained. > > > > The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet another interface for _Everything_ that already exists. > > and device is still have to implement all the existing things because the device does not know which driver will operate. > > > > And that too some register based inefficient interface. > > Just to reset the device one needs to fully setup the new message interface but device still have to be working. > > That defeats the whole purpose of reset_1 and reset_2 in the device. > > > > This does not bring anything better for the PCI devices at all. > > > > A transport binding should be defined for the bus binding. > > A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. > > > > If we are creating something brand-new, for PCI the only thing needed is: > > 1. Reset the device > > 2. Create an admin virtqueue > > 3. Transport everything needed through this virtqueue including features, configs, control. > > > > And this will work for any other bus or msg based too given only contract needed is to creating the aq. > > I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. > > You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". > > When the virtio-msg bus over PCI it will communicate through this device with an external > system connected through the PCI bus. > The driver will enumerate virtio devices available behind this bus and register them so that > the corresponding virtio drivers are probed for them. > All virtio-msg messages required to communicate with those devices will be transferred through > a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the > virtqueues with all the devices on the bus. > > So the PCI device is not one virtio device but one bus behind which there can be many devices. > > Is this making the concept a bit clearer ? > > Cheers > Bertrand > So this is fundamentally pci over virtio-msg? > > > >>>> > >>>> And these are just a few questions that comes to the top of my head. There could > >>>> be plenty more. > >>>> > >>>> How can we expect all the virtio-msg bus implementations to adhere to the same > >>>> format so that the interoperability offered by the Virtio spec is guaranteed? > >>> > >>> We spent a lot of time thinking on that (this started around 2 years ago) and we > >>> discussed several use cases and did some PoC to try to have everything covered > >>> (secure to non secure and vm to vm using ffa, system to system over PCI or hardware > >>> messaging system, PCI, Xen specific implementation) to check the needs and try to > >>> cover as much as we can. > >>> > >>> Now there might be cases we missed but we think that having a purely message based > >>> interface between the bus and the transport and split responsibilities the way we did > >>> is allowing lots of different bus implementations without affecting the transport and > >>> driver/device implementations on top. > >>> > >>> We identified that a common use case will be for the bus to transfer messages using > >>> FIFOs to optimize speed (at the end you need to have a way to share memory between > >>> both sides so why not using a part of it to transfer the messages to and reduce the number > >>> of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in > >>> practice (so we might standardize the FIFO format in the future to allow even more code > >>> reuse between busses). > >>> > >> > >> Not just the FIFO format, but how that FIFO gets shared between the device and > >> the host also needs to be documented. Maybe for this initial transport version, > >> you can start with defining the FF-A bus implementation? > > ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:52 ` Michael S. Tsirkin @ 2026-02-25 10:55 ` Bertrand Marquis 2026-02-25 10:58 ` Michael S. Tsirkin 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 10:55 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Parav Pandit, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi > On 25 Feb 2026, at 11:52, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Wed, Feb 25, 2026 at 10:35:41AM +0000, Bertrand Marquis wrote: >> Hi Parav, >> >>> On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: >>> >>>> >>>> From: Manivannan Sadhasivam <mani@kernel.org> >>>> Sent: 25 February 2026 03:37 PM >>>> >>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: >>>>> Hi Manivannan, >>>>> >>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: >>>>>> >>>>>> Hi Bertrand, >>>>>> >>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>>>>> Hi Parav, >>>>>>> >>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com> >>>>>>>>> Sent: 20 February 2026 05:25 AM >>>>>>>>> >>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: >>>>>>>>>> Hi Bill, >>>>>>>>>> >>>>>>>>>>> From: Bill Mills <bill.mills@linaro.org> >>>>>>>>>>> Sent: 26 January 2026 10:02 PM >>>>>>>>>>> >>>>>>>>>>> This series adds the virtio-msg transport layer. >>>>>>>>>>> >>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in >>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one >>>>>>>>>>> more transport that performs its transport layer operations by sending and >>>>>>>>>>> receiving messages. >>>>>>>>>>> >>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>>>>>>>> * common / device level >>>>>>>>>>> * bus level >>>>>>>>>>> >>>>>>>>>>> The common / device level defines the messages exchanged between the driver >>>>>>>>>>> and a device. This common part should lead to a common driver holding most >>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>>>>>>>> main focus of this version of the patch series. >>>>>>>>>>> >>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines >>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>>>>>>>> receive operations. A number of bus implementations are envisioned, >>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations >>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card >>>>>>>>>>> and its driver. >>>>>>>>>>> >>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations >>>>>>>>>>> doing the same thing in different ways for no good reason. However the >>>>>>>>>>> different environments will require different things. Instead of trying to >>>>>>>>>>> anticipate all needs and provide something very abstract, we think >>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time, >>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to >>>>>>>>>>> standardize a bus level message for that. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I would review more, had first round of sparse review. >>>>>>>>>> Please find few comments/questions below. >>>>>>>>> >>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and >>>>>>>>> then add performance features on top as appropriate. >>>>>>>> Sounds good. Simple but complete is needed. >>>>>>> >>>>>>> Agree. >>>>>>> >>>>>>>> >>>>>>>>> So one way to address some of these comments is to show how >>>>>>>>> they can be addressed with a feature bit down the road. >>>>>>>>> >>>>>>>>> >>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>>>>>>>>> >>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>>>>>>>>> Needs 32-bits. >>>>>>>>>> >>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>>>>>>>>> But more below. >>>>>>>>>> >>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to >>>> operate >>>>>>>>> the virtqueues. >>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >>>>>>>>> >>>>>>>>> surely this can be an optional transport feature bit. >>>>>>>>> >>>>>>>> How is this optional? >>>>>>> >>>>>>> As said in a previous mail, we have messages already for that. >>>>>>> Please confirm if that answer your question. >>>>>>> >>>>>>>> How can one implement a transport without defining the basic data transfer semantics? >>>>>>> >>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>>>>> >>>>>> >>>>>> I'd love to have this transport over PCI because it addresses the shortcomings >>>>>> of the existing PCI transport which just assumes that every config space access\ >>>>>> is trap and emulate. >>>>> >>>>> Agree and AMD did exactly that in their demonstrator. >>>>> I will give you answers here as i know them but Edgar will probably give you more >>>>> details (and probably fix my mistakes). >>>>> >>>>>> >>>>>> But that being said, I somewhat agree with Parav that we should define the bus >>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For >>>>>> instance, if we try to use this transport over PCI, we've got questions like: >>>>>> >>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with >>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? >>>>> >>>>> One bus is appearing as one pci device with its own Vendor ID, >>>>> >>>> >>>> What should be the 'own Vendor ID' here? >>>> >>>> The existing virtio-pci driver binds to all devices with the Vendor ID of >>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own >>>> VID for exposing the Virtio devices? That would mean, the drivers on the host >>>> need update as well, which will not scale. >>>> >>>> It would be good if the existing virtio-pci devices can use this new transport >>>> with only device side modifications. >>>> >>>>>> >>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config >>>>>> space or through some other means? >>>>> >>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI >>>>> device (ending up being memory shared between both sides) >>>>> >>>> >>>> What should be the BAR number and size? >>>> >>>>>> >>>>>> 3. How the notification be delivered from the device to the host? Through >>>>>> INT-X/MSI/MSI-X or even polling? >>>>> >>>>> Notifications are delivered through MSI. >>>>> >>>> >>>> So no INT-X or MSI-X? Why so? >>>> >>>> Anyhow, my objective is not to get answers for my above questions here in this >>>> thread, but to state the reality that it would be hard for us to make use of >>>> this new transport without defining the bus implementation. >>>> >>> +1 to most of the points that Manivannan explained. >>> >>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet another interface for _Everything_ that already exists. >>> and device is still have to implement all the existing things because the device does not know which driver will operate. >>> >>> And that too some register based inefficient interface. >>> Just to reset the device one needs to fully setup the new message interface but device still have to be working. >>> That defeats the whole purpose of reset_1 and reset_2 in the device. >>> >>> This does not bring anything better for the PCI devices at all. >>> >>> A transport binding should be defined for the bus binding. >>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. >>> >>> If we are creating something brand-new, for PCI the only thing needed is: >>> 1. Reset the device >>> 2. Create an admin virtqueue >>> 3. Transport everything needed through this virtqueue including features, configs, control. >>> >>> And this will work for any other bus or msg based too given only contract needed is to creating the aq. >> >> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. >> >> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". >> >> When the virtio-msg bus over PCI it will communicate through this device with an external >> system connected through the PCI bus. >> The driver will enumerate virtio devices available behind this bus and register them so that >> the corresponding virtio drivers are probed for them. >> All virtio-msg messages required to communicate with those devices will be transferred through >> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the >> virtqueues with all the devices on the bus. >> >> So the PCI device is not one virtio device but one bus behind which there can be many devices. >> >> Is this making the concept a bit clearer ? >> >> Cheers >> Bertrand >> > > So this is fundamentally pci over virtio-msg? we are not doing anything PCI, we use PCI bars and device as a way to implement a virtio-msg bus. The devices on top are virtio devices not PCI devices and they use a virtio-msg transport not PCI. PCI here is only used as a communication system between 2 heterogenous systems connected by PCI where one system is seen by the other as a PCI device. pci over virtio-msg is making me think we provide some pci functionality over virtio-msg. Cheers Bertrand > > >>> >>>>>> >>>>>> And these are just a few questions that comes to the top of my head. There could >>>>>> be plenty more. >>>>>> >>>>>> How can we expect all the virtio-msg bus implementations to adhere to the same >>>>>> format so that the interoperability offered by the Virtio spec is guaranteed? >>>>> >>>>> We spent a lot of time thinking on that (this started around 2 years ago) and we >>>>> discussed several use cases and did some PoC to try to have everything covered >>>>> (secure to non secure and vm to vm using ffa, system to system over PCI or hardware >>>>> messaging system, PCI, Xen specific implementation) to check the needs and try to >>>>> cover as much as we can. >>>>> >>>>> Now there might be cases we missed but we think that having a purely message based >>>>> interface between the bus and the transport and split responsibilities the way we did >>>>> is allowing lots of different bus implementations without affecting the transport and >>>>> driver/device implementations on top. >>>>> >>>>> We identified that a common use case will be for the bus to transfer messages using >>>>> FIFOs to optimize speed (at the end you need to have a way to share memory between >>>>> both sides so why not using a part of it to transfer the messages to and reduce the number >>>>> of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in >>>>> practice (so we might standardize the FIFO format in the future to allow even more code >>>>> reuse between busses). >>>>> >>>> >>>> Not just the FIFO format, but how that FIFO gets shared between the device and >>>> the host also needs to be documented. Maybe for this initial transport version, >>>> you can start with defining the FF-A bus implementation? IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:55 ` Bertrand Marquis @ 2026-02-25 10:58 ` Michael S. Tsirkin 0 siblings, 0 replies; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 10:58 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 10:55:05AM +0000, Bertrand Marquis wrote: > Hi > > > On 25 Feb 2026, at 11:52, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Feb 25, 2026 at 10:35:41AM +0000, Bertrand Marquis wrote: > >> Hi Parav, > >> > >>> On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: > >>> > >>>> > >>>> From: Manivannan Sadhasivam <mani@kernel.org> > >>>> Sent: 25 February 2026 03:37 PM > >>>> > >>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > >>>>> Hi Manivannan, > >>>>> > >>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > >>>>>> > >>>>>> Hi Bertrand, > >>>>>> > >>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > >>>>>>> Hi Parav, > >>>>>>> > >>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com> > >>>>>>>>> Sent: 20 February 2026 05:25 AM > >>>>>>>>> > >>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > >>>>>>>>>> Hi Bill, > >>>>>>>>>> > >>>>>>>>>>> From: Bill Mills <bill.mills@linaro.org> > >>>>>>>>>>> Sent: 26 January 2026 10:02 PM > >>>>>>>>>>> > >>>>>>>>>>> This series adds the virtio-msg transport layer. > >>>>>>>>>>> > >>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in > >>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one > >>>>>>>>>>> more transport that performs its transport layer operations by sending and > >>>>>>>>>>> receiving messages. > >>>>>>>>>>> > >>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: > >>>>>>>>>>> * common / device level > >>>>>>>>>>> * bus level > >>>>>>>>>>> > >>>>>>>>>>> The common / device level defines the messages exchanged between the driver > >>>>>>>>>>> and a device. This common part should lead to a common driver holding most > >>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport > >>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device > >>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > >>>>>>>>>>> main focus of this version of the patch series. > >>>>>>>>>>> > >>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines > >>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and > >>>>>>>>>>> receive operations. A number of bus implementations are envisioned, > >>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations > >>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card > >>>>>>>>>>> and its driver. > >>>>>>>>>>> > >>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations > >>>>>>>>>>> doing the same thing in different ways for no good reason. However the > >>>>>>>>>>> different environments will require different things. Instead of trying to > >>>>>>>>>>> anticipate all needs and provide something very abstract, we think > >>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time, > >>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to > >>>>>>>>>>> standardize a bus level message for that. > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> I would review more, had first round of sparse review. > >>>>>>>>>> Please find few comments/questions below. > >>>>>>>>> > >>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and > >>>>>>>>> then add performance features on top as appropriate. > >>>>>>>> Sounds good. Simple but complete is needed. > >>>>>>> > >>>>>>> Agree. > >>>>>>> > >>>>>>>> > >>>>>>>>> So one way to address some of these comments is to show how > >>>>>>>>> they can be addressed with a feature bit down the road. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. > >>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > >>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > >>>>>>>>>> > >>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > >>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > >>>>>>>>>> Needs 32-bits. > >>>>>>>>>> > >>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > >>>>>>>>>> But more below. > >>>>>>>>>> > >>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > >>>> operate > >>>>>>>>> the virtqueues. > >>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >>>>>>>>> > >>>>>>>>> surely this can be an optional transport feature bit. > >>>>>>>>> > >>>>>>>> How is this optional? > >>>>>>> > >>>>>>> As said in a previous mail, we have messages already for that. > >>>>>>> Please confirm if that answer your question. > >>>>>>> > >>>>>>>> How can one implement a transport without defining the basic data transfer semantics? > >>>>>>> > >>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > >>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue. > >>>>>>> > >>>>>> > >>>>>> I'd love to have this transport over PCI because it addresses the shortcomings > >>>>>> of the existing PCI transport which just assumes that every config space access\ > >>>>>> is trap and emulate. > >>>>> > >>>>> Agree and AMD did exactly that in their demonstrator. > >>>>> I will give you answers here as i know them but Edgar will probably give you more > >>>>> details (and probably fix my mistakes). > >>>>> > >>>>>> > >>>>>> But that being said, I somewhat agree with Parav that we should define the bus > >>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For > >>>>>> instance, if we try to use this transport over PCI, we've got questions like: > >>>>>> > >>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with > >>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > >>>>> > >>>>> One bus is appearing as one pci device with its own Vendor ID, > >>>>> > >>>> > >>>> What should be the 'own Vendor ID' here? > >>>> > >>>> The existing virtio-pci driver binds to all devices with the Vendor ID of > >>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > >>>> VID for exposing the Virtio devices? That would mean, the drivers on the host > >>>> need update as well, which will not scale. > >>>> > >>>> It would be good if the existing virtio-pci devices can use this new transport > >>>> with only device side modifications. > >>>> > >>>>>> > >>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config > >>>>>> space or through some other means? > >>>>> > >>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI > >>>>> device (ending up being memory shared between both sides) > >>>>> > >>>> > >>>> What should be the BAR number and size? > >>>> > >>>>>> > >>>>>> 3. How the notification be delivered from the device to the host? Through > >>>>>> INT-X/MSI/MSI-X or even polling? > >>>>> > >>>>> Notifications are delivered through MSI. > >>>>> > >>>> > >>>> So no INT-X or MSI-X? Why so? > >>>> > >>>> Anyhow, my objective is not to get answers for my above questions here in this > >>>> thread, but to state the reality that it would be hard for us to make use of > >>>> this new transport without defining the bus implementation. > >>>> > >>> +1 to most of the points that Manivannan explained. > >>> > >>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet another interface for _Everything_ that already exists. > >>> and device is still have to implement all the existing things because the device does not know which driver will operate. > >>> > >>> And that too some register based inefficient interface. > >>> Just to reset the device one needs to fully setup the new message interface but device still have to be working. > >>> That defeats the whole purpose of reset_1 and reset_2 in the device. > >>> > >>> This does not bring anything better for the PCI devices at all. > >>> > >>> A transport binding should be defined for the bus binding. > >>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. > >>> > >>> If we are creating something brand-new, for PCI the only thing needed is: > >>> 1. Reset the device > >>> 2. Create an admin virtqueue > >>> 3. Transport everything needed through this virtqueue including features, configs, control. > >>> > >>> And this will work for any other bus or msg based too given only contract needed is to creating the aq. > >> > >> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. > >> > >> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". > >> > >> When the virtio-msg bus over PCI it will communicate through this device with an external > >> system connected through the PCI bus. > >> The driver will enumerate virtio devices available behind this bus and register them so that > >> the corresponding virtio drivers are probed for them. > >> All virtio-msg messages required to communicate with those devices will be transferred through > >> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the > >> virtqueues with all the devices on the bus. > >> > >> So the PCI device is not one virtio device but one bus behind which there can be many devices. > >> > >> Is this making the concept a bit clearer ? > >> > >> Cheers > >> Bertrand > >> > > > > So this is fundamentally pci over virtio-msg? > > we are not doing anything PCI, we use PCI bars and device as a way to implement a virtio-msg > bus. The devices on top are virtio devices not PCI devices and they use a virtio-msg transport > not PCI. > > PCI here is only used as a communication system between 2 heterogenous systems connected > by PCI where one system is seen by the other as a PCI device. > > pci over virtio-msg is making me think we provide some pci functionality over virtio-msg. > > Cheers > Bertrand > it might make sense to write this up a bit more formally then. ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:35 ` Bertrand Marquis 2026-02-25 10:52 ` Michael S. Tsirkin @ 2026-02-25 14:45 ` Parav Pandit 2026-02-25 14:49 ` Michael S. Tsirkin 1 sibling, 1 reply; 105+ messages in thread From: Parav Pandit @ 2026-02-25 14:45 UTC (permalink / raw) To: Bertrand Marquis Cc: Manivannan Sadhasivam, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Bertrand Marquis <Bertrand.Marquis@arm.com> > Sent: 25 February 2026 04:06 PM > > Hi Parav, > > > On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: > > > >> > >> From: Manivannan Sadhasivam <mani@kernel.org> > >> Sent: 25 February 2026 03:37 PM > >> > >> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > >>> Hi Manivannan, > >>> > >>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > >>>> > >>>> Hi Bertrand, > >>>> > >>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > >>>>> Hi Parav, > >>>>> > >>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>>> From: Michael S. Tsirkin <mst@redhat.com> > >>>>>>> Sent: 20 February 2026 05:25 AM > >>>>>>> > >>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > >>>>>>>> Hi Bill, > >>>>>>>> > >>>>>>>>> From: Bill Mills <bill.mills@linaro.org> > >>>>>>>>> Sent: 26 January 2026 10:02 PM > >>>>>>>>> > >>>>>>>>> This series adds the virtio-msg transport layer. > >>>>>>>>> > >>>>>>>>> The individuals and organizations involved in this effort have had difficulty in > >>>>>>>>> using the existing virtio-transports in various situations and desire to add one > >>>>>>>>> more transport that performs its transport layer operations by sending and > >>>>>>>>> receiving messages. > >>>>>>>>> > >>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: > >>>>>>>>> * common / device level > >>>>>>>>> * bus level > >>>>>>>>> > >>>>>>>>> The common / device level defines the messages exchanged between the driver > >>>>>>>>> and a device. This common part should lead to a common driver holding most > >>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport > >>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device > >>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > >>>>>>>>> main focus of this version of the patch series. > >>>>>>>>> > >>>>>>>>> The virtio-msg bus level implements the normal things a bus defines > >>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and > >>>>>>>>> receive operations. A number of bus implementations are envisioned, > >>>>>>>>> some of which will be reusable and general purpose. Other bus implementations > >>>>>>>>> might be unique to a given situation, for example only used by a PCIe card > >>>>>>>>> and its driver. > >>>>>>>>> > >>>>>>>>> The standard bus messages are an effort to avoid different bus implementations > >>>>>>>>> doing the same thing in different ways for no good reason. However the > >>>>>>>>> different environments will require different things. Instead of trying to > >>>>>>>>> anticipate all needs and provide something very abstract, we think > >>>>>>>>> implementation specific messages will be needed at the bus level. Over time, > >>>>>>>>> if we see similar messages across multiple bus implementations, we will move to > >>>>>>>>> standardize a bus level message for that. > >>>>>>>>> > >>>>>>>> > >>>>>>>> I would review more, had first round of sparse review. > >>>>>>>> Please find few comments/questions below. > >>>>>>> > >>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and > >>>>>>> then add performance features on top as appropriate. > >>>>>> Sounds good. Simple but complete is needed. > >>>>> > >>>>> Agree. > >>>>> > >>>>>> > >>>>>>> So one way to address some of these comments is to show how > >>>>>>> they can be addressed with a feature bit down the road. > >>>>>>> > >>>>>>> > >>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. > >>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > >>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > >>>>>>>> > >>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > >>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > >>>>>>>> Needs 32-bits. > >>>>>>>> > >>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > >>>>>>>> But more below. > >>>>>>>> > >>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > >> operate > >>>>>>> the virtqueues. > >>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >>>>>>> > >>>>>>> surely this can be an optional transport feature bit. > >>>>>>> > >>>>>> How is this optional? > >>>>> > >>>>> As said in a previous mail, we have messages already for that. > >>>>> Please confirm if that answer your question. > >>>>> > >>>>>> How can one implement a transport without defining the basic data transfer semantics? > >>>>> > >>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > >>>>> If anything is missing, we are more than happy to discuss it and solve the issue. > >>>>> > >>>> > >>>> I'd love to have this transport over PCI because it addresses the shortcomings > >>>> of the existing PCI transport which just assumes that every config space access\ > >>>> is trap and emulate. > >>> > >>> Agree and AMD did exactly that in their demonstrator. > >>> I will give you answers here as i know them but Edgar will probably give you more > >>> details (and probably fix my mistakes). > >>> > >>>> > >>>> But that being said, I somewhat agree with Parav that we should define the bus > >>>> implementations in the spec to avoid fixing the ABI in the implementations. For > >>>> instance, if we try to use this transport over PCI, we've got questions like: > >>>> > >>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with > >>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > >>> > >>> One bus is appearing as one pci device with its own Vendor ID, > >>> > >> > >> What should be the 'own Vendor ID' here? > >> > >> The existing virtio-pci driver binds to all devices with the Vendor ID of > >> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > >> VID for exposing the Virtio devices? That would mean, the drivers on the host > >> need update as well, which will not scale. > >> > >> It would be good if the existing virtio-pci devices can use this new transport > >> with only device side modifications. > >> > >>>> > >>>> 2. How the Virtio messages should be transferred? Is it through endpoint config > >>>> space or through some other means? > >>> > >>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI > >>> device (ending up being memory shared between both sides) > >>> > >> > >> What should be the BAR number and size? > >> > >>>> > >>>> 3. How the notification be delivered from the device to the host? Through > >>>> INT-X/MSI/MSI-X or even polling? > >>> > >>> Notifications are delivered through MSI. > >>> > >> > >> So no INT-X or MSI-X? Why so? > >> > >> Anyhow, my objective is not to get answers for my above questions here in this > >> thread, but to state the reality that it would be hard for us to make use of > >> this new transport without defining the bus implementation. > >> > > +1 to most of the points that Manivannan explained. > > > > The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet > another interface for _Everything_ that already exists. > > and device is still have to implement all the existing things because the device does not know which driver will operate. > > > > And that too some register based inefficient interface. > > Just to reset the device one needs to fully setup the new message interface but device still have to be working. > > That defeats the whole purpose of reset_1 and reset_2 in the device. > > > > This does not bring anything better for the PCI devices at all. > > > > A transport binding should be defined for the bus binding. > > A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. > > > > If we are creating something brand-new, for PCI the only thing needed is: > > 1. Reset the device > > 2. Create an admin virtqueue > > 3. Transport everything needed through this virtqueue including features, configs, control. > > > > And this will work for any other bus or msg based too given only contract needed is to creating the aq. > > I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. > > You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". > > When the virtio-msg bus over PCI it will communicate through this device with an external > system connected through the PCI bus. > The driver will enumerate virtio devices available behind this bus and register them so that > the corresponding virtio drivers are probed for them. > All virtio-msg messages required to communicate with those devices will be transferred through > a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the > virtqueues with all the devices on the bus. > > So the PCI device is not one virtio device but one bus behind which there can be many devices. > > Is this making the concept a bit clearer ? > Yes. This makes a lot of sense now. This is a virtio-msg-transport device that needs its own device id in the table. And its binding to the PCI transport. So that device producer can implement this standard device and driver developer can develop the driver for multiplexing by reading the spec. > Cheers > Bertrand > > > > > >>>> > >>>> And these are just a few questions that comes to the top of my head. There could > >>>> be plenty more. > >>>> > >>>> How can we expect all the virtio-msg bus implementations to adhere to the same > >>>> format so that the interoperability offered by the Virtio spec is guaranteed? > >>> > >>> We spent a lot of time thinking on that (this started around 2 years ago) and we > >>> discussed several use cases and did some PoC to try to have everything covered > >>> (secure to non secure and vm to vm using ffa, system to system over PCI or hardware > >>> messaging system, PCI, Xen specific implementation) to check the needs and try to > >>> cover as much as we can. > >>> > >>> Now there might be cases we missed but we think that having a purely message based > >>> interface between the bus and the transport and split responsibilities the way we did > >>> is allowing lots of different bus implementations without affecting the transport and > >>> driver/device implementations on top. > >>> > >>> We identified that a common use case will be for the bus to transfer messages using > >>> FIFOs to optimize speed (at the end you need to have a way to share memory between > >>> both sides so why not using a part of it to transfer the messages to and reduce the number > >>> of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in > >>> practice (so we might standardize the FIFO format in the future to allow even more code > >>> reuse between busses). > >>> > >> > >> Not just the FIFO format, but how that FIFO gets shared between the device and > >> the host also needs to be documented. Maybe for this initial transport version, > >> you can start with defining the FF-A bus implementation? > > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy > the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 14:45 ` Parav Pandit @ 2026-02-25 14:49 ` Michael S. Tsirkin 2026-02-25 14:53 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 14:49 UTC (permalink / raw) To: Parav Pandit Cc: Bertrand Marquis, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 02:45:35PM +0000, Parav Pandit wrote: > > > From: Bertrand Marquis <Bertrand.Marquis@arm.com> > > Sent: 25 February 2026 04:06 PM > > > > Hi Parav, > > > > > On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: > > > > > >> > > >> From: Manivannan Sadhasivam <mani@kernel.org> > > >> Sent: 25 February 2026 03:37 PM > > >> > > >> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > > >>> Hi Manivannan, > > >>> > > >>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > > >>>> > > >>>> Hi Bertrand, > > >>>> > > >>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > > >>>>> Hi Parav, > > >>>>> > > >>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>> From: Michael S. Tsirkin <mst@redhat.com> > > >>>>>>> Sent: 20 February 2026 05:25 AM > > >>>>>>> > > >>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > > >>>>>>>> Hi Bill, > > >>>>>>>> > > >>>>>>>>> From: Bill Mills <bill.mills@linaro.org> > > >>>>>>>>> Sent: 26 January 2026 10:02 PM > > >>>>>>>>> > > >>>>>>>>> This series adds the virtio-msg transport layer. > > >>>>>>>>> > > >>>>>>>>> The individuals and organizations involved in this effort have had difficulty in > > >>>>>>>>> using the existing virtio-transports in various situations and desire to add one > > >>>>>>>>> more transport that performs its transport layer operations by sending and > > >>>>>>>>> receiving messages. > > >>>>>>>>> > > >>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: > > >>>>>>>>> * common / device level > > >>>>>>>>> * bus level > > >>>>>>>>> > > >>>>>>>>> The common / device level defines the messages exchanged between the driver > > >>>>>>>>> and a device. This common part should lead to a common driver holding most > > >>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > > >>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport > > >>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device > > >>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > > >>>>>>>>> main focus of this version of the patch series. > > >>>>>>>>> > > >>>>>>>>> The virtio-msg bus level implements the normal things a bus defines > > >>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and > > >>>>>>>>> receive operations. A number of bus implementations are envisioned, > > >>>>>>>>> some of which will be reusable and general purpose. Other bus implementations > > >>>>>>>>> might be unique to a given situation, for example only used by a PCIe card > > >>>>>>>>> and its driver. > > >>>>>>>>> > > >>>>>>>>> The standard bus messages are an effort to avoid different bus implementations > > >>>>>>>>> doing the same thing in different ways for no good reason. However the > > >>>>>>>>> different environments will require different things. Instead of trying to > > >>>>>>>>> anticipate all needs and provide something very abstract, we think > > >>>>>>>>> implementation specific messages will be needed at the bus level. Over time, > > >>>>>>>>> if we see similar messages across multiple bus implementations, we will move to > > >>>>>>>>> standardize a bus level message for that. > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> I would review more, had first round of sparse review. > > >>>>>>>> Please find few comments/questions below. > > >>>>>>> > > >>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and > > >>>>>>> then add performance features on top as appropriate. > > >>>>>> Sounds good. Simple but complete is needed. > > >>>>> > > >>>>> Agree. > > >>>>> > > >>>>>> > > >>>>>>> So one way to address some of these comments is to show how > > >>>>>>> they can be addressed with a feature bit down the road. > > >>>>>>> > > >>>>>>> > > >>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. > > >>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > > >>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > > >>>>>>>> > > >>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > > >>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > > >>>>>>>> Needs 32-bits. > > >>>>>>>> > > >>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > > >>>>>>>> But more below. > > >>>>>>>> > > >>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > > >> operate > > >>>>>>> the virtqueues. > > >>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > >>>>>>> > > >>>>>>> surely this can be an optional transport feature bit. > > >>>>>>> > > >>>>>> How is this optional? > > >>>>> > > >>>>> As said in a previous mail, we have messages already for that. > > >>>>> Please confirm if that answer your question. > > >>>>> > > >>>>>> How can one implement a transport without defining the basic data transfer semantics? > > >>>>> > > >>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > > >>>>> If anything is missing, we are more than happy to discuss it and solve the issue. > > >>>>> > > >>>> > > >>>> I'd love to have this transport over PCI because it addresses the shortcomings > > >>>> of the existing PCI transport which just assumes that every config space access\ > > >>>> is trap and emulate. > > >>> > > >>> Agree and AMD did exactly that in their demonstrator. > > >>> I will give you answers here as i know them but Edgar will probably give you more > > >>> details (and probably fix my mistakes). > > >>> > > >>>> > > >>>> But that being said, I somewhat agree with Parav that we should define the bus > > >>>> implementations in the spec to avoid fixing the ABI in the implementations. For > > >>>> instance, if we try to use this transport over PCI, we've got questions like: > > >>>> > > >>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with > > >>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > >>> > > >>> One bus is appearing as one pci device with its own Vendor ID, > > >>> > > >> > > >> What should be the 'own Vendor ID' here? > > >> > > >> The existing virtio-pci driver binds to all devices with the Vendor ID of > > >> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > > >> VID for exposing the Virtio devices? That would mean, the drivers on the host > > >> need update as well, which will not scale. > > >> > > >> It would be good if the existing virtio-pci devices can use this new transport > > >> with only device side modifications. > > >> > > >>>> > > >>>> 2. How the Virtio messages should be transferred? Is it through endpoint config > > >>>> space or through some other means? > > >>> > > >>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI > > >>> device (ending up being memory shared between both sides) > > >>> > > >> > > >> What should be the BAR number and size? > > >> > > >>>> > > >>>> 3. How the notification be delivered from the device to the host? Through > > >>>> INT-X/MSI/MSI-X or even polling? > > >>> > > >>> Notifications are delivered through MSI. > > >>> > > >> > > >> So no INT-X or MSI-X? Why so? > > >> > > >> Anyhow, my objective is not to get answers for my above questions here in this > > >> thread, but to state the reality that it would be hard for us to make use of > > >> this new transport without defining the bus implementation. > > >> > > > +1 to most of the points that Manivannan explained. > > > > > > The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet > > another interface for _Everything_ that already exists. > > > and device is still have to implement all the existing things because the device does not know which driver will operate. > > > > > > And that too some register based inefficient interface. > > > Just to reset the device one needs to fully setup the new message interface but device still have to be working. > > > That defeats the whole purpose of reset_1 and reset_2 in the device. > > > > > > This does not bring anything better for the PCI devices at all. > > > > > > A transport binding should be defined for the bus binding. > > > A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. > > > > > > If we are creating something brand-new, for PCI the only thing needed is: > > > 1. Reset the device > > > 2. Create an admin virtqueue > > > 3. Transport everything needed through this virtqueue including features, configs, control. > > > > > > And this will work for any other bus or msg based too given only contract needed is to creating the aq. > > > > I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. > > > > You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". > > > > When the virtio-msg bus over PCI it will communicate through this device with an external > > system connected through the PCI bus. > > The driver will enumerate virtio devices available behind this bus and register them so that > > the corresponding virtio drivers are probed for them. > > All virtio-msg messages required to communicate with those devices will be transferred through > > a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the > > virtqueues with all the devices on the bus. > > > > So the PCI device is not one virtio device but one bus behind which there can be many devices. > > > > Is this making the concept a bit clearer ? > > > Yes. This makes a lot of sense now. > > This is a virtio-msg-transport device that needs its own device id in the table. > And its binding to the PCI transport. ok. how about an rfc of that idea on the list? > So that device producer can implement this standard device and driver developer can develop the driver for multiplexing by reading the spec. > > > Cheers > > Bertrand > > > > > > > > > >>>> > > >>>> And these are just a few questions that comes to the top of my head. There could > > >>>> be plenty more. > > >>>> > > >>>> How can we expect all the virtio-msg bus implementations to adhere to the same > > >>>> format so that the interoperability offered by the Virtio spec is guaranteed? > > >>> > > >>> We spent a lot of time thinking on that (this started around 2 years ago) and we > > >>> discussed several use cases and did some PoC to try to have everything covered > > >>> (secure to non secure and vm to vm using ffa, system to system over PCI or hardware > > >>> messaging system, PCI, Xen specific implementation) to check the needs and try to > > >>> cover as much as we can. > > >>> > > >>> Now there might be cases we missed but we think that having a purely message based > > >>> interface between the bus and the transport and split responsibilities the way we did > > >>> is allowing lots of different bus implementations without affecting the transport and > > >>> driver/device implementations on top. > > >>> > > >>> We identified that a common use case will be for the bus to transfer messages using > > >>> FIFOs to optimize speed (at the end you need to have a way to share memory between > > >>> both sides so why not using a part of it to transfer the messages to and reduce the number > > >>> of data exchanges and copies) and this will be used by PCI, Xen, FF-A and others in > > >>> practice (so we might standardize the FIFO format in the future to allow even more code > > >>> reuse between busses). > > >>> > > >> > > >> Not just the FIFO format, but how that FIFO gets shared between the device and > > >> the host also needs to be documented. Maybe for this initial transport version, > > >> you can start with defining the FF-A bus implementation? > > > > > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended > > recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy > > the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 14:49 ` Michael S. Tsirkin @ 2026-02-25 14:53 ` Bertrand Marquis 2026-02-25 15:00 ` Parav Pandit 2026-02-25 16:42 ` Edgar E. Iglesias 0 siblings, 2 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 14:53 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Parav Pandit, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi, > On 25 Feb 2026, at 15:49, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Wed, Feb 25, 2026 at 02:45:35PM +0000, Parav Pandit wrote: >> >>> From: Bertrand Marquis <Bertrand.Marquis@arm.com> >>> Sent: 25 February 2026 04:06 PM >>> >>> Hi Parav, >>> >>>> On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: >>>> >>>>> >>>>> From: Manivannan Sadhasivam <mani@kernel.org> >>>>> Sent: 25 February 2026 03:37 PM >>>>> >>>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: >>>>>> Hi Manivannan, >>>>>> >>>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: >>>>>>> >>>>>>> Hi Bertrand, >>>>>>> >>>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>>>>>> Hi Parav, >>>>>>>> >>>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com> >>>>>>>>>> Sent: 20 February 2026 05:25 AM >>>>>>>>>> >>>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: >>>>>>>>>>> Hi Bill, >>>>>>>>>>> >>>>>>>>>>>> From: Bill Mills <bill.mills@linaro.org> >>>>>>>>>>>> Sent: 26 January 2026 10:02 PM >>>>>>>>>>>> >>>>>>>>>>>> This series adds the virtio-msg transport layer. >>>>>>>>>>>> >>>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in >>>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one >>>>>>>>>>>> more transport that performs its transport layer operations by sending and >>>>>>>>>>>> receiving messages. >>>>>>>>>>>> >>>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>>>>>>>>> * common / device level >>>>>>>>>>>> * bus level >>>>>>>>>>>> >>>>>>>>>>>> The common / device level defines the messages exchanged between the driver >>>>>>>>>>>> and a device. This common part should lead to a common driver holding most >>>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>>>>>>>>> main focus of this version of the patch series. >>>>>>>>>>>> >>>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines >>>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>>>>>>>>> receive operations. A number of bus implementations are envisioned, >>>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations >>>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card >>>>>>>>>>>> and its driver. >>>>>>>>>>>> >>>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations >>>>>>>>>>>> doing the same thing in different ways for no good reason. However the >>>>>>>>>>>> different environments will require different things. Instead of trying to >>>>>>>>>>>> anticipate all needs and provide something very abstract, we think >>>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time, >>>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to >>>>>>>>>>>> standardize a bus level message for that. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I would review more, had first round of sparse review. >>>>>>>>>>> Please find few comments/questions below. >>>>>>>>>> >>>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and >>>>>>>>>> then add performance features on top as appropriate. >>>>>>>>> Sounds good. Simple but complete is needed. >>>>>>>> >>>>>>>> Agree. >>>>>>>> >>>>>>>>> >>>>>>>>>> So one way to address some of these comments is to show how >>>>>>>>>> they can be addressed with a feature bit down the road. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>>>>>>>>>> >>>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>>>>>>>>>> Needs 32-bits. >>>>>>>>>>> >>>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>>>>>>>>>> But more below. >>>>>>>>>>> >>>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to >>>>> operate >>>>>>>>>> the virtqueues. >>>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >>>>>>>>>> >>>>>>>>>> surely this can be an optional transport feature bit. >>>>>>>>>> >>>>>>>>> How is this optional? >>>>>>>> >>>>>>>> As said in a previous mail, we have messages already for that. >>>>>>>> Please confirm if that answer your question. >>>>>>>> >>>>>>>>> How can one implement a transport without defining the basic data transfer semantics? >>>>>>>> >>>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>>>>>> >>>>>>> >>>>>>> I'd love to have this transport over PCI because it addresses the shortcomings >>>>>>> of the existing PCI transport which just assumes that every config space access\ >>>>>>> is trap and emulate. >>>>>> >>>>>> Agree and AMD did exactly that in their demonstrator. >>>>>> I will give you answers here as i know them but Edgar will probably give you more >>>>>> details (and probably fix my mistakes). >>>>>> >>>>>>> >>>>>>> But that being said, I somewhat agree with Parav that we should define the bus >>>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For >>>>>>> instance, if we try to use this transport over PCI, we've got questions like: >>>>>>> >>>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with >>>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? >>>>>> >>>>>> One bus is appearing as one pci device with its own Vendor ID, >>>>>> >>>>> >>>>> What should be the 'own Vendor ID' here? >>>>> >>>>> The existing virtio-pci driver binds to all devices with the Vendor ID of >>>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own >>>>> VID for exposing the Virtio devices? That would mean, the drivers on the host >>>>> need update as well, which will not scale. >>>>> >>>>> It would be good if the existing virtio-pci devices can use this new transport >>>>> with only device side modifications. >>>>> >>>>>>> >>>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config >>>>>>> space or through some other means? >>>>>> >>>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI >>>>>> device (ending up being memory shared between both sides) >>>>>> >>>>> >>>>> What should be the BAR number and size? >>>>> >>>>>>> >>>>>>> 3. How the notification be delivered from the device to the host? Through >>>>>>> INT-X/MSI/MSI-X or even polling? >>>>>> >>>>>> Notifications are delivered through MSI. >>>>>> >>>>> >>>>> So no INT-X or MSI-X? Why so? >>>>> >>>>> Anyhow, my objective is not to get answers for my above questions here in this >>>>> thread, but to state the reality that it would be hard for us to make use of >>>>> this new transport without defining the bus implementation. >>>>> >>>> +1 to most of the points that Manivannan explained. >>>> >>>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet >>> another interface for _Everything_ that already exists. >>>> and device is still have to implement all the existing things because the device does not know which driver will operate. >>>> >>>> And that too some register based inefficient interface. >>>> Just to reset the device one needs to fully setup the new message interface but device still have to be working. >>>> That defeats the whole purpose of reset_1 and reset_2 in the device. >>>> >>>> This does not bring anything better for the PCI devices at all. >>>> >>>> A transport binding should be defined for the bus binding. >>>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. >>>> >>>> If we are creating something brand-new, for PCI the only thing needed is: >>>> 1. Reset the device >>>> 2. Create an admin virtqueue >>>> 3. Transport everything needed through this virtqueue including features, configs, control. >>>> >>>> And this will work for any other bus or msg based too given only contract needed is to creating the aq. >>> >>> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. >>> >>> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". >>> >>> When the virtio-msg bus over PCI it will communicate through this device with an external >>> system connected through the PCI bus. >>> The driver will enumerate virtio devices available behind this bus and register them so that >>> the corresponding virtio drivers are probed for them. >>> All virtio-msg messages required to communicate with those devices will be transferred through >>> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the >>> virtqueues with all the devices on the bus. >>> >>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>> >>> Is this making the concept a bit clearer ? >>> >> Yes. This makes a lot of sense now. >> >> This is a virtio-msg-transport device that needs its own device id in the table. >> And its binding to the PCI transport. > > ok. how about an rfc of that idea on the list? I will let Edgar answer on this. Cheers Bertrand IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 14:53 ` Bertrand Marquis @ 2026-02-25 15:00 ` Parav Pandit 2026-02-25 15:07 ` Parav Pandit ` (2 more replies) 2026-02-25 16:42 ` Edgar E. Iglesias 1 sibling, 3 replies; 105+ messages in thread From: Parav Pandit @ 2026-02-25 15:00 UTC (permalink / raw) To: Bertrand Marquis, Michael S. Tsirkin Cc: Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Bertrand Marquis <Bertrand.Marquis@arm.com> > Sent: 25 February 2026 08:24 PM > > Hi, > > > On 25 Feb 2026, at 15:49, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Feb 25, 2026 at 02:45:35PM +0000, Parav Pandit wrote: > >> > >>> From: Bertrand Marquis <Bertrand.Marquis@arm.com> > >>> Sent: 25 February 2026 04:06 PM > >>> > >>> Hi Parav, > >>> > >>>> On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: > >>>> > >>>>> > >>>>> From: Manivannan Sadhasivam <mani@kernel.org> > >>>>> Sent: 25 February 2026 03:37 PM > >>>>> > >>>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > >>>>>> Hi Manivannan, > >>>>>> > >>>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > >>>>>>> > >>>>>>> Hi Bertrand, > >>>>>>> > >>>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > >>>>>>>> Hi Parav, > >>>>>>>> > >>>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com> > >>>>>>>>>> Sent: 20 February 2026 05:25 AM > >>>>>>>>>> > >>>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > >>>>>>>>>>> Hi Bill, > >>>>>>>>>>> > >>>>>>>>>>>> From: Bill Mills <bill.mills@linaro.org> > >>>>>>>>>>>> Sent: 26 January 2026 10:02 PM > >>>>>>>>>>>> > >>>>>>>>>>>> This series adds the virtio-msg transport layer. > >>>>>>>>>>>> > >>>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in > >>>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one > >>>>>>>>>>>> more transport that performs its transport layer operations by sending and > >>>>>>>>>>>> receiving messages. > >>>>>>>>>>>> > >>>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: > >>>>>>>>>>>> * common / device level > >>>>>>>>>>>> * bus level > >>>>>>>>>>>> > >>>>>>>>>>>> The common / device level defines the messages exchanged between the driver > >>>>>>>>>>>> and a device. This common part should lead to a common driver holding most > >>>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >>>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport > >>>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device > >>>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > >>>>>>>>>>>> main focus of this version of the patch series. > >>>>>>>>>>>> > >>>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines > >>>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and > >>>>>>>>>>>> receive operations. A number of bus implementations are envisioned, > >>>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations > >>>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card > >>>>>>>>>>>> and its driver. > >>>>>>>>>>>> > >>>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations > >>>>>>>>>>>> doing the same thing in different ways for no good reason. However the > >>>>>>>>>>>> different environments will require different things. Instead of trying to > >>>>>>>>>>>> anticipate all needs and provide something very abstract, we think > >>>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time, > >>>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to > >>>>>>>>>>>> standardize a bus level message for that. > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I would review more, had first round of sparse review. > >>>>>>>>>>> Please find few comments/questions below. > >>>>>>>>>> > >>>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and > >>>>>>>>>> then add performance features on top as appropriate. > >>>>>>>>> Sounds good. Simple but complete is needed. > >>>>>>>> > >>>>>>>> Agree. > >>>>>>>> > >>>>>>>>> > >>>>>>>>>> So one way to address some of these comments is to show how > >>>>>>>>>> they can be addressed with a feature bit down the road. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. > >>>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > >>>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > >>>>>>>>>>> > >>>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > >>>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > >>>>>>>>>>> Needs 32-bits. > >>>>>>>>>>> > >>>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > >>>>>>>>>>> But more below. > >>>>>>>>>>> > >>>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses > to > >>>>> operate > >>>>>>>>>> the virtqueues. > >>>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >>>>>>>>>> > >>>>>>>>>> surely this can be an optional transport feature bit. > >>>>>>>>>> > >>>>>>>>> How is this optional? > >>>>>>>> > >>>>>>>> As said in a previous mail, we have messages already for that. > >>>>>>>> Please confirm if that answer your question. > >>>>>>>> > >>>>>>>>> How can one implement a transport without defining the basic data transfer semantics? > >>>>>>>> > >>>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > >>>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue. > >>>>>>>> > >>>>>>> > >>>>>>> I'd love to have this transport over PCI because it addresses the shortcomings > >>>>>>> of the existing PCI transport which just assumes that every config space access\ > >>>>>>> is trap and emulate. > >>>>>> > >>>>>> Agree and AMD did exactly that in their demonstrator. > >>>>>> I will give you answers here as i know them but Edgar will probably give you more > >>>>>> details (and probably fix my mistakes). > >>>>>> > >>>>>>> > >>>>>>> But that being said, I somewhat agree with Parav that we should define the bus > >>>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For > >>>>>>> instance, if we try to use this transport over PCI, we've got questions like: > >>>>>>> > >>>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with > >>>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > >>>>>> > >>>>>> One bus is appearing as one pci device with its own Vendor ID, > >>>>>> > >>>>> > >>>>> What should be the 'own Vendor ID' here? > >>>>> > >>>>> The existing virtio-pci driver binds to all devices with the Vendor ID of > >>>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > >>>>> VID for exposing the Virtio devices? That would mean, the drivers on the host > >>>>> need update as well, which will not scale. > >>>>> > >>>>> It would be good if the existing virtio-pci devices can use this new transport > >>>>> with only device side modifications. > >>>>> > >>>>>>> > >>>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config > >>>>>>> space or through some other means? > >>>>>> > >>>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI > >>>>>> device (ending up being memory shared between both sides) > >>>>>> > >>>>> > >>>>> What should be the BAR number and size? > >>>>> > >>>>>>> > >>>>>>> 3. How the notification be delivered from the device to the host? Through > >>>>>>> INT-X/MSI/MSI-X or even polling? > >>>>>> > >>>>>> Notifications are delivered through MSI. > >>>>>> > >>>>> > >>>>> So no INT-X or MSI-X? Why so? > >>>>> > >>>>> Anyhow, my objective is not to get answers for my above questions here in this > >>>>> thread, but to state the reality that it would be hard for us to make use of > >>>>> this new transport without defining the bus implementation. > >>>>> > >>>> +1 to most of the points that Manivannan explained. > >>>> > >>>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet > >>> another interface for _Everything_ that already exists. > >>>> and device is still have to implement all the existing things because the device does not know which driver will operate. > >>>> > >>>> And that too some register based inefficient interface. > >>>> Just to reset the device one needs to fully setup the new message interface but device still have to be working. > >>>> That defeats the whole purpose of reset_1 and reset_2 in the device. > >>>> > >>>> This does not bring anything better for the PCI devices at all. > >>>> > >>>> A transport binding should be defined for the bus binding. > >>>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. > >>>> > >>>> If we are creating something brand-new, for PCI the only thing needed is: > >>>> 1. Reset the device > >>>> 2. Create an admin virtqueue > >>>> 3. Transport everything needed through this virtqueue including features, configs, control. > >>>> > >>>> And this will work for any other bus or msg based too given only contract needed is to creating the aq. > >>> > >>> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. > >>> > >>> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". > >>> > >>> When the virtio-msg bus over PCI it will communicate through this device with an external > >>> system connected through the PCI bus. > >>> The driver will enumerate virtio devices available behind this bus and register them so that > >>> the corresponding virtio drivers are probed for them. > >>> All virtio-msg messages required to communicate with those devices will be transferred through > >>> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the > >>> virtqueues with all the devices on the bus. > >>> > >>> So the PCI device is not one virtio device but one bus behind which there can be many devices. > >>> > >>> Is this making the concept a bit clearer ? > >>> > >> Yes. This makes a lot of sense now. > >> > >> This is a virtio-msg-transport device that needs its own device id in the table. > >> And its binding to the PCI transport. > > > > ok. how about an rfc of that idea on the list? > > > I will let Edgar answer on this. > Sure. Also please explain how is this different than SR-IOV devices? On a PCI device there are some child devices exposed using some virtio-msg bus transport. And all of them are going to share same PCI BDF. These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. So what would this virtio-msg-transport device achieve? If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. But underlying transport being PCI, its back to square one for the originally described motivation. > Cheers > Bertrand > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended > recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy > the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:00 ` Parav Pandit @ 2026-02-25 15:07 ` Parav Pandit 2026-02-25 15:12 ` Bertrand Marquis 2026-02-25 15:11 ` Manivannan Sadhasivam 2026-02-25 15:23 ` Demi Marie Obenour 2 siblings, 1 reply; 105+ messages in thread From: Parav Pandit @ 2026-02-25 15:07 UTC (permalink / raw) To: Parav Pandit, Bertrand Marquis, Michael S. Tsirkin Cc: Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Bertrand, > From: Parav Pandit <parav@nvidia.com> > Sent: 25 February 2026 08:30 PM [..] > > >>> So the PCI device is not one virtio device but one bus behind which there can be many devices. > > >>> > > >>> Is this making the concept a bit clearer ? > > >>> > > >> Yes. This makes a lot of sense now. > > >> > > >> This is a virtio-msg-transport device that needs its own device id in the table. > > >> And its binding to the PCI transport. > > > > > > ok. how about an rfc of that idea on the list? > > > > > > I will let Edgar answer on this. > > > Sure. > Also please explain how is this different than SR-IOV devices? > On a PCI device there are some child devices exposed using some virtio-msg bus transport. > And all of them are going to share same PCI BDF. > These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > So what would this virtio-msg-transport device achieve? > > If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > But underlying transport being PCI, its back to square one for the originally described motivation. If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). This would help this series to progress faster without getting distracted on PCI side. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:07 ` Parav Pandit @ 2026-02-25 15:12 ` Bertrand Marquis 2026-02-25 15:15 ` Michael S. Tsirkin ` (2 more replies) 0 siblings, 3 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 15:12 UTC (permalink / raw) To: Parav Pandit Cc: Michael S. Tsirkin, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Parav, > On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: > > Hi Bertrand, > >> From: Parav Pandit <parav@nvidia.com> >> Sent: 25 February 2026 08:30 PM > > [..] >>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>>>>> >>>>>> Is this making the concept a bit clearer ? >>>>>> >>>>> Yes. This makes a lot of sense now. >>>>> >>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>> And its binding to the PCI transport. >>>> >>>> ok. how about an rfc of that idea on the list? >>> >>> >>> I will let Edgar answer on this. >>> >> Sure. >> Also please explain how is this different than SR-IOV devices? >> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >> And all of them are going to share same PCI BDF. >> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >> So what would this virtio-msg-transport device achieve? >> >> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >> But underlying transport being PCI, its back to square one for the originally described motivation. > > If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). > > This would help this series to progress faster without getting distracted on PCI side. Agree the example of virtio-msg over PCI made the discussion divert a bit. Maybe focusing on use cases like: - FF-A - messaging unit + shared memory - hypervisor based events + shared memory Would make the discussion focus more. Cheers Bertrand IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:12 ` Bertrand Marquis @ 2026-02-25 15:15 ` Michael S. Tsirkin 2026-02-25 15:36 ` Demi Marie Obenour 2026-02-26 5:40 ` Manivannan Sadhasivam 2 siblings, 0 replies; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 15:15 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 03:12:11PM +0000, Bertrand Marquis wrote: > Hi Parav, > > > On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: > > > > Hi Bertrand, > > > >> From: Parav Pandit <parav@nvidia.com> > >> Sent: 25 February 2026 08:30 PM > > > > [..] > >>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. > >>>>>> > >>>>>> Is this making the concept a bit clearer ? > >>>>>> > >>>>> Yes. This makes a lot of sense now. > >>>>> > >>>>> This is a virtio-msg-transport device that needs its own device id in the table. > >>>>> And its binding to the PCI transport. > >>>> > >>>> ok. how about an rfc of that idea on the list? > >>> > >>> > >>> I will let Edgar answer on this. > >>> > >> Sure. > >> Also please explain how is this different than SR-IOV devices? > >> On a PCI device there are some child devices exposed using some virtio-msg bus transport. > >> And all of them are going to share same PCI BDF. > >> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > >> So what would this virtio-msg-transport device achieve? > >> > >> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > >> But underlying transport being PCI, its back to square one for the originally described motivation. > > > > If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). > > > > This would help this series to progress faster without getting distracted on PCI side. > > Agree the example of virtio-msg over PCI made the discussion divert a bit. > > Maybe focusing on use cases like: > - FF-A > - messaging unit + shared memory > - hypervisor based events + shared memory > > > Would make the discussion focus more. > > Cheers > Bertrand > > indeed, possible. but do include ff-a bit for reference then. even if down the road it is not part of the spec (though no idea why and I would like to see). -- MST ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:12 ` Bertrand Marquis 2026-02-25 15:15 ` Michael S. Tsirkin @ 2026-02-25 15:36 ` Demi Marie Obenour 2026-02-25 15:40 ` Bertrand Marquis 2026-02-26 5:40 ` Manivannan Sadhasivam 2 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 15:36 UTC (permalink / raw) To: Bertrand Marquis, Parav Pandit Cc: Michael S. Tsirkin, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 2168 bytes --] On 2/25/26 10:12, Bertrand Marquis wrote: > Hi Parav, > >> On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: >> >> Hi Bertrand, >> >>> From: Parav Pandit <parav@nvidia.com> >>> Sent: 25 February 2026 08:30 PM >> >> [..] >>>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>>>>>> >>>>>>> Is this making the concept a bit clearer ? >>>>>>> >>>>>> Yes. This makes a lot of sense now. >>>>>> >>>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>>> And its binding to the PCI transport. >>>>> >>>>> ok. how about an rfc of that idea on the list? >>>> >>>> >>>> I will let Edgar answer on this. >>>> >>> Sure. >>> Also please explain how is this different than SR-IOV devices? >>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >>> And all of them are going to share same PCI BDF. >>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >>> So what would this virtio-msg-transport device achieve? >>> >>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >>> But underlying transport being PCI, its back to square one for the originally described motivation. >> >> If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). >> >> This would help this series to progress faster without getting distracted on PCI side. > > Agree the example of virtio-msg over PCI made the discussion divert a bit. > > Maybe focusing on use cases like: > - FF-A > - messaging unit + shared memory > - hypervisor based events + shared memory Is "hypervisor based events + shared memory" only really intended for Xen and other true type-1 hypervisors? Should KVM and other type-2 hypervisors use a different approach, like virtio-vhost-user? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:36 ` Demi Marie Obenour @ 2026-02-25 15:40 ` Bertrand Marquis 2026-02-25 15:48 ` Demi Marie Obenour 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 15:40 UTC (permalink / raw) To: Demi Marie Obenour Cc: Parav Pandit, Michael S. Tsirkin, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > On 25 Feb 2026, at 16:36, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 2/25/26 10:12, Bertrand Marquis wrote: >> Hi Parav, >> >>> On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: >>> >>> Hi Bertrand, >>> >>>> From: Parav Pandit <parav@nvidia.com> >>>> Sent: 25 February 2026 08:30 PM >>> >>> [..] >>>>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>>>>>>> >>>>>>>> Is this making the concept a bit clearer ? >>>>>>>> >>>>>>> Yes. This makes a lot of sense now. >>>>>>> >>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>>>> And its binding to the PCI transport. >>>>>> >>>>>> ok. how about an rfc of that idea on the list? >>>>> >>>>> >>>>> I will let Edgar answer on this. >>>>> >>>> Sure. >>>> Also please explain how is this different than SR-IOV devices? >>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >>>> And all of them are going to share same PCI BDF. >>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >>>> So what would this virtio-msg-transport device achieve? >>>> >>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >>>> But underlying transport being PCI, its back to square one for the originally described motivation. >>> >>> If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). >>> >>> This would help this series to progress faster without getting distracted on PCI side. >> >> Agree the example of virtio-msg over PCI made the discussion divert a bit. >> >> Maybe focusing on use cases like: >> - FF-A >> - messaging unit + shared memory >> - hypervisor based events + shared memory > > Is "hypervisor based events + shared memory" only really intended for > Xen and other true type-1 hypervisors? Should KVM and other type-2 > hypervisors use a different approach, like virtio-vhost-user? Well there is definitely a use case in xen but any other type 1 or type 2 hypervisor could benefit from using virtio-msg and have asynchronous handling instead of a trap and emulate based implementation. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:40 ` Bertrand Marquis @ 2026-02-25 15:48 ` Demi Marie Obenour 2026-02-25 15:51 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 15:48 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Michael S. Tsirkin, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 2821 bytes --] On 2/25/26 10:40, Bertrand Marquis wrote: > > >> On 25 Feb 2026, at 16:36, Demi Marie Obenour <demiobenour@gmail.com> wrote: >> >> On 2/25/26 10:12, Bertrand Marquis wrote: >>> Hi Parav, >>> >>>> On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: >>>> >>>> Hi Bertrand, >>>> >>>>> From: Parav Pandit <parav@nvidia.com> >>>>> Sent: 25 February 2026 08:30 PM >>>> >>>> [..] >>>>>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>>>>>>>> >>>>>>>>> Is this making the concept a bit clearer ? >>>>>>>>> >>>>>>>> Yes. This makes a lot of sense now. >>>>>>>> >>>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>>>>> And its binding to the PCI transport. >>>>>>> >>>>>>> ok. how about an rfc of that idea on the list? >>>>>> >>>>>> >>>>>> I will let Edgar answer on this. >>>>>> >>>>> Sure. >>>>> Also please explain how is this different than SR-IOV devices? >>>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >>>>> And all of them are going to share same PCI BDF. >>>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >>>>> So what would this virtio-msg-transport device achieve? >>>>> >>>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >>>>> But underlying transport being PCI, its back to square one for the originally described motivation. >>>> >>>> If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). >>>> >>>> This would help this series to progress faster without getting distracted on PCI side. >>> >>> Agree the example of virtio-msg over PCI made the discussion divert a bit. >>> >>> Maybe focusing on use cases like: >>> - FF-A >>> - messaging unit + shared memory >>> - hypervisor based events + shared memory >> >> Is "hypervisor based events + shared memory" only really intended for >> Xen and other true type-1 hypervisors? Should KVM and other type-2 >> hypervisors use a different approach, like virtio-vhost-user? > > Well there is definitely a use case in xen but any other type 1 or type 2 > hypervisor could benefit from using virtio-msg and have asynchronous > handling instead of a trap and emulate based implementation. Do irqfds and ioeventfds already provide this? What interface should the device-implementing VM use to provide the device? Should it be a new type of virtio device? Should this be standardized? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:48 ` Demi Marie Obenour @ 2026-02-25 15:51 ` Bertrand Marquis 2026-02-25 16:15 ` Demi Marie Obenour 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 15:51 UTC (permalink / raw) To: Demi Marie Obenour Cc: Parav Pandit, Michael S. Tsirkin, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > On 25 Feb 2026, at 16:48, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 2/25/26 10:40, Bertrand Marquis wrote: >> >> >>> On 25 Feb 2026, at 16:36, Demi Marie Obenour <demiobenour@gmail.com> wrote: >>> >>> On 2/25/26 10:12, Bertrand Marquis wrote: >>>> Hi Parav, >>>> >>>>> On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: >>>>> >>>>> Hi Bertrand, >>>>> >>>>>> From: Parav Pandit <parav@nvidia.com> >>>>>> Sent: 25 February 2026 08:30 PM >>>>> >>>>> [..] >>>>>>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>>>>>>>>> >>>>>>>>>> Is this making the concept a bit clearer ? >>>>>>>>>> >>>>>>>>> Yes. This makes a lot of sense now. >>>>>>>>> >>>>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>>>>>> And its binding to the PCI transport. >>>>>>>> >>>>>>>> ok. how about an rfc of that idea on the list? >>>>>>> >>>>>>> >>>>>>> I will let Edgar answer on this. >>>>>>> >>>>>> Sure. >>>>>> Also please explain how is this different than SR-IOV devices? >>>>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >>>>>> And all of them are going to share same PCI BDF. >>>>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >>>>>> So what would this virtio-msg-transport device achieve? >>>>>> >>>>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >>>>>> But underlying transport being PCI, its back to square one for the originally described motivation. >>>>> >>>>> If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). >>>>> >>>>> This would help this series to progress faster without getting distracted on PCI side. >>>> >>>> Agree the example of virtio-msg over PCI made the discussion divert a bit. >>>> >>>> Maybe focusing on use cases like: >>>> - FF-A >>>> - messaging unit + shared memory >>>> - hypervisor based events + shared memory >>> >>> Is "hypervisor based events + shared memory" only really intended for >>> Xen and other true type-1 hypervisors? Should KVM and other type-2 >>> hypervisors use a different approach, like virtio-vhost-user? >> >> Well there is definitely a use case in xen but any other type 1 or type 2 >> hypervisor could benefit from using virtio-msg and have asynchronous >> handling instead of a trap and emulate based implementation. > > Do irqfds and ioeventfds already provide this? What interface should > the device-implementing VM use to provide the device? Should it be > a new type of virtio device? Should this be standardized? There are 2 sides: - virtio-msg support in qemu (working on that) - your new virtio device/driver Once the first point is available you can create a new device type if you need or just reuse any existing ones (block, net, you name it) and build something on top which is probably the easiest solution. As said once the transport will be in linux and qemu, any existing virtio driver will work and that is the same for virtio devices existing already in qemu. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:51 ` Bertrand Marquis @ 2026-02-25 16:15 ` Demi Marie Obenour 0 siblings, 0 replies; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 16:15 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Michael S. Tsirkin, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 5270 bytes --] On 2/25/26 10:51, Bertrand Marquis wrote: > > >> On 25 Feb 2026, at 16:48, Demi Marie Obenour <demiobenour@gmail.com> wrote: >> >> On 2/25/26 10:40, Bertrand Marquis wrote: >>> >>> >>>> On 25 Feb 2026, at 16:36, Demi Marie Obenour <demiobenour@gmail.com> wrote: >>>> >>>> On 2/25/26 10:12, Bertrand Marquis wrote: >>>>> Hi Parav, >>>>> >>>>>> On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: >>>>>> >>>>>> Hi Bertrand, >>>>>> >>>>>>> From: Parav Pandit <parav@nvidia.com> >>>>>>> Sent: 25 February 2026 08:30 PM >>>>>> >>>>>> [..] >>>>>>>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>>>>>>>>>> >>>>>>>>>>> Is this making the concept a bit clearer ? >>>>>>>>>>> >>>>>>>>>> Yes. This makes a lot of sense now. >>>>>>>>>> >>>>>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>>>>>>> And its binding to the PCI transport. >>>>>>>>> >>>>>>>>> ok. how about an rfc of that idea on the list? >>>>>>>> >>>>>>>> >>>>>>>> I will let Edgar answer on this. >>>>>>>> >>>>>>> Sure. >>>>>>> Also please explain how is this different than SR-IOV devices? >>>>>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >>>>>>> And all of them are going to share same PCI BDF. >>>>>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >>>>>>> So what would this virtio-msg-transport device achieve? >>>>>>> >>>>>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >>>>>>> But underlying transport being PCI, its back to square one for the originally described motivation. >>>>>> >>>>>> If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). >>>>>> >>>>>> This would help this series to progress faster without getting distracted on PCI side. >>>>> >>>>> Agree the example of virtio-msg over PCI made the discussion divert a bit. >>>>> >>>>> Maybe focusing on use cases like: >>>>> - FF-A >>>>> - messaging unit + shared memory >>>>> - hypervisor based events + shared memory >>>> >>>> Is "hypervisor based events + shared memory" only really intended for >>>> Xen and other true type-1 hypervisors? Should KVM and other type-2 >>>> hypervisors use a different approach, like virtio-vhost-user? >>> >>> Well there is definitely a use case in xen but any other type 1 or type 2 >>> hypervisor could benefit from using virtio-msg and have asynchronous >>> handling instead of a trap and emulate based implementation. >> >> Do irqfds and ioeventfds already provide this? What interface should >> the device-implementing VM use to provide the device? Should it be >> a new type of virtio device? Should this be standardized? > > There are 2 sides: > - virtio-msg support in qemu (working on that) > - your new virtio device/driver > > Once the first point is available you can create a new device type if you need > or just reuse any existing ones (block, net, you name it) and build something > on top which is probably the easiest solution. > > As said once the transport will be in linux and qemu, any existing virtio driver > will work and that is the same for virtio devices existing already in qemu. I'm not actually using QEMU due to its large attack surface. I'm writing my own device implementation, either in Cloud Hypervisor or as a vhost-user device. It will implement a vhost-user device that can be used by a different VM. I'm going to call the guest implementing the device the "backend", and the guest using the device the "frontend". The threat model is that both assume the other might be malicious, but denial of service is not a serious concern. Latency and throughput _are_ serious concerns, as this will be used for real-time sound and video streams. The main four things I am trying to figure out are: 1. Most importantly, is this the best approach? Does it make sense to abstract over virtio-msg and vhost-user like this, or should they be completely separate, with separate userspace APIs? If they should be separate, I'll implement virtio-vhost-user instead. 2. Where should I translate from vhost-user semantics to virtio-msg semantics? I can do this in the device model, the guest kernel, or in guest userspace. All three have advantages and disadvantages. An implementation in the device model makes it easier to protect the frontend VM from the backend VM, but adds additional complexity in the backend VM's VMM. 3. How do I make my implementation interoperable with the common Linux virtio-msg device implementation API? If I develop a driver in parallel with the one you are working on, we will need to coordinate to avoid exposing diverging APIs to userspace. 4. Is the virtio device I am writing something that could be standardized? Is its driver one that would be upstreamable? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:12 ` Bertrand Marquis 2026-02-25 15:15 ` Michael S. Tsirkin 2026-02-25 15:36 ` Demi Marie Obenour @ 2026-02-26 5:40 ` Manivannan Sadhasivam 2026-02-26 7:05 ` Bertrand Marquis 2 siblings, 1 reply; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-26 5:40 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 03:12:11PM +0000, Bertrand Marquis wrote: > Hi Parav, > > > On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: > > > > Hi Bertrand, > > > >> From: Parav Pandit <parav@nvidia.com> > >> Sent: 25 February 2026 08:30 PM > > > > [..] > >>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. > >>>>>> > >>>>>> Is this making the concept a bit clearer ? > >>>>>> > >>>>> Yes. This makes a lot of sense now. > >>>>> > >>>>> This is a virtio-msg-transport device that needs its own device id in the table. > >>>>> And its binding to the PCI transport. > >>>> > >>>> ok. how about an rfc of that idea on the list? > >>> > >>> > >>> I will let Edgar answer on this. > >>> > >> Sure. > >> Also please explain how is this different than SR-IOV devices? > >> On a PCI device there are some child devices exposed using some virtio-msg bus transport. > >> And all of them are going to share same PCI BDF. > >> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > >> So what would this virtio-msg-transport device achieve? > >> > >> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > >> But underlying transport being PCI, its back to square one for the originally described motivation. > > > > If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). > > > > This would help this series to progress faster without getting distracted on PCI side. > > Agree the example of virtio-msg over PCI made the discussion divert a bit. > I brought it up *only* because I felt that the bus implementations need to be defined in the spec, because it would be hard for us to use virtio-msg without defining the actual bus implementation. Merely defining the 'bus requirements' is not enough. > Maybe focusing on use cases like: > - FF-A > - messaging unit + shared memory > - hypervisor based events + shared memory > IMO, for the initial version, you can atleast define one bus implementation. It could be one among the three you mentioned above. PCI bus implementation could very well come later once this gets in. - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-26 5:40 ` Manivannan Sadhasivam @ 2026-02-26 7:05 ` Bertrand Marquis 0 siblings, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-26 7:05 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Manivannan, > On 26 Feb 2026, at 06:40, Manivannan Sadhasivam <mani@kernel.org> wrote: > > On Wed, Feb 25, 2026 at 03:12:11PM +0000, Bertrand Marquis wrote: >> Hi Parav, >> >>> On 25 Feb 2026, at 16:07, Parav Pandit <parav@nvidia.com> wrote: >>> >>> Hi Bertrand, >>> >>>> From: Parav Pandit <parav@nvidia.com> >>>> Sent: 25 February 2026 08:30 PM >>> >>> [..] >>>>>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>>>>>>> >>>>>>>> Is this making the concept a bit clearer ? >>>>>>>> >>>>>>> Yes. This makes a lot of sense now. >>>>>>> >>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>>>> And its binding to the PCI transport. >>>>>> >>>>>> ok. how about an rfc of that idea on the list? >>>>> >>>>> >>>>> I will let Edgar answer on this. >>>>> >>>> Sure. >>>> Also please explain how is this different than SR-IOV devices? >>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >>>> And all of them are going to share same PCI BDF. >>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >>>> So what would this virtio-msg-transport device achieve? >>>> >>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >>>> But underlying transport being PCI, its back to square one for the originally described motivation. >>> >>> If the objective of the virtio-msg layer is to really support FF-A or similar other non-PCI transports, lets please focus the discussion on such use case and binding of that specific non-PCI bus. (like the link you pointed on arm site). >>> >>> This would help this series to progress faster without getting distracted on PCI side. >> >> Agree the example of virtio-msg over PCI made the discussion divert a bit. >> > > I brought it up *only* because I felt that the bus implementations need to be > defined in the spec, because it would be hard for us to use virtio-msg without > defining the actual bus implementation. Merely defining the 'bus requirements' > is not enough. We could have A bus implementation specified in the spec but not all bus implementations. A big part of the design is to allow implementation on several transport mechanisms (at the end you only need to have a way to share or access remote memory and transport messages). As of now the identified ones are: - FF-A - PCI for system to system (Edgar) - Xen (grantables and events) - Arm MHU based (cpu to coprocessor) But we know that there are others working on other hypervisor specific solutions. > >> Maybe focusing on use cases like: >> - FF-A >> - messaging unit + shared memory >> - hypervisor based events + shared memory >> > > IMO, for the initial version, you can atleast define one bus implementation. It > could be one among the three you mentioned above. PCI bus implementation could > very well come later once this gets in. As said implementation over FF-A has a public specification. It is not in the virtio specification but i am happy to give a link to it in the spec. Cheers Bertrand > > - Mani > > -- > மணிவண்ணன் சதாசிவம் IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:00 ` Parav Pandit 2026-02-25 15:07 ` Parav Pandit @ 2026-02-25 15:11 ` Manivannan Sadhasivam 2026-02-25 15:15 ` Parav Pandit 2026-02-25 15:23 ` Demi Marie Obenour 2 siblings, 1 reply; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-25 15:11 UTC (permalink / raw) To: Parav Pandit Cc: Bertrand Marquis, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: [...] > > >> Yes. This makes a lot of sense now. > > >> > > >> This is a virtio-msg-transport device that needs its own device id in the table. > > >> And its binding to the PCI transport. > > > > > > ok. how about an rfc of that idea on the list? > > > > > > I will let Edgar answer on this. > > > Sure. > Also please explain how is this different than SR-IOV devices? > On a PCI device there are some child devices exposed using some virtio-msg bus transport. > And all of them are going to share same PCI BDF. > These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > So what would this virtio-msg-transport device achieve? > IDK why you are comparing this transport with SR-IOV, which altogether serves a different purpose. > If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > But underlying transport being PCI, its back to square one for the originally described motivation. > Even with PCI as the physical bus, we don't need any trap and emulate assumption with this transport, because the communication is message based, not writing and reading registers directly in the BAR memory. - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:11 ` Manivannan Sadhasivam @ 2026-02-25 15:15 ` Parav Pandit 2026-02-26 5:36 ` Manivannan Sadhasivam 0 siblings, 1 reply; 105+ messages in thread From: Parav Pandit @ 2026-02-25 15:15 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Bertrand Marquis, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Manivannan Sadhasivam <mani@kernel.org> > Sent: 25 February 2026 08:41 PM > > On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: > > [...] > > > > >> Yes. This makes a lot of sense now. > > > >> > > > >> This is a virtio-msg-transport device that needs its own device id in the table. > > > >> And its binding to the PCI transport. > > > > > > > > ok. how about an rfc of that idea on the list? > > > > > > > > > I will let Edgar answer on this. > > > > > Sure. > > Also please explain how is this different than SR-IOV devices? > > On a PCI device there are some child devices exposed using some virtio-msg bus transport. > > And all of them are going to share same PCI BDF. > > These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > > So what would this virtio-msg-transport device achieve? > > > > IDK why you are comparing this transport with SR-IOV, which altogether serves a > different purpose. > In the example showed there are multiple devices behind one PCI device. Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices. > > If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > > But underlying transport being PCI, its back to square one for the originally described motivation. > > > > Even with PCI as the physical bus, we don't need any trap and emulate assumption > with this transport, because the communication is message based, not writing and > reading registers directly in the BAR memory. > That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport. So I do not know why t+e was mentioned. I don’t have the link to that email at the moment. > - Mani > > -- > மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:15 ` Parav Pandit @ 2026-02-26 5:36 ` Manivannan Sadhasivam 2026-02-26 5:59 ` Parav Pandit ` (2 more replies) 0 siblings, 3 replies; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-26 5:36 UTC (permalink / raw) To: Parav Pandit Cc: Bertrand Marquis, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 03:15:39PM +0000, Parav Pandit wrote: > > > From: Manivannan Sadhasivam <mani@kernel.org> > > Sent: 25 February 2026 08:41 PM > > > > On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: > > > > [...] > > > > > > >> Yes. This makes a lot of sense now. > > > > >> > > > > >> This is a virtio-msg-transport device that needs its own device id in the table. > > > > >> And its binding to the PCI transport. > > > > > > > > > > ok. how about an rfc of that idea on the list? > > > > > > > > > > > > I will let Edgar answer on this. > > > > > > > Sure. > > > Also please explain how is this different than SR-IOV devices? > > > On a PCI device there are some child devices exposed using some virtio-msg bus transport. > > > And all of them are going to share same PCI BDF. > > > These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > > > So what would this virtio-msg-transport device achieve? > > > > > > > IDK why you are comparing this transport with SR-IOV, which altogether serves a > > different purpose. > > > In the example showed there are multiple devices behind one PCI device. > Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices. > That could be one usecase, but not the actual design. > > > If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > > > But underlying transport being PCI, its back to square one for the originally described motivation. > > > > > > > Even with PCI as the physical bus, we don't need any trap and emulate assumption > > with this transport, because the communication is message based, not writing and > > reading registers directly in the BAR memory. > > > That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport. > So I do not know why t+e was mentioned. > I don’t have the link to that email at the moment. > I mentioned that because to run the existing Virtio PCI one would need an emulated PCI device. I think it is also possible to expose custom designed HW as a Virtio PCI device, but that's not the majority. One primary example is how the virtqueue configuration happens. The driver selects the queue by writing to the 'queue_select' and immediately it expects the device to select that virtqueue and give response in other fields such as 'queue_size'. This only happens in an emulated PCI bus. But in a real PCI bus, the device on the other end cannot synchronize itself with the host. And this is what being fixed by the message based communication. We tried using the existing Virtio PCI transport in a design where two SoCs are connected using PCIe bus, both running Linux. One will act as Virtio Frontend and another as backend. Then we ran into all sort issues due to the above mentioned assumption/expectation by the transport. - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-26 5:36 ` Manivannan Sadhasivam @ 2026-02-26 5:59 ` Parav Pandit 2026-02-26 6:19 ` Manivannan Sadhasivam 2026-02-26 7:01 ` Bertrand Marquis 2026-02-26 19:20 ` Demi Marie Obenour 2 siblings, 1 reply; 105+ messages in thread From: Parav Pandit @ 2026-02-26 5:59 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Bertrand Marquis, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Manivannan Sadhasivam <mani@kernel.org> > Sent: 26 February 2026 11:06 AM > > On Wed, Feb 25, 2026 at 03:15:39PM +0000, Parav Pandit wrote: > > > > > From: Manivannan Sadhasivam <mani@kernel.org> > > > Sent: 25 February 2026 08:41 PM > > > > > > On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: > > > > > > [...] > > > > > > > > >> Yes. This makes a lot of sense now. > > > > > >> > > > > > >> This is a virtio-msg-transport device that needs its own device id in the table. > > > > > >> And its binding to the PCI transport. > > > > > > > > > > > > ok. how about an rfc of that idea on the list? > > > > > > > > > > > > > > > I will let Edgar answer on this. > > > > > > > > > Sure. > > > > Also please explain how is this different than SR-IOV devices? > > > > On a PCI device there are some child devices exposed using some virtio-msg bus transport. > > > > And all of them are going to share same PCI BDF. > > > > These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > > > > So what would this virtio-msg-transport device achieve? > > > > > > > > > > IDK why you are comparing this transport with SR-IOV, which altogether serves a > > > different purpose. > > > > > In the example showed there are multiple devices behind one PCI device. > > Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices. > > > > That could be one usecase, but not the actual design. > I didn’t understand the design part. Why that cannot be the design if the intent is to have multiple devices? > > > > If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > > > > But underlying transport being PCI, its back to square one for the originally described motivation. > > > > > > > > > > Even with PCI as the physical bus, we don't need any trap and emulate assumption > > > with this transport, because the communication is message based, not writing and > > > reading registers directly in the BAR memory. > > > > > That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport. > > So I do not know why t+e was mentioned. > > I don’t have the link to that email at the moment. > > > > I mentioned that because to run the existing Virtio PCI one would need an > emulated PCI device. I think it is also possible to expose custom designed HW as > a Virtio PCI device, but that's not the majority. > > One primary example is how the virtqueue configuration happens. The driver > selects the queue by writing to the 'queue_select' and immediately it expects > the device to select that virtqueue and give response in other fields such as > 'queue_size'. This only happens in an emulated PCI bus. But in a real PCI bus, > the device on the other end cannot synchronize itself with the host. And this is > what being fixed by the message based communication. > Yes, I am familiar with this inefficiency. Well for such fundamental case, all resource creation can be directly move to an admin command interface being basic facility for long enough period. > We tried using the existing Virtio PCI transport in a design where two SoCs are > connected using PCIe bus, both running Linux. One will act as Virtio Frontend > and another as backend. Then we ran into all sort issues due to the above > mentioned assumption/expectation by the transport. > When you say front end meaning the driver, and backend being the device? If yes, then then it is very common for us as well to have modern interface for resource creation such as virtqueue. I am still yet to catch up on Bertrand, Edgar's, Demi's and other emails from yesterday. > - Mani > > -- > மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-26 5:59 ` Parav Pandit @ 2026-02-26 6:19 ` Manivannan Sadhasivam 0 siblings, 0 replies; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-26 6:19 UTC (permalink / raw) To: Parav Pandit Cc: Bertrand Marquis, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Thu, Feb 26, 2026 at 05:59:52AM +0000, Parav Pandit wrote: > > > > From: Manivannan Sadhasivam <mani@kernel.org> > > Sent: 26 February 2026 11:06 AM > > > > On Wed, Feb 25, 2026 at 03:15:39PM +0000, Parav Pandit wrote: > > > > > > > From: Manivannan Sadhasivam <mani@kernel.org> > > > > Sent: 25 February 2026 08:41 PM > > > > > > > > On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: > > > > > > > > [...] > > > > > > > > > > >> Yes. This makes a lot of sense now. > > > > > > >> > > > > > > >> This is a virtio-msg-transport device that needs its own device id in the table. > > > > > > >> And its binding to the PCI transport. > > > > > > > > > > > > > > ok. how about an rfc of that idea on the list? > > > > > > > > > > > > > > > > > > I will let Edgar answer on this. > > > > > > > > > > > Sure. > > > > > Also please explain how is this different than SR-IOV devices? > > > > > On a PCI device there are some child devices exposed using some virtio-msg bus transport. > > > > > And all of them are going to share same PCI BDF. > > > > > These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > > > > > So what would this virtio-msg-transport device achieve? > > > > > > > > > > > > > IDK why you are comparing this transport with SR-IOV, which altogether serves a > > > > different purpose. > > > > > > > In the example showed there are multiple devices behind one PCI device. > > > Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices. > > > > > > > That could be one usecase, but not the actual design. > > > I didn’t understand the design part. > Why that cannot be the design if the intent is to have multiple devices? > PCI is going to be *one* of the bus implementations for virtio-msg. Other busses could also expose multiple devices if the underlying protocol supports. So I didn't want to tie it with SR-IOV alone. > > > > > If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > > > > > But underlying transport being PCI, its back to square one for the originally described motivation. > > > > > > > > > > > > > Even with PCI as the physical bus, we don't need any trap and emulate assumption > > > > with this transport, because the communication is message based, not writing and > > > > reading registers directly in the BAR memory. > > > > > > > That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport. > > > So I do not know why t+e was mentioned. > > > I don’t have the link to that email at the moment. > > > > > > > I mentioned that because to run the existing Virtio PCI one would need an > > emulated PCI device. I think it is also possible to expose custom designed HW as > > a Virtio PCI device, but that's not the majority. > > > > One primary example is how the virtqueue configuration happens. The driver > > selects the queue by writing to the 'queue_select' and immediately it expects > > the device to select that virtqueue and give response in other fields such as > > 'queue_size'. This only happens in an emulated PCI bus. But in a real PCI bus, > > the device on the other end cannot synchronize itself with the host. And this is > > what being fixed by the message based communication. > > > Yes, I am familiar with this inefficiency. > Well for such fundamental case, all resource creation can be directly move to an admin command interface being basic facility for long enough period. > I'm not aware of this 'admin command interface'. Could you elaborate? > > We tried using the existing Virtio PCI transport in a design where two SoCs are > > connected using PCIe bus, both running Linux. One will act as Virtio Frontend > > and another as backend. Then we ran into all sort issues due to the above > > mentioned assumption/expectation by the transport. > > > When you say front end meaning the driver, and backend being the device? Yes! > If yes, then then it is very common for us as well to have modern interface for resource creation such as virtqueue. > When I talked to Michael a couple of years back at Vienna LPC about these issues, he suggested going for a separate transport than fixing the existing PCI transport (which might not be possible while preserving backwards compatibility). That's when I encountered this new virtio-msg transport and it suited my needs. - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-26 5:36 ` Manivannan Sadhasivam 2026-02-26 5:59 ` Parav Pandit @ 2026-02-26 7:01 ` Bertrand Marquis 2026-02-26 7:28 ` Manivannan Sadhasivam 2026-02-26 19:20 ` Demi Marie Obenour 2 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-26 7:01 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Manivannan, > On 26 Feb 2026, at 06:36, Manivannan Sadhasivam <mani@kernel.org> wrote: > > On Wed, Feb 25, 2026 at 03:15:39PM +0000, Parav Pandit wrote: >> >>> From: Manivannan Sadhasivam <mani@kernel.org> >>> Sent: 25 February 2026 08:41 PM >>> >>> On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: >>> >>> [...] >>> >>>>>>> Yes. This makes a lot of sense now. >>>>>>> >>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>>>> And its binding to the PCI transport. >>>>>> >>>>>> ok. how about an rfc of that idea on the list? >>>>> >>>>> >>>>> I will let Edgar answer on this. >>>>> >>>> Sure. >>>> Also please explain how is this different than SR-IOV devices? >>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >>>> And all of them are going to share same PCI BDF. >>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >>>> So what would this virtio-msg-transport device achieve? >>>> >>> >>> IDK why you are comparing this transport with SR-IOV, which altogether serves a >>> different purpose. >>> >> In the example showed there are multiple devices behind one PCI device. >> Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices. >> > > That could be one usecase, but not the actual design. > >>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >>>> But underlying transport being PCI, its back to square one for the originally described motivation. >>>> >>> >>> Even with PCI as the physical bus, we don't need any trap and emulate assumption >>> with this transport, because the communication is message based, not writing and >>> reading registers directly in the BAR memory. >>> >> That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport. >> So I do not know why t+e was mentioned. >> I don’t have the link to that email at the moment. >> > > I mentioned that because to run the existing Virtio PCI one would need an > emulated PCI device. I think it is also possible to expose custom designed HW as > a Virtio PCI device, but that's not the majority. > > One primary example is how the virtqueue configuration happens. The driver > selects the queue by writing to the 'queue_select' and immediately it expects > the device to select that virtqueue and give response in other fields such as > 'queue_size'. This only happens in an emulated PCI bus. But in a real PCI bus, > the device on the other end cannot synchronize itself with the host. And this is > what being fixed by the message based communication. > > We tried using the existing Virtio PCI transport in a design where two SoCs are > connected using PCIe bus, both running Linux. One will act as Virtio Frontend > and another as backend. Then we ran into all sort issues due to the above > mentioned assumption/expectation by the transport. > Definitely check Edgar's work using virtio-msg as it seems that it is exactly what you are looking for. Cheers Bertrand > - Mani > > -- > மணிவண்ணன் சதாசிவம் IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-26 7:01 ` Bertrand Marquis @ 2026-02-26 7:28 ` Manivannan Sadhasivam 0 siblings, 0 replies; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-26 7:28 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Thu, Feb 26, 2026 at 07:01:25AM +0000, Bertrand Marquis wrote: > Hi Manivannan, > > > On 26 Feb 2026, at 06:36, Manivannan Sadhasivam <mani@kernel.org> wrote: > > > > On Wed, Feb 25, 2026 at 03:15:39PM +0000, Parav Pandit wrote: > >> > >>> From: Manivannan Sadhasivam <mani@kernel.org> > >>> Sent: 25 February 2026 08:41 PM > >>> > >>> On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: > >>> > >>> [...] > >>> > >>>>>>> Yes. This makes a lot of sense now. > >>>>>>> > >>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. > >>>>>>> And its binding to the PCI transport. > >>>>>> > >>>>>> ok. how about an rfc of that idea on the list? > >>>>> > >>>>> > >>>>> I will let Edgar answer on this. > >>>>> > >>>> Sure. > >>>> Also please explain how is this different than SR-IOV devices? > >>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. > >>>> And all of them are going to share same PCI BDF. > >>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > >>>> So what would this virtio-msg-transport device achieve? > >>>> > >>> > >>> IDK why you are comparing this transport with SR-IOV, which altogether serves a > >>> different purpose. > >>> > >> In the example showed there are multiple devices behind one PCI device. > >> Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices. > >> > > > > That could be one usecase, but not the actual design. > > > >>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > >>>> But underlying transport being PCI, its back to square one for the originally described motivation. > >>>> > >>> > >>> Even with PCI as the physical bus, we don't need any trap and emulate assumption > >>> with this transport, because the communication is message based, not writing and > >>> reading registers directly in the BAR memory. > >>> > >> That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport. > >> So I do not know why t+e was mentioned. > >> I don’t have the link to that email at the moment. > >> > > > > I mentioned that because to run the existing Virtio PCI one would need an > > emulated PCI device. I think it is also possible to expose custom designed HW as > > a Virtio PCI device, but that's not the majority. > > > > One primary example is how the virtqueue configuration happens. The driver > > selects the queue by writing to the 'queue_select' and immediately it expects > > the device to select that virtqueue and give response in other fields such as > > 'queue_size'. This only happens in an emulated PCI bus. But in a real PCI bus, > > the device on the other end cannot synchronize itself with the host. And this is > > what being fixed by the message based communication. > > > > We tried using the existing Virtio PCI transport in a design where two SoCs are > > connected using PCIe bus, both running Linux. One will act as Virtio Frontend > > and another as backend. Then we ran into all sort issues due to the above > > mentioned assumption/expectation by the transport. > > > > Definitely check Edgar's work using virtio-msg as it seems that it is exactly what > you are looking for. > I already have a prototype based on the intial virtio-msg transport kernel patches by Viresh: https://github.com/Mani-Sadhasivam/linux/tree/pci-epf-virtio-msg I guess some of it, especially the queue implementation was inspired from Edgar's work I believe. - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-26 5:36 ` Manivannan Sadhasivam 2026-02-26 5:59 ` Parav Pandit 2026-02-26 7:01 ` Bertrand Marquis @ 2026-02-26 19:20 ` Demi Marie Obenour 2026-02-26 22:08 ` Edgar E. Iglesias 2 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-26 19:20 UTC (permalink / raw) To: Manivannan Sadhasivam, Parav Pandit Cc: Bertrand Marquis, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 3697 bytes --] On 2/26/26 00:36, Manivannan Sadhasivam wrote: > On Wed, Feb 25, 2026 at 03:15:39PM +0000, Parav Pandit wrote: >> >>> From: Manivannan Sadhasivam <mani@kernel.org> >>> Sent: 25 February 2026 08:41 PM >>> >>> On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: >>> >>> [...] >>> >>>>>>> Yes. This makes a lot of sense now. >>>>>>> >>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>>>>> And its binding to the PCI transport. >>>>>> >>>>>> ok. how about an rfc of that idea on the list? >>>>> >>>>> >>>>> I will let Edgar answer on this. >>>>> >>>> Sure. >>>> Also please explain how is this different than SR-IOV devices? >>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. >>>> And all of them are going to share same PCI BDF. >>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. >>>> So what would this virtio-msg-transport device achieve? >>>> >>> >>> IDK why you are comparing this transport with SR-IOV, which altogether serves a >>> different purpose. >>> >> In the example showed there are multiple devices behind one PCI device. >> Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices. >> > > That could be one usecase, but not the actual design. > >>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. >>>> But underlying transport being PCI, its back to square one for the originally described motivation. >>>> >>> >>> Even with PCI as the physical bus, we don't need any trap and emulate assumption >>> with this transport, because the communication is message based, not writing and >>> reading registers directly in the BAR memory. >>> >> That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport. >> So I do not know why t+e was mentioned. >> I don’t have the link to that email at the moment. >> > > I mentioned that because to run the existing Virtio PCI one would need an > emulated PCI device. I think it is also possible to expose custom designed HW as > a Virtio PCI device, but that's not the majority. I suspect that HW virtio-PCI implementations do exist. That's the purpose of ACCESS_PLATFORM if I recall correctly. > One primary example is how the virtqueue configuration happens. The driver > selects the queue by writing to the 'queue_select' and immediately it expects > the device to select that virtqueue and give response in other fields such as > 'queue_size'. This only happens in an emulated PCI bus. But in a real PCI bus, > the device on the other end cannot synchronize itself with the host. And this is > what being fixed by the message based communication. > > We tried using the existing Virtio PCI transport in a design where two SoCs are > connected using PCIe bus, both running Linux. One will act as Virtio Frontend > and another as backend. Then we ran into all sort issues due to the above > mentioned assumption/expectation by the transport. My understanding is that PCIe guarantees TLP ordering and that reads are implemented as messages that require a response. If so, the write and read TLPs will appear in the appropriate order and could be handled correctly. However, this would require the PCIe endpoint support software-defined MMIO reads. Is this something that is supported, or is it not an option due to latency requirements, at least without wasting a CPU core on polling? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-26 19:20 ` Demi Marie Obenour @ 2026-02-26 22:08 ` Edgar E. Iglesias 0 siblings, 0 replies; 105+ messages in thread From: Edgar E. Iglesias @ 2026-02-26 22:08 UTC (permalink / raw) To: Demi Marie Obenour Cc: Manivannan Sadhasivam, Parav Pandit, Bertrand Marquis, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Thu, Feb 26, 2026 at 02:20:01PM -0500, Demi Marie Obenour wrote: > On 2/26/26 00:36, Manivannan Sadhasivam wrote: > > On Wed, Feb 25, 2026 at 03:15:39PM +0000, Parav Pandit wrote: > >> > >>> From: Manivannan Sadhasivam <mani@kernel.org> > >>> Sent: 25 February 2026 08:41 PM > >>> > >>> On Wed, Feb 25, 2026 at 03:00:19PM +0000, Parav Pandit wrote: > >>> > >>> [...] > >>> > >>>>>>> Yes. This makes a lot of sense now. > >>>>>>> > >>>>>>> This is a virtio-msg-transport device that needs its own device id in the table. > >>>>>>> And its binding to the PCI transport. > >>>>>> > >>>>>> ok. how about an rfc of that idea on the list? > >>>>> > >>>>> > >>>>> I will let Edgar answer on this. > >>>>> > >>>> Sure. > >>>> Also please explain how is this different than SR-IOV devices? > >>>> On a PCI device there are some child devices exposed using some virtio-msg bus transport. > >>>> And all of them are going to share same PCI BDF. > >>>> These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > >>>> So what would this virtio-msg-transport device achieve? > >>>> > >>> > >>> IDK why you are comparing this transport with SR-IOV, which altogether serves a > >>> different purpose. > >>> > >> In the example showed there are multiple devices behind one PCI device. > >> Why is it not similar to SR-IOV that does exactly that, on one PCI link, exposes multiple devices. > >> > > > > That could be one usecase, but not the actual design. > > > >>>> If there was no PCI bus when implementing the virtio-msg transport, it would make sense to avoid trap-emulate story. > >>>> But underlying transport being PCI, its back to square one for the originally described motivation. > >>>> > >>> > >>> Even with PCI as the physical bus, we don't need any trap and emulate assumption > >>> with this transport, because the communication is message based, not writing and > >>> reading registers directly in the BAR memory. > >>> > >> That is true. Somewhere in this thread, there was mention of trap+emulation as reason to create some msg based transport. > >> So I do not know why t+e was mentioned. > >> I don’t have the link to that email at the moment. > >> > > > > I mentioned that because to run the existing Virtio PCI one would need an > > emulated PCI device. I think it is also possible to expose custom designed HW as > > a Virtio PCI device, but that's not the majority. > > I suspect that HW virtio-PCI implementations do exist. > That's the purpose of ACCESS_PLATFORM if I recall correctly. > > > One primary example is how the virtqueue configuration happens. The driver > > selects the queue by writing to the 'queue_select' and immediately it expects > > the device to select that virtqueue and give response in other fields such as > > 'queue_size'. This only happens in an emulated PCI bus. But in a real PCI bus, > > the device on the other end cannot synchronize itself with the host. And this is > > what being fixed by the message based communication. > > > > We tried using the existing Virtio PCI transport in a design where two SoCs are > > connected using PCIe bus, both running Linux. One will act as Virtio Frontend > > and another as backend. Then we ran into all sort issues due to the above > > mentioned assumption/expectation by the transport. > My understanding is that PCIe guarantees TLP ordering and that reads > are implemented as messages that require a response. If so, the > write and read TLPs will appear in the appropriate order and could > be handled correctly. > > However, this would require the PCIe endpoint support software-defined > MMIO reads. Is this something that is supported, or is it not an > option due to latency requirements, at least without wasting a CPU > core on polling? Right, I think it would be hard to get something like that to perform well. We would also need the HW logic to hand-over memory accesses to SW and back.. There are several benefits with a message-based transport compared to trap+emulate, for example: 1. Trap and emulation is not really available in AMP scenarios. It could be worked around by trapping locally and converting each memory access into some kind of message (like Xen IOREQs) but this adds complexity and latency. With an end-to-end message based transport we can natively speak across system boundaries. 2. Real-time. Imagine a guest with a single core running some kind of real-time sensitive workload. Whenever guests access a register, traps and emulates, the guest "stalls" for the time it takes to respond to the trapped access. This can lead to high response times which can be hard to quantify depending on the device implementation. With a message based transport, guests crafts messages, send them over and can do other work (e.g take interrupts) while waiting for responses. Best regards, Edgar ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:00 ` Parav Pandit 2026-02-25 15:07 ` Parav Pandit 2026-02-25 15:11 ` Manivannan Sadhasivam @ 2026-02-25 15:23 ` Demi Marie Obenour 2 siblings, 0 replies; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 15:23 UTC (permalink / raw) To: Parav Pandit, Bertrand Marquis, Michael S. Tsirkin Cc: Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 11546 bytes --] On 2/25/26 10:00, Parav Pandit wrote: > >> From: Bertrand Marquis <Bertrand.Marquis@arm.com> >> Sent: 25 February 2026 08:24 PM >> >> Hi, >> >>> On 25 Feb 2026, at 15:49, Michael S. Tsirkin <mst@redhat.com> wrote: >>> >>> On Wed, Feb 25, 2026 at 02:45:35PM +0000, Parav Pandit wrote: >>>> >>>>> From: Bertrand Marquis <Bertrand.Marquis@arm.com> >>>>> Sent: 25 February 2026 04:06 PM >>>>> >>>>> Hi Parav, >>>>> >>>>>> On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: >>>>>> >>>>>>> >>>>>>> From: Manivannan Sadhasivam <mani@kernel.org> >>>>>>> Sent: 25 February 2026 03:37 PM >>>>>>> >>>>>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: >>>>>>>> Hi Manivannan, >>>>>>>> >>>>>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: >>>>>>>>> >>>>>>>>> Hi Bertrand, >>>>>>>>> >>>>>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>>>>>>>> Hi Parav, >>>>>>>>>> >>>>>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com> >>>>>>>>>>>> Sent: 20 February 2026 05:25 AM >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: >>>>>>>>>>>>> Hi Bill, >>>>>>>>>>>>> >>>>>>>>>>>>>> From: Bill Mills <bill.mills@linaro.org> >>>>>>>>>>>>>> Sent: 26 January 2026 10:02 PM >>>>>>>>>>>>>> >>>>>>>>>>>>>> This series adds the virtio-msg transport layer. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in >>>>>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one >>>>>>>>>>>>>> more transport that performs its transport layer operations by sending and >>>>>>>>>>>>>> receiving messages. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>>>>>>>>>>> * common / device level >>>>>>>>>>>>>> * bus level >>>>>>>>>>>>>> >>>>>>>>>>>>>> The common / device level defines the messages exchanged between the driver >>>>>>>>>>>>>> and a device. This common part should lead to a common driver holding most >>>>>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>>>>>>>>>>> main focus of this version of the patch series. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines >>>>>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>>>>>>>>>>> receive operations. A number of bus implementations are envisioned, >>>>>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations >>>>>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card >>>>>>>>>>>>>> and its driver. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations >>>>>>>>>>>>>> doing the same thing in different ways for no good reason. However the >>>>>>>>>>>>>> different environments will require different things. Instead of trying to >>>>>>>>>>>>>> anticipate all needs and provide something very abstract, we think >>>>>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time, >>>>>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to >>>>>>>>>>>>>> standardize a bus level message for that. >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I would review more, had first round of sparse review. >>>>>>>>>>>>> Please find few comments/questions below. >>>>>>>>>>>> >>>>>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and >>>>>>>>>>>> then add performance features on top as appropriate. >>>>>>>>>>> Sounds good. Simple but complete is needed. >>>>>>>>>> >>>>>>>>>> Agree. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> So one way to address some of these comments is to show how >>>>>>>>>>>> they can be addressed with a feature bit down the road. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>>>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>>>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>>>>>>>>>>>> >>>>>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>>>>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>>>>>>>>>>>> Needs 32-bits. >>>>>>>>>>>>> >>>>>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>>>>>>>>>>>> But more below. >>>>>>>>>>>>> >>>>>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses >> to >>>>>>> operate >>>>>>>>>>>> the virtqueues. >>>>>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >>>>>>>>>>>> >>>>>>>>>>>> surely this can be an optional transport feature bit. >>>>>>>>>>>> >>>>>>>>>>> How is this optional? >>>>>>>>>> >>>>>>>>>> As said in a previous mail, we have messages already for that. >>>>>>>>>> Please confirm if that answer your question. >>>>>>>>>> >>>>>>>>>>> How can one implement a transport without defining the basic data transfer semantics? >>>>>>>>>> >>>>>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>>>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>>>>>>>> >>>>>>>>> >>>>>>>>> I'd love to have this transport over PCI because it addresses the shortcomings >>>>>>>>> of the existing PCI transport which just assumes that every config space access\ >>>>>>>>> is trap and emulate. >>>>>>>> >>>>>>>> Agree and AMD did exactly that in their demonstrator. >>>>>>>> I will give you answers here as i know them but Edgar will probably give you more >>>>>>>> details (and probably fix my mistakes). >>>>>>>> >>>>>>>>> >>>>>>>>> But that being said, I somewhat agree with Parav that we should define the bus >>>>>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For >>>>>>>>> instance, if we try to use this transport over PCI, we've got questions like: >>>>>>>>> >>>>>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with >>>>>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? >>>>>>>> >>>>>>>> One bus is appearing as one pci device with its own Vendor ID, >>>>>>>> >>>>>>> >>>>>>> What should be the 'own Vendor ID' here? >>>>>>> >>>>>>> The existing virtio-pci driver binds to all devices with the Vendor ID of >>>>>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own >>>>>>> VID for exposing the Virtio devices? That would mean, the drivers on the host >>>>>>> need update as well, which will not scale. >>>>>>> >>>>>>> It would be good if the existing virtio-pci devices can use this new transport >>>>>>> with only device side modifications. >>>>>>> >>>>>>>>> >>>>>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config >>>>>>>>> space or through some other means? >>>>>>>> >>>>>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI >>>>>>>> device (ending up being memory shared between both sides) >>>>>>>> >>>>>>> >>>>>>> What should be the BAR number and size? >>>>>>> >>>>>>>>> >>>>>>>>> 3. How the notification be delivered from the device to the host? Through >>>>>>>>> INT-X/MSI/MSI-X or even polling? >>>>>>>> >>>>>>>> Notifications are delivered through MSI. >>>>>>>> >>>>>>> >>>>>>> So no INT-X or MSI-X? Why so? >>>>>>> >>>>>>> Anyhow, my objective is not to get answers for my above questions here in this >>>>>>> thread, but to state the reality that it would be hard for us to make use of >>>>>>> this new transport without defining the bus implementation. >>>>>>> >>>>>> +1 to most of the points that Manivannan explained. >>>>>> >>>>>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet >>>>> another interface for _Everything_ that already exists. >>>>>> and device is still have to implement all the existing things because the device does not know which driver will operate. >>>>>> >>>>>> And that too some register based inefficient interface. >>>>>> Just to reset the device one needs to fully setup the new message interface but device still have to be working. >>>>>> That defeats the whole purpose of reset_1 and reset_2 in the device. >>>>>> >>>>>> This does not bring anything better for the PCI devices at all. >>>>>> >>>>>> A transport binding should be defined for the bus binding. >>>>>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. >>>>>> >>>>>> If we are creating something brand-new, for PCI the only thing needed is: >>>>>> 1. Reset the device >>>>>> 2. Create an admin virtqueue >>>>>> 3. Transport everything needed through this virtqueue including features, configs, control. >>>>>> >>>>>> And this will work for any other bus or msg based too given only contract needed is to creating the aq. >>>>> >>>>> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. >>>>> >>>>> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". >>>>> >>>>> When the virtio-msg bus over PCI it will communicate through this device with an external >>>>> system connected through the PCI bus. >>>>> The driver will enumerate virtio devices available behind this bus and register them so that >>>>> the corresponding virtio drivers are probed for them. >>>>> All virtio-msg messages required to communicate with those devices will be transferred through >>>>> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the >>>>> virtqueues with all the devices on the bus. >>>>> >>>>> So the PCI device is not one virtio device but one bus behind which there can be many devices. >>>>> >>>>> Is this making the concept a bit clearer ? >>>>> >>>> Yes. This makes a lot of sense now. >>>> >>>> This is a virtio-msg-transport device that needs its own device id in the table. >>>> And its binding to the PCI transport. >>> >>> ok. how about an rfc of that idea on the list? >> >> >> I will let Edgar answer on this. >> > Sure. > Also please explain how is this different than SR-IOV devices? > On a PCI device there are some child devices exposed using some virtio-msg bus transport. > And all of them are going to share same PCI BDF. > These individual devices cannot be mapped to different VMs (secure or regular) in a performant way either. > So what would this virtio-msg-transport device achieve? Since the message transport is more peer-to-peer, would it make sense to use this to allow VMs to implement virtio devices for use by other VMs? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 14:53 ` Bertrand Marquis 2026-02-25 15:00 ` Parav Pandit @ 2026-02-25 16:42 ` Edgar E. Iglesias 1 sibling, 0 replies; 105+ messages in thread From: Edgar E. Iglesias @ 2026-02-25 16:42 UTC (permalink / raw) To: Bertrand Marquis Cc: Michael S. Tsirkin, Parav Pandit, Manivannan Sadhasivam, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 02:53:58PM +0000, Bertrand Marquis wrote: > Hi, > > > On 25 Feb 2026, at 15:49, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Feb 25, 2026 at 02:45:35PM +0000, Parav Pandit wrote: > >> > >>> From: Bertrand Marquis <Bertrand.Marquis@arm.com> > >>> Sent: 25 February 2026 04:06 PM > >>> > >>> Hi Parav, > >>> > >>>> On 25 Feb 2026, at 11:24, Parav Pandit <parav@nvidia.com> wrote: > >>>> > >>>>> > >>>>> From: Manivannan Sadhasivam <mani@kernel.org> > >>>>> Sent: 25 February 2026 03:37 PM > >>>>> > >>>>> On Wed, Feb 25, 2026 at 08:03:48AM +0000, Bertrand Marquis wrote: > >>>>>> Hi Manivannan, > >>>>>> > >>>>>>> On 25 Feb 2026, at 08:45, Manivannan Sadhasivam <mani@kernel.org> wrote: > >>>>>>> > >>>>>>> Hi Bertrand, > >>>>>>> > >>>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > >>>>>>>> Hi Parav, > >>>>>>>> > >>>>>>>>> On 20 Feb 2026, at 07:13, Parav Pandit <parav@nvidia.com> wrote: > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com> > >>>>>>>>>> Sent: 20 February 2026 05:25 AM > >>>>>>>>>> > >>>>>>>>>> On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: > >>>>>>>>>>> Hi Bill, > >>>>>>>>>>> > >>>>>>>>>>>> From: Bill Mills <bill.mills@linaro.org> > >>>>>>>>>>>> Sent: 26 January 2026 10:02 PM > >>>>>>>>>>>> > >>>>>>>>>>>> This series adds the virtio-msg transport layer. > >>>>>>>>>>>> > >>>>>>>>>>>> The individuals and organizations involved in this effort have had difficulty in > >>>>>>>>>>>> using the existing virtio-transports in various situations and desire to add one > >>>>>>>>>>>> more transport that performs its transport layer operations by sending and > >>>>>>>>>>>> receiving messages. > >>>>>>>>>>>> > >>>>>>>>>>>> Implementations of virtio-msg will normally be done in multiple layers: > >>>>>>>>>>>> * common / device level > >>>>>>>>>>>> * bus level > >>>>>>>>>>>> > >>>>>>>>>>>> The common / device level defines the messages exchanged between the driver > >>>>>>>>>>>> and a device. This common part should lead to a common driver holding most > >>>>>>>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >>>>>>>>>>>> The kernel implementation in [3] shows this separation. As with other transport > >>>>>>>>>>>> layers, virtio-msg should not require modifications to existing virtio device > >>>>>>>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the > >>>>>>>>>>>> main focus of this version of the patch series. > >>>>>>>>>>>> > >>>>>>>>>>>> The virtio-msg bus level implements the normal things a bus defines > >>>>>>>>>>>> (enumeration, dma operations, etc) but also implements the message send and > >>>>>>>>>>>> receive operations. A number of bus implementations are envisioned, > >>>>>>>>>>>> some of which will be reusable and general purpose. Other bus implementations > >>>>>>>>>>>> might be unique to a given situation, for example only used by a PCIe card > >>>>>>>>>>>> and its driver. > >>>>>>>>>>>> > >>>>>>>>>>>> The standard bus messages are an effort to avoid different bus implementations > >>>>>>>>>>>> doing the same thing in different ways for no good reason. However the > >>>>>>>>>>>> different environments will require different things. Instead of trying to > >>>>>>>>>>>> anticipate all needs and provide something very abstract, we think > >>>>>>>>>>>> implementation specific messages will be needed at the bus level. Over time, > >>>>>>>>>>>> if we see similar messages across multiple bus implementations, we will move to > >>>>>>>>>>>> standardize a bus level message for that. > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I would review more, had first round of sparse review. > >>>>>>>>>>> Please find few comments/questions below. > >>>>>>>>>> > >>>>>>>>>> I'd like to comment that I think it makes sense to have a basic simple transport and > >>>>>>>>>> then add performance features on top as appropriate. > >>>>>>>>> Sounds good. Simple but complete is needed. > >>>>>>>> > >>>>>>>> Agree. > >>>>>>>> > >>>>>>>>> > >>>>>>>>>> So one way to address some of these comments is to show how > >>>>>>>>>> they can be addressed with a feature bit down the road. > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> 1. device number should be 32-bit in struct virtio_msg_header. > >>>>>>>>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > >>>>>>>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. > >>>>>>>>>>> > >>>>>>>>>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > >>>>>>>>>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > >>>>>>>>>>> Needs 32-bits. > >>>>>>>>>>> > >>>>>>>>>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > >>>>>>>>>>> But more below. > >>>>>>>>>>> > >>>>>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > >>>>> operate > >>>>>>>>>> the virtqueues. > >>>>>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >>>>>>>>>> > >>>>>>>>>> surely this can be an optional transport feature bit. > >>>>>>>>>> > >>>>>>>>> How is this optional? > >>>>>>>> > >>>>>>>> As said in a previous mail, we have messages already for that. > >>>>>>>> Please confirm if that answer your question. > >>>>>>>> > >>>>>>>>> How can one implement a transport without defining the basic data transfer semantics? > >>>>>>>> > >>>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > >>>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue. > >>>>>>>> > >>>>>>> > >>>>>>> I'd love to have this transport over PCI because it addresses the shortcomings > >>>>>>> of the existing PCI transport which just assumes that every config space access\ > >>>>>>> is trap and emulate. > >>>>>> > >>>>>> Agree and AMD did exactly that in their demonstrator. > >>>>>> I will give you answers here as i know them but Edgar will probably give you more > >>>>>> details (and probably fix my mistakes). > >>>>>> > >>>>>>> > >>>>>>> But that being said, I somewhat agree with Parav that we should define the bus > >>>>>>> implementations in the spec to avoid fixing the ABI in the implementations. For > >>>>>>> instance, if we try to use this transport over PCI, we've got questions like: > >>>>>>> > >>>>>>> 1. How the device should be bind to the virtio-msg-pci bus driver and not with > >>>>>>> the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > >>>>>> > >>>>>> One bus is appearing as one pci device with its own Vendor ID, > >>>>>> > >>>>> > >>>>> What should be the 'own Vendor ID' here? > >>>>> > >>>>> The existing virtio-pci driver binds to all devices with the Vendor ID of > >>>>> PCI_VENDOR_ID_REDHAT_QUMRANET. So are you expecting the Vendors to use their own > >>>>> VID for exposing the Virtio devices? That would mean, the drivers on the host > >>>>> need update as well, which will not scale. > >>>>> > >>>>> It would be good if the existing virtio-pci devices can use this new transport > >>>>> with only device side modifications. > >>>>> > >>>>>>> > >>>>>>> 2. How the Virtio messages should be transferred? Is it through endpoint config > >>>>>>> space or through some other means? > >>>>>> > >>>>>> The virtio messages are transfered using FIFOs stored in the BAR of the PCI > >>>>>> device (ending up being memory shared between both sides) > >>>>>> > >>>>> > >>>>> What should be the BAR number and size? > >>>>> > >>>>>>> > >>>>>>> 3. How the notification be delivered from the device to the host? Through > >>>>>>> INT-X/MSI/MSI-X or even polling? > >>>>>> > >>>>>> Notifications are delivered through MSI. > >>>>>> > >>>>> > >>>>> So no INT-X or MSI-X? Why so? > >>>>> > >>>>> Anyhow, my objective is not to get answers for my above questions here in this > >>>>> thread, but to state the reality that it would be hard for us to make use of > >>>>> this new transport without defining the bus implementation. > >>>>> > >>>> +1 to most of the points that Manivannan explained. > >>>> > >>>> The whole new definition of message layer for the PCI does not make any sense at all where expectation for the device is to build yet > >>> another interface for _Everything_ that already exists. > >>>> and device is still have to implement all the existing things because the device does not know which driver will operate. > >>>> > >>>> And that too some register based inefficient interface. > >>>> Just to reset the device one needs to fully setup the new message interface but device still have to be working. > >>>> That defeats the whole purpose of reset_1 and reset_2 in the device. > >>>> > >>>> This does not bring anything better for the PCI devices at all. > >>>> > >>>> A transport binding should be defined for the bus binding. > >>>> A bus that chooses a msg interface should be listed that way and bus choose inline messages can continue the way they are. > >>>> > >>>> If we are creating something brand-new, for PCI the only thing needed is: > >>>> 1. Reset the device > >>>> 2. Create an admin virtqueue > >>>> 3. Transport everything needed through this virtqueue including features, configs, control. > >>>> > >>>> And this will work for any other bus or msg based too given only contract needed is to creating the aq. > >>> > >>> I think you misunderstood a bit the point of virtio-msg bus over PCI so let me try to explain. > >>> > >>> You see one PCI device (regular, not virtio) which is a "virtio-msg bus over PCI". > >>> > >>> When the virtio-msg bus over PCI it will communicate through this device with an external > >>> system connected through the PCI bus. > >>> The driver will enumerate virtio devices available behind this bus and register them so that > >>> the corresponding virtio drivers are probed for them. > >>> All virtio-msg messages required to communicate with those devices will be transferred through > >>> a FIFO stored in the BAR of the pci device and standard PCI DMA will be used to share the > >>> virtqueues with all the devices on the bus. > >>> > >>> So the PCI device is not one virtio device but one bus behind which there can be many devices. > >>> > >>> Is this making the concept a bit clearer ? > >>> > >> Yes. This makes a lot of sense now. > >> > >> This is a virtio-msg-transport device that needs its own device id in the table. > >> And its binding to the PCI transport. > > > > ok. how about an rfc of that idea on the list? > > > I will let Edgar answer on this. Yes, I agree it makes sense to document the virtio-msg PCI bus. We've been looking at a virtio-msg PCI bus for several reasons. One is to share devices between two SoCs connected over PCI, e.g., an x86 host and an ARM endpoint. To enable virtio-msg, we need shared memory for FIFOs plus notifications in both directions. We also need DMA for virtqueue access. In the simple case where the endpoint shares devices to the host, we have a BAR0 with notification registers (host to EP) and a prefetchable BAR1 with RAM for FIFOs. We implement software FIFOs in RAM and move virtio-msg messages over these. When the host needs to notify the EP, it writes to the notification registers in BAR0. When the EP needs to notify the host, it raises an interrupt (we're using MSI-X now, but it could be MSI or INTX). When the device model running on the EP needs to access the host's VQs, it uses ordinary PCI DMA. When the host wants to share a device with the EP, we need an additional large BAR2 (e.g., 4 GB) to expose EP memory to the host for DMA. The virtio-msg queues work the same way as in the first scenario, but DMA to the VQs is done over BAR2. This is still a work in progress, but we have a setup that demonstrates EP-to-host sharing. In our setup, we're using virtio-msg to share devices between host kernels, e.g., dom0 on the x86 and dom0 on the ARM side. To share with guests we're currently re-sharing using virtio-pci to support unmodified non-virtio-msg-capable guests. If we wanted to do virtio-msg end-to-end between guests, we'd need to multiply the PCI functions to address the IOMMU issue that Parav mentioned, either by using multiple functions or multiple VFs (SR-IOV) (one per guest). This is something we've talked about but not yet tried. A second reason for looking at PCI was to provide a hypervisor-neutral and easy way to access and try virtio-msg. So we created a QEMU model of the EP-to-host sharing mechanisms described above. This is what we've submitted to QEMU as the virtio-msg AMP PCI device. https://lore.kernel.org/qemu-devel/20260224155721.612314-1-edgar.iglesias@gmail.com/ ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 7:45 ` Manivannan Sadhasivam 2026-02-25 8:03 ` Bertrand Marquis @ 2026-02-25 12:53 ` Demi Marie Obenour 2026-02-25 13:09 ` Manivannan Sadhasivam 1 sibling, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 12:53 UTC (permalink / raw) To: Manivannan Sadhasivam, Bertrand Marquis Cc: Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 2057 bytes --] On 2/25/26 02:45, Manivannan Sadhasivam wrote: > Hi Bertrand, > > On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >> Hi Parav, (snip) >>> How can one implement a transport without defining the basic data transfer semantics? >> >> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >> If anything is missing, we are more than happy to discuss it and solve the issue. >> > > I'd love to have this transport over PCI because it addresses the shortcomings > of the existing PCI transport which just assumes that every config space access\ > is trap and emulate. Would you mind elaborating on "over PCI"? I can see two meanings for this: 1. Virtio-msg over a physical PCI bus for heterogeneous multiprocessing environments. 2. Virtio-msg over an emulated PCI bus for QEMU, Cloud Hypervisor, and other existing VMMs. Which one are you referring to? > But that being said, I somewhat agree with Parav that we should define the bus > implementations in the spec to avoid fixing the ABI in the implementations. For > instance, if we try to use this transport over PCI, we've got questions like:> > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > 2. How the Virtio messages should be transferred? Is it through endpoint config > space or through some other means? > > 3. How the notification be delivered from the device to the host? Through > INT-X/MSI/MSI-X or even polling? > > And these are just a few questions that comes to the top of my head. There could > be plenty more. > > How can we expect all the virtio-msg bus implementations to adhere to the same > format so that the interoperability offered by the Virtio spec is guaranteed? This really ought to be included in the virtio standard. FF-A is an Arm specification, but PCI is not, and virtio already references PCI. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 12:53 ` Demi Marie Obenour @ 2026-02-25 13:09 ` Manivannan Sadhasivam 2026-02-25 13:12 ` Demi Marie Obenour 0 siblings, 1 reply; 105+ messages in thread From: Manivannan Sadhasivam @ 2026-02-25 13:09 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bertrand Marquis, Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 07:53:38AM -0500, Demi Marie Obenour wrote: > On 2/25/26 02:45, Manivannan Sadhasivam wrote: > > Hi Bertrand, > > > > On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: > >> Hi Parav, > > (snip) > > >>> How can one implement a transport without defining the basic data transfer semantics? > >> > >> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. > >> If anything is missing, we are more than happy to discuss it and solve the issue. > >> > > > > I'd love to have this transport over PCI because it addresses the shortcomings > > of the existing PCI transport which just assumes that every config space access\ > > is trap and emulate. > > Would you mind elaborating on "over PCI"? > > I can see two meanings for this: > > 1. Virtio-msg over a physical PCI bus for heterogeneous multiprocessing > environments. > > 2. Virtio-msg over an emulated PCI bus for QEMU, Cloud Hypervisor, > and other existing VMMs. > > Which one are you referring to? > Physical PCI bus, that's why I mentioned 'trap and emulate' which only applies to emulated PCI bus. > > But that being said, I somewhat agree with Parav that we should define the bus > > implementations in the spec to avoid fixing the ABI in the implementations. For > > instance, if we try to use this transport over PCI, we've got questions like:> > > 1. How the device should be bind to the virtio-msg-pci bus driver and not with > > the existing virtio-pci driver? Should it use a new Vendor ID or Sub-IDs? > > > > 2. How the Virtio messages should be transferred? Is it through endpoint config > > space or through some other means? > > > > 3. How the notification be delivered from the device to the host? Through > > INT-X/MSI/MSI-X or even polling? > > > > And these are just a few questions that comes to the top of my head. There could > > be plenty more. > > > > How can we expect all the virtio-msg bus implementations to adhere to the same > > format so that the interoperability offered by the Virtio spec is guaranteed? > This really ought to be included in the virtio standard. FF-A is an > Arm specification, but PCI is not, and virtio already references PCI. Yes! - Mani -- மணிவண்ணன் சதாசிவம் ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 13:09 ` Manivannan Sadhasivam @ 2026-02-25 13:12 ` Demi Marie Obenour 2026-02-25 13:29 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 13:12 UTC (permalink / raw) To: Manivannan Sadhasivam Cc: Bertrand Marquis, Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 1403 bytes --] On 2/25/26 08:09, Manivannan Sadhasivam wrote: > On Wed, Feb 25, 2026 at 07:53:38AM -0500, Demi Marie Obenour wrote: >> On 2/25/26 02:45, Manivannan Sadhasivam wrote: >>> Hi Bertrand, >>> >>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>> Hi Parav, >> >> (snip) >> >>>>> How can one implement a transport without defining the basic data transfer semantics? >>>> >>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>> >>> >>> I'd love to have this transport over PCI because it addresses the shortcomings >>> of the existing PCI transport which just assumes that every config space access\ >>> is trap and emulate. >> >> Would you mind elaborating on "over PCI"? >> >> I can see two meanings for this: >> >> 1. Virtio-msg over a physical PCI bus for heterogeneous multiprocessing >> environments. >> >> 2. Virtio-msg over an emulated PCI bus for QEMU, Cloud Hypervisor, >> and other existing VMMs. >> >> Which one are you referring to? >> > > Physical PCI bus, that's why I mentioned 'trap and emulate' which only applies > to emulated PCI bus. Is virtio-msg appropriate when the PCI bus is emulated, or should the existing PCI transport be used instead? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 13:12 ` Demi Marie Obenour @ 2026-02-25 13:29 ` Bertrand Marquis 2026-02-25 15:19 ` Demi Marie Obenour 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 13:29 UTC (permalink / raw) To: Demi Marie Obenour Cc: Manivannan Sadhasivam, Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 25 Feb 2026, at 14:12, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 2/25/26 08:09, Manivannan Sadhasivam wrote: >> On Wed, Feb 25, 2026 at 07:53:38AM -0500, Demi Marie Obenour wrote: >>> On 2/25/26 02:45, Manivannan Sadhasivam wrote: >>>> Hi Bertrand, >>>> >>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>>> Hi Parav, >>> >>> (snip) >>> >>>>>> How can one implement a transport without defining the basic data transfer semantics? >>>>> >>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>>> >>>> >>>> I'd love to have this transport over PCI because it addresses the shortcomings >>>> of the existing PCI transport which just assumes that every config space access\ >>>> is trap and emulate. >>> >>> Would you mind elaborating on "over PCI"? >>> >>> I can see two meanings for this: >>> >>> 1. Virtio-msg over a physical PCI bus for heterogeneous multiprocessing >>> environments. >>> >>> 2. Virtio-msg over an emulated PCI bus for QEMU, Cloud Hypervisor, >>> and other existing VMMs. >>> >>> Which one are you referring to? >>> >> >> Physical PCI bus, that's why I mentioned 'trap and emulate' which only applies >> to emulated PCI bus. > > Is virtio-msg appropriate when the PCI bus is emulated, or should > the existing PCI transport be used instead? No existing PCI transport should be used, virtio msg over PCI is just for a specific case and definitely not when virtio devices are pci devices. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 13:29 ` Bertrand Marquis @ 2026-02-25 15:19 ` Demi Marie Obenour 2026-02-25 15:27 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 15:19 UTC (permalink / raw) To: Bertrand Marquis Cc: Manivannan Sadhasivam, Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 3282 bytes --] On 2/25/26 08:29, Bertrand Marquis wrote: > Hi Demi, > >> On 25 Feb 2026, at 14:12, Demi Marie Obenour <demiobenour@gmail.com> wrote: >> >> On 2/25/26 08:09, Manivannan Sadhasivam wrote: >>> On Wed, Feb 25, 2026 at 07:53:38AM -0500, Demi Marie Obenour wrote: >>>> On 2/25/26 02:45, Manivannan Sadhasivam wrote: >>>>> Hi Bertrand, >>>>> >>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>>>> Hi Parav, >>>> >>>> (snip) >>>> >>>>>>> How can one implement a transport without defining the basic data transfer semantics? >>>>>> >>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>>>> >>>>> >>>>> I'd love to have this transport over PCI because it addresses the shortcomings >>>>> of the existing PCI transport which just assumes that every config space access\ >>>>> is trap and emulate. >>>> >>>> Would you mind elaborating on "over PCI"? >>>> >>>> I can see two meanings for this: >>>> >>>> 1. Virtio-msg over a physical PCI bus for heterogeneous multiprocessing >>>> environments. >>>> >>>> 2. Virtio-msg over an emulated PCI bus for QEMU, Cloud Hypervisor, >>>> and other existing VMMs. >>>> >>>> Which one are you referring to? >>>> >>> >>> Physical PCI bus, that's why I mentioned 'trap and emulate' which only applies >>> to emulated PCI bus. >> >> Is virtio-msg appropriate when the PCI bus is emulated, or should >> the existing PCI transport be used instead? > > No existing PCI transport should be used, virtio msg over PCI is just for a specific case > and definitely not when virtio devices are pci devices. Is virtio-msg an appropriate way for a VM to _implement_ a virtio device? More specifically: Suppose I want to implement a virtio device in a VM, rather than on the host, for isolation reasons. This is on KVM, so the Xen transport isn't feasible, and it needs to be portable, so the FF-A transport isn't feasible. The AMP transport _could_ be feasible, but it requires a "virtio-vfio" device that speaks the vfio-user protocol. Such a device does not currently exist. There are a few other options: 1. I can implement the virtio-vhost-user protocol. This was proposed but never standardized. It has the advantage of being a good fit for the vhost-user protocol, and thus providing easy interoperability with many frontend VMMs like QEMU. 2. I can implement virtio-msg over PCI, but have the driver speak the _device_ side of the protocol. This is a better fit to the virtio-msg-pci device, which is going to be standardized. However, it doesn't map as well to the vhost-user protocol, and thus will make it harder to interoperate with existing frontend VMMs. 3. I can implement virtio-msg over PCI, have the driver speak the device side of the protocol, and translate this to vhost-user in the VMM. This leads to additional complexity and also creates a specific device that is not likely to be used for other purposes. 4. I can try to implement whatever future interface will be available for QEMU to implement devices. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:19 ` Demi Marie Obenour @ 2026-02-25 15:27 ` Bertrand Marquis 0 siblings, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 15:27 UTC (permalink / raw) To: Demi Marie Obenour Cc: Manivannan Sadhasivam, Parav Pandit, Michael S. Tsirkin, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 25 Feb 2026, at 16:19, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 2/25/26 08:29, Bertrand Marquis wrote: >> Hi Demi, >> >>> On 25 Feb 2026, at 14:12, Demi Marie Obenour <demiobenour@gmail.com> wrote: >>> >>> On 2/25/26 08:09, Manivannan Sadhasivam wrote: >>>> On Wed, Feb 25, 2026 at 07:53:38AM -0500, Demi Marie Obenour wrote: >>>>> On 2/25/26 02:45, Manivannan Sadhasivam wrote: >>>>>> Hi Bertrand, >>>>>> >>>>>> On Fri, Feb 20, 2026 at 09:02:12AM +0000, Bertrand Marquis wrote: >>>>>>> Hi Parav, >>>>> >>>>> (snip) >>>>> >>>>>>>> How can one implement a transport without defining the basic data transfer semantics? >>>>>>> >>>>>>> We did a lot of experiments and we are feature equivalent to PCI, MMIO or Channel I/O. >>>>>>> If anything is missing, we are more than happy to discuss it and solve the issue. >>>>>>> >>>>>> >>>>>> I'd love to have this transport over PCI because it addresses the shortcomings >>>>>> of the existing PCI transport which just assumes that every config space access\ >>>>>> is trap and emulate. >>>>> >>>>> Would you mind elaborating on "over PCI"? >>>>> >>>>> I can see two meanings for this: >>>>> >>>>> 1. Virtio-msg over a physical PCI bus for heterogeneous multiprocessing >>>>> environments. >>>>> >>>>> 2. Virtio-msg over an emulated PCI bus for QEMU, Cloud Hypervisor, >>>>> and other existing VMMs. >>>>> >>>>> Which one are you referring to? >>>>> >>>> >>>> Physical PCI bus, that's why I mentioned 'trap and emulate' which only applies >>>> to emulated PCI bus. >>> >>> Is virtio-msg appropriate when the PCI bus is emulated, or should >>> the existing PCI transport be used instead? >> >> No existing PCI transport should be used, virtio msg over PCI is just for a specific case >> and definitely not when virtio devices are pci devices. > > Is virtio-msg an appropriate way for a VM to _implement_ a virtio device? > > More specifically: > > Suppose I want to implement a virtio device in a VM, rather than > on the host, for isolation reasons. This is on KVM, so the Xen > transport isn't feasible, and it needs to be portable, so the FF-A > transport isn't feasible. The AMP transport _could_ be feasible, but > it requires a "virtio-vfio" device that speaks the vfio-user protocol. > Such a device does not currently exist. Right now virtio device/driver implementations are independent of the transport so the idea is you can implement it and reuse in KVM (which will probably continue to use PCI), xen (which currently use PCI or mmio using ioreq but will also support virtio-msg). > > There are a few other options: > > 1. I can implement the virtio-vhost-user protocol. This was > proposed but never standardized. It has the advantage of being > a good fit for the vhost-user protocol, and thus providing easy > interoperability with many frontend VMMs like QEMU. That is a solution if available. > > 2. I can implement virtio-msg over PCI, but have the driver speak > the _device_ side of the protocol. This is a better fit to the > virtio-msg-pci device, which is going to be standardized. However, > it doesn't map as well to the vhost-user protocol, and thus will > make it harder to interoperate with existing frontend VMMs. We need to stop speaking of PCI, it is giving a wrong idea. In your use case it does not make sense if PCI is emulated you would do virtio over PCI or use virtio over Messages if available but virtio-msg over PCI is never a solution which should only be used in cases where you have a physical PCI connection between 2 systems. > > 3. I can implement virtio-msg over PCI, have the driver speak the > device side of the protocol, and translate this to vhost-user in > the VMM. This leads to additional complexity and also creates a > specific device that is not likely to be used for other purposes. same as above > > 4. I can try to implement whatever future interface will be available > for QEMU to implement devices. I would do this. The plan is to add virtio-msg support in qemu so that any device implementation available in qemu could be used with virtio-msg or virtio-pci or virtio-mmio. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-20 6:13 ` Parav Pandit 2026-02-20 9:02 ` Bertrand Marquis @ 2026-02-20 10:03 ` Michael S. Tsirkin 2026-02-25 5:09 ` Parav Pandit 1 sibling, 1 reply; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-20 10:03 UTC (permalink / raw) To: Parav Pandit Cc: Bill Mills, virtio-comment@lists.linux.dev, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Fri, Feb 20, 2026 at 06:13:55AM +0000, Parav Pandit wrote: > > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate > > the virtqueues. > > > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > > > surely this can be an optional transport feature bit. > > > How is this optional? > How can one implement a transport without defining the basic data transfer semantics? > For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. > How does it work? I'm not sure what do foo/bar refer to, or what TCP transporting means. The simplest way to do TCP on top of virtio is to layer it above virtio net. That uses VQs for data transfers. -- MST ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-20 10:03 ` Michael S. Tsirkin @ 2026-02-25 5:09 ` Parav Pandit 2026-02-25 7:25 ` Michael S. Tsirkin 0 siblings, 1 reply; 105+ messages in thread From: Parav Pandit @ 2026-02-25 5:09 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Bill Mills, virtio-comment@lists.linux.dev, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Michael S. Tsirkin <mst@redhat.com> > Sent: 20 February 2026 03:33 PM > > On Fri, Feb 20, 2026 at 06:13:55AM +0000, Parav Pandit wrote: > > > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > operate > > > the virtqueues. > > > > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > > > > > surely this can be an optional transport feature bit. > > > > > How is this optional? > > How can one implement a transport without defining the basic data transfer semantics? > > For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. > > How does it work? > > I'm not sure what do foo/bar refer to, or what TCP transporting means. This proposal is subset of past proposal of [1]. Proposal [1] covered transporting virtio control and data operations using something other than MMIO and PCI. And current proposal is similar, except that it didn't define the transport binding at all for the specific bus. It is only a partial 'control-transport'. [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ So foo and bar are the definitions I expect as listed in the patch-5 of [1]. If it has to be done by the bus, lets write up this as 'control-transport'. > The simplest way to do TCP on top of virtio is to layer it above virtio > net. That uses VQs for data transfers. > The intention of this proposal is not to do TCP on top of virtio. The intention of this proposal is to do virtio on transports other than MMIO, PCI and channel. Such a transport can be anything - not defined in virtio spec. It could be FFA, some two SoC as written in cover letter example, or it can be something else such as TCP or UDP or vsock or whatever else. And hence, data transfer part must be scoped properly. Maybe I am yet to read this text in the 1600 lines of proposed patch... > -- > MST ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 5:09 ` Parav Pandit @ 2026-02-25 7:25 ` Michael S. Tsirkin 2026-02-25 9:18 ` Parav Pandit 0 siblings, 1 reply; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 7:25 UTC (permalink / raw) To: Parav Pandit Cc: Bill Mills, virtio-comment@lists.linux.dev, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 05:09:45AM +0000, Parav Pandit wrote: > > > > From: Michael S. Tsirkin <mst@redhat.com> > > Sent: 20 February 2026 03:33 PM > > > > On Fri, Feb 20, 2026 at 06:13:55AM +0000, Parav Pandit wrote: > > > > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > > operate > > > > the virtqueues. > > > > > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > > > > > > > surely this can be an optional transport feature bit. > > > > > > > How is this optional? > > > How can one implement a transport without defining the basic data transfer semantics? > > > For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. > > > How does it work? > > > > I'm not sure what do foo/bar refer to, or what TCP transporting means. > This proposal is subset of past proposal of [1]. > > Proposal [1] covered transporting virtio control and data operations using something other than MMIO and PCI. > And current proposal is similar, except that it didn't define the transport binding at all for the specific bus. > It is only a partial 'control-transport'. > > [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ > > So foo and bar are the definitions I expect as listed in the patch-5 of [1]. > If it has to be done by the bus, lets write up this as 'control-transport'. > > > The simplest way to do TCP on top of virtio is to layer it above virtio > > net. That uses VQs for data transfers. > > > The intention of this proposal is not to do TCP on top of virtio. > The intention of this proposal is to do virtio on transports other than MMIO, PCI and channel. > Such a transport can be anything - not defined in virtio spec. > It could be FFA, some two SoC as written in cover letter example, or it can be something else such as TCP or UDP or vsock or whatever else. I feel this "anything" is simply too broad a requirement. I did not see any demand for virtio over TCP. And, making it work with existing drivers will be a mess. We can scope this for buses that can do DMA for now. > > And hence, data transfer part must be scoped properly. > Maybe I am yet to read this text in the 1600 lines of proposed patch... Once we get into "support everything in the most abstract way possible" we have already lost. No one asked for virtio over carrier pigeons. Let us not over engineer please. -- MST ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 7:25 ` Michael S. Tsirkin @ 2026-02-25 9:18 ` Parav Pandit 2026-02-25 9:22 ` Michael S. Tsirkin 0 siblings, 1 reply; 105+ messages in thread From: Parav Pandit @ 2026-02-25 9:18 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Bill Mills, virtio-comment@lists.linux.dev, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Michael S. Tsirkin <mst@redhat.com> > Sent: 25 February 2026 12:55 PM > > On Wed, Feb 25, 2026 at 05:09:45AM +0000, Parav Pandit wrote: > > > > > > > From: Michael S. Tsirkin <mst@redhat.com> > > > Sent: 20 February 2026 03:33 PM > > > > > > On Fri, Feb 20, 2026 at 06:13:55AM +0000, Parav Pandit wrote: > > > > > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > > > operate > > > > > the virtqueues. > > > > > > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > > > > > > > > > surely this can be an optional transport feature bit. > > > > > > > > > How is this optional? > > > > How can one implement a transport without defining the basic data transfer semantics? > > > > For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. > > > > How does it work? > > > > > > I'm not sure what do foo/bar refer to, or what TCP transporting means. > > This proposal is subset of past proposal of [1]. > > > > Proposal [1] covered transporting virtio control and data operations using something other than MMIO and PCI. > > And current proposal is similar, except that it didn't define the transport binding at all for the specific bus. > > It is only a partial 'control-transport'. > > > > [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ > > > > So foo and bar are the definitions I expect as listed in the patch-5 of [1]. > > If it has to be done by the bus, lets write up this as 'control-transport'. > > > > > The simplest way to do TCP on top of virtio is to layer it above virtio > > > net. That uses VQs for data transfers. > > > > > The intention of this proposal is not to do TCP on top of virtio. > > The intention of this proposal is to do virtio on transports other than MMIO, PCI and channel. > > Such a transport can be anything - not defined in virtio spec. > > > It could be FFA, some two SoC as written in cover letter example, or it can be something else such as TCP or UDP or vsock or whatever else. > > I feel this "anything" is simply too broad a requirement. > I did not see any demand for virtio over TCP. > And, making it work with existing drivers will be a mess. Why would it be a mess? Because of load/store semantics? If so, would the message layer also faces the same challenge? > We can scope this for buses that can do DMA for now. That looks reasonable good start. However, "Appendix C. Creating New Transports" needs modification. Following two likely needs to stay outside of the "control transport". A transport provides a mechanism for the device to send device notifications to the driver, such as used buffer notifications. A transport provides a mechanism for the driver to send driver notifications to the device, such as available buffer notifications. We also need to fix the Appendix to describe about control transport and full transport. Current definition of Appendix C is listing MMIO And PCI transport, yet it misses virtqueue implementation by _full_ transport. > > > > > > > And hence, data transfer part must be scoped properly. > > Maybe I am yet to read this text in the 1600 lines of proposed patch... > > > Once we get into "support everything in the most abstract way possible" > we have already lost. Unlikely. Nvme over TCP is present since 2021. NVMe over RDMA is present since 2017. iSCSI over TCP is present from 2014. NFS .. List continues... The idea is to not define most abstract thing. The idea is to define the practical spec that cater to requirements already listed, including the TCP one in [1]. > No one asked for virtio over carrier pigeons. CSP user already explained the use case of virtio over network transport in [1]. Considering above many use cases already done on similar non virtio devices as pigeons is severe undermining the scope of virtio. One can say, I only want to engineer control transport ignoring needs of [1]. However, we must have the doors and vision open so that users of [1] can also use it without creating yet another 'fabric transport'. At the same time, we must define the binding to the bus where it is going to be used for the DMA. For example, cover letter mentions about " between a host processor and its co-processors" And " between normal and secure worlds" How can one implement a driver if the bus driver does not know how to enumerate/discover it? In the proposed example, of host processor and co-processor if they are connected via Xenbus or AMBA bus, how should driver discover this device? If this bus binding is not part of the virtio spec, how can it be ever implemented and yet comply to the virtio spec? Can someone please explain this? I frankly expect an ARM FF-A bus binding transport section in virtio-spec. So that device and driver can inter-operate implemented by two different entities. Only virtio level messages do not seem sufficient. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 9:18 ` Parav Pandit @ 2026-02-25 9:22 ` Michael S. Tsirkin 2026-02-25 9:35 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 9:22 UTC (permalink / raw) To: Parav Pandit Cc: Bill Mills, virtio-comment@lists.linux.dev, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 09:18:58AM +0000, Parav Pandit wrote: > > > From: Michael S. Tsirkin <mst@redhat.com> > > Sent: 25 February 2026 12:55 PM > > > > On Wed, Feb 25, 2026 at 05:09:45AM +0000, Parav Pandit wrote: > > > > > > > > > > From: Michael S. Tsirkin <mst@redhat.com> > > > > Sent: 20 February 2026 03:33 PM > > > > > > > > On Fri, Feb 20, 2026 at 06:13:55AM +0000, Parav Pandit wrote: > > > > > > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > > > > operate > > > > > > the virtqueues. > > > > > > > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > > > > > > > > > > > surely this can be an optional transport feature bit. > > > > > > > > > > > How is this optional? > > > > > How can one implement a transport without defining the basic data transfer semantics? > > > > > For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. > > > > > How does it work? > > > > > > > > I'm not sure what do foo/bar refer to, or what TCP transporting means. > > > This proposal is subset of past proposal of [1]. > > > > > > Proposal [1] covered transporting virtio control and data operations using something other than MMIO and PCI. > > > And current proposal is similar, except that it didn't define the transport binding at all for the specific bus. > > > It is only a partial 'control-transport'. > > > > > > [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ > > > > > > So foo and bar are the definitions I expect as listed in the patch-5 of [1]. > > > If it has to be done by the bus, lets write up this as 'control-transport'. > > > > > > > The simplest way to do TCP on top of virtio is to layer it above virtio > > > > net. That uses VQs for data transfers. > > > > > > > The intention of this proposal is not to do TCP on top of virtio. > > > The intention of this proposal is to do virtio on transports other than MMIO, PCI and channel. > > > Such a transport can be anything - not defined in virtio spec. > > > > > It could be FFA, some two SoC as written in cover letter example, or it can be something else such as TCP or UDP or vsock or whatever else. > > > > I feel this "anything" is simply too broad a requirement. > > I did not see any demand for virtio over TCP. > > And, making it work with existing drivers will be a mess. > Why would it be a mess? Because of load/store semantics? > If so, would the message layer also faces the same challenge? not sure what "the message layer" is. > > We can scope this for buses that can do DMA for now. > That looks reasonable good start. > However, "Appendix C. Creating New Transports" needs modification. > > Following two likely needs to stay outside of the "control transport". > > A transport provides a mechanism for the device to send device notifications to the driver, such as used > buffer notifications. > A transport provides a mechanism for the driver to send driver notifications to the device, such as available > buffer notifications. Well existing transports sure do include these. > We also need to fix the Appendix to describe about control transport and full transport. > Current definition of Appendix C is listing MMIO And PCI transport, yet it misses virtqueue implementation by _full_ transport. because they all share a single virtqueue implementation. > > > > > > > > > > > > And hence, data transfer part must be scoped properly. > > > Maybe I am yet to read this text in the 1600 lines of proposed patch... > > > > > > Once we get into "support everything in the most abstract way possible" > > we have already lost. > Unlikely. > Nvme over TCP is present since 2021. > NVMe over RDMA is present since 2017. > iSCSI over TCP is present from 2014. > NFS .. > List continues... > > The idea is to not define most abstract thing. > The idea is to define the practical spec that cater to requirements already listed, including the TCP one in [1]. I don't object to virtio over TCP. just do not want to block this work on that. > > No one asked for virtio over carrier pigeons. > CSP user already explained the use case of virtio over network transport in [1]. > Considering above many use cases already done on similar non virtio devices as pigeons is severe undermining the scope of virtio. > > One can say, I only want to engineer control transport ignoring needs of [1]. > However, we must have the doors and vision open so that users of [1] can also use it without creating yet another 'fabric transport'. > > At the same time, we must define the binding to the bus where it is going to be used for the DMA. > > For example, cover letter mentions about " between a host processor and its co-processors" > And " between normal and secure worlds" > > How can one implement a driver if the bus driver does not know how to enumerate/discover it? > > In the proposed example, of host processor and co-processor if they are connected via Xenbus or AMBA bus, > how should driver discover this device? > > If this bus binding is not part of the virtio spec, how can it be ever implemented and yet comply to the virtio spec? > Can someone please explain this? > > I frankly expect an ARM FF-A bus binding transport section in virtio-spec. > So that device and driver can inter-operate implemented by two different entities. > Only virtio level messages do not seem sufficient. I agree at least one should be shown at least as an rfc. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 9:22 ` Michael S. Tsirkin @ 2026-02-25 9:35 ` Bertrand Marquis 2026-02-25 9:54 ` Michael S. Tsirkin 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 9:35 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Michael and Parav, > On 25 Feb 2026, at 10:22, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Wed, Feb 25, 2026 at 09:18:58AM +0000, Parav Pandit wrote: >> >>> From: Michael S. Tsirkin <mst@redhat.com> >>> Sent: 25 February 2026 12:55 PM >>> >>> On Wed, Feb 25, 2026 at 05:09:45AM +0000, Parav Pandit wrote: >>>> >>>> >>>>> From: Michael S. Tsirkin <mst@redhat.com> >>>>> Sent: 20 February 2026 03:33 PM >>>>> >>>>> On Fri, Feb 20, 2026 at 06:13:55AM +0000, Parav Pandit wrote: >>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to >>>>> operate >>>>>>> the virtqueues. >>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >>>>>>> >>>>>>> surely this can be an optional transport feature bit. >>>>>>> >>>>>> How is this optional? >>>>>> How can one implement a transport without defining the basic data transfer semantics? >>>>>> For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. >>>>>> How does it work? >>>>> >>>>> I'm not sure what do foo/bar refer to, or what TCP transporting means. >>>> This proposal is subset of past proposal of [1]. >>>> >>>> Proposal [1] covered transporting virtio control and data operations using something other than MMIO and PCI. >>>> And current proposal is similar, except that it didn't define the transport binding at all for the specific bus. >>>> It is only a partial 'control-transport'. >>>> >>>> [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ >>>> >>>> So foo and bar are the definitions I expect as listed in the patch-5 of [1]. >>>> If it has to be done by the bus, lets write up this as 'control-transport'. >>>> >>>>> The simplest way to do TCP on top of virtio is to layer it above virtio >>>>> net. That uses VQs for data transfers. >>>>> >>>> The intention of this proposal is not to do TCP on top of virtio. >>>> The intention of this proposal is to do virtio on transports other than MMIO, PCI and channel. >>>> Such a transport can be anything - not defined in virtio spec. >>> >>>> It could be FFA, some two SoC as written in cover letter example, or it can be something else such as TCP or UDP or vsock or whatever else. >>> >>> I feel this "anything" is simply too broad a requirement. >>> I did not see any demand for virtio over TCP. >>> And, making it work with existing drivers will be a mess. >> Why would it be a mess? Because of load/store semantics? >> If so, would the message layer also faces the same challenge? > > not sure what "the message layer" is. > >>> We can scope this for buses that can do DMA for now. >> That looks reasonable good start. >> However, "Appendix C. Creating New Transports" needs modification. >> >> Following two likely needs to stay outside of the "control transport". >> >> A transport provides a mechanism for the device to send device notifications to the driver, such as used >> buffer notifications. >> A transport provides a mechanism for the driver to send driver notifications to the device, such as available >> buffer notifications. > > > Well existing transports sure do include these. > >> We also need to fix the Appendix to describe about control transport and full transport. >> Current definition of Appendix C is listing MMIO And PCI transport, yet it misses virtqueue implementation by _full_ transport. > > because they all share a single virtqueue implementation. > >>> >>> >>> >>>> >>>> And hence, data transfer part must be scoped properly. >>>> Maybe I am yet to read this text in the 1600 lines of proposed patch... >>> >>> >>> Once we get into "support everything in the most abstract way possible" >>> we have already lost. >> Unlikely. >> Nvme over TCP is present since 2021. >> NVMe over RDMA is present since 2017. >> iSCSI over TCP is present from 2014. >> NFS .. >> List continues... >> >> The idea is to not define most abstract thing. >> The idea is to define the practical spec that cater to requirements already listed, including the TCP one in [1]. > > I don't object to virtio over TCP. just do not want to block this > work on that. > > >>> No one asked for virtio over carrier pigeons. >> CSP user already explained the use case of virtio over network transport in [1]. >> Considering above many use cases already done on similar non virtio devices as pigeons is severe undermining the scope of virtio. >> >> One can say, I only want to engineer control transport ignoring needs of [1]. >> However, we must have the doors and vision open so that users of [1] can also use it without creating yet another 'fabric transport'. >> >> At the same time, we must define the binding to the bus where it is going to be used for the DMA. >> >> For example, cover letter mentions about " between a host processor and its co-processors" >> And " between normal and secure worlds" >> >> How can one implement a driver if the bus driver does not know how to enumerate/discover it? >> >> In the proposed example, of host processor and co-processor if they are connected via Xenbus or AMBA bus, >> how should driver discover this device? >> >> If this bus binding is not part of the virtio spec, how can it be ever implemented and yet comply to the virtio spec? >> Can someone please explain this? >> >> I frankly expect an ARM FF-A bus binding transport section in virtio-spec. >> So that device and driver can inter-operate implemented by two different entities. >> Only virtio level messages do not seem sufficient. > > I agree at least one should be shown at least as an rfc. Here is a link to the current virtio message bus over FF-A specification: https://developer.arm.com/documentation/den0153/0101/ 1.0 Alpha 1 version in sync with this version of the virtio message spec will be released later today, you will be redirected to 1.0 Alpha 0 version until it is which is in sync with the previous RFC we sent. Regards Bertrand IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 9:35 ` Bertrand Marquis @ 2026-02-25 9:54 ` Michael S. Tsirkin 2026-02-25 10:01 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 9:54 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 09:35:29AM +0000, Bertrand Marquis wrote: > Hi Michael and Parav, > > > On 25 Feb 2026, at 10:22, Michael S. Tsirkin <mst@redhat.com> wrote: > > > > On Wed, Feb 25, 2026 at 09:18:58AM +0000, Parav Pandit wrote: > >> > >>> From: Michael S. Tsirkin <mst@redhat.com> > >>> Sent: 25 February 2026 12:55 PM > >>> > >>> On Wed, Feb 25, 2026 at 05:09:45AM +0000, Parav Pandit wrote: > >>>> > >>>> > >>>>> From: Michael S. Tsirkin <mst@redhat.com> > >>>>> Sent: 20 February 2026 03:33 PM > >>>>> > >>>>> On Fri, Feb 20, 2026 at 06:13:55AM +0000, Parav Pandit wrote: > >>>>>>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to > >>>>> operate > >>>>>>> the virtqueues. > >>>>>>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > >>>>>>> > >>>>>>> surely this can be an optional transport feature bit. > >>>>>>> > >>>>>> How is this optional? > >>>>>> How can one implement a transport without defining the basic data transfer semantics? > >>>>>> For example for TCP transporting, driver side OS is implementing header format foo, and device side is using header format bar. > >>>>>> How does it work? > >>>>> > >>>>> I'm not sure what do foo/bar refer to, or what TCP transporting means. > >>>> This proposal is subset of past proposal of [1]. > >>>> > >>>> Proposal [1] covered transporting virtio control and data operations using something other than MMIO and PCI. > >>>> And current proposal is similar, except that it didn't define the transport binding at all for the specific bus. > >>>> It is only a partial 'control-transport'. > >>>> > >>>> [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ > >>>> > >>>> So foo and bar are the definitions I expect as listed in the patch-5 of [1]. > >>>> If it has to be done by the bus, lets write up this as 'control-transport'. > >>>> > >>>>> The simplest way to do TCP on top of virtio is to layer it above virtio > >>>>> net. That uses VQs for data transfers. > >>>>> > >>>> The intention of this proposal is not to do TCP on top of virtio. > >>>> The intention of this proposal is to do virtio on transports other than MMIO, PCI and channel. > >>>> Such a transport can be anything - not defined in virtio spec. > >>> > >>>> It could be FFA, some two SoC as written in cover letter example, or it can be something else such as TCP or UDP or vsock or whatever else. > >>> > >>> I feel this "anything" is simply too broad a requirement. > >>> I did not see any demand for virtio over TCP. > >>> And, making it work with existing drivers will be a mess. > >> Why would it be a mess? Because of load/store semantics? > >> If so, would the message layer also faces the same challenge? > > > > not sure what "the message layer" is. > > > >>> We can scope this for buses that can do DMA for now. > >> That looks reasonable good start. > >> However, "Appendix C. Creating New Transports" needs modification. > >> > >> Following two likely needs to stay outside of the "control transport". > >> > >> A transport provides a mechanism for the device to send device notifications to the driver, such as used > >> buffer notifications. > >> A transport provides a mechanism for the driver to send driver notifications to the device, such as available > >> buffer notifications. > > > > > > Well existing transports sure do include these. > > > >> We also need to fix the Appendix to describe about control transport and full transport. > >> Current definition of Appendix C is listing MMIO And PCI transport, yet it misses virtqueue implementation by _full_ transport. > > > > because they all share a single virtqueue implementation. > > > >>> > >>> > >>> > >>>> > >>>> And hence, data transfer part must be scoped properly. > >>>> Maybe I am yet to read this text in the 1600 lines of proposed patch... > >>> > >>> > >>> Once we get into "support everything in the most abstract way possible" > >>> we have already lost. > >> Unlikely. > >> Nvme over TCP is present since 2021. > >> NVMe over RDMA is present since 2017. > >> iSCSI over TCP is present from 2014. > >> NFS .. > >> List continues... > >> > >> The idea is to not define most abstract thing. > >> The idea is to define the practical spec that cater to requirements already listed, including the TCP one in [1]. > > > > I don't object to virtio over TCP. just do not want to block this > > work on that. > > > > > >>> No one asked for virtio over carrier pigeons. > >> CSP user already explained the use case of virtio over network transport in [1]. > >> Considering above many use cases already done on similar non virtio devices as pigeons is severe undermining the scope of virtio. > >> > >> One can say, I only want to engineer control transport ignoring needs of [1]. > >> However, we must have the doors and vision open so that users of [1] can also use it without creating yet another 'fabric transport'. > >> > >> At the same time, we must define the binding to the bus where it is going to be used for the DMA. > >> > >> For example, cover letter mentions about " between a host processor and its co-processors" > >> And " between normal and secure worlds" > >> > >> How can one implement a driver if the bus driver does not know how to enumerate/discover it? > >> > >> In the proposed example, of host processor and co-processor if they are connected via Xenbus or AMBA bus, > >> how should driver discover this device? > >> > >> If this bus binding is not part of the virtio spec, how can it be ever implemented and yet comply to the virtio spec? > >> Can someone please explain this? > >> > >> I frankly expect an ARM FF-A bus binding transport section in virtio-spec. > >> So that device and driver can inter-operate implemented by two different entities. > >> Only virtio level messages do not seem sufficient. > > > > I agree at least one should be shown at least as an rfc. > > Here is a link to the current virtio message bus over FF-A specification: > https://developer.arm.com/documentation/den0153/0101/ > > 1.0 Alpha 1 version in sync with this version of the virtio message spec will be released later today, you will be redirected to > 1.0 Alpha 0 version until it is which is in sync with the previous RFC we sent. > > Regards > Bertrand > > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. thanks. pls include that in the series as [PATCH RFC 5/4] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 9:54 ` Michael S. Tsirkin @ 2026-02-25 10:01 ` Bertrand Marquis 2026-02-25 10:08 ` Michael S. Tsirkin 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 10:01 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Michael, [snip] >>>> >>>> >>>> I frankly expect an ARM FF-A bus binding transport section in virtio-spec. >>>> So that device and driver can inter-operate implemented by two different entities. >>>> Only virtio level messages do not seem sufficient. >>> >>> I agree at least one should be shown at least as an rfc. >> >> Here is a link to the current virtio message bus over FF-A specification: >> https://developer.arm.com/documentation/den0153/0101/ >> >> 1.0 Alpha 1 version in sync with this version of the virtio message spec will be released later today, you will be redirected to >> 1.0 Alpha 0 version until it is which is in sync with the previous RFC we sent. >> >> Regards >> Bertrand >> >> >> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. > > > thanks. pls include that in the series as [PATCH RFC 5/4] > We are not planning to have this included in the virtio specification but maintained by arm as an independent specification. FF-A is an arm specification and as such virtio message bus over FF-A is arm specific. We can reference it as a bus specification example in the virtio specification if you want (as an external link). Regards Bertrand IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 10:01 ` Bertrand Marquis @ 2026-02-25 10:08 ` Michael S. Tsirkin 0 siblings, 0 replies; 105+ messages in thread From: Michael S. Tsirkin @ 2026-02-25 10:08 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine On Wed, Feb 25, 2026 at 10:01:45AM +0000, Bertrand Marquis wrote: > Hi Michael, > > [snip] > > >>>> > >>>> > >>>> I frankly expect an ARM FF-A bus binding transport section in virtio-spec. > >>>> So that device and driver can inter-operate implemented by two different entities. > >>>> Only virtio level messages do not seem sufficient. > >>> > >>> I agree at least one should be shown at least as an rfc. > >> > >> Here is a link to the current virtio message bus over FF-A specification: > >> https://developer.arm.com/documentation/den0153/0101/ > >> > >> 1.0 Alpha 1 version in sync with this version of the virtio message spec will be released later today, you will be redirected to > >> 1.0 Alpha 0 version until it is which is in sync with the previous RFC we sent. > >> > >> Regards > >> Bertrand > >> > >> > >> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. > > > > > > thanks. pls include that in the series as [PATCH RFC 5/4] > > > > We are not planning to have this included in the virtio specification but maintained by arm as an independent specification. > FF-A is an arm specification and as such virtio message bus over FF-A is arm specific. > > We can reference it as a bus specification example in the virtio specification if you want (as an external link). Surely, you should. But for a variety of reasons, including the fact that they should be developed in lockstep, and others, please include it in the list with "RFC do not apply" subject. > Regards > Bertrand > > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-19 23:54 ` Michael S. Tsirkin 2026-02-20 6:13 ` Parav Pandit @ 2026-02-20 8:58 ` Bertrand Marquis 1 sibling, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-20 8:58 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Michael, > On 20 Feb 2026, at 00:54, Michael S. Tsirkin <mst@redhat.com> wrote: > > On Fri, Feb 13, 2026 at 01:52:06PM +0000, Parav Pandit wrote: >> Hi Bill, >> >>> From: Bill Mills <bill.mills@linaro.org> >>> Sent: 26 January 2026 10:02 PM >>> >>> This series adds the virtio-msg transport layer. >>> >>> The individuals and organizations involved in this effort have had difficulty in >>> using the existing virtio-transports in various situations and desire to add one >>> more transport that performs its transport layer operations by sending and >>> receiving messages. >>> >>> Implementations of virtio-msg will normally be done in multiple layers: >>> * common / device level >>> * bus level >>> >>> The common / device level defines the messages exchanged between the driver >>> and a device. This common part should lead to a common driver holding most >>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>> The kernel implementation in [3] shows this separation. As with other transport >>> layers, virtio-msg should not require modifications to existing virtio device >>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>> main focus of this version of the patch series. >>> >>> The virtio-msg bus level implements the normal things a bus defines >>> (enumeration, dma operations, etc) but also implements the message send and >>> receive operations. A number of bus implementations are envisioned, >>> some of which will be reusable and general purpose. Other bus implementations >>> might be unique to a given situation, for example only used by a PCIe card >>> and its driver. >>> >>> The standard bus messages are an effort to avoid different bus implementations >>> doing the same thing in different ways for no good reason. However the >>> different environments will require different things. Instead of trying to >>> anticipate all needs and provide something very abstract, we think >>> implementation specific messages will be needed at the bus level. Over time, >>> if we see similar messages across multiple bus implementations, we will move to >>> standardize a bus level message for that. >>> >> >> I would review more, had first round of sparse review. >> Please find few comments/questions below. > > I'd like to comment that I think it makes sense to have a basic simple transport and > then add performance features on top as appropriate. > So one way to address some of these comments is to show how > they can be addressed with a feature bit down the road. Agree and i will investigate the possibility to introduce a solution (now or in the future) to use a feature bit to enable/disable out-of-order when supported as i think this is a good solution. > > >> 1. device number should be 32-bit in struct virtio_msg_header. >>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >> >> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >> Needs 32-bits. >> >> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >> But more below. >> >> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate the virtqueues. >> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > surely this can be an optional transport feature bit. GET/SET_CONFIG should cover this already. > >> Also, the queue notification message is missing at bus level. >> But I dont think queue notification via message good idea anyway. >> We rather need a more higher-level message for virtqueues. >> Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. >> Without this msg transport is incomplete. > > this too can be a basic method and upgrade to a different one > with a feature negotiation I am not quite familiar with those principles but definitely introducing new messages now or in the future is possible and will be done. Main question is what must be introduced in the first version and what needs to be investigated more to be added in the future. > > >> 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in >> >> 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request for reads. > > here too, relaxed ordering could be a feature bit. Ack. Cheers Bertrand > >> 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. >> This requires caching all the values on the driver side before sending the transport request. >> I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. >> and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. >> It may require more plumbing, but it is cleaner interface when a new transport binding is created. >> >> 8. We should have the message response header for req-resp method with status code. >> (even though they are plain MMIO writes for pci). >> As opposed to that, proposed messages are doing composite tasks, hence we need the status response. >> >> 9. virtio stack cannot handle hotplug devices today, at least Linux. >> So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. >> A graceful method is needed. >> >>> We are working on a few reusable bus implementations: >>> >>> * virtio-msg-ffa based on Arm FF-A interface for use between: >>> * normal world and secure world >>> * host and VM or VM to VM >>> * Can be used w/ or with out a hypervisor >>> * Any Hypervisor that implements FF-A can be used >>> * We have this working with pKVM and Xen >>> * We have this working with Trusty and OP-TEE >>> >>> * virtio-msg-amp for use between heterogenous systems >>> * The main processors and its co-processors on an AMP SOC >>> * Two or more systems connected via PCIe >>> * Minimal requirements: bi-directional interrupts and >>> at least one shared memory area >>> * hvac-demo has 2 demos of this >>> * This is working on two hardware platforms >>> >>> * virtio-msg-loopback for userspace implemented devices >>> * Allows user space to provide devices to its own kernel >>> * This is similar to fuse, cuse or loopback block devices but for virtio >>> * hvac-demo has a demo >>> >>> We also anticipate a few more: >>> >>> * virtio-msg-xen specific to Xen >>> * Usable on any Xen system (including x86 where FF-A does not exist) >>> * Using Xen events and page grants >>> >>> * virtio-msg over admin virtqueues >>> * This allows any virtio-pci device that supports admin virtqueues to also >>> support a virtio-msg bus that supports sub devices >>> * [We are looking for collaborators for this work] >>> >>> Changes since RFC2: >>> >>> Spec Functional: >>> * Made the common message header 8 bytes and added a token for optional use >>> when parallel outstanding requests are possible >>> * Made the 8 byte fields align to 8 byte offsets. >>> This effects the {SET,GET}_VQUEUE messages >>> * Many conformance cases have been tightened. >>> >>> Spec Editorial: >>> * Major restructure to better align with virtio spec >>> * Conformance model now matches other transports >>> * Use C structures to define messages instead of tables >>> * Added a section to describe how responses are matched to requests >>> This includes the use of the new token field >>> * Redefined / better described error handling between transport and bus >>> layers to eliminate the need for the bus to generate fake response messages >>> * Included editorial feedback from RFC2 >>> >>> Eco-system: >>> * Added virtio-msg-loopback demo >>> * Arm has published the first draft of the virtio-msg over FFA spec [6] >>> * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE >>> secure world >>> * LKML RFC has been sent [7] >>> * QEMU RFC has been sent [8] >>> >>> This is the first non-RFC patch series. The known short comings have been >>> addressed. We ask for review in earnest on this series and thank you for >>> any feedback you can provide. >>> >>> Background info and work in progress implementations: >>> * HVAC project page with intro slides [1] >>> * HVAC demo repo w/ instructions in README.md [2] >>> * Kernel w/ virtio-msg common level and ffa support [3] >>> * QEMU w/ support for one form of virtio-msg-amp [4] >>> * Portable RTOS library w/ one form of virtio-msg-amp [5] >>> >>> In addition to the QEMU system based demos in the hvac-demo repo, we also have >>> two hardware systems running: >>> * AMD x86 + AMD Arm Versal connected via PCIe >>> * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr >>> >>> Please note that although the demos work, a few have not been aligned >>> with this version of the spec. >>> >>> [1] >>> https://linaro.atlassian.net/wiki/spaces/HVAC/overview >>> 05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6390 >>> 50419790817373%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIs >>> IldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JnCO8QgAIiyGAprBLkprUd8A3162wCFpmGic3JKoHXo%3D&reserved=0 >>> [2] https://github.com/wmamills/hvac- >>> demo&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7 >>> C0%7C639050419790868944%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj >>> oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XeBuS8Hr%2FDETKfSyidKKGoGq6OTUcOYoeSGDouUMcf4%3D&reserved=0 >>> [3] >>> https://git.kernel.org/pub/scm/linux/kernel/git/vireshk%252 >>> Flinux.git%2Flog%2F%3Fh%3Dvirtio%2Fmsg&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d1 >>> 5727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419790913867%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw >>> LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PuLq68Y0ZNCYbfrgDZlipfm44xHZp1%2F%2 >>> BLu%2FaKQBBB%2Fo%3D&reserved=0 >>> [4] https://github.com/edgarigl/qemu/commits/edgar/virtio- >>> msg- >>> rfc%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0% >>> 7C0%7C639050419790960971%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFO >>> IjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xyowKD9KG3IasQTiYCzfCRGqCLSF%2Fa1g4%2F2%2ByKb1OoA%3D&reserved=0 >>> [5] https://github.com/arnopo/open-amp/commits/virtio- >>> msg%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0 >>> %7C0%7C639050419791007707%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF >>> OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=69OwO2wXi0%2FEhDDSR2uLG2iis5RYzI%2FLnlfkcRYao9o%3D&reserved=0 >>> [6] https://documentation-/ >>> service.arm.com%2Fstatic%2F68f647791134f773ab3f0a7c&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d9 >>> 5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419791052848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy >>> dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cD5VqKnTuIFqlN2XV3j%2FHQ >>> Wvsz8UVe5PM0ZjMU3o0sM%3D&reserved=0 >>> [7] >>> https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linar >>> o.org%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C >>> 0%7C0%7C639050419791093596%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI >>> kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NmhmL%2BuyVqnpHT5r9EfI2uzPf9LSXLga7bUw3FO5x4w%3D&reserved=0 >>> [8] https://mail.gnu.org/archive/html/qemu-devel/2025- >>> 10%2Fmsg07438.html&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd >>> 9ccc17a%7C0%7C0%7C639050419791139232%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJ >>> XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=osI1FttOO9%2B01e%2B5EtwnYe%2Fl1%2BIvFjH%2Blyz2ipM3RM >>> Q%3D&reserved=0 >>> >>> Bertrand Marquis (2): >>> virtio-msg: add new command for bus normative >>> virtio-msg: add conformance entries in conformance chapter >>> >>> Bill Mills (2): >>> virtio-msg: Add virtio-msg, a message based virtio transport layer >>> virtio-msg: link virtio-msg content >>> >>> commands.tex | 3 +- >>> conformance.tex | 105 ++- >>> content.tex | 1 + >>> transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ >>> 4 files changed, 1746 insertions(+), 3 deletions(-) >>> create mode 100644 transport-msg.tex >>> >>> -- >>> 2.34.1 >>> >> > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-13 13:52 ` Parav Pandit 2026-02-13 19:45 ` Demi Marie Obenour 2026-02-19 23:54 ` Michael S. Tsirkin @ 2026-02-20 8:40 ` Bertrand Marquis 2026-02-25 4:58 ` Parav Pandit 2 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-20 8:40 UTC (permalink / raw) To: Parav Pandit Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Parav, > On 13 Feb 2026, at 14:52, Parav Pandit <parav@nvidia.com> wrote: > > Hi Bill, > >> From: Bill Mills <bill.mills@linaro.org> >> Sent: 26 January 2026 10:02 PM >> >> This series adds the virtio-msg transport layer. >> >> The individuals and organizations involved in this effort have had difficulty in >> using the existing virtio-transports in various situations and desire to add one >> more transport that performs its transport layer operations by sending and >> receiving messages. >> >> Implementations of virtio-msg will normally be done in multiple layers: >> * common / device level >> * bus level >> >> The common / device level defines the messages exchanged between the driver >> and a device. This common part should lead to a common driver holding most >> of the virtio specifics and can be shared by all virtio-msg bus implementations. >> The kernel implementation in [3] shows this separation. As with other transport >> layers, virtio-msg should not require modifications to existing virtio device >> implementations (virtio-net, virtio-blk etc). The common / device level is the >> main focus of this version of the patch series. >> >> The virtio-msg bus level implements the normal things a bus defines >> (enumeration, dma operations, etc) but also implements the message send and >> receive operations. A number of bus implementations are envisioned, >> some of which will be reusable and general purpose. Other bus implementations >> might be unique to a given situation, for example only used by a PCIe card >> and its driver. >> >> The standard bus messages are an effort to avoid different bus implementations >> doing the same thing in different ways for no good reason. However the >> different environments will require different things. Instead of trying to >> anticipate all needs and provide something very abstract, we think >> implementation specific messages will be needed at the bus level. Over time, >> if we see similar messages across multiple bus implementations, we will move to >> standardize a bus level message for that. >> > > I would review more, had first round of sparse review. > Please find few comments/questions below. > > 1. device number should be 32-bit in struct virtio_msg_header. > From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > Also mapping PCI BDF wont be enough in 16-bits considering domain field. That is a very interesting feedback, we will definitely take this into account. We will have to decide how big and I will answer that in Demi's mail as there might be some drawbacks with having very big sizes for the device ID. In any case this point was raised by you, Peter and Demi and we will definitely handle it in v2. > > 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > Needs 32-bits. The point of the transport is not to transfer data, this should be done using virtqueues. Transport is only there to allow access to registers, features and configuration, the main transfers are to be done using virtqueues and i would not expect a network driver to use MMIO registers to write network packet payloads but to use virtqueues for that (as vsock or net is doing). So the msg_size here is just to fit transport specific requests. > > 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > But more below. Good point, we will check that to make this coherent with other transports. > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate the virtqueues. > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? Equivalent of virtio_mem_read/write is provided by GET/SET CONFIG which allows you to access the configuration area. So those are the equivalent of mem_read/write. Does that answer your question ? > Also, the queue notification message is missing at bus level. Queue notifications are provided by event_avail and event_used messages. > But I dont think queue notification via message good idea anyway. As said in the spec, a bus can handle this using MSI or interrupts and generate the messages in the interface between the bus and the transport. It is not an hard requirement to have those done using messages transfered and the bus can do what it wants, but it must inform the transport using a message. > We rather need a more higher-level message for virtqueues. > Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. > Without this msg transport is incomplete. I am not completely understanding what you mean here, are your concerns covered by the event messages ? > > 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in We went back and forth on the naming, msg_id could be misunderstood as message opcode. I guess we will have to find the best consensus here. > > 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request for reads. This was pointed out in an other mail and i agree here. I will investigate how we could make this a feature bit which i think could be a good solution. > > 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. Set vqueue has an index to specify the virtqueue index but you have to specific all fields in one go that is true. Do you need a solution where you could set some fields to a specific value to say "keep current" and only update part of the vqueue configuration ? > This requires caching all the values on the driver side before sending the transport request. > I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. > and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. > It may require more plumbing, but it is cleaner interface when a new transport binding is created. Admin queue useable with the message transport but I would be interested to understand exactly the model you are referring to with create/destroy model. Could you elaborate a bit so that i could understand what messages you would expect and how this would work ? > > 8. We should have the message response header for req-resp method with status code. > (even though they are plain MMIO writes for pci). > As opposed to that, proposed messages are doing composite tasks, hence we need the status response. You want an error code in the header, is that what you mean ? We went into this discussion while designing this, the main issue is that most drivers could not handle an error as this is something not possible in MMIO or PCI so we decided instead to say that in case of error a valid answer must be generated (all 0 when doing get_config for example). > > 9. virtio stack cannot handle hotplug devices today, at least Linux. > So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. > A graceful method is needed. Hotplug could be handled, remove is more problematic in Linux (you could declare a new device instance at any time and this should work). Could you give more details on what a graceful method would need ? Regards Bertrand > >> We are working on a few reusable bus implementations: >> >> * virtio-msg-ffa based on Arm FF-A interface for use between: >> * normal world and secure world >> * host and VM or VM to VM >> * Can be used w/ or with out a hypervisor >> * Any Hypervisor that implements FF-A can be used >> * We have this working with pKVM and Xen >> * We have this working with Trusty and OP-TEE >> >> * virtio-msg-amp for use between heterogenous systems >> * The main processors and its co-processors on an AMP SOC >> * Two or more systems connected via PCIe >> * Minimal requirements: bi-directional interrupts and >> at least one shared memory area >> * hvac-demo has 2 demos of this >> * This is working on two hardware platforms >> >> * virtio-msg-loopback for userspace implemented devices >> * Allows user space to provide devices to its own kernel >> * This is similar to fuse, cuse or loopback block devices but for virtio >> * hvac-demo has a demo >> >> We also anticipate a few more: >> >> * virtio-msg-xen specific to Xen >> * Usable on any Xen system (including x86 where FF-A does not exist) >> * Using Xen events and page grants >> >> * virtio-msg over admin virtqueues >> * This allows any virtio-pci device that supports admin virtqueues to also >> support a virtio-msg bus that supports sub devices >> * [We are looking for collaborators for this work] >> >> Changes since RFC2: >> >> Spec Functional: >> * Made the common message header 8 bytes and added a token for optional use >> when parallel outstanding requests are possible >> * Made the 8 byte fields align to 8 byte offsets. >> This effects the {SET,GET}_VQUEUE messages >> * Many conformance cases have been tightened. >> >> Spec Editorial: >> * Major restructure to better align with virtio spec >> * Conformance model now matches other transports >> * Use C structures to define messages instead of tables >> * Added a section to describe how responses are matched to requests >> This includes the use of the new token field >> * Redefined / better described error handling between transport and bus >> layers to eliminate the need for the bus to generate fake response messages >> * Included editorial feedback from RFC2 >> >> Eco-system: >> * Added virtio-msg-loopback demo >> * Arm has published the first draft of the virtio-msg over FFA spec [6] >> * virtio-msg over FFA has been demonstrated with both Trusty and OP-TEE >> secure world >> * LKML RFC has been sent [7] >> * QEMU RFC has been sent [8] >> >> This is the first non-RFC patch series. The known short comings have been >> addressed. We ask for review in earnest on this series and thank you for >> any feedback you can provide. >> >> Background info and work in progress implementations: >> * HVAC project page with intro slides [1] >> * HVAC demo repo w/ instructions in README.md [2] >> * Kernel w/ virtio-msg common level and ffa support [3] >> * QEMU w/ support for one form of virtio-msg-amp [4] >> * Portable RTOS library w/ one form of virtio-msg-amp [5] >> >> In addition to the QEMU system based demos in the hvac-demo repo, we also have >> two hardware systems running: >> * AMD x86 + AMD Arm Versal connected via PCIe >> * ST STM32MP157 A7 Linux using virtio-i2c provided by M4 Zephyr >> >> Please note that although the demos work, a few have not been aligned >> with this version of the spec. >> >> [1] >> https://linaro.atlassian.net/wiki/spaces/HVAC/overview >> 05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C6390 >> 50419790817373%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIs >> IldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=JnCO8QgAIiyGAprBLkprUd8A3162wCFpmGic3JKoHXo%3D&reserved=0 >> [2] https://github.com/wmamills/hvac- >> demo&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0%7 >> C0%7C639050419790868944%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIj >> oiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=XeBuS8Hr%2FDETKfSyidKKGoGq6OTUcOYoeSGDouUMcf4%3D&reserved=0 >> [3] >> https://git.kernel.org/pub/scm/linux/kernel/git/vireshk%252 >> Flinux.git%2Flog%2F%3Fh%3Dvirtio%2Fmsg&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d1 >> 5727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419790913867%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIw >> LjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=PuLq68Y0ZNCYbfrgDZlipfm44xHZp1%2F%2 >> BLu%2FaKQBBB%2Fo%3D&reserved=0 >> [4] https://github.com/edgarigl/qemu/commits/edgar/virtio- >> msg- >> rfc%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0% >> 7C0%7C639050419790960971%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFO >> IjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xyowKD9KG3IasQTiYCzfCRGqCLSF%2Fa1g4%2F2%2ByKb1OoA%3D&reserved=0 >> [5] https://github.com/arnopo/open-amp/commits/virtio- >> msg%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C0 >> %7C0%7C639050419791007707%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkF >> OIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=69OwO2wXi0%2FEhDDSR2uLG2iis5RYzI%2FLnlfkcRYao9o%3D&reserved=0 >> [6] https://documentation-/ >> service.arm.com%2Fstatic%2F68f647791134f773ab3f0a7c&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d9 >> 5%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C639050419791052848%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRy >> dWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=cD5VqKnTuIFqlN2XV3j%2FHQ >> Wvsz8UVe5PM0ZjMU3o0sM%3D&reserved=0 >> [7] >> https://lore.kernel.org/all/cover.1753865268.git.viresh.kumar@linar >> o.org%2F&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd9ccc17a%7C >> 0%7C0%7C639050419791093596%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsI >> kFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=NmhmL%2BuyVqnpHT5r9EfI2uzPf9LSXLga7bUw3FO5x4w%3D&reserved=0 >> [8] https://mail.gnu.org/archive/html/qemu-devel/2025- >> 10%2Fmsg07438.html&data=05%7C02%7Cparav%40nvidia.com%7Cf7f430294c37485df25808de5cf88d95%7C43083d15727340c1b7db39efd >> 9ccc17a%7C0%7C0%7C639050419791139232%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJ >> XaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=osI1FttOO9%2B01e%2B5EtwnYe%2Fl1%2BIvFjH%2Blyz2ipM3RM >> Q%3D&reserved=0 >> >> Bertrand Marquis (2): >> virtio-msg: add new command for bus normative >> virtio-msg: add conformance entries in conformance chapter >> >> Bill Mills (2): >> virtio-msg: Add virtio-msg, a message based virtio transport layer >> virtio-msg: link virtio-msg content >> >> commands.tex | 3 +- >> conformance.tex | 105 ++- >> content.tex | 1 + >> transport-msg.tex | 1640 +++++++++++++++++++++++++++++++++++++++++++++ >> 4 files changed, 1746 insertions(+), 3 deletions(-) >> create mode 100644 transport-msg.tex >> >> -- >> 2.34.1 >> > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* RE: [PATCH v1 0/4] virtio-msg transport layer 2026-02-20 8:40 ` Bertrand Marquis @ 2026-02-25 4:58 ` Parav Pandit 2026-02-25 7:52 ` Bertrand Marquis 0 siblings, 1 reply; 105+ messages in thread From: Parav Pandit @ 2026-02-25 4:58 UTC (permalink / raw) To: Bertrand Marquis Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine > From: Bertrand Marquis <Bertrand.Marquis@arm.com> > Sent: 20 February 2026 02:10 PM > > Hi Parav, > > > On 13 Feb 2026, at 14:52, Parav Pandit <parav@nvidia.com> wrote: > > > > Hi Bill, > > > >> From: Bill Mills <bill.mills@linaro.org> > >> Sent: 26 January 2026 10:02 PM > >> > >> This series adds the virtio-msg transport layer. > >> > >> The individuals and organizations involved in this effort have had difficulty in > >> using the existing virtio-transports in various situations and desire to add one > >> more transport that performs its transport layer operations by sending and > >> receiving messages. > >> > >> Implementations of virtio-msg will normally be done in multiple layers: > >> * common / device level > >> * bus level > >> > >> The common / device level defines the messages exchanged between the driver > >> and a device. This common part should lead to a common driver holding most > >> of the virtio specifics and can be shared by all virtio-msg bus implementations. > >> The kernel implementation in [3] shows this separation. As with other transport > >> layers, virtio-msg should not require modifications to existing virtio device > >> implementations (virtio-net, virtio-blk etc). The common / device level is the > >> main focus of this version of the patch series. > >> > >> The virtio-msg bus level implements the normal things a bus defines > >> (enumeration, dma operations, etc) but also implements the message send and > >> receive operations. A number of bus implementations are envisioned, > >> some of which will be reusable and general purpose. Other bus implementations > >> might be unique to a given situation, for example only used by a PCIe card > >> and its driver. > >> > >> The standard bus messages are an effort to avoid different bus implementations > >> doing the same thing in different ways for no good reason. However the > >> different environments will require different things. Instead of trying to > >> anticipate all needs and provide something very abstract, we think > >> implementation specific messages will be needed at the bus level. Over time, > >> if we see similar messages across multiple bus implementations, we will move to > >> standardize a bus level message for that. > >> > > > > I would review more, had first round of sparse review. > > Please find few comments/questions below. > > > > 1. device number should be 32-bit in struct virtio_msg_header. > > From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. > > Also mapping PCI BDF wont be enough in 16-bits considering domain field. > > That is a very interesting feedback, we will definitely take this into account. > We will have to decide how big and I will answer that in Demi's mail as there might be some drawbacks with > having very big sizes for the device ID. > I am slowly catching up on the thread. There are two types of device id needed. One is UUID style to uniquely identify the device that may show up using two transports to the driver. With that a driver can create single virtio_device object, which is reachable via two different transports. This offers performance, resiliency. This is likely a bigger string which is not efficient to use during every message transaction. Second one is: within a transport, a device id to identify the communication. I was referring to this device id to be u32, so that transport can support more than 64K devices. > In any case this point was raised by you, Peter and Demi and we will definitely handle it in v2. > Ok. > > > > 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. > > For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. > > Needs 32-bits. > > The point of the transport is not to transfer data, this should be done using virtqueues. How comes virtqueues live outside of the transport? I don't understand this at all. Transport is supposed to transport the descriptors and data also _of_ the virtqueue. For example you can see that proposal [1] is nearly complete that enables virtio msg layer over various transports. This includes control operations and data operations both. Patch-4 defines the control operations (similar to this proposal) Patch-5 defines the binding of data operation with the specific underlying transports such as tcp, rdma etc. In your case, patch-5 would be for system specific bus such as FF-A or others that others have acked for. But it must be documented somewhere. [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ > Transport is only there to allow access to registers, features and configuration, the main transfers are to > be done using virtqueues and i would not expect a network driver to use MMIO registers to write network > packet payloads but to use virtqueues for that (as vsock or net is doing). > Very strange. If that is your intention, this transport thing must be renamed to control-transport to succinctly indicate its objective is for 'control operation'. > So the msg_size here is just to fit transport specific requests. > > > > > 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) > > But more below. > > Good point, we will check that to make this coherent with other transports. > Ok. > > > > 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate > the virtqueues. > > Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? > > Equivalent of virtio_mem_read/write is provided by GET/SET CONFIG which allows you to access the configuration area. > So those are the equivalent of mem_read/write. > Does that answer your question ? > No. but I understood that what is proposed here is a partial proposal to have transport only for control operation. And data transfer implementation is undefined. > > Also, the queue notification message is missing at bus level. > > Queue notifications are provided by event_avail and event_used messages. > This does not make sense either. The operations that related towards the virtqueues should be part of the virtqueue operations. By not defining specific bus specific details, we don't seem to gain anything significant. So while transport message are good, it needs to cover virtqueue data exchanges too as previously proposed in [1]. > > But I dont think queue notification via message good idea anyway. > > As said in the spec, a bus can handle this using MSI or interrupts and generate the messages in the interface > between the bus and the transport. It is not an hard requirement to have those done using messages transfered and > the bus can do what it wants, but it must inform the transport using a message. > > > We rather need a more higher-level message for virtqueues. > > Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. > > Without this msg transport is incomplete. > > I am not completely understanding what you mean here, are your concerns covered by the event messages ? > We need transport messages agnostic of the transport like patch-4 in [1]. And transport binding defined for each bus of fabric like [2]. So that message interface can support [1] as well by only extending the virtqueue bindings. > > > > 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in > > We went back and forth on the naming, msg_id could be misunderstood as message opcode. > I guess we will have to find the best consensus here. > Ok. > > > > 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request > for reads. > > This was pointed out in an other mail and i agree here. > I will investigate how we could make this a feature bit which i think could be a good solution. > Ok. > > > > 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. > > Set vqueue has an index to specify the virtqueue index but you have to specific all fields in one go that is true. > Do you need a solution where you could set some fields to a specific value to say "keep current" and only update part of the vqueue > configuration ? > I believe so, otherwise it cannot work with existing drivers without driver side caching them. > > This requires caching all the values on the driver side before sending the transport request. > > I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. > > and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. > > It may require more plumbing, but it is cleaner interface when a new transport binding is created. > > Admin queue useable with the message transport but I would be interested to understand exactly > the model you are referring to with create/destroy model. > Could you elaborate a bit so that i could understand what messages you would expect and how this would work ? > The suggestion is to not use SET_VQUEUE legacy. The suggestion is to use admin virtqueue to create queues and destroy queues, as they are merely an object. And new transport like your proposal can adapt to the modern style. So only admin queue configuration would be the only message. Rest of the other queues can be created directly using the admin queue. One can say it is orthogonal feature and I agree with that reasoning. The part that bothers me is that the new transport ends up adding some legacy bits like set vqueue. > > > > 8. We should have the message response header for req-resp method with status code. > > (even though they are plain MMIO writes for pci). > > As opposed to that, proposed messages are doing composite tasks, hence we need the status response. > > You want an error code in the header, is that what you mean ? Yes, in the response header. > We went into this discussion while designing this, the main issue is that most drivers could not handle an > error as this is something not possible in MMIO or PCI so we decided instead to say that in case of error > a valid answer must be generated (all 0 when doing get_config for example). > Well, what is proposed here is a generic transport for future to come. So even if driver does not handle error for MMIO/PCI register writes, transport would have commands for future to come. So creating V1 structure in future just for the status wouldn't be useful. For LOAD/STORE type of messages, driver can always ignore the error. The messages are used for bus discovery too which is outside of existing drivers scope. So status is useful there too. > > > > 9. virtio stack cannot handle hotplug devices today, at least Linux. > > So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. > > A graceful method is needed. > > Hotplug could be handled, remove is more problematic in Linux (you could declare a new device instance > at any time and this should work). > > Could you give more details on what a graceful method would need ? > Graceful would need a flow something like below. 1. Remove device request (sent by device side bus) Device continue to operate while the request is serviced 2. Driver acts on the request and unloads the driver/reset the device etc. 3. Driver (and device) eventually stops using the device by stopping all the bus activities. 4. Bus driver notifies to the device, it is safe to remove and remove request is completed. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 4:58 ` Parav Pandit @ 2026-02-25 7:52 ` Bertrand Marquis 2026-02-25 12:46 ` Demi Marie Obenour 2026-02-25 15:17 ` Bertrand Marquis 0 siblings, 2 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 7:52 UTC (permalink / raw) To: Parav Pandit Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Parav, > On 25 Feb 2026, at 05:58, Parav Pandit <parav@nvidia.com> wrote: > > >> From: Bertrand Marquis <Bertrand.Marquis@arm.com> >> Sent: 20 February 2026 02:10 PM >> >> Hi Parav, >> >>> On 13 Feb 2026, at 14:52, Parav Pandit <parav@nvidia.com> wrote: >>> >>> Hi Bill, >>> >>>> From: Bill Mills <bill.mills@linaro.org> >>>> Sent: 26 January 2026 10:02 PM >>>> >>>> This series adds the virtio-msg transport layer. >>>> >>>> The individuals and organizations involved in this effort have had difficulty in >>>> using the existing virtio-transports in various situations and desire to add one >>>> more transport that performs its transport layer operations by sending and >>>> receiving messages. >>>> >>>> Implementations of virtio-msg will normally be done in multiple layers: >>>> * common / device level >>>> * bus level >>>> >>>> The common / device level defines the messages exchanged between the driver >>>> and a device. This common part should lead to a common driver holding most >>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>> The kernel implementation in [3] shows this separation. As with other transport >>>> layers, virtio-msg should not require modifications to existing virtio device >>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>> main focus of this version of the patch series. >>>> >>>> The virtio-msg bus level implements the normal things a bus defines >>>> (enumeration, dma operations, etc) but also implements the message send and >>>> receive operations. A number of bus implementations are envisioned, >>>> some of which will be reusable and general purpose. Other bus implementations >>>> might be unique to a given situation, for example only used by a PCIe card >>>> and its driver. >>>> >>>> The standard bus messages are an effort to avoid different bus implementations >>>> doing the same thing in different ways for no good reason. However the >>>> different environments will require different things. Instead of trying to >>>> anticipate all needs and provide something very abstract, we think >>>> implementation specific messages will be needed at the bus level. Over time, >>>> if we see similar messages across multiple bus implementations, we will move to >>>> standardize a bus level message for that. >>>> >>> >>> I would review more, had first round of sparse review. >>> Please find few comments/questions below. >>> >>> 1. device number should be 32-bit in struct virtio_msg_header. >>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >> >> That is a very interesting feedback, we will definitely take this into account. >> We will have to decide how big and I will answer that in Demi's mail as there might be some drawbacks with >> having very big sizes for the device ID. >> > I am slowly catching up on the thread. > There are two types of device id needed. > One is UUID style to uniquely identify the device that may show up using two transports to the driver. > With that a driver can create single virtio_device object, which is reachable via two different transports. > This offers performance, resiliency. > This is likely a bigger string which is not efficient to use during every message transaction. > > Second one is: within a transport, a device id to identify the communication. > I was referring to this device id to be u32, so that transport can support more than 64K devices. We will increase the device number size to support more than 64k devices. For the UUID part, I feel it should be something provided as device information per device, so we could add a non-mandatory field (nil-UUID when no ID available) and transfer that information as part of GET_DEVICE_INFO. > > > >> In any case this point was raised by you, Peter and Demi and we will definitely handle it in v2. >> > Ok. > >>> >>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>> Needs 32-bits. >> >> The point of the transport is not to transfer data, this should be done using virtqueues. > How comes virtqueues live outside of the transport? > I don't understand this at all. > Transport is supposed to transport the descriptors and data also _of_ the virtqueue. virtqueues do not really live outside of the transport but are defined and used in the same way as they are in PCI or MMIO. The transport provide ways to configure the virtqueues but relies on the fact that virtqueue content is available to both sides through DMA, shared memory or any other custom mean. > > For example you can see that proposal [1] is nearly complete that enables virtio msg layer over various transports. > This includes control operations and data operations both. > Patch-4 defines the control operations (similar to this proposal) > Patch-5 defines the binding of data operation with the specific underlying transports such as tcp, rdma etc. > > In your case, patch-5 would be for system specific bus such as FF-A or others that others have acked for. > But it must be documented somewhere. Correct me if I'm wrong but this is centered on transfering virqueue content over messages and having something that is network friendly. Even though some of the principles could be reused, I think the use case is a bit different. In our use cases we have remote memory access but what we cannot afford is trap-and-emulate to access registers (MMIO or PCI) or a hardware specific solution (CCW) (for scheduling issues and performance reasons). Now what could be done easily is extend the virtio-msg proposal with a solution to transfer virtqueues over messages to and answer to this use case. > > > [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ > >> Transport is only there to allow access to registers, features and configuration, the main transfers are to >> be done using virtqueues and i would not expect a network driver to use MMIO registers to write network >> packet payloads but to use virtqueues for that (as vsock or net is doing). >> > Very strange. > If that is your intention, this transport thing must be renamed to control-transport to succinctly indicate its objective is for 'control operation'. The proposal is following what PCI, MMIO and CCW transports are defining as they do not mention how virtqueues data is transmitted. > >> So the msg_size here is just to fit transport specific requests. >> >>> >>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>> But more below. >> >> Good point, we will check that to make this coherent with other transports. >> > Ok. >>> >>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate >> the virtqueues. >>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >> >> Equivalent of virtio_mem_read/write is provided by GET/SET CONFIG which allows you to access the configuration area. >> So those are the equivalent of mem_read/write. >> Does that answer your question ? >> > No. but I understood that what is proposed here is a partial proposal to have transport only for control operation. > And data transfer implementation is undefined. But configuration register read/write which is not going through virtqueues is still covered. Can you confirm and ack we do not need any new message added for mem_read/write ? > >>> Also, the queue notification message is missing at bus level. >> >> Queue notifications are provided by event_avail and event_used messages. >> > This does not make sense either. > The operations that related towards the virtqueues should be part of the virtqueue operations. > > By not defining specific bus specific details, we don't seem to gain anything significant. > So while transport message are good, it needs to cover virtqueue data exchanges too as previously proposed in [1]. Please see my previous answer and discussions with Demi for notifications handling. > >>> But I dont think queue notification via message good idea anyway. >> >> As said in the spec, a bus can handle this using MSI or interrupts and generate the messages in the interface >> between the bus and the transport. It is not an hard requirement to have those done using messages transfered and >> the bus can do what it wants, but it must inform the transport using a message. >> >>> We rather need a more higher-level message for virtqueues. >>> Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. >>> Without this msg transport is incomplete. >> >> I am not completely understanding what you mean here, are your concerns covered by the event messages ? >> > We need transport messages agnostic of the transport like patch-4 in [1]. > And transport binding defined for each bus of fabric like [2]. > So that message interface can support [1] as well by only extending the virtqueue bindings. I do think that we could easily support your use case of transfering the messages over network. What is not covered by our proposal is definitely virtqueue data transfer but this could be easily done as part of a bus implementation without impacting the transport itself. > > >>> >>> 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in >> >> We went back and forth on the naming, msg_id could be misunderstood as message opcode. >> I guess we will have to find the best consensus here. >> > Ok. > >>> >>> 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request >> for reads. >> >> This was pointed out in an other mail and i agree here. >> I will investigate how we could make this a feature bit which i think could be a good solution. >> > Ok. >>> >>> 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. >> >> Set vqueue has an index to specify the virtqueue index but you have to specific all fields in one go that is true. >> Do you need a solution where you could set some fields to a specific value to say "keep current" and only update part of the vqueue >> configuration ? >> > I believe so, otherwise it cannot work with existing drivers without driver side caching them. I will investigate that possibility. > >>> This requires caching all the values on the driver side before sending the transport request. >>> I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. >>> and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. >>> It may require more plumbing, but it is cleaner interface when a new transport binding is created. >> >> Admin queue useable with the message transport but I would be interested to understand exactly >> the model you are referring to with create/destroy model. >> Could you elaborate a bit so that i could understand what messages you would expect and how this would work ? >> > The suggestion is to not use SET_VQUEUE legacy. > The suggestion is to use admin virtqueue to create queues and destroy queues, as they are merely an object. > And new transport like your proposal can adapt to the modern style. > So only admin queue configuration would be the only message. > Rest of the other queues can be created directly using the admin queue. In a way, set vqueue message could be seen like that as it is a one time operation. In most cases, drivers are configuring a virtqueue in one go which is optimized here as we need only one message transfer. If we include the proposal to also have an enable/disable directly in the message this could allow for even less messages. Using the admin solution on top of virtio message is something possible and not prevented by this solution. In my mind we have an object here as you create/destroy queues in one go and the fact that a bus can transfer those requests asynchronously gives a solution equivalent to what would be provided by admin vqueues. > > One can say it is orthogonal feature and I agree with that reasoning. > The part that bothers me is that the new transport ends up adding some legacy bits like set vqueue. We still need to support legacy to have existing implementation working and here we can optimize them a bit by not transferring one message per field. > >>> >>> 8. We should have the message response header for req-resp method with status code. >>> (even though they are plain MMIO writes for pci). >>> As opposed to that, proposed messages are doing composite tasks, hence we need the status response. >> >> You want an error code in the header, is that what you mean ? > Yes, in the response header. > >> We went into this discussion while designing this, the main issue is that most drivers could not handle an >> error as this is something not possible in MMIO or PCI so we decided instead to say that in case of error >> a valid answer must be generated (all 0 when doing get_config for example). >> > Well, what is proposed here is a generic transport for future to come. > So even if driver does not handle error for MMIO/PCI register writes, transport would have commands for future to come. > So creating V1 structure in future just for the status wouldn't be useful. We do handle that but the error transport is handled by the bus and is returned as an error code to the transport (msg send returning an error number) instead of having an error field in the message. We felt this design was making things simpler and more compliant to existing designs. > > For LOAD/STORE type of messages, driver can always ignore the error. > The messages are used for bus discovery too which is outside of existing drivers scope. > So status is useful there too. I will keep that solution in mind to replace the current one but please consider what I said upper and confirm. > > >>> >>> 9. virtio stack cannot handle hotplug devices today, at least Linux. >>> So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. >>> A graceful method is needed. >> >> Hotplug could be handled, remove is more problematic in Linux (you could declare a new device instance >> at any time and this should work). >> >> Could you give more details on what a graceful method would need ? >> > Graceful would need a flow something like below. > 1. Remove device request (sent by device side bus) > Device continue to operate while the request is serviced > > 2. Driver acts on the request and unloads the driver/reset the device etc. > > 3. Driver (and device) eventually stops using the device by stopping all the bus activities. > > 4. Bus driver notifies to the device, it is safe to remove and remove request is completed. This is a transport side handling possible depending on what the operating system provides as solutions to handle things this way. Here we have a generic message saying "this device is out" which leaves space to handle this gracefully on top and this can happen in lots of cases where the device is offline without it doing it knowingly (crash, power down, you name it). In a graceful scenario i would expect the driver to ask the device to power down through some device specific means which would result in the DEVICE_BUS_STATE_REMOVED messages being sent once it has turn itself off ending in your scenario. Do you agree or did i miss something ? Cheers Bertrand > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 7:52 ` Bertrand Marquis @ 2026-02-25 12:46 ` Demi Marie Obenour 2026-02-25 13:05 ` Bertrand Marquis 2026-02-25 15:17 ` Bertrand Marquis 1 sibling, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 12:46 UTC (permalink / raw) To: Bertrand Marquis, Parav Pandit Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 7208 bytes --] On 2/25/26 02:52, Bertrand Marquis wrote: > Hi Parav, > >> On 25 Feb 2026, at 05:58, Parav Pandit <parav@nvidia.com> wrote: >> >> >>> From: Bertrand Marquis <Bertrand.Marquis@arm.com> >>> Sent: 20 February 2026 02:10 PM >>> >>> Hi Parav, >>> >>>> On 13 Feb 2026, at 14:52, Parav Pandit <parav@nvidia.com> wrote: >>>> >>>> Hi Bill, >>>> >>>>> From: Bill Mills <bill.mills@linaro.org> >>>>> Sent: 26 January 2026 10:02 PM >>>>> >>>>> This series adds the virtio-msg transport layer. >>>>> >>>>> The individuals and organizations involved in this effort have had difficulty in >>>>> using the existing virtio-transports in various situations and desire to add one >>>>> more transport that performs its transport layer operations by sending and >>>>> receiving messages. >>>>> >>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>> * common / device level >>>>> * bus level >>>>> >>>>> The common / device level defines the messages exchanged between the driver >>>>> and a device. This common part should lead to a common driver holding most >>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>> main focus of this version of the patch series. >>>>> >>>>> The virtio-msg bus level implements the normal things a bus defines >>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>> receive operations. A number of bus implementations are envisioned, >>>>> some of which will be reusable and general purpose. Other bus implementations >>>>> might be unique to a given situation, for example only used by a PCIe card >>>>> and its driver. >>>>> >>>>> The standard bus messages are an effort to avoid different bus implementations >>>>> doing the same thing in different ways for no good reason. However the >>>>> different environments will require different things. Instead of trying to >>>>> anticipate all needs and provide something very abstract, we think >>>>> implementation specific messages will be needed at the bus level. Over time, >>>>> if we see similar messages across multiple bus implementations, we will move to >>>>> standardize a bus level message for that. >>>>> >>>> >>>> I would review more, had first round of sparse review. >>>> Please find few comments/questions below. >>>> >>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>> >>> That is a very interesting feedback, we will definitely take this into account. >>> We will have to decide how big and I will answer that in Demi's mail as there might be some drawbacks with >>> having very big sizes for the device ID. >>> >> I am slowly catching up on the thread. >> There are two types of device id needed. >> One is UUID style to uniquely identify the device that may show up using two transports to the driver. >> With that a driver can create single virtio_device object, which is reachable via two different transports. >> This offers performance, resiliency. >> This is likely a bigger string which is not efficient to use during every message transaction. >> >> Second one is: within a transport, a device id to identify the communication. >> I was referring to this device id to be u32, so that transport can support more than 64K devices. > > We will increase the device number size to support more than 64k devices. > For the UUID part, I feel it should be something provided as device information per device, so > we could add a non-mandatory field (nil-UUID when no ID available) and transfer that information > as part of GET_DEVICE_INFO. When would it not be available? >>>> 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. >>> >>> Set vqueue has an index to specify the virtqueue index but you have to specific all fields in one go that is true. >>> Do you need a solution where you could set some fields to a specific value to say "keep current" and only update part of the vqueue >>> configuration ? >>> >> I believe so, otherwise it cannot work with existing drivers without driver side caching them. > > I will investigate that possibility. With the current spec, adding virtqueues one at a time requires time quadratic in the number of virtqueues. That's not good. >>>> This requires caching all the values on the driver side before sending the transport request. >>>> I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. >>>> and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. >>>> It may require more plumbing, but it is cleaner interface when a new transport binding is created. >>> >>> Admin queue useable with the message transport but I would be interested to understand exactly >>> the model you are referring to with create/destroy model. >>> Could you elaborate a bit so that i could understand what messages you would expect and how this would work ? >>> >> The suggestion is to not use SET_VQUEUE legacy. >> The suggestion is to use admin virtqueue to create queues and destroy queues, as they are merely an object. >> And new transport like your proposal can adapt to the modern style. >> So only admin queue configuration would be the only message. >> Rest of the other queues can be created directly using the admin queue. > > In a way, set vqueue message could be seen like that as it is a one time operation. > In most cases, drivers are configuring a virtqueue in one go which is optimized here > as we need only one message transfer. > If we include the proposal to also have an enable/disable directly in the message this > could allow for even less messages. > > Using the admin solution on top of virtio message is something possible and not prevented > by this solution. > > In my mind we have an object here as you create/destroy queues in one go and the fact > that a bus can transfer those requests asynchronously gives a solution equivalent to what > would be provided by admin vqueues. Why disadvantages does using admin vqueues have? >> One can say it is orthogonal feature and I agree with that reasoning. >> The part that bothers me is that the new transport ends up adding some legacy bits like set vqueue. > > We still need to support legacy to have existing implementation working and here we can optimize > them a bit by not transferring one message per field. Which existing implementations are you referring to? It appears that these are mostly in embedded systems. If they are shipped as a single unit, the drivers and devices could be updated at the same time and there is no need for backwards compatibility. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 12:46 ` Demi Marie Obenour @ 2026-02-25 13:05 ` Bertrand Marquis 2026-02-25 13:09 ` Demi Marie Obenour 0 siblings, 1 reply; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 13:05 UTC (permalink / raw) To: Demi Marie Obenour Cc: Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Demi, > On 25 Feb 2026, at 13:46, Demi Marie Obenour <demiobenour@gmail.com> wrote: > > On 2/25/26 02:52, Bertrand Marquis wrote: >> Hi Parav, >> >>> On 25 Feb 2026, at 05:58, Parav Pandit <parav@nvidia.com> wrote: >>> >>> >>>> From: Bertrand Marquis <Bertrand.Marquis@arm.com> >>>> Sent: 20 February 2026 02:10 PM >>>> >>>> Hi Parav, >>>> >>>>> On 13 Feb 2026, at 14:52, Parav Pandit <parav@nvidia.com> wrote: >>>>> >>>>> Hi Bill, >>>>> >>>>>> From: Bill Mills <bill.mills@linaro.org> >>>>>> Sent: 26 January 2026 10:02 PM >>>>>> >>>>>> This series adds the virtio-msg transport layer. >>>>>> >>>>>> The individuals and organizations involved in this effort have had difficulty in >>>>>> using the existing virtio-transports in various situations and desire to add one >>>>>> more transport that performs its transport layer operations by sending and >>>>>> receiving messages. >>>>>> >>>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>>> * common / device level >>>>>> * bus level >>>>>> >>>>>> The common / device level defines the messages exchanged between the driver >>>>>> and a device. This common part should lead to a common driver holding most >>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>>> main focus of this version of the patch series. >>>>>> >>>>>> The virtio-msg bus level implements the normal things a bus defines >>>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>>> receive operations. A number of bus implementations are envisioned, >>>>>> some of which will be reusable and general purpose. Other bus implementations >>>>>> might be unique to a given situation, for example only used by a PCIe card >>>>>> and its driver. >>>>>> >>>>>> The standard bus messages are an effort to avoid different bus implementations >>>>>> doing the same thing in different ways for no good reason. However the >>>>>> different environments will require different things. Instead of trying to >>>>>> anticipate all needs and provide something very abstract, we think >>>>>> implementation specific messages will be needed at the bus level. Over time, >>>>>> if we see similar messages across multiple bus implementations, we will move to >>>>>> standardize a bus level message for that. >>>>>> >>>>> >>>>> I would review more, had first round of sparse review. >>>>> Please find few comments/questions below. >>>>> >>>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>>> >>>> That is a very interesting feedback, we will definitely take this into account. >>>> We will have to decide how big and I will answer that in Demi's mail as there might be some drawbacks with >>>> having very big sizes for the device ID. >>>> >>> I am slowly catching up on the thread. >>> There are two types of device id needed. >>> One is UUID style to uniquely identify the device that may show up using two transports to the driver. >>> With that a driver can create single virtio_device object, which is reachable via two different transports. >>> This offers performance, resiliency. >>> This is likely a bigger string which is not efficient to use during every message transaction. >>> >>> Second one is: within a transport, a device id to identify the communication. >>> I was referring to this device id to be u32, so that transport can support more than 64K devices. >> >> We will increase the device number size to support more than 64k devices. >> For the UUID part, I feel it should be something provided as device information per device, so >> we could add a non-mandatory field (nil-UUID when no ID available) and transfer that information >> as part of GET_DEVICE_INFO. > > When would it not be available? I am not aware of implementations assigned UUIDs per device. In any case the field can always be there and the nil-UUID used if something does not have one. > >>>>> 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. >>>> >>>> Set vqueue has an index to specify the virtqueue index but you have to specific all fields in one go that is true. >>>> Do you need a solution where you could set some fields to a specific value to say "keep current" and only update part of the vqueue >>>> configuration ? >>>> >>> I believe so, otherwise it cannot work with existing drivers without driver side caching them. >> >> I will investigate that possibility. > > With the current spec, adding virtqueues one at a time requires time > quadratic in the number of virtqueues. That's not good. Solution i am looking at would still allow to transmit everything in one go but would leave it open to only update/modify some specific fields (using an optional field mask to not modify some parts). > >>>>> This requires caching all the values on the driver side before sending the transport request. >>>>> I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. >>>>> and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. >>>>> It may require more plumbing, but it is cleaner interface when a new transport binding is created. >>>> >>>> Admin queue useable with the message transport but I would be interested to understand exactly >>>> the model you are referring to with create/destroy model. >>>> Could you elaborate a bit so that i could understand what messages you would expect and how this would work ? >>>> >>> The suggestion is to not use SET_VQUEUE legacy. >>> The suggestion is to use admin virtqueue to create queues and destroy queues, as they are merely an object. >>> And new transport like your proposal can adapt to the modern style. >>> So only admin queue configuration would be the only message. >>> Rest of the other queues can be created directly using the admin queue. >> >> In a way, set vqueue message could be seen like that as it is a one time operation. >> In most cases, drivers are configuring a virtqueue in one go which is optimized here >> as we need only one message transfer. >> If we include the proposal to also have an enable/disable directly in the message this >> could allow for even less messages. >> >> Using the admin solution on top of virtio message is something possible and not prevented >> by this solution. >> >> In my mind we have an object here as you create/destroy queues in one go and the fact >> that a bus can transfer those requests asynchronously gives a solution equivalent to what >> would be provided by admin vqueues. > > Why disadvantages does using admin vqueues have? You need a virtqueue to use admin virtqueues so we need to have a solution to set this up. Once there i agree you can use that instead but it would not change the fact that we need a solution to setup a virtqueue. Right now nothing is forbidding to use them, but we still need to have a way to configure vqueues in the transport. > >>> One can say it is orthogonal feature and I agree with that reasoning. >>> The part that bothers me is that the new transport ends up adding some legacy bits like set vqueue. >> >> We still need to support legacy to have existing implementation working and here we can optimize >> them a bit by not transferring one message per field. > > Which existing implementations are you referring to? It appears > that these are mostly in embedded systems. If they are shipped as > a single unit, the drivers and devices could be updated at the same > time and there is no need for backwards compatibility. I am a bit lost here, do you mean we should or not have a solution to update individual fields ? In any case the change i suggest would not remove the solution (preferred) to do everything in one go (plus the possibiity to enable/disable the queue in the same message) but would also allow ot update independent fields if wanted. Cheers Bertrand > -- > Sincerely, > Demi Marie Obenour (she/her/hers)<OpenPGP_0xB288B55FFF9C22C1.asc> IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 13:05 ` Bertrand Marquis @ 2026-02-25 13:09 ` Demi Marie Obenour 0 siblings, 0 replies; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 13:09 UTC (permalink / raw) To: Bertrand Marquis Cc: Parav Pandit, Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 8979 bytes --] On 2/25/26 08:05, Bertrand Marquis wrote: > Hi Demi, > >> On 25 Feb 2026, at 13:46, Demi Marie Obenour <demiobenour@gmail.com> wrote: >> >> On 2/25/26 02:52, Bertrand Marquis wrote: >>> Hi Parav, >>> >>>> On 25 Feb 2026, at 05:58, Parav Pandit <parav@nvidia.com> wrote: >>>> >>>> >>>>> From: Bertrand Marquis <Bertrand.Marquis@arm.com> >>>>> Sent: 20 February 2026 02:10 PM >>>>> >>>>> Hi Parav, >>>>> >>>>>> On 13 Feb 2026, at 14:52, Parav Pandit <parav@nvidia.com> wrote: >>>>>> >>>>>> Hi Bill, >>>>>> >>>>>>> From: Bill Mills <bill.mills@linaro.org> >>>>>>> Sent: 26 January 2026 10:02 PM >>>>>>> >>>>>>> This series adds the virtio-msg transport layer. >>>>>>> >>>>>>> The individuals and organizations involved in this effort have had difficulty in >>>>>>> using the existing virtio-transports in various situations and desire to add one >>>>>>> more transport that performs its transport layer operations by sending and >>>>>>> receiving messages. >>>>>>> >>>>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>>>> * common / device level >>>>>>> * bus level >>>>>>> >>>>>>> The common / device level defines the messages exchanged between the driver >>>>>>> and a device. This common part should lead to a common driver holding most >>>>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>>>> main focus of this version of the patch series. >>>>>>> >>>>>>> The virtio-msg bus level implements the normal things a bus defines >>>>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>>>> receive operations. A number of bus implementations are envisioned, >>>>>>> some of which will be reusable and general purpose. Other bus implementations >>>>>>> might be unique to a given situation, for example only used by a PCIe card >>>>>>> and its driver. >>>>>>> >>>>>>> The standard bus messages are an effort to avoid different bus implementations >>>>>>> doing the same thing in different ways for no good reason. However the >>>>>>> different environments will require different things. Instead of trying to >>>>>>> anticipate all needs and provide something very abstract, we think >>>>>>> implementation specific messages will be needed at the bus level. Over time, >>>>>>> if we see similar messages across multiple bus implementations, we will move to >>>>>>> standardize a bus level message for that. >>>>>>> >>>>>> >>>>>> I would review more, had first round of sparse review. >>>>>> Please find few comments/questions below. >>>>>> >>>>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>>>> >>>>> That is a very interesting feedback, we will definitely take this into account. >>>>> We will have to decide how big and I will answer that in Demi's mail as there might be some drawbacks with >>>>> having very big sizes for the device ID. >>>>> >>>> I am slowly catching up on the thread. >>>> There are two types of device id needed. >>>> One is UUID style to uniquely identify the device that may show up using two transports to the driver. >>>> With that a driver can create single virtio_device object, which is reachable via two different transports. >>>> This offers performance, resiliency. >>>> This is likely a bigger string which is not efficient to use during every message transaction. >>>> >>>> Second one is: within a transport, a device id to identify the communication. >>>> I was referring to this device id to be u32, so that transport can support more than 64K devices. >>> >>> We will increase the device number size to support more than 64k devices. >>> For the UUID part, I feel it should be something provided as device information per device, so >>> we could add a non-mandatory field (nil-UUID when no ID available) and transfer that information >>> as part of GET_DEVICE_INFO. >> >> When would it not be available? > > I am not aware of implementations assigned UUIDs per device. > In any case the field can always be there and the nil-UUID used if something does not have one. I expect that the UUID will be set out of band by the management stack. >>>>>> 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. >>>>> >>>>> Set vqueue has an index to specify the virtqueue index but you have to specific all fields in one go that is true. >>>>> Do you need a solution where you could set some fields to a specific value to say "keep current" and only update part of the vqueue >>>>> configuration ? >>>>> >>>> I believe so, otherwise it cannot work with existing drivers without driver side caching them. >>> >>> I will investigate that possibility. >> >> With the current spec, adding virtqueues one at a time requires time >> quadratic in the number of virtqueues. That's not good. > > Solution i am looking at would still allow to transmit everything in one go but would leave it open > to only update/modify some specific fields (using an optional field mask to not modify some parts). Please ignore this comment. I didn't read the part about the virtqueue index. >>>>>> This requires caching all the values on the driver side before sending the transport request. >>>>>> I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. >>>>>> and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. >>>>>> It may require more plumbing, but it is cleaner interface when a new transport binding is created. >>>>> >>>>> Admin queue useable with the message transport but I would be interested to understand exactly >>>>> the model you are referring to with create/destroy model. >>>>> Could you elaborate a bit so that i could understand what messages you would expect and how this would work ? >>>>> >>>> The suggestion is to not use SET_VQUEUE legacy. >>>> The suggestion is to use admin virtqueue to create queues and destroy queues, as they are merely an object. >>>> And new transport like your proposal can adapt to the modern style. >>>> So only admin queue configuration would be the only message. >>>> Rest of the other queues can be created directly using the admin queue. >>> >>> In a way, set vqueue message could be seen like that as it is a one time operation. >>> In most cases, drivers are configuring a virtqueue in one go which is optimized here >>> as we need only one message transfer. >>> If we include the proposal to also have an enable/disable directly in the message this >>> could allow for even less messages. >>> >>> Using the admin solution on top of virtio message is something possible and not prevented >>> by this solution. >>> >>> In my mind we have an object here as you create/destroy queues in one go and the fact >>> that a bus can transfer those requests asynchronously gives a solution equivalent to what >>> would be provided by admin vqueues. >> >> Why disadvantages does using admin vqueues have? > > You need a virtqueue to use admin virtqueues so we need to have a solution to set this up. > Once there i agree you can use that instead but it would not change the fact that we need > a solution to setup a virtqueue. > Right now nothing is forbidding to use them, but we still need to have a way to configure vqueues > in the transport. Oh, that makes sense. >>>> One can say it is orthogonal feature and I agree with that reasoning. >>>> The part that bothers me is that the new transport ends up adding some legacy bits like set vqueue. >>> >>> We still need to support legacy to have existing implementation working and here we can optimize >>> them a bit by not transferring one message per field. >> >> Which existing implementations are you referring to? It appears >> that these are mostly in embedded systems. If they are shipped as >> a single unit, the drivers and devices could be updated at the same >> time and there is no need for backwards compatibility. > > I am a bit lost here, do you mean we should or not have a solution to update individual fields ? > In any case the change i suggest would not remove the solution (preferred) to do everything > in one go (plus the possibiity to enable/disable the queue in the same message) but would > also allow ot update independent fields if wanted. I meant being able to update individual virtqueues, as opposed to having to setup all virtqueues at once. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 7:52 ` Bertrand Marquis 2026-02-25 12:46 ` Demi Marie Obenour @ 2026-02-25 15:17 ` Bertrand Marquis 1 sibling, 0 replies; 105+ messages in thread From: Bertrand Marquis @ 2026-02-25 15:17 UTC (permalink / raw) To: Parav Pandit Cc: Bill Mills (bill.mills@linaro.org), virtio-comment@lists.linux.dev, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine Hi Parav, > On 25 Feb 2026, at 08:52, Bertrand Marquis <Bertrand.Marquis@arm.com> wrote: > > Hi Parav, > >> On 25 Feb 2026, at 05:58, Parav Pandit <parav@nvidia.com> wrote: >> >> >>> From: Bertrand Marquis <Bertrand.Marquis@arm.com> >>> Sent: 20 February 2026 02:10 PM >>> >>> Hi Parav, >>> >>>> On 13 Feb 2026, at 14:52, Parav Pandit <parav@nvidia.com> wrote: >>>> >>>> Hi Bill, >>>> >>>>> From: Bill Mills <bill.mills@linaro.org> >>>>> Sent: 26 January 2026 10:02 PM >>>>> >>>>> This series adds the virtio-msg transport layer. >>>>> >>>>> The individuals and organizations involved in this effort have had difficulty in >>>>> using the existing virtio-transports in various situations and desire to add one >>>>> more transport that performs its transport layer operations by sending and >>>>> receiving messages. >>>>> >>>>> Implementations of virtio-msg will normally be done in multiple layers: >>>>> * common / device level >>>>> * bus level >>>>> >>>>> The common / device level defines the messages exchanged between the driver >>>>> and a device. This common part should lead to a common driver holding most >>>>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>>>> The kernel implementation in [3] shows this separation. As with other transport >>>>> layers, virtio-msg should not require modifications to existing virtio device >>>>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>>>> main focus of this version of the patch series. >>>>> >>>>> The virtio-msg bus level implements the normal things a bus defines >>>>> (enumeration, dma operations, etc) but also implements the message send and >>>>> receive operations. A number of bus implementations are envisioned, >>>>> some of which will be reusable and general purpose. Other bus implementations >>>>> might be unique to a given situation, for example only used by a PCIe card >>>>> and its driver. >>>>> >>>>> The standard bus messages are an effort to avoid different bus implementations >>>>> doing the same thing in different ways for no good reason. However the >>>>> different environments will require different things. Instead of trying to >>>>> anticipate all needs and provide something very abstract, we think >>>>> implementation specific messages will be needed at the bus level. Over time, >>>>> if we see similar messages across multiple bus implementations, we will move to >>>>> standardize a bus level message for that. >>>>> >>>> >>>> I would review more, had first round of sparse review. >>>> Please find few comments/questions below. >>>> >>>> 1. device number should be 32-bit in struct virtio_msg_header. >>>> From SIOV_R2 experiences, we learnt that some uses have use case for more than 64k devices. >>>> Also mapping PCI BDF wont be enough in 16-bits considering domain field. >>> >>> That is a very interesting feedback, we will definitely take this into account. >>> We will have to decide how big and I will answer that in Demi's mail as there might be some drawbacks with >>> having very big sizes for the device ID. >>> >> I am slowly catching up on the thread. >> There are two types of device id needed. >> One is UUID style to uniquely identify the device that may show up using two transports to the driver. >> With that a driver can create single virtio_device object, which is reachable via two different transports. >> This offers performance, resiliency. >> This is likely a bigger string which is not efficient to use during every message transaction. >> >> Second one is: within a transport, a device id to identify the communication. >> I was referring to this device id to be u32, so that transport can support more than 64K devices. > > We will increase the device number size to support more than 64k devices. > For the UUID part, I feel it should be something provided as device information per device, so > we could add a non-mandatory field (nil-UUID when no ID available) and transfer that information > as part of GET_DEVICE_INFO. > >> >> >> >>> In any case this point was raised by you, Peter and Demi and we will definitely handle it in v2. >>> >> Ok. >> >>>> >>>> 2. msg_size of 16-bits for 64KB-8 bytes is too less for data transfer. >>>> For example, a TCP stream wants to send 64KB of data + payload, needs more than 64KB data. >>>> Needs 32-bits. >>> >>> The point of the transport is not to transfer data, this should be done using virtqueues. >> How comes virtqueues live outside of the transport? >> I don't understand this at all. >> Transport is supposed to transport the descriptors and data also _of_ the virtqueue. > > virtqueues do not really live outside of the transport but are defined and used in the same way > as they are in PCI or MMIO. The transport provide ways to configure the virtqueues but relies on > the fact that virtqueue content is available to both sides through DMA, shared memory or any other > custom mean. > >> >> For example you can see that proposal [1] is nearly complete that enables virtio msg layer over various transports. >> This includes control operations and data operations both. >> Patch-4 defines the control operations (similar to this proposal) >> Patch-5 defines the binding of data operation with the specific underlying transports such as tcp, rdma etc. >> >> In your case, patch-5 would be for system specific bus such as FF-A or others that others have acked for. >> But it must be documented somewhere. > > Correct me if I'm wrong but this is centered on transfering virqueue content over messages and having > something that is network friendly. Even though some of the principles could be reused, I think the use > case is a bit different. > > In our use cases we have remote memory access but what we cannot afford is trap-and-emulate to > access registers (MMIO or PCI) or a hardware specific solution (CCW) (for scheduling issues and > performance reasons). > > Now what could be done easily is extend the virtio-msg proposal with a solution to transfer virtqueues > over messages to and answer to this use case. > >> >> >> [1] https://yhbt.net/lore/virtio-comment/20231023104647.290759-2-pizhenwei@bytedance.com/ >> >>> Transport is only there to allow access to registers, features and configuration, the main transfers are to >>> be done using virtqueues and i would not expect a network driver to use MMIO registers to write network >>> packet payloads but to use virtqueues for that (as vsock or net is doing). >>> >> Very strange. >> If that is your intention, this transport thing must be renamed to control-transport to succinctly indicate its objective is for 'control operation'. > > The proposal is following what PCI, MMIO and CCW transports are defining as they do not mention how virtqueues data is transmitted. > >> >>> So the msg_size here is just to fit transport specific requests. >>> >>>> >>>> 3. BUS_MSG_EVENT_DEVICE to have symmetric name as ADDED and REMOVED (instead of READY) >>>> But more below. >>> >>> Good point, we will check that to make this coherent with other transports. >>> >> Ok. >>>> >>>> 4. I dont find the transport messages to read and write to the driver memory supplied in VIRTIO_MSG_SET_VQUEUE addresses to operate >>> the virtqueues. >>>> Dont we need VIRTIO_MEM_READ, VIRTIO_MEM_WRITE request and response? >>> >>> Equivalent of virtio_mem_read/write is provided by GET/SET CONFIG which allows you to access the configuration area. >>> So those are the equivalent of mem_read/write. >>> Does that answer your question ? >>> >> No. but I understood that what is proposed here is a partial proposal to have transport only for control operation. >> And data transfer implementation is undefined. > > But configuration register read/write which is not going through virtqueues is still covered. > Can you confirm and ack we do not need any new message added for mem_read/write ? > >> >>>> Also, the queue notification message is missing at bus level. >>> >>> Queue notifications are provided by event_avail and event_used messages. >>> >> This does not make sense either. >> The operations that related towards the virtqueues should be part of the virtqueue operations. >> >> By not defining specific bus specific details, we don't seem to gain anything significant. >> So while transport message are good, it needs to cover virtqueue data exchanges too as previously proposed in [1]. > > Please see my previous answer and discussions with Demi for notifications handling. > >> >>>> But I dont think queue notification via message good idea anyway. >>> >>> As said in the spec, a bus can handle this using MSI or interrupts and generate the messages in the interface >>> between the bus and the transport. It is not an hard requirement to have those done using messages transfered and >>> the bus can do what it wants, but it must inform the transport using a message. >>> >>>> We rather need a more higher-level message for virtqueues. >>>> Such as posting the descriptors to device. And this should translate into a VIRTIO_MSG_DESC_SEND or bus specific binding. >>>> Without this msg transport is incomplete. >>> >>> I am not completely understanding what you mean here, are your concerns covered by the event messages ? >>> >> We need transport messages agnostic of the transport like patch-4 in [1]. >> And transport binding defined for each bus of fabric like [2]. >> So that message interface can support [1] as well by only extending the virtqueue bindings. > > I do think that we could easily support your use case of transfering the messages over network. > What is not covered by our proposal is definitely virtqueue data transfer but this could be easily > done as part of a bus implementation without impacting the transport itself. > >> >> >>>> >>>> 5. msg_id to be renamed to msg_opcode, and token to be renamed to msg_id as it identifies the msg (as written in the description) in >>> >>> We went back and forth on the naming, msg_id could be misunderstood as message opcode. >>> I guess we will have to find the best consensus here. >>> >> Ok. >> >>>> >>>> 6. Bus requirements ordering is too strict for implementing any performant data path as data response may not be in same order as request >>> for reads. >>> >>> This was pointed out in an other mail and i agree here. >>> I will investigate how we could make this a feature bit which i think could be a good solution. >>> >> Ok. >>>> >>>> 7. VIRTIO_MSG_SET_VQUEUE does not have bit field for individual addresses. >>> >>> Set vqueue has an index to specify the virtqueue index but you have to specific all fields in one go that is true. >>> Do you need a solution where you could set some fields to a specific value to say "keep current" and only update part of the vqueue >>> configuration ? >>> >> I believe so, otherwise it cannot work with existing drivers without driver side caching them. > > I will investigate that possibility. > >> >>>> This requires caching all the values on the driver side before sending the transport request. >>>> I think it is time for virtio spec to shift to virt queue create and destroy model using the admin queue interface. >>>> and no need to bring this VIRTIO_MSG_SET_VQUEUE legacy to new transport bindings. >>>> It may require more plumbing, but it is cleaner interface when a new transport binding is created. >>> >>> Admin queue useable with the message transport but I would be interested to understand exactly >>> the model you are referring to with create/destroy model. >>> Could you elaborate a bit so that i could understand what messages you would expect and how this would work ? >>> >> The suggestion is to not use SET_VQUEUE legacy. >> The suggestion is to use admin virtqueue to create queues and destroy queues, as they are merely an object. >> And new transport like your proposal can adapt to the modern style. >> So only admin queue configuration would be the only message. >> Rest of the other queues can be created directly using the admin queue. > > In a way, set vqueue message could be seen like that as it is a one time operation. > In most cases, drivers are configuring a virtqueue in one go which is optimized here > as we need only one message transfer. > If we include the proposal to also have an enable/disable directly in the message this > could allow for even less messages. > > Using the admin solution on top of virtio message is something possible and not prevented > by this solution. > > In my mind we have an object here as you create/destroy queues in one go and the fact > that a bus can transfer those requests asynchronously gives a solution equivalent to what > would be provided by admin vqueues. > >> >> One can say it is orthogonal feature and I agree with that reasoning. >> The part that bothers me is that the new transport ends up adding some legacy bits like set vqueue. > > We still need to support legacy to have existing implementation working and here we can optimize > them a bit by not transferring one message per field. > >> >>>> >>>> 8. We should have the message response header for req-resp method with status code. >>>> (even though they are plain MMIO writes for pci). >>>> As opposed to that, proposed messages are doing composite tasks, hence we need the status response. >>> >>> You want an error code in the header, is that what you mean ? >> Yes, in the response header. > >> >>> We went into this discussion while designing this, the main issue is that most drivers could not handle an >>> error as this is something not possible in MMIO or PCI so we decided instead to say that in case of error >>> a valid answer must be generated (all 0 when doing get_config for example). >>> >> Well, what is proposed here is a generic transport for future to come. >> So even if driver does not handle error for MMIO/PCI register writes, transport would have commands for future to come. >> So creating V1 structure in future just for the status wouldn't be useful. > > We do handle that but the error transport is handled by the bus and is returned as an error code to > the transport (msg send returning an error number) instead of having an error field in the message. > > We felt this design was making things simpler and more compliant to existing designs. > >> >> For LOAD/STORE type of messages, driver can always ignore the error. >> The messages are used for bus discovery too which is outside of existing drivers scope. >> So status is useful there too. > > I will keep that solution in mind to replace the current one but please consider what I said upper and > confirm. > >> >> >>>> >>>> 9. virtio stack cannot handle hotplug devices today, at least Linux. >>>> So if the intention is to fit into such a system, DEVICE_BUS_STATE_REMOVED will not be enough. >>>> A graceful method is needed. >>> >>> Hotplug could be handled, remove is more problematic in Linux (you could declare a new device instance >>> at any time and this should work). >>> >>> Could you give more details on what a graceful method would need ? >>> >> Graceful would need a flow something like below. >> 1. Remove device request (sent by device side bus) >> Device continue to operate while the request is serviced >> >> 2. Driver acts on the request and unloads the driver/reset the device etc. >> >> 3. Driver (and device) eventually stops using the device by stopping all the bus activities. >> >> 4. Bus driver notifies to the device, it is safe to remove and remove request is completed. > > This is a transport side handling possible depending on what the operating system provides as solutions to > handle things this way. > > Here we have a generic message saying "this device is out" which leaves space to handle this gracefully on > top and this can happen in lots of cases where the device is offline without it doing it knowingly (crash, power > down, you name it). > > In a graceful scenario i would expect the driver to ask the device to power down through some device specific > means which would result in the DEVICE_BUS_STATE_REMOVED messages being sent once it has turn > itself off ending in your scenario. > > Do you agree or did i miss something ? Coming back on this subject, we have one message defined for a backend to signal that a device is removed. This is a standard bus message but a specific bus wanting to design a more graceful way to handle device remove could do so without entering in contradiction with the spec. I am not sure that adding more standard bus messages to handle this case would really be needed here as any bus can do it using its own way (with probably a strong dependency on the upper OS) but without any link or consequence on the generic transport part. Are you ok with this and letting the message simple in the spec ? I am ok to add some kind of mention that this is possible or recommended to do but not specified here for example. Cheers Bertrand IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills ` (8 preceding siblings ...) 2026-02-13 13:52 ` Parav Pandit @ 2026-02-24 17:57 ` Demi Marie Obenour 2026-02-25 15:21 ` Alex Bennée 9 siblings, 1 reply; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-24 17:57 UTC (permalink / raw) To: Bill Mills, virtio-comment Cc: Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Alex Bennee, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 3623 bytes --] On 1/26/26 11:32, Bill Mills wrote: > This series adds the virtio-msg transport layer. > > The individuals and organizations involved in this effort have had difficulty in > using the existing virtio-transports in various situations and desire to add one > more transport that performs its transport layer operations by sending and > receiving messages. > > Implementations of virtio-msg will normally be done in multiple layers: > * common / device level > * bus level > > The common / device level defines the messages exchanged between the driver > and a device. This common part should lead to a common driver holding most > of the virtio specifics and can be shared by all virtio-msg bus implementations. > The kernel implementation in [3] shows this separation. As with other transport > layers, virtio-msg should not require modifications to existing virtio device > implementations (virtio-net, virtio-blk etc). The common / device level is the > main focus of this version of the patch series. > > The virtio-msg bus level implements the normal things a bus defines > (enumeration, dma operations, etc) but also implements the message send and > receive operations. A number of bus implementations are envisioned, > some of which will be reusable and general purpose. Other bus implementations > might be unique to a given situation, for example only used by a PCIe card > and its driver. > > The standard bus messages are an effort to avoid different bus implementations > doing the same thing in different ways for no good reason. However the > different environments will require different things. Instead of trying to > anticipate all needs and provide something very abstract, we think > implementation specific messages will be needed at the bus level. Over time, > if we see similar messages across multiple bus implementations, we will move to > standardize a bus level message for that. > > We are working on a few reusable bus implementations: > > * virtio-msg-ffa based on Arm FF-A interface for use between: > * normal world and secure world > * host and VM or VM to VM > * Can be used w/ or with out a hypervisor > * Any Hypervisor that implements FF-A can be used > * We have this working with pKVM and Xen > * We have this working with Trusty and OP-TEE > > * virtio-msg-amp for use between heterogenous systems > * The main processors and its co-processors on an AMP SOC > * Two or more systems connected via PCIe > * Minimal requirements: bi-directional interrupts and > at least one shared memory area > * hvac-demo has 2 demos of this > * This is working on two hardware platforms > > * virtio-msg-loopback for userspace implemented devices > * Allows user space to provide devices to its own kernel > * This is similar to fuse, cuse or loopback block devices but for virtio > * hvac-demo has a demo > > We also anticipate a few more: > > * virtio-msg-xen specific to Xen > * Usable on any Xen system (including x86 where FF-A does not exist) > * Using Xen events and page grants > > * virtio-msg over admin virtqueues > * This allows any virtio-pci device that supports admin virtqueues to also > support a virtio-msg bus that supports sub devices > * [We are looking for collaborators for this work] Which transport, if any, would be suitable for communication between a KVM VM and a userspace process or between two KVM VMs? Would this require a new bus? If so, should it be implemented in terms of vhost-user? -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-24 17:57 ` Demi Marie Obenour @ 2026-02-25 15:21 ` Alex Bennée 2026-02-25 15:46 ` Demi Marie Obenour 0 siblings, 1 reply; 105+ messages in thread From: Alex Bennée @ 2026-02-25 15:21 UTC (permalink / raw) To: Demi Marie Obenour Cc: Bill Mills, virtio-comment, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Armelle Laine Demi Marie Obenour <demiobenour@gmail.com> writes: > On 1/26/26 11:32, Bill Mills wrote: >> This series adds the virtio-msg transport layer. >> >> The individuals and organizations involved in this effort have had difficulty in >> using the existing virtio-transports in various situations and desire to add one >> more transport that performs its transport layer operations by sending and >> receiving messages. >> >> Implementations of virtio-msg will normally be done in multiple layers: >> * common / device level >> * bus level >> >> The common / device level defines the messages exchanged between the driver >> and a device. This common part should lead to a common driver holding most >> of the virtio specifics and can be shared by all virtio-msg bus implementations. >> The kernel implementation in [3] shows this separation. As with other transport >> layers, virtio-msg should not require modifications to existing virtio device >> implementations (virtio-net, virtio-blk etc). The common / device level is the >> main focus of this version of the patch series. >> >> The virtio-msg bus level implements the normal things a bus defines >> (enumeration, dma operations, etc) but also implements the message send and >> receive operations. A number of bus implementations are envisioned, >> some of which will be reusable and general purpose. Other bus implementations >> might be unique to a given situation, for example only used by a PCIe card >> and its driver. >> >> The standard bus messages are an effort to avoid different bus implementations >> doing the same thing in different ways for no good reason. However the >> different environments will require different things. Instead of trying to >> anticipate all needs and provide something very abstract, we think >> implementation specific messages will be needed at the bus level. Over time, >> if we see similar messages across multiple bus implementations, we will move to >> standardize a bus level message for that. >> >> We are working on a few reusable bus implementations: >> >> * virtio-msg-ffa based on Arm FF-A interface for use between: >> * normal world and secure world >> * host and VM or VM to VM >> * Can be used w/ or with out a hypervisor >> * Any Hypervisor that implements FF-A can be used >> * We have this working with pKVM and Xen >> * We have this working with Trusty and OP-TEE >> >> * virtio-msg-amp for use between heterogenous systems >> * The main processors and its co-processors on an AMP SOC >> * Two or more systems connected via PCIe >> * Minimal requirements: bi-directional interrupts and >> at least one shared memory area >> * hvac-demo has 2 demos of this >> * This is working on two hardware platforms >> >> * virtio-msg-loopback for userspace implemented devices >> * Allows user space to provide devices to its own kernel >> * This is similar to fuse, cuse or loopback block devices but for virtio >> * hvac-demo has a demo >> >> We also anticipate a few more: >> >> * virtio-msg-xen specific to Xen >> * Usable on any Xen system (including x86 where FF-A does not exist) >> * Using Xen events and page grants >> >> * virtio-msg over admin virtqueues >> * This allows any virtio-pci device that supports admin virtqueues to also >> support a virtio-msg bus that supports sub devices >> * [We are looking for collaborators for this work] > > Which transport, if any, would be suitable for communication > between a KVM VM and a userspace process I think vhost-user already covers this case well enough. However there is no reason the VMM could terminate a virtio-msg bus and pass the messages across vhost-user to a device backend. > or between two KVM VMs? > Would this require a new bus? If so, should it be implemented in > terms of vhost-user? If you want the backend for a VirtIO device in another KVM VM that I suspect you want to avoid a round-trip through the user-space of the VMM. If FF-A isn't an option the hypervisor could support some sort of minimal hypercall transport to pass messages between the two. -- Alex Bennée Virtualisation Tech Lead @ Linaro ^ permalink raw reply [flat|nested] 105+ messages in thread
* Re: [PATCH v1 0/4] virtio-msg transport layer 2026-02-25 15:21 ` Alex Bennée @ 2026-02-25 15:46 ` Demi Marie Obenour 0 siblings, 0 replies; 105+ messages in thread From: Demi Marie Obenour @ 2026-02-25 15:46 UTC (permalink / raw) To: Alex Bennée Cc: Bill Mills, virtio-comment, Bertrand Marquis, Edgar E . Iglesias, Arnaud Pouliquen, Viresh Kumar, Armelle Laine [-- Attachment #1.1.1: Type: text/plain, Size: 5173 bytes --] On 2/25/26 10:21, Alex Bennée wrote: > Demi Marie Obenour <demiobenour@gmail.com> writes: > >> On 1/26/26 11:32, Bill Mills wrote: >>> This series adds the virtio-msg transport layer. >>> >>> The individuals and organizations involved in this effort have had difficulty in >>> using the existing virtio-transports in various situations and desire to add one >>> more transport that performs its transport layer operations by sending and >>> receiving messages. >>> >>> Implementations of virtio-msg will normally be done in multiple layers: >>> * common / device level >>> * bus level >>> >>> The common / device level defines the messages exchanged between the driver >>> and a device. This common part should lead to a common driver holding most >>> of the virtio specifics and can be shared by all virtio-msg bus implementations. >>> The kernel implementation in [3] shows this separation. As with other transport >>> layers, virtio-msg should not require modifications to existing virtio device >>> implementations (virtio-net, virtio-blk etc). The common / device level is the >>> main focus of this version of the patch series. >>> >>> The virtio-msg bus level implements the normal things a bus defines >>> (enumeration, dma operations, etc) but also implements the message send and >>> receive operations. A number of bus implementations are envisioned, >>> some of which will be reusable and general purpose. Other bus implementations >>> might be unique to a given situation, for example only used by a PCIe card >>> and its driver. >>> >>> The standard bus messages are an effort to avoid different bus implementations >>> doing the same thing in different ways for no good reason. However the >>> different environments will require different things. Instead of trying to >>> anticipate all needs and provide something very abstract, we think >>> implementation specific messages will be needed at the bus level. Over time, >>> if we see similar messages across multiple bus implementations, we will move to >>> standardize a bus level message for that. >>> >>> We are working on a few reusable bus implementations: >>> >>> * virtio-msg-ffa based on Arm FF-A interface for use between: >>> * normal world and secure world >>> * host and VM or VM to VM >>> * Can be used w/ or with out a hypervisor >>> * Any Hypervisor that implements FF-A can be used >>> * We have this working with pKVM and Xen >>> * We have this working with Trusty and OP-TEE >>> >>> * virtio-msg-amp for use between heterogenous systems >>> * The main processors and its co-processors on an AMP SOC >>> * Two or more systems connected via PCIe >>> * Minimal requirements: bi-directional interrupts and >>> at least one shared memory area >>> * hvac-demo has 2 demos of this >>> * This is working on two hardware platforms >>> >>> * virtio-msg-loopback for userspace implemented devices >>> * Allows user space to provide devices to its own kernel >>> * This is similar to fuse, cuse or loopback block devices but for virtio >>> * hvac-demo has a demo >>> >>> We also anticipate a few more: >>> >>> * virtio-msg-xen specific to Xen >>> * Usable on any Xen system (including x86 where FF-A does not exist) >>> * Using Xen events and page grants >>> >>> * virtio-msg over admin virtqueues >>> * This allows any virtio-pci device that supports admin virtqueues to also >>> support a virtio-msg bus that supports sub devices >>> * [We are looking for collaborators for this work] >> >> Which transport, if any, would be suitable for communication >> between a KVM VM and a userspace process > > I think vhost-user already covers this case well enough. However there > is no reason the VMM could terminate a virtio-msg bus and pass the > messages across vhost-user to a device backend. Does this provide any advantages over the existing virtio-PCI transport? >> or between two KVM VMs? >> Would this require a new bus? If so, should it be implemented in >> terms of vhost-user? > > If you want the backend for a VirtIO device in another KVM VM that I > suspect you want to avoid a round-trip through the user-space of the > VMM. If FF-A isn't an option the hypervisor could support some sort of > minimal hypercall transport to pass messages between the two. My plan was to use irqfd + ioeventfd + shared memory to avoid round-trips through userspace in the data path. That's not a good fit for any of the proposed transports, though. Do you have suggestions for how to implement a virtio device in another KVM VM? I'd prefer to not need to patch KVM itself unless it provides a big advantage of some sort. I know that virtio-vhost-user already exists, but am not sure if something based on virtio-msg would be a better option. Specifically, there could be a virtio device that passed driver-to-device messages to the backend VM, and passed device-to-driver messages to the frontend VMM. If you think that there is an even better option, I would greatly appreciate the suggestion. -- Sincerely, Demi Marie Obenour (she/her/hers) [-- Attachment #1.1.2: OpenPGP public key --] [-- Type: application/pgp-keys, Size: 7253 bytes --] [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 105+ messages in thread
end of thread, other threads:[~2026-02-26 22:08 UTC | newest] Thread overview: 105+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-01-26 16:32 [PATCH v1 0/4] virtio-msg transport layer Bill Mills 2026-01-26 16:32 ` [PATCH v1 1/4] virtio-msg: add new command for bus normative Bill Mills 2026-02-03 19:42 ` Edgar E. Iglesias 2026-01-26 16:32 ` [PATCH v1 2/4] virtio-msg: Add virtio-msg, a message based virtio transport layer Bill Mills 2026-02-06 16:28 ` Peter Hilber 2026-02-10 9:39 ` Bertrand Marquis 2026-02-12 11:16 ` Peter Hilber 2026-02-20 8:23 ` Bertrand Marquis 2026-02-26 13:53 ` Peter Hilber 2026-02-13 19:09 ` Demi Marie Obenour 2026-02-20 8:52 ` Bertrand Marquis 2026-02-21 2:04 ` Demi Marie Obenour 2026-02-23 7:44 ` Bertrand Marquis 2026-02-24 15:41 ` Demi Marie Obenour 2026-02-24 16:14 ` Bertrand Marquis 2026-02-24 17:36 ` Edgar E. Iglesias 2026-02-24 17:14 ` Demi Marie Obenour 2026-02-24 17:20 ` Bertrand Marquis 2026-02-24 17:46 ` Demi Marie Obenour 2026-02-25 7:26 ` Bertrand Marquis 2026-02-25 12:36 ` Demi Marie Obenour 2026-02-25 12:46 ` Bertrand Marquis 2026-01-26 16:32 ` [PATCH v1 3/4] virtio-msg: link virtio-msg content Bill Mills 2026-02-03 19:43 ` Edgar E. Iglesias 2026-01-26 16:32 ` [PATCH v1 4/4] virtio-msg: add conformance entries in conformance chapter Bill Mills 2026-02-03 19:43 ` Edgar E. Iglesias 2026-01-26 21:47 ` [PATCH v1 0/4] virtio-msg transport layer Demi Marie Obenour 2026-02-03 13:21 ` Michael S. Tsirkin 2026-02-03 19:48 ` Edgar E. Iglesias 2026-02-03 19:55 ` Michael S. Tsirkin 2026-02-04 8:33 ` Bertrand Marquis 2026-02-04 13:50 ` Arnaud POULIQUEN 2026-02-04 3:29 ` Viresh Kumar 2026-02-04 5:34 ` Manivannan Sadhasivam 2026-02-13 13:52 ` Parav Pandit 2026-02-13 19:45 ` Demi Marie Obenour 2026-02-19 17:31 ` Armelle Laine 2026-02-20 8:55 ` Bertrand Marquis 2026-02-19 23:54 ` Michael S. Tsirkin 2026-02-20 6:13 ` Parav Pandit 2026-02-20 9:02 ` Bertrand Marquis 2026-02-25 7:45 ` Manivannan Sadhasivam 2026-02-25 8:03 ` Bertrand Marquis 2026-02-25 8:12 ` Michael S. Tsirkin 2026-02-25 10:06 ` Manivannan Sadhasivam 2026-02-25 10:10 ` Michael S. Tsirkin 2026-02-25 10:14 ` Bertrand Marquis 2026-02-25 10:22 ` Michael S. Tsirkin 2026-02-25 10:53 ` Manivannan Sadhasivam 2026-02-25 10:24 ` Parav Pandit 2026-02-25 10:35 ` Bertrand Marquis 2026-02-25 10:52 ` Michael S. Tsirkin 2026-02-25 10:55 ` Bertrand Marquis 2026-02-25 10:58 ` Michael S. Tsirkin 2026-02-25 14:45 ` Parav Pandit 2026-02-25 14:49 ` Michael S. Tsirkin 2026-02-25 14:53 ` Bertrand Marquis 2026-02-25 15:00 ` Parav Pandit 2026-02-25 15:07 ` Parav Pandit 2026-02-25 15:12 ` Bertrand Marquis 2026-02-25 15:15 ` Michael S. Tsirkin 2026-02-25 15:36 ` Demi Marie Obenour 2026-02-25 15:40 ` Bertrand Marquis 2026-02-25 15:48 ` Demi Marie Obenour 2026-02-25 15:51 ` Bertrand Marquis 2026-02-25 16:15 ` Demi Marie Obenour 2026-02-26 5:40 ` Manivannan Sadhasivam 2026-02-26 7:05 ` Bertrand Marquis 2026-02-25 15:11 ` Manivannan Sadhasivam 2026-02-25 15:15 ` Parav Pandit 2026-02-26 5:36 ` Manivannan Sadhasivam 2026-02-26 5:59 ` Parav Pandit 2026-02-26 6:19 ` Manivannan Sadhasivam 2026-02-26 7:01 ` Bertrand Marquis 2026-02-26 7:28 ` Manivannan Sadhasivam 2026-02-26 19:20 ` Demi Marie Obenour 2026-02-26 22:08 ` Edgar E. Iglesias 2026-02-25 15:23 ` Demi Marie Obenour 2026-02-25 16:42 ` Edgar E. Iglesias 2026-02-25 12:53 ` Demi Marie Obenour 2026-02-25 13:09 ` Manivannan Sadhasivam 2026-02-25 13:12 ` Demi Marie Obenour 2026-02-25 13:29 ` Bertrand Marquis 2026-02-25 15:19 ` Demi Marie Obenour 2026-02-25 15:27 ` Bertrand Marquis 2026-02-20 10:03 ` Michael S. Tsirkin 2026-02-25 5:09 ` Parav Pandit 2026-02-25 7:25 ` Michael S. Tsirkin 2026-02-25 9:18 ` Parav Pandit 2026-02-25 9:22 ` Michael S. Tsirkin 2026-02-25 9:35 ` Bertrand Marquis 2026-02-25 9:54 ` Michael S. Tsirkin 2026-02-25 10:01 ` Bertrand Marquis 2026-02-25 10:08 ` Michael S. Tsirkin 2026-02-20 8:58 ` Bertrand Marquis 2026-02-20 8:40 ` Bertrand Marquis 2026-02-25 4:58 ` Parav Pandit 2026-02-25 7:52 ` Bertrand Marquis 2026-02-25 12:46 ` Demi Marie Obenour 2026-02-25 13:05 ` Bertrand Marquis 2026-02-25 13:09 ` Demi Marie Obenour 2026-02-25 15:17 ` Bertrand Marquis 2026-02-24 17:57 ` Demi Marie Obenour 2026-02-25 15:21 ` Alex Bennée 2026-02-25 15:46 ` Demi Marie Obenour
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox