[virtio-dev] [PATCH v3 0/4] Introduce aux. notifications and virtio-vhost-user

[virtio-dev] [PATCH v3 0/4] Introduce aux. notifications and virtio-vhost-user

[Usama Arif]

Hi,

This patch series introduces device and driver auxiliary notifications
as a new set of virtio device resources, as well as vhost-user device
backend that uses these resources.

Driver auxiliary notifications allow the device to send notifications
other than configuration changes and used buffer notifications to the
driver, these are optional and their meaning is **device-specific**.

Device auxiliary notifcations allow the driver to send notifcations
other than available buffer notifications to the device for example
through a device register, these are optional and their meaning is
**device-specific**.

These resources are used in the last patch by the virtio-vhost-user
device in order to send/receive notifications to/from the driver
regarding activity on the vhost-user virtqueues. By standardizing
these resources as standalone virtio device resources, other future
devices will be able to use them as well.

The last patch introduces the vhost-user device backend which facilitates
vhost-user device emulation through vhost-user protocol exchanges and
access to shared memory. Software-defined networking, storage, and other
I/O appliances can provide services through this device.

This device is based on Wei Wang's vhost-pci work. The virtio-vhost-user
device differs from vhost-pci because it is a single virtio
device type that exposes the vhost-user protocol instead of a family of
new virtio device types, one for each vhost-user device type.

A HTML version with the changes is available at [1].

For more information about virtio-vhost-user, see [2].

These patches are based on the work initially done by Stefan Hajnoczi [2]
and continued by Nikos Dragazis [3], [4].

A working prototype implementing this spec can be reproduced using
instructions in [5] utilizing QEMU [6] and DPDK [7].
This is also based on the work initially done by Stefan
and continued later by Nikos. The prototype code uses the terms
"doorbell"/"device-specific notification" and "master"/"slave" instead of
"device auxiliary notification"/"driver auxiliary notification" and
"frontend"/"backend". This is based on older work, however, their
functionality is the same. If these virtio-spec changes get approved,
I will refractor the patches for QEMU/DPDK and send them for review
according to the final patches in the respective mailing list.

Thanks and looking forward to your response!
Usama

[1] https://uarif1.github.io/vvu/v3/virtio-v1.2-cs01
[2] https://wiki.qemu.org/Features/VirtioVhostUser
[3] https://ndragazis.github.io/dpdk-vhost-vvu-demo.html
[4] https://lists.oasis-open.org/archives/virtio-dev/202005/msg00132.html
[5] https://uarif1.github.io/vvu/dpdk-vvu-instructions
[6] https://github.com/uarif1/qemu/tree/vvu
[7] https://github.com/uarif1/dpdk/tree/vvu
----------
v2->v3:
Mostly addressing comments made by Michael S. Tsirkin:
- Give a clearer relation between backend driver, backend device and frontend at
  beginning of the Vhost-user Device Backend section.
- change uuid to id.
- Add information about inflight memory.

v1->v2:
- Shortened device aux. notification section
- Make data width of device aux. notifications device specific and not
  limited to 2 bytes.
- Added information about the minimum number of MSIX vectors.
- Split virtio-mmio implementation of driver aux. notifications into 2
  registers, DeviceAuxNotificationIndex and DeviceAuxNotificationData.
- Made the shared memory section in virtio vhost-user clearer and added
  the shared memory would look like when
  VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS has been negotiated.

Usama Arif (4):
  content: Introduce driver/device auxiliary notifications
  content: Introduce driver/device aux. notification cfg type for PCI
  content: Introduce driver/device auxiliary notifications for MMIO
  vhost-user: add vhost-user device type

 conformance.tex       |  29 +++-
 content.tex           | 254 +++++++++++++++++++++++++++++-----
 introduction.tex      |   3 +
 virtio-vhost-user.tex | 309 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 554 insertions(+), 41 deletions(-)
 create mode 100644 virtio-vhost-user.tex

-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v3 1/4] content: Introduce driver/device auxiliary notifications

[Usama Arif]

Driver auxiliary notifications allow the device to send notifications
other than configuration changes and used buffer notifications to the
driver, these are optional and their meaning is device-specific.

Device auxiliary notifications allow the driver to send notifications
other than available buffer notifications to the device for example
through a device register, these are optional and their meaning is
device-specific.

These device-specific notifications are needed later when adding support
for virtio-vhost-user device.

Signed-off-by: Usama Arif <usama.arif@bytedance.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 35 ++++++++++++++++++++++-------------
 1 file changed, 22 insertions(+), 13 deletions(-)

diff --git a/content.tex b/content.tex
index e863709..49f9f2a 100644
--- a/content.tex
+++ b/content.tex
@@ -160,29 +160,38 @@ \subsection{Legacy Interface: A Note on Feature
 Specification text within these sections generally does not apply
 to non-transitional devices.
 
-\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
-/ Notifications}
+\section{Notifications}\label{sec:Basic Facilities of a Virtio Device / Notifications}
 
 The notion of sending a notification (driver to device or device
 to driver) plays an important role in this specification. The
 modus operandi of the notifications is transport specific.
 
-There are three types of notifications: 
+There are five types of notifications:
 \begin{itemize}
 \item configuration change notification
 \item available buffer notification
-\item used buffer notification. 
+\item used buffer notification
+\item driver auxiliary notification
+\item device auxiliary notification
 \end{itemize}
 
-Configuration change notifications and used buffer notifications are sent
-by the device, the recipient is the driver. A configuration change
-notification indicates that the device configuration space has changed; a
-used buffer notification indicates that a buffer may have been made used
-on the virtqueue designated by the notification.
-
-Available buffer notifications are sent by the driver, the recipient is
-the device. This type of notification indicates that a buffer may have
-been made available on the virtqueue designated by the notification.
+Configuration change notifications, used buffer notifications and
+driver auxiliary notifications are sent by the device,
+the recipient is the driver. A configuration change notification indicates
+that the device configuration space has changed; a used buffer notification
+indicates that a buffer may have been made used on the virtqueue designated
+by the notification; driver auxiliary notifications allow the
+device to send notifications other than configuration changes and used
+buffer notifications to the driver, these are optional and their meaning
+is device-specific.
+
+Available buffer notifications and device auxiliary notifications
+are sent by the driver, the recipient is the device. Available buffer
+notifications indicate that a buffer may have been made available on the
+virtqueue designated by the notification; device auxiliary
+notifications allow the driver to send notifications other than available
+buffer notifications to the device for example through a device register, these
+are optional and their meaning is device-specific.
 
 The semantics, the transport-specific implementations, and other
 important aspects of the different notifications are specified in detail
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v3 2/4] content: Introduce driver/device aux. notification cfg type for PCI

[Usama Arif]

This includes the PCI device conformances for these notification
capabilities.

Signed-off-by: Usama Arif <usama.arif@bytedance.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 conformance.tex |   2 +
 content.tex     | 191 ++++++++++++++++++++++++++++++++++++++++++------
 2 files changed, 171 insertions(+), 22 deletions(-)

diff --git a/conformance.tex b/conformance.tex
index c3c1d3e..f6914b0 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -370,6 +370,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device-specific configuration}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
+\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
 \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Non-transitional Device With Legacy Driver}
diff --git a/content.tex b/content.tex
index 49f9f2a..33362b7 100644
--- a/content.tex
+++ b/content.tex
@@ -719,6 +719,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 \item ISR Status
 \item Device-specific configuration (optional)
 \item PCI configuration access
+\item Driver auxiliary notifications (optional)
+\item Device auxiliary notifications (optional)
 \end{itemize}
 
 Each structure can be mapped by a Base Address register (BAR) belonging to
@@ -765,19 +767,23 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
 
 \begin{lstlisting}
 /* Common configuration */
-#define VIRTIO_PCI_CAP_COMMON_CFG        1
+#define VIRTIO_PCI_CAP_COMMON_CFG             1
 /* Notifications */
-#define VIRTIO_PCI_CAP_NOTIFY_CFG        2
+#define VIRTIO_PCI_CAP_NOTIFY_CFG             2
 /* ISR Status */
-#define VIRTIO_PCI_CAP_ISR_CFG           3
+#define VIRTIO_PCI_CAP_ISR_CFG                3
 /* Device specific configuration */
-#define VIRTIO_PCI_CAP_DEVICE_CFG        4
+#define VIRTIO_PCI_CAP_DEVICE_CFG             4
 /* PCI configuration access */
-#define VIRTIO_PCI_CAP_PCI_CFG           5
+#define VIRTIO_PCI_CAP_PCI_CFG                5
+/* Device auxiliary notification */
+#define VIRTIO_PCI_CAP_DEVICE_AUX_NOTIFY_CFG  6
+/* Driver auxiliary notification */
+#define VIRTIO_PCI_CAP_DRIVER_AUX_NOTIFY_CFG  7
 /* Shared memory region */
-#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
+#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG      8
 /* Vendor-specific data */
-#define VIRTIO_PCI_CAP_VENDOR_CFG        9
+#define VIRTIO_PCI_CAP_VENDOR_CFG             9
 \end{lstlisting}
 
         Any other value is reserved for future use.
@@ -1158,11 +1164,11 @@ \subsubsection{ISR status capability}\label{sec:Virtio Transport Options / Virti
 The ISR bits allow the driver to distinguish between device-specific configuration
 change interrupts and normal virtqueue interrupts:
 
-\begin{tabular}{ |l||l|l|l| }
+\begin{tabular}{ |l||l|l|l|l| }
 \hline
-Bits       & 0                               & 1               &  2 to 31 \\
+Bits       & 0                               & 1                                & 2                      & 3 to 31 \\
 \hline
-Purpose    & Queue Interrupt  & Device Configuration Interrupt & Reserved \\
+Purpose    & Queue Interrupt  & Device Configuration Interrupt & Driver Auxiliary Notification Interrupt & Reserved \\
 \hline
 \end{tabular}
 
@@ -1208,6 +1214,135 @@ \subsubsection{Device-specific configuration}\label{sec:Virtio Transport Options
 
 The \field{offset} for the device-specific configuration MUST be 4-byte aligned.
 
+\subsubsection{Device auxiliary notification capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
+
+The device auxiliary notification \ref{sec:Basic Facilities of a Virtio Device / Notifications}
+location is found using the VIRTIO_PCI_CAP_DEVICE_AUX_NOTIFY_CFG capability. This
+capability is immediately followed by an additional field, like so:
+
+\begin{lstlisting}
+struct virtio_pci_dev_aux_notification_cap {
+        struct virtio_pci_cap cap;
+        le32 dev_aux_notification_off_multiplier;
+};
+\end{lstlisting}
+
+The device auxiliary notification address within a BAR is calculated as follows:
+
+\begin{lstlisting}
+cap.offset + dev_aux_notification_idx * dev_aux_notification_off_multiplier
+\end{lstlisting}
+
+The \field{cap.offset} and \field{dev_aux_notification_off_multiplier} are taken
+from the device auxiliary notification capability structure above, and the
+\field{dev_aux_notification_idx} is the device auxiliary notification index.
+There is no restriction for the mapping between device auxiliary notifications
+and dev_aux_notification_idx. The number of device auxiliary notifications and
+their purpose is device-specific.
+
+\devicenormative{\paragraph}{Device auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
+
+The data width of the device auxiliary notifications is device specific.
+
+For devices that provide a data width of 1 byte:
+
+The \field{cap.offset} MUST be byte aligned.
+
+The value \field{cap.length} presented by the device MUST be at least 1
+and MUST be large enough to support device auxiliary notification offsets for all supported
+device auxiliary notifications in all possible configurations.
+
+The value \field{cap.length} presented by the device MUST satisfy:
+
+\begin{lstlisting}
+cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 1
+\end{lstlisting}
+
+where \field{max_dev_aux_notification_idx} is the maximum device auxiliary
+notification index and is dependent on the device.
+
+For devices that provide a data width of 2 bytes:
+
+The \field{cap.offset} MUST be 2-byte aligned.
+
+The device MUST either present \field{dev_aux_notification_off_multiplier} as an
+even power of 2, or present \field{dev_aux_notification_off_multiplier} as 0.
+
+The value \field{cap.length} presented by the device MUST be at least 2
+and MUST be large enough to support device auxiliary notification offsets for
+all supported device auxiliary notifications in all possible configurations.
+
+The value \field{cap.length} presented by the device MUST satisfy:
+
+\begin{lstlisting}
+cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 2
+\end{lstlisting}
+
+where \field{max_dev_aux_notification_idx} is the maximum device auxiliary
+notification index and is dependent on the device.
+
+For devices that provide a data width of 4 bytes:
+
+The \field{cap.offset} MUST be 4-byte aligned.
+
+The device MUST either present \field{dev_aux_notification_off_multiplier} as a
+number that is a power of 2 that is also a multiple 4, or present
+\field{dev_aux_notification_off_multiplier} as 0.
+
+The value \field{cap.length} presented by the device MUST be at least 4
+and MUST be large enough to support device auxiliary notification offsets for all supported
+device auxiliary notifications in all possible configurations.
+
+The value \field{cap.length} presented by the device MUST satisfy:
+
+\begin{lstlisting}
+cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 4
+\end{lstlisting}
+
+where \field{max_dev_aux_notification_idx} is the maximum device auxiliary
+notification index and is dependent on the device.
+
+\subsubsection{Driver auxiliary notification capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
+
+The Driver auxiliary notification
+\ref{sec:Basic Facilities of a Virtio Device / Notifications} location
+is found using the VIRTIO_PCI_CAP_DRIVER_AUX_NOTIFY_CFG capability. The
+driver auxiliary notification structure allows MSI-X vectors to be
+configured for notification interrupts. If MSI-X is not available, bit 2
+of the ISR status \ref{sec:Virtio Transport Options / Virtio Over PCI
+Bus / PCI Device Layout / ISR status capability} indicates that a
+driver auxiliary notification occurred. If MSI-X is not available,
+it is not possible to determine which driver auxiliary notification occured,
+hence only a single notification is supported.
+
+The driver auxiliary notification structure is the following:
+
+\begin{lstlisting}
+struct virtio_pci_driver_aux_notification_cfg {
+	le16 driver_aux_notification_select;              /* read-write */
+	le16 driver_aux_notification_msix_vector;         /* read-write */
+};
+\end{lstlisting}
+
+The driver indicates which notification is of interest by writing the
+\field{driver_aux_notification_select} field. The driver then writes the MSI-X
+vector or VIRTIO_MSI_NO_VECTOR to \field{driver_aux_notification_msix_vector} to
+change the MSI-X vector for that notification.
+
+The mapping between notifications and notification indices is
+device-specific. The total number of notifications is also
+device-specific.
+
+\devicenormative{\paragraph}{Driver auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
+
+Device MUST ignore writes to \field{driver_aux_notification_msix_vector} if the
+value written to \field{driver_aux_notification_select} is not a valid notification
+index.
+
+Device MUST return VIRTIO_MSI_NO_VECTOR for reads from
+\field{driver_aux_notification_msix_vector} if the value written to
+\field{driver_aux_notification_select} is not a valid notification index.
+
 \subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
 
 Shared memory regions \ref{sec:Basic Facilities of a Virtio
@@ -1519,15 +1654,17 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 \paragraph{MSI-X Vector Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
 
 When MSI-X capability is present and enabled in the device
-(through standard PCI configuration space) \field{config_msix_vector} and \field{queue_msix_vector} are used to map configuration change and queue
-interrupts to MSI-X vectors. In this case, the ISR Status is unused.
+(through standard PCI configuration space) \field{config_msix_vector},
+\field{queue_msix_vector} and \field{driver_aux_notification_msix_vector} are used
+to map configuration change, queue and device-specific interrupts to
+MSI-X vectors respectively. In this case, the ISR Status is unused.
 
 Writing a valid MSI-X Table entry number, 0 to 0x7FF, to
-\field{config_msix_vector}/\field{queue_msix_vector} maps interrupts triggered
-by the configuration change/selected queue events respectively to
-the corresponding MSI-X vector. To disable interrupts for an
-event type, the driver unmaps this event by writing a special NO_VECTOR
-value:
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}
+maps interrupts triggered by the configuration change/selected
+queue/device-specific events respectively to the corresponding MSI-X
+vector. To disable interrupts for an event type, the driver unmaps this
+event by writing a special NO_VECTOR value:
 
 \begin{lstlisting}
 /* Vector value used to disable MSI for queue */
@@ -1541,6 +1678,14 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 
 A device that has an MSI-X capability SHOULD support at least 2
 and at most 0x800 MSI-X vectors.
+
+A device that has an MSI-X capability MUST support atleast:
+\begin{lstlisting}
+num_msix_vectors >= 1 /* config_msix_vector */ + num_vqs /* queue_msix_vectors */ + num_driver_aux_notifications
+\end{lstlisting}
+where num_vqs is the number of virtqueues and num_driver_aux_notifications is
+the number of driver auxiliary notifications.
+
 Device MUST report the number of vectors supported in
 \field{Table Size} in the MSI-X Capability as specified in
 \hyperref[intro:PCI]{[PCI]}.
@@ -1554,16 +1699,18 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 vector 0 to MSI-X \field{Table Size}.
 Device MUST support unmapping any event type.
 
-The device MUST return vector mapped to a given event,
-(NO_VECTOR if unmapped) on read of \field{config_msix_vector}/\field{queue_msix_vector}.
-The device MUST have all queue and configuration change
-events are unmapped upon reset.
+The device MUST return the vector mapped to a given event,
+(NO_VECTOR if unmapped) on read of
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}.
+The device MUST have all queue/configuration change/device-specific
+events unmapped upon reset.
 
 Devices SHOULD NOT cause mapping an event to vector to fail
 unless it is impossible for the device to satisfy the mapping
 request.  Devices MUST report mapping
 failures by returning the NO_VECTOR value when the relevant
-\field{config_msix_vector}/\field{queue_msix_vector} field is read. 
+\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}
+field is read.
 
 \drivernormative{\subparagraph}{MSI-X Vector Configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
 
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v3 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Usama Arif]

This includes the additions to the corresponding device and driver
conformances.

Signed-off-by: Usama Arif <usama.arif@bytedance.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 content.tex | 25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)

diff --git a/content.tex b/content.tex
index 33362b7..8968fcd 100644
--- a/content.tex
+++ b/content.tex
@@ -2049,6 +2049,8 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
         in at least one of the active virtual queues.
       \item [Configuration Change Notification] - bit 1 - the interrupt was
         asserted because the configuration of the device has changed.
+      \item [Device-specific Driver Auxiliary Notification] - bit 2 - the interrupt was
+        asserted because a device-specific event occurred to notify the driver.
     \end{description}
   }
   \hline 
@@ -2121,6 +2123,19 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
     apply to the queue selected by writing to \field{QueueSel}.
   }
   \hline
+  \mmioreg{DeviceAuxNotificationIndex}{Device Auxiliary Notifier}{0xd0}{W}{
+    Writing to this register selects the device auxiliary
+    notification index to which \field{DeviceAuxiliaryNotificationData} apply to.
+    The number of device auxiliary notifications and their purpose is
+    device-specific.
+  }
+  \mmioreg{DeviceAuxNotificationData}{Device Auxiliary Notification Data}{0xd4}{W}{
+    Writing to this register notifies the device that a device auxiliary
+    notification has occured corresponding to the index indicated by
+    \field{DeviceAuxNotificationIndex}. The value written to this register has
+    a device-specific meaning.
+  }
+  \hline
   \mmioreg{ConfigGeneration}{Configuration atomicity value}{0x0fc}{R}{
     Reading from this register returns a value describing a version of the device-specific configuration space (see \field{Config}).
     The driver can then access the configuration space and, when finished, read \field{ConfigGeneration} again.
@@ -2156,6 +2171,9 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
 
 The device MUST NOT access virtual queue contents when \field{QueueReady} is zero (0x0).
 
+The device MUST ignore device auxiliary notifications for invalid device auxiliary notification
+indices.
+
 If VIRTIO_F_RING_RESET has been negotiated, the device MUST present a 0 in
 \field{QueueReset} on reset.
 
@@ -2234,6 +2252,8 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
 Drivers not expecting shared memory MUST NOT use the shared
 memory registers.
 
+Drivers not expecting device auxiliary notifications MUST NOT use the device auxiliary notification register.
+
 Further initialization MUST follow the procedure described in
 \ref{sec:General Initialization And Device Operation / Device Initialization}~\nameref{sec:General Initialization And Device Operation / Device Initialization}.
 
@@ -2287,8 +2307,9 @@ \subsubsection{Notifications From The Device}\label{sec:Virtio Transport Options
 The memory mapped virtio device is using a single, dedicated
 interrupt signal, which is asserted when at least one of the
 bits described in the description of \field{InterruptStatus}
-is set. This is how the device sends a used buffer notification
-or a configuration change notification to the device.
+is set. This is how the device sends a
+used buffer/configuration change/device-specific notification to the
+device.
 
 \drivernormative{\paragraph}{Notifications From The Device}{Virtio Transport Options / Virtio Over MMIO / MMIO-specific Initialization And Device Operation / Notifications From The Device}
 After receiving an interrupt, the driver MUST read
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v3 4/4] vhost-user: add vhost-user device type

[Usama Arif]

The vhost-user device backend facilitates vhost-user device emulation
through vhost-user protocol exchanges and access to shared memory.
Software-defined networking, storage, and other I/O appliances can
provide services through this device.

This device is based on Wei Wang's vhost-pci work.  The virtio
vhost-user device differs from vhost-pci because it is a single virtio
device type that exposes the vhost-user protocol instead of a family of
new virtio device types, one for each vhost-user device type.

This device supports vhost-user backend and vhost-user frontend
reconnection. It also contains an ID so that vhost-user backend programs
can identify a specific device among many without using bus addresses.

virtio-vhost-user makes use of additional resources introduced in earlier
patches including device aux. notifications, driver aux. notifications,
as well as shared memory.

Signed-off-by: Usama Arif <usama.arif@bytedance.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
---
 conformance.tex       |  27 +++-
 content.tex           |   3 +
 introduction.tex      |   3 +
 virtio-vhost-user.tex | 309 ++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 338 insertions(+), 4 deletions(-)
 create mode 100644 virtio-vhost-user.tex

diff --git a/conformance.tex b/conformance.tex
index f6914b0..434cde3 100644
--- a/conformance.tex
+++ b/conformance.tex
@@ -32,8 +32,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Driver Conformance / Memory Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / I2C Adapter Driver Conformance},
 \ref{sec:Conformance / Driver Conformance / SCMI Driver Conformance},
-\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance} or
-\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}.
+\ref{sec:Conformance / Driver Conformance / GPIO Driver Conformance},
+\ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance} or
+\ref{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}
@@ -59,8 +60,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \ref{sec:Conformance / Device Conformance / Memory Device Conformance},
 \ref{sec:Conformance / Device Conformance / I2C Adapter Device Conformance},
 \ref{sec:Conformance / Device Conformance / SCMI Device Conformance},
-\ref{sec:Conformance / Device Conformance / GPIO Device Conformance} or
-\ref{sec:Conformance / Device Conformance / PMEM Device Conformance}.
+\ref{sec:Conformance / Device Conformance / GPIO Device Conformance},
+\ref{sec:Conformance / Device Conformance / PMEM Device Conformance} or
+\ref{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}.
 
     \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}.
   \end{itemize}
@@ -335,6 +337,15 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{drivernormative:Device Types / PMEM Device / Device Initialization}
 \end{itemize}
 
+\conformance{\subsection}{Vhost-user Backend Driver Conformance}\label{sec:Conformance / Driver Conformance / Vhost-user Backend Driver Conformance}
+
+A vhost-user backend driver MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device configuration layout}
+\item \ref{drivernormative:Device Types / Vhost-user Device Backend / Device Initialization}
+\end{itemize}
+
 \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance}
 
 A device MUST conform to the following normative statements:
@@ -623,6 +634,14 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
 \item \ref{devicenormative:Device Types / PMEM Device / Device Operation / Virtqueue return}
 \end{itemize}
 
+\conformance{\subsection}{Vhost-user Backend Device Conformance}\label{sec:Conformance / Device Conformance / Vhost-user Backend Device Conformance}
+
+A Vhost-user backend device MUST conform to the following normative statements:
+
+\begin{itemize}
+\item \ref{devicenormative:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
+\end{itemize}
+
 \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}
 A conformant implementation MUST be either transitional or
 non-transitional, see \ref{intro:Legacy
diff --git a/content.tex b/content.tex
index 8968fcd..5627ba1 100644
--- a/content.tex
+++ b/content.tex
@@ -3167,6 +3167,8 @@ \chapter{Device Types}\label{sec:Device Types}
 \hline
 42         &   RDMA device \\
 \hline
+43         &   vhost-user device backend \ \\
+\hline
 \end{tabular}
 
 Some of the devices above are unspecified by this document,
@@ -7028,6 +7030,7 @@ \subsubsection{Legacy Interface: Framing Requirements}\label{sec:Device
 \input{virtio-scmi.tex}
 \input{virtio-gpio.tex}
 \input{virtio-pmem.tex}
+\input{virtio-vhost-user.tex}
 
 \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits}
 
diff --git a/introduction.tex b/introduction.tex
index 287c5fc..c10f39c 100644
--- a/introduction.tex
+++ b/introduction.tex
@@ -98,6 +98,9 @@ \section{Normative References}\label{sec:Normative References}
 	\phantomsection\label{intro:SEC1}\textbf{[SEC1]} &
     Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000.
 	\newline\url{https://www.secg.org/sec1-v2.pdf}\\
+	\phantomsection\label{intro:Vhost-user Protocol}\textbf{[Vhost-user Protocol]}
+	& Vhost-user Protocol,
+	\newline\url{https://qemu.readthedocs.io/en/latest/interop/vhost-user.html}\\
 
 \end{longtable}
 
diff --git a/virtio-vhost-user.tex b/virtio-vhost-user.tex
new file mode 100644
index 0000000..9677760
--- /dev/null
+++ b/virtio-vhost-user.tex
@@ -0,0 +1,309 @@
+\section{Vhost-user Device Backend}\label{sec:Device Types / Vhost-user Device
+Backend}
+
+The vhost-user device backend facilitates vhost-user device emulation through
+vhost-user protocol exchanges and access to shared memory. Software-defined
+networking, storage, and other I/O appliances can provide services through this
+device.
+
+This section relies on definitions from the \hyperref[intro:Vhost-user
+Protocol]{Vhost-user Protocol}. Knowledge of the vhost-user protocol is a
+prerequisite for understanding this device.
+
+The \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol} was originally
+designed for processes on a single system communicating over UNIX domain
+sockets. The virtio vhost-user backend device and vhost-user backend
+driver together serve to implement the vhost-user backend. The backend driver
+communicates with the frontend via the backend device rather than using
+the UNIX domain socket directly. This allows the backend and frontend to run on
+two separate systems such as a virtual machine and a hypervisor.
+
+Existing vhost-user backend programs that communicate over UNIX domain sockets
+can support the virtio vhost-user device backend without invasive changes
+because the pre-existing vhost-user wire protocol is used.
+
+\subsection{Device ID}\label{sec:Device Types / Vhost-user Device Backend / Device ID}
+  43
+
+\subsection{Virtqueues}\label{sec:Device Types / Vhost-user Device Backend / Virtqueues}
+
+\begin{description}
+\item[0] rxq (device-to-driver vhost-user protocol messages)
+\item[1] txq (driver-to-device vhost-user protocol messages)
+\end{description}
+
+\subsection{Feature bits}\label{sec:Device Types / Vhost-user Device Backend / Feature bits}
+
+No feature bits are defined at this time.
+
+\subsection{Device configuration layout}\label{sec:Device Types / Vhost-user Device Backend / Device configuration layout}
+
+  All fields of this configuration are always available.
+
+\begin{lstlisting}
+struct virtio_vhost_user_config {
+        le32 status;
+#define VIRTIO_VHOST_USER_STATUS_BACKEND_UP (1 << 0)
+#define VIRTIO_VHOST_USER_STATUS_FRONTEND_UP (1 << 1)
+        le32 max_vhost_queues;
+        u8 id[16];
+};
+\end{lstlisting}
+
+\begin{description}
+\item[\field{status}] contains the vhost-user operational status. The default
+    value of this field is 0.
+
+    The driver sets VIRTIO_VHOST_USER_STATUS_BACKEND_UP to indicate readiness for
+    the vhost-user frontend to connect. The vhost-user frontend cannot connect
+    unless the driver has set this bit first.
+
+    When the vhost-user frontend connects, the device sets VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
+    to indicate it.
+
+    When the driver clears VIRTIO_VHOST_USER_STATUS_BACKEND_UP while the
+    vhost-user frontend is connected, the vhost-user frontend is disconnected.
+
+    When the vhost-user frontend disconnects, both
+    VIRTIO_VHOST_USER_STATUS_BACKEND_UP and VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
+    are cleared by the device. Communication can be restarted by the driver
+    setting VIRTIO_VHOST_USER_STATUS_BACKEND_UP again.
+
+    A configuration change notification is sent when the device changes
+    this field, unless a write to the field by the driver caused the change.
+
+\item[\field{max_vhost_queues}] is the maximum number of vhost-user queues
+    supported by this device. This field is always greater than 0.
+
+\item[\field{id}] is the identifier for this device. If the device has no ID
+    then this field contains all zeroes. The ID allows vhost-user backend programs
+    to identify a specific vhost-user device backend among many without relying on bus
+    addresses.
+\end{description}
+
+\drivernormative{\subsubsection}{Device configuration layout}{Device Types / Vhost-user Device Backend / Device configuration layout}
+
+The driver MUST NOT write to device configuration fields other than \field{status}.
+
+The driver MUST NOT set undefined bits in the \field{status} configuration field.
+
+\subsection{Device Initialization}\label{sec:Device Types / Vhost-user Device Backend / Device Initialization}
+
+The driver checks if the virtio features supported by it are provided by the device,
+initializes the rxq/txq virtqueues and then sets VIRTIO_VHOST_USER_STATUS_BACKEND_UP to
+the \field{status} field of the device configuration structure.
+
+\drivernormative{\subsubsection}{Device Initialization}{Device Types / Vhost-user Device Backend / Device Initialization}
+
+The driver SHOULD check the \field{max_vhost_queues} configuration field to
+determine how many queues the vhost-user backend will be able to support.
+
+The driver SHOULD fetch the \field{id} configuration field to allow
+vhost-user backend programs to identify a specific device among many.
+
+The driver SHOULD place at least one buffer in rxq before setting the
+VIRTIO_VHOST_USER_STATUS_BACKEND_UP bit in the \field{status} configuration field.
+
+The driver MUST handle rxq virtqueue notifications that occur before the
+configuration change notification. It is possible that a vhost-user protocol
+message from the vhost-user frontend arrives before the driver has seen the
+configuration change notification for the VIRTIO_VHOST_USER_STATUS_FRONTEND_UP
+\field{status} change.
+
+\subsection{Device Operation}\label{sec:Device Types / Vhost-user Device Backend / Device Operation}
+
+Device operation consists of operating request queues and response queues.
+
+\subsubsection{Device Operation: Request Queues}\label{sec:Device Types / Vhost-user Device Backend / Device Operation / Device Operation: RX/TX Queues}
+
+The driver receives vhost-user protocol messages from the vhost-user frontend on
+rxq. The driver sends responses to the vhost-user frontend on txq.
+
+The driver sends backend-initiated requests on txq. The driver receives
+responses from the vhost-user frontend on rxq.
+
+All virtqueues offer in-order guaranteed delivery semantics for vhost-user
+protocol messages.
+
+Each buffer is a vhost-user protocol message as defined by the
+\hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}.
+
+The appropriate size of rxq buffers is at least as large as the largest message
+defined by the \hyperref[intro:Vhost-user Protocol]{Vhost-user Protocol}
+standard version that the driver supports. If the vhost-user frontend sends a
+message that is too large for an rxq buffer, then DEVICE_NEEDS_RESET is set and
+the driver must reset the device.
+
+File descriptor passing is handled differently by the vhost-user device
+backend. When a frontend-initiated message is received that carries one or more file
+descriptors according to the vhost-user protocol, additional device resources
+become available to the driver.
+
+\subsection{Additional Device Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources}
+
+The vhost-user device backend uses the following facilities from virtio device
+\ref{sec:Basic Facilities of a Virtio Device} for the vhost-user frontend and
+backend to exchange notifications and data through the device:
+
+\begin{description}
+  \item[Device auxiliary notification] \ref{sec:Basic Facilities of a Virtio Device / Notifications}
+The driver signals the vhost-user frontend through device auxiliary notifications. The signal does not
+carry any data, it is purely an event.
+  \item[Driver auxiliary notification] \ref{sec:Basic Facilities of a Virtio Device / Notifications}
+The vhost-user frontend signals the driver for events besides virtqueue activity
+and configuration changes by sending driver auxiliary notification.
+  \item[Shared memory] \ref{sec:Basic Facilities of a Virtio Device / Shared Memory Regions}
+The vhost-user frontend gives access to memory that can be mapped by the driver.
+\end{description}
+
+\subsubsection{Device auxiliary notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Device auxiliary notifications}
+
+The vhost-user device backend provides all (or part) of the following device
+auxiliary notifications:
+
+\begin{description}
+\item[0] Vring call for vhost-user queue 0
+\item[\ldots]
+\item[N-1] Vring call for vhost-user queue N-1
+\item[N] Vring err for vhost-user queue 0
+\item[\ldots]
+\item[2N-1] Vring err for vhost-user queue N-1
+\item[2N] Log
+\end{description}
+
+where N is the number of the vhost-user virtqueues.
+
+\subsubsection{Driver auxiliary notifications}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Driver auxiliary notifications}
+
+The vhost-user device backend provides all (or part) of the following driver
+auxiliary notifications:
+
+\begin{description}
+\item[0] Vring kick for vhost-user queue 0
+\item[\ldots]
+\item[N-1] Vring kick for vhost-user queue N-1
+\end{description}
+
+where N is the number of the vhost-user virtqueues.
+
+\subsubsection{Shared Memory}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory}
+
+The vhost-user device backend provides all (or part) of the following shared memory.
+The device MUST assign shmids' as per the following table:
+
+\begin{tabular}{|l|l|}
+  \hline
+  Virtio Vhost-user shared memory region & Virtio shmid \\
+  \hline \hline
+  Reserved (invalid) & 0 \\
+  \hline
+  Vhost-user memory & 1 \\
+  \hline
+  Log memory & 2 \\
+  \hline
+  Inflight memory & 3 \\
+  \hline
+  Reserved for future use & 4 and greater \\
+  \hline
+\end{tabular}
+
+The log memory is only available if VHOST_USER_PROTOCOL_F_LOG_SHMFD has been
+been negotiated between the frontend and backend. Similarly, inflight memory
+is only available if VHOST_USER_PROTOCOL_F_INFLIGHT_SHMFD has been negotiated.
+
+When the VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS protocol feature is not
+present, the vhost-user device backend provides the following vhost-user
+memory regions as a single shared memory region with shmid 1:
+
+\begin{description}
+\item[0] Vhost-user memory region 0
+\item[1] Vhost-user memory region 1
+\item[\ldots]
+\item[M-1] Vhost-user memory region M-1
+\end{description}
+
+where M is the total number of vhost-user memory regions that can be mapped
+by the driver using the VHOST_USER_SET_MEM_TABLE protocol message.
+
+When the VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS protocol feature has been
+negotiated, the vhost-user device backend still provides all the vhost-user
+memory regions as a single shared memory, however the number of vhost-user
+memory regions exported in the single shared memory is dynamic, i.e.
+VHOST_USER_ADD_MEM_REG message adds a vhost-user memory regions to the
+shared memory that can be mapped by the driver and VHOST_USER_REM_MEM_REG
+removes the region from the shared memory.
+One possible method the shared memory could be managed with is memory
+compaction. In this method, the device would remove the vhost-user memory
+region from the shared memory when VHOST_USER_REM_MEM_REG is received for the
+specific vhost-user memory region. When VHOST_USER_ADD_MEM_REG protocol
+message is received, the device would run memory compaction on the shared memory
+to remove any fragmentation caused by previously received VHOST_USER_REM_MEM_REG
+protocol messages before adding the new vhost-user memory region to the
+shared memory. The driver would then make the corresponding shifts in the mmap
+addresses of the shifted memory regions.
+
+\devicenormative{\paragraph}{Shared Memory layout}{Device Types / Vhost-user Device Backend / Additional Device Resources / Shared Memory layout}
+
+The device exports all memory regions reported by the vhost-user frontend as a
+single shared memory\ref{sec:Basic Facilities of a Virtio Device /
+Shared Memory Regions}.
+
+The size of this shared memory exported by the device MUST be at least
+as much as the sum of the sizes of all the memory regions reported by the
+vhost-user frontend.
+
+When the VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS protocol feature is not
+present, the vhost-user memory regions exported by the device MUST be laid out
+contiguously in the same order in which they are reported by the frontend with
+vhost-user messages.
+The offsets in which the memory regions are mapped inside the shared memory
+region MUST be the following:
+
+\begin{description}
+\item[0] Offset for vhost-user memory region 0
+\item[SIZE0] Offset for vhost-user memory region 1
+\item[\ldots]
+\item[SIZE0 + SIZE1 + \ldots] Offset for vhost-user memory region M
+\end{description}
+
+where SIZEi is the size of the vhost-user memory region i.
+
+When the VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS protocol feature has been
+negotiated, the memory regions exported by the device might not be contiguously
+laid out.
+
+\subsubsection{Availability of Additional Resources}\label{sec:Device Types / Vhost-user Device Backend / Additional Device Resources / Availability of Additional Resources}
+
+The following vhost-user protocol messages convey access to additional device
+resources:
+
+\begin{description}
+\item[VHOST_USER_SET_MEM_TABLE] Contents of vhost-user memory regions are made
+available to the driver in the Shared Memory Region. Region contents are laid out
+in the same order as the vhost-user memory region list.
+\item[VHOST_USER_ADD_MEM_REG] Contents of the vhost-user memory region are made
+available to the driver in the Shared Memory Region.
+\item[VHOST_USER_REM_MEM_REG] Contents of the vhost-user memory region are no
+longer available to the driver in the Shared Memory Region.
+\item[VHOST_USER_SET_LOG_BASE] Contents of the log memory region are available
+to the driver in the Shared Memory Region.
+\item[VHOST_USER_SET_LOG_FD] The log doorbell is available to the driver.
+Writes to the log doorbell before this message is received produce no effect.
+\item[VHOST_USER_GET_INFLIGHT_FD] Contents of the inflight memory region are available
+in the Shared Memory Region.
+\item[VHOST_USER_SET_VRING_KICK] The vring kick notification for this queue is
+available to the driver. The first notification may occur before the driver has
+processed this message. If a kick arrives before the corresponding
+VHOST_USER_SET_VRING_KICK message has been processed, the vhost-user backend
+should ignore the unknown kick and peek at the vring when
+VHOST_USER_SET_VRING_ENABLE is processed.
+\item[VHOST_USER_SET_VRING_CALL] The vring call device auxiliary notification
+for this queue is available to the driver. Writes to the vring call device
+auxiliary notification before this message is received produce no effect.
+\item[VHOST_USER_SET_VRING_ERR] The vring err device auxiliary notification for
+this queue is available to the driver. Writes to the vring err device auxiliary
+notification before this message is received produce no effect.
+\item[VHOST_USER_SET_SLAVE_REQ_FD] The driver may send vhost-user protocol
+backend messages on txq. Backend-initiated messages put onto txq before this
+message is received are discarded by the device.
+\end{description}
-- 
2.25.1


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v3 1/4] content: Introduce driver/device auxiliary notifications

[Cornelia Huck]

On Fri, Oct 07 2022, Usama Arif <usama.arif@bytedance.com> wrote:

> Driver auxiliary notifications allow the device to send notifications
> other than configuration changes and used buffer notifications to the
> driver, these are optional and their meaning is device-specific.
>
> Device auxiliary notifications allow the driver to send notifications
> other than available buffer notifications to the device for example
> through a device register, these are optional and their meaning is
> device-specific.
>
> These device-specific notifications are needed later when adding support
> for virtio-vhost-user device.
>
> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  content.tex | 35 ++++++++++++++++++++++-------------
>  1 file changed, 22 insertions(+), 13 deletions(-)
>
> diff --git a/content.tex b/content.tex
> index e863709..49f9f2a 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -160,29 +160,38 @@ \subsection{Legacy Interface: A Note on Feature
>  Specification text within these sections generally does not apply
>  to non-transitional devices.
>  
> -\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
> -/ Notifications}
> +\section{Notifications}\label{sec:Basic Facilities of a Virtio Device / Notifications}
>  
>  The notion of sending a notification (driver to device or device
>  to driver) plays an important role in this specification. The
>  modus operandi of the notifications is transport specific.
>  
> -There are three types of notifications: 
> +There are five types of notifications:
>  \begin{itemize}
>  \item configuration change notification
>  \item available buffer notification
> -\item used buffer notification. 
> +\item used buffer notification
> +\item driver auxiliary notification
> +\item device auxiliary notification

Should we postfix the two new entries with "(optional)"?

>  \end{itemize}
>  
> -Configuration change notifications and used buffer notifications are sent
> -by the device, the recipient is the driver. A configuration change
> -notification indicates that the device configuration space has changed; a
> -used buffer notification indicates that a buffer may have been made used
> -on the virtqueue designated by the notification.
> -
> -Available buffer notifications are sent by the driver, the recipient is
> -the device. This type of notification indicates that a buffer may have
> -been made available on the virtqueue designated by the notification.
> +Configuration change notifications, used buffer notifications and
> +driver auxiliary notifications are sent by the device,
> +the recipient is the driver. A configuration change notification indicates
> +that the device configuration space has changed; a used buffer notification
> +indicates that a buffer may have been made used on the virtqueue designated
> +by the notification; driver auxiliary notifications allow the
> +device to send notifications other than configuration changes and used
> +buffer notifications to the driver, these are optional and their meaning
> +is device-specific.
> +
> +Available buffer notifications and device auxiliary notifications
> +are sent by the driver, the recipient is the device. Available buffer
> +notifications indicate that a buffer may have been made available on the
> +virtqueue designated by the notification; device auxiliary
> +notifications allow the driver to send notifications other than available
> +buffer notifications to the device for example through a device register, these
> +are optional and their meaning is device-specific.
>  
>  The semantics, the transport-specific implementations, and other
>  important aspects of the different notifications are specified in detail


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] [PATCH v3 1/4] content: Introduce driver/device auxiliary notifications

[Usama Arif]

On 11/10/2022 10:20, Cornelia Huck wrote:
> On Fri, Oct 07 2022, Usama Arif <usama.arif@bytedance.com> wrote:
> 
>> Driver auxiliary notifications allow the device to send notifications
>> other than configuration changes and used buffer notifications to the
>> driver, these are optional and their meaning is device-specific.
>>
>> Device auxiliary notifications allow the driver to send notifications
>> other than available buffer notifications to the device for example
>> through a device register, these are optional and their meaning is
>> device-specific.
>>
>> These device-specific notifications are needed later when adding support
>> for virtio-vhost-user device.
>>
>> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> ---
>>   content.tex | 35 ++++++++++++++++++++++-------------
>>   1 file changed, 22 insertions(+), 13 deletions(-)
>>
>> diff --git a/content.tex b/content.tex
>> index e863709..49f9f2a 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -160,29 +160,38 @@ \subsection{Legacy Interface: A Note on Feature
>>   Specification text within these sections generally does not apply
>>   to non-transitional devices.
>>   
>> -\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
>> -/ Notifications}
>> +\section{Notifications}\label{sec:Basic Facilities of a Virtio Device / Notifications}
>>   
>>   The notion of sending a notification (driver to device or device
>>   to driver) plays an important role in this specification. The
>>   modus operandi of the notifications is transport specific.
>>   
>> -There are three types of notifications:
>> +There are five types of notifications:
>>   \begin{itemize}
>>   \item configuration change notification
>>   \item available buffer notification
>> -\item used buffer notification.
>> +\item used buffer notification
>> +\item driver auxiliary notification
>> +\item device auxiliary notification
> 
> Should we postfix the two new entries with "(optional)"?
>

I can add that in the next version I send. I have "these are optional 
and their meaning is device-specific." in the description below as well.

>>   \end{itemize}
>>   
>> -Configuration change notifications and used buffer notifications are sent
>> -by the device, the recipient is the driver. A configuration change
>> -notification indicates that the device configuration space has changed; a
>> -used buffer notification indicates that a buffer may have been made used
>> -on the virtqueue designated by the notification.
>> -
>> -Available buffer notifications are sent by the driver, the recipient is
>> -the device. This type of notification indicates that a buffer may have
>> -been made available on the virtqueue designated by the notification.
>> +Configuration change notifications, used buffer notifications and
>> +driver auxiliary notifications are sent by the device,
>> +the recipient is the driver. A configuration change notification indicates
>> +that the device configuration space has changed; a used buffer notification
>> +indicates that a buffer may have been made used on the virtqueue designated
>> +by the notification; driver auxiliary notifications allow the
>> +device to send notifications other than configuration changes and used
>> +buffer notifications to the driver, these are optional and their meaning
>> +is device-specific.
>> +
>> +Available buffer notifications and device auxiliary notifications
>> +are sent by the driver, the recipient is the device. Available buffer
>> +notifications indicate that a buffer may have been made available on the
>> +virtqueue designated by the notification; device auxiliary
>> +notifications allow the driver to send notifications other than available
>> +buffer notifications to the device for example through a device register, these
>> +are optional and their meaning is device-specific.
>>   
>>   The semantics, the transport-specific implementations, and other
>>   important aspects of the different notifications are specified in detail
> 

[virtio-dev] Re: [PATCH v3 0/4] Introduce aux. notifications and virtio-vhost-user

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4705 bytes --]

On Fri, Oct 07, 2022 at 05:56:39PM +0100, Usama Arif wrote:
> Hi,
> 
> This patch series introduces device and driver auxiliary notifications
> as a new set of virtio device resources, as well as vhost-user device
> backend that uses these resources.
> 
> Driver auxiliary notifications allow the device to send notifications
> other than configuration changes and used buffer notifications to the
> driver, these are optional and their meaning is **device-specific**.
> 
> Device auxiliary notifcations allow the driver to send notifcations
> other than available buffer notifications to the device for example
> through a device register, these are optional and their meaning is
> **device-specific**.
> 
> These resources are used in the last patch by the virtio-vhost-user
> device in order to send/receive notifications to/from the driver
> regarding activity on the vhost-user virtqueues. By standardizing
> these resources as standalone virtio device resources, other future
> devices will be able to use them as well.

Hi Michael,
Do you have time to review this? I have been too involved in
virtio-vhost-user in the past to do a truly independent review and your
VIRTIO + vhost expertise would be perfect here.

Thanks,
Stefan

> 
> The last patch introduces the vhost-user device backend which facilitates
> vhost-user device emulation through vhost-user protocol exchanges and
> access to shared memory. Software-defined networking, storage, and other
> I/O appliances can provide services through this device.
> 
> This device is based on Wei Wang's vhost-pci work. The virtio-vhost-user
> device differs from vhost-pci because it is a single virtio
> device type that exposes the vhost-user protocol instead of a family of
> new virtio device types, one for each vhost-user device type.
> 
> A HTML version with the changes is available at [1].
> 
> For more information about virtio-vhost-user, see [2].
> 
> These patches are based on the work initially done by Stefan Hajnoczi [2]
> and continued by Nikos Dragazis [3], [4].
> 
> A working prototype implementing this spec can be reproduced using
> instructions in [5] utilizing QEMU [6] and DPDK [7].
> This is also based on the work initially done by Stefan
> and continued later by Nikos. The prototype code uses the terms
> "doorbell"/"device-specific notification" and "master"/"slave" instead of
> "device auxiliary notification"/"driver auxiliary notification" and
> "frontend"/"backend". This is based on older work, however, their
> functionality is the same. If these virtio-spec changes get approved,
> I will refractor the patches for QEMU/DPDK and send them for review
> according to the final patches in the respective mailing list.
> 
> Thanks and looking forward to your response!
> Usama
> 
> [1] https://uarif1.github.io/vvu/v3/virtio-v1.2-cs01
> [2] https://wiki.qemu.org/Features/VirtioVhostUser
> [3] https://ndragazis.github.io/dpdk-vhost-vvu-demo.html
> [4] https://lists.oasis-open.org/archives/virtio-dev/202005/msg00132.html
> [5] https://uarif1.github.io/vvu/dpdk-vvu-instructions
> [6] https://github.com/uarif1/qemu/tree/vvu
> [7] https://github.com/uarif1/dpdk/tree/vvu
> ----------
> v2->v3:
> Mostly addressing comments made by Michael S. Tsirkin:
> - Give a clearer relation between backend driver, backend device and frontend at
>   beginning of the Vhost-user Device Backend section.
> - change uuid to id.
> - Add information about inflight memory.
> 
> v1->v2:
> - Shortened device aux. notification section
> - Make data width of device aux. notifications device specific and not
>   limited to 2 bytes.
> - Added information about the minimum number of MSIX vectors.
> - Split virtio-mmio implementation of driver aux. notifications into 2
>   registers, DeviceAuxNotificationIndex and DeviceAuxNotificationData.
> - Made the shared memory section in virtio vhost-user clearer and added
>   the shared memory would look like when
>   VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS has been negotiated.
> 
> Usama Arif (4):
>   content: Introduce driver/device auxiliary notifications
>   content: Introduce driver/device aux. notification cfg type for PCI
>   content: Introduce driver/device auxiliary notifications for MMIO
>   vhost-user: add vhost-user device type
> 
>  conformance.tex       |  29 +++-
>  content.tex           | 254 +++++++++++++++++++++++++++++-----
>  introduction.tex      |   3 +
>  virtio-vhost-user.tex | 309 ++++++++++++++++++++++++++++++++++++++++++
>  4 files changed, 554 insertions(+), 41 deletions(-)
>  create mode 100644 virtio-vhost-user.tex
> 
> -- 
> 2.25.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v3 1/4] content: Introduce driver/device auxiliary notifications

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4605 bytes --]

On Fri, Oct 07, 2022 at 05:56:40PM +0100, Usama Arif wrote:
> Driver auxiliary notifications allow the device to send notifications
> other than configuration changes and used buffer notifications to the
> driver, these are optional and their meaning is device-specific.
> 
> Device auxiliary notifications allow the driver to send notifications
> other than available buffer notifications to the device for example
> through a device register, these are optional and their meaning is
> device-specific.
> 
> These device-specific notifications are needed later when adding support
> for virtio-vhost-user device.
> 
> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  content.tex | 35 ++++++++++++++++++++++-------------
>  1 file changed, 22 insertions(+), 13 deletions(-)
> 
> diff --git a/content.tex b/content.tex
> index e863709..49f9f2a 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -160,29 +160,38 @@ \subsection{Legacy Interface: A Note on Feature
>  Specification text within these sections generally does not apply
>  to non-transitional devices.
>  
> -\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
> -/ Notifications}
> +\section{Notifications}\label{sec:Basic Facilities of a Virtio Device / Notifications}
>  
>  The notion of sending a notification (driver to device or device
>  to driver) plays an important role in this specification. The
>  modus operandi of the notifications is transport specific.
>  
> -There are three types of notifications: 
> +There are five types of notifications:
>  \begin{itemize}
>  \item configuration change notification
>  \item available buffer notification
> -\item used buffer notification. 
> +\item used buffer notification
> +\item driver auxiliary notification
> +\item device auxiliary notification
>  \end{itemize}
>  
> -Configuration change notifications and used buffer notifications are sent
> -by the device, the recipient is the driver. A configuration change
> -notification indicates that the device configuration space has changed; a
> -used buffer notification indicates that a buffer may have been made used
> -on the virtqueue designated by the notification.
> -
> -Available buffer notifications are sent by the driver, the recipient is
> -the device. This type of notification indicates that a buffer may have
> -been made available on the virtqueue designated by the notification.
> +Configuration change notifications, used buffer notifications and
> +driver auxiliary notifications are sent by the device,
> +the recipient is the driver. A configuration change notification indicates
> +that the device configuration space has changed; a used buffer notification
> +indicates that a buffer may have been made used on the virtqueue designated
> +by the notification; driver auxiliary notifications allow the
> +device to send notifications other than configuration changes and used
> +buffer notifications to the driver, these are optional and their meaning
> +is device-specific.

The meaning of "optional" is confusing and I suggest dropping it.

Auxiliary notifications are not used by all device types. In that sense
they are "optional".

But a device types that do use them, like virtio-vhost-user, may require
them. You cannot implement a virtio-vhost-user device without auxiliary
notifications, so they are mandatory (not "optional") for that device
type.

Finally, it is also possible for a device type to define a feature bit
that enables auxiliary notifications. Drivers/devices that don't support
auxiliary notifications leave the feature bit cleared and still work
properly. In this case they are truly "optional".

"Optional" is confusing. Just saying "their meaning is device-specific"
is enough.

> +
> +Available buffer notifications and device auxiliary notifications
> +are sent by the driver, the recipient is the device. Available buffer
> +notifications indicate that a buffer may have been made available on the
> +virtqueue designated by the notification; device auxiliary
> +notifications allow the driver to send notifications other than available
> +buffer notifications to the device for example through a device register, these
> +are optional and their meaning is device-specific.
>  
>  The semantics, the transport-specific implementations, and other
>  important aspects of the different notifications are specified in detail
> -- 
> 2.25.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v3 2/4] content: Introduce driver/device aux. notification cfg type for PCI

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 15605 bytes --]

On Fri, Oct 07, 2022 at 05:56:41PM +0100, Usama Arif wrote:
> This includes the PCI device conformances for these notification
> capabilities.
> 
> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

You can keep this, but please see comments below.

> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
> ---
>  conformance.tex |   2 +
>  content.tex     | 191 ++++++++++++++++++++++++++++++++++++++++++------
>  2 files changed, 171 insertions(+), 22 deletions(-)
> 
> diff --git a/conformance.tex b/conformance.tex
> index c3c1d3e..f6914b0 100644
> --- a/conformance.tex
> +++ b/conformance.tex
> @@ -370,6 +370,8 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets}
>  \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
>  \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
>  \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device-specific configuration}
> +\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
> +\item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
>  \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
>  \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / PCI configuration access capability}
>  \item \ref{devicenormative:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / Non-transitional Device With Legacy Driver}
> diff --git a/content.tex b/content.tex
> index 49f9f2a..33362b7 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -719,6 +719,8 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
>  \item ISR Status
>  \item Device-specific configuration (optional)
>  \item PCI configuration access
> +\item Driver auxiliary notifications (optional)
> +\item Device auxiliary notifications (optional)
>  \end{itemize}
>  
>  Each structure can be mapped by a Base Address register (BAR) belonging to
> @@ -765,19 +767,23 @@ \subsection{Virtio Structure PCI Capabilities}\label{sec:Virtio Transport Option
>  
>  \begin{lstlisting}
>  /* Common configuration */
> -#define VIRTIO_PCI_CAP_COMMON_CFG        1
> +#define VIRTIO_PCI_CAP_COMMON_CFG             1
>  /* Notifications */
> -#define VIRTIO_PCI_CAP_NOTIFY_CFG        2
> +#define VIRTIO_PCI_CAP_NOTIFY_CFG             2
>  /* ISR Status */
> -#define VIRTIO_PCI_CAP_ISR_CFG           3
> +#define VIRTIO_PCI_CAP_ISR_CFG                3
>  /* Device specific configuration */
> -#define VIRTIO_PCI_CAP_DEVICE_CFG        4
> +#define VIRTIO_PCI_CAP_DEVICE_CFG             4
>  /* PCI configuration access */
> -#define VIRTIO_PCI_CAP_PCI_CFG           5
> +#define VIRTIO_PCI_CAP_PCI_CFG                5
> +/* Device auxiliary notification */
> +#define VIRTIO_PCI_CAP_DEVICE_AUX_NOTIFY_CFG  6
> +/* Driver auxiliary notification */
> +#define VIRTIO_PCI_CAP_DRIVER_AUX_NOTIFY_CFG  7
>  /* Shared memory region */
> -#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG 8
> +#define VIRTIO_PCI_CAP_SHARED_MEMORY_CFG      8
>  /* Vendor-specific data */
> -#define VIRTIO_PCI_CAP_VENDOR_CFG        9
> +#define VIRTIO_PCI_CAP_VENDOR_CFG             9
>  \end{lstlisting}
>  
>          Any other value is reserved for future use.
> @@ -1158,11 +1164,11 @@ \subsubsection{ISR status capability}\label{sec:Virtio Transport Options / Virti
>  The ISR bits allow the driver to distinguish between device-specific configuration
>  change interrupts and normal virtqueue interrupts:
>  
> -\begin{tabular}{ |l||l|l|l| }
> +\begin{tabular}{ |l||l|l|l|l| }
>  \hline
> -Bits       & 0                               & 1               &  2 to 31 \\
> +Bits       & 0                               & 1                                & 2                      & 3 to 31 \\
>  \hline
> -Purpose    & Queue Interrupt  & Device Configuration Interrupt & Reserved \\
> +Purpose    & Queue Interrupt  & Device Configuration Interrupt & Driver Auxiliary Notification Interrupt & Reserved \\
>  \hline
>  \end{tabular}
>  
> @@ -1208,6 +1214,135 @@ \subsubsection{Device-specific configuration}\label{sec:Virtio Transport Options
>  
>  The \field{offset} for the device-specific configuration MUST be 4-byte aligned.
>  
> +\subsubsection{Device auxiliary notification capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
> +
> +The device auxiliary notification \ref{sec:Basic Facilities of a Virtio Device / Notifications}
> +location is found using the VIRTIO_PCI_CAP_DEVICE_AUX_NOTIFY_CFG capability. This
> +capability is immediately followed by an additional field, like so:
> +
> +\begin{lstlisting}
> +struct virtio_pci_dev_aux_notification_cap {
> +        struct virtio_pci_cap cap;
> +        le32 dev_aux_notification_off_multiplier;
> +};
> +\end{lstlisting}
> +
> +The device auxiliary notification address within a BAR is calculated as follows:
> +
> +\begin{lstlisting}
> +cap.offset + dev_aux_notification_idx * dev_aux_notification_off_multiplier
> +\end{lstlisting}
> +
> +The \field{cap.offset} and \field{dev_aux_notification_off_multiplier} are taken
> +from the device auxiliary notification capability structure above, and the
> +\field{dev_aux_notification_idx} is the device auxiliary notification index.
> +There is no restriction for the mapping between device auxiliary notifications
> +and dev_aux_notification_idx. The number of device auxiliary notifications and
> +their purpose is device-specific.
> +
> +\devicenormative{\paragraph}{Device auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Device auxiliary notification capability}
> +
> +The data width of the device auxiliary notifications is device specific.
> +
> +For devices that provide a data width of 1 byte:
> +
> +The \field{cap.offset} MUST be byte aligned.

How can it not be byte-aligned? The offset field is in units of bytes. I
suggest dropping this sentence.

> +
> +The value \field{cap.length} presented by the device MUST be at least 1
> +and MUST be large enough to support device auxiliary notification offsets for all supported
> +device auxiliary notifications in all possible configurations.
> +
> +The value \field{cap.length} presented by the device MUST satisfy:
> +
> +\begin{lstlisting}
> +cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 1
> +\end{lstlisting}
> +
> +where \field{max_dev_aux_notification_idx} is the maximum device auxiliary
> +notification index and is dependent on the device.
> +
> +For devices that provide a data width of 2 bytes:
> +
> +The \field{cap.offset} MUST be 2-byte aligned.
> +
> +The device MUST either present \field{dev_aux_notification_off_multiplier} as an
> +even power of 2, or present \field{dev_aux_notification_off_multiplier} as 0.
> +
> +The value \field{cap.length} presented by the device MUST be at least 2
> +and MUST be large enough to support device auxiliary notification offsets for
> +all supported device auxiliary notifications in all possible configurations.
> +
> +The value \field{cap.length} presented by the device MUST satisfy:
> +
> +\begin{lstlisting}
> +cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 2
> +\end{lstlisting}
> +
> +where \field{max_dev_aux_notification_idx} is the maximum device auxiliary
> +notification index and is dependent on the device.
> +
> +For devices that provide a data width of 4 bytes:
> +
> +The \field{cap.offset} MUST be 4-byte aligned.
> +
> +The device MUST either present \field{dev_aux_notification_off_multiplier} as a
> +number that is a power of 2 that is also a multiple 4, or present
> +\field{dev_aux_notification_off_multiplier} as 0.
> +
> +The value \field{cap.length} presented by the device MUST be at least 4
> +and MUST be large enough to support device auxiliary notification offsets for all supported
> +device auxiliary notifications in all possible configurations.
> +
> +The value \field{cap.length} presented by the device MUST satisfy:
> +
> +\begin{lstlisting}
> +cap.length >= max_dev_aux_notification_idx * dev_aux_notification_off_multiplier + 4
> +\end{lstlisting}
> +
> +where \field{max_dev_aux_notification_idx} is the maximum device auxiliary
> +notification index and is dependent on the device.
> +
> +\subsubsection{Driver auxiliary notification capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
> +
> +The Driver auxiliary notification
> +\ref{sec:Basic Facilities of a Virtio Device / Notifications} location
> +is found using the VIRTIO_PCI_CAP_DRIVER_AUX_NOTIFY_CFG capability. The
> +driver auxiliary notification structure allows MSI-X vectors to be
> +configured for notification interrupts. If MSI-X is not available, bit 2
> +of the ISR status \ref{sec:Virtio Transport Options / Virtio Over PCI
> +Bus / PCI Device Layout / ISR status capability} indicates that a
> +driver auxiliary notification occurred. If MSI-X is not available,
> +it is not possible to determine which driver auxiliary notification occured,
> +hence only a single notification is supported.
> +
> +The driver auxiliary notification structure is the following:
> +
> +\begin{lstlisting}
> +struct virtio_pci_driver_aux_notification_cfg {
> +	le16 driver_aux_notification_select;              /* read-write */
> +	le16 driver_aux_notification_msix_vector;         /* read-write */
> +};
> +\end{lstlisting}
> +
> +The driver indicates which notification is of interest by writing the
> +\field{driver_aux_notification_select} field. The driver then writes the MSI-X
> +vector or VIRTIO_MSI_NO_VECTOR to \field{driver_aux_notification_msix_vector} to
> +change the MSI-X vector for that notification.
> +
> +The mapping between notifications and notification indices is
> +device-specific. The total number of notifications is also
> +device-specific.
> +
> +\devicenormative{\paragraph}{Driver auxiliary notification capability}{Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Driver auxiliary notification capability}
> +
> +Device MUST ignore writes to \field{driver_aux_notification_msix_vector} if the
> +value written to \field{driver_aux_notification_select} is not a valid notification
> +index.
> +
> +Device MUST return VIRTIO_MSI_NO_VECTOR for reads from
> +\field{driver_aux_notification_msix_vector} if the value written to
> +\field{driver_aux_notification_select} is not a valid notification index.
> +
>  \subsubsection{Shared memory capability}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI Device Layout / Shared memory capability}
>  
>  Shared memory regions \ref{sec:Basic Facilities of a Virtio
> @@ -1519,15 +1654,17 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
>  \paragraph{MSI-X Vector Configuration}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
>  
>  When MSI-X capability is present and enabled in the device
> -(through standard PCI configuration space) \field{config_msix_vector} and \field{queue_msix_vector} are used to map configuration change and queue
> -interrupts to MSI-X vectors. In this case, the ISR Status is unused.
> +(through standard PCI configuration space) \field{config_msix_vector},
> +\field{queue_msix_vector} and \field{driver_aux_notification_msix_vector} are used
> +to map configuration change, queue and device-specific interrupts to
> +MSI-X vectors respectively. In this case, the ISR Status is unused.
>  
>  Writing a valid MSI-X Table entry number, 0 to 0x7FF, to
> -\field{config_msix_vector}/\field{queue_msix_vector} maps interrupts triggered
> -by the configuration change/selected queue events respectively to
> -the corresponding MSI-X vector. To disable interrupts for an
> -event type, the driver unmaps this event by writing a special NO_VECTOR
> -value:
> +\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}
> +maps interrupts triggered by the configuration change/selected
> +queue/device-specific events respectively to the corresponding MSI-X
> +vector. To disable interrupts for an event type, the driver unmaps this
> +event by writing a special NO_VECTOR value:
>  
>  \begin{lstlisting}
>  /* Vector value used to disable MSI for queue */
> @@ -1541,6 +1678,14 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
>  
>  A device that has an MSI-X capability SHOULD support at least 2
>  and at most 0x800 MSI-X vectors.
> +
> +A device that has an MSI-X capability MUST support atleast:

"at least"

> +\begin{lstlisting}
> +num_msix_vectors >= 1 /* config_msix_vector */ + num_vqs /* queue_msix_vectors */ + num_driver_aux_notifications
> +\end{lstlisting}
> +where num_vqs is the number of virtqueues and num_driver_aux_notifications is
> +the number of driver auxiliary notifications.

Is this a new requirement? I'm not aware of any limit on num_vqs vs
MSI-X vectors. I think it's possible to have a device that supports more
vqs than MSI-X vectors. We probably shouldn't introduce a requirement
that makes existing devices non-compliant.

> +
>  Device MUST report the number of vectors supported in
>  \field{Table Size} in the MSI-X Capability as specified in
>  \hyperref[intro:PCI]{[PCI]}.
> @@ -1554,16 +1699,18 @@ \subsubsection{Device Initialization}\label{sec:Virtio Transport Options / Virti
>  vector 0 to MSI-X \field{Table Size}.
>  Device MUST support unmapping any event type.
>  
> -The device MUST return vector mapped to a given event,
> -(NO_VECTOR if unmapped) on read of \field{config_msix_vector}/\field{queue_msix_vector}.
> -The device MUST have all queue and configuration change
> -events are unmapped upon reset.
> +The device MUST return the vector mapped to a given event,
> +(NO_VECTOR if unmapped) on read of
> +\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}.
> +The device MUST have all queue/configuration change/device-specific
> +events unmapped upon reset.
>  
>  Devices SHOULD NOT cause mapping an event to vector to fail
>  unless it is impossible for the device to satisfy the mapping
>  request.  Devices MUST report mapping
>  failures by returning the NO_VECTOR value when the relevant
> -\field{config_msix_vector}/\field{queue_msix_vector} field is read. 
> +\field{config_msix_vector}/\field{queue_msix_vector}/\field{driver_aux_notification_msix_vector}
> +field is read.
>  
>  \drivernormative{\subparagraph}{MSI-X Vector Configuration}{Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Device Initialization / MSI-X Vector Configuration}
>  
> -- 
> 2.25.1
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [PATCH v3 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 495 bytes --]

On Fri, Oct 07, 2022 at 05:56:42PM +0100, Usama Arif wrote:
> This includes the additions to the corresponding device and driver
> conformances.
> 
> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>

You can keep this. One thought:

I realized that virtio-mmio support won't be very useful until
virtio-mmio gets MSI-X support because drivers currently cannot
differentiate between device auxiliary notifications.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

[virtio-dev] Re: [External] Re: [PATCH v3 1/4] content: Introduce driver/device auxiliary notifications

[Usama Arif]

On 27/10/2022 21:05, Stefan Hajnoczi wrote:
> On Fri, Oct 07, 2022 at 05:56:40PM +0100, Usama Arif wrote:
>> Driver auxiliary notifications allow the device to send notifications
>> other than configuration changes and used buffer notifications to the
>> driver, these are optional and their meaning is device-specific.
>>
>> Device auxiliary notifications allow the driver to send notifications
>> other than available buffer notifications to the device for example
>> through a device register, these are optional and their meaning is
>> device-specific.
>>
>> These device-specific notifications are needed later when adding support
>> for virtio-vhost-user device.
>>
>> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Signed-off-by: Nikos Dragazis <ndragazis@arrikto.com>
>> ---
>>   content.tex | 35 ++++++++++++++++++++++-------------
>>   1 file changed, 22 insertions(+), 13 deletions(-)
>>
>> diff --git a/content.tex b/content.tex
>> index e863709..49f9f2a 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -160,29 +160,38 @@ \subsection{Legacy Interface: A Note on Feature
>>   Specification text within these sections generally does not apply
>>   to non-transitional devices.
>>   
>> -\section{Notifications}\label{sec:Basic Facilities of a Virtio Device
>> -/ Notifications}
>> +\section{Notifications}\label{sec:Basic Facilities of a Virtio Device / Notifications}
>>   
>>   The notion of sending a notification (driver to device or device
>>   to driver) plays an important role in this specification. The
>>   modus operandi of the notifications is transport specific.
>>   
>> -There are three types of notifications:
>> +There are five types of notifications:
>>   \begin{itemize}
>>   \item configuration change notification
>>   \item available buffer notification
>> -\item used buffer notification.
>> +\item used buffer notification
>> +\item driver auxiliary notification
>> +\item device auxiliary notification
>>   \end{itemize}
>>   
>> -Configuration change notifications and used buffer notifications are sent
>> -by the device, the recipient is the driver. A configuration change
>> -notification indicates that the device configuration space has changed; a
>> -used buffer notification indicates that a buffer may have been made used
>> -on the virtqueue designated by the notification.
>> -
>> -Available buffer notifications are sent by the driver, the recipient is
>> -the device. This type of notification indicates that a buffer may have
>> -been made available on the virtqueue designated by the notification.
>> +Configuration change notifications, used buffer notifications and
>> +driver auxiliary notifications are sent by the device,
>> +the recipient is the driver. A configuration change notification indicates
>> +that the device configuration space has changed; a used buffer notification
>> +indicates that a buffer may have been made used on the virtqueue designated
>> +by the notification; driver auxiliary notifications allow the
>> +device to send notifications other than configuration changes and used
>> +buffer notifications to the driver, these are optional and their meaning
>> +is device-specific.
> 
> The meaning of "optional" is confusing and I suggest dropping it.
> 
> Auxiliary notifications are not used by all device types. In that sense
> they are "optional".
> 
> But a device types that do use them, like virtio-vhost-user, may require
> them. You cannot implement a virtio-vhost-user device without auxiliary
> notifications, so they are mandatory (not "optional") for that device
> type.
> 
> Finally, it is also possible for a device type to define a feature bit
> that enables auxiliary notifications. Drivers/devices that don't support
> auxiliary notifications leave the feature bit cleared and still work
> properly. In this case they are truly "optional".
> 
> "Optional" is confusing. Just saying "their meaning is device-specific"
> is enough.
> 

Hi,

Yes agreed, it could be the case that auxiliary notifications are 
necessary for certain devices like VVU, so adding optional here can be 
confusing.

Will drop optional in next revision.

Thanks

>> +
>> +Available buffer notifications and device auxiliary notifications
>> +are sent by the driver, the recipient is the device. Available buffer
>> +notifications indicate that a buffer may have been made available on the
>> +virtqueue designated by the notification; device auxiliary
>> +notifications allow the driver to send notifications other than available
>> +buffer notifications to the device for example through a device register, these
>> +are optional and their meaning is device-specific.
>>   
>>   The semantics, the transport-specific implementations, and other
>>   important aspects of the different notifications are specified in detail
>> -- 
>> 2.25.1
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v3 0/4] Introduce aux. notifications and virtio-vhost-user

[Michael S. Tsirkin]

On Thu, Oct 27, 2022 at 03:54:41PM -0400, Stefan Hajnoczi wrote:
> On Fri, Oct 07, 2022 at 05:56:39PM +0100, Usama Arif wrote:
> > Hi,
> > 
> > This patch series introduces device and driver auxiliary notifications
> > as a new set of virtio device resources, as well as vhost-user device
> > backend that uses these resources.
> > 
> > Driver auxiliary notifications allow the device to send notifications
> > other than configuration changes and used buffer notifications to the
> > driver, these are optional and their meaning is **device-specific**.
> > 
> > Device auxiliary notifcations allow the driver to send notifcations
> > other than available buffer notifications to the device for example
> > through a device register, these are optional and their meaning is
> > **device-specific**.
> > 
> > These resources are used in the last patch by the virtio-vhost-user
> > device in order to send/receive notifications to/from the driver
> > regarding activity on the vhost-user virtqueues. By standardizing
> > these resources as standalone virtio device resources, other future
> > devices will be able to use them as well.
> 
> Hi Michael,
> Do you have time to review this? I have been too involved in
> virtio-vhost-user in the past to do a truly independent review and your
> VIRTIO + vhost expertise would be perfect here.
> 
> Thanks,
> Stefan


Plan to, just trying to merge a ton of stuff before qemu freeze.

> > 
> > The last patch introduces the vhost-user device backend which facilitates
> > vhost-user device emulation through vhost-user protocol exchanges and
> > access to shared memory. Software-defined networking, storage, and other
> > I/O appliances can provide services through this device.
> > 
> > This device is based on Wei Wang's vhost-pci work. The virtio-vhost-user
> > device differs from vhost-pci because it is a single virtio
> > device type that exposes the vhost-user protocol instead of a family of
> > new virtio device types, one for each vhost-user device type.
> > 
> > A HTML version with the changes is available at [1].
> > 
> > For more information about virtio-vhost-user, see [2].
> > 
> > These patches are based on the work initially done by Stefan Hajnoczi [2]
> > and continued by Nikos Dragazis [3], [4].
> > 
> > A working prototype implementing this spec can be reproduced using
> > instructions in [5] utilizing QEMU [6] and DPDK [7].
> > This is also based on the work initially done by Stefan
> > and continued later by Nikos. The prototype code uses the terms
> > "doorbell"/"device-specific notification" and "master"/"slave" instead of
> > "device auxiliary notification"/"driver auxiliary notification" and
> > "frontend"/"backend". This is based on older work, however, their
> > functionality is the same. If these virtio-spec changes get approved,
> > I will refractor the patches for QEMU/DPDK and send them for review
> > according to the final patches in the respective mailing list.
> > 
> > Thanks and looking forward to your response!
> > Usama
> > 
> > [1] https://uarif1.github.io/vvu/v3/virtio-v1.2-cs01
> > [2] https://wiki.qemu.org/Features/VirtioVhostUser
> > [3] https://ndragazis.github.io/dpdk-vhost-vvu-demo.html
> > [4] https://lists.oasis-open.org/archives/virtio-dev/202005/msg00132.html
> > [5] https://uarif1.github.io/vvu/dpdk-vvu-instructions
> > [6] https://github.com/uarif1/qemu/tree/vvu
> > [7] https://github.com/uarif1/dpdk/tree/vvu
> > ----------
> > v2->v3:
> > Mostly addressing comments made by Michael S. Tsirkin:
> > - Give a clearer relation between backend driver, backend device and frontend at
> >   beginning of the Vhost-user Device Backend section.
> > - change uuid to id.
> > - Add information about inflight memory.
> > 
> > v1->v2:
> > - Shortened device aux. notification section
> > - Make data width of device aux. notifications device specific and not
> >   limited to 2 bytes.
> > - Added information about the minimum number of MSIX vectors.
> > - Split virtio-mmio implementation of driver aux. notifications into 2
> >   registers, DeviceAuxNotificationIndex and DeviceAuxNotificationData.
> > - Made the shared memory section in virtio vhost-user clearer and added
> >   the shared memory would look like when
> >   VHOST_USER_PROTOCOL_F_CONFIGURE_MEM_SLOTS has been negotiated.
> > 
> > Usama Arif (4):
> >   content: Introduce driver/device auxiliary notifications
> >   content: Introduce driver/device aux. notification cfg type for PCI
> >   content: Introduce driver/device auxiliary notifications for MMIO
> >   vhost-user: add vhost-user device type
> > 
> >  conformance.tex       |  29 +++-
> >  content.tex           | 254 +++++++++++++++++++++++++++++-----
> >  introduction.tex      |   3 +
> >  virtio-vhost-user.tex | 309 ++++++++++++++++++++++++++++++++++++++++++
> >  4 files changed, 554 insertions(+), 41 deletions(-)
> >  create mode 100644 virtio-vhost-user.tex
> > 
> > -- 
> > 2.25.1
> > 



---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [External] Re: [PATCH v3 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Usama Arif]

On 27/10/2022 22:27, Stefan Hajnoczi wrote:
> On Fri, Oct 07, 2022 at 05:56:42PM +0100, Usama Arif wrote:
>> This includes the additions to the corresponding device and driver
>> conformances.
>>
>> Signed-off-by: Usama Arif <usama.arif@bytedance.com>
>> Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> 
> You can keep this. One thought:
> 
> I realized that virtio-mmio support won't be very useful until
> virtio-mmio gets MSI-X support because drivers currently cannot
> differentiate between device auxiliary notifications.
> 
> Stefan

Did you mean drivers currently cannot differentiate between "driver" 
auxiliary notifications? The driver will be able to be able to choose 
the index it wants to notifiy using DeviceAuxNotificationIndex.

I can add the following line in the Interrupt Status change of the patch 
if it makes sense?
"As MSI-X is not available, it is not possible to determine which driver 
auxiliary notification occured, hence only a single notification is 
supported."

Thanks,
Usama

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v3 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Michael S. Tsirkin]

On Thu, Oct 27, 2022 at 05:27:28PM -0400, Stefan Hajnoczi wrote:
> On Fri, Oct 07, 2022 at 05:56:42PM +0100, Usama Arif wrote:
> > This includes the additions to the corresponding device and driver
> > conformances.
> > 
> > Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> 
> You can keep this. One thought:
> 
> I realized that virtio-mmio support won't be very useful until
> virtio-mmio gets MSI-X support because drivers currently cannot
> differentiate between device auxiliary notifications.
> 
> Stefan


The big issue with MSI is it actually needs a ton of registers
for interrupt rebalancing, masking etc and they need to be fast.
Maybe a status bit like pci has makes sense here.

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [External] Re: [PATCH v3 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Michael S. Tsirkin]

On Mon, Oct 31, 2022 at 02:22:22PM +0000, Usama Arif wrote:
> 
> 
> On 27/10/2022 22:27, Stefan Hajnoczi wrote:
> > On Fri, Oct 07, 2022 at 05:56:42PM +0100, Usama Arif wrote:
> > > This includes the additions to the corresponding device and driver
> > > conformances.
> > > 
> > > Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > 
> > You can keep this. One thought:
> > 
> > I realized that virtio-mmio support won't be very useful until
> > virtio-mmio gets MSI-X support because drivers currently cannot
> > differentiate between device auxiliary notifications.
> > 
> > Stefan
> 
> Did you mean drivers currently cannot differentiate between "driver"
> auxiliary notifications? The driver will be able to be able to choose the
> index it wants to notifiy using DeviceAuxNotificationIndex.
> 
> I can add the following line in the Interrupt Status change of the patch if
> it makes sense?
> "As MSI-X is not available, it is not possible to determine which driver

drop "As MSI-X is not available," msi-x is a pci technology.

> auxiliary notification occured, hence only a single notification is
> supported."
> 
> Thanks,
> Usama


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

Re: [virtio-dev] Re: [PATCH v3 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1757 bytes --]

On Mon, Oct 31, 2022 at 11:45:55AM -0400, Michael S. Tsirkin wrote:
> On Thu, Oct 27, 2022 at 05:27:28PM -0400, Stefan Hajnoczi wrote:
> > On Fri, Oct 07, 2022 at 05:56:42PM +0100, Usama Arif wrote:
> > > This includes the additions to the corresponding device and driver
> > > conformances.
> > > 
> > > Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > 
> > You can keep this. One thought:
> > 
> > I realized that virtio-mmio support won't be very useful until
> > virtio-mmio gets MSI-X support because drivers currently cannot
> > differentiate between device auxiliary notifications.
> > 
> > Stefan
> 
> 
> The big issue with MSI is it actually needs a ton of registers
> for interrupt rebalancing, masking etc and they need to be fast.
> Maybe a status bit like pci has makes sense here.

This patch does add an ISR bit:

diff --git a/content.tex b/content.tex
index 33362b7..8968fcd 100644
--- a/content.tex
+++ b/content.tex
@@ -2049,6 +2049,8 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
         in at least one of the active virtual queues.
       \item [Configuration Change Notification] - bit 1 - the interrupt was
         asserted because the configuration of the device has changed.
+      \item [Device-specific Driver Auxiliary Notification] - bit 2 - the interrupt was
+        asserted because a device-specific event occurred to notify the driver.

The problem is that the driver doesn't know which device-specific event
occurred, so the mechanism is less useful than the MSI-X approach where
the driver knows exactly which (of potentially many) device-specific
events occurred.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] Re: [External] Re: [PATCH v3 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1361 bytes --]

On Mon, Oct 31, 2022 at 02:22:22PM +0000, Usama Arif wrote:
> On 27/10/2022 22:27, Stefan Hajnoczi wrote:
> > On Fri, Oct 07, 2022 at 05:56:42PM +0100, Usama Arif wrote:
> > > This includes the additions to the corresponding device and driver
> > > conformances.
> > > 
> > > Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > 
> > You can keep this. One thought:
> > 
> > I realized that virtio-mmio support won't be very useful until
> > virtio-mmio gets MSI-X support because drivers currently cannot
> > differentiate between device auxiliary notifications.
> > 
> > Stefan
> 
> Did you mean drivers currently cannot differentiate between "driver"
> auxiliary notifications? The driver will be able to be able to choose the
> index it wants to notifiy using DeviceAuxNotificationIndex.

Yes, sorry!

> 
> I can add the following line in the Interrupt Status change of the patch if
> it makes sense?
> "As MSI-X is not available, it is not possible to determine which driver
> auxiliary notification occured, hence only a single notification is
> supported."

Yes, that is how I understood the mechanism to work. Depending on the
nature of the device-specific event, it may be unworkable for the driver
to have no way of identifying the exact event.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [virtio-dev] Re: [PATCH v3 3/4] content: Introduce driver/device auxiliary notifications for MMIO

[Michael S. Tsirkin]

On Mon, Oct 31, 2022 at 01:24:30PM -0400, Stefan Hajnoczi wrote:
> On Mon, Oct 31, 2022 at 11:45:55AM -0400, Michael S. Tsirkin wrote:
> > On Thu, Oct 27, 2022 at 05:27:28PM -0400, Stefan Hajnoczi wrote:
> > > On Fri, Oct 07, 2022 at 05:56:42PM +0100, Usama Arif wrote:
> > > > This includes the additions to the corresponding device and driver
> > > > conformances.
> > > > 
> > > > Signed-off-by: Usama Arif <usama.arif@bytedance.com>
> > > > Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
> > > 
> > > You can keep this. One thought:
> > > 
> > > I realized that virtio-mmio support won't be very useful until
> > > virtio-mmio gets MSI-X support because drivers currently cannot
> > > differentiate between device auxiliary notifications.
> > > 
> > > Stefan
> > 
> > 
> > The big issue with MSI is it actually needs a ton of registers
> > for interrupt rebalancing, masking etc and they need to be fast.
> > Maybe a status bit like pci has makes sense here.
> 
> This patch does add an ISR bit:
> 
> diff --git a/content.tex b/content.tex
> index 33362b7..8968fcd 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -2049,6 +2049,8 @@ \subsection{MMIO Device Register Layout}\label{sec:Virtio Transport Options / Vi
>          in at least one of the active virtual queues.
>        \item [Configuration Change Notification] - bit 1 - the interrupt was
>          asserted because the configuration of the device has changed.
> +      \item [Device-specific Driver Auxiliary Notification] - bit 2 - the interrupt was
> +        asserted because a device-specific event occurred to notify the driver.
> 
> The problem is that the driver doesn't know which device-specific event
> occurred, so the mechanism is less useful than the MSI-X approach where
> the driver knows exactly which (of potentially many) device-specific
> events occurred.
> 
> Stefan


Right I figured that out after sending this. So it's limited to
one event type.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org