RE: [PATCH] VIRTIO_F_USED_EVENT_AUTO_DISABLE: add new used buffer notification suppression mechanism

Lege Wang
2024-07-05T03:35:23Z

hi

Thanks for your comments in previous discussions.
I'd like to confirm that do you understand this proposal's idea now. If yes,
I may start to prepare a more formal V2.

Regards,
Xiaoguang Wang

> -----Original Message-----
> From: Lege Wang <lege.wang@jaguarmicro.com>
> Sent: Monday, July 1, 2024 11:45 AM
> To: virtio-comment@lists.linux.dev
> Cc: vattunuru@marvell.com; ndabilpuram@marvell.com; parav@nvidia.com;
> mst@redhat.com; Leo Liu <leo.liu@jaguarmicro.com>; Angus Chen
> <angus.chen@jaguarmicro.com>; Lege Wang <lege.wang@jaguarmicro.com>
> Subject: [PATCH] VIRTIO_F_USED_EVENT_AUTO_DISABLE: add new used buffer
> notification suppression mechanism
>
> Currently for the PCI transport, used buffer notification suppression
> informations are located in guest memory, for example, pvirtq_event_suppress
> for packed ring and virtq_avail for split ring, then PCI devices simulated
> by hypervisors, such as QEMU, can vist these suppression informations
> directly before issuing used event, the only overheads may be just proper
> memory barriers, but for some hardware PCI devices offered by
> DPU(Data Processing Unit), DPU would need to submit DMAs to copy guest
> suppression informations to DPU internal memory, then check whether
> used event should be notified by examing this internal memory, which
> is obvious expensive, and this DMA operation is needed every time
> DPU attempts to issue used event, DPU may also choose to add a hardware
> component to polling suppression informations by submit DMAs continuously
> to get newest used event suppression infos, but it's also expensive.
>
> To help improve this sitiuations, add a new feature:
> VIRTIO_F_USED_EVENT_AUTO_DISABLE. When this feature bit is negotiated,
> devices will place new notification area for used buffer suppression
> notification in PCI bar specfied by virtio_pci_notify_cap, also for better
> used buffer notification coalescing, the device will disable later used
> buffer notifications automatically after sending one used buffer notification
> to driver, and after the driver finishes handling current used buffer event,
> it needs to notify the device to enable used buffer notification again by
> writing above new notification area in PCI bar. Device will receive
> this notification reliably, and device knows that it can issue used event
> now, but it may choose appropriate timing to send used events according to
> its notification coalescing policy , now it also does not need to DMA guest
> memory to obtain notification suppression.
>
> Signed-off-by: Xiaoguang Wang <lege.wang@jaguarmicro.com>
> ---
> content.tex | 38 ++++++++++++++++++++++++++++++++++++++
> transport-pci.tex | 26 +++++++++++++++++++++++++-
> 2 files changed, 63 insertions(+), 1 deletion(-)
>
> diff --git a/content.tex b/content.tex
> index 0a62dce..84505da 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -448,6 +448,7 @@ \section{Driver Notifications} \label{sec:Basic Facilities
> of a Virtio Device /
> The driver is sometimes required to send an available buffer
> notification to the device.
>
> +\subsection{Available Buffer Notifications} \label{sec:Basic Facilities of a
> Virtio Device / Driver notifications / Available Buffer Notifications}
> When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
> this notification contains either a virtqueue index if
> VIRTIO_F_NOTIF_CONFIG_DATA is not negotiated or device supplied
> virtqueue
> @@ -488,6 +489,35 @@ \section{Driver Notifications} \label{sec:Basic
> Facilities of a Virtio Device /
> has been negotiated, these notifications would then have
> identical \field{next_off} and \field{next_wrap} values.
>
> +\subsection{Used Buffer Event Enable Notifications} \label{sec:Basic Facilities
> of a Virtio Device / Driver notifications / Used Buffer Event Enable
> Notifications}
> +This kind of notification is only necessary when
> VIRTIO_F_USED_EVENT_AUTO_DISABLE has been negotiated.
> +When this feature bit is negotiated, the device will disable used buffer
> notification
> +automatically after sending one used buffer notification to the driver, and
> after the driver
> +finishes handling current used buffer event, it needs to notify the device to
> enable used
> +buffer notification again.
> +
> +\begin{description}
> +\item [vq_index or vq_notif_config_data] Either virtqueue index or device
> + supplied queue notification config data corresponding to a virtqueue.
> +\item [used_off] Offset
> + within the ring where the latest used ring entry the driver has
> processed.
> + Without VIRTIO_F_RING_PACKED this refers to the 14 least significant
> bits of the
> + offset (in units of descriptor entries) within the descriptor ring where
> the latest used
> + ring entry the driver has processed. With VIRTIO_F_RING_PACKED this
> refers to the offset
> + (in units of descriptor entries) within the descriptor ring where the
> latest used ring
> + entry the driver has processed.
> +\item [used_wrap] Wrap Counter.
> + With VIRTIO_F_RING_PACKED this is the wrap counter
> + referring to the latest used descriptor where the driver has processed.
> + Without VIRTIO_F_RING_PACKED this refers to the most significant
> bit(bit 14) of the
> + offset (in units of descriptor entries) within the descriptor ring where
> the latest used ring
> + entry the driver has processed..
> +\item [enable_notify] Whether to enable used buffer notification.
> + The driver may benefit from just informing device the latest used ring
> entry
> + it has processed, but not enabling used buffer notification. This field
> only
> + occupies one bit, if set to 1, it will enable used buffer notification,
> otherwise not.
> +\end{description}
> +
> \input{shared-mem.tex}
>
> \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device /
> Exporting Objects}
> @@ -872,6 +902,14 @@ \chapter{Reserved Feature Bits}\label{sec:Reserved
> Feature Bits}
> \ref{devicenormative:Basic Facilities of a Virtio Device / Feature Bits} for
> handling features reserved for future use.
>
> + \item[VIRTIO_F_USED_EVENT_AUTO_DISABLE(42)] This feature indicates
> that the device
> + will disable used buffer notification automatically after sending one used
> buffer notification.
> + Note: when VIRTIO_F_RING_PACKED has not been negotiated, the
> maximum queue
> + size MUST NOT be greater than 16384, when VIRTIO_F_RING_PACKED has
> been negotiated,
> + the maximum queue size MUST NOT be greater than 16383.
> +
> + See \ref{sec:Basic Facilities of a Virtio Device / Driver notifications / Used
> Buffer Event Enable Notifications}
> +
> \end{description}
>
> \drivernormative{\section}{Reserved Feature Bits}{Reserved Feature Bits}
> diff --git a/transport-pci.tex b/transport-pci.tex
> index a5c6719..62ba9dc 100644
> --- a/transport-pci.tex
> +++ b/transport-pci.tex
> @@ -551,6 +551,13 @@ \subsubsection{Notification structure
> layout}\label{sec:Virtio Transport Options
> cap.offset + queue_notify_off * notify_off_multiplier
> \end{lstlisting}
>
> +If the VIRTIO_F_USED_EVENT_AUTO_DISABLE feature has been negotiated,
> \field{notify_off_multiplier} is also
> +combined with the \field{queue_notify_off} to derive the Queue Notify
> address for enabling
> +used buffer notifications within a BAR for a virtqueue, but add a 4 bytes
> offset:
> +\begin{lstlisting}
> + cap.offset + queue_notify_off * notify_off_multiplier + 4
> +\end{lstlisting}
> +
> The \field{cap.offset} and \field{notify_off_multiplier} are taken from the
> notification capability structure above, and the \field{queue_notify_off} is
> taken from the common configuration structure.
> @@ -563,7 +570,7 @@ \subsubsection{Notification structure
> layout}\label{sec:Virtio Transport Options
> \devicenormative{\paragraph}{Notification capability}{Virtio Transport
> Options / Virtio Over PCI Bus / PCI Device Layout / Notification capability}
> The device MUST present at least one notification capability.
>
> -For devices not offering VIRTIO_F_NOTIFICATION_DATA:
> +For devices not offering VIRTIO_F_NOTIFICATION_DATA or
> VIRTIO_F_USED_EVENT_AUTO_DISABLE:
>
> The \field{cap.offset} MUST be 2-byte aligned.
>
> @@ -596,6 +603,23 @@ \subsubsection{Notification structure
> layout}\label{sec:Virtio Transport Options
> cap.length >= queue_notify_off * notify_off_multiplier + 4
> \end{lstlisting}
>
> +For devices offering VIRTIO_F_USED_EVENT_AUTO_DISABLE:
> +
> +The device MUST either present \field{notify_off_multiplier} as a
> +number that is a power of 2 that is also a multiple 4,
> +or present \field{notify_off_multiplier} as 0.
> +
> +The \field{cap.offset} MUST be 4-byte aligned.
> +
> +The value \field{cap.length} presented by the device MUST be at least 8
> +and MUST be large enough to support queue notification offsets
> +for all supported queues in all possible configurations.
> +
> +For all queues, the value \field{cap.length} presented by the device MUST
> satisfy:
> +\begin{lstlisting}
> +cap.length >= queue_notify_off * notify_off_multiplier + 8
> +\end{lstlisting}
> +
> \subsubsection{ISR status capability}\label{sec:Virtio Transport Options /
> Virtio Over PCI Bus / PCI Device Layout / ISR status capability}
>
> The VIRTIO_PCI_CAP_ISR_CFG capability
> --
> 2.34.1