* [PATCH v6] virtio-net: support the virtqueue coalescing moderation
@ 2023-02-22 3:21 Heng Qi
2023-02-22 4:13 ` Parav Pandit
2023-02-22 8:09 ` [virtio-comment] " Alvaro Karsz
0 siblings, 2 replies; 12+ messages in thread
From: Heng Qi @ 2023-02-22 3:21 UTC (permalink / raw)
To: virtio-dev, virtio-comment
Cc: Michael S . Tsirkin, Parav Pandit, Alvaro Karsz, David Edmondson,
Jason Wang, Xuan Zhuo, Cornelia Huck
Currently, coalescing parameters are grouped for all transmit and receive
virtqueues. This patch supports setting or getting the parameters for a
specified virtqueue, and a typical application of this function is netdim[1].
When the traffic between virtqueues is unbalanced, for example, one virtqueue
is busy and another virtqueue is idle, then it will be very useful to
control coalescing parameters at the virtqueue granularity.
[1] https://docs.kernel.org/networking/net_dim.html
Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
---
This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html .
v5->v6:
1. Explain that the device may set a different value than the one passed in by the driver. @David Edmondson
v4->v5:
1. Add the correspondence between virtio_net_ctrl_coal and virtio_net_ctrl_coal_vq and control commands. @Michael S. Tsirkin
2. Add read and write attributes for each field. @Michael S. Tsirkin
3. A clearer description of how to set coalescing parameters for vq reset. @Michael S. Tsirkin
4. Fix some syntax errors. @Michael S. Tsirkin, @David Edmondson
v3->v4:
1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz
2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin
4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
v2->v3:
1. Add the netdim link. @Parav Pandit
2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
3. _VQ_GET is explained more. @Michael S. Tsirkin
4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
7. Fix some typos. @Michael S. Tsirkin
v1->v2:
1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
device-types/net/description.tex | 100 +++++++++++++++++++++++++++++--
1 file changed, 95 insertions(+), 5 deletions(-)
diff --git a/device-types/net/description.tex b/device-types/net/description.tex
index e71e33b..0d9f4b9 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
\item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
channel.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing.
+
\item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
\item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
@@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
\item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
\item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
\item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
+\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
\end{description}
\subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
@@ -1508,6 +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can
send control commands for dynamically changing the coalescing parameters.
+If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
+\begin{itemize}
+\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set coalescing parameters of a given
+ enabled transmit/receive virtqueue.
+\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device responds with
+ coalescing parameters of a given enabled transmit/receive virtqueue.
+\end{itemize}
+
\begin{note}
The behavior of the device in response to these commands is best-effort:
the device may generate notifications more or less frequently than specified.
@@ -1519,25 +1530,76 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
le32 max_usecs;
};
+struct virtio_net_ctrl_coal_vq {
+ le16 vqn;
+ le16 reserved;
+ struct virtio_net_ctrl_coal coal;
+};
+
#define VIRTIO_NET_CTRL_NOTF_COAL 6
#define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
#define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
+ #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
\end{lstlisting}
+The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the
+virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all
+transmit/receive virtqueues.
+
+The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure
+to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}.
+
+The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and
+\field{max_packets} of the specified virtqueue from the device by setting \field{vqn}
+in the virtio_net_ctrl_coal_vq structure.
+
+# Read/Write attributes for coalescing parameters
+\begin{itemize}
+\item For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs}
+ and \field{max_packets} are write-only for a driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, \field{reserved}, \field{max_usecs}
+ and \field{max_packets} are write-only for a driver.
+\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} and \field{reserved} are write-only
+ for a driver, and, \field{max_usecs} and \field{max_packets} are read-only for a driver.
+\end{itemize}
+
Coalescing parameters:
\begin{itemize}
+\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue.
\item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification.
\item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification.
\item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification.
\item \field{max_packets} for TX: Maximum number of packets to send before a TX notification.
\end{itemize}
-The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
+\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
+
+\field{reserved} is reserved and it is ignored by a device.
+
+The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
\begin{enumerate}
-\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues.
-\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive
+ virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled
+ transmit/receive virtqueue whose number is \field{vqn}.
+\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN.
+\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN.
\end{enumerate}
+If coalescing parameters are being set, the device applies the last coalescing parameters received for a
+virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues:
+# Command sequence
+Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues.
+\begin{itemize}
+\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values.
+\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values from command1.
+\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 0, the device responds with coalescing parameters of virtqueue0 set by command2.
+\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous values.
+\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing parameters for virtqueue1 and virtqueue3, and overrides the values set by command4.
+\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 1, the device responds with coalescing parameters of virtqueue1 set by command5.
+\end{itemize}
+
\subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1549,6 +1611,16 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent.
+When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL class to set a coalescing parameter,
+it may set the parameter to a value close to a power of 2. For example:
+If the device receives \field{max_usecs} = 7 from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set \field{max_usecs} = 8 for a given enabled virtqueue.
+
+When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands,
+it saves the values of coalescing parameters as global values, and the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command
+does not change the global values. If the device is reset, the global values will be set to 0.
+
+When a virtqueue is reset, its coalescing parameters are set to the global values.
+
\subparagraph{RX Example}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Example}
If, for example:
@@ -1585,11 +1657,29 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
\drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
-If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
+has been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
+
+A driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with VIRTIO_NET_ERR.
\devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
-A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters.
+
+A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters.
+
+A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the given virtqueue is disabled.
+
+The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters for all transmit/receive
+virtqueues respectively and values of coalescing parameters are recorded as global values by a device.
+The device MUST set the global values of coalescing parameters to 0 after being reset.
+The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values.
+
+After disabling and re-enabling a virtqueue, the device MUST revert coalescing parameters of the virtqueue to the global values.
+
+A device MAY set the coalescing parameter to a value close to a power of 2 value.
+
+A device MUST ignore \field{reserved}.
A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
--
2.19.1.6.gb485710b
^ permalink raw reply related [flat|nested] 12+ messages in thread* RE: [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 3:21 [PATCH v6] virtio-net: support the virtqueue coalescing moderation Heng Qi @ 2023-02-22 4:13 ` Parav Pandit 2023-02-22 5:31 ` [virtio-dev] " Heng Qi ` (2 more replies) 2023-02-22 8:09 ` [virtio-comment] " Alvaro Karsz 1 sibling, 3 replies; 12+ messages in thread From: Parav Pandit @ 2023-02-22 4:13 UTC (permalink / raw) To: Heng Qi, virtio-dev@lists.oasis-open.org, virtio-comment@lists.oasis-open.org Cc: Michael S . Tsirkin, Alvaro Karsz, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck > From: Heng Qi <hengqi@linux.alibaba.com> > Sent: Tuesday, February 21, 2023 10:22 PM > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue > notification coalescing. > + s/notification/notifications should be plural as multiple notifications are coalesced like the below description of VIRTIO_NET_F_NOTF_COAL. > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications > coalescing. > > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. > @@ -139,6 +141,7 @@ \subsubsection{Feature bit > requirements}\label{sec:Device Types / Network Device > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or > VIRTIO_NET_F_HOST_TSO6. > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. > \end{description} > > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / > Network Device / Feature bits / Legacy Interface: Feature bits} @@ -1508,6 > +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / > Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, > the driver can send control commands for dynamically changing the coalescing > parameters. > > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: > +\begin{itemize} > +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command > to set coalescing parameters of a given > + enabled transmit/receive virtqueue. > +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command > to a device, and the device responds with > + coalescing parameters of a given enabled transmit/receive virtqueue. > +\end{itemize} > + > \begin{note} > The behavior of the device in response to these commands is best-effort: > the device may generate notifications more or less frequently than specified. > @@ -1519,25 +1530,76 @@ \subsubsection{Control > Virtqueue}\label{sec:Device Types / Network Device / Devi > le32 max_usecs; > }; > > +struct virtio_net_ctrl_coal_vq { > + le16 vqn; > + le16 reserved; > + struct virtio_net_ctrl_coal coal; > +}; > + > #define VIRTIO_NET_CTRL_NOTF_COAL 6 > #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 #define > + VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 > \end{lstlisting} > > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the > virtio_net_ctrl_coal > +structure to set \field{max_usecs} and \field{max_packets} for all > transmit/receive virtqueues. > + > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the > +virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and > \field{max_packets} for the supplied virtqueue number \field{vqn}. > + > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of > +\field{max_usecs} and \field{max_packets} of the specified virtqueue > +from the device by setting \field{vqn} in the virtio_net_ctrl_coal_vq structure. > + > +# Read/Write attributes for coalescing parameters \begin{itemize} \item > +For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} > + and \field{max_packets} are write-only for a driver. > +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, Virtio spec is using vq number and vq index terminology interchangeably. For example, a new patch about admin vq, uses aq_start_index. MMIO device has vq_index register too. I am inclined to vq_index given current state of spec and new additions by Michael. Michael, Can you please suggest vqn or vq_index to use? > \field{reserved}, \field{max_usecs} > + and \field{max_packets} are write-only for a driver. > +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} > and \field{reserved} are write-only > + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only > for a driver. Remove this trailing "for a driver", it is a duplicate. > +\end{itemize} > + > Coalescing parameters: > \begin{itemize} > +\item \field{vqn}: The virtqueue number of an enabled transmit or receive > virtqueue. > \item \field{max_usecs} for RX: Maximum number of microseconds to delay a > RX notification. > \item \field{max_usecs} for TX: Maximum number of microseconds to delay a > TX notification. > \item \field{max_packets} for RX: Maximum number of packets to receive > before a RX notification. > \item \field{max_packets} for TX: Maximum number of packets to send before > a TX notification. > \end{itemize} > > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: > +\field{vqn} points to an enabled transmit/receive virtqueue, and its value > satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. > + This calculation description is not needed here. It is covered somewhere else. > +\field{reserved} is reserved and it is ignored by a device. > + > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: > \begin{enumerate} > -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and > \field{max_packets} parameters for all transmit virtqueues. > -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and > \field{max_packets} parameters for all receive virtqueues. > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and > \field{max_packets} parameters for an enabled transmit/receive > + virtqueue whose number is \field{vqn}. > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the > \field{max_usecs} and \field{max_packets} parameters for an enabled > + transmit/receive virtqueue whose number is \field{vqn}. > +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each > virtqueue of transmitq1\ldots transmitqN. > +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each > virtqueue of receiveq1\ldots receiveqN. > \end{enumerate} > > +If coalescing parameters are being set, the device applies the last > +coalescing parameters received for a virtqueue, regardless of the command > used to set the parameters. For example with 2 pairs of virtqueues: > +# Command sequence > +Each of the following commands sets \field{max_usecs} and > \field{max_packets} parameters for virtqueues. > +\begin{itemize} > +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing > parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 > retain their previous parameter values. > +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = > 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values > from command1. > +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = > 0, the device responds with coalescing parameters of virtqueue0 set by > command2. > +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = > 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous > values. > +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing > parameters for virtqueue1 and virtqueue3, and overrides the values set by > command4. > +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = > 1, the device responds with coalescing parameters of virtqueue1 set by > command5. > +\end{itemize} > + > \subparagraph{Operation}\label{sec:Device Types / Network Device / Device > Operation / Control Virtqueue / Notifications Coalescing / Operation} > > The device sends a used buffer notification once the notification conditions are > met and if the notifications are not suppressed as explained in \ref{sec:Basic > Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. > @@ -1549,6 +1611,16 @@ \subsubsection{Control > Virtqueue}\label{sec:Device Types / Network Device / Devi > > When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the > notification conditions are met after every packet received/sent. > > +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL > class > +to set a coalescing parameter, it may set the parameter to a value close to a > power of 2. For example: > +If the device receives \field{max_usecs} = 7 from the > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set > \field{max_usecs} = 8 for a given enabled virtqueue. > + > +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, it saves the values of > +coalescing parameters as global values, and the > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command does not change the global > values. If the device is reset, the global values will be set to 0. > + > +When a virtqueue is reset, its coalescing parameters are set to the global > values. > + A VQ reset operation disables (or destroys) the VQ in the device. Hence, a vq under reset doesn't have any valid parameters. Therefore, above wording should be, When a virtqueue is enabled after it is reset, its coalescing parameters are set to global values as configured by VIRTIO_NET_CTRL_NOTF_COAL_TX_SET or VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. > \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device > Operation / Control Virtqueue / Notifications Coalescing / RX Example} > > If, for example: > @@ -1585,11 +1657,29 @@ \subsubsection{Control > Virtqueue}\label{sec:Device Types / Network Device / Devi > > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / > Network Device / Device Operation / Control Virtqueue / Notifications > Coalescing} > > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver > MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. > +If neither the VIRTIO_NET_F_NOTF_COAL nor the > VIRTIO_NET_F_VQ_NOTF_COAL > +feature has been negotiated, the driver MUST NOT issue > VIRTIO_NET_CTRL_NOTF_COAL commands. > + > +A driver MUST ignore the values of coalescing parameters received from the > VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with > VIRTIO_NET_ERR. > > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / > Network Device / Device Operation / Control Virtqueue / Notifications > Coalescing} > > -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands > with VIRTIO_NET_ERR if it was not able to change the parameters. > +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it > was not able to change the parameters. > + > +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET > command with VIRTIO_NET_ERR if it was not able to change the parameters. > + > +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and > VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if > the given virtqueue is disabled. > + > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters > for all transmit/receive virtqueues respectively and values of coalescing > parameters are recorded as global values by a device. > +The device MUST set the global values of coalescing parameters to 0 after > being reset. > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing > parameters for a given enabled virtqueue without changing the global values. > + > +After disabling and re-enabling a virtqueue, the device MUST revert coalescing > parameters of the virtqueue to the global values. > + > +A device MAY set the coalescing parameter to a value close to a power of 2 > value. > + > +A device MUST ignore \field{reserved}. > > A device SHOULD NOT send used buffer notifications to the driver if the > notifications are suppressed, even if the notification conditions are met. > > -- > 2.19.1.6.gb485710b ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [virtio-dev] RE: [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 4:13 ` Parav Pandit @ 2023-02-22 5:31 ` Heng Qi 2023-02-22 6:29 ` Michael S. Tsirkin 2023-02-23 10:01 ` [virtio-comment] " David Edmondson 2 siblings, 0 replies; 12+ messages in thread From: Heng Qi @ 2023-02-22 5:31 UTC (permalink / raw) To: Parav Pandit, virtio-dev@lists.oasis-open.org, virtio-comment@lists.oasis-open.org Cc: Michael S . Tsirkin, Alvaro Karsz, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck Hi, Parav. 在 2023/2/22 下午12:13, Parav Pandit 写道: >> From: Heng Qi <hengqi@linux.alibaba.com> >> Sent: Tuesday, February 21, 2023 10:22 PM >> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue >> notification coalescing. >> + > s/notification/notifications > should be plural as multiple notifications are coalesced like the below description of VIRTIO_NET_F_NOTF_COAL. I see, but I saw the usage of "notification coalesing" on the professional technical articles exist as well. I can modify it if you want to be consistent with the following.:) > >> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications >> coalescing. >> >> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. >> @@ -139,6 +141,7 @@ \subsubsection{Feature bit >> requirements}\label{sec:Device Types / Network Device >> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or >> VIRTIO_NET_F_HOST_TSO6. >> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. >> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >> \end{description} >> >> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / >> Network Device / Feature bits / Legacy Interface: Feature bits} @@ -1508,6 >> +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / >> Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, >> the driver can send control commands for dynamically changing the coalescing >> parameters. >> >> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: >> +\begin{itemize} >> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command >> to set coalescing parameters of a given >> + enabled transmit/receive virtqueue. >> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command >> to a device, and the device responds with >> + coalescing parameters of a given enabled transmit/receive virtqueue. >> +\end{itemize} >> + >> \begin{note} >> The behavior of the device in response to these commands is best-effort: >> the device may generate notifications more or less frequently than specified. >> @@ -1519,25 +1530,76 @@ \subsubsection{Control >> Virtqueue}\label{sec:Device Types / Network Device / Devi >> le32 max_usecs; >> }; >> >> +struct virtio_net_ctrl_coal_vq { >> + le16 vqn; >> + le16 reserved; >> + struct virtio_net_ctrl_coal coal; >> +}; >> + >> #define VIRTIO_NET_CTRL_NOTF_COAL 6 >> #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 >> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 >> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 #define >> + VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 >> \end{lstlisting} >> >> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the >> virtio_net_ctrl_coal >> +structure to set \field{max_usecs} and \field{max_packets} for all >> transmit/receive virtqueues. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the >> +virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and >> \field{max_packets} for the supplied virtqueue number \field{vqn}. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of >> +\field{max_usecs} and \field{max_packets} of the specified virtqueue >> +from the device by setting \field{vqn} in the virtio_net_ctrl_coal_vq structure. >> + >> +# Read/Write attributes for coalescing parameters \begin{itemize} \item >> +For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} >> + and \field{max_packets} are write-only for a driver. >> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, > Virtio spec is using vq number and vq index terminology interchangeably. > For example, a new patch about admin vq, uses aq_start_index. > > MMIO device has vq_index register too. > I am inclined to vq_index given current state of spec and new additions by Michael. Yes, both vq number and vq index exist in the existing spec. For example this paragraph: " \begin{description} \item [vqn] VQ number to be notified. \item [next_off] Offset ... \end{description} " > > Michael, > Can you please suggest vqn or vq_index to use? Let's hear Michael's opinion. > >> \field{reserved}, \field{max_usecs} >> + and \field{max_packets} are write-only for a driver. >> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} >> and \field{reserved} are write-only > >> + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only >> for a driver. > Remove this trailing "for a driver", it is a duplicate. Ok. > >> +\end{itemize} >> + >> Coalescing parameters: >> \begin{itemize} >> +\item \field{vqn}: The virtqueue number of an enabled transmit or receive >> virtqueue. >> \item \field{max_usecs} for RX: Maximum number of microseconds to delay a >> RX notification. >> \item \field{max_usecs} for TX: Maximum number of microseconds to delay a >> TX notification. >> \item \field{max_packets} for RX: Maximum number of packets to receive >> before a RX notification. >> \item \field{max_packets} for TX: Maximum number of packets to send before >> a TX notification. >> \end{itemize} >> >> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: >> +\field{vqn} points to an enabled transmit/receive virtqueue, and its value >> satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. >> + > This calculation description is not needed here. It is covered somewhere else. Ok, I'll drop this. > >> +\field{reserved} is reserved and it is ignored by a device. >> + >> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: >> \begin{enumerate} >> -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and >> \field{max_packets} parameters for all transmit virtqueues. >> -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and >> \field{max_packets} parameters for all receive virtqueues. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and >> \field{max_packets} parameters for an enabled transmit/receive >> + virtqueue whose number is \field{vqn}. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the >> \field{max_usecs} and \field{max_packets} parameters for an enabled >> + transmit/receive virtqueue whose number is \field{vqn}. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each >> virtqueue of transmitq1\ldots transmitqN. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each >> virtqueue of receiveq1\ldots receiveqN. >> \end{enumerate} >> >> +If coalescing parameters are being set, the device applies the last >> +coalescing parameters received for a virtqueue, regardless of the command >> used to set the parameters. For example with 2 pairs of virtqueues: >> +# Command sequence >> +Each of the following commands sets \field{max_usecs} and >> \field{max_packets} parameters for virtqueues. >> +\begin{itemize} >> +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing >> parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 >> retain their previous parameter values. >> +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = >> 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values >> from command1. >> +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = >> 0, the device responds with coalescing parameters of virtqueue0 set by >> command2. >> +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = >> 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous >> values. >> +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing >> parameters for virtqueue1 and virtqueue3, and overrides the values set by >> command4. >> +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = >> 1, the device responds with coalescing parameters of virtqueue1 set by >> command5. >> +\end{itemize} >> + >> \subparagraph{Operation}\label{sec:Device Types / Network Device / Device >> Operation / Control Virtqueue / Notifications Coalescing / Operation} >> >> The device sends a used buffer notification once the notification conditions are >> met and if the notifications are not suppressed as explained in \ref{sec:Basic >> Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. >> @@ -1549,6 +1611,16 @@ \subsubsection{Control >> Virtqueue}\label{sec:Device Types / Network Device / Devi >> >> When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the >> notification conditions are met after every packet received/sent. >> >> +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL >> class >> +to set a coalescing parameter, it may set the parameter to a value close to a >> power of 2. For example: >> +If the device receives \field{max_usecs} = 7 from the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set >> \field{max_usecs} = 8 for a given enabled virtqueue. >> + >> +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, it saves the values of >> +coalescing parameters as global values, and the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command does not change the global >> values. If the device is reset, the global values will be set to 0. >> + >> +When a virtqueue is reset, its coalescing parameters are set to the global >> values. >> + > A VQ reset operation disables (or destroys) the VQ in the device. > Hence, a vq under reset doesn't have any valid parameters. > Therefore, above wording should be, > > When a virtqueue is enabled after it is reset, its coalescing parameters are set to global values as configured by VIRTIO_NET_CTRL_NOTF_COAL_TX_SET or VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. Thanks for your suggestion! I think the following description can clarify? “When a virtqueue is enabled after it is reset, its coalescing parameters are set to global values.” Because the global values have been mentioned above: " +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, it saves the values of +coalescing parameters as global values, and the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command does not change the global values. If the device is reset, the global values will be set to 0. " and it may be 0 after device reset, or it may be the values set by VIRTIO_NET_CTRL_NOTF_COAL_RX_SET or VIRTIO_NET_CTRL_NOTF_COAL_TX_SET. Thank! > >> \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device >> Operation / Control Virtqueue / Notifications Coalescing / RX Example} >> >> If, for example: >> @@ -1585,11 +1657,29 @@ \subsubsection{Control >> Virtqueue}\label{sec:Device Types / Network Device / Devi >> >> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / >> Network Device / Device Operation / Control Virtqueue / Notifications >> Coalescing} >> >> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver >> MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. >> +If neither the VIRTIO_NET_F_NOTF_COAL nor the >> VIRTIO_NET_F_VQ_NOTF_COAL >> +feature has been negotiated, the driver MUST NOT issue >> VIRTIO_NET_CTRL_NOTF_COAL commands. >> + >> +A driver MUST ignore the values of coalescing parameters received from the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with >> VIRTIO_NET_ERR. >> >> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / >> Network Device / Device Operation / Control Virtqueue / Notifications >> Coalescing} >> >> -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands >> with VIRTIO_NET_ERR if it was not able to change the parameters. >> +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it >> was not able to change the parameters. >> + >> +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET >> command with VIRTIO_NET_ERR if it was not able to change the parameters. >> + >> +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if >> the given virtqueue is disabled. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters >> for all transmit/receive virtqueues respectively and values of coalescing >> parameters are recorded as global values by a device. >> +The device MUST set the global values of coalescing parameters to 0 after >> being reset. >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing >> parameters for a given enabled virtqueue without changing the global values. >> + >> +After disabling and re-enabling a virtqueue, the device MUST revert coalescing >> parameters of the virtqueue to the global values. >> + >> +A device MAY set the coalescing parameter to a value close to a power of 2 >> value. >> + >> +A device MUST ignore \field{reserved}. >> >> A device SHOULD NOT send used buffer notifications to the driver if the >> notifications are suppressed, even if the notification conditions are met. >> >> -- >> 2.19.1.6.gb485710b > > --------------------------------------------------------------------- > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 4:13 ` Parav Pandit 2023-02-22 5:31 ` [virtio-dev] " Heng Qi @ 2023-02-22 6:29 ` Michael S. Tsirkin 2023-02-22 6:49 ` [virtio-comment] " Heng Qi 2023-02-22 14:28 ` [virtio-comment] " Parav Pandit 2023-02-23 10:01 ` [virtio-comment] " David Edmondson 2 siblings, 2 replies; 12+ messages in thread From: Michael S. Tsirkin @ 2023-02-22 6:29 UTC (permalink / raw) To: Parav Pandit Cc: Heng Qi, virtio-dev@lists.oasis-open.org, virtio-comment@lists.oasis-open.org, Alvaro Karsz, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck On Wed, Feb 22, 2023 at 04:13:06AM +0000, Parav Pandit wrote: > > > From: Heng Qi <hengqi@linux.alibaba.com> > > Sent: Tuesday, February 21, 2023 10:22 PM > > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue > > notification coalescing. > > + > s/notification/notifications > should be plural as multiple notifications are coalesced like the below description of VIRTIO_NET_F_NOTF_COAL. Hmm nope - this is the "reduced relative" construct. It's always in a singular form. The full form would be "coalescing of notifications of virtqueues" and that indeed would use a plural form. But technical writing is usually using the reduced relative form. > > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications > > coalescing. > > > > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. > > @@ -139,6 +141,7 @@ \subsubsection{Feature bit > > requirements}\label{sec:Device Types / Network Device > > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. > > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or > > VIRTIO_NET_F_HOST_TSO6. > > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. > > \end{description} > > > > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / > > Network Device / Feature bits / Legacy Interface: Feature bits} @@ -1508,6 > > +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / > > Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, > > the driver can send control commands for dynamically changing the coalescing > > parameters. > > > > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: > > +\begin{itemize} > > +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command > > to set coalescing parameters of a given > > + enabled transmit/receive virtqueue. > > +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command > > to a device, and the device responds with > > + coalescing parameters of a given enabled transmit/receive virtqueue. > > +\end{itemize} > > + > > \begin{note} > > The behavior of the device in response to these commands is best-effort: > > the device may generate notifications more or less frequently than specified. > > @@ -1519,25 +1530,76 @@ \subsubsection{Control > > Virtqueue}\label{sec:Device Types / Network Device / Devi > > le32 max_usecs; > > }; > > > > +struct virtio_net_ctrl_coal_vq { > > + le16 vqn; > > + le16 reserved; > > + struct virtio_net_ctrl_coal coal; > > +}; > > + > > #define VIRTIO_NET_CTRL_NOTF_COAL 6 > > #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 > > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 > > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 #define > > + VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 > > \end{lstlisting} > > > > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > > +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the > > virtio_net_ctrl_coal > > +structure to set \field{max_usecs} and \field{max_packets} for all > > transmit/receive virtqueues. > > + > > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the > > +virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and > > \field{max_packets} for the supplied virtqueue number \field{vqn}. > > + > > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of > > +\field{max_usecs} and \field{max_packets} of the specified virtqueue > > +from the device by setting \field{vqn} in the virtio_net_ctrl_coal_vq structure. > > + > > > +# Read/Write attributes for coalescing parameters \begin{itemize} \item > > +For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > > VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} > > + and \field{max_packets} are write-only for a driver. > > +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, > > Virtio spec is using vq number and vq index terminology interchangeably. > For example, a new patch about admin vq, uses aq_start_index. > > MMIO device has vq_index register too. > I am inclined to vq_index given current state of spec [mst@tuck virtio]$ git grep vq_index|wc -l 0 > and new additions by Michael. admin vq? Good point, I will change that to vqn too. > Michael, > Can you please suggest vqn or vq_index to use? [mst@tuck virtio]$ git grep vqn|wc -l 5 I'd say vqn has precedent. > > \field{reserved}, \field{max_usecs} > > + and \field{max_packets} are write-only for a driver. > > +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} > > and \field{reserved} are write-only > > > > + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only > > for a driver. > Remove this trailing "for a driver", it is a duplicate. Nope it's required, without it, it's unclear which side is doing reading and which writing. Unfortunate that it's a bit verbose but we don't have better terms in the spec right now. > > +\end{itemize} > > + > > Coalescing parameters: > > \begin{itemize} > > +\item \field{vqn}: The virtqueue number of an enabled transmit or receive > > virtqueue. > > \item \field{max_usecs} for RX: Maximum number of microseconds to delay a > > RX notification. > > \item \field{max_usecs} for TX: Maximum number of microseconds to delay a > > TX notification. > > \item \field{max_packets} for RX: Maximum number of packets to receive > > before a RX notification. > > \item \field{max_packets} for TX: Maximum number of packets to send before > > a TX notification. > > \end{itemize} > > > > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: > > +\field{vqn} points to an enabled transmit/receive virtqueue, and its value > > satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. > > + > This calculation description is not needed here. It is covered somewhere else. I agree. > > +\field{reserved} is reserved and it is ignored by a device. > > + > > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: > > \begin{enumerate} > > -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and > > \field{max_packets} parameters for all transmit virtqueues. > > -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and > > \field{max_packets} parameters for all receive virtqueues. > > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and > > \field{max_packets} parameters for an enabled transmit/receive > > + virtqueue whose number is \field{vqn}. > > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the > > \field{max_usecs} and \field{max_packets} parameters for an enabled > > + transmit/receive virtqueue whose number is \field{vqn}. > > +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the > > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each > > virtqueue of transmitq1\ldots transmitqN. > > +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the > > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each > > virtqueue of receiveq1\ldots receiveqN. > > \end{enumerate} > > > > +If coalescing parameters are being set, the device applies the last > > +coalescing parameters received for a virtqueue, regardless of the command > > used to set the parameters. For example with 2 pairs of virtqueues: > > +# Command sequence > > +Each of the following commands sets \field{max_usecs} and > > \field{max_packets} parameters for virtqueues. > > +\begin{itemize} > > +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing > > parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 > > retain their previous parameter values. > > +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = > > 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values > > from command1. > > +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = > > 0, the device responds with coalescing parameters of virtqueue0 set by > > command2. > > +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = > > 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous > > values. > > +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing > > parameters for virtqueue1 and virtqueue3, and overrides the values set by > > command4. > > +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = > > 1, the device responds with coalescing parameters of virtqueue1 set by > > command5. > > +\end{itemize} > > + > > \subparagraph{Operation}\label{sec:Device Types / Network Device / Device > > Operation / Control Virtqueue / Notifications Coalescing / Operation} > > > > The device sends a used buffer notification once the notification conditions are > > met and if the notifications are not suppressed as explained in \ref{sec:Basic > > Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. > > @@ -1549,6 +1611,16 @@ \subsubsection{Control > > Virtqueue}\label{sec:Device Types / Network Device / Devi > > > > When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the > > notification conditions are met after every packet received/sent. > > > > +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL > > class > > +to set a coalescing parameter, it may set the parameter to a value close to a > > power of 2. For example: > > +If the device receives \field{max_usecs} = 7 from the > > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set > > \field{max_usecs} = 8 for a given enabled virtqueue. > > + > > +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > > +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, it saves the values of > > +coalescing parameters as global values, and the > > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command does not change the global > > values. If the device is reset, the global values will be set to 0. > > + > > +When a virtqueue is reset, its coalescing parameters are set to the global > > values. > > + > A VQ reset operation disables (or destroys) the VQ in the device. > Hence, a vq under reset doesn't have any valid parameters. > Therefore, above wording should be, > > When a virtqueue is enabled after it is reset, its coalescing parameters are set to global values as configured by VIRTIO_NET_CTRL_NOTF_COAL_TX_SET or VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. Right though this repeats "it" twice. At which point the second it becomes slightly confusing. Not too bad but ""enabled after virtqueue reset" would address this. > > \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device > > Operation / Control Virtqueue / Notifications Coalescing / RX Example} > > > > If, for example: > > @@ -1585,11 +1657,29 @@ \subsubsection{Control > > Virtqueue}\label{sec:Device Types / Network Device / Devi > > > > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / > > Network Device / Device Operation / Control Virtqueue / Notifications > > Coalescing} > > > > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver > > MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. > > +If neither the VIRTIO_NET_F_NOTF_COAL nor the > > VIRTIO_NET_F_VQ_NOTF_COAL > > +feature has been negotiated, the driver MUST NOT issue > > VIRTIO_NET_CTRL_NOTF_COAL commands. > > + > > +A driver MUST ignore the values of coalescing parameters received from the > > VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with > > VIRTIO_NET_ERR. > > > > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / > > Network Device / Device Operation / Control Virtqueue / Notifications > > Coalescing} > > > > -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands > > with VIRTIO_NET_ERR if it was not able to change the parameters. > > +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > > VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it > > was not able to change the parameters. > > + > > +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET > > command with VIRTIO_NET_ERR if it was not able to change the parameters. > > + > > +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and > > VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if > > the given virtqueue is disabled. > > + > > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and > > +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters > > for all transmit/receive virtqueues respectively and values of coalescing > > parameters are recorded as global values by a device. > > +The device MUST set the global values of coalescing parameters to 0 after > > being reset. > > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing > > parameters for a given enabled virtqueue without changing the global values. > > + > > +After disabling and re-enabling a virtqueue, the device MUST revert coalescing > > parameters of the virtqueue to the global values. > > + > > +A device MAY set the coalescing parameter to a value close to a power of 2 > > value. > > + > > +A device MUST ignore \field{reserved}. > > > > A device SHOULD NOT send used buffer notifications to the driver if the > > notifications are suppressed, even if the notification conditions are met. > > > > -- > > 2.19.1.6.gb485710b ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [virtio-comment] Re: [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 6:29 ` Michael S. Tsirkin @ 2023-02-22 6:49 ` Heng Qi 2023-02-22 14:28 ` [virtio-comment] " Parav Pandit 1 sibling, 0 replies; 12+ messages in thread From: Heng Qi @ 2023-02-22 6:49 UTC (permalink / raw) To: Michael S. Tsirkin, Parav Pandit Cc: virtio-dev@lists.oasis-open.org, virtio-comment@lists.oasis-open.org, Alvaro Karsz, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck 在 2023/2/22 下午2:29, Michael S. Tsirkin 写道: > On Wed, Feb 22, 2023 at 04:13:06AM +0000, Parav Pandit wrote: >>> From: Heng Qi <hengqi@linux.alibaba.com> >>> Sent: Tuesday, February 21, 2023 10:22 PM >>> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue >>> notification coalescing. >>> + >> s/notification/notifications >> should be plural as multiple notifications are coalesced like the below description of VIRTIO_NET_F_NOTF_COAL. > Hmm nope - this is the "reduced relative" construct. It's always in a singular > form. The full form would be "coalescing of notifications of virtqueues" > and that indeed would use a plural form. But technical writing is > usually using the reduced relative form. > >>> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications >>> coalescing. >>> >>> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. >>> @@ -139,6 +141,7 @@ \subsubsection{Feature bit >>> requirements}\label{sec:Device Types / Network Device >>> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >>> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or >>> VIRTIO_NET_F_HOST_TSO6. >>> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. >>> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >>> \end{description} >>> >>> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / >>> Network Device / Feature bits / Legacy Interface: Feature bits} @@ -1508,6 >>> +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / >>> Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, >>> the driver can send control commands for dynamically changing the coalescing >>> parameters. >>> >>> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: >>> +\begin{itemize} >>> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command >>> to set coalescing parameters of a given >>> + enabled transmit/receive virtqueue. >>> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command >>> to a device, and the device responds with >>> + coalescing parameters of a given enabled transmit/receive virtqueue. >>> +\end{itemize} >>> + >>> \begin{note} >>> The behavior of the device in response to these commands is best-effort: >>> the device may generate notifications more or less frequently than specified. >>> @@ -1519,25 +1530,76 @@ \subsubsection{Control >>> Virtqueue}\label{sec:Device Types / Network Device / Devi >>> le32 max_usecs; >>> }; >>> >>> +struct virtio_net_ctrl_coal_vq { >>> + le16 vqn; >>> + le16 reserved; >>> + struct virtio_net_ctrl_coal coal; >>> +}; >>> + >>> #define VIRTIO_NET_CTRL_NOTF_COAL 6 >>> #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 >>> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 >>> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 #define >>> + VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 >>> \end{lstlisting} >>> >>> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >>> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the >>> virtio_net_ctrl_coal >>> +structure to set \field{max_usecs} and \field{max_packets} for all >>> transmit/receive virtqueues. >>> + >>> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the >>> +virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and >>> \field{max_packets} for the supplied virtqueue number \field{vqn}. >>> + >>> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of >>> +\field{max_usecs} and \field{max_packets} of the specified virtqueue >>> +from the device by setting \field{vqn} in the virtio_net_ctrl_coal_vq structure. >>> + >>> +# Read/Write attributes for coalescing parameters \begin{itemize} \item >>> +For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >>> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} >>> + and \field{max_packets} are write-only for a driver. >>> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, >> Virtio spec is using vq number and vq index terminology interchangeably. >> For example, a new patch about admin vq, uses aq_start_index. >> >> MMIO device has vq_index register too. >> I am inclined to vq_index given current state of spec > [mst@tuck virtio]$ git grep vq_index|wc -l > 0 > >> and new additions by Michael. > admin vq? Good point, I will change that to vqn too. > >> Michael, >> Can you please suggest vqn or vq_index to use? > [mst@tuck virtio]$ git grep vqn|wc -l > 5 > > I'd say vqn has precedent. > > >>> \field{reserved}, \field{max_usecs} >>> + and \field{max_packets} are write-only for a driver. >>> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} >>> and \field{reserved} are write-only >> >>> + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only >>> for a driver. >> Remove this trailing "for a driver", it is a duplicate. > Nope it's required, without it, it's unclear which side is doing reading > and which writing. Unfortunate that it's a bit verbose but we don't have > better terms in the spec right now. > >>> +\end{itemize} >>> + >>> Coalescing parameters: >>> \begin{itemize} >>> +\item \field{vqn}: The virtqueue number of an enabled transmit or receive >>> virtqueue. >>> \item \field{max_usecs} for RX: Maximum number of microseconds to delay a >>> RX notification. >>> \item \field{max_usecs} for TX: Maximum number of microseconds to delay a >>> TX notification. >>> \item \field{max_packets} for RX: Maximum number of packets to receive >>> before a RX notification. >>> \item \field{max_packets} for TX: Maximum number of packets to send before >>> a TX notification. >>> \end{itemize} >>> >>> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: >>> +\field{vqn} points to an enabled transmit/receive virtqueue, and its value >>> satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. >>> + >> This calculation description is not needed here. It is covered somewhere else. > > I agree. > >>> +\field{reserved} is reserved and it is ignored by a device. >>> + >>> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: >>> \begin{enumerate} >>> -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and >>> \field{max_packets} parameters for all transmit virtqueues. >>> -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and >>> \field{max_packets} parameters for all receive virtqueues. >>> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and >>> \field{max_packets} parameters for an enabled transmit/receive >>> + virtqueue whose number is \field{vqn}. >>> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the >>> \field{max_usecs} and \field{max_packets} parameters for an enabled >>> + transmit/receive virtqueue whose number is \field{vqn}. >>> +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the >>> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each >>> virtqueue of transmitq1\ldots transmitqN. >>> +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the >>> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each >>> virtqueue of receiveq1\ldots receiveqN. >>> \end{enumerate} >>> >>> +If coalescing parameters are being set, the device applies the last >>> +coalescing parameters received for a virtqueue, regardless of the command >>> used to set the parameters. For example with 2 pairs of virtqueues: >>> +# Command sequence >>> +Each of the following commands sets \field{max_usecs} and >>> \field{max_packets} parameters for virtqueues. >>> +\begin{itemize} >>> +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing >>> parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 >>> retain their previous parameter values. >>> +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = >>> 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values >>> from command1. >>> +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = >>> 0, the device responds with coalescing parameters of virtqueue0 set by >>> command2. >>> +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = >>> 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous >>> values. >>> +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing >>> parameters for virtqueue1 and virtqueue3, and overrides the values set by >>> command4. >>> +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = >>> 1, the device responds with coalescing parameters of virtqueue1 set by >>> command5. >>> +\end{itemize} >>> + >>> \subparagraph{Operation}\label{sec:Device Types / Network Device / Device >>> Operation / Control Virtqueue / Notifications Coalescing / Operation} >>> >>> The device sends a used buffer notification once the notification conditions are >>> met and if the notifications are not suppressed as explained in \ref{sec:Basic >>> Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. >>> @@ -1549,6 +1611,16 @@ \subsubsection{Control >>> Virtqueue}\label{sec:Device Types / Network Device / Devi >>> >>> When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the >>> notification conditions are met after every packet received/sent. >>> >>> +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL >>> class >>> +to set a coalescing parameter, it may set the parameter to a value close to a >>> power of 2. For example: >>> +If the device receives \field{max_usecs} = 7 from the >>> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set >>> \field{max_usecs} = 8 for a given enabled virtqueue. >>> + >>> +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >>> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, it saves the values of >>> +coalescing parameters as global values, and the >>> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command does not change the global >>> values. If the device is reset, the global values will be set to 0. >>> + >>> +When a virtqueue is reset, its coalescing parameters are set to the global >>> values. >>> + >> A VQ reset operation disables (or destroys) the VQ in the device. >> Hence, a vq under reset doesn't have any valid parameters. >> Therefore, above wording should be, >> >> When a virtqueue is enabled after it is reset, its coalescing parameters are set to global values as configured by VIRTIO_NET_CTRL_NOTF_COAL_TX_SET or VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. > Right though this repeats "it" twice. At which point the second it > becomes slightly confusing. Not too bad but ""enabled after virtqueue > reset" would address this. Ok, I'll modify this. Thanks. > > >>> \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device >>> Operation / Control Virtqueue / Notifications Coalescing / RX Example} >>> >>> If, for example: >>> @@ -1585,11 +1657,29 @@ \subsubsection{Control >>> Virtqueue}\label{sec:Device Types / Network Device / Devi >>> >>> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / >>> Network Device / Device Operation / Control Virtqueue / Notifications >>> Coalescing} >>> >>> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver >>> MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. >>> +If neither the VIRTIO_NET_F_NOTF_COAL nor the >>> VIRTIO_NET_F_VQ_NOTF_COAL >>> +feature has been negotiated, the driver MUST NOT issue >>> VIRTIO_NET_CTRL_NOTF_COAL commands. >>> + >>> +A driver MUST ignore the values of coalescing parameters received from the >>> VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with >>> VIRTIO_NET_ERR. >>> >>> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / >>> Network Device / Device Operation / Control Virtqueue / Notifications >>> Coalescing} >>> >>> -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands >>> with VIRTIO_NET_ERR if it was not able to change the parameters. >>> +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >>> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it >>> was not able to change the parameters. >>> + >>> +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET >>> command with VIRTIO_NET_ERR if it was not able to change the parameters. >>> + >>> +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and >>> VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if >>> the given virtqueue is disabled. >>> + >>> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >>> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters >>> for all transmit/receive virtqueues respectively and values of coalescing >>> parameters are recorded as global values by a device. >>> +The device MUST set the global values of coalescing parameters to 0 after >>> being reset. >>> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing >>> parameters for a given enabled virtqueue without changing the global values. >>> + >>> +After disabling and re-enabling a virtqueue, the device MUST revert coalescing >>> parameters of the virtqueue to the global values. >>> + >>> +A device MAY set the coalescing parameter to a value close to a power of 2 >>> value. >>> + >>> +A device MUST ignore \field{reserved}. >>> >>> A device SHOULD NOT send used buffer notifications to the driver if the >>> notifications are suppressed, even if the notification conditions are met. >>> >>> -- >>> 2.19.1.6.gb485710b > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 12+ messages in thread
* [virtio-comment] RE: [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 6:29 ` Michael S. Tsirkin 2023-02-22 6:49 ` [virtio-comment] " Heng Qi @ 2023-02-22 14:28 ` Parav Pandit 2023-02-22 15:21 ` [virtio-dev] " Heng Qi 1 sibling, 1 reply; 12+ messages in thread From: Parav Pandit @ 2023-02-22 14:28 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Heng Qi, virtio-dev@lists.oasis-open.org, virtio-comment@lists.oasis-open.org, Alvaro Karsz, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck > From: Michael S. Tsirkin <mst@redhat.com> > Sent: Wednesday, February 22, 2023 1:29 AM > > MMIO device has vq_index register too. > > I am inclined to vq_index given current state of spec > > [mst@tuck virtio]$ git grep vq_index|wc -l > 0 > :) Grep for "virtqueue index". We need to change "virtqueue index" to "virtqueue number". > > and new additions by Michael. > > admin vq? Good point, I will change that to vqn too. Ok. sounds good. > > > Michael, > > Can you please suggest vqn or vq_index to use? > > [mst@tuck virtio]$ git grep vqn|wc -l > 5 > > I'd say vqn has precedent. > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [virtio-dev] RE: [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 14:28 ` [virtio-comment] " Parav Pandit @ 2023-02-22 15:21 ` Heng Qi 0 siblings, 0 replies; 12+ messages in thread From: Heng Qi @ 2023-02-22 15:21 UTC (permalink / raw) To: Parav Pandit, Michael S. Tsirkin Cc: virtio-dev@lists.oasis-open.org, virtio-comment@lists.oasis-open.org, Alvaro Karsz, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck Hi, Parav. 在 2023/2/22 下午10:28, Parav Pandit 写道: > >> From: Michael S. Tsirkin <mst@redhat.com> >> Sent: Wednesday, February 22, 2023 1:29 AM >>> MMIO device has vq_index register too. >>> I am inclined to vq_index given current state of spec >> [mst@tuck virtio]$ git grep vq_index|wc -l >> 0 >> > :) > > Grep for "virtqueue index". > We need to change "virtqueue index" to "virtqueue number". Yes, this is more unified in the spec. :) This version is closed, please help review the v7 version [1] if you have time. [1] https://lists.oasis-open.org/archives/virtio-dev/202302/msg00520.html Thanks very much for your response! > >>> and new additions by Michael. >> admin vq? Good point, I will change that to vqn too. > Ok. sounds good. > >>> Michael, >>> Can you please suggest vqn or vq_index to use? >> [mst@tuck virtio]$ git grep vqn|wc -l >> 5 >> >> I'd say vqn has precedent. >> > --------------------------------------------------------------------- > To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org > For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org ^ permalink raw reply [flat|nested] 12+ messages in thread
* [virtio-comment] Re: [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 4:13 ` Parav Pandit 2023-02-22 5:31 ` [virtio-dev] " Heng Qi 2023-02-22 6:29 ` Michael S. Tsirkin @ 2023-02-23 10:01 ` David Edmondson 2 siblings, 0 replies; 12+ messages in thread From: David Edmondson @ 2023-02-23 10:01 UTC (permalink / raw) To: Parav Pandit Cc: Heng Qi, virtio-dev@lists.oasis-open.org, virtio-comment@lists.oasis-open.org, Michael S . Tsirkin, Alvaro Karsz, Jason Wang, Xuan Zhuo, Cornelia Huck On Wednesday, 2023-02-22 at 04:13:06 UTC, Parav Pandit wrote: >> From: Heng Qi <hengqi@linux.alibaba.com> >> Sent: Tuesday, February 21, 2023 10:22 PM > >> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue >> notification coalescing. >> + > s/notification/notifications > should be plural as multiple notifications are coalesced like the below description of VIRTIO_NET_F_NOTF_COAL. Using the plural sounds weird, and isn't it implied by "coalescing"? If there were not multiple, there would be nothing to coalesce :-) >> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications >> coalescing. >> >> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. >> @@ -139,6 +141,7 @@ \subsubsection{Feature bit >> requirements}\label{sec:Device Types / Network Device >> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or >> VIRTIO_NET_F_HOST_TSO6. >> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. >> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >> \end{description} >> >> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / >> Network Device / Feature bits / Legacy Interface: Feature bits} @@ -1508,6 >> +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / >> Network Device / Devi If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, >> the driver can send control commands for dynamically changing the coalescing >> parameters. >> >> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: >> +\begin{itemize} >> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command >> to set coalescing parameters of a given >> + enabled transmit/receive virtqueue. >> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command >> to a device, and the device responds with >> + coalescing parameters of a given enabled transmit/receive virtqueue. >> +\end{itemize} >> + >> \begin{note} >> The behavior of the device in response to these commands is best-effort: >> the device may generate notifications more or less frequently than specified. >> @@ -1519,25 +1530,76 @@ \subsubsection{Control >> Virtqueue}\label{sec:Device Types / Network Device / Devi >> le32 max_usecs; >> }; >> >> +struct virtio_net_ctrl_coal_vq { >> + le16 vqn; >> + le16 reserved; >> + struct virtio_net_ctrl_coal coal; >> +}; >> + >> #define VIRTIO_NET_CTRL_NOTF_COAL 6 >> #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 >> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 >> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 #define >> + VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 >> \end{lstlisting} >> >> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the >> virtio_net_ctrl_coal >> +structure to set \field{max_usecs} and \field{max_packets} for all >> transmit/receive virtqueues. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the >> +virtio_net_ctrl_coal_vq structure to set \field{max_usecs} and >> \field{max_packets} for the supplied virtqueue number \field{vqn}. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of >> +\field{max_usecs} and \field{max_packets} of the specified virtqueue >> +from the device by setting \field{vqn} in the virtio_net_ctrl_coal_vq structure. >> + > >> +# Read/Write attributes for coalescing parameters \begin{itemize} \item >> +For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} >> + and \field{max_packets} are write-only for a driver. >> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, > > Virtio spec is using vq number and vq index terminology interchangeably. > For example, a new patch about admin vq, uses aq_start_index. > > MMIO device has vq_index register too. > I am inclined to vq_index given current state of spec and new additions by Michael. > > Michael, > Can you please suggest vqn or vq_index to use? > >> \field{reserved}, \field{max_usecs} >> + and \field{max_packets} are write-only for a driver. >> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} >> and \field{reserved} are write-only > > >> + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only >> for a driver. > Remove this trailing "for a driver", it is a duplicate. > >> +\end{itemize} >> + >> Coalescing parameters: >> \begin{itemize} >> +\item \field{vqn}: The virtqueue number of an enabled transmit or receive >> virtqueue. >> \item \field{max_usecs} for RX: Maximum number of microseconds to delay a >> RX notification. >> \item \field{max_usecs} for TX: Maximum number of microseconds to delay a >> TX notification. >> \item \field{max_packets} for RX: Maximum number of packets to receive >> before a RX notification. >> \item \field{max_packets} for TX: Maximum number of packets to send before >> a TX notification. >> \end{itemize} >> >> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: >> +\field{vqn} points to an enabled transmit/receive virtqueue, and its value >> satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. >> + > This calculation description is not needed here. It is covered somewhere else. > >> +\field{reserved} is reserved and it is ignored by a device. >> + >> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: >> \begin{enumerate} >> -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and >> \field{max_packets} parameters for all transmit virtqueues. >> -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and >> \field{max_packets} parameters for all receive virtqueues. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and >> \field{max_packets} parameters for an enabled transmit/receive >> + virtqueue whose number is \field{vqn}. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the >> \field{max_usecs} and \field{max_packets} parameters for an enabled >> + transmit/receive virtqueue whose number is \field{vqn}. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each >> virtqueue of transmitq1\ldots transmitqN. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each >> virtqueue of receiveq1\ldots receiveqN. >> \end{enumerate} >> >> +If coalescing parameters are being set, the device applies the last >> +coalescing parameters received for a virtqueue, regardless of the command >> used to set the parameters. For example with 2 pairs of virtqueues: >> +# Command sequence >> +Each of the following commands sets \field{max_usecs} and >> \field{max_packets} parameters for virtqueues. >> +\begin{itemize} >> +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing >> parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 >> retain their previous parameter values. >> +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = >> 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values >> from command1. >> +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = >> 0, the device responds with coalescing parameters of virtqueue0 set by >> command2. >> +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = >> 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous >> values. >> +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing >> parameters for virtqueue1 and virtqueue3, and overrides the values set by >> command4. >> +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = >> 1, the device responds with coalescing parameters of virtqueue1 set by >> command5. >> +\end{itemize} >> + >> \subparagraph{Operation}\label{sec:Device Types / Network Device / Device >> Operation / Control Virtqueue / Notifications Coalescing / Operation} >> >> The device sends a used buffer notification once the notification conditions are >> met and if the notifications are not suppressed as explained in \ref{sec:Basic >> Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. >> @@ -1549,6 +1611,16 @@ \subsubsection{Control >> Virtqueue}\label{sec:Device Types / Network Device / Devi >> >> When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the >> notification conditions are met after every packet received/sent. >> >> +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL >> class >> +to set a coalescing parameter, it may set the parameter to a value close to a >> power of 2. For example: >> +If the device receives \field{max_usecs} = 7 from the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set >> \field{max_usecs} = 8 for a given enabled virtqueue. >> + >> +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, it saves the values of >> +coalescing parameters as global values, and the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command does not change the global >> values. If the device is reset, the global values will be set to 0. >> + >> +When a virtqueue is reset, its coalescing parameters are set to the global >> values. >> + > A VQ reset operation disables (or destroys) the VQ in the device. > Hence, a vq under reset doesn't have any valid parameters. > Therefore, above wording should be, > > When a virtqueue is enabled after it is reset, its coalescing > parameters are set to global values as configured by > VIRTIO_NET_CTRL_NOTF_COAL_TX_SET or VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. > >> \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device >> Operation / Control Virtqueue / Notifications Coalescing / RX Example} >> >> If, for example: >> @@ -1585,11 +1657,29 @@ \subsubsection{Control >> Virtqueue}\label{sec:Device Types / Network Device / Devi >> >> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / >> Network Device / Device Operation / Control Virtqueue / Notifications >> Coalescing} >> >> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver >> MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. >> +If neither the VIRTIO_NET_F_NOTF_COAL nor the >> VIRTIO_NET_F_VQ_NOTF_COAL >> +feature has been negotiated, the driver MUST NOT issue >> VIRTIO_NET_CTRL_NOTF_COAL commands. >> + >> +A driver MUST ignore the values of coalescing parameters received from the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with >> VIRTIO_NET_ERR. >> >> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / >> Network Device / Device Operation / Control Virtqueue / Notifications >> Coalescing} >> >> -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands >> with VIRTIO_NET_ERR if it was not able to change the parameters. >> +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it >> was not able to change the parameters. >> + >> +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET >> command with VIRTIO_NET_ERR if it was not able to change the parameters. >> + >> +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if >> the given virtqueue is disabled. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and >> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters >> for all transmit/receive virtqueues respectively and values of coalescing >> parameters are recorded as global values by a device. >> +The device MUST set the global values of coalescing parameters to 0 after >> being reset. >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing >> parameters for a given enabled virtqueue without changing the global values. >> + >> +After disabling and re-enabling a virtqueue, the device MUST revert coalescing >> parameters of the virtqueue to the global values. >> + >> +A device MAY set the coalescing parameter to a value close to a power of 2 >> value. >> + >> +A device MUST ignore \field{reserved}. >> >> A device SHOULD NOT send used buffer notifications to the driver if the >> notifications are suppressed, even if the notification conditions are met. >> >> -- >> 2.19.1.6.gb485710b -- Tonight I think I'll walk alone, I'll find my soul as I go home. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [virtio-comment] [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 3:21 [PATCH v6] virtio-net: support the virtqueue coalescing moderation Heng Qi 2023-02-22 4:13 ` Parav Pandit @ 2023-02-22 8:09 ` Alvaro Karsz 2023-02-22 8:32 ` Heng Qi 1 sibling, 1 reply; 12+ messages in thread From: Alvaro Karsz @ 2023-02-22 8:09 UTC (permalink / raw) To: Heng Qi Cc: virtio-dev, virtio-comment, Michael S . Tsirkin, Parav Pandit, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck Hi Heng, > > Currently, coalescing parameters are grouped for all transmit and receive > virtqueues. This patch supports setting or getting the parameters for a > specified virtqueue, and a typical application of this function is netdim[1]. > > When the traffic between virtqueues is unbalanced, for example, one virtqueue > is busy and another virtqueue is idle, then it will be very useful to > control coalescing parameters at the virtqueue granularity. > > [1] https://docs.kernel.org/networking/net_dim.html > > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com> > Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > --- > This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html . > > v5->v6: > 1. Explain that the device may set a different value than the one passed in by the driver. @David Edmondson > > v4->v5: > 1. Add the correspondence between virtio_net_ctrl_coal and virtio_net_ctrl_coal_vq and control commands. @Michael S. Tsirkin > 2. Add read and write attributes for each field. @Michael S. Tsirkin > 3. A clearer description of how to set coalescing parameters for vq reset. @Michael S. Tsirkin > 4. Fix some syntax errors. @Michael S. Tsirkin, @David Edmondson > > v3->v4: > 1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz > 2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > 3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin > 4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > > v2->v3: > 1. Add the netdim link. @Parav Pandit > 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz > 3. _VQ_GET is explained more. @Michael S. Tsirkin > 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin > 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin > 7. Fix some typos. @Michael S. Tsirkin > > v1->v2: > 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin > 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin > 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin > 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz > 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > > device-types/net/description.tex | 100 +++++++++++++++++++++++++++++-- > 1 file changed, 95 insertions(+), 5 deletions(-) > > diff --git a/device-types/net/description.tex b/device-types/net/description.tex > index e71e33b..0d9f4b9 100644 > --- a/device-types/net/description.tex > +++ b/device-types/net/description.tex > @@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits > \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control > channel. > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. > + > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. > > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. > @@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. > \end{description} > > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits} > @@ -1508,6 +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi > If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can > send control commands for dynamically changing the coalescing parameters. > > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: > +\begin{itemize} > +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set coalescing parameters of a given > + enabled transmit/receive virtqueue. > +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device responds with > + coalescing parameters of a given enabled transmit/receive virtqueue. > +\end{itemize} > + > \begin{note} > The behavior of the device in response to these commands is best-effort: > the device may generate notifications more or less frequently than specified. > @@ -1519,25 +1530,76 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi > le32 max_usecs; > }; > > +struct virtio_net_ctrl_coal_vq { > + le16 vqn; > + le16 reserved; > + struct virtio_net_ctrl_coal coal; > +}; > + > #define VIRTIO_NET_CTRL_NOTF_COAL 6 > #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 > \end{lstlisting} > > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the > +virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all > +transmit/receive virtqueues. > + > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure > +to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}. > + > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and > +\field{max_packets} of the specified virtqueue from the device by setting \field{vqn} > +in the virtio_net_ctrl_coal_vq structure. > + > +# Read/Write attributes for coalescing parameters > +\begin{itemize} > +\item For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} > + and \field{max_packets} are write-only for a driver. > +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, \field{reserved}, \field{max_usecs} > + and \field{max_packets} are write-only for a driver. > +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} and \field{reserved} are write-only > + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only for a driver. > +\end{itemize} > + > Coalescing parameters: > \begin{itemize} > +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. > \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. > \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. > \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. > \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. > \end{itemize} > > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: > +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. > + > +\field{reserved} is reserved and it is ignored by a device. > + > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: > \begin{enumerate} > -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. > -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive > + virtqueue whose number is \field{vqn}. > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled > + transmit/receive virtqueue whose number is \field{vqn}. > +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. You mention later that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets the global parameters and repeating VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET for each transmitq doesn't, so it's not exactly the same effect. Same for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. > +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. > \end{enumerate} > > +If coalescing parameters are being set, the device applies the last coalescing parameters received for a > +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: > +# Command sequence > +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. > +\begin{itemize} > +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. > +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values from command1. > +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 0, the device responds with coalescing parameters of virtqueue0 set by command2. > +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous values. > +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. > +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 1, the device responds with coalescing parameters of virtqueue1 set by command5. > +\end{itemize} > + > \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} > > The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. > @@ -1549,6 +1611,16 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi > > When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent. > > +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL class to set a coalescing parameter, > +it may set the parameter to a value close to a power of 2. For example: > +If the device receives \field{max_usecs} = 7 from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set \field{max_usecs} = 8 for a given enabled virtqueue. > + > +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, > +it saves the values of coalescing parameters as global values, and the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command > +does not change the global values. If the device is reset, the global values will be set to 0. > + > +When a virtqueue is reset, its coalescing parameters are set to the global values. > + > \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Example} > > If, for example: > @@ -1585,11 +1657,29 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi > > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} > > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. > +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature > +has been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. > + > +A driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with VIRTIO_NET_ERR. > > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} > > -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. > +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters. > + > +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters. > + > +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the given virtqueue is disabled. > + > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters for all transmit/receive > +virtqueues respectively and values of coalescing parameters are recorded as global values by a device. > +The device MUST set the global values of coalescing parameters to 0 after being reset. > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. > + > +After disabling and re-enabling a virtqueue, the device MUST revert coalescing parameters of the virtqueue to the global values. > + > +A device MAY set the coalescing parameter to a value close to a power of 2 value. > + > +A device MUST ignore \field{reserved}. > > A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. > > -- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [virtio-comment] [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 8:09 ` [virtio-comment] " Alvaro Karsz @ 2023-02-22 8:32 ` Heng Qi 2023-02-22 11:48 ` Michael S. Tsirkin 0 siblings, 1 reply; 12+ messages in thread From: Heng Qi @ 2023-02-22 8:32 UTC (permalink / raw) To: Alvaro Karsz Cc: virtio-dev, virtio-comment, Michael S . Tsirkin, Parav Pandit, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck Hi, Alvaro. 在 2023/2/22 下午4:09, Alvaro Karsz 写道: > Hi Heng, > >> Currently, coalescing parameters are grouped for all transmit and receive >> virtqueues. This patch supports setting or getting the parameters for a >> specified virtqueue, and a typical application of this function is netdim[1]. >> >> When the traffic between virtqueues is unbalanced, for example, one virtqueue >> is busy and another virtqueue is idle, then it will be very useful to >> control coalescing parameters at the virtqueue granularity. >> >> [1] https://docs.kernel.org/networking/net_dim.html >> >> Signed-off-by: Heng Qi <hengqi@linux.alibaba.com> >> Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com> >> --- >> This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html . >> >> v5->v6: >> 1. Explain that the device may set a different value than the one passed in by the driver. @David Edmondson >> >> v4->v5: >> 1. Add the correspondence between virtio_net_ctrl_coal and virtio_net_ctrl_coal_vq and control commands. @Michael S. Tsirkin >> 2. Add read and write attributes for each field. @Michael S. Tsirkin >> 3. A clearer description of how to set coalescing parameters for vq reset. @Michael S. Tsirkin >> 4. Fix some syntax errors. @Michael S. Tsirkin, @David Edmondson >> >> v3->v4: >> 1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz >> 2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >> 3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin >> 4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >> >> v2->v3: >> 1. Add the netdim link. @Parav Pandit >> 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz >> 3. _VQ_GET is explained more. @Michael S. Tsirkin >> 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin >> 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >> 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin >> 7. Fix some typos. @Michael S. Tsirkin >> >> v1->v2: >> 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin >> 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin >> 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin >> 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >> 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz >> 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >> >> device-types/net/description.tex | 100 +++++++++++++++++++++++++++++-- >> 1 file changed, 95 insertions(+), 5 deletions(-) >> >> diff --git a/device-types/net/description.tex b/device-types/net/description.tex >> index e71e33b..0d9f4b9 100644 >> --- a/device-types/net/description.tex >> +++ b/device-types/net/description.tex >> @@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits >> \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control >> channel. >> >> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. >> + >> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. >> >> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. >> @@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device >> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. >> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. >> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >> \end{description} >> >> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits} >> @@ -1508,6 +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi >> If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can >> send control commands for dynamically changing the coalescing parameters. >> >> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: >> +\begin{itemize} >> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set coalescing parameters of a given >> + enabled transmit/receive virtqueue. >> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device responds with >> + coalescing parameters of a given enabled transmit/receive virtqueue. >> +\end{itemize} >> + >> \begin{note} >> The behavior of the device in response to these commands is best-effort: >> the device may generate notifications more or less frequently than specified. >> @@ -1519,25 +1530,76 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi >> le32 max_usecs; >> }; >> >> +struct virtio_net_ctrl_coal_vq { >> + le16 vqn; >> + le16 reserved; >> + struct virtio_net_ctrl_coal coal; >> +}; >> + >> #define VIRTIO_NET_CTRL_NOTF_COAL 6 >> #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 >> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 >> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 >> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 >> \end{lstlisting} >> >> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the >> +virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all >> +transmit/receive virtqueues. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure >> +to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and >> +\field{max_packets} of the specified virtqueue from the device by setting \field{vqn} >> +in the virtio_net_ctrl_coal_vq structure. >> + >> +# Read/Write attributes for coalescing parameters >> +\begin{itemize} >> +\item For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} >> + and \field{max_packets} are write-only for a driver. >> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, \field{reserved}, \field{max_usecs} >> + and \field{max_packets} are write-only for a driver. >> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} and \field{reserved} are write-only >> + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only for a driver. >> +\end{itemize} >> + >> Coalescing parameters: >> \begin{itemize} >> +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. >> \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. >> \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. >> \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. >> \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. >> \end{itemize} >> >> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: >> +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. >> + >> +\field{reserved} is reserved and it is ignored by a device. >> + >> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: >> \begin{enumerate} >> -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. >> -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive >> + virtqueue whose number is \field{vqn}. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled >> + transmit/receive virtqueue whose number is \field{vqn}. >> +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. > You mention later that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets the > global parameters and repeating VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET for > each transmitq doesn't, so it's not exactly the same effect. I think this is mainly talking about the process of setting coalescing parameters for virtqueues, and it doesn't seem to involve the process of " recording global values". How about this: "When setting the coalescing parameters, the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command has the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN." But this is a bit verbose. . Thanks. > Same for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. > >> +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. >> \end{enumerate} >> >> +If coalescing parameters are being set, the device applies the last coalescing parameters received for a >> +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: >> +# Command sequence >> +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. >> +\begin{itemize} >> +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. >> +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values from command1. >> +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 0, the device responds with coalescing parameters of virtqueue0 set by command2. >> +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous values. >> +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. >> +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 1, the device responds with coalescing parameters of virtqueue1 set by command5. >> +\end{itemize} >> + >> \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} >> >> The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. >> @@ -1549,6 +1611,16 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi >> >> When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent. >> >> +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL class to set a coalescing parameter, >> +it may set the parameter to a value close to a power of 2. For example: >> +If the device receives \field{max_usecs} = 7 from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set \field{max_usecs} = 8 for a given enabled virtqueue. >> + >> +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, >> +it saves the values of coalescing parameters as global values, and the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command >> +does not change the global values. If the device is reset, the global values will be set to 0. >> + >> +When a virtqueue is reset, its coalescing parameters are set to the global values. >> + >> \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Example} >> >> If, for example: >> @@ -1585,11 +1657,29 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi >> >> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} >> >> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. >> +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature >> +has been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. >> + >> +A driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with VIRTIO_NET_ERR. >> >> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} >> >> -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. >> +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters. >> + >> +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters. >> + >> +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the given virtqueue is disabled. >> + >> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters for all transmit/receive >> +virtqueues respectively and values of coalescing parameters are recorded as global values by a device. >> +The device MUST set the global values of coalescing parameters to 0 after being reset. >> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. >> + >> +After disabling and re-enabling a virtqueue, the device MUST revert coalescing parameters of the virtqueue to the global values. >> + >> +A device MAY set the coalescing parameter to a value close to a power of 2 value. >> + >> +A device MUST ignore \field{reserved}. >> >> A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. >> >> -- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [virtio-comment] [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 8:32 ` Heng Qi @ 2023-02-22 11:48 ` Michael S. Tsirkin 2023-02-22 12:50 ` Heng Qi 0 siblings, 1 reply; 12+ messages in thread From: Michael S. Tsirkin @ 2023-02-22 11:48 UTC (permalink / raw) To: Heng Qi Cc: Alvaro Karsz, virtio-dev, virtio-comment, Parav Pandit, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck On Wed, Feb 22, 2023 at 04:32:56PM +0800, Heng Qi wrote: > > Hi, Alvaro. > > 在 2023/2/22 下午4:09, Alvaro Karsz 写道: > > Hi Heng, > > > > > Currently, coalescing parameters are grouped for all transmit and receive > > > virtqueues. This patch supports setting or getting the parameters for a > > > specified virtqueue, and a typical application of this function is netdim[1]. > > > > > > When the traffic between virtqueues is unbalanced, for example, one virtqueue > > > is busy and another virtqueue is idle, then it will be very useful to > > > control coalescing parameters at the virtqueue granularity. > > > > > > [1] https://docs.kernel.org/networking/net_dim.html > > > > > > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com> > > > Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > > > --- > > > This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html . > > > > > > v5->v6: > > > 1. Explain that the device may set a different value than the one passed in by the driver. @David Edmondson > > > > > > v4->v5: > > > 1. Add the correspondence between virtio_net_ctrl_coal and virtio_net_ctrl_coal_vq and control commands. @Michael S. Tsirkin > > > 2. Add read and write attributes for each field. @Michael S. Tsirkin > > > 3. A clearer description of how to set coalescing parameters for vq reset. @Michael S. Tsirkin > > > 4. Fix some syntax errors. @Michael S. Tsirkin, @David Edmondson > > > > > > v3->v4: > > > 1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz > > > 2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > > > 3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin > > > 4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > > > > > > v2->v3: > > > 1. Add the netdim link. @Parav Pandit > > > 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz > > > 3. _VQ_GET is explained more. @Michael S. Tsirkin > > > 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin > > > 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > > > 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin > > > 7. Fix some typos. @Michael S. Tsirkin > > > > > > v1->v2: > > > 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin > > > 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin > > > 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin > > > 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > > > 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz > > > 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz > > > > > > device-types/net/description.tex | 100 +++++++++++++++++++++++++++++-- > > > 1 file changed, 95 insertions(+), 5 deletions(-) > > > > > > diff --git a/device-types/net/description.tex b/device-types/net/description.tex > > > index e71e33b..0d9f4b9 100644 > > > --- a/device-types/net/description.tex > > > +++ b/device-types/net/description.tex > > > @@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits > > > \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control > > > channel. > > > > > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. > > > + > > > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. > > > > > > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. > > > @@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device > > > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. > > > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. > > > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. > > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. > > > \end{description} > > > > > > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits} > > > @@ -1508,6 +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi > > > If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can > > > send control commands for dynamically changing the coalescing parameters. > > > > > > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: > > > +\begin{itemize} > > > +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set coalescing parameters of a given > > > + enabled transmit/receive virtqueue. > > > +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device responds with > > > + coalescing parameters of a given enabled transmit/receive virtqueue. > > > +\end{itemize} > > > + > > > \begin{note} > > > The behavior of the device in response to these commands is best-effort: > > > the device may generate notifications more or less frequently than specified. > > > @@ -1519,25 +1530,76 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi > > > le32 max_usecs; > > > }; > > > > > > +struct virtio_net_ctrl_coal_vq { > > > + le16 vqn; > > > + le16 reserved; > > > + struct virtio_net_ctrl_coal coal; > > > +}; > > > + > > > #define VIRTIO_NET_CTRL_NOTF_COAL 6 > > > #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 > > > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 > > > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 > > > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 > > > \end{lstlisting} > > > > > > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the > > > +virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all > > > +transmit/receive virtqueues. > > > + > > > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure > > > +to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}. > > > + > > > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and > > > +\field{max_packets} of the specified virtqueue from the device by setting \field{vqn} > > > +in the virtio_net_ctrl_coal_vq structure. > > > + > > > +# Read/Write attributes for coalescing parameters > > > +\begin{itemize} > > > +\item For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} > > > + and \field{max_packets} are write-only for a driver. > > > +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, \field{reserved}, \field{max_usecs} > > > + and \field{max_packets} are write-only for a driver. > > > +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} and \field{reserved} are write-only > > > + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only for a driver. > > > +\end{itemize} > > > + > > > Coalescing parameters: > > > \begin{itemize} > > > +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. > > > \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. > > > \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. > > > \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. > > > \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. > > > \end{itemize} > > > > > > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: > > > +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. > > > + > > > +\field{reserved} is reserved and it is ignored by a device. > > > + > > > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: > > > \begin{enumerate} > > > -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. > > > -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. > > > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive > > > + virtqueue whose number is \field{vqn}. > > > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled > > > + transmit/receive virtqueue whose number is \field{vqn}. > > > +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. > > You mention later that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets the > > global parameters and repeating VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET for > > each transmitq doesn't, so it's not exactly the same effect. > > I think this is mainly talking about the process of setting coalescing > parameters for virtqueues, and it doesn't seem to involve the process of " > recording global values". > > How about this: > "When setting the coalescing parameters, the > VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command has the same effect as the > VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of > transmitq1\ldots transmitqN." > > But this is a bit verbose. . > > Thanks. It's not really clear because "when" implies time not the desired effect. Just write out what it does, it is no longer the same IMHO. > > Same for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. > > > > > +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. > > > \end{enumerate} > > > > > > +If coalescing parameters are being set, the device applies the last coalescing parameters received for a > > > +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: > > > +# Command sequence > > > +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. > > > +\begin{itemize} > > > +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. > > > +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values from command1. > > > +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 0, the device responds with coalescing parameters of virtqueue0 set by command2. > > > +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous values. > > > +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. > > > +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 1, the device responds with coalescing parameters of virtqueue1 set by command5. > > > +\end{itemize} > > > + > > > \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} > > > > > > The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. > > > @@ -1549,6 +1611,16 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi > > > > > > When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent. > > > > > > +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL class to set a coalescing parameter, > > > +it may set the parameter to a value close to a power of 2. For example: > > > +If the device receives \field{max_usecs} = 7 from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set \field{max_usecs} = 8 for a given enabled virtqueue. > > > + > > > +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, > > > +it saves the values of coalescing parameters as global values, and the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command > > > +does not change the global values. If the device is reset, the global values will be set to 0. > > > + > > > +When a virtqueue is reset, its coalescing parameters are set to the global values. > > > + > > > \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Example} > > > > > > If, for example: > > > @@ -1585,11 +1657,29 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi > > > > > > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} > > > > > > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. > > > +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature > > > +has been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. > > > + > > > +A driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with VIRTIO_NET_ERR. > > > > > > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} > > > > > > -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. > > > +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters. > > > + > > > +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters. > > > + > > > +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the given virtqueue is disabled. > > > + > > > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters for all transmit/receive > > > +virtqueues respectively and values of coalescing parameters are recorded as global values by a device. > > > +The device MUST set the global values of coalescing parameters to 0 after being reset. > > > +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. > > > + > > > +After disabling and re-enabling a virtqueue, the device MUST revert coalescing parameters of the virtqueue to the global values. > > > + > > > +A device MAY set the coalescing parameter to a value close to a power of 2 value. > > > + > > > +A device MUST ignore \field{reserved}. > > > > > > A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. > > > > > > -- ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [virtio-comment] [PATCH v6] virtio-net: support the virtqueue coalescing moderation 2023-02-22 11:48 ` Michael S. Tsirkin @ 2023-02-22 12:50 ` Heng Qi 0 siblings, 0 replies; 12+ messages in thread From: Heng Qi @ 2023-02-22 12:50 UTC (permalink / raw) To: Michael S. Tsirkin Cc: Alvaro Karsz, virtio-dev, virtio-comment, Parav Pandit, David Edmondson, Jason Wang, Xuan Zhuo, Cornelia Huck 在 2023/2/22 下午7:48, Michael S. Tsirkin 写道: > On Wed, Feb 22, 2023 at 04:32:56PM +0800, Heng Qi wrote: >> Hi, Alvaro. >> >> 在 2023/2/22 下午4:09, Alvaro Karsz 写道: >>> Hi Heng, >>> >>>> Currently, coalescing parameters are grouped for all transmit and receive >>>> virtqueues. This patch supports setting or getting the parameters for a >>>> specified virtqueue, and a typical application of this function is netdim[1]. >>>> >>>> When the traffic between virtqueues is unbalanced, for example, one virtqueue >>>> is busy and another virtqueue is idle, then it will be very useful to >>>> control coalescing parameters at the virtqueue granularity. >>>> >>>> [1] https://docs.kernel.org/networking/net_dim.html >>>> >>>> Signed-off-by: Heng Qi <hengqi@linux.alibaba.com> >>>> Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com> >>>> --- >>>> This patch is on top of Alvaro's latest v7 patch: https://lists.oasis-open.org/archives/virtio-dev/202302/msg00431.html . >>>> >>>> v5->v6: >>>> 1. Explain that the device may set a different value than the one passed in by the driver. @David Edmondson >>>> >>>> v4->v5: >>>> 1. Add the correspondence between virtio_net_ctrl_coal and virtio_net_ctrl_coal_vq and control commands. @Michael S. Tsirkin >>>> 2. Add read and write attributes for each field. @Michael S. Tsirkin >>>> 3. A clearer description of how to set coalescing parameters for vq reset. @Michael S. Tsirkin >>>> 4. Fix some syntax errors. @Michael S. Tsirkin, @David Edmondson >>>> >>>> v3->v4: >>>> 1. Include virtio_net_ctrl_coal in the virtio_net_ctrl_coal_vq structure. @Alvaro Karsz >>>> 2. Add consideration of vq reset. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >>>> 3. Avoid too many examples by giving a comprehensive example. @Michael S. Tsirkin >>>> 4. Fix typos and streamline clarifications. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >>>> >>>> v2->v3: >>>> 1. Add the netdim link. @Parav Pandit >>>> 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz >>>> 3. _VQ_GET is explained more. @Michael S. Tsirkin >>>> 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin >>>> 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >>>> 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin >>>> 7. Fix some typos. @Michael S. Tsirkin >>>> >>>> v1->v2: >>>> 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin >>>> 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin >>>> 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin >>>> 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >>>> 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz >>>> 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz >>>> >>>> device-types/net/description.tex | 100 +++++++++++++++++++++++++++++-- >>>> 1 file changed, 95 insertions(+), 5 deletions(-) >>>> >>>> diff --git a/device-types/net/description.tex b/device-types/net/description.tex >>>> index e71e33b..0d9f4b9 100644 >>>> --- a/device-types/net/description.tex >>>> +++ b/device-types/net/description.tex >>>> @@ -83,6 +83,8 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits >>>> \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control >>>> channel. >>>> >>>> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports virtqueue notification coalescing. >>>> + >>>> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing. >>>> >>>> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets. >>>> @@ -139,6 +141,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device >>>> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >>>> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6. >>>> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ. >>>> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ. >>>> \end{description} >>>> >>>> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits} >>>> @@ -1508,6 +1511,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi >>>> If the VIRTIO_NET_F_NOTF_COAL feature is negotiated, the driver can >>>> send control commands for dynamically changing the coalescing parameters. >>>> >>>> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated: >>>> +\begin{itemize} >>>> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set coalescing parameters of a given >>>> + enabled transmit/receive virtqueue. >>>> +\item a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device responds with >>>> + coalescing parameters of a given enabled transmit/receive virtqueue. >>>> +\end{itemize} >>>> + >>>> \begin{note} >>>> The behavior of the device in response to these commands is best-effort: >>>> the device may generate notifications more or less frequently than specified. >>>> @@ -1519,25 +1530,76 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi >>>> le32 max_usecs; >>>> }; >>>> >>>> +struct virtio_net_ctrl_coal_vq { >>>> + le16 vqn; >>>> + le16 reserved; >>>> + struct virtio_net_ctrl_coal coal; >>>> +}; >>>> + >>>> #define VIRTIO_NET_CTRL_NOTF_COAL 6 >>>> #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0 >>>> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1 >>>> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 >>>> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3 >>>> \end{lstlisting} >>>> >>>> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands use the >>>> +virtio_net_ctrl_coal structure to set \field{max_usecs} and \field{max_packets} for all >>>> +transmit/receive virtqueues. >>>> + >>>> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command uses the virtio_net_ctrl_coal_vq structure >>>> +to set \field{max_usecs} and \field{max_packets} for the supplied virtqueue number \field{vqn}. >>>> + >>>> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command gets the values of \field{max_usecs} and >>>> +\field{max_packets} of the specified virtqueue from the device by setting \field{vqn} >>>> +in the virtio_net_ctrl_coal_vq structure. >>>> + >>>> +# Read/Write attributes for coalescing parameters >>>> +\begin{itemize} >>>> +\item For commands VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, \field{max_usecs} >>>> + and \field{max_packets} are write-only for a driver. >>>> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, \field{vqn}, \field{reserved}, \field{max_usecs} >>>> + and \field{max_packets} are write-only for a driver. >>>> +\item For the command VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, \field{vqn} and \field{reserved} are write-only >>>> + for a driver, and, \field{max_usecs} and \field{max_packets} are read-only for a driver. >>>> +\end{itemize} >>>> + >>>> Coalescing parameters: >>>> \begin{itemize} >>>> +\item \field{vqn}: The virtqueue number of an enabled transmit or receive virtqueue. >>>> \item \field{max_usecs} for RX: Maximum number of microseconds to delay a RX notification. >>>> \item \field{max_usecs} for TX: Maximum number of microseconds to delay a TX notification. >>>> \item \field{max_packets} for RX: Maximum number of packets to receive before a RX notification. >>>> \item \field{max_packets} for TX: Maximum number of packets to send before a TX notification. >>>> \end{itemize} >>>> >>>> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands: >>>> +\field{vqn} points to an enabled transmit/receive virtqueue, and its value satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $. >>>> + >>>> +\field{reserved} is reserved and it is ignored by a device. >>>> + >>>> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands: >>>> \begin{enumerate} >>>> -\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all transmit virtqueues. >>>> -\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{max_usecs} and \field{max_packets} parameters for all receive virtqueues. >>>> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_usecs} and \field{max_packets} parameters for an enabled transmit/receive >>>> + virtqueue whose number is \field{vqn}. >>>> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device returns the \field{max_usecs} and \field{max_packets} parameters for an enabled >>>> + transmit/receive virtqueue whose number is \field{vqn}. >>>> +\item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of transmitq1\ldots transmitqN. >>> You mention later that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets the >>> global parameters and repeating VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET for >>> each transmitq doesn't, so it's not exactly the same effect. >> I think this is mainly talking about the process of setting coalescing >> parameters for virtqueues, and it doesn't seem to involve the process of " >> recording global values". >> >> How about this: >> "When setting the coalescing parameters, the >> VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command has the same effect as the >> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of >> transmitq1\ldots transmitqN." >> >> But this is a bit verbose. . >> >> Thanks. > It's not really clear because "when" implies time not the desired > effect. Just write out what it does, it is no longer the same IMHO. > I see. Thanks. > >>> Same for VIRTIO_NET_CTRL_NOTF_COAL_RX_SET. >>> >>>> +\item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: have the same effect as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each virtqueue of receiveq1\ldots receiveqN. >>>> \end{enumerate} >>>> >>>> +If coalescing parameters are being set, the device applies the last coalescing parameters received for a >>>> +virtqueue, regardless of the command used to set the parameters. For example with 2 pairs of virtqueues: >>>> +# Command sequence >>>> +Each of the following commands sets \field{max_usecs} and \field{max_packets} parameters for virtqueues. >>>> +\begin{itemize} >>>> +\item Command1: VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets coalescing parameters for virtqueue0 and virtqueue2, and, virtqueue1 and virtqueue3 retain their previous parameter values. >>>> +\item Command2: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 0 sets coalescing parameters for virtqueue0, and virtqueue2 retains the values from command1. >>>> +\item Command3: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 0, the device responds with coalescing parameters of virtqueue0 set by command2. >>>> +\item Command4: VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET with \field{vqn} = 1 sets coalescing parameters for virtqueue1, and virtqueue3 retains its previous values. >>>> +\item Command5: VIRTIO_NET_CTRL_NOTF_COAL_TX_SET sets coalescing parameters for virtqueue1 and virtqueue3, and overrides the values set by command4. >>>> +\item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vqn} = 1, the device responds with coalescing parameters of virtqueue1 set by command5. >>>> +\end{itemize} >>>> + >>>> \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation} >>>> >>>> The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}. >>>> @@ -1549,6 +1611,16 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi >>>> >>>> When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent. >>>> >>>> +When a device receives a command of the VIRTIO_NET_CTRL_NOTF_COAL class to set a coalescing parameter, >>>> +it may set the parameter to a value close to a power of 2. For example: >>>> +If the device receives \field{max_usecs} = 7 from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command, it may set \field{max_usecs} = 8 for a given enabled virtqueue. >>>> + >>>> +When the device receives the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands, >>>> +it saves the values of coalescing parameters as global values, and the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command >>>> +does not change the global values. If the device is reset, the global values will be set to 0. >>>> + >>>> +When a virtqueue is reset, its coalescing parameters are set to the global values. >>>> + >>>> \subparagraph{RX Example}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Example} >>>> >>>> If, for example: >>>> @@ -1585,11 +1657,29 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi >>>> >>>> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} >>>> >>>> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. >>>> +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature >>>> +has been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands. >>>> + >>>> +A driver MUST ignore the values of coalescing parameters received from the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command if a device responds with VIRTIO_NET_ERR. >>>> >>>> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing} >>>> >>>> -A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters. >>>> +A device SHOULD respond to VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands with VIRTIO_NET_ERR if it was not able to change the parameters. >>>> + >>>> +A device MUST respond to the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with VIRTIO_NET_ERR if it was not able to change the parameters. >>>> + >>>> +A device MUST respond to VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET commands with VIRTIO_NET_ERR if the given virtqueue is disabled. >>>> + >>>> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and VIRTIO_NET_CTRL_NOTF_COAL_RX_SET commands set coalescing parameters for all transmit/receive >>>> +virtqueues respectively and values of coalescing parameters are recorded as global values by a device. >>>> +The device MUST set the global values of coalescing parameters to 0 after being reset. >>>> +The VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command sets the coalescing parameters for a given enabled virtqueue without changing the global values. >>>> + >>>> +After disabling and re-enabling a virtqueue, the device MUST revert coalescing parameters of the virtqueue to the global values. >>>> + >>>> +A device MAY set the coalescing parameter to a value close to a power of 2 value. >>>> + >>>> +A device MUST ignore \field{reserved}. >>>> >>>> A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met. >>>> >>>> -- ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2023-02-23 10:01 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2023-02-22 3:21 [PATCH v6] virtio-net: support the virtqueue coalescing moderation Heng Qi 2023-02-22 4:13 ` Parav Pandit 2023-02-22 5:31 ` [virtio-dev] " Heng Qi 2023-02-22 6:29 ` Michael S. Tsirkin 2023-02-22 6:49 ` [virtio-comment] " Heng Qi 2023-02-22 14:28 ` [virtio-comment] " Parav Pandit 2023-02-22 15:21 ` [virtio-dev] " Heng Qi 2023-02-23 10:01 ` [virtio-comment] " David Edmondson 2023-02-22 8:09 ` [virtio-comment] " Alvaro Karsz 2023-02-22 8:32 ` Heng Qi 2023-02-22 11:48 ` Michael S. Tsirkin 2023-02-22 12:50 ` Heng Qi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox