* Re: [virtio-dev] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 14:43 [PATCH v3] virtio-net: support the virtqueue coalescing moderation Heng Qi
@ 2023-02-16 14:49 ` Heng Qi
2023-02-16 15:12 ` Parav Pandit
2023-02-16 16:08 ` Michael S. Tsirkin
2023-02-16 16:17 ` Michael S. Tsirkin
` (3 subsequent siblings)
4 siblings, 2 replies; 29+ messages in thread
From: Heng Qi @ 2023-02-16 14:49 UTC (permalink / raw)
To: virtio-comment, virtio-dev, Michael S . Tsirkin
Cc: Parav Pandit, Yuri Benditovich, Jason Wang, Cornelia Huck,
Xuan Zhuo
在 2023/2/16 下午10:43, Heng Qi 写道:
> Currently coalescing parameters are grouped for all transmit and receive
> virtqueues. This patch supports setting or getting the parameters for a
> specified virtqueue, and a typical application of this function is netdim[1].
>
> When the traffic between virtqueues is unbalanced, for example, one virtqueue
> is busy and another virtqueue is idle, then it will be very useful to
> control coalescing parameters at the virtqueue granularity.
>
> [1] https://docs.kernel.org/networking/net_dim.html
>
> Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
> v2->v3:
> 1. Add the netdim link. @Parav Pandit
> 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
> 3. _VQ_GET is explained more. @Michael S. Tsirkin
> 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
> 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
> 7. Fix some typos. @Michael S. Tsirkin
Hi Michael.
I know you suggested in v2 that I split the virtqueue SET and GET
commands in series,
but I have a bit of a problem trying to split because there are some
statements where SET and GET are glued together.
Please see the following places.
If you think the split is better, I'll split it again and send it right
away.
Thanks.
>
> v1->v2:
> 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
> 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
> 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
> 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
> 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
>
> content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--
> 1 file changed, 126 insertions(+), 4 deletions(-)
>
> diff --git a/content.tex b/content.tex
> index e863709..7f99503 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> channel.
>
> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> + notifications coalescing.
> +
> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
>
> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> \end{description}
>
> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> };
>
> #define VIRTIO_NET_CTRL_NOTF_COAL 6
> - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
here
> +
> \end{lstlisting}
>
> Coalescing parameters:
> @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \end{itemize}
>
>
> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
here
> \begin{enumerate}
> \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> + transmit/receive virtqueue whose number is \field{vqn}.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> + responds them to the driver.
> \end{enumerate}
>
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> + parameters of a enabled transmit/receive virtqueue.
> +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
> +
> +\begin{lstlisting}
> +struct virtio_net_ctrl_coal_vq {
> + le16 vqn;
> + le16 reserved;
> + le32 max_packets;
> + le32 max_usecs;
> +};
> +\end{lstlisting}
> +
> +Virtqueue coalescing parameters:
> +\begin{itemize}
> +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> +\end{itemize}
> +
> +\field{reserved} is reserved and it is ignored by the device.
> +
> +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> +The conversion between \field{vqn} and transmitq/receiveq index:
> +\begin{itemize}
> +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> +\end{itemize}
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> +
> +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> +virtqueue, regardless of the command used to deliver the parameters. For example:
> +# Command sequence 1:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> +# Command sequence 2:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
>
> If, for example:
> @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>
> \begin{itemize}
> \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of setting coalescing parameters for a receive virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 0
> +\end{itemize}
> +
> +The device will operate as follows:
> +\begin{itemize}
> +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a receive virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 0.
> \end{itemize}
>
> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> +and then responds them to the driver.
> +
> \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
>
> If, for example:
> @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> \end{itemize}
>
> +Example of setting coalescing parameters for a transmit virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 1
> +\end{itemize}
> +
> +The device will operate as follows:
> +
> +\begin{itemize}
> +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a transmit virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 1.
> +\end{itemize}
> +
> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> +and then responds them to the driver.
> +
> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>
> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
> +
> +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
>
> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>
> A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
>
> +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
> +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
here
> +
> +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> +
> A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
>
> Upon reset, a device MUST initialize all coalescing parameters to 0.
^ permalink raw reply [flat|nested] 29+ messages in thread* RE: [virtio-dev] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 14:49 ` [virtio-dev] " Heng Qi
@ 2023-02-16 15:12 ` Parav Pandit
2023-02-16 16:08 ` Michael S. Tsirkin
1 sibling, 0 replies; 29+ messages in thread
From: Parav Pandit @ 2023-02-16 15:12 UTC (permalink / raw)
To: Heng Qi, virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, Michael S . Tsirkin
Cc: Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
> From: Heng Qi <hengqi@linux.alibaba.com>
> Sent: Thursday, February 16, 2023 9:49 AM
> Hi Michael.
>
> I know you suggested in v2 that I split the virtqueue SET and GET commands in
> series, but I have a bit of a problem trying to split because there are some
> statements where SET and GET are glued together.
> Please see the following places.
>
> If you think the split is better, I'll split it again and send it right away.
I agree with Michael, better to split.
It should be possible.
First patch without next patch consideration.
Second patch can modify the previous line, this is fine.
Its common practice and better even if it touches some of the same lines.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [virtio-dev] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 14:49 ` [virtio-dev] " Heng Qi
2023-02-16 15:12 ` Parav Pandit
@ 2023-02-16 16:08 ` Michael S. Tsirkin
2023-02-17 6:14 ` [virtio-comment] " Heng Qi
1 sibling, 1 reply; 29+ messages in thread
From: Michael S. Tsirkin @ 2023-02-16 16:08 UTC (permalink / raw)
To: Heng Qi
Cc: virtio-comment, virtio-dev, Parav Pandit, Yuri Benditovich,
Jason Wang, Cornelia Huck, Xuan Zhuo
On Thu, Feb 16, 2023 at 10:49:25PM +0800, Heng Qi wrote:
>
>
> 在 2023/2/16 下午10:43, Heng Qi 写道:
> > Currently coalescing parameters are grouped for all transmit and receive
> > virtqueues. This patch supports setting or getting the parameters for a
> > specified virtqueue, and a typical application of this function is netdim[1].
> >
> > When the traffic between virtqueues is unbalanced, for example, one virtqueue
> > is busy and another virtqueue is idle, then it will be very useful to
> > control coalescing parameters at the virtqueue granularity.
> >
> > [1] https://docs.kernel.org/networking/net_dim.html
> >
> > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> > Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> > v2->v3:
> > 1. Add the netdim link. @Parav Pandit
> > 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
> > 3. _VQ_GET is explained more. @Michael S. Tsirkin
> > 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
> > 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
> > 7. Fix some typos. @Michael S. Tsirkin
>
> Hi Michael.
>
> I know you suggested in v2 that I split the virtqueue SET and GET commands
> in series,
> but I have a bit of a problem trying to split because there are some
> statements where SET and GET are glued together.
> Please see the following places.
>
> If you think the split is better, I'll split it again and send it right
> away.
>
> Thanks.
I don't mind that much. Lots of tiny patches are annoying too. Parav
seems to want this though.
> >
> > v1->v2:
> > 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
> > 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
> > 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
> > 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
> > 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> >
> > content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--
> > 1 file changed, 126 insertions(+), 4 deletions(-)
> >
> > diff --git a/content.tex b/content.tex
> > index e863709..7f99503 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> > \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> > channel.
> > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> > + notifications coalescing.
> > +
> > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
> > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> > @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > \end{description}
> > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> > @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > };
> > #define VIRTIO_NET_CTRL_NOTF_COAL 6
> > - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
>
> here
>
> > +
> > \end{lstlisting}
> > Coalescing parameters:
> > @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > \end{itemize}
> > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
>
> here
>
> > \begin{enumerate}
> > \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> > \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> > + transmit/receive virtqueue whose number is \field{vqn}.
> > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> > + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> > + responds them to the driver.
> > \end{enumerate}
> > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> > +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> > + parameters of a enabled transmit/receive virtqueue.
an enabled
> > +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> > + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
an enabled
> > +
> > +\begin{lstlisting}
> > +struct virtio_net_ctrl_coal_vq {
> > + le16 vqn;
> > + le16 reserved;
> > + le32 max_packets;
> > + le32 max_usecs;
> > +};
> > +\end{lstlisting}
> > +
> > +Virtqueue coalescing parameters:
> > +\begin{itemize}
> > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > +\end{itemize}
> > +
> > +\field{reserved} is reserved and it is ignored by the device.
> > +
> > +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> > +The conversion between \field{vqn} and transmitq/receiveq index:
> > +\begin{itemize}
> > +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> > +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> > +\end{itemize}
> > +
> > +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
has the same effect
> > +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> > +
> > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
has the same effect
> > +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> > +
> > +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> > +virtqueue, regardless of the command used to deliver the parameters. For example:
> > +# Command sequence 1:
> > +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> > +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > +
> > +# Command sequence 2:
> > +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > +
> > \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
> > If, for example:
> > @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > \begin{itemize}
> > \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> > -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\end{itemize}
> > +
> > +Example of setting coalescing parameters for a receive virtqueue:
> > +\begin{itemize}
> > +\item \field{max_packets} = 15
> > +\item \field{max_usecs} = 10
> > +\item \field{vqn} = 0
> > +\end{itemize}
> > +
> > +The device will operate as follows:
> > +\begin{itemize}
> > +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> > +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
use tense consistently:
elapsed -> elapse,
was -> is
one refers to packets I guess but technically could refer to usecs.
one -> packet
usecs -> microsecons , eschew abbreviation
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\end{itemize}
> > +
> > +Example of getting coalescing parameters from a receive virtqueue:
> > +\begin{itemize}
> > +\item \field{vqn} = 0.
> > \end{itemize}
> > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> > +and then responds them to the driver.
respond is not a transitive verb
responds -> reports?
> > +
> > \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
> > If, for example:
> > @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > \end{itemize}
> > +Example of setting coalescing parameters for a transmit virtqueue:
> > +\begin{itemize}
> > +\item \field{max_packets} = 15
> > +\item \field{max_usecs} = 10
> > +\item \field{vqn} = 1
> > +\end{itemize}
> > +
> > +The device will operate as follows:
> > +
> > +\begin{itemize}
> > +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> > +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\end{itemize}
> > +
> > +Example of getting coalescing parameters from a transmit virtqueue:
> > +\begin{itemize}
> > +\item \field{vqn} = 1.
> > +\end{itemize}
> > +
> > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> > +and then responds them to the driver.
same
> > +
> > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> > +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > +
> > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> > +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
either all perfect or all in simple tense
and maybe just put the formula here.
> > +
> > +A driver MAY NOT get exactly the same value as the coalescing
> > parameter it was set to,
This is really a device conformance statement not a driver one.
And MAY NOT means it is not allowed.
just drop this here. or write that driver MUST NOT rely on getting same
# back.
> for example it MAY get a value that is a
> > power of 2.
this confuses more than it clarifies.
> > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> > A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
> > +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
virtqueue number, not virtqueue with number.
> > +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
>
> here
>
> > +
> > +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> > +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> > +
same as driver. MAY NOT means not allowed. should be MAY.
going overvoard with examples. keep them out of conformance sections
these must be clear by themselves.
> > A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
> > Upon reset, a device MUST initialize all coalescing parameters to 0.
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-comment] Re: [virtio-dev] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 16:08 ` Michael S. Tsirkin
@ 2023-02-17 6:14 ` Heng Qi
0 siblings, 0 replies; 29+ messages in thread
From: Heng Qi @ 2023-02-17 6:14 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, virtio-dev, Parav Pandit, Yuri Benditovich,
Jason Wang, Cornelia Huck, Xuan Zhuo, Alvaro Karsz
On Thu, Feb 16, 2023 at 11:08:54AM -0500, Michael S. Tsirkin wrote:
> On Thu, Feb 16, 2023 at 10:49:25PM +0800, Heng Qi wrote:
> >
> >
> > 在 2023/2/16 下午10:43, Heng Qi 写道:
> > > Currently coalescing parameters are grouped for all transmit and receive
> > > virtqueues. This patch supports setting or getting the parameters for a
> > > specified virtqueue, and a typical application of this function is netdim[1].
> > >
> > > When the traffic between virtqueues is unbalanced, for example, one virtqueue
> > > is busy and another virtqueue is idle, then it will be very useful to
> > > control coalescing parameters at the virtqueue granularity.
> > >
> > > [1] https://docs.kernel.org/networking/net_dim.html
> > >
> > > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> > > Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > ---
> > > v2->v3:
> > > 1. Add the netdim link. @Parav Pandit
> > > 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
> > > 3. _VQ_GET is explained more. @Michael S. Tsirkin
> > > 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
> > > 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > > 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
> > > 7. Fix some typos. @Michael S. Tsirkin
> >
> > Hi Michael.
> >
> > I know you suggested in v2 that I split the virtqueue SET and GET commands
> > in series,
> > but I have a bit of a problem trying to split because there are some
> > statements where SET and GET are glued together.
> > Please see the following places.
> >
> > If you think the split is better, I'll split it again and send it right
> > away.
> >
> > Thanks.
>
>
> I don't mind that much. Lots of tiny patches are annoying too. Parav
> seems to want this though.
>
> > >
> > > v1->v2:
> > > 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
> > > 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
> > > 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
> > > 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > > 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
> > > 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > >
> > > content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--
> > > 1 file changed, 126 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/content.tex b/content.tex
> > > index e863709..7f99503 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> > > \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> > > channel.
> > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> > > + notifications coalescing.
> > > +
> > > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
> > > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> > > @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> > > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> > > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > > \end{description}
> > > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> > > @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > };
> > > #define VIRTIO_NET_CTRL_NOTF_COAL 6
> > > - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > > + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> > > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> > > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> >
> > here
> >
> > > +
> > > \end{lstlisting}
> > > Coalescing parameters:
> > > @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > \end{itemize}
> > > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> > > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> >
> > here
> >
> > > \begin{enumerate}
> > > \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> > > \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> > > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> > > + transmit/receive virtqueue whose number is \field{vqn}.
> > > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> > > + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> > > + responds them to the driver.
> > > \end{enumerate}
> > > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> > > +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> > > + parameters of a enabled transmit/receive virtqueue.
>
> an enabled
>
Will fix.
> > > +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> > > + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
>
> an enabled
Will fix.
>
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_net_ctrl_coal_vq {
> > > + le16 vqn;
> > > + le16 reserved;
> > > + le32 max_packets;
> > > + le32 max_usecs;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +Virtqueue coalescing parameters:
> > > +\begin{itemize}
> > > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > > +\end{itemize}
> > > +
> > > +\field{reserved} is reserved and it is ignored by the device.
> > > +
> > > +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> > > +The conversion between \field{vqn} and transmitq/receiveq index:
> > > +\begin{itemize}
> > > +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> > > +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> > > +\end{itemize}
> > > +
> > > +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
>
> has the same effect
Ok.
>
> > > +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> > > +
> > > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
>
>
> has the same effect
Ok.
>
> > > +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> > > +
> > > +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> > > +virtqueue, regardless of the command used to deliver the parameters. For example:
> > > +# Command sequence 1:
> > > +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > > +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > > +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> > > +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > > +
> > > +# Command sequence 2:
> > > +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > > +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > > +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > > +
> > > \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
> > > If, for example:
> > > @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > \begin{itemize}
> > > \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> > > -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > > +\end{itemize}
> > > +
> > > +Example of setting coalescing parameters for a receive virtqueue:
> > > +\begin{itemize}
> > > +\item \field{max_packets} = 15
> > > +\item \field{max_usecs} = 10
> > > +\item \field{vqn} = 0
> > > +\end{itemize}
> > > +
> > > +The device will operate as follows:
> > > +\begin{itemize}
> > > +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> > > +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
>
> use tense consistently:
> elapsed -> elapse,
> was -> is
>
> one refers to packets I guess but technically could refer to usecs.
>
> one -> packet
>
> usecs -> microsecons , eschew abbreviation
Thanks for pointing this out, I think this is more clear.
>
>
> > > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > > +\end{itemize}
> > > +
> > > +Example of getting coalescing parameters from a receive virtqueue:
> > > +\begin{itemize}
> > > +\item \field{vqn} = 0.
> > > \end{itemize}
> > > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> > > +and then responds them to the driver.
>
> respond is not a transitive verb
> responds -> reports?
Ahh, will fix. But how about "returns" ?
>
> > > +
> > > \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
> > > If, for example:
> > > @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > > \end{itemize}
> > > +Example of setting coalescing parameters for a transmit virtqueue:
> > > +\begin{itemize}
> > > +\item \field{max_packets} = 15
> > > +\item \field{max_usecs} = 10
> > > +\item \field{vqn} = 1
> > > +\end{itemize}
> > > +
> > > +The device will operate as follows:
> > > +
> > > +\begin{itemize}
> > > +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> > > +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> > > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > > +\end{itemize}
> > > +
> > > +Example of getting coalescing parameters from a transmit virtqueue:
> > > +\begin{itemize}
> > > +\item \field{vqn} = 1.
> > > +\end{itemize}
> > > +
> > > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> > > +and then responds them to the driver.
>
> same
>
> > > +
> > > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> > > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > > +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> > > +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > > +
> > > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> > > +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
>
> either all perfect or all in simple tense
> and maybe just put the formula here.
Sure.
>
>
> > > +
> > > +A driver MAY NOT get exactly the same value as the coalescing
> > > parameter it was set to,
>
>
>
> This is really a device conformance statement not a driver one.
> And MAY NOT means it is not allowed.
> just drop this here. or write that driver MUST NOT rely on getting same
> # back.
OK, I've covered this in device conformance, and it seems better to drop this here.
>
> > for example it MAY get a value that is a
> > > power of 2.
>
> this confuses more than it clarifies.
>
> > > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> > > A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
> > > +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
>
> virtqueue number, not virtqueue with number.
Ok.
>
> > > +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
> >
> > here
> >
> > > +
> > > +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> > > +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> > > +
>
> same as driver. MAY NOT means not allowed. should be MAY.
> going overvoard with examples. keep them out of conformance sections
> these must be clear by themselves.
I see. Thanks a lot for your suggestion!
>
> > > A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
> > > Upon reset, a device MUST initialize all coalescing parameters to 0.
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 14:43 [PATCH v3] virtio-net: support the virtqueue coalescing moderation Heng Qi
2023-02-16 14:49 ` [virtio-dev] " Heng Qi
@ 2023-02-16 16:17 ` Michael S. Tsirkin
2023-02-17 5:24 ` Heng Qi
2023-02-17 2:47 ` Parav Pandit
` (2 subsequent siblings)
4 siblings, 1 reply; 29+ messages in thread
From: Michael S. Tsirkin @ 2023-02-16 16:17 UTC (permalink / raw)
To: Heng Qi
Cc: virtio-comment, virtio-dev, Parav Pandit, Yuri Benditovich,
Jason Wang, Cornelia Huck, Xuan Zhuo
On Thu, Feb 16, 2023 at 10:43:01PM +0800, Heng Qi wrote:
> Currently coalescing parameters are grouped for all transmit and receive
> virtqueues. This patch supports setting or getting the parameters for a
> specified virtqueue, and a typical application of this function is netdim[1].
>
> When the traffic between virtqueues is unbalanced, for example, one virtqueue
> is busy and another virtqueue is idle, then it will be very useful to
> control coalescing parameters at the virtqueue granularity.
>
> [1] https://docs.kernel.org/networking/net_dim.html
>
> Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
> v2->v3:
> 1. Add the netdim link. @Parav Pandit
> 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
> 3. _VQ_GET is explained more. @Michael S. Tsirkin
> 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
> 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
> 7. Fix some typos. @Michael S. Tsirkin
>
> v1->v2:
> 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
> 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
> 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
> 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
> 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
>
> content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--
> 1 file changed, 126 insertions(+), 4 deletions(-)
>
> diff --git a/content.tex b/content.tex
> index e863709..7f99503 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> channel.
>
> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> + notifications coalescing.
> +
> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
>
> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> \end{description}
>
> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> };
>
> #define VIRTIO_NET_CTRL_NOTF_COAL 6
> - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> +
> \end{lstlisting}
>
> Coalescing parameters:
> @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \end{itemize}
>
>
> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> \begin{enumerate}
> \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> + transmit/receive virtqueue whose number is \field{vqn}.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> + responds them to the driver.
> \end{enumerate}
>
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> + parameters of a enabled transmit/receive virtqueue.
> +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
> +
> +\begin{lstlisting}
> +struct virtio_net_ctrl_coal_vq {
> + le16 vqn;
> + le16 reserved;
> + le32 max_packets;
> + le32 max_usecs;
> +};
> +\end{lstlisting}
> +
> +Virtqueue coalescing parameters:
> +\begin{itemize}
> +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> +\end{itemize}
> +
> +\field{reserved} is reserved and it is ignored by the device.
> +
> +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> +The conversion between \field{vqn} and transmitq/receiveq index:
> +\begin{itemize}
> +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> +\end{itemize}
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> +
> +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> +virtqueue, regardless of the command used to deliver the parameters. For example:
> +# Command sequence 1:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> +# Command sequence 2:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
>
> If, for example:
> @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>
> \begin{itemize}
> \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of setting coalescing parameters for a receive virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 0
> +\end{itemize}
> +
> +The device will operate as follows:
> +\begin{itemize}
> +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a receive virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 0.
> \end{itemize}
>
> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> +and then responds them to the driver.
> +
> \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
>
> If, for example:
> @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> \end{itemize}
>
> +Example of setting coalescing parameters for a transmit virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 1
> +\end{itemize}
> +
> +The device will operate as follows:
> +
> +\begin{itemize}
> +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a transmit virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 1.
> +\end{itemize}
> +
> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> +and then responds them to the driver.
> +
> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>
> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
> +
> +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
>
> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>
> A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
>
> +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
> +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
What if vq is disabled? do parameters reset?
What about VIRTIO_NET_CTRL_NOTF_COAL?
what if it is envoked while some vq is disabled then vq is enabled?
do parameters for this vq stay 0?
> +
> +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> +
> A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
>
> Upon reset, a device MUST initialize all coalescing parameters to 0.
> --
> 2.19.1.6.gb485710b
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 16:17 ` Michael S. Tsirkin
@ 2023-02-17 5:24 ` Heng Qi
2023-02-17 10:37 ` Michael S. Tsirkin
2023-02-17 13:50 ` Parav Pandit
0 siblings, 2 replies; 29+ messages in thread
From: Heng Qi @ 2023-02-17 5:24 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, virtio-dev, Parav Pandit, Yuri Benditovich,
Jason Wang, Cornelia Huck, Xuan Zhuo
On Thu, Feb 16, 2023 at 11:17:48AM -0500, Michael S. Tsirkin wrote:
> On Thu, Feb 16, 2023 at 10:43:01PM +0800, Heng Qi wrote:
> > Currently coalescing parameters are grouped for all transmit and receive
> > virtqueues. This patch supports setting or getting the parameters for a
> > specified virtqueue, and a typical application of this function is netdim[1].
> >
> > When the traffic between virtqueues is unbalanced, for example, one virtqueue
> > is busy and another virtqueue is idle, then it will be very useful to
> > control coalescing parameters at the virtqueue granularity.
> >
> > [1] https://docs.kernel.org/networking/net_dim.html
> >
> > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> > Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > ---
> > v2->v3:
> > 1. Add the netdim link. @Parav Pandit
> > 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
> > 3. _VQ_GET is explained more. @Michael S. Tsirkin
> > 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
> > 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
> > 7. Fix some typos. @Michael S. Tsirkin
> >
> > v1->v2:
> > 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
> > 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
> > 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
> > 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
> > 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> >
> > content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--
> > 1 file changed, 126 insertions(+), 4 deletions(-)
> >
> > diff --git a/content.tex b/content.tex
> > index e863709..7f99503 100644
> > --- a/content.tex
> > +++ b/content.tex
> > @@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> > \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> > channel.
> >
> > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> > + notifications coalescing.
> > +
> > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
> >
> > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> > @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > \end{description}
> >
> > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> > @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > };
> >
> > #define VIRTIO_NET_CTRL_NOTF_COAL 6
> > - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> > +
> > \end{lstlisting}
> >
> > Coalescing parameters:
> > @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > \end{itemize}
> >
> >
> > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> > \begin{enumerate}
> > \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> > \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> > + transmit/receive virtqueue whose number is \field{vqn}.
> > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> > + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> > + responds them to the driver.
> > \end{enumerate}
> >
> > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> > +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> > + parameters of a enabled transmit/receive virtqueue.
> > +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> > + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
> > +
> > +\begin{lstlisting}
> > +struct virtio_net_ctrl_coal_vq {
> > + le16 vqn;
> > + le16 reserved;
> > + le32 max_packets;
> > + le32 max_usecs;
> > +};
> > +\end{lstlisting}
> > +
> > +Virtqueue coalescing parameters:
> > +\begin{itemize}
> > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > +\end{itemize}
> > +
> > +\field{reserved} is reserved and it is ignored by the device.
> > +
> > +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> > +The conversion between \field{vqn} and transmitq/receiveq index:
> > +\begin{itemize}
> > +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> > +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> > +\end{itemize}
> > +
> > +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> > +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> > +
> > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> > +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> > +
> > +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> > +virtqueue, regardless of the command used to deliver the parameters. For example:
> > +# Command sequence 1:
> > +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> > +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > +
> > +# Command sequence 2:
> > +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > +
> > \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
> >
> > If, for example:
> > @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >
> > \begin{itemize}
> > \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> > -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\end{itemize}
> > +
> > +Example of setting coalescing parameters for a receive virtqueue:
> > +\begin{itemize}
> > +\item \field{max_packets} = 15
> > +\item \field{max_usecs} = 10
> > +\item \field{vqn} = 0
> > +\end{itemize}
> > +
> > +The device will operate as follows:
> > +\begin{itemize}
> > +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> > +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\end{itemize}
> > +
> > +Example of getting coalescing parameters from a receive virtqueue:
> > +\begin{itemize}
> > +\item \field{vqn} = 0.
> > \end{itemize}
> >
> > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> > +and then responds them to the driver.
> > +
> > \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
> >
> > If, for example:
> > @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > \end{itemize}
> >
> > +Example of setting coalescing parameters for a transmit virtqueue:
> > +\begin{itemize}
> > +\item \field{max_packets} = 15
> > +\item \field{max_usecs} = 10
> > +\item \field{vqn} = 1
> > +\end{itemize}
> > +
> > +The device will operate as follows:
> > +
> > +\begin{itemize}
> > +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> > +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\end{itemize}
> > +
> > +Example of getting coalescing parameters from a transmit virtqueue:
> > +\begin{itemize}
> > +\item \field{vqn} = 1.
> > +\end{itemize}
> > +
> > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> > +and then responds them to the driver.
> > +
> > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> >
> > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> > +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > +
> > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> > +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
> > +
> > +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
> >
> > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> >
> > A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
> >
> > +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
> > +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
>
> What if vq is disabled? do parameters reset?
Yes, max_packets and max_usecs SHOULD be reset to 0.
When the virtqueue is re-enabled, it means that notification conditions are met after each packet is sent/received.
As described by Alvaro in "[PATCH v5] virtio-net: Fix and update VIRTIO_NET_F_NOTF_COAL feature":
"+When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent."
> What about VIRTIO_NET_CTRL_NOTF_COAL?
I think it should be handled in the same way as the above VQ_SET, that is, reset the corresponding virtqeuue parameters to 0.
> what if it is envoked while some vq is disabled then vq is enabled?
> do parameters for this vq stay 0?
I think there are two kinds of sequences:
#Seq1:
1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0)
2. VQ_SET command arrives with max_packets=10, max_usecs = 8
3. vq is enabled (vq params are max_packets = 0, max_usecs = 0)
After #3 is done, vq params should be max_packets = 0, max_usecs = 0, because the device cannot apply parameters when vq is disabled.
Other reason is that parameters determined based on the previous traffic information before vq is re-enabled, and it is not applicable now.
#Seq1:
1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0)
2. vq is enabled (vq params are max_packets = 0, max_usecs = 0)
3. VQ_SET command arrives with max_packets=10, max_usecs = 8
After 3 is done, vq params should be max_packets = 10, max_usecs = 8
Thanks.
>
> > +
> > +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> > +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> > +
> > A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
> >
> > Upon reset, a device MUST initialize all coalescing parameters to 0.
> > --
> > 2.19.1.6.gb485710b
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 5:24 ` Heng Qi
@ 2023-02-17 10:37 ` Michael S. Tsirkin
2023-02-17 11:17 ` [virtio-comment] Re: [virtio-dev] " Alvaro Karsz
2023-02-17 13:50 ` Parav Pandit
1 sibling, 1 reply; 29+ messages in thread
From: Michael S. Tsirkin @ 2023-02-17 10:37 UTC (permalink / raw)
To: Heng Qi
Cc: virtio-comment, virtio-dev, Parav Pandit, Yuri Benditovich,
Jason Wang, Cornelia Huck, Xuan Zhuo
On Fri, Feb 17, 2023 at 01:24:04PM +0800, Heng Qi wrote:
> On Thu, Feb 16, 2023 at 11:17:48AM -0500, Michael S. Tsirkin wrote:
> > On Thu, Feb 16, 2023 at 10:43:01PM +0800, Heng Qi wrote:
> > > Currently coalescing parameters are grouped for all transmit and receive
> > > virtqueues. This patch supports setting or getting the parameters for a
> > > specified virtqueue, and a typical application of this function is netdim[1].
> > >
> > > When the traffic between virtqueues is unbalanced, for example, one virtqueue
> > > is busy and another virtqueue is idle, then it will be very useful to
> > > control coalescing parameters at the virtqueue granularity.
> > >
> > > [1] https://docs.kernel.org/networking/net_dim.html
> > >
> > > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> > > Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > ---
> > > v2->v3:
> > > 1. Add the netdim link. @Parav Pandit
> > > 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
> > > 3. _VQ_GET is explained more. @Michael S. Tsirkin
> > > 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
> > > 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > > 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
> > > 7. Fix some typos. @Michael S. Tsirkin
> > >
> > > v1->v2:
> > > 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
> > > 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
> > > 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
> > > 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > > 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
> > > 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> > >
> > > content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--
> > > 1 file changed, 126 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/content.tex b/content.tex
> > > index e863709..7f99503 100644
> > > --- a/content.tex
> > > +++ b/content.tex
> > > @@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> > > \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> > > channel.
> > >
> > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> > > + notifications coalescing.
> > > +
> > > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
> > >
> > > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> > > @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> > > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> > > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> > > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > > \end{description}
> > >
> > > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> > > @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > };
> > >
> > > #define VIRTIO_NET_CTRL_NOTF_COAL 6
> > > - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > > + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> > > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> > > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> > > +
> > > \end{lstlisting}
> > >
> > > Coalescing parameters:
> > > @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > \end{itemize}
> > >
> > >
> > > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> > > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> > > \begin{enumerate}
> > > \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> > > \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> > > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> > > + transmit/receive virtqueue whose number is \field{vqn}.
> > > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> > > + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> > > + responds them to the driver.
> > > \end{enumerate}
> > >
> > > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> > > +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> > > + parameters of a enabled transmit/receive virtqueue.
> > > +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> > > + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
> > > +
> > > +\begin{lstlisting}
> > > +struct virtio_net_ctrl_coal_vq {
> > > + le16 vqn;
> > > + le16 reserved;
> > > + le32 max_packets;
> > > + le32 max_usecs;
> > > +};
> > > +\end{lstlisting}
> > > +
> > > +Virtqueue coalescing parameters:
> > > +\begin{itemize}
> > > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > > +\end{itemize}
> > > +
> > > +\field{reserved} is reserved and it is ignored by the device.
> > > +
> > > +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> > > +The conversion between \field{vqn} and transmitq/receiveq index:
> > > +\begin{itemize}
> > > +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> > > +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> > > +\end{itemize}
> > > +
> > > +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> > > +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> > > +
> > > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> > > +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> > > +
> > > +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> > > +virtqueue, regardless of the command used to deliver the parameters. For example:
> > > +# Command sequence 1:
> > > +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > > +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > > +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> > > +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > > +
> > > +# Command sequence 2:
> > > +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > > +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > > +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > > +
> > > \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
> > >
> > > If, for example:
> > > @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > >
> > > \begin{itemize}
> > > \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> > > -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > > +\end{itemize}
> > > +
> > > +Example of setting coalescing parameters for a receive virtqueue:
> > > +\begin{itemize}
> > > +\item \field{max_packets} = 15
> > > +\item \field{max_usecs} = 10
> > > +\item \field{vqn} = 0
> > > +\end{itemize}
> > > +
> > > +The device will operate as follows:
> > > +\begin{itemize}
> > > +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> > > +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> > > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > > +\end{itemize}
> > > +
> > > +Example of getting coalescing parameters from a receive virtqueue:
> > > +\begin{itemize}
> > > +\item \field{vqn} = 0.
> > > \end{itemize}
> > >
> > > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> > > +and then responds them to the driver.
> > > +
> > > \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
> > >
> > > If, for example:
> > > @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > > \end{itemize}
> > >
> > > +Example of setting coalescing parameters for a transmit virtqueue:
> > > +\begin{itemize}
> > > +\item \field{max_packets} = 15
> > > +\item \field{max_usecs} = 10
> > > +\item \field{vqn} = 1
> > > +\end{itemize}
> > > +
> > > +The device will operate as follows:
> > > +
> > > +\begin{itemize}
> > > +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> > > +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> > > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > > +\end{itemize}
> > > +
> > > +Example of getting coalescing parameters from a transmit virtqueue:
> > > +\begin{itemize}
> > > +\item \field{vqn} = 1.
> > > +\end{itemize}
> > > +
> > > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> > > +and then responds them to the driver.
> > > +
> > > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> > >
> > > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > > +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> > > +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > > +
> > > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> > > +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
> > > +
> > > +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
> > >
> > > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> > >
> > > A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
> > >
> > > +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
> > > +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
> >
> > What if vq is disabled? do parameters reset?
>
> Yes, max_packets and max_usecs SHOULD be reset to 0.
> When the virtqueue is re-enabled, it means that notification conditions are met after each packet is sent/received.
>
> As described by Alvaro in "[PATCH v5] virtio-net: Fix and update VIRTIO_NET_F_NOTF_COAL feature":
> "+When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent."
Oh. Hmm.
What Alvaro's patch does not describe is what happens when VQ is reset.
Alvaro you said you have hardware implementing this right?
How does the command interact with vq reset?
> > What about VIRTIO_NET_CTRL_NOTF_COAL?
>
> I think it should be handled in the same way as the above VQ_SET, that is, reset the corresponding virtqeuue parameters to 0.
sounds consistent. but the problem is, I don't think this is
how we previously behaved. and RING_RESET is out in 1.2.
So we need something compatible. I am sorry.
I suspect that instead we can say that existing hardware has a default set of
parameters for tx and rx. And global commands change that
besides changing all enabled VQs.
That is a side effect beyond just changing all VQs.
Hmm.
Alvaro?
> > what if it is envoked while some vq is disabled then vq is enabled?
> > do parameters for this vq stay 0?
>
> I think there are two kinds of sequences:
> #Seq1:
> 1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0)
> 2. VQ_SET command arrives with max_packets=10, max_usecs = 8
will not this VQ_SET fail?
> 3. vq is enabled (vq params are max_packets = 0, max_usecs = 0)
> After #3 is done, vq params should be max_packets = 0, max_usecs = 0, because the device cannot apply parameters when vq is disabled.
> Other reason is that parameters determined based on the previous traffic information before vq is re-enabled, and it is not applicable now.
>
> #Seq1:
> 1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0)
> 2. vq is enabled (vq params are max_packets = 0, max_usecs = 0)
> 3. VQ_SET command arrives with max_packets=10, max_usecs = 8
> After 3 is done, vq params should be max_packets = 10, max_usecs = 8
>
> Thanks.
> >
> > > +
> > > +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> > > +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> > > +
> > > A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
> > >
> > > Upon reset, a device MUST initialize all coalescing parameters to 0.
> > > --
> > > 2.19.1.6.gb485710b
^ permalink raw reply [flat|nested] 29+ messages in thread* [virtio-comment] Re: [virtio-dev] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 10:37 ` Michael S. Tsirkin
@ 2023-02-17 11:17 ` Alvaro Karsz
2023-02-17 11:35 ` Michael S. Tsirkin
0 siblings, 1 reply; 29+ messages in thread
From: Alvaro Karsz @ 2023-02-17 11:17 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Heng Qi, virtio-comment, virtio-dev, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
> > Yes, max_packets and max_usecs SHOULD be reset to 0.
> > When the virtqueue is re-enabled, it means that notification conditions are met after each packet is sent/received.
> >
> > As described by Alvaro in "[PATCH v5] virtio-net: Fix and update VIRTIO_NET_F_NOTF_COAL feature":
> > "+When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent."
>
> Oh. Hmm.
>
> What Alvaro's patch does not describe is what happens when VQ is reset.
>
> Alvaro you said you have hardware implementing this right?
> How does the command interact with vq reset?
>
The device doesn't offer VIRTIO_F_RING_RESET at the moment.
>
> > > What about VIRTIO_NET_CTRL_NOTF_COAL?
> >
> > I think it should be handled in the same way as the above VQ_SET, that is, reset the corresponding virtqeuue parameters to 0.
>
> sounds consistent. but the problem is, I don't think this is
> how we previously behaved. and RING_RESET is out in 1.2.
> So we need something compatible. I am sorry.
>
> I suspect that instead we can say that existing hardware has a default set of
> parameters for tx and rx. And global commands change that
> besides changing all enabled VQs.
>
> That is a side effect beyond just changing all VQs.
>
> Hmm.
> Alvaro?
>
This is indeed a good point.
We mention the device reset case, but nothing about VQ reset.
I feel that no matter how we handle this, we break something.
Having default coalescing values may collide with "Upon reset, a
device MUST initialize all coalescing parameters to 0."
We can say that VQ reset doesn't affect the "global parameters" and a
device reset does, but this collides with "Device Requirements:
Virtqueue Reset".
In fact, resetting coalescing values after vq reset may be derived
from "Upon reset, a device MUST initialize all coalescing parameters
to 0".
This is consistent with "Device Requirements: Virtqueue Reset".
I can add in my patch some clarifications.
This will break the linux virtio_net ethtool implementation a little,
we'll need to fix it.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-dev] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 11:17 ` [virtio-comment] Re: [virtio-dev] " Alvaro Karsz
@ 2023-02-17 11:35 ` Michael S. Tsirkin
2023-02-17 12:17 ` Alvaro Karsz
2023-02-17 16:12 ` Parav Pandit
0 siblings, 2 replies; 29+ messages in thread
From: Michael S. Tsirkin @ 2023-02-17 11:35 UTC (permalink / raw)
To: Alvaro Karsz
Cc: Heng Qi, virtio-comment, virtio-dev, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
On Fri, Feb 17, 2023 at 01:17:50PM +0200, Alvaro Karsz wrote:
> > > Yes, max_packets and max_usecs SHOULD be reset to 0.
> > > When the virtqueue is re-enabled, it means that notification conditions are met after each packet is sent/received.
> > >
> > > As described by Alvaro in "[PATCH v5] virtio-net: Fix and update VIRTIO_NET_F_NOTF_COAL feature":
> > > "+When the device has \field{max_usecs} = 0 or \field{max_packets} = 0, the notification conditions are met after every packet received/sent."
> >
> > Oh. Hmm.
> >
> > What Alvaro's patch does not describe is what happens when VQ is reset.
> >
> > Alvaro you said you have hardware implementing this right?
> > How does the command interact with vq reset?
> >
>
> The device doesn't offer VIRTIO_F_RING_RESET at the moment.
>
> >
> > > > What about VIRTIO_NET_CTRL_NOTF_COAL?
> > >
> > > I think it should be handled in the same way as the above VQ_SET, that is, reset the corresponding virtqeuue parameters to 0.
> >
> > sounds consistent. but the problem is, I don't think this is
> > how we previously behaved. and RING_RESET is out in 1.2.
> > So we need something compatible. I am sorry.
> >
> > I suspect that instead we can say that existing hardware has a default set of
> > parameters for tx and rx. And global commands change that
> > besides changing all enabled VQs.
> >
> > That is a side effect beyond just changing all VQs.
> >
> > Hmm.
> > Alvaro?
> >
>
> This is indeed a good point.
> We mention the device reset case, but nothing about VQ reset.
>
> I feel that no matter how we handle this, we break something.
>
> Having default coalescing values may collide with "Upon reset, a
> device MUST initialize all coalescing parameters to 0."
No this is after device reset.
> We can say that VQ reset doesn't affect the "global parameters" and a
> device reset does, but this collides with "Device Requirements:
> Virtqueue Reset".
>
> In fact, resetting coalescing values after vq reset may be derived
> from "Upon reset, a device MUST initialize all coalescing parameters
> to 0".
> This is consistent with "Device Requirements: Virtqueue Reset".
>
> I can add in my patch some clarifications.
>
> This will break the linux virtio_net ethtool implementation a little,
> we'll need to fix it.
Not good. I feel we must come up with spec that is backwards compatible.
Hmm, maybe this is why Parav kept talking about modes.
I did not realize at the time, sorry Parav.
I still feel modes are not the best way to describe things so I propose this:
- in addition to per vq parameters, device that supports global TX/RX
commands and ring reset maintains two sets of default parameters: for RX and TX
- existing commands change default and change all enabled vqs
of the correct type (RX/TX) to the same value
- vq reset changes a vq to the default
- device reset changes defaults to 0 and changes all vqs to 0
note how defaults are only used for ring reset. is "vq reset parameter"
a better name? I feel we will repeat "reset" too many times in a
sentence if we call it that though.
So fundamentally the only change is with RING_RESET, then
default is not always 0, it can be set by the global command.
--
MST
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-dev] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 11:35 ` Michael S. Tsirkin
@ 2023-02-17 12:17 ` Alvaro Karsz
2023-02-17 16:11 ` [virtio-comment] " Parav Pandit
2023-02-17 16:12 ` Parav Pandit
1 sibling, 1 reply; 29+ messages in thread
From: Alvaro Karsz @ 2023-02-17 12:17 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Heng Qi, virtio-comment, virtio-dev, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
> Not good. I feel we must come up with spec that is backwards compatible.
> Hmm, maybe this is why Parav kept talking about modes.
> I did not realize at the time, sorry Parav.
>
> I still feel modes are not the best way to describe things so I propose this:
> - in addition to per vq parameters, device that supports global TX/RX
> commands and ring reset maintains two sets of default parameters: for RX and TX
> - existing commands change default and change all enabled vqs
> of the correct type (RX/TX) to the same value
> - vq reset changes a vq to the default
> - device reset changes defaults to 0 and changes all vqs to 0
>
> note how defaults are only used for ring reset. is "vq reset parameter"
> a better name? I feel we will repeat "reset" too many times in a
> sentence if we call it that though.
>
> So fundamentally the only change is with RING_RESET, then
> default is not always 0, it can be set by the global command.
I like the idea, maintaining default coalescing parameters is
compatible with vq reset.
"After a queue has been reset by the driver, the device MUST NOT
execute any requests from that virtqueue, or notify the driver for it.
The device MUST reset any state of a virtqueue to the *default state*,
including the available state and the used state."
I wonder if some of that should be included in my patch.
Most of that is not relevant before introducing the per vq command.
So maybe I could just mention in the device conformance that:
"Coalescing parameters MUST/SHOULD persist a virtqueue reset."
And then Heng can edit/add on top of that.
Now that the existing commands do more than just iterating through all
the virtqueues, maybe we should be able to modify all virtqueues with
VIRTIO_NET_F_VQ_NOTF_COAL.
I think that using a special vqn is messy, we can split "le32
reserved" to "le16 flags" and "le16 reserved".
Then we can have:
- A flag to apply parameters to all receive VQs, vqn is ignored in
this case. (same as VIRTIO_NET_CTRL_NOTF_COAL_RX_SET but without
changing the default parameters).
- A flag to apply parameters to all transmit VQs, vqn is ignored in
this case. (same as VIRTIO_NET_CTRL_NOTF_COAL_TX_SET but without
changing the default parameters).
And maybe a flag to apply to all vqs (transmit + receive), I actually
am not sure how useful this is..
^ permalink raw reply [flat|nested] 29+ messages in thread
* [virtio-comment] RE: [virtio-dev] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 12:17 ` Alvaro Karsz
@ 2023-02-17 16:11 ` Parav Pandit
0 siblings, 0 replies; 29+ messages in thread
From: Parav Pandit @ 2023-02-17 16:11 UTC (permalink / raw)
To: Alvaro Karsz, Michael S. Tsirkin
Cc: Heng Qi, virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, Yuri Benditovich, Jason Wang,
Cornelia Huck, Xuan Zhuo
> From: Alvaro Karsz <alvaro.karsz@solid-run.com>
> Sent: Friday, February 17, 2023 7:18 AM
> I wonder if some of that should be included in my patch.
> Most of that is not relevant before introducing the per vq command.
> So maybe I could just mention in the device conformance that:
> "Coalescing parameters MUST/SHOULD persist a virtqueue reset."
>
For sure coalescing parameters must not persist on a vq reset in the device.
VQ reset operation destroys or disables the VQ.
Device should not have any obligation to maintain some parameter on the disabled/destroyed object.
User likely disabled the VQ because user may want to have a fresh start for this VQ for same or different use.
And hence, reconfiguring the new addresses, new vq size, new vq notifications parameters..
^ permalink raw reply [flat|nested] 29+ messages in thread
* [virtio-comment] RE: [virtio-dev] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 11:35 ` Michael S. Tsirkin
2023-02-17 12:17 ` Alvaro Karsz
@ 2023-02-17 16:12 ` Parav Pandit
2023-02-17 17:17 ` Heng Qi
1 sibling, 1 reply; 29+ messages in thread
From: Parav Pandit @ 2023-02-17 16:12 UTC (permalink / raw)
To: Michael S. Tsirkin, Alvaro Karsz
Cc: Heng Qi, virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, Yuri Benditovich, Jason Wang,
Cornelia Huck, Xuan Zhuo
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Friday, February 17, 2023 6:35 AM
> > We mention the device reset case, but nothing about VQ reset.
> >
> > I feel that no matter how we handle this, we break something.
> >
> > Having default coalescing values may collide with "Upon reset, a
> > device MUST initialize all coalescing parameters to 0."
>
> No this is after device reset.
>
> > We can say that VQ reset doesn't affect the "global parameters" and a
> > device reset does, but this collides with "Device Requirements:
> > Virtqueue Reset".
> >
Not really.
When the device resets, VQ objects are destroyed in the device.
So VQ's notifications parameters doesn't exists on device reset.
And so the same case with VQ reset.
When a VQ is reset (disabled), VQ's notifications configuration is removed in the device too.
Just like its desc ring and other addresses are invalid.
> > In fact, resetting coalescing values after vq reset may be derived
> > from "Upon reset, a device MUST initialize all coalescing parameters
> > to 0".
> > This is consistent with "Device Requirements: Virtqueue Reset".
> >
> > I can add in my patch some clarifications.
> >
> > This will break the linux virtio_net ethtool implementation a little,
> > we'll need to fix it.
>
> Not good. I feel we must come up with spec that is backwards compatible.
> Hmm, maybe this is why Parav kept talking about modes.
> I did not realize at the time, sorry Parav.
>
> I still feel modes are not the best way to describe things so I propose this:
> - in addition to per vq parameters, device that supports global TX/RX
> commands and ring reset maintains two sets of default parameters: for RX and
> TX
> - existing commands change default and change all enabled vqs
> of the correct type (RX/TX) to the same value
> - vq reset changes a vq to the default
> - device reset changes defaults to 0 and changes all vqs to 0
>
> note how defaults are only used for ring reset. is "vq reset parameter"
> a better name? I feel we will repeat "reset" too many times in a sentence if we
> call it that though.
>
> So fundamentally the only change is with RING_RESET, then default is not
> always 0, it can be set by the global command.
True default is not zero when global are configured.
It is ok to report VQ parameters on GET command when globals are configured.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [virtio-dev] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 16:12 ` Parav Pandit
@ 2023-02-17 17:17 ` Heng Qi
2023-02-17 17:22 ` [virtio-comment] " Parav Pandit
0 siblings, 1 reply; 29+ messages in thread
From: Heng Qi @ 2023-02-17 17:17 UTC (permalink / raw)
To: Parav Pandit
Cc: Michael S. Tsirkin, Alvaro Karsz,
virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, Yuri Benditovich, Jason Wang,
Cornelia Huck, Xuan Zhuo
On Fri, Feb 17, 2023 at 04:12:34PM +0000, Parav Pandit wrote:
>
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Friday, February 17, 2023 6:35 AM
>
> > > We mention the device reset case, but nothing about VQ reset.
> > >
> > > I feel that no matter how we handle this, we break something.
> > >
> > > Having default coalescing values may collide with "Upon reset, a
> > > device MUST initialize all coalescing parameters to 0."
> >
> > No this is after device reset.
> >
> > > We can say that VQ reset doesn't affect the "global parameters" and a
> > > device reset does, but this collides with "Device Requirements:
> > > Virtqueue Reset".
> > >
> Not really.
> When the device resets, VQ objects are destroyed in the device.
> So VQ's notifications parameters doesn't exists on device reset.
>
> And so the same case with VQ reset.
> When a VQ is reset (disabled), VQ's notifications configuration is removed in the device too.
> Just like its desc ring and other addresses are invalid.
Yes, but there seems to be such a situation: when the device is reactivated, as the specification says,
all parameters are set to 0 (use parameters as the default configuration on the device).
When CTRL_COAL_SET and CTRL_COAL_VQ_SET are sent, the configuration is updated (the parameters of each vq may be different,
but the global parameter configuration may be recorded), at this time, if vq is reset,
should the parameters be 0 or a recorded global parameters after it is re-enabled?
>
> > > In fact, resetting coalescing values after vq reset may be derived
> > > from "Upon reset, a device MUST initialize all coalescing parameters
> > > to 0".
> > > This is consistent with "Device Requirements: Virtqueue Reset".
> > >
> > > I can add in my patch some clarifications.
> > >
> > > This will break the linux virtio_net ethtool implementation a little,
> > > we'll need to fix it.
> >
> > Not good. I feel we must come up with spec that is backwards compatible.
> > Hmm, maybe this is why Parav kept talking about modes.
> > I did not realize at the time, sorry Parav.
> >
> > I still feel modes are not the best way to describe things so I propose this:
> > - in addition to per vq parameters, device that supports global TX/RX
> > commands and ring reset maintains two sets of default parameters: for RX and
> > TX
> > - existing commands change default and change all enabled vqs
> > of the correct type (RX/TX) to the same value
> > - vq reset changes a vq to the default
> > - device reset changes defaults to 0 and changes all vqs to 0
> >
> > note how defaults are only used for ring reset. is "vq reset parameter"
> > a better name? I feel we will repeat "reset" too many times in a sentence if we
> > call it that though.
> >
> > So fundamentally the only change is with RING_RESET, then default is not
> > always 0, it can be set by the global command.
>
> True default is not zero when global are configured.
> It is ok to report VQ parameters on GET command when globals are configured.
^ permalink raw reply [flat|nested] 29+ messages in thread
* [virtio-comment] RE: [virtio-dev] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 17:17 ` Heng Qi
@ 2023-02-17 17:22 ` Parav Pandit
0 siblings, 0 replies; 29+ messages in thread
From: Parav Pandit @ 2023-02-17 17:22 UTC (permalink / raw)
To: Heng Qi
Cc: Michael S. Tsirkin, Alvaro Karsz,
virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, Yuri Benditovich, Jason Wang,
Cornelia Huck, Xuan Zhuo
> From: Heng Qi <hengqi@linux.alibaba.com>
> Sent: Friday, February 17, 2023 12:17 PM
> Yes, but there seems to be such a situation: when the device is reactivated, as
> the specification says, all parameters are set to 0 (use parameters as the default
> configuration on the device).
> When CTRL_COAL_SET and CTRL_COAL_VQ_SET are sent, the configuration is
> updated (the parameters of each vq may be different, but the global parameter
> configuration may be recorded), at this time, if vq is reset, should the
> parameters be 0 or a recorded global parameters after it is re-enabled?
Recorded global parameters should be fine to report on an enabled vq even if the COAL_VQ_SET was not done.
This is because the global value reflects on the VQ.
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 29+ messages in thread
* RE: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 5:24 ` Heng Qi
2023-02-17 10:37 ` Michael S. Tsirkin
@ 2023-02-17 13:50 ` Parav Pandit
2023-02-17 16:43 ` Heng Qi
1 sibling, 1 reply; 29+ messages in thread
From: Parav Pandit @ 2023-02-17 13:50 UTC (permalink / raw)
To: Heng Qi, Michael S. Tsirkin
Cc: virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, Yuri Benditovich, Jason Wang,
Cornelia Huck, Xuan Zhuo
> From: Heng Qi <hengqi@linux.alibaba.com>
> Sent: Friday, February 17, 2023 12:24 AM
> I think there are two kinds of sequences:
> #Seq1:
> 1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0) 2.
> VQ_SET command arrives with max_packets=10, max_usecs = 8 3. vq is enabled
> (vq params are max_packets = 0, max_usecs = 0) After #3 is done, vq params
> should be max_packets = 0, max_usecs = 0, because the device cannot apply
> parameters when vq is disabled.
> Other reason is that parameters determined based on the previous traffic
> information before vq is re-enabled, and it is not applicable now.
From dev point of view, above is not a valid sequence.
VQ is the object whose lifecycle (create/delete/query/modify) is triggered by the driver.
So step 2 cannot happen before step 3.
If driver does that, driver should get error code INVALID_VQ on step 2.
>
> #Seq1:
> 1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0) 2. vq
> is enabled (vq params are max_packets = 0, max_usecs = 0) 3. VQ_SET
> command arrives with max_packets=10, max_usecs = 8 After 3 is done, vq
> params should be max_packets = 10, max_usecs = 8
>
> Thanks.
>
> >
> > > +
> > > +A device MAY NOT set the coalescing parameter to the exact same
> > > +value as the one passed in by a driver. For example, the value of
> \field{max_packets} set by the driver is 15, but the device MAY store a value
> that is a power of 2, that is, 16.
> > > +
> > > A device SHOULD NOT send used buffer notifications to the driver, if the
> notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio
> Device / Virtqueues / Used Buffer Notification Suppression}, even if the
> coalescing counters expired.
> > >
> > > Upon reset, a device MUST initialize all coalescing parameters to 0.
> > > --
> > > 2.19.1.6.gb485710b
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 13:50 ` Parav Pandit
@ 2023-02-17 16:43 ` Heng Qi
0 siblings, 0 replies; 29+ messages in thread
From: Heng Qi @ 2023-02-17 16:43 UTC (permalink / raw)
To: Parav Pandit, Michael S. Tsirkin
Cc: virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, Yuri Benditovich, Jason Wang,
Cornelia Huck, Xuan Zhuo
在 2023/2/17 下午9:50, Parav Pandit 写道:
>
>> From: Heng Qi <hengqi@linux.alibaba.com>
>> Sent: Friday, February 17, 2023 12:24 AM
>> I think there are two kinds of sequences:
>> #Seq1:
>> 1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0) 2.
>> VQ_SET command arrives with max_packets=10, max_usecs = 8 3. vq is enabled
>> (vq params are max_packets = 0, max_usecs = 0) After #3 is done, vq params
>> should be max_packets = 0, max_usecs = 0, because the device cannot apply
>> parameters when vq is disabled.
>> Other reason is that parameters determined based on the previous traffic
>> information before vq is re-enabled, and it is not applicable now.
> From dev point of view, above is not a valid sequence.
> VQ is the object whose lifecycle (create/delete/query/modify) is triggered by the driver.
> So step 2 cannot happen before step 3.
> If driver does that, driver should get error code INVALID_VQ on step 2.
>
Yes, This example is not appropriate. The driver must not make any
requests on vq while it is being reset,
unless the reset of vq has been completed.
>> #Seq1:
>> 1. vq is disabled (vq params are reset to max_packets = 0, max_usecs = 0) 2. vq
>> is enabled (vq params are max_packets = 0, max_usecs = 0) 3. VQ_SET
>> command arrives with max_packets=10, max_usecs = 8 After 3 is done, vq
>> params should be max_packets = 10, max_usecs = 8
>>
>> Thanks.
>>
>>>> +
>>>> +A device MAY NOT set the coalescing parameter to the exact same
>>>> +value as the one passed in by a driver. For example, the value of
>> \field{max_packets} set by the driver is 15, but the device MAY store a value
>> that is a power of 2, that is, 16.
>>>> +
>>>> A device SHOULD NOT send used buffer notifications to the driver, if the
>> notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio
>> Device / Virtqueues / Used Buffer Notification Suppression}, even if the
>> coalescing counters expired.
>>>> Upon reset, a device MUST initialize all coalescing parameters to 0.
>>>> --
>>>> 2.19.1.6.gb485710b
^ permalink raw reply [flat|nested] 29+ messages in thread
* RE: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 14:43 [PATCH v3] virtio-net: support the virtqueue coalescing moderation Heng Qi
2023-02-16 14:49 ` [virtio-dev] " Heng Qi
2023-02-16 16:17 ` Michael S. Tsirkin
@ 2023-02-17 2:47 ` Parav Pandit
2023-02-17 4:56 ` [virtio-comment] Re: [virtio-dev] " Heng Qi
2023-02-17 8:42 ` [virtio-comment] " Alvaro Karsz
2023-02-17 10:04 ` Michael S. Tsirkin
4 siblings, 1 reply; 29+ messages in thread
From: Parav Pandit @ 2023-02-17 2:47 UTC (permalink / raw)
To: Heng Qi, virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org
Cc: Michael S . Tsirkin, Yuri Benditovich, Jason Wang, Cornelia Huck,
Xuan Zhuo
> From: Heng Qi <hengqi@linux.alibaba.com>
> Sent: Thursday, February 16, 2023 9:43 AM
[..]
> #define VIRTIO_NET_CTRL_NOTF_COAL 6
> - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
Unrelated change. Please remove this hunk.
> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 #define
> + VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> +
> \end{lstlisting}
>
> Coalescing parameters:
> @@ -4514,12 +4521,67 @@ \subsubsection{Control
> Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize}
>
>
> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> \begin{enumerate}
> \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and
> \field{tx_max_packets} parameters.
> \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and
> \field{rx_max_packets} parameters.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets}
> and \field{max_usecs} parameters for a enabled
> + transmit/receive virtqueue whose number is \field{vqn}.
In the spec we have mix of notion as vqn or vq index.
For example, recent aq patch from Michael refers to aq vq index.
Mmio has vq index register too for long time.
And some places we don't refer to index vs number. (pcie queue_select register).
We all understand that they refer to same.
But it is time to get consistency for new additions.
So lets agree on either one of the naming convention.
Michael,
I am inclined to name it as vq index given existings references and with your aq vq index patch.
Do you have preference?
If its index, above should be changed from vqn/vq_index.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the
> \field{max_packets} and \field{max_usecs} parameters of
> + a enabled transmit/receive virtqueue whose number is
> \field{vqn}, and then
> + responds them to the driver.
> \end{enumerate}
It is confusing line of "device gets".
It is better to write it as,
The device returns max_packets and max_usecs for the enabled transmit or receive virtuque whose index is vq_index.
>
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to
> set the coalescing
> + parameters of a enabled transmit/receive virtqueue.
> +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a
> device, and the device
> + responds to the driver with the coalescing parameters of a enabled
> transmit/receive virtqueue.
> +
> +\begin{lstlisting}
> +struct virtio_net_ctrl_coal_vq {
> + le16 vqn;
> + le16 reserved;
> + le32 max_packets;
> + le32 max_usecs;
> +};
> +\end{lstlisting}
> +
> +Virtqueue coalescing parameters:
> +\begin{itemize}
> +\item \field{vqn}: The virtqueue number of the enabled transmit or receive
> virtqueue, excluding the control virtqueue.
Transmit and receive vq is self-explanatory. No need to mention control vq.
The word "excluding" doesn't fit here well given it refers to only single vq at a time.
> +\item \field{max_packets}: The maximum number of packets sent/received by
> the specified virtqueue before a TX/RX notification.
> +\item \field{max_usecs}: The maximum number of TX/RX usecs that the
> specified virtqueue delays a TX/RX notification.
> +\end{itemize}
> +
> +\field{reserved} is reserved and it is ignored by the device.
> +
> +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> +The conversion between \field{vqn} and transmitq/receiveq index:
> +\begin{itemize}
> +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> +\end{itemize}
> +
No need to provide the math here, it is about enabled vq.
The spec somewhere else already describes what the valid index of enabled vq.
> +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the
> +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each
> virtqueue of receiveq1\ldots receiveqN.
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the
> +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each
> virtqueue of transmitq1\ldots transmitqN.
> +
> +If coalescing parameters are being set, the device applies the last
> +coalescing parameters received for a virtqueue, regardless of the command
> used to deliver the parameters. For example:
used to set the parameters.
> +# Command sequence 1:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $
> \field{max_packets}
> += 15 $ and $ \field{max_usecs} = 10 $ 2.
> +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $
> +\field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ After #2 command is
> applied by the device, the coalescing parameters of receiveq1 are $
> \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, and the coalescing
> parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and
> $ \field{max_usecs} = 10 $.
> +
> +# Command sequence 2:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $,
> $
> +\field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ 2.
> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets}
> = 15 $ and $ \field{max_usecs} = 10 $ After #2 command is applied by the
> device, the coalescing parameters of receiveq1\ldots receiveqN are $
> \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> \subparagraph{RX Notifications}\label{sec:Device Types / Network Device /
> Device Operation / Control Virtqueue / Notifications Coalescing / RX
> Notifications}
>
> If, for example:
> @@ -4532,9 +4594,33 @@ \subsubsection{Control
> Virtqueue}\label{sec:Device Types / Network Device / Devi
>
> \begin{itemize}
> \item The device will count received packets until it accumulates 15, or until 10
> usecs elapsed since the first one was received.
> -\item If the notifications are not suppressed by the driver, the device will send
> an used buffer notification, otherwise, the device will not send an used buffer
> notification as long as the notifications are suppressed.
> +\item If the notifications are not suppressed by the driver, the device will send
> an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the
> notifications are suppressed.
> +\end{itemize}
> +
> +Example of setting coalescing parameters for a receive virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 0
> +\end{itemize}
> +
> +The device will operate as follows:
> +\begin{itemize}
> +\item The device applies the coalescing parameters to receiveq1 because its
> virtqueue number is 0.
> +\item The device will count received packets on receiveq1 until it accumulates
> 15, or until 10 usecs elapsed since the first one was received.
> +\item If the notifications are not suppressed by the driver, the device will send
> an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the
> notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a receive virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 0.
> \end{itemize}
>
> +The device gets the values of the \field{max_packets} and
> +\field{max_usecs} parameters from the receiveq1 whose virtqueue number is
> 0, and then responds them to the driver.
> +
The device returns the value of X and Y parameters for the supplied vq index.
> \subparagraph{TX Notifications}\label{sec:Device Types / Network Device /
> Device Operation / Control Virtqueue / Notifications Coalescing / TX
> Notifications}
>
> If, for example:
> @@ -4550,14 +4636,50 @@ \subsubsection{Control
> Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the
> notifications are not suppressed by the driver, the device will send an used
> buffer notification, otherwise, the device will not send an used buffer
> notification as long as the notifications are suppressed.
> \end{itemize}
>
> +Example of setting coalescing parameters for a transmit virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 1
> +\end{itemize}
> +
> +The device will operate as follows:
> +
> +\begin{itemize}
> +\item The device applies the coalescing parameters to transmitq1 because its
> virtqueue number is 1.
> +\item The device will count sent packets on transmitq1 until it accumulates 15,
> or until 10 usecs elapsed since the first one was sent.
> +\item If the notifications are not suppressed by the driver, the device will send
> an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the
> notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a transmit virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 1.
> +\end{itemize}
> +
> +The device gets the values of the \field{max_packets} and
> +\field{max_usecs} parameters from the transmitq1 whose virtqueue number
> is 1, and then responds them to the driver.
> +
> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types /
> Network Device / Device Operation / Control Virtqueue / Notifications
> Coalescing}
>
> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver
> MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +If neither the VIRTIO_NET_F_NOTF_COAL nor the
> VIRTIO_NET_F_VQ_NOTF_COAL
> +feature has not been negotiated, the driver MUST NOT issue
> VIRTIO_NET_CTRL_NOTF_COAL commands.
> +
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST
> +set \field{vqn} to a legal value, that is, \field{vqn} points to an enabled
> transmit/receive virtqueue.
> +
> +A driver MAY NOT get exactly the same value as the coalescing parameter it
> was set to, for example it MAY get a value that is a power of 2.
>
It MAY get a value that is close to power of 2 value of the previously set value.
> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types /
> Network Device / Device Operation / Control Virtqueue / Notifications
> Coalescing}
>
> A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands
> with VIRTIO_NET_ERR if it was not able to change the parameters.
>
Silently dropping the command is not good, we cannot change past now.
But lets please fix the below one.
Please add the note around above text that,
Note: VQ_COAL is exception to it, which must return NET_ERR if device is unable to change the parameters.
> +When a device receives a virtqueue command to set/get coalescing
> +parameters for a virtqueue with number \field{vqn}, if the virtqueue is not
> enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
> +
Device MUST respond with an error on the object that doesn't exist.
Please change it to MUST.
Please add below text. Need some wording change for above SHOULD.
A device MUST respond to the VIRTIO_NET_CTRL_VQ_NOTF_COAL_GET and SET command with
with VIRTIO_NET_ERR if it is not able to change the parameters.
> +A device MAY NOT set the coalescing parameter to the exact same value
> +as the one passed in by a driver.
s/as the one passed in by a driver/as the one set by the driver
For example, the value of \field{max_packets}
> set by the driver is 15, but the device MAY store a value that is a power of 2,
> that is, 16.
> +
> A device SHOULD NOT send used buffer notifications to the driver, if the
> notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio
> Device / Virtqueues / Used Buffer Notification Suppression}, even if the
> coalescing counters expired.
>
> Upon reset, a device MUST initialize all coalescing parameters to 0.
> --
> 2.19.1.6.gb485710b
^ permalink raw reply [flat|nested] 29+ messages in thread* [virtio-comment] Re: [virtio-dev] RE: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 2:47 ` Parav Pandit
@ 2023-02-17 4:56 ` Heng Qi
0 siblings, 0 replies; 29+ messages in thread
From: Heng Qi @ 2023-02-17 4:56 UTC (permalink / raw)
To: Parav Pandit, virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org
Cc: Michael S . Tsirkin, Yuri Benditovich, Jason Wang, Cornelia Huck,
Xuan Zhuo
Hi, Parav. Thanks for your suggested revisions.
在 2023/2/17 上午10:47, Parav Pandit 写道:
>> From: Heng Qi <hengqi@linux.alibaba.com>
>> Sent: Thursday, February 16, 2023 9:43 AM
> [..]
>> #define VIRTIO_NET_CTRL_NOTF_COAL 6
>> - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
>> + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> Unrelated change. Please remove this hunk.
Ok. Will fix.
>
>> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
>> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2 #define
>> + VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
>> +
>> \end{lstlisting}
>>
>> Coalescing parameters:
>> @@ -4514,12 +4521,67 @@ \subsubsection{Control
>> Virtqueue}\label{sec:Device Types / Network Device / Devi \end{itemize}
>>
>>
>> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
>> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
>> \begin{enumerate}
>> \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and
>> \field{tx_max_packets} parameters.
>> \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and
>> \field{rx_max_packets} parameters.
>> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets}
>> and \field{max_usecs} parameters for a enabled
>> + transmit/receive virtqueue whose number is \field{vqn}.
> In the spec we have mix of notion as vqn or vq index.
> For example, recent aq patch from Michael refers to aq vq index.
> Mmio has vq index register too for long time.
>
> And some places we don't refer to index vs number. (pcie queue_select register).
> We all understand that they refer to same.
> But it is time to get consistency for new additions.
I totally agree.
I see that "\item [vqn] VQ number to be notified." is currently present
in \section{Driver Notifications},
and also
"When VIRTIO_F_NOTIFICATION_DATA has not been negotiated,
The driver sends an available buffer notification to the device by
writing the 16-bit virtqueue index”
These all seem to exist at the same time, and I think we should indeed
unify.
> So lets agree on either one of the naming convention.
>
> Michael,
> I am inclined to name it as vq index given existings references and with your aq vq index patch.
> Do you have preference?
> If its index, above should be changed from vqn/vq_index.
>
>> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the
>> \field{max_packets} and \field{max_usecs} parameters of
>> + a enabled transmit/receive virtqueue whose number is
>> \field{vqn}, and then
>> + responds them to the driver.
>> \end{enumerate}
> It is confusing line of "device gets".
>
> It is better to write it as,
> The device returns max_packets and max_usecs for the enabled transmit or receive virtuque whose index is vq_index.
Ok. Will modify.
>
>> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
>> +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to
>> set the coalescing
>> + parameters of a enabled transmit/receive virtqueue.
>> +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a
>> device, and the device
>> + responds to the driver with the coalescing parameters of a enabled
>> transmit/receive virtqueue.
>> +
>> +\begin{lstlisting}
>> +struct virtio_net_ctrl_coal_vq {
>> + le16 vqn;
>> + le16 reserved;
>> + le32 max_packets;
>> + le32 max_usecs;
>> +};
>> +\end{lstlisting}
>> +
>> +Virtqueue coalescing parameters:
>> +\begin{itemize}
>> +\item \field{vqn}: The virtqueue number of the enabled transmit or receive
>> virtqueue, excluding the control virtqueue.
> Transmit and receive vq is self-explanatory. No need to mention control vq.
> The word "excluding" doesn't fit here well given it refers to only single vq at a time.
Ok. Will remove.
>> +\item \field{max_packets}: The maximum number of packets sent/received by
>> the specified virtqueue before a TX/RX notification.
>> +\item \field{max_usecs}: The maximum number of TX/RX usecs that the
>> specified virtqueue delays a TX/RX notification.
>> +\end{itemize}
>> +
>> +\field{reserved} is reserved and it is ignored by the device.
>> +
>> +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
>> +The conversion between \field{vqn} and transmitq/receiveq index:
>> +\begin{itemize}
>> +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
>> +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
>> +\end{itemize}
>> +
> No need to provide the math here, it is about enabled vq.
> The spec somewhere else already describes what the valid index of enabled vq.
I saw this:
"
\begin{description}
\item[0] receiveq1
\item[1] transmitq1
\item[\ldots]
\item[2(N-1)] receiveqN
\item[2(N-1)+1] transmitqN
\item[2N]controlq
\end{description}
"
, it seems that simply giving the conversion formula here is also
helpful for readers to understand?
>
>> +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the
>> +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each
>> virtqueue of receiveq1\ldots receiveqN.
>> +
>> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the
>> +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command repeated for each
>> virtqueue of transmitq1\ldots transmitqN.
>> +
>> +If coalescing parameters are being set, the device applies the last
>> +coalescing parameters received for a virtqueue, regardless of the command
>> used to deliver the parameters. For example:
> used to set the parameters.
Ok.
>
>> +# Command sequence 1:
>> +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $
>> \field{max_packets}
>> += 15 $ and $ \field{max_usecs} = 10 $ 2.
>> +VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $
>> +\field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ After #2 command is
>> applied by the device, the coalescing parameters of receiveq1 are $
>> \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $, and the coalescing
>> parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and
>> $ \field{max_usecs} = 10 $.
>> +
>> +# Command sequence 2:
>> +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $,
>> $
>> +\field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $ 2.
>> +VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets}
>> = 15 $ and $ \field{max_usecs} = 10 $ After #2 command is applied by the
>> device, the coalescing parameters of receiveq1\ldots receiveqN are $
>> \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
>> +
>> \subparagraph{RX Notifications}\label{sec:Device Types / Network Device /
>> Device Operation / Control Virtqueue / Notifications Coalescing / RX
>> Notifications}
>>
>> If, for example:
>> @@ -4532,9 +4594,33 @@ \subsubsection{Control
>> Virtqueue}\label{sec:Device Types / Network Device / Devi
>>
>> \begin{itemize}
>> \item The device will count received packets until it accumulates 15, or until 10
>> usecs elapsed since the first one was received.
>> -\item If the notifications are not suppressed by the driver, the device will send
>> an used buffer notification, otherwise, the device will not send an used buffer
>> notification as long as the notifications are suppressed.
>> +\item If the notifications are not suppressed by the driver, the device will send
>> an used buffer notification, otherwise,
>> + the device will not send an used buffer notification as long as the
>> notifications are suppressed.
>> +\end{itemize}
>> +
>> +Example of setting coalescing parameters for a receive virtqueue:
>> +\begin{itemize}
>> +\item \field{max_packets} = 15
>> +\item \field{max_usecs} = 10
>> +\item \field{vqn} = 0
>> +\end{itemize}
>> +
>> +The device will operate as follows:
>> +\begin{itemize}
>> +\item The device applies the coalescing parameters to receiveq1 because its
>> virtqueue number is 0.
>> +\item The device will count received packets on receiveq1 until it accumulates
>> 15, or until 10 usecs elapsed since the first one was received.
>> +\item If the notifications are not suppressed by the driver, the device will send
>> an used buffer notification, otherwise,
>> + the device will not send an used buffer notification as long as the
>> notifications are suppressed.
>> +\end{itemize}
>> +
>> +Example of getting coalescing parameters from a receive virtqueue:
>> +\begin{itemize}
>> +\item \field{vqn} = 0.
>> \end{itemize}
>>
>> +The device gets the values of the \field{max_packets} and
>> +\field{max_usecs} parameters from the receiveq1 whose virtqueue number is
>> 0, and then responds them to the driver.
>> +
> The device returns the value of X and Y parameters for the supplied vq index.
Ok.
>
>> \subparagraph{TX Notifications}\label{sec:Device Types / Network Device /
>> Device Operation / Control Virtqueue / Notifications Coalescing / TX
>> Notifications}
>>
>> If, for example:
>> @@ -4550,14 +4636,50 @@ \subsubsection{Control
>> Virtqueue}\label{sec:Device Types / Network Device / Devi \item If the
>> notifications are not suppressed by the driver, the device will send an used
>> buffer notification, otherwise, the device will not send an used buffer
>> notification as long as the notifications are suppressed.
>> \end{itemize}
>>
>> +Example of setting coalescing parameters for a transmit virtqueue:
>> +\begin{itemize}
>> +\item \field{max_packets} = 15
>> +\item \field{max_usecs} = 10
>> +\item \field{vqn} = 1
>> +\end{itemize}
>> +
>> +The device will operate as follows:
>> +
>> +\begin{itemize}
>> +\item The device applies the coalescing parameters to transmitq1 because its
>> virtqueue number is 1.
>> +\item The device will count sent packets on transmitq1 until it accumulates 15,
>> or until 10 usecs elapsed since the first one was sent.
>> +\item If the notifications are not suppressed by the driver, the device will send
>> an used buffer notification, otherwise,
>> + the device will not send an used buffer notification as long as the
>> notifications are suppressed.
>> +\end{itemize}
>> +
>> +Example of getting coalescing parameters from a transmit virtqueue:
>> +\begin{itemize}
>> +\item \field{vqn} = 1.
>> +\end{itemize}
>> +
>> +The device gets the values of the \field{max_packets} and
>> +\field{max_usecs} parameters from the transmitq1 whose virtqueue number
>> is 1, and then responds them to the driver.
>> +
>> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types /
>> Network Device / Device Operation / Control Virtqueue / Notifications
>> Coalescing}
>>
>> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver
>> MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
>> +If neither the VIRTIO_NET_F_NOTF_COAL nor the
>> VIRTIO_NET_F_VQ_NOTF_COAL
>> +feature has not been negotiated, the driver MUST NOT issue
>> VIRTIO_NET_CTRL_NOTF_COAL commands.
>> +
>> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST
>> +set \field{vqn} to a legal value, that is, \field{vqn} points to an enabled
>> transmit/receive virtqueue.
>> +
>> +A driver MAY NOT get exactly the same value as the coalescing parameter it
>> was set to, for example it MAY get a value that is a power of 2.
>>
> It MAY get a value that is close to power of 2 value of the previously set value.
Ok. This is in line with Michael's suggestion.
>
>> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types /
>> Network Device / Device Operation / Control Virtqueue / Notifications
>> Coalescing}
>>
>> A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands
>> with VIRTIO_NET_ERR if it was not able to change the parameters.
>>
> Silently dropping the command is not good, we cannot change past now.
> But lets please fix the below one.
> Please add the note around above text that,
> Note: VQ_COAL is exception to it, which must return NET_ERR if device is unable to change the parameters.
Why are the corresponding behaviors of VIRTIO_NET_CTRL_NOTF_COAL and
VIRTIO_NET_CTRL_VQ_NOTF_COAL_SET inconsistent in the same scenario?
VIRTIO_NET_CTRL_NOTF_COAL --> SHOULD, but
VIRTIO_NET_CTRL_VQ_NOTF_COAL_SET --> MUST?
My guess is that when VIRTIO_NET_CTRL_NOTF_COAL is set for
receiveq1\ldots receiveqN, if recieveq1 is not set successfully because
it is disabled,
all other receiveq2\ldots receiveqN are set successfully. At this time,
the device can choose not to return an error?
>> +When a device receives a virtqueue command to set/get coalescing
>> +parameters for a virtqueue with number \field{vqn}, if the virtqueue is not
>> enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
>> +
> Device MUST respond with an error on the object that doesn't exist.
> Please change it to MUST.
>
> Please add below text. Need some wording change for above SHOULD.
>
> A device MUST respond to the VIRTIO_NET_CTRL_VQ_NOTF_COAL_GET and SET command with
> with VIRTIO_NET_ERR if it is not able to change the parameters.
When the device receives _VQ_NOTF_COAL_GET, it will not try to change
the parameters. And seeing here,
we seem to be able to avoid splitting the patch, because it seems
redundant to say this description once for GET and SET?
>
>
>> +A device MAY NOT set the coalescing parameter to the exact same value
>> +as the one passed in by a driver.
> s/as the one passed in by a driver/as the one set by the driver
Ok.
>
> For example, the value of \field{max_packets}
This is how I described it in the original v3 patch, did you miss this
sentence?
Thanks. :)
>> set by the driver is 15, but the device MAY store a value that is a power of 2,
>> that is, 16.
>> +
>> A device SHOULD NOT send used buffer notifications to the driver, if the
>> notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio
>> Device / Virtqueues / Used Buffer Notification Suppression}, even if the
>> coalescing counters expired.
>>
>> Upon reset, a device MUST initialize all coalescing parameters to 0.
>> --
>> 2.19.1.6.gb485710b
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 14:43 [PATCH v3] virtio-net: support the virtqueue coalescing moderation Heng Qi
` (2 preceding siblings ...)
2023-02-17 2:47 ` Parav Pandit
@ 2023-02-17 8:42 ` Alvaro Karsz
2023-02-17 9:31 ` [virtio-dev] " Heng Qi
2023-02-17 10:04 ` Michael S. Tsirkin
4 siblings, 1 reply; 29+ messages in thread
From: Alvaro Karsz @ 2023-02-17 8:42 UTC (permalink / raw)
To: Heng Qi
Cc: virtio-comment, virtio-dev, Michael S . Tsirkin, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
Hi Heng,
> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> + notifications coalescing.
> +
> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
>
> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> \end{description}
>
> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> };
>
> #define VIRTIO_NET_CTRL_NOTF_COAL 6
> - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> +
> \end{lstlisting}
>
> Coalescing parameters:
> @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \end{itemize}
>
>
> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> \begin{enumerate}
> \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> + transmit/receive virtqueue whose number is \field{vqn}.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> + responds them to the driver.
> \end{enumerate}
>
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> + parameters of a enabled transmit/receive virtqueue.
> +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
> +
> +\begin{lstlisting}
> +struct virtio_net_ctrl_coal_vq {
> + le16 vqn;
> + le16 reserved;
> + le32 max_packets;
> + le32 max_usecs;
> +};
> +\end{lstlisting}
> +
Maybe we can use struct virtio_net_ctrl_coal inside struct
virtio_net_ctrl_coal_vq instead of repeating max_usecs and
max_packets?
I'm not sure if it would be confusing, what do you think?
> +Virtqueue coalescing parameters:
> +\begin{itemize}
> +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> +\end{itemize}
> +
> +\field{reserved} is reserved and it is ignored by the device.
> +
max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
("Maximum number of packets to receive/send before a RX/TX notification").
The fact that this is applied to all VQs or to a specific one is
derived from the command.
Same for max_usecs.
Maybe we can join the coalescing parameters somehow instead of
repeating the explanations?
> +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> +The conversion between \field{vqn} and transmitq/receiveq index:
> +\begin{itemize}
> +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> +\end{itemize}
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> +
> +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> +virtqueue, regardless of the command used to deliver the parameters. For example:
> +# Command sequence 1:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> +# Command sequence 2:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
>
> If, for example:
> @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>
> \begin{itemize}
> \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
Both sentences seem similar to me.
> +\end{itemize}
> +
> +Example of setting coalescing parameters for a receive virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 0
> +\end{itemize}
> +
> +The device will operate as follows:
> +\begin{itemize}
> +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a receive virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 0.
> \end{itemize}
>
> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> +and then responds them to the driver.
> +
> \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
>
> If, for example:
> @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> \end{itemize}
>
> +Example of setting coalescing parameters for a transmit virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 1
> +\end{itemize}
> +
> +The device will operate as follows:
> +
> +\begin{itemize}
> +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a transmit virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 1.
> +\end{itemize}
> +
> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> +and then responds them to the driver.
> +
> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>
> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
> +
> +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
>
Maybe we can add here that the driver must ignore the values received
from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, if the device responds with
VIRTIO_NET_ERR?
> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>
> A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
>
> +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
> +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
> +
> +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> +
> A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
>
> Upon reset, a device MUST initialize all coalescing parameters to 0.
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 8:42 ` [virtio-comment] " Alvaro Karsz
@ 2023-02-17 9:31 ` Heng Qi
2023-02-17 9:40 ` Alvaro Karsz
0 siblings, 1 reply; 29+ messages in thread
From: Heng Qi @ 2023-02-17 9:31 UTC (permalink / raw)
To: Alvaro Karsz
Cc: virtio-comment, virtio-dev, Michael S . Tsirkin, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
On Fri, Feb 17, 2023 at 10:42:21AM +0200, Alvaro Karsz wrote:
> Hi Heng,
>
> > +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> > + notifications coalescing.
> > +
> > \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
> >
> > \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> > @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> > \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> > \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> > +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> > \end{description}
> >
> > \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> > @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > };
> >
> > #define VIRTIO_NET_CTRL_NOTF_COAL 6
> > - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> > #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> > + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> > +
> > \end{lstlisting}
> >
> > Coalescing parameters:
> > @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > \end{itemize}
> >
> >
> > -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> > +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> > \begin{enumerate}
> > \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> > \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> > + transmit/receive virtqueue whose number is \field{vqn}.
> > +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> > + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> > + responds them to the driver.
> > \end{enumerate}
> >
> > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> > +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> > + parameters of a enabled transmit/receive virtqueue.
> > +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> > + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
> > +
> > +\begin{lstlisting}
> > +struct virtio_net_ctrl_coal_vq {
> > + le16 vqn;
> > + le16 reserved;
> > + le32 max_packets;
> > + le32 max_usecs;
> > +};
> > +\end{lstlisting}
> > +
>
> Maybe we can use struct virtio_net_ctrl_coal inside struct
> virtio_net_ctrl_coal_vq instead of repeating max_usecs and
> max_packets?
> I'm not sure if it would be confusing, what do you think?
>
Hi Alvaro.
I guess you mean one of the following two forms:
#1
struct virtio_net_ctrl_coal {
le32 max_packets;
le32 max_usecs;
};
struct virtio_net_ctrl_coal_vq {
le16 vqn;
le16 reserved;
struct virtio_net_ctrl_coal coal;
} coal_vq;
#2
struct virtio_net_ctrl_coal {
le32 max_packets;
le32 max_usecs;
le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
};
If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the presence of vqn and reserved is awkward.
If it's #2, I think this is a bit like the v1 version, using virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.
I think we should hear from Michael and Parav.
> > +Virtqueue coalescing parameters:
> > +\begin{itemize}
> > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > +\end{itemize}
> > +
> > +\field{reserved} is reserved and it is ignored by the device.
> > +
>
> max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
> VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
> ("Maximum number of packets to receive/send before a RX/TX notification").
> The fact that this is applied to all VQs or to a specific one is
> derived from the command.
> Same for max_usecs.
> Maybe we can join the coalescing parameters somehow instead of
> repeating the explanations?
>
> > +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> > +The conversion between \field{vqn} and transmitq/receiveq index:
> > +\begin{itemize}
> > +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> > +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> > +\end{itemize}
> > +
> > +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> > +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> > +
> > +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> > +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> > +
> > +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> > +virtqueue, regardless of the command used to deliver the parameters. For example:
> > +# Command sequence 1:
> > +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> > +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > +
> > +# Command sequence 2:
> > +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> > +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> > +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> > +
> > \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
> >
> > If, for example:
> > @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >
> > \begin{itemize}
> > \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> > -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
>
> Both sentences seem similar to me.
Yes, this is an irrelevant modification, I will remove this modification.
>
> > +\end{itemize}
> > +
> > +Example of setting coalescing parameters for a receive virtqueue:
> > +\begin{itemize}
> > +\item \field{max_packets} = 15
> > +\item \field{max_usecs} = 10
> > +\item \field{vqn} = 0
> > +\end{itemize}
> > +
> > +The device will operate as follows:
> > +\begin{itemize}
> > +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> > +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\end{itemize}
> > +
> > +Example of getting coalescing parameters from a receive virtqueue:
> > +\begin{itemize}
> > +\item \field{vqn} = 0.
> > \end{itemize}
> >
> > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> > +and then responds them to the driver.
> > +
> > \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
> >
> > If, for example:
> > @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> > \end{itemize}
> >
> > +Example of setting coalescing parameters for a transmit virtqueue:
> > +\begin{itemize}
> > +\item \field{max_packets} = 15
> > +\item \field{max_usecs} = 10
> > +\item \field{vqn} = 1
> > +\end{itemize}
> > +
> > +The device will operate as follows:
> > +
> > +\begin{itemize}
> > +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> > +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> > +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> > + the device will not send an used buffer notification as long as the notifications are suppressed.
> > +\end{itemize}
> > +
> > +Example of getting coalescing parameters from a transmit virtqueue:
> > +\begin{itemize}
> > +\item \field{vqn} = 1.
> > +\end{itemize}
> > +
> > +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> > +and then responds them to the driver.
> > +
> > \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> >
> > -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> > +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> > +
> > +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> > +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
> > +
> > +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
> >
>
> Maybe we can add here that the driver must ignore the values received
> from VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET, if the device responds with
> VIRTIO_NET_ERR?
Good idea.
Thanks.
>
> > \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
> >
> > A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
> >
> > +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
> > +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
> > +
> > +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> > +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> > +
> > A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
> >
> > Upon reset, a device MUST initialize all coalescing parameters to 0.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 9:31 ` [virtio-dev] " Heng Qi
@ 2023-02-17 9:40 ` Alvaro Karsz
2023-02-17 10:06 ` Heng Qi
2023-02-17 10:07 ` Michael S. Tsirkin
0 siblings, 2 replies; 29+ messages in thread
From: Alvaro Karsz @ 2023-02-17 9:40 UTC (permalink / raw)
To: Heng Qi
Cc: virtio-comment, virtio-dev, Michael S . Tsirkin, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
> > Maybe we can use struct virtio_net_ctrl_coal inside struct
> > virtio_net_ctrl_coal_vq instead of repeating max_usecs and
> > max_packets?
> > I'm not sure if it would be confusing, what do you think?
> >
>
> Hi Alvaro.
>
> I guess you mean one of the following two forms:
>
> #1
> struct virtio_net_ctrl_coal {
> le32 max_packets;
> le32 max_usecs;
> };
>
> struct virtio_net_ctrl_coal_vq {
> le16 vqn;
> le16 reserved;
> struct virtio_net_ctrl_coal coal;
> } coal_vq;
>
> #2
> struct virtio_net_ctrl_coal {
> le32 max_packets;
> le32 max_usecs;
> le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
> le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
> };
>
> If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the presence of vqn and reserved is awkward.
> If it's #2, I think this is a bit like the v1 version, using virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.
>
> I think we should hear from Michael and Parav.
>
I meant #1.
We can see virtio_net_ctrl_coal as a struct holding coalescing
parameters, regardless of the commands.
Yes, let's wait for more comments on that.
> > > +Virtqueue coalescing parameters:
> > > +\begin{itemize}
> > > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > > +\end{itemize}
> > > +
> > > +\field{reserved} is reserved and it is ignored by the device.
> > > +
> >
> > max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
> > VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
> > ("Maximum number of packets to receive/send before a RX/TX notification").
> > The fact that this is applied to all VQs or to a specific one is
> > derived from the command.
> > Same for max_usecs.
> > Maybe we can join the coalescing parameters somehow instead of
> > repeating the explanations?
> >
Any thoughts on this part?
---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 9:40 ` Alvaro Karsz
@ 2023-02-17 10:06 ` Heng Qi
2023-02-17 10:09 ` Michael S. Tsirkin
2023-02-17 10:07 ` Michael S. Tsirkin
1 sibling, 1 reply; 29+ messages in thread
From: Heng Qi @ 2023-02-17 10:06 UTC (permalink / raw)
To: Alvaro Karsz
Cc: virtio-comment, virtio-dev, Michael S . Tsirkin, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
On Fri, Feb 17, 2023 at 11:40:15AM +0200, Alvaro Karsz wrote:
> > > Maybe we can use struct virtio_net_ctrl_coal inside struct
> > > virtio_net_ctrl_coal_vq instead of repeating max_usecs and
> > > max_packets?
> > > I'm not sure if it would be confusing, what do you think?
> > >
> >
> > Hi Alvaro.
> >
> > I guess you mean one of the following two forms:
> >
> > #1
> > struct virtio_net_ctrl_coal {
> > le32 max_packets;
> > le32 max_usecs;
> > };
> >
> > struct virtio_net_ctrl_coal_vq {
> > le16 vqn;
> > le16 reserved;
> > struct virtio_net_ctrl_coal coal;
> > } coal_vq;
> >
> > #2
> > struct virtio_net_ctrl_coal {
> > le32 max_packets;
> > le32 max_usecs;
> > le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
> > le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
> > };
> >
> > If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the presence of vqn and reserved is awkward.
> > If it's #2, I think this is a bit like the v1 version, using virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.
> >
> > I think we should hear from Michael and Parav.
> >
>
> I meant #1.
> We can see virtio_net_ctrl_coal as a struct holding coalescing
> parameters, regardless of the commands.
> Yes, let's wait for more comments on that.
>
> > > > +Virtqueue coalescing parameters:
> > > > +\begin{itemize}
> > > > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > > > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > > > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > > > +\end{itemize}
> > > > +
> > > > +\field{reserved} is reserved and it is ignored by the device.
> > > > +
> > >
> > > max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
> > > VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
> > > ("Maximum number of packets to receive/send before a RX/TX notification").
> > > The fact that this is applied to all VQs or to a specific one is
> > > derived from the command.
> > > Same for max_usecs.
> > > Maybe we can join the coalescing parameters somehow instead of
> > > repeating the explanations?
> > >
>
> Any thoughts on this part?
Good idea, and if so, is there a good way to expose vqn to the interpretation of max_packets ?
#1
\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue.
\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
#2
\item \field{max_packets}: Maximum number of packets to receive/send before a RX/TX notification.
Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 10:06 ` Heng Qi
@ 2023-02-17 10:09 ` Michael S. Tsirkin
0 siblings, 0 replies; 29+ messages in thread
From: Michael S. Tsirkin @ 2023-02-17 10:09 UTC (permalink / raw)
To: Heng Qi
Cc: Alvaro Karsz, virtio-comment, virtio-dev, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
On Fri, Feb 17, 2023 at 06:06:03PM +0800, Heng Qi wrote:
> On Fri, Feb 17, 2023 at 11:40:15AM +0200, Alvaro Karsz wrote:
> > > > Maybe we can use struct virtio_net_ctrl_coal inside struct
> > > > virtio_net_ctrl_coal_vq instead of repeating max_usecs and
> > > > max_packets?
> > > > I'm not sure if it would be confusing, what do you think?
> > > >
> > >
> > > Hi Alvaro.
> > >
> > > I guess you mean one of the following two forms:
> > >
> > > #1
> > > struct virtio_net_ctrl_coal {
> > > le32 max_packets;
> > > le32 max_usecs;
> > > };
> > >
> > > struct virtio_net_ctrl_coal_vq {
> > > le16 vqn;
> > > le16 reserved;
> > > struct virtio_net_ctrl_coal coal;
> > > } coal_vq;
> > >
> > > #2
> > > struct virtio_net_ctrl_coal {
> > > le32 max_packets;
> > > le32 max_usecs;
> > > le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
> > > le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
> > > };
> > >
> > > If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the presence of vqn and reserved is awkward.
> > > If it's #2, I think this is a bit like the v1 version, using virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.
> > >
> > > I think we should hear from Michael and Parav.
> > >
> >
> > I meant #1.
> > We can see virtio_net_ctrl_coal as a struct holding coalescing
> > parameters, regardless of the commands.
> > Yes, let's wait for more comments on that.
> >
> > > > > +Virtqueue coalescing parameters:
> > > > > +\begin{itemize}
> > > > > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > > > > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > > > > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > > > > +\end{itemize}
> > > > > +
> > > > > +\field{reserved} is reserved and it is ignored by the device.
> > > > > +
> > > >
> > > > max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
> > > > VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
> > > > ("Maximum number of packets to receive/send before a RX/TX notification").
> > > > The fact that this is applied to all VQs or to a specific one is
> > > > derived from the command.
> > > > Same for max_usecs.
> > > > Maybe we can join the coalescing parameters somehow instead of
> > > > repeating the explanations?
> > > >
> >
> > Any thoughts on this part?
>
> Good idea, and if so, is there a good way to expose vqn to the interpretation of max_packets ?
not sure what you are asking here.
>
> #1
> \item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue.
an enabled - 1st time you mention a virtqueue.
> \item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
>
> #2
> \item \field{max_packets}: Maximum number of packets to receive/send before a RX/TX notification.
>
> Thanks.
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 9:40 ` Alvaro Karsz
2023-02-17 10:06 ` Heng Qi
@ 2023-02-17 10:07 ` Michael S. Tsirkin
2023-02-17 15:54 ` Parav Pandit
2023-02-17 16:36 ` [virtio-comment] " Heng Qi
1 sibling, 2 replies; 29+ messages in thread
From: Michael S. Tsirkin @ 2023-02-17 10:07 UTC (permalink / raw)
To: Alvaro Karsz
Cc: Heng Qi, virtio-comment, virtio-dev, Parav Pandit,
Yuri Benditovich, Jason Wang, Cornelia Huck, Xuan Zhuo
On Fri, Feb 17, 2023 at 11:40:15AM +0200, Alvaro Karsz wrote:
> > > Maybe we can use struct virtio_net_ctrl_coal inside struct
> > > virtio_net_ctrl_coal_vq instead of repeating max_usecs and
> > > max_packets?
> > > I'm not sure if it would be confusing, what do you think?
> > >
> >
> > Hi Alvaro.
> >
> > I guess you mean one of the following two forms:
> >
> > #1
> > struct virtio_net_ctrl_coal {
> > le32 max_packets;
> > le32 max_usecs;
> > };
> >
> > struct virtio_net_ctrl_coal_vq {
> > le16 vqn;
> > le16 reserved;
> > struct virtio_net_ctrl_coal coal;
> > } coal_vq;
> >
> > #2
> > struct virtio_net_ctrl_coal {
> > le32 max_packets;
> > le32 max_usecs;
> > le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
> > le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
> > };
> >
> > If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the presence of vqn and reserved is awkward.
> > If it's #2, I think this is a bit like the v1 version, using virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.
> >
> > I think we should hear from Michael and Parav.
> >
>
> I meant #1.
> We can see virtio_net_ctrl_coal as a struct holding coalescing
> parameters, regardless of the commands.
> Yes, let's wait for more comments on that.
Reusing virtio_net_ctrl_coal is a nice thought. Makes it a bit clearer
these have exactly the same role.
Whether to put vqn first or last does not matter imho.
> > > > +Virtqueue coalescing parameters:
> > > > +\begin{itemize}
> > > > +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> > > > +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> > > > +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> > > > +\end{itemize}
> > > > +
> > > > +\field{reserved} is reserved and it is ignored by the device.
> > > > +
> > >
> > > max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
> > > VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
> > > ("Maximum number of packets to receive/send before a RX/TX notification").
> > > The fact that this is applied to all VQs or to a specific one is
> > > derived from the command.
> > > Same for max_usecs.
> > > Maybe we can join the coalescing parameters somehow instead of
> > > repeating the explanations?
> > >
>
> Any thoughts on this part?
I think I agree. Generally I think we should first of all describe the
new VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, moving all explanation text
to that.
Then just explain that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_RX_SET have the same effect
as VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET repeated for
all currently enabled tx/rx vqs.
Plus maybe a single annotated example where there's a mix of
VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands. For example with 2 vq pairs:
1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets for vq 0 and 2, vq 1 and 3 retain reset value
2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 0, vq 2 retains value from 1
3. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 1, vq 3 retains reset value
4. VIRTIO_NET_CTRL_NOTF_COAL_TX_SET overrides command 3
no need for many examples.
--
MST
^ permalink raw reply [flat|nested] 29+ messages in thread* RE: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 10:07 ` Michael S. Tsirkin
@ 2023-02-17 15:54 ` Parav Pandit
2023-02-17 16:36 ` [virtio-comment] " Heng Qi
1 sibling, 0 replies; 29+ messages in thread
From: Parav Pandit @ 2023-02-17 15:54 UTC (permalink / raw)
To: Michael S. Tsirkin, Alvaro Karsz
Cc: Heng Qi, virtio-comment@lists.oasis-open.org,
virtio-dev@lists.oasis-open.org, Yuri Benditovich, Jason Wang,
Cornelia Huck, Xuan Zhuo
> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Friday, February 17, 2023 5:07 AM
> > > #1
> > > struct virtio_net_ctrl_coal {
> > > le32 max_packets;
> > > le32 max_usecs;
> > > };
> > >
> > > struct virtio_net_ctrl_coal_vq {
> > > le16 vqn;
> > > le16 reserved;
> > > struct virtio_net_ctrl_coal coal; } coal_vq;
> > > I think we should hear from Michael and Parav.
> > >
> >
> > I meant #1.
> > We can see virtio_net_ctrl_coal as a struct holding coalescing
> > parameters, regardless of the commands.
> > Yes, let's wait for more comments on that.
>
+1 for #1.
> Reusing virtio_net_ctrl_coal is a nice thought. Makes it a bit clearer these have
> exactly the same role.
> Whether to put vqn first or last does not matter imho.
>
Putting VQN first looks better to me, to say for vqn -> below are the parameters.
We may find more parameters in future too in same struct.
At that point also having vqn first reads better.
> > > > > +Virtqueue coalescing parameters:
> > > > > +\begin{itemize}
> > > > > +\item \field{vqn}: The virtqueue number of the enabled transmit or
> receive virtqueue, excluding the control virtqueue.
> > > > > +\item \field{max_packets}: The maximum number of packets
> sent/received by the specified virtqueue before a TX/RX notification.
> > > > > +\item \field{max_usecs}: The maximum number of TX/RX usecs that
> the specified virtqueue delays a TX/RX notification.
> > > > > +\end{itemize}
> > > > > +
> > > > > +\field{reserved} is reserved and it is ignored by the device.
> > > > > +
> > > >
> > > > max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> and
> > > > with VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
> > > > ("Maximum number of packets to receive/send before a RX/TX
> notification").
> > > > The fact that this is applied to all VQs or to a specific one is
> > > > derived from the command.
> > > > Same for max_usecs.
> > > > Maybe we can join the coalescing parameters somehow instead of
> > > > repeating the explanations?
> > > >
> >
> > Any thoughts on this part?
>
> I think I agree. Generally I think we should first of all describe the new
> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, moving all explanation text to that.
>
> Then just explain that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and
> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET have the same effect as
> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET repeated for all currently enabled
> tx/rx vqs.
> Plus maybe a single annotated example where there's a mix of
> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
> VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands. For example with 2 vq
> pairs:
>
> 1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets for vq 0 and 2, vq 1 and 3 retain
> reset value 2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 0, vq 2 retains
> value from 1 3. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 1, vq 3 retains
> reset value 4. VIRTIO_NET_CTRL_NOTF_COAL_TX_SET overrides command 3
>
> no need for many examples.
+1.
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 10:07 ` Michael S. Tsirkin
2023-02-17 15:54 ` Parav Pandit
@ 2023-02-17 16:36 ` Heng Qi
1 sibling, 0 replies; 29+ messages in thread
From: Heng Qi @ 2023-02-17 16:36 UTC (permalink / raw)
To: Michael S. Tsirkin, Alvaro Karsz
Cc: virtio-comment, virtio-dev, Parav Pandit, Yuri Benditovich,
Jason Wang, Cornelia Huck, Xuan Zhuo
在 2023/2/17 下午6:07, Michael S. Tsirkin 写道:
> On Fri, Feb 17, 2023 at 11:40:15AM +0200, Alvaro Karsz wrote:
>>>> Maybe we can use struct virtio_net_ctrl_coal inside struct
>>>> virtio_net_ctrl_coal_vq instead of repeating max_usecs and
>>>> max_packets?
>>>> I'm not sure if it would be confusing, what do you think?
>>>>
>>> Hi Alvaro.
>>>
>>> I guess you mean one of the following two forms:
>>>
>>> #1
>>> struct virtio_net_ctrl_coal {
>>> le32 max_packets;
>>> le32 max_usecs;
>>> };
>>>
>>> struct virtio_net_ctrl_coal_vq {
>>> le16 vqn;
>>> le16 reserved;
>>> struct virtio_net_ctrl_coal coal;
>>> } coal_vq;
>>>
>>> #2
>>> struct virtio_net_ctrl_coal {
>>> le32 max_packets;
>>> le32 max_usecs;
>>> le16 vqn; // if _F_VQ_NOTF_COAL is negotiated
>>> le16 reserved; // if _F_VQ_NOTF_COAL is negotiated
>>> };
>>>
>>> If it's #1, I think the format is a bit ugly, it's not semantic to use coal_vq to send global commands when _F_VQ_NOTF_COAL is not negotiated, and the presence of vqn and reserved is awkward.
>>> If it's #2, I think this is a bit like the v1 version, using virtio_net_ctrl_coal as a virtual queue to send commands does not seem to be semantic, but it is indeed more unified in function.
>>>
>>> I think we should hear from Michael and Parav.
>>>
>> I meant #1.
>> We can see virtio_net_ctrl_coal as a struct holding coalescing
>> parameters, regardless of the commands.
>> Yes, let's wait for more comments on that.
> Reusing virtio_net_ctrl_coal is a nice thought. Makes it a bit clearer
> these have exactly the same role.
> Whether to put vqn first or last does not matter imho.
>
>>>>> +Virtqueue coalescing parameters:
>>>>> +\begin{itemize}
>>>>> +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
>>>>> +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
>>>>> +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
>>>>> +\end{itemize}
>>>>> +
>>>>> +\field{reserved} is reserved and it is ignored by the device.
>>>>> +
>>>> max_packets is the same with VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET and with
>>>> VIRTIO_NET_CTRL_NOTF_COAL_[T/R]X_SET.
>>>> ("Maximum number of packets to receive/send before a RX/TX notification").
>>>> The fact that this is applied to all VQs or to a specific one is
>>>> derived from the command.
>>>> Same for max_usecs.
>>>> Maybe we can join the coalescing parameters somehow instead of
>>>> repeating the explanations?
>>>>
>> Any thoughts on this part?
> I think I agree. Generally I think we should first of all describe the
> new VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, moving all explanation text
> to that.
>
> Then just explain that VIRTIO_NET_CTRL_NOTF_COAL_TX_SET and
> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET have the same effect
> as VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET repeated for
> all currently enabled tx/rx vqs.
> Plus maybe a single annotated example where there's a mix of
> VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET, VIRTIO_NET_CTRL_NOTF_COAL_RX_SET and
> VIRTIO_NET_CTRL_NOTF_COAL_TX_SET commands. For example with 2 vq pairs:
>
> 1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET sets for vq 0 and 2, vq 1 and 3 retain reset value
> 2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 0, vq 2 retains value from 1
> 3. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET sets vq 1, vq 3 retains reset value
> 4. VIRTIO_NET_CTRL_NOTF_COAL_TX_SET overrides command 3
>
> no need for many examples.
Good idea. This is a clear and comprehensive example.
Thanks.
>
This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.
In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.
Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-16 14:43 [PATCH v3] virtio-net: support the virtqueue coalescing moderation Heng Qi
` (3 preceding siblings ...)
2023-02-17 8:42 ` [virtio-comment] " Alvaro Karsz
@ 2023-02-17 10:04 ` Michael S. Tsirkin
2023-02-17 10:09 ` [virtio-comment] " Heng Qi
4 siblings, 1 reply; 29+ messages in thread
From: Michael S. Tsirkin @ 2023-02-17 10:04 UTC (permalink / raw)
To: Heng Qi
Cc: virtio-comment, virtio-dev, Parav Pandit, Yuri Benditovich,
Jason Wang, Cornelia Huck, Xuan Zhuo
On Thu, Feb 16, 2023 at 10:43:01PM +0800, Heng Qi wrote:
> Currently coalescing parameters are grouped for all transmit and receive
> virtqueues. This patch supports setting or getting the parameters for a
> specified virtqueue, and a typical application of this function is netdim[1].
>
> When the traffic between virtqueues is unbalanced, for example, one virtqueue
> is busy and another virtqueue is idle, then it will be very useful to
> control coalescing parameters at the virtqueue granularity.
>
> [1] https://docs.kernel.org/networking/net_dim.html
>
> Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> ---
> v2->v3:
> 1. Add the netdim link. @Parav Pandit
> 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
> 3. _VQ_GET is explained more. @Michael S. Tsirkin
> 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
> 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
> 7. Fix some typos. @Michael S. Tsirkin
>
> v1->v2:
> 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
> 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
> 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
> 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
> 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
> 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
>
> content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--
> 1 file changed, 126 insertions(+), 4 deletions(-)
>
> diff --git a/content.tex b/content.tex
> index e863709..7f99503 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
> \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
> channel.
>
> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
> + notifications coalescing.
btw no "the" here. 1st time we mention it. and notification not
notifications - reduced relative here, uses the singular form.
> +
> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
>
> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
> @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
> \end{description}
>
> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
> @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> };
>
> #define VIRTIO_NET_CTRL_NOTF_COAL 6
> - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
> +
> \end{lstlisting}
>
> Coalescing parameters:
> @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \end{itemize}
>
>
> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
> \begin{enumerate}
> \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
> \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
> + transmit/receive virtqueue whose number is \field{vqn}.
> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
> + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
> + responds them to the driver.
> \end{enumerate}
>
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
> +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
> + parameters of a enabled transmit/receive virtqueue.
> +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
> + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
> +
> +\begin{lstlisting}
> +struct virtio_net_ctrl_coal_vq {
> + le16 vqn;
> + le16 reserved;
> + le32 max_packets;
> + le32 max_usecs;
> +};
> +\end{lstlisting}
> +
> +Virtqueue coalescing parameters:
> +\begin{itemize}
> +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
> +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
> +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
> +\end{itemize}
> +
> +\field{reserved} is reserved and it is ignored by the device.
> +
> +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
> +The conversion between \field{vqn} and transmitq/receiveq index:
> +\begin{itemize}
> +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
> +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
> +\end{itemize}
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> +command repeated for each virtqueue of receiveq1\ldots receiveqN.
> +
> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
> +command repeated for each virtqueue of transmitq1\ldots transmitqN.
> +
> +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
> +virtqueue, regardless of the command used to deliver the parameters. For example:
> +# Command sequence 1:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
> +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> +# Command sequence 2:
> +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
> +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
> +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
> +
> \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
>
> If, for example:
> @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>
> \begin{itemize}
> \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of setting coalescing parameters for a receive virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 0
> +\end{itemize}
> +
> +The device will operate as follows:
> +\begin{itemize}
> +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
> +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a receive virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 0.
> \end{itemize}
>
> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
> +and then responds them to the driver.
> +
> \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
>
> If, for example:
> @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
> \end{itemize}
>
> +Example of setting coalescing parameters for a transmit virtqueue:
> +\begin{itemize}
> +\item \field{max_packets} = 15
> +\item \field{max_usecs} = 10
> +\item \field{vqn} = 1
> +\end{itemize}
> +
> +The device will operate as follows:
> +
> +\begin{itemize}
> +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
> +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
> + the device will not send an used buffer notification as long as the notifications are suppressed.
> +\end{itemize}
> +
> +Example of getting coalescing parameters from a transmit virtqueue:
> +\begin{itemize}
> +\item \field{vqn} = 1.
> +\end{itemize}
> +
> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
> +and then responds them to the driver.
> +
> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>
> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
> +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
> +
> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
> +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
> +
> +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
>
> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>
> A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
>
> +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
> +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
> +
> +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
> +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
> +
> A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
>
> Upon reset, a device MUST initialize all coalescing parameters to 0.
> --
> 2.19.1.6.gb485710b
^ permalink raw reply [flat|nested] 29+ messages in thread* Re: [virtio-comment] Re: [PATCH v3] virtio-net: support the virtqueue coalescing moderation
2023-02-17 10:04 ` Michael S. Tsirkin
@ 2023-02-17 10:09 ` Heng Qi
0 siblings, 0 replies; 29+ messages in thread
From: Heng Qi @ 2023-02-17 10:09 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: virtio-comment, virtio-dev, Parav Pandit, Yuri Benditovich,
Jason Wang, Cornelia Huck, Xuan Zhuo
在 2023/2/17 下午6:04, Michael S. Tsirkin 写道:
> On Thu, Feb 16, 2023 at 10:43:01PM +0800, Heng Qi wrote:
>> Currently coalescing parameters are grouped for all transmit and receive
>> virtqueues. This patch supports setting or getting the parameters for a
>> specified virtqueue, and a typical application of this function is netdim[1].
>>
>> When the traffic between virtqueues is unbalanced, for example, one virtqueue
>> is busy and another virtqueue is idle, then it will be very useful to
>> control coalescing parameters at the virtqueue granularity.
>>
>> [1] https://docs.kernel.org/networking/net_dim.html
>>
>> Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
>> Reviewed-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>> ---
>> v2->v3:
>> 1. Add the netdim link. @Parav Pandit
>> 2. VIRTIO_NET_F_VQ_NOTF_COAL no longer depends on VIRTIO_NET_F_NOTF_COAL. @Michael S. Tsirkin, @Alvaro Karsz
>> 3. _VQ_GET is explained more. @Michael S. Tsirkin
>> 4. Add more examples to avoid misunderstandings. @Michael S. Tsirkin
>> 5. Clarify some statements. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
>> 6. Adjust the virtio_net_ctrl_coal_vq structure. @Michael S. Tsirkin
>> 7. Fix some typos. @Michael S. Tsirkin
>>
>> v1->v2:
>> 1. Rename VIRTIO_NET_F_PERQUEUE_NOTF_COAL to VIRTIO_NET_F_VQ_NOTF_COAL. @Michael S. Tsirkin
>> 2. Use the \field{vqn} instead of the qid. @Michael S. Tsirkin
>> 3. Unify tx and rx control structres into one structure virtio_net_ctrl_coal_vq. @Michael S. Tsirkin
>> 4. Add a new control command VIRTIO_NET_CTRL_NOTF_COAL_VQ. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
>> 5. The special value 0xFFF is removed because VIRTIO_NET_CTRL_NOTF_COAL can be used. @Alvaro Karsz
>> 6. Clarify some special scenarios. @Michael S. Tsirkin, @Parav Pandit, @Alvaro Karsz
>>
>> content.tex | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++--
>> 1 file changed, 126 insertions(+), 4 deletions(-)
>>
>> diff --git a/content.tex b/content.tex
>> index e863709..7f99503 100644
>> --- a/content.tex
>> +++ b/content.tex
>> @@ -3084,6 +3084,9 @@ \subsection{Feature bits}\label{sec:Device Types / Network Device / Feature bits
>> \item[VIRTIO_NET_F_CTRL_MAC_ADDR(23)] Set MAC address through control
>> channel.
>>
>> +\item[VIRTIO_NET_F_VQ_NOTF_COAL(52)] Device supports the virtqueue
>> + notifications coalescing.
>
> btw no "the" here. 1st time we mention it. and notification not
> notifications - reduced relative here, uses the singular form.
Will fix.
Thanks.
>
>> +
>> \item[VIRTIO_NET_F_NOTF_COAL(53)] Device supports notifications coalescing.
>>
>> \item[VIRTIO_NET_F_GUEST_USO4 (54)] Driver can receive USOv4 packets.
>> @@ -3140,6 +3143,7 @@ \subsubsection{Feature bit requirements}\label{sec:Device Types / Network Device
>> \item[VIRTIO_NET_F_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
>> \item[VIRTIO_NET_F_RSC_EXT] Requires VIRTIO_NET_F_HOST_TSO4 or VIRTIO_NET_F_HOST_TSO6.
>> \item[VIRTIO_NET_F_RSS] Requires VIRTIO_NET_F_CTRL_VQ.
>> +\item[VIRTIO_NET_F_VQ_NOTF_COAL] Requires VIRTIO_NET_F_CTRL_VQ.
>> \end{description}
>>
>> \subsubsection{Legacy Interface: Feature bits}\label{sec:Device Types / Network Device / Feature bits / Legacy Interface: Feature bits}
>> @@ -4501,8 +4505,11 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>> };
>>
>> #define VIRTIO_NET_CTRL_NOTF_COAL 6
>> - #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
>> + #define VIRTIO_NET_CTRL_NOTF_COAL_TX_SET 0
>> #define VIRTIO_NET_CTRL_NOTF_COAL_RX_SET 1
>> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET 2
>> + #define VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET 3
>> +
>> \end{lstlisting}
>>
>> Coalescing parameters:
>> @@ -4514,12 +4521,67 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>> \end{itemize}
>>
>>
>> -The class VIRTIO_NET_CTRL_NOTF_COAL has 2 commands:
>> +The class VIRTIO_NET_CTRL_NOTF_COAL has 4 commands:
>> \begin{enumerate}
>> \item VIRTIO_NET_CTRL_NOTF_COAL_TX_SET: set the \field{tx_usecs} and \field{tx_max_packets} parameters.
>> \item VIRTIO_NET_CTRL_NOTF_COAL_RX_SET: set the \field{rx_usecs} and \field{rx_max_packets} parameters.
>> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET: set the \field{max_packets} and \field{max_usecs} parameters for a enabled
>> + transmit/receive virtqueue whose number is \field{vqn}.
>> +\item VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET: the device gets the \field{max_packets} and \field{max_usecs} parameters of
>> + a enabled transmit/receive virtqueue whose number is \field{vqn}, and then
>> + responds them to the driver.
>> \end{enumerate}
>>
>> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated:
>> +1. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command to set the coalescing
>> + parameters of a enabled transmit/receive virtqueue.
>> +2. a driver can send a VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command to a device, and the device
>> + responds to the driver with the coalescing parameters of a enabled transmit/receive virtqueue.
>> +
>> +\begin{lstlisting}
>> +struct virtio_net_ctrl_coal_vq {
>> + le16 vqn;
>> + le16 reserved;
>> + le32 max_packets;
>> + le32 max_usecs;
>> +};
>> +\end{lstlisting}
>> +
>> +Virtqueue coalescing parameters:
>> +\begin{itemize}
>> +\item \field{vqn}: The virtqueue number of the enabled transmit or receive virtqueue, excluding the control virtqueue.
>> +\item \field{max_packets}: The maximum number of packets sent/received by the specified virtqueue before a TX/RX notification.
>> +\item \field{max_usecs}: The maximum number of TX/RX usecs that the specified virtqueue delays a TX/RX notification.
>> +\end{itemize}
>> +
>> +\field{reserved} is reserved and it is ignored by the device.
>> +
>> +THe value of \field{vqn} satisfies $ 0 \leq vqn < max_virtqueue_pairs \ast 2 $.
>> +The conversion between \field{vqn} and transmitq/receiveq index:
>> +\begin{itemize}
>> +$ \lfloor vqn / 2 \rfloor $ is the index of the corresponding receiveq.
>> +$ \lfloor (vqn / 2) + 1 \rfloor $ is the index of the corresponding tranmitq.
>> +\end{itemize}
>> +
>> +The VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
>> +command repeated for each virtqueue of receiveq1\ldots receiveqN.
>> +
>> +The VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command is the same as the VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET
>> +command repeated for each virtqueue of transmitq1\ldots transmitqN.
>> +
>> +If coalescing parameters are being set, the device applies the last coalescing parameters received for a
>> +virtqueue, regardless of the command used to deliver the parameters. For example:
>> +# Command sequence 1:
>> +1. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
>> +2. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
>> +After #2 command is applied by the device, the coalescing parameters of receiveq1 are $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $,
>> +and the coalescing parameters of receiveq2\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
>> +
>> +# Command sequence 2:
>> +1. VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET command with $ \field{vqn} = 0 $, $ \field{max_packets} = 12 $ and $ \field{max_usecs} = 8 $
>> +2. VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command with $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $
>> +After #2 command is applied by the device, the coalescing parameters of receiveq1\ldots receiveqN are $ \field{max_packets} = 15 $ and $ \field{max_usecs} = 10 $.
>> +
>> \subparagraph{RX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / RX Notifications}
>>
>> If, for example:
>> @@ -4532,9 +4594,33 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>>
>> \begin{itemize}
>> \item The device will count received packets until it accumulates 15, or until 10 usecs elapsed since the first one was received.
>> -\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
>> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
>> + the device will not send an used buffer notification as long as the notifications are suppressed.
>> +\end{itemize}
>> +
>> +Example of setting coalescing parameters for a receive virtqueue:
>> +\begin{itemize}
>> +\item \field{max_packets} = 15
>> +\item \field{max_usecs} = 10
>> +\item \field{vqn} = 0
>> +\end{itemize}
>> +
>> +The device will operate as follows:
>> +\begin{itemize}
>> +\item The device applies the coalescing parameters to receiveq1 because its virtqueue number is 0.
>> +\item The device will count received packets on receiveq1 until it accumulates 15, or until 10 usecs elapsed since the first one was received.
>> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
>> + the device will not send an used buffer notification as long as the notifications are suppressed.
>> +\end{itemize}
>> +
>> +Example of getting coalescing parameters from a receive virtqueue:
>> +\begin{itemize}
>> +\item \field{vqn} = 0.
>> \end{itemize}
>>
>> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the receiveq1 whose virtqueue number is 0,
>> +and then responds them to the driver.
>> +
>> \subparagraph{TX Notifications}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / TX Notifications}
>>
>> If, for example:
>> @@ -4550,14 +4636,50 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>> \item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise, the device will not send an used buffer notification as long as the notifications are suppressed.
>> \end{itemize}
>>
>> +Example of setting coalescing parameters for a transmit virtqueue:
>> +\begin{itemize}
>> +\item \field{max_packets} = 15
>> +\item \field{max_usecs} = 10
>> +\item \field{vqn} = 1
>> +\end{itemize}
>> +
>> +The device will operate as follows:
>> +
>> +\begin{itemize}
>> +\item The device applies the coalescing parameters to transmitq1 because its virtqueue number is 1.
>> +\item The device will count sent packets on transmitq1 until it accumulates 15, or until 10 usecs elapsed since the first one was sent.
>> +\item If the notifications are not suppressed by the driver, the device will send an used buffer notification, otherwise,
>> + the device will not send an used buffer notification as long as the notifications are suppressed.
>> +\end{itemize}
>> +
>> +Example of getting coalescing parameters from a transmit virtqueue:
>> +\begin{itemize}
>> +\item \field{vqn} = 1.
>> +\end{itemize}
>> +
>> +The device gets the values of the \field{max_packets} and \field{max_usecs} parameters from the transmitq1 whose virtqueue number is 1,
>> +and then responds them to the driver.
>> +
>> \drivernormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>>
>> -If the VIRTIO_NET_F_NOTF_COAL feature has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
>> +If neither the VIRTIO_NET_F_NOTF_COAL nor the VIRTIO_NET_F_VQ_NOTF_COAL feature
>> +has not been negotiated, the driver MUST NOT issue VIRTIO_NET_CTRL_NOTF_COAL commands.
>> +
>> +If the VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated, the driver MUST set \field{vqn}
>> +to a legal value, that is, \field{vqn} points to an enabled transmit/receive virtqueue.
>> +
>> +A driver MAY NOT get exactly the same value as the coalescing parameter it was set to, for example it MAY get a value that is a power of 2.
>>
>> \devicenormative{\subparagraph}{Notifications Coalescing}{Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing}
>>
>> A device SHOULD respond to the VIRTIO_NET_CTRL_NOTF_COAL commands with VIRTIO_NET_ERR if it was not able to change the parameters.
>>
>> +When a device receives a virtqueue command to set/get coalescing parameters for a virtqueue with number \field{vqn},
>> +if the virtqueue is not enabled, the device SHOULD respond to the command with VIRTIO_NET_ERR.
>> +
>> +A device MAY NOT set the coalescing parameter to the exact same value as the one passed in by a driver. For example,
>> +the value of \field{max_packets} set by the driver is 15, but the device MAY store a value that is a power of 2, that is, 16.
>> +
>> A device SHOULD NOT send used buffer notifications to the driver, if the notifications are suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}, even if the coalescing counters expired.
>>
>> Upon reset, a device MUST initialize all coalescing parameters to 0.
>> --
>> 2.19.1.6.gb485710b
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
^ permalink raw reply [flat|nested] 29+ messages in thread