[PATCH v5] virtio-net: clarify coalescing parameters settings

[PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

The device can set any initial coalescing parameters (0 or non-zero)
for the receive/send queue before the setting command is executed,
not just 0, enhancing device performance even without DIM enabled.

So we need to clarify descriptions that don't fit the behavior.

Fixes: https://github.com/oasis-tcs/virtio-spec/issues/194
Suggested-by: Jason Wang <jasowang@redhat.com>
Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
Reviewed-by: Parav Pandit <parav@nvidia.com>
Reviewed-by: Jason Wang <jasowang@redhat.com>
---
v4->v5:
  - Add RB tags from Parav and Jason. Thanks!

v3->v4:
  - Doesn't force the device to remember more stuff. @Parav

v2->v3:
  - Clarify description to be more generic. @Parav

v1->v2:
  - Update description. @Jason

 device-types/net/description.tex | 14 +++++++++++---
 1 file changed, 11 insertions(+), 3 deletions(-)

diff --git a/device-types/net/description.tex b/device-types/net/description.tex
index 61cce1f..00c1b36 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
 \end{enumerate}
 
+If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated,
+the device may apply any coalescing parameters to each transmit/receive virtqueue
+before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.
+
+The driver can query the coalescing parameters of any enabled transmit/receive
+virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any
+VIRTIO_NET_CTRL_NOTF_COAL set command is done.
+
 The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
 
 If coalescing parameters are being set, the device applies the last coalescing parameters set for a
@@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
 
 Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
-to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
+to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.
 
 Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
-to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
+to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values.
 
 The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
 the device MAY generate notifications more or less frequently than specified.
 
 A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
 
-Upon reset, a device MUST initialize all coalescing parameters to 0.
+Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.
 
 \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}
 
-- 
2.32.0.3.g01195cf9f

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Tue, 28 May 2024 12:47:02 +0800, Heng Qi <hengqi@linux.alibaba.com> wrote:
> The device can set any initial coalescing parameters (0 or non-zero)
> for the receive/send queue before the setting command is executed,
> not just 0, enhancing device performance even without DIM enabled.
> 
> So we need to clarify descriptions that don't fit the behavior.
> 
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/194
> Suggested-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> Reviewed-by: Parav Pandit <parav@nvidia.com>
> Reviewed-by: Jason Wang <jasowang@redhat.com>

Hi, Cornelia!

I'm wondering if our voting infrastructure is already working properly, and if
so, can we turn on voting for this update patch?

Thanks a lot!

> ---
> v4->v5:
>   - Add RB tags from Parav and Jason. Thanks!
> 
> v3->v4:
>   - Doesn't force the device to remember more stuff. @Parav
> 
> v2->v3:
>   - Clarify description to be more generic. @Parav
> 
> v1->v2:
>   - Update description. @Jason
> 
>  device-types/net/description.tex | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> index 61cce1f..00c1b36 100644
> --- a/device-types/net/description.tex
> +++ b/device-types/net/description.tex
> @@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
>  \end{enumerate}
>  
> +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated,
> +the device may apply any coalescing parameters to each transmit/receive virtqueue
> +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.
> +
> +The driver can query the coalescing parameters of any enabled transmit/receive
> +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any
> +VIRTIO_NET_CTRL_NOTF_COAL set command is done.
> +
>  The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
>  
>  If coalescing parameters are being set, the device applies the last coalescing parameters set for a
> @@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>  VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
>  
>  Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
> -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
> +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.
>  
>  Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values.
>  
>  The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
>  the device MAY generate notifications more or less frequently than specified.
>  
>  A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
>  
> -Upon reset, a device MUST initialize all coalescing parameters to 0.
> +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.
>  
>  \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}
>  
> -- 
> 2.32.0.3.g01195cf9f
> 
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Tue, 28 May 2024 12:50:34 +0800, Heng Qi <hengqi@linux.alibaba.com> wrote:
> On Tue, 28 May 2024 12:47:02 +0800, Heng Qi <hengqi@linux.alibaba.com> wrote:
> > The device can set any initial coalescing parameters (0 or non-zero)
> > for the receive/send queue before the setting command is executed,
> > not just 0, enhancing device performance even without DIM enabled.
> > 
> > So we need to clarify descriptions that don't fit the behavior.
> > 
> > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/194
> > Suggested-by: Jason Wang <jasowang@redhat.com>
> > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> > Reviewed-by: Parav Pandit <parav@nvidia.com>
> > Reviewed-by: Jason Wang <jasowang@redhat.com>
> 
> Hi, Cornelia!
> 
> I'm wondering if our voting infrastructure is already working properly, and if
> so, can we turn on voting for this update patch?

Do I need to repost a patch with the subject prefix [Request for vote]?

> 
> Thanks a lot!
> 
> > ---
> > v4->v5:
> >   - Add RB tags from Parav and Jason. Thanks!
> > 
> > v3->v4:
> >   - Doesn't force the device to remember more stuff. @Parav
> > 
> > v2->v3:
> >   - Clarify description to be more generic. @Parav
> > 
> > v1->v2:
> >   - Update description. @Jason
> > 
> >  device-types/net/description.tex | 14 +++++++++++---
> >  1 file changed, 11 insertions(+), 3 deletions(-)
> > 
> > diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> > index 61cce1f..00c1b36 100644
> > --- a/device-types/net/description.tex
> > +++ b/device-types/net/description.tex
> > @@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
> >  \end{enumerate}
> >  
> > +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated,
> > +the device may apply any coalescing parameters to each transmit/receive virtqueue
> > +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.
> > +
> > +The driver can query the coalescing parameters of any enabled transmit/receive
> > +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any
> > +VIRTIO_NET_CTRL_NOTF_COAL set command is done.
> > +
> >  The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> >  
> >  If coalescing parameters are being set, the device applies the last coalescing parameters set for a
> > @@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >  VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
> >  
> >  Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
> > -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
> > +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.
> >  
> >  Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> > -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> > +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values.
> >  
> >  The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> >  the device MAY generate notifications more or less frequently than specified.
> >  
> >  A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> >  
> > -Upon reset, a device MUST initialize all coalescing parameters to 0.
> > +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.
> >  
> >  \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}
> >  
> > -- 
> > 2.32.0.3.g01195cf9f
> > 
> > 
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Cornelia Huck]

On Fri, May 31 2024, Heng Qi <hengqi@linux.alibaba.com> wrote:

> On Tue, 28 May 2024 12:50:34 +0800, Heng Qi <hengqi@linux.alibaba.com> wrote:
>> On Tue, 28 May 2024 12:47:02 +0800, Heng Qi <hengqi@linux.alibaba.com> wrote:
>> > The device can set any initial coalescing parameters (0 or non-zero)
>> > for the receive/send queue before the setting command is executed,
>> > not just 0, enhancing device performance even without DIM enabled.
>> > 
>> > So we need to clarify descriptions that don't fit the behavior.
>> > 
>> > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/194
>> > Suggested-by: Jason Wang <jasowang@redhat.com>
>> > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
>> > Reviewed-by: Parav Pandit <parav@nvidia.com>
>> > Reviewed-by: Jason Wang <jasowang@redhat.com>
>> 
>> Hi, Cornelia!
>> 
>> I'm wondering if our voting infrastructure is already working properly, and if
>> so, can we turn on voting for this update patch?

Oh, seems I forgot to send that mail out -- platform still has issues :(

>
> Do I need to repost a patch with the subject prefix [Request for vote]?

This one is already sitting on my list, no need to do so (and we haven't
actually finalized what it should look like, I'm still waiting for more
feedback before proposing a patch.)

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Halil Pasic]

On Tue, 28 May 2024 12:47:02 +0800
Heng Qi <hengqi@linux.alibaba.com> wrote:

> The device can set any initial coalescing parameters (0 or non-zero)
> for the receive/send queue before the setting command is executed,
> not just 0, enhancing device performance even without DIM enabled.
> 
> So we need to clarify descriptions that don't fit the behavior.

Sorry I'm late to the party -- again! Just for my understanding: how/why
is this a clarification and not just a (basically incompatible) change?

I mean if I read this correctly, before the driver had the guaranty
that if the parameters are not set by the driver, negotiating the
feature does not introduce any coalescing. After this in theory
the device could just pick some max value and potentially introduce
maximal latency in certain scenarios.

I understand that it is probably in the best interest of the devices to
not pick stupid defaults. But it is also probably in the best interest
of the driver to set those params, and if the driver is going to set its
values, the devices defaults are moot unless we assume that those may end
up being used by the driver as a hint when deciding which parameter
values to choose.

What am I missing here?

Regards,
Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Fri, 7 Jun 2024 22:02:46 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> On Tue, 28 May 2024 12:47:02 +0800
> Heng Qi <hengqi@linux.alibaba.com> wrote:
> 
> > The device can set any initial coalescing parameters (0 or non-zero)
> > for the receive/send queue before the setting command is executed,
> > not just 0, enhancing device performance even without DIM enabled.
> > 
> > So we need to clarify descriptions that don't fit the behavior.
> 
> Sorry I'm late to the party -- again! Just for my understanding: how/why
> is this a clarification and not just a (basically incompatible) change?

In my opinion, "clarification" means that something may have been described
incorrectly before, and we now need to discuss, explain clearly, and correct
the possibly incorrect description.

> 
> I mean if I read this correctly, before the driver had the guaranty
> that if the parameters are not set by the driver, negotiating the
> feature does not introduce any coalescing. After this in theory
> the device could just pick some max value and potentially introduce
> maximal latency in certain scenarios.

"maximum latency" also means "throughput improvement".

> 
> I understand that it is probably in the best interest of the devices to
> not pick stupid defaults. But it is also probably in the best interest
> of the driver to set those params, and if the driver is going to set its
> values, the devices defaults are moot unless we assume that those may end
> up being used by the driver as a hint when deciding which parameter
> values to choose.

"Any values" is compatible with "0 for max-usecs and max-frames", and the device
can choose "no coalescing".

"No coalescing" means "latency friendly", but it also means "a lot of
interruptions and throughput unfriendly". If the device chooses a stupid
maximum value, it is his choice (spec should give more devices choices instead of
forcing them to choose "0" which is not the best practice). We can't talk about
performance for drivers when the devices tend to choose any "stupid" designs.

We need relaxe the restrictions and makes the spec more reasonable.

Thanks.

> What am I missing here?
> 
> Regards,
> Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Halil Pasic]

On Sat, 8 Jun 2024 10:34:22 +0800
Heng Qi <hengqi@linux.alibaba.com> wrote:

> On Fri, 7 Jun 2024 22:02:46 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> > On Tue, 28 May 2024 12:47:02 +0800
> > Heng Qi <hengqi@linux.alibaba.com> wrote:
> >   
> > > The device can set any initial coalescing parameters (0 or non-zero)
> > > for the receive/send queue before the setting command is executed,
> > > not just 0, enhancing device performance even without DIM enabled.
> > > 
> > > So we need to clarify descriptions that don't fit the behavior.  
> > 
> > Sorry I'm late to the party -- again! Just for my understanding: how/why
> > is this a clarification and not just a (basically incompatible) change?  
> 
> In my opinion, "clarification" means that something may have been described
> incorrectly before, and we now need to discuss, explain clearly, and correct
> the possibly incorrect description.
> 

I figure the difference in perceived semantics of the word
"clarification" is at the root of my confusion. Let us have a look at
https://dictionary.cambridge.org/de/worterbuch/englisch/clarification

According to my understanding a "clarification", while an improvement in
ease of understanding and/or decrease of ambiguity (possibly to no
ambiguity at all) implies that what receiving a clarification is not
outright wrong.

When rectifying something that is outright incorrect or wrong, I would
refer to that with words like "correction", "fix", "erratum" or
"corrigendum".

> > 
> > I mean if I read this correctly, before the driver had the guaranty
> > that if the parameters are not set by the driver, negotiating the
> > feature does not introduce any coalescing. After this in theory
> > the device could just pick some max value and potentially introduce
> > maximal latency in certain scenarios.  
> 
> "maximum latency" also means "throughput improvement".
> 

Under certain assumptions. But not necessarily. Again my concern is
mostly the type of change. The virtio standard maintain a revision
history appendix, and I would like to avoid the nature of this change
being misrepresented there. If Connie and/or Michael think it is worth
fixing, I believe it can be fixed with an editorial change.

AFAIU VIRTIO_NET_F_NOTF_COAL and VIRTIO_NET_F_VQ_NOTF_COAL are about to
land with virtio-1.3, i.e. there is no released/standardized virtio
version where the "initialize to 0" is released. In that sense it looks
like we are still on time to change this. But I am not 100% certain. In
any case I don't think this as a huge impact and I'm fine going ahead
with the change.

> > 
> > I understand that it is probably in the best interest of the devices to
> > not pick stupid defaults. But it is also probably in the best interest
> > of the driver to set those params, and if the driver is going to set its
> > values, the devices defaults are moot unless we assume that those may end
> > up being used by the driver as a hint when deciding which parameter
> > values to choose.  
> 
> "Any values" is compatible with "0 for max-usecs and max-frames", and the device
> can choose "no coalescing".
> 
> "No coalescing" means "latency friendly", but it also means "a lot of
> interruptions and throughput unfriendly". 

To what extent does virtio's "normal" notification suppression
(VIRTIO_F_EVENT_IDX) would alleviate that in practice? At least in
theory the interruption suppression could save us there right? 


> If the device chooses a stupid
> maximum value, it is his choice (spec should give more devices choices instead of
> forcing them to choose "0" which is not the best practice). We can't talk about
> performance for drivers when the devices tend to choose any "stupid" designs.
> 
> We need relaxe the restrictions and makes the spec more reasonable.
> 

Hm, I see Linux virtio-net changes have landed with v6.0 and if I read
those correctly the driver -- contrary to my initial expectation --
negotiates the feature, but does not set the parameters explicitly and
thus keeps the defaults (until userspace decides to set the parameters).
So it does matter whether the defaults are guaranteed to be 0 or not,
and if not it does matter what defaults are chosen by the device.

One could even argue that those patches have been reviewed under the
assumption that the device needs to use 0 as the default parameter value.

Well no strong opinions here. If the community is fine with it, I'm fine
with it as well.

Thank you!

Regards,
Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> On Sat, 8 Jun 2024 10:34:22 +0800
> Heng Qi <hengqi@linux.alibaba.com> wrote:
> 
> > On Fri, 7 Jun 2024 22:02:46 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> > > On Tue, 28 May 2024 12:47:02 +0800
> > > Heng Qi <hengqi@linux.alibaba.com> wrote:
> > >   
> > > > The device can set any initial coalescing parameters (0 or non-zero)
> > > > for the receive/send queue before the setting command is executed,
> > > > not just 0, enhancing device performance even without DIM enabled.
> > > > 
> > > > So we need to clarify descriptions that don't fit the behavior.  
> > > 
> > > Sorry I'm late to the party -- again! Just for my understanding: how/why
> > > is this a clarification and not just a (basically incompatible) change?  
> > 
> > In my opinion, "clarification" means that something may have been described
> > incorrectly before, and we now need to discuss, explain clearly, and correct
> > the possibly incorrect description.
> > 
> 
> I figure the difference in perceived semantics of the word
> "clarification" is at the root of my confusion. Let us have a look at
> https://dictionary.cambridge.org/de/worterbuch/englisch/clarification
> 
> According to my understanding a "clarification", while an improvement in
> ease of understanding and/or decrease of ambiguity (possibly to no
> ambiguity at all) implies that what receiving a clarification is not
> outright wrong.
> 
> When rectifying something that is outright incorrect or wrong, I would
> refer to that with words like "correction", "fix", "erratum" or
> "corrigendum".
> 
> > > 
> > > I mean if I read this correctly, before the driver had the guaranty
> > > that if the parameters are not set by the driver, negotiating the
> > > feature does not introduce any coalescing. After this in theory
> > > the device could just pick some max value and potentially introduce
> > > maximal latency in certain scenarios.  
> > 
> > "maximum latency" also means "throughput improvement".
> > 
> 
> Under certain assumptions. But not necessarily. Again my concern is
> mostly the type of change. The virtio standard maintain a revision
> history appendix, and I would like to avoid the nature of this change
> being misrepresented there. If Connie and/or Michael think it is worth
> fixing, I believe it can be fixed with an editorial change.
> 
> AFAIU VIRTIO_NET_F_NOTF_COAL and VIRTIO_NET_F_VQ_NOTF_COAL are about to
> land with virtio-1.3, i.e. there is no released/standardized virtio
> version where the "initialize to 0" is released. In that sense it looks
> like we are still on time to change this. But I am not 100% certain. In
> any case I don't think this as a huge impact and I'm fine going ahead
> with the change.

Sorry for the late reply, I'm on vacation.

I agree with this, and I prefer to release this patch as a correction for
virtio1.3 instead of a new patch for virtio1.4, because if devices support moderation coalescing based on virtio1.3,
and after virtio1.4 is released, these devices need to be updated again for a
more reasonable coalescing parameters.

Cornelia and Michael, what do you think?

> 
> > > 
> > > I understand that it is probably in the best interest of the devices to
> > > not pick stupid defaults. But it is also probably in the best interest
> > > of the driver to set those params, and if the driver is going to set its
> > > values, the devices defaults are moot unless we assume that those may end
> > > up being used by the driver as a hint when deciding which parameter
> > > values to choose.  
> > 
> > "Any values" is compatible with "0 for max-usecs and max-frames", and the device
> > can choose "no coalescing".
> > 
> > "No coalescing" means "latency friendly", but it also means "a lot of
> > interruptions and throughput unfriendly". 
> 
> To what extent does virtio's "normal" notification suppression
> (VIRTIO_F_EVENT_IDX) would alleviate that in practice? At least in
> theory the interruption suppression could save us there right? 
> 

Interruption suppression is useful, but many scenes still have a lot of
interruptions.

> 
> > If the device chooses a stupid
> > maximum value, it is his choice (spec should give more devices choices instead of
> > forcing them to choose "0" which is not the best practice). We can't talk about
> > performance for drivers when the devices tend to choose any "stupid" designs.
> > 
> > We need relaxe the restrictions and makes the spec more reasonable.
> > 
> 
> Hm, I see Linux virtio-net changes have landed with v6.0 and if I read
> those correctly the driver -- contrary to my initial expectation --
> negotiates the feature, but does not set the parameters explicitly and

How does the driver know what parameters to set? The parameters should be
exposed by each device.

> thus keeps the defaults (until userspace decides to set the parameters).
> So it does matter whether the defaults are guaranteed to be 0 or not,
> and if not it does matter what defaults are chosen by the device.

Didn't follow this. More below.

> 
> One could even argue that those patches have been reviewed under the
> assumption that the device needs to use 0 as the default parameter value.

The default value should not be explicitly specified in the spec, because one
size does not fit multiple devices. The source of this problem is that we are
missing fields like default_{rx, tx}_coalesicng_params that indicate the device
capabilities. No?

Thanks.

> 
> Well no strong opinions here. If the community is fine with it, I'm fine
> with it as well.
> 
> Thank you!
> 
> Regards,
> Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Mon, Jun 10, 2024 at 09:35:45PM +0800, Heng Qi wrote:
> On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> > On Sat, 8 Jun 2024 10:34:22 +0800
> > Heng Qi <hengqi@linux.alibaba.com> wrote:
> > 
> > > On Fri, 7 Jun 2024 22:02:46 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> > > > On Tue, 28 May 2024 12:47:02 +0800
> > > > Heng Qi <hengqi@linux.alibaba.com> wrote:
> > > >   
> > > > > The device can set any initial coalescing parameters (0 or non-zero)
> > > > > for the receive/send queue before the setting command is executed,
> > > > > not just 0, enhancing device performance even without DIM enabled.
> > > > > 
> > > > > So we need to clarify descriptions that don't fit the behavior.  
> > > > 
> > > > Sorry I'm late to the party -- again! Just for my understanding: how/why
> > > > is this a clarification and not just a (basically incompatible) change?  
> > > 
> > > In my opinion, "clarification" means that something may have been described
> > > incorrectly before, and we now need to discuss, explain clearly, and correct
> > > the possibly incorrect description.
> > > 
> > 
> > I figure the difference in perceived semantics of the word
> > "clarification" is at the root of my confusion. Let us have a look at
> > https://dictionary.cambridge.org/de/worterbuch/englisch/clarification
> > 
> > According to my understanding a "clarification", while an improvement in
> > ease of understanding and/or decrease of ambiguity (possibly to no
> > ambiguity at all) implies that what receiving a clarification is not
> > outright wrong.
> > 
> > When rectifying something that is outright incorrect or wrong, I would
> > refer to that with words like "correction", "fix", "erratum" or
> > "corrigendum".
> > 
> > > > 
> > > > I mean if I read this correctly, before the driver had the guaranty
> > > > that if the parameters are not set by the driver, negotiating the
> > > > feature does not introduce any coalescing. After this in theory
> > > > the device could just pick some max value and potentially introduce
> > > > maximal latency in certain scenarios.  
> > > 
> > > "maximum latency" also means "throughput improvement".
> > > 
> > 
> > Under certain assumptions. But not necessarily. Again my concern is
> > mostly the type of change. The virtio standard maintain a revision
> > history appendix, and I would like to avoid the nature of this change
> > being misrepresented there. If Connie and/or Michael think it is worth
> > fixing, I believe it can be fixed with an editorial change.
> > 
> > AFAIU VIRTIO_NET_F_NOTF_COAL and VIRTIO_NET_F_VQ_NOTF_COAL are about to
> > land with virtio-1.3, i.e. there is no released/standardized virtio
> > version where the "initialize to 0" is released. In that sense it looks
> > like we are still on time to change this. But I am not 100% certain. In
> > any case I don't think this as a huge impact and I'm fine going ahead
> > with the change.
> 
> Sorry for the late reply, I'm on vacation.
> 
> I agree with this, and I prefer to release this patch as a correction for
> virtio1.3 instead of a new patch for virtio1.4, because if devices support moderation coalescing based on virtio1.3,
> and after virtio1.4 is released, these devices need to be updated again for a
> more reasonable coalescing parameters.
> 
> Cornelia and Michael, what do you think?

The TC is just voting to start the public review process.
We can defer that by a couple more weeks if there is
a known issue to address.

For that I expect we want a final patch  and a couple of acks
on list from TC members by end of the ballot, June 13.



> > 
> > > > 
> > > > I understand that it is probably in the best interest of the devices to
> > > > not pick stupid defaults. But it is also probably in the best interest
> > > > of the driver to set those params, and if the driver is going to set its
> > > > values, the devices defaults are moot unless we assume that those may end
> > > > up being used by the driver as a hint when deciding which parameter
> > > > values to choose.  
> > > 
> > > "Any values" is compatible with "0 for max-usecs and max-frames", and the device
> > > can choose "no coalescing".
> > > 
> > > "No coalescing" means "latency friendly", but it also means "a lot of
> > > interruptions and throughput unfriendly". 
> > 
> > To what extent does virtio's "normal" notification suppression
> > (VIRTIO_F_EVENT_IDX) would alleviate that in practice? At least in
> > theory the interruption suppression could save us there right? 
> > 
> 
> Interruption suppression is useful, but many scenes still have a lot of
> interruptions.
> 
> > 
> > > If the device chooses a stupid
> > > maximum value, it is his choice (spec should give more devices choices instead of
> > > forcing them to choose "0" which is not the best practice). We can't talk about
> > > performance for drivers when the devices tend to choose any "stupid" designs.
> > > 
> > > We need relaxe the restrictions and makes the spec more reasonable.
> > > 
> > 
> > Hm, I see Linux virtio-net changes have landed with v6.0 and if I read
> > those correctly the driver -- contrary to my initial expectation --
> > negotiates the feature, but does not set the parameters explicitly and
> 
> How does the driver know what parameters to set? The parameters should be
> exposed by each device.
> 
> > thus keeps the defaults (until userspace decides to set the parameters).
> > So it does matter whether the defaults are guaranteed to be 0 or not,
> > and if not it does matter what defaults are chosen by the device.
> 
> Didn't follow this. More below.
> 
> > 
> > One could even argue that those patches have been reviewed under the
> > assumption that the device needs to use 0 as the default parameter value.
> 
> The default value should not be explicitly specified in the spec, because one
> size does not fit multiple devices. The source of this problem is that we are
> missing fields like default_{rx, tx}_coalesicng_params that indicate the device
> capabilities. No?
> 
> Thanks.
> 
> > 
> > Well no strong opinions here. If the community is fine with it, I'm fine
> > with it as well.
> > 
> > Thank you!
> > 
> > Regards,
> > Halil

RE: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, June 10, 2024 8:20 PM
> 
> On Mon, Jun 10, 2024 at 09:35:45PM +0800, Heng Qi wrote:
> > On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <pasic@linux.ibm.com>
> wrote:
> > > On Sat, 8 Jun 2024 10:34:22 +0800
> > > Heng Qi <hengqi@linux.alibaba.com> wrote:
> > >
> > > > On Fri, 7 Jun 2024 22:02:46 +0200, Halil Pasic <pasic@linux.ibm.com>
> wrote:
> > > > > On Tue, 28 May 2024 12:47:02 +0800 Heng Qi
> > > > > <hengqi@linux.alibaba.com> wrote:
> > > > >
> > > > > > The device can set any initial coalescing parameters (0 or
> > > > > > non-zero) for the receive/send queue before the setting
> > > > > > command is executed, not just 0, enhancing device performance even
> without DIM enabled.
> > > > > >
> > > > > > So we need to clarify descriptions that don't fit the behavior.
> > > > >
> > > > > Sorry I'm late to the party -- again! Just for my understanding:
> > > > > how/why is this a clarification and not just a (basically incompatible)
> change?
> > > >
> > > > In my opinion, "clarification" means that something may have been
> > > > described incorrectly before, and we now need to discuss, explain
> > > > clearly, and correct the possibly incorrect description.
> > > >
> > >
> > > I figure the difference in perceived semantics of the word
> > > "clarification" is at the root of my confusion. Let us have a look
> > > at
> > > https://dictionary.cambridge.org/de/worterbuch/englisch/clarificatio
> > > n
> > >
> > > According to my understanding a "clarification", while an
> > > improvement in ease of understanding and/or decrease of ambiguity
> > > (possibly to no ambiguity at all) implies that what receiving a
> > > clarification is not outright wrong.
> > >
> > > When rectifying something that is outright incorrect or wrong, I
> > > would refer to that with words like "correction", "fix", "erratum"
> > > or "corrigendum".
> > >
> > > > >
> > > > > I mean if I read this correctly, before the driver had the
> > > > > guaranty that if the parameters are not set by the driver,
> > > > > negotiating the feature does not introduce any coalescing. After
> > > > > this in theory the device could just pick some max value and
> > > > > potentially introduce maximal latency in certain scenarios.
> > > >
> > > > "maximum latency" also means "throughput improvement".
> > > >
> > >
> > > Under certain assumptions. But not necessarily. Again my concern is
> > > mostly the type of change. The virtio standard maintain a revision
> > > history appendix, and I would like to avoid the nature of this
> > > change being misrepresented there. If Connie and/or Michael think it
> > > is worth fixing, I believe it can be fixed with an editorial change.
> > >
> > > AFAIU VIRTIO_NET_F_NOTF_COAL and VIRTIO_NET_F_VQ_NOTF_COAL
> are about
> > > to land with virtio-1.3, i.e. there is no released/standardized
> > > virtio version where the "initialize to 0" is released. In that
> > > sense it looks like we are still on time to change this. But I am
> > > not 100% certain. In any case I don't think this as a huge impact
> > > and I'm fine going ahead with the change.
> >
> > Sorry for the late reply, I'm on vacation.
> >
> > I agree with this, and I prefer to release this patch as a correction
> > for
> > virtio1.3 instead of a new patch for virtio1.4, because if devices
> > support moderation coalescing based on virtio1.3, and after virtio1.4
> > is released, these devices need to be updated again for a more reasonable
> coalescing parameters.

Heng's patch is bringing only the relaxation to the device that it can have nonzero value on reset.

I don't have strong opinion either way.
Either approach looks fine to me as it does not seem to break any functionality for driver and device.
It only improves the performance when driver has not set them.

'Clarify' to 'correct' or 'fix' subject line would have been obviously better as Pasic Halil mentioned, but probably while merging the patch, it can be corrected.

> >
> > Cornelia and Michael, what do you think?
> 
> The TC is just voting to start the public review process.
> We can defer that by a couple more weeks if there is a known issue to address.
> 
> For that I expect we want a final patch  and a couple of acks on list from TC
> members by end of the ballot, June 13.
> 
> 
> 
> > >
> > > > >
> > > > > I understand that it is probably in the best interest of the
> > > > > devices to not pick stupid defaults. But it is also probably in
> > > > > the best interest of the driver to set those params, and if the
> > > > > driver is going to set its values, the devices defaults are moot
> > > > > unless we assume that those may end up being used by the driver
> > > > > as a hint when deciding which parameter values to choose.
> > > >
> > > > "Any values" is compatible with "0 for max-usecs and max-frames",
> > > > and the device can choose "no coalescing".
> > > >
> > > > "No coalescing" means "latency friendly", but it also means "a lot
> > > > of interruptions and throughput unfriendly".
> > >
> > > To what extent does virtio's "normal" notification suppression
> > > (VIRTIO_F_EVENT_IDX) would alleviate that in practice? At least in
> > > theory the interruption suppression could save us there right?
> > >
> >
> > Interruption suppression is useful, but many scenes still have a lot
> > of interruptions.
> >
> > >
> > > > If the device chooses a stupid
> > > > maximum value, it is his choice (spec should give more devices
> > > > choices instead of forcing them to choose "0" which is not the
> > > > best practice). We can't talk about performance for drivers when the
> devices tend to choose any "stupid" designs.
> > > >
> > > > We need relaxe the restrictions and makes the spec more reasonable.
> > > >
> > >
> > > Hm, I see Linux virtio-net changes have landed with v6.0 and if I
> > > read those correctly the driver -- contrary to my initial
> > > expectation -- negotiates the feature, but does not set the
> > > parameters explicitly and
> >
> > How does the driver know what parameters to set? The parameters should
> > be exposed by each device.
> >
> > > thus keeps the defaults (until userspace decides to set the parameters).
> > > So it does matter whether the defaults are guaranteed to be 0 or
> > > not, and if not it does matter what defaults are chosen by the device.
> >
> > Didn't follow this. More below.
> >
> > >
> > > One could even argue that those patches have been reviewed under the
> > > assumption that the device needs to use 0 as the default parameter value.
> >
> > The default value should not be explicitly specified in the spec,
> > because one size does not fit multiple devices. The source of this
> > problem is that we are missing fields like default_{rx,
> > tx}_coalesicng_params that indicate the device capabilities. No?
> >
> > Thanks.
> >
> > >
> > > Well no strong opinions here. If the community is fine with it, I'm
> > > fine with it as well.
> > >
> > > Thank you!
> > >
> > > Regards,
> > > Halil
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Halil Pasic]

On Mon, 10 Jun 2024 21:35:45 +0800
Heng Qi <hengqi@linux.alibaba.com> wrote:

> On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
[..]

> > > If the device chooses a stupid
> > > maximum value, it is his choice (spec should give more devices choices instead of
> > > forcing them to choose "0" which is not the best practice). We can't talk about
> > > performance for drivers when the devices tend to choose any "stupid" designs.
> > > 
> > > We need relaxe the restrictions and makes the spec more reasonable.
> > >   
> > 
> > Hm, I see Linux virtio-net changes have landed with v6.0 and if I read
> > those correctly the driver -- contrary to my initial expectation --
> > negotiates the feature, but does not set the parameters explicitly and  
> 
> How does the driver know what parameters to set? 

Disclaimer: I'm not very knowledgeable when it comes to networking and
NICs. Please be patient with me.

If I understood that properly have previously stated that it is basically
a trade-off between "latency friendly"  (downside: overhead and not so
throughput friendly) and "throughput friendly" (downside: not so latency
friendly). And that makes sense to me.

So I would think, the answer to the question what is the best trade-off
should also depend on the workload.

Now as far as I understand although we call the parameters max_usecs and 
max_packets the notification condition is dictated by those two values. I.e.
there won't be a notification unless the compound condition is met.

When there is no traffic 0 and 0 looks like reasonable values to me: I
want the first of possibly many packets ASAP. Depending on the actual
load, maybe one could employ some sort of a heuristics to keep good
balance -- maybe based on a frequency of interrupts. Maybe DIM is
actually exactly what I have in mind.

You seem more knowledgeable on the topic. How is this usually done? How
are the optimal values correlated with device characteristics?

> The parameters should be
> exposed by each device.

I would like to better understand why. My intention is not to question
the correctness of this statement, but to gain a better understanding.

> 
> > thus keeps the defaults (until userspace decides to set the parameters).
> > So it does matter whether the defaults are guaranteed to be 0 or not,
> > and if not it does matter what defaults are chosen by the device.  
> 
> Didn't follow this. More below.
>

Maybe my statement was wrong. So let me make a question out of it. What
entity or entities do we expect to change the parameter values, and when
or under which conditions do we expect them to change the parameter
values?
 
> > 
> > One could even argue that those patches have been reviewed under the
> > assumption that the device needs to use 0 as the default parameter value.  
> 
> The default value should not be explicitly specified in the spec, because one
> size does not fit multiple devices.

This ties in to my previous question about the relationship between
device characteristics and the optimal values for max_usecs and
max_packets.

> The source of this problem is that we are
> missing fields like default_{rx, tx}_coalesicng_params that indicate the device
> capabilities. No?

I don't think so! In my understanding is that with your proposal
after a reset, the OS/driver has no problem obtaining the defaults that
indicate the device capabilities. But I would really like to better
understand that device capabilities part.

And yes another way around this would have been say:
* let us introduce those defaults fields, e.g. to the config space
* make the driver read those values as a part of the initialization
* and set those values as the initial parameters.
But frankly I see no benefit of that over what you propose here.

Regards,
Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Mon, 10 Jun 2024 22:19:00 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> On Mon, 10 Jun 2024 21:35:45 +0800
> Heng Qi <hengqi@linux.alibaba.com> wrote:
> 
> > On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> [..]
> 
> > > > If the device chooses a stupid
> > > > maximum value, it is his choice (spec should give more devices choices instead of
> > > > forcing them to choose "0" which is not the best practice). We can't talk about
> > > > performance for drivers when the devices tend to choose any "stupid" designs.
> > > > 
> > > > We need relaxe the restrictions and makes the spec more reasonable.
> > > >   
> > > 
> > > Hm, I see Linux virtio-net changes have landed with v6.0 and if I read
> > > those correctly the driver -- contrary to my initial expectation --
> > > negotiates the feature, but does not set the parameters explicitly and  
> > 
> > How does the driver know what parameters to set? 
> 
> Disclaimer: I'm not very knowledgeable when it comes to networking and
> NICs. Please be patient with me.
> 
> If I understood that properly have previously stated that it is basically
> a trade-off between "latency friendly"  (downside: overhead and not so
> throughput friendly) and "throughput friendly" (downside: not so latency
> friendly). And that makes sense to me.
> 

Yes, I am correcting the description about the trade-off thing.

> So I would think, the answer to the question what is the best trade-off
> should also depend on the workload.

device-side dim or driver-side dim to solve this problem. But, when the device
is reset, the driver may not enable dim, the device or driver needs to  have a
static coalescing parameters (0 or non-zero) for the trade-off.

(don't trust dim too much now, I'm doing some updates to optimize dim.)


> 
> Now as far as I understand although we call the parameters max_usecs and 
> max_packets the notification condition is dictated by those two values. I.e.
> there won't be a notification unless the compound condition is met.

Right, just one of the conditions needs to be met.

> 
> When there is no traffic 0 and 0 looks like reasonable values to me: I
> want the first of possibly many packets ASAP. Depending on the actual
> load, maybe one could employ some sort of a heuristics to keep good
> balance -- maybe based on a frequency of interrupts. Maybe DIM is
> actually exactly what I have in mind.

This patch is not to solve the scenario where dim exists, but what the
static value of the coalescing parameter is when the device is reset.
(please check some hhistorical discussion of this topic)

> 
> You seem more knowledgeable on the topic. How is this usually done? How
> are the optimal values correlated with device characteristics?


NICs such as mlx, ena, ice, and HiSilicon all have a non-zero static
coalescing value. Although they all support netdim, the static value is
still useful when dim is disabled.

As for the specific value, I think it is best to notify the driver through the
device itself, such as adding capability fileds or this:
https://lore.kernel.org/all/20240426065441.120710-3-hengqi@linux.alibaba.com/

> 
> > The parameters should be
> > exposed by each device.
> 
> I would like to better understand why. My intention is not to question
> the correctness of this statement, but to gain a better understanding.
> 
> > 
> > > thus keeps the defaults (until userspace decides to set the parameters).
> > > So it does matter whether the defaults are guaranteed to be 0 or not,
> > > and if not it does matter what defaults are chosen by the device.  
> > 
> > Didn't follow this. More below.
> >
> 
> Maybe my statement was wrong. So let me make a question out of it. What
> entity or entities do we expect to change the parameter values, and when
> or under which conditions do we expect them to change the parameter
> values?

Changed by the driver when the load changes or user..

>  
> > > 
> > > One could even argue that those patches have been reviewed under the
> > > assumption that the device needs to use 0 as the default parameter value.  
> > 
> > The default value should not be explicitly specified in the spec, because one
> > size does not fit multiple devices.
> 
> This ties in to my previous question about the relationship between
> device characteristics and the optimal values for max_usecs and
> max_packets.

Device characteristics and optimal max_usecs/frames are preferably communicated
via device capability, but at least should not be forced to a specific value by
the driver.

> 
> > The source of this problem is that we are
> > missing fields like default_{rx, tx}_coalesicng_params that indicate the device
> > capabilities. No?
> 
> I don't think so! In my understanding is that with your proposal
> after a reset, the OS/driver has no problem obtaining the defaults that
> indicate the device capabilities. But I would really like to better
> understand that device capabilities part.
> 
> And yes another way around this would have been say:
> * let us introduce those defaults fields, e.g. to the config space

Do you want to solve it by adding another new feature?
Otherwise, how do you solve the compatibility problem?

> * make the driver read those values as a part of the initialization

The spec is in conflict with this, because the driver can only read 0 from
devices that conform to the spec. I submitted this patch to achieve this goal:
https://lore.kernel.org/all/20240426065441.120710-3-hengqi@linux.alibaba.com/

> * and set those values as the initial parameters.

I don't support this, the device doesn't want to accept any initial
unreasonable settings.

Thanks.

> But frankly I see no benefit of that over what you propose here.
> 
> Regards,
> Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Cornelia Huck]

On Mon, Jun 10 2024, "Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Mon, Jun 10, 2024 at 09:35:45PM +0800, Heng Qi wrote:
>> On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
>> > On Sat, 8 Jun 2024 10:34:22 +0800
>> > Heng Qi <hengqi@linux.alibaba.com> wrote:
>> > 
>> > > On Fri, 7 Jun 2024 22:02:46 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
>> > > > On Tue, 28 May 2024 12:47:02 +0800
>> > > > Heng Qi <hengqi@linux.alibaba.com> wrote:
>> > > >   
>> > > > > The device can set any initial coalescing parameters (0 or non-zero)
>> > > > > for the receive/send queue before the setting command is executed,
>> > > > > not just 0, enhancing device performance even without DIM enabled.
>> > > > > 
>> > > > > So we need to clarify descriptions that don't fit the behavior.  
>> > > > 
>> > > > Sorry I'm late to the party -- again! Just for my understanding: how/why
>> > > > is this a clarification and not just a (basically incompatible) change?  
>> > > 
>> > > In my opinion, "clarification" means that something may have been described
>> > > incorrectly before, and we now need to discuss, explain clearly, and correct
>> > > the possibly incorrect description.
>> > > 
>> > 
>> > I figure the difference in perceived semantics of the word
>> > "clarification" is at the root of my confusion. Let us have a look at
>> > https://dictionary.cambridge.org/de/worterbuch/englisch/clarification
>> > 
>> > According to my understanding a "clarification", while an improvement in
>> > ease of understanding and/or decrease of ambiguity (possibly to no
>> > ambiguity at all) implies that what receiving a clarification is not
>> > outright wrong.
>> > 
>> > When rectifying something that is outright incorrect or wrong, I would
>> > refer to that with words like "correction", "fix", "erratum" or
>> > "corrigendum".
>> > 
>> > > > 
>> > > > I mean if I read this correctly, before the driver had the guaranty
>> > > > that if the parameters are not set by the driver, negotiating the
>> > > > feature does not introduce any coalescing. After this in theory
>> > > > the device could just pick some max value and potentially introduce
>> > > > maximal latency in certain scenarios.  
>> > > 
>> > > "maximum latency" also means "throughput improvement".
>> > > 
>> > 
>> > Under certain assumptions. But not necessarily. Again my concern is
>> > mostly the type of change. The virtio standard maintain a revision
>> > history appendix, and I would like to avoid the nature of this change
>> > being misrepresented there. If Connie and/or Michael think it is worth
>> > fixing, I believe it can be fixed with an editorial change.
>> > 
>> > AFAIU VIRTIO_NET_F_NOTF_COAL and VIRTIO_NET_F_VQ_NOTF_COAL are about to
>> > land with virtio-1.3, i.e. there is no released/standardized virtio
>> > version where the "initialize to 0" is released. In that sense it looks
>> > like we are still on time to change this. But I am not 100% certain. In
>> > any case I don't think this as a huge impact and I'm fine going ahead
>> > with the change.
>> 
>> Sorry for the late reply, I'm on vacation.
>> 
>> I agree with this, and I prefer to release this patch as a correction for
>> virtio1.3 instead of a new patch for virtio1.4, because if devices support moderation coalescing based on virtio1.3,
>> and after virtio1.4 is released, these devices need to be updated again for a
>> more reasonable coalescing parameters.
>> 
>> Cornelia and Michael, what do you think?
>
> The TC is just voting to start the public review process.
> We can defer that by a couple more weeks if there is
> a known issue to address.
>
> For that I expect we want a final patch  and a couple of acks
> on list from TC members by end of the ballot, June 13.

FTR, I'd be fine with that if there's a consensus, but I'll not be
around to handle this personally. I also have no objection to proceeding
with the process.

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Tue, Jun 11, 2024 at 06:40:33PM +0800, Heng Qi wrote:
> On Mon, 10 Jun 2024 22:19:00 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> > On Mon, 10 Jun 2024 21:35:45 +0800
> > Heng Qi <hengqi@linux.alibaba.com> wrote:
> > 
> > > On Mon, 10 Jun 2024 14:46:02 +0200, Halil Pasic <pasic@linux.ibm.com> wrote:
> > [..]
> > 
> > > > > If the device chooses a stupid
> > > > > maximum value, it is his choice (spec should give more devices choices instead of
> > > > > forcing them to choose "0" which is not the best practice). We can't talk about
> > > > > performance for drivers when the devices tend to choose any "stupid" designs.
> > > > > 
> > > > > We need relaxe the restrictions and makes the spec more reasonable.
> > > > >   
> > > > 
> > > > Hm, I see Linux virtio-net changes have landed with v6.0 and if I read
> > > > those correctly the driver -- contrary to my initial expectation --
> > > > negotiates the feature, but does not set the parameters explicitly and  
> > > 
> > > How does the driver know what parameters to set? 
> > 
> > Disclaimer: I'm not very knowledgeable when it comes to networking and
> > NICs. Please be patient with me.
> > 
> > If I understood that properly have previously stated that it is basically
> > a trade-off between "latency friendly"  (downside: overhead and not so
> > throughput friendly) and "throughput friendly" (downside: not so latency
> > friendly). And that makes sense to me.
> > 
> 
> Yes, I am correcting the description about the trade-off thing.

It does not look like this thing is ready yet.


How we can we make progress with
the realease but sure we don't make backwards compat a pain?

Ideas?



> > So I would think, the answer to the question what is the best trade-off
> > should also depend on the workload.
> 
> device-side dim or driver-side dim to solve this problem. But, when the device
> is reset, the driver may not enable dim, the device or driver needs to  have a
> static coalescing parameters (0 or non-zero) for the trade-off.
> 
> (don't trust dim too much now, I'm doing some updates to optimize dim.)
> 
> 
> > 
> > Now as far as I understand although we call the parameters max_usecs and 
> > max_packets the notification condition is dictated by those two values. I.e.
> > there won't be a notification unless the compound condition is met.
> 
> Right, just one of the conditions needs to be met.
> 
> > 
> > When there is no traffic 0 and 0 looks like reasonable values to me: I
> > want the first of possibly many packets ASAP. Depending on the actual
> > load, maybe one could employ some sort of a heuristics to keep good
> > balance -- maybe based on a frequency of interrupts. Maybe DIM is
> > actually exactly what I have in mind.
> 
> This patch is not to solve the scenario where dim exists, but what the
> static value of the coalescing parameter is when the device is reset.
> (please check some hhistorical discussion of this topic)
> 
> > 
> > You seem more knowledgeable on the topic. How is this usually done? How
> > are the optimal values correlated with device characteristics?
> 
> 
> NICs such as mlx, ena, ice, and HiSilicon all have a non-zero static
> coalescing value. Although they all support netdim, the static value is
> still useful when dim is disabled.
> 
> As for the specific value, I think it is best to notify the driver through the
> device itself, such as adding capability fileds or this:
> https://lore.kernel.org/all/20240426065441.120710-3-hengqi@linux.alibaba.com/
> 
> > 
> > > The parameters should be
> > > exposed by each device.
> > 
> > I would like to better understand why. My intention is not to question
> > the correctness of this statement, but to gain a better understanding.
> > 
> > > 
> > > > thus keeps the defaults (until userspace decides to set the parameters).
> > > > So it does matter whether the defaults are guaranteed to be 0 or not,
> > > > and if not it does matter what defaults are chosen by the device.  
> > > 
> > > Didn't follow this. More below.
> > >
> > 
> > Maybe my statement was wrong. So let me make a question out of it. What
> > entity or entities do we expect to change the parameter values, and when
> > or under which conditions do we expect them to change the parameter
> > values?
> 
> Changed by the driver when the load changes or user..
> 
> >  
> > > > 
> > > > One could even argue that those patches have been reviewed under the
> > > > assumption that the device needs to use 0 as the default parameter value.  
> > > 
> > > The default value should not be explicitly specified in the spec, because one
> > > size does not fit multiple devices.
> > 
> > This ties in to my previous question about the relationship between
> > device characteristics and the optimal values for max_usecs and
> > max_packets.
> 
> Device characteristics and optimal max_usecs/frames are preferably communicated
> via device capability, but at least should not be forced to a specific value by
> the driver.
> 
> > 
> > > The source of this problem is that we are
> > > missing fields like default_{rx, tx}_coalesicng_params that indicate the device
> > > capabilities. No?
> > 
> > I don't think so! In my understanding is that with your proposal
> > after a reset, the OS/driver has no problem obtaining the defaults that
> > indicate the device capabilities. But I would really like to better
> > understand that device capabilities part.
> > 
> > And yes another way around this would have been say:
> > * let us introduce those defaults fields, e.g. to the config space
> 
> Do you want to solve it by adding another new feature?
> Otherwise, how do you solve the compatibility problem?
> 
> > * make the driver read those values as a part of the initialization
> 
> The spec is in conflict with this, because the driver can only read 0 from
> devices that conform to the spec. I submitted this patch to achieve this goal:
> https://lore.kernel.org/all/20240426065441.120710-3-hengqi@linux.alibaba.com/
> 
> > * and set those values as the initial parameters.
> 
> I don't support this, the device doesn't want to accept any initial
> unreasonable settings.
> 
> Thanks.
> 
> > But frankly I see no benefit of that over what you propose here.
> > 
> > Regards,
> > Halil

RE: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Tuesday, June 11, 2024 10:00 PM


> How we can we make progress with
> the realease but sure we don't make backwards compat a pain?
> 
> Ideas?
>
There is no functional break with this relaxation.
Device set some non-zero defaults and driver didn't modify them.... 
Anything broken? Unlikely.
And device/driver has better performance, is that a problem? Unlikely.

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Tue, May 28, 2024 at 12:47:02PM +0800, Heng Qi wrote:
> The device can set any initial coalescing parameters (0 or non-zero)
> for the receive/send queue before the setting command is executed,
> not just 0, enhancing device performance even without DIM enabled.
> 
> So we need to clarify descriptions that don't fit the behavior.
> 
> Fixes: https://github.com/oasis-tcs/virtio-spec/issues/194
> Suggested-by: Jason Wang <jasowang@redhat.com>
> Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> Reviewed-by: Parav Pandit <parav@nvidia.com>
> Reviewed-by: Jason Wang <jasowang@redhat.com>
> ---
> v4->v5:
>   - Add RB tags from Parav and Jason. Thanks!
> 
> v3->v4:
>   - Doesn't force the device to remember more stuff. @Parav
> 
> v2->v3:
>   - Clarify description to be more generic. @Parav
> 
> v1->v2:
>   - Update description. @Jason
> 
>  device-types/net/description.tex | 14 +++++++++++---
>  1 file changed, 11 insertions(+), 3 deletions(-)
> 
> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> index 61cce1f..00c1b36 100644
> --- a/device-types/net/description.tex
> +++ b/device-types/net/description.tex
> @@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
>  \end{enumerate}
>  
> +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated,
> +the device may apply any coalescing parameters to each transmit/receive virtqueue
> +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.

may outside of conformance section, not good.

> +
> +The driver can query the coalescing parameters of any enabled transmit/receive
> +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any
> +VIRTIO_NET_CTRL_NOTF_COAL set command is done.
> +

Vague. There are just 3 set commands ust list them.

>  The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
>  
>  If coalescing parameters are being set, the device applies the last coalescing parameters set for a

This needs more work but I guess we can skip the above part.

> @@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>  VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
>  
>  Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
> -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
> +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.
>  

If device does not initialize them then what?
Actually something like 
0 -> the default TX coalescing parameters chosen by the device?


>  Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values.
>  
>  The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
>  the device MAY generate notifications more or less frequently than specified.
>  
>  A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
>  
> -Upon reset, a device MUST initialize all coalescing parameters to 0.
> +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.

Confusing.
This seems to allow any vq to have a different set of parameters.
Now if driver calls VIRTIO_NET_CTRL_NOTF_COAL_RX_SET then what?


>  
>  \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}




>  
> -- 
> 2.32.0.3.g01195cf9f

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> 
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Tuesday, June 11, 2024 10:00 PM
> 
> 
> > How we can we make progress with
> > the realease but sure we don't make backwards compat a pain?
> > 
> > Ideas?
> >
> There is no functional break with this relaxation.
> Device set some non-zero defaults and driver didn't modify them.... 
> Anything broken? Unlikely.
> And device/driver has better performance, is that a problem? Unlikely.
> 

Yes, it is possible. Driver can cache values it sets
and never query device with get.
Before anything is set, driver will report incorrect values.
What will break as a result? Hard to predict.


Given the patch is big, I am inclined to say it just should use
a feature bit.

-- 
MST

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> > 
> > 
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Tuesday, June 11, 2024 10:00 PM
> > 
> > 
> > > How we can we make progress with
> > > the realease but sure we don't make backwards compat a pain?
> > > 
> > > Ideas?
> > >
> > There is no functional break with this relaxation.
> > Device set some non-zero defaults and driver didn't modify them.... 
> > Anything broken? Unlikely.
> > And device/driver has better performance, is that a problem? Unlikely.
> > 

Sorry, my vacation just ended.

> 
> Yes, it is possible. Driver can cache values it sets
> and never query device with get.

Don't we already have a lot of behaviors to drive queries from devices?
RSS context, device stats.

> Before anything is set, driver will report incorrect values.

Devices that are widely supported and supported by good practices should have
any initialization value. Just reporting 0 is incorrect value. Although the
spec now says so.

> What will break as a result? Hard to predict.
> 
> 
> Given the patch is big, I am inclined to say it just should use
> a feature bit.
> 

This doesn't seem to break anything in my opinion, we just told the device that
now you can set more values.

Using new feature bits does not seem necessary.

Thanks.

> -- 
> MST
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Tue, 11 Jun 2024 19:03:11 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Tue, May 28, 2024 at 12:47:02PM +0800, Heng Qi wrote:
> > The device can set any initial coalescing parameters (0 or non-zero)
> > for the receive/send queue before the setting command is executed,
> > not just 0, enhancing device performance even without DIM enabled.
> > 
> > So we need to clarify descriptions that don't fit the behavior.
> > 
> > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/194
> > Suggested-by: Jason Wang <jasowang@redhat.com>
> > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> > Reviewed-by: Parav Pandit <parav@nvidia.com>
> > Reviewed-by: Jason Wang <jasowang@redhat.com>
> > ---
> > v4->v5:
> >   - Add RB tags from Parav and Jason. Thanks!
> > 
> > v3->v4:
> >   - Doesn't force the device to remember more stuff. @Parav
> > 
> > v2->v3:
> >   - Clarify description to be more generic. @Parav
> > 
> > v1->v2:
> >   - Update description. @Jason
> > 
> >  device-types/net/description.tex | 14 +++++++++++---
> >  1 file changed, 11 insertions(+), 3 deletions(-)
> > 
> > diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> > index 61cce1f..00c1b36 100644
> > --- a/device-types/net/description.tex
> > +++ b/device-types/net/description.tex
> > @@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
> >  \end{enumerate}
> >  
> > +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated,
> > +the device may apply any coalescing parameters to each transmit/receive virtqueue
> > +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.
> 
> may outside of conformance section, not good.

This is outside of the conformance section. Do you want inside?

> 
> > +
> > +The driver can query the coalescing parameters of any enabled transmit/receive
> > +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any
> > +VIRTIO_NET_CTRL_NOTF_COAL set command is done.
> > +
> 
> Vague. There are just 3 set commands ust list them.
> 
> >  The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> >  
> >  If coalescing parameters are being set, the device applies the last coalescing parameters set for a
> 
> This needs more work but I guess we can skip the above part.
> 
> > @@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >  VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
> >  
> >  Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
> > -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
> > +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.
> >  
> 
> If device does not initialize them then what?

The device's customized behavior, at least, does not conform to the spec.

> Actually something like 
> 0 -> the default TX coalescing parameters chosen by the device?

Deefault coalescing parameters should also have a concrete "value".

> 
> 
> >  Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> > -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> > +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values.
> >  
> >  The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> >  the device MAY generate notifications more or less frequently than specified.
> >  
> >  A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> >  
> > -Upon reset, a device MUST initialize all coalescing parameters to 0.
> > +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.
> 
> Confusing.
> This seems to allow any vq to have a different set of parameters.

YES.

> Now if driver calls VIRTIO_NET_CTRL_NOTF_COAL_RX_SET then what?
> 

Then the coalescing parameters of all RX queues are updated to the values
set by VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, and the coalescing parameters of all
TX queues continue to retain the previous values.

Thanks.

> 
> >  
> >  \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}
> 
> 
> 
> 
> >  
> > -- 
> > 2.32.0.3.g01195cf9f
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/16/2024 7:27 PM, Heng Qi wrote:
> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
>>>
>>>> From: Michael S. Tsirkin <mst@redhat.com>
>>>> Sent: Tuesday, June 11, 2024 10:00 PM
>>>
>>>> How we can we make progress with
>>>> the realease but sure we don't make backwards compat a pain?
>>>>
>>>> Ideas?
>>>>
>>> There is no functional break with this relaxation.
>>> Device set some non-zero defaults and driver didn't modify them....
>>> Anything broken? Unlikely.
Generally it's inappropriate to leave this decision making to the device 
for what would be the best / most performant default config, as the 
device is generally considered agnostic to the guest load... Unless the 
device is specially hard wired to some fixed guest setup that users 
couldn't change, it doesn't seem logical that the device could derive 
the best or most performant config on driver's behalf. What if the guest 
wants best latency for its load but the device just blindlessly guess 
the guest might prefer throughput friendly that it miserably uses 
latency impacting non-zero default? Could this device side change for 
the default config regress boot time performance (which may need best 
latency over throughput)?

>>> And device/driver has better performance, is that a problem? Unlikely.
>>>
Even for rare case with a hard wired setup, the way to tackle the very 
problem using device's default is still quite questionable. Usually the 
mgmt software or network config utility should be equipped with some 
default value if need be. And we know the guest has the best position to 
impose the best / most performant config for its own load. What is the 
issue or use case that this initial config couldn't be applied by the 
guest mgmt software ahead but has to resort to the device to load some 
default (which is odd and irrelevant to any guest load), before the 
interface is brought up for operation i.e. performing I/O?

> Sorry, my vacation just ended.
>
>> Yes, it is possible. Driver can cache values it sets
>> and never query device with get.
> Don't we already have a lot of behaviors to drive queries from devices?
> RSS context, device stats.
>
>> Before anything is set, driver will report incorrect values.
> Devices that are widely supported and supported by good practices should have
> any initialization value. Just reporting 0 is incorrect value. Although the
> spec now says so.
I don't have an aligned view here, sorry. As I recall having 0 as the 
default is just to keep device started in a state where coalescing is 
disabled, so it's backward compatible and consistent with a 
non-coalescing supporting driver - such that it won't yield surprising 
effect (for e.g. regressed latency) inadvertently after user's getting 
driver software upgraded. Unlike the other virtio-net features that 
could 100% improve performance in all aspect, this coalescing feature is 
more of a performance tuning knob that may improve performance metrics 
(such as cpu usage or throughput) of one dimension while demoting the 
others (such as latency, jitter or connection rate) from the equation. 
That said, there's not a single and fixed set of default config that 
device could supply which is able to satisfy all kind of guest load. 
Rather than rely on the device to offer a matching default for driver 
(which I think it's technically wrong), I'd lean to having guest 
software or network utility to apply the initial config for the guest, 
where they should have best knowledge for the specific guest workload 
than what device could do.

>
>> What will break as a result? Hard to predict.
>>
>>
>> Given the patch is big, I am inclined to say it just should use
>> a feature bit.
>>
> This doesn't seem to break anything in my opinion, we just told the device that
> now you can set more values.
Another thing this could break is live migration between devices with 
varied default. How do you make sure the guest doesn't rely on some 
default from the source device, while on the destination it just doesn't 
get the same default coalescing value? To get rid of this side effect 
the guest would still need to apply the initial config for its own, 
anyway... Which eventually would render this proposal with arbitrary 
default rather pointless.


Thanks,
-Siwei
>
> Using new feature bits does not seem necessary.
>
> Thanks.
>
>> -- 
>> MST
>>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> 
> 
> On 6/16/2024 7:27 PM, Heng Qi wrote:
> > On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> >>>
> >>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>> Sent: Tuesday, June 11, 2024 10:00 PM
> >>>
> >>>> How we can we make progress with
> >>>> the realease but sure we don't make backwards compat a pain?
> >>>>
> >>>> Ideas?
> >>>>
> >>> There is no functional break with this relaxation.
> >>> Device set some non-zero defaults and driver didn't modify them....
> >>> Anything broken? Unlikely.
> Generally it's inappropriate to leave this decision making to the device 
> for what would be the best / most performant default config, as the 
> device is generally considered agnostic to the guest load...

Instead, the performance of the virtual machine and the driver depends heavily
on how the device is implemented, just as we have proposed various ways to
offload the data queue in the device to the hardware. The reason why most
devices use software to simulate ctrlq instead of using hardware offload is
that the driver has no requirements for the performance of ctrlq before, that is,
the device implementation is responsible for and meets the driver's performance
requirements.


>Unless the 
> device is specially hard wired to some fixed guest setup that users 
> couldn't change, it doesn't seem logical that the device could derive 
> the best or most performant config on driver's behalf. What if the guest 
> wants best latency for its load but the device just blindlessly guess 
> the guest might prefer throughput friendly that it miserably uses 
> latency impacting non-zero default? 

The device does not want to guess and cannot guess. This patch does not force
the device to choose a non-zero value, but relaxes it to allow the device to
choose 0 or non-zero, which is very friendly to virtual machines with different
performance requirements, right?

>Could this device side change for 
> the default config regress boot time performance (which may need best 
> latency over throughput)?

Don't make these assumptions, what if the driver needs better throughput?

> 
> >>> And device/driver has better performance, is that a problem? Unlikely.
> >>>
> Even for rare case with a hard wired setup, the way to tackle the very 
> problem using device's default is still quite questionable. Usually the 
> mgmt software or network config utility should be equipped with some 
> default value if need be. And we know the guest has the best position to 
> impose the best / most performant config for its own load. What is the 
> issue or use case that this initial config couldn't be applied by the 
> guest mgmt software ahead but has to resort to the device to load some 
> default (which is odd and irrelevant to any guest load), before the 
> interface is brought up for operation i.e. performing I/O?

Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards
have a default value that is not 0.
(0 is actually a kind of default value)

> 
> > Sorry, my vacation just ended.
> >
> >> Yes, it is possible. Driver can cache values it sets
> >> and never query device with get.
> > Don't we already have a lot of behaviors to drive queries from devices?
> > RSS context, device stats.
> >
> >> Before anything is set, driver will report incorrect values.
> > Devices that are widely supported and supported by good practices should have
> > any initialization value. Just reporting 0 is incorrect value. Although the
> > spec now says so.
> I don't have an aligned view here, sorry. As I recall having 0 as the 
> default is just to keep device started in a state where coalescing is 
> disabled, so it's backward compatible and consistent with a 
> non-coalescing supporting driver - such that it won't yield surprising 
> effect (for e.g. regressed latency) inadvertently after user's getting 
> driver software upgraded. Unlike the other virtio-net features that 
> could 100% improve performance in all aspect, this coalescing feature is 
> more of a performance tuning knob that may improve performance metrics 
> (such as cpu usage or throughput) of one dimension while demoting the 
> others (such as latency, jitter or connection rate) from the equation. 
> That said, there's not a single and fixed set of default config that 
> device could supply which is able to satisfy all kind of guest load. 
> Rather than rely on the device to offer a matching default for driver 
> (which I think it's technically wrong), I'd lean to having guest 
> software or network utility to apply the initial config for the guest, 
> where they should have best knowledge for the specific guest workload 
> than what device could do.

Before this feature, a good device implementation should also support coalescing
(of course we don't necessarily assume it has coalescing). In addition, virtual
machines that tend to favor latency and throughput exist. If the device supported
by the manufacturer needs to provide a low-latency virtual machine, please
continue to keep the default value of 0.

> 
> >
> >> What will break as a result? Hard to predict.
> >>
> >>
> >> Given the patch is big, I am inclined to say it just should use
> >> a feature bit.
> 
> > This doesn't seem to break anything in my opinion, we just told the device that
> > now you can set more values.
> Another thing this could break is live migration between devices with 
> varied default. How do you make sure the guest doesn't rely on some 
> default from the source device, while on the destination it just doesn't 
> get the same default coalescingjjj value? To get rid of this side effect 
> the guest would still need to apply the initial config for its own, 
> anyway... Which eventually would render this proposal with arbitrary 
> default rather pointless.

I don't quite understand why this would affect hot migration, the values
would be migrated over.

Thanks.

> 
> 
> Thanks,
> -Siwei
> >
> > Using new feature bits does not seem necessary.
> >
> > Thanks.
> >
> >> -- 
> >> MST
> >>
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/20/2024 12:40 AM, Heng Qi wrote:
> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
>>
>> On 6/16/2024 7:27 PM, Heng Qi wrote:
>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
>>>>>> From: Michael S. Tsirkin <mst@redhat.com>
>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
>>>>>> How we can we make progress with
>>>>>> the realease but sure we don't make backwards compat a pain?
>>>>>>
>>>>>> Ideas?
>>>>>>
>>>>> There is no functional break with this relaxation.
>>>>> Device set some non-zero defaults and driver didn't modify them....
>>>>> Anything broken? Unlikely.
>> Generally it's inappropriate to leave this decision making to the device
>> for what would be the best / most performant default config, as the
>> device is generally considered agnostic to the guest load...
> Instead, the performance of the virtual machine and the driver depends heavily
> on how the device is implemented, just as we have proposed various ways to
> offload the data queue in the device to the hardware. The reason why most
> devices use software to simulate ctrlq instead of using hardware offload is
> that the driver has no requirements for the performance of ctrlq before, that is,
> the device implementation is responsible for and meets the driver's performance
> requirements.
I am not sure I follow the arguments of ctrlq being s/w or h/w, I 
thought that it was for the debate why we need coalescing on hardware 
offload device in the first place, instead of reusing event index or 
similar s/w based notification suppression. I don't doubt the value of 
coalescing, but how's it relevant to the default value disposition?  
Generally the default config disposition is not considered to be part of 
device implementation, especially when it comes to the situation where 
device can't easily figure out the specific workload to occur in the 
guest, and there's no perfect single default value that could meet every 
single performance metric across the board. This is a typical tuning 
knob left up to the user to adjust, why the device or driver needs to 
set or load the initial value? The driver just needs to start with 
certain value, be it 0 or non-zero, which guest user can override at any 
point of time, depending on his/her need, and that's it! I guess I still 
don't understand your user case here, why device / driver default is of 
such importance.

>> Unless the
>> device is specially hard wired to some fixed guest setup that users
>> couldn't change, it doesn't seem logical that the device could derive
>> the best or most performant config on driver's behalf. What if the guest
>> wants best latency for its load but the device just blindlessly guess
>> the guest might prefer throughput friendly that it miserably uses
>> latency impacting non-zero default?
> The device does not want to guess and cannot guess. This patch does not force
> the device to choose a non-zero value, but relaxes it to allow the device to
> choose 0 or non-zero, which is very friendly to virtual machines with different
> performance requirements, right?
I don't understand the friendly part - do you imply your VM users are on 
kinda fixed wired setup that they cannot change these coalescing 
parameters after driver is loaded? Can the owner of the VM in control 
apply certain initial config for the coalescing parameters to the VM 
image? Or is it the problem of the guest driver that doesn't yet expose 
coalescing parameters to the end user? Otherwise I would think that 
guest user should be able to set parameters accordingly that would best 
fit the specific performance requirement of their own. How the device 
could even help here? I don't feel there's a lot of value to grant the 
device or host admin the flexibility to policy the *best* config on 
guest user's behalf, to be honest. And you seem to admit the fact that 
the default doesn't really matter, be it 0 or non-zero.

>
>> Could this device side change for
>> the default config regress boot time performance (which may need best
>> latency over throughput)?
> Don't make these assumptions, what if the driver needs better throughput?
There's a misconception here: what we think that driver may need in 
terms on performance does actually reflect what guest user would like to 
have. Driver cannot read guest user's mind to make the decision, either.

In history there was drastic change in the Linux virtio-net driver that 
ever changed the default disposition for XPS (and RPS as well) from 
throughput and long-lived connection oriented to concurrency and 
short-lived oriented, which regressed a lot of existing setups that 
expects sustained throughput and packet rate after kernel upgrade. 
Although the occurrence of such drastic change for default disposition 
is not so welcome (that is one of the reasons why I valued consistent 
initial value and back compatible  behavior), I don't see people yelling 
at virtio spec for less flexibility of offering the default disposition, 
given that the guest user can override the config any time with their 
own tooling or script, there's no problem at all for them to just set 
the corresponding config back explicitly to what it was before.

>
>>>>> And device/driver has better performance, is that a problem? Unlikely.
>>>>>
>> Even for rare case with a hard wired setup, the way to tackle the very
>> problem using device's default is still quite questionable. Usually the
>> mgmt software or network config utility should be equipped with some
>> default value if need be. And we know the guest has the best position to
>> impose the best / most performant config for its own load. What is the
>> issue or use case that this initial config couldn't be applied by the
>> guest mgmt software ahead but has to resort to the device to load some
>> default (which is odd and irrelevant to any guest load), before the
>> interface is brought up for operation i.e. performing I/O?
> Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards
> have a default value that is not 0.
You seems to be referencing your own setup basically, and the question 
is still left answered - why the initial config can't be done through 
the mgmt software or network config utility within the guest?

> (0 is actually a kind of default value)
>
>>> Sorry, my vacation just ended.
>>>
>>>> Yes, it is possible. Driver can cache values it sets
>>>> and never query device with get.
>>> Don't we already have a lot of behaviors to drive queries from devices?
>>> RSS context, device stats.
>>>
>>>> Before anything is set, driver will report incorrect values.
>>> Devices that are widely supported and supported by good practices should have
>>> any initialization value. Just reporting 0 is incorrect value. Although the
>>> spec now says so.
>> I don't have an aligned view here, sorry. As I recall having 0 as the
>> default is just to keep device started in a state where coalescing is
>> disabled, so it's backward compatible and consistent with a
>> non-coalescing supporting driver - such that it won't yield surprising
>> effect (for e.g. regressed latency) inadvertently after user's getting
>> driver software upgraded. Unlike the other virtio-net features that
>> could 100% improve performance in all aspect, this coalescing feature is
>> more of a performance tuning knob that may improve performance metrics
>> (such as cpu usage or throughput) of one dimension while demoting the
>> others (such as latency, jitter or connection rate) from the equation.
>> That said, there's not a single and fixed set of default config that
>> device could supply which is able to satisfy all kind of guest load.
>> Rather than rely on the device to offer a matching default for driver
>> (which I think it's technically wrong), I'd lean to having guest
>> software or network utility to apply the initial config for the guest,
>> where they should have best knowledge for the specific guest workload
>> than what device could do.
> Before this feature, a good device implementation should also support coalescing
> (of course we don't necessarily assume it has coalescing).
Again, I don't doubt the value of supporting coalescing.

>   In addition, virtual
> machines that tend to favor latency and throughput exist. If the device supported
> by the manufacturer needs to provide a low-latency virtual machine, please
> continue to keep the default value of 0.
No, that's not what I was asking. There's no such requirement for any 
vendor to provide a low-latency or high throughput VM. The more general 
use case is - the setup for real world workload might just be too 
complex that end users would prefer low-latency on some virtual NIC or 
even some specific queues, while the other queues of a virtual NIC, or 
reset virtual NIC might have very different dispositions. Due to the 
needs and dynamics of workload scaling up & down, they might have more 
or less queues or virtual NICs to configure, so these disposition would 
need to be readjusted at any point of time, for which there's no easy 
way for device to adapt to easily. The guest user should have best 
knowledge for the specific guest workload and setup than what device 
could/should offer.

>
>>>> What will break as a result? Hard to predict.
>>>>
>>>>
>>>> Given the patch is big, I am inclined to say it just should use
>>>> a feature bit.
>>> This doesn't seem to break anything in my opinion, we just told the device that
>>> now you can set more values.
>> Another thing this could break is live migration between devices with
>> varied default. How do you make sure the guest doesn't rely on some
>> default from the source device, while on the destination it just doesn't
>> get the same default coalescingjjj value? To get rid of this side effect
>> the guest would still need to apply the initial config for its own,
>> anyway... Which eventually would render this proposal with arbitrary
>> default rather pointless.
> I don't quite understand why this would affect hot migration, the values
> would be migrated over.
Then I don't see this has been discussed (wouldn't the initial value be 
part of virtio device state to migrate over?) in the thread or described 
in the proposed text. And your proposed per-queue coalescing parameters 
(default plus current value) would have to be described too as part of 
virtio-net device state, so that people don't misunderstand your proposal.

Thanks,
-Siwei

> Thanks.
>
>>
>> Thanks,
>> -Siwei
>>> Using new feature bits does not seem necessary.
>>>
>>> Thanks.
>>>
>>>> -- 
>>>> MST
>>>>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> 
> 
> On 6/20/2024 12:40 AM, Heng Qi wrote:
> > On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> >>
> >> On 6/16/2024 7:27 PM, Heng Qi wrote:
> >>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> >>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
> >>>>>> How we can we make progress with
> >>>>>> the realease but sure we don't make backwards compat a pain?
> >>>>>>
> >>>>>> Ideas?
> >>>>>>
> >>>>> There is no functional break with this relaxation.
> >>>>> Device set some non-zero defaults and driver didn't modify them....
> >>>>> Anything broken? Unlikely.
> >> Generally it's inappropriate to leave this decision making to the device
> >> for what would be the best / most performant default config, as the
> >> device is generally considered agnostic to the guest load...
> > Instead, the performance of the virtual machine and the driver depends heavily
> > on how the device is implemented, just as we have proposed various ways to
> > offload the data queue in the device to the hardware. The reason why most
> > devices use software to simulate ctrlq instead of using hardware offload is
> > that the driver has no requirements for the performance of ctrlq before, that is,
> > the device implementation is responsible for and meets the driver's performance
> > requirements.
> I am not sure I follow the arguments of ctrlq being s/w or h/w, I 
> thought that it was for the debate why we need coalescing on hardware 
> offload device in the first place, instead of reusing event index or 
> similar s/w based notification suppression. I don't doubt the value of 
> coalescing, but how's it relevant to the default value disposition?  
> Generally the default config disposition is not considered to be part of 
> device implementation, especially when it comes to the situation where 
> device can't easily figure out the specific workload to occur in the 
> guest, and there's no perfect single default value that could meet every 
> single performance metric across the board. This is a typical tuning 
> knob left up to the user to adjust, why the device or driver needs to 
> set or load the initial value? The driver just needs to start with 
> certain value, be it 0 or non-zero, which guest user can override at any 
> point of time, depending on his/her need, and that's it! I guess I still 
> don't understand your user case here, why device / driver default is of 
> such importance.

I've explained that, and I understand your argument is why default value is needed,
and users should be able to adjust them, right?

The default value is working when the user doesn't adjust them.
It's not practical to rely entirely on user adjustment, and for devices
that serve hundreds of thousands of customers, the device implementation
has to be comprehensive.

Thanks.

> 
> >> Unless the
> >> device is specially hard wired to some fixed guest setup that users
> >> couldn't change, it doesn't seem logical that the device could derive
> >> the best or most performant config on driver's behalf. What if the guest
> >> wants best latency for its load but the device just blindlessly guess
> >> the guest might prefer throughput friendly that it miserably uses
> >> latency impacting non-zero default?
> > The device does not want to guess and cannot guess. This patch does not force
> > the device to choose a non-zero value, but relaxes it to allow the device to
> > choose 0 or non-zero, which is very friendly to virtual machines with different
> > performance requirements, right?
> I don't understand the friendly part - do you imply your VM users are on 
> kinda fixed wired setup that they cannot change these coalescing 
> parameters after driver is loaded? Can the owner of the VM in control 
> apply certain initial config for the coalescing parameters to the VM 
> image? Or is it the problem of the guest driver that doesn't yet expose 
> coalescing parameters to the end user? Otherwise I would think that 
> guest user should be able to set parameters accordingly that would best 
> fit the specific performance requirement of their own. How the device 
> could even help here? I don't feel there's a lot of value to grant the 
> device or host admin the flexibility to policy the *best* config on 
> guest user's behalf, to be honest. And you seem to admit the fact that 
> the default doesn't really matter, be it 0 or non-zero.
> 
> >
> >> Could this device side change for
> >> the default config regress boot time performance (which may need best
> >> latency over throughput)?
> > Don't make these assumptions, what if the driver needs better throughput?
> There's a misconception here: what we think that driver may need in 
> terms on performance does actually reflect what guest user would like to 
> have. Driver cannot read guest user's mind to make the decision, either.
> 
> In history there was drastic change in the Linux virtio-net driver that 
> ever changed the default disposition for XPS (and RPS as well) from 
> throughput and long-lived connection oriented to concurrency and 
> short-lived oriented, which regressed a lot of existing setups that 
> expects sustained throughput and packet rate after kernel upgrade. 
> Although the occurrence of such drastic change for default disposition 
> is not so welcome (that is one of the reasons why I valued consistent 
> initial value and back compatible  behavior), I don't see people yelling 
> at virtio spec for less flexibility of offering the default disposition, 
> given that the guest user can override the config any time with their 
> own tooling or script, there's no problem at all for them to just set 
> the corresponding config back explicitly to what it was before.
> 
> >
> >>>>> And device/driver has better performance, is that a problem? Unlikely.
> >>>>>
> >> Even for rare case with a hard wired setup, the way to tackle the very
> >> problem using device's default is still quite questionable. Usually the
> >> mgmt software or network config utility should be equipped with some
> >> default value if need be. And we know the guest has the best position to
> >> impose the best / most performant config for its own load. What is the
> >> issue or use case that this initial config couldn't be applied by the
> >> guest mgmt software ahead but has to resort to the device to load some
> >> default (which is odd and irrelevant to any guest load), before the
> >> interface is brought up for operation i.e. performing I/O?
> > Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards
> > have a default value that is not 0.
> You seems to be referencing your own setup basically, and the question 
> is still left answered - why the initial config can't be done through 
> the mgmt software or network config utility within the guest?
> 
> > (0 is actually a kind of default value)
> >
> >>> Sorry, my vacation just ended.
> >>>
> >>>> Yes, it is possible. Driver can cache values it sets
> >>>> and never query device with get.
> >>> Don't we already have a lot of behaviors to drive queries from devices?
> >>> RSS context, device stats.
> >>>
> >>>> Before anything is set, driver will report incorrect values.
> >>> Devices that are widely supported and supported by good practices should have
> >>> any initialization value. Just reporting 0 is incorrect value. Although the
> >>> spec now says so.
> >> I don't have an aligned view here, sorry. As I recall having 0 as the
> >> default is just to keep device started in a state where coalescing is
> >> disabled, so it's backward compatible and consistent with a
> >> non-coalescing supporting driver - such that it won't yield surprising
> >> effect (for e.g. regressed latency) inadvertently after user's getting
> >> driver software upgraded. Unlike the other virtio-net features that
> >> could 100% improve performance in all aspect, this coalescing feature is
> >> more of a performance tuning knob that may improve performance metrics
> >> (such as cpu usage or throughput) of one dimension while demoting the
> >> others (such as latency, jitter or connection rate) from the equation.
> >> That said, there's not a single and fixed set of default config that
> >> device could supply which is able to satisfy all kind of guest load.
> >> Rather than rely on the device to offer a matching default for driver
> >> (which I think it's technically wrong), I'd lean to having guest
> >> software or network utility to apply the initial config for the guest,
> >> where they should have best knowledge for the specific guest workload
> >> than what device could do.
> > Before this feature, a good device implementation should also support coalescing
> > (of course we don't necessarily assume it has coalescing).
> Again, I don't doubt the value of supporting coalescing.
> 
> >   In addition, virtual
> > machines that tend to favor latency and throughput exist. If the device supported
> > by the manufacturer needs to provide a low-latency virtual machine, please
> > continue to keep the default value of 0.
> No, that's not what I was asking. There's no such requirement for any 
> vendor to provide a low-latency or high throughput VM. The more general 
> use case is - the setup for real world workload might just be too 
> complex that end users would prefer low-latency on some virtual NIC or 
> even some specific queues, while the other queues of a virtual NIC, or 
> reset virtual NIC might have very different dispositions. Due to the 
> needs and dynamics of workload scaling up & down, they might have more 
> or less queues or virtual NICs to configure, so these disposition would 
> need to be readjusted at any point of time, for which there's no easy 
> way for device to adapt to easily. The guest user should have best 
> knowledge for the specific guest workload and setup than what device 
> could/should offer.
> 
> >
> >>>> What will break as a result? Hard to predict.
> >>>>
> >>>>
> >>>> Given the patch is big, I am inclined to say it just should use
> >>>> a feature bit.
> >>> This doesn't seem to break anything in my opinion, we just told the device that
> >>> now you can set more values.
> >> Another thing this could break is live migration between devices with
> >> varied default. How do you make sure the guest doesn't rely on some
> >> default from the source device, while on the destination it just doesn't
> >> get the same default coalescingjjj value? To get rid of this side effect
> >> the guest would still need to apply the initial config for its own,
> >> anyway... Which eventually would render this proposal with arbitrary
> >> default rather pointless.
> > I don't quite understand why this would affect hot migration, the values
> > would be migrated over.
> Then I don't see this has been discussed (wouldn't the initial value be 
> part of virtio device state to migrate over?) in the thread or described 
> in the proposed text. And your proposed per-queue coalescing parameters 
> (default plus current value) would have to be described too as part of 
> virtio-net device state, so that people don't misunderstand your proposal.
> 
> Thanks,
> -Siwei
> 
> > Thanks.
> >
> >>
> >> Thanks,
> >> -Siwei
> >>> Using new feature bits does not seem necessary.
> >>>
> >>> Thanks.
> >>>
> >>>> -- 
> >>>> MST
> >>>>
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/20/2024 8:24 PM, Heng Qi wrote:
> On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
>>
>> On 6/20/2024 12:40 AM, Heng Qi wrote:
>>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
>>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
>>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
>>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
>>>>>>>> From: Michael S. Tsirkin <mst@redhat.com>
>>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
>>>>>>>> How we can we make progress with
>>>>>>>> the realease but sure we don't make backwards compat a pain?
>>>>>>>>
>>>>>>>> Ideas?
>>>>>>>>
>>>>>>> There is no functional break with this relaxation.
>>>>>>> Device set some non-zero defaults and driver didn't modify them....
>>>>>>> Anything broken? Unlikely.
>>>> Generally it's inappropriate to leave this decision making to the device
>>>> for what would be the best / most performant default config, as the
>>>> device is generally considered agnostic to the guest load...
>>> Instead, the performance of the virtual machine and the driver depends heavily
>>> on how the device is implemented, just as we have proposed various ways to
>>> offload the data queue in the device to the hardware. The reason why most
>>> devices use software to simulate ctrlq instead of using hardware offload is
>>> that the driver has no requirements for the performance of ctrlq before, that is,
>>> the device implementation is responsible for and meets the driver's performance
>>> requirements.
>> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
>> thought that it was for the debate why we need coalescing on hardware
>> offload device in the first place, instead of reusing event index or
>> similar s/w based notification suppression. I don't doubt the value of
>> coalescing, but how's it relevant to the default value disposition?
>> Generally the default config disposition is not considered to be part of
>> device implementation, especially when it comes to the situation where
>> device can't easily figure out the specific workload to occur in the
>> guest, and there's no perfect single default value that could meet every
>> single performance metric across the board. This is a typical tuning
>> knob left up to the user to adjust, why the device or driver needs to
>> set or load the initial value? The driver just needs to start with
>> certain value, be it 0 or non-zero, which guest user can override at any
>> point of time, depending on his/her need, and that's it! I guess I still
>> don't understand your user case here, why device / driver default is of
>> such importance.
> I've explained that, and I understand your argument is why default value is needed,
> and users should be able to adjust them, right?
Sounds about right. You'll soon realize that there's no perfect default 
that could work with everyone - in that it's just a static value to 
begin with, no matter whatever initial value the device comes up with, 
one user or another will come over to you and complain that what is 
loaded from the device doesn't match the workload they have, so those 
users are still expected to adjust manually tweaking for their own. As 
long as it's a tunable that guest user can control and override anytime, 
I don't feel it too much different what initial config the device would 
start with.

> The default value is working when the user doesn't adjust them.
> It's not practical to rely entirely on user adjustment,
That's where the adaptive interrupt moderation framework (e.g. DIM for 
Linux) could come to the play, I think?

>   and for devices
> that serve hundreds of thousands of customers, the device implementation
> has to be comprehensive.
Would you mind elaborating the technical difficulty for why the device 
implementation has to be comprehensive to serve hundreds or thousands of 
customers (i.e. adaptive moderation in the device), and is there better 
way to improve the current interface (ctrlq v.s. doorbell/MMIO register) 
that is hardware offload implementation wise. I feel people reading the 
thread without understanding the full background would become even more 
confused as to why it's relevant to your proposal, to me it's completely 
distinct use case or problem area that we are talking about.

Thanks,
-Siwei

>
> Thanks.
>
>>>> Unless the
>>>> device is specially hard wired to some fixed guest setup that users
>>>> couldn't change, it doesn't seem logical that the device could derive
>>>> the best or most performant config on driver's behalf. What if the guest
>>>> wants best latency for its load but the device just blindlessly guess
>>>> the guest might prefer throughput friendly that it miserably uses
>>>> latency impacting non-zero default?
>>> The device does not want to guess and cannot guess. This patch does not force
>>> the device to choose a non-zero value, but relaxes it to allow the device to
>>> choose 0 or non-zero, which is very friendly to virtual machines with different
>>> performance requirements, right?
>> I don't understand the friendly part - do you imply your VM users are on
>> kinda fixed wired setup that they cannot change these coalescing
>> parameters after driver is loaded? Can the owner of the VM in control
>> apply certain initial config for the coalescing parameters to the VM
>> image? Or is it the problem of the guest driver that doesn't yet expose
>> coalescing parameters to the end user? Otherwise I would think that
>> guest user should be able to set parameters accordingly that would best
>> fit the specific performance requirement of their own. How the device
>> could even help here? I don't feel there's a lot of value to grant the
>> device or host admin the flexibility to policy the *best* config on
>> guest user's behalf, to be honest. And you seem to admit the fact that
>> the default doesn't really matter, be it 0 or non-zero.
>>
>>>> Could this device side change for
>>>> the default config regress boot time performance (which may need best
>>>> latency over throughput)?
>>> Don't make these assumptions, what if the driver needs better throughput?
>> There's a misconception here: what we think that driver may need in
>> terms on performance does actually reflect what guest user would like to
>> have. Driver cannot read guest user's mind to make the decision, either.
>>
>> In history there was drastic change in the Linux virtio-net driver that
>> ever changed the default disposition for XPS (and RPS as well) from
>> throughput and long-lived connection oriented to concurrency and
>> short-lived oriented, which regressed a lot of existing setups that
>> expects sustained throughput and packet rate after kernel upgrade.
>> Although the occurrence of such drastic change for default disposition
>> is not so welcome (that is one of the reasons why I valued consistent
>> initial value and back compatible  behavior), I don't see people yelling
>> at virtio spec for less flexibility of offering the default disposition,
>> given that the guest user can override the config any time with their
>> own tooling or script, there's no problem at all for them to just set
>> the corresponding config back explicitly to what it was before.
>>
>>>>>>> And device/driver has better performance, is that a problem? Unlikely.
>>>>>>>
>>>> Even for rare case with a hard wired setup, the way to tackle the very
>>>> problem using device's default is still quite questionable. Usually the
>>>> mgmt software or network config utility should be equipped with some
>>>> default value if need be. And we know the guest has the best position to
>>>> impose the best / most performant config for its own load. What is the
>>>> issue or use case that this initial config couldn't be applied by the
>>>> guest mgmt software ahead but has to resort to the device to load some
>>>> default (which is odd and irrelevant to any guest load), before the
>>>> interface is brought up for operation i.e. performing I/O?
>>> Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards
>>> have a default value that is not 0.
>> You seems to be referencing your own setup basically, and the question
>> is still left answered - why the initial config can't be done through
>> the mgmt software or network config utility within the guest?
>>
>>> (0 is actually a kind of default value)
>>>
>>>>> Sorry, my vacation just ended.
>>>>>
>>>>>> Yes, it is possible. Driver can cache values it sets
>>>>>> and never query device with get.
>>>>> Don't we already have a lot of behaviors to drive queries from devices?
>>>>> RSS context, device stats.
>>>>>
>>>>>> Before anything is set, driver will report incorrect values.
>>>>> Devices that are widely supported and supported by good practices should have
>>>>> any initialization value. Just reporting 0 is incorrect value. Although the
>>>>> spec now says so.
>>>> I don't have an aligned view here, sorry. As I recall having 0 as the
>>>> default is just to keep device started in a state where coalescing is
>>>> disabled, so it's backward compatible and consistent with a
>>>> non-coalescing supporting driver - such that it won't yield surprising
>>>> effect (for e.g. regressed latency) inadvertently after user's getting
>>>> driver software upgraded. Unlike the other virtio-net features that
>>>> could 100% improve performance in all aspect, this coalescing feature is
>>>> more of a performance tuning knob that may improve performance metrics
>>>> (such as cpu usage or throughput) of one dimension while demoting the
>>>> others (such as latency, jitter or connection rate) from the equation.
>>>> That said, there's not a single and fixed set of default config that
>>>> device could supply which is able to satisfy all kind of guest load.
>>>> Rather than rely on the device to offer a matching default for driver
>>>> (which I think it's technically wrong), I'd lean to having guest
>>>> software or network utility to apply the initial config for the guest,
>>>> where they should have best knowledge for the specific guest workload
>>>> than what device could do.
>>> Before this feature, a good device implementation should also support coalescing
>>> (of course we don't necessarily assume it has coalescing).
>> Again, I don't doubt the value of supporting coalescing.
>>
>>>    In addition, virtual
>>> machines that tend to favor latency and throughput exist. If the device supported
>>> by the manufacturer needs to provide a low-latency virtual machine, please
>>> continue to keep the default value of 0.
>> No, that's not what I was asking. There's no such requirement for any
>> vendor to provide a low-latency or high throughput VM. The more general
>> use case is - the setup for real world workload might just be too
>> complex that end users would prefer low-latency on some virtual NIC or
>> even some specific queues, while the other queues of a virtual NIC, or
>> reset virtual NIC might have very different dispositions. Due to the
>> needs and dynamics of workload scaling up & down, they might have more
>> or less queues or virtual NICs to configure, so these disposition would
>> need to be readjusted at any point of time, for which there's no easy
>> way for device to adapt to easily. The guest user should have best
>> knowledge for the specific guest workload and setup than what device
>> could/should offer.
>>
>>>>>> What will break as a result? Hard to predict.
>>>>>>
>>>>>>
>>>>>> Given the patch is big, I am inclined to say it just should use
>>>>>> a feature bit.
>>>>> This doesn't seem to break anything in my opinion, we just told the device that
>>>>> now you can set more values.
>>>> Another thing this could break is live migration between devices with
>>>> varied default. How do you make sure the guest doesn't rely on some
>>>> default from the source device, while on the destination it just doesn't
>>>> get the same default coalescingjjj value? To get rid of this side effect
>>>> the guest would still need to apply the initial config for its own,
>>>> anyway... Which eventually would render this proposal with arbitrary
>>>> default rather pointless.
>>> I don't quite understand why this would affect hot migration, the values
>>> would be migrated over.
>> Then I don't see this has been discussed (wouldn't the initial value be
>> part of virtio device state to migrate over?) in the thread or described
>> in the proposed text. And your proposed per-queue coalescing parameters
>> (default plus current value) would have to be described too as part of
>> virtio-net device state, so that people don't misunderstand your proposal.
>>
>> Thanks,
>> -Siwei
>>
>>> Thanks.
>>>
>>>> Thanks,
>>>> -Siwei
>>>>> Using new feature bits does not seem necessary.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>> -- 
>>>>>> MST
>>>>>>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi]

On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> 
> 
> On 6/20/2024 8:24 PM, Heng Qi wrote:
> > On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> >>
> >> On 6/20/2024 12:40 AM, Heng Qi wrote:
> >>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> >>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
> >>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> >>>>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
> >>>>>>>> How we can we make progress with
> >>>>>>>> the realease but sure we don't make backwards compat a pain?
> >>>>>>>>
> >>>>>>>> Ideas?
> >>>>>>>>
> >>>>>>> There is no functional break with this relaxation.
> >>>>>>> Device set some non-zero defaults and driver didn't modify them....
> >>>>>>> Anything broken? Unlikely.
> >>>> Generally it's inappropriate to leave this decision making to the device
> >>>> for what would be the best / most performant default config, as the
> >>>> device is generally considered agnostic to the guest load...
> >>> Instead, the performance of the virtual machine and the driver depends heavily
> >>> on how the device is implemented, just as we have proposed various ways to
> >>> offload the data queue in the device to the hardware. The reason why most
> >>> devices use software to simulate ctrlq instead of using hardware offload is
> >>> that the driver has no requirements for the performance of ctrlq before, that is,
> >>> the device implementation is responsible for and meets the driver's performance
> >>> requirements.
> >> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
> >> thought that it was for the debate why we need coalescing on hardware
> >> offload device in the first place, instead of reusing event index or
> >> similar s/w based notification suppression. I don't doubt the value of
> >> coalescing, but how's it relevant to the default value disposition?
> >> Generally the default config disposition is not considered to be part of
> >> device implementation, especially when it comes to the situation where
> >> device can't easily figure out the specific workload to occur in the
> >> guest, and there's no perfect single default value that could meet every
> >> single performance metric across the board. This is a typical tuning
> >> knob left up to the user to adjust, why the device or driver needs to
> >> set or load the initial value? The driver just needs to start with
> >> certain value, be it 0 or non-zero, which guest user can override at any
> >> point of time, depending on his/her need, and that's it! I guess I still
> >> don't understand your user case here, why device / driver default is of
> >> such importance.
> > I've explained that, and I understand your argument is why default value is needed,
> > and users should be able to adjust them, right?
> Sounds about right. You'll soon realize that there's no perfect default 
> that could work with everyone - in that it's just a static value to 
> begin with, no matter whatever initial value the device comes up with, 
> one user or another will come over to you and complain that what is 
> loaded from the device doesn't match the workload they have, so those 
> users are still expected to adjust manually tweaking for their own. As 
> long as it's a tunable that guest user can control and override anytime, 
> I don't feel it too much different what initial config the device would 
> start with.

When you have a large number of customers and they buy your machines,
how many users do you think have experience adjusting this value? More below.

> 
> > The default value is working when the user doesn't adjust them.
> > It's not practical to rely entirely on user adjustment,
> That's where the adaptive interrupt moderation framework (e.g. DIM for 
> Linux) could come to the play, I think?

Last time, this work is only useful if DIM is not enabled. I don't want to
explain it again, I've said it many times to different people in the history
discussion, don't you want to check any history discussion?

> 
> >   and for devices
> > that serve hundreds of thousands of customers, the device implementation
> > has to be comprehensive.
> Would you mind elaborating the technical difficulty for why the device 
> implementation has to be comprehensive to serve hundreds or thousands of 
> customers (i.e. adaptive moderation in the device), and is there better 
> way to improve the current interface (ctrlq v.s. doorbell/MMIO register) 
> that is hardware offload implementation wise. I feel people reading the 
> thread without understanding the full background would become even more 
> confused as to why it's relevant to your proposal, to me it's completely 
> distinct use case or problem area that we are talking about.

Regardless of whether DIM can be enabled by default (it cannot accelerate all
scenarios, right?), therefore, a considerable number of machines on the
cloud vendors' line still have to rely on static values to provide boot-up
performance. The vendors will also make optimizations to optimize scenarios
like ping-pong, so the boot-up performance is very good.

Thanks.

> 
> Thanks,
> -Siwei
> 
> >
> > Thanks.
> >
> >>>> Unless the
> >>>> device is specially hard wired to some fixed guest setup that users
> >>>> couldn't change, it doesn't seem logical that the device could derive
> >>>> the best or most performant config on driver's behalf. What if the guest
> >>>> wants best latency for its load but the device just blindlessly guess
> >>>> the guest might prefer throughput friendly that it miserably uses
> >>>> latency impacting non-zero default?
> >>> The device does not want to guess and cannot guess. This patch does not force
> >>> the device to choose a non-zero value, but relaxes it to allow the device to
> >>> choose 0 or non-zero, which is very friendly to virtual machines with different
> >>> performance requirements, right?
> >> I don't understand the friendly part - do you imply your VM users are on
> >> kinda fixed wired setup that they cannot change these coalescing
> >> parameters after driver is loaded? Can the owner of the VM in control
> >> apply certain initial config for the coalescing parameters to the VM
> >> image? Or is it the problem of the guest driver that doesn't yet expose
> >> coalescing parameters to the end user? Otherwise I would think that
> >> guest user should be able to set parameters accordingly that would best
> >> fit the specific performance requirement of their own. How the device
> >> could even help here? I don't feel there's a lot of value to grant the
> >> device or host admin the flexibility to policy the *best* config on
> >> guest user's behalf, to be honest. And you seem to admit the fact that
> >> the default doesn't really matter, be it 0 or non-zero.
> >>
> >>>> Could this device side change for
> >>>> the default config regress boot time performance (which may need best
> >>>> latency over throughput)?
> >>> Don't make these assumptions, what if the driver needs better throughput?
> >> There's a misconception here: what we think that driver may need in
> >> terms on performance does actually reflect what guest user would like to
> >> have. Driver cannot read guest user's mind to make the decision, either.
> >>
> >> In history there was drastic change in the Linux virtio-net driver that
> >> ever changed the default disposition for XPS (and RPS as well) from
> >> throughput and long-lived connection oriented to concurrency and
> >> short-lived oriented, which regressed a lot of existing setups that
> >> expects sustained throughput and packet rate after kernel upgrade.
> >> Although the occurrence of such drastic change for default disposition
> >> is not so welcome (that is one of the reasons why I valued consistent
> >> initial value and back compatible  behavior), I don't see people yelling
> >> at virtio spec for less flexibility of offering the default disposition,
> >> given that the guest user can override the config any time with their
> >> own tooling or script, there's no problem at all for them to just set
> >> the corresponding config back explicitly to what it was before.
> >>
> >>>>>>> And device/driver has better performance, is that a problem? Unlikely.
> >>>>>>>
> >>>> Even for rare case with a hard wired setup, the way to tackle the very
> >>>> problem using device's default is still quite questionable. Usually the
> >>>> mgmt software or network config utility should be equipped with some
> >>>> default value if need be. And we know the guest has the best position to
> >>>> impose the best / most performant config for its own load. What is the
> >>>> issue or use case that this initial config couldn't be applied by the
> >>>> guest mgmt software ahead but has to resort to the device to load some
> >>>> default (which is odd and irrelevant to any guest load), before the
> >>>> interface is brought up for operation i.e. performing I/O?
> >>> Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards
> >>> have a default value that is not 0.
> >> You seems to be referencing your own setup basically, and the question
> >> is still left answered - why the initial config can't be done through
> >> the mgmt software or network config utility within the guest?
> >>
> >>> (0 is actually a kind of default value)
> >>>
> >>>>> Sorry, my vacation just ended.
> >>>>>
> >>>>>> Yes, it is possible. Driver can cache values it sets
> >>>>>> and never query device with get.
> >>>>> Don't we already have a lot of behaviors to drive queries from devices?
> >>>>> RSS context, device stats.
> >>>>>
> >>>>>> Before anything is set, driver will report incorrect values.
> >>>>> Devices that are widely supported and supported by good practices should have
> >>>>> any initialization value. Just reporting 0 is incorrect value. Although the
> >>>>> spec now says so.
> >>>> I don't have an aligned view here, sorry. As I recall having 0 as the
> >>>> default is just to keep device started in a state where coalescing is
> >>>> disabled, so it's backward compatible and consistent with a
> >>>> non-coalescing supporting driver - such that it won't yield surprising
> >>>> effect (for e.g. regressed latency) inadvertently after user's getting
> >>>> driver software upgraded. Unlike the other virtio-net features that
> >>>> could 100% improve performance in all aspect, this coalescing feature is
> >>>> more of a performance tuning knob that may improve performance metrics
> >>>> (such as cpu usage or throughput) of one dimension while demoting the
> >>>> others (such as latency, jitter or connection rate) from the equation.
> >>>> That said, there's not a single and fixed set of default config that
> >>>> device could supply which is able to satisfy all kind of guest load.
> >>>> Rather than rely on the device to offer a matching default for driver
> >>>> (which I think it's technically wrong), I'd lean to having guest
> >>>> software or network utility to apply the initial config for the guest,
> >>>> where they should have best knowledge for the specific guest workload
> >>>> than what device could do.
> >>> Before this feature, a good device implementation should also support coalescing
> >>> (of course we don't necessarily assume it has coalescing).
> >> Again, I don't doubt the value of supporting coalescing.
> >>
> >>>    In addition, virtual
> >>> machines that tend to favor latency and throughput exist. If the device supported
> >>> by the manufacturer needs to provide a low-latency virtual machine, please
> >>> continue to keep the default value of 0.
> >> No, that's not what I was asking. There's no such requirement for any
> >> vendor to provide a low-latency or high throughput VM. The more general
> >> use case is - the setup for real world workload might just be too
> >> complex that end users would prefer low-latency on some virtual NIC or
> >> even some specific queues, while the other queues of a virtual NIC, or
> >> reset virtual NIC might have very different dispositions. Due to the
> >> needs and dynamics of workload scaling up & down, they might have more
> >> or less queues or virtual NICs to configure, so these disposition would
> >> need to be readjusted at any point of time, for which there's no easy
> >> way for device to adapt to easily. The guest user should have best
> >> knowledge for the specific guest workload and setup than what device
> >> could/should offer.
> >>
> >>>>>> What will break as a result? Hard to predict.
> >>>>>>
> >>>>>>
> >>>>>> Given the patch is big, I am inclined to say it just should use
> >>>>>> a feature bit.
> >>>>> This doesn't seem to break anything in my opinion, we just told the device that
> >>>>> now you can set more values.
> >>>> Another thing this could break is live migration between devices with
> >>>> varied default. How do you make sure the guest doesn't rely on some
> >>>> default from the source device, while on the destination it just doesn't
> >>>> get the same default coalescingjjj value? To get rid of this side effect
> >>>> the guest would still need to apply the initial config for its own,
> >>>> anyway... Which eventually would render this proposal with arbitrary
> >>>> default rather pointless.
> >>> I don't quite understand why this would affect hot migration, the values
> >>> would be migrated over.
> >> Then I don't see this has been discussed (wouldn't the initial value be
> >> part of virtio device state to migrate over?) in the thread or described
> >> in the proposed text. And your proposed per-queue coalescing parameters
> >> (default plus current value) would have to be described too as part of
> >> virtio-net device state, so that people don't misunderstand your proposal.
> >>
> >> Thanks,
> >> -Siwei
> >>
> >>> Thanks.
> >>>
> >>>> Thanks,
> >>>> -Siwei
> >>>>> Using new feature bits does not seem necessary.
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>>> -- 
> >>>>>> MST
> >>>>>>
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/21/2024 6:34 PM, Heng Qi wrote:
> On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>  wrote:
>> On 6/20/2024 8:24 PM, Heng Qi wrote:
>>> On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>  wrote:
>>>> On 6/20/2024 12:40 AM, Heng Qi wrote:
>>>>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>  wrote:
>>>>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
>>>>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin"<mst@redhat.com>  wrote:
>>>>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
>>>>>>>>>> From: Michael S. Tsirkin<mst@redhat.com>
>>>>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
>>>>>>>>>> How we can we make progress with
>>>>>>>>>> the realease but sure we don't make backwards compat a pain?
>>>>>>>>>>
>>>>>>>>>> Ideas?
>>>>>>>>>>
>>>>>>>>> There is no functional break with this relaxation.
>>>>>>>>> Device set some non-zero defaults and driver didn't modify them....
>>>>>>>>> Anything broken? Unlikely.
>>>>>> Generally it's inappropriate to leave this decision making to the device
>>>>>> for what would be the best / most performant default config, as the
>>>>>> device is generally considered agnostic to the guest load...
>>>>> Instead, the performance of the virtual machine and the driver depends heavily
>>>>> on how the device is implemented, just as we have proposed various ways to
>>>>> offload the data queue in the device to the hardware. The reason why most
>>>>> devices use software to simulate ctrlq instead of using hardware offload is
>>>>> that the driver has no requirements for the performance of ctrlq before, that is,
>>>>> the device implementation is responsible for and meets the driver's performance
>>>>> requirements.
>>>> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
>>>> thought that it was for the debate why we need coalescing on hardware
>>>> offload device in the first place, instead of reusing event index or
>>>> similar s/w based notification suppression. I don't doubt the value of
>>>> coalescing, but how's it relevant to the default value disposition?
>>>> Generally the default config disposition is not considered to be part of
>>>> device implementation, especially when it comes to the situation where
>>>> device can't easily figure out the specific workload to occur in the
>>>> guest, and there's no perfect single default value that could meet every
>>>> single performance metric across the board. This is a typical tuning
>>>> knob left up to the user to adjust, why the device or driver needs to
>>>> set or load the initial value? The driver just needs to start with
>>>> certain value, be it 0 or non-zero, which guest user can override at any
>>>> point of time, depending on his/her need, and that's it! I guess I still
>>>> don't understand your user case here, why device / driver default is of
>>>> such importance.
>>> I've explained that, and I understand your argument is why default value is needed,
>>> and users should be able to adjust them, right?
>> Sounds about right. You'll soon realize that there's no perfect default
>> that could work with everyone - in that it's just a static value to
>> begin with, no matter whatever initial value the device comes up with,
>> one user or another will come over to you and complain that what is
>> loaded from the device doesn't match the workload they have, so those
>> users are still expected to adjust manually tweaking for their own. As
>> long as it's a tunable that guest user can control and override anytime,
>> I don't feel it too much different what initial config the device would
>> start with.
> When you have a large number of customers and they buy your machines,
> how many users do you think have experience adjusting this value? More below.
This is not what this proposal alone could address, as it just shifts 
the sheer pain to the device and host admin rather than provide any 
flexibility. Technically I believe the right solution is to seek 
adaptive coalescing that doesn't require user to tune anything. Noted 
adaptive coalescing / suppression could be done from the guest or by the 
device. For e.g. the s/w based event index based notification 
suppression could also work with hardware if the event index is somehow 
propagated through a doorbell register rather than load from host memory 
over the PCI transport.

>>> The default value is working when the user doesn't adjust them.
>>> It's not practical to rely entirely on user adjustment,
>> That's where the adaptive interrupt moderation framework (e.g. DIM for
>> Linux) could come to the play, I think?
> Last time, this work is only useful if DIM is not enabled. I don't want to
> explain it again, I've said it many times to different people in the history
> discussion, don't you want to check any history discussion?
I did before I replied. I thought both Halil and Michael tried hard to 
convey is that what you claimed to work by just getting the initial 
config loaded from device will not be sufficient to address all the 
performance requirements in broad and general means other than satisfy 
the goal set by yourself. What I saw is that you kept proving yourself 
by trying to reference your own use case and design - this not just 
mixed different things up but also ended up with circular argument that 
wasted everyone's time unfortunately.

>>>    and for devices
>>> that serve hundreds of thousands of customers, the device implementation
>>> has to be comprehensive.
>> Would you mind elaborating the technical difficulty for why the device
>> implementation has to be comprehensive to serve hundreds or thousands of
>> customers (i.e. adaptive moderation in the device), and is there better
>> way to improve the current interface (ctrlq v.s. doorbell/MMIO register)
>> that is hardware offload implementation wise. I feel people reading the
>> thread without understanding the full background would become even more
>> confused as to why it's relevant to your proposal, to me it's completely
>> distinct use case or problem area that we are talking about.
> Regardless of whether DIM can be enabled by default (it cannot accelerate all
> scenarios, right?), therefore, a considerable number of machines on the
> cloud vendors' line still have to rely on static values to provide boot-up
> performance.
Sorry, this is not a good excuse for people to accept a sub-optimal 
solution. Please talk to the other hardware vendor or read my above 
reply carefully.

-Siwei

>   The vendors will also make optimizations to optimize scenarios
> like ping-pong, so the boot-up performance is very good.
>
> Thanks.
>
>> Thanks,
>> -Siwei
>>
>>> Thanks.
>>>
>>>>>> Unless the
>>>>>> device is specially hard wired to some fixed guest setup that users
>>>>>> couldn't change, it doesn't seem logical that the device could derive
>>>>>> the best or most performant config on driver's behalf. What if the guest
>>>>>> wants best latency for its load but the device just blindlessly guess
>>>>>> the guest might prefer throughput friendly that it miserably uses
>>>>>> latency impacting non-zero default?
>>>>> The device does not want to guess and cannot guess. This patch does not force
>>>>> the device to choose a non-zero value, but relaxes it to allow the device to
>>>>> choose 0 or non-zero, which is very friendly to virtual machines with different
>>>>> performance requirements, right?
>>>> I don't understand the friendly part - do you imply your VM users are on
>>>> kinda fixed wired setup that they cannot change these coalescing
>>>> parameters after driver is loaded? Can the owner of the VM in control
>>>> apply certain initial config for the coalescing parameters to the VM
>>>> image? Or is it the problem of the guest driver that doesn't yet expose
>>>> coalescing parameters to the end user? Otherwise I would think that
>>>> guest user should be able to set parameters accordingly that would best
>>>> fit the specific performance requirement of their own. How the device
>>>> could even help here? I don't feel there's a lot of value to grant the
>>>> device or host admin the flexibility to policy the *best* config on
>>>> guest user's behalf, to be honest. And you seem to admit the fact that
>>>> the default doesn't really matter, be it 0 or non-zero.
>>>>
>>>>>> Could this device side change for
>>>>>> the default config regress boot time performance (which may need best
>>>>>> latency over throughput)?
>>>>> Don't make these assumptions, what if the driver needs better throughput?
>>>> There's a misconception here: what we think that driver may need in
>>>> terms on performance does actually reflect what guest user would like to
>>>> have. Driver cannot read guest user's mind to make the decision, either.
>>>>
>>>> In history there was drastic change in the Linux virtio-net driver that
>>>> ever changed the default disposition for XPS (and RPS as well) from
>>>> throughput and long-lived connection oriented to concurrency and
>>>> short-lived oriented, which regressed a lot of existing setups that
>>>> expects sustained throughput and packet rate after kernel upgrade.
>>>> Although the occurrence of such drastic change for default disposition
>>>> is not so welcome (that is one of the reasons why I valued consistent
>>>> initial value and back compatible  behavior), I don't see people yelling
>>>> at virtio spec for less flexibility of offering the default disposition,
>>>> given that the guest user can override the config any time with their
>>>> own tooling or script, there's no problem at all for them to just set
>>>> the corresponding config back explicitly to what it was before.
>>>>
>>>>>>>>> And device/driver has better performance, is that a problem? Unlikely.
>>>>>>>>>
>>>>>> Even for rare case with a hard wired setup, the way to tackle the very
>>>>>> problem using device's default is still quite questionable. Usually the
>>>>>> mgmt software or network config utility should be equipped with some
>>>>>> default value if need be. And we know the guest has the best position to
>>>>>> impose the best / most performant config for its own load. What is the
>>>>>> issue or use case that this initial config couldn't be applied by the
>>>>>> guest mgmt software ahead but has to resort to the device to load some
>>>>>> default (which is odd and irrelevant to any guest load), before the
>>>>>> interface is brought up for operation i.e. performing I/O?
>>>>> Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards
>>>>> have a default value that is not 0.
>>>> You seems to be referencing your own setup basically, and the question
>>>> is still left answered - why the initial config can't be done through
>>>> the mgmt software or network config utility within the guest?
>>>>
>>>>> (0 is actually a kind of default value)
>>>>>
>>>>>>> Sorry, my vacation just ended.
>>>>>>>
>>>>>>>> Yes, it is possible. Driver can cache values it sets
>>>>>>>> and never query device with get.
>>>>>>> Don't we already have a lot of behaviors to drive queries from devices?
>>>>>>> RSS context, device stats.
>>>>>>>
>>>>>>>> Before anything is set, driver will report incorrect values.
>>>>>>> Devices that are widely supported and supported by good practices should have
>>>>>>> any initialization value. Just reporting 0 is incorrect value. Although the
>>>>>>> spec now says so.
>>>>>> I don't have an aligned view here, sorry. As I recall having 0 as the
>>>>>> default is just to keep device started in a state where coalescing is
>>>>>> disabled, so it's backward compatible and consistent with a
>>>>>> non-coalescing supporting driver - such that it won't yield surprising
>>>>>> effect (for e.g. regressed latency) inadvertently after user's getting
>>>>>> driver software upgraded. Unlike the other virtio-net features that
>>>>>> could 100% improve performance in all aspect, this coalescing feature is
>>>>>> more of a performance tuning knob that may improve performance metrics
>>>>>> (such as cpu usage or throughput) of one dimension while demoting the
>>>>>> others (such as latency, jitter or connection rate) from the equation.
>>>>>> That said, there's not a single and fixed set of default config that
>>>>>> device could supply which is able to satisfy all kind of guest load.
>>>>>> Rather than rely on the device to offer a matching default for driver
>>>>>> (which I think it's technically wrong), I'd lean to having guest
>>>>>> software or network utility to apply the initial config for the guest,
>>>>>> where they should have best knowledge for the specific guest workload
>>>>>> than what device could do.
>>>>> Before this feature, a good device implementation should also support coalescing
>>>>> (of course we don't necessarily assume it has coalescing).
>>>> Again, I don't doubt the value of supporting coalescing.
>>>>
>>>>>     In addition, virtual
>>>>> machines that tend to favor latency and throughput exist. If the device supported
>>>>> by the manufacturer needs to provide a low-latency virtual machine, please
>>>>> continue to keep the default value of 0.
>>>> No, that's not what I was asking. There's no such requirement for any
>>>> vendor to provide a low-latency or high throughput VM. The more general
>>>> use case is - the setup for real world workload might just be too
>>>> complex that end users would prefer low-latency on some virtual NIC or
>>>> even some specific queues, while the other queues of a virtual NIC, or
>>>> reset virtual NIC might have very different dispositions. Due to the
>>>> needs and dynamics of workload scaling up & down, they might have more
>>>> or less queues or virtual NICs to configure, so these disposition would
>>>> need to be readjusted at any point of time, for which there's no easy
>>>> way for device to adapt to easily. The guest user should have best
>>>> knowledge for the specific guest workload and setup than what device
>>>> could/should offer.
>>>>
>>>>>>>> What will break as a result? Hard to predict.
>>>>>>>>
>>>>>>>>
>>>>>>>> Given the patch is big, I am inclined to say it just should use
>>>>>>>> a feature bit.
>>>>>>> This doesn't seem to break anything in my opinion, we just told the device that
>>>>>>> now you can set more values.
>>>>>> Another thing this could break is live migration between devices with
>>>>>> varied default. How do you make sure the guest doesn't rely on some
>>>>>> default from the source device, while on the destination it just doesn't
>>>>>> get the same default coalescingjjj value? To get rid of this side effect
>>>>>> the guest would still need to apply the initial config for its own,
>>>>>> anyway... Which eventually would render this proposal with arbitrary
>>>>>> default rather pointless.
>>>>> I don't quite understand why this would affect hot migration, the values
>>>>> would be migrated over.
>>>> Then I don't see this has been discussed (wouldn't the initial value be
>>>> part of virtio device state to migrate over?) in the thread or described
>>>> in the proposed text. And your proposed per-queue coalescing parameters
>>>> (default plus current value) would have to be described too as part of
>>>> virtio-net device state, so that people don't misunderstand your proposal.
>>>>
>>>> Thanks,
>>>> -Siwei
>>>>
>>>>> Thanks.
>>>>>
>>>>>> Thanks,
>>>>>> -Siwei
>>>>>>> Using new feature bits does not seem necessary.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>> -- 
>>>>>>>> MST
>>>>>>>>

RE: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Parav Pandit]

Hi Si-Wei,

> From: Si-Wei Liu <si-wei.liu@oracle.com>
> Sent: Tuesday, June 25, 2024 10:22 AM
> 
> On 6/21/2024 6:34 PM, Heng Qi wrote:
> > On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>
> wrote:
> >> On 6/20/2024 8:24 PM, Heng Qi wrote:
> >>> On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>
> wrote:
> >>>> On 6/20/2024 12:40 AM, Heng Qi wrote:
> >>>>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu"<si-
> wei.liu@oracle.com>  wrote:
> >>>>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
> >>>>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S.
> Tsirkin"<mst@redhat.com>  wrote:
> >>>>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> >>>>>>>>>> From: Michael S. Tsirkin<mst@redhat.com>
> >>>>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM How we can we make
> >>>>>>>>>> progress with the realease but sure we don't make backwards
> >>>>>>>>>> compat a pain?
> >>>>>>>>>>
> >>>>>>>>>> Ideas?
> >>>>>>>>>>
> >>>>>>>>> There is no functional break with this relaxation.
> >>>>>>>>> Device set some non-zero defaults and driver didn't modify
> them....
> >>>>>>>>> Anything broken? Unlikely.
> >>>>>> Generally it's inappropriate to leave this decision making to the
> >>>>>> device for what would be the best / most performant default
> >>>>>> config, as the device is generally considered agnostic to the guest
> load...
> >>>>> Instead, the performance of the virtual machine and the driver
> >>>>> depends heavily on how the device is implemented, just as we have
> >>>>> proposed various ways to offload the data queue in the device to
> >>>>> the hardware. The reason why most devices use software to simulate
> >>>>> ctrlq instead of using hardware offload is that the driver has no
> >>>>> requirements for the performance of ctrlq before, that is, the
> >>>>> device implementation is responsible for and meets the driver's
> performance requirements.
> >>>> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
> >>>> thought that it was for the debate why we need coalescing on
> >>>> hardware offload device in the first place, instead of reusing
> >>>> event index or similar s/w based notification suppression. I don't
> >>>> doubt the value of coalescing, but how's it relevant to the default value
> disposition?
> >>>> Generally the default config disposition is not considered to be
> >>>> part of device implementation, especially when it comes to the
> >>>> situation where device can't easily figure out the specific
> >>>> workload to occur in the guest, and there's no perfect single
> >>>> default value that could meet every single performance metric
> >>>> across the board. This is a typical tuning knob left up to the user
> >>>> to adjust, why the device or driver needs to set or load the
> >>>> initial value? The driver just needs to start with certain value,
> >>>> be it 0 or non-zero, which guest user can override at any point of
> >>>> time, depending on his/her need, and that's it! I guess I still
> >>>> don't understand your user case here, why device / driver default is of
> such importance.
> >>> I've explained that, and I understand your argument is why default
> >>> value is needed, and users should be able to adjust them, right?
> >> Sounds about right. You'll soon realize that there's no perfect
> >> default that could work with everyone - in that it's just a static
> >> value to begin with, no matter whatever initial value the device
> >> comes up with, one user or another will come over to you and complain
> >> that what is loaded from the device doesn't match the workload they
> >> have, so those users are still expected to adjust manually tweaking
> >> for their own. As long as it's a tunable that guest user can control
> >> and override anytime, I don't feel it too much different what initial
> >> config the device would start with.
> > When you have a large number of customers and they buy your machines,
> > how many users do you think have experience adjusting this value? More
> below.
> This is not what this proposal alone could address, as it just shifts the sheer
> pain to the device and host admin rather than provide any flexibility.
> Technically I believe the right solution is to seek adaptive coalescing that
> doesn't require user to tune anything. Noted adaptive coalescing /
> suppression could be done from the guest or by the device. For e.g. the s/w
> based event index based notification suppression could also work with
> hardware if the event index is somehow propagated through a doorbell
> register rather than load from host memory over the PCI transport.
> 
> >>> The default value is working when the user doesn't adjust them.
> >>> It's not practical to rely entirely on user adjustment,
> >> That's where the adaptive interrupt moderation framework (e.g. DIM
> >> for
> >> Linux) could come to the play, I think?
> > Last time, this work is only useful if DIM is not enabled. I don't
> > want to explain it again, I've said it many times to different people
> > in the history discussion, don't you want to check any history discussion?
> I did before I replied. I thought both Halil and Michael tried hard to convey is
> that what you claimed to work by just getting the initial config loaded from
> device will not be sufficient to address all the performance requirements in
> broad and general means other than satisfy the goal set by yourself. What I
> saw is that you kept proving yourself by trying to reference your own use
> case and design - this not just mixed different things up but also ended up
> with circular argument that wasted everyone's time unfortunately.
> 
> >>>    and for devices
> >>> that serve hundreds of thousands of customers, the device
> >>> implementation has to be comprehensive.
> >> Would you mind elaborating the technical difficulty for why the
> >> device implementation has to be comprehensive to serve hundreds or
> >> thousands of customers (i.e. adaptive moderation in the device), and
> >> is there better way to improve the current interface (ctrlq v.s.
> >> doorbell/MMIO register) that is hardware offload implementation wise.
> >> I feel people reading the thread without understanding the full
> >> background would become even more confused as to why it's relevant to
> >> your proposal, to me it's completely distinct use case or problem area that
> we are talking about.
> > Regardless of whether DIM can be enabled by default (it cannot
> > accelerate all scenarios, right?), therefore, a considerable number of
> > machines on the cloud vendors' line still have to rely on static
> > values to provide boot-up performance.
> Sorry, this is not a good excuse for people to accept a sub-optimal solution.
> Please talk to the other hardware vendor or read my above reply carefully.
> 
> -Siwei
> 
> >   The vendors will also make optimizations to optimize scenarios like
> > ping-pong, so the boot-up performance is very good.
> >
> > Thanks.
> >
> >> Thanks,
> >> -Siwei
> >>
> >>> Thanks.
> >>>
> >>>>>> Unless the
> >>>>>> device is specially hard wired to some fixed guest setup that
> >>>>>> users couldn't change, it doesn't seem logical that the device
> >>>>>> could derive the best or most performant config on driver's
> >>>>>> behalf. What if the guest wants best latency for its load but the
> >>>>>> device just blindlessly guess the guest might prefer throughput
> >>>>>> friendly that it miserably uses latency impacting non-zero default?
> >>>>> The device does not want to guess and cannot guess. This patch
> >>>>> does not force the device to choose a non-zero value, but relaxes
> >>>>> it to allow the device to choose 0 or non-zero, which is very
> >>>>> friendly to virtual machines with different performance requirements,
> right?
> >>>> I don't understand the friendly part - do you imply your VM users
> >>>> are on kinda fixed wired setup that they cannot change these
> >>>> coalescing parameters after driver is loaded? Can the owner of the
> >>>> VM in control apply certain initial config for the coalescing
> >>>> parameters to the VM image? Or is it the problem of the guest
> >>>> driver that doesn't yet expose coalescing parameters to the end
> >>>> user? Otherwise I would think that guest user should be able to set
> >>>> parameters accordingly that would best fit the specific performance
> >>>> requirement of their own. How the device could even help here? I
> >>>> don't feel there's a lot of value to grant the device or host admin
> >>>> the flexibility to policy the *best* config on guest user's behalf,
> >>>> to be honest. And you seem to admit the fact that the default doesn't
> really matter, be it 0 or non-zero.
> >>>>
> >>>>>> Could this device side change for the default config regress boot
> >>>>>> time performance (which may need best latency over throughput)?
> >>>>> Don't make these assumptions, what if the driver needs better
> throughput?
> >>>> There's a misconception here: what we think that driver may need in
> >>>> terms on performance does actually reflect what guest user would
> >>>> like to have. Driver cannot read guest user's mind to make the decision,
> either.
> >>>>
> >>>> In history there was drastic change in the Linux virtio-net driver
> >>>> that ever changed the default disposition for XPS (and RPS as well)
> >>>> from throughput and long-lived connection oriented to concurrency
> >>>> and short-lived oriented, which regressed a lot of existing setups
> >>>> that expects sustained throughput and packet rate after kernel
> upgrade.
> >>>> Although the occurrence of such drastic change for default
> >>>> disposition is not so welcome (that is one of the reasons why I
> >>>> valued consistent initial value and back compatible  behavior), I
> >>>> don't see people yelling at virtio spec for less flexibility of
> >>>> offering the default disposition, given that the guest user can
> >>>> override the config any time with their own tooling or script,
> >>>> there's no problem at all for them to just set the corresponding config
> back explicitly to what it was before.
> >>>>
> >>>>>>>>> And device/driver has better performance, is that a problem?
> Unlikely.
> >>>>>>>>>
> >>>>>> Even for rare case with a hard wired setup, the way to tackle the
> >>>>>> very problem using device's default is still quite questionable.
> >>>>>> Usually the mgmt software or network config utility should be
> >>>>>> equipped with some default value if need be. And we know the
> >>>>>> guest has the best position to impose the best / most performant
> >>>>>> config for its own load. What is the issue or use case that this
> >>>>>> initial config couldn't be applied by the guest mgmt software
> >>>>>> ahead but has to resort to the device to load some default (which
> >>>>>> is odd and irrelevant to any guest load), before the interface is
> brought up for operation i.e. performing I/O?
> >>>>> Use cases are everywhere, Alibaba Cloud, MLX and all other modern
> >>>>> network cards have a default value that is not 0.
> >>>> You seems to be referencing your own setup basically, and the
> >>>> question is still left answered - why the initial config can't be
> >>>> done through the mgmt software or network config utility within the
> guest?
> >>>>
> >>>>> (0 is actually a kind of default value)
> >>>>>
> >>>>>>> Sorry, my vacation just ended.
> >>>>>>>
> >>>>>>>> Yes, it is possible. Driver can cache values it sets and never
> >>>>>>>> query device with get.
> >>>>>>> Don't we already have a lot of behaviors to drive queries from
> devices?
> >>>>>>> RSS context, device stats.
> >>>>>>>
> >>>>>>>> Before anything is set, driver will report incorrect values.
> >>>>>>> Devices that are widely supported and supported by good
> >>>>>>> practices should have any initialization value. Just reporting 0
> >>>>>>> is incorrect value. Although the spec now says so.
> >>>>>> I don't have an aligned view here, sorry. As I recall having 0 as
> >>>>>> the default is just to keep device started in a state where
> >>>>>> coalescing is disabled, so it's backward compatible and
> >>>>>> consistent with a non-coalescing supporting driver - such that it
> >>>>>> won't yield surprising effect (for e.g. regressed latency)
> >>>>>> inadvertently after user's getting driver software upgraded.
> >>>>>> Unlike the other virtio-net features that could 100% improve
> >>>>>> performance in all aspect, this coalescing feature is more of a
> >>>>>> performance tuning knob that may improve performance metrics
> >>>>>> (such as cpu usage or throughput) of one dimension while demoting
> the others (such as latency, jitter or connection rate) from the equation.
> >>>>>> That said, there's not a single and fixed set of default config
> >>>>>> that device could supply which is able to satisfy all kind of guest load.
> >>>>>> Rather than rely on the device to offer a matching default for
> >>>>>> driver (which I think it's technically wrong), I'd lean to having
> >>>>>> guest software or network utility to apply the initial config for
> >>>>>> the guest, where they should have best knowledge for the specific
> >>>>>> guest workload than what device could do.
> >>>>> Before this feature, a good device implementation should also
> >>>>> support coalescing (of course we don't necessarily assume it has
> coalescing).
> >>>> Again, I don't doubt the value of supporting coalescing.
> >>>>
> >>>>>     In addition, virtual
> >>>>> machines that tend to favor latency and throughput exist. If the
> >>>>> device supported by the manufacturer needs to provide a
> >>>>> low-latency virtual machine, please continue to keep the default value
> of 0.
> >>>> No, that's not what I was asking. There's no such requirement for
> >>>> any vendor to provide a low-latency or high throughput VM. The more
> >>>> general use case is - the setup for real world workload might just
> >>>> be too complex that end users would prefer low-latency on some
> >>>> virtual NIC or even some specific queues, while the other queues of
> >>>> a virtual NIC, or reset virtual NIC might have very different
> >>>> dispositions. Due to the needs and dynamics of workload scaling up
> >>>> & down, they might have more or less queues or virtual NICs to
> >>>> configure, so these disposition would need to be readjusted at any
> >>>> point of time, for which there's no easy way for device to adapt to
> >>>> easily. The guest user should have best knowledge for the specific
> >>>> guest workload and setup than what device could/should offer.
> >>>>
> >>>>>>>> What will break as a result? Hard to predict.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Given the patch is big, I am inclined to say it just should use
> >>>>>>>> a feature bit.
> >>>>>>> This doesn't seem to break anything in my opinion, we just told
> >>>>>>> the device that now you can set more values.
> >>>>>> Another thing this could break is live migration between devices
> >>>>>> with varied default. How do you make sure the guest doesn't rely
> >>>>>> on some default from the source device, while on the destination
> >>>>>> it just doesn't get the same default coalescingjjj value? To get
> >>>>>> rid of this side effect the guest would still need to apply the
> >>>>>> initial config for its own, anyway... Which eventually would
> >>>>>> render this proposal with arbitrary default rather pointless.
> >>>>> I don't quite understand why this would affect hot migration, the
> >>>>> values would be migrated over.
> >>>> Then I don't see this has been discussed (wouldn't the initial
> >>>> value be part of virtio device state to migrate over?) in the
> >>>> thread or described in the proposed text. And your proposed
> >>>> per-queue coalescing parameters (default plus current value) would
> >>>> have to be described too as part of virtio-net device state, so that
> people don't misunderstand your proposal.
> >>>>
> >>>> Thanks,
> >>>> -Siwei
> >>>>
> >>>>> Thanks.
> >>>>>
> >>>>>> Thanks,
> >>>>>> -Siwei
> >>>>>>> Using new feature bits does not seem necessary.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>> --
> >>>>>>>> MST
> >>>>>>>>

I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.

Flow is:
1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL

3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.

4. the virtio-net driver even after negotiating VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled.
vi->rq[queue].dim_enabled = false by default.

5. The virtio net driver enables dim only on user request via ethtool.

So here, just because one is using a new driver and new device, it may get lower performance.

Do we agree to above problem description?
If no, I likely missed something in the long discussion and need to read all of it. :(

If yes, few solutions are:
1. A user intervention is must to avoid this regression. The user in the guest VM must enable DIM if the device supports it.
Very hard to do this plumbing.

2. A net driver enables the DIM by default as long as VIRTIO_NET_F_VQ_NOTF_COAL is negotiated.
(without ethtool setting)

3. A device relaxes the limitation and continue to apply the coalescing, until driver overrides it.

In my humble opinion, Heng is solving it using option #3, that tends to work with existing and future drivers who may/may not enable the DIM.

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Mon, Jun 17, 2024 at 10:35:38AM +0800, Heng Qi wrote:
> On Tue, 11 Jun 2024 19:03:11 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Tue, May 28, 2024 at 12:47:02PM +0800, Heng Qi wrote:
> > > The device can set any initial coalescing parameters (0 or non-zero)
> > > for the receive/send queue before the setting command is executed,
> > > not just 0, enhancing device performance even without DIM enabled.
> > > 
> > > So we need to clarify descriptions that don't fit the behavior.
> > > 
> > > Fixes: https://github.com/oasis-tcs/virtio-spec/issues/194
> > > Suggested-by: Jason Wang <jasowang@redhat.com>
> > > Signed-off-by: Heng Qi <hengqi@linux.alibaba.com>
> > > Reviewed-by: Parav Pandit <parav@nvidia.com>
> > > Reviewed-by: Jason Wang <jasowang@redhat.com>
> > > ---
> > > v4->v5:
> > >   - Add RB tags from Parav and Jason. Thanks!
> > > 
> > > v3->v4:
> > >   - Doesn't force the device to remember more stuff. @Parav
> > > 
> > > v2->v3:
> > >   - Clarify description to be more generic. @Parav
> > > 
> > > v1->v2:
> > >   - Update description. @Jason
> > > 
> > >  device-types/net/description.tex | 14 +++++++++++---
> > >  1 file changed, 11 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> > > index 61cce1f..00c1b36 100644
> > > --- a/device-types/net/description.tex
> > > +++ b/device-types/net/description.tex
> > > @@ -1803,6 +1803,14 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > >                                          for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
> > >  \end{enumerate}
> > >  
> > > +If the VIRTIO_NET_F_NOTF_COAL or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated,
> > > +the device may apply any coalescing parameters to each transmit/receive virtqueue
> > > +before the driver successfully performs one of the VIRTIO_NET_CTRL_NOTF_COAL set commands.
> > 
> > may outside of conformance section, not good.
> 
> This is outside of the conformance section. Do you want inside?

needs to be rephrased to avoid using reserved keywords.

> > 
> > > +
> > > +The driver can query the coalescing parameters of any enabled transmit/receive
> > > +virtqueue using the VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET command, before or after any
> > > +VIRTIO_NET_CTRL_NOTF_COAL set command is done.
> > > +
> > 
> > Vague. There are just 3 set commands ust list them.
> > 
> > >  The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> > >  
> > >  If coalescing parameters are being set, the device applies the last coalescing parameters set for a
> > 
> > This needs more work but I guess we can skip the above part.
> > 
> > > @@ -1885,17 +1893,17 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > >  VIRTIO_NET_ERR if the designated virtqueue is not an enabled transmit or receive virtqueue.
> > >  
> > >  Upon disabling and re-enabling a transmit virtqueue, the device MUST set the coalescing parameters of the virtqueue
> > > -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, to 0.
> > > +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver did not set any TX coalescing parameters, the device MAY initialize them to any values.
> > >  
> > 
> > If device does not initialize them then what?
> 
> The device's customized behavior, at least, does not conform to the spec.

Then why do you say "may"?


> > Actually something like 
> > 0 -> the default TX coalescing parameters chosen by the device?
> 
> Deefault coalescing parameters should also have a concrete "value".

and I just proposed how to say this in a clear way.


> > 
> > 
> > >  Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> > > -to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> > > +to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, the device MAY initialize them to any values.
> > >  
> > >  The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > >  the device MAY generate notifications more or less frequently than specified.
> > >  
> > >  A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> > >  
> > > -Upon reset, a device MUST initialize all coalescing parameters to 0.
> > > +Upon reset, a device MAY set any coalescing parameters for all transmit and receive virtqueues.
> > 
> > Confusing.
> > This seems to allow any vq to have a different set of parameters.
> 
> YES.
> 
> > Now if driver calls VIRTIO_NET_CTRL_NOTF_COAL_RX_SET then what?
> > 
> 
> Then the coalescing parameters of all RX queues are updated to the values
> set by VIRTIO_NET_CTRL_NOTF_COAL_RX_SET, and the coalescing parameters of all
> TX queues continue to retain the previous values.
> 
> Thanks.

It's not at all nice that there's apparently no way for driver to
get back to the default values.


> > 
> > >  
> > >  \paragraph{Device Statistics}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Device Statistics}
> > 
> > 
> > 
> > 
> > >  
> > > -- 
> > > 2.32.0.3.g01195cf9f
> > 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Jason Wang]

On Sat, Jun 22, 2024 at 7:46 AM Si-Wei Liu <si-wei.liu@oracle.com> wrote:
>
>
>
> On 6/20/2024 8:24 PM, Heng Qi wrote:
> > On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> >>
> >> On 6/20/2024 12:40 AM, Heng Qi wrote:
> >>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> >>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
> >>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> >>>>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
> >>>>>>>> How we can we make progress with
> >>>>>>>> the realease but sure we don't make backwards compat a pain?
> >>>>>>>>
> >>>>>>>> Ideas?
> >>>>>>>>
> >>>>>>> There is no functional break with this relaxation.
> >>>>>>> Device set some non-zero defaults and driver didn't modify them....
> >>>>>>> Anything broken? Unlikely.
> >>>> Generally it's inappropriate to leave this decision making to the device
> >>>> for what would be the best / most performant default config, as the
> >>>> device is generally considered agnostic to the guest load...
> >>> Instead, the performance of the virtual machine and the driver depends heavily
> >>> on how the device is implemented, just as we have proposed various ways to
> >>> offload the data queue in the device to the hardware. The reason why most
> >>> devices use software to simulate ctrlq instead of using hardware offload is
> >>> that the driver has no requirements for the performance of ctrlq before, that is,
> >>> the device implementation is responsible for and meets the driver's performance
> >>> requirements.
> >> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
> >> thought that it was for the debate why we need coalescing on hardware
> >> offload device in the first place, instead of reusing event index or
> >> similar s/w based notification suppression. I don't doubt the value of
> >> coalescing, but how's it relevant to the default value disposition?
> >> Generally the default config disposition is not considered to be part of
> >> device implementation, especially when it comes to the situation where
> >> device can't easily figure out the specific workload to occur in the
> >> guest, and there's no perfect single default value that could meet every
> >> single performance metric across the board. This is a typical tuning
> >> knob left up to the user to adjust, why the device or driver needs to
> >> set or load the initial value? The driver just needs to start with
> >> certain value, be it 0 or non-zero, which guest user can override at any
> >> point of time, depending on his/her need, and that's it! I guess I still
> >> don't understand your user case here, why device / driver default is of
> >> such importance.
> > I've explained that, and I understand your argument is why default value is needed,
> > and users should be able to adjust them, right?
> Sounds about right. You'll soon realize that there's no perfect default
> that could work with everyone - in that it's just a static value to
> begin with, no matter whatever initial value the device comes up with,
> one user or another will come over to you and complain that what is
> loaded from the device doesn't match the workload they have, so those
> users are still expected to adjust manually tweaking for their own. As
> long as it's a tunable that guest user can control and override anytime,
> I don't feel it too much different what initial config the device would
> start with.

Think hard of these, I think there's probably no conflict between the
points of each.

Generally, it's very hard to have a default value for virtio-net moderation as

1) various implementations of virtio-net
2) various setups of virtio-net (from virtual to physical, from 1G to 100G)
3) various kind of workloads

But it doesn't mean the device can not have something by default which
the vendor thinks it works well for in most cases. And anyhow the
value could be overridden by the driver if it wants (e.g several Linux
networking drivers will try to set a good default moderation value
during probe).

Thanks

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Tue, Jun 25, 2024 at 03:53:33PM +0800, Jason Wang wrote:
> On Sat, Jun 22, 2024 at 7:46 AM Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> >
> >
> >
> > On 6/20/2024 8:24 PM, Heng Qi wrote:
> > > On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> > >>
> > >> On 6/20/2024 12:40 AM, Heng Qi wrote:
> > >>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> > >>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
> > >>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > >>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> > >>>>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> > >>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
> > >>>>>>>> How we can we make progress with
> > >>>>>>>> the realease but sure we don't make backwards compat a pain?
> > >>>>>>>>
> > >>>>>>>> Ideas?
> > >>>>>>>>
> > >>>>>>> There is no functional break with this relaxation.
> > >>>>>>> Device set some non-zero defaults and driver didn't modify them....
> > >>>>>>> Anything broken? Unlikely.
> > >>>> Generally it's inappropriate to leave this decision making to the device
> > >>>> for what would be the best / most performant default config, as the
> > >>>> device is generally considered agnostic to the guest load...
> > >>> Instead, the performance of the virtual machine and the driver depends heavily
> > >>> on how the device is implemented, just as we have proposed various ways to
> > >>> offload the data queue in the device to the hardware. The reason why most
> > >>> devices use software to simulate ctrlq instead of using hardware offload is
> > >>> that the driver has no requirements for the performance of ctrlq before, that is,
> > >>> the device implementation is responsible for and meets the driver's performance
> > >>> requirements.
> > >> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
> > >> thought that it was for the debate why we need coalescing on hardware
> > >> offload device in the first place, instead of reusing event index or
> > >> similar s/w based notification suppression. I don't doubt the value of
> > >> coalescing, but how's it relevant to the default value disposition?
> > >> Generally the default config disposition is not considered to be part of
> > >> device implementation, especially when it comes to the situation where
> > >> device can't easily figure out the specific workload to occur in the
> > >> guest, and there's no perfect single default value that could meet every
> > >> single performance metric across the board. This is a typical tuning
> > >> knob left up to the user to adjust, why the device or driver needs to
> > >> set or load the initial value? The driver just needs to start with
> > >> certain value, be it 0 or non-zero, which guest user can override at any
> > >> point of time, depending on his/her need, and that's it! I guess I still
> > >> don't understand your user case here, why device / driver default is of
> > >> such importance.
> > > I've explained that, and I understand your argument is why default value is needed,
> > > and users should be able to adjust them, right?
> > Sounds about right. You'll soon realize that there's no perfect default
> > that could work with everyone - in that it's just a static value to
> > begin with, no matter whatever initial value the device comes up with,
> > one user or another will come over to you and complain that what is
> > loaded from the device doesn't match the workload they have, so those
> > users are still expected to adjust manually tweaking for their own. As
> > long as it's a tunable that guest user can control and override anytime,
> > I don't feel it too much different what initial config the device would
> > start with.
> 
> Think hard of these, I think there's probably no conflict between the
> points of each.
> 
> Generally, it's very hard to have a default value for virtio-net moderation as
> 
> 1) various implementations of virtio-net
> 2) various setups of virtio-net (from virtual to physical, from 1G to 100G)
> 3) various kind of workloads
> 
> But it doesn't mean the device can not have something by default which
> the vendor thinks it works well for in most cases. And anyhow the
> value could be overridden by the driver if it wants (e.g several Linux
> networking drivers will try to set a good default moderation value
> during probe).
> 
> Thanks

Do we want to just burn a feature bit on this then?
I also think we need to have a way to reset to that
default value. Maybe setting #packets to 0 should do that?


-- 
MST

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Jason Wang]

On Tue, Jun 25, 2024 at 4:06 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Tue, Jun 25, 2024 at 03:53:33PM +0800, Jason Wang wrote:
> > On Sat, Jun 22, 2024 at 7:46 AM Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> > >
> > >
> > >
> > > On 6/20/2024 8:24 PM, Heng Qi wrote:
> > > > On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> > > >>
> > > >> On 6/20/2024 12:40 AM, Heng Qi wrote:
> > > >>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> > > >>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
> > > >>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > >>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> > > >>>>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> > > >>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
> > > >>>>>>>> How we can we make progress with
> > > >>>>>>>> the realease but sure we don't make backwards compat a pain?
> > > >>>>>>>>
> > > >>>>>>>> Ideas?
> > > >>>>>>>>
> > > >>>>>>> There is no functional break with this relaxation.
> > > >>>>>>> Device set some non-zero defaults and driver didn't modify them....
> > > >>>>>>> Anything broken? Unlikely.
> > > >>>> Generally it's inappropriate to leave this decision making to the device
> > > >>>> for what would be the best / most performant default config, as the
> > > >>>> device is generally considered agnostic to the guest load...
> > > >>> Instead, the performance of the virtual machine and the driver depends heavily
> > > >>> on how the device is implemented, just as we have proposed various ways to
> > > >>> offload the data queue in the device to the hardware. The reason why most
> > > >>> devices use software to simulate ctrlq instead of using hardware offload is
> > > >>> that the driver has no requirements for the performance of ctrlq before, that is,
> > > >>> the device implementation is responsible for and meets the driver's performance
> > > >>> requirements.
> > > >> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
> > > >> thought that it was for the debate why we need coalescing on hardware
> > > >> offload device in the first place, instead of reusing event index or
> > > >> similar s/w based notification suppression. I don't doubt the value of
> > > >> coalescing, but how's it relevant to the default value disposition?
> > > >> Generally the default config disposition is not considered to be part of
> > > >> device implementation, especially when it comes to the situation where
> > > >> device can't easily figure out the specific workload to occur in the
> > > >> guest, and there's no perfect single default value that could meet every
> > > >> single performance metric across the board. This is a typical tuning
> > > >> knob left up to the user to adjust, why the device or driver needs to
> > > >> set or load the initial value? The driver just needs to start with
> > > >> certain value, be it 0 or non-zero, which guest user can override at any
> > > >> point of time, depending on his/her need, and that's it! I guess I still
> > > >> don't understand your user case here, why device / driver default is of
> > > >> such importance.
> > > > I've explained that, and I understand your argument is why default value is needed,
> > > > and users should be able to adjust them, right?
> > > Sounds about right. You'll soon realize that there's no perfect default
> > > that could work with everyone - in that it's just a static value to
> > > begin with, no matter whatever initial value the device comes up with,
> > > one user or another will come over to you and complain that what is
> > > loaded from the device doesn't match the workload they have, so those
> > > users are still expected to adjust manually tweaking for their own. As
> > > long as it's a tunable that guest user can control and override anytime,
> > > I don't feel it too much different what initial config the device would
> > > start with.
> >
> > Think hard of these, I think there's probably no conflict between the
> > points of each.
> >
> > Generally, it's very hard to have a default value for virtio-net moderation as
> >
> > 1) various implementations of virtio-net
> > 2) various setups of virtio-net (from virtual to physical, from 1G to 100G)
> > 3) various kind of workloads
> >
> > But it doesn't mean the device can not have something by default which
> > the vendor thinks it works well for in most cases. And anyhow the
> > value could be overridden by the driver if it wants (e.g several Linux
> > networking drivers will try to set a good default moderation value
> > during probe).
> >
> > Thanks
>
> Do we want to just burn a feature bit on this then?

It might be more safe to do that.

> I also think we need to have a way to reset to that
> default value. Maybe setting #packets to 0 should do that?

Or a dedicated command.

Thanks

>
>
> --
> MST
>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Tue, Jun 25, 2024 at 04:13:44PM +0800, Jason Wang wrote:
> > I also think we need to have a way to reset to that
> > default value. Maybe setting #packets to 0 should do that?
> 
> Or a dedicated command.

Kind of seems like an overkill, no?

-- 
MST

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/24/2024 10:56 PM, Parav Pandit wrote:
> Hi Si-Wei,
>
>> From: Si-Wei Liu <si-wei.liu@oracle.com>
>> Sent: Tuesday, June 25, 2024 10:22 AM
>>
>> On 6/21/2024 6:34 PM, Heng Qi wrote:
>>> On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>
>> wrote:
>>>> On 6/20/2024 8:24 PM, Heng Qi wrote:
>>>>> On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>
>> wrote:
>>>>>> On 6/20/2024 12:40 AM, Heng Qi wrote:
>>>>>>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu"<si-
>> wei.liu@oracle.com>  wrote:
>>>>>>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
>>>>>>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S.
>> Tsirkin"<mst@redhat.com>  wrote:
>>>>>>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
>>>>>>>>>>>> From: Michael S. Tsirkin<mst@redhat.com>
>>>>>>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM How we can we make
>>>>>>>>>>>> progress with the realease but sure we don't make backwards
>>>>>>>>>>>> compat a pain?
>>>>>>>>>>>>
>>>>>>>>>>>> Ideas?
>>>>>>>>>>>>
>>>>>>>>>>> There is no functional break with this relaxation.
>>>>>>>>>>> Device set some non-zero defaults and driver didn't modify
>> them....
>>>>>>>>>>> Anything broken? Unlikely.
>>>>>>>> Generally it's inappropriate to leave this decision making to the
>>>>>>>> device for what would be the best / most performant default
>>>>>>>> config, as the device is generally considered agnostic to the guest
>> load...
>>>>>>> Instead, the performance of the virtual machine and the driver
>>>>>>> depends heavily on how the device is implemented, just as we have
>>>>>>> proposed various ways to offload the data queue in the device to
>>>>>>> the hardware. The reason why most devices use software to simulate
>>>>>>> ctrlq instead of using hardware offload is that the driver has no
>>>>>>> requirements for the performance of ctrlq before, that is, the
>>>>>>> device implementation is responsible for and meets the driver's
>> performance requirements.
>>>>>> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
>>>>>> thought that it was for the debate why we need coalescing on
>>>>>> hardware offload device in the first place, instead of reusing
>>>>>> event index or similar s/w based notification suppression. I don't
>>>>>> doubt the value of coalescing, but how's it relevant to the default value
>> disposition?
>>>>>> Generally the default config disposition is not considered to be
>>>>>> part of device implementation, especially when it comes to the
>>>>>> situation where device can't easily figure out the specific
>>>>>> workload to occur in the guest, and there's no perfect single
>>>>>> default value that could meet every single performance metric
>>>>>> across the board. This is a typical tuning knob left up to the user
>>>>>> to adjust, why the device or driver needs to set or load the
>>>>>> initial value? The driver just needs to start with certain value,
>>>>>> be it 0 or non-zero, which guest user can override at any point of
>>>>>> time, depending on his/her need, and that's it! I guess I still
>>>>>> don't understand your user case here, why device / driver default is of
>> such importance.
>>>>> I've explained that, and I understand your argument is why default
>>>>> value is needed, and users should be able to adjust them, right?
>>>> Sounds about right. You'll soon realize that there's no perfect
>>>> default that could work with everyone - in that it's just a static
>>>> value to begin with, no matter whatever initial value the device
>>>> comes up with, one user or another will come over to you and complain
>>>> that what is loaded from the device doesn't match the workload they
>>>> have, so those users are still expected to adjust manually tweaking
>>>> for their own. As long as it's a tunable that guest user can control
>>>> and override anytime, I don't feel it too much different what initial
>>>> config the device would start with.
>>> When you have a large number of customers and they buy your machines,
>>> how many users do you think have experience adjusting this value? More
>> below.
>> This is not what this proposal alone could address, as it just shifts the sheer
>> pain to the device and host admin rather than provide any flexibility.
>> Technically I believe the right solution is to seek adaptive coalescing that
>> doesn't require user to tune anything. Noted adaptive coalescing /
>> suppression could be done from the guest or by the device. For e.g. the s/w
>> based event index based notification suppression could also work with
>> hardware if the event index is somehow propagated through a doorbell
>> register rather than load from host memory over the PCI transport.
>>
>>>>> The default value is working when the user doesn't adjust them.
>>>>> It's not practical to rely entirely on user adjustment,
>>>> That's where the adaptive interrupt moderation framework (e.g. DIM
>>>> for
>>>> Linux) could come to the play, I think?
>>> Last time, this work is only useful if DIM is not enabled. I don't
>>> want to explain it again, I've said it many times to different people
>>> in the history discussion, don't you want to check any history discussion?
>> I did before I replied. I thought both Halil and Michael tried hard to convey is
>> that what you claimed to work by just getting the initial config loaded from
>> device will not be sufficient to address all the performance requirements in
>> broad and general means other than satisfy the goal set by yourself. What I
>> saw is that you kept proving yourself by trying to reference your own use
>> case and design - this not just mixed different things up but also ended up
>> with circular argument that wasted everyone's time unfortunately.
>>
>>>>>     and for devices
>>>>> that serve hundreds of thousands of customers, the device
>>>>> implementation has to be comprehensive.
>>>> Would you mind elaborating the technical difficulty for why the
>>>> device implementation has to be comprehensive to serve hundreds or
>>>> thousands of customers (i.e. adaptive moderation in the device), and
>>>> is there better way to improve the current interface (ctrlq v.s.
>>>> doorbell/MMIO register) that is hardware offload implementation wise.
>>>> I feel people reading the thread without understanding the full
>>>> background would become even more confused as to why it's relevant to
>>>> your proposal, to me it's completely distinct use case or problem area that
>> we are talking about.
>>> Regardless of whether DIM can be enabled by default (it cannot
>>> accelerate all scenarios, right?), therefore, a considerable number of
>>> machines on the cloud vendors' line still have to rely on static
>>> values to provide boot-up performance.
>> Sorry, this is not a good excuse for people to accept a sub-optimal solution.
>> Please talk to the other hardware vendor or read my above reply carefully.
>>
>> -Siwei
>>
>>>    The vendors will also make optimizations to optimize scenarios like
>>> ping-pong, so the boot-up performance is very good.
>>>
>>> Thanks.
>>>
>>>> Thanks,
>>>> -Siwei
>>>>
>>>>> Thanks.
>>>>>
>>>>>>>> Unless the
>>>>>>>> device is specially hard wired to some fixed guest setup that
>>>>>>>> users couldn't change, it doesn't seem logical that the device
>>>>>>>> could derive the best or most performant config on driver's
>>>>>>>> behalf. What if the guest wants best latency for its load but the
>>>>>>>> device just blindlessly guess the guest might prefer throughput
>>>>>>>> friendly that it miserably uses latency impacting non-zero default?
>>>>>>> The device does not want to guess and cannot guess. This patch
>>>>>>> does not force the device to choose a non-zero value, but relaxes
>>>>>>> it to allow the device to choose 0 or non-zero, which is very
>>>>>>> friendly to virtual machines with different performance requirements,
>> right?
>>>>>> I don't understand the friendly part - do you imply your VM users
>>>>>> are on kinda fixed wired setup that they cannot change these
>>>>>> coalescing parameters after driver is loaded? Can the owner of the
>>>>>> VM in control apply certain initial config for the coalescing
>>>>>> parameters to the VM image? Or is it the problem of the guest
>>>>>> driver that doesn't yet expose coalescing parameters to the end
>>>>>> user? Otherwise I would think that guest user should be able to set
>>>>>> parameters accordingly that would best fit the specific performance
>>>>>> requirement of their own. How the device could even help here? I
>>>>>> don't feel there's a lot of value to grant the device or host admin
>>>>>> the flexibility to policy the *best* config on guest user's behalf,
>>>>>> to be honest. And you seem to admit the fact that the default doesn't
>> really matter, be it 0 or non-zero.
>>>>>>>> Could this device side change for the default config regress boot
>>>>>>>> time performance (which may need best latency over throughput)?
>>>>>>> Don't make these assumptions, what if the driver needs better
>> throughput?
>>>>>> There's a misconception here: what we think that driver may need in
>>>>>> terms on performance does actually reflect what guest user would
>>>>>> like to have. Driver cannot read guest user's mind to make the decision,
>> either.
>>>>>> In history there was drastic change in the Linux virtio-net driver
>>>>>> that ever changed the default disposition for XPS (and RPS as well)
>>>>>> from throughput and long-lived connection oriented to concurrency
>>>>>> and short-lived oriented, which regressed a lot of existing setups
>>>>>> that expects sustained throughput and packet rate after kernel
>> upgrade.
>>>>>> Although the occurrence of such drastic change for default
>>>>>> disposition is not so welcome (that is one of the reasons why I
>>>>>> valued consistent initial value and back compatible  behavior), I
>>>>>> don't see people yelling at virtio spec for less flexibility of
>>>>>> offering the default disposition, given that the guest user can
>>>>>> override the config any time with their own tooling or script,
>>>>>> there's no problem at all for them to just set the corresponding config
>> back explicitly to what it was before.
>>>>>>>>>>> And device/driver has better performance, is that a problem?
>> Unlikely.
>>>>>>>> Even for rare case with a hard wired setup, the way to tackle the
>>>>>>>> very problem using device's default is still quite questionable.
>>>>>>>> Usually the mgmt software or network config utility should be
>>>>>>>> equipped with some default value if need be. And we know the
>>>>>>>> guest has the best position to impose the best / most performant
>>>>>>>> config for its own load. What is the issue or use case that this
>>>>>>>> initial config couldn't be applied by the guest mgmt software
>>>>>>>> ahead but has to resort to the device to load some default (which
>>>>>>>> is odd and irrelevant to any guest load), before the interface is
>> brought up for operation i.e. performing I/O?
>>>>>>> Use cases are everywhere, Alibaba Cloud, MLX and all other modern
>>>>>>> network cards have a default value that is not 0.
>>>>>> You seems to be referencing your own setup basically, and the
>>>>>> question is still left answered - why the initial config can't be
>>>>>> done through the mgmt software or network config utility within the
>> guest?
>>>>>>> (0 is actually a kind of default value)
>>>>>>>
>>>>>>>>> Sorry, my vacation just ended.
>>>>>>>>>
>>>>>>>>>> Yes, it is possible. Driver can cache values it sets and never
>>>>>>>>>> query device with get.
>>>>>>>>> Don't we already have a lot of behaviors to drive queries from
>> devices?
>>>>>>>>> RSS context, device stats.
>>>>>>>>>
>>>>>>>>>> Before anything is set, driver will report incorrect values.
>>>>>>>>> Devices that are widely supported and supported by good
>>>>>>>>> practices should have any initialization value. Just reporting 0
>>>>>>>>> is incorrect value. Although the spec now says so.
>>>>>>>> I don't have an aligned view here, sorry. As I recall having 0 as
>>>>>>>> the default is just to keep device started in a state where
>>>>>>>> coalescing is disabled, so it's backward compatible and
>>>>>>>> consistent with a non-coalescing supporting driver - such that it
>>>>>>>> won't yield surprising effect (for e.g. regressed latency)
>>>>>>>> inadvertently after user's getting driver software upgraded.
>>>>>>>> Unlike the other virtio-net features that could 100% improve
>>>>>>>> performance in all aspect, this coalescing feature is more of a
>>>>>>>> performance tuning knob that may improve performance metrics
>>>>>>>> (such as cpu usage or throughput) of one dimension while demoting
>> the others (such as latency, jitter or connection rate) from the equation.
>>>>>>>> That said, there's not a single and fixed set of default config
>>>>>>>> that device could supply which is able to satisfy all kind of guest load.
>>>>>>>> Rather than rely on the device to offer a matching default for
>>>>>>>> driver (which I think it's technically wrong), I'd lean to having
>>>>>>>> guest software or network utility to apply the initial config for
>>>>>>>> the guest, where they should have best knowledge for the specific
>>>>>>>> guest workload than what device could do.
>>>>>>> Before this feature, a good device implementation should also
>>>>>>> support coalescing (of course we don't necessarily assume it has
>> coalescing).
>>>>>> Again, I don't doubt the value of supporting coalescing.
>>>>>>
>>>>>>>      In addition, virtual
>>>>>>> machines that tend to favor latency and throughput exist. If the
>>>>>>> device supported by the manufacturer needs to provide a
>>>>>>> low-latency virtual machine, please continue to keep the default value
>> of 0.
>>>>>> No, that's not what I was asking. There's no such requirement for
>>>>>> any vendor to provide a low-latency or high throughput VM. The more
>>>>>> general use case is - the setup for real world workload might just
>>>>>> be too complex that end users would prefer low-latency on some
>>>>>> virtual NIC or even some specific queues, while the other queues of
>>>>>> a virtual NIC, or reset virtual NIC might have very different
>>>>>> dispositions. Due to the needs and dynamics of workload scaling up
>>>>>> & down, they might have more or less queues or virtual NICs to
>>>>>> configure, so these disposition would need to be readjusted at any
>>>>>> point of time, for which there's no easy way for device to adapt to
>>>>>> easily. The guest user should have best knowledge for the specific
>>>>>> guest workload and setup than what device could/should offer.
>>>>>>
>>>>>>>>>> What will break as a result? Hard to predict.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Given the patch is big, I am inclined to say it just should use
>>>>>>>>>> a feature bit.
>>>>>>>>> This doesn't seem to break anything in my opinion, we just told
>>>>>>>>> the device that now you can set more values.
>>>>>>>> Another thing this could break is live migration between devices
>>>>>>>> with varied default. How do you make sure the guest doesn't rely
>>>>>>>> on some default from the source device, while on the destination
>>>>>>>> it just doesn't get the same default coalescingjjj value? To get
>>>>>>>> rid of this side effect the guest would still need to apply the
>>>>>>>> initial config for its own, anyway... Which eventually would
>>>>>>>> render this proposal with arbitrary default rather pointless.
>>>>>>> I don't quite understand why this would affect hot migration, the
>>>>>>> values would be migrated over.
>>>>>> Then I don't see this has been discussed (wouldn't the initial
>>>>>> value be part of virtio device state to migrate over?) in the
>>>>>> thread or described in the proposed text. And your proposed
>>>>>> per-queue coalescing parameters (default plus current value) would
>>>>>> have to be described too as part of virtio-net device state, so that
>> people don't misunderstand your proposal.
>>>>>> Thanks,
>>>>>> -Siwei
>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> -Siwei
>>>>>>>>> Using new feature bits does not seem necessary.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> MST
>>>>>>>>>>
> I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
>
> Flow is:
> 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
>
> 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
>
> 4. the virtio-net driver even after negotiating VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled.
> vi->rq[queue].dim_enabled = false by default.
>
> 5. The virtio net driver enables dim only on user request via ethtool.
>
> So here, just because one is using a new driver and new device, it may get lower performance.
Impossible, older driver / device doesn't have coalescing advertise / 
enabled, and new device & new driver with coalescing negotiated starts 
safe with 0 with coalescing disabled effectively, how can it get lower 
performance?

>
> Do we agree to above problem description?
This narrative has two problems:

- The term "lower performance" is misleading and inaccurate, the fact or 
implication of coalescing is that it yields worse latency, causing 
additional delay and jitter that would hurt quite a lot of real world 
workload that is latency sensitive.  So, rather than say coalescing 
helps performance, it should clearly tell the truth that the coalescing 
parameter is no more than a tuning knob that has performance 
impact/implication, being positive or negative it is very subject to the 
guest workload. So the default disposition or initial parameters loaded 
from device just reflects the preference of the host admin/ device 
owner/ cloud vendor of one's own, for e.g. it is optimized for 
throughput oriented load versus latency oriented. Be noted, generally 
end users don't care what coalescing is about, they do not connect 
performance to "throughput" or "bandwidth" as that in your mind.

- The specific problem or use case is narrow scoped and specifically 
tailored (e.g. for a hard wired guest setup that is known to the device 
owner / host admin already), while for some reason the advantage of the 
proposed solution is overly exaggerated. For one hand, the ethtool 
interface is very Linux specific that exposed quite a lit lower level 
device implementation details to normal end users. In case of static and 
fixed configuration, it definitely requires dedicated knowledge of the 
user to fine tune for their load. For the other hand, if we are talking 
about normal users who don't have such knowledge AND don't care so much 
about performance, then whatever coalescing parameter will just work 
with them, and they don't really care about the default. However, For 
those normal users who care about performance but still care about 
getting the best or well balanced performance (N.B. I say performance in 
general, not limited to only throughput or latency in one dimension) for 
their load, what options do they have? IMHO what they really like is to 
lean on guest/driver/device to auto tune by itself rather than for them 
to take the deep learning curve to fiddle with the low level device 
parameters manually by themselves, right?

If you agree with the above, I guess we could proceed with unlocking the 
other good options, as listed below. To me, the question I got from the 
above problem statement but still left unanswered is that, what makes 
the cloud vendor think that they could have better posture to decide 
certain default values for guest users' behalf? Why it is a MUST to do 
so for every cloud vendor in general?

Thanks,
-Siwei

> If no, I likely missed something in the long discussion and need to read all of it. :(
>
> If yes, few solutions are:
> 1. A user intervention is must to avoid this regression. The user in the guest VM must enable DIM if the device supports it.
> Very hard to do this plumbing.
>
> 2. A net driver enables the DIM by default as long as VIRTIO_NET_F_VQ_NOTF_COAL is negotiated.
> (without ethtool setting)
>
> 3. A device relaxes the limitation and continue to apply the coalescing, until driver overrides it.
>
> In my humble opinion, Heng is solving it using option #3, that tends to work with existing and future drivers who may/may not enable the DIM.
>
>
>
>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Halil Pasic]

On Tue, 25 Jun 2024 18:14:15 -0700
Si-Wei Liu <si-wei.liu@oracle.com> wrote:

> On 6/24/2024 10:56 PM, Parav Pandit wrote:
[..]
> > I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> > The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
> >
> > Flow is:
> > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> > 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
> >
> > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.

I have certainly understood this differently. When
VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).

If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
to/MUST do the coalescing according to the parameters as described by
the virtio spec.

Michael, Jason: Can you guys weigh in on this?

> >
> > 4. the virtio-net driver even after negotiating VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled.
> > vi->rq[queue].dim_enabled = false by default.
> >
> > 5. The virtio net driver enables dim only on user request via ethtool.
> >
> > So here, just because one is using a new driver and new device, it may get lower performance.  
> Impossible, older driver / device doesn't have coalescing advertise / 
> enabled, and new device & new driver with coalescing negotiated starts 
> safe with 0 with coalescing disabled effectively, how can it get lower 
> performance?

Si-Wei and Parav, I think it is the other way around. Without the
change proposed here, with no user/management software intervention
assumed, we are guaranteed to not see any performance regression
but also no performance benefit because of the coalescing mechanism,
because with zeros as default it is essentially dormant.

With the change proposed here however, we open up the possibility
to see either a performance improvement or a performance degradation
depending on the defaults chosen by the device and the workload (sill
assuming new device & new driver & no intervention by management
software).


> 
> >
> > Do we agree to above problem description?  
> This narrative has two problems:
> 
> - The term "lower performance" is misleading and inaccurate, the fact or 
> implication of coalescing is that it yields worse latency, causing 
> additional delay and jitter that would hurt quite a lot of real world 
> workload that is latency sensitive.  So, rather than say coalescing 
> helps performance, it should clearly tell the truth that the coalescing 
> parameter is no more than a tuning knob that has performance 
> impact/implication, being positive or negative it is very subject to the 
> guest workload. So the default disposition or initial parameters loaded 
> from device just reflects the preference of the host admin/ device 
> owner/ cloud vendor of one's own, for e.g. it is optimized for 
> throughput oriented load versus latency oriented. Be noted, generally 
> end users don't care what coalescing is about, they do not connect 
> performance to "throughput" or "bandwidth" as that in your mind.
> 

Si-Wei, I sympathies with your argument but I'm willing to tolerate this
change nevertheless.

Yes the device and via the device whoever controls these parameters can
shoot the guest in the foot, if the guest admin is not going out of his
way to do something about controlling coalescing.

But via the same mechanism we could end up in a better place as well.

Because I assume that the device and its "admin" are at least trying to
do good, I'm not straight out against this proposal, as long nothing
wonky happens with the code which is already out there, in a sense of a
split brain scenario where Linux thinks those values are 0 but they
aren't. Unfortunately can't tell you how we stand on this.

[..]
> > If no, I likely missed something in the long discussion and need to read all of it. :(
> >
> > If yes, few solutions are:
> > 1. A user intervention is must to avoid this regression. The user in the guest VM must enable DIM if the device supports it.

I don't think burdening up the users with avoiding regressions is a good
idea.

Regards,
Halil

> > Very hard to do this plumbing.
> >
> > 2. A net driver enables the DIM by default as long as VIRTIO_NET_F_VQ_NOTF_COAL is negotiated.
> > (without ethtool setting)
> >
> > 3. A device relaxes the limitation and continue to apply the coalescing, until driver overrides it.
> >
> > In my humble opinion, Heng is solving it using option #3, that tends to work with existing and future drivers who may/may not enable the DIM.
> >
> >
> >
> >  
> 

RE: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Parav Pandit]

Hi Halil,

> From: Halil Pasic <pasic@linux.ibm.com>
> Sent: Thursday, June 27, 2024 4:08 PM
> 
> On Tue, 25 Jun 2024 18:14:15 -0700
> Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> 
> > On 6/24/2024 10:56 PM, Parav Pandit wrote:
> [..]
> > > I saw the need of this proposal slightly differently in the discussion with
> Heng in v4.
> > > The way I understood is, proposed relaxation enables below Linux driver
> flow to work as equally as without device offering
> VIRTIO_NET_F_VQ_NOTF_COAL.
> > >
> > > Flow is:
> > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL 2. The
> > > virtio-net driver negotiated VIRTNET_FEATURES that has
> > > VIRTIO_NET_F_VQ_NOTF_COAL
> > >
> > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not
> applying any coalescing on the VQ, in a good hope that driver will perform VQ
> notification coalescing.
> 
> I have certainly understood this differently. When
> VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not
> supposed/allowed to do any interrupt coalescing (notification suppression
> may still apply).
> 
If device does not do interrupt coalescing, it will be worse because of interrupt per packet.

> If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> to/MUST do the coalescing according to the parameters as described by the
> virtio spec.
> 
Yes.

> Michael, Jason: Can you guys weigh in on this?
> 
> > >
> > > 4. the virtio-net driver even after negotiating
> VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled.
> > > vi->rq[queue].dim_enabled = false by default.
> > >
> > > 5. The virtio net driver enables dim only on user request via ethtool.
> > >
> > > So here, just because one is using a new driver and new device, it may get
> lower performance.
> > Impossible, older driver / device doesn't have coalescing advertise /
> > enabled, and new device & new driver with coalescing negotiated starts
> > safe with 0 with coalescing disabled effectively, how can it get lower
> > performance?
> 
> Si-Wei and Parav, I think it is the other way around. Without the change
> proposed here, with no user/management software intervention assumed,
> we are guaranteed to not see any performance regression but also no
> performance benefit because of the coalescing mechanism, because with
> zeros as default it is essentially dormant.
> 
I will avoid quoting the term "performance regression".
I believe there will be behavior regression.
For example, devices do apply notification suppression today when VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated.
Not doing so will not allow virtio-net to come this far.

By setting them to zero, device stops applying what it was doing until now.
And it results in "behavior regression".
One can argue that now the latency is best because very first packet will trigger a notification.

The proposed change in my humble view is allowing the device to restore its behavior when driver has NOT configured notification params and yet negotiated VIRTIO_NET_F_VQ_NOTF_COAL.

> With the change proposed here however, we open up the possibility to see
> either a performance improvement or a performance degradation depending
> on the defaults chosen by the device and the workload (sill assuming new
> device & new driver & no intervention by management software).
> 
Exactly. I agree with you.
It seems fine because it restores the behavior. Whether that behavior is good/bad/average - it is upto the device implementation.

> 
> >
> > >
> > > Do we agree to above problem description?
> > This narrative has two problems:
> >
> > - The term "lower performance" is misleading and inaccurate, the fact
> > or implication of coalescing is that it yields worse latency, causing
> > additional delay and jitter that would hurt quite a lot of real world
> > workload that is latency sensitive.  So, rather than say coalescing
> > helps performance, it should clearly tell the truth that the
> > coalescing parameter is no more than a tuning knob that has
> > performance impact/implication, being positive or negative it is very
> > subject to the guest workload. So the default disposition or initial
> > parameters loaded from device just reflects the preference of the host
> > admin/ device owner/ cloud vendor of one's own, for e.g. it is
> > optimized for throughput oriented load versus latency oriented. Be
> > noted, generally end users don't care what coalescing is about, they
> > do not connect performance to "throughput" or "bandwidth" as that in your
> mind.
> >
> 
> Si-Wei, I sympathies with your argument but I'm willing to tolerate this change
> nevertheless.
> 
> Yes the device and via the device whoever controls these parameters can
> shoot the guest in the foot, if the guest admin is not going out of his way to do
> something about controlling coalescing.
> 
Right.

> But via the same mechanism we could end up in a better place as well.
> 
Right if not better, at least it brings to the same place as where "this feature is not offered".

In the second example for txq, current Linux dim does not have netdim for TX. And negotiating this feature (and not doing actual TXQ level moderation) results in zero tx moderation.

All of these would have been fine, if the Linux net driver would have implemented and enabled netdim whenever the feature is negotiated.

Driver negotiating and driver not controlling is the main issue.
The device is eagerly hoping that driver will control, but it doesn't.

The proposed change overcomes this limitation, that until driver controls, it let it control the moderation.
Once driver takes over (by issuing first command), device stops applying its own logic.

> Because I assume that the device and its "admin" are at least trying to do
> good, I'm not straight out against this proposal, as long nothing wonky
> happens with the code which is already out there, in a sense of a split brain
> scenario where Linux thinks those values are 0 but they aren't. Unfortunately
> can't tell you how we stand on this.
> 
> [..]
> > > If no, I likely missed something in the long discussion and need to
> > > read all of it. :(
> > >
> > > If yes, few solutions are:
> > > 1. A user intervention is must to avoid this regression. The user in the guest
> VM must enable DIM if the device supports it.
> 
> I don't think burdening up the users with avoiding regressions is a good idea.
> 
The user is the vnet driver who doesnt even has netdim implemented for the TX.
So device is in dark, should it offer this feature or not?
If it offers (without the proposed change), there is behavior regression until the driver fix is done.

Hence, I agree to Halil's point.
We should relax the requirement to restore the behavior.
New feature bit won't help the behavior restoration..

> Regards,
> Halil
> 
> > > Very hard to do this plumbing.
> > >
> > > 2. A net driver enables the DIM by default as long as
> VIRTIO_NET_F_VQ_NOTF_COAL is negotiated.
> > > (without ethtool setting)
> > >
> > > 3. A device relaxes the limitation and continue to apply the coalescing, until
> driver overrides it.
> > >
> > > In my humble opinion, Heng is solving it using option #3, that tends to
> work with existing and future drivers who may/may not enable the DIM.
> > >
> > >
> > >
> > >
> >

RE: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Parav Pandit]

> From: Si-Wei Liu <si-wei.liu@oracle.com>
> Sent: Wednesday, June 26, 2024 6:44 AM
> 
> On 6/24/2024 10:56 PM, Parav Pandit wrote:
> > Hi Si-Wei,
> >
> >> From: Si-Wei Liu <si-wei.liu@oracle.com>
> >> Sent: Tuesday, June 25, 2024 10:22 AM
> >>
> >> On 6/21/2024 6:34 PM, Heng Qi wrote:
> >>> On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei
> >>> Liu"<si-wei.liu@oracle.com>
> >> wrote:
> >>>> On 6/20/2024 8:24 PM, Heng Qi wrote:
> >>>>> On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei
> >>>>> Liu"<si-wei.liu@oracle.com>
> >> wrote:
> >>>>>> On 6/20/2024 12:40 AM, Heng Qi wrote:
> >>>>>>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu"<si-
> >> wei.liu@oracle.com>  wrote:
> >>>>>>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
> >>>>>>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S.
> >> Tsirkin"<mst@redhat.com>  wrote:
> >>>>>>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> >>>>>>>>>>>> From: Michael S. Tsirkin<mst@redhat.com>
> >>>>>>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM How we can we make
> >>>>>>>>>>>> progress with the realease but sure we don't make backwards
> >>>>>>>>>>>> compat a pain?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ideas?
> >>>>>>>>>>>>
> >>>>>>>>>>> There is no functional break with this relaxation.
> >>>>>>>>>>> Device set some non-zero defaults and driver didn't modify
> >> them....
> >>>>>>>>>>> Anything broken? Unlikely.
> >>>>>>>> Generally it's inappropriate to leave this decision making to
> >>>>>>>> the device for what would be the best / most performant default
> >>>>>>>> config, as the device is generally considered agnostic to the
> >>>>>>>> guest
> >> load...
> >>>>>>> Instead, the performance of the virtual machine and the driver
> >>>>>>> depends heavily on how the device is implemented, just as we
> >>>>>>> have proposed various ways to offload the data queue in the
> >>>>>>> device to the hardware. The reason why most devices use software
> >>>>>>> to simulate ctrlq instead of using hardware offload is that the
> >>>>>>> driver has no requirements for the performance of ctrlq before,
> >>>>>>> that is, the device implementation is responsible for and meets
> >>>>>>> the driver's
> >> performance requirements.
> >>>>>> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
> >>>>>> thought that it was for the debate why we need coalescing on
> >>>>>> hardware offload device in the first place, instead of reusing
> >>>>>> event index or similar s/w based notification suppression. I
> >>>>>> don't doubt the value of coalescing, but how's it relevant to the
> >>>>>> default value
> >> disposition?
> >>>>>> Generally the default config disposition is not considered to be
> >>>>>> part of device implementation, especially when it comes to the
> >>>>>> situation where device can't easily figure out the specific
> >>>>>> workload to occur in the guest, and there's no perfect single
> >>>>>> default value that could meet every single performance metric
> >>>>>> across the board. This is a typical tuning knob left up to the
> >>>>>> user to adjust, why the device or driver needs to set or load the
> >>>>>> initial value? The driver just needs to start with certain value,
> >>>>>> be it 0 or non-zero, which guest user can override at any point
> >>>>>> of time, depending on his/her need, and that's it! I guess I
> >>>>>> still don't understand your user case here, why device / driver
> >>>>>> default is of
> >> such importance.
> >>>>> I've explained that, and I understand your argument is why default
> >>>>> value is needed, and users should be able to adjust them, right?
> >>>> Sounds about right. You'll soon realize that there's no perfect
> >>>> default that could work with everyone - in that it's just a static
> >>>> value to begin with, no matter whatever initial value the device
> >>>> comes up with, one user or another will come over to you and
> >>>> complain that what is loaded from the device doesn't match the
> >>>> workload they have, so those users are still expected to adjust
> >>>> manually tweaking for their own. As long as it's a tunable that
> >>>> guest user can control and override anytime, I don't feel it too
> >>>> much different what initial config the device would start with.
> >>> When you have a large number of customers and they buy your
> >>> machines, how many users do you think have experience adjusting this
> >>> value? More
> >> below.
> >> This is not what this proposal alone could address, as it just shifts
> >> the sheer pain to the device and host admin rather than provide any
> flexibility.
> >> Technically I believe the right solution is to seek adaptive
> >> coalescing that doesn't require user to tune anything. Noted adaptive
> >> coalescing / suppression could be done from the guest or by the
> >> device. For e.g. the s/w based event index based notification
> >> suppression could also work with hardware if the event index is
> >> somehow propagated through a doorbell register rather than load from
> host memory over the PCI transport.
> >>
> >>>>> The default value is working when the user doesn't adjust them.
> >>>>> It's not practical to rely entirely on user adjustment,
> >>>> That's where the adaptive interrupt moderation framework (e.g. DIM
> >>>> for
> >>>> Linux) could come to the play, I think?
> >>> Last time, this work is only useful if DIM is not enabled. I don't
> >>> want to explain it again, I've said it many times to different
> >>> people in the history discussion, don't you want to check any history
> discussion?
> >> I did before I replied. I thought both Halil and Michael tried hard
> >> to convey is that what you claimed to work by just getting the
> >> initial config loaded from device will not be sufficient to address
> >> all the performance requirements in broad and general means other
> >> than satisfy the goal set by yourself. What I saw is that you kept
> >> proving yourself by trying to reference your own use case and design
> >> - this not just mixed different things up but also ended up with circular
> argument that wasted everyone's time unfortunately.
> >>
> >>>>>     and for devices
> >>>>> that serve hundreds of thousands of customers, the device
> >>>>> implementation has to be comprehensive.
> >>>> Would you mind elaborating the technical difficulty for why the
> >>>> device implementation has to be comprehensive to serve hundreds or
> >>>> thousands of customers (i.e. adaptive moderation in the device),
> >>>> and is there better way to improve the current interface (ctrlq v.s.
> >>>> doorbell/MMIO register) that is hardware offload implementation wise.
> >>>> I feel people reading the thread without understanding the full
> >>>> background would become even more confused as to why it's relevant
> >>>> to your proposal, to me it's completely distinct use case or
> >>>> problem area that
> >> we are talking about.
> >>> Regardless of whether DIM can be enabled by default (it cannot
> >>> accelerate all scenarios, right?), therefore, a considerable number
> >>> of machines on the cloud vendors' line still have to rely on static
> >>> values to provide boot-up performance.
> >> Sorry, this is not a good excuse for people to accept a sub-optimal solution.
> >> Please talk to the other hardware vendor or read my above reply carefully.
> >>
> >> -Siwei
> >>
> >>>    The vendors will also make optimizations to optimize scenarios
> >>> like ping-pong, so the boot-up performance is very good.
> >>>
> >>> Thanks.
> >>>
> >>>> Thanks,
> >>>> -Siwei
> >>>>
> >>>>> Thanks.
> >>>>>
> >>>>>>>> Unless the
> >>>>>>>> device is specially hard wired to some fixed guest setup that
> >>>>>>>> users couldn't change, it doesn't seem logical that the device
> >>>>>>>> could derive the best or most performant config on driver's
> >>>>>>>> behalf. What if the guest wants best latency for its load but
> >>>>>>>> the device just blindlessly guess the guest might prefer
> >>>>>>>> throughput friendly that it miserably uses latency impacting non-
> zero default?
> >>>>>>> The device does not want to guess and cannot guess. This patch
> >>>>>>> does not force the device to choose a non-zero value, but
> >>>>>>> relaxes it to allow the device to choose 0 or non-zero, which is
> >>>>>>> very friendly to virtual machines with different performance
> >>>>>>> requirements,
> >> right?
> >>>>>> I don't understand the friendly part - do you imply your VM users
> >>>>>> are on kinda fixed wired setup that they cannot change these
> >>>>>> coalescing parameters after driver is loaded? Can the owner of
> >>>>>> the VM in control apply certain initial config for the coalescing
> >>>>>> parameters to the VM image? Or is it the problem of the guest
> >>>>>> driver that doesn't yet expose coalescing parameters to the end
> >>>>>> user? Otherwise I would think that guest user should be able to
> >>>>>> set parameters accordingly that would best fit the specific
> >>>>>> performance requirement of their own. How the device could even
> >>>>>> help here? I don't feel there's a lot of value to grant the
> >>>>>> device or host admin the flexibility to policy the *best* config
> >>>>>> on guest user's behalf, to be honest. And you seem to admit the
> >>>>>> fact that the default doesn't
> >> really matter, be it 0 or non-zero.
> >>>>>>>> Could this device side change for the default config regress
> >>>>>>>> boot time performance (which may need best latency over
> throughput)?
> >>>>>>> Don't make these assumptions, what if the driver needs better
> >> throughput?
> >>>>>> There's a misconception here: what we think that driver may need
> >>>>>> in terms on performance does actually reflect what guest user
> >>>>>> would like to have. Driver cannot read guest user's mind to make
> >>>>>> the decision,
> >> either.
> >>>>>> In history there was drastic change in the Linux virtio-net
> >>>>>> driver that ever changed the default disposition for XPS (and RPS
> >>>>>> as well) from throughput and long-lived connection oriented to
> >>>>>> concurrency and short-lived oriented, which regressed a lot of
> >>>>>> existing setups that expects sustained throughput and packet rate
> >>>>>> after kernel
> >> upgrade.
> >>>>>> Although the occurrence of such drastic change for default
> >>>>>> disposition is not so welcome (that is one of the reasons why I
> >>>>>> valued consistent initial value and back compatible  behavior), I
> >>>>>> don't see people yelling at virtio spec for less flexibility of
> >>>>>> offering the default disposition, given that the guest user can
> >>>>>> override the config any time with their own tooling or script,
> >>>>>> there's no problem at all for them to just set the corresponding
> >>>>>> config
> >> back explicitly to what it was before.
> >>>>>>>>>>> And device/driver has better performance, is that a problem?
> >> Unlikely.
> >>>>>>>> Even for rare case with a hard wired setup, the way to tackle
> >>>>>>>> the very problem using device's default is still quite questionable.
> >>>>>>>> Usually the mgmt software or network config utility should be
> >>>>>>>> equipped with some default value if need be. And we know the
> >>>>>>>> guest has the best position to impose the best / most
> >>>>>>>> performant config for its own load. What is the issue or use
> >>>>>>>> case that this initial config couldn't be applied by the guest
> >>>>>>>> mgmt software ahead but has to resort to the device to load
> >>>>>>>> some default (which is odd and irrelevant to any guest load),
> >>>>>>>> before the interface is
> >> brought up for operation i.e. performing I/O?
> >>>>>>> Use cases are everywhere, Alibaba Cloud, MLX and all other
> >>>>>>> modern network cards have a default value that is not 0.
> >>>>>> You seems to be referencing your own setup basically, and the
> >>>>>> question is still left answered - why the initial config can't be
> >>>>>> done through the mgmt software or network config utility within
> >>>>>> the
> >> guest?
> >>>>>>> (0 is actually a kind of default value)
> >>>>>>>
> >>>>>>>>> Sorry, my vacation just ended.
> >>>>>>>>>
> >>>>>>>>>> Yes, it is possible. Driver can cache values it sets and
> >>>>>>>>>> never query device with get.
> >>>>>>>>> Don't we already have a lot of behaviors to drive queries from
> >> devices?
> >>>>>>>>> RSS context, device stats.
> >>>>>>>>>
> >>>>>>>>>> Before anything is set, driver will report incorrect values.
> >>>>>>>>> Devices that are widely supported and supported by good
> >>>>>>>>> practices should have any initialization value. Just reporting
> >>>>>>>>> 0 is incorrect value. Although the spec now says so.
> >>>>>>>> I don't have an aligned view here, sorry. As I recall having 0
> >>>>>>>> as the default is just to keep device started in a state where
> >>>>>>>> coalescing is disabled, so it's backward compatible and
> >>>>>>>> consistent with a non-coalescing supporting driver - such that
> >>>>>>>> it won't yield surprising effect (for e.g. regressed latency)
> >>>>>>>> inadvertently after user's getting driver software upgraded.
> >>>>>>>> Unlike the other virtio-net features that could 100% improve
> >>>>>>>> performance in all aspect, this coalescing feature is more of a
> >>>>>>>> performance tuning knob that may improve performance metrics
> >>>>>>>> (such as cpu usage or throughput) of one dimension while
> >>>>>>>> demoting
> >> the others (such as latency, jitter or connection rate) from the equation.
> >>>>>>>> That said, there's not a single and fixed set of default config
> >>>>>>>> that device could supply which is able to satisfy all kind of guest
> load.
> >>>>>>>> Rather than rely on the device to offer a matching default for
> >>>>>>>> driver (which I think it's technically wrong), I'd lean to
> >>>>>>>> having guest software or network utility to apply the initial
> >>>>>>>> config for the guest, where they should have best knowledge for
> >>>>>>>> the specific guest workload than what device could do.
> >>>>>>> Before this feature, a good device implementation should also
> >>>>>>> support coalescing (of course we don't necessarily assume it has
> >> coalescing).
> >>>>>> Again, I don't doubt the value of supporting coalescing.
> >>>>>>
> >>>>>>>      In addition, virtual
> >>>>>>> machines that tend to favor latency and throughput exist. If the
> >>>>>>> device supported by the manufacturer needs to provide a
> >>>>>>> low-latency virtual machine, please continue to keep the default
> >>>>>>> value
> >> of 0.
> >>>>>> No, that's not what I was asking. There's no such requirement for
> >>>>>> any vendor to provide a low-latency or high throughput VM. The
> >>>>>> more general use case is - the setup for real world workload
> >>>>>> might just be too complex that end users would prefer low-latency
> >>>>>> on some virtual NIC or even some specific queues, while the other
> >>>>>> queues of a virtual NIC, or reset virtual NIC might have very
> >>>>>> different dispositions. Due to the needs and dynamics of workload
> >>>>>> scaling up & down, they might have more or less queues or virtual
> >>>>>> NICs to configure, so these disposition would need to be
> >>>>>> readjusted at any point of time, for which there's no easy way
> >>>>>> for device to adapt to easily. The guest user should have best
> >>>>>> knowledge for the specific guest workload and setup than what device
> could/should offer.
> >>>>>>
> >>>>>>>>>> What will break as a result? Hard to predict.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Given the patch is big, I am inclined to say it just should
> >>>>>>>>>> use a feature bit.
> >>>>>>>>> This doesn't seem to break anything in my opinion, we just
> >>>>>>>>> told the device that now you can set more values.
> >>>>>>>> Another thing this could break is live migration between
> >>>>>>>> devices with varied default. How do you make sure the guest
> >>>>>>>> doesn't rely on some default from the source device, while on
> >>>>>>>> the destination it just doesn't get the same default
> >>>>>>>> coalescingjjj value? To get rid of this side effect the guest
> >>>>>>>> would still need to apply the initial config for its own,
> >>>>>>>> anyway... Which eventually would render this proposal with
> arbitrary default rather pointless.
> >>>>>>> I don't quite understand why this would affect hot migration,
> >>>>>>> the values would be migrated over.
> >>>>>> Then I don't see this has been discussed (wouldn't the initial
> >>>>>> value be part of virtio device state to migrate over?) in the
> >>>>>> thread or described in the proposed text. And your proposed
> >>>>>> per-queue coalescing parameters (default plus current value)
> >>>>>> would have to be described too as part of virtio-net device
> >>>>>> state, so that
> >> people don't misunderstand your proposal.
> >>>>>> Thanks,
> >>>>>> -Siwei
> >>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> -Siwei
> >>>>>>>>> Using new feature bits does not seem necessary.
> >>>>>>>>>
> >>>>>>>>> Thanks.
> >>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> MST
> >>>>>>>>>>
> > I saw the need of this proposal slightly differently in the discussion with Heng
> in v4.
> > The way I understood is, proposed relaxation enables below Linux driver
> flow to work as equally as without device offering
> VIRTIO_NET_F_VQ_NOTF_COAL.
> >
> > Flow is:
> > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL 2. The
> > virtio-net driver negotiated VIRTNET_FEATURES that has
> > VIRTIO_NET_F_VQ_NOTF_COAL
> >
> > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not
> applying any coalescing on the VQ, in a good hope that driver will perform VQ
> notification coalescing.
> >
> > 4. the virtio-net driver even after negotiating
> VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled.
> > vi->rq[queue].dim_enabled = false by default.
> >
> > 5. The virtio net driver enables dim only on user request via ethtool.
> >
> > So here, just because one is using a new driver and new device, it may get
> lower performance.
> Impossible, older driver / device doesn't have coalescing advertise / enabled,
> and new device & new driver with coalescing negotiated starts safe with 0 with
> coalescing disabled effectively, how can it get lower performance?
The current Linux driver does not even have the code for netdim for tx, not enabling netdim by default.
And enabling netdim for rx requires user space.

So in it current form, the device implementation wont apply any notification suppression (which otherwise it did in absence of this feature).

> 
> >
> > Do we agree to above problem description?
> This narrative has two problems:
> 
> - The term "lower performance" is misleading and inaccurate, the fact or
> implication of coalescing is that it yields worse latency, causing additional delay
> and jitter that would hurt quite a lot of real world workload that is latency
> sensitive.  So, rather than say coalescing helps performance, it should clearly
> tell the truth that the coalescing parameter is no more than a tuning knob that
> has performance impact/implication, being positive or negative it is very
> subject to the guest workload. So the default disposition or initial parameters
> loaded from device just reflects the preference of the host admin/ device
> owner/ cloud vendor of one's own, for e.g. it is optimized for throughput
> oriented load versus latency oriented. Be noted, generally end users don't care
> what coalescing is about, they do not connect performance to "throughput"
> or "bandwidth" as that in your mind.
> 
> - The specific problem or use case is narrow scoped and specifically tailored
> (e.g. for a hard wired guest setup that is known to the device owner / host
> admin already), while for some reason the advantage of the proposed solution
> is overly exaggerated. For one hand, the ethtool interface is very Linux specific
> that exposed quite a lit lower level device implementation details to normal
> end users. In case of static and fixed configuration, it definitely requires
> dedicated knowledge of the user to fine tune for their load. For the other
> hand, if we are talking about normal users who don't have such knowledge
> AND don't care so much about performance, then whatever coalescing
> parameter will just work with them, and they don't really care about the
> default. However, For those normal users who care about performance but
> still care about getting the best or well balanced performance (N.B. I say
> performance in general, not limited to only throughput or latency in one
> dimension) for their load, what options do they have? IMHO what they really
> like is to lean on guest/driver/device to auto tune by itself rather than for them
> to take the deep learning curve to fiddle with the low level device parameters
> manually by themselves, right?
> 
> If you agree with the above, I guess we could proceed with unlocking the
> other good options, as listed below. To me, the question I got from the above
> problem statement but still left unanswered is that, what makes the cloud
> vendor think that they could have better posture to decide certain default
> values for guest users' behalf? Why it is a MUST to do so for every cloud
> vendor in general?
> 
It likely does not.
The intent is to restore the behavior to same as what it used to be when VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated.

> Thanks,
> -Siwei
> 
> > If no, I likely missed something in the long discussion and need to
> > read all of it. :(
> >
> > If yes, few solutions are:
> > 1. A user intervention is must to avoid this regression. The user in the guest
> VM must enable DIM if the device supports it.
> > Very hard to do this plumbing.
> >
> > 2. A net driver enables the DIM by default as long as
> VIRTIO_NET_F_VQ_NOTF_COAL is negotiated.
> > (without ethtool setting)
> >
> > 3. A device relaxes the limitation and continue to apply the coalescing, until
> driver overrides it.
> >
> > In my humble opinion, Heng is solving it using option #3, that tends to work
> with existing and future drivers who may/may not enable the DIM.
> >
> >
> >
> >

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> On Tue, 25 Jun 2024 18:14:15 -0700
> Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> 
> > On 6/24/2024 10:56 PM, Parav Pandit wrote:
> [..]
> > > I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> > > The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
> > >
> > > Flow is:
> > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> > > 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
> > >
> > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
> 
> I have certainly understood this differently. When
> VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
> 
> If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> to/MUST do the coalescing according to the parameters as described by
> the virtio spec.
> 
> Michael, Jason: Can you guys weigh in on this?

I still don't understand why this change is needed.
We have this text:

	The device may generate notifications more or less frequently than
	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.

and

	The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
	the device MAY generate notifications more or less frequently than specified.

So with no spec changes, devices already can do what this patch says they can do -
send notifications less frequently.

Re-reading this spec text, maybe the confusion is that it mentions set
commands specifically? And it's also stuck in the middle where it's easy
to miss.

So it would seem that the following should be sufficient, and it looks
like a small clarification we could just apply and include in the
vote for the csd. What do you guys think?


Signed-off-by: Michael S. Tsirkin <mst@redhat.com>

---- 


diff --git a/device-types/net/description.tex b/device-types/net/description.tex
index 76585b0..d6788df 100644
--- a/device-types/net/description.tex
+++ b/device-types/net/description.tex
@@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
                                         for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
 \end{enumerate}
 
-The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
-
 If coalescing parameters are being set, the device applies the last coalescing parameters set for a
 virtqueue, regardless of the command used to set the parameters. Use the following command sequence
 with two pairs of virtqueues as an example:
@@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
 \end{itemize}
 
+The device can generate notifications more or less frequently
+than specified by the coalescing parameters.
+
 \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
 
 The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
@@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
 Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
 to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
 
-The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
+The behavior of the device in response to specific coalescing parameters is best-effort:
 the device MAY generate notifications more or less frequently than specified.
 
 A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.

-- 
MST

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Thu, Jun 27, 2024 at 12:13:50PM +0000, Parav Pandit wrote:
> The intent is to restore the behavior to same as what it used to be when VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated.

This is not what the commit log or the github issue say though.
The more I look at the change the less I understand it.

Can the issue be clarified please?
	Current spec text ABC is bad for performance because DEF,
	change it to GHI - as a result device can get better
	performance by JKL.


-- 
MST

RE: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Parav Pandit]

Hi Michael,

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, June 27, 2024 6:05 PM
> 
> On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> > On Tue, 25 Jun 2024 18:14:15 -0700
> > Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> >
> > > On 6/24/2024 10:56 PM, Parav Pandit wrote:
> > [..]
> > > > I saw the need of this proposal slightly differently in the discussion with
> Heng in v4.
> > > > The way I understood is, proposed relaxation enables below Linux driver
> flow to work as equally as without device offering
> VIRTIO_NET_F_VQ_NOTF_COAL.
> > > >
> > > > Flow is:
> > > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL 2. The
> > > > virtio-net driver negotiated VIRTNET_FEATURES that has
> > > > VIRTIO_NET_F_VQ_NOTF_COAL
> > > >
> > > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not
> applying any coalescing on the VQ, in a good hope that driver will perform VQ
> notification coalescing.
> >
> > I have certainly understood this differently. When
> > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not
> supposed/allowed to do any interrupt coalescing (notification suppression
> may still apply).
> >
> > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > to/MUST do the coalescing according to the parameters as described by
> > the virtio spec.
> >
> > Michael, Jason: Can you guys weigh in on this?
> 
> I still don't understand why this change is needed.
> We have this text:
> 
> 	The device may generate notifications more or less frequently than
> 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL
> class.
> 
> and
> 
> 	The behavior of the device in response to set commands of the
> VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> 	the device MAY generate notifications more or less frequently than
> specified.
> 
> So with no spec changes, devices already can do what this patch says they can
> do - send notifications less frequently.
> 
> Re-reading this spec text, maybe the confusion is that it mentions set
> commands specifically? And it's also stuck in the middle where it's easy to
> miss.
> 
> So it would seem that the following should be sufficient, and it looks like a
> small clarification we could just apply and include in the vote for the csd. What
> do you guys think?
> 
> 
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> 
> ----
> 
> 
> diff --git a/device-types/net/description.tex b/device-
> types/net/description.tex
> index 76585b0..d6788df 100644
> --- a/device-types/net/description.tex
> +++ b/device-types/net/description.tex
> @@ -1711,8 +1711,6 @@ \subsubsection{Control
> Virtqueue}\label{sec:Device Types / Network Device / Devi
>                                          for an enabled transmit/receive virtqueue whose index is
> \field{vq_index}.
>  \end{enumerate}
> 
> -The device may generate notifications more or less frequently than specified
> by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> -
>  If coalescing parameters are being set, the device applies the last coalescing
> parameters set for a  virtqueue, regardless of the command used to set the
> parameters. Use the following command sequence  with two pairs of
> virtqueues as an example:
> @@ -1726,6 +1724,9 @@ \subsubsection{Control
> Virtqueue}\label{sec:Device Types / Network Device / Devi  \item Command6:
> VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device
> responds with coalescing parameters of index 1 set by command5.
>  \end{itemize}
> 
> +The device can generate notifications more or less frequently than
> +specified by the coalescing parameters.
> +
>  \subparagraph{Operation}\label{sec:Device Types / Network Device / Device
> Operation / Control Virtqueue / Notifications Coalescing / Operation}
> 
>  The device sends a used buffer notification once the notification conditions
> are met and if the notifications are not suppressed as explained in
> \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer
> Notification Suppression}.
> @@ -1798,7 +1799,7 @@ \subsubsection{Control
> Virtqueue}\label{sec:Device Types / Network Device / Devi  Upon disabling
> and re-enabling a receive virtqueue, the device MUST set the coalescing
> parameters of the virtqueue  to those configured through the
> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set
> any RX coalescing parameters, to 0.
> 
> -The behavior of the device in response to set commands of the
> VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> +The behavior of the device in response to specific coalescing parameters is
> best-effort:
>  the device MAY generate notifications more or less frequently than specified.
> 
>  A device SHOULD NOT send used buffer notifications to the driver if the
> notifications are suppressed, even if the notification conditions are met.
> 

Your changes above are good and they are applicable when the specific coalescing parameters are set by the driver.

The issue that is being resolved is, when these parameters are NOT set by the driver, when feature is negotiated.
What should device do?
The parameters are not specified...

The spec says " Upon reset, a device MUST initialize all coalescing parameters to 0.".

And Heng attempt is to clarify that by adding a changing above line something like,

"when driver does not set coalescing params, but have negotiated the feature, device may generate notifications more or less frequently than specified".

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Thu, Jun 27, 2024 at 12:45:49PM +0000, Parav Pandit wrote:
> Hi Michael,
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, June 27, 2024 6:05 PM
> > 
> > On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> > > On Tue, 25 Jun 2024 18:14:15 -0700
> > > Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> > >
> > > > On 6/24/2024 10:56 PM, Parav Pandit wrote:
> > > [..]
> > > > > I saw the need of this proposal slightly differently in the discussion with
> > Heng in v4.
> > > > > The way I understood is, proposed relaxation enables below Linux driver
> > flow to work as equally as without device offering
> > VIRTIO_NET_F_VQ_NOTF_COAL.
> > > > >
> > > > > Flow is:
> > > > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL 2. The
> > > > > virtio-net driver negotiated VIRTNET_FEATURES that has
> > > > > VIRTIO_NET_F_VQ_NOTF_COAL
> > > > >
> > > > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not
> > applying any coalescing on the VQ, in a good hope that driver will perform VQ
> > notification coalescing.
> > >
> > > I have certainly understood this differently. When
> > > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not
> > supposed/allowed to do any interrupt coalescing (notification suppression
> > may still apply).
> > >
> > > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > > to/MUST do the coalescing according to the parameters as described by
> > > the virtio spec.
> > >
> > > Michael, Jason: Can you guys weigh in on this?
> > 
> > I still don't understand why this change is needed.
> > We have this text:
> > 
> > 	The device may generate notifications more or less frequently than
> > 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL
> > class.
> > 
> > and
> > 
> > 	The behavior of the device in response to set commands of the
> > VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > 	the device MAY generate notifications more or less frequently than
> > specified.
> > 
> > So with no spec changes, devices already can do what this patch says they can
> > do - send notifications less frequently.
> > 
> > Re-reading this spec text, maybe the confusion is that it mentions set
> > commands specifically? And it's also stuck in the middle where it's easy to
> > miss.
> > 
> > So it would seem that the following should be sufficient, and it looks like a
> > small clarification we could just apply and include in the vote for the csd. What
> > do you guys think?
> > 
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > 
> > ----
> > 
> > 
> > diff --git a/device-types/net/description.tex b/device-
> > types/net/description.tex
> > index 76585b0..d6788df 100644
> > --- a/device-types/net/description.tex
> > +++ b/device-types/net/description.tex
> > @@ -1711,8 +1711,6 @@ \subsubsection{Control
> > Virtqueue}\label{sec:Device Types / Network Device / Devi
> >                                          for an enabled transmit/receive virtqueue whose index is
> > \field{vq_index}.
> >  \end{enumerate}
> > 
> > -The device may generate notifications more or less frequently than specified
> > by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> > -
> >  If coalescing parameters are being set, the device applies the last coalescing
> > parameters set for a  virtqueue, regardless of the command used to set the
> > parameters. Use the following command sequence  with two pairs of
> > virtqueues as an example:
> > @@ -1726,6 +1724,9 @@ \subsubsection{Control
> > Virtqueue}\label{sec:Device Types / Network Device / Devi  \item Command6:
> > VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device
> > responds with coalescing parameters of index 1 set by command5.
> >  \end{itemize}
> > 
> > +The device can generate notifications more or less frequently than
> > +specified by the coalescing parameters.
> > +
> >  \subparagraph{Operation}\label{sec:Device Types / Network Device / Device
> > Operation / Control Virtqueue / Notifications Coalescing / Operation}
> > 
> >  The device sends a used buffer notification once the notification conditions
> > are met and if the notifications are not suppressed as explained in
> > \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer
> > Notification Suppression}.
> > @@ -1798,7 +1799,7 @@ \subsubsection{Control
> > Virtqueue}\label{sec:Device Types / Network Device / Devi  Upon disabling
> > and re-enabling a receive virtqueue, the device MUST set the coalescing
> > parameters of the virtqueue  to those configured through the
> > VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set
> > any RX coalescing parameters, to 0.
> > 
> > -The behavior of the device in response to set commands of the
> > VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > +The behavior of the device in response to specific coalescing parameters is
> > best-effort:
> >  the device MAY generate notifications more or less frequently than specified.
> > 
> >  A device SHOULD NOT send used buffer notifications to the driver if the
> > notifications are suppressed, even if the notification conditions are met.
> > 
> 
> Your changes above are good and they are applicable when the specific coalescing parameters are set by the driver.

What makes you say this?
No they apply universally whenever parameters are specified by the spec:
whether parameters are set by driver or by device.
Is that unclear somehow?
I suspect you are just reading the wrong part of the patch ...


> The issue that is being resolved is, when these parameters are NOT set by the driver, when feature is negotiated.
> What should device do?
> The parameters are not specified...

They are specified - set to 0.

> The spec says " Upon reset, a device MUST initialize all coalescing parameters to 0.".

yes. and device can then coalesce if it wants to.

> And Heng attempt is to clarify that by adding a changing above line something like,
> 
> "when driver does not set coalescing params, but have negotiated the feature, device may generate notifications more or less frequently than specified".


Maybe that is the intention, but the proposed text is instead saying the
parameters themselves are !0. Which is what triggered the whole
mess with a misvote: commit log says it's a clarification when it is actually
a behaviour change.


-- 
MST

RE: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Thursday, June 27, 2024 6:23 PM
> 
> On Thu, Jun 27, 2024 at 12:45:49PM +0000, Parav Pandit wrote:
> > Hi Michael,
> >
> > > From: Michael S. Tsirkin <mst@redhat.com>
> > > Sent: Thursday, June 27, 2024 6:05 PM
> > >
> > > On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> > > > On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu
> > > > <si-wei.liu@oracle.com> wrote:
> > > >
> > > > > On 6/24/2024 10:56 PM, Parav Pandit wrote:
> > > > [..]
> > > > > > I saw the need of this proposal slightly differently in the
> > > > > > discussion with
> > > Heng in v4.
> > > > > > The way I understood is, proposed relaxation enables below
> > > > > > Linux driver
> > > flow to work as equally as without device offering
> > > VIRTIO_NET_F_VQ_NOTF_COAL.
> > > > > >
> > > > > > Flow is:
> > > > > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL 2. The
> > > > > > virtio-net driver negotiated VIRTNET_FEATURES that has
> > > > > > VIRTIO_NET_F_VQ_NOTF_COAL
> > > > > >
> > > > > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is
> > > > > > not
> > > applying any coalescing on the VQ, in a good hope that driver will
> > > perform VQ notification coalescing.
> > > >
> > > > I have certainly understood this differently. When
> > > > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not
> > > supposed/allowed to do any interrupt coalescing (notification
> > > suppression may still apply).
> > > >
> > > > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > > > to/MUST do the coalescing according to the parameters as described
> > > > by the virtio spec.
> > > >
> > > > Michael, Jason: Can you guys weigh in on this?
> > >
> > > I still don't understand why this change is needed.
> > > We have this text:
> > >
> > > 	The device may generate notifications more or less frequently than
> > > 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL
> class.
> > >
> > > and
> > >
> > > 	The behavior of the device in response to set commands of the
> > > VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > > 	the device MAY generate notifications more or less frequently than
> > > specified.
> > >
> > > So with no spec changes, devices already can do what this patch says
> > > they can do - send notifications less frequently.
> > >
> > > Re-reading this spec text, maybe the confusion is that it mentions
> > > set commands specifically? And it's also stuck in the middle where
> > > it's easy to miss.
> > >
> > > So it would seem that the following should be sufficient, and it
> > > looks like a small clarification we could just apply and include in
> > > the vote for the csd. What do you guys think?
> > >
> > >
> > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > >
> > > ----
> > >
> > >
> > > diff --git a/device-types/net/description.tex b/device-
> > > types/net/description.tex index 76585b0..d6788df 100644
> > > --- a/device-types/net/description.tex
> > > +++ b/device-types/net/description.tex
> > > @@ -1711,8 +1711,6 @@ \subsubsection{Control
> > > Virtqueue}\label{sec:Device Types / Network Device / Devi
> > >                                          for an enabled
> > > transmit/receive virtqueue whose index is \field{vq_index}.
> > >  \end{enumerate}
> > >
> > > -The device may generate notifications more or less frequently than
> > > specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> > > -
> > >  If coalescing parameters are being set, the device applies the last
> > > coalescing parameters set for a  virtqueue, regardless of the
> > > command used to set the parameters. Use the following command
> > > sequence  with two pairs of virtqueues as an example:
> > > @@ -1726,6 +1724,9 @@ \subsubsection{Control
> > > Virtqueue}\label{sec:Device Types / Network Device / Devi  \item
> Command6:
> > > VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the
> > > device responds with coalescing parameters of index 1 set by command5.
> > >  \end{itemize}
> > >
> > > +The device can generate notifications more or less frequently than
> > > +specified by the coalescing parameters.
> > > +
> > >  \subparagraph{Operation}\label{sec:Device Types / Network Device /
> > > Device Operation / Control Virtqueue / Notifications Coalescing /
> > > Operation}
> > >
> > >  The device sends a used buffer notification once the notification
> > > conditions are met and if the notifications are not suppressed as
> > > explained in \ref{sec:Basic Facilities of a Virtio Device /
> > > Virtqueues / Used Buffer Notification Suppression}.
> > > @@ -1798,7 +1799,7 @@ \subsubsection{Control
> > > Virtqueue}\label{sec:Device Types / Network Device / Devi  Upon
> > > disabling and re-enabling a receive virtqueue, the device MUST set
> > > the coalescing parameters of the virtqueue  to those configured
> > > through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the
> > > driver did not set any RX coalescing parameters, to 0.
> > >
> > > -The behavior of the device in response to set commands of the
> > > VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > > +The behavior of the device in response to specific coalescing
> > > +parameters is
> > > best-effort:
> > >  the device MAY generate notifications more or less frequently than
> specified.
> > >
> > >  A device SHOULD NOT send used buffer notifications to the driver if
> > > the notifications are suppressed, even if the notification conditions are
> met.
> > >
> >
> > Your changes above are good and they are applicable when the specific
> coalescing parameters are set by the driver.
> 
> What makes you say this?
Because I was reading this in the CVQ section of the spec where CVQ commands are setting it.

> No they apply universally whenever parameters are specified by the spec:
> whether parameters are set by driver or by device.
I see. Upon reset they are set to 0 in the below text, means no coalescing, means quick interrupts, right?

> Is that unclear somehow?
> I suspect you are just reading the wrong part of the patch ...
> 
> 
> > The issue that is being resolved is, when these parameters are NOT set by the
> driver, when feature is negotiated.
> > What should device do?
> > The parameters are not specified...
> 
> They are specified - set to 0.
> 
> > The spec says " Upon reset, a device MUST initialize all coalescing parameters
> to 0.".
> 
> yes. and device can then coalesce if it wants to.
> 
You mean 0 means coalescing is disabled (and not quick notifications) ?
And hence, it is device chosen value?
If so, that interpretation would work too.

> > And Heng attempt is to clarify that by adding a changing above line
> > something like,
> >
> > "when driver does not set coalescing params, but have negotiated the
> feature, device may generate notifications more or less frequently than
> specified".
> 
> 
> Maybe that is the intention, but the proposed text is instead saying the
> parameters themselves are !0. 
A clarification like above would be better or 0 means device chosen parameters would also work fine.

> Which is what triggered the whole mess with a
> misvote: commit log says it's a clarification when it is actually a behaviour
> change.
> 
It is a behavior change to fix/restore.
But if you suggest: 0 means device chosen value, that will work too to restore current regression.

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Thu, Jun 27, 2024 at 01:03:51PM +0000, Parav Pandit wrote:
> 
> > From: Michael S. Tsirkin <mst@redhat.com>
> > Sent: Thursday, June 27, 2024 6:23 PM
> > 
> > On Thu, Jun 27, 2024 at 12:45:49PM +0000, Parav Pandit wrote:
> > > Hi Michael,
> > >
> > > > From: Michael S. Tsirkin <mst@redhat.com>
> > > > Sent: Thursday, June 27, 2024 6:05 PM
> > > >
> > > > On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> > > > > On Tue, 25 Jun 2024 18:14:15 -0700 Si-Wei Liu
> > > > > <si-wei.liu@oracle.com> wrote:
> > > > >
> > > > > > On 6/24/2024 10:56 PM, Parav Pandit wrote:
> > > > > [..]
> > > > > > > I saw the need of this proposal slightly differently in the
> > > > > > > discussion with
> > > > Heng in v4.
> > > > > > > The way I understood is, proposed relaxation enables below
> > > > > > > Linux driver
> > > > flow to work as equally as without device offering
> > > > VIRTIO_NET_F_VQ_NOTF_COAL.
> > > > > > >
> > > > > > > Flow is:
> > > > > > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL 2. The
> > > > > > > virtio-net driver negotiated VIRTNET_FEATURES that has
> > > > > > > VIRTIO_NET_F_VQ_NOTF_COAL
> > > > > > >
> > > > > > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is
> > > > > > > not
> > > > applying any coalescing on the VQ, in a good hope that driver will
> > > > perform VQ notification coalescing.
> > > > >
> > > > > I have certainly understood this differently. When
> > > > > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not
> > > > supposed/allowed to do any interrupt coalescing (notification
> > > > suppression may still apply).
> > > > >
> > > > > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > > > > to/MUST do the coalescing according to the parameters as described
> > > > > by the virtio spec.
> > > > >
> > > > > Michael, Jason: Can you guys weigh in on this?
> > > >
> > > > I still don't understand why this change is needed.
> > > > We have this text:
> > > >
> > > > 	The device may generate notifications more or less frequently than
> > > > 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL
> > class.
> > > >
> > > > and
> > > >
> > > > 	The behavior of the device in response to set commands of the
> > > > VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > > > 	the device MAY generate notifications more or less frequently than
> > > > specified.
> > > >
> > > > So with no spec changes, devices already can do what this patch says
> > > > they can do - send notifications less frequently.
> > > >
> > > > Re-reading this spec text, maybe the confusion is that it mentions
> > > > set commands specifically? And it's also stuck in the middle where
> > > > it's easy to miss.
> > > >
> > > > So it would seem that the following should be sufficient, and it
> > > > looks like a small clarification we could just apply and include in
> > > > the vote for the csd. What do you guys think?
> > > >
> > > >
> > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > >
> > > > ----
> > > >
> > > >
> > > > diff --git a/device-types/net/description.tex b/device-
> > > > types/net/description.tex index 76585b0..d6788df 100644
> > > > --- a/device-types/net/description.tex
> > > > +++ b/device-types/net/description.tex
> > > > @@ -1711,8 +1711,6 @@ \subsubsection{Control
> > > > Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > >                                          for an enabled
> > > > transmit/receive virtqueue whose index is \field{vq_index}.
> > > >  \end{enumerate}
> > > >
> > > > -The device may generate notifications more or less frequently than
> > > > specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> > > > -
> > > >  If coalescing parameters are being set, the device applies the last
> > > > coalescing parameters set for a  virtqueue, regardless of the
> > > > command used to set the parameters. Use the following command
> > > > sequence  with two pairs of virtqueues as an example:
> > > > @@ -1726,6 +1724,9 @@ \subsubsection{Control
> > > > Virtqueue}\label{sec:Device Types / Network Device / Devi  \item
> > Command6:
> > > > VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the
> > > > device responds with coalescing parameters of index 1 set by command5.
> > > >  \end{itemize}
> > > >
> > > > +The device can generate notifications more or less frequently than
> > > > +specified by the coalescing parameters.
> > > > +
> > > >  \subparagraph{Operation}\label{sec:Device Types / Network Device /
> > > > Device Operation / Control Virtqueue / Notifications Coalescing /
> > > > Operation}
> > > >
> > > >  The device sends a used buffer notification once the notification
> > > > conditions are met and if the notifications are not suppressed as
> > > > explained in \ref{sec:Basic Facilities of a Virtio Device /
> > > > Virtqueues / Used Buffer Notification Suppression}.
> > > > @@ -1798,7 +1799,7 @@ \subsubsection{Control
> > > > Virtqueue}\label{sec:Device Types / Network Device / Devi  Upon
> > > > disabling and re-enabling a receive virtqueue, the device MUST set
> > > > the coalescing parameters of the virtqueue  to those configured
> > > > through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the
> > > > driver did not set any RX coalescing parameters, to 0.
> > > >
> > > > -The behavior of the device in response to set commands of the
> > > > VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > > > +The behavior of the device in response to specific coalescing
> > > > +parameters is
> > > > best-effort:
> > > >  the device MAY generate notifications more or less frequently than
> > specified.
> > > >
> > > >  A device SHOULD NOT send used buffer notifications to the driver if
> > > > the notifications are suppressed, even if the notification conditions are
> > met.
> > > >
> > >
> > > Your changes above are good and they are applicable when the specific
> > coalescing parameters are set by the driver.
> > 
> > What makes you say this?
> Because I was reading this in the CVQ section of the spec where CVQ commands are setting it.


So? Reset also sets it.

> > No they apply universally whenever parameters are specified by the spec:
> > whether parameters are set by driver or by device.
> I see. Upon reset they are set to 0 in the below text, means no coalescing, means quick interrupts, right?

But device can make them as slow as it likes, spec says so.


> > Is that unclear somehow?
> > I suspect you are just reading the wrong part of the patch ...
> > 
> > 
> > > The issue that is being resolved is, when these parameters are NOT set by the
> > driver, when feature is negotiated.
> > > What should device do?
> > > The parameters are not specified...
> > 
> > They are specified - set to 0.
> > 
> > > The spec says " Upon reset, a device MUST initialize all coalescing parameters
> > to 0.".
> > 
> > yes. and device can then coalesce if it wants to.
> > 
> You mean 0 means coalescing is disabled (and not quick notifications) ?
> And hence, it is device chosen value?
> If so, that interpretation would work too.

No 0 means send notification after any # of packets.


> > > And Heng attempt is to clarify that by adding a changing above line
> > > something like,
> > >
> > > "when driver does not set coalescing params, but have negotiated the
> > feature, device may generate notifications more or less frequently than
> > specified".
> > 
> > 
> > Maybe that is the intention, but the proposed text is instead saying the
> > parameters themselves are !0. 
> A clarification like above would be better or 0 means device chosen parameters would also work fine.


> > Which is what triggered the whole mess with a
> > misvote: commit log says it's a clarification when it is actually a behaviour
> > change.
> > 
> It is a behavior change to fix/restore.
> But if you suggest: 0 means device chosen value, that will work too to restore current regression.

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/27/2024 5:35 AM, Michael S. Tsirkin wrote:
> On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
>> On Tue, 25 Jun 2024 18:14:15 -0700
>> Si-Wei Liu <si-wei.liu@oracle.com> wrote:
>>
>>> On 6/24/2024 10:56 PM, Parav Pandit wrote:
>> [..]
>>>> I saw the need of this proposal slightly differently in the discussion with Heng in v4.
>>>> The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
>>>>
>>>> Flow is:
>>>> 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
>>>> 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
>>>>
>>>> 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
>> I have certainly understood this differently. When
>> VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
>>
>> If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
>> to/MUST do the coalescing according to the parameters as described by
>> the virtio spec.
>>
>> Michael, Jason: Can you guys weigh in on this?
> I still don't understand why this change is needed.
> We have this text:
>
> 	The device may generate notifications more or less frequently than
> 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
>
> and
>
> 	The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> 	the device MAY generate notifications more or less frequently than specified.
>
> So with no spec changes, devices already can do what this patch says they can do -
> send notifications less frequently.
Right. If I still recall it correctly, the above text was to accommodate 
the interaction with existing used buffer notification mechanism, 
F_EVENT_IDX or whatnot. When coalescing is in place and effective, 
interrupt delivery is a collaborative consideration based on the 
coalescing parameters specified *AND * event index / NO_INTERRUPT flag 
as well.
>
> Re-reading this spec text, maybe the confusion is that it mentions set
> commands specifically? And it's also stuck in the middle where it's easy
> to miss.
>
> So it would seem that the following should be sufficient, and it looks
> like a small clarification we could just apply and include in the
> vote for the csd. What do you guys think?

The revised text below looks good to me.

Thanks,
-Siwei
>
>
> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>
> ----
>
>
> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> index 76585b0..d6788df 100644
> --- a/device-types/net/description.tex
> +++ b/device-types/net/description.tex
> @@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>                                           for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
>   \end{enumerate}
>   
> -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> -
>   If coalescing parameters are being set, the device applies the last coalescing parameters set for a
>   virtqueue, regardless of the command used to set the parameters. Use the following command sequence
>   with two pairs of virtqueues as an example:
> @@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>   \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
>   \end{itemize}
>   
> +The device can generate notifications more or less frequently
> +than specified by the coalescing parameters.
> +
>   \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
>   
>   The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
> @@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>   Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
>   to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
>   
> -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> +The behavior of the device in response to specific coalescing parameters is best-effort:
>   the device MAY generate notifications more or less frequently than specified.
>   
>   A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/27/2024 5:52 AM, Michael S. Tsirkin wrote:
> On Thu, Jun 27, 2024 at 12:45:49PM +0000, Parav Pandit wrote:
>> Hi Michael,
>>
>>> From: Michael S. Tsirkin <mst@redhat.com>
>>> Sent: Thursday, June 27, 2024 6:05 PM
>>>
>>> On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
>>>> On Tue, 25 Jun 2024 18:14:15 -0700
>>>> Si-Wei Liu <si-wei.liu@oracle.com> wrote:
>>>>
>>>>> On 6/24/2024 10:56 PM, Parav Pandit wrote:
>>>> [..]
>>>>>> I saw the need of this proposal slightly differently in the discussion with
>>> Heng in v4.
>>>>>> The way I understood is, proposed relaxation enables below Linux driver
>>> flow to work as equally as without device offering
>>> VIRTIO_NET_F_VQ_NOTF_COAL.
>>>>>> Flow is:
>>>>>> 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL 2. The
>>>>>> virtio-net driver negotiated VIRTNET_FEATURES that has
>>>>>> VIRTIO_NET_F_VQ_NOTF_COAL
>>>>>>
>>>>>> 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not
>>> applying any coalescing on the VQ, in a good hope that driver will perform VQ
>>> notification coalescing.
>>>> I have certainly understood this differently. When
>>>> VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not
>>> supposed/allowed to do any interrupt coalescing (notification suppression
>>> may still apply).
>>>> If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
>>>> to/MUST do the coalescing according to the parameters as described by
>>>> the virtio spec.
>>>>
>>>> Michael, Jason: Can you guys weigh in on this?
>>> I still don't understand why this change is needed.
>>> We have this text:
>>>
>>> 	The device may generate notifications more or less frequently than
>>> 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL
>>> class.
>>>
>>> and
>>>
>>> 	The behavior of the device in response to set commands of the
>>> VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
>>> 	the device MAY generate notifications more or less frequently than
>>> specified.
>>>
>>> So with no spec changes, devices already can do what this patch says they can
>>> do - send notifications less frequently.
>>>
>>> Re-reading this spec text, maybe the confusion is that it mentions set
>>> commands specifically? And it's also stuck in the middle where it's easy to
>>> miss.
>>>
>>> So it would seem that the following should be sufficient, and it looks like a
>>> small clarification we could just apply and include in the vote for the csd. What
>>> do you guys think?
>>>
>>>
>>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>
>>> ----
>>>
>>>
>>> diff --git a/device-types/net/description.tex b/device-
>>> types/net/description.tex
>>> index 76585b0..d6788df 100644
>>> --- a/device-types/net/description.tex
>>> +++ b/device-types/net/description.tex
>>> @@ -1711,8 +1711,6 @@ \subsubsection{Control
>>> Virtqueue}\label{sec:Device Types / Network Device / Devi
>>>                                           for an enabled transmit/receive virtqueue whose index is
>>> \field{vq_index}.
>>>   \end{enumerate}
>>>
>>> -The device may generate notifications more or less frequently than specified
>>> by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
>>> -
>>>   If coalescing parameters are being set, the device applies the last coalescing
>>> parameters set for a  virtqueue, regardless of the command used to set the
>>> parameters. Use the following command sequence  with two pairs of
>>> virtqueues as an example:
>>> @@ -1726,6 +1724,9 @@ \subsubsection{Control
>>> Virtqueue}\label{sec:Device Types / Network Device / Devi  \item Command6:
>>> VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device
>>> responds with coalescing parameters of index 1 set by command5.
>>>   \end{itemize}
>>>
>>> +The device can generate notifications more or less frequently than
>>> +specified by the coalescing parameters.
>>> +
>>>   \subparagraph{Operation}\label{sec:Device Types / Network Device / Device
>>> Operation / Control Virtqueue / Notifications Coalescing / Operation}
>>>
>>>   The device sends a used buffer notification once the notification conditions
>>> are met and if the notifications are not suppressed as explained in
>>> \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer
>>> Notification Suppression}.
>>> @@ -1798,7 +1799,7 @@ \subsubsection{Control
>>> Virtqueue}\label{sec:Device Types / Network Device / Devi  Upon disabling
>>> and re-enabling a receive virtqueue, the device MUST set the coalescing
>>> parameters of the virtqueue  to those configured through the
>>> VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set
>>> any RX coalescing parameters, to 0.
>>>
>>> -The behavior of the device in response to set commands of the
>>> VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
>>> +The behavior of the device in response to specific coalescing parameters is
>>> best-effort:
>>>   the device MAY generate notifications more or less frequently than specified.
>>>
>>>   A device SHOULD NOT send used buffer notifications to the driver if the
>>> notifications are suppressed, even if the notification conditions are met.
>>>
>> Your changes above are good and they are applicable when the specific coalescing parameters are set by the driver.
> What makes you say this?
> No they apply universally whenever parameters are specified by the spec:
> whether parameters are set by driver or by device.
> Is that unclear somehow?
> I suspect you are just reading the wrong part of the patch ...
>
>
>> The issue that is being resolved is, when these parameters are NOT set by the driver, when feature is negotiated.
>> What should device do?
>> The parameters are not specified...
> They are specified - set to 0.
>
>> The spec says " Upon reset, a device MUST initialize all coalescing parameters to 0.".
> yes. and device can then coalesce if it wants to.
Well, maybe coalesce here in this context is a little confusing but I 
guess I got what you're saying: setting 0 effectively disables the 
driver's control of coalescing, right. And device may have its own way 
to suppress notification...

>
>> And Heng attempt is to clarify that by adding a changing above line something like,
>>
>> "when driver does not set coalescing params, but have negotiated the feature, device may generate notifications more or less frequently than specified".
>
> Maybe that is the intention, but the proposed text is instead saying the
> parameters themselves are !0. Which is what triggered the whole
> mess with a misvote: commit log says it's a clarification when it is actually
> a behaviour change.
>

+1.

Whatever beautified wording there was, the proposed spec fundamentally 
leads to a behavior change.

-Siwei

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Thu, Jun 27, 2024 at 10:14:49AM -0700, Si-Wei Liu wrote:
> 
> 
> On 6/27/2024 5:35 AM, Michael S. Tsirkin wrote:
> > On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> > > On Tue, 25 Jun 2024 18:14:15 -0700
> > > Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> > > 
> > > > On 6/24/2024 10:56 PM, Parav Pandit wrote:
> > > [..]
> > > > > I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> > > > > The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
> > > > > 
> > > > > Flow is:
> > > > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> > > > > 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
> > > > > 
> > > > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
> > > I have certainly understood this differently. When
> > > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
> > > 
> > > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > > to/MUST do the coalescing according to the parameters as described by
> > > the virtio spec.
> > > 
> > > Michael, Jason: Can you guys weigh in on this?
> > I still don't understand why this change is needed.
> > We have this text:
> > 
> > 	The device may generate notifications more or less frequently than
> > 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> > 
> > and
> > 
> > 	The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > 	the device MAY generate notifications more or less frequently than specified.
> > 
> > So with no spec changes, devices already can do what this patch says they can do -
> > send notifications less frequently.
> Right. If I still recall it correctly, the above text was to accommodate the
> interaction with existing used buffer notification mechanism, F_EVENT_IDX or
> whatnot.
> When coalescing is in place and effective, interrupt delivery is a
> collaborative consideration based on the coalescing parameters specified
> *AND * event index / NO_INTERRUPT flag as well.

No, not really.

For that we have the next sentence:

A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.


And of course event index never makes you send interrupts
more frequently.

> > 
> > Re-reading this spec text, maybe the confusion is that it mentions set
> > commands specifically? And it's also stuck in the middle where it's easy
> > to miss.
> > 
> > So it would seem that the following should be sufficient, and it looks
> > like a small clarification we could just apply and include in the
> > vote for the csd. What do you guys think?
> 
> The revised text below looks good to me.
> 
> Thanks,
> -Siwei
> > 
> > 
> > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > 
> > ----
> > 
> > 
> > diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> > index 76585b0..d6788df 100644
> > --- a/device-types/net/description.tex
> > +++ b/device-types/net/description.tex
> > @@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >                                           for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
> >   \end{enumerate}
> > -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> > -
> >   If coalescing parameters are being set, the device applies the last coalescing parameters set for a
> >   virtqueue, regardless of the command used to set the parameters. Use the following command sequence
> >   with two pairs of virtqueues as an example:
> > @@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >   \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
> >   \end{itemize}
> > +The device can generate notifications more or less frequently
> > +than specified by the coalescing parameters.
> > +
> >   \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
> >   The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
> > @@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >   Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> >   to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> > -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > +The behavior of the device in response to specific coalescing parameters is best-effort:
> >   the device MAY generate notifications more or less frequently than specified.
> >   A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> > 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/27/2024 3:18 PM, Michael S. Tsirkin wrote:
> On Thu, Jun 27, 2024 at 10:14:49AM -0700, Si-Wei Liu wrote:
>>
>> On 6/27/2024 5:35 AM, Michael S. Tsirkin wrote:
>>> On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
>>>> On Tue, 25 Jun 2024 18:14:15 -0700
>>>> Si-Wei Liu <si-wei.liu@oracle.com> wrote:
>>>>
>>>>> On 6/24/2024 10:56 PM, Parav Pandit wrote:
>>>> [..]
>>>>>> I saw the need of this proposal slightly differently in the discussion with Heng in v4.
>>>>>> The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
>>>>>>
>>>>>> Flow is:
>>>>>> 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
>>>>>> 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
>>>>>>
>>>>>> 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
>>>> I have certainly understood this differently. When
>>>> VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
>>>>
>>>> If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
>>>> to/MUST do the coalescing according to the parameters as described by
>>>> the virtio spec.
>>>>
>>>> Michael, Jason: Can you guys weigh in on this?
>>> I still don't understand why this change is needed.
>>> We have this text:
>>>
>>> 	The device may generate notifications more or less frequently than
>>> 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
>>>
>>> and
>>>
>>> 	The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
>>> 	the device MAY generate notifications more or less frequently than specified.
>>>
>>> So with no spec changes, devices already can do what this patch says they can do -
>>> send notifications less frequently.
>> Right. If I still recall it correctly, the above text was to accommodate the
>> interaction with existing used buffer notification mechanism, F_EVENT_IDX or
>> whatnot.
>> When coalescing is in place and effective, interrupt delivery is a
>> collaborative consideration based on the coalescing parameters specified
>> *AND * event index / NO_INTERRUPT flag as well.
> No, not really.
>
> For that we have the next sentence:
>
> A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
>
>
> And of course event index never makes you send interrupts
> more frequently.
Hmmm, this next sentence you reference above was indeed for the 
interaction between coalescing and used buffer suppression. Then what's 
the best-effort part was about, really? Round-up or round down the set 
value to the power of 2 to save space? How is it relevant to our 
discussion? I think even with rounding it shouldn't be too off? (as 
said, by best-effort v.s. give up)

-Siwei

>
>>> Re-reading this spec text, maybe the confusion is that it mentions set
>>> commands specifically? And it's also stuck in the middle where it's easy
>>> to miss.
>>>
>>> So it would seem that the following should be sufficient, and it looks
>>> like a small clarification we could just apply and include in the
>>> vote for the csd. What do you guys think?
>> The revised text below looks good to me.
>>
>> Thanks,
>> -Siwei
>>>
>>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>
>>> ----
>>>
>>>
>>> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
>>> index 76585b0..d6788df 100644
>>> --- a/device-types/net/description.tex
>>> +++ b/device-types/net/description.tex
>>> @@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>>>                                            for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
>>>    \end{enumerate}
>>> -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
>>> -
>>>    If coalescing parameters are being set, the device applies the last coalescing parameters set for a
>>>    virtqueue, regardless of the command used to set the parameters. Use the following command sequence
>>>    with two pairs of virtqueues as an example:
>>> @@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>>>    \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
>>>    \end{itemize}
>>> +The device can generate notifications more or less frequently
>>> +than specified by the coalescing parameters.
>>> +
>>>    \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
>>>    The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
>>> @@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>>>    Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
>>>    to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
>>> -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
>>> +The behavior of the device in response to specific coalescing parameters is best-effort:
>>>    the device MAY generate notifications more or less frequently than specified.
>>>    A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
>>>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Jason Wang]

On Fri, Jun 28, 2024 at 2:57 PM Si-Wei Liu <si-wei.liu@oracle.com> wrote:
>
>
>
> On 6/27/2024 3:18 PM, Michael S. Tsirkin wrote:
> > On Thu, Jun 27, 2024 at 10:14:49AM -0700, Si-Wei Liu wrote:
> >>
> >> On 6/27/2024 5:35 AM, Michael S. Tsirkin wrote:
> >>> On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> >>>> On Tue, 25 Jun 2024 18:14:15 -0700
> >>>> Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> >>>>
> >>>>> On 6/24/2024 10:56 PM, Parav Pandit wrote:
> >>>> [..]
> >>>>>> I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> >>>>>> The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
> >>>>>>
> >>>>>> Flow is:
> >>>>>> 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> >>>>>> 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
> >>>>>>
> >>>>>> 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
> >>>> I have certainly understood this differently. When
> >>>> VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
> >>>>
> >>>> If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> >>>> to/MUST do the coalescing according to the parameters as described by
> >>>> the virtio spec.
> >>>>
> >>>> Michael, Jason: Can you guys weigh in on this?
> >>> I still don't understand why this change is needed.
> >>> We have this text:
> >>>
> >>>     The device may generate notifications more or less frequently than
> >>>     specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> >>>
> >>> and
> >>>
> >>>     The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> >>>     the device MAY generate notifications more or less frequently than specified.
> >>>
> >>> So with no spec changes, devices already can do what this patch says they can do -
> >>> send notifications less frequently.
> >> Right. If I still recall it correctly, the above text was to accommodate the
> >> interaction with existing used buffer notification mechanism, F_EVENT_IDX or
> >> whatnot.
> >> When coalescing is in place and effective, interrupt delivery is a
> >> collaborative consideration based on the coalescing parameters specified
> >> *AND * event index / NO_INTERRUPT flag as well.
> > No, not really.
> >
> > For that we have the next sentence:
> >
> > A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> >
> >
> > And of course event index never makes you send interrupts
> > more frequently.
> Hmmm, this next sentence you reference above was indeed for the
> interaction between coalescing and used buffer suppression. Then what's
> the best-effort part was about, really? Round-up or round down the set
> value to the power of 2 to save space? How is it relevant to our
> discussion? I think even with rounding it shouldn't be too off? (as
> said, by best-effort v.s. give up)

Would it help if we add something like below in the guidance?

When the device wants to send a notification it needs:

1) check notification suppression, don't send interrupt if it is
suppressed otherwise goto 2)
2) check event index, don't send interrupt if event index is not
crossed otherwise goto 3)
3) check coalescing, don't send interrupt if the coalescing limit is
not exceed, otherwise send the interrupt

(Just to demonstrate the idea, needs tweaking for sure)

THanks


>
> -Siwei
>
> >
> >>> Re-reading this spec text, maybe the confusion is that it mentions set
> >>> commands specifically? And it's also stuck in the middle where it's easy
> >>> to miss.
> >>>
> >>> So it would seem that the following should be sufficient, and it looks
> >>> like a small clarification we could just apply and include in the
> >>> vote for the csd. What do you guys think?
> >> The revised text below looks good to me.
> >>
> >> Thanks,
> >> -Siwei
> >>>
> >>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >>>
> >>> ----
> >>>
> >>>
> >>> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> >>> index 76585b0..d6788df 100644
> >>> --- a/device-types/net/description.tex
> >>> +++ b/device-types/net/description.tex
> >>> @@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >>>                                            for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
> >>>    \end{enumerate}
> >>> -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> >>> -
> >>>    If coalescing parameters are being set, the device applies the last coalescing parameters set for a
> >>>    virtqueue, regardless of the command used to set the parameters. Use the following command sequence
> >>>    with two pairs of virtqueues as an example:
> >>> @@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >>>    \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
> >>>    \end{itemize}
> >>> +The device can generate notifications more or less frequently
> >>> +than specified by the coalescing parameters.
> >>> +
> >>>    \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
> >>>    The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
> >>> @@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >>>    Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> >>>    to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> >>> -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> >>> +The behavior of the device in response to specific coalescing parameters is best-effort:
> >>>    the device MAY generate notifications more or less frequently than specified.
> >>>    A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> >>>
>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Si-Wei Liu]

On 6/28/2024 1:23 AM, Jason Wang wrote:
> On Fri, Jun 28, 2024 at 2:57 PM Si-Wei Liu <si-wei.liu@oracle.com> wrote:
>>
>>
>> On 6/27/2024 3:18 PM, Michael S. Tsirkin wrote:
>>> On Thu, Jun 27, 2024 at 10:14:49AM -0700, Si-Wei Liu wrote:
>>>> On 6/27/2024 5:35 AM, Michael S. Tsirkin wrote:
>>>>> On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
>>>>>> On Tue, 25 Jun 2024 18:14:15 -0700
>>>>>> Si-Wei Liu <si-wei.liu@oracle.com> wrote:
>>>>>>
>>>>>>> On 6/24/2024 10:56 PM, Parav Pandit wrote:
>>>>>> [..]
>>>>>>>> I saw the need of this proposal slightly differently in the discussion with Heng in v4.
>>>>>>>> The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
>>>>>>>>
>>>>>>>> Flow is:
>>>>>>>> 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
>>>>>>>> 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
>>>>>>>>
>>>>>>>> 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
>>>>>> I have certainly understood this differently. When
>>>>>> VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
>>>>>>
>>>>>> If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
>>>>>> to/MUST do the coalescing according to the parameters as described by
>>>>>> the virtio spec.
>>>>>>
>>>>>> Michael, Jason: Can you guys weigh in on this?
>>>>> I still don't understand why this change is needed.
>>>>> We have this text:
>>>>>
>>>>>      The device may generate notifications more or less frequently than
>>>>>      specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
>>>>>
>>>>> and
>>>>>
>>>>>      The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
>>>>>      the device MAY generate notifications more or less frequently than specified.
>>>>>
>>>>> So with no spec changes, devices already can do what this patch says they can do -
>>>>> send notifications less frequently.
>>>> Right. If I still recall it correctly, the above text was to accommodate the
>>>> interaction with existing used buffer notification mechanism, F_EVENT_IDX or
>>>> whatnot.
>>>> When coalescing is in place and effective, interrupt delivery is a
>>>> collaborative consideration based on the coalescing parameters specified
>>>> *AND * event index / NO_INTERRUPT flag as well.
>>> No, not really.
>>>
>>> For that we have the next sentence:
>>>
>>> A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
>>>
>>>
>>> And of course event index never makes you send interrupts
>>> more frequently.
>> Hmmm, this next sentence you reference above was indeed for the
>> interaction between coalescing and used buffer suppression. Then what's
>> the best-effort part was about, really? Round-up or round down the set
>> value to the power of 2 to save space? How is it relevant to our
>> discussion? I think even with rounding it shouldn't be too off? (as
>> said, by best-effort v.s. give up)
> Would it help if we add something like below in the guidance?
>
> When the device wants to send a notification it needs:
>
> 1) check notification suppression, don't send interrupt if it is
> suppressed otherwise goto 2)
> 2) check event index, don't send interrupt if event index is not
> crossed otherwise goto 3)
> 3) check coalescing, don't send interrupt if the coalescing limit is
> not exceed, otherwise send the interrupt
>
> (Just to demonstrate the idea, needs tweaking for sure)
Yeah, this looks good to me, in principle. I recall there was detailed 
description like this in the early revisions from the first set of 
coalescing proposal, but those useful descriptions were gone for some 
reason. We should really come up with clear clarifications to help 
readers set expectation rather than being vague or worrying too much 
about verbosity. I was having a hard time in understanding what the 
best-effort part is about...

-Siwei
>
> THanks
>
>
>> -Siwei
>>
>>>>> Re-reading this spec text, maybe the confusion is that it mentions set
>>>>> commands specifically? And it's also stuck in the middle where it's easy
>>>>> to miss.
>>>>>
>>>>> So it would seem that the following should be sufficient, and it looks
>>>>> like a small clarification we could just apply and include in the
>>>>> vote for the csd. What do you guys think?
>>>> The revised text below looks good to me.
>>>>
>>>> Thanks,
>>>> -Siwei
>>>>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
>>>>>
>>>>> ----
>>>>>
>>>>>
>>>>> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
>>>>> index 76585b0..d6788df 100644
>>>>> --- a/device-types/net/description.tex
>>>>> +++ b/device-types/net/description.tex
>>>>> @@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>>>>>                                             for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
>>>>>     \end{enumerate}
>>>>> -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
>>>>> -
>>>>>     If coalescing parameters are being set, the device applies the last coalescing parameters set for a
>>>>>     virtqueue, regardless of the command used to set the parameters. Use the following command sequence
>>>>>     with two pairs of virtqueues as an example:
>>>>> @@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>>>>>     \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
>>>>>     \end{itemize}
>>>>> +The device can generate notifications more or less frequently
>>>>> +than specified by the coalescing parameters.
>>>>> +
>>>>>     \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
>>>>>     The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
>>>>> @@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
>>>>>     Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
>>>>>     to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
>>>>> -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
>>>>> +The behavior of the device in response to specific coalescing parameters is best-effort:
>>>>>     the device MAY generate notifications more or less frequently than specified.
>>>>>     A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
>>>>>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Halil Pasic]

On Thu, 27 Jun 2024 08:35:16 -0400
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> > On Tue, 25 Jun 2024 18:14:15 -0700
> > Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> >   
> > > On 6/24/2024 10:56 PM, Parav Pandit wrote:  
> > [..]  
> > > > I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> > > > The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
> > > >
> > > > Flow is:
> > > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> > > > 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
> > > >
> > > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.  
> > 
> > I have certainly understood this differently. When
> > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
> > 
> > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > to/MUST do the coalescing according to the parameters as described by
> > the virtio spec.
> > 
> > Michael, Jason: Can you guys weigh in on this?  
> 
> I still don't understand why this change is needed.
> We have this text:
> 
> 	The device may generate notifications more or less frequently than
> 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.

The less frequently could be understood like due to notification
suppression.

> 
> and
> 
> 	The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> 	the device MAY generate notifications more or less frequently than specified.

To me best-effort implies making an effort. Here we talk about
intentionally ignoring the parameter values. For me stuff like not being
precise about the counting or timekeeping, or running into the danger of
having to drop packets because the queue is about to fill up, would
qualify as best-effort deviations from the specified behavior.

I believe we want the drivers written under the assumption that the
workings of a notification coalescing in any device that implements it
are close enough to what is described in the spec. Just my opinion.

Regards,
Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Sat, Jun 29, 2024 at 08:47:47AM +0200, Halil Pasic wrote:
> On Thu, 27 Jun 2024 08:35:16 -0400
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
> 
> > On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> > > On Tue, 25 Jun 2024 18:14:15 -0700
> > > Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> > >   
> > > > On 6/24/2024 10:56 PM, Parav Pandit wrote:  
> > > [..]  
> > > > > I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> > > > > The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
> > > > >
> > > > > Flow is:
> > > > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> > > > > 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
> > > > >
> > > > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.  
> > > 
> > > I have certainly understood this differently. When
> > > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
> > > 
> > > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > > to/MUST do the coalescing according to the parameters as described by
> > > the virtio spec.
> > > 
> > > Michael, Jason: Can you guys weigh in on this?  
> > 
> > I still don't understand why this change is needed.
> > We have this text:
> > 
> > 	The device may generate notifications more or less frequently than
> > 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> 
> The less frequently could be understood like due to notification
> suppression.

I think you are missing that there is a separate text about notification suppression
immediately after it:
	A device SHOULD NOT send used buffer notifications to the driver
	if the notifications are suppressed, even if the notification conditions
	are met.
I think that makes it clear this text is separate from suppression.


> > 
> > and
> > 
> > 	The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > 	the device MAY generate notifications more or less frequently than specified.
> 
> To me best-effort implies making an effort. Here we talk about
> intentionally ignoring the parameter values. For me stuff like not being
> precise about the counting or timekeeping, or running into the danger of
> having to drop packets because the queue is about to fill up, would
> qualify as best-effort deviations from the specified behavior.
> 
> I believe we want the drivers written under the assumption that the
> workings of a notification coalescing in any device that implements it
> are close enough to what is described in the spec. Just my opinion.
> 
> Regards,
> Halil

OK, I guess we can clarify:
	in particular, the device MAY coalesce notifications when
	coalescing parameters are set to 0.

will that make it better?

-- 
MST

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Fri, Jun 28, 2024 at 12:31:45PM -0700, Si-Wei Liu wrote:
> 
> 
> On 6/28/2024 1:23 AM, Jason Wang wrote:
> > On Fri, Jun 28, 2024 at 2:57 PM Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> > > 
> > > 
> > > On 6/27/2024 3:18 PM, Michael S. Tsirkin wrote:
> > > > On Thu, Jun 27, 2024 at 10:14:49AM -0700, Si-Wei Liu wrote:
> > > > > On 6/27/2024 5:35 AM, Michael S. Tsirkin wrote:
> > > > > > On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> > > > > > > On Tue, 25 Jun 2024 18:14:15 -0700
> > > > > > > Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> > > > > > > 
> > > > > > > > On 6/24/2024 10:56 PM, Parav Pandit wrote:
> > > > > > > [..]
> > > > > > > > > I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> > > > > > > > > The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
> > > > > > > > > 
> > > > > > > > > Flow is:
> > > > > > > > > 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> > > > > > > > > 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
> > > > > > > > > 
> > > > > > > > > 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
> > > > > > > I have certainly understood this differently. When
> > > > > > > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
> > > > > > > 
> > > > > > > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > > > > > > to/MUST do the coalescing according to the parameters as described by
> > > > > > > the virtio spec.
> > > > > > > 
> > > > > > > Michael, Jason: Can you guys weigh in on this?
> > > > > > I still don't understand why this change is needed.
> > > > > > We have this text:
> > > > > > 
> > > > > >      The device may generate notifications more or less frequently than
> > > > > >      specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> > > > > > 
> > > > > > and
> > > > > > 
> > > > > >      The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > > > > >      the device MAY generate notifications more or less frequently than specified.
> > > > > > 
> > > > > > So with no spec changes, devices already can do what this patch says they can do -
> > > > > > send notifications less frequently.
> > > > > Right. If I still recall it correctly, the above text was to accommodate the
> > > > > interaction with existing used buffer notification mechanism, F_EVENT_IDX or
> > > > > whatnot.
> > > > > When coalescing is in place and effective, interrupt delivery is a
> > > > > collaborative consideration based on the coalescing parameters specified
> > > > > *AND * event index / NO_INTERRUPT flag as well.
> > > > No, not really.
> > > > 
> > > > For that we have the next sentence:
> > > > 
> > > > A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> > > > 
> > > > 
> > > > And of course event index never makes you send interrupts
> > > > more frequently.
> > > Hmmm, this next sentence you reference above was indeed for the
> > > interaction between coalescing and used buffer suppression. Then what's
> > > the best-effort part was about, really? Round-up or round down the set
> > > value to the power of 2 to save space? How is it relevant to our
> > > discussion? I think even with rounding it shouldn't be too off? (as
> > > said, by best-effort v.s. give up)
> > Would it help if we add something like below in the guidance?
> > 
> > When the device wants to send a notification it needs:
> > 
> > 1) check notification suppression, don't send interrupt if it is
> > suppressed otherwise goto 2)
> > 2) check event index, don't send interrupt if event index is not
> > crossed otherwise goto 3)
> > 3) check coalescing, don't send interrupt if the coalescing limit is
> > not exceed, otherwise send the interrupt
> > 
> > (Just to demonstrate the idea, needs tweaking for sure)
> Yeah, this looks good to me, in principle. I recall there was detailed
> description like this in the early revisions from the first set of
> coalescing proposal, but those useful descriptions were gone for some
> reason. We should really come up with clear clarifications to help readers
> set expectation rather than being vague or worrying too much about
> verbosity. I was having a hard time in understanding what the best-effort
> part is about...
> 
> -Siwei

It means it's a hint from the driver.
As another example, devices might have a coarse
grained clock, or a limited packet counter.
Or whatever. No need to guess.

There's a balance with verbosity - more text needs to
add value not just repeat same thing many times.
It also needs to be written in an exceptionally clear way
otherwise we are confusing the reader instead of
helping.


And in this case, I think I agree - instead of saying it
is best effort and then clarifying, we should straight away
say this is a hint.

The text in the conformance section should be thus changed:

-The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
+The device MAY treat the coalescing parameters as a hint:
 the device MAY generate notifications more or less frequently than specified.


The text in non conformance section is fine, but let's make it
consistent:
-The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
+	The device can treat coalescing parameters as a hint,
+	generating notifications more or less frequently
+	than specified by the coalescing parameters.

> > 
> > THanks
> > 
> > 
> > > -Siwei
> > > 
> > > > > > Re-reading this spec text, maybe the confusion is that it mentions set
> > > > > > commands specifically? And it's also stuck in the middle where it's easy
> > > > > > to miss.
> > > > > > 
> > > > > > So it would seem that the following should be sufficient, and it looks
> > > > > > like a small clarification we could just apply and include in the
> > > > > > vote for the csd. What do you guys think?
> > > > > The revised text below looks good to me.
> > > > > 
> > > > > Thanks,
> > > > > -Siwei
> > > > > > Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> > > > > > 
> > > > > > ----
> > > > > > 
> > > > > > 
> > > > > > diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> > > > > > index 76585b0..d6788df 100644
> > > > > > --- a/device-types/net/description.tex
> > > > > > +++ b/device-types/net/description.tex
> > > > > > @@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > > > >                                             for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
> > > > > >     \end{enumerate}
> > > > > > -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> > > > > > -
> > > > > >     If coalescing parameters are being set, the device applies the last coalescing parameters set for a
> > > > > >     virtqueue, regardless of the command used to set the parameters. Use the following command sequence
> > > > > >     with two pairs of virtqueues as an example:
> > > > > > @@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > > > >     \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
> > > > > >     \end{itemize}
> > > > > > +The device can generate notifications more or less frequently
> > > > > > +than specified by the coalescing parameters.
> > > > > > +
> > > > > >     \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
> > > > > >     The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
> > > > > > @@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> > > > > >     Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> > > > > >     to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> > > > > > -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> > > > > > +The behavior of the device in response to specific coalescing parameters is best-effort:
> > > > > >     the device MAY generate notifications more or less frequently than specified.
> > > > > >     A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> > > > > > 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Halil Pasic]

On Fri, 28 Jun 2024 16:23:17 +0800
Jason Wang <jasowang@redhat.com> wrote:

> > Hmmm, this next sentence you reference above was indeed for the
> > interaction between coalescing and used buffer suppression. Then what's
> > the best-effort part was about, really? Round-up or round down the set
> > value to the power of 2 to save space? How is it relevant to our
> > discussion? I think even with rounding it shouldn't be too off? (as
> > said, by best-effort v.s. give up)  
> 
> Would it help if we add something like below in the guidance?
>

It would IMHO greatly benefit clarity, provided we do get it right.
 
> When the device wants to send a notification it needs:
> 
> 1) check notification suppression, don't send interrupt if it is
> suppressed otherwise goto 2)
> 2) check event index, don't send interrupt if event index is not
> crossed otherwise goto 3)
> 3) check coalescing, don't send interrupt if the coalescing limit is
> not exceed, otherwise send the interrupt
> 
> (Just to demonstrate the idea, needs tweaking for sure)

Yes I agree this indeed needs tweaking. Let me make some points.

1) for virtio-net we have 3 distinct mechanisms that police used
buffer notification avoidance, but 2 of the three are actually
mutually exclusive if I understand it correctly. Namely we have
coalescing guarded by VIRTIO_NET_F_NOTF_COAL and then depending on
whether _F_EVENT_IDX was negotiated or not, we have "event_idx based" or
"crude flag based" notification suppression.

2) Let me just quote the entire section on the good old used buffer
notification suppression
"""
2.7.7.2 Device Requirements: Used Buffer Notification Suppression

If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:

    * The device MUST ignore the used_event value.
    * After the device writes a descriptor index into the used ring:
       -  If flags is 1, the device SHOULD NOT send a notification.
       -  If flags is 0, the device MUST send a notification.

Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:

    * The device MUST ignore the lower bit of flags.
    * After the device writes a descriptor index into the used ring:
       - If the idx field in the used ring (which determined where that
      descriptor index was placed) was equal to used_event, the device
      MUST send a notification.
       - Otherwise the device SHOULD NOT send a notification.

Note: For example, if used_event is 0, then a device using

VIRTIO_F_EVENT_IDX would send a used buffer notification to the driver
after the first buffer is used (and again after the 65536th buffer,
etc).
""" 

Frankly, a MUST is a must, and I don't see very little leeway
for the coalescing to avoid any notifications. But on the other hand,
that is the very purpose of the coalescing so IMHO we have a problem
here.

The only loophole here is "after", which could be stretched to
"eventually, before the end of the world". Which would basically reduce
the MUST ad-absurdum, and I would not like that.

3) AFAIU if multiple suppression mechanisms are active
concurrently we are aiming for some sort of an logical AND semantic i.e.
a further reduction over just a single one being employed (and not OR).

But IMHO the both the wording for the event_idx based and curde
suppression, as well as for coalescing.

Let me quote the full description for the coalescing:

"""
5.1.6.5.9.1 Operation
The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in 2.7.7.

When the device has non-zero max_usecs and non-zero max_packets, it starts counting microseconds and packets upon receiving/sending a packet. The device counts packets and microseconds for each receive virtqueue and transmit virtqueue separately. In this case, the notification conditions are met when max_usecs microseconds elapse, or upon sending/receiving max_packets packets, whichever happens first. Afterwards, the device waits for the next packet and starts counting packets and microseconds again.

When the device has max_usecs = 0 or max_packets = 0, the notification conditions are met after every packet received/sent. 
"""

Here "Afterwards" is ambiguous. What we want here is "afterwards" ==
"after the notification has been sent", but one can also read it as
"after the condition has been met".

4) I see two ways to make sense out of it.

My preferred way would be to replace "send a notification" with
"generate a notification" in the main text including "2.7.7.2 Device
Requirements: Used Buffer Notification Suppression", along with adding an
explanation which states that generated notifications are sent
immediately unless a device has an active facility that under a certain
set of conditions  retains and possibly coalesces notifications. 

Please notice that here coalescence is used with its dictionary
meaning (see https://dictionary.cambridge.org/dictionary/english/coalesce), 
and that the current description of NET_F_NOTIF_COAL is more akin to a
suppression mechanism where notifications are suppressed or discarded and
not coalesced.

The notification coalescing facility would then retain the first
used buffer notification request for each queue (those are still
generated by the same rules), until either the packet count or the
timeout type max frequency (or delay we need to clarify that) condition
is met, and coalesce any further notification requests with the retained
one until it is sent.

The other way, which I'm not sure is actually viable, is basically to
try to clarify the hell out of the current approach as taken by
"5.1.6.5.9.1 Operation" by making it crystal clear when exactly are the
"notification conditions met" and what is the complement state of that,
and add a sentence to 2.7.7.2 that references device specific mechanisms
to do more suppression.  And  of course we would have to find a nice
definition for "if the notifications are not suppressed as explained in
2.7.7." such that loss of initiative is not possible.

IMHO any solution where things can stall because we would have sent
an used buffer notification without "coalescing" but we did not because
of coalescing  is not acceptable. This could happen if we cross
event_idx when the coalescing notification conditions are not met and
thus do not notify, and then when the coalescing notification condition
are finally met (e.g. via timeout) the event_id type suppression
is active again because we crossed event_idx when coalescing
notification condition was not met, and we end up not notifying again.

Sorry for the wall of text.

Regards,
Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Michael S. Tsirkin]

On Tue, Jul 02, 2024 at 10:37:35PM +0200, Halil Pasic wrote:
> On Fri, 28 Jun 2024 16:23:17 +0800
> Jason Wang <jasowang@redhat.com> wrote:
> 
> > > Hmmm, this next sentence you reference above was indeed for the
> > > interaction between coalescing and used buffer suppression. Then what's
> > > the best-effort part was about, really? Round-up or round down the set
> > > value to the power of 2 to save space? How is it relevant to our
> > > discussion? I think even with rounding it shouldn't be too off? (as
> > > said, by best-effort v.s. give up)  
> > 
> > Would it help if we add something like below in the guidance?
> >
> 
> It would IMHO greatly benefit clarity, provided we do get it right.
>  
> > When the device wants to send a notification it needs:
> > 
> > 1) check notification suppression, don't send interrupt if it is
> > suppressed otherwise goto 2)
> > 2) check event index, don't send interrupt if event index is not
> > crossed otherwise goto 3)
> > 3) check coalescing, don't send interrupt if the coalescing limit is
> > not exceed, otherwise send the interrupt
> > 
> > (Just to demonstrate the idea, needs tweaking for sure)
> 
> Yes I agree this indeed needs tweaking. Let me make some points.

Very good points below.


> 1) for virtio-net we have 3 distinct mechanisms that police used
> buffer notification avoidance, but 2 of the three are actually
> mutually exclusive if I understand it correctly. Namely we have
> coalescing guarded by VIRTIO_NET_F_NOTF_COAL and then depending on
> whether _F_EVENT_IDX was negotiated or not, we have "event_idx based" or
> "crude flag based" notification suppression.
> 
> 2) Let me just quote the entire section on the good old used buffer
> notification suppression
> """
> 2.7.7.2 Device Requirements: Used Buffer Notification Suppression
> 
> If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
> 
>     * The device MUST ignore the used_event value.
>     * After the device writes a descriptor index into the used ring:
>        -  If flags is 1, the device SHOULD NOT send a notification.
>        -  If flags is 0, the device MUST send a notification.
> 
> Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
> 
>     * The device MUST ignore the lower bit of flags.
>     * After the device writes a descriptor index into the used ring:
>        - If the idx field in the used ring (which determined where that
>       descriptor index was placed) was equal to used_event, the device
>       MUST send a notification.
>        - Otherwise the device SHOULD NOT send a notification.
> 
> Note: For example, if used_event is 0, then a device using
> 
> VIRTIO_F_EVENT_IDX would send a used buffer notification to the driver
> after the first buffer is used (and again after the 65536th buffer,
> etc).
> """ 
> 
> Frankly, a MUST is a must, and I don't see very little leeway
> for the coalescing to avoid any notifications. But on the other hand,
> that is the very purpose of the coalescing so IMHO we have a problem
> here.
> 
> The only loophole here is "after", which could be stretched to
> "eventually, before the end of the world". Which would basically reduce
> the MUST ad-absurdum, and I would not like that.

I agree as such but I also think we can just go ahead and make this
explicit. That is s/after/at some point in time after/.

And then let's add text that actually explains how
- devices can defer notifications for performance
- some devices can have device specific mechanisms
  for driver to control how much notifications are deferred.


WDYT?



> 3) AFAIU if multiple suppression mechanisms are active
> concurrently we are aiming for some sort of an logical AND semantic i.e.
> a further reduction over just a single one being employed (and not OR).
> 
> But IMHO the both the wording for the event_idx based and curde
> suppression, as well as for coalescing.
> 
> Let me quote the full description for the coalescing:
> 
> """
> 5.1.6.5.9.1 Operation
> The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in 2.7.7.
> 
> When the device has non-zero max_usecs and non-zero max_packets, it starts counting microseconds and packets upon receiving/sending a packet. The device counts packets and microseconds for each receive virtqueue and transmit virtqueue separately. In this case, the notification conditions are met when max_usecs microseconds elapse, or upon sending/receiving max_packets packets, whichever happens first. Afterwards, the device waits for the next packet and starts counting packets and microseconds again.
> 
> When the device has max_usecs = 0 or max_packets = 0, the notification conditions are met after every packet received/sent. 
> """
> 
> Here "Afterwards" is ambiguous. What we want here is "afterwards" ==
> "after the notification has been sent", but one can also read it as
> "after the condition has been met".
> 
> 4) I see two ways to make sense out of it.
> 
> My preferred way would be to replace "send a notification" with
> "generate a notification" in the main text including "2.7.7.2 Device
> Requirements: Used Buffer Notification Suppression", along with adding an
> explanation which states that generated notifications are sent
> immediately unless a device has an active facility that under a certain
> set of conditions  retains and possibly coalesces notifications. 
> 
> Please notice that here coalescence is used with its dictionary
> meaning (see https://dictionary.cambridge.org/dictionary/english/coalesce), 
> and that the current description of NET_F_NOTIF_COAL is more akin to a
> suppression mechanism where notifications are suppressed or discarded and
> not coalesced.
> 
> The notification coalescing facility would then retain the first
> used buffer notification request for each queue (those are still
> generated by the same rules), until either the packet count or the
> timeout type max frequency (or delay we need to clarify that) condition
> is met, and coalesce any further notification requests with the retained
> one until it is sent.
> 
> The other way, which I'm not sure is actually viable, is basically to
> try to clarify the hell out of the current approach as taken by
> "5.1.6.5.9.1 Operation" by making it crystal clear when exactly are the
> "notification conditions met" and what is the complement state of that,
> and add a sentence to 2.7.7.2 that references device specific mechanisms
> to do more suppression.  And  of course we would have to find a nice
> definition for "if the notifications are not suppressed as explained in
> 2.7.7." such that loss of initiative is not possible.
> 
> IMHO any solution where things can stall because we would have sent
> an used buffer notification without "coalescing" but we did not because
> of coalescing  is not acceptable. This could happen if we cross
> event_idx when the coalescing notification conditions are not met and
> thus do not notify, and then when the coalescing notification condition
> are finally met (e.g. via timeout) the event_id type suppression
> is active again because we crossed event_idx when coalescing
> notification condition was not met, and we end up not notifying again.
> 
> Sorry for the wall of text.
> 
> Regards,
> Halil
> 
> 
> 

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Halil Pasic]

On Sun, 30 Jun 2024 12:55:35 -0400
"Michael S. Tsirkin" <mst@redhat.com> wrote:

> On Sat, Jun 29, 2024 at 08:47:47AM +0200, Halil Pasic wrote:
[..] 
> > > > I have certainly understood this differently. When
> > > > VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
> > > > 
> > > > If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> > > > to/MUST do the coalescing according to the parameters as described by
> > > > the virtio spec.
> > > > 
> > > > Michael, Jason: Can you guys weigh in on this?    
> > > 
> > > I still don't understand why this change is needed.
> > > We have this text:
> > > 
> > > 	The device may generate notifications more or less frequently than
> > > 	specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.  
> > 
> > The less frequently could be understood like due to notification
> > suppression.  
> 
> I think you are missing that there is a separate text about notification suppression
> immediately after it:
> 	A device SHOULD NOT send used buffer notifications to the driver
> 	if the notifications are suppressed, even if the notification conditions
> 	are met.
> I think that makes it clear this text is separate from suppression.
> 

No I did not miss it, and I don't think that the two sentences being in
adjacent paragraphs makes it clear that this is separate from
suppression.

For me the "notification conditions" part and the "even" is making it
sound like normally meeting these conditions would mean that a
notification should be sent but then notification suppression overrules
that, so there will be no notification even though if it wasn't for
the suppression there would be one.
 
[..]
> 
> OK, I guess we can clarify:
> 	in particular, the device MAY coalesce notifications when
> 	coalescing parameters are set to 0.
> 
> will that make it better?
> 

I don't think so. Sorry, there are things I would like to understand
before committing. Back then when I voted on the coalescing I convinced
myself that the interaction between used buffer notification suppression
and used buffer notification coalescing on the TX and RX queues of
virtio-net is specified with enough clarity. Now I'm asking myself
if I came to that conclusion along the lines "nothing else would make
any sense, not spelling it out properly is not tragic". Please
see my comment on Jason's proposal.

Regards,
Halil

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Jason Wang]

On Wed, Jul 3, 2024 at 4:38 AM Halil Pasic <pasic@linux.ibm.com> wrote:
>
> On Fri, 28 Jun 2024 16:23:17 +0800
> Jason Wang <jasowang@redhat.com> wrote:
>
> > > Hmmm, this next sentence you reference above was indeed for the
> > > interaction between coalescing and used buffer suppression. Then what's
> > > the best-effort part was about, really? Round-up or round down the set
> > > value to the power of 2 to save space? How is it relevant to our
> > > discussion? I think even with rounding it shouldn't be too off? (as
> > > said, by best-effort v.s. give up)
> >
> > Would it help if we add something like below in the guidance?
> >
>
> It would IMHO greatly benefit clarity, provided we do get it right.

Exactly.

>
> > When the device wants to send a notification it needs:
> >
> > 1) check notification suppression, don't send interrupt if it is
> > suppressed otherwise goto 2)
> > 2) check event index, don't send interrupt if event index is not
> > crossed otherwise goto 3)
> > 3) check coalescing, don't send interrupt if the coalescing limit is
> > not exceed, otherwise send the interrupt
> >
> > (Just to demonstrate the idea, needs tweaking for sure)
>
> Yes I agree this indeed needs tweaking. Let me make some points.
>
> 1) for virtio-net we have 3 distinct mechanisms that police used
> buffer notification avoidance, but 2 of the three are actually
> mutually exclusive if I understand it correctly. Namely we have
> coalescing guarded by VIRTIO_NET_F_NOTF_COAL and then depending on
> whether _F_EVENT_IDX was negotiated or not, we have "event_idx based" or
> "crude flag based" notification suppression.
>
> 2) Let me just quote the entire section on the good old used buffer
> notification suppression
> """
> 2.7.7.2 Device Requirements: Used Buffer Notification Suppression
>
> If the VIRTIO_F_EVENT_IDX feature bit is not negotiated:
>
>     * The device MUST ignore the used_event value.
>     * After the device writes a descriptor index into the used ring:
>        -  If flags is 1, the device SHOULD NOT send a notification.
>        -  If flags is 0, the device MUST send a notification.
>
> Otherwise, if the VIRTIO_F_EVENT_IDX feature bit is negotiated:
>
>     * The device MUST ignore the lower bit of flags.
>     * After the device writes a descriptor index into the used ring:
>        - If the idx field in the used ring (which determined where that
>       descriptor index was placed) was equal to used_event, the device
>       MUST send a notification.
>        - Otherwise the device SHOULD NOT send a notification.
>
> Note: For example, if used_event is 0, then a device using
>
> VIRTIO_F_EVENT_IDX would send a used buffer notification to the driver
> after the first buffer is used (and again after the 65536th buffer,
> etc).
> """
>
> Frankly, a MUST is a must, and I don't see very little leeway
> for the coalescing to avoid any notifications. But on the other hand,
> that is the very purpose of the coalescing so IMHO we have a problem
> here.
>
> The only loophole here is "after", which could be stretched to
> "eventually, before the end of the world". Which would basically reduce
> the MUST ad-absurdum, and I would not like that.
>
> 3) AFAIU if multiple suppression mechanisms are active
> concurrently we are aiming for some sort of an logical AND semantic i.e.
> a further reduction over just a single one being employed (and not OR).
>
> But IMHO the both the wording for the event_idx based and curde
> suppression, as well as for coalescing.
>
> Let me quote the full description for the coalescing:
>
> """
> 5.1.6.5.9.1 Operation
> The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in 2.7.7.
>
> When the device has non-zero max_usecs and non-zero max_packets, it starts counting microseconds and packets upon receiving/sending a packet. The device counts packets and microseconds for each receive virtqueue and transmit virtqueue separately. In this case, the notification conditions are met when max_usecs microseconds elapse, or upon sending/receiving max_packets packets, whichever happens first. Afterwards, the device waits for the next packet and starts counting packets and microseconds again.
>
> When the device has max_usecs = 0 or max_packets = 0, the notification conditions are met after every packet received/sent.
> """
>
> Here "Afterwards" is ambiguous. What we want here is "afterwards" ==
> "after the notification has been sent", but one can also read it as
> "after the condition has been met".

I agree and unfortunately it could be understood differently by
vendors (and we don't even have an open source software implementation
so far to refer to).

Btw, here is the one that is used by Intel E810. It seems better than
the "afterwards" we used here.

"""
Interrupt throttling is a mechanism that guarantees a minimum gap
between two consecutive interrupts (other than possible jitter caused
by handling the interrupts). The E810 counts the time since the last
interrupt is scheduled and compares it against the ITR setting. If an
event associated with this ITR happens before the ITR expires, the
interrupt assertion is delayed until the ITR expires. If the ITR
expires before any event associated with this interrupt, the interrupt
logic is “armed” and the interrupt can be asserted the moment the
event happens.
"""

And if we follow this it should be "after the notification has been sent".

>
> 4) I see two ways to make sense out of it.
>
> My preferred way would be to replace "send a notification" with
> "generate a notification" in the main text including "2.7.7.2 Device
> Requirements: Used Buffer Notification Suppression", along with adding an
> explanation which states that generated notifications are sent
> immediately unless a device has an active facility that under a certain
> set of conditions  retains and possibly coalesces notifications.
>
> Please notice that here coalescence is used with its dictionary
> meaning (see https://dictionary.cambridge.org/dictionary/english/coalesce),

Probably, because we don't tell the driver how many interrupts were
coalesced. I guess that's why it is usually named as "interrupt
moderation" for most of the nic vendors.

> and that the current description of NET_F_NOTIF_COAL is more akin to a
> suppression mechanism where notifications are suppressed or discarded and
> not coalesced.
>
> The notification coalescing facility would then retain the first
> used buffer notification request for each queue (those are still
> generated by the same rules), until either the packet count or the
> timeout type max frequency (or delay we need to clarify that) condition
> is met, and coalesce any further notification requests with the retained
> one until it is sent.
>
> The other way, which I'm not sure is actually viable, is basically to
> try to clarify the hell out of the current approach as taken by
> "5.1.6.5.9.1 Operation" by making it crystal clear when exactly are the
> "notification conditions met" and what is the complement state of that,
> and add a sentence to 2.7.7.2 that references device specific mechanisms
> to do more suppression.  And  of course we would have to find a nice
> definition for "if the notifications are not suppressed as explained in
> 2.7.7." such that loss of initiative is not possible.

This is what I try to achieve. Not sure if it's the best. Or we can
have a pseudo code to explain if it helps.


>
> IMHO any solution where things can stall because we would have sent
> an used buffer notification without "coalescing" but we did not because
> of coalescing  is not acceptable. This could happen if we cross
> event_idx when the coalescing notification conditions are not met and
> thus do not notify, and then when the coalescing notification condition
> are finally met (e.g. via timeout) the event_id type suppression
> is active again because we crossed event_idx when coalescing
> notification condition was not met, and we end up not notifying again.
>
> Sorry for the wall of text.
>
> Regards,
> Halil

Thanks

>
>
>
>

Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Jason Wang]

On Sat, Jun 29, 2024 at 3:32 AM Si-Wei Liu <si-wei.liu@oracle.com> wrote:
>
>
>
> On 6/28/2024 1:23 AM, Jason Wang wrote:
> > On Fri, Jun 28, 2024 at 2:57 PM Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> >>
> >>
> >> On 6/27/2024 3:18 PM, Michael S. Tsirkin wrote:
> >>> On Thu, Jun 27, 2024 at 10:14:49AM -0700, Si-Wei Liu wrote:
> >>>> On 6/27/2024 5:35 AM, Michael S. Tsirkin wrote:
> >>>>> On Thu, Jun 27, 2024 at 12:37:32PM +0200, Halil Pasic wrote:
> >>>>>> On Tue, 25 Jun 2024 18:14:15 -0700
> >>>>>> Si-Wei Liu <si-wei.liu@oracle.com> wrote:
> >>>>>>
> >>>>>>> On 6/24/2024 10:56 PM, Parav Pandit wrote:
> >>>>>> [..]
> >>>>>>>> I saw the need of this proposal slightly differently in the discussion with Heng in v4.
> >>>>>>>> The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.
> >>>>>>>>
> >>>>>>>> Flow is:
> >>>>>>>> 1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
> >>>>>>>> 2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL
> >>>>>>>>
> >>>>>>>> 3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.
> >>>>>> I have certainly understood this differently. When
> >>>>>> VIRTIO_NET_F_VQ_NOTF_COAL is not negotiated then the device is not supposed/allowed to do any interrupt coalescing (notification suppression may still apply).
> >>>>>>
> >>>>>> If VIRTIO_NET_F_VQ_NOTF_COAL is negotiated the device is supposed
> >>>>>> to/MUST do the coalescing according to the parameters as described by
> >>>>>> the virtio spec.
> >>>>>>
> >>>>>> Michael, Jason: Can you guys weigh in on this?
> >>>>> I still don't understand why this change is needed.
> >>>>> We have this text:
> >>>>>
> >>>>>      The device may generate notifications more or less frequently than
> >>>>>      specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> >>>>>
> >>>>> and
> >>>>>
> >>>>>      The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> >>>>>      the device MAY generate notifications more or less frequently than specified.
> >>>>>
> >>>>> So with no spec changes, devices already can do what this patch says they can do -
> >>>>> send notifications less frequently.
> >>>> Right. If I still recall it correctly, the above text was to accommodate the
> >>>> interaction with existing used buffer notification mechanism, F_EVENT_IDX or
> >>>> whatnot.
> >>>> When coalescing is in place and effective, interrupt delivery is a
> >>>> collaborative consideration based on the coalescing parameters specified
> >>>> *AND * event index / NO_INTERRUPT flag as well.
> >>> No, not really.
> >>>
> >>> For that we have the next sentence:
> >>>
> >>> A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> >>>
> >>>
> >>> And of course event index never makes you send interrupts
> >>> more frequently.
> >> Hmmm, this next sentence you reference above was indeed for the
> >> interaction between coalescing and used buffer suppression. Then what's
> >> the best-effort part was about, really? Round-up or round down the set
> >> value to the power of 2 to save space? How is it relevant to our
> >> discussion? I think even with rounding it shouldn't be too off? (as
> >> said, by best-effort v.s. give up)
> > Would it help if we add something like below in the guidance?
> >
> > When the device wants to send a notification it needs:
> >
> > 1) check notification suppression, don't send interrupt if it is
> > suppressed otherwise goto 2)
> > 2) check event index, don't send interrupt if event index is not
> > crossed otherwise goto 3)
> > 3) check coalescing, don't send interrupt if the coalescing limit is
> > not exceed, otherwise send the interrupt
> >
> > (Just to demonstrate the idea, needs tweaking for sure)
> Yeah, this looks good to me, in principle. I recall there was detailed
> description like this in the early revisions from the first set of
> coalescing proposal, but those useful descriptions were gone for some
> reason. We should really come up with clear clarifications to help
> readers set expectation rather than being vague or worrying too much
> about verbosity.

Yes.

> I was having a hard time in understanding what the
> best-effort part is about...

Being verbose might be helpful for vendors. Otherwise the may guess
(since we don't have a reference software implementation so far).

Thanks

>
> -Siwei
> >
> > THanks
> >
> >
> >> -Siwei
> >>
> >>>>> Re-reading this spec text, maybe the confusion is that it mentions set
> >>>>> commands specifically? And it's also stuck in the middle where it's easy
> >>>>> to miss.
> >>>>>
> >>>>> So it would seem that the following should be sufficient, and it looks
> >>>>> like a small clarification we could just apply and include in the
> >>>>> vote for the csd. What do you guys think?
> >>>> The revised text below looks good to me.
> >>>>
> >>>> Thanks,
> >>>> -Siwei
> >>>>> Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
> >>>>>
> >>>>> ----
> >>>>>
> >>>>>
> >>>>> diff --git a/device-types/net/description.tex b/device-types/net/description.tex
> >>>>> index 76585b0..d6788df 100644
> >>>>> --- a/device-types/net/description.tex
> >>>>> +++ b/device-types/net/description.tex
> >>>>> @@ -1711,8 +1711,6 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >>>>>                                             for an enabled transmit/receive virtqueue whose index is \field{vq_index}.
> >>>>>     \end{enumerate}
> >>>>> -The device may generate notifications more or less frequently than specified by set commands of the VIRTIO_NET_CTRL_NOTF_COAL class.
> >>>>> -
> >>>>>     If coalescing parameters are being set, the device applies the last coalescing parameters set for a
> >>>>>     virtqueue, regardless of the command used to set the parameters. Use the following command sequence
> >>>>>     with two pairs of virtqueues as an example:
> >>>>> @@ -1726,6 +1724,9 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >>>>>     \item Command6: VIRTIO_NET_CTRL_NOTF_COAL_VQ_GET with \field{vq_index} = 1, the device responds with coalescing parameters of index 1 set by command5.
> >>>>>     \end{itemize}
> >>>>> +The device can generate notifications more or less frequently
> >>>>> +than specified by the coalescing parameters.
> >>>>> +
> >>>>>     \subparagraph{Operation}\label{sec:Device Types / Network Device / Device Operation / Control Virtqueue / Notifications Coalescing / Operation}
> >>>>>     The device sends a used buffer notification once the notification conditions are met and if the notifications are not suppressed as explained in \ref{sec:Basic Facilities of a Virtio Device / Virtqueues / Used Buffer Notification Suppression}.
> >>>>> @@ -1798,7 +1799,7 @@ \subsubsection{Control Virtqueue}\label{sec:Device Types / Network Device / Devi
> >>>>>     Upon disabling and re-enabling a receive virtqueue, the device MUST set the coalescing parameters of the virtqueue
> >>>>>     to those configured through the VIRTIO_NET_CTRL_NOTF_COAL_RX_SET command, or, if the driver did not set any RX coalescing parameters, to 0.
> >>>>> -The behavior of the device in response to set commands of the VIRTIO_NET_CTRL_NOTF_COAL class is best-effort:
> >>>>> +The behavior of the device in response to specific coalescing parameters is best-effort:
> >>>>>     the device MAY generate notifications more or less frequently than specified.
> >>>>>     A device SHOULD NOT send used buffer notifications to the driver if the notifications are suppressed, even if the notification conditions are met.
> >>>>>
>