RE: [PATCH v5] virtio-net: clarify coalescing parameters settings

Parav Pandit
2024-06-25T05:56:47Z

Hi Si-Wei,

> From: Si-Wei Liu <si-wei.liu@oracle.com>
> Sent: Tuesday, June 25, 2024 10:22 AM
>
> On 6/21/2024 6:34 PM, Heng Qi wrote:
> > On Fri, 21 Jun 2024 16:46:27 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>
> wrote:
> >> On 6/20/2024 8:24 PM, Heng Qi wrote:
> >>> On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu"<si-wei.liu@oracle.com>
> wrote:
> >>>> On 6/20/2024 12:40 AM, Heng Qi wrote:
> >>>>> On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu"<si-
> wei.liu@oracle.com> wrote:
> >>>>>> On 6/16/2024 7:27 PM, Heng Qi wrote:
> >>>>>>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S.
> Tsirkin"<mst@redhat.com> wrote:
> >>>>>>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> >>>>>>>>>> From: Michael S. Tsirkin<mst@redhat.com>
> >>>>>>>>>> Sent: Tuesday, June 11, 2024 10:00 PM How we can we make
> >>>>>>>>>> progress with the realease but sure we don't make backwards
> >>>>>>>>>> compat a pain?
> >>>>>>>>>>
> >>>>>>>>>> Ideas?
> >>>>>>>>>>
> >>>>>>>>> There is no functional break with this relaxation.
> >>>>>>>>> Device set some non-zero defaults and driver didn't modify
> them....
> >>>>>>>>> Anything broken? Unlikely.
> >>>>>> Generally it's inappropriate to leave this decision making to the
> >>>>>> device for what would be the best / most performant default
> >>>>>> config, as the device is generally considered agnostic to the guest
> load...
> >>>>> Instead, the performance of the virtual machine and the driver
> >>>>> depends heavily on how the device is implemented, just as we have
> >>>>> proposed various ways to offload the data queue in the device to
> >>>>> the hardware. The reason why most devices use software to simulate
> >>>>> ctrlq instead of using hardware offload is that the driver has no
> >>>>> requirements for the performance of ctrlq before, that is, the
> >>>>> device implementation is responsible for and meets the driver's
> performance requirements.
> >>>> I am not sure I follow the arguments of ctrlq being s/w or h/w, I
> >>>> thought that it was for the debate why we need coalescing on
> >>>> hardware offload device in the first place, instead of reusing
> >>>> event index or similar s/w based notification suppression. I don't
> >>>> doubt the value of coalescing, but how's it relevant to the default value
> disposition?
> >>>> Generally the default config disposition is not considered to be
> >>>> part of device implementation, especially when it comes to the
> >>>> situation where device can't easily figure out the specific
> >>>> workload to occur in the guest, and there's no perfect single
> >>>> default value that could meet every single performance metric
> >>>> across the board. This is a typical tuning knob left up to the user
> >>>> to adjust, why the device or driver needs to set or load the
> >>>> initial value? The driver just needs to start with certain value,
> >>>> be it 0 or non-zero, which guest user can override at any point of
> >>>> time, depending on his/her need, and that's it! I guess I still
> >>>> don't understand your user case here, why device / driver default is of
> such importance.
> >>> I've explained that, and I understand your argument is why default
> >>> value is needed, and users should be able to adjust them, right?
> >> Sounds about right. You'll soon realize that there's no perfect
> >> default that could work with everyone - in that it's just a static
> >> value to begin with, no matter whatever initial value the device
> >> comes up with, one user or another will come over to you and complain
> >> that what is loaded from the device doesn't match the workload they
> >> have, so those users are still expected to adjust manually tweaking
> >> for their own. As long as it's a tunable that guest user can control
> >> and override anytime, I don't feel it too much different what initial
> >> config the device would start with.
> > When you have a large number of customers and they buy your machines,
> > how many users do you think have experience adjusting this value? More
> below.
> This is not what this proposal alone could address, as it just shifts the sheer
> pain to the device and host admin rather than provide any flexibility.
> Technically I believe the right solution is to seek adaptive coalescing that
> doesn't require user to tune anything. Noted adaptive coalescing /
> suppression could be done from the guest or by the device. For e.g. the s/w
> based event index based notification suppression could also work with
> hardware if the event index is somehow propagated through a doorbell
> register rather than load from host memory over the PCI transport.
>
> >>> The default value is working when the user doesn't adjust them.
> >>> It's not practical to rely entirely on user adjustment,
> >> That's where the adaptive interrupt moderation framework (e.g. DIM
> >> for
> >> Linux) could come to the play, I think?
> > Last time, this work is only useful if DIM is not enabled. I don't
> > want to explain it again, I've said it many times to different people
> > in the history discussion, don't you want to check any history discussion?
> I did before I replied. I thought both Halil and Michael tried hard to convey is
> that what you claimed to work by just getting the initial config loaded from
> device will not be sufficient to address all the performance requirements in
> broad and general means other than satisfy the goal set by yourself. What I
> saw is that you kept proving yourself by trying to reference your own use
> case and design - this not just mixed different things up but also ended up
> with circular argument that wasted everyone's time unfortunately.
>
> >>> and for devices
> >>> that serve hundreds of thousands of customers, the device
> >>> implementation has to be comprehensive.
> >> Would you mind elaborating the technical difficulty for why the
> >> device implementation has to be comprehensive to serve hundreds or
> >> thousands of customers (i.e. adaptive moderation in the device), and
> >> is there better way to improve the current interface (ctrlq v.s.
> >> doorbell/MMIO register) that is hardware offload implementation wise.
> >> I feel people reading the thread without understanding the full
> >> background would become even more confused as to why it's relevant to
> >> your proposal, to me it's completely distinct use case or problem area that
> we are talking about.
> > Regardless of whether DIM can be enabled by default (it cannot
> > accelerate all scenarios, right?), therefore, a considerable number of
> > machines on the cloud vendors' line still have to rely on static
> > values to provide boot-up performance.
> Sorry, this is not a good excuse for people to accept a sub-optimal solution.
> Please talk to the other hardware vendor or read my above reply carefully.
>
> -Siwei
>
> > The vendors will also make optimizations to optimize scenarios like
> > ping-pong, so the boot-up performance is very good.
> >
> > Thanks.
> >
> >> Thanks,
> >> -Siwei
> >>
> >>> Thanks.
> >>>
> >>>>>> Unless the
> >>>>>> device is specially hard wired to some fixed guest setup that
> >>>>>> users couldn't change, it doesn't seem logical that the device
> >>>>>> could derive the best or most performant config on driver's
> >>>>>> behalf. What if the guest wants best latency for its load but the
> >>>>>> device just blindlessly guess the guest might prefer throughput
> >>>>>> friendly that it miserably uses latency impacting non-zero default?
> >>>>> The device does not want to guess and cannot guess. This patch
> >>>>> does not force the device to choose a non-zero value, but relaxes
> >>>>> it to allow the device to choose 0 or non-zero, which is very
> >>>>> friendly to virtual machines with different performance requirements,
> right?
> >>>> I don't understand the friendly part - do you imply your VM users
> >>>> are on kinda fixed wired setup that they cannot change these
> >>>> coalescing parameters after driver is loaded? Can the owner of the
> >>>> VM in control apply certain initial config for the coalescing
> >>>> parameters to the VM image? Or is it the problem of the guest
> >>>> driver that doesn't yet expose coalescing parameters to the end
> >>>> user? Otherwise I would think that guest user should be able to set
> >>>> parameters accordingly that would best fit the specific performance
> >>>> requirement of their own. How the device could even help here? I
> >>>> don't feel there's a lot of value to grant the device or host admin
> >>>> the flexibility to policy the *best* config on guest user's behalf,
> >>>> to be honest. And you seem to admit the fact that the default doesn't
> really matter, be it 0 or non-zero.
> >>>>
> >>>>>> Could this device side change for the default config regress boot
> >>>>>> time performance (which may need best latency over throughput)?
> >>>>> Don't make these assumptions, what if the driver needs better
> throughput?
> >>>> There's a misconception here: what we think that driver may need in
> >>>> terms on performance does actually reflect what guest user would
> >>>> like to have. Driver cannot read guest user's mind to make the decision,
> either.
> >>>>
> >>>> In history there was drastic change in the Linux virtio-net driver
> >>>> that ever changed the default disposition for XPS (and RPS as well)
> >>>> from throughput and long-lived connection oriented to concurrency
> >>>> and short-lived oriented, which regressed a lot of existing setups
> >>>> that expects sustained throughput and packet rate after kernel
> upgrade.
> >>>> Although the occurrence of such drastic change for default
> >>>> disposition is not so welcome (that is one of the reasons why I
> >>>> valued consistent initial value and back compatible  behavior), I
> >>>> don't see people yelling at virtio spec for less flexibility of
> >>>> offering the default disposition, given that the guest user can
> >>>> override the config any time with their own tooling or script,
> >>>> there's no problem at all for them to just set the corresponding config
> back explicitly to what it was before.
> >>>>
> >>>>>>>>> And device/driver has better performance, is that a problem?
> Unlikely.
> >>>>>>>>>
> >>>>>> Even for rare case with a hard wired setup, the way to tackle the
> >>>>>> very problem using device's default is still quite questionable.
> >>>>>> Usually the mgmt software or network config utility should be
> >>>>>> equipped with some default value if need be. And we know the
> >>>>>> guest has the best position to impose the best / most performant
> >>>>>> config for its own load. What is the issue or use case that this
> >>>>>> initial config couldn't be applied by the guest mgmt software
> >>>>>> ahead but has to resort to the device to load some default (which
> >>>>>> is odd and irrelevant to any guest load), before the interface is
> brought up for operation i.e. performing I/O?
> >>>>> Use cases are everywhere, Alibaba Cloud, MLX and all other modern
> >>>>> network cards have a default value that is not 0.
> >>>> You seems to be referencing your own setup basically, and the
> >>>> question is still left answered - why the initial config can't be
> >>>> done through the mgmt software or network config utility within the
> guest?
> >>>>
> >>>>> (0 is actually a kind of default value)
> >>>>>
> >>>>>>> Sorry, my vacation just ended.
> >>>>>>>
> >>>>>>>> Yes, it is possible. Driver can cache values it sets and never
> >>>>>>>> query device with get.
> >>>>>>> Don't we already have a lot of behaviors to drive queries from
> devices?
> >>>>>>> RSS context, device stats.
> >>>>>>>
> >>>>>>>> Before anything is set, driver will report incorrect values.
> >>>>>>> Devices that are widely supported and supported by good
> >>>>>>> practices should have any initialization value. Just reporting 0
> >>>>>>> is incorrect value. Although the spec now says so.
> >>>>>> I don't have an aligned view here, sorry. As I recall having 0 as
> >>>>>> the default is just to keep device started in a state where
> >>>>>> coalescing is disabled, so it's backward compatible and
> >>>>>> consistent with a non-coalescing supporting driver - such that it
> >>>>>> won't yield surprising effect (for e.g. regressed latency)
> >>>>>> inadvertently after user's getting driver software upgraded.
> >>>>>> Unlike the other virtio-net features that could 100% improve
> >>>>>> performance in all aspect, this coalescing feature is more of a
> >>>>>> performance tuning knob that may improve performance metrics
> >>>>>> (such as cpu usage or throughput) of one dimension while demoting
> the others (such as latency, jitter or connection rate) from the equation.
> >>>>>> That said, there's not a single and fixed set of default config
> >>>>>> that device could supply which is able to satisfy all kind of guest load.
> >>>>>> Rather than rely on the device to offer a matching default for
> >>>>>> driver (which I think it's technically wrong), I'd lean to having
> >>>>>> guest software or network utility to apply the initial config for
> >>>>>> the guest, where they should have best knowledge for the specific
> >>>>>> guest workload than what device could do.
> >>>>> Before this feature, a good device implementation should also
> >>>>> support coalescing (of course we don't necessarily assume it has
> coalescing).
> >>>> Again, I don't doubt the value of supporting coalescing.
> >>>>
> >>>>> In addition, virtual
> >>>>> machines that tend to favor latency and throughput exist. If the
> >>>>> device supported by the manufacturer needs to provide a
> >>>>> low-latency virtual machine, please continue to keep the default value
> of 0.
> >>>> No, that's not what I was asking. There's no such requirement for
> >>>> any vendor to provide a low-latency or high throughput VM. The more
> >>>> general use case is - the setup for real world workload might just
> >>>> be too complex that end users would prefer low-latency on some
> >>>> virtual NIC or even some specific queues, while the other queues of
> >>>> a virtual NIC, or reset virtual NIC might have very different
> >>>> dispositions. Due to the needs and dynamics of workload scaling up
> >>>> & down, they might have more or less queues or virtual NICs to
> >>>> configure, so these disposition would need to be readjusted at any
> >>>> point of time, for which there's no easy way for device to adapt to
> >>>> easily. The guest user should have best knowledge for the specific
> >>>> guest workload and setup than what device could/should offer.
> >>>>
> >>>>>>>> What will break as a result? Hard to predict.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Given the patch is big, I am inclined to say it just should use
> >>>>>>>> a feature bit.
> >>>>>>> This doesn't seem to break anything in my opinion, we just told
> >>>>>>> the device that now you can set more values.
> >>>>>> Another thing this could break is live migration between devices
> >>>>>> with varied default. How do you make sure the guest doesn't rely
> >>>>>> on some default from the source device, while on the destination
> >>>>>> it just doesn't get the same default coalescingjjj value? To get
> >>>>>> rid of this side effect the guest would still need to apply the
> >>>>>> initial config for its own, anyway... Which eventually would
> >>>>>> render this proposal with arbitrary default rather pointless.
> >>>>> I don't quite understand why this would affect hot migration, the
> >>>>> values would be migrated over.
> >>>> Then I don't see this has been discussed (wouldn't the initial
> >>>> value be part of virtio device state to migrate over?) in the
> >>>> thread or described in the proposed text. And your proposed
> >>>> per-queue coalescing parameters (default plus current value) would
> >>>> have to be described too as part of virtio-net device state, so that
> people don't misunderstand your proposal.
> >>>>
> >>>> Thanks,
> >>>> -Siwei
> >>>>
> >>>>> Thanks.
> >>>>>
> >>>>>> Thanks,
> >>>>>> -Siwei
> >>>>>>> Using new feature bits does not seem necessary.
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>> --
> >>>>>>>> MST
> >>>>>>>>

I saw the need of this proposal slightly differently in the discussion with Heng in v4.
The way I understood is, proposed relaxation enables below Linux driver flow to work as equally as without device offering VIRTIO_NET_F_VQ_NOTF_COAL.

Flow is:
1. The device offered feature VIRTIO_NET_F_VQ_NOTF_COAL
2. The virtio-net driver negotiated VIRTNET_FEATURES that has VIRTIO_NET_F_VQ_NOTF_COAL

3. Because VIRTIO_NET_F_VQ_NOTF_COAL is negotiated, device is not applying any coalescing on the VQ, in a good hope that driver will perform VQ notification coalescing.

4. the virtio-net driver even after negotiating VIRTIO_NET_F_VQ_NOTF_COAL, still kept the its dim disabled.
vi->rq[queue].dim_enabled = false by default.

5. The virtio net driver enables dim only on user request via ethtool.

So here, just because one is using a new driver and new device, it may get lower performance.

Do we agree to above problem description?
If no, I likely missed something in the long discussion and need to read all of it. :(

If yes, few solutions are:
1. A user intervention is must to avoid this regression. The user in the guest VM must enable DIM if the device supports it.
Very hard to do this plumbing.

2. A net driver enables the DIM by default as long as VIRTIO_NET_F_VQ_NOTF_COAL is negotiated.
(without ethtool setting)

3. A device relaxes the limitation and continue to apply the coalescing, until driver overrides it.

In my humble opinion, Heng is solving it using option #3, that tends to work with existing and future drivers who may/may not enable the DIM.