Re: [PATCH v5] virtio-net: clarify coalescing parameters settings

[Heng Qi <hengqi@linux.alibaba.com>]

On Thu, 20 Jun 2024 18:21:03 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> 
> 
> On 6/20/2024 12:40 AM, Heng Qi wrote:
> > On Mon, 17 Jun 2024 16:31:25 -0700, "Si-Wei Liu" <si-wei.liu@oracle.com> wrote:
> >>
> >> On 6/16/2024 7:27 PM, Heng Qi wrote:
> >>> On Thu, 13 Jun 2024 02:13:50 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> >>>> On Tue, Jun 11, 2024 at 05:43:18PM +0000, Parav Pandit wrote:
> >>>>>> From: Michael S. Tsirkin <mst@redhat.com>
> >>>>>> Sent: Tuesday, June 11, 2024 10:00 PM
> >>>>>> How we can we make progress with
> >>>>>> the realease but sure we don't make backwards compat a pain?
> >>>>>>
> >>>>>> Ideas?
> >>>>>>
> >>>>> There is no functional break with this relaxation.
> >>>>> Device set some non-zero defaults and driver didn't modify them....
> >>>>> Anything broken? Unlikely.
> >> Generally it's inappropriate to leave this decision making to the device
> >> for what would be the best / most performant default config, as the
> >> device is generally considered agnostic to the guest load...
> > Instead, the performance of the virtual machine and the driver depends heavily
> > on how the device is implemented, just as we have proposed various ways to
> > offload the data queue in the device to the hardware. The reason why most
> > devices use software to simulate ctrlq instead of using hardware offload is
> > that the driver has no requirements for the performance of ctrlq before, that is,
> > the device implementation is responsible for and meets the driver's performance
> > requirements.
> I am not sure I follow the arguments of ctrlq being s/w or h/w, I 
> thought that it was for the debate why we need coalescing on hardware 
> offload device in the first place, instead of reusing event index or 
> similar s/w based notification suppression. I don't doubt the value of 
> coalescing, but how's it relevant to the default value disposition?  
> Generally the default config disposition is not considered to be part of 
> device implementation, especially when it comes to the situation where 
> device can't easily figure out the specific workload to occur in the 
> guest, and there's no perfect single default value that could meet every 
> single performance metric across the board. This is a typical tuning 
> knob left up to the user to adjust, why the device or driver needs to 
> set or load the initial value? The driver just needs to start with 
> certain value, be it 0 or non-zero, which guest user can override at any 
> point of time, depending on his/her need, and that's it! I guess I still 
> don't understand your user case here, why device / driver default is of 
> such importance.

I've explained that, and I understand your argument is why default value is needed,
and users should be able to adjust them, right?

The default value is working when the user doesn't adjust them.
It's not practical to rely entirely on user adjustment, and for devices
that serve hundreds of thousands of customers, the device implementation
has to be comprehensive.

Thanks.

> 
> >> Unless the
> >> device is specially hard wired to some fixed guest setup that users
> >> couldn't change, it doesn't seem logical that the device could derive
> >> the best or most performant config on driver's behalf. What if the guest
> >> wants best latency for its load but the device just blindlessly guess
> >> the guest might prefer throughput friendly that it miserably uses
> >> latency impacting non-zero default?
> > The device does not want to guess and cannot guess. This patch does not force
> > the device to choose a non-zero value, but relaxes it to allow the device to
> > choose 0 or non-zero, which is very friendly to virtual machines with different
> > performance requirements, right?
> I don't understand the friendly part - do you imply your VM users are on 
> kinda fixed wired setup that they cannot change these coalescing 
> parameters after driver is loaded? Can the owner of the VM in control 
> apply certain initial config for the coalescing parameters to the VM 
> image? Or is it the problem of the guest driver that doesn't yet expose 
> coalescing parameters to the end user? Otherwise I would think that 
> guest user should be able to set parameters accordingly that would best 
> fit the specific performance requirement of their own. How the device 
> could even help here? I don't feel there's a lot of value to grant the 
> device or host admin the flexibility to policy the *best* config on 
> guest user's behalf, to be honest. And you seem to admit the fact that 
> the default doesn't really matter, be it 0 or non-zero.
> 
> >
> >> Could this device side change for
> >> the default config regress boot time performance (which may need best
> >> latency over throughput)?
> > Don't make these assumptions, what if the driver needs better throughput?
> There's a misconception here: what we think that driver may need in 
> terms on performance does actually reflect what guest user would like to 
> have. Driver cannot read guest user's mind to make the decision, either.
> 
> In history there was drastic change in the Linux virtio-net driver that 
> ever changed the default disposition for XPS (and RPS as well) from 
> throughput and long-lived connection oriented to concurrency and 
> short-lived oriented, which regressed a lot of existing setups that 
> expects sustained throughput and packet rate after kernel upgrade. 
> Although the occurrence of such drastic change for default disposition 
> is not so welcome (that is one of the reasons why I valued consistent 
> initial value and back compatible  behavior), I don't see people yelling 
> at virtio spec for less flexibility of offering the default disposition, 
> given that the guest user can override the config any time with their 
> own tooling or script, there's no problem at all for them to just set 
> the corresponding config back explicitly to what it was before.
> 
> >
> >>>>> And device/driver has better performance, is that a problem? Unlikely.
> >>>>>
> >> Even for rare case with a hard wired setup, the way to tackle the very
> >> problem using device's default is still quite questionable. Usually the
> >> mgmt software or network config utility should be equipped with some
> >> default value if need be. And we know the guest has the best position to
> >> impose the best / most performant config for its own load. What is the
> >> issue or use case that this initial config couldn't be applied by the
> >> guest mgmt software ahead but has to resort to the device to load some
> >> default (which is odd and irrelevant to any guest load), before the
> >> interface is brought up for operation i.e. performing I/O?
> > Use cases are everywhere, Alibaba Cloud, MLX and all other modern network cards
> > have a default value that is not 0.
> You seems to be referencing your own setup basically, and the question 
> is still left answered - why the initial config can't be done through 
> the mgmt software or network config utility within the guest?
> 
> > (0 is actually a kind of default value)
> >
> >>> Sorry, my vacation just ended.
> >>>
> >>>> Yes, it is possible. Driver can cache values it sets
> >>>> and never query device with get.
> >>> Don't we already have a lot of behaviors to drive queries from devices?
> >>> RSS context, device stats.
> >>>
> >>>> Before anything is set, driver will report incorrect values.
> >>> Devices that are widely supported and supported by good practices should have
> >>> any initialization value. Just reporting 0 is incorrect value. Although the
> >>> spec now says so.
> >> I don't have an aligned view here, sorry. As I recall having 0 as the
> >> default is just to keep device started in a state where coalescing is
> >> disabled, so it's backward compatible and consistent with a
> >> non-coalescing supporting driver - such that it won't yield surprising
> >> effect (for e.g. regressed latency) inadvertently after user's getting
> >> driver software upgraded. Unlike the other virtio-net features that
> >> could 100% improve performance in all aspect, this coalescing feature is
> >> more of a performance tuning knob that may improve performance metrics
> >> (such as cpu usage or throughput) of one dimension while demoting the
> >> others (such as latency, jitter or connection rate) from the equation.
> >> That said, there's not a single and fixed set of default config that
> >> device could supply which is able to satisfy all kind of guest load.
> >> Rather than rely on the device to offer a matching default for driver
> >> (which I think it's technically wrong), I'd lean to having guest
> >> software or network utility to apply the initial config for the guest,
> >> where they should have best knowledge for the specific guest workload
> >> than what device could do.
> > Before this feature, a good device implementation should also support coalescing
> > (of course we don't necessarily assume it has coalescing).
> Again, I don't doubt the value of supporting coalescing.
> 
> >   In addition, virtual
> > machines that tend to favor latency and throughput exist. If the device supported
> > by the manufacturer needs to provide a low-latency virtual machine, please
> > continue to keep the default value of 0.
> No, that's not what I was asking. There's no such requirement for any 
> vendor to provide a low-latency or high throughput VM. The more general 
> use case is - the setup for real world workload might just be too 
> complex that end users would prefer low-latency on some virtual NIC or 
> even some specific queues, while the other queues of a virtual NIC, or 
> reset virtual NIC might have very different dispositions. Due to the 
> needs and dynamics of workload scaling up & down, they might have more 
> or less queues or virtual NICs to configure, so these disposition would 
> need to be readjusted at any point of time, for which there's no easy 
> way for device to adapt to easily. The guest user should have best 
> knowledge for the specific guest workload and setup than what device 
> could/should offer.
> 
> >
> >>>> What will break as a result? Hard to predict.
> >>>>
> >>>>
> >>>> Given the patch is big, I am inclined to say it just should use
> >>>> a feature bit.
> >>> This doesn't seem to break anything in my opinion, we just told the device that
> >>> now you can set more values.
> >> Another thing this could break is live migration between devices with
> >> varied default. How do you make sure the guest doesn't rely on some
> >> default from the source device, while on the destination it just doesn't
> >> get the same default coalescingjjj value? To get rid of this side effect
> >> the guest would still need to apply the initial config for its own,
> >> anyway... Which eventually would render this proposal with arbitrary
> >> default rather pointless.
> > I don't quite understand why this would affect hot migration, the values
> > would be migrated over.
> Then I don't see this has been discussed (wouldn't the initial value be 
> part of virtio device state to migrate over?) in the thread or described 
> in the proposed text. And your proposed per-queue coalescing parameters 
> (default plus current value) would have to be described too as part of 
> virtio-net device state, so that people don't misunderstand your proposal.
> 
> Thanks,
> -Siwei
> 
> > Thanks.
> >
> >>
> >> Thanks,
> >> -Siwei
> >>> Using new feature bits does not seem necessary.
> >>>
> >>> Thanks.
> >>>
> >>>> -- 
> >>>> MST
> >>>>
>