[virtio-dev] RE: [virtio-comment] [PATCH 5/5] virtio-pci: implement VIRTIO_F_QUEUE_STATE

Parav Pandit
2023-09-13T06:39:50Z

> From: Jason Wang <jasowang@redhat.com>
> Sent: Wednesday, September 13, 2023 10:15 AM

[..]
> > > > [1]
> > > > https://lists.oasis-open.org/archives/virtio-comment/202309/msg000
> > > > 61.h
> > > > tml
> > >
> > > The series works for stateless devices. Before we introduce device
> > > states in the spec, we can't migrate stateful devices. So the device
> > > context doesn't make much sense right now.
> > The series works for stateful devices too. The device context covers it.
>
> How? Can it be used for migrating any existing stateful devices? Don't we need
> to define what context means for a specific stateful device before you can
> introduce things like device context? Please go through the archives for the
> relevant discussions (e.g virtio-FS), it's not as simple as introducing a device
> context API.
>
A device will have its own context for example RSS definition, or flow filters tomorrow.
The device context will be extended post the first series.

> And what's more, how can it handle the migration compatibility?
It will be taken care in follow on as we all know that this to be checked.
I will include the notes of future follow up work items in v1, which will be taken care post this series.

> > > Dirty page tracking in virtio is not a must for live migration to
> > > work. It can be done via platform facilities or even software. And
> > > to make it more efficient, it needs to utilize transport facilities instead of a
> general one.
> > >
> > It is also optional in the spec proposal.
> > Most platforms claimed are not able to do efficiently either,
>
> Most platforms are working towards an efficient way. But we are talking about
> different things, hardware based dirty page logging is not a must, that is what
> I'm saying. For example, KVM doesn't use hardware to log dirty pages.
>
I also said same, that hw based dirty page logging is not must. :)
One day hw mmu will be able to track everything efficiently. I have not seen it happening yet.

> > hence the vfio subsystem added the support for it.
>
> As an open standard, if it is designed for a specific software subsystem on a
> specific OS, it's a failure.
>
It is not.
One need accept that, in certain areas virtio is following the trails of advancement already done in sw stack.
So that virtio spec advancement fits in to supply such use cases.
And blocking such advancement of virtio spec to promote only_mediation approach is not good either.

BTW: One can say the mediation approach is also designed for specific software subsystem and hence failure.
I will stay away from quoting it, as I don’t see it this way.

> >
> > > The FLR, P2P demonstrates the fragility of a simple passthrough
> > > method and how it conflicts with live migration and complicates the device
> implementation.
> > Huh, it shows the opposite.
> > It shows that both will seamlessly work.
>
> Have you even tried your proposal with a prototype device?
Of course, it is delivered to user for 1.5 years ago before bringing it to the spec with virtio-net and virtio-blk devices.

>
> >
> > > And it means you need to audit all PCI features and do workaround if
> > > there're any possible issues (or using a whitelist).
> > No need for any of this.
>
> You need to prove this otherwise it's fragile. It's the duty of the author to justify
> not the reviewer.
>
One cannot post patches and nor review giant series in one go.
Hence the work to be split on a logical boundary.
Features provisioning, pci layout etc is secondary tasks to take care of.

> For example FLR is required to be done in 100ms. How could you achieve this
> during the live migration? How does it affect the downtime and FRS?
>
Good technical question to discuss instead of passthrough vs mediation. :)

Device administration work is separate from the device operational part.
The device context records what is the current device context, when the FLR occurs, the device stops all the operations.
And on next read of the device context the FLRed context is returned.

> >
> > > This is tricky and we are migrating virtio not virtio-pci. If we
> > > don't use simple passthrough we don't need to care about this.
> > >
> > Exactly, we are migrating virtio device for the PCI transport.
>
> No, the migration facility is a general requirement for all transport.
It is for all transport. One can extend when do for MMIO.

> Starting from a PCI specific (actually your proposal does not even cover all even
> for PCI) solution which may easily end up with issues in other transports.
>
Like?

> Even if you want to migrate virtio for PCI, please at least read Qemu migration
> codes for virtio and PCI, then you will soon realize that a lot of things are
> missing in your proposal.
>
Device context is something that will be extended.
VIRTIO_PCI_CAP_PCI_CFG will also be added as optional item for PCI transport.

> > As usual, if you have to keep arguing about not doing passhthrough, we are
> surely past that point.
>
> Who is "we"?
>
We = You and me.
From 2021, you keep objecting that passthrough must not be done.
And blocking the work done by other technical committee members to improve the virtio spec to make that happen is simply wrong.

> Is something like what you said here passed the vote and written
> to the spec?
Not only me.
The virtio technical committee has agreed for nested and hardware-based implementation _both_.

" hardware-based implementations" is part of the virtio specification charter with ballot of [1].

[1] https://lists.oasis-open.org/archives/virtio/202104/msg00038.html

And passthrough hardware-based device is in the charter that we strive to support.

> We all know the current virtio spec is not built upon passthrough.

This efforts improve the passthrough hw based implementation that should not be blocked.

> > Virtio does not need to stay in the weird umbrella to always mediate etc.
>
> It's not the mediation, we're not doing vDPA, the device model we had in
> hardware and we present to guests are all virtio devices. It's the trap and
> emulation which is fundamental in the world of virtualization for the past
> decades. It's the model we used to virtualize standard devices. If you want to
> debate this methodology, virtio community is clearly the wrong forum.
>
I am not debating it at all. You keep bringing up the point of mediation.

The proposal of [1] is clear that wants to do hardware based passthrough devices with least amount of virtio level mediation.

So somewhere mode of virtualizing has been used, that’s fine, it can continue with full virtualization, mediation,

And also hardware based passthrough device.

> >
> > Series [1] will be enhanced further to support virtio passthrough device for
> device context and more.
> > Even further we like to extend the support.
> >
> > > Since the functionality proposed in this series focus on the minimal
> > > set of the functionality for migration, it is virtio specific and
> > > self contained so nothing special is required to work in the nest.
> >
> > Maybe it is.
> >
> > Again, I repeat and like to converge the admin commands between
> passthrough and non-passthrough cases.
>
> You need to prove at least that your proposal can work for the passthrough
> before we can try to converge.
>
What do you mean by "prove"? virtio specification development is not proof based method.

If you want to participate, please review the patches and help community to improve.

> > If we can converge it is good.
> > If not both modes can expand.
> > It is not either or as use cases are different.
>
> Admin commands are not the cure for all, I've stated drawbacks in other
> threads. Not repeating it again here.
He he, sure, I am not attempting to cure all.
One solution does not fit all cases.
Admin commands are used to solve the specific problem for which the AQ is designed for.

One can make argument saying take pci fabric to 10 km distance, don’t bring new virtio tcp transport...

Drawing boundaries around virtio spec in certain way only makes it further inferior. So please do not block advancements bring in [1].
We really would like to make it more robust with your rich experience and inputs, if you care to participate.

[1] https://lists.oasis-open.org/archives/virtio-comment/202309/msg00061.html