[virtio-dev] RE: [virtio-comment] [PATCH 5/5] virtio-pci: implement VIRTIO_F_QUEUE_STATE

Parav Pandit
2023-09-11T08:12:15Z

> From: Zhu, Lingshan <lingshan.zhu@intel.com>
> Sent: Monday, September 11, 2023 1:28 PM

> > Basic facilities are added in [1] for passthrough devices.
> > You can leverage them in your v2 for supporting p2p devices, dirty page
> tracking, passthrough support, shorter downtime and more.
> Basic facilities should be better not depend on others, but admin vq can re-use
> the basic facilities.
>
> For P2P, what if the devices are placed in different IOMMU group?
IOMMU grouping is one OS specific notion of an older API.
Hypervisor needs to do right setup anyway for using PCI spec define access control and other semantics which is outside the scope of [1].
It is outside primarily because proposal [1] is not migrating the whole "PCI device".
It is migrating the virtio device, so that we can migrate from PCI VF member to some software based device too.
And vis-versa.

> > QOS is such a broad term that is hard to debate unless you get to a specific
> point.
> E.g., there can be hundreds or thousands of VMs, how many admin vq are
> required to serve them when LM? To converge, no timeout.
How many RSS queues are required to reach 800Gbs NIC performance at what q depth at what interrupt moderation level?
Such details are outside the scope of virtio specification.
Those are implementation details of the device.

Similarly here for AQ too.
The inherent nature of AQ to queue commands and execute them out of order in the device is the fundamental reason, AQ is introduced.
And one can have more AQs to do unrelated work, mainly from the hypervisor owner device who wants to enqueue unrelated commands in parallel.

> >> E.g., do you need 100 admin vqs for 1000 VMs? How do you decide the
> >> number in HW implementation and how does the driver get informed?
> > Usually just one AQ is enough as proposal [1] is built around inherent
> downtime reduction.
> > You can ask similar question for RSS, how does hw device how many RSS
> > queues are needed. 😊
> > Device exposes number of supported AQs that driver is free to use.
> RSS is not a must for the transition through maybe performance overhead.
> But if the host can not finish Live Migration in the due time, then it is a failed
> LM.
It can aborts the LM and restore it back by resuming the device.

> >
> > Most sane sys admins do not migrate 1000 VMs at same time for obvious
> reasons.
> > But when such requirements arise, a device may support it.
> > Just like how a net device can support from 1 to 32K txqueues at spec level.
> The orchestration layer may do that for host upgrade or power-saving.
> And the VMs may be required to migrate together, for example:
> a cluster of VMs in the same subnet.
>
Sure. AQ of depth 1K can support 1K outstanding commands at a time for 1000 member devices.

> Lets do not introduce new frangibility
I don’t see any frangibility added by [1].
If you see one, please let me know.

> >

> >>>> Remember CSP migrates all VMs on a host for powersaving or upgrade.
> >>> I am not sure why the migration reason has any influence on the design.
> >> Because this design is for live migration.
> >>> The CSPs that we had discussed, care for performance more and hence
> >> prefers passthrough instead or mediation and don’t seem to be doing
> >> any nesting.
> >>> CPU doesnt have support for 3 level of page table nesting either.
> >>> I agree that there could be other users who care for nested functionality.
> >>>
> >>> Any ways, nesting and non-nesting are two different requirements.
> >> The LM facility should server both,
> > I don’t see how PCI spec let you do it.
> > PCI community already handed over this to SR-PCIM interface outside of the
> PCI spec domain.
> > Hence, its done over admin queue for passthrough devices.
> >
> > If you can explain, how your proposal addresses passthrough support without
> mediation and also does DMA, I am very interested to learn that.
> Do you mean nested?
Before nesting, just like to see basic single level passthrough to see functional and performant like [1].

> Why this series can not support nested?
I don’t see all the aspects that I covered in series [1] ranging from flr, device context migration, virtio level reset, dirty page tracking, p2p support, etc. covered in some device, vq suspend resume piece.

[1] https://lists.oasis-open.org/archives/virtio-comment/202309/msg00061.html

> >> And it does not serve bare-metal live migration either.
> > A bare-metal migration seems a distance theory as one need side cpu,
> memory accessor apart from device accessor.
> > But somehow if that exists, there will be similar admin device to migrate it
> may be TDDISP will own this whole piece one day.
> Bare metal live migration require other components like firmware OS and
> partitioning, that's why the device live migration should not be a blocker.
Device migration is not blocker.
In-fact it facilitates for this future in case if that happens where side cpu like DPU or similar sideband virtio admin device can migrate over its admin vq.

Long ago when admin commands were discussed, this was discussed too where a admin device may not be an owner device.