[virtio-dev] Re: [virtio-comment] [PATCH 5/5] virtio-pci: implement VIRTIO_F_QUEUE_STATE

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 9/11/2023 4:12 PM, Parav Pandit wrote:
>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
>> Sent: Monday, September 11, 2023 1:28 PM
>
>>> Basic facilities are added in [1] for passthrough devices.
>>> You can leverage them in your v2 for supporting p2p devices, dirty page
>> tracking, passthrough support, shorter downtime and more.
>> Basic facilities should be better not depend on others, but admin vq can re-use
>> the basic facilities.
>>
>> For P2P, what if the devices are placed in different IOMMU group?
> IOMMU grouping is one OS specific notion of an older API.
> Hypervisor needs to do right setup anyway for using PCI spec define access control and other semantics which is outside the scope of [1].
> It is outside primarily because proposal [1] is not migrating the whole "PCI device".
> It is migrating the virtio device, so that we can migrate from PCI VF member to some software based device too.
> And vis-versa.
Since you talked about P2P, IOMMU is basically for address space 
isolation. For security reasons, it is usually
suggest to passthrough all devices in one IOMMU group to a single guest.

That means, if you want the VF to perform P2P with the PF there the AQ 
resides, you have to place them in the same
IOMMU group and passthrough them all to a guest. So how this AQ serve 
other purposes?
>
>>> QOS is such a broad term that is hard to debate unless you get to a specific
>> point.
>> E.g., there can be hundreds or thousands of VMs, how many admin vq are
>> required to serve them when LM? To converge, no timeout.
> How many RSS queues are required to reach 800Gbs NIC performance at what q depth at what interrupt moderation level?
> Such details are outside the scope of virtio specification.
> Those are implementation details of the device.
>
> Similarly here for AQ too.
> The inherent nature of AQ to queue commands and execute them out of order in the device is the fundamental reason, AQ is introduced.
> And one can have more AQs to do unrelated work, mainly from the hypervisor owner device who wants to enqueue unrelated commands in parallel.
As pointed above insufficient RSS capabilities may cause performance 
overhead, not not a failure, the device still stay functional.
But too few AQ to serve too high volume of VMs may be a problem.
Yes the number of AQs are negotiable, but how many exactly should the HW 
provide?

Naming a number or an algorithm for the ratio of devices / num_of_AQs is 
beyond this topic, but I made my point clear.
>
>>>> E.g., do you need 100 admin vqs for 1000 VMs? How do you decide the
>>>> number in HW implementation and how does the driver get informed?
>>> Usually just one AQ is enough as proposal [1] is built around inherent
>> downtime reduction.
>>> You can ask similar question for RSS, how does hw device how many RSS
>>> queues are needed. 😊
>>> Device exposes number of supported AQs that driver is free to use.
>> RSS is not a must for the transition through maybe performance overhead.
>> But if the host can not finish Live Migration in the due time, then it is a failed
>> LM.
> It can aborts the LM and restore it back by resuming the device.
aborts means fail
>
>>> Most sane sys admins do not migrate 1000 VMs at same time for obvious
>> reasons.
>>> But when such requirements arise, a device may support it.
>>> Just like how a net device can support from 1 to 32K txqueues at spec level.
>> The orchestration layer may do that for host upgrade or power-saving.
>> And the VMs may be required to migrate together, for example:
>> a cluster of VMs in the same subnet.
>>
> Sure. AQ of depth 1K can support 1K outstanding commands at a time for 1000 member devices.
PCI transition is FIFO, can depth = 1K introduce significant latency? 
And 1K depths is
almost identical to 2 X 500 queue depths, so still the same problem, how 
many resource
does the HW need to reserve to serve the worst case?

Let's forget the numbers, the point is clear.
>
>> Lets do not introduce new frangibility
> I don’t see any frangibility added by [1].
> If you see one, please let me know.
The resource and latency explained above.
>
>>>>>> Remember CSP migrates all VMs on a host for powersaving or upgrade.
>>>>> I am not sure why the migration reason has any influence on the design.
>>>> Because this design is for live migration.
>>>>> The CSPs that we had discussed, care for performance more and hence
>>>> prefers passthrough instead or mediation and don’t seem to be doing
>>>> any nesting.
>>>>> CPU doesnt have support for 3 level of page table nesting either.
>>>>> I agree that there could be other users who care for nested functionality.
>>>>>
>>>>> Any ways, nesting and non-nesting are two different requirements.
>>>> The LM facility should server both,
>>> I don’t see how PCI spec let you do it.
>>> PCI community already handed over this to SR-PCIM interface outside of the
>> PCI spec domain.
>>> Hence, its done over admin queue for passthrough devices.
>>>
>>> If you can explain, how your proposal addresses passthrough support without
>> mediation and also does DMA, I am very interested to learn that.
>> Do you mean nested?
> Before nesting, just like to see basic single level passthrough to see functional and performant like [1].
I think we have discussed about this, the nested guest is not aware of 
the admin vq and can not access it,
because the admin vq is a host facility.
>
>> Why this series can not support nested?
> I don’t see all the aspects that I covered in series [1] ranging from flr, device context migration, virtio level reset, dirty page tracking, p2p support, etc. covered in some device, vq suspend resume piece.
>
> [1] https://lists.oasis-open.org/archives/virtio-comment/202309/msg00061.html
We have discussed many other issues in this thread.
>
>>>> And it does not serve bare-metal live migration either.
>>> A bare-metal migration seems a distance theory as one need side cpu,
>> memory accessor apart from device accessor.
>>> But somehow if that exists, there will be similar admin device to migrate it
>> may be TDDISP will own this whole piece one day.
>> Bare metal live migration require other components like firmware OS and
>> partitioning, that's why the device live migration should not be a blocker.
> Device migration is not blocker.
> In-fact it facilitates for this future in case if that happens where side cpu like DPU or similar sideband virtio admin device can migrate over its admin vq.
>
> Long ago when admin commands were discussed, this was discussed too where a admin device may not be an owner device.
The admin vq can not migrate it self therefore baremetal can not be 
migrated by admin vq


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org