[virtio-dev] Re: [virtio-comment] [PATCH 5/5] virtio-pci: implement VIRTIO_F_QUEUE_STATE

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 9/12/2023 5:21 PM, Parav Pandit wrote:
>> From: Zhu, Lingshan <lingshan.zhu@intel.com>
>> Sent: Tuesday, September 12, 2023 2:33 PM
>> admin vq require fixed and dedicated resource to serve the VMs, the question
>> still remains, does is scale to server big amount of devices migration? how many
>> admin vqs do you need to serve 10 VMs, how many for 100? and so on? How to
>> scale?
>>
> Yes, it scales within the AQ and across multiple AQs.
> Please consult your board designers to know such limits for your device.
scales require multiple AQs, then how many should a vendor provide for the
worst case?

I am boring for the same repeating questions.
>   
>> If one admin vq can serve 100 VMs, can it migrate 1000VMs in reasonable time?
>> If not, how many exactly.
>>
> Yes, it can serve both 100 and 1000 VMs in reasonable time.
I am not sure, the aq is limitless? Can serve thousands of VMs
in a reasonable time? Like in 300ms?

If you say, that require multiple AQ, then how many should a vendor provide?

Don't say the board designer own the risks.
>
>> And register does not need to scale, it resides on the VF and only serve
>> the VF.
>>
> Since its per VF, by nature it is linearly growing entity that the board design needs to support read and write with guaranteed timing.
> It clearly scaled poor than queue.
Please read my series. For example, we introduce a new bit SUSPEND in 
the \field{device status}, any scalability issues here?
>   
>> It does not reside on the PF to migrate the VFs.
> Hence it does not scale and cannot do parallel operation within the VF, unless each register is replicated.
Why its not scale? It is a per device facility.
Why do you need parallel operation against the LM facility?
That doesn't make a lot of sense.
>
> Using register of a queue for bulk data transfer is solved question when the virtio spec was born.
> I don’t see a point to discuss it.
> Snippet from spec: " As a device can have zero or more virtqueues for bulk data transport"
Where do you see the series intends to transfer bulk data through registers?
>
>> VFs config space can use the device dedicated resource like the bandwidth.
>>
>> for AQ, still you need to reserve resource and how much?
> It depends on your board, please consult your board designer to know depending on the implementation.
>  From spec point of view, it should not be same as any other virtqueue.
so the vendor own the risk to implement AQ LM? Why they have to?
>>> No. I do not agree. It can fail and very hard for board designers.
>>> AQs are more reliable way to transport bulk data in scalable manner for tens
>> of member devices.
>> Really? How often do you observe virtio config space fail?
> On Intel Icelake server we have seen it failing with 128 VFs.
> And device needs to do very weird things to support 1000+ VFs forever expanding config space, which is not the topic of this discussion anyway.
That is your setup problem.
>
>
>>>> Please allow me to provide an extreme example, is one single admin vq
>>>> limitless, that can serve hundreds to thousands of VMs migration?
>>> It is left to the device implementation. Just like RSS and multi queue support?
>>> Is one Q enough for 800Gbps to 10Mbps link?
>>> Answer is: Not the scope of specification, spec provide the framework to scale
>> this way, but not impose on the device.
>> Even if not support RSS or MQ, the device still can work with
>> performance overhead, not fail.
>>
> _work_ is subjective.
> The financial transaction (application) failed. Packeted worked.
> LM commands were successful, but it was not timely.
>
> Same same..
>   
>> Insufficient bandwidth & resource caused live migration fail is totally
>> different.
> Very abstract point and unrelated to administration commands.
It is your design facing the problem.
>
>>>> If not, two or
>>>> three or what number?
>>> It really does not matter. Its wrong point to discuss here.
>>> Number of queues and command execution depends on the device
>> implementation.
>>> A financial transaction application can timeout when a device queuing delay
>> for virtio net rx queue is long.
>>> And we don’t put details about such things in specification.
>>> Spec takes the requirements and provides driver device interface to
>> implement and scale.
>>> I still don’t follow the motivation behind the question.
>>> Is your question: How many admin queues are needed to migrate N member
>> devices? If so, it is implementation specific.
>>> It is similar to how such things depend on implementation for 30 virtio device
>> types.
>>> And if are implying that because it is implementation specific, that is why
>> administration queue should not be used, but some configuration register
>> should be used.
>>> Than you should propose a config register interface to post virtqueue
>> descriptors that way for 30 device types!
>> if so, leave it as undefined? A potential risk for device implantation?
>
>> Then why must the admin vq?
> Because administration commands and admin vq does not impose devices to implement thousands of registers which must have time bound completion guarantee.
> The large part of industry including SIOV devices led by Intel and others are moving away from register access mode.
>
> To summarize, administration commands and queue offer following benefits.
>
> 1. Ability to do bulk data transfer between driver and device
>
> 2. Ability to parallelize the work within driver and within device within single or multiple virtqueues
>
> 3. Eliminates implementing PCI read/write MMIO registers which demand low latency response interval
>
> 4. Better utilize host cpu as no one needs to poll on the device register for completion
>
> 5. Ability to handle variability in command completion by device and ability to notify the driver
>
> If this does not satisfy you, please refer to some of the past email discussions during administration virtuqueue time.
I think you mixed up the facility and the implementation in my series, 
please read.


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org