Re: [virtio-comment] About the plan of Admin Queue

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 7/3/2023 1:54 PM, Xuan Zhuo wrote:
> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
>>>> open.org> On Behalf Of Zhu, Lingshan
>>>> Sent: Friday, June 30, 2023 6:33 AM
>>>>
>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
>>>> backend?
>>> Yes, why not.
>>>
>>>>>>> The key point is who want to create a new device.
>>>>>> DPU can come with a certain number of pre-created ADIs, just make
>>>>>> sure the orchestration SW is aware of their device IDs.
>>>>>>
>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
>>> Xuan partly explained it.
>>>
>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
>>>>>> wants to create and hot plug in a device to a guest?
>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
>>>>>> machines?
>>>>> In your scenario, the supervisor is on the os, which controls the DPU
>>>>> to create new devices.
>>>>>
>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
>>>>> entire host is for users. Of course, there are situations where the
>>>>> vendor manager are in the HOST. But for bare metal machines, the host
>>>>> belongs to the customer, the vendor manager is only in the DPU.
>>>>>
>>>>> So when the customers buy a new nic for the host, the vendor manager
>>>>> will plug a device to the host from the DPU.
>>>> I understand once a customer orders a new NIC, you wants to present the NIC
>>>> to the host.
>>>> However you only owns the DPU and the customer owns the host, that means
>>>> this creation and hot plug must be transparent to the host and there may not be
>>>> a host driver help handling an interrupt/probe.
>>>>
>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
>>>
>>>> However this is not PCI which has a tree/switch and can enumerate devices to
>>>> the host by spanning the device across the PCI hierarchy.
>>>>
>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
>>>
>>>> To address an ADI, there is only a device_id.
>>>>
>>> SIOV device must have a unique identifier at PCI bus level for sure.
>>> I cannot speak more about it in this forum due to other logistics issue.
>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
>>>
>>>> So, do you mind share how your DPU offload the device model? What kind of
>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
>>>> its own?
>>>>
>>> It is a virtio nic, blk and other virtio devices for us.
>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
>>>
>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
>>> You might find a Linux reference model useful at [1].
>>> A usage model already exists in one OS and in use for non virtio devices.
>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
>>>
>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
>>>
>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
>> well I see SF facing the similar challenge, I can add a command for the
>> driver to query all existing SIOV ADIs of a device,
>> and the device return ADIs id and status. Looks good? and work for you
>> @Xuan?
> Can the device notifies the driver by a interrupt?
Of course this can be an option as previously discussed.
That would be a config interrupt(with proper suppression) from the 
management device,
which requires a new cap in a bar, then you still need to fetch the 
information.

A proposal:

a new cap, at least contains:
total number of the ADIs.

Then once any ADIs are created or removed, the PF driver get a notification.

Works for you?
>
> Thanks.
>
>
>> Thanks
>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/