[virtio-comment] About the plan of Admin Queue

[virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

Hi,

hi, I would want to know some plans and progress about admin queue.

At the current spec, it seems that there is only one framework and no
specific commands. I'd like to know if anyone is currently working on this and
what the plans are.

We also faced some similar issues, and we think admin queue is a good way to
manage sr-iov.


Thanks.




This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
> Hi,
>
> hi, I would want to know some plans and progress about admin queue.
>
> At the current spec, it seems that there is only one framework and no
> specific commands. I'd like to know if anyone is currently working on this and
> what the plans are.
>
> We also faced some similar issues, and we think admin queue is a good way to
> manage sr-iov.
>
>
> Thanks.
I plan to rebase original transport vq on admin vq.

https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html

Thanks,
Zhu Lingshan
>
>
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
> > Hi,
> >
> > hi, I would want to know some plans and progress about admin queue.
> >
> > At the current spec, it seems that there is only one framework and no
> > specific commands. I'd like to know if anyone is currently working on this and
> > what the plans are.
> >
> > We also faced some similar issues, and we think admin queue is a good way to
> > manage sr-iov.
> >
> >
> > Thanks.
> I plan to rebase original transport vq on admin vq.
>
> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html


I review the patch, that is for S-IOV, right?

I think it is good.

I would if all is configured by the transport vq/admin vq from the OS?

Can we create a managed dev from the backend?

Such as, the DPU sends a command to the driver, then the driver creates a new
managed dev.

Thanks.


>
> Thanks,
> Zhu Lingshan
> >
> >
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 6/30/2023 1:54 PM, Xuan Zhuo wrote:
> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
>>> Hi,
>>>
>>> hi, I would want to know some plans and progress about admin queue.
>>>
>>> At the current spec, it seems that there is only one framework and no
>>> specific commands. I'd like to know if anyone is currently working on this and
>>> what the plans are.
>>>
>>> We also faced some similar issues, and we think admin queue is a good way to
>>> manage sr-iov.
>>>
>>>
>>> Thanks.
>> I plan to rebase original transport vq on admin vq.
>>
>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
>
> I review the patch, that is for S-IOV, right?
Yes, for SIOV and other similar devices
>
> I think it is good.
>
> I would if all is configured by the transport vq/admin vq from the OS?
For SIOV ADIs, this transport vq is the transport layer, so they are 
configured by the OS through transport vq.
>
> Can we create a managed dev from the backend?
>
> Such as, the DPU sends a command to the driver, then the driver creates a new
> managed dev.
I think the group owner, usually the PCI PF is the management device.

Thanks
>
> Thanks.
>
>
>> Thanks,
>> Zhu Lingshan
>>>
>>>
>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>
>>
>> This publicly archived list offers a means to provide input to the
>> OASIS Virtual I/O Device (VIRTIO) TC.
>>
>> In order to verify user consent to the Feedback License terms and
>> to minimize spam in the list archive, subscription is required
>> before posting.
>>
>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>> List help: virtio-comment-help@lists.oasis-open.org
>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>> Committee: https://www.oasis-open.org/committees/virtio/
>> Join OASIS: https://www.oasis-open.org/join/
>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/30/2023 1:54 PM, Xuan Zhuo wrote:
> > On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
> >>> Hi,
> >>>
> >>> hi, I would want to know some plans and progress about admin queue.
> >>>
> >>> At the current spec, it seems that there is only one framework and no
> >>> specific commands. I'd like to know if anyone is currently working on this and
> >>> what the plans are.
> >>>
> >>> We also faced some similar issues, and we think admin queue is a good way to
> >>> manage sr-iov.
> >>>
> >>>
> >>> Thanks.
> >> I plan to rebase original transport vq on admin vq.
> >>
> >> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
> >
> > I review the patch, that is for S-IOV, right?
> Yes, for SIOV and other similar devices
> >
> > I think it is good.
> >
> > I would if all is configured by the transport vq/admin vq from the OS?
> For SIOV ADIs, this transport vq is the transport layer, so they are
> configured by the OS through transport vq.
> >
> > Can we create a managed dev from the backend?
> >
> > Such as, the DPU sends a command to the driver, then the driver creates a new
> > managed dev.
> I think the group owner, usually the PCI PF is the management device.


I mean the DPU hot-plug a new device. Not the managament device create a new
device.

The managament device is in the OS, we want the device is plugged by the DPU.

Thanks.


>
> Thanks
> >
> > Thanks.
> >
> >
> >> Thanks,
> >> Zhu Lingshan
> >>>
> >>>
> >>>
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>> Committee: https://www.oasis-open.org/committees/virtio/
> >>> Join OASIS: https://www.oasis-open.org/join/
> >>>
> >>
> >> This publicly archived list offers a means to provide input to the
> >> OASIS Virtual I/O Device (VIRTIO) TC.
> >>
> >> In order to verify user consent to the Feedback License terms and
> >> to minimize spam in the list archive, subscription is required
> >> before posting.
> >>
> >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >> List help: virtio-comment-help@lists.oasis-open.org
> >> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >> Committee: https://www.oasis-open.org/committees/virtio/
> >> Join OASIS: https://www.oasis-open.org/join/
> >>
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 6/30/2023 3:46 PM, Xuan Zhuo wrote:
> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote:
>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
>>>>> Hi,
>>>>>
>>>>> hi, I would want to know some plans and progress about admin queue.
>>>>>
>>>>> At the current spec, it seems that there is only one framework and no
>>>>> specific commands. I'd like to know if anyone is currently working on this and
>>>>> what the plans are.
>>>>>
>>>>> We also faced some similar issues, and we think admin queue is a good way to
>>>>> manage sr-iov.
>>>>>
>>>>>
>>>>> Thanks.
>>>> I plan to rebase original transport vq on admin vq.
>>>>
>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
>>> I review the patch, that is for S-IOV, right?
>> Yes, for SIOV and other similar devices
>>> I think it is good.
>>>
>>> I would if all is configured by the transport vq/admin vq from the OS?
>> For SIOV ADIs, this transport vq is the transport layer, so they are
>> configured by the OS through transport vq.
>>> Can we create a managed dev from the backend?
>>>
>>> Such as, the DPU sends a command to the driver, then the driver creates a new
>>> managed dev.
>> I think the group owner, usually the PCI PF is the management device.
>
> I mean the DPU hot-plug a new device. Not the managament device create a new
> device.
>
> The managament device is in the OS, we want the device is plugged by the DPU.
The PCI management(SIOV group owner) device is on the DPU, when create 
an ADI,
OS send a command to the DPU/PF through transport vq,
then the PF hot plugged in a new ADI through the transport vq
specific channel. Or did I misunderstand your question?

Thanks
>
> Thanks.
>
>
>> Thanks
>>> Thanks.
>>>
>>>
>>>> Thanks,
>>>> Zhu Lingshan
>>>>>
>>>>>
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>
>>>> This publicly archived list offers a means to provide input to the
>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>
>>>> In order to verify user consent to the Feedback License terms and
>>>> to minimize spam in the list archive, subscription is required
>>>> before posting.
>>>>
>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/30/2023 3:46 PM, Xuan Zhuo wrote:
> > On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/30/2023 1:54 PM, Xuan Zhuo wrote:
> >>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
> >>>>> Hi,
> >>>>>
> >>>>> hi, I would want to know some plans and progress about admin queue.
> >>>>>
> >>>>> At the current spec, it seems that there is only one framework and no
> >>>>> specific commands. I'd like to know if anyone is currently working on this and
> >>>>> what the plans are.
> >>>>>
> >>>>> We also faced some similar issues, and we think admin queue is a good way to
> >>>>> manage sr-iov.
> >>>>>
> >>>>>
> >>>>> Thanks.
> >>>> I plan to rebase original transport vq on admin vq.
> >>>>
> >>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
> >>> I review the patch, that is for S-IOV, right?
> >> Yes, for SIOV and other similar devices
> >>> I think it is good.
> >>>
> >>> I would if all is configured by the transport vq/admin vq from the OS?
> >> For SIOV ADIs, this transport vq is the transport layer, so they are
> >> configured by the OS through transport vq.
> >>> Can we create a managed dev from the backend?
> >>>
> >>> Such as, the DPU sends a command to the driver, then the driver creates a new
> >>> managed dev.
> >> I think the group owner, usually the PCI PF is the management device.
> >
> > I mean the DPU hot-plug a new device. Not the managament device create a new
> > device.
> >
> > The managament device is in the OS, we want the device is plugged by the DPU.
> The PCI management(SIOV group owner) device is on the DPU, when create
> an ADI,
> OS send a command to the DPU/PF through transport vq,
> then the PF hot plugged in a new ADI through the transport vq
> specific channel. Or did I misunderstand your question?

Your first step is the os send a command. Right?

Can we let the DPU notify the driver to create a new devicer from the backend?

The key point is who want to create a new device.

Thanks.




>
> Thanks
> >
> > Thanks.
> >
> >
> >> Thanks
> >>> Thanks.
> >>>
> >>>
> >>>> Thanks,
> >>>> Zhu Lingshan
> >>>>>
> >>>>>
> >>>>> This publicly archived list offers a means to provide input to the
> >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>
> >>>>> In order to verify user consent to the Feedback License terms and
> >>>>> to minimize spam in the list archive, subscription is required
> >>>>> before posting.
> >>>>>
> >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>
> >>>> This publicly archived list offers a means to provide input to the
> >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>
> >>>> In order to verify user consent to the Feedback License terms and
> >>>> to minimize spam in the list archive, subscription is required
> >>>> before posting.
> >>>>
> >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>> Committee: https://www.oasis-open.org/committees/virtio/
> >>> Join OASIS: https://www.oasis-open.org/join/
> >>>
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 6/30/2023 3:56 PM, Xuan Zhuo wrote:
> On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/30/2023 3:46 PM, Xuan Zhuo wrote:
>>> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote:
>>>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> hi, I would want to know some plans and progress about admin queue.
>>>>>>>
>>>>>>> At the current spec, it seems that there is only one framework and no
>>>>>>> specific commands. I'd like to know if anyone is currently working on this and
>>>>>>> what the plans are.
>>>>>>>
>>>>>>> We also faced some similar issues, and we think admin queue is a good way to
>>>>>>> manage sr-iov.
>>>>>>>
>>>>>>>
>>>>>>> Thanks.
>>>>>> I plan to rebase original transport vq on admin vq.
>>>>>>
>>>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
>>>>> I review the patch, that is for S-IOV, right?
>>>> Yes, for SIOV and other similar devices
>>>>> I think it is good.
>>>>>
>>>>> I would if all is configured by the transport vq/admin vq from the OS?
>>>> For SIOV ADIs, this transport vq is the transport layer, so they are
>>>> configured by the OS through transport vq.
>>>>> Can we create a managed dev from the backend?
>>>>>
>>>>> Such as, the DPU sends a command to the driver, then the driver creates a new
>>>>> managed dev.
>>>> I think the group owner, usually the PCI PF is the management device.
>>> I mean the DPU hot-plug a new device. Not the managament device create a new
>>> device.
>>>
>>> The managament device is in the OS, we want the device is plugged by the DPU.
>> The PCI management(SIOV group owner) device is on the DPU, when create
>> an ADI,
>> OS send a command to the DPU/PF through transport vq,
>> then the PF hot plugged in a new ADI through the transport vq
>> specific channel. Or did I misunderstand your question?
> Your first step is the os send a command. Right?
>
> Can we let the DPU notify the driver to create a new devicer from the backend?
>
> The key point is who want to create a new device.
DPU can come with a certain number of pre-created ADIs, just make sure the
orchestration SW is aware of their device IDs.

If you want the DPU randomly create ADIs and notify the driver, I think
we need interrupt, e.g., re-use config interrupt. But why DPU
wants to create and hot plug in a device to a guest?
Shall the host handle that or DPU pre-create then expose to baremteal 
machines?

Thanks
>
> Thanks.
>
>
>
>
>> Thanks
>>> Thanks.
>>>
>>>
>>>> Thanks
>>>>> Thanks.
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Zhu Lingshan
>>>>>>>
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>
>>>>>> This publicly archived list offers a means to provide input to the
>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>
>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>> to minimize spam in the list archive, subscription is required
>>>>>> before posting.
>>>>>>
>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 6/30/2023 4:32 PM, Zhu, Lingshan wrote:
>
>
> On 6/30/2023 3:56 PM, Xuan Zhuo wrote:
>> On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" 
>> <lingshan.zhu@intel.com> wrote:
>>>
>>> On 6/30/2023 3:46 PM, Xuan Zhuo wrote:
>>>> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" 
>>>> <lingshan.zhu@intel.com> wrote:
>>>>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote:
>>>>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" 
>>>>>> <lingshan.zhu@intel.com> wrote:
>>>>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> hi, I would want to know some plans and progress about admin 
>>>>>>>> queue.
>>>>>>>>
>>>>>>>> At the current spec, it seems that there is only one framework 
>>>>>>>> and no
>>>>>>>> specific commands. I'd like to know if anyone is currently 
>>>>>>>> working on this and
>>>>>>>> what the plans are.
>>>>>>>>
>>>>>>>> We also faced some similar issues, and we think admin queue is 
>>>>>>>> a good way to
>>>>>>>> manage sr-iov.
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>> I plan to rebase original transport vq on admin vq.
>>>>>>>
>>>>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html 
>>>>>>>
>>>>>> I review the patch, that is for S-IOV, right?
>>>>> Yes, for SIOV and other similar devices
>>>>>> I think it is good.
>>>>>>
>>>>>> I would if all is configured by the transport vq/admin vq from 
>>>>>> the OS?
>>>>> For SIOV ADIs, this transport vq is the transport layer, so they are
>>>>> configured by the OS through transport vq.
>>>>>> Can we create a managed dev from the backend?
>>>>>>
>>>>>> Such as, the DPU sends a command to the driver, then the driver 
>>>>>> creates a new
>>>>>> managed dev.
>>>>> I think the group owner, usually the PCI PF is the management device.
>>>> I mean the DPU hot-plug a new device. Not the managament device 
>>>> create a new
>>>> device.
>>>>
>>>> The managament device is in the OS, we want the device is plugged 
>>>> by the DPU.
>>> The PCI management(SIOV group owner) device is on the DPU, when create
>>> an ADI,
>>> OS send a command to the DPU/PF through transport vq,
>>> then the PF hot plugged in a new ADI through the transport vq
>>> specific channel. Or did I misunderstand your question?
>> Your first step is the os send a command. Right?
>>
>> Can we let the DPU notify the driver to create a new devicer from the 
>> backend?
>>
>> The key point is who want to create a new device.
> DPU can come with a certain number of pre-created ADIs, just make sure 
> the
> orchestration SW is aware of their device IDs.
>
> If you want the DPU randomly create ADIs and notify the driver, I think
> we need interrupt, e.g., re-use config interrupt. But why DPU
> wants to create and hot plug in a device to a guest?
> Shall the host handle that or DPU pre-create then expose to baremteal 
> machines?
>
> Thanks
And how about this, I can try provide a new command can help query the 
onboard ADIs,
report the IDs, then the driver can probe them by the IDs and read their 
config space
to detect their config like num_vqs

Thanks
>>
>> Thanks.
>>
>>
>>
>>
>>> Thanks
>>>> Thanks.
>>>>
>>>>
>>>>> Thanks
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Zhu Lingshan
>>>>>>>>
>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>
>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>> before posting.
>>>>>>>>
>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>> List archive: 
>>>>>>>> https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>> Feedback License: 
>>>>>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>> List Guidelines: 
>>>>>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>> Feedback License: 
>>>>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>> List Guidelines: 
>>>>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>
>>>>>> This publicly archived list offers a means to provide input to the
>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>
>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>> to minimize spam in the list archive, subscription is required
>>>>>> before posting.
>>>>>>
>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>> Feedback License: 
>>>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>> List Guidelines: 
>>>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Fri, 30 Jun 2023 16:32:44 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/30/2023 3:56 PM, Xuan Zhuo wrote:
> > On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/30/2023 3:46 PM, Xuan Zhuo wrote:
> >>> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote:
> >>>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> hi, I would want to know some plans and progress about admin queue.
> >>>>>>>
> >>>>>>> At the current spec, it seems that there is only one framework and no
> >>>>>>> specific commands. I'd like to know if anyone is currently working on this and
> >>>>>>> what the plans are.
> >>>>>>>
> >>>>>>> We also faced some similar issues, and we think admin queue is a good way to
> >>>>>>> manage sr-iov.
> >>>>>>>
> >>>>>>>
> >>>>>>> Thanks.
> >>>>>> I plan to rebase original transport vq on admin vq.
> >>>>>>
> >>>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
> >>>>> I review the patch, that is for S-IOV, right?
> >>>> Yes, for SIOV and other similar devices
> >>>>> I think it is good.
> >>>>>
> >>>>> I would if all is configured by the transport vq/admin vq from the OS?
> >>>> For SIOV ADIs, this transport vq is the transport layer, so they are
> >>>> configured by the OS through transport vq.
> >>>>> Can we create a managed dev from the backend?
> >>>>>
> >>>>> Such as, the DPU sends a command to the driver, then the driver creates a new
> >>>>> managed dev.
> >>>> I think the group owner, usually the PCI PF is the management device.
> >>> I mean the DPU hot-plug a new device. Not the managament device create a new
> >>> device.
> >>>
> >>> The managament device is in the OS, we want the device is plugged by the DPU.
> >> The PCI management(SIOV group owner) device is on the DPU, when create
> >> an ADI,
> >> OS send a command to the DPU/PF through transport vq,
> >> then the PF hot plugged in a new ADI through the transport vq
> >> specific channel. Or did I misunderstand your question?
> > Your first step is the os send a command. Right?
> >
> > Can we let the DPU notify the driver to create a new devicer from the backend?
> >
> > The key point is who want to create a new device.
> DPU can come with a certain number of pre-created ADIs, just make sure the
> orchestration SW is aware of their device IDs.
>
> If you want the DPU randomly create ADIs and notify the driver, I think
> we need interrupt, e.g., re-use config interrupt. But why DPU
> wants to create and hot plug in a device to a guest?
> Shall the host handle that or DPU pre-create then expose to baremteal
> machines?


In your scenario, the supervisor is on the os, which controls the DPU to create
new devices.

In the cloud scenario, the vendor manager is in the DPU, and the entire
host is for users. Of course, there are situations where the vendor manager are
in the HOST. But for bare metal machines, the host belongs to the customer, the
vendor manager is only in the DPU.

So when the customers buy a new nic for the host, the vendor manager will plug a
device to the host from the DPU.

Thanks.


>
> Thanks
> >
> > Thanks.
> >
> >
> >
> >
> >> Thanks
> >>> Thanks.
> >>>
> >>>
> >>>> Thanks
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>>> Thanks,
> >>>>>> Zhu Lingshan
> >>>>>>>
> >>>>>>> This publicly archived list offers a means to provide input to the
> >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>>
> >>>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>>> to minimize spam in the list archive, subscription is required
> >>>>>>> before posting.
> >>>>>>>
> >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>>
> >>>>>> This publicly archived list offers a means to provide input to the
> >>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>
> >>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>> to minimize spam in the list archive, subscription is required
> >>>>>> before posting.
> >>>>>>
> >>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>
> >>>>> This publicly archived list offers a means to provide input to the
> >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>
> >>>>> In order to verify user consent to the Feedback License terms and
> >>>>> to minimize spam in the list archive, subscription is required
> >>>>> before posting.
> >>>>>
> >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 6/30/2023 5:14 PM, Xuan Zhuo wrote:
> On Fri, 30 Jun 2023 16:32:44 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/30/2023 3:56 PM, Xuan Zhuo wrote:
>>> On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/30/2023 3:46 PM, Xuan Zhuo wrote:
>>>>> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote:
>>>>>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> hi, I would want to know some plans and progress about admin queue.
>>>>>>>>>
>>>>>>>>> At the current spec, it seems that there is only one framework and no
>>>>>>>>> specific commands. I'd like to know if anyone is currently working on this and
>>>>>>>>> what the plans are.
>>>>>>>>>
>>>>>>>>> We also faced some similar issues, and we think admin queue is a good way to
>>>>>>>>> manage sr-iov.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>> I plan to rebase original transport vq on admin vq.
>>>>>>>>
>>>>>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
>>>>>>> I review the patch, that is for S-IOV, right?
>>>>>> Yes, for SIOV and other similar devices
>>>>>>> I think it is good.
>>>>>>>
>>>>>>> I would if all is configured by the transport vq/admin vq from the OS?
>>>>>> For SIOV ADIs, this transport vq is the transport layer, so they are
>>>>>> configured by the OS through transport vq.
>>>>>>> Can we create a managed dev from the backend?
>>>>>>>
>>>>>>> Such as, the DPU sends a command to the driver, then the driver creates a new
>>>>>>> managed dev.
>>>>>> I think the group owner, usually the PCI PF is the management device.
>>>>> I mean the DPU hot-plug a new device. Not the managament device create a new
>>>>> device.
>>>>>
>>>>> The managament device is in the OS, we want the device is plugged by the DPU.
>>>> The PCI management(SIOV group owner) device is on the DPU, when create
>>>> an ADI,
>>>> OS send a command to the DPU/PF through transport vq,
>>>> then the PF hot plugged in a new ADI through the transport vq
>>>> specific channel. Or did I misunderstand your question?
>>> Your first step is the os send a command. Right?
>>>
>>> Can we let the DPU notify the driver to create a new devicer from the backend?
>>>
>>> The key point is who want to create a new device.
>> DPU can come with a certain number of pre-created ADIs, just make sure the
>> orchestration SW is aware of their device IDs.
>>
>> If you want the DPU randomly create ADIs and notify the driver, I think
>> we need interrupt, e.g., re-use config interrupt. But why DPU
>> wants to create and hot plug in a device to a guest?
>> Shall the host handle that or DPU pre-create then expose to baremteal
>> machines?
>
> In your scenario, the supervisor is on the os, which controls the DPU to create
> new devices.
>
> In the cloud scenario, the vendor manager is in the DPU, and the entire
> host is for users. Of course, there are situations where the vendor manager are
> in the HOST. But for bare metal machines, the host belongs to the customer, the
> vendor manager is only in the DPU.
>
> So when the customers buy a new nic for the host, the vendor manager will plug a
> device to the host from the DPU.
I understand once a customer orders a new NIC, you wants to present the 
NIC to the host.
However you only owns the DPU and the customer owns the host, that means 
this creation
and hot plug must be transparent to the host and there may not be a host 
driver help
handling an interrupt/probe.

However this is not PCI which has a tree/switch and can enumerate 
devices to the
host by spanning the device across the PCI hierarchy.

To address an ADI, there is only a device_id.

So, do you mind share how your DPU offload the device model? What kind 
of device your DPU provide
to the host? Lets see whether DPU can mediate this by its own?

>
> Thanks.
>
>
>> Thanks
>>> Thanks.
>>>
>>>
>>>
>>>
>>>> Thanks
>>>>> Thanks.
>>>>>
>>>>>
>>>>>> Thanks
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Zhu Lingshan
>>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>>
>>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>>> before posting.
>>>>>>>>>
>>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>>
>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>
>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>> before posting.
>>>>>>>>
>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] About the plan of Admin Queue

[Parav Pandit]

> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> open.org> On Behalf Of Zhu, Lingshan
> Sent: Friday, June 30, 2023 6:33 AM
> 


> >>> Can we let the DPU notify the driver to create a new devicer from the
> backend?
Yes, why not.

> >>>
> >>> The key point is who want to create a new device.
> >> DPU can come with a certain number of pre-created ADIs, just make
> >> sure the orchestration SW is aware of their device IDs.
> >>

Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
Xuan partly explained it.

> >> If you want the DPU randomly create ADIs and notify the driver, I
> >> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >> wants to create and hot plug in a device to a guest?
> >> Shall the host handle that or DPU pre-create then expose to baremteal
> >> machines?
> >
> > In your scenario, the supervisor is on the os, which controls the DPU
> > to create new devices.
> >
> > In the cloud scenario, the vendor manager is in the DPU, and the
> > entire host is for users. Of course, there are situations where the
> > vendor manager are in the HOST. But for bare metal machines, the host
> > belongs to the customer, the vendor manager is only in the DPU.
> >
> > So when the customers buy a new nic for the host, the vendor manager
> > will plug a device to the host from the DPU.
> I understand once a customer orders a new NIC, you wants to present the NIC
> to the host.
> However you only owns the DPU and the customer owns the host, that means
> this creation and hot plug must be transparent to the host and there may not be
> a host driver help handling an interrupt/probe.
> 
That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.

> However this is not PCI which has a tree/switch and can enumerate devices to
> the host by spanning the device across the PCI hierarchy.
>
Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.

> To address an ADI, there is only a device_id.
>
SIOV device must have a unique identifier at PCI bus level for sure.
I cannot speak more about it in this forum due to other logistics issue.
But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
 
> So, do you mind share how your DPU offload the device model? What kind of
> device your DPU provide to the host? Lets see whether DPU can mediate this by
> its own?
> 
It is a virtio nic, blk and other virtio devices for us.
A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.

For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
You might find a Linux reference model useful at [1].
A usage model already exists in one OS and in use for non virtio devices.
This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.

For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.

[1] https://github.com/Mellanox/scalablefunctions/wiki

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 6/30/2023 7:35 PM, Parav Pandit wrote:
>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
>> open.org> On Behalf Of Zhu, Lingshan
>> Sent: Friday, June 30, 2023 6:33 AM
>>
>
>>>>> Can we let the DPU notify the driver to create a new devicer from the
>> backend?
> Yes, why not.
>
>>>>> The key point is who want to create a new device.
>>>> DPU can come with a certain number of pre-created ADIs, just make
>>>> sure the orchestration SW is aware of their device IDs.
>>>>
> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> Xuan partly explained it.
>
>>>> If you want the DPU randomly create ADIs and notify the driver, I
>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
>>>> wants to create and hot plug in a device to a guest?
>>>> Shall the host handle that or DPU pre-create then expose to baremteal
>>>> machines?
>>> In your scenario, the supervisor is on the os, which controls the DPU
>>> to create new devices.
>>>
>>> In the cloud scenario, the vendor manager is in the DPU, and the
>>> entire host is for users. Of course, there are situations where the
>>> vendor manager are in the HOST. But for bare metal machines, the host
>>> belongs to the customer, the vendor manager is only in the DPU.
>>>
>>> So when the customers buy a new nic for the host, the vendor manager
>>> will plug a device to the host from the DPU.
>> I understand once a customer orders a new NIC, you wants to present the NIC
>> to the host.
>> However you only owns the DPU and the customer owns the host, that means
>> this creation and hot plug must be transparent to the host and there may not be
>> a host driver help handling an interrupt/probe.
>>
> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
>
>> However this is not PCI which has a tree/switch and can enumerate devices to
>> the host by spanning the device across the PCI hierarchy.
>>
> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
>
>> To address an ADI, there is only a device_id.
>>
> SIOV device must have a unique identifier at PCI bus level for sure.
> I cannot speak more about it in this forum due to other logistics issue.
> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
>   
>> So, do you mind share how your DPU offload the device model? What kind of
>> device your DPU provide to the host? Lets see whether DPU can mediate this by
>> its own?
>>
> It is a virtio nic, blk and other virtio devices for us.
> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
>
> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> You might find a Linux reference model useful at [1].
> A usage model already exists in one OS and in use for non virtio devices.
> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
>
> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
>
> [1] https://github.com/Mellanox/scalablefunctions/wiki
well I see SF facing the similar challenge, I can add a command for the 
driver to query all existing SIOV ADIs of a device,
and the device return ADIs id and status. Looks good? and work for you 
@Xuan?

Thanks


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> >> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> >> open.org> On Behalf Of Zhu, Lingshan
> >> Sent: Friday, June 30, 2023 6:33 AM
> >>
> >
> >>>>> Can we let the DPU notify the driver to create a new devicer from the
> >> backend?
> > Yes, why not.
> >
> >>>>> The key point is who want to create a new device.
> >>>> DPU can come with a certain number of pre-created ADIs, just make
> >>>> sure the orchestration SW is aware of their device IDs.
> >>>>
> > Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > Xuan partly explained it.
> >
> >>>> If you want the DPU randomly create ADIs and notify the driver, I
> >>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >>>> wants to create and hot plug in a device to a guest?
> >>>> Shall the host handle that or DPU pre-create then expose to baremteal
> >>>> machines?
> >>> In your scenario, the supervisor is on the os, which controls the DPU
> >>> to create new devices.
> >>>
> >>> In the cloud scenario, the vendor manager is in the DPU, and the
> >>> entire host is for users. Of course, there are situations where the
> >>> vendor manager are in the HOST. But for bare metal machines, the host
> >>> belongs to the customer, the vendor manager is only in the DPU.
> >>>
> >>> So when the customers buy a new nic for the host, the vendor manager
> >>> will plug a device to the host from the DPU.
> >> I understand once a customer orders a new NIC, you wants to present the NIC
> >> to the host.
> >> However you only owns the DPU and the customer owns the host, that means
> >> this creation and hot plug must be transparent to the host and there may not be
> >> a host driver help handling an interrupt/probe.
> >>
> > That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> >
> >> However this is not PCI which has a tree/switch and can enumerate devices to
> >> the host by spanning the device across the PCI hierarchy.
> >>
> > Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> >
> >> To address an ADI, there is only a device_id.
> >>
> > SIOV device must have a unique identifier at PCI bus level for sure.
> > I cannot speak more about it in this forum due to other logistics issue.
> > But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> >
> >> So, do you mind share how your DPU offload the device model? What kind of
> >> device your DPU provide to the host? Lets see whether DPU can mediate this by
> >> its own?
> >>
> > It is a virtio nic, blk and other virtio devices for us.
> > A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> >
> > For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > You might find a Linux reference model useful at [1].
> > A usage model already exists in one OS and in use for non virtio devices.
> > This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> >
> > For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> >
> > [1] https://github.com/Mellanox/scalablefunctions/wiki
> well I see SF facing the similar challenge, I can add a command for the
> driver to query all existing SIOV ADIs of a device,
> and the device return ADIs id and status. Looks good? and work for you
> @Xuan?

Can the device notifies the driver by a interrupt?

Thanks.


>
> Thanks
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 7/3/2023 1:54 PM, Xuan Zhuo wrote:
> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
>>>> open.org> On Behalf Of Zhu, Lingshan
>>>> Sent: Friday, June 30, 2023 6:33 AM
>>>>
>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
>>>> backend?
>>> Yes, why not.
>>>
>>>>>>> The key point is who want to create a new device.
>>>>>> DPU can come with a certain number of pre-created ADIs, just make
>>>>>> sure the orchestration SW is aware of their device IDs.
>>>>>>
>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
>>> Xuan partly explained it.
>>>
>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
>>>>>> wants to create and hot plug in a device to a guest?
>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
>>>>>> machines?
>>>>> In your scenario, the supervisor is on the os, which controls the DPU
>>>>> to create new devices.
>>>>>
>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
>>>>> entire host is for users. Of course, there are situations where the
>>>>> vendor manager are in the HOST. But for bare metal machines, the host
>>>>> belongs to the customer, the vendor manager is only in the DPU.
>>>>>
>>>>> So when the customers buy a new nic for the host, the vendor manager
>>>>> will plug a device to the host from the DPU.
>>>> I understand once a customer orders a new NIC, you wants to present the NIC
>>>> to the host.
>>>> However you only owns the DPU and the customer owns the host, that means
>>>> this creation and hot plug must be transparent to the host and there may not be
>>>> a host driver help handling an interrupt/probe.
>>>>
>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
>>>
>>>> However this is not PCI which has a tree/switch and can enumerate devices to
>>>> the host by spanning the device across the PCI hierarchy.
>>>>
>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
>>>
>>>> To address an ADI, there is only a device_id.
>>>>
>>> SIOV device must have a unique identifier at PCI bus level for sure.
>>> I cannot speak more about it in this forum due to other logistics issue.
>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
>>>
>>>> So, do you mind share how your DPU offload the device model? What kind of
>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
>>>> its own?
>>>>
>>> It is a virtio nic, blk and other virtio devices for us.
>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
>>>
>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
>>> You might find a Linux reference model useful at [1].
>>> A usage model already exists in one OS and in use for non virtio devices.
>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
>>>
>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
>>>
>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
>> well I see SF facing the similar challenge, I can add a command for the
>> driver to query all existing SIOV ADIs of a device,
>> and the device return ADIs id and status. Looks good? and work for you
>> @Xuan?
> Can the device notifies the driver by a interrupt?
Of course this can be an option as previously discussed.
That would be a config interrupt(with proper suppression) from the 
management device,
which requires a new cap in a bar, then you still need to fetch the 
information.

A proposal:

a new cap, at least contains:
total number of the ADIs.

Then once any ADIs are created or removed, the PF driver get a notification.

Works for you?
>
> Thanks.
>
>
>> Thanks
>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Jason Wang]

On Fri, Jun 30, 2023 at 1:59 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >
> >
> > On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
> > > Hi,
> > >
> > > hi, I would want to know some plans and progress about admin queue.
> > >
> > > At the current spec, it seems that there is only one framework and no
> > > specific commands. I'd like to know if anyone is currently working on this and
> > > what the plans are.
> > >
> > > We also faced some similar issues, and we think admin queue is a good way to
> > > manage sr-iov.
> > >
> > >
> > > Thanks.
> > I plan to rebase original transport vq on admin vq.
> >
> > https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
>
>
> I review the patch, that is for S-IOV, right?

It is not limited to SIOV. So you won't see any SIOV terminology in
the proposal.

>
> I think it is good.
>
> I would if all is configured by the transport vq/admin vq from the OS?

One thing to clarify is that, we will define a set of admin commands,
and admin virtqueue is one interface for that. The admin commands is
designed to be more general to be carried via other type of interfaces
(if not, let's fix):

1) MMIO
2) Other transport that already use DMA
3) software IPC

Thanks

>
> Can we create a managed dev from the backend?
>
> Such as, the DPU sends a command to the driver, then the driver creates a new
> managed dev.
>
> Thanks.
>
>
> >
> > Thanks,
> > Zhu Lingshan
> > >
> > >
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Mon, 3 Jul 2023 16:10:18 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Fri, Jun 30, 2023 at 1:59 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > > On 6/20/2023 2:44 PM, Xuan Zhuo wrote:
> > > > Hi,
> > > >
> > > > hi, I would want to know some plans and progress about admin queue.
> > > >
> > > > At the current spec, it seems that there is only one framework and no
> > > > specific commands. I'd like to know if anyone is currently working on this and
> > > > what the plans are.
> > > >
> > > > We also faced some similar issues, and we think admin queue is a good way to
> > > > manage sr-iov.
> > > >
> > > >
> > > > Thanks.
> > > I plan to rebase original transport vq on admin vq.
> > >
> > > https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html
> >
> >
> > I review the patch, that is for S-IOV, right?
>
> It is not limited to SIOV. So you won't see any SIOV terminology in
> the proposal.


I got.


>
> >
> > I think it is good.
> >
> > I would if all is configured by the transport vq/admin vq from the OS?
>
> One thing to clarify is that, we will define a set of admin commands,
> and admin virtqueue is one interface for that. The admin commands is
> designed to be more general to be carried via other type of interfaces
> (if not, let's fix):
>
> 1) MMIO
> 2) Other transport that already use DMA
> 3) software IPC


Great!!

Thanks.


>
> Thanks
>
> >
> > Can we create a managed dev from the backend?
> >
> > Such as, the DPU sends a command to the driver, then the driver creates a new
> > managed dev.
> >
> > Thanks.
> >
> >
> > >
> > > Thanks,
> > > Zhu Lingshan
> > > >
> > > >
> > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Mon, 3 Jul 2023 16:01:35 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 7/3/2023 1:54 PM, Xuan Zhuo wrote:
> > On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> >>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> >>>> open.org> On Behalf Of Zhu, Lingshan
> >>>> Sent: Friday, June 30, 2023 6:33 AM
> >>>>
> >>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> >>>> backend?
> >>> Yes, why not.
> >>>
> >>>>>>> The key point is who want to create a new device.
> >>>>>> DPU can come with a certain number of pre-created ADIs, just make
> >>>>>> sure the orchestration SW is aware of their device IDs.
> >>>>>>
> >>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> >>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> >>> Xuan partly explained it.
> >>>
> >>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> >>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >>>>>> wants to create and hot plug in a device to a guest?
> >>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> >>>>>> machines?
> >>>>> In your scenario, the supervisor is on the os, which controls the DPU
> >>>>> to create new devices.
> >>>>>
> >>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> >>>>> entire host is for users. Of course, there are situations where the
> >>>>> vendor manager are in the HOST. But for bare metal machines, the host
> >>>>> belongs to the customer, the vendor manager is only in the DPU.
> >>>>>
> >>>>> So when the customers buy a new nic for the host, the vendor manager
> >>>>> will plug a device to the host from the DPU.
> >>>> I understand once a customer orders a new NIC, you wants to present the NIC
> >>>> to the host.
> >>>> However you only owns the DPU and the customer owns the host, that means
> >>>> this creation and hot plug must be transparent to the host and there may not be
> >>>> a host driver help handling an interrupt/probe.
> >>>>
> >>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> >>>
> >>>> However this is not PCI which has a tree/switch and can enumerate devices to
> >>>> the host by spanning the device across the PCI hierarchy.
> >>>>
> >>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> >>>
> >>>> To address an ADI, there is only a device_id.
> >>>>
> >>> SIOV device must have a unique identifier at PCI bus level for sure.
> >>> I cannot speak more about it in this forum due to other logistics issue.
> >>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> >>>
> >>>> So, do you mind share how your DPU offload the device model? What kind of
> >>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> >>>> its own?
> >>>>
> >>> It is a virtio nic, blk and other virtio devices for us.
> >>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> >>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> >>>
> >>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> >>> You might find a Linux reference model useful at [1].
> >>> A usage model already exists in one OS and in use for non virtio devices.
> >>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> >>>
> >>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> >>>
> >>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> >> well I see SF facing the similar challenge, I can add a command for the
> >> driver to query all existing SIOV ADIs of a device,
> >> and the device return ADIs id and status. Looks good? and work for you
> >> @Xuan?
> > Can the device notifies the driver by a interrupt?
> Of course this can be an option as previously discussed.
> That would be a config interrupt(with proper suppression) from the
> management device,
> which requires a new cap in a bar, then you still need to fetch the
> information.
>
> A proposal:
>
> a new cap, at least contains:
> total number of the ADIs.
>
> Then once any ADIs are created or removed, the PF driver get a notification.
>
> Works for you?


Look good to me.

Thanks.


> >
> > Thanks.
> >
> >
> >> Thanks
> >>
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 7/3/2023 4:21 PM, Xuan Zhuo wrote:
> On Mon, 3 Jul 2023 16:01:35 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 7/3/2023 1:54 PM, Xuan Zhuo wrote:
>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
>>>>>> open.org> On Behalf Of Zhu, Lingshan
>>>>>> Sent: Friday, June 30, 2023 6:33 AM
>>>>>>
>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
>>>>>> backend?
>>>>> Yes, why not.
>>>>>
>>>>>>>>> The key point is who want to create a new device.
>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
>>>>>>>> sure the orchestration SW is aware of their device IDs.
>>>>>>>>
>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
>>>>> Xuan partly explained it.
>>>>>
>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
>>>>>>>> wants to create and hot plug in a device to a guest?
>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
>>>>>>>> machines?
>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
>>>>>>> to create new devices.
>>>>>>>
>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
>>>>>>> entire host is for users. Of course, there are situations where the
>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
>>>>>>> belongs to the customer, the vendor manager is only in the DPU.
>>>>>>>
>>>>>>> So when the customers buy a new nic for the host, the vendor manager
>>>>>>> will plug a device to the host from the DPU.
>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
>>>>>> to the host.
>>>>>> However you only owns the DPU and the customer owns the host, that means
>>>>>> this creation and hot plug must be transparent to the host and there may not be
>>>>>> a host driver help handling an interrupt/probe.
>>>>>>
>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
>>>>>
>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
>>>>>> the host by spanning the device across the PCI hierarchy.
>>>>>>
>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
>>>>>
>>>>>> To address an ADI, there is only a device_id.
>>>>>>
>>>>> SIOV device must have a unique identifier at PCI bus level for sure.
>>>>> I cannot speak more about it in this forum due to other logistics issue.
>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
>>>>>
>>>>>> So, do you mind share how your DPU offload the device model? What kind of
>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
>>>>>> its own?
>>>>>>
>>>>> It is a virtio nic, blk and other virtio devices for us.
>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
>>>>>
>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
>>>>> You might find a Linux reference model useful at [1].
>>>>> A usage model already exists in one OS and in use for non virtio devices.
>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
>>>>>
>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
>>>>>
>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
>>>> well I see SF facing the similar challenge, I can add a command for the
>>>> driver to query all existing SIOV ADIs of a device,
>>>> and the device return ADIs id and status. Looks good? and work for you
>>>> @Xuan?
>>> Can the device notifies the driver by a interrupt?
>> Of course this can be an option as previously discussed.
>> That would be a config interrupt(with proper suppression) from the
>> management device,
>> which requires a new cap in a bar, then you still need to fetch the
>> information.
>>
>> A proposal:
>>
>> a new cap, at least contains:
>> total number of the ADIs.
>>
>> Then once any ADIs are created or removed, the PF driver get a notification.
>>
>> Works for you?
>
> Look good to me.
OK, you can review the details when I post the series

Thanks
>
> Thanks.
>
>
>>> Thanks.
>>>
>>>
>>>> Thanks
>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Michael S. Tsirkin]

OK it looks like at least Parav and Jason understand what's
going on, maybe one of you can explain:

1- there's a management stack that can bind an ID
  to a group of properties
2- the proposed command then binds an ID to a VF
3- after this, VF can be assigned to a VM

Is this a fair summary?

In that case what is gained compared to
configuring properties on the VF directly?
Why is not the ID an internal property of the management?

-- 
MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] About the plan of Admin Queue

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, July 3, 2023 9:05 AM
> 
> OK it looks like at least Parav and Jason understand what's going on, maybe one
> of you can explain:
>
Yet to catch up on the thread today, will be able to respond in few hours.
 
> 1- there's a management stack that can bind an ID
>   to a group of properties
> 2- the proposed command then binds an ID to a VF
> 3- after this, VF can be assigned to a VM
> 
> Is this a fair summary?
> 
> In that case what is gained compared to
> configuring properties on the VF directly?
> Why is not the ID an internal property of the management?
> 
> --
> MST


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] About the plan of Admin Queue

[Parav Pandit]

> From: Michael S. Tsirkin <mst@redhat.com>
> Sent: Monday, July 3, 2023 9:05 AM
> 
> OK it looks like at least Parav and Jason understand what's going on, maybe one
> of you can explain:
> 
> 1- there's a management stack that can bind an ID
>   to a group of properties
> 2- the proposed command then binds an ID to a VF
> 3- after this, VF can be assigned to a VM
> 
> Is this a fair summary?
> 
> In that case what is gained compared to
> configuring properties on the VF directly?
> Why is not the ID an internal property of the management?

For VFs I also believe we can live with an internal property.

For migratable SIOV devices which are spawned by the management system has similar ID assigned by the mgmt. system.
Having such node local ID is needed anyway.

There is a struggle (scale wise) to make such ID network wide unique if each virtio device to expose such id at transport level.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 7/4/2023 4:38 AM, Parav Pandit wrote:
>
>> From: Michael S. Tsirkin <mst@redhat.com>
>> Sent: Monday, July 3, 2023 9:05 AM
>>
>> OK it looks like at least Parav and Jason understand what's going on, maybe one
>> of you can explain:
>>
>> 1- there's a management stack that can bind an ID
>>    to a group of properties
>> 2- the proposed command then binds an ID to a VF
>> 3- after this, VF can be assigned to a VM
>>
>> Is this a fair summary?
>>
>> In that case what is gained compared to
>> configuring properties on the VF directly?
>> Why is not the ID an internal property of the management?
> For VFs I also believe we can live with an internal property.
>
> For migratable SIOV devices which are spawned by the management system has similar ID assigned by the mgmt. system.
> Having such node local ID is needed anyway.
>
> There is a struggle (scale wise) to make such ID network wide unique if each virtio device to expose such id at transport level.
Maybe you don't want a guest talking to the PF through admin queue 
directly, so there can be a wrapper for SIOV ADIs, therefore the ID is 
not a problem for migration.


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Mon, 3 Jul 2023 09:05:09 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> OK it looks like at least Parav and Jason understand what's
> going on, maybe one of you can explain:
>
> 1- there's a management stack that can bind an ID
>   to a group of properties
> 2- the proposed command then binds an ID to a VF
> 3- after this, VF can be assigned to a VM
>
> Is this a fair summary?

This sounds like what I need.

This thread discusses another point, the device notifies the pf driver that
there is a new device.

Thanks.


>
> In that case what is gained compared to
> configuring properties on the VF directly?
> Why is not the ID an internal property of the management?
>
> --
> MST
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Mon, 3 Jul 2023 09:05:09 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> OK it looks like at least Parav and Jason understand what's
> going on, maybe one of you can explain:
>
> 1- there's a management stack that can bind an ID
>   to a group of properties
> 2- the proposed command then binds an ID to a VF
> 3- after this, VF can be assigned to a VM
>
> Is this a fair summary?
>
> In that case what is gained compared to
> configuring properties on the VF directly?

I have been thinking about this question for a long time, and there maybe is a
gap of understanding here.

In our scenario, we should have two roles, one is the user and the other is the
vendor's management software. Do you think that the management
software is running on the host, and it manages the DPU?

But in the scenario I mentioned, the vendor's management software runs on
the DPU, and all hosts belong to the user.

The problem we want to solve is that this user may want 1000 network cards. So
it must use VFs as a network cards. It cannot and will not configure detailed
NIC properties based on the admin queue.

I don't know if I found the key to the problem.

Thanks.



> Why is not the ID an internal property of the management?
>
> --
> MST
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] About the plan of Admin Queue

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Tuesday, July 4, 2023 8:14 AM

> But in the scenario I mentioned, the vendor's management software runs on
> the DPU, and all hosts belong to the user.
>
How did vendor management software configure the parameters on the src side?
Same channel can be used on destination side, right?
 
> The problem we want to solve is that this user may want 1000 network cards. So
> it must use VFs as a network cards. It cannot and will not configure detailed NIC
> properties based on the admin queue.
> 
Sure.
It will not.
The idea is to use same out of band channel as what is used on the src side, also on the destination side.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: RE: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Tue, 4 Jul 2023 13:15:23 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Tuesday, July 4, 2023 8:14 AM
>
> > But in the scenario I mentioned, the vendor's management software runs on
> > the DPU, and all hosts belong to the user.
> >
> How did vendor management software configure the parameters on the src side?


Sorry, the src side you mean the host, right?


> Same channel can be used on destination side, right?

Do you mean that the bandwidth of the net device can be configured by the user?
I think the user can only do this on the platform.


Thanks.


>
> > The problem we want to solve is that this user may want 1000 network cards. So
> > it must use VFs as a network cards. It cannot and will not configure detailed NIC
> > properties based on the admin queue.
> >
> Sure.
> It will not.
> The idea is to use same out of band channel as what is used on the src side, also on the destination side.




This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: RE: [virtio-comment] About the plan of Admin Queue

[Parav Pandit]

> From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> Sent: Wednesday, July 5, 2023 12:30 AM
> 
> On Tue, 4 Jul 2023 13:15:23 +0000, Parav Pandit <parav@nvidia.com> wrote:
> >
> > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > Sent: Tuesday, July 4, 2023 8:14 AM
> >
> > > But in the scenario I mentioned, the vendor's management software
> > > runs on the DPU, and all hosts belong to the user.
> > >
> > How did vendor management software configure the parameters on the src
> side?
> 
> 
> Sorry, the src side you mean the host, right?
> 
No. in your live migration (LM) example src side mean src side of hypervisor where a virtio device is used for the first time.

> 
> > Same channel can be used on destination side, right?
> 
> Do you mean that the bandwidth of the net device can be configured by the
> user?

No no.
> I think the user can only do this on the platform.
> 
Right, only by the platform.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: RE: RE: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Wed, 5 Jul 2023 04:35:36 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Wednesday, July 5, 2023 12:30 AM
> >
> > On Tue, 4 Jul 2023 13:15:23 +0000, Parav Pandit <parav@nvidia.com> wrote:
> > >
> > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > > Sent: Tuesday, July 4, 2023 8:14 AM
> > >
> > > > But in the scenario I mentioned, the vendor's management software
> > > > runs on the DPU, and all hosts belong to the user.
> > > >
> > > How did vendor management software configure the parameters on the src
> > side?
> >
> >
> > Sorry, the src side you mean the host, right?
> >
> No. in your live migration (LM) example src side mean src side of hypervisor where a virtio device is used for the first time.


Oh, I mistake.


>
> >
> > > Same channel can be used on destination side, right?
> >
> > Do you mean that the bandwidth of the net device can be configured by the
> > user?
>
> No no.
> > I think the user can only do this on the platform.
> >
> Right, only by the platform.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: RE: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Tue, 4 Jul 2023 13:15:23 +0000, Parav Pandit <parav@nvidia.com> wrote:
>
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Tuesday, July 4, 2023 8:14 AM
>
> > But in the scenario I mentioned, the vendor's management software runs on
> > the DPU, and all hosts belong to the user.
> >
> How did vendor management software configure the parameters on the src side?
> Same channel can be used on destination side, right?

Yes.

>
> > The problem we want to solve is that this user may want 1000 network cards. So
> > it must use VFs as a network cards. It cannot and will not configure detailed NIC
> > properties based on the admin queue.
> >
> Sure.
> It will not.
> The idea is to use same out of band channel as what is used on the src side, also on the destination side.

Sure.


Thanks.

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> >> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> >> open.org> On Behalf Of Zhu, Lingshan
> >> Sent: Friday, June 30, 2023 6:33 AM
> >>
> >
> >>>>> Can we let the DPU notify the driver to create a new devicer from the
> >> backend?
> > Yes, why not.
> >
> >>>>> The key point is who want to create a new device.
> >>>> DPU can come with a certain number of pre-created ADIs, just make
> >>>> sure the orchestration SW is aware of their device IDs.
> >>>>
> > Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > Xuan partly explained it.
> >
> >>>> If you want the DPU randomly create ADIs and notify the driver, I
> >>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >>>> wants to create and hot plug in a device to a guest?
> >>>> Shall the host handle that or DPU pre-create then expose to baremteal
> >>>> machines?
> >>> In your scenario, the supervisor is on the os, which controls the DPU
> >>> to create new devices.
> >>>
> >>> In the cloud scenario, the vendor manager is in the DPU, and the
> >>> entire host is for users. Of course, there are situations where the
> >>> vendor manager are in the HOST. But for bare metal machines, the host
> >>> belongs to the customer, the vendor manager is only in the DPU.
> >>>
> >>> So when the customers buy a new nic for the host, the vendor manager
> >>> will plug a device to the host from the DPU.
> >> I understand once a customer orders a new NIC, you wants to present the NIC
> >> to the host.
> >> However you only owns the DPU and the customer owns the host, that means
> >> this creation and hot plug must be transparent to the host and there may not be
> >> a host driver help handling an interrupt/probe.
> >>
> > That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> >
> >> However this is not PCI which has a tree/switch and can enumerate devices to
> >> the host by spanning the device across the PCI hierarchy.
> >>
> > Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> >
> >> To address an ADI, there is only a device_id.
> >>
> > SIOV device must have a unique identifier at PCI bus level for sure.
> > I cannot speak more about it in this forum due to other logistics issue.
> > But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> >
> >> So, do you mind share how your DPU offload the device model? What kind of
> >> device your DPU provide to the host? Lets see whether DPU can mediate this by
> >> its own?
> >>
> > It is a virtio nic, blk and other virtio devices for us.
> > A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> >
> > For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > You might find a Linux reference model useful at [1].
> > A usage model already exists in one OS and in use for non virtio devices.
> > This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> >
> > For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> >
> > [1] https://github.com/Mellanox/scalablefunctions/wiki
> well I see SF facing the similar challenge, I can add a command for the
> driver to query all existing SIOV ADIs of a device,
> and the device return ADIs id and status. Looks good? and work for you
> @Xuan?

Could I have your plan for this?

If you do not mind, I'd like to add a command to query VF's info. Such
as mac, ip, etc.

Thanks.



>
> Thanks
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
>>>> open.org> On Behalf Of Zhu, Lingshan
>>>> Sent: Friday, June 30, 2023 6:33 AM
>>>>
>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
>>>> backend?
>>> Yes, why not.
>>>
>>>>>>> The key point is who want to create a new device.
>>>>>> DPU can come with a certain number of pre-created ADIs, just make
>>>>>> sure the orchestration SW is aware of their device IDs.
>>>>>>
>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
>>> Xuan partly explained it.
>>>
>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
>>>>>> wants to create and hot plug in a device to a guest?
>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
>>>>>> machines?
>>>>> In your scenario, the supervisor is on the os, which controls the DPU
>>>>> to create new devices.
>>>>>
>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
>>>>> entire host is for users. Of course, there are situations where the
>>>>> vendor manager are in the HOST. But for bare metal machines, the host
>>>>> belongs to the customer, the vendor manager is only in the DPU.
>>>>>
>>>>> So when the customers buy a new nic for the host, the vendor manager
>>>>> will plug a device to the host from the DPU.
>>>> I understand once a customer orders a new NIC, you wants to present the NIC
>>>> to the host.
>>>> However you only owns the DPU and the customer owns the host, that means
>>>> this creation and hot plug must be transparent to the host and there may not be
>>>> a host driver help handling an interrupt/probe.
>>>>
>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
>>>
>>>> However this is not PCI which has a tree/switch and can enumerate devices to
>>>> the host by spanning the device across the PCI hierarchy.
>>>>
>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
>>>
>>>> To address an ADI, there is only a device_id.
>>>>
>>> SIOV device must have a unique identifier at PCI bus level for sure.
>>> I cannot speak more about it in this forum due to other logistics issue.
>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
>>>
>>>> So, do you mind share how your DPU offload the device model? What kind of
>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
>>>> its own?
>>>>
>>> It is a virtio nic, blk and other virtio devices for us.
>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
>>>
>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
>>> You might find a Linux reference model useful at [1].
>>> A usage model already exists in one OS and in use for non virtio devices.
>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
>>>
>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
>>>
>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
>> well I see SF facing the similar challenge, I can add a command for the
>> driver to query all existing SIOV ADIs of a device,
>> and the device return ADIs id and status. Looks good? and work for you
>> @Xuan?
> Could I have your plan for this?
>
> If you do not mind, I'd like to add a command to query VF's info. Such
> as mac, ip, etc.
I think the query commands for SIOV is a little more complex, e.g.,
need to report device type and its scale(e.g., features, mq).
There can be thousands of SIOV ADIs and we don't want output flood.

We have discussed implementation a config interrupt to report new 
created / deleted
ADIs on the DPU side, therefore there must be a cap contains related 
information,
my rough approach of the process is:
1) a cap contains the total number of existing ADIs and the max dev id
2) driver queries detailed information of a certain ADI or a bunch of 
ADIs in a [dev_id....dev_id2] range.

I am not sure whether a NIC stores its IP

Thanks
>
> Thanks.
>
>
>
>> Thanks
>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> >>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> >>>> open.org> On Behalf Of Zhu, Lingshan
> >>>> Sent: Friday, June 30, 2023 6:33 AM
> >>>>
> >>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> >>>> backend?
> >>> Yes, why not.
> >>>
> >>>>>>> The key point is who want to create a new device.
> >>>>>> DPU can come with a certain number of pre-created ADIs, just make
> >>>>>> sure the orchestration SW is aware of their device IDs.
> >>>>>>
> >>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> >>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> >>> Xuan partly explained it.
> >>>
> >>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> >>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >>>>>> wants to create and hot plug in a device to a guest?
> >>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> >>>>>> machines?
> >>>>> In your scenario, the supervisor is on the os, which controls the DPU
> >>>>> to create new devices.
> >>>>>
> >>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> >>>>> entire host is for users. Of course, there are situations where the
> >>>>> vendor manager are in the HOST. But for bare metal machines, the host
> >>>>> belongs to the customer, the vendor manager is only in the DPU.
> >>>>>
> >>>>> So when the customers buy a new nic for the host, the vendor manager
> >>>>> will plug a device to the host from the DPU.
> >>>> I understand once a customer orders a new NIC, you wants to present the NIC
> >>>> to the host.
> >>>> However you only owns the DPU and the customer owns the host, that means
> >>>> this creation and hot plug must be transparent to the host and there may not be
> >>>> a host driver help handling an interrupt/probe.
> >>>>
> >>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> >>>
> >>>> However this is not PCI which has a tree/switch and can enumerate devices to
> >>>> the host by spanning the device across the PCI hierarchy.
> >>>>
> >>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> >>>
> >>>> To address an ADI, there is only a device_id.
> >>>>
> >>> SIOV device must have a unique identifier at PCI bus level for sure.
> >>> I cannot speak more about it in this forum due to other logistics issue.
> >>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> >>>
> >>>> So, do you mind share how your DPU offload the device model? What kind of
> >>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> >>>> its own?
> >>>>
> >>> It is a virtio nic, blk and other virtio devices for us.
> >>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> >>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> >>>
> >>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> >>> You might find a Linux reference model useful at [1].
> >>> A usage model already exists in one OS and in use for non virtio devices.
> >>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> >>>
> >>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> >>>
> >>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> >> well I see SF facing the similar challenge, I can add a command for the
> >> driver to query all existing SIOV ADIs of a device,
> >> and the device return ADIs id and status. Looks good? and work for you
> >> @Xuan?
> > Could I have your plan for this?
> >
> > If you do not mind, I'd like to add a command to query VF's info. Such
> > as mac, ip, etc.
> I think the query commands for SIOV is a little more complex, e.g.,
> need to report device type and its scale(e.g., features, mq).
> There can be thousands of SIOV ADIs and we don't want output flood.
>
> We have discussed implementation a config interrupt to report new
> created / deleted
> ADIs on the DPU side, therefore there must be a cap contains related
> information,
> my rough approach of the process is:
> 1) a cap contains the total number of existing ADIs and the max dev id
> 2) driver queries detailed information of a certain ADI or a bunch of
> ADIs in a [dev_id....dev_id2] range.

Yes, Admin Queue can obtain the info of the specific one or more devices.

>
> I am not sure whether a NIC stores its IP


IP is the other topic. I want the Admin Queue manage the switch.
So the switch know about the IP of every device, and the
Admin Queue will has the ability to config the IP of the device inside the
switch.


Thanks.


>
> Thanks
> >
> > Thanks.
> >
> >
> >
> >> Thanks
> >>
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Zhu, Lingshan]

On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
>>>>>> open.org> On Behalf Of Zhu, Lingshan
>>>>>> Sent: Friday, June 30, 2023 6:33 AM
>>>>>>
>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
>>>>>> backend?
>>>>> Yes, why not.
>>>>>
>>>>>>>>> The key point is who want to create a new device.
>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
>>>>>>>> sure the orchestration SW is aware of their device IDs.
>>>>>>>>
>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
>>>>> Xuan partly explained it.
>>>>>
>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
>>>>>>>> wants to create and hot plug in a device to a guest?
>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
>>>>>>>> machines?
>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
>>>>>>> to create new devices.
>>>>>>>
>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
>>>>>>> entire host is for users. Of course, there are situations where the
>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
>>>>>>> belongs to the customer, the vendor manager is only in the DPU.
>>>>>>>
>>>>>>> So when the customers buy a new nic for the host, the vendor manager
>>>>>>> will plug a device to the host from the DPU.
>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
>>>>>> to the host.
>>>>>> However you only owns the DPU and the customer owns the host, that means
>>>>>> this creation and hot plug must be transparent to the host and there may not be
>>>>>> a host driver help handling an interrupt/probe.
>>>>>>
>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
>>>>>
>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
>>>>>> the host by spanning the device across the PCI hierarchy.
>>>>>>
>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
>>>>>
>>>>>> To address an ADI, there is only a device_id.
>>>>>>
>>>>> SIOV device must have a unique identifier at PCI bus level for sure.
>>>>> I cannot speak more about it in this forum due to other logistics issue.
>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
>>>>>
>>>>>> So, do you mind share how your DPU offload the device model? What kind of
>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
>>>>>> its own?
>>>>>>
>>>>> It is a virtio nic, blk and other virtio devices for us.
>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
>>>>>
>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
>>>>> You might find a Linux reference model useful at [1].
>>>>> A usage model already exists in one OS and in use for non virtio devices.
>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
>>>>>
>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
>>>>>
>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
>>>> well I see SF facing the similar challenge, I can add a command for the
>>>> driver to query all existing SIOV ADIs of a device,
>>>> and the device return ADIs id and status. Looks good? and work for you
>>>> @Xuan?
>>> Could I have your plan for this?
>>>
>>> If you do not mind, I'd like to add a command to query VF's info. Such
>>> as mac, ip, etc.
>> I think the query commands for SIOV is a little more complex, e.g.,
>> need to report device type and its scale(e.g., features, mq).
>> There can be thousands of SIOV ADIs and we don't want output flood.
>>
>> We have discussed implementation a config interrupt to report new
>> created / deleted
>> ADIs on the DPU side, therefore there must be a cap contains related
>> information,
>> my rough approach of the process is:
>> 1) a cap contains the total number of existing ADIs and the max dev id
>> 2) driver queries detailed information of a certain ADI or a bunch of
>> ADIs in a [dev_id....dev_id2] range.
> Yes, Admin Queue can obtain the info of the specific one or more devices.
>
>> I am not sure whether a NIC stores its IP
>
> IP is the other topic. I want the Admin Queue manage the switch.
> So the switch know about the IP of every device, and the
> Admin Queue will has the ability to config the IP of the device inside the
> switch.
DPU onboard switch? OVS? Does it beyond virtio spec?
>
>
> Thanks.
>
>
>> Thanks
>>> Thanks.
>>>
>>>
>>>
>>>> Thanks
>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> >>>>>> open.org> On Behalf Of Zhu, Lingshan
> >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> >>>>>>
> >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> >>>>>> backend?
> >>>>> Yes, why not.
> >>>>>
> >>>>>>>>> The key point is who want to create a new device.
> >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> >>>>>>>> sure the orchestration SW is aware of their device IDs.
> >>>>>>>>
> >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> >>>>> Xuan partly explained it.
> >>>>>
> >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >>>>>>>> wants to create and hot plug in a device to a guest?
> >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> >>>>>>>> machines?
> >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> >>>>>>> to create new devices.
> >>>>>>>
> >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> >>>>>>> entire host is for users. Of course, there are situations where the
> >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> >>>>>>>
> >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> >>>>>>> will plug a device to the host from the DPU.
> >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> >>>>>> to the host.
> >>>>>> However you only owns the DPU and the customer owns the host, that means
> >>>>>> this creation and hot plug must be transparent to the host and there may not be
> >>>>>> a host driver help handling an interrupt/probe.
> >>>>>>
> >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> >>>>>
> >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> >>>>>> the host by spanning the device across the PCI hierarchy.
> >>>>>>
> >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> >>>>>
> >>>>>> To address an ADI, there is only a device_id.
> >>>>>>
> >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> >>>>> I cannot speak more about it in this forum due to other logistics issue.
> >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> >>>>>
> >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> >>>>>> its own?
> >>>>>>
> >>>>> It is a virtio nic, blk and other virtio devices for us.
> >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> >>>>>
> >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> >>>>> You might find a Linux reference model useful at [1].
> >>>>> A usage model already exists in one OS and in use for non virtio devices.
> >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> >>>>>
> >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> >>>>>
> >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> >>>> well I see SF facing the similar challenge, I can add a command for the
> >>>> driver to query all existing SIOV ADIs of a device,
> >>>> and the device return ADIs id and status. Looks good? and work for you
> >>>> @Xuan?
> >>> Could I have your plan for this?
> >>>
> >>> If you do not mind, I'd like to add a command to query VF's info. Such
> >>> as mac, ip, etc.
> >> I think the query commands for SIOV is a little more complex, e.g.,
> >> need to report device type and its scale(e.g., features, mq).
> >> There can be thousands of SIOV ADIs and we don't want output flood.
> >>
> >> We have discussed implementation a config interrupt to report new
> >> created / deleted
> >> ADIs on the DPU side, therefore there must be a cap contains related
> >> information,
> >> my rough approach of the process is:
> >> 1) a cap contains the total number of existing ADIs and the max dev id
> >> 2) driver queries detailed information of a certain ADI or a bunch of
> >> ADIs in a [dev_id....dev_id2] range.
> > Yes, Admin Queue can obtain the info of the specific one or more devices.
> >
> >> I am not sure whether a NIC stores its IP
> >
> > IP is the other topic. I want the Admin Queue manage the switch.
> > So the switch know about the IP of every device, and the
> > Admin Queue will has the ability to config the IP of the device inside the
> > switch.
> DPU onboard switch? OVS? Does it beyond virtio spec?

YES.

For SIOV, I think this is MUST. Maybe you have one simple implementation.
But you have to solve the IP steering. So admin queue should has the ability
to config the IP steering.

Thanks.

> >
> >
> > Thanks.
> >
> >
> >> Thanks
> >>> Thanks.
> >>>
> >>>
> >>>
> >>>> Thanks
> >>>>
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>> Committee: https://www.oasis-open.org/committees/virtio/
> >>> Join OASIS: https://www.oasis-open.org/join/
> >>>
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Jason Wang]

On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >
> >
> > On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>
> > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > >>>>>> open.org> On Behalf Of Zhu, Lingshan
> > >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > >>>>>>
> > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > >>>>>> backend?
> > >>>>> Yes, why not.
> > >>>>>
> > >>>>>>>>> The key point is who want to create a new device.
> > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > >>>>>>>> sure the orchestration SW is aware of their device IDs.
> > >>>>>>>>
> > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > >>>>> Xuan partly explained it.
> > >>>>>
> > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > >>>>>>>> wants to create and hot plug in a device to a guest?
> > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > >>>>>>>> machines?
> > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > >>>>>>> to create new devices.
> > >>>>>>>
> > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > >>>>>>> entire host is for users. Of course, there are situations where the
> > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > >>>>>>>
> > >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > >>>>>>> will plug a device to the host from the DPU.
> > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > >>>>>> to the host.
> > >>>>>> However you only owns the DPU and the customer owns the host, that means
> > >>>>>> this creation and hot plug must be transparent to the host and there may not be
> > >>>>>> a host driver help handling an interrupt/probe.
> > >>>>>>
> > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > >>>>>
> > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > >>>>>> the host by spanning the device across the PCI hierarchy.
> > >>>>>>
> > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > >>>>>
> > >>>>>> To address an ADI, there is only a device_id.
> > >>>>>>
> > >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > >>>>> I cannot speak more about it in this forum due to other logistics issue.
> > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > >>>>>
> > >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > >>>>>> its own?
> > >>>>>>
> > >>>>> It is a virtio nic, blk and other virtio devices for us.
> > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > >>>>>
> > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > >>>>> You might find a Linux reference model useful at [1].
> > >>>>> A usage model already exists in one OS and in use for non virtio devices.
> > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > >>>>>
> > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > >>>>>
> > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > >>>> well I see SF facing the similar challenge, I can add a command for the
> > >>>> driver to query all existing SIOV ADIs of a device,
> > >>>> and the device return ADIs id and status. Looks good? and work for you
> > >>>> @Xuan?
> > >>> Could I have your plan for this?
> > >>>
> > >>> If you do not mind, I'd like to add a command to query VF's info. Such
> > >>> as mac, ip, etc.
> > >> I think the query commands for SIOV is a little more complex, e.g.,
> > >> need to report device type and its scale(e.g., features, mq).
> > >> There can be thousands of SIOV ADIs and we don't want output flood.
> > >>
> > >> We have discussed implementation a config interrupt to report new
> > >> created / deleted
> > >> ADIs on the DPU side, therefore there must be a cap contains related
> > >> information,
> > >> my rough approach of the process is:
> > >> 1) a cap contains the total number of existing ADIs and the max dev id
> > >> 2) driver queries detailed information of a certain ADI or a bunch of
> > >> ADIs in a [dev_id....dev_id2] range.
> > > Yes, Admin Queue can obtain the info of the specific one or more devices.
> > >
> > >> I am not sure whether a NIC stores its IP
> > >
> > > IP is the other topic. I want the Admin Queue manage the switch.
> > > So the switch know about the IP of every device, and the
> > > Admin Queue will has the ability to config the IP of the device inside the
> > > switch.
> > DPU onboard switch? OVS? Does it beyond virtio spec?
>
> YES.

Adding Washizu.

We can have a switch/dpa defined in the networking device for sure.

>
> For SIOV, I think this is MUST.

A learning bridge would be fine as a starter. It's better not to
couple new scalable capability with any device specific features.

> Maybe you have one simple implementation.
> But you have to solve the IP steering. So admin queue should has the ability
> to config the IP steering.

I think not. Those L2/LN tables/filters are networking specific.
Control virtqueue is better than admin virtqueue here.

Thanks

>
> Thanks.
>
> > >
> > >
> > > Thanks.
> > >
> > >
> > >> Thanks
> > >>> Thanks.
> > >>>
> > >>>
> > >>>
> > >>>> Thanks
> > >>>>
> > >>> This publicly archived list offers a means to provide input to the
> > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>
> > >>> In order to verify user consent to the Feedback License terms and
> > >>> to minimize spam in the list archive, subscription is required
> > >>> before posting.
> > >>>
> > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>> List help: virtio-comment-help@lists.oasis-open.org
> > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>> Join OASIS: https://www.oasis-open.org/join/
> > >>>
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >>
> > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > >>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > >>>>>>
> > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > >>>>>> backend?
> > > >>>>> Yes, why not.
> > > >>>>>
> > > >>>>>>>>> The key point is who want to create a new device.
> > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > >>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > >>>>>>>>
> > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > >>>>> Xuan partly explained it.
> > > >>>>>
> > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > >>>>>>>> wants to create and hot plug in a device to a guest?
> > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > >>>>>>>> machines?
> > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > >>>>>>> to create new devices.
> > > >>>>>>>
> > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > >>>>>>> entire host is for users. Of course, there are situations where the
> > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > >>>>>>>
> > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > >>>>>>> will plug a device to the host from the DPU.
> > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > >>>>>> to the host.
> > > >>>>>> However you only owns the DPU and the customer owns the host, that means
> > > >>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > >>>>>> a host driver help handling an interrupt/probe.
> > > >>>>>>
> > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > >>>>>
> > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > >>>>>> the host by spanning the device across the PCI hierarchy.
> > > >>>>>>
> > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > >>>>>
> > > >>>>>> To address an ADI, there is only a device_id.
> > > >>>>>>
> > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > >>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > >>>>>
> > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > >>>>>> its own?
> > > >>>>>>
> > > >>>>> It is a virtio nic, blk and other virtio devices for us.
> > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > >>>>>
> > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > >>>>> You might find a Linux reference model useful at [1].
> > > >>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > >>>>>
> > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > >>>>>
> > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > >>>> well I see SF facing the similar challenge, I can add a command for the
> > > >>>> driver to query all existing SIOV ADIs of a device,
> > > >>>> and the device return ADIs id and status. Looks good? and work for you
> > > >>>> @Xuan?
> > > >>> Could I have your plan for this?
> > > >>>
> > > >>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > >>> as mac, ip, etc.
> > > >> I think the query commands for SIOV is a little more complex, e.g.,
> > > >> need to report device type and its scale(e.g., features, mq).
> > > >> There can be thousands of SIOV ADIs and we don't want output flood.
> > > >>
> > > >> We have discussed implementation a config interrupt to report new
> > > >> created / deleted
> > > >> ADIs on the DPU side, therefore there must be a cap contains related
> > > >> information,
> > > >> my rough approach of the process is:
> > > >> 1) a cap contains the total number of existing ADIs and the max dev id
> > > >> 2) driver queries detailed information of a certain ADI or a bunch of
> > > >> ADIs in a [dev_id....dev_id2] range.
> > > > Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > >
> > > >> I am not sure whether a NIC stores its IP
> > > >
> > > > IP is the other topic. I want the Admin Queue manage the switch.
> > > > So the switch know about the IP of every device, and the
> > > > Admin Queue will has the ability to config the IP of the device inside the
> > > > switch.
> > > DPU onboard switch? OVS? Does it beyond virtio spec?
> >
> > YES.
>
> Adding Washizu.
>
> We can have a switch/dpa defined in the networking device for sure.

Yes, I think we should introduce that for the sr-iov. Or for other.

I would like to know who is doing this?

Another question, @Jason are you referring to a new device type or a
new virtio-net feature.

>
> >
> > For SIOV, I think this is MUST.
>
> A learning bridge would be fine as a starter. It's better not to
> couple new scalable capability with any device specific features.
>
> > Maybe you have one simple implementation.
> > But you have to solve the IP steering. So admin queue should has the ability
> > to config the IP steering.
>
> I think not. Those L2/LN tables/filters are networking specific.

Let us assume that there is a switch/bridge firstly.
The VFs may be passed to different VMs.

I also think this is the networking specific. But I want to config
the ip for every vf from the pf. Because the user of the vf may be unreliable.
We need a manager to config the ip for every vf.


> Control virtqueue is better than admin virtqueue here.

by cq?

What case?

Thanks.

>
> Thanks
>
> >
> > Thanks.
> >
> > > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > > >> Thanks
> > > >>> Thanks.
> > > >>>
> > > >>>
> > > >>>
> > > >>>> Thanks
> > > >>>>
> > > >>> This publicly archived list offers a means to provide input to the
> > > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>
> > > >>> In order to verify user consent to the Feedback License terms and
> > > >>> to minimize spam in the list archive, subscription is required
> > > >>> before posting.
> > > >>>
> > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Jason Wang]

On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >
> > > >
> > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > >>
> > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > > >>>>>>
> > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > > >>>>>> backend?
> > > > >>>>> Yes, why not.
> > > > >>>>>
> > > > >>>>>>>>> The key point is who want to create a new device.
> > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > > >>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > > >>>>>>>>
> > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > > >>>>> Xuan partly explained it.
> > > > >>>>>
> > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > > >>>>>>>> wants to create and hot plug in a device to a guest?
> > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > > >>>>>>>> machines?
> > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > > >>>>>>> to create new devices.
> > > > >>>>>>>
> > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > > >>>>>>> entire host is for users. Of course, there are situations where the
> > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > > >>>>>>>
> > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > > >>>>>>> will plug a device to the host from the DPU.
> > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > > >>>>>> to the host.
> > > > >>>>>> However you only owns the DPU and the customer owns the host, that means
> > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > > >>>>>> a host driver help handling an interrupt/probe.
> > > > >>>>>>
> > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > > >>>>>
> > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > > >>>>>> the host by spanning the device across the PCI hierarchy.
> > > > >>>>>>
> > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > > >>>>>
> > > > >>>>>> To address an ADI, there is only a device_id.
> > > > >>>>>>
> > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > > >>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > > >>>>>
> > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > > >>>>>> its own?
> > > > >>>>>>
> > > > >>>>> It is a virtio nic, blk and other virtio devices for us.
> > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > > >>>>>
> > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > > >>>>> You might find a Linux reference model useful at [1].
> > > > >>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > > >>>>>
> > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > > >>>>>
> > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > > >>>> well I see SF facing the similar challenge, I can add a command for the
> > > > >>>> driver to query all existing SIOV ADIs of a device,
> > > > >>>> and the device return ADIs id and status. Looks good? and work for you
> > > > >>>> @Xuan?
> > > > >>> Could I have your plan for this?
> > > > >>>
> > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > > >>> as mac, ip, etc.
> > > > >> I think the query commands for SIOV is a little more complex, e.g.,
> > > > >> need to report device type and its scale(e.g., features, mq).
> > > > >> There can be thousands of SIOV ADIs and we don't want output flood.
> > > > >>
> > > > >> We have discussed implementation a config interrupt to report new
> > > > >> created / deleted
> > > > >> ADIs on the DPU side, therefore there must be a cap contains related
> > > > >> information,
> > > > >> my rough approach of the process is:
> > > > >> 1) a cap contains the total number of existing ADIs and the max dev id
> > > > >> 2) driver queries detailed information of a certain ADI or a bunch of
> > > > >> ADIs in a [dev_id....dev_id2] range.
> > > > > Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > > >
> > > > >> I am not sure whether a NIC stores its IP
> > > > >
> > > > > IP is the other topic. I want the Admin Queue manage the switch.
> > > > > So the switch know about the IP of every device, and the
> > > > > Admin Queue will has the ability to config the IP of the device inside the
> > > > > switch.
> > > > DPU onboard switch? OVS? Does it beyond virtio spec?
> > >
> > > YES.
> >
> > Adding Washizu.
> >
> > We can have a switch/dpa defined in the networking device for sure.
>
> Yes, I think we should introduce that for the sr-iov. Or for other.

This should be a general one as a switch should be transport independent.

>
> I would like to know who is doing this?

Washizu, could you confirm if you want to do this or not?

>
> Another question, @Jason are you referring to a new device type or a
> new virtio-net feature.

Extending virtio-net should be fine, did you see any issues for this?

>
> >
> > >
> > > For SIOV, I think this is MUST.
> >
> > A learning bridge would be fine as a starter. It's better not to
> > couple new scalable capability with any device specific features.
> >
> > > Maybe you have one simple implementation.
> > > But you have to solve the IP steering. So admin queue should has the ability
> > > to config the IP steering.
> >
> > I think not. Those L2/LN tables/filters are networking specific.
>
> Let us assume that there is a switch/bridge firstly.
> The VFs may be passed to different VMs.
>
> I also think this is the networking specific. But I want to config
> the ip for every vf from the pf.

What do you mean by ip here (e.g who is the user for this ip?)

> Because the user of the vf may be unreliable.
> We need a manager to config the ip for every vf.

Did you mean you're using a tunnel or not?

>
>
> > Control virtqueue is better than admin virtqueue here.
>
> by cq?
>
> What case?

We've already used control virtqueue for steering.

Thanks

>
> Thanks.
>
> >
> > Thanks
> >
> > >
> > > Thanks.
> > >
> > > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > >> Thanks
> > > > >>> Thanks.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>> Thanks
> > > > >>>>
> > > > >>> This publicly archived list offers a means to provide input to the
> > > > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>
> > > > >>> In order to verify user consent to the Feedback License terms and
> > > > >>> to minimize spam in the list archive, subscription is required
> > > > >>> before posting.
> > > > >>>
> > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>
> > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Thu, 27 Jul 2023 16:28:07 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >
> > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > >
> > > > >
> > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > > >>
> > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > > > >>>>>>
> > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > > > >>>>>> backend?
> > > > > >>>>> Yes, why not.
> > > > > >>>>>
> > > > > >>>>>>>>> The key point is who want to create a new device.
> > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > > > >>>>>>>>
> > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > > > >>>>> Xuan partly explained it.
> > > > > >>>>>
> > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > > > >>>>>>>> wants to create and hot plug in a device to a guest?
> > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > > > >>>>>>>> machines?
> > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > > > >>>>>>> to create new devices.
> > > > > >>>>>>>
> > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > > > >>>>>>> entire host is for users. Of course, there are situations where the
> > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > > > >>>>>>>
> > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > > > >>>>>>> will plug a device to the host from the DPU.
> > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > > > >>>>>> to the host.
> > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means
> > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > > > >>>>>> a host driver help handling an interrupt/probe.
> > > > > >>>>>>
> > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > > > >>>>>
> > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > > > >>>>>> the host by spanning the device across the PCI hierarchy.
> > > > > >>>>>>
> > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > > > >>>>>
> > > > > >>>>>> To address an ADI, there is only a device_id.
> > > > > >>>>>>
> > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > > > >>>>>
> > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > > > >>>>>> its own?
> > > > > >>>>>>
> > > > > >>>>> It is a virtio nic, blk and other virtio devices for us.
> > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > > > >>>>>
> > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > > > >>>>> You might find a Linux reference model useful at [1].
> > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > > > >>>>>
> > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > > > >>>>>
> > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > > > >>>> well I see SF facing the similar challenge, I can add a command for the
> > > > > >>>> driver to query all existing SIOV ADIs of a device,
> > > > > >>>> and the device return ADIs id and status. Looks good? and work for you
> > > > > >>>> @Xuan?
> > > > > >>> Could I have your plan for this?
> > > > > >>>
> > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > > > >>> as mac, ip, etc.
> > > > > >> I think the query commands for SIOV is a little more complex, e.g.,
> > > > > >> need to report device type and its scale(e.g., features, mq).
> > > > > >> There can be thousands of SIOV ADIs and we don't want output flood.
> > > > > >>
> > > > > >> We have discussed implementation a config interrupt to report new
> > > > > >> created / deleted
> > > > > >> ADIs on the DPU side, therefore there must be a cap contains related
> > > > > >> information,
> > > > > >> my rough approach of the process is:
> > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id
> > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of
> > > > > >> ADIs in a [dev_id....dev_id2] range.
> > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > > > >
> > > > > >> I am not sure whether a NIC stores its IP
> > > > > >
> > > > > > IP is the other topic. I want the Admin Queue manage the switch.
> > > > > > So the switch know about the IP of every device, and the
> > > > > > Admin Queue will has the ability to config the IP of the device inside the
> > > > > > switch.
> > > > > DPU onboard switch? OVS? Does it beyond virtio spec?
> > > >
> > > > YES.
> > >
> > > Adding Washizu.
> > >
> > > We can have a switch/dpa defined in the networking device for sure.
> >
> > Yes, I think we should introduce that for the sr-iov. Or for other.
>
> This should be a general one as a switch should be transport independent.

I agree.


>
> >
> > I would like to know who is doing this?
>
> Washizu, could you confirm if you want to do this or not?
>
> >
> > Another question, @Jason are you referring to a new device type or a
> > new virtio-net feature.
>
> Extending virtio-net should be fine, did you see any issues for this?

You kwnow the packet will be steered to different virtio-net devices.
So I think this is odd if the feature is the extending of virtio-net.

How do you think about this?


>
> >
> > >
> > > >
> > > > For SIOV, I think this is MUST.
> > >
> > > A learning bridge would be fine as a starter. It's better not to
> > > couple new scalable capability with any device specific features.
> > >
> > > > Maybe you have one simple implementation.
> > > > But you have to solve the IP steering. So admin queue should has the ability
> > > > to config the IP steering.
> > >
> > > I think not. Those L2/LN tables/filters are networking specific.
> >
> > Let us assume that there is a switch/bridge firstly.
> > The VFs may be passed to different VMs.
> >
> > I also think this is the networking specific. But I want to config
> > the ip for every vf from the pf.
>
> What do you mean by ip here (e.g who is the user for this ip?)

1. the admin queue config the ip for the vf to the switch
2. the vf get the ip by dhcp from the switch. (or other way)

Here we config an ip for the vf, and we also limit that
just this vf can use the ip for security. Other vf can not use the ip configured
by self.


>
> > Because the user of the vf may be unreliable.
> > We need a manager to config the ip for every vf.
>
> Did you mean you're using a tunnel or not?

No.

Multiple VMs on a host may be used by different users. Some users are not
trustworthy. We want to prevent a user from maliciously using ip

>
> >
> >
> > > Control virtqueue is better than admin virtqueue here.
> >
> > by cq?
> >
> > What case?
>
> We've already used control virtqueue for steering.
>


More details?

You config the steering rule to other virtio-net device by the cq of one
virtio-net?

Thanks.



> Thanks
>
> >
> > Thanks.
> >
> > >
> > > Thanks
> > >
> > > >
> > > > Thanks.
> > > >
> > > > > >
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > >
> > > > > >> Thanks
> > > > > >>> Thanks.
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>> Thanks
> > > > > >>>>
> > > > > >>> This publicly archived list offers a means to provide input to the
> > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > >>>
> > > > > >>> In order to verify user consent to the Feedback License terms and
> > > > > >>> to minimize spam in the list archive, subscription is required
> > > > > >>> before posting.
> > > > > >>>
> > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > >>> List help: virtio-comment-help@lists.oasis-open.org
> > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > > >>> Join OASIS: https://www.oasis-open.org/join/
> > > > > >>>
> > > > >
> > > > >
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and
> > > > > to minimize spam in the list archive, subscription is required
> > > > > before posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Jason Wang]

On Thu, Jul 27, 2023 at 4:41 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Thu, 27 Jul 2023 16:28:07 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > >
> > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > > >
> > > > > >
> > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > > > >>
> > > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > > > > >>>>>>
> > > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > > > > >>>>>> backend?
> > > > > > >>>>> Yes, why not.
> > > > > > >>>>>
> > > > > > >>>>>>>>> The key point is who want to create a new device.
> > > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > > > > >>>>>>>>
> > > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > > > > >>>>> Xuan partly explained it.
> > > > > > >>>>>
> > > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > > > > >>>>>>>> wants to create and hot plug in a device to a guest?
> > > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > > > > >>>>>>>> machines?
> > > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > > > > >>>>>>> to create new devices.
> > > > > > >>>>>>>
> > > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > > > > >>>>>>> entire host is for users. Of course, there are situations where the
> > > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > > > > >>>>>>>
> > > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > > > > >>>>>>> will plug a device to the host from the DPU.
> > > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > > > > >>>>>> to the host.
> > > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means
> > > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > > > > >>>>>> a host driver help handling an interrupt/probe.
> > > > > > >>>>>>
> > > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > > > > >>>>>
> > > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > > > > >>>>>> the host by spanning the device across the PCI hierarchy.
> > > > > > >>>>>>
> > > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > > > > >>>>>
> > > > > > >>>>>> To address an ADI, there is only a device_id.
> > > > > > >>>>>>
> > > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > > > > >>>>>
> > > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > > > > >>>>>> its own?
> > > > > > >>>>>>
> > > > > > >>>>> It is a virtio nic, blk and other virtio devices for us.
> > > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > > > > >>>>>
> > > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > > > > >>>>> You might find a Linux reference model useful at [1].
> > > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > > > > >>>>>
> > > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > > > > >>>>>
> > > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > > > > >>>> well I see SF facing the similar challenge, I can add a command for the
> > > > > > >>>> driver to query all existing SIOV ADIs of a device,
> > > > > > >>>> and the device return ADIs id and status. Looks good? and work for you
> > > > > > >>>> @Xuan?
> > > > > > >>> Could I have your plan for this?
> > > > > > >>>
> > > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > > > > >>> as mac, ip, etc.
> > > > > > >> I think the query commands for SIOV is a little more complex, e.g.,
> > > > > > >> need to report device type and its scale(e.g., features, mq).
> > > > > > >> There can be thousands of SIOV ADIs and we don't want output flood.
> > > > > > >>
> > > > > > >> We have discussed implementation a config interrupt to report new
> > > > > > >> created / deleted
> > > > > > >> ADIs on the DPU side, therefore there must be a cap contains related
> > > > > > >> information,
> > > > > > >> my rough approach of the process is:
> > > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id
> > > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of
> > > > > > >> ADIs in a [dev_id....dev_id2] range.
> > > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > > > > >
> > > > > > >> I am not sure whether a NIC stores its IP
> > > > > > >
> > > > > > > IP is the other topic. I want the Admin Queue manage the switch.
> > > > > > > So the switch know about the IP of every device, and the
> > > > > > > Admin Queue will has the ability to config the IP of the device inside the
> > > > > > > switch.
> > > > > > DPU onboard switch? OVS? Does it beyond virtio spec?
> > > > >
> > > > > YES.
> > > >
> > > > Adding Washizu.
> > > >
> > > > We can have a switch/dpa defined in the networking device for sure.
> > >
> > > Yes, I think we should introduce that for the sr-iov. Or for other.
> >
> > This should be a general one as a switch should be transport independent.
>
> I agree.
>
>
> >
> > >
> > > I would like to know who is doing this?
> >
> > Washizu, could you confirm if you want to do this or not?
> >
> > >
> > > Another question, @Jason are you referring to a new device type or a
> > > new virtio-net feature.
> >
> > Extending virtio-net should be fine, did you see any issues for this?
>
> You kwnow the packet will be steered to different virtio-net devices.
> So I think this is odd if the feature is the extending of virtio-net.
>
> How do you think about this?

I think this is the common model of modern NICs? For example the PF
with switch is usually an ethernet device itself.

>
>
> >
> > >
> > > >
> > > > >
> > > > > For SIOV, I think this is MUST.
> > > >
> > > > A learning bridge would be fine as a starter. It's better not to
> > > > couple new scalable capability with any device specific features.
> > > >
> > > > > Maybe you have one simple implementation.
> > > > > But you have to solve the IP steering. So admin queue should has the ability
> > > > > to config the IP steering.
> > > >
> > > > I think not. Those L2/LN tables/filters are networking specific.
> > >
> > > Let us assume that there is a switch/bridge firstly.
> > > The VFs may be passed to different VMs.
> > >
> > > I also think this is the networking specific. But I want to config
> > > the ip for every vf from the pf.
> >
> > What do you mean by ip here (e.g who is the user for this ip?)
>
> 1. the admin queue config the ip for the vf to the switch
> 2. the vf get the ip by dhcp from the switch. (or other way)
>
> Here we config an ip for the vf, and we also limit that
> just this vf can use the ip for security. Other vf can not use the ip configured
> by self.

It looks like a feature which is orthogonal to switch.

>
>
> >
> > > Because the user of the vf may be unreliable.
> > > We need a manager to config the ip for every vf.
> >
> > Did you mean you're using a tunnel or not?
>
> No.
>
> Multiple VMs on a host may be used by different users. Some users are not
> trustworthy. We want to prevent a user from maliciously using ip

Ok, so it's an independent feature which could be done separately.

>
> >
> > >
> > >
> > > > Control virtqueue is better than admin virtqueue here.
> > >
> > > by cq?
> > >
> > > What case?
> >
> > We've already used control virtqueue for steering.
> >
>
>
> More details?
>
> You config the steering rule to other virtio-net device by the cq of one
> virtio-net?

Exactly, we had already had the owner and group definition. So it's
not that hard? Actually, there should be layers in the middle for
example, the switch should see only ports and there should be some
compositions of ports to a device I guess.

Thanks

>
> Thanks.
>
>
>
> > Thanks
> >
> > >
> > > Thanks.
> > >
> > > >
> > > > Thanks
> > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > >
> > > > > > >> Thanks
> > > > > > >>> Thanks.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>>> Thanks
> > > > > > >>>>
> > > > > > >>> This publicly archived list offers a means to provide input to the
> > > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > >>>
> > > > > > >>> In order to verify user consent to the Feedback License terms and
> > > > > > >>> to minimize spam in the list archive, subscription is required
> > > > > > >>> before posting.
> > > > > > >>>
> > > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > >>> List help: virtio-comment-help@lists.oasis-open.org
> > > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > >>> Join OASIS: https://www.oasis-open.org/join/
> > > > > > >>>
> > > > > >
> > > > > >
> > > > > > This publicly archived list offers a means to provide input to the
> > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > >
> > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > to minimize spam in the list archive, subscription is required
> > > > > > before posting.
> > > > > >
> > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > >
> > > > >
> > > > > This publicly archived list offers a means to provide input to the
> > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >
> > > > > In order to verify user consent to the Feedback License terms and
> > > > > to minimize spam in the list archive, subscription is required
> > > > > before posting.
> > > > >
> > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > >
> > > >
> > >
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Thu, 27 Jul 2023 16:56:55 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Thu, Jul 27, 2023 at 4:41 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Thu, 27 Jul 2023 16:28:07 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >
> > > > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > > > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > > >
> > > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > > > >
> > > > > > >
> > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > > > > >>
> > > > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > > > > > >>>>>>
> > > > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > > > > > >>>>>> backend?
> > > > > > > >>>>> Yes, why not.
> > > > > > > >>>>>
> > > > > > > >>>>>>>>> The key point is who want to create a new device.
> > > > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > > > > > >>>>>>>>
> > > > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > > > > > >>>>> Xuan partly explained it.
> > > > > > > >>>>>
> > > > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > > > > > >>>>>>>> wants to create and hot plug in a device to a guest?
> > > > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > > > > > >>>>>>>> machines?
> > > > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > > > > > >>>>>>> to create new devices.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > > > > > >>>>>>> entire host is for users. Of course, there are situations where the
> > > > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > > > > > >>>>>>>
> > > > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > > > > > >>>>>>> will plug a device to the host from the DPU.
> > > > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > > > > > >>>>>> to the host.
> > > > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means
> > > > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > > > > > >>>>>> a host driver help handling an interrupt/probe.
> > > > > > > >>>>>>
> > > > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > > > > > >>>>>
> > > > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > > > > > >>>>>> the host by spanning the device across the PCI hierarchy.
> > > > > > > >>>>>>
> > > > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > > > > > >>>>>
> > > > > > > >>>>>> To address an ADI, there is only a device_id.
> > > > > > > >>>>>>
> > > > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > > > > > >>>>>
> > > > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > > > > > >>>>>> its own?
> > > > > > > >>>>>>
> > > > > > > >>>>> It is a virtio nic, blk and other virtio devices for us.
> > > > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > > > > > >>>>>
> > > > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > > > > > >>>>> You might find a Linux reference model useful at [1].
> > > > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > > > > > >>>>>
> > > > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > > > > > >>>>>
> > > > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > > > > > >>>> well I see SF facing the similar challenge, I can add a command for the
> > > > > > > >>>> driver to query all existing SIOV ADIs of a device,
> > > > > > > >>>> and the device return ADIs id and status. Looks good? and work for you
> > > > > > > >>>> @Xuan?
> > > > > > > >>> Could I have your plan for this?
> > > > > > > >>>
> > > > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > > > > > >>> as mac, ip, etc.
> > > > > > > >> I think the query commands for SIOV is a little more complex, e.g.,
> > > > > > > >> need to report device type and its scale(e.g., features, mq).
> > > > > > > >> There can be thousands of SIOV ADIs and we don't want output flood.
> > > > > > > >>
> > > > > > > >> We have discussed implementation a config interrupt to report new
> > > > > > > >> created / deleted
> > > > > > > >> ADIs on the DPU side, therefore there must be a cap contains related
> > > > > > > >> information,
> > > > > > > >> my rough approach of the process is:
> > > > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id
> > > > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of
> > > > > > > >> ADIs in a [dev_id....dev_id2] range.
> > > > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > > > > > >
> > > > > > > >> I am not sure whether a NIC stores its IP
> > > > > > > >
> > > > > > > > IP is the other topic. I want the Admin Queue manage the switch.
> > > > > > > > So the switch know about the IP of every device, and the
> > > > > > > > Admin Queue will has the ability to config the IP of the device inside the
> > > > > > > > switch.
> > > > > > > DPU onboard switch? OVS? Does it beyond virtio spec?
> > > > > >
> > > > > > YES.
> > > > >
> > > > > Adding Washizu.
> > > > >
> > > > > We can have a switch/dpa defined in the networking device for sure.
> > > >
> > > > Yes, I think we should introduce that for the sr-iov. Or for other.
> > >
> > > This should be a general one as a switch should be transport independent.
> >
> > I agree.
> >
> >
> > >
> > > >
> > > > I would like to know who is doing this?
> > >
> > > Washizu, could you confirm if you want to do this or not?
> > >
> > > >
> > > > Another question, @Jason are you referring to a new device type or a
> > > > new virtio-net feature.
> > >
> > > Extending virtio-net should be fine, did you see any issues for this?
> >
> > You kwnow the packet will be steered to different virtio-net devices.
> > So I think this is odd if the feature is the extending of virtio-net.
> >
> > How do you think about this?
>
> I think this is the common model of modern NICs? For example the PF
> with switch is usually an ethernet device itself.

Great!


>
> >
> >
> > >
> > > >
> > > > >
> > > > > >
> > > > > > For SIOV, I think this is MUST.
> > > > >
> > > > > A learning bridge would be fine as a starter. It's better not to
> > > > > couple new scalable capability with any device specific features.
> > > > >
> > > > > > Maybe you have one simple implementation.
> > > > > > But you have to solve the IP steering. So admin queue should has the ability
> > > > > > to config the IP steering.
> > > > >
> > > > > I think not. Those L2/LN tables/filters are networking specific.
> > > >
> > > > Let us assume that there is a switch/bridge firstly.
> > > > The VFs may be passed to different VMs.
> > > >
> > > > I also think this is the networking specific. But I want to config
> > > > the ip for every vf from the pf.
> > >
> > > What do you mean by ip here (e.g who is the user for this ip?)
> >
> > 1. the admin queue config the ip for the vf to the switch
> > 2. the vf get the ip by dhcp from the switch. (or other way)
> >
> > Here we config an ip for the vf, and we also limit that
> > just this vf can use the ip for security. Other vf can not use the ip configured
> > by self.
>
> It looks like a feature which is orthogonal to switch.
>
> >
> >
> > >
> > > > Because the user of the vf may be unreliable.
> > > > We need a manager to config the ip for every vf.
> > >
> > > Did you mean you're using a tunnel or not?
> >
> > No.
> >
> > Multiple VMs on a host may be used by different users. Some users are not
> > trustworthy. We want to prevent a user from maliciously using ip
>
> Ok, so it's an independent feature which could be done separately.
>
> >
> > >
> > > >
> > > >
> > > > > Control virtqueue is better than admin virtqueue here.
> > > >
> > > > by cq?
> > > >
> > > > What case?
> > >
> > > We've already used control virtqueue for steering.

Now, the spec includes this?

Do you mean that you config the steering by the control virtqueue of pf?

If so, I think is ok.

Thanks.

> > >
> >
> >
> > More details?
> >
> > You config the steering rule to other virtio-net device by the cq of one
> > virtio-net?
>
> Exactly, we had already had the owner and group definition. So it's
> not that hard? Actually, there should be layers in the middle for
> example, the switch should see only ports and there should be some
> compositions of ports to a device I guess.
>
> Thanks
>
> >
> > Thanks.
> >
> >
> >
> > > Thanks
> > >
> > > >
> > > > Thanks.
> > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > >> Thanks
> > > > > > > >>> Thanks.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>> Thanks
> > > > > > > >>>>
> > > > > > > >>> This publicly archived list offers a means to provide input to the
> > > > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > > >>>
> > > > > > > >>> In order to verify user consent to the Feedback License terms and
> > > > > > > >>> to minimize spam in the list archive, subscription is required
> > > > > > > >>> before posting.
> > > > > > > >>>
> > > > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > >>> List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > >>> Join OASIS: https://www.oasis-open.org/join/
> > > > > > > >>>
> > > > > > >
> > > > > > >
> > > > > > > This publicly archived list offers a means to provide input to the
> > > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > > >
> > > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > > to minimize spam in the list archive, subscription is required
> > > > > > > before posting.
> > > > > > >
> > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > > >
> > > > > >
> > > > > > This publicly archived list offers a means to provide input to the
> > > > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > > > >
> > > > > > In order to verify user consent to the Feedback License terms and
> > > > > > to minimize spam in the list archive, subscription is required
> > > > > > before posting.
> > > > > >
> > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > > > Join OASIS: https://www.oasis-open.org/join/
> > > > > >
> > > > >
> > > >
> > >
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >>
> > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > >>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > >>>>>>
> > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > >>>>>> backend?
> > > >>>>> Yes, why not.
> > > >>>>>
> > > >>>>>>>>> The key point is who want to create a new device.
> > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > >>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > >>>>>>>>
> > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > >>>>> Xuan partly explained it.
> > > >>>>>
> > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > >>>>>>>> wants to create and hot plug in a device to a guest?
> > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > >>>>>>>> machines?
> > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > >>>>>>> to create new devices.
> > > >>>>>>>
> > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > >>>>>>> entire host is for users. Of course, there are situations where the
> > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > >>>>>>>
> > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > >>>>>>> will plug a device to the host from the DPU.
> > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > >>>>>> to the host.
> > > >>>>>> However you only owns the DPU and the customer owns the host, that means
> > > >>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > >>>>>> a host driver help handling an interrupt/probe.
> > > >>>>>>
> > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > >>>>>
> > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > >>>>>> the host by spanning the device across the PCI hierarchy.
> > > >>>>>>
> > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > >>>>>
> > > >>>>>> To address an ADI, there is only a device_id.
> > > >>>>>>
> > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > >>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > >>>>>
> > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > >>>>>> its own?
> > > >>>>>>
> > > >>>>> It is a virtio nic, blk and other virtio devices for us.
> > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > >>>>>
> > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > >>>>> You might find a Linux reference model useful at [1].
> > > >>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > >>>>>
> > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > >>>>>
> > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > >>>> well I see SF facing the similar challenge, I can add a command for the
> > > >>>> driver to query all existing SIOV ADIs of a device,
> > > >>>> and the device return ADIs id and status. Looks good? and work for you
> > > >>>> @Xuan?
> > > >>> Could I have your plan for this?
> > > >>>
> > > >>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > >>> as mac, ip, etc.
> > > >> I think the query commands for SIOV is a little more complex, e.g.,
> > > >> need to report device type and its scale(e.g., features, mq).
> > > >> There can be thousands of SIOV ADIs and we don't want output flood.
> > > >>
> > > >> We have discussed implementation a config interrupt to report new
> > > >> created / deleted
> > > >> ADIs on the DPU side, therefore there must be a cap contains related
> > > >> information,
> > > >> my rough approach of the process is:
> > > >> 1) a cap contains the total number of existing ADIs and the max dev id
> > > >> 2) driver queries detailed information of a certain ADI or a bunch of
> > > >> ADIs in a [dev_id....dev_id2] range.
> > > > Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > >
> > > >> I am not sure whether a NIC stores its IP
> > > >
> > > > IP is the other topic. I want the Admin Queue manage the switch.
> > > > So the switch know about the IP of every device, and the
> > > > Admin Queue will has the ability to config the IP of the device inside the
> > > > switch.
> > > DPU onboard switch? OVS? Does it beyond virtio spec?
> >
> > YES.
>
> Adding Washizu.
>
> We can have a switch/dpa defined in the networking device for sure.
>
> >
> > For SIOV, I think this is MUST.
>
> A learning bridge would be fine as a starter. It's better not to
> couple new scalable capability with any device specific features.
>
> > Maybe you have one simple implementation.
> > But you have to solve the IP steering. So admin queue should has the ability
> > to config the IP steering.
>
> I think not. Those L2/LN tables/filters are networking specific.
> Control virtqueue is better than admin virtqueue here.

So I think some things like ring size, queue num, interrupts etc. are configured
by the admin queue. VF's ip, mac should be configured by PF's cvq.

Is that right?

@Michael @Jason

Thanks.


>
> Thanks
>
> >
> > Thanks.
> >
> > > >
> > > >
> > > > Thanks.
> > > >
> > > >
> > > >> Thanks
> > > >>> Thanks.
> > > >>>
> > > >>>
> > > >>>
> > > >>>> Thanks
> > > >>>>
> > > >>> This publicly archived list offers a means to provide input to the
> > > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>
> > > >>> In order to verify user consent to the Feedback License terms and
> > > >>> to minimize spam in the list archive, subscription is required
> > > >>> before posting.
> > > >>>
> > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Jason Wang]

On Fri, Jul 28, 2023 at 2:14 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >
> > > >
> > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > >>
> > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > > >>>>>>
> > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > > >>>>>> backend?
> > > > >>>>> Yes, why not.
> > > > >>>>>
> > > > >>>>>>>>> The key point is who want to create a new device.
> > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > > >>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > > >>>>>>>>
> > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > > >>>>> Xuan partly explained it.
> > > > >>>>>
> > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > > >>>>>>>> wants to create and hot plug in a device to a guest?
> > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > > >>>>>>>> machines?
> > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > > >>>>>>> to create new devices.
> > > > >>>>>>>
> > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > > >>>>>>> entire host is for users. Of course, there are situations where the
> > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > > >>>>>>>
> > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > > >>>>>>> will plug a device to the host from the DPU.
> > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > > >>>>>> to the host.
> > > > >>>>>> However you only owns the DPU and the customer owns the host, that means
> > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > > >>>>>> a host driver help handling an interrupt/probe.
> > > > >>>>>>
> > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > > >>>>>
> > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > > >>>>>> the host by spanning the device across the PCI hierarchy.
> > > > >>>>>>
> > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > > >>>>>
> > > > >>>>>> To address an ADI, there is only a device_id.
> > > > >>>>>>
> > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > > >>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > > >>>>>
> > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > > >>>>>> its own?
> > > > >>>>>>
> > > > >>>>> It is a virtio nic, blk and other virtio devices for us.
> > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > > >>>>>
> > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > > >>>>> You might find a Linux reference model useful at [1].
> > > > >>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > > >>>>>
> > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > > >>>>>
> > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > > >>>> well I see SF facing the similar challenge, I can add a command for the
> > > > >>>> driver to query all existing SIOV ADIs of a device,
> > > > >>>> and the device return ADIs id and status. Looks good? and work for you
> > > > >>>> @Xuan?
> > > > >>> Could I have your plan for this?
> > > > >>>
> > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > > >>> as mac, ip, etc.
> > > > >> I think the query commands for SIOV is a little more complex, e.g.,
> > > > >> need to report device type and its scale(e.g., features, mq).
> > > > >> There can be thousands of SIOV ADIs and we don't want output flood.
> > > > >>
> > > > >> We have discussed implementation a config interrupt to report new
> > > > >> created / deleted
> > > > >> ADIs on the DPU side, therefore there must be a cap contains related
> > > > >> information,
> > > > >> my rough approach of the process is:
> > > > >> 1) a cap contains the total number of existing ADIs and the max dev id
> > > > >> 2) driver queries detailed information of a certain ADI or a bunch of
> > > > >> ADIs in a [dev_id....dev_id2] range.
> > > > > Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > > >
> > > > >> I am not sure whether a NIC stores its IP
> > > > >
> > > > > IP is the other topic. I want the Admin Queue manage the switch.
> > > > > So the switch know about the IP of every device, and the
> > > > > Admin Queue will has the ability to config the IP of the device inside the
> > > > > switch.
> > > > DPU onboard switch? OVS? Does it beyond virtio spec?
> > >
> > > YES.
> >
> > Adding Washizu.
> >
> > We can have a switch/dpa defined in the networking device for sure.
> >
> > >
> > > For SIOV, I think this is MUST.
> >
> > A learning bridge would be fine as a starter. It's better not to
> > couple new scalable capability with any device specific features.
> >
> > > Maybe you have one simple implementation.
> > > But you have to solve the IP steering. So admin queue should has the ability
> > > to config the IP steering.
> >
> > I think not. Those L2/LN tables/filters are networking specific.
> > Control virtqueue is better than admin virtqueue here.
>
> So I think some things like ring size, queue num, interrupts etc. are configured
> by the admin queue. VF's ip, mac should be configured by PF's cvq.
>
> Is that right?
>
> @Michael @Jason

That's my understanding.

Thanks

>
> Thanks.
>
>
> >
> > Thanks
> >
> > >
> > > Thanks.
> > >
> > > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > >> Thanks
> > > > >>> Thanks.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>> Thanks
> > > > >>>>
> > > > >>> This publicly archived list offers a means to provide input to the
> > > > >>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>
> > > > >>> In order to verify user consent to the Feedback License terms and
> > > > >>> to minimize spam in the list archive, subscription is required
> > > > >>> before posting.
> > > > >>>
> > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>
> > > >
> > > >
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and
> > > > to minimize spam in the list archive, subscription is required
> > > > before posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

RE: [virtio-comment] About the plan of Admin Queue

[Parav Pandit]

> From: Jason Wang <jasowang@redhat.com>
> Sent: Monday, July 31, 2023 6:50 AM
> > So I think some things like ring size, queue num, interrupts etc. are
> > configured by the admin queue. VF's ip, mac should be configured by PF's cvq.

VF's mac should be configured by itself using its own cvq. There is no mediation required.

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Wed, 2 Aug 2023 15:01:21 +0900, Yui Washizu <yui.washidu@gmail.com> wrote:
>
> On 2023/07/27 17:28, Jason Wang wrote:
> > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>
> >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan
> >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM
> >>>>>>>>>>>
> >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> >>>>>>>>>>> backend?
> >>>>>>>>>> Yes, why not.
> >>>>>>>>>>
> >>>>>>>>>>>>>> The key point is who want to create a new device.
> >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs.
> >>>>>>>>>>>>>
> >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> >>>>>>>>>> Xuan partly explained it.
> >>>>>>>>>>
> >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >>>>>>>>>>>>> wants to create and hot plug in a device to a guest?
> >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> >>>>>>>>>>>>> machines?
> >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> >>>>>>>>>>>> to create new devices.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> >>>>>>>>>>>> entire host is for users. Of course, there are situations where the
> >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager
> >>>>>>>>>>>> will plug a device to the host from the DPU.
> >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> >>>>>>>>>>> to the host.
> >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means
> >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be
> >>>>>>>>>>> a host driver help handling an interrupt/probe.
> >>>>>>>>>>>
> >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> >>>>>>>>>>
> >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> >>>>>>>>>>> the host by spanning the device across the PCI hierarchy.
> >>>>>>>>>>>
> >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> >>>>>>>>>>
> >>>>>>>>>>> To address an ADI, there is only a device_id.
> >>>>>>>>>>>
> >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue.
> >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> >>>>>>>>>>
> >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of
> >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> >>>>>>>>>>> its own?
> >>>>>>>>>>>
> >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us.
> >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> >>>>>>>>>>
> >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> >>>>>>>>>> You might find a Linux reference model useful at [1].
> >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices.
> >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> >>>>>>>>>>
> >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> >>>>>>>>>>
> >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the
> >>>>>>>>> driver to query all existing SIOV ADIs of a device,
> >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you
> >>>>>>>>> @Xuan?
> >>>>>>>> Could I have your plan for this?
> >>>>>>>>
> >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such
> >>>>>>>> as mac, ip, etc.
> >>>>>>> I think the query commands for SIOV is a little more complex, e.g.,
> >>>>>>> need to report device type and its scale(e.g., features, mq).
> >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood.
> >>>>>>>
> >>>>>>> We have discussed implementation a config interrupt to report new
> >>>>>>> created / deleted
> >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related
> >>>>>>> information,
> >>>>>>> my rough approach of the process is:
> >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id
> >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of
> >>>>>>> ADIs in a [dev_id....dev_id2] range.
> >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices.
> >>>>>>
> >>>>>>> I am not sure whether a NIC stores its IP
> >>>>>> IP is the other topic. I want the Admin Queue manage the switch.
> >>>>>> So the switch know about the IP of every device, and the
> >>>>>> Admin Queue will has the ability to config the IP of the device inside the
> >>>>>> switch.
> >>>>> DPU onboard switch? OVS? Does it beyond virtio spec?
> >>>> YES.
> >>> Adding Washizu.
> >>>
> >>> We can have a switch/dpa defined in the networking device for sure.
> >> Yes, I think we should introduce that for the sr-iov. Or for other.
> > This should be a general one as a switch should be transport independent.
> >
> >> I would like to know who is doing this?
> > Washizu, could you confirm if you want to do this or not?
>
>
> Does this mean adding a switch definition to the virtio spec?
>
>
> If so, it will be necessary for the implementation of my plan,
>
> but it may take time (probably several months?) to get started,
>
> as I'm currently working on another task (virtio-net SR-IOV feature in
> qemu).
>
> Anyone is welcome to work on adding the switch definition in the meantime,
>
> it's completely fine with me.
>
> I think I'll work on that if no one has finished the work.

OK, I got.

Because we have a need in this area, I will push the work in this area.

Thanks.

>
>
>
> >
> >> Another question, @Jason are you referring to a new device type or a
> >> new virtio-net feature.
> > Extending virtio-net should be fine, did you see any issues for this?
> >
> >>>> For SIOV, I think this is MUST.
> >>> A learning bridge would be fine as a starter. It's better not to
> >>> couple new scalable capability with any device specific features.
> >>>
> >>>> Maybe you have one simple implementation.
> >>>> But you have to solve the IP steering. So admin queue should has the ability
> >>>> to config the IP steering.
> >>> I think not. Those L2/LN tables/filters are networking specific.
> >> Let us assume that there is a switch/bridge firstly.
> >> The VFs may be passed to different VMs.
> >>
> >> I also think this is the networking specific. But I want to config
> >> the ip for every vf from the pf.
> > What do you mean by ip here (e.g who is the user for this ip?)
> >
> >> Because the user of the vf may be unreliable.
> >> We need a manager to config the ip for every vf.
> > Did you mean you're using a tunnel or not?
> >
> >>
> >>> Control virtqueue is better than admin virtqueue here.
> >> by cq?
> >>
> >> What case?
> > We've already used control virtqueue for steering.
> >
> > Thanks
> >
> >> Thanks.
> >>
> >>> Thanks
> >>>
> >>>> Thanks.
> >>>>
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>>> Thanks
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>> This publicly archived list offers a means to provide input to the
> >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>>>
> >>>>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>>>> to minimize spam in the list archive, subscription is required
> >>>>>>>> before posting.
> >>>>>>>>
> >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>>>
> >>>>>
> >>>>> This publicly archived list offers a means to provide input to the
> >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>
> >>>>> In order to verify user consent to the Feedback License terms and
> >>>>> to minimize spam in the list archive, subscription is required
> >>>>> before posting.
> >>>>>
> >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>
> >>>> This publicly archived list offers a means to provide input to the
> >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>
> >>>> In order to verify user consent to the Feedback License terms and
> >>>> to minimize spam in the list archive, subscription is required
> >>>> before posting.
> >>>>
> >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Jason Wang]

On Wed, Aug 2, 2023 at 2:01 PM Yui Washizu <yui.washidu@gmail.com> wrote:
>
>
> On 2023/07/27 17:28, Jason Wang wrote:
> > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>
> >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan
> >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM
> >>>>>>>>>>>
> >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> >>>>>>>>>>> backend?
> >>>>>>>>>> Yes, why not.
> >>>>>>>>>>
> >>>>>>>>>>>>>> The key point is who want to create a new device.
> >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs.
> >>>>>>>>>>>>>
> >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> >>>>>>>>>> Xuan partly explained it.
> >>>>>>>>>>
> >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> >>>>>>>>>>>>> wants to create and hot plug in a device to a guest?
> >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> >>>>>>>>>>>>> machines?
> >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> >>>>>>>>>>>> to create new devices.
> >>>>>>>>>>>>
> >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> >>>>>>>>>>>> entire host is for users. Of course, there are situations where the
> >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> >>>>>>>>>>>>
> >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager
> >>>>>>>>>>>> will plug a device to the host from the DPU.
> >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> >>>>>>>>>>> to the host.
> >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means
> >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be
> >>>>>>>>>>> a host driver help handling an interrupt/probe.
> >>>>>>>>>>>
> >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> >>>>>>>>>>
> >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> >>>>>>>>>>> the host by spanning the device across the PCI hierarchy.
> >>>>>>>>>>>
> >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> >>>>>>>>>>
> >>>>>>>>>>> To address an ADI, there is only a device_id.
> >>>>>>>>>>>
> >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue.
> >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> >>>>>>>>>>
> >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of
> >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> >>>>>>>>>>> its own?
> >>>>>>>>>>>
> >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us.
> >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> >>>>>>>>>>
> >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> >>>>>>>>>> You might find a Linux reference model useful at [1].
> >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices.
> >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> >>>>>>>>>>
> >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> >>>>>>>>>>
> >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the
> >>>>>>>>> driver to query all existing SIOV ADIs of a device,
> >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you
> >>>>>>>>> @Xuan?
> >>>>>>>> Could I have your plan for this?
> >>>>>>>>
> >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such
> >>>>>>>> as mac, ip, etc.
> >>>>>>> I think the query commands for SIOV is a little more complex, e.g.,
> >>>>>>> need to report device type and its scale(e.g., features, mq).
> >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood.
> >>>>>>>
> >>>>>>> We have discussed implementation a config interrupt to report new
> >>>>>>> created / deleted
> >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related
> >>>>>>> information,
> >>>>>>> my rough approach of the process is:
> >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id
> >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of
> >>>>>>> ADIs in a [dev_id....dev_id2] range.
> >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices.
> >>>>>>
> >>>>>>> I am not sure whether a NIC stores its IP
> >>>>>> IP is the other topic. I want the Admin Queue manage the switch.
> >>>>>> So the switch know about the IP of every device, and the
> >>>>>> Admin Queue will has the ability to config the IP of the device inside the
> >>>>>> switch.
> >>>>> DPU onboard switch? OVS? Does it beyond virtio spec?
> >>>> YES.
> >>> Adding Washizu.
> >>>
> >>> We can have a switch/dpa defined in the networking device for sure.
> >> Yes, I think we should introduce that for the sr-iov. Or for other.
> > This should be a general one as a switch should be transport independent.
> >
> >> I would like to know who is doing this?
> > Washizu, could you confirm if you want to do this or not?
>
>
> Does this mean adding a switch definition to the virtio spec?
>
>
> If so, it will be necessary for the implementation of my plan,
>
> but it may take time (probably several months?) to get started,
>
> as I'm currently working on another task (virtio-net SR-IOV feature in
> qemu).
>
> Anyone is welcome to work on adding the switch definition in the meantime,
>
> it's completely fine with me.
>
> I think I'll work on that if no one has finished the work.

Ok, great, I think we need to start this by considering reusing one of
the existing DPA spec. (For example, ofdpa? or any other?)

Thanks

>
>
>
> >
> >> Another question, @Jason are you referring to a new device type or a
> >> new virtio-net feature.
> > Extending virtio-net should be fine, did you see any issues for this?
> >
> >>>> For SIOV, I think this is MUST.
> >>> A learning bridge would be fine as a starter. It's better not to
> >>> couple new scalable capability with any device specific features.
> >>>
> >>>> Maybe you have one simple implementation.
> >>>> But you have to solve the IP steering. So admin queue should has the ability
> >>>> to config the IP steering.
> >>> I think not. Those L2/LN tables/filters are networking specific.
> >> Let us assume that there is a switch/bridge firstly.
> >> The VFs may be passed to different VMs.
> >>
> >> I also think this is the networking specific. But I want to config
> >> the ip for every vf from the pf.
> > What do you mean by ip here (e.g who is the user for this ip?)
> >
> >> Because the user of the vf may be unreliable.
> >> We need a manager to config the ip for every vf.
> > Did you mean you're using a tunnel or not?
> >
> >>
> >>> Control virtqueue is better than admin virtqueue here.
> >> by cq?
> >>
> >> What case?
> > We've already used control virtqueue for steering.
> >
> > Thanks
> >
> >> Thanks.
> >>
> >>> Thanks
> >>>
> >>>> Thanks.
> >>>>
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>>> Thanks
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>>
> >>>>>>>> This publicly archived list offers a means to provide input to the
> >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>>>
> >>>>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>>>> to minimize spam in the list archive, subscription is required
> >>>>>>>> before posting.
> >>>>>>>>
> >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>>>
> >>>>>
> >>>>> This publicly archived list offers a means to provide input to the
> >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>
> >>>>> In order to verify user consent to the Feedback License terms and
> >>>>> to minimize spam in the list archive, subscription is required
> >>>>> before posting.
> >>>>>
> >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>
> >>>> This publicly archived list offers a means to provide input to the
> >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>
> >>>> In order to verify user consent to the Feedback License terms and
> >>>> to minimize spam in the list archive, subscription is required
> >>>> before posting.
> >>>>
> >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Wed, 2 Aug 2023 14:15:55 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Wed, Aug 2, 2023 at 2:01 PM Yui Washizu <yui.washidu@gmail.com> wrote:
> >
> >
> > On 2023/07/27 17:28, Jason Wang wrote:
> > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>>>>
> > >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan
> > >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > >>>>>>>>>>>
> > >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > >>>>>>>>>>> backend?
> > >>>>>>>>>> Yes, why not.
> > >>>>>>>>>>
> > >>>>>>>>>>>>>> The key point is who want to create a new device.
> > >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs.
> > >>>>>>>>>>>>>
> > >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > >>>>>>>>>> Xuan partly explained it.
> > >>>>>>>>>>
> > >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > >>>>>>>>>>>>> wants to create and hot plug in a device to a guest?
> > >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > >>>>>>>>>>>>> machines?
> > >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > >>>>>>>>>>>> to create new devices.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > >>>>>>>>>>>> entire host is for users. Of course, there are situations where the
> > >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > >>>>>>>>>>>> will plug a device to the host from the DPU.
> > >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > >>>>>>>>>>> to the host.
> > >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means
> > >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be
> > >>>>>>>>>>> a host driver help handling an interrupt/probe.
> > >>>>>>>>>>>
> > >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > >>>>>>>>>>
> > >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > >>>>>>>>>>> the host by spanning the device across the PCI hierarchy.
> > >>>>>>>>>>>
> > >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > >>>>>>>>>>
> > >>>>>>>>>>> To address an ADI, there is only a device_id.
> > >>>>>>>>>>>
> > >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue.
> > >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > >>>>>>>>>>
> > >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > >>>>>>>>>>> its own?
> > >>>>>>>>>>>
> > >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us.
> > >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > >>>>>>>>>>
> > >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > >>>>>>>>>> You might find a Linux reference model useful at [1].
> > >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices.
> > >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > >>>>>>>>>>
> > >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > >>>>>>>>>>
> > >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the
> > >>>>>>>>> driver to query all existing SIOV ADIs of a device,
> > >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you
> > >>>>>>>>> @Xuan?
> > >>>>>>>> Could I have your plan for this?
> > >>>>>>>>
> > >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such
> > >>>>>>>> as mac, ip, etc.
> > >>>>>>> I think the query commands for SIOV is a little more complex, e.g.,
> > >>>>>>> need to report device type and its scale(e.g., features, mq).
> > >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood.
> > >>>>>>>
> > >>>>>>> We have discussed implementation a config interrupt to report new
> > >>>>>>> created / deleted
> > >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related
> > >>>>>>> information,
> > >>>>>>> my rough approach of the process is:
> > >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id
> > >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of
> > >>>>>>> ADIs in a [dev_id....dev_id2] range.
> > >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices.
> > >>>>>>
> > >>>>>>> I am not sure whether a NIC stores its IP
> > >>>>>> IP is the other topic. I want the Admin Queue manage the switch.
> > >>>>>> So the switch know about the IP of every device, and the
> > >>>>>> Admin Queue will has the ability to config the IP of the device inside the
> > >>>>>> switch.
> > >>>>> DPU onboard switch? OVS? Does it beyond virtio spec?
> > >>>> YES.
> > >>> Adding Washizu.
> > >>>
> > >>> We can have a switch/dpa defined in the networking device for sure.
> > >> Yes, I think we should introduce that for the sr-iov. Or for other.
> > > This should be a general one as a switch should be transport independent.
> > >
> > >> I would like to know who is doing this?
> > > Washizu, could you confirm if you want to do this or not?
> >
> >
> > Does this mean adding a switch definition to the virtio spec?
> >
> >
> > If so, it will be necessary for the implementation of my plan,
> >
> > but it may take time (probably several months?) to get started,
> >
> > as I'm currently working on another task (virtio-net SR-IOV feature in
> > qemu).
> >
> > Anyone is welcome to work on adding the switch definition in the meantime,
> >
> > it's completely fine with me.
> >
> > I think I'll work on that if no one has finished the work.
>
> Ok, great, I think we need to start this by considering reusing one of
> the existing DPA spec. (For example, ofdpa? or any other?)

We have a need on this area. I want to start this work now.

Could you give me the link of the spec?

Thanks



>
> Thanks
>
> >
> >
> >
> > >
> > >> Another question, @Jason are you referring to a new device type or a
> > >> new virtio-net feature.
> > > Extending virtio-net should be fine, did you see any issues for this?
> > >
> > >>>> For SIOV, I think this is MUST.
> > >>> A learning bridge would be fine as a starter. It's better not to
> > >>> couple new scalable capability with any device specific features.
> > >>>
> > >>>> Maybe you have one simple implementation.
> > >>>> But you have to solve the IP steering. So admin queue should has the ability
> > >>>> to config the IP steering.
> > >>> I think not. Those L2/LN tables/filters are networking specific.
> > >> Let us assume that there is a switch/bridge firstly.
> > >> The VFs may be passed to different VMs.
> > >>
> > >> I also think this is the networking specific. But I want to config
> > >> the ip for every vf from the pf.
> > > What do you mean by ip here (e.g who is the user for this ip?)
> > >
> > >> Because the user of the vf may be unreliable.
> > >> We need a manager to config the ip for every vf.
> > > Did you mean you're using a tunnel or not?
> > >
> > >>
> > >>> Control virtqueue is better than admin virtqueue here.
> > >> by cq?
> > >>
> > >> What case?
> > > We've already used control virtqueue for steering.
> > >
> > > Thanks
> > >
> > >> Thanks.
> > >>
> > >>> Thanks
> > >>>
> > >>>> Thanks.
> > >>>>
> > >>>>>>
> > >>>>>> Thanks.
> > >>>>>>
> > >>>>>>
> > >>>>>>> Thanks
> > >>>>>>>> Thanks.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>> Thanks
> > >>>>>>>>>
> > >>>>>>>> This publicly archived list offers a means to provide input to the
> > >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>>>>>>
> > >>>>>>>> In order to verify user consent to the Feedback License terms and
> > >>>>>>>> to minimize spam in the list archive, subscription is required
> > >>>>>>>> before posting.
> > >>>>>>>>
> > >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> > >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>>>>>>> Join OASIS: https://www.oasis-open.org/join/
> > >>>>>>>>
> > >>>>>
> > >>>>> This publicly archived list offers a means to provide input to the
> > >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>>>
> > >>>>> In order to verify user consent to the Feedback License terms and
> > >>>>> to minimize spam in the list archive, subscription is required
> > >>>>> before posting.
> > >>>>>
> > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > >>>>>
> > >>>> This publicly archived list offers a means to provide input to the
> > >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > >>>>
> > >>>> In order to verify user consent to the Feedback License terms and
> > >>>> to minimize spam in the list archive, subscription is required
> > >>>> before posting.
> > >>>>
> > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > >>>> Join OASIS: https://www.oasis-open.org/join/
> > >>>>
> >
>
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Jason Wang]

On Wed, Aug 2, 2023 at 2:37 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Wed, 2 Aug 2023 14:15:55 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Wed, Aug 2, 2023 at 2:01 PM Yui Washizu <yui.washidu@gmail.com> wrote:
> > >
> > >
> > > On 2023/07/27 17:28, Jason Wang wrote:
> > > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >>>>>
> > > >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > >>>>>>>>>>>
> > > >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > >>>>>>>>>>> backend?
> > > >>>>>>>>>> Yes, why not.
> > > >>>>>>>>>>
> > > >>>>>>>>>>>>>> The key point is who want to create a new device.
> > > >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > >>>>>>>>>>>>>
> > > >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > >>>>>>>>>> Xuan partly explained it.
> > > >>>>>>>>>>
> > > >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > >>>>>>>>>>>>> wants to create and hot plug in a device to a guest?
> > > >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > >>>>>>>>>>>>> machines?
> > > >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > >>>>>>>>>>>> to create new devices.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > >>>>>>>>>>>> entire host is for users. Of course, there are situations where the
> > > >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > >>>>>>>>>>>>
> > > >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > >>>>>>>>>>>> will plug a device to the host from the DPU.
> > > >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > >>>>>>>>>>> to the host.
> > > >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means
> > > >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > >>>>>>>>>>> a host driver help handling an interrupt/probe.
> > > >>>>>>>>>>>
> > > >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > >>>>>>>>>>> the host by spanning the device across the PCI hierarchy.
> > > >>>>>>>>>>>
> > > >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> To address an ADI, there is only a device_id.
> > > >>>>>>>>>>>
> > > >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > >>>>>>>>>>
> > > >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > >>>>>>>>>>> its own?
> > > >>>>>>>>>>>
> > > >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us.
> > > >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > >>>>>>>>>>
> > > >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > >>>>>>>>>> You might find a Linux reference model useful at [1].
> > > >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > >>>>>>>>>>
> > > >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > >>>>>>>>>>
> > > >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the
> > > >>>>>>>>> driver to query all existing SIOV ADIs of a device,
> > > >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you
> > > >>>>>>>>> @Xuan?
> > > >>>>>>>> Could I have your plan for this?
> > > >>>>>>>>
> > > >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > >>>>>>>> as mac, ip, etc.
> > > >>>>>>> I think the query commands for SIOV is a little more complex, e.g.,
> > > >>>>>>> need to report device type and its scale(e.g., features, mq).
> > > >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood.
> > > >>>>>>>
> > > >>>>>>> We have discussed implementation a config interrupt to report new
> > > >>>>>>> created / deleted
> > > >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related
> > > >>>>>>> information,
> > > >>>>>>> my rough approach of the process is:
> > > >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id
> > > >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of
> > > >>>>>>> ADIs in a [dev_id....dev_id2] range.
> > > >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > >>>>>>
> > > >>>>>>> I am not sure whether a NIC stores its IP
> > > >>>>>> IP is the other topic. I want the Admin Queue manage the switch.
> > > >>>>>> So the switch know about the IP of every device, and the
> > > >>>>>> Admin Queue will has the ability to config the IP of the device inside the
> > > >>>>>> switch.
> > > >>>>> DPU onboard switch? OVS? Does it beyond virtio spec?
> > > >>>> YES.
> > > >>> Adding Washizu.
> > > >>>
> > > >>> We can have a switch/dpa defined in the networking device for sure.
> > > >> Yes, I think we should introduce that for the sr-iov. Or for other.
> > > > This should be a general one as a switch should be transport independent.
> > > >
> > > >> I would like to know who is doing this?
> > > > Washizu, could you confirm if you want to do this or not?
> > >
> > >
> > > Does this mean adding a switch definition to the virtio spec?
> > >
> > >
> > > If so, it will be necessary for the implementation of my plan,
> > >
> > > but it may take time (probably several months?) to get started,
> > >
> > > as I'm currently working on another task (virtio-net SR-IOV feature in
> > > qemu).
> > >
> > > Anyone is welcome to work on adding the switch definition in the meantime,
> > >
> > > it's completely fine with me.
> > >
> > > I think I'll work on that if no one has finished the work.
> >
> > Ok, great, I think we need to start this by considering reusing one of
> > the existing DPA spec. (For example, ofdpa? or any other?)
>
> We have a need on this area. I want to start this work now.
>
> Could you give me the link of the spec?

I'm not sure this is the best one, we can hear from others. I mention
it since it has an emulation code that is done in Qemu (rocker
switch).

The spec is:

https://docs.broadcom.com/doc/12378911

(not sure this is the recent one though).

Thanks

>
> Thanks
>
>
>
> >
> > Thanks
> >
> > >
> > >
> > >
> > > >
> > > >> Another question, @Jason are you referring to a new device type or a
> > > >> new virtio-net feature.
> > > > Extending virtio-net should be fine, did you see any issues for this?
> > > >
> > > >>>> For SIOV, I think this is MUST.
> > > >>> A learning bridge would be fine as a starter. It's better not to
> > > >>> couple new scalable capability with any device specific features.
> > > >>>
> > > >>>> Maybe you have one simple implementation.
> > > >>>> But you have to solve the IP steering. So admin queue should has the ability
> > > >>>> to config the IP steering.
> > > >>> I think not. Those L2/LN tables/filters are networking specific.
> > > >> Let us assume that there is a switch/bridge firstly.
> > > >> The VFs may be passed to different VMs.
> > > >>
> > > >> I also think this is the networking specific. But I want to config
> > > >> the ip for every vf from the pf.
> > > > What do you mean by ip here (e.g who is the user for this ip?)
> > > >
> > > >> Because the user of the vf may be unreliable.
> > > >> We need a manager to config the ip for every vf.
> > > > Did you mean you're using a tunnel or not?
> > > >
> > > >>
> > > >>> Control virtqueue is better than admin virtqueue here.
> > > >> by cq?
> > > >>
> > > >> What case?
> > > > We've already used control virtqueue for steering.
> > > >
> > > > Thanks
> > > >
> > > >> Thanks.
> > > >>
> > > >>> Thanks
> > > >>>
> > > >>>> Thanks.
> > > >>>>
> > > >>>>>>
> > > >>>>>> Thanks.
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>> Thanks
> > > >>>>>>>> Thanks.
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> Thanks
> > > >>>>>>>>>
> > > >>>>>>>> This publicly archived list offers a means to provide input to the
> > > >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>>>>>
> > > >>>>>>>> In order to verify user consent to the Feedback License terms and
> > > >>>>>>>> to minimize spam in the list archive, subscription is required
> > > >>>>>>>> before posting.
> > > >>>>>>>>
> > > >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>>>>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>>>>>
> > > >>>>>
> > > >>>>> This publicly archived list offers a means to provide input to the
> > > >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>>
> > > >>>>> In order to verify user consent to the Feedback License terms and
> > > >>>>> to minimize spam in the list archive, subscription is required
> > > >>>>> before posting.
> > > >>>>>
> > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>>
> > > >>>> This publicly archived list offers a means to provide input to the
> > > >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>
> > > >>>> In order to verify user consent to the Feedback License terms and
> > > >>>> to minimize spam in the list archive, subscription is required
> > > >>>> before posting.
> > > >>>>
> > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>
> > >
> >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/

Re: [virtio-comment] About the plan of Admin Queue

[Xuan Zhuo]

On Wed, 2 Aug 2023 14:53:28 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Wed, Aug 2, 2023 at 2:37 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Wed, 2 Aug 2023 14:15:55 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > On Wed, Aug 2, 2023 at 2:01 PM Yui Washizu <yui.washidu@gmail.com> wrote:
> > > >
> > > >
> > > > On 2023/07/27 17:28, Jason Wang wrote:
> > > > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > > > >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > > > >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > >>>>>
> > > > >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote:
> > > > >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote:
> > > > >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> > > > >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote:
> > > > >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis-
> > > > >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan
> > > > >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the
> > > > >>>>>>>>>>> backend?
> > > > >>>>>>>>>> Yes, why not.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>>>>> The key point is who want to create a new device.
> > > > >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make
> > > > >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs.
> > > > >>>>>>>>>>>>>
> > > > >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted.
> > > > >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host.
> > > > >>>>>>>>>> Xuan partly explained it.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I
> > > > >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU
> > > > >>>>>>>>>>>>> wants to create and hot plug in a device to a guest?
> > > > >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal
> > > > >>>>>>>>>>>>> machines?
> > > > >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU
> > > > >>>>>>>>>>>> to create new devices.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the
> > > > >>>>>>>>>>>> entire host is for users. Of course, there are situations where the
> > > > >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host
> > > > >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU.
> > > > >>>>>>>>>>>>
> > > > >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager
> > > > >>>>>>>>>>>> will plug a device to the host from the DPU.
> > > > >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC
> > > > >>>>>>>>>>> to the host.
> > > > >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means
> > > > >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be
> > > > >>>>>>>>>>> a host driver help handling an interrupt/probe.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to
> > > > >>>>>>>>>>> the host by spanning the device across the PCI hierarchy.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> To address an ADI, there is only a device_id.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure.
> > > > >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue.
> > > > >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of
> > > > >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by
> > > > >>>>>>>>>>> its own?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us.
> > > > >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries.
> > > > >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago.
> > > > >>>>>>>>>> You might find a Linux reference model useful at [1].
> > > > >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices.
> > > > >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki
> > > > >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the
> > > > >>>>>>>>> driver to query all existing SIOV ADIs of a device,
> > > > >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you
> > > > >>>>>>>>> @Xuan?
> > > > >>>>>>>> Could I have your plan for this?
> > > > >>>>>>>>
> > > > >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such
> > > > >>>>>>>> as mac, ip, etc.
> > > > >>>>>>> I think the query commands for SIOV is a little more complex, e.g.,
> > > > >>>>>>> need to report device type and its scale(e.g., features, mq).
> > > > >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood.
> > > > >>>>>>>
> > > > >>>>>>> We have discussed implementation a config interrupt to report new
> > > > >>>>>>> created / deleted
> > > > >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related
> > > > >>>>>>> information,
> > > > >>>>>>> my rough approach of the process is:
> > > > >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id
> > > > >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of
> > > > >>>>>>> ADIs in a [dev_id....dev_id2] range.
> > > > >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices.
> > > > >>>>>>
> > > > >>>>>>> I am not sure whether a NIC stores its IP
> > > > >>>>>> IP is the other topic. I want the Admin Queue manage the switch.
> > > > >>>>>> So the switch know about the IP of every device, and the
> > > > >>>>>> Admin Queue will has the ability to config the IP of the device inside the
> > > > >>>>>> switch.
> > > > >>>>> DPU onboard switch? OVS? Does it beyond virtio spec?
> > > > >>>> YES.
> > > > >>> Adding Washizu.
> > > > >>>
> > > > >>> We can have a switch/dpa defined in the networking device for sure.
> > > > >> Yes, I think we should introduce that for the sr-iov. Or for other.
> > > > > This should be a general one as a switch should be transport independent.
> > > > >
> > > > >> I would like to know who is doing this?
> > > > > Washizu, could you confirm if you want to do this or not?
> > > >
> > > >
> > > > Does this mean adding a switch definition to the virtio spec?
> > > >
> > > >
> > > > If so, it will be necessary for the implementation of my plan,
> > > >
> > > > but it may take time (probably several months?) to get started,
> > > >
> > > > as I'm currently working on another task (virtio-net SR-IOV feature in
> > > > qemu).
> > > >
> > > > Anyone is welcome to work on adding the switch definition in the meantime,
> > > >
> > > > it's completely fine with me.
> > > >
> > > > I think I'll work on that if no one has finished the work.
> > >
> > > Ok, great, I think we need to start this by considering reusing one of
> > > the existing DPA spec. (For example, ofdpa? or any other?)
> >
> > We have a need on this area. I want to start this work now.
> >
> > Could you give me the link of the spec?
>
> I'm not sure this is the best one, we can hear from others. I mention
> it since it has an emulation code that is done in Qemu (rocker
> switch).
>
> The spec is:
>
> https://docs.broadcom.com/doc/12378911

OK.

I will study it. But I think we can start with a simple model. For virtio-net,
we don't need a full-featured switch. I would mainly consider the SR-IOV case.

Thanks.

>
> (not sure this is the recent one though).
>
> Thanks
>
> >
> > Thanks
> >
> >
> >
> > >
> > > Thanks
> > >
> > > >
> > > >
> > > >
> > > > >
> > > > >> Another question, @Jason are you referring to a new device type or a
> > > > >> new virtio-net feature.
> > > > > Extending virtio-net should be fine, did you see any issues for this?
> > > > >
> > > > >>>> For SIOV, I think this is MUST.
> > > > >>> A learning bridge would be fine as a starter. It's better not to
> > > > >>> couple new scalable capability with any device specific features.
> > > > >>>
> > > > >>>> Maybe you have one simple implementation.
> > > > >>>> But you have to solve the IP steering. So admin queue should has the ability
> > > > >>>> to config the IP steering.
> > > > >>> I think not. Those L2/LN tables/filters are networking specific.
> > > > >> Let us assume that there is a switch/bridge firstly.
> > > > >> The VFs may be passed to different VMs.
> > > > >>
> > > > >> I also think this is the networking specific. But I want to config
> > > > >> the ip for every vf from the pf.
> > > > > What do you mean by ip here (e.g who is the user for this ip?)
> > > > >
> > > > >> Because the user of the vf may be unreliable.
> > > > >> We need a manager to config the ip for every vf.
> > > > > Did you mean you're using a tunnel or not?
> > > > >
> > > > >>
> > > > >>> Control virtqueue is better than admin virtqueue here.
> > > > >> by cq?
> > > > >>
> > > > >> What case?
> > > > > We've already used control virtqueue for steering.
> > > > >
> > > > > Thanks
> > > > >
> > > > >> Thanks.
> > > > >>
> > > > >>> Thanks
> > > > >>>
> > > > >>>> Thanks.
> > > > >>>>
> > > > >>>>>>
> > > > >>>>>> Thanks.
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>> Thanks
> > > > >>>>>>>> Thanks.
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>> Thanks
> > > > >>>>>>>>>
> > > > >>>>>>>> This publicly archived list offers a means to provide input to the
> > > > >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>>>>>>
> > > > >>>>>>>> In order to verify user consent to the Feedback License terms and
> > > > >>>>>>>> to minimize spam in the list archive, subscription is required
> > > > >>>>>>>> before posting.
> > > > >>>>>>>>
> > > > >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>>>>>>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>>>>>>
> > > > >>>>>
> > > > >>>>> This publicly archived list offers a means to provide input to the
> > > > >>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>>>
> > > > >>>>> In order to verify user consent to the Feedback License terms and
> > > > >>>>> to minimize spam in the list archive, subscription is required
> > > > >>>>> before posting.
> > > > >>>>>
> > > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>>>
> > > > >>>> This publicly archived list offers a means to provide input to the
> > > > >>>> OASIS Virtual I/O Device (VIRTIO) TC.
> > > > >>>>
> > > > >>>> In order to verify user consent to the Feedback License terms and
> > > > >>>> to minimize spam in the list archive, subscription is required
> > > > >>>> before posting.
> > > > >>>>
> > > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > > >>>>
> > > >
> > >
> > >
> > > This publicly archived list offers a means to provide input to the
> > > OASIS Virtual I/O Device (VIRTIO) TC.
> > >
> > > In order to verify user consent to the Feedback License terms and
> > > to minimize spam in the list archive, subscription is required
> > > before posting.
> > >
> > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > List help: virtio-comment-help@lists.oasis-open.org
> > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > Committee: https://www.oasis-open.org/committees/virtio/
> > > Join OASIS: https://www.oasis-open.org/join/
> > >
> >
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

This publicly archived list offers a means to provide input to the
OASIS Virtual I/O Device (VIRTIO) TC.

In order to verify user consent to the Feedback License terms and
to minimize spam in the list archive, subscription is required
before posting.

Subscribe: virtio-comment-subscribe@lists.oasis-open.org
Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
List help: virtio-comment-help@lists.oasis-open.org
List archive: https://lists.oasis-open.org/archives/virtio-comment/
Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
Committee: https://www.oasis-open.org/committees/virtio/
Join OASIS: https://www.oasis-open.org/join/