* [virtio-comment] About the plan of Admin Queue @ 2023-06-20 6:44 Xuan Zhuo 2023-06-20 8:11 ` Zhu, Lingshan 0 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-06-20 6:44 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: virtio-comment Hi, hi, I would want to know some plans and progress about admin queue. At the current spec, it seems that there is only one framework and no specific commands. I'd like to know if anyone is currently working on this and what the plans are. We also faced some similar issues, and we think admin queue is a good way to manage sr-iov. Thanks. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-20 6:44 [virtio-comment] About the plan of Admin Queue Xuan Zhuo @ 2023-06-20 8:11 ` Zhu, Lingshan 2023-06-30 5:54 ` Xuan Zhuo 0 siblings, 1 reply; 49+ messages in thread From: Zhu, Lingshan @ 2023-06-20 8:11 UTC (permalink / raw) To: Xuan Zhuo, Michael S. Tsirkin; +Cc: virtio-comment On 6/20/2023 2:44 PM, Xuan Zhuo wrote: > Hi, > > hi, I would want to know some plans and progress about admin queue. > > At the current spec, it seems that there is only one framework and no > specific commands. I'd like to know if anyone is currently working on this and > what the plans are. > > We also faced some similar issues, and we think admin queue is a good way to > manage sr-iov. > > > Thanks. I plan to rebase original transport vq on admin vq. https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html Thanks, Zhu Lingshan > > > > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-20 8:11 ` Zhu, Lingshan @ 2023-06-30 5:54 ` Xuan Zhuo 2023-06-30 6:19 ` Zhu, Lingshan 2023-07-03 8:10 ` Jason Wang 0 siblings, 2 replies; 49+ messages in thread From: Xuan Zhuo @ 2023-06-30 5:54 UTC (permalink / raw) To: Zhu, Lingshan; +Cc: virtio-comment, Michael S. Tsirkin On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 6/20/2023 2:44 PM, Xuan Zhuo wrote: > > Hi, > > > > hi, I would want to know some plans and progress about admin queue. > > > > At the current spec, it seems that there is only one framework and no > > specific commands. I'd like to know if anyone is currently working on this and > > what the plans are. > > > > We also faced some similar issues, and we think admin queue is a good way to > > manage sr-iov. > > > > > > Thanks. > I plan to rebase original transport vq on admin vq. > > https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html I review the patch, that is for S-IOV, right? I think it is good. I would if all is configured by the transport vq/admin vq from the OS? Can we create a managed dev from the backend? Such as, the DPU sends a command to the driver, then the driver creates a new managed dev. Thanks. > > Thanks, > Zhu Lingshan > > > > > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 5:54 ` Xuan Zhuo @ 2023-06-30 6:19 ` Zhu, Lingshan 2023-06-30 7:46 ` Xuan Zhuo 2023-07-03 8:10 ` Jason Wang 1 sibling, 1 reply; 49+ messages in thread From: Zhu, Lingshan @ 2023-06-30 6:19 UTC (permalink / raw) To: Xuan Zhuo; +Cc: virtio-comment, Michael S. Tsirkin On 6/30/2023 1:54 PM, Xuan Zhuo wrote: > On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >> >> On 6/20/2023 2:44 PM, Xuan Zhuo wrote: >>> Hi, >>> >>> hi, I would want to know some plans and progress about admin queue. >>> >>> At the current spec, it seems that there is only one framework and no >>> specific commands. I'd like to know if anyone is currently working on this and >>> what the plans are. >>> >>> We also faced some similar issues, and we think admin queue is a good way to >>> manage sr-iov. >>> >>> >>> Thanks. >> I plan to rebase original transport vq on admin vq. >> >> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html > > I review the patch, that is for S-IOV, right? Yes, for SIOV and other similar devices > > I think it is good. > > I would if all is configured by the transport vq/admin vq from the OS? For SIOV ADIs, this transport vq is the transport layer, so they are configured by the OS through transport vq. > > Can we create a managed dev from the backend? > > Such as, the DPU sends a command to the driver, then the driver creates a new > managed dev. I think the group owner, usually the PCI PF is the management device. Thanks > > Thanks. > > >> Thanks, >> Zhu Lingshan >>> >>> >>> >>> This publicly archived list offers a means to provide input to the >>> OASIS Virtual I/O Device (VIRTIO) TC. >>> >>> In order to verify user consent to the Feedback License terms and >>> to minimize spam in the list archive, subscription is required >>> before posting. >>> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>> List help: virtio-comment-help@lists.oasis-open.org >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>> Committee: https://www.oasis-open.org/committees/virtio/ >>> Join OASIS: https://www.oasis-open.org/join/ >>> >> >> This publicly archived list offers a means to provide input to the >> OASIS Virtual I/O Device (VIRTIO) TC. >> >> In order to verify user consent to the Feedback License terms and >> to minimize spam in the list archive, subscription is required >> before posting. >> >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >> List help: virtio-comment-help@lists.oasis-open.org >> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >> Committee: https://www.oasis-open.org/committees/virtio/ >> Join OASIS: https://www.oasis-open.org/join/ >> > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 6:19 ` Zhu, Lingshan @ 2023-06-30 7:46 ` Xuan Zhuo 2023-06-30 7:54 ` Zhu, Lingshan 0 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-06-30 7:46 UTC (permalink / raw) To: Zhu, Lingshan; +Cc: virtio-comment, Michael S. Tsirkin On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 6/30/2023 1:54 PM, Xuan Zhuo wrote: > > On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >> > >> On 6/20/2023 2:44 PM, Xuan Zhuo wrote: > >>> Hi, > >>> > >>> hi, I would want to know some plans and progress about admin queue. > >>> > >>> At the current spec, it seems that there is only one framework and no > >>> specific commands. I'd like to know if anyone is currently working on this and > >>> what the plans are. > >>> > >>> We also faced some similar issues, and we think admin queue is a good way to > >>> manage sr-iov. > >>> > >>> > >>> Thanks. > >> I plan to rebase original transport vq on admin vq. > >> > >> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html > > > > I review the patch, that is for S-IOV, right? > Yes, for SIOV and other similar devices > > > > I think it is good. > > > > I would if all is configured by the transport vq/admin vq from the OS? > For SIOV ADIs, this transport vq is the transport layer, so they are > configured by the OS through transport vq. > > > > Can we create a managed dev from the backend? > > > > Such as, the DPU sends a command to the driver, then the driver creates a new > > managed dev. > I think the group owner, usually the PCI PF is the management device. I mean the DPU hot-plug a new device. Not the managament device create a new device. The managament device is in the OS, we want the device is plugged by the DPU. Thanks. > > Thanks > > > > Thanks. > > > > > >> Thanks, > >> Zhu Lingshan > >>> > >>> > >>> > >>> This publicly archived list offers a means to provide input to the > >>> OASIS Virtual I/O Device (VIRTIO) TC. > >>> > >>> In order to verify user consent to the Feedback License terms and > >>> to minimize spam in the list archive, subscription is required > >>> before posting. > >>> > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>> List help: virtio-comment-help@lists.oasis-open.org > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>> Committee: https://www.oasis-open.org/committees/virtio/ > >>> Join OASIS: https://www.oasis-open.org/join/ > >>> > >> > >> This publicly archived list offers a means to provide input to the > >> OASIS Virtual I/O Device (VIRTIO) TC. > >> > >> In order to verify user consent to the Feedback License terms and > >> to minimize spam in the list archive, subscription is required > >> before posting. > >> > >> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >> List help: virtio-comment-help@lists.oasis-open.org > >> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >> Committee: https://www.oasis-open.org/committees/virtio/ > >> Join OASIS: https://www.oasis-open.org/join/ > >> > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 7:46 ` Xuan Zhuo @ 2023-06-30 7:54 ` Zhu, Lingshan 2023-06-30 7:56 ` Xuan Zhuo 0 siblings, 1 reply; 49+ messages in thread From: Zhu, Lingshan @ 2023-06-30 7:54 UTC (permalink / raw) To: Xuan Zhuo; +Cc: virtio-comment, Michael S. Tsirkin On 6/30/2023 3:46 PM, Xuan Zhuo wrote: > On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >> >> On 6/30/2023 1:54 PM, Xuan Zhuo wrote: >>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote: >>>>> Hi, >>>>> >>>>> hi, I would want to know some plans and progress about admin queue. >>>>> >>>>> At the current spec, it seems that there is only one framework and no >>>>> specific commands. I'd like to know if anyone is currently working on this and >>>>> what the plans are. >>>>> >>>>> We also faced some similar issues, and we think admin queue is a good way to >>>>> manage sr-iov. >>>>> >>>>> >>>>> Thanks. >>>> I plan to rebase original transport vq on admin vq. >>>> >>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html >>> I review the patch, that is for S-IOV, right? >> Yes, for SIOV and other similar devices >>> I think it is good. >>> >>> I would if all is configured by the transport vq/admin vq from the OS? >> For SIOV ADIs, this transport vq is the transport layer, so they are >> configured by the OS through transport vq. >>> Can we create a managed dev from the backend? >>> >>> Such as, the DPU sends a command to the driver, then the driver creates a new >>> managed dev. >> I think the group owner, usually the PCI PF is the management device. > > I mean the DPU hot-plug a new device. Not the managament device create a new > device. > > The managament device is in the OS, we want the device is plugged by the DPU. The PCI management(SIOV group owner) device is on the DPU, when create an ADI, OS send a command to the DPU/PF through transport vq, then the PF hot plugged in a new ADI through the transport vq specific channel. Or did I misunderstand your question? Thanks > > Thanks. > > >> Thanks >>> Thanks. >>> >>> >>>> Thanks, >>>> Zhu Lingshan >>>>> >>>>> >>>>> This publicly archived list offers a means to provide input to the >>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>> >>>>> In order to verify user consent to the Feedback License terms and >>>>> to minimize spam in the list archive, subscription is required >>>>> before posting. >>>>> >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>> >>>> This publicly archived list offers a means to provide input to the >>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>> >>>> In order to verify user consent to the Feedback License terms and >>>> to minimize spam in the list archive, subscription is required >>>> before posting. >>>> >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>> List help: virtio-comment-help@lists.oasis-open.org >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>> Join OASIS: https://www.oasis-open.org/join/ >>>> >>> This publicly archived list offers a means to provide input to the >>> OASIS Virtual I/O Device (VIRTIO) TC. >>> >>> In order to verify user consent to the Feedback License terms and >>> to minimize spam in the list archive, subscription is required >>> before posting. >>> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>> List help: virtio-comment-help@lists.oasis-open.org >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>> Committee: https://www.oasis-open.org/committees/virtio/ >>> Join OASIS: https://www.oasis-open.org/join/ >>> This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 7:54 ` Zhu, Lingshan @ 2023-06-30 7:56 ` Xuan Zhuo 2023-06-30 8:32 ` Zhu, Lingshan 0 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-06-30 7:56 UTC (permalink / raw) To: Zhu, Lingshan; +Cc: virtio-comment, Michael S. Tsirkin On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 6/30/2023 3:46 PM, Xuan Zhuo wrote: > > On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >> > >> On 6/30/2023 1:54 PM, Xuan Zhuo wrote: > >>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote: > >>>>> Hi, > >>>>> > >>>>> hi, I would want to know some plans and progress about admin queue. > >>>>> > >>>>> At the current spec, it seems that there is only one framework and no > >>>>> specific commands. I'd like to know if anyone is currently working on this and > >>>>> what the plans are. > >>>>> > >>>>> We also faced some similar issues, and we think admin queue is a good way to > >>>>> manage sr-iov. > >>>>> > >>>>> > >>>>> Thanks. > >>>> I plan to rebase original transport vq on admin vq. > >>>> > >>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html > >>> I review the patch, that is for S-IOV, right? > >> Yes, for SIOV and other similar devices > >>> I think it is good. > >>> > >>> I would if all is configured by the transport vq/admin vq from the OS? > >> For SIOV ADIs, this transport vq is the transport layer, so they are > >> configured by the OS through transport vq. > >>> Can we create a managed dev from the backend? > >>> > >>> Such as, the DPU sends a command to the driver, then the driver creates a new > >>> managed dev. > >> I think the group owner, usually the PCI PF is the management device. > > > > I mean the DPU hot-plug a new device. Not the managament device create a new > > device. > > > > The managament device is in the OS, we want the device is plugged by the DPU. > The PCI management(SIOV group owner) device is on the DPU, when create > an ADI, > OS send a command to the DPU/PF through transport vq, > then the PF hot plugged in a new ADI through the transport vq > specific channel. Or did I misunderstand your question? Your first step is the os send a command. Right? Can we let the DPU notify the driver to create a new devicer from the backend? The key point is who want to create a new device. Thanks. > > Thanks > > > > Thanks. > > > > > >> Thanks > >>> Thanks. > >>> > >>> > >>>> Thanks, > >>>> Zhu Lingshan > >>>>> > >>>>> > >>>>> This publicly archived list offers a means to provide input to the > >>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>> > >>>>> In order to verify user consent to the Feedback License terms and > >>>>> to minimize spam in the list archive, subscription is required > >>>>> before posting. > >>>>> > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>> > >>>> This publicly archived list offers a means to provide input to the > >>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>> > >>>> In order to verify user consent to the Feedback License terms and > >>>> to minimize spam in the list archive, subscription is required > >>>> before posting. > >>>> > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>> List help: virtio-comment-help@lists.oasis-open.org > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>> Join OASIS: https://www.oasis-open.org/join/ > >>>> > >>> This publicly archived list offers a means to provide input to the > >>> OASIS Virtual I/O Device (VIRTIO) TC. > >>> > >>> In order to verify user consent to the Feedback License terms and > >>> to minimize spam in the list archive, subscription is required > >>> before posting. > >>> > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>> List help: virtio-comment-help@lists.oasis-open.org > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>> Committee: https://www.oasis-open.org/committees/virtio/ > >>> Join OASIS: https://www.oasis-open.org/join/ > >>> > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 7:56 ` Xuan Zhuo @ 2023-06-30 8:32 ` Zhu, Lingshan 2023-06-30 9:07 ` Zhu, Lingshan 2023-06-30 9:14 ` Xuan Zhuo 0 siblings, 2 replies; 49+ messages in thread From: Zhu, Lingshan @ 2023-06-30 8:32 UTC (permalink / raw) To: Xuan Zhuo; +Cc: virtio-comment, Michael S. Tsirkin On 6/30/2023 3:56 PM, Xuan Zhuo wrote: > On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >> >> On 6/30/2023 3:46 PM, Xuan Zhuo wrote: >>> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >>>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote: >>>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >>>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote: >>>>>>> Hi, >>>>>>> >>>>>>> hi, I would want to know some plans and progress about admin queue. >>>>>>> >>>>>>> At the current spec, it seems that there is only one framework and no >>>>>>> specific commands. I'd like to know if anyone is currently working on this and >>>>>>> what the plans are. >>>>>>> >>>>>>> We also faced some similar issues, and we think admin queue is a good way to >>>>>>> manage sr-iov. >>>>>>> >>>>>>> >>>>>>> Thanks. >>>>>> I plan to rebase original transport vq on admin vq. >>>>>> >>>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html >>>>> I review the patch, that is for S-IOV, right? >>>> Yes, for SIOV and other similar devices >>>>> I think it is good. >>>>> >>>>> I would if all is configured by the transport vq/admin vq from the OS? >>>> For SIOV ADIs, this transport vq is the transport layer, so they are >>>> configured by the OS through transport vq. >>>>> Can we create a managed dev from the backend? >>>>> >>>>> Such as, the DPU sends a command to the driver, then the driver creates a new >>>>> managed dev. >>>> I think the group owner, usually the PCI PF is the management device. >>> I mean the DPU hot-plug a new device. Not the managament device create a new >>> device. >>> >>> The managament device is in the OS, we want the device is plugged by the DPU. >> The PCI management(SIOV group owner) device is on the DPU, when create >> an ADI, >> OS send a command to the DPU/PF through transport vq, >> then the PF hot plugged in a new ADI through the transport vq >> specific channel. Or did I misunderstand your question? > Your first step is the os send a command. Right? > > Can we let the DPU notify the driver to create a new devicer from the backend? > > The key point is who want to create a new device. DPU can come with a certain number of pre-created ADIs, just make sure the orchestration SW is aware of their device IDs. If you want the DPU randomly create ADIs and notify the driver, I think we need interrupt, e.g., re-use config interrupt. But why DPU wants to create and hot plug in a device to a guest? Shall the host handle that or DPU pre-create then expose to baremteal machines? Thanks > > Thanks. > > > > >> Thanks >>> Thanks. >>> >>> >>>> Thanks >>>>> Thanks. >>>>> >>>>> >>>>>> Thanks, >>>>>> Zhu Lingshan >>>>>>> >>>>>>> This publicly archived list offers a means to provide input to the >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>>>> >>>>>>> In order to verify user consent to the Feedback License terms and >>>>>>> to minimize spam in the list archive, subscription is required >>>>>>> before posting. >>>>>>> >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>>>> >>>>>> This publicly archived list offers a means to provide input to the >>>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>>> >>>>>> In order to verify user consent to the Feedback License terms and >>>>>> to minimize spam in the list archive, subscription is required >>>>>> before posting. >>>>>> >>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>>> >>>>> This publicly archived list offers a means to provide input to the >>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>> >>>>> In order to verify user consent to the Feedback License terms and >>>>> to minimize spam in the list archive, subscription is required >>>>> before posting. >>>>> >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>> This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 8:32 ` Zhu, Lingshan @ 2023-06-30 9:07 ` Zhu, Lingshan 2023-06-30 9:14 ` Xuan Zhuo 1 sibling, 0 replies; 49+ messages in thread From: Zhu, Lingshan @ 2023-06-30 9:07 UTC (permalink / raw) To: Xuan Zhuo; +Cc: virtio-comment, Michael S. Tsirkin On 6/30/2023 4:32 PM, Zhu, Lingshan wrote: > > > On 6/30/2023 3:56 PM, Xuan Zhuo wrote: >> On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" >> <lingshan.zhu@intel.com> wrote: >>> >>> On 6/30/2023 3:46 PM, Xuan Zhuo wrote: >>>> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" >>>> <lingshan.zhu@intel.com> wrote: >>>>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote: >>>>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" >>>>>> <lingshan.zhu@intel.com> wrote: >>>>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> hi, I would want to know some plans and progress about admin >>>>>>>> queue. >>>>>>>> >>>>>>>> At the current spec, it seems that there is only one framework >>>>>>>> and no >>>>>>>> specific commands. I'd like to know if anyone is currently >>>>>>>> working on this and >>>>>>>> what the plans are. >>>>>>>> >>>>>>>> We also faced some similar issues, and we think admin queue is >>>>>>>> a good way to >>>>>>>> manage sr-iov. >>>>>>>> >>>>>>>> >>>>>>>> Thanks. >>>>>>> I plan to rebase original transport vq on admin vq. >>>>>>> >>>>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html >>>>>>> >>>>>> I review the patch, that is for S-IOV, right? >>>>> Yes, for SIOV and other similar devices >>>>>> I think it is good. >>>>>> >>>>>> I would if all is configured by the transport vq/admin vq from >>>>>> the OS? >>>>> For SIOV ADIs, this transport vq is the transport layer, so they are >>>>> configured by the OS through transport vq. >>>>>> Can we create a managed dev from the backend? >>>>>> >>>>>> Such as, the DPU sends a command to the driver, then the driver >>>>>> creates a new >>>>>> managed dev. >>>>> I think the group owner, usually the PCI PF is the management device. >>>> I mean the DPU hot-plug a new device. Not the managament device >>>> create a new >>>> device. >>>> >>>> The managament device is in the OS, we want the device is plugged >>>> by the DPU. >>> The PCI management(SIOV group owner) device is on the DPU, when create >>> an ADI, >>> OS send a command to the DPU/PF through transport vq, >>> then the PF hot plugged in a new ADI through the transport vq >>> specific channel. Or did I misunderstand your question? >> Your first step is the os send a command. Right? >> >> Can we let the DPU notify the driver to create a new devicer from the >> backend? >> >> The key point is who want to create a new device. > DPU can come with a certain number of pre-created ADIs, just make sure > the > orchestration SW is aware of their device IDs. > > If you want the DPU randomly create ADIs and notify the driver, I think > we need interrupt, e.g., re-use config interrupt. But why DPU > wants to create and hot plug in a device to a guest? > Shall the host handle that or DPU pre-create then expose to baremteal > machines? > > Thanks And how about this, I can try provide a new command can help query the onboard ADIs, report the IDs, then the driver can probe them by the IDs and read their config space to detect their config like num_vqs Thanks >> >> Thanks. >> >> >> >> >>> Thanks >>>> Thanks. >>>> >>>> >>>>> Thanks >>>>>> Thanks. >>>>>> >>>>>> >>>>>>> Thanks, >>>>>>> Zhu Lingshan >>>>>>>> >>>>>>>> This publicly archived list offers a means to provide input to the >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>>>>> >>>>>>>> In order to verify user consent to the Feedback License terms and >>>>>>>> to minimize spam in the list archive, subscription is required >>>>>>>> before posting. >>>>>>>> >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>>>>> List archive: >>>>>>>> https://lists.oasis-open.org/archives/virtio-comment/ >>>>>>>> Feedback License: >>>>>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>>>>> List Guidelines: >>>>>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>>>>> >>>>>>> This publicly archived list offers a means to provide input to the >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>>>> >>>>>>> In order to verify user consent to the Feedback License terms and >>>>>>> to minimize spam in the list archive, subscription is required >>>>>>> before posting. >>>>>>> >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>>>> Feedback License: >>>>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>>>> List Guidelines: >>>>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>>>> >>>>>> This publicly archived list offers a means to provide input to the >>>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>>> >>>>>> In order to verify user consent to the Feedback License terms and >>>>>> to minimize spam in the list archive, subscription is required >>>>>> before posting. >>>>>> >>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>>> Feedback License: >>>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>>> List Guidelines: >>>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>>> > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 8:32 ` Zhu, Lingshan 2023-06-30 9:07 ` Zhu, Lingshan @ 2023-06-30 9:14 ` Xuan Zhuo 2023-06-30 10:33 ` Zhu, Lingshan 1 sibling, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-06-30 9:14 UTC (permalink / raw) To: Zhu, Lingshan; +Cc: virtio-comment, Michael S. Tsirkin On Fri, 30 Jun 2023 16:32:44 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 6/30/2023 3:56 PM, Xuan Zhuo wrote: > > On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >> > >> On 6/30/2023 3:46 PM, Xuan Zhuo wrote: > >>> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote: > >>>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote: > >>>>>>> Hi, > >>>>>>> > >>>>>>> hi, I would want to know some plans and progress about admin queue. > >>>>>>> > >>>>>>> At the current spec, it seems that there is only one framework and no > >>>>>>> specific commands. I'd like to know if anyone is currently working on this and > >>>>>>> what the plans are. > >>>>>>> > >>>>>>> We also faced some similar issues, and we think admin queue is a good way to > >>>>>>> manage sr-iov. > >>>>>>> > >>>>>>> > >>>>>>> Thanks. > >>>>>> I plan to rebase original transport vq on admin vq. > >>>>>> > >>>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html > >>>>> I review the patch, that is for S-IOV, right? > >>>> Yes, for SIOV and other similar devices > >>>>> I think it is good. > >>>>> > >>>>> I would if all is configured by the transport vq/admin vq from the OS? > >>>> For SIOV ADIs, this transport vq is the transport layer, so they are > >>>> configured by the OS through transport vq. > >>>>> Can we create a managed dev from the backend? > >>>>> > >>>>> Such as, the DPU sends a command to the driver, then the driver creates a new > >>>>> managed dev. > >>>> I think the group owner, usually the PCI PF is the management device. > >>> I mean the DPU hot-plug a new device. Not the managament device create a new > >>> device. > >>> > >>> The managament device is in the OS, we want the device is plugged by the DPU. > >> The PCI management(SIOV group owner) device is on the DPU, when create > >> an ADI, > >> OS send a command to the DPU/PF through transport vq, > >> then the PF hot plugged in a new ADI through the transport vq > >> specific channel. Or did I misunderstand your question? > > Your first step is the os send a command. Right? > > > > Can we let the DPU notify the driver to create a new devicer from the backend? > > > > The key point is who want to create a new device. > DPU can come with a certain number of pre-created ADIs, just make sure the > orchestration SW is aware of their device IDs. > > If you want the DPU randomly create ADIs and notify the driver, I think > we need interrupt, e.g., re-use config interrupt. But why DPU > wants to create and hot plug in a device to a guest? > Shall the host handle that or DPU pre-create then expose to baremteal > machines? In your scenario, the supervisor is on the os, which controls the DPU to create new devices. In the cloud scenario, the vendor manager is in the DPU, and the entire host is for users. Of course, there are situations where the vendor manager are in the HOST. But for bare metal machines, the host belongs to the customer, the vendor manager is only in the DPU. So when the customers buy a new nic for the host, the vendor manager will plug a device to the host from the DPU. Thanks. > > Thanks > > > > Thanks. > > > > > > > > > >> Thanks > >>> Thanks. > >>> > >>> > >>>> Thanks > >>>>> Thanks. > >>>>> > >>>>> > >>>>>> Thanks, > >>>>>> Zhu Lingshan > >>>>>>> > >>>>>>> This publicly archived list offers a means to provide input to the > >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>>>> > >>>>>>> In order to verify user consent to the Feedback License terms and > >>>>>>> to minimize spam in the list archive, subscription is required > >>>>>>> before posting. > >>>>>>> > >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>>>> > >>>>>> This publicly archived list offers a means to provide input to the > >>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>>> > >>>>>> In order to verify user consent to the Feedback License terms and > >>>>>> to minimize spam in the list archive, subscription is required > >>>>>> before posting. > >>>>>> > >>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>>> > >>>>> This publicly archived list offers a means to provide input to the > >>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>> > >>>>> In order to verify user consent to the Feedback License terms and > >>>>> to minimize spam in the list archive, subscription is required > >>>>> before posting. > >>>>> > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>> > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 9:14 ` Xuan Zhuo @ 2023-06-30 10:33 ` Zhu, Lingshan 2023-06-30 11:35 ` Parav Pandit 0 siblings, 1 reply; 49+ messages in thread From: Zhu, Lingshan @ 2023-06-30 10:33 UTC (permalink / raw) To: Xuan Zhuo; +Cc: virtio-comment, Michael S. Tsirkin On 6/30/2023 5:14 PM, Xuan Zhuo wrote: > On Fri, 30 Jun 2023 16:32:44 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >> >> On 6/30/2023 3:56 PM, Xuan Zhuo wrote: >>> On Fri, 30 Jun 2023 15:54:40 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >>>> On 6/30/2023 3:46 PM, Xuan Zhuo wrote: >>>>> On Fri, 30 Jun 2023 14:19:47 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >>>>>> On 6/30/2023 1:54 PM, Xuan Zhuo wrote: >>>>>>> On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >>>>>>>> On 6/20/2023 2:44 PM, Xuan Zhuo wrote: >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> hi, I would want to know some plans and progress about admin queue. >>>>>>>>> >>>>>>>>> At the current spec, it seems that there is only one framework and no >>>>>>>>> specific commands. I'd like to know if anyone is currently working on this and >>>>>>>>> what the plans are. >>>>>>>>> >>>>>>>>> We also faced some similar issues, and we think admin queue is a good way to >>>>>>>>> manage sr-iov. >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks. >>>>>>>> I plan to rebase original transport vq on admin vq. >>>>>>>> >>>>>>>> https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html >>>>>>> I review the patch, that is for S-IOV, right? >>>>>> Yes, for SIOV and other similar devices >>>>>>> I think it is good. >>>>>>> >>>>>>> I would if all is configured by the transport vq/admin vq from the OS? >>>>>> For SIOV ADIs, this transport vq is the transport layer, so they are >>>>>> configured by the OS through transport vq. >>>>>>> Can we create a managed dev from the backend? >>>>>>> >>>>>>> Such as, the DPU sends a command to the driver, then the driver creates a new >>>>>>> managed dev. >>>>>> I think the group owner, usually the PCI PF is the management device. >>>>> I mean the DPU hot-plug a new device. Not the managament device create a new >>>>> device. >>>>> >>>>> The managament device is in the OS, we want the device is plugged by the DPU. >>>> The PCI management(SIOV group owner) device is on the DPU, when create >>>> an ADI, >>>> OS send a command to the DPU/PF through transport vq, >>>> then the PF hot plugged in a new ADI through the transport vq >>>> specific channel. Or did I misunderstand your question? >>> Your first step is the os send a command. Right? >>> >>> Can we let the DPU notify the driver to create a new devicer from the backend? >>> >>> The key point is who want to create a new device. >> DPU can come with a certain number of pre-created ADIs, just make sure the >> orchestration SW is aware of their device IDs. >> >> If you want the DPU randomly create ADIs and notify the driver, I think >> we need interrupt, e.g., re-use config interrupt. But why DPU >> wants to create and hot plug in a device to a guest? >> Shall the host handle that or DPU pre-create then expose to baremteal >> machines? > > In your scenario, the supervisor is on the os, which controls the DPU to create > new devices. > > In the cloud scenario, the vendor manager is in the DPU, and the entire > host is for users. Of course, there are situations where the vendor manager are > in the HOST. But for bare metal machines, the host belongs to the customer, the > vendor manager is only in the DPU. > > So when the customers buy a new nic for the host, the vendor manager will plug a > device to the host from the DPU. I understand once a customer orders a new NIC, you wants to present the NIC to the host. However you only owns the DPU and the customer owns the host, that means this creation and hot plug must be transparent to the host and there may not be a host driver help handling an interrupt/probe. However this is not PCI which has a tree/switch and can enumerate devices to the host by spanning the device across the PCI hierarchy. To address an ADI, there is only a device_id. So, do you mind share how your DPU offload the device model? What kind of device your DPU provide to the host? Lets see whether DPU can mediate this by its own? > > Thanks. > > >> Thanks >>> Thanks. >>> >>> >>> >>> >>>> Thanks >>>>> Thanks. >>>>> >>>>> >>>>>> Thanks >>>>>>> Thanks. >>>>>>> >>>>>>> >>>>>>>> Thanks, >>>>>>>> Zhu Lingshan >>>>>>>>> This publicly archived list offers a means to provide input to the >>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>>>>>> >>>>>>>>> In order to verify user consent to the Feedback License terms and >>>>>>>>> to minimize spam in the list archive, subscription is required >>>>>>>>> before posting. >>>>>>>>> >>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>>>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>>>>>> >>>>>>>> This publicly archived list offers a means to provide input to the >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>>>>> >>>>>>>> In order to verify user consent to the Feedback License terms and >>>>>>>> to minimize spam in the list archive, subscription is required >>>>>>>> before posting. >>>>>>>> >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>>>>> >>>>>>> This publicly archived list offers a means to provide input to the >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. >>>>>>> >>>>>>> In order to verify user consent to the Feedback License terms and >>>>>>> to minimize spam in the list archive, subscription is required >>>>>>> before posting. >>>>>>> >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>>>>>> List help: virtio-comment-help@lists.oasis-open.org >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ >>>>>>> Join OASIS: https://www.oasis-open.org/join/ >>>>>>> This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* RE: [virtio-comment] About the plan of Admin Queue 2023-06-30 10:33 ` Zhu, Lingshan @ 2023-06-30 11:35 ` Parav Pandit 2023-07-03 4:29 ` Zhu, Lingshan 0 siblings, 1 reply; 49+ messages in thread From: Parav Pandit @ 2023-06-30 11:35 UTC (permalink / raw) To: Zhu, Lingshan, Xuan Zhuo Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin > From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > open.org> On Behalf Of Zhu, Lingshan > Sent: Friday, June 30, 2023 6:33 AM > > >>> Can we let the DPU notify the driver to create a new devicer from the > backend? Yes, why not. > >>> > >>> The key point is who want to create a new device. > >> DPU can come with a certain number of pre-created ADIs, just make > >> sure the orchestration SW is aware of their device IDs. > >> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. Xuan partly explained it. > >> If you want the DPU randomly create ADIs and notify the driver, I > >> think we need interrupt, e.g., re-use config interrupt. But why DPU > >> wants to create and hot plug in a device to a guest? > >> Shall the host handle that or DPU pre-create then expose to baremteal > >> machines? > > > > In your scenario, the supervisor is on the os, which controls the DPU > > to create new devices. > > > > In the cloud scenario, the vendor manager is in the DPU, and the > > entire host is for users. Of course, there are situations where the > > vendor manager are in the HOST. But for bare metal machines, the host > > belongs to the customer, the vendor manager is only in the DPU. > > > > So when the customers buy a new nic for the host, the vendor manager > > will plug a device to the host from the DPU. > I understand once a customer orders a new NIC, you wants to present the NIC > to the host. > However you only owns the DPU and the customer owns the host, that means > this creation and hot plug must be transparent to the host and there may not be > a host driver help handling an interrupt/probe. > That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > However this is not PCI which has a tree/switch and can enumerate devices to > the host by spanning the device across the PCI hierarchy. > Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > To address an ADI, there is only a device_id. > SIOV device must have a unique identifier at PCI bus level for sure. I cannot speak more about it in this forum due to other logistics issue. But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > So, do you mind share how your DPU offload the device model? What kind of > device your DPU provide to the host? Lets see whether DPU can mediate this by > its own? > It is a virtio nic, blk and other virtio devices for us. A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. You might find a Linux reference model useful at [1]. A usage model already exists in one OS and in use for non virtio devices. This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. [1] https://github.com/Mellanox/scalablefunctions/wiki ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 11:35 ` Parav Pandit @ 2023-07-03 4:29 ` Zhu, Lingshan 2023-07-03 5:54 ` Xuan Zhuo 2023-07-27 2:30 ` Xuan Zhuo 0 siblings, 2 replies; 49+ messages in thread From: Zhu, Lingshan @ 2023-07-03 4:29 UTC (permalink / raw) To: Parav Pandit, Xuan Zhuo Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin On 6/30/2023 7:35 PM, Parav Pandit wrote: >> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- >> open.org> On Behalf Of Zhu, Lingshan >> Sent: Friday, June 30, 2023 6:33 AM >> > >>>>> Can we let the DPU notify the driver to create a new devicer from the >> backend? > Yes, why not. > >>>>> The key point is who want to create a new device. >>>> DPU can come with a certain number of pre-created ADIs, just make >>>> sure the orchestration SW is aware of their device IDs. >>>> > Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > Xuan partly explained it. > >>>> If you want the DPU randomly create ADIs and notify the driver, I >>>> think we need interrupt, e.g., re-use config interrupt. But why DPU >>>> wants to create and hot plug in a device to a guest? >>>> Shall the host handle that or DPU pre-create then expose to baremteal >>>> machines? >>> In your scenario, the supervisor is on the os, which controls the DPU >>> to create new devices. >>> >>> In the cloud scenario, the vendor manager is in the DPU, and the >>> entire host is for users. Of course, there are situations where the >>> vendor manager are in the HOST. But for bare metal machines, the host >>> belongs to the customer, the vendor manager is only in the DPU. >>> >>> So when the customers buy a new nic for the host, the vendor manager >>> will plug a device to the host from the DPU. >> I understand once a customer orders a new NIC, you wants to present the NIC >> to the host. >> However you only owns the DPU and the customer owns the host, that means >> this creation and hot plug must be transparent to the host and there may not be >> a host driver help handling an interrupt/probe. >> > That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > >> However this is not PCI which has a tree/switch and can enumerate devices to >> the host by spanning the device across the PCI hierarchy. >> > Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > >> To address an ADI, there is only a device_id. >> > SIOV device must have a unique identifier at PCI bus level for sure. > I cannot speak more about it in this forum due to other logistics issue. > But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > >> So, do you mind share how your DPU offload the device model? What kind of >> device your DPU provide to the host? Lets see whether DPU can mediate this by >> its own? >> > It is a virtio nic, blk and other virtio devices for us. > A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > You might find a Linux reference model useful at [1]. > A usage model already exists in one OS and in use for non virtio devices. > This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > [1] https://github.com/Mellanox/scalablefunctions/wiki well I see SF facing the similar challenge, I can add a command for the driver to query all existing SIOV ADIs of a device, and the device return ADIs id and status. Looks good? and work for you @Xuan? Thanks This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 4:29 ` Zhu, Lingshan @ 2023-07-03 5:54 ` Xuan Zhuo 2023-07-03 8:01 ` Zhu, Lingshan 2023-07-27 2:30 ` Xuan Zhuo 1 sibling, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-03 5:54 UTC (permalink / raw) To: Zhu, Lingshan Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 6/30/2023 7:35 PM, Parav Pandit wrote: > >> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > >> open.org> On Behalf Of Zhu, Lingshan > >> Sent: Friday, June 30, 2023 6:33 AM > >> > > > >>>>> Can we let the DPU notify the driver to create a new devicer from the > >> backend? > > Yes, why not. > > > >>>>> The key point is who want to create a new device. > >>>> DPU can come with a certain number of pre-created ADIs, just make > >>>> sure the orchestration SW is aware of their device IDs. > >>>> > > Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > Xuan partly explained it. > > > >>>> If you want the DPU randomly create ADIs and notify the driver, I > >>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > >>>> wants to create and hot plug in a device to a guest? > >>>> Shall the host handle that or DPU pre-create then expose to baremteal > >>>> machines? > >>> In your scenario, the supervisor is on the os, which controls the DPU > >>> to create new devices. > >>> > >>> In the cloud scenario, the vendor manager is in the DPU, and the > >>> entire host is for users. Of course, there are situations where the > >>> vendor manager are in the HOST. But for bare metal machines, the host > >>> belongs to the customer, the vendor manager is only in the DPU. > >>> > >>> So when the customers buy a new nic for the host, the vendor manager > >>> will plug a device to the host from the DPU. > >> I understand once a customer orders a new NIC, you wants to present the NIC > >> to the host. > >> However you only owns the DPU and the customer owns the host, that means > >> this creation and hot plug must be transparent to the host and there may not be > >> a host driver help handling an interrupt/probe. > >> > > That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > >> However this is not PCI which has a tree/switch and can enumerate devices to > >> the host by spanning the device across the PCI hierarchy. > >> > > Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > >> To address an ADI, there is only a device_id. > >> > > SIOV device must have a unique identifier at PCI bus level for sure. > > I cannot speak more about it in this forum due to other logistics issue. > > But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > >> So, do you mind share how your DPU offload the device model? What kind of > >> device your DPU provide to the host? Lets see whether DPU can mediate this by > >> its own? > >> > > It is a virtio nic, blk and other virtio devices for us. > > A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > You might find a Linux reference model useful at [1]. > > A usage model already exists in one OS and in use for non virtio devices. > > This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > [1] https://github.com/Mellanox/scalablefunctions/wiki > well I see SF facing the similar challenge, I can add a command for the > driver to query all existing SIOV ADIs of a device, > and the device return ADIs id and status. Looks good? and work for you > @Xuan? Can the device notifies the driver by a interrupt? Thanks. > > Thanks > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 5:54 ` Xuan Zhuo @ 2023-07-03 8:01 ` Zhu, Lingshan 2023-07-03 8:21 ` Xuan Zhuo 0 siblings, 1 reply; 49+ messages in thread From: Zhu, Lingshan @ 2023-07-03 8:01 UTC (permalink / raw) To: Xuan Zhuo Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On 7/3/2023 1:54 PM, Xuan Zhuo wrote: > On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >> >> On 6/30/2023 7:35 PM, Parav Pandit wrote: >>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- >>>> open.org> On Behalf Of Zhu, Lingshan >>>> Sent: Friday, June 30, 2023 6:33 AM >>>> >>>>>>> Can we let the DPU notify the driver to create a new devicer from the >>>> backend? >>> Yes, why not. >>> >>>>>>> The key point is who want to create a new device. >>>>>> DPU can come with a certain number of pre-created ADIs, just make >>>>>> sure the orchestration SW is aware of their device IDs. >>>>>> >>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. >>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. >>> Xuan partly explained it. >>> >>>>>> If you want the DPU randomly create ADIs and notify the driver, I >>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU >>>>>> wants to create and hot plug in a device to a guest? >>>>>> Shall the host handle that or DPU pre-create then expose to baremteal >>>>>> machines? >>>>> In your scenario, the supervisor is on the os, which controls the DPU >>>>> to create new devices. >>>>> >>>>> In the cloud scenario, the vendor manager is in the DPU, and the >>>>> entire host is for users. Of course, there are situations where the >>>>> vendor manager are in the HOST. But for bare metal machines, the host >>>>> belongs to the customer, the vendor manager is only in the DPU. >>>>> >>>>> So when the customers buy a new nic for the host, the vendor manager >>>>> will plug a device to the host from the DPU. >>>> I understand once a customer orders a new NIC, you wants to present the NIC >>>> to the host. >>>> However you only owns the DPU and the customer owns the host, that means >>>> this creation and hot plug must be transparent to the host and there may not be >>>> a host driver help handling an interrupt/probe. >>>> >>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. >>> >>>> However this is not PCI which has a tree/switch and can enumerate devices to >>>> the host by spanning the device across the PCI hierarchy. >>>> >>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. >>> >>>> To address an ADI, there is only a device_id. >>>> >>> SIOV device must have a unique identifier at PCI bus level for sure. >>> I cannot speak more about it in this forum due to other logistics issue. >>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. >>> >>>> So, do you mind share how your DPU offload the device model? What kind of >>>> device your DPU provide to the host? Lets see whether DPU can mediate this by >>>> its own? >>>> >>> It is a virtio nic, blk and other virtio devices for us. >>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. >>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. >>> >>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. >>> You might find a Linux reference model useful at [1]. >>> A usage model already exists in one OS and in use for non virtio devices. >>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. >>> >>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. >>> >>> [1] https://github.com/Mellanox/scalablefunctions/wiki >> well I see SF facing the similar challenge, I can add a command for the >> driver to query all existing SIOV ADIs of a device, >> and the device return ADIs id and status. Looks good? and work for you >> @Xuan? > Can the device notifies the driver by a interrupt? Of course this can be an option as previously discussed. That would be a config interrupt(with proper suppression) from the management device, which requires a new cap in a bar, then you still need to fetch the information. A proposal: a new cap, at least contains: total number of the ADIs. Then once any ADIs are created or removed, the PF driver get a notification. Works for you? > > Thanks. > > >> Thanks >> > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 8:01 ` Zhu, Lingshan @ 2023-07-03 8:21 ` Xuan Zhuo 2023-07-03 8:23 ` Zhu, Lingshan 0 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-03 8:21 UTC (permalink / raw) To: Zhu, Lingshan Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On Mon, 3 Jul 2023 16:01:35 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 7/3/2023 1:54 PM, Xuan Zhuo wrote: > > On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >> > >> On 6/30/2023 7:35 PM, Parav Pandit wrote: > >>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > >>>> open.org> On Behalf Of Zhu, Lingshan > >>>> Sent: Friday, June 30, 2023 6:33 AM > >>>> > >>>>>>> Can we let the DPU notify the driver to create a new devicer from the > >>>> backend? > >>> Yes, why not. > >>> > >>>>>>> The key point is who want to create a new device. > >>>>>> DPU can come with a certain number of pre-created ADIs, just make > >>>>>> sure the orchestration SW is aware of their device IDs. > >>>>>> > >>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > >>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > >>> Xuan partly explained it. > >>> > >>>>>> If you want the DPU randomly create ADIs and notify the driver, I > >>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > >>>>>> wants to create and hot plug in a device to a guest? > >>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > >>>>>> machines? > >>>>> In your scenario, the supervisor is on the os, which controls the DPU > >>>>> to create new devices. > >>>>> > >>>>> In the cloud scenario, the vendor manager is in the DPU, and the > >>>>> entire host is for users. Of course, there are situations where the > >>>>> vendor manager are in the HOST. But for bare metal machines, the host > >>>>> belongs to the customer, the vendor manager is only in the DPU. > >>>>> > >>>>> So when the customers buy a new nic for the host, the vendor manager > >>>>> will plug a device to the host from the DPU. > >>>> I understand once a customer orders a new NIC, you wants to present the NIC > >>>> to the host. > >>>> However you only owns the DPU and the customer owns the host, that means > >>>> this creation and hot plug must be transparent to the host and there may not be > >>>> a host driver help handling an interrupt/probe. > >>>> > >>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > >>> > >>>> However this is not PCI which has a tree/switch and can enumerate devices to > >>>> the host by spanning the device across the PCI hierarchy. > >>>> > >>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > >>> > >>>> To address an ADI, there is only a device_id. > >>>> > >>> SIOV device must have a unique identifier at PCI bus level for sure. > >>> I cannot speak more about it in this forum due to other logistics issue. > >>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > >>> > >>>> So, do you mind share how your DPU offload the device model? What kind of > >>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > >>>> its own? > >>>> > >>> It is a virtio nic, blk and other virtio devices for us. > >>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > >>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > >>> > >>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > >>> You might find a Linux reference model useful at [1]. > >>> A usage model already exists in one OS and in use for non virtio devices. > >>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > >>> > >>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > >>> > >>> [1] https://github.com/Mellanox/scalablefunctions/wiki > >> well I see SF facing the similar challenge, I can add a command for the > >> driver to query all existing SIOV ADIs of a device, > >> and the device return ADIs id and status. Looks good? and work for you > >> @Xuan? > > Can the device notifies the driver by a interrupt? > Of course this can be an option as previously discussed. > That would be a config interrupt(with proper suppression) from the > management device, > which requires a new cap in a bar, then you still need to fetch the > information. > > A proposal: > > a new cap, at least contains: > total number of the ADIs. > > Then once any ADIs are created or removed, the PF driver get a notification. > > Works for you? Look good to me. Thanks. > > > > Thanks. > > > > > >> Thanks > >> > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 8:21 ` Xuan Zhuo @ 2023-07-03 8:23 ` Zhu, Lingshan 0 siblings, 0 replies; 49+ messages in thread From: Zhu, Lingshan @ 2023-07-03 8:23 UTC (permalink / raw) To: Xuan Zhuo Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On 7/3/2023 4:21 PM, Xuan Zhuo wrote: > On Mon, 3 Jul 2023 16:01:35 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >> >> On 7/3/2023 1:54 PM, Xuan Zhuo wrote: >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- >>>>>> open.org> On Behalf Of Zhu, Lingshan >>>>>> Sent: Friday, June 30, 2023 6:33 AM >>>>>> >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the >>>>>> backend? >>>>> Yes, why not. >>>>> >>>>>>>>> The key point is who want to create a new device. >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make >>>>>>>> sure the orchestration SW is aware of their device IDs. >>>>>>>> >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. >>>>> Xuan partly explained it. >>>>> >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU >>>>>>>> wants to create and hot plug in a device to a guest? >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal >>>>>>>> machines? >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU >>>>>>> to create new devices. >>>>>>> >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the >>>>>>> entire host is for users. Of course, there are situations where the >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host >>>>>>> belongs to the customer, the vendor manager is only in the DPU. >>>>>>> >>>>>>> So when the customers buy a new nic for the host, the vendor manager >>>>>>> will plug a device to the host from the DPU. >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC >>>>>> to the host. >>>>>> However you only owns the DPU and the customer owns the host, that means >>>>>> this creation and hot plug must be transparent to the host and there may not be >>>>>> a host driver help handling an interrupt/probe. >>>>>> >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. >>>>> >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to >>>>>> the host by spanning the device across the PCI hierarchy. >>>>>> >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. >>>>> >>>>>> To address an ADI, there is only a device_id. >>>>>> >>>>> SIOV device must have a unique identifier at PCI bus level for sure. >>>>> I cannot speak more about it in this forum due to other logistics issue. >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. >>>>> >>>>>> So, do you mind share how your DPU offload the device model? What kind of >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by >>>>>> its own? >>>>>> >>>>> It is a virtio nic, blk and other virtio devices for us. >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. >>>>> >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. >>>>> You might find a Linux reference model useful at [1]. >>>>> A usage model already exists in one OS and in use for non virtio devices. >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. >>>>> >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. >>>>> >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki >>>> well I see SF facing the similar challenge, I can add a command for the >>>> driver to query all existing SIOV ADIs of a device, >>>> and the device return ADIs id and status. Looks good? and work for you >>>> @Xuan? >>> Can the device notifies the driver by a interrupt? >> Of course this can be an option as previously discussed. >> That would be a config interrupt(with proper suppression) from the >> management device, >> which requires a new cap in a bar, then you still need to fetch the >> information. >> >> A proposal: >> >> a new cap, at least contains: >> total number of the ADIs. >> >> Then once any ADIs are created or removed, the PF driver get a notification. >> >> Works for you? > > Look good to me. OK, you can review the details when I post the series Thanks > > Thanks. > > >>> Thanks. >>> >>> >>>> Thanks >>>> >>> This publicly archived list offers a means to provide input to the >>> OASIS Virtual I/O Device (VIRTIO) TC. >>> >>> In order to verify user consent to the Feedback License terms and >>> to minimize spam in the list archive, subscription is required >>> before posting. >>> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>> List help: virtio-comment-help@lists.oasis-open.org >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>> Committee: https://www.oasis-open.org/committees/virtio/ >>> Join OASIS: https://www.oasis-open.org/join/ >>> This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 4:29 ` Zhu, Lingshan 2023-07-03 5:54 ` Xuan Zhuo @ 2023-07-27 2:30 ` Xuan Zhuo 2023-07-27 3:56 ` Zhu, Lingshan 1 sibling, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-27 2:30 UTC (permalink / raw) To: Zhu, Lingshan Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 6/30/2023 7:35 PM, Parav Pandit wrote: > >> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > >> open.org> On Behalf Of Zhu, Lingshan > >> Sent: Friday, June 30, 2023 6:33 AM > >> > > > >>>>> Can we let the DPU notify the driver to create a new devicer from the > >> backend? > > Yes, why not. > > > >>>>> The key point is who want to create a new device. > >>>> DPU can come with a certain number of pre-created ADIs, just make > >>>> sure the orchestration SW is aware of their device IDs. > >>>> > > Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > Xuan partly explained it. > > > >>>> If you want the DPU randomly create ADIs and notify the driver, I > >>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > >>>> wants to create and hot plug in a device to a guest? > >>>> Shall the host handle that or DPU pre-create then expose to baremteal > >>>> machines? > >>> In your scenario, the supervisor is on the os, which controls the DPU > >>> to create new devices. > >>> > >>> In the cloud scenario, the vendor manager is in the DPU, and the > >>> entire host is for users. Of course, there are situations where the > >>> vendor manager are in the HOST. But for bare metal machines, the host > >>> belongs to the customer, the vendor manager is only in the DPU. > >>> > >>> So when the customers buy a new nic for the host, the vendor manager > >>> will plug a device to the host from the DPU. > >> I understand once a customer orders a new NIC, you wants to present the NIC > >> to the host. > >> However you only owns the DPU and the customer owns the host, that means > >> this creation and hot plug must be transparent to the host and there may not be > >> a host driver help handling an interrupt/probe. > >> > > That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > >> However this is not PCI which has a tree/switch and can enumerate devices to > >> the host by spanning the device across the PCI hierarchy. > >> > > Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > >> To address an ADI, there is only a device_id. > >> > > SIOV device must have a unique identifier at PCI bus level for sure. > > I cannot speak more about it in this forum due to other logistics issue. > > But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > >> So, do you mind share how your DPU offload the device model? What kind of > >> device your DPU provide to the host? Lets see whether DPU can mediate this by > >> its own? > >> > > It is a virtio nic, blk and other virtio devices for us. > > A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > You might find a Linux reference model useful at [1]. > > A usage model already exists in one OS and in use for non virtio devices. > > This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > [1] https://github.com/Mellanox/scalablefunctions/wiki > well I see SF facing the similar challenge, I can add a command for the > driver to query all existing SIOV ADIs of a device, > and the device return ADIs id and status. Looks good? and work for you > @Xuan? Could I have your plan for this? If you do not mind, I'd like to add a command to query VF's info. Such as mac, ip, etc. Thanks. > > Thanks > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 2:30 ` Xuan Zhuo @ 2023-07-27 3:56 ` Zhu, Lingshan 2023-07-27 6:09 ` Xuan Zhuo 0 siblings, 1 reply; 49+ messages in thread From: Zhu, Lingshan @ 2023-07-27 3:56 UTC (permalink / raw) To: Xuan Zhuo Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >> >> On 6/30/2023 7:35 PM, Parav Pandit wrote: >>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- >>>> open.org> On Behalf Of Zhu, Lingshan >>>> Sent: Friday, June 30, 2023 6:33 AM >>>> >>>>>>> Can we let the DPU notify the driver to create a new devicer from the >>>> backend? >>> Yes, why not. >>> >>>>>>> The key point is who want to create a new device. >>>>>> DPU can come with a certain number of pre-created ADIs, just make >>>>>> sure the orchestration SW is aware of their device IDs. >>>>>> >>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. >>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. >>> Xuan partly explained it. >>> >>>>>> If you want the DPU randomly create ADIs and notify the driver, I >>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU >>>>>> wants to create and hot plug in a device to a guest? >>>>>> Shall the host handle that or DPU pre-create then expose to baremteal >>>>>> machines? >>>>> In your scenario, the supervisor is on the os, which controls the DPU >>>>> to create new devices. >>>>> >>>>> In the cloud scenario, the vendor manager is in the DPU, and the >>>>> entire host is for users. Of course, there are situations where the >>>>> vendor manager are in the HOST. But for bare metal machines, the host >>>>> belongs to the customer, the vendor manager is only in the DPU. >>>>> >>>>> So when the customers buy a new nic for the host, the vendor manager >>>>> will plug a device to the host from the DPU. >>>> I understand once a customer orders a new NIC, you wants to present the NIC >>>> to the host. >>>> However you only owns the DPU and the customer owns the host, that means >>>> this creation and hot plug must be transparent to the host and there may not be >>>> a host driver help handling an interrupt/probe. >>>> >>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. >>> >>>> However this is not PCI which has a tree/switch and can enumerate devices to >>>> the host by spanning the device across the PCI hierarchy. >>>> >>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. >>> >>>> To address an ADI, there is only a device_id. >>>> >>> SIOV device must have a unique identifier at PCI bus level for sure. >>> I cannot speak more about it in this forum due to other logistics issue. >>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. >>> >>>> So, do you mind share how your DPU offload the device model? What kind of >>>> device your DPU provide to the host? Lets see whether DPU can mediate this by >>>> its own? >>>> >>> It is a virtio nic, blk and other virtio devices for us. >>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. >>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. >>> >>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. >>> You might find a Linux reference model useful at [1]. >>> A usage model already exists in one OS and in use for non virtio devices. >>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. >>> >>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. >>> >>> [1] https://github.com/Mellanox/scalablefunctions/wiki >> well I see SF facing the similar challenge, I can add a command for the >> driver to query all existing SIOV ADIs of a device, >> and the device return ADIs id and status. Looks good? and work for you >> @Xuan? > Could I have your plan for this? > > If you do not mind, I'd like to add a command to query VF's info. Such > as mac, ip, etc. I think the query commands for SIOV is a little more complex, e.g., need to report device type and its scale(e.g., features, mq). There can be thousands of SIOV ADIs and we don't want output flood. We have discussed implementation a config interrupt to report new created / deleted ADIs on the DPU side, therefore there must be a cap contains related information, my rough approach of the process is: 1) a cap contains the total number of existing ADIs and the max dev id 2) driver queries detailed information of a certain ADI or a bunch of ADIs in a [dev_id....dev_id2] range. I am not sure whether a NIC stores its IP Thanks > > Thanks. > > > >> Thanks >> > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 3:56 ` Zhu, Lingshan @ 2023-07-27 6:09 ` Xuan Zhuo 2023-07-27 6:17 ` Zhu, Lingshan 0 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-27 6:09 UTC (permalink / raw) To: Zhu, Lingshan Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >> > >> On 6/30/2023 7:35 PM, Parav Pandit wrote: > >>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > >>>> open.org> On Behalf Of Zhu, Lingshan > >>>> Sent: Friday, June 30, 2023 6:33 AM > >>>> > >>>>>>> Can we let the DPU notify the driver to create a new devicer from the > >>>> backend? > >>> Yes, why not. > >>> > >>>>>>> The key point is who want to create a new device. > >>>>>> DPU can come with a certain number of pre-created ADIs, just make > >>>>>> sure the orchestration SW is aware of their device IDs. > >>>>>> > >>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > >>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > >>> Xuan partly explained it. > >>> > >>>>>> If you want the DPU randomly create ADIs and notify the driver, I > >>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > >>>>>> wants to create and hot plug in a device to a guest? > >>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > >>>>>> machines? > >>>>> In your scenario, the supervisor is on the os, which controls the DPU > >>>>> to create new devices. > >>>>> > >>>>> In the cloud scenario, the vendor manager is in the DPU, and the > >>>>> entire host is for users. Of course, there are situations where the > >>>>> vendor manager are in the HOST. But for bare metal machines, the host > >>>>> belongs to the customer, the vendor manager is only in the DPU. > >>>>> > >>>>> So when the customers buy a new nic for the host, the vendor manager > >>>>> will plug a device to the host from the DPU. > >>>> I understand once a customer orders a new NIC, you wants to present the NIC > >>>> to the host. > >>>> However you only owns the DPU and the customer owns the host, that means > >>>> this creation and hot plug must be transparent to the host and there may not be > >>>> a host driver help handling an interrupt/probe. > >>>> > >>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > >>> > >>>> However this is not PCI which has a tree/switch and can enumerate devices to > >>>> the host by spanning the device across the PCI hierarchy. > >>>> > >>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > >>> > >>>> To address an ADI, there is only a device_id. > >>>> > >>> SIOV device must have a unique identifier at PCI bus level for sure. > >>> I cannot speak more about it in this forum due to other logistics issue. > >>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > >>> > >>>> So, do you mind share how your DPU offload the device model? What kind of > >>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > >>>> its own? > >>>> > >>> It is a virtio nic, blk and other virtio devices for us. > >>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > >>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > >>> > >>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > >>> You might find a Linux reference model useful at [1]. > >>> A usage model already exists in one OS and in use for non virtio devices. > >>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > >>> > >>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > >>> > >>> [1] https://github.com/Mellanox/scalablefunctions/wiki > >> well I see SF facing the similar challenge, I can add a command for the > >> driver to query all existing SIOV ADIs of a device, > >> and the device return ADIs id and status. Looks good? and work for you > >> @Xuan? > > Could I have your plan for this? > > > > If you do not mind, I'd like to add a command to query VF's info. Such > > as mac, ip, etc. > I think the query commands for SIOV is a little more complex, e.g., > need to report device type and its scale(e.g., features, mq). > There can be thousands of SIOV ADIs and we don't want output flood. > > We have discussed implementation a config interrupt to report new > created / deleted > ADIs on the DPU side, therefore there must be a cap contains related > information, > my rough approach of the process is: > 1) a cap contains the total number of existing ADIs and the max dev id > 2) driver queries detailed information of a certain ADI or a bunch of > ADIs in a [dev_id....dev_id2] range. Yes, Admin Queue can obtain the info of the specific one or more devices. > > I am not sure whether a NIC stores its IP IP is the other topic. I want the Admin Queue manage the switch. So the switch know about the IP of every device, and the Admin Queue will has the ability to config the IP of the device inside the switch. Thanks. > > Thanks > > > > Thanks. > > > > > > > >> Thanks > >> > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 6:09 ` Xuan Zhuo @ 2023-07-27 6:17 ` Zhu, Lingshan 2023-07-27 6:20 ` Xuan Zhuo 0 siblings, 1 reply; 49+ messages in thread From: Zhu, Lingshan @ 2023-07-27 6:17 UTC (permalink / raw) To: Xuan Zhuo Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >> >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- >>>>>> open.org> On Behalf Of Zhu, Lingshan >>>>>> Sent: Friday, June 30, 2023 6:33 AM >>>>>> >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the >>>>>> backend? >>>>> Yes, why not. >>>>> >>>>>>>>> The key point is who want to create a new device. >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make >>>>>>>> sure the orchestration SW is aware of their device IDs. >>>>>>>> >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. >>>>> Xuan partly explained it. >>>>> >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU >>>>>>>> wants to create and hot plug in a device to a guest? >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal >>>>>>>> machines? >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU >>>>>>> to create new devices. >>>>>>> >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the >>>>>>> entire host is for users. Of course, there are situations where the >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host >>>>>>> belongs to the customer, the vendor manager is only in the DPU. >>>>>>> >>>>>>> So when the customers buy a new nic for the host, the vendor manager >>>>>>> will plug a device to the host from the DPU. >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC >>>>>> to the host. >>>>>> However you only owns the DPU and the customer owns the host, that means >>>>>> this creation and hot plug must be transparent to the host and there may not be >>>>>> a host driver help handling an interrupt/probe. >>>>>> >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. >>>>> >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to >>>>>> the host by spanning the device across the PCI hierarchy. >>>>>> >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. >>>>> >>>>>> To address an ADI, there is only a device_id. >>>>>> >>>>> SIOV device must have a unique identifier at PCI bus level for sure. >>>>> I cannot speak more about it in this forum due to other logistics issue. >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. >>>>> >>>>>> So, do you mind share how your DPU offload the device model? What kind of >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by >>>>>> its own? >>>>>> >>>>> It is a virtio nic, blk and other virtio devices for us. >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. >>>>> >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. >>>>> You might find a Linux reference model useful at [1]. >>>>> A usage model already exists in one OS and in use for non virtio devices. >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. >>>>> >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. >>>>> >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki >>>> well I see SF facing the similar challenge, I can add a command for the >>>> driver to query all existing SIOV ADIs of a device, >>>> and the device return ADIs id and status. Looks good? and work for you >>>> @Xuan? >>> Could I have your plan for this? >>> >>> If you do not mind, I'd like to add a command to query VF's info. Such >>> as mac, ip, etc. >> I think the query commands for SIOV is a little more complex, e.g., >> need to report device type and its scale(e.g., features, mq). >> There can be thousands of SIOV ADIs and we don't want output flood. >> >> We have discussed implementation a config interrupt to report new >> created / deleted >> ADIs on the DPU side, therefore there must be a cap contains related >> information, >> my rough approach of the process is: >> 1) a cap contains the total number of existing ADIs and the max dev id >> 2) driver queries detailed information of a certain ADI or a bunch of >> ADIs in a [dev_id....dev_id2] range. > Yes, Admin Queue can obtain the info of the specific one or more devices. > >> I am not sure whether a NIC stores its IP > > IP is the other topic. I want the Admin Queue manage the switch. > So the switch know about the IP of every device, and the > Admin Queue will has the ability to config the IP of the device inside the > switch. DPU onboard switch? OVS? Does it beyond virtio spec? > > > Thanks. > > >> Thanks >>> Thanks. >>> >>> >>> >>>> Thanks >>>> >>> This publicly archived list offers a means to provide input to the >>> OASIS Virtual I/O Device (VIRTIO) TC. >>> >>> In order to verify user consent to the Feedback License terms and >>> to minimize spam in the list archive, subscription is required >>> before posting. >>> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org >>> List help: virtio-comment-help@lists.oasis-open.org >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists >>> Committee: https://www.oasis-open.org/committees/virtio/ >>> Join OASIS: https://www.oasis-open.org/join/ >>> This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 6:17 ` Zhu, Lingshan @ 2023-07-27 6:20 ` Xuan Zhuo 2023-07-27 8:03 ` Jason Wang 0 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-27 6:20 UTC (permalink / raw) To: Zhu, Lingshan Cc: virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >> > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > >>>>>> open.org> On Behalf Of Zhu, Lingshan > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > >>>>>> > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > >>>>>> backend? > >>>>> Yes, why not. > >>>>> > >>>>>>>>> The key point is who want to create a new device. > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > >>>>>>>> sure the orchestration SW is aware of their device IDs. > >>>>>>>> > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > >>>>> Xuan partly explained it. > >>>>> > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > >>>>>>>> wants to create and hot plug in a device to a guest? > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > >>>>>>>> machines? > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > >>>>>>> to create new devices. > >>>>>>> > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > >>>>>>> entire host is for users. Of course, there are situations where the > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > >>>>>>> > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > >>>>>>> will plug a device to the host from the DPU. > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > >>>>>> to the host. > >>>>>> However you only owns the DPU and the customer owns the host, that means > >>>>>> this creation and hot plug must be transparent to the host and there may not be > >>>>>> a host driver help handling an interrupt/probe. > >>>>>> > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > >>>>> > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > >>>>>> the host by spanning the device across the PCI hierarchy. > >>>>>> > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > >>>>> > >>>>>> To address an ADI, there is only a device_id. > >>>>>> > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > >>>>> I cannot speak more about it in this forum due to other logistics issue. > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > >>>>> > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > >>>>>> its own? > >>>>>> > >>>>> It is a virtio nic, blk and other virtio devices for us. > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > >>>>> > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > >>>>> You might find a Linux reference model useful at [1]. > >>>>> A usage model already exists in one OS and in use for non virtio devices. > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > >>>>> > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > >>>>> > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > >>>> well I see SF facing the similar challenge, I can add a command for the > >>>> driver to query all existing SIOV ADIs of a device, > >>>> and the device return ADIs id and status. Looks good? and work for you > >>>> @Xuan? > >>> Could I have your plan for this? > >>> > >>> If you do not mind, I'd like to add a command to query VF's info. Such > >>> as mac, ip, etc. > >> I think the query commands for SIOV is a little more complex, e.g., > >> need to report device type and its scale(e.g., features, mq). > >> There can be thousands of SIOV ADIs and we don't want output flood. > >> > >> We have discussed implementation a config interrupt to report new > >> created / deleted > >> ADIs on the DPU side, therefore there must be a cap contains related > >> information, > >> my rough approach of the process is: > >> 1) a cap contains the total number of existing ADIs and the max dev id > >> 2) driver queries detailed information of a certain ADI or a bunch of > >> ADIs in a [dev_id....dev_id2] range. > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > >> I am not sure whether a NIC stores its IP > > > > IP is the other topic. I want the Admin Queue manage the switch. > > So the switch know about the IP of every device, and the > > Admin Queue will has the ability to config the IP of the device inside the > > switch. > DPU onboard switch? OVS? Does it beyond virtio spec? YES. For SIOV, I think this is MUST. Maybe you have one simple implementation. But you have to solve the IP steering. So admin queue should has the ability to config the IP steering. Thanks. > > > > > > Thanks. > > > > > >> Thanks > >>> Thanks. > >>> > >>> > >>> > >>>> Thanks > >>>> > >>> This publicly archived list offers a means to provide input to the > >>> OASIS Virtual I/O Device (VIRTIO) TC. > >>> > >>> In order to verify user consent to the Feedback License terms and > >>> to minimize spam in the list archive, subscription is required > >>> before posting. > >>> > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>> List help: virtio-comment-help@lists.oasis-open.org > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>> Committee: https://www.oasis-open.org/committees/virtio/ > >>> Join OASIS: https://www.oasis-open.org/join/ > >>> > > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 6:20 ` Xuan Zhuo @ 2023-07-27 8:03 ` Jason Wang 2023-07-27 8:07 ` Xuan Zhuo 2023-07-28 6:09 ` Xuan Zhuo 0 siblings, 2 replies; 49+ messages in thread From: Jason Wang @ 2023-07-27 8:03 UTC (permalink / raw) To: Xuan Zhuo Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Washizu Yui On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > >> > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > >>>>>> > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > >>>>>> backend? > > >>>>> Yes, why not. > > >>>>> > > >>>>>>>>> The key point is who want to create a new device. > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > >>>>>>>> > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > >>>>> Xuan partly explained it. > > >>>>> > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > >>>>>>>> wants to create and hot plug in a device to a guest? > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > >>>>>>>> machines? > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > >>>>>>> to create new devices. > > >>>>>>> > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > >>>>>>> entire host is for users. Of course, there are situations where the > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > >>>>>>> > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > >>>>>>> will plug a device to the host from the DPU. > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > >>>>>> to the host. > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > >>>>>> a host driver help handling an interrupt/probe. > > >>>>>> > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > >>>>> > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > >>>>>> the host by spanning the device across the PCI hierarchy. > > >>>>>> > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > >>>>> > > >>>>>> To address an ADI, there is only a device_id. > > >>>>>> > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > >>>>> > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > >>>>>> its own? > > >>>>>> > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > >>>>> > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > >>>>> You might find a Linux reference model useful at [1]. > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > >>>>> > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > >>>>> > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > >>>> well I see SF facing the similar challenge, I can add a command for the > > >>>> driver to query all existing SIOV ADIs of a device, > > >>>> and the device return ADIs id and status. Looks good? and work for you > > >>>> @Xuan? > > >>> Could I have your plan for this? > > >>> > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > >>> as mac, ip, etc. > > >> I think the query commands for SIOV is a little more complex, e.g., > > >> need to report device type and its scale(e.g., features, mq). > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > >> > > >> We have discussed implementation a config interrupt to report new > > >> created / deleted > > >> ADIs on the DPU side, therefore there must be a cap contains related > > >> information, > > >> my rough approach of the process is: > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > >> ADIs in a [dev_id....dev_id2] range. > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > >> I am not sure whether a NIC stores its IP > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > So the switch know about the IP of every device, and the > > > Admin Queue will has the ability to config the IP of the device inside the > > > switch. > > DPU onboard switch? OVS? Does it beyond virtio spec? > > YES. Adding Washizu. We can have a switch/dpa defined in the networking device for sure. > > For SIOV, I think this is MUST. A learning bridge would be fine as a starter. It's better not to couple new scalable capability with any device specific features. > Maybe you have one simple implementation. > But you have to solve the IP steering. So admin queue should has the ability > to config the IP steering. I think not. Those L2/LN tables/filters are networking specific. Control virtqueue is better than admin virtqueue here. Thanks > > Thanks. > > > > > > > > > > Thanks. > > > > > > > > >> Thanks > > >>> Thanks. > > >>> > > >>> > > >>> > > >>>> Thanks > > >>>> > > >>> This publicly archived list offers a means to provide input to the > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > >>> > > >>> In order to verify user consent to the Feedback License terms and > > >>> to minimize spam in the list archive, subscription is required > > >>> before posting. > > >>> > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > >>> List help: virtio-comment-help@lists.oasis-open.org > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > >>> Join OASIS: https://www.oasis-open.org/join/ > > >>> > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 8:03 ` Jason Wang @ 2023-07-27 8:07 ` Xuan Zhuo 2023-07-27 8:28 ` Jason Wang 2023-07-28 6:09 ` Xuan Zhuo 1 sibling, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-27 8:07 UTC (permalink / raw) To: Jason Wang Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Washizu Yui On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > >> > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > >>>>>> > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > >>>>>> backend? > > > >>>>> Yes, why not. > > > >>>>> > > > >>>>>>>>> The key point is who want to create a new device. > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > > >>>>>>>> > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > >>>>> Xuan partly explained it. > > > >>>>> > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > >>>>>>>> wants to create and hot plug in a device to a guest? > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > >>>>>>>> machines? > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > >>>>>>> to create new devices. > > > >>>>>>> > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > >>>>>>> entire host is for users. Of course, there are situations where the > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > >>>>>>> > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > >>>>>>> will plug a device to the host from the DPU. > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > >>>>>> to the host. > > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > > >>>>>> a host driver help handling an interrupt/probe. > > > >>>>>> > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > >>>>> > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > >>>>>> the host by spanning the device across the PCI hierarchy. > > > >>>>>> > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > >>>>> > > > >>>>>> To address an ADI, there is only a device_id. > > > >>>>>> > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > >>>>> > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > >>>>>> its own? > > > >>>>>> > > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > >>>>> > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > >>>>> You might find a Linux reference model useful at [1]. > > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > >>>>> > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > >>>>> > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > >>>> well I see SF facing the similar challenge, I can add a command for the > > > >>>> driver to query all existing SIOV ADIs of a device, > > > >>>> and the device return ADIs id and status. Looks good? and work for you > > > >>>> @Xuan? > > > >>> Could I have your plan for this? > > > >>> > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > > >>> as mac, ip, etc. > > > >> I think the query commands for SIOV is a little more complex, e.g., > > > >> need to report device type and its scale(e.g., features, mq). > > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > > >> > > > >> We have discussed implementation a config interrupt to report new > > > >> created / deleted > > > >> ADIs on the DPU side, therefore there must be a cap contains related > > > >> information, > > > >> my rough approach of the process is: > > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > > >> ADIs in a [dev_id....dev_id2] range. > > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > > > >> I am not sure whether a NIC stores its IP > > > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > > So the switch know about the IP of every device, and the > > > > Admin Queue will has the ability to config the IP of the device inside the > > > > switch. > > > DPU onboard switch? OVS? Does it beyond virtio spec? > > > > YES. > > Adding Washizu. > > We can have a switch/dpa defined in the networking device for sure. Yes, I think we should introduce that for the sr-iov. Or for other. I would like to know who is doing this? Another question, @Jason are you referring to a new device type or a new virtio-net feature. > > > > > For SIOV, I think this is MUST. > > A learning bridge would be fine as a starter. It's better not to > couple new scalable capability with any device specific features. > > > Maybe you have one simple implementation. > > But you have to solve the IP steering. So admin queue should has the ability > > to config the IP steering. > > I think not. Those L2/LN tables/filters are networking specific. Let us assume that there is a switch/bridge firstly. The VFs may be passed to different VMs. I also think this is the networking specific. But I want to config the ip for every vf from the pf. Because the user of the vf may be unreliable. We need a manager to config the ip for every vf. > Control virtqueue is better than admin virtqueue here. by cq? What case? Thanks. > > Thanks > > > > > Thanks. > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > >> Thanks > > > >>> Thanks. > > > >>> > > > >>> > > > >>> > > > >>>> Thanks > > > >>>> > > > >>> This publicly archived list offers a means to provide input to the > > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > > >>> > > > >>> In order to verify user consent to the Feedback License terms and > > > >>> to minimize spam in the list archive, subscription is required > > > >>> before posting. > > > >>> > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > >>> List help: virtio-comment-help@lists.oasis-open.org > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > > >>> Join OASIS: https://www.oasis-open.org/join/ > > > >>> > > > > > > > > > This publicly archived list offers a means to provide input to the > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > In order to verify user consent to the Feedback License terms and > > > to minimize spam in the list archive, subscription is required > > > before posting. > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > List help: virtio-comment-help@lists.oasis-open.org > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 8:07 ` Xuan Zhuo @ 2023-07-27 8:28 ` Jason Wang 2023-07-27 8:30 ` Xuan Zhuo [not found] ` <aafe1885-0ec2-66ca-4511-f2606bc881ee@gmail.com> 0 siblings, 2 replies; 49+ messages in thread From: Jason Wang @ 2023-07-27 8:28 UTC (permalink / raw) To: Xuan Zhuo Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Washizu Yui On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > >> > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > > >>>>>> > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > > >>>>>> backend? > > > > >>>>> Yes, why not. > > > > >>>>> > > > > >>>>>>>>> The key point is who want to create a new device. > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > > > >>>>>>>> > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > > >>>>> Xuan partly explained it. > > > > >>>>> > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > > >>>>>>>> wants to create and hot plug in a device to a guest? > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > > >>>>>>>> machines? > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > > >>>>>>> to create new devices. > > > > >>>>>>> > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > > >>>>>>> entire host is for users. Of course, there are situations where the > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > > >>>>>>> > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > > >>>>>>> will plug a device to the host from the DPU. > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > > >>>>>> to the host. > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > > > >>>>>> a host driver help handling an interrupt/probe. > > > > >>>>>> > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > > >>>>> > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > > >>>>>> the host by spanning the device across the PCI hierarchy. > > > > >>>>>> > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > > >>>>> > > > > >>>>>> To address an ADI, there is only a device_id. > > > > >>>>>> > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > > >>>>> > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > > >>>>>> its own? > > > > >>>>>> > > > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > >>>>> > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > > >>>>> You might find a Linux reference model useful at [1]. > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > >>>>> > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > >>>>> > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > > >>>> well I see SF facing the similar challenge, I can add a command for the > > > > >>>> driver to query all existing SIOV ADIs of a device, > > > > >>>> and the device return ADIs id and status. Looks good? and work for you > > > > >>>> @Xuan? > > > > >>> Could I have your plan for this? > > > > >>> > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > > > >>> as mac, ip, etc. > > > > >> I think the query commands for SIOV is a little more complex, e.g., > > > > >> need to report device type and its scale(e.g., features, mq). > > > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > > > >> > > > > >> We have discussed implementation a config interrupt to report new > > > > >> created / deleted > > > > >> ADIs on the DPU side, therefore there must be a cap contains related > > > > >> information, > > > > >> my rough approach of the process is: > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > > > >> ADIs in a [dev_id....dev_id2] range. > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > > > > > >> I am not sure whether a NIC stores its IP > > > > > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > > > So the switch know about the IP of every device, and the > > > > > Admin Queue will has the ability to config the IP of the device inside the > > > > > switch. > > > > DPU onboard switch? OVS? Does it beyond virtio spec? > > > > > > YES. > > > > Adding Washizu. > > > > We can have a switch/dpa defined in the networking device for sure. > > Yes, I think we should introduce that for the sr-iov. Or for other. This should be a general one as a switch should be transport independent. > > I would like to know who is doing this? Washizu, could you confirm if you want to do this or not? > > Another question, @Jason are you referring to a new device type or a > new virtio-net feature. Extending virtio-net should be fine, did you see any issues for this? > > > > > > > > > For SIOV, I think this is MUST. > > > > A learning bridge would be fine as a starter. It's better not to > > couple new scalable capability with any device specific features. > > > > > Maybe you have one simple implementation. > > > But you have to solve the IP steering. So admin queue should has the ability > > > to config the IP steering. > > > > I think not. Those L2/LN tables/filters are networking specific. > > Let us assume that there is a switch/bridge firstly. > The VFs may be passed to different VMs. > > I also think this is the networking specific. But I want to config > the ip for every vf from the pf. What do you mean by ip here (e.g who is the user for this ip?) > Because the user of the vf may be unreliable. > We need a manager to config the ip for every vf. Did you mean you're using a tunnel or not? > > > > Control virtqueue is better than admin virtqueue here. > > by cq? > > What case? We've already used control virtqueue for steering. Thanks > > Thanks. > > > > > Thanks > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > >> Thanks > > > > >>> Thanks. > > > > >>> > > > > >>> > > > > >>> > > > > >>>> Thanks > > > > >>>> > > > > >>> This publicly archived list offers a means to provide input to the > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > >>> > > > > >>> In order to verify user consent to the Feedback License terms and > > > > >>> to minimize spam in the list archive, subscription is required > > > > >>> before posting. > > > > >>> > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > >>> List help: virtio-comment-help@lists.oasis-open.org > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > >>> Join OASIS: https://www.oasis-open.org/join/ > > > > >>> > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > to minimize spam in the list archive, subscription is required > > > > before posting. > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > In order to verify user consent to the Feedback License terms and > > > to minimize spam in the list archive, subscription is required > > > before posting. > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > List help: virtio-comment-help@lists.oasis-open.org > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 8:28 ` Jason Wang @ 2023-07-27 8:30 ` Xuan Zhuo 2023-07-27 8:56 ` Jason Wang [not found] ` <aafe1885-0ec2-66ca-4511-f2606bc881ee@gmail.com> 1 sibling, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-27 8:30 UTC (permalink / raw) To: Jason Wang Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Washizu Yui On Thu, 27 Jul 2023 16:28:07 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > >> > > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > > > >>>>>> > > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > > > >>>>>> backend? > > > > > >>>>> Yes, why not. > > > > > >>>>> > > > > > >>>>>>>>> The key point is who want to create a new device. > > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > > > > >>>>>>>> > > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > > > >>>>> Xuan partly explained it. > > > > > >>>>> > > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > > > >>>>>>>> wants to create and hot plug in a device to a guest? > > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > > > >>>>>>>> machines? > > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > > > >>>>>>> to create new devices. > > > > > >>>>>>> > > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > > > >>>>>>> entire host is for users. Of course, there are situations where the > > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > > > >>>>>>> > > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > > > >>>>>>> will plug a device to the host from the DPU. > > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > > > >>>>>> to the host. > > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > > > > >>>>>> a host driver help handling an interrupt/probe. > > > > > >>>>>> > > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > > > >>>>> > > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > > > >>>>>> the host by spanning the device across the PCI hierarchy. > > > > > >>>>>> > > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > > > >>>>> > > > > > >>>>>> To address an ADI, there is only a device_id. > > > > > >>>>>> > > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > > > >>>>> > > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > > > >>>>>> its own? > > > > > >>>>>> > > > > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > > >>>>> > > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > > > >>>>> You might find a Linux reference model useful at [1]. > > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > > >>>>> > > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > > >>>>> > > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > > > >>>> well I see SF facing the similar challenge, I can add a command for the > > > > > >>>> driver to query all existing SIOV ADIs of a device, > > > > > >>>> and the device return ADIs id and status. Looks good? and work for you > > > > > >>>> @Xuan? > > > > > >>> Could I have your plan for this? > > > > > >>> > > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > > > > >>> as mac, ip, etc. > > > > > >> I think the query commands for SIOV is a little more complex, e.g., > > > > > >> need to report device type and its scale(e.g., features, mq). > > > > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > > > > >> > > > > > >> We have discussed implementation a config interrupt to report new > > > > > >> created / deleted > > > > > >> ADIs on the DPU side, therefore there must be a cap contains related > > > > > >> information, > > > > > >> my rough approach of the process is: > > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > > > > >> ADIs in a [dev_id....dev_id2] range. > > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > > > > > > > >> I am not sure whether a NIC stores its IP > > > > > > > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > > > > So the switch know about the IP of every device, and the > > > > > > Admin Queue will has the ability to config the IP of the device inside the > > > > > > switch. > > > > > DPU onboard switch? OVS? Does it beyond virtio spec? > > > > > > > > YES. > > > > > > Adding Washizu. > > > > > > We can have a switch/dpa defined in the networking device for sure. > > > > Yes, I think we should introduce that for the sr-iov. Or for other. > > This should be a general one as a switch should be transport independent. I agree. > > > > > I would like to know who is doing this? > > Washizu, could you confirm if you want to do this or not? > > > > > Another question, @Jason are you referring to a new device type or a > > new virtio-net feature. > > Extending virtio-net should be fine, did you see any issues for this? You kwnow the packet will be steered to different virtio-net devices. So I think this is odd if the feature is the extending of virtio-net. How do you think about this? > > > > > > > > > > > > > > For SIOV, I think this is MUST. > > > > > > A learning bridge would be fine as a starter. It's better not to > > > couple new scalable capability with any device specific features. > > > > > > > Maybe you have one simple implementation. > > > > But you have to solve the IP steering. So admin queue should has the ability > > > > to config the IP steering. > > > > > > I think not. Those L2/LN tables/filters are networking specific. > > > > Let us assume that there is a switch/bridge firstly. > > The VFs may be passed to different VMs. > > > > I also think this is the networking specific. But I want to config > > the ip for every vf from the pf. > > What do you mean by ip here (e.g who is the user for this ip?) 1. the admin queue config the ip for the vf to the switch 2. the vf get the ip by dhcp from the switch. (or other way) Here we config an ip for the vf, and we also limit that just this vf can use the ip for security. Other vf can not use the ip configured by self. > > > Because the user of the vf may be unreliable. > > We need a manager to config the ip for every vf. > > Did you mean you're using a tunnel or not? No. Multiple VMs on a host may be used by different users. Some users are not trustworthy. We want to prevent a user from maliciously using ip > > > > > > > > Control virtqueue is better than admin virtqueue here. > > > > by cq? > > > > What case? > > We've already used control virtqueue for steering. > More details? You config the steering rule to other virtio-net device by the cq of one virtio-net? Thanks. > Thanks > > > > > Thanks. > > > > > > > > Thanks > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > >> Thanks > > > > > >>> Thanks. > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>>> Thanks > > > > > >>>> > > > > > >>> This publicly archived list offers a means to provide input to the > > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > > >>> > > > > > >>> In order to verify user consent to the Feedback License terms and > > > > > >>> to minimize spam in the list archive, subscription is required > > > > > >>> before posting. > > > > > >>> > > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > >>> List help: virtio-comment-help@lists.oasis-open.org > > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > > >>> Join OASIS: https://www.oasis-open.org/join/ > > > > > >>> > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > > to minimize spam in the list archive, subscription is required > > > > > before posting. > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > to minimize spam in the list archive, subscription is required > > > > before posting. > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 8:30 ` Xuan Zhuo @ 2023-07-27 8:56 ` Jason Wang 2023-07-27 9:01 ` Xuan Zhuo 0 siblings, 1 reply; 49+ messages in thread From: Jason Wang @ 2023-07-27 8:56 UTC (permalink / raw) To: Xuan Zhuo Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Washizu Yui On Thu, Jul 27, 2023 at 4:41 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Thu, 27 Jul 2023 16:28:07 +0800, Jason Wang <jasowang@redhat.com> wrote: > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > > > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > > > > > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > >> > > > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > > > > >>>>>> > > > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > > > > >>>>>> backend? > > > > > > >>>>> Yes, why not. > > > > > > >>>>> > > > > > > >>>>>>>>> The key point is who want to create a new device. > > > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > > > > > >>>>>>>> > > > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > > > > >>>>> Xuan partly explained it. > > > > > > >>>>> > > > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > > > > >>>>>>>> wants to create and hot plug in a device to a guest? > > > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > > > > >>>>>>>> machines? > > > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > > > > >>>>>>> to create new devices. > > > > > > >>>>>>> > > > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > > > > >>>>>>> entire host is for users. Of course, there are situations where the > > > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > > > > >>>>>>> > > > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > > > > >>>>>>> will plug a device to the host from the DPU. > > > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > > > > >>>>>> to the host. > > > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > > > > > >>>>>> a host driver help handling an interrupt/probe. > > > > > > >>>>>> > > > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > > > > >>>>> > > > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > > > > >>>>>> the host by spanning the device across the PCI hierarchy. > > > > > > >>>>>> > > > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > > > > >>>>> > > > > > > >>>>>> To address an ADI, there is only a device_id. > > > > > > >>>>>> > > > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > > > > >>>>> > > > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > > > > >>>>>> its own? > > > > > > >>>>>> > > > > > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > > > >>>>> > > > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > > > > >>>>> You might find a Linux reference model useful at [1]. > > > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > > > >>>>> > > > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > > > >>>>> > > > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > > > > >>>> well I see SF facing the similar challenge, I can add a command for the > > > > > > >>>> driver to query all existing SIOV ADIs of a device, > > > > > > >>>> and the device return ADIs id and status. Looks good? and work for you > > > > > > >>>> @Xuan? > > > > > > >>> Could I have your plan for this? > > > > > > >>> > > > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > > > > > >>> as mac, ip, etc. > > > > > > >> I think the query commands for SIOV is a little more complex, e.g., > > > > > > >> need to report device type and its scale(e.g., features, mq). > > > > > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > > > > > >> > > > > > > >> We have discussed implementation a config interrupt to report new > > > > > > >> created / deleted > > > > > > >> ADIs on the DPU side, therefore there must be a cap contains related > > > > > > >> information, > > > > > > >> my rough approach of the process is: > > > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > > > > > >> ADIs in a [dev_id....dev_id2] range. > > > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > > > > > > > > > >> I am not sure whether a NIC stores its IP > > > > > > > > > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > > > > > So the switch know about the IP of every device, and the > > > > > > > Admin Queue will has the ability to config the IP of the device inside the > > > > > > > switch. > > > > > > DPU onboard switch? OVS? Does it beyond virtio spec? > > > > > > > > > > YES. > > > > > > > > Adding Washizu. > > > > > > > > We can have a switch/dpa defined in the networking device for sure. > > > > > > Yes, I think we should introduce that for the sr-iov. Or for other. > > > > This should be a general one as a switch should be transport independent. > > I agree. > > > > > > > > > > I would like to know who is doing this? > > > > Washizu, could you confirm if you want to do this or not? > > > > > > > > Another question, @Jason are you referring to a new device type or a > > > new virtio-net feature. > > > > Extending virtio-net should be fine, did you see any issues for this? > > You kwnow the packet will be steered to different virtio-net devices. > So I think this is odd if the feature is the extending of virtio-net. > > How do you think about this? I think this is the common model of modern NICs? For example the PF with switch is usually an ethernet device itself. > > > > > > > > > > > > > > > > > > > > > For SIOV, I think this is MUST. > > > > > > > > A learning bridge would be fine as a starter. It's better not to > > > > couple new scalable capability with any device specific features. > > > > > > > > > Maybe you have one simple implementation. > > > > > But you have to solve the IP steering. So admin queue should has the ability > > > > > to config the IP steering. > > > > > > > > I think not. Those L2/LN tables/filters are networking specific. > > > > > > Let us assume that there is a switch/bridge firstly. > > > The VFs may be passed to different VMs. > > > > > > I also think this is the networking specific. But I want to config > > > the ip for every vf from the pf. > > > > What do you mean by ip here (e.g who is the user for this ip?) > > 1. the admin queue config the ip for the vf to the switch > 2. the vf get the ip by dhcp from the switch. (or other way) > > Here we config an ip for the vf, and we also limit that > just this vf can use the ip for security. Other vf can not use the ip configured > by self. It looks like a feature which is orthogonal to switch. > > > > > > > Because the user of the vf may be unreliable. > > > We need a manager to config the ip for every vf. > > > > Did you mean you're using a tunnel or not? > > No. > > Multiple VMs on a host may be used by different users. Some users are not > trustworthy. We want to prevent a user from maliciously using ip Ok, so it's an independent feature which could be done separately. > > > > > > > > > > > > > Control virtqueue is better than admin virtqueue here. > > > > > > by cq? > > > > > > What case? > > > > We've already used control virtqueue for steering. > > > > > More details? > > You config the steering rule to other virtio-net device by the cq of one > virtio-net? Exactly, we had already had the owner and group definition. So it's not that hard? Actually, there should be layers in the middle for example, the switch should see only ports and there should be some compositions of ports to a device I guess. Thanks > > Thanks. > > > > > Thanks > > > > > > > > Thanks. > > > > > > > > > > > Thanks > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > >> Thanks > > > > > > >>> Thanks. > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > >>>> Thanks > > > > > > >>>> > > > > > > >>> This publicly archived list offers a means to provide input to the > > > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > >>> > > > > > > >>> In order to verify user consent to the Feedback License terms and > > > > > > >>> to minimize spam in the list archive, subscription is required > > > > > > >>> before posting. > > > > > > >>> > > > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > > >>> List help: virtio-comment-help@lists.oasis-open.org > > > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > > > >>> Join OASIS: https://www.oasis-open.org/join/ > > > > > > >>> > > > > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > > > to minimize spam in the list archive, subscription is required > > > > > > before posting. > > > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > > to minimize spam in the list archive, subscription is required > > > > > before posting. > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 8:56 ` Jason Wang @ 2023-07-27 9:01 ` Xuan Zhuo 0 siblings, 0 replies; 49+ messages in thread From: Xuan Zhuo @ 2023-07-27 9:01 UTC (permalink / raw) To: Jason Wang Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Washizu Yui On Thu, 27 Jul 2023 16:56:55 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Thu, Jul 27, 2023 at 4:41 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > On Thu, 27 Jul 2023 16:28:07 +0800, Jason Wang <jasowang@redhat.com> wrote: > > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > > > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > > > > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > > > > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > > > > > > > > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > >> > > > > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > > > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > > > > > >>>>>> > > > > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > > > > > >>>>>> backend? > > > > > > > >>>>> Yes, why not. > > > > > > > >>>>> > > > > > > > >>>>>>>>> The key point is who want to create a new device. > > > > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > > > > > > >>>>>>>> > > > > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > > > > > >>>>> Xuan partly explained it. > > > > > > > >>>>> > > > > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > > > > > >>>>>>>> wants to create and hot plug in a device to a guest? > > > > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > > > > > >>>>>>>> machines? > > > > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > > > > > >>>>>>> to create new devices. > > > > > > > >>>>>>> > > > > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > > > > > >>>>>>> entire host is for users. Of course, there are situations where the > > > > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > > > > > >>>>>>> > > > > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > > > > > >>>>>>> will plug a device to the host from the DPU. > > > > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > > > > > >>>>>> to the host. > > > > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > > > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > > > > > > >>>>>> a host driver help handling an interrupt/probe. > > > > > > > >>>>>> > > > > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > > > > > >>>>> > > > > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > > > > > >>>>>> the host by spanning the device across the PCI hierarchy. > > > > > > > >>>>>> > > > > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > > > > > >>>>> > > > > > > > >>>>>> To address an ADI, there is only a device_id. > > > > > > > >>>>>> > > > > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > > > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > > > > > >>>>> > > > > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > > > > > >>>>>> its own? > > > > > > > >>>>>> > > > > > > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > > > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > > > > >>>>> > > > > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > > > > > >>>>> You might find a Linux reference model useful at [1]. > > > > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > > > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > > > > >>>>> > > > > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > > > > >>>>> > > > > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > > > > > >>>> well I see SF facing the similar challenge, I can add a command for the > > > > > > > >>>> driver to query all existing SIOV ADIs of a device, > > > > > > > >>>> and the device return ADIs id and status. Looks good? and work for you > > > > > > > >>>> @Xuan? > > > > > > > >>> Could I have your plan for this? > > > > > > > >>> > > > > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > > > > > > >>> as mac, ip, etc. > > > > > > > >> I think the query commands for SIOV is a little more complex, e.g., > > > > > > > >> need to report device type and its scale(e.g., features, mq). > > > > > > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > > > > > > >> > > > > > > > >> We have discussed implementation a config interrupt to report new > > > > > > > >> created / deleted > > > > > > > >> ADIs on the DPU side, therefore there must be a cap contains related > > > > > > > >> information, > > > > > > > >> my rough approach of the process is: > > > > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > > > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > > > > > > >> ADIs in a [dev_id....dev_id2] range. > > > > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > > > > > > > > > > > >> I am not sure whether a NIC stores its IP > > > > > > > > > > > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > > > > > > So the switch know about the IP of every device, and the > > > > > > > > Admin Queue will has the ability to config the IP of the device inside the > > > > > > > > switch. > > > > > > > DPU onboard switch? OVS? Does it beyond virtio spec? > > > > > > > > > > > > YES. > > > > > > > > > > Adding Washizu. > > > > > > > > > > We can have a switch/dpa defined in the networking device for sure. > > > > > > > > Yes, I think we should introduce that for the sr-iov. Or for other. > > > > > > This should be a general one as a switch should be transport independent. > > > > I agree. > > > > > > > > > > > > > > > I would like to know who is doing this? > > > > > > Washizu, could you confirm if you want to do this or not? > > > > > > > > > > > Another question, @Jason are you referring to a new device type or a > > > > new virtio-net feature. > > > > > > Extending virtio-net should be fine, did you see any issues for this? > > > > You kwnow the packet will be steered to different virtio-net devices. > > So I think this is odd if the feature is the extending of virtio-net. > > > > How do you think about this? > > I think this is the common model of modern NICs? For example the PF > with switch is usually an ethernet device itself. Great! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > For SIOV, I think this is MUST. > > > > > > > > > > A learning bridge would be fine as a starter. It's better not to > > > > > couple new scalable capability with any device specific features. > > > > > > > > > > > Maybe you have one simple implementation. > > > > > > But you have to solve the IP steering. So admin queue should has the ability > > > > > > to config the IP steering. > > > > > > > > > > I think not. Those L2/LN tables/filters are networking specific. > > > > > > > > Let us assume that there is a switch/bridge firstly. > > > > The VFs may be passed to different VMs. > > > > > > > > I also think this is the networking specific. But I want to config > > > > the ip for every vf from the pf. > > > > > > What do you mean by ip here (e.g who is the user for this ip?) > > > > 1. the admin queue config the ip for the vf to the switch > > 2. the vf get the ip by dhcp from the switch. (or other way) > > > > Here we config an ip for the vf, and we also limit that > > just this vf can use the ip for security. Other vf can not use the ip configured > > by self. > > It looks like a feature which is orthogonal to switch. > > > > > > > > > > > > Because the user of the vf may be unreliable. > > > > We need a manager to config the ip for every vf. > > > > > > Did you mean you're using a tunnel or not? > > > > No. > > > > Multiple VMs on a host may be used by different users. Some users are not > > trustworthy. We want to prevent a user from maliciously using ip > > Ok, so it's an independent feature which could be done separately. > > > > > > > > > > > > > > > > > > > Control virtqueue is better than admin virtqueue here. > > > > > > > > by cq? > > > > > > > > What case? > > > > > > We've already used control virtqueue for steering. Now, the spec includes this? Do you mean that you config the steering by the control virtqueue of pf? If so, I think is ok. Thanks. > > > > > > > > > More details? > > > > You config the steering rule to other virtio-net device by the cq of one > > virtio-net? > > Exactly, we had already had the owner and group definition. So it's > not that hard? Actually, there should be layers in the middle for > example, the switch should see only ports and there should be some > compositions of ports to a device I guess. > > Thanks > > > > > Thanks. > > > > > > > > > Thanks > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > >> Thanks > > > > > > > >>> Thanks. > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>>> Thanks > > > > > > > >>>> > > > > > > > >>> This publicly archived list offers a means to provide input to the > > > > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > >>> > > > > > > > >>> In order to verify user consent to the Feedback License terms and > > > > > > > >>> to minimize spam in the list archive, subscription is required > > > > > > > >>> before posting. > > > > > > > >>> > > > > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > > > >>> List help: virtio-comment-help@lists.oasis-open.org > > > > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > > > > >>> Join OASIS: https://www.oasis-open.org/join/ > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > > > > to minimize spam in the list archive, subscription is required > > > > > > > before posting. > > > > > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > > > to minimize spam in the list archive, subscription is required > > > > > > before posting. > > > > > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
[parent not found: <aafe1885-0ec2-66ca-4511-f2606bc881ee@gmail.com>]
* Re: [virtio-comment] About the plan of Admin Queue [not found] ` <aafe1885-0ec2-66ca-4511-f2606bc881ee@gmail.com> @ 2023-08-02 6:13 ` Xuan Zhuo 2023-08-02 6:15 ` Jason Wang 1 sibling, 0 replies; 49+ messages in thread From: Xuan Zhuo @ 2023-08-02 6:13 UTC (permalink / raw) To: Yui Washizu Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Jason Wang On Wed, 2 Aug 2023 15:01:21 +0900, Yui Washizu <yui.washidu@gmail.com> wrote: > > On 2023/07/27 17:28, Jason Wang wrote: > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>> > >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan > >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM > >>>>>>>>>>> > >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > >>>>>>>>>>> backend? > >>>>>>>>>> Yes, why not. > >>>>>>>>>> > >>>>>>>>>>>>>> The key point is who want to create a new device. > >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs. > >>>>>>>>>>>>> > >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > >>>>>>>>>> Xuan partly explained it. > >>>>>>>>>> > >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > >>>>>>>>>>>>> wants to create and hot plug in a device to a guest? > >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > >>>>>>>>>>>>> machines? > >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > >>>>>>>>>>>> to create new devices. > >>>>>>>>>>>> > >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > >>>>>>>>>>>> entire host is for users. Of course, there are situations where the > >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU. > >>>>>>>>>>>> > >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager > >>>>>>>>>>>> will plug a device to the host from the DPU. > >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > >>>>>>>>>>> to the host. > >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means > >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be > >>>>>>>>>>> a host driver help handling an interrupt/probe. > >>>>>>>>>>> > >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > >>>>>>>>>> > >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > >>>>>>>>>>> the host by spanning the device across the PCI hierarchy. > >>>>>>>>>>> > >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > >>>>>>>>>> > >>>>>>>>>>> To address an ADI, there is only a device_id. > >>>>>>>>>>> > >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure. > >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue. > >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > >>>>>>>>>> > >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of > >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > >>>>>>>>>>> its own? > >>>>>>>>>>> > >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us. > >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > >>>>>>>>>> > >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > >>>>>>>>>> You might find a Linux reference model useful at [1]. > >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices. > >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > >>>>>>>>>> > >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > >>>>>>>>>> > >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the > >>>>>>>>> driver to query all existing SIOV ADIs of a device, > >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you > >>>>>>>>> @Xuan? > >>>>>>>> Could I have your plan for this? > >>>>>>>> > >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such > >>>>>>>> as mac, ip, etc. > >>>>>>> I think the query commands for SIOV is a little more complex, e.g., > >>>>>>> need to report device type and its scale(e.g., features, mq). > >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood. > >>>>>>> > >>>>>>> We have discussed implementation a config interrupt to report new > >>>>>>> created / deleted > >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related > >>>>>>> information, > >>>>>>> my rough approach of the process is: > >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id > >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of > >>>>>>> ADIs in a [dev_id....dev_id2] range. > >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices. > >>>>>> > >>>>>>> I am not sure whether a NIC stores its IP > >>>>>> IP is the other topic. I want the Admin Queue manage the switch. > >>>>>> So the switch know about the IP of every device, and the > >>>>>> Admin Queue will has the ability to config the IP of the device inside the > >>>>>> switch. > >>>>> DPU onboard switch? OVS? Does it beyond virtio spec? > >>>> YES. > >>> Adding Washizu. > >>> > >>> We can have a switch/dpa defined in the networking device for sure. > >> Yes, I think we should introduce that for the sr-iov. Or for other. > > This should be a general one as a switch should be transport independent. > > > >> I would like to know who is doing this? > > Washizu, could you confirm if you want to do this or not? > > > Does this mean adding a switch definition to the virtio spec? > > > If so, it will be necessary for the implementation of my plan, > > but it may take time (probably several months?) to get started, > > as I'm currently working on another task (virtio-net SR-IOV feature in > qemu). > > Anyone is welcome to work on adding the switch definition in the meantime, > > it's completely fine with me. > > I think I'll work on that if no one has finished the work. OK, I got. Because we have a need in this area, I will push the work in this area. Thanks. > > > > > > >> Another question, @Jason are you referring to a new device type or a > >> new virtio-net feature. > > Extending virtio-net should be fine, did you see any issues for this? > > > >>>> For SIOV, I think this is MUST. > >>> A learning bridge would be fine as a starter. It's better not to > >>> couple new scalable capability with any device specific features. > >>> > >>>> Maybe you have one simple implementation. > >>>> But you have to solve the IP steering. So admin queue should has the ability > >>>> to config the IP steering. > >>> I think not. Those L2/LN tables/filters are networking specific. > >> Let us assume that there is a switch/bridge firstly. > >> The VFs may be passed to different VMs. > >> > >> I also think this is the networking specific. But I want to config > >> the ip for every vf from the pf. > > What do you mean by ip here (e.g who is the user for this ip?) > > > >> Because the user of the vf may be unreliable. > >> We need a manager to config the ip for every vf. > > Did you mean you're using a tunnel or not? > > > >> > >>> Control virtqueue is better than admin virtqueue here. > >> by cq? > >> > >> What case? > > We've already used control virtqueue for steering. > > > > Thanks > > > >> Thanks. > >> > >>> Thanks > >>> > >>>> Thanks. > >>>> > >>>>>> > >>>>>> Thanks. > >>>>>> > >>>>>> > >>>>>>> Thanks > >>>>>>>> Thanks. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> Thanks > >>>>>>>>> > >>>>>>>> This publicly archived list offers a means to provide input to the > >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>>>>> > >>>>>>>> In order to verify user consent to the Feedback License terms and > >>>>>>>> to minimize spam in the list archive, subscription is required > >>>>>>>> before posting. > >>>>>>>> > >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>>>>> > >>>>> > >>>>> This publicly archived list offers a means to provide input to the > >>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>> > >>>>> In order to verify user consent to the Feedback License terms and > >>>>> to minimize spam in the list archive, subscription is required > >>>>> before posting. > >>>>> > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>> > >>>> This publicly archived list offers a means to provide input to the > >>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>> > >>>> In order to verify user consent to the Feedback License terms and > >>>> to minimize spam in the list archive, subscription is required > >>>> before posting. > >>>> > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>> List help: virtio-comment-help@lists.oasis-open.org > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>> Join OASIS: https://www.oasis-open.org/join/ > >>>> This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue [not found] ` <aafe1885-0ec2-66ca-4511-f2606bc881ee@gmail.com> 2023-08-02 6:13 ` Xuan Zhuo @ 2023-08-02 6:15 ` Jason Wang 2023-08-02 6:34 ` Xuan Zhuo 1 sibling, 1 reply; 49+ messages in thread From: Jason Wang @ 2023-08-02 6:15 UTC (permalink / raw) To: Yui Washizu Cc: Xuan Zhuo, Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit On Wed, Aug 2, 2023 at 2:01 PM Yui Washizu <yui.washidu@gmail.com> wrote: > > > On 2023/07/27 17:28, Jason Wang wrote: > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>> > >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan > >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM > >>>>>>>>>>> > >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > >>>>>>>>>>> backend? > >>>>>>>>>> Yes, why not. > >>>>>>>>>> > >>>>>>>>>>>>>> The key point is who want to create a new device. > >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs. > >>>>>>>>>>>>> > >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > >>>>>>>>>> Xuan partly explained it. > >>>>>>>>>> > >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > >>>>>>>>>>>>> wants to create and hot plug in a device to a guest? > >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > >>>>>>>>>>>>> machines? > >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > >>>>>>>>>>>> to create new devices. > >>>>>>>>>>>> > >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > >>>>>>>>>>>> entire host is for users. Of course, there are situations where the > >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU. > >>>>>>>>>>>> > >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager > >>>>>>>>>>>> will plug a device to the host from the DPU. > >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > >>>>>>>>>>> to the host. > >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means > >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be > >>>>>>>>>>> a host driver help handling an interrupt/probe. > >>>>>>>>>>> > >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > >>>>>>>>>> > >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > >>>>>>>>>>> the host by spanning the device across the PCI hierarchy. > >>>>>>>>>>> > >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > >>>>>>>>>> > >>>>>>>>>>> To address an ADI, there is only a device_id. > >>>>>>>>>>> > >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure. > >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue. > >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > >>>>>>>>>> > >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of > >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > >>>>>>>>>>> its own? > >>>>>>>>>>> > >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us. > >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > >>>>>>>>>> > >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > >>>>>>>>>> You might find a Linux reference model useful at [1]. > >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices. > >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > >>>>>>>>>> > >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > >>>>>>>>>> > >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the > >>>>>>>>> driver to query all existing SIOV ADIs of a device, > >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you > >>>>>>>>> @Xuan? > >>>>>>>> Could I have your plan for this? > >>>>>>>> > >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such > >>>>>>>> as mac, ip, etc. > >>>>>>> I think the query commands for SIOV is a little more complex, e.g., > >>>>>>> need to report device type and its scale(e.g., features, mq). > >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood. > >>>>>>> > >>>>>>> We have discussed implementation a config interrupt to report new > >>>>>>> created / deleted > >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related > >>>>>>> information, > >>>>>>> my rough approach of the process is: > >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id > >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of > >>>>>>> ADIs in a [dev_id....dev_id2] range. > >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices. > >>>>>> > >>>>>>> I am not sure whether a NIC stores its IP > >>>>>> IP is the other topic. I want the Admin Queue manage the switch. > >>>>>> So the switch know about the IP of every device, and the > >>>>>> Admin Queue will has the ability to config the IP of the device inside the > >>>>>> switch. > >>>>> DPU onboard switch? OVS? Does it beyond virtio spec? > >>>> YES. > >>> Adding Washizu. > >>> > >>> We can have a switch/dpa defined in the networking device for sure. > >> Yes, I think we should introduce that for the sr-iov. Or for other. > > This should be a general one as a switch should be transport independent. > > > >> I would like to know who is doing this? > > Washizu, could you confirm if you want to do this or not? > > > Does this mean adding a switch definition to the virtio spec? > > > If so, it will be necessary for the implementation of my plan, > > but it may take time (probably several months?) to get started, > > as I'm currently working on another task (virtio-net SR-IOV feature in > qemu). > > Anyone is welcome to work on adding the switch definition in the meantime, > > it's completely fine with me. > > I think I'll work on that if no one has finished the work. Ok, great, I think we need to start this by considering reusing one of the existing DPA spec. (For example, ofdpa? or any other?) Thanks > > > > > > >> Another question, @Jason are you referring to a new device type or a > >> new virtio-net feature. > > Extending virtio-net should be fine, did you see any issues for this? > > > >>>> For SIOV, I think this is MUST. > >>> A learning bridge would be fine as a starter. It's better not to > >>> couple new scalable capability with any device specific features. > >>> > >>>> Maybe you have one simple implementation. > >>>> But you have to solve the IP steering. So admin queue should has the ability > >>>> to config the IP steering. > >>> I think not. Those L2/LN tables/filters are networking specific. > >> Let us assume that there is a switch/bridge firstly. > >> The VFs may be passed to different VMs. > >> > >> I also think this is the networking specific. But I want to config > >> the ip for every vf from the pf. > > What do you mean by ip here (e.g who is the user for this ip?) > > > >> Because the user of the vf may be unreliable. > >> We need a manager to config the ip for every vf. > > Did you mean you're using a tunnel or not? > > > >> > >>> Control virtqueue is better than admin virtqueue here. > >> by cq? > >> > >> What case? > > We've already used control virtqueue for steering. > > > > Thanks > > > >> Thanks. > >> > >>> Thanks > >>> > >>>> Thanks. > >>>> > >>>>>> > >>>>>> Thanks. > >>>>>> > >>>>>> > >>>>>>> Thanks > >>>>>>>> Thanks. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> Thanks > >>>>>>>>> > >>>>>>>> This publicly archived list offers a means to provide input to the > >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>>>>> > >>>>>>>> In order to verify user consent to the Feedback License terms and > >>>>>>>> to minimize spam in the list archive, subscription is required > >>>>>>>> before posting. > >>>>>>>> > >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>>>>> > >>>>> > >>>>> This publicly archived list offers a means to provide input to the > >>>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>>> > >>>>> In order to verify user consent to the Feedback License terms and > >>>>> to minimize spam in the list archive, subscription is required > >>>>> before posting. > >>>>> > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>>> List help: virtio-comment-help@lists.oasis-open.org > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>>> Join OASIS: https://www.oasis-open.org/join/ > >>>>> > >>>> This publicly archived list offers a means to provide input to the > >>>> OASIS Virtual I/O Device (VIRTIO) TC. > >>>> > >>>> In order to verify user consent to the Feedback License terms and > >>>> to minimize spam in the list archive, subscription is required > >>>> before posting. > >>>> > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > >>>> List help: virtio-comment-help@lists.oasis-open.org > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > >>>> Committee: https://www.oasis-open.org/committees/virtio/ > >>>> Join OASIS: https://www.oasis-open.org/join/ > >>>> > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-08-02 6:15 ` Jason Wang @ 2023-08-02 6:34 ` Xuan Zhuo 2023-08-02 6:53 ` Jason Wang 0 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-08-02 6:34 UTC (permalink / raw) To: Jason Wang Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Yui Washizu On Wed, 2 Aug 2023 14:15:55 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Wed, Aug 2, 2023 at 2:01 PM Yui Washizu <yui.washidu@gmail.com> wrote: > > > > > > On 2023/07/27 17:28, Jason Wang wrote: > > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > >>>>> > > >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan > > >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM > > >>>>>>>>>>> > > >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > >>>>>>>>>>> backend? > > >>>>>>>>>> Yes, why not. > > >>>>>>>>>> > > >>>>>>>>>>>>>> The key point is who want to create a new device. > > >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs. > > >>>>>>>>>>>>> > > >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > >>>>>>>>>> Xuan partly explained it. > > >>>>>>>>>> > > >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > >>>>>>>>>>>>> wants to create and hot plug in a device to a guest? > > >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > >>>>>>>>>>>>> machines? > > >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > >>>>>>>>>>>> to create new devices. > > >>>>>>>>>>>> > > >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > >>>>>>>>>>>> entire host is for users. Of course, there are situations where the > > >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > >>>>>>>>>>>> > > >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager > > >>>>>>>>>>>> will plug a device to the host from the DPU. > > >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > >>>>>>>>>>> to the host. > > >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means > > >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be > > >>>>>>>>>>> a host driver help handling an interrupt/probe. > > >>>>>>>>>>> > > >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > >>>>>>>>>> > > >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > >>>>>>>>>>> the host by spanning the device across the PCI hierarchy. > > >>>>>>>>>>> > > >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > >>>>>>>>>> > > >>>>>>>>>>> To address an ADI, there is only a device_id. > > >>>>>>>>>>> > > >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue. > > >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > >>>>>>>>>> > > >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of > > >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > >>>>>>>>>>> its own? > > >>>>>>>>>>> > > >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us. > > >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > >>>>>>>>>> > > >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > >>>>>>>>>> You might find a Linux reference model useful at [1]. > > >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices. > > >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > >>>>>>>>>> > > >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > >>>>>>>>>> > > >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the > > >>>>>>>>> driver to query all existing SIOV ADIs of a device, > > >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you > > >>>>>>>>> @Xuan? > > >>>>>>>> Could I have your plan for this? > > >>>>>>>> > > >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such > > >>>>>>>> as mac, ip, etc. > > >>>>>>> I think the query commands for SIOV is a little more complex, e.g., > > >>>>>>> need to report device type and its scale(e.g., features, mq). > > >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood. > > >>>>>>> > > >>>>>>> We have discussed implementation a config interrupt to report new > > >>>>>>> created / deleted > > >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related > > >>>>>>> information, > > >>>>>>> my rough approach of the process is: > > >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id > > >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of > > >>>>>>> ADIs in a [dev_id....dev_id2] range. > > >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices. > > >>>>>> > > >>>>>>> I am not sure whether a NIC stores its IP > > >>>>>> IP is the other topic. I want the Admin Queue manage the switch. > > >>>>>> So the switch know about the IP of every device, and the > > >>>>>> Admin Queue will has the ability to config the IP of the device inside the > > >>>>>> switch. > > >>>>> DPU onboard switch? OVS? Does it beyond virtio spec? > > >>>> YES. > > >>> Adding Washizu. > > >>> > > >>> We can have a switch/dpa defined in the networking device for sure. > > >> Yes, I think we should introduce that for the sr-iov. Or for other. > > > This should be a general one as a switch should be transport independent. > > > > > >> I would like to know who is doing this? > > > Washizu, could you confirm if you want to do this or not? > > > > > > Does this mean adding a switch definition to the virtio spec? > > > > > > If so, it will be necessary for the implementation of my plan, > > > > but it may take time (probably several months?) to get started, > > > > as I'm currently working on another task (virtio-net SR-IOV feature in > > qemu). > > > > Anyone is welcome to work on adding the switch definition in the meantime, > > > > it's completely fine with me. > > > > I think I'll work on that if no one has finished the work. > > Ok, great, I think we need to start this by considering reusing one of > the existing DPA spec. (For example, ofdpa? or any other?) We have a need on this area. I want to start this work now. Could you give me the link of the spec? Thanks > > Thanks > > > > > > > > > > > > >> Another question, @Jason are you referring to a new device type or a > > >> new virtio-net feature. > > > Extending virtio-net should be fine, did you see any issues for this? > > > > > >>>> For SIOV, I think this is MUST. > > >>> A learning bridge would be fine as a starter. It's better not to > > >>> couple new scalable capability with any device specific features. > > >>> > > >>>> Maybe you have one simple implementation. > > >>>> But you have to solve the IP steering. So admin queue should has the ability > > >>>> to config the IP steering. > > >>> I think not. Those L2/LN tables/filters are networking specific. > > >> Let us assume that there is a switch/bridge firstly. > > >> The VFs may be passed to different VMs. > > >> > > >> I also think this is the networking specific. But I want to config > > >> the ip for every vf from the pf. > > > What do you mean by ip here (e.g who is the user for this ip?) > > > > > >> Because the user of the vf may be unreliable. > > >> We need a manager to config the ip for every vf. > > > Did you mean you're using a tunnel or not? > > > > > >> > > >>> Control virtqueue is better than admin virtqueue here. > > >> by cq? > > >> > > >> What case? > > > We've already used control virtqueue for steering. > > > > > > Thanks > > > > > >> Thanks. > > >> > > >>> Thanks > > >>> > > >>>> Thanks. > > >>>> > > >>>>>> > > >>>>>> Thanks. > > >>>>>> > > >>>>>> > > >>>>>>> Thanks > > >>>>>>>> Thanks. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>>> Thanks > > >>>>>>>>> > > >>>>>>>> This publicly archived list offers a means to provide input to the > > >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > > >>>>>>>> > > >>>>>>>> In order to verify user consent to the Feedback License terms and > > >>>>>>>> to minimize spam in the list archive, subscription is required > > >>>>>>>> before posting. > > >>>>>>>> > > >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org > > >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > > >>>>>>>> Join OASIS: https://www.oasis-open.org/join/ > > >>>>>>>> > > >>>>> > > >>>>> This publicly archived list offers a means to provide input to the > > >>>>> OASIS Virtual I/O Device (VIRTIO) TC. > > >>>>> > > >>>>> In order to verify user consent to the Feedback License terms and > > >>>>> to minimize spam in the list archive, subscription is required > > >>>>> before posting. > > >>>>> > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > >>>>> List help: virtio-comment-help@lists.oasis-open.org > > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/ > > >>>>> Join OASIS: https://www.oasis-open.org/join/ > > >>>>> > > >>>> This publicly archived list offers a means to provide input to the > > >>>> OASIS Virtual I/O Device (VIRTIO) TC. > > >>>> > > >>>> In order to verify user consent to the Feedback License terms and > > >>>> to minimize spam in the list archive, subscription is required > > >>>> before posting. > > >>>> > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > >>>> List help: virtio-comment-help@lists.oasis-open.org > > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > >>>> Committee: https://www.oasis-open.org/committees/virtio/ > > >>>> Join OASIS: https://www.oasis-open.org/join/ > > >>>> > > > > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-08-02 6:34 ` Xuan Zhuo @ 2023-08-02 6:53 ` Jason Wang 2023-08-02 6:55 ` Xuan Zhuo 0 siblings, 1 reply; 49+ messages in thread From: Jason Wang @ 2023-08-02 6:53 UTC (permalink / raw) To: Xuan Zhuo Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Yui Washizu On Wed, Aug 2, 2023 at 2:37 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Wed, 2 Aug 2023 14:15:55 +0800, Jason Wang <jasowang@redhat.com> wrote: > > On Wed, Aug 2, 2023 at 2:01 PM Yui Washizu <yui.washidu@gmail.com> wrote: > > > > > > > > > On 2023/07/27 17:28, Jason Wang wrote: > > > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > > >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > >>>>> > > > >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan > > > >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > >>>>>>>>>>> > > > >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > >>>>>>>>>>> backend? > > > >>>>>>>>>> Yes, why not. > > > >>>>>>>>>> > > > >>>>>>>>>>>>>> The key point is who want to create a new device. > > > >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs. > > > >>>>>>>>>>>>> > > > >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > >>>>>>>>>> Xuan partly explained it. > > > >>>>>>>>>> > > > >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > >>>>>>>>>>>>> wants to create and hot plug in a device to a guest? > > > >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > >>>>>>>>>>>>> machines? > > > >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > >>>>>>>>>>>> to create new devices. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > >>>>>>>>>>>> entire host is for users. Of course, there are situations where the > > > >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > >>>>>>>>>>>> will plug a device to the host from the DPU. > > > >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > >>>>>>>>>>> to the host. > > > >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means > > > >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be > > > >>>>>>>>>>> a host driver help handling an interrupt/probe. > > > >>>>>>>>>>> > > > >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > >>>>>>>>>> > > > >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > >>>>>>>>>>> the host by spanning the device across the PCI hierarchy. > > > >>>>>>>>>>> > > > >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > >>>>>>>>>> > > > >>>>>>>>>>> To address an ADI, there is only a device_id. > > > >>>>>>>>>>> > > > >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue. > > > >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > >>>>>>>>>> > > > >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > >>>>>>>>>>> its own? > > > >>>>>>>>>>> > > > >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us. > > > >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > >>>>>>>>>> > > > >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > >>>>>>>>>> You might find a Linux reference model useful at [1]. > > > >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices. > > > >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > >>>>>>>>>> > > > >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > >>>>>>>>>> > > > >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the > > > >>>>>>>>> driver to query all existing SIOV ADIs of a device, > > > >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you > > > >>>>>>>>> @Xuan? > > > >>>>>>>> Could I have your plan for this? > > > >>>>>>>> > > > >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such > > > >>>>>>>> as mac, ip, etc. > > > >>>>>>> I think the query commands for SIOV is a little more complex, e.g., > > > >>>>>>> need to report device type and its scale(e.g., features, mq). > > > >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood. > > > >>>>>>> > > > >>>>>>> We have discussed implementation a config interrupt to report new > > > >>>>>>> created / deleted > > > >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related > > > >>>>>>> information, > > > >>>>>>> my rough approach of the process is: > > > >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id > > > >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of > > > >>>>>>> ADIs in a [dev_id....dev_id2] range. > > > >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices. > > > >>>>>> > > > >>>>>>> I am not sure whether a NIC stores its IP > > > >>>>>> IP is the other topic. I want the Admin Queue manage the switch. > > > >>>>>> So the switch know about the IP of every device, and the > > > >>>>>> Admin Queue will has the ability to config the IP of the device inside the > > > >>>>>> switch. > > > >>>>> DPU onboard switch? OVS? Does it beyond virtio spec? > > > >>>> YES. > > > >>> Adding Washizu. > > > >>> > > > >>> We can have a switch/dpa defined in the networking device for sure. > > > >> Yes, I think we should introduce that for the sr-iov. Or for other. > > > > This should be a general one as a switch should be transport independent. > > > > > > > >> I would like to know who is doing this? > > > > Washizu, could you confirm if you want to do this or not? > > > > > > > > > Does this mean adding a switch definition to the virtio spec? > > > > > > > > > If so, it will be necessary for the implementation of my plan, > > > > > > but it may take time (probably several months?) to get started, > > > > > > as I'm currently working on another task (virtio-net SR-IOV feature in > > > qemu). > > > > > > Anyone is welcome to work on adding the switch definition in the meantime, > > > > > > it's completely fine with me. > > > > > > I think I'll work on that if no one has finished the work. > > > > Ok, great, I think we need to start this by considering reusing one of > > the existing DPA spec. (For example, ofdpa? or any other?) > > We have a need on this area. I want to start this work now. > > Could you give me the link of the spec? I'm not sure this is the best one, we can hear from others. I mention it since it has an emulation code that is done in Qemu (rocker switch). The spec is: https://docs.broadcom.com/doc/12378911 (not sure this is the recent one though). Thanks > > Thanks > > > > > > > Thanks > > > > > > > > > > > > > > > > > > >> Another question, @Jason are you referring to a new device type or a > > > >> new virtio-net feature. > > > > Extending virtio-net should be fine, did you see any issues for this? > > > > > > > >>>> For SIOV, I think this is MUST. > > > >>> A learning bridge would be fine as a starter. It's better not to > > > >>> couple new scalable capability with any device specific features. > > > >>> > > > >>>> Maybe you have one simple implementation. > > > >>>> But you have to solve the IP steering. So admin queue should has the ability > > > >>>> to config the IP steering. > > > >>> I think not. Those L2/LN tables/filters are networking specific. > > > >> Let us assume that there is a switch/bridge firstly. > > > >> The VFs may be passed to different VMs. > > > >> > > > >> I also think this is the networking specific. But I want to config > > > >> the ip for every vf from the pf. > > > > What do you mean by ip here (e.g who is the user for this ip?) > > > > > > > >> Because the user of the vf may be unreliable. > > > >> We need a manager to config the ip for every vf. > > > > Did you mean you're using a tunnel or not? > > > > > > > >> > > > >>> Control virtqueue is better than admin virtqueue here. > > > >> by cq? > > > >> > > > >> What case? > > > > We've already used control virtqueue for steering. > > > > > > > > Thanks > > > > > > > >> Thanks. > > > >> > > > >>> Thanks > > > >>> > > > >>>> Thanks. > > > >>>> > > > >>>>>> > > > >>>>>> Thanks. > > > >>>>>> > > > >>>>>> > > > >>>>>>> Thanks > > > >>>>>>>> Thanks. > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>> > > > >>>>>>>>> Thanks > > > >>>>>>>>> > > > >>>>>>>> This publicly archived list offers a means to provide input to the > > > >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > > > >>>>>>>> > > > >>>>>>>> In order to verify user consent to the Feedback License terms and > > > >>>>>>>> to minimize spam in the list archive, subscription is required > > > >>>>>>>> before posting. > > > >>>>>>>> > > > >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org > > > >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > > > >>>>>>>> Join OASIS: https://www.oasis-open.org/join/ > > > >>>>>>>> > > > >>>>> > > > >>>>> This publicly archived list offers a means to provide input to the > > > >>>>> OASIS Virtual I/O Device (VIRTIO) TC. > > > >>>>> > > > >>>>> In order to verify user consent to the Feedback License terms and > > > >>>>> to minimize spam in the list archive, subscription is required > > > >>>>> before posting. > > > >>>>> > > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > >>>>> List help: virtio-comment-help@lists.oasis-open.org > > > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/ > > > >>>>> Join OASIS: https://www.oasis-open.org/join/ > > > >>>>> > > > >>>> This publicly archived list offers a means to provide input to the > > > >>>> OASIS Virtual I/O Device (VIRTIO) TC. > > > >>>> > > > >>>> In order to verify user consent to the Feedback License terms and > > > >>>> to minimize spam in the list archive, subscription is required > > > >>>> before posting. > > > >>>> > > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > >>>> List help: virtio-comment-help@lists.oasis-open.org > > > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > >>>> Committee: https://www.oasis-open.org/committees/virtio/ > > > >>>> Join OASIS: https://www.oasis-open.org/join/ > > > >>>> > > > > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-08-02 6:53 ` Jason Wang @ 2023-08-02 6:55 ` Xuan Zhuo 0 siblings, 0 replies; 49+ messages in thread From: Xuan Zhuo @ 2023-08-02 6:55 UTC (permalink / raw) To: Jason Wang Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Yui Washizu On Wed, 2 Aug 2023 14:53:28 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Wed, Aug 2, 2023 at 2:37 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > On Wed, 2 Aug 2023 14:15:55 +0800, Jason Wang <jasowang@redhat.com> wrote: > > > On Wed, Aug 2, 2023 at 2:01 PM Yui Washizu <yui.washidu@gmail.com> wrote: > > > > > > > > > > > > On 2023/07/27 17:28, Jason Wang wrote: > > > > > On Thu, Jul 27, 2023 at 4:20 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > >> On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > > > >>> On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > >>>> On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > >>>>> > > > > >>>>> On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > >>>>>> On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > >>>>>>> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > > >>>>>>>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > >>>>>>>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > > >>>>>>>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > > >>>>>>>>>>> open.org> On Behalf Of Zhu, Lingshan > > > > >>>>>>>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > > >>>>>>>>>>> > > > > >>>>>>>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > > >>>>>>>>>>> backend? > > > > >>>>>>>>>> Yes, why not. > > > > >>>>>>>>>> > > > > >>>>>>>>>>>>>> The key point is who want to create a new device. > > > > >>>>>>>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > > >>>>>>>>>>>>> sure the orchestration SW is aware of their device IDs. > > > > >>>>>>>>>>>>> > > > > >>>>>>>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > > >>>>>>>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > > >>>>>>>>>> Xuan partly explained it. > > > > >>>>>>>>>> > > > > >>>>>>>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > > >>>>>>>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > > >>>>>>>>>>>>> wants to create and hot plug in a device to a guest? > > > > >>>>>>>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > > >>>>>>>>>>>>> machines? > > > > >>>>>>>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > > >>>>>>>>>>>> to create new devices. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > > >>>>>>>>>>>> entire host is for users. Of course, there are situations where the > > > > >>>>>>>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > > >>>>>>>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > > >>>>>>>>>>>> > > > > >>>>>>>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > > >>>>>>>>>>>> will plug a device to the host from the DPU. > > > > >>>>>>>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > > >>>>>>>>>>> to the host. > > > > >>>>>>>>>>> However you only owns the DPU and the customer owns the host, that means > > > > >>>>>>>>>>> this creation and hot plug must be transparent to the host and there may not be > > > > >>>>>>>>>>> a host driver help handling an interrupt/probe. > > > > >>>>>>>>>>> > > > > >>>>>>>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > > >>>>>>>>>> > > > > >>>>>>>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > > >>>>>>>>>>> the host by spanning the device across the PCI hierarchy. > > > > >>>>>>>>>>> > > > > >>>>>>>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > > >>>>>>>>>> > > > > >>>>>>>>>>> To address an ADI, there is only a device_id. > > > > >>>>>>>>>>> > > > > >>>>>>>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > > >>>>>>>>>> I cannot speak more about it in this forum due to other logistics issue. > > > > >>>>>>>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > > >>>>>>>>>> > > > > >>>>>>>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > > >>>>>>>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > > >>>>>>>>>>> its own? > > > > >>>>>>>>>>> > > > > >>>>>>>>>> It is a virtio nic, blk and other virtio devices for us. > > > > >>>>>>>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > > >>>>>>>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > >>>>>>>>>> > > > > >>>>>>>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > > >>>>>>>>>> You might find a Linux reference model useful at [1]. > > > > >>>>>>>>>> A usage model already exists in one OS and in use for non virtio devices. > > > > >>>>>>>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > >>>>>>>>>> > > > > >>>>>>>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > >>>>>>>>>> > > > > >>>>>>>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > > >>>>>>>>> well I see SF facing the similar challenge, I can add a command for the > > > > >>>>>>>>> driver to query all existing SIOV ADIs of a device, > > > > >>>>>>>>> and the device return ADIs id and status. Looks good? and work for you > > > > >>>>>>>>> @Xuan? > > > > >>>>>>>> Could I have your plan for this? > > > > >>>>>>>> > > > > >>>>>>>> If you do not mind, I'd like to add a command to query VF's info. Such > > > > >>>>>>>> as mac, ip, etc. > > > > >>>>>>> I think the query commands for SIOV is a little more complex, e.g., > > > > >>>>>>> need to report device type and its scale(e.g., features, mq). > > > > >>>>>>> There can be thousands of SIOV ADIs and we don't want output flood. > > > > >>>>>>> > > > > >>>>>>> We have discussed implementation a config interrupt to report new > > > > >>>>>>> created / deleted > > > > >>>>>>> ADIs on the DPU side, therefore there must be a cap contains related > > > > >>>>>>> information, > > > > >>>>>>> my rough approach of the process is: > > > > >>>>>>> 1) a cap contains the total number of existing ADIs and the max dev id > > > > >>>>>>> 2) driver queries detailed information of a certain ADI or a bunch of > > > > >>>>>>> ADIs in a [dev_id....dev_id2] range. > > > > >>>>>> Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > >>>>>> > > > > >>>>>>> I am not sure whether a NIC stores its IP > > > > >>>>>> IP is the other topic. I want the Admin Queue manage the switch. > > > > >>>>>> So the switch know about the IP of every device, and the > > > > >>>>>> Admin Queue will has the ability to config the IP of the device inside the > > > > >>>>>> switch. > > > > >>>>> DPU onboard switch? OVS? Does it beyond virtio spec? > > > > >>>> YES. > > > > >>> Adding Washizu. > > > > >>> > > > > >>> We can have a switch/dpa defined in the networking device for sure. > > > > >> Yes, I think we should introduce that for the sr-iov. Or for other. > > > > > This should be a general one as a switch should be transport independent. > > > > > > > > > >> I would like to know who is doing this? > > > > > Washizu, could you confirm if you want to do this or not? > > > > > > > > > > > > Does this mean adding a switch definition to the virtio spec? > > > > > > > > > > > > If so, it will be necessary for the implementation of my plan, > > > > > > > > but it may take time (probably several months?) to get started, > > > > > > > > as I'm currently working on another task (virtio-net SR-IOV feature in > > > > qemu). > > > > > > > > Anyone is welcome to work on adding the switch definition in the meantime, > > > > > > > > it's completely fine with me. > > > > > > > > I think I'll work on that if no one has finished the work. > > > > > > Ok, great, I think we need to start this by considering reusing one of > > > the existing DPA spec. (For example, ofdpa? or any other?) > > > > We have a need on this area. I want to start this work now. > > > > Could you give me the link of the spec? > > I'm not sure this is the best one, we can hear from others. I mention > it since it has an emulation code that is done in Qemu (rocker > switch). > > The spec is: > > https://docs.broadcom.com/doc/12378911 OK. I will study it. But I think we can start with a simple model. For virtio-net, we don't need a full-featured switch. I would mainly consider the SR-IOV case. Thanks. > > (not sure this is the recent one though). > > Thanks > > > > > Thanks > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > > > > > >> Another question, @Jason are you referring to a new device type or a > > > > >> new virtio-net feature. > > > > > Extending virtio-net should be fine, did you see any issues for this? > > > > > > > > > >>>> For SIOV, I think this is MUST. > > > > >>> A learning bridge would be fine as a starter. It's better not to > > > > >>> couple new scalable capability with any device specific features. > > > > >>> > > > > >>>> Maybe you have one simple implementation. > > > > >>>> But you have to solve the IP steering. So admin queue should has the ability > > > > >>>> to config the IP steering. > > > > >>> I think not. Those L2/LN tables/filters are networking specific. > > > > >> Let us assume that there is a switch/bridge firstly. > > > > >> The VFs may be passed to different VMs. > > > > >> > > > > >> I also think this is the networking specific. But I want to config > > > > >> the ip for every vf from the pf. > > > > > What do you mean by ip here (e.g who is the user for this ip?) > > > > > > > > > >> Because the user of the vf may be unreliable. > > > > >> We need a manager to config the ip for every vf. > > > > > Did you mean you're using a tunnel or not? > > > > > > > > > >> > > > > >>> Control virtqueue is better than admin virtqueue here. > > > > >> by cq? > > > > >> > > > > >> What case? > > > > > We've already used control virtqueue for steering. > > > > > > > > > > Thanks > > > > > > > > > >> Thanks. > > > > >> > > > > >>> Thanks > > > > >>> > > > > >>>> Thanks. > > > > >>>> > > > > >>>>>> > > > > >>>>>> Thanks. > > > > >>>>>> > > > > >>>>>> > > > > >>>>>>> Thanks > > > > >>>>>>>> Thanks. > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>>> Thanks > > > > >>>>>>>>> > > > > >>>>>>>> This publicly archived list offers a means to provide input to the > > > > >>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > >>>>>>>> > > > > >>>>>>>> In order to verify user consent to the Feedback License terms and > > > > >>>>>>>> to minimize spam in the list archive, subscription is required > > > > >>>>>>>> before posting. > > > > >>>>>>>> > > > > >>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > >>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > >>>>>>>> List help: virtio-comment-help@lists.oasis-open.org > > > > >>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > >>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > >>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > >>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > >>>>>>>> Join OASIS: https://www.oasis-open.org/join/ > > > > >>>>>>>> > > > > >>>>> > > > > >>>>> This publicly archived list offers a means to provide input to the > > > > >>>>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > >>>>> > > > > >>>>> In order to verify user consent to the Feedback License terms and > > > > >>>>> to minimize spam in the list archive, subscription is required > > > > >>>>> before posting. > > > > >>>>> > > > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > >>>>> List help: virtio-comment-help@lists.oasis-open.org > > > > >>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > >>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > >>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > >>>>> Join OASIS: https://www.oasis-open.org/join/ > > > > >>>>> > > > > >>>> This publicly archived list offers a means to provide input to the > > > > >>>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > >>>> > > > > >>>> In order to verify user consent to the Feedback License terms and > > > > >>>> to minimize spam in the list archive, subscription is required > > > > >>>> before posting. > > > > >>>> > > > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > >>>> List help: virtio-comment-help@lists.oasis-open.org > > > > >>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > >>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > >>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > >>>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > >>>> Join OASIS: https://www.oasis-open.org/join/ > > > > >>>> > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > In order to verify user consent to the Feedback License terms and > > > to minimize spam in the list archive, subscription is required > > > before posting. > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > List help: virtio-comment-help@lists.oasis-open.org > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-27 8:03 ` Jason Wang 2023-07-27 8:07 ` Xuan Zhuo @ 2023-07-28 6:09 ` Xuan Zhuo 2023-07-31 1:20 ` Jason Wang 1 sibling, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-28 6:09 UTC (permalink / raw) To: Jason Wang Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Washizu Yui On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > >> > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > >>>>>> > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > >>>>>> backend? > > > >>>>> Yes, why not. > > > >>>>> > > > >>>>>>>>> The key point is who want to create a new device. > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > > >>>>>>>> > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > >>>>> Xuan partly explained it. > > > >>>>> > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > >>>>>>>> wants to create and hot plug in a device to a guest? > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > >>>>>>>> machines? > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > >>>>>>> to create new devices. > > > >>>>>>> > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > >>>>>>> entire host is for users. Of course, there are situations where the > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > >>>>>>> > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > >>>>>>> will plug a device to the host from the DPU. > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > >>>>>> to the host. > > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > > >>>>>> a host driver help handling an interrupt/probe. > > > >>>>>> > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > >>>>> > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > >>>>>> the host by spanning the device across the PCI hierarchy. > > > >>>>>> > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > >>>>> > > > >>>>>> To address an ADI, there is only a device_id. > > > >>>>>> > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > >>>>> > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > >>>>>> its own? > > > >>>>>> > > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > >>>>> > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > >>>>> You might find a Linux reference model useful at [1]. > > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > >>>>> > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > >>>>> > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > >>>> well I see SF facing the similar challenge, I can add a command for the > > > >>>> driver to query all existing SIOV ADIs of a device, > > > >>>> and the device return ADIs id and status. Looks good? and work for you > > > >>>> @Xuan? > > > >>> Could I have your plan for this? > > > >>> > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > > >>> as mac, ip, etc. > > > >> I think the query commands for SIOV is a little more complex, e.g., > > > >> need to report device type and its scale(e.g., features, mq). > > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > > >> > > > >> We have discussed implementation a config interrupt to report new > > > >> created / deleted > > > >> ADIs on the DPU side, therefore there must be a cap contains related > > > >> information, > > > >> my rough approach of the process is: > > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > > >> ADIs in a [dev_id....dev_id2] range. > > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > > > >> I am not sure whether a NIC stores its IP > > > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > > So the switch know about the IP of every device, and the > > > > Admin Queue will has the ability to config the IP of the device inside the > > > > switch. > > > DPU onboard switch? OVS? Does it beyond virtio spec? > > > > YES. > > Adding Washizu. > > We can have a switch/dpa defined in the networking device for sure. > > > > > For SIOV, I think this is MUST. > > A learning bridge would be fine as a starter. It's better not to > couple new scalable capability with any device specific features. > > > Maybe you have one simple implementation. > > But you have to solve the IP steering. So admin queue should has the ability > > to config the IP steering. > > I think not. Those L2/LN tables/filters are networking specific. > Control virtqueue is better than admin virtqueue here. So I think some things like ring size, queue num, interrupts etc. are configured by the admin queue. VF's ip, mac should be configured by PF's cvq. Is that right? @Michael @Jason Thanks. > > Thanks > > > > > Thanks. > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > >> Thanks > > > >>> Thanks. > > > >>> > > > >>> > > > >>> > > > >>>> Thanks > > > >>>> > > > >>> This publicly archived list offers a means to provide input to the > > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > > >>> > > > >>> In order to verify user consent to the Feedback License terms and > > > >>> to minimize spam in the list archive, subscription is required > > > >>> before posting. > > > >>> > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > >>> List help: virtio-comment-help@lists.oasis-open.org > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > > >>> Join OASIS: https://www.oasis-open.org/join/ > > > >>> > > > > > > > > > This publicly archived list offers a means to provide input to the > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > In order to verify user consent to the Feedback License terms and > > > to minimize spam in the list archive, subscription is required > > > before posting. > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > List help: virtio-comment-help@lists.oasis-open.org > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-28 6:09 ` Xuan Zhuo @ 2023-07-31 1:20 ` Jason Wang 2023-07-31 2:02 ` Parav Pandit 0 siblings, 1 reply; 49+ messages in thread From: Jason Wang @ 2023-07-31 1:20 UTC (permalink / raw) To: Xuan Zhuo Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Parav Pandit, Washizu Yui On Fri, Jul 28, 2023 at 2:14 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Thu, 27 Jul 2023 16:03:56 +0800, Jason Wang <jasowang@redhat.com> wrote: > > On Thu, Jul 27, 2023 at 2:23 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > On Thu, 27 Jul 2023 14:17:53 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > > > > On 7/27/2023 2:09 PM, Xuan Zhuo wrote: > > > > > On Thu, 27 Jul 2023 11:56:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > >> > > > > >> On 7/27/2023 10:30 AM, Xuan Zhuo wrote: > > > > >>> On Mon, 3 Jul 2023 12:29:32 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > >>>> On 6/30/2023 7:35 PM, Parav Pandit wrote: > > > > >>>>>> From: virtio-comment@lists.oasis-open.org <virtio-comment@lists.oasis- > > > > >>>>>> open.org> On Behalf Of Zhu, Lingshan > > > > >>>>>> Sent: Friday, June 30, 2023 6:33 AM > > > > >>>>>> > > > > >>>>>>>>> Can we let the DPU notify the driver to create a new devicer from the > > > > >>>>>> backend? > > > > >>>>> Yes, why not. > > > > >>>>> > > > > >>>>>>>>> The key point is who want to create a new device. > > > > >>>>>>>> DPU can come with a certain number of pre-created ADIs, just make > > > > >>>>>>>> sure the orchestration SW is aware of their device IDs. > > > > >>>>>>>> > > > > >>>>> Cloud often need these devices to be created dynamically, many a time after the host OS is booted. > > > > >>>>> To be more generic, those devices to be created and connected to the host regardless of the life cycle of the host. > > > > >>>>> Xuan partly explained it. > > > > >>>>> > > > > >>>>>>>> If you want the DPU randomly create ADIs and notify the driver, I > > > > >>>>>>>> think we need interrupt, e.g., re-use config interrupt. But why DPU > > > > >>>>>>>> wants to create and hot plug in a device to a guest? > > > > >>>>>>>> Shall the host handle that or DPU pre-create then expose to baremteal > > > > >>>>>>>> machines? > > > > >>>>>>> In your scenario, the supervisor is on the os, which controls the DPU > > > > >>>>>>> to create new devices. > > > > >>>>>>> > > > > >>>>>>> In the cloud scenario, the vendor manager is in the DPU, and the > > > > >>>>>>> entire host is for users. Of course, there are situations where the > > > > >>>>>>> vendor manager are in the HOST. But for bare metal machines, the host > > > > >>>>>>> belongs to the customer, the vendor manager is only in the DPU. > > > > >>>>>>> > > > > >>>>>>> So when the customers buy a new nic for the host, the vendor manager > > > > >>>>>>> will plug a device to the host from the DPU. > > > > >>>>>> I understand once a customer orders a new NIC, you wants to present the NIC > > > > >>>>>> to the host. > > > > >>>>>> However you only owns the DPU and the customer owns the host, that means > > > > >>>>>> this creation and hot plug must be transparent to the host and there may not be > > > > >>>>>> a host driver help handling an interrupt/probe. > > > > >>>>>> > > > > >>>>> That is ok. when driver is loaded, it would query about its child devices and probe it, if we strictly want to follow SIOV model. > > > > >>>>> > > > > >>>>>> However this is not PCI which has a tree/switch and can enumerate devices to > > > > >>>>>> the host by spanning the device across the PCI hierarchy. > > > > >>>>>> > > > > >>>>> Those enumeration is triggered by the parent PCI device and pci bridge and switch will also discover it. > > > > >>>>> > > > > >>>>>> To address an ADI, there is only a device_id. > > > > >>>>>> > > > > >>>>> SIOV device must have a unique identifier at PCI bus level for sure. > > > > >>>>> I cannot speak more about it in this forum due to other logistics issue. > > > > >>>>> But assume that there is PCI level unique identifier for SIOV device that switches on the path will learn about. > > > > >>>>> > > > > >>>>>> So, do you mind share how your DPU offload the device model? What kind of > > > > >>>>>> device your DPU provide to the host? Lets see whether DPU can mediate this by > > > > >>>>>> its own? > > > > >>>>>> > > > > >>>>> It is a virtio nic, blk and other virtio devices for us. > > > > >>>>> A DPU hotplugs a device, host side either gets interrupt or later gets to know about it when explicitly queries. > > > > >>>>> There is no mediation per say here, it is just a dpu based SIOV device like a regular PF. > > > > >>>>> > > > > >>>>> For non virtio DPU device, I implemented them in Linux for dpus 2 years ago. > > > > >>>>> You might find a Linux reference model useful at [1]. > > > > >>>>> A usage model already exists in one OS and in use for non virtio devices. > > > > >>>>> This certainly works without SIOV unique PCI device identifiers, because DPU (non-host) managed SIOV device spec still does not exist. > > > > >>>>> > > > > >>>>> For virtio, I think we should wait for this piece to be defined and leverage that, instead of virtio tc creating its own. > > > > >>>>> > > > > >>>>> [1] https://github.com/Mellanox/scalablefunctions/wiki > > > > >>>> well I see SF facing the similar challenge, I can add a command for the > > > > >>>> driver to query all existing SIOV ADIs of a device, > > > > >>>> and the device return ADIs id and status. Looks good? and work for you > > > > >>>> @Xuan? > > > > >>> Could I have your plan for this? > > > > >>> > > > > >>> If you do not mind, I'd like to add a command to query VF's info. Such > > > > >>> as mac, ip, etc. > > > > >> I think the query commands for SIOV is a little more complex, e.g., > > > > >> need to report device type and its scale(e.g., features, mq). > > > > >> There can be thousands of SIOV ADIs and we don't want output flood. > > > > >> > > > > >> We have discussed implementation a config interrupt to report new > > > > >> created / deleted > > > > >> ADIs on the DPU side, therefore there must be a cap contains related > > > > >> information, > > > > >> my rough approach of the process is: > > > > >> 1) a cap contains the total number of existing ADIs and the max dev id > > > > >> 2) driver queries detailed information of a certain ADI or a bunch of > > > > >> ADIs in a [dev_id....dev_id2] range. > > > > > Yes, Admin Queue can obtain the info of the specific one or more devices. > > > > > > > > > >> I am not sure whether a NIC stores its IP > > > > > > > > > > IP is the other topic. I want the Admin Queue manage the switch. > > > > > So the switch know about the IP of every device, and the > > > > > Admin Queue will has the ability to config the IP of the device inside the > > > > > switch. > > > > DPU onboard switch? OVS? Does it beyond virtio spec? > > > > > > YES. > > > > Adding Washizu. > > > > We can have a switch/dpa defined in the networking device for sure. > > > > > > > > For SIOV, I think this is MUST. > > > > A learning bridge would be fine as a starter. It's better not to > > couple new scalable capability with any device specific features. > > > > > Maybe you have one simple implementation. > > > But you have to solve the IP steering. So admin queue should has the ability > > > to config the IP steering. > > > > I think not. Those L2/LN tables/filters are networking specific. > > Control virtqueue is better than admin virtqueue here. > > So I think some things like ring size, queue num, interrupts etc. are configured > by the admin queue. VF's ip, mac should be configured by PF's cvq. > > Is that right? > > @Michael @Jason That's my understanding. Thanks > > Thanks. > > > > > > Thanks > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > >> Thanks > > > > >>> Thanks. > > > > >>> > > > > >>> > > > > >>> > > > > >>>> Thanks > > > > >>>> > > > > >>> This publicly archived list offers a means to provide input to the > > > > >>> OASIS Virtual I/O Device (VIRTIO) TC. > > > > >>> > > > > >>> In order to verify user consent to the Feedback License terms and > > > > >>> to minimize spam in the list archive, subscription is required > > > > >>> before posting. > > > > >>> > > > > >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > >>> List help: virtio-comment-help@lists.oasis-open.org > > > > >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > >>> Committee: https://www.oasis-open.org/committees/virtio/ > > > > >>> Join OASIS: https://www.oasis-open.org/join/ > > > > >>> > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > to minimize spam in the list archive, subscription is required > > > > before posting. > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > In order to verify user consent to the Feedback License terms and > > > to minimize spam in the list archive, subscription is required > > > before posting. > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > List help: virtio-comment-help@lists.oasis-open.org > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* RE: [virtio-comment] About the plan of Admin Queue 2023-07-31 1:20 ` Jason Wang @ 2023-07-31 2:02 ` Parav Pandit 0 siblings, 0 replies; 49+ messages in thread From: Parav Pandit @ 2023-07-31 2:02 UTC (permalink / raw) To: Jason Wang, Xuan Zhuo Cc: Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin, Washizu Yui > From: Jason Wang <jasowang@redhat.com> > Sent: Monday, July 31, 2023 6:50 AM > > So I think some things like ring size, queue num, interrupts etc. are > > configured by the admin queue. VF's ip, mac should be configured by PF's cvq. VF's mac should be configured by itself using its own cvq. There is no mediation required. ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-06-30 5:54 ` Xuan Zhuo 2023-06-30 6:19 ` Zhu, Lingshan @ 2023-07-03 8:10 ` Jason Wang 2023-07-03 8:20 ` Xuan Zhuo 1 sibling, 1 reply; 49+ messages in thread From: Jason Wang @ 2023-07-03 8:10 UTC (permalink / raw) To: Xuan Zhuo; +Cc: Zhu, Lingshan, virtio-comment, Michael S. Tsirkin On Fri, Jun 30, 2023 at 1:59 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > On 6/20/2023 2:44 PM, Xuan Zhuo wrote: > > > Hi, > > > > > > hi, I would want to know some plans and progress about admin queue. > > > > > > At the current spec, it seems that there is only one framework and no > > > specific commands. I'd like to know if anyone is currently working on this and > > > what the plans are. > > > > > > We also faced some similar issues, and we think admin queue is a good way to > > > manage sr-iov. > > > > > > > > > Thanks. > > I plan to rebase original transport vq on admin vq. > > > > https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html > > > I review the patch, that is for S-IOV, right? It is not limited to SIOV. So you won't see any SIOV terminology in the proposal. > > I think it is good. > > I would if all is configured by the transport vq/admin vq from the OS? One thing to clarify is that, we will define a set of admin commands, and admin virtqueue is one interface for that. The admin commands is designed to be more general to be carried via other type of interfaces (if not, let's fix): 1) MMIO 2) Other transport that already use DMA 3) software IPC Thanks > > Can we create a managed dev from the backend? > > Such as, the DPU sends a command to the driver, then the driver creates a new > managed dev. > > Thanks. > > > > > > Thanks, > > Zhu Lingshan > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > In order to verify user consent to the Feedback License terms and > > > to minimize spam in the list archive, subscription is required > > > before posting. > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > List help: virtio-comment-help@lists.oasis-open.org > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > > This publicly archived list offers a means to provide input to the > OASIS Virtual I/O Device (VIRTIO) TC. > > In order to verify user consent to the Feedback License terms and > to minimize spam in the list archive, subscription is required > before posting. > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > List help: virtio-comment-help@lists.oasis-open.org > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > Committee: https://www.oasis-open.org/committees/virtio/ > Join OASIS: https://www.oasis-open.org/join/ > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 8:10 ` Jason Wang @ 2023-07-03 8:20 ` Xuan Zhuo 2023-07-03 13:05 ` Michael S. Tsirkin 0 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-03 8:20 UTC (permalink / raw) To: Jason Wang; +Cc: Zhu, Lingshan, virtio-comment, Michael S. Tsirkin On Mon, 3 Jul 2023 16:10:18 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Fri, Jun 30, 2023 at 1:59 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > On Tue, 20 Jun 2023 16:11:12 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote: > > > > > > > > > On 6/20/2023 2:44 PM, Xuan Zhuo wrote: > > > > Hi, > > > > > > > > hi, I would want to know some plans and progress about admin queue. > > > > > > > > At the current spec, it seems that there is only one framework and no > > > > specific commands. I'd like to know if anyone is currently working on this and > > > > what the plans are. > > > > > > > > We also faced some similar issues, and we think admin queue is a good way to > > > > manage sr-iov. > > > > > > > > > > > > Thanks. > > > I plan to rebase original transport vq on admin vq. > > > > > > https://lists.oasis-open.org/archives/virtio-comment/202208/msg00140.html > > > > > > I review the patch, that is for S-IOV, right? > > It is not limited to SIOV. So you won't see any SIOV terminology in > the proposal. I got. > > > > > I think it is good. > > > > I would if all is configured by the transport vq/admin vq from the OS? > > One thing to clarify is that, we will define a set of admin commands, > and admin virtqueue is one interface for that. The admin commands is > designed to be more general to be carried via other type of interfaces > (if not, let's fix): > > 1) MMIO > 2) Other transport that already use DMA > 3) software IPC Great!! Thanks. > > Thanks > > > > > Can we create a managed dev from the backend? > > > > Such as, the DPU sends a command to the driver, then the driver creates a new > > managed dev. > > > > Thanks. > > > > > > > > > > Thanks, > > > Zhu Lingshan > > > > > > > > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > > > In order to verify user consent to the Feedback License terms and > > > > to minimize spam in the list archive, subscription is required > > > > before posting. > > > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > > List help: virtio-comment-help@lists.oasis-open.org > > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > > > > > > > This publicly archived list offers a means to provide input to the > > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > > > In order to verify user consent to the Feedback License terms and > > > to minimize spam in the list archive, subscription is required > > > before posting. > > > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > > List help: virtio-comment-help@lists.oasis-open.org > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > > Committee: https://www.oasis-open.org/committees/virtio/ > > > Join OASIS: https://www.oasis-open.org/join/ > > > > > > > This publicly archived list offers a means to provide input to the > > OASIS Virtual I/O Device (VIRTIO) TC. > > > > In order to verify user consent to the Feedback License terms and > > to minimize spam in the list archive, subscription is required > > before posting. > > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org > > List help: virtio-comment-help@lists.oasis-open.org > > List archive: https://lists.oasis-open.org/archives/virtio-comment/ > > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf > > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists > > Committee: https://www.oasis-open.org/committees/virtio/ > > Join OASIS: https://www.oasis-open.org/join/ > > > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 8:20 ` Xuan Zhuo @ 2023-07-03 13:05 ` Michael S. Tsirkin 2023-07-03 13:06 ` Parav Pandit ` (3 more replies) 0 siblings, 4 replies; 49+ messages in thread From: Michael S. Tsirkin @ 2023-07-03 13:05 UTC (permalink / raw) To: Xuan Zhuo; +Cc: Jason Wang, Zhu, Lingshan, virtio-comment, parav OK it looks like at least Parav and Jason understand what's going on, maybe one of you can explain: 1- there's a management stack that can bind an ID to a group of properties 2- the proposed command then binds an ID to a VF 3- after this, VF can be assigned to a VM Is this a fair summary? In that case what is gained compared to configuring properties on the VF directly? Why is not the ID an internal property of the management? -- MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* RE: [virtio-comment] About the plan of Admin Queue 2023-07-03 13:05 ` Michael S. Tsirkin @ 2023-07-03 13:06 ` Parav Pandit 2023-07-03 20:38 ` Parav Pandit ` (2 subsequent siblings) 3 siblings, 0 replies; 49+ messages in thread From: Parav Pandit @ 2023-07-03 13:06 UTC (permalink / raw) To: Michael S. Tsirkin, Xuan Zhuo Cc: Jason Wang, Zhu, Lingshan, virtio-comment@lists.oasis-open.org > From: Michael S. Tsirkin <mst@redhat.com> > Sent: Monday, July 3, 2023 9:05 AM > > OK it looks like at least Parav and Jason understand what's going on, maybe one > of you can explain: > Yet to catch up on the thread today, will be able to respond in few hours. > 1- there's a management stack that can bind an ID > to a group of properties > 2- the proposed command then binds an ID to a VF > 3- after this, VF can be assigned to a VM > > Is this a fair summary? > > In that case what is gained compared to > configuring properties on the VF directly? > Why is not the ID an internal property of the management? > > -- > MST This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* RE: [virtio-comment] About the plan of Admin Queue 2023-07-03 13:05 ` Michael S. Tsirkin 2023-07-03 13:06 ` Parav Pandit @ 2023-07-03 20:38 ` Parav Pandit 2023-07-04 3:48 ` Zhu, Lingshan 2023-07-04 12:11 ` Xuan Zhuo 2023-07-04 12:14 ` Xuan Zhuo 3 siblings, 1 reply; 49+ messages in thread From: Parav Pandit @ 2023-07-03 20:38 UTC (permalink / raw) To: Michael S. Tsirkin, Xuan Zhuo Cc: Jason Wang, Zhu, Lingshan, virtio-comment@lists.oasis-open.org > From: Michael S. Tsirkin <mst@redhat.com> > Sent: Monday, July 3, 2023 9:05 AM > > OK it looks like at least Parav and Jason understand what's going on, maybe one > of you can explain: > > 1- there's a management stack that can bind an ID > to a group of properties > 2- the proposed command then binds an ID to a VF > 3- after this, VF can be assigned to a VM > > Is this a fair summary? > > In that case what is gained compared to > configuring properties on the VF directly? > Why is not the ID an internal property of the management? For VFs I also believe we can live with an internal property. For migratable SIOV devices which are spawned by the management system has similar ID assigned by the mgmt. system. Having such node local ID is needed anyway. There is a struggle (scale wise) to make such ID network wide unique if each virtio device to expose such id at transport level. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 20:38 ` Parav Pandit @ 2023-07-04 3:48 ` Zhu, Lingshan 0 siblings, 0 replies; 49+ messages in thread From: Zhu, Lingshan @ 2023-07-04 3:48 UTC (permalink / raw) To: Parav Pandit, Michael S. Tsirkin, Xuan Zhuo Cc: Jason Wang, virtio-comment@lists.oasis-open.org On 7/4/2023 4:38 AM, Parav Pandit wrote: > >> From: Michael S. Tsirkin <mst@redhat.com> >> Sent: Monday, July 3, 2023 9:05 AM >> >> OK it looks like at least Parav and Jason understand what's going on, maybe one >> of you can explain: >> >> 1- there's a management stack that can bind an ID >> to a group of properties >> 2- the proposed command then binds an ID to a VF >> 3- after this, VF can be assigned to a VM >> >> Is this a fair summary? >> >> In that case what is gained compared to >> configuring properties on the VF directly? >> Why is not the ID an internal property of the management? > For VFs I also believe we can live with an internal property. > > For migratable SIOV devices which are spawned by the management system has similar ID assigned by the mgmt. system. > Having such node local ID is needed anyway. > > There is a struggle (scale wise) to make such ID network wide unique if each virtio device to expose such id at transport level. Maybe you don't want a guest talking to the PF through admin queue directly, so there can be a wrapper for SIOV ADIs, therefore the ID is not a problem for migration. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 13:05 ` Michael S. Tsirkin 2023-07-03 13:06 ` Parav Pandit 2023-07-03 20:38 ` Parav Pandit @ 2023-07-04 12:11 ` Xuan Zhuo 2023-07-04 12:14 ` Xuan Zhuo 3 siblings, 0 replies; 49+ messages in thread From: Xuan Zhuo @ 2023-07-04 12:11 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: Jason Wang, Zhu, Lingshan, virtio-comment, parav On Mon, 3 Jul 2023 09:05:09 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote: > OK it looks like at least Parav and Jason understand what's > going on, maybe one of you can explain: > > 1- there's a management stack that can bind an ID > to a group of properties > 2- the proposed command then binds an ID to a VF > 3- after this, VF can be assigned to a VM > > Is this a fair summary? This sounds like what I need. This thread discusses another point, the device notifies the pf driver that there is a new device. Thanks. > > In that case what is gained compared to > configuring properties on the VF directly? > Why is not the ID an internal property of the management? > > -- > MST > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: [virtio-comment] About the plan of Admin Queue 2023-07-03 13:05 ` Michael S. Tsirkin ` (2 preceding siblings ...) 2023-07-04 12:11 ` Xuan Zhuo @ 2023-07-04 12:14 ` Xuan Zhuo 2023-07-04 13:15 ` Parav Pandit 3 siblings, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-04 12:14 UTC (permalink / raw) To: Michael S. Tsirkin; +Cc: Jason Wang, Zhu, Lingshan, virtio-comment, parav On Mon, 3 Jul 2023 09:05:09 -0400, "Michael S. Tsirkin" <mst@redhat.com> wrote: > OK it looks like at least Parav and Jason understand what's > going on, maybe one of you can explain: > > 1- there's a management stack that can bind an ID > to a group of properties > 2- the proposed command then binds an ID to a VF > 3- after this, VF can be assigned to a VM > > Is this a fair summary? > > In that case what is gained compared to > configuring properties on the VF directly? I have been thinking about this question for a long time, and there maybe is a gap of understanding here. In our scenario, we should have two roles, one is the user and the other is the vendor's management software. Do you think that the management software is running on the host, and it manages the DPU? But in the scenario I mentioned, the vendor's management software runs on the DPU, and all hosts belong to the user. The problem we want to solve is that this user may want 1000 network cards. So it must use VFs as a network cards. It cannot and will not configure detailed NIC properties based on the admin queue. I don't know if I found the key to the problem. Thanks. > Why is not the ID an internal property of the management? > > -- > MST > This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* RE: [virtio-comment] About the plan of Admin Queue 2023-07-04 12:14 ` Xuan Zhuo @ 2023-07-04 13:15 ` Parav Pandit 2023-07-05 4:30 ` Xuan Zhuo 2023-07-05 4:38 ` Xuan Zhuo 0 siblings, 2 replies; 49+ messages in thread From: Parav Pandit @ 2023-07-04 13:15 UTC (permalink / raw) To: Xuan Zhuo, Michael S. Tsirkin Cc: Jason Wang, Zhu, Lingshan, virtio-comment@lists.oasis-open.org > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > Sent: Tuesday, July 4, 2023 8:14 AM > But in the scenario I mentioned, the vendor's management software runs on > the DPU, and all hosts belong to the user. > How did vendor management software configure the parameters on the src side? Same channel can be used on destination side, right? > The problem we want to solve is that this user may want 1000 network cards. So > it must use VFs as a network cards. It cannot and will not configure detailed NIC > properties based on the admin queue. > Sure. It will not. The idea is to use same out of band channel as what is used on the src side, also on the destination side. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: RE: [virtio-comment] About the plan of Admin Queue 2023-07-04 13:15 ` Parav Pandit @ 2023-07-05 4:30 ` Xuan Zhuo 2023-07-05 4:35 ` Parav Pandit 2023-07-05 4:38 ` Xuan Zhuo 1 sibling, 1 reply; 49+ messages in thread From: Xuan Zhuo @ 2023-07-05 4:30 UTC (permalink / raw) To: Parav Pandit Cc: Jason Wang, Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin On Tue, 4 Jul 2023 13:15:23 +0000, Parav Pandit <parav@nvidia.com> wrote: > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > > Sent: Tuesday, July 4, 2023 8:14 AM > > > But in the scenario I mentioned, the vendor's management software runs on > > the DPU, and all hosts belong to the user. > > > How did vendor management software configure the parameters on the src side? Sorry, the src side you mean the host, right? > Same channel can be used on destination side, right? Do you mean that the bandwidth of the net device can be configured by the user? I think the user can only do this on the platform. Thanks. > > > The problem we want to solve is that this user may want 1000 network cards. So > > it must use VFs as a network cards. It cannot and will not configure detailed NIC > > properties based on the admin queue. > > > Sure. > It will not. > The idea is to use same out of band channel as what is used on the src side, also on the destination side. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* RE: RE: [virtio-comment] About the plan of Admin Queue 2023-07-05 4:30 ` Xuan Zhuo @ 2023-07-05 4:35 ` Parav Pandit 2023-07-05 4:36 ` Xuan Zhuo 0 siblings, 1 reply; 49+ messages in thread From: Parav Pandit @ 2023-07-05 4:35 UTC (permalink / raw) To: Xuan Zhuo Cc: Jason Wang, Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > Sent: Wednesday, July 5, 2023 12:30 AM > > On Tue, 4 Jul 2023 13:15:23 +0000, Parav Pandit <parav@nvidia.com> wrote: > > > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > > > Sent: Tuesday, July 4, 2023 8:14 AM > > > > > But in the scenario I mentioned, the vendor's management software > > > runs on the DPU, and all hosts belong to the user. > > > > > How did vendor management software configure the parameters on the src > side? > > > Sorry, the src side you mean the host, right? > No. in your live migration (LM) example src side mean src side of hypervisor where a virtio device is used for the first time. > > > Same channel can be used on destination side, right? > > Do you mean that the bandwidth of the net device can be configured by the > user? No no. > I think the user can only do this on the platform. > Right, only by the platform. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: RE: RE: [virtio-comment] About the plan of Admin Queue 2023-07-05 4:35 ` Parav Pandit @ 2023-07-05 4:36 ` Xuan Zhuo 0 siblings, 0 replies; 49+ messages in thread From: Xuan Zhuo @ 2023-07-05 4:36 UTC (permalink / raw) To: Parav Pandit Cc: Jason Wang, Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin On Wed, 5 Jul 2023 04:35:36 +0000, Parav Pandit <parav@nvidia.com> wrote: > > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > > Sent: Wednesday, July 5, 2023 12:30 AM > > > > On Tue, 4 Jul 2023 13:15:23 +0000, Parav Pandit <parav@nvidia.com> wrote: > > > > > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > > > > Sent: Tuesday, July 4, 2023 8:14 AM > > > > > > > But in the scenario I mentioned, the vendor's management software > > > > runs on the DPU, and all hosts belong to the user. > > > > > > > How did vendor management software configure the parameters on the src > > side? > > > > > > Sorry, the src side you mean the host, right? > > > No. in your live migration (LM) example src side mean src side of hypervisor where a virtio device is used for the first time. Oh, I mistake. > > > > > > Same channel can be used on destination side, right? > > > > Do you mean that the bandwidth of the net device can be configured by the > > user? > > No no. > > I think the user can only do this on the platform. > > > Right, only by the platform. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
* Re: RE: [virtio-comment] About the plan of Admin Queue 2023-07-04 13:15 ` Parav Pandit 2023-07-05 4:30 ` Xuan Zhuo @ 2023-07-05 4:38 ` Xuan Zhuo 1 sibling, 0 replies; 49+ messages in thread From: Xuan Zhuo @ 2023-07-05 4:38 UTC (permalink / raw) To: Parav Pandit Cc: Jason Wang, Zhu, Lingshan, virtio-comment@lists.oasis-open.org, Michael S. Tsirkin On Tue, 4 Jul 2023 13:15:23 +0000, Parav Pandit <parav@nvidia.com> wrote: > > > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com> > > Sent: Tuesday, July 4, 2023 8:14 AM > > > But in the scenario I mentioned, the vendor's management software runs on > > the DPU, and all hosts belong to the user. > > > How did vendor management software configure the parameters on the src side? > Same channel can be used on destination side, right? Yes. > > > The problem we want to solve is that this user may want 1000 network cards. So > > it must use VFs as a network cards. It cannot and will not configure detailed NIC > > properties based on the admin queue. > > > Sure. > It will not. > The idea is to use same out of band channel as what is used on the src side, also on the destination side. Sure. Thanks. This publicly archived list offers a means to provide input to the OASIS Virtual I/O Device (VIRTIO) TC. In order to verify user consent to the Feedback License terms and to minimize spam in the list archive, subscription is required before posting. Subscribe: virtio-comment-subscribe@lists.oasis-open.org Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org List help: virtio-comment-help@lists.oasis-open.org List archive: https://lists.oasis-open.org/archives/virtio-comment/ Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists Committee: https://www.oasis-open.org/committees/virtio/ Join OASIS: https://www.oasis-open.org/join/ ^ permalink raw reply [flat|nested] 49+ messages in thread
end of thread, other threads:[~2023-08-02 7:00 UTC | newest]
Thread overview: 49+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-06-20 6:44 [virtio-comment] About the plan of Admin Queue Xuan Zhuo
2023-06-20 8:11 ` Zhu, Lingshan
2023-06-30 5:54 ` Xuan Zhuo
2023-06-30 6:19 ` Zhu, Lingshan
2023-06-30 7:46 ` Xuan Zhuo
2023-06-30 7:54 ` Zhu, Lingshan
2023-06-30 7:56 ` Xuan Zhuo
2023-06-30 8:32 ` Zhu, Lingshan
2023-06-30 9:07 ` Zhu, Lingshan
2023-06-30 9:14 ` Xuan Zhuo
2023-06-30 10:33 ` Zhu, Lingshan
2023-06-30 11:35 ` Parav Pandit
2023-07-03 4:29 ` Zhu, Lingshan
2023-07-03 5:54 ` Xuan Zhuo
2023-07-03 8:01 ` Zhu, Lingshan
2023-07-03 8:21 ` Xuan Zhuo
2023-07-03 8:23 ` Zhu, Lingshan
2023-07-27 2:30 ` Xuan Zhuo
2023-07-27 3:56 ` Zhu, Lingshan
2023-07-27 6:09 ` Xuan Zhuo
2023-07-27 6:17 ` Zhu, Lingshan
2023-07-27 6:20 ` Xuan Zhuo
2023-07-27 8:03 ` Jason Wang
2023-07-27 8:07 ` Xuan Zhuo
2023-07-27 8:28 ` Jason Wang
2023-07-27 8:30 ` Xuan Zhuo
2023-07-27 8:56 ` Jason Wang
2023-07-27 9:01 ` Xuan Zhuo
[not found] ` <aafe1885-0ec2-66ca-4511-f2606bc881ee@gmail.com>
2023-08-02 6:13 ` Xuan Zhuo
2023-08-02 6:15 ` Jason Wang
2023-08-02 6:34 ` Xuan Zhuo
2023-08-02 6:53 ` Jason Wang
2023-08-02 6:55 ` Xuan Zhuo
2023-07-28 6:09 ` Xuan Zhuo
2023-07-31 1:20 ` Jason Wang
2023-07-31 2:02 ` Parav Pandit
2023-07-03 8:10 ` Jason Wang
2023-07-03 8:20 ` Xuan Zhuo
2023-07-03 13:05 ` Michael S. Tsirkin
2023-07-03 13:06 ` Parav Pandit
2023-07-03 20:38 ` Parav Pandit
2023-07-04 3:48 ` Zhu, Lingshan
2023-07-04 12:11 ` Xuan Zhuo
2023-07-04 12:14 ` Xuan Zhuo
2023-07-04 13:15 ` Parav Pandit
2023-07-05 4:30 ` Xuan Zhuo
2023-07-05 4:35 ` Parav Pandit
2023-07-05 4:36 ` Xuan Zhuo
2023-07-05 4:38 ` Xuan Zhuo
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox