[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

[Xuan Zhuo <xuanzhuo@linux.alibaba.com>]

On Tue, 27 Jun 2023 10:57:13 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>
>
> On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>
> >> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> >>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> >>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> >>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
> >>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> >>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
> >>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>>>> On the other hand, generally the virtio devices are not created by the user
> >>>>>>>>> inside the guest OS. This requires some management platform to participate.
> >>>>>>>>>
> >>>>>>>>> So the usage of this command:
> >>>>>>>>> 1. The user purchases a virtio network card on the management platform,
> >>>>>>>>>         and sets the ip, queue number, etc. The user obtains the identity of
> >>>>>>>>>         the network card.
> >>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
> >>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
> >>>>>>>>>         of the patch
> >>>>>>>>>
> >>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> >>>>>>>>> ---
> >>>>>>>>>       admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> >>>>>>>>>       1 file changed, 40 insertions(+), 1 deletion(-)
> >>>>>>>>>
> >>>>>>>>> diff --git a/admin.tex b/admin.tex
> >>>>>>>>> index 2efd4d7..64d0667 100644
> >>>>>>>>> --- a/admin.tex
> >>>>>>>>> +++ b/admin.tex
> >>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>>>       \hline \hline
> >>>>>>>>>       0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
> >>>>>>>>>       0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
> >>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
> >>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
> >>>>>>>>>       \hline
> >>>>>>>>>       0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
> >>>>>>>>>       \hline
> >>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
> >>>>>>>>>       \field{VF Enable} refer to registers within the SR-IOV Extended
> >>>>>>>>>       Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> >>>>>>>>>
> >>>>>>>>> +\subsubsection{Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
> >>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
> >>>>>>>> passthrough-ed the
> >>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
> >>>>>>>> issue(affects host pci).
> >>>>>>> No care about the passthrough, we always created VFs by the PF.
> >>>>>>>
> >>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
> >>>>>>> by the user in the OS. The devices does not know about it.
> >>>>>> OK, perhaps just say create VFs from a PF in the OS?
> >>>>> YES.
> >>>>>
> >>>>>
> >>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
> >>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
> >>>>>>>> I failed to understand this, once a VF is created, it has a personality,
> >>>>>>>> e.g.,
> >>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
> >>>>>>>>
> >>>>>>>> I am not familiar with the background, What do you mean by virtio device
> >>>>>>>> choose
> >>>>>>>> one VF to bind?
> >>>>>>> On the cloud, the nic is created by the management platform, the
> >>>>>>> user can not create a new nic inside the OS.
> >>>>>>>
> >>>>>>> So after echo sriov_numvfs, the user just got some VFs,
> >>>>>>> there is not backend virtio-net devices.
> >>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
> >>>>>> the orchestration software and it assign properly selected a VF to a
> >>>>>> guest on demands.
> >>>>> Yes, but we do not need to care about the guest. Because VF may only be used
> >>>>> in host, such as docker.
> >>>>>
> >>>>> The problem is that the user (you can think of this as the orchestration
> >>>>> software) creates some VFs, these are only some PCI devices, which virtio
> >>>>> devices will work on these VFs. I think that creating a vf and creating a
> >>>>> virtio-net device are two different things. One is done by user in the OS, one
> >>>>> is done on the management platform.  So we need to bind them together.
> >>>> If the VFs are created through sriov_numvfs, once created, the VF device and
> >>>> its personality are determined.
> >>>>
> >>>> PCI spec says:
> >>>> All VFs associated with a PF must be the same device type as the PF,
> >>>> (e.g., the same network
> >>>> device type or the same storage device type.)
> >>>>
> >>>> So how can the creating process be splitted into separated steps?
> >>>>
> >>>> Are we discussing something beyond the spec?
> >>> NO.
> >>>
> >>> The device types are same.
> >>>
> >>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> >>> these are managed by the management platform. On the cloud, there is an abstract
> >>> object in the backend, which contains things that are generally configured on
> >>> the management platform. It is something that users purchase.
> >>> Under the virtio standard it is similar to device.
> >>>
> >>> In my understanding, we just created a pci vf, and virtio works on top of pci,
> >>> so there must be two steps here (If I mistake, please point out.). When we create
> >>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> >>> scenarios, we can immediately have a backend default device respond when the
> >>> driver probe the vf.  But in our scenario, each device is independent.
> >> Once a VF is crated, there comes with some default configurations, like
> >> MTU and MAC.
> >> Do you mean first step creation and second step initialize it?
> > Not exactly correct,
> >
> > The first step is just to create a vf, at this time there can be a default
> > virtio-net, it doesn't matter.
> >
> > In the second step, we can bind a backend device to this vf.
> What is "bind a backend device to this vf"? The VF itself is a backend
> device, right?


I think that the VF is a transport. I want to bind
the virtio backend device to this transport.


> >
> > Not just for initialization for new divice, we also want to support live
> > migration.
> >
> > For example, on the host, we create a vf and passthrough it into a guest os,
> > this guest is migrated from another host, and its corresponding network card is
> > also migrated to this host. We need to bind this vf to the migrated network
> > card.
> >
> > So just initialization is not enough.
> I think you mean to restore the source side NIC configuration?
> If so, I think this should be managed in the orchestration layer and
> supported by virtio/vdpa APIs, e.g., at least restore common config

I will post our case later.

Thanks.



>
> Thanks
> >
> > Thanks
> >
> >> If so, current spec only allow the user to config MAC through control vq.
> >> vDPA allows to provision a device with proper configuration, maybe that
> >> can be the solution?
> >>
> >> For binding, maybe the orchestration layer manages the pool and it knows
> >> how to initialize
> >> the device
> >>
> >> Thanks
> >>> Thanks.
> >>>
> >>>> Thanks
> >>>>> Thanks.
> >>>>>
> >>>>>
> >>>>>
> >>>>>> So I am confused what the intention of this patch.
> >>>>>>> Thanks.
> >>>>>>>
> >>>>>>>
> >>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> >>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
> >>>>>>>>> +inside the guest OS. This requires some management platform to participate.
> >>>>>>>>> +
> >>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
> >>>>>>>>> +devices.
> >>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
> >>>>>>>> Provision a device on demands and assign it to a guest?
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>>> +
> >>>>>>>>> +\begin{lstlisting}
> >>>>>>>>> +struct virtio_admin_cmd_bind {
> >>>>>>>>> +    u64 identity;
> >>>>>>>>> +};
> >>>>>>>>> +\end{lstlisting}
> >>>>>>>>> +
> >>>>>>>>> +The user got the \field{identity} from the management platform, that is not
> >>>>>>>>> +included by this spec.
> >>>>>>>>> +
> >>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
> >>>>>>>>> +
> >>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
> >>>>>>>>> +device.
> >>>>>>>>> +
> >>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
> >>>>>>>>> +
> >>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
> >>>>>>>>> +
> >>>>>>>>> +If the PF device can not find the device by the \field{identity},
> >>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
> >>>>>>>>> +
> >>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
> >>>>>>>>> +device of this group member specified by the \field{group_member_id}.
> >>>>>>>>> +
> >>>>>>>>>       \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
> >>>>>>>>>
> >>>>>>>>>       An administration virtqueue of an owner device is used to submit
> >>>>>>> This publicly archived list offers a means to provide input to the
> >>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>>
> >>>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>>> to minimize spam in the list archive, subscription is required
> >>>>>>> before posting.
> >>>>>>>
> >>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>>
> >>>>>> This publicly archived list offers a means to provide input to the
> >>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>>>>
> >>>>>> In order to verify user consent to the Feedback License terms and
> >>>>>> to minimize spam in the list archive, subscription is required
> >>>>>> before posting.
> >>>>>>
> >>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>>>>> List help: virtio-comment-help@lists.oasis-open.org
> >>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>>>>> Committee: https://www.oasis-open.org/committees/virtio/
> >>>>>> Join OASIS: https://www.oasis-open.org/join/
> >>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> >>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
> >>>>>
> >>> This publicly archived list offers a means to provide input to the
> >>> OASIS Virtual I/O Device (VIRTIO) TC.
> >>>
> >>> In order to verify user consent to the Feedback License terms and
> >>> to minimize spam in the list archive, subscription is required
> >>> before posting.
> >>>
> >>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> >>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> >>> List help: virtio-comment-help@lists.oasis-open.org
> >>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> >>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> >>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> >>> Committee: https://www.oasis-open.org/committees/virtio/
> >>> Join OASIS: https://www.oasis-open.org/join/
> >>>
> > This publicly archived list offers a means to provide input to the
> > OASIS Virtual I/O Device (VIRTIO) TC.
> >
> > In order to verify user consent to the Feedback License terms and
> > to minimize spam in the list archive, subscription is required
> > before posting.
> >
> > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > List help: virtio-comment-help@lists.oasis-open.org
> > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> > Committee: https://www.oasis-open.org/committees/virtio/
> > Join OASIS: https://www.oasis-open.org/join/
> >
>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org