[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

["Michael S. Tsirkin" <mst@redhat.com>]

On Mon, Jun 26, 2023 at 12:19:04PM +0000, Parav Pandit wrote:
> 
> 
> > From: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > Sent: Monday, June 26, 2023 6:51 AM
> > 
> > On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > >
> > >
> > > On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> > > > On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>
> > > >> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
> > > >>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
> > > >>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan"
> > <lingshan.zhu@intel.com> wrote:
> > > >>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
> > > >>>>>>> The VFs of the SR-IOV are created by the user inside the guest
> > > >>>>>>> OS, so the virtio devices don't know about these VFs. Because
> > > >>>>>>> each VF may be assigned a different role by the user, the virtio device
> > can not choose one VF to bind random.
> > > >>>>>>> So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> On the other hand, generally the virtio devices are not
> > > >>>>>>> created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>>
> > > >>>>>>> So the usage of this command:
> > > >>>>>>> 1. The user purchases a virtio network card on the management
> > platform,
> > > >>>>>>>        and sets the ip, queue number, etc. The user obtains the identity
> > of
> > > >>>>>>>        the network card.
> > > >>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs 3. The
> > > >>>>>>> user binds the net crad to a VF with identity through the command
> > > >>>>>>>        of the patch
> > > >>>>>>>
> > > >>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
> > > >>>>>>> ---
> > > >>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
> > > >>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
> > > >>>>>>>
> > > >>>>>>> diff --git a/admin.tex b/admin.tex index 2efd4d7..64d0667
> > > >>>>>>> 100644
> > > >>>>>>> --- a/admin.tex
> > > >>>>>>> +++ b/admin.tex
> > > >>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \hline \hline
> > > >>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver
> > list of commands supported for this group type    \\
> > > >>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list
> > of commands used for this group type \\
> > > >>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to
> > one group member \\
> > > >>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct
> > virtio_admin_cmd}    \\
> > > >>>>>>>      \hline
> > > >>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly
> > using a different structure)    \\
> > > >>>>>>>      \hline
> > > >>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration
> > commands}\label{sec:Basic Facilities of a Virti
> > > >>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
> > > >>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
> > > >>>>>>>
> > > >>>>>>> +\subsubsection{Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +The VFs of the SR-IOV are created by the user inside the
> > > >>>>>>> +guest OS, so the virtio
> > > >>>>>> If the VFs are create in a guest OS, I assume that means the
> > > >>>>>> user has passthrough-ed the PF to the guest. For nested, I am
> > > >>>>>> not sure whether this is a security issue(affects host pci).
> > > >>>>> No care about the passthrough, we always created VFs by the PF.
> > > >>>>>
> > > >>>>> I should not say "inside the guest OS". I just want to say that
> > > >>>>> the VF is create by the user in the OS. The devices does not know
> > about it.
> > > >>>> OK, perhaps just say create VFs from a PF in the OS?
> > > >>> YES.
> > > >>>
> > > >>>
> > > >>>>>>> +devices don't know about these VFs. Because each VF may be
> > > >>>>>>> +assigned a different role by the user, the virtio device can not
> > choose one VF to bind random.
> > > >>>>>> I failed to understand this, once a VF is created, it has a
> > > >>>>>> personality, e.g., create a virtio-net VF from a virtio-net PF,
> > > >>>>>> and PF knows that.
> > > >>>>>>
> > > >>>>>> I am not familiar with the background, What do you mean by
> > > >>>>>> virtio device choose one VF to bind?
> > > >>>>> On the cloud, the nic is created by the management platform, the
> > > >>>>> user can not create a new nic inside the OS.
> > > >>>>>
> > > >>>>> So after echo sriov_numvfs, the user just got some VFs, there is
> > > >>>>> not backend virtio-net devices.
> > > >>>> I think it is not a "user" mange the VFs, the VFs usually
> > > >>>> provisioned by the orchestration software and it assign properly
> > > >>>> selected a VF to a guest on demands.
> > > >>> Yes, but we do not need to care about the guest. Because VF may
> > > >>> only be used in host, such as docker.
> > > >>>
> > > >>> The problem is that the user (you can think of this as the
> > > >>> orchestration
> > > >>> software) creates some VFs, these are only some PCI devices, which
> > > >>> virtio devices will work on these VFs. I think that creating a vf
> > > >>> and creating a virtio-net device are two different things. One is
> > > >>> done by user in the OS, one is done on the management platform.  So we
> > need to bind them together.
> > > >> If the VFs are created through sriov_numvfs, once created, the VF
> > > >> device and its personality are determined.
> > > >>
> > > >> PCI spec says:
> > > >> All VFs associated with a PF must be the same device type as the
> > > >> PF, (e.g., the same network device type or the same storage device
> > > >> type.)
> > > >>
> > > >> So how can the creating process be splitted into separated steps?
> > > >>
> > > >> Are we discussing something beyond the spec?
> > > > NO.
> > > >
> > > > The device types are same.
> > > >
> > > > How do we configure the ip, mac, etc of the virtio-net device? In
> > > > the cloud, these are managed by the management platform. On the
> > > > cloud, there is an abstract object in the backend, which contains
> > > > things that are generally configured on the management platform. It is
> > something that users purchase.
> > > > Under the virtio standard it is similar to device.
> > > >
> > > > In my understanding, we just created a pci vf, and virtio works on
> > > > top of pci, so there must be two steps here (If I mistake, please
> > > > point out.). When we create a vf, it doesn't mean that the backend
> > > > deivce is ready. Of course, in some scenarios, we can immediately
> > > > have a backend default device respond when the driver probe the vf.  But in
> > our scenario, each device is independent.
> > > Once a VF is crated, there comes with some default configurations,
> > > like MTU and MAC.
> > > Do you mean first step creation and second step initialize it?
> > 
> > Not exactly correct,
> > 
> > The first step is just to create a vf, at this time there can be a default virtio-net, it
> > doesn't matter.
> > 
> > In the second step, we can bind a backend device to this vf.
> > 
> > Not just for initialization for new divice, we also want to support live migration.
> > 
> > For example, on the host, we create a vf and passthrough it into a guest os, this
> > guest is migrated from another host, and its corresponding network card is also
> > migrated to this host. We need to bind this vf to the migrated network card.
> > 
> > So just initialization is not enough.
> >
> 
> Yet to catch up on the thread, so likely I am missing something.
> 
> The flow is for one OS (Linux) is:
> 1. user enable SR-IOV on the PF device in a host, which creates SR-IOV VFs in the device.
> (num_vfs and vf enable bit in the PCI capability)
> 
> 2. VFs are created at the PCI level in the host system and also inside the device
> 
> 3. A user on the host may bind these VFs to the VF driver (virtionet/blk or vfio or vp_vdpa or some other)
> 
> Between step #2 and #3, a user may configure one or multiple attributes of the VF.
> This includes feature bits, config space fields, vf msix vectors and more.
> This is to be using admin command.
> These admin commands definition is due.

To be frank, I am not sure binding to an ID necessarily needs to be
gated on provisioning commands.
What was not explained at all is what purpose does this extra level
of indirection serve.


> 
>  
> > Thanks
> > 
> > > If so, current spec only allow the user to config MAC through control vq.
> > > vDPA allows to provision a device with proper configuration, maybe
> > > that can be the solution?
> > >
> > > For binding, maybe the orchestration layer manages the pool and it
> > > knows how to initialize the device
> > >
> > > Thanks
> > > >
> > > > Thanks.
> > > >
> > > >> Thanks
> > > >>> Thanks.
> > > >>>
> > > >>>
> > > >>>
> > > >>>> So I am confused what the intention of this patch.
> > > >>>>> Thanks.
> > > >>>>>
> > > >>>>>
> > > >>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
> > > >>>>>>> +On the other hand, generally the virtio devices are not
> > > >>>>>>> +created by the user inside the guest OS. This requires some
> > management platform to participate.
> > > >>>>>>> +
> > > >>>>>>> +So we introduce a new admin queue command to bind the VFs and
> > > >>>>>>> +the virtio devices.
> > > >>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can
> > help?
> > > >>>>>> Provision a device on demands and assign it to a guest?
> > > >>>>>>
> > > >>>>>> Thanks
> > > >>>>>>> +
> > > >>>>>>> +\begin{lstlisting}
> > > >>>>>>> +struct virtio_admin_cmd_bind {
> > > >>>>>>> +    u64 identity;
> > > >>>>>>> +};
> > > >>>>>>> +\end{lstlisting}
> > > >>>>>>> +
> > > >>>>>>> +The user got the \field{identity} from the management
> > > >>>>>>> +platform, that is not included by this spec.
> > > >>>>>>> +
> > > >>>>>>> +\drivernormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the
> > \field{group_member_id} MUST be set.
> > > >>>>>>> +
> > > >>>>>>> +The \field{identity} is passed by the user. It is the
> > > >>>>>>> +identity of the virtio device.
> > > >>>>>>> +
> > > >>>>>>> +\devicenormative{\paragraph}{Group administration
> > > >>>>>>> +commands}{Basic Facilities of a Virtio Device / Device groups
> > > >>>>>>> +/ Group administration commands / Bind the device for member}
> > > >>>>>>> +
> > > >>>>>>> +Every device MUST have one unique \field{identity} in the host.
> > > >>>>>>> +
> > > >>>>>>> +If the PF device can not find the device by the
> > > >>>>>>> +\field{identity}, the \field{status} MUST be set to
> > VIRTIO_ADMIN_STATUS_EINVAL.
> > > >>>>>>> +
> > > >>>>>>> +If the device is found by the \field{identity}, the device
> > > >>>>>>> +MUST work as the device of this group member specified by the
> > \field{group_member_id}.
> > > >>>>>>> +
> > > >>>>>>>      \section{Administration Virtqueues}\label{sec:Basic
> > > >>>>>>> Facilities of a Virtio Device / Administration Virtqueues}
> > > >>>>>>>
> > > >>>>>>>      An administration virtqueue of an owner device is used to
> > > >>>>>>> submit
> > > >>>>> This publicly archived list offers a means to provide input to
> > > >>>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>>
> > > >>>>> In order to verify user consent to the Feedback License terms
> > > >>>>> and to minimize spam in the list archive, subscription is
> > > >>>>> required before posting.
> > > >>>>>
> > > >>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>>> List archive:
> > > >>>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>>> Feedback License:
> > > >>>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>>> List Guidelines:
> > > >>>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>>
> > > >>>> This publicly archived list offers a means to provide input to
> > > >>>> the OASIS Virtual I/O Device (VIRTIO) TC.
> > > >>>>
> > > >>>> In order to verify user consent to the Feedback License terms and
> > > >>>> to minimize spam in the list archive, subscription is required
> > > >>>> before posting.
> > > >>>>
> > > >>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > >>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > >>>> List help: virtio-comment-help@lists.oasis-open.org
> > > >>>> List archive:
> > > >>>> https://lists.oasis-open.org/archives/virtio-comment/
> > > >>>> Feedback License:
> > > >>>> https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > >>>> List Guidelines:
> > > >>>> https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > >>>> Committee: https://www.oasis-open.org/committees/virtio/
> > > >>>> Join OASIS: https://www.oasis-open.org/join/
> > > >>>>
> > > >>> ------------------------------------------------------------------
> > > >>> --- To unsubscribe, e-mail:
> > > >>> virtio-dev-unsubscribe@lists.oasis-open.org
> > > >>> For additional commands, e-mail:
> > > >>> virtio-dev-help@lists.oasis-open.org
> > > >>>
> > > > This publicly archived list offers a means to provide input to the
> > > > OASIS Virtual I/O Device (VIRTIO) TC.
> > > >
> > > > In order to verify user consent to the Feedback License terms and to
> > > > minimize spam in the list archive, subscription is required before
> > > > posting.
> > > >
> > > > Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> > > > Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> > > > List help: virtio-comment-help@lists.oasis-open.org
> > > > List archive: https://lists.oasis-open.org/archives/virtio-comment/
> > > > Feedback License:
> > > > https://www.oasis-open.org/who/ipr/feedback_license.pdf
> > > > List Guidelines:
> > > > https://www.oasis-open.org/policies-guidelines/mailing-lists
> > > > Committee: https://www.oasis-open.org/committees/virtio/
> > > > Join OASIS: https://www.oasis-open.org/join/
> > > >
> > >


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org