[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>>>
>>>>>>>>> So the usage of this command:
>>>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>>>         and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>>>         the network card.
>>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>>>         of the patch
>>>>>>>>>
>>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>>>> ---
>>>>>>>>>       admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>       1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>>>
>>>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>>>> --- a/admin.tex
>>>>>>>>> +++ b/admin.tex
>>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>       \hline \hline
>>>>>>>>>       0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>>>       0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>       \hline
>>>>>>>>>       0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>>>       \hline
>>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>       \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>>>       Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>>>
>>>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>>>> passthrough-ed the
>>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>>>> issue(affects host pci).
>>>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>>>
>>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>>>> by the user in the OS. The devices does not know about it.
>>>>>> OK, perhaps just say create VFs from a PF in the OS?
>>>>> YES.
>>>>>
>>>>>
>>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>>>> e.g.,
>>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>>>
>>>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>>>> choose
>>>>>>>> one VF to bind?
>>>>>>> On the cloud, the nic is created by the management platform, the
>>>>>>> user can not create a new nic inside the OS.
>>>>>>>
>>>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>>>> there is not backend virtio-net devices.
>>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>>>> the orchestration software and it assign properly selected a VF to a
>>>>>> guest on demands.
>>>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>>>> in host, such as docker.
>>>>>
>>>>> The problem is that the user (you can think of this as the orchestration
>>>>> software) creates some VFs, these are only some PCI devices, which virtio
>>>>> devices will work on these VFs. I think that creating a vf and creating a
>>>>> virtio-net device are two different things. One is done by user in the OS, one
>>>>> is done on the management platform.  So we need to bind them together.
>>>> If the VFs are created through sriov_numvfs, once created, the VF device and
>>>> its personality are determined.
>>>>
>>>> PCI spec says:
>>>> All VFs associated with a PF must be the same device type as the PF,
>>>> (e.g., the same network
>>>> device type or the same storage device type.)
>>>>
>>>> So how can the creating process be splitted into separated steps?
>>>>
>>>> Are we discussing something beyond the spec?
>>> NO.
>>>
>>> The device types are same.
>>>
>>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
>>> these are managed by the management platform. On the cloud, there is an abstract
>>> object in the backend, which contains things that are generally configured on
>>> the management platform. It is something that users purchase.
>>> Under the virtio standard it is similar to device.
>>>
>>> In my understanding, we just created a pci vf, and virtio works on top of pci,
>>> so there must be two steps here (If I mistake, please point out.). When we create
>>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
>>> scenarios, we can immediately have a backend default device respond when the
>>> driver probe the vf.  But in our scenario, each device is independent.
>> Once a VF is crated, there comes with some default configurations, like
>> MTU and MAC.
>> Do you mean first step creation and second step initialize it?
> Not exactly correct,
>
> The first step is just to create a vf, at this time there can be a default
> virtio-net, it doesn't matter.
>
> In the second step, we can bind a backend device to this vf.
What is "bind a backend device to this vf"? The VF itself is a backend 
device, right?
>
> Not just for initialization for new divice, we also want to support live
> migration.
>
> For example, on the host, we create a vf and passthrough it into a guest os,
> this guest is migrated from another host, and its corresponding network card is
> also migrated to this host. We need to bind this vf to the migrated network
> card.
>
> So just initialization is not enough.
I think you mean to restore the source side NIC configuration?
If so, I think this should be managed in the orchestration layer and
supported by virtio/vdpa APIs, e.g., at least restore common config

Thanks
>
> Thanks
>
>> If so, current spec only allow the user to config MAC through control vq.
>> vDPA allows to provision a device with proper configuration, maybe that
>> can be the solution?
>>
>> For binding, maybe the orchestration layer manages the pool and it knows
>> how to initialize
>> the device
>>
>> Thanks
>>> Thanks.
>>>
>>>> Thanks
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>> So I am confused what the intention of this patch.
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>>>> +
>>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>>>> +devices.
>>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>> +
>>>>>>>>> +\begin{lstlisting}
>>>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>>>> +    u64 identity;
>>>>>>>>> +};
>>>>>>>>> +\end{lstlisting}
>>>>>>>>> +
>>>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>>>> +included by this spec.
>>>>>>>>> +
>>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>>>> +
>>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>>>> +device.
>>>>>>>>> +
>>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>> +
>>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>>>> +
>>>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>>>> +
>>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>>>> +
>>>>>>>>>       \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>>>
>>>>>>>>>       An administration virtqueue of an owner device is used to submit
>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>
>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>> before posting.
>>>>>>>
>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>
>>>>>> This publicly archived list offers a means to provide input to the
>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>
>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>> to minimize spam in the list archive, subscription is required
>>>>>> before posting.
>>>>>>
>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org