[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>
>>>>>>> So the usage of this command:
>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>        and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>        the network card.
>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>        of the patch
>>>>>>>
>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>> ---
>>>>>>>      admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>      1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>> --- a/admin.tex
>>>>>>> +++ b/admin.tex
>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>      \hline \hline
>>>>>>>      0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>      0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>      \hline
>>>>>>>      0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>      \hline
>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>      \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>      Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>
>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>> +
>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>> passthrough-ed the
>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>> issue(affects host pci).
>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>
>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>> by the user in the OS. The devices does not know about it.
>>>> OK, perhaps just say create VFs from a PF in the OS?
>>> YES.
>>>
>>>
>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>> e.g.,
>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>
>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>> choose
>>>>>> one VF to bind?
>>>>> On the cloud, the nic is created by the management platform, the
>>>>> user can not create a new nic inside the OS.
>>>>>
>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>> there is not backend virtio-net devices.
>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>> the orchestration software and it assign properly selected a VF to a
>>>> guest on demands.
>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>> in host, such as docker.
>>>
>>> The problem is that the user (you can think of this as the orchestration
>>> software) creates some VFs, these are only some PCI devices, which virtio
>>> devices will work on these VFs. I think that creating a vf and creating a
>>> virtio-net device are two different things. One is done by user in the OS, one
>>> is done on the management platform.  So we need to bind them together.
>> If the VFs are created through sriov_numvfs, once created, the VF device and
>> its personality are determined.
>>
>> PCI spec says:
>> All VFs associated with a PF must be the same device type as the PF,
>> (e.g., the same network
>> device type or the same storage device type.)
>>
>> So how can the creating process be splitted into separated steps?
>>
>> Are we discussing something beyond the spec?
> NO.
>
> The device types are same.
>
> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
> these are managed by the management platform. On the cloud, there is an abstract
> object in the backend, which contains things that are generally configured on
> the management platform. It is something that users purchase.
> Under the virtio standard it is similar to device.
>
> In my understanding, we just created a pci vf, and virtio works on top of pci,
> so there must be two steps here (If I mistake, please point out.). When we create
> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
> scenarios, we can immediately have a backend default device respond when the
> driver probe the vf.  But in our scenario, each device is independent.
Once a VF is crated, there comes with some default configurations, like 
MTU and MAC.
Do you mean first step creation and second step initialize it?
If so, current spec only allow the user to config MAC through control vq.
vDPA allows to provision a device with proper configuration, maybe that 
can be the solution?

For binding, maybe the orchestration layer manages the pool and it knows 
how to initialize
the device

Thanks
>
> Thanks.
>
>> Thanks
>>> Thanks.
>>>
>>>
>>>
>>>> So I am confused what the intention of this patch.
>>>>> Thanks.
>>>>>
>>>>>
>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>> +
>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>> +devices.
>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>
>>>>>> Thanks
>>>>>>> +
>>>>>>> +\begin{lstlisting}
>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>> +    u64 identity;
>>>>>>> +};
>>>>>>> +\end{lstlisting}
>>>>>>> +
>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>> +included by this spec.
>>>>>>> +
>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>> +
>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>> +
>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>> +device.
>>>>>>> +
>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>> +
>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>> +
>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>> +
>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>> +
>>>>>>>      \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>
>>>>>>>      An administration virtqueue of an owner device is used to submit
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>
>>>> This publicly archived list offers a means to provide input to the
>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>
>>>> In order to verify user consent to the Feedback License terms and
>>>> to minimize spam in the list archive, subscription is required
>>>> before posting.
>>>>
>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
>
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
>
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org