[virtio-dev] Re: [virtio-comment] Re: [virtio-dev] Re: [virtio-comment] [RFC PATCH] admin-queue: bind the group member to the device

["Zhu, Lingshan" <lingshan.zhu@intel.com>]

On 6/27/2023 4:14 PM, Xuan Zhuo wrote:
> On Tue, 27 Jun 2023 10:57:13 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>
>> On 6/26/2023 6:50 PM, Xuan Zhuo wrote:
>>> On Mon, 26 Jun 2023 17:56:01 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>> On 6/26/2023 5:16 PM, Xuan Zhuo wrote:
>>>>> On Mon, 26 Jun 2023 16:59:48 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>> On 6/26/2023 4:09 PM, Xuan Zhuo wrote:
>>>>>>> On Mon, 26 Jun 2023 15:57:33 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>> On 6/26/2023 3:08 PM, Xuan Zhuo wrote:
>>>>>>>>> On Mon, 26 Jun 2023 14:43:17 +0800, "Zhu, Lingshan" <lingshan.zhu@intel.com> wrote:
>>>>>>>>>> On 6/26/2023 2:22 PM, Xuan Zhuo wrote:
>>>>>>>>>>> The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>>>> devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>>>> by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>>>> So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>>>> On the other hand, generally the virtio devices are not created by the user
>>>>>>>>>>> inside the guest OS. This requires some management platform to participate.
>>>>>>>>>>>
>>>>>>>>>>> So the usage of this command:
>>>>>>>>>>> 1. The user purchases a virtio network card on the management platform,
>>>>>>>>>>>          and sets the ip, queue number, etc. The user obtains the identity of
>>>>>>>>>>>          the network card.
>>>>>>>>>>> 2. The user creates a VF with echo 8 > sriov_numvfs
>>>>>>>>>>> 3. The user binds the net crad to a VF with identity through the command
>>>>>>>>>>>          of the patch
>>>>>>>>>>>
>>>>>>>>>>> Signed-off-by: Xuan Zhuo <xuanzhuo@linux.alibaba.com>
>>>>>>>>>>> ---
>>>>>>>>>>>        admin.tex | 41 ++++++++++++++++++++++++++++++++++++++++-
>>>>>>>>>>>        1 file changed, 40 insertions(+), 1 deletion(-)
>>>>>>>>>>>
>>>>>>>>>>> diff --git a/admin.tex b/admin.tex
>>>>>>>>>>> index 2efd4d7..64d0667 100644
>>>>>>>>>>> --- a/admin.tex
>>>>>>>>>>> +++ b/admin.tex
>>>>>>>>>>> @@ -115,7 +115,8 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>>>        \hline \hline
>>>>>>>>>>>        0x0000 & VIRTIO_ADMIN_CMD_LIST_QUERY & Provides to driver list of commands supported for this group type    \\
>>>>>>>>>>>        0x0001 & VIRTIO_ADMIN_CMD_LIST_USE & Provides to device list of commands used for this group type \\
>>>>>>>>>>> -0x0002 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>>> +0x0002 & VIRTIO_ADMIN_CMD_BIND_DEVICE & Bind the device to one group member \\
>>>>>>>>>>> +0x0003 - 0x7FFF & - & Commands using \field{struct virtio_admin_cmd}    \\
>>>>>>>>>>>        \hline
>>>>>>>>>>>        0x8000 - 0xFFFF & - & Reserved for future commands (possibly using a different structure)    \\
>>>>>>>>>>>        \hline
>>>>>>>>>>> @@ -429,6 +430,44 @@ \subsection{Group administration commands}\label{sec:Basic Facilities of a Virti
>>>>>>>>>>>        \field{VF Enable} refer to registers within the SR-IOV Extended
>>>>>>>>>>>        Capability as specified by \hyperref[intro:PCIe]{[PCIe]}.
>>>>>>>>>>>
>>>>>>>>>>> +\subsubsection{Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +The VFs of the SR-IOV are created by the user inside the guest OS, so the virtio
>>>>>>>>>> If the VFs are create in a guest OS, I assume that means the user has
>>>>>>>>>> passthrough-ed the
>>>>>>>>>> PF to the guest. For nested, I am not sure whether this is a security
>>>>>>>>>> issue(affects host pci).
>>>>>>>>> No care about the passthrough, we always created VFs by the PF.
>>>>>>>>>
>>>>>>>>> I should not say "inside the guest OS". I just want to say that the VF is create
>>>>>>>>> by the user in the OS. The devices does not know about it.
>>>>>>>> OK, perhaps just say create VFs from a PF in the OS?
>>>>>>> YES.
>>>>>>>
>>>>>>>
>>>>>>>>>>> +devices don't know about these VFs. Because each VF may be assigned a different role
>>>>>>>>>>> +by the user, the virtio device can not choose one VF to bind random.
>>>>>>>>>> I failed to understand this, once a VF is created, it has a personality,
>>>>>>>>>> e.g.,
>>>>>>>>>> create a virtio-net VF from a virtio-net PF, and PF knows that.
>>>>>>>>>>
>>>>>>>>>> I am not familiar with the background, What do you mean by virtio device
>>>>>>>>>> choose
>>>>>>>>>> one VF to bind?
>>>>>>>>> On the cloud, the nic is created by the management platform, the
>>>>>>>>> user can not create a new nic inside the OS.
>>>>>>>>>
>>>>>>>>> So after echo sriov_numvfs, the user just got some VFs,
>>>>>>>>> there is not backend virtio-net devices.
>>>>>>>> I think it is not a "user" mange the VFs, the VFs usually provisioned by
>>>>>>>> the orchestration software and it assign properly selected a VF to a
>>>>>>>> guest on demands.
>>>>>>> Yes, but we do not need to care about the guest. Because VF may only be used
>>>>>>> in host, such as docker.
>>>>>>>
>>>>>>> The problem is that the user (you can think of this as the orchestration
>>>>>>> software) creates some VFs, these are only some PCI devices, which virtio
>>>>>>> devices will work on these VFs. I think that creating a vf and creating a
>>>>>>> virtio-net device are two different things. One is done by user in the OS, one
>>>>>>> is done on the management platform.  So we need to bind them together.
>>>>>> If the VFs are created through sriov_numvfs, once created, the VF device and
>>>>>> its personality are determined.
>>>>>>
>>>>>> PCI spec says:
>>>>>> All VFs associated with a PF must be the same device type as the PF,
>>>>>> (e.g., the same network
>>>>>> device type or the same storage device type.)
>>>>>>
>>>>>> So how can the creating process be splitted into separated steps?
>>>>>>
>>>>>> Are we discussing something beyond the spec?
>>>>> NO.
>>>>>
>>>>> The device types are same.
>>>>>
>>>>> How do we configure the ip, mac, etc of the virtio-net device? In the cloud,
>>>>> these are managed by the management platform. On the cloud, there is an abstract
>>>>> object in the backend, which contains things that are generally configured on
>>>>> the management platform. It is something that users purchase.
>>>>> Under the virtio standard it is similar to device.
>>>>>
>>>>> In my understanding, we just created a pci vf, and virtio works on top of pci,
>>>>> so there must be two steps here (If I mistake, please point out.). When we create
>>>>> a vf, it doesn't mean that the backend deivce is ready. Of course, in some
>>>>> scenarios, we can immediately have a backend default device respond when the
>>>>> driver probe the vf.  But in our scenario, each device is independent.
>>>> Once a VF is crated, there comes with some default configurations, like
>>>> MTU and MAC.
>>>> Do you mean first step creation and second step initialize it?
>>> Not exactly correct,
>>>
>>> The first step is just to create a vf, at this time there can be a default
>>> virtio-net, it doesn't matter.
>>>
>>> In the second step, we can bind a backend device to this vf.
>> What is "bind a backend device to this vf"? The VF itself is a backend
>> device, right?
>
> I think that the VF is a transport. I want to bind
> the virtio backend device to this transport.
still confused, PCIe VF is a device, transport refers to PCI since we 
are talking about VFs.
Not sure what is the binding because the VF comes with its transport, but
if you want to address a VF, shall the orchestrator simply use the num 
of a VF or its BDF?
>
>
>>> Not just for initialization for new divice, we also want to support live
>>> migration.
>>>
>>> For example, on the host, we create a vf and passthrough it into a guest os,
>>> this guest is migrated from another host, and its corresponding network card is
>>> also migrated to this host. We need to bind this vf to the migrated network
>>> card.
>>>
>>> So just initialization is not enough.
>> I think you mean to restore the source side NIC configuration?
>> If so, I think this should be managed in the orchestration layer and
>> supported by virtio/vdpa APIs, e.g., at least restore common config
> I will post our case later.
>
> Thanks.
>
>
>
>> Thanks
>>> Thanks
>>>
>>>> If so, current spec only allow the user to config MAC through control vq.
>>>> vDPA allows to provision a device with proper configuration, maybe that
>>>> can be the solution?
>>>>
>>>> For binding, maybe the orchestration layer manages the pool and it knows
>>>> how to initialize
>>>> the device
>>>>
>>>> Thanks
>>>>> Thanks.
>>>>>
>>>>>> Thanks
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> So I am confused what the intention of this patch.
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> +So only the user knows how to bind the virtio devices to the VFs.
>>>>>>>>>>> +On the other hand, generally the virtio devices are not created by the user
>>>>>>>>>>> +inside the guest OS. This requires some management platform to participate.
>>>>>>>>>>> +
>>>>>>>>>>> +So we introduce a new admin queue command to bind the VFs and the virtio
>>>>>>>>>>> +devices.
>>>>>>>>>> Sorry, failed to process this. Maybe an orchestration sw layer can help?
>>>>>>>>>> Provision a device on demands and assign it to a guest?
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>> +
>>>>>>>>>>> +\begin{lstlisting}
>>>>>>>>>>> +struct virtio_admin_cmd_bind {
>>>>>>>>>>> +    u64 identity;
>>>>>>>>>>> +};
>>>>>>>>>>> +\end{lstlisting}
>>>>>>>>>>> +
>>>>>>>>>>> +The user got the \field{identity} from the management platform, that is not
>>>>>>>>>>> +included by this spec.
>>>>>>>>>>> +
>>>>>>>>>>> +\drivernormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +VIRTIO_ADMIN_CMD_BIND_DEVICE requires that the \field{group_member_id} MUST be set.
>>>>>>>>>>> +
>>>>>>>>>>> +The \field{identity} is passed by the user. It is the identity of the virtio
>>>>>>>>>>> +device.
>>>>>>>>>>> +
>>>>>>>>>>> +\devicenormative{\paragraph}{Group administration commands}{Basic Facilities of a Virtio Device / Device groups / Group administration commands / Bind the device for member}
>>>>>>>>>>> +
>>>>>>>>>>> +Every device MUST have one unique \field{identity} in the host.
>>>>>>>>>>> +
>>>>>>>>>>> +If the PF device can not find the device by the \field{identity},
>>>>>>>>>>> +the \field{status} MUST be set to VIRTIO_ADMIN_STATUS_EINVAL.
>>>>>>>>>>> +
>>>>>>>>>>> +If the device is found by the \field{identity}, the device MUST work as the
>>>>>>>>>>> +device of this group member specified by the \field{group_member_id}.
>>>>>>>>>>> +
>>>>>>>>>>>        \section{Administration Virtqueues}\label{sec:Basic Facilities of a Virtio Device / Administration Virtqueues}
>>>>>>>>>>>
>>>>>>>>>>>        An administration virtqueue of an owner device is used to submit
>>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>>
>>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>>> before posting.
>>>>>>>>>
>>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>>
>>>>>>>> This publicly archived list offers a means to provide input to the
>>>>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>>>>
>>>>>>>> In order to verify user consent to the Feedback License terms and
>>>>>>>> to minimize spam in the list archive, subscription is required
>>>>>>>> before posting.
>>>>>>>>
>>>>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
>>>>>>> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>>>>>>>
>>>>> This publicly archived list offers a means to provide input to the
>>>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>>>
>>>>> In order to verify user consent to the Feedback License terms and
>>>>> to minimize spam in the list archive, subscription is required
>>>>> before posting.
>>>>>
>>>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>>>> List help: virtio-comment-help@lists.oasis-open.org
>>>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>>>> Committee: https://www.oasis-open.org/committees/virtio/
>>>>> Join OASIS: https://www.oasis-open.org/join/
>>>>>
>>> This publicly archived list offers a means to provide input to the
>>> OASIS Virtual I/O Device (VIRTIO) TC.
>>>
>>> In order to verify user consent to the Feedback License terms and
>>> to minimize spam in the list archive, subscription is required
>>> before posting.
>>>
>>> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
>>> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
>>> List help: virtio-comment-help@lists.oasis-open.org
>>> List archive: https://lists.oasis-open.org/archives/virtio-comment/
>>> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
>>> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
>>> Committee: https://www.oasis-open.org/committees/virtio/
>>> Join OASIS: https://www.oasis-open.org/join/
>>>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org