From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kirti Wankhede Subject: Re: [libvirt] [PATCH v7 0/4] Add Mediated device support Date: Sat, 3 Sep 2016 23:17:41 +0530 Message-ID: <3b920471-c69a-ccac-aa53-ae258442e82e@nvidia.com> References: <1472097235-6332-1-git-send-email-kwankhede@nvidia.com> <20160830101638.49df467d@t450s.home> <78fedd65-6d62-e849-ff3b-d5105b2da816@redhat.com> <20160901105948.62f750aa@t450s.home> <98bbdbbf-c388-9120-3306-64f0cfb820a7@nvidia.com> <8682faeb-0331-f014-c13e-03c20f3f2bdf@redhat.com> <2a195ed1-f6aa-ffab-3f5c-4121de264d05@redhat.com> <3169fcf3-1c1f-38cc-eb6d-3e8b4b8b1dd9@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Cc: "Song, Jike" , "cjia@nvidia.com" , "kvm@vger.kernel.org" , "libvir-list@redhat.com" , "Tian, Kevin" , "qemu-devel@nongnu.org" , "kraxel@redhat.com" , Laine Stump , "bjsdjshi@linux.vnet.ibm.com" To: Paolo Bonzini , John Ferlan , Michal Privoznik , Alex Williamson Return-path: In-Reply-To: <3169fcf3-1c1f-38cc-eb6d-3e8b4b8b1dd9@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Sender: "Qemu-devel" List-Id: kvm.vger.kernel.org On 9/3/2016 6:37 PM, Paolo Bonzini wrote: >=20 >=20 > On 03/09/2016 13:56, John Ferlan wrote: >> On 09/02/2016 05:48 PM, Paolo Bonzini wrote: >>> On 02/09/2016 20:33, Kirti Wankhede wrote: >>>> We could even do: >>>>>> >>>>>> echo $UUID1:$GROUPA > create >>>>>> >>>>>> where $GROUPA is the group ID of a previously created mdev device in= to >>>>>> which $UUID1 is to be created and added to the same group. >>>> >>> >>> >From the point of view of libvirt, I think I prefer Alex's idea. >>> could be an additional element in the nodedev-create XML: >>> >>> >>> my-vgpu >>> pci_0000_86_00_0 >>> >>> >>> 0695d332-7831-493f-9e71-1c85c8911a08 >>> group1 >>> >>> >>> >>> (should group also be a UUID?) >> I replied to earlier mail too, group number doesn't need to be UUID. It should be a unique number. I think in the discussion in bof someone mentioned about using domain's unique number that libvirt generates. That should also work. >> As long as create_group handles all the work and all libvirt does is >> call it, get the return status/error, and handle deleting the vGPU on >> error, then I guess it's doable. >> Yes that is the idea. Libvirt doesn't have to care about the groups. With Alex's proposal, as you mentioned above, libvirt have to provide group number to mdev_create, check return status and handle error case. echo $UUID1:$GROUP1 > mdev_create echo $UUID2:$GROUP1 > mdev_create would create two mdev devices assigned to same domain. >> Alternatively having multiple in the XML and performing = a >> single *mdev/create_group is an option. >=20 > I don't really like the idea of a single nodedev-create creating > multiple devices, but that would work too. >=20 >> That is, what is the "output" from create_group that gets added to the >> domain XML? How is that found? >=20 > A new sysfs path is created, whose name depends on the UUID. The UUID > is used in a element in the domain XML and the sysfs path > appears in the QEMU command line. Kirti and Neo had examples in their > presentation at KVM Forum. >=20 > If you create multiple devices in the same group, they are added to the > same IOMMU group so they must be used by the same VM. However they > don't have to be available from the beginning; they could be > hotplugged/hot-unplugged later, since from the point of view of the VM > those are just another PCI device. >=20 >> Also, once the domain is running can a >> vGPU be added to the group? Removed? What allows/prevents? >=20 > Kirti?... :) Yes, vGPU could be hot-plugged or hot-unplugged. This also depends on does vendor driver want to support that. For example, domain is running with two vGPUs $UUID1 and $UUID2 and user tried to hot-unplug vGPU $UUID2, vendor driver knows that domain is running and vGPU is being used in guest, so vendor driver can fail offline/close() call if they don't support hot-unplug. Similarly for hot-plug vendor driver can fail create call to not to support hot-plug. >=20 > In principle I don't think anything should block vGPUs from different > groups being added to the same VM, but I have to defer to Alex and Kirti > again on this. >=20 No, there should be one group per VM. >>> Since John brought up the topic of minimal XML, in this case it will be >>> like this: >>> >>> >>> my-vgpu >>> pci_0000_86_00_0 >>> >>> >>> >>> >>> >>> The uuid will be autogenerated by libvirt and if there's no (as >>> is common for VMs with only 1 vGPU) it will be a single-device group. >> >> The could be ignored as it seems existing libvirt code wants to >> generate a name via udevGenerateDeviceName for other devices. I haven't >> studied it long enough, but I believe that's how those pci_####* names >> created. >=20 > Yeah that makes sense. So we get down to a minimal XML that has just > parent, and capability with type in it; additional elements could be > name (ignored anyway), and within capability uuid and group. > Yes, this seems good. I would like to have one more capability here. Pulling here some suggestion from my previous mail: In the directory structure, a 'params' can take optional parameters. Libvirt then can set 'params' and then create mdev device. For example, param say 'disable_console_vnc=3D1' is set for type 11, then devices created of type 11 will have that param set unless it is cleared. =E2=94=94=E2=94=80=E2=94=80 mdev_supported_types =E2=94=9C=E2=94=80=E2=94=80 11 =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 create =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 description =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 max_instances =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 params =E2=94=9C=E2=94=80=E2=94=80 12 =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 create =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 description =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 max_instances =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 params =E2=94=94=E2=94=80=E2=94=80 13 =E2=94=9C=E2=94=80=E2=94=80 create =E2=94=9C=E2=94=80=E2=94=80 description =E2=94=94=E2=94=80=E2=94=80 max_instances =E2=94=94=E2=94=80=E2=94=80 params So with that XML format would be: my-vgpu pci_0000_86_00_0 group1 disable_console_vnc=3D1 and 'params' field should be just a string to libvirt and its optional also. If user want to provide extra parameter while creating vGPU device they should provide it in XML file as above to nodedev-create. Very initial proposal was to have this extra paramter list as a string to mdev_create itself as: echo $UUID1:$PARAMS > mdev_create I would like to know others opinions on whether it should be part of mdev_create input or a separate write to 'params' file in sysfs as in above directory structure. Kirti. > Thanks, >=20 > Paolo >=20