From mboxrd@z Thu Jan 1 00:00:00 1970 From: Neo Jia Subject: Re: [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU. Date: Tue, 16 Feb 2016 00:48:55 -0800 Message-ID: <20160216084855.GA7717@nvidia.com> References: <1454488111.4967.39.camel@redhat.com> <1454527963.18969.8.camel@redhat.com> <20160216071304.GA6867@nvidia.com> <20160216073647.GB6867@nvidia.com> <20160216075310.GC6867@nvidia.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: Alex Williamson , Gerd Hoffmann , Kirti Wankhede , Paolo Bonzini , "Ruan, Shuai" , "Song, Jike" , "Lv, Zhiyuan" , "kvm@vger.kernel.org" , qemu-devel To: "Tian, Kevin" Return-path: Received: from hqemgate16.nvidia.com ([216.228.121.65]:10249 "EHLO hqemgate16.nvidia.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752195AbcBPItB convert rfc822-to-8bit (ORCPT ); Tue, 16 Feb 2016 03:49:01 -0500 Content-Disposition: inline In-Reply-To: Sender: kvm-owner@vger.kernel.org List-ID: On Tue, Feb 16, 2016 at 08:10:42AM +0000, Tian, Kevin wrote: > > From: Neo Jia [mailto:cjia@nvidia.com] > > Sent: Tuesday, February 16, 2016 3:53 PM > >=20 > > On Tue, Feb 16, 2016 at 07:40:47AM +0000, Tian, Kevin wrote: > > > > From: Neo Jia [mailto:cjia@nvidia.com] > > > > Sent: Tuesday, February 16, 2016 3:37 PM > > > > > > > > On Tue, Feb 16, 2016 at 07:27:09AM +0000, Tian, Kevin wrote: > > > > > > From: Neo Jia [mailto:cjia@nvidia.com] > > > > > > Sent: Tuesday, February 16, 2016 3:13 PM > > > > > > > > > > > > On Tue, Feb 16, 2016 at 06:49:30AM +0000, Tian, Kevin wrote= : > > > > > > > > From: Alex Williamson [mailto:alex.williamson@redhat.co= m] > > > > > > > > Sent: Thursday, February 04, 2016 3:33 AM > > > > > > > > > > > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote: > > > > > > > > > =A0 Hi, > > > > > > > > > > > > > > > > > > > Actually I have a long puzzle in this area. Definit= ely libvirt will use UUID > > to > > > > > > > > > > mark a VM. And obviously UUID is not recorded withi= n KVM. Then how > > does > > > > > > > > > > libvirt talk to KVM based on UUID? It could be a go= od reference to this > > design. > > > > > > > > > > > > > > > > > > libvirt keeps track which qemu instance belongs to wh= ich vm. > > > > > > > > > qemu also gets started with "-uuid ...", so one can q= uery qemu via > > > > > > > > > monitor ("info uuid") to figure what the uuid is.=A0=A0= It is also in the > > > > > > > > > smbios tables so the guest can see it in the system i= nformation table. > > > > > > > > > > > > > > > > > > The uuid is not visible to the kernel though, the kvm= kernel driver > > > > > > > > > doesn't know what the uuid is (and neither does vfio)= =2E=A0=A0qemu uses file > > > > > > > > > handles to talk to both kvm and vfio.=A0=A0qemu notif= ies both kvm and vfio > > > > > > > > > about anything relevant events (guest address space c= hanges etc) and > > > > > > > > > connects file descriptors (eventfd -> irqfd). > > > > > > > > > > > > > > > > I think the original link to using a VM UUID for the vG= PU comes from > > > > > > > > NVIDIA having a userspace component which might get lau= nched from a udev > > > > > > > > event as the vGPU is created or the set of vGPUs within= that UUID is > > > > > > > > started.=A0=A0Using the VM UUID then gives them a way t= o associate that > > > > > > > > userspace process with a VM instance.=A0=A0Maybe it cou= ld register with > > > > > > > > libvirt for some sort of service provided for the VM, I= don't know. > > > > > > > > > > > > > > Intel doesn't have this requirement. It should be enough = as long as > > > > > > > libvirt maintains which sysfs vgpu node is associated to = a VM UUID. > > > > > > > > > > > > > > > > > > > > > > > > qemu needs a sysfs node as handle to the vfio device,= something > > > > > > > > > like /sys/devices/virtual/vgpu/.=A0=A0 ca= n be a uuid if you > > want > > > > > > > > > have it that way, but it could be pretty much anythin= g.=A0=A0The sysfs node > > > > > > > > > will probably show up as-is in the libvirt xml when a= ssign a vgpu to a > > > > > > > > > vm.=A0=A0So the name should be something stable (i.e.= when using a uuid as > > > > > > > > > name you should better not generate a new one on each= boot). > > > > > > > > > > > > > > > > Actually I don't think there's really a persistent nami= ng issue, that's > > > > > > > > probably where we diverge from the SR-IOV model.=A0=A0S= R-IOV cannot > > > > > > > > dynamically add a new VF, it needs to reset the number = of VFs to zero, > > > > > > > > then re-allocate all of them up to the new desired coun= t.=A0=A0That has some > > > > > > > > obvious implications.=A0=A0I think with both vendors he= re, we can > > > > > > > > dynamically allocate new vGPUs, so I would expect that = libvirt would > > > > > > > > create each vGPU instance as it's needed.=A0=A0None wou= ld be created by > > > > > > > > default without user interaction. > > > > > > > > > > > > > > > > Personally I think using a UUID makes sense, but it nee= ds to be > > > > > > > > userspace policy whether that UUID has any implicit mea= ning like > > > > > > > > matching the VM UUID.=A0=A0Having an index within a UUI= D bothers me a bit, > > > > > > > > but it doesn't seem like too much of a concession to en= able the use case > > > > > > > > that NVIDIA is trying to achieve.=A0=A0Thanks, > > > > > > > > > > > > > > > > > > > > > > I would prefer to making UUID an optional parameter, whil= e not tieing > > > > > > > sysfs vgpu naming to UUID. This would be more flexible to= different > > > > > > > scenarios where UUID might not be required. > > > > > > > > > > > > Hi Kevin, > > > > > > > > > > > > Happy Chinese New Year! > > > > > > > > > > > > I think having UUID as the vgpu device name will allow us t= o have an gpu vendor > > > > > > agnostic solution for the upper layer software stack such a= s QEMU, who is > > > > > > supposed to open the device. > > > > > > > > > > > > > > > > Qemu can use whatever sysfs path provided to open the device,= regardless > > > > > of whether there is an UUID within the path... > > > > > > > > > > > > > Hi Kevin, > > > > > > > > Then it will provide even more benefit of using UUID as libvirt= can be > > > > implemented as gpu vendor agnostic, right? :-) > > > > > > > > The UUID can be VM UUID or vGPU group object UUID which really = depends on the > > > > high level software stack, again the benefit is gpu vendor agno= stic. > > > > > > > > > > There is case where libvirt is not used while another mgmt. stack= doesn't use > > > UUID, e.g. in some Xen scenarios. So it's not about GPU vendor ag= nostic. It's > > > about high level mgmt. stack agnostic. That's why we need make UU= ID as > > > optional in this vGPU-core framework. > >=20 > > Hi Kevin, > >=20 > > As long as you have to create an object to represent vGPU or vGPU g= roup, you > > will have UUID, no matter which management stack you are going to u= se. > >=20 > > UUID is the most agnostic way to represent an object, I think. > >=20 > > (a bit off topic since we are supposed to focus on VFIO on KVM) > >=20 > > Since now you are talking about Xen, I am very happy to discuss tha= t with you. > > You can check how Xen has managed its object via UUID in xapi. > >=20 >=20 > Well, I'm not the expert in this area. IMHO UUID is just an user leve= l > attribute, which can be associated to any sysfs node and managed by > mgmt. stack itself, and then the sysfs path can be opened as the > bridge between user/kernel. I don't understand the necessity of bindi= ng=20 > UUID internally within vGPU core framework here. Alex gave one exampl= e > of udev, but I didn't quite catch why only UUID can work there. Maybe > you can elaborate that requirement. Hi Kevin, UUID is just a way to represent an object. It is not binding, it is just a representation. I think here we are jus= t creating a convenient and generic way to represent a virtual gpu device= on sysfs. Having the UUID as part of the virtual gpu device name allows us easily= find out the mapping. UUID can be anything, you can always use an UUID to present VMID in the= example you listed below, so you are actually gaining flexibility by using UUID= instead of VMID as it can be supported by both KVM and Xen. :-) Thanks, Neo >=20 > P.S. taking my daily Xen development experience for example, I just u= se=20 > xl w/o need to bother managing UUID (Xen hypervisor only uses VMID > instead of UUID). I don't want to eliminate such flexibility in this = design. :-) >=20 > Thanks > Kevin