From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:40164) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aVtZ0-0003PS-7u for qemu-devel@nongnu.org; Tue, 16 Feb 2016 23:18:08 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aVtYv-0006Tj-5J for qemu-devel@nongnu.org; Tue, 16 Feb 2016 23:18:06 -0500 Received: from hqemgate14.nvidia.com ([216.228.121.143]:17355) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aVtYu-0006Td-ND for qemu-devel@nongnu.org; Tue, 16 Feb 2016 23:18:00 -0500 Date: Tue, 16 Feb 2016 20:17:43 -0800 From: Neo Jia Message-ID: <20160217041743.GA7903@nvidia.com> References: <1454527963.18969.8.camel@redhat.com> <20160216071304.GA6867@nvidia.com> <20160216073647.GB6867@nvidia.com> <20160216075310.GC6867@nvidia.com> <20160216084855.GA7717@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: Subject: Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU. List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Tian, Kevin" Cc: "Ruan, Shuai" , "Song, Jike" , "kvm@vger.kernel.org" , Kirti Wankhede , qemu-devel , Alex Williamson , Gerd Hoffmann , Paolo Bonzini , "Lv, Zhiyuan" On Wed, Feb 17, 2016 at 03:31:24AM +0000, Tian, Kevin wrote: > > From: Neo Jia [mailto:cjia@nvidia.com] > > Sent: Tuesday, February 16, 2016 4:49 PM > >=20 > > On Tue, Feb 16, 2016 at 08:10:42AM +0000, Tian, Kevin wrote: > > > > From: Neo Jia [mailto:cjia@nvidia.com] > > > > Sent: Tuesday, February 16, 2016 3:53 PM > > > > > > > > On Tue, Feb 16, 2016 at 07:40:47AM +0000, Tian, Kevin wrote: > > > > > > From: Neo Jia [mailto:cjia@nvidia.com] > > > > > > Sent: Tuesday, February 16, 2016 3:37 PM > > > > > > > > > > > > On Tue, Feb 16, 2016 at 07:27:09AM +0000, Tian, Kevin wrote: > > > > > > > > From: Neo Jia [mailto:cjia@nvidia.com] > > > > > > > > Sent: Tuesday, February 16, 2016 3:13 PM > > > > > > > > > > > > > > > > On Tue, Feb 16, 2016 at 06:49:30AM +0000, Tian, Kevin wrote= : > > > > > > > > > > From: Alex Williamson [mailto:alex.williamson@redhat.co= m] > > > > > > > > > > Sent: Thursday, February 04, 2016 3:33 AM > > > > > > > > > > > > > > > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote: > > > > > > > > > > > =A0 Hi, > > > > > > > > > > > > > > > > > > > > > > > Actually I have a long puzzle in this area. Definit= ely libvirt will use > > UUID > > > > to > > > > > > > > > > > > mark a VM. And obviously UUID is not recorded withi= n KVM. Then > > how > > > > does > > > > > > > > > > > > libvirt talk to KVM based on UUID? It could be a go= od reference to > > this > > > > design. > > > > > > > > > > > > > > > > > > > > > > libvirt keeps track which qemu instance belongs to wh= ich vm. > > > > > > > > > > > qemu also gets started with "-uuid ...", so one can q= uery qemu via > > > > > > > > > > > monitor ("info uuid") to figure what the uuid is.=A0= =A0It is also in the > > > > > > > > > > > smbios tables so the guest can see it in the system i= nformation table. > > > > > > > > > > > > > > > > > > > > > > The uuid is not visible to the kernel though, the kvm= kernel driver > > > > > > > > > > > doesn't know what the uuid is (and neither does vfio)= .=A0=A0qemu uses > > file > > > > > > > > > > > handles to talk to both kvm and vfio.=A0=A0qemu notif= ies both kvm and > > vfio > > > > > > > > > > > about anything relevant events (guest address space c= hanges etc) > > and > > > > > > > > > > > connects file descriptors (eventfd -> irqfd). > > > > > > > > > > > > > > > > > > > > I think the original link to using a VM UUID for the vG= PU comes from > > > > > > > > > > NVIDIA having a userspace component which might get lau= nched from > > a udev > > > > > > > > > > event as the vGPU is created or the set of vGPUs within= that UUID is > > > > > > > > > > started.=A0=A0Using the VM UUID then gives them a way t= o associate that > > > > > > > > > > userspace process with a VM instance.=A0=A0Maybe it cou= ld register with > > > > > > > > > > libvirt for some sort of service provided for the VM, I= don't know. > > > > > > > > > > > > > > > > > > Intel doesn't have this requirement. It should be enough = as long as > > > > > > > > > libvirt maintains which sysfs vgpu node is associated to = a VM UUID. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > qemu needs a sysfs node as handle to the vfio device,= something > > > > > > > > > > > like /sys/devices/virtual/vgpu/.=A0=A0 ca= n be a uuid if > > you > > > > want > > > > > > > > > > > have it that way, but it could be pretty much anythin= g.=A0=A0The sysfs node > > > > > > > > > > > will probably show up as-is in the libvirt xml when a= ssign a vgpu to > > a > > > > > > > > > > > vm.=A0=A0So the name should be something stable (i.e.= when using a uuid > > as > > > > > > > > > > > name you should better not generate a new one on each= boot). > > > > > > > > > > > > > > > > > > > > Actually I don't think there's really a persistent nami= ng issue, that's > > > > > > > > > > probably where we diverge from the SR-IOV model.=A0=A0S= R-IOV cannot > > > > > > > > > > dynamically add a new VF, it needs to reset the number = of VFs to zero, > > > > > > > > > > then re-allocate all of them up to the new desired coun= t.=A0=A0That has some > > > > > > > > > > obvious implications.=A0=A0I think with both vendors he= re, we can > > > > > > > > > > dynamically allocate new vGPUs, so I would expect that = libvirt would > > > > > > > > > > create each vGPU instance as it's needed.=A0=A0None wou= ld be created by > > > > > > > > > > default without user interaction. > > > > > > > > > > > > > > > > > > > > Personally I think using a UUID makes sense, but it nee= ds to be > > > > > > > > > > userspace policy whether that UUID has any implicit mea= ning like > > > > > > > > > > matching the VM UUID.=A0=A0Having an index within a UUI= D bothers me a > > bit, > > > > > > > > > > but it doesn't seem like too much of a concession to en= able the use case > > > > > > > > > > that NVIDIA is trying to achieve.=A0=A0Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would prefer to making UUID an optional parameter, whil= e not tieing > > > > > > > > > sysfs vgpu naming to UUID. This would be more flexible to= different > > > > > > > > > scenarios where UUID might not be required. > > > > > > > > > > > > > > > > Hi Kevin, > > > > > > > > > > > > > > > > Happy Chinese New Year! > > > > > > > > > > > > > > > > I think having UUID as the vgpu device name will allow us t= o have an gpu > > vendor > > > > > > > > agnostic solution for the upper layer software stack such a= s QEMU, who is > > > > > > > > supposed to open the device. > > > > > > > > > > > > > > > > > > > > > > Qemu can use whatever sysfs path provided to open the device,= regardless > > > > > > > of whether there is an UUID within the path... > > > > > > > > > > > > > > > > > > > Hi Kevin, > > > > > > > > > > > > Then it will provide even more benefit of using UUID as libvirt= can be > > > > > > implemented as gpu vendor agnostic, right? :-) > > > > > > > > > > > > The UUID can be VM UUID or vGPU group object UUID which really = depends on > > the > > > > > > high level software stack, again the benefit is gpu vendor agno= stic. > > > > > > > > > > > > > > > > There is case where libvirt is not used while another mgmt. stack= doesn't use > > > > > UUID, e.g. in some Xen scenarios. So it's not about GPU vendor ag= nostic. It's > > > > > about high level mgmt. stack agnostic. That's why we need make UU= ID as > > > > > optional in this vGPU-core framework. > > > > > > > > Hi Kevin, > > > > > > > > As long as you have to create an object to represent vGPU or vGPU g= roup, you > > > > will have UUID, no matter which management stack you are going to u= se. > > > > > > > > UUID is the most agnostic way to represent an object, I think. > > > > > > > > (a bit off topic since we are supposed to focus on VFIO on KVM) > > > > > > > > Since now you are talking about Xen, I am very happy to discuss tha= t with you. > > > > You can check how Xen has managed its object via UUID in xapi. > > > > > > > > > > Well, I'm not the expert in this area. IMHO UUID is just an user leve= l > > > attribute, which can be associated to any sysfs node and managed by > > > mgmt. stack itself, and then the sysfs path can be opened as the > > > bridge between user/kernel. I don't understand the necessity of bindi= ng > > > UUID internally within vGPU core framework here. Alex gave one exampl= e > > > of udev, but I didn't quite catch why only UUID can work there. Maybe > > > you can elaborate that requirement. > >=20 > > Hi Kevin, > >=20 > > UUID is just a way to represent an object. > >=20 > > It is not binding, it is just a representation. I think here we are jus= t > > creating a convenient and generic way to represent a virtual gpu device= on > > sysfs. > >=20 > > Having the UUID as part of the virtual gpu device name allows us easily= find out > > the mapping. > >=20 > > UUID can be anything, you can always use an UUID to present VMID in the= example > > you listed below, so you are actually gaining flexibility by using UUID= instead > > of VMID as it can be supported by both KVM and Xen. :-) > >=20 > > Thanks, > > Neo > >=20 >=20 > Thanks Neo. I understand UUID has its merit in many usages. As you > may see from my earlier reply, my main concern is whether it's a must > to record this information within kernel vGPU-core framework. We can > still make it hypervisor agnostic even not using UUID, as long as there's > a unified namespace created for all vgpus, like: > vgpu-vendor-0, vgpu-vendor-1, ... >=20 > Then high level mgmt. stack can associate UUID to that namespace. So > I hope you can help elaborate below description: >=20 > > Having the UUID as part of the virtual gpu device name allows us easily= find out > > the mapping. >=20 Hi Kevin, The answer is simple, having a UUID as part of the device name will give yo= u a unique sysfs path that will be opened by QEMU. vgpu-vendor-0 and vgpu-vendor-1 will not be unique as we can have multiple virtual gpu devices per VM coming from same or different physical devices. If you are worried about losing meaningful name here, we can create a sysfs= file to capture the vendor device description if you like. Thanks, Neo > so we can evaluate carefully whether UUID must be included in the vgpu > device name or not. If not using UUID is OK, it can remove one more > user space dependency on using vgpu features. >=20 > Thanks > Kevin