From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46796) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aVaRh-0002x0-Fj for qemu-devel@nongnu.org; Tue, 16 Feb 2016 02:53:19 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aVaRe-0001Mf-7W for qemu-devel@nongnu.org; Tue, 16 Feb 2016 02:53:17 -0500 Received: from hqemgate14.nvidia.com ([216.228.121.143]:11915) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aVaRd-0001MJ-NT for qemu-devel@nongnu.org; Tue, 16 Feb 2016 02:53:13 -0500 Date: Mon, 15 Feb 2016 23:53:10 -0800 From: Neo Jia Message-ID: <20160216075310.GC6867@nvidia.com> References: <20160202083114.GB9895@nvidia.com> <1454433079.30910.3.camel@redhat.com> <1454488111.4967.39.camel@redhat.com> <1454527963.18969.8.camel@redhat.com> <20160216071304.GA6867@nvidia.com> <20160216073647.GB6867@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: Subject: Re: [Qemu-devel] [RFC PATCH v1 1/1] vGPU core driver : to provide common interface for vGPU. List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Tian, Kevin" Cc: "Ruan, Shuai" , "Song, Jike" , "kvm@vger.kernel.org" , Kirti Wankhede , qemu-devel , Alex Williamson , Gerd Hoffmann , Paolo Bonzini , "Lv, Zhiyuan" On Tue, Feb 16, 2016 at 07:40:47AM +0000, Tian, Kevin wrote: > > From: Neo Jia [mailto:cjia@nvidia.com] > > Sent: Tuesday, February 16, 2016 3:37 PM > >=20 > > On Tue, Feb 16, 2016 at 07:27:09AM +0000, Tian, Kevin wrote: > > > > From: Neo Jia [mailto:cjia@nvidia.com] > > > > Sent: Tuesday, February 16, 2016 3:13 PM > > > > > > > > On Tue, Feb 16, 2016 at 06:49:30AM +0000, Tian, Kevin wrote: > > > > > > From: Alex Williamson [mailto:alex.williamson@redhat.com] > > > > > > Sent: Thursday, February 04, 2016 3:33 AM > > > > > > > > > > > > On Wed, 2016-02-03 at 09:28 +0100, Gerd Hoffmann wrote: > > > > > > > =A0 Hi, > > > > > > > > > > > > > > > Actually I have a long puzzle in this area. Definitely libv= irt will use UUID to > > > > > > > > mark a VM. And obviously UUID is not recorded within KVM. T= hen how does > > > > > > > > libvirt talk to KVM based on UUID? It could be a good refer= ence to this design. > > > > > > > > > > > > > > libvirt keeps track which qemu instance belongs to which vm. > > > > > > > qemu also gets started with "-uuid ...", so one can query qem= u via > > > > > > > monitor ("info uuid") to figure what the uuid is.=A0=A0It is = also in the > > > > > > > smbios tables so the guest can see it in the system informati= on table. > > > > > > > > > > > > > > The uuid is not visible to the kernel though, the kvm kernel = driver > > > > > > > doesn't know what the uuid is (and neither does vfio).=A0=A0q= emu uses file > > > > > > > handles to talk to both kvm and vfio.=A0=A0qemu notifies both= kvm and vfio > > > > > > > about anything relevant events (guest address space changes e= tc) and > > > > > > > connects file descriptors (eventfd -> irqfd). > > > > > > > > > > > > I think the original link to using a VM UUID for the vGPU comes= from > > > > > > NVIDIA having a userspace component which might get launched fr= om a udev > > > > > > event as the vGPU is created or the set of vGPUs within that UU= ID is > > > > > > started.=A0=A0Using the VM UUID then gives them a way to associ= ate that > > > > > > userspace process with a VM instance.=A0=A0Maybe it could regis= ter with > > > > > > libvirt for some sort of service provided for the VM, I don't k= now. > > > > > > > > > > Intel doesn't have this requirement. It should be enough as long = as > > > > > libvirt maintains which sysfs vgpu node is associated to a VM UUI= D. > > > > > > > > > > > > > > > > > > qemu needs a sysfs node as handle to the vfio device, somethi= ng > > > > > > > like /sys/devices/virtual/vgpu/.=A0=A0 can be a u= uid if you want > > > > > > > have it that way, but it could be pretty much anything.=A0=A0= The sysfs node > > > > > > > will probably show up as-is in the libvirt xml when assign a = vgpu to a > > > > > > > vm.=A0=A0So the name should be something stable (i.e. when us= ing a uuid as > > > > > > > name you should better not generate a new one on each boot). > > > > > > > > > > > > Actually I don't think there's really a persistent naming issue= , that's > > > > > > probably where we diverge from the SR-IOV model.=A0=A0SR-IOV ca= nnot > > > > > > dynamically add a new VF, it needs to reset the number of VFs t= o zero, > > > > > > then re-allocate all of them up to the new desired count.=A0=A0= That has some > > > > > > obvious implications.=A0=A0I think with both vendors here, we c= an > > > > > > dynamically allocate new vGPUs, so I would expect that libvirt = would > > > > > > create each vGPU instance as it's needed.=A0=A0None would be cr= eated by > > > > > > default without user interaction. > > > > > > > > > > > > Personally I think using a UUID makes sense, but it needs to be > > > > > > userspace policy whether that UUID has any implicit meaning lik= e > > > > > > matching the VM UUID.=A0=A0Having an index within a UUID bother= s me a bit, > > > > > > but it doesn't seem like too much of a concession to enable the= use case > > > > > > that NVIDIA is trying to achieve.=A0=A0Thanks, > > > > > > > > > > > > > > > > I would prefer to making UUID an optional parameter, while not ti= eing > > > > > sysfs vgpu naming to UUID. This would be more flexible to differe= nt > > > > > scenarios where UUID might not be required. > > > > > > > > Hi Kevin, > > > > > > > > Happy Chinese New Year! > > > > > > > > I think having UUID as the vgpu device name will allow us to have a= n gpu vendor > > > > agnostic solution for the upper layer software stack such as QEMU, = who is > > > > supposed to open the device. > > > > > > > > > > Qemu can use whatever sysfs path provided to open the device, regardl= ess > > > of whether there is an UUID within the path... > > > > >=20 > > Hi Kevin, > >=20 > > Then it will provide even more benefit of using UUID as libvirt can be > > implemented as gpu vendor agnostic, right? :-) > >=20 > > The UUID can be VM UUID or vGPU group object UUID which really depends = on the > > high level software stack, again the benefit is gpu vendor agnostic. > >=20 >=20 > There is case where libvirt is not used while another mgmt. stack doesn't= use > UUID, e.g. in some Xen scenarios. So it's not about GPU vendor agnostic. = It's > about high level mgmt. stack agnostic. That's why we need make UUID as > optional in this vGPU-core framework. Hi Kevin, As long as you have to create an object to represent vGPU or vGPU group, yo= u will have UUID, no matter which management stack you are going to use. UUID is the most agnostic way to represent an object, I think. (a bit off topic since we are supposed to focus on VFIO on KVM) Since now you are talking about Xen, I am very happy to discuss that with y= ou. You can check how Xen has managed its object via UUID in xapi. Thanks, Neo >=20 > Thanks > Kevin