From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel Date: Thu, 19 Nov 2015 21:22:51 -0700 Message-ID: <1447993371.4697.257.camel@redhat.com> References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by gabe.freedesktop.org (Postfix) with ESMTPS id 81FC289DF7 for ; Thu, 19 Nov 2015 20:22:55 -0800 (PST) In-Reply-To: <564E8C51.6070706@intel.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: Jike Song Cc: "igvt-g@ml01.01.org" , "Reddy, Raghuveer" , Stefano Stabellini , "White, Michael L" , "Cowperthwaite, David J" , "intel-gfx@lists.freedesktop.org" , "Li, Susie" , "Dong, Eddie" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xen.org" , qemu-devel , "Zhou, Chao" , Paolo Bonzini , "Zhu, Libo" , "Wang, Hongbo" List-Id: intel-gfx@lists.freedesktop.org T24gRnJpLCAyMDE1LTExLTIwIGF0IDEwOjU4ICswODAwLCBKaWtlIFNvbmcgd3JvdGU6Cj4gT24g MTEvMTkvMjAxNSAxMTo1MiBQTSwgQWxleCBXaWxsaWFtc29uIHdyb3RlOgo+ID4gT24gVGh1LCAy MDE1LTExLTE5IGF0IDE1OjMyICswMDAwLCBTdGVmYW5vIFN0YWJlbGxpbmkgd3JvdGU6Cj4gPj4g T24gVGh1LCAxOSBOb3YgMjAxNSwgSmlrZSBTb25nIHdyb3RlOgo+ID4+PiBIaSBBbGV4LCB0aGFu a3MgZm9yIHRoZSBkaXNjdXNzaW9uLgo+ID4+Pgo+ID4+PiBJbiBhZGRpdGlvbiB0byBLZXZpbidz IHJlcGxpZXMsIEkgaGF2ZSBhIGhpZ2gtbGV2ZWwgcXVlc3Rpb246IGNhbiBWRklPCj4gPj4+IGJl IHVzZWQgYnkgUUVNVSBmb3IgYm90aCBLVk0gYW5kIFhlbj8KPiA+Pgo+ID4+IE5vLiBWRklPIGNh bm5vdCBiZSB1c2VkIHdpdGggWGVuIHRvZGF5LiBXaGVuIHJ1bm5pbmcgb24gWGVuLCB0aGUgSU9N TVUKPiA+PiBpcyBvd25lZCBieSBYZW4uCj4gPgo+ID4gUmlnaHQsIGJ1dCBpbiB0aGlzIGNhc2Ug d2UncmUgdGFsa2luZyBhYm91dCBkZXZpY2UgTU1Vcywgd2hpY2ggYXJlIG93bmVkCj4gPiBieSB0 aGUgZGV2aWNlIGRyaXZlciB3aGljaCBJIHRoaW5rIGlzIHJ1bm5pbmcgaW4gZG9tMCwgcmlnaHQ/ ICBUaGlzCj4gPiBwcm9wb3NhbCBkb2Vzbid0IHJlcXVpcmUgc3VwcG9ydCBvZiB0aGUgc3lzdGVt IElPTU1VLCB0aGUgZG9tMCBkcml2ZXIKPiA+IG1hcHMgSU9WQSB0cmFuc2xhdGlvbnMganVzdCBh cyBpdCB3b3VsZCBmb3IgaXRzZWxmLiAgV2UncmUgbGFyZ2VseQo+ID4gcHJvcG9zaW5nIHVzZSBv ZiB0aGUgVkZJTyBBUEkgdG8gcHJvdmlkZSBhIGNvbW1vbiBpbnRlcmZhY2UgdG8gZXhwb3NlIGEK PiA+IFBDSShlKSBkZXZpY2UgdG8gUUVNVSwgYnV0IHdoYXQgaGFwcGVucyBpbiB0aGUgdkdQVSB2 ZW5kb3IgZGV2aWNlIGFuZAo+ID4gSU9NTVUgYmFja2VuZHMgaXMgc3BlY2lmaWMgdG8gdGhlIGRl dmljZSBhbmQgcGVyaGFwcyBldmVuIHNwZWNpZmljIHRvCj4gPiB0aGUgaHlwZXJ2aXNvci4gIFRo YW5rcywKPiAKPiBMZXQgbWUgY29uY2x1ZGUgdGhpcywgYW5kIHBsZWFzZSBjb3JyZWN0IG1lIGlu IGNhc2Ugb2YgYW55IG1pc3JlYWQ6IHRoZQo+IHZHUFUgaW50ZXJmYWNlIGJldHdlZW4ga2VybmVs IGFuZCBRRU1VIHdpbGwgYmUgdGhyb3VnaCBWRklPLCB3aXRoIGEgbmV3Cj4gVkZJTyBiYWNrZW5k IChpbnN0ZWFkIG9mIHRoZSBleGlzdGluZyB0eXBlMSksIGZvciBib3RoIEtWTUdUIGFuZCBYZW5H VD8KCk15IHByaW1hcnkgY29uY2VybiBpcyBLVk0gYW5kIFFFTVUgdXBzdHJlYW0sIHRoZSBwcm9w b3NhbCBpcyBub3QKc3BlY2lmaWNhbGx5IGRpcmVjdGVkIGF0IFhlbkdULCBidXQgZG9lcyBub3Qg ZXhjbHVkZSBpdCBlaXRoZXIuICBYZW4gaXMKd2VsY29tZSB0byBhZG9wdCB0aGlzIHByb3Bvc2Fs IGFzIHdlbGwsIGl0IHNpbXBseSBkZWZpbmVzIHRoZSBjaGFubmVsCnRocm91Z2ggd2hpY2ggdkdQ VXMgYXJlIGV4cG9zZWQgdG8gUUVNVSBhcyB0aGUgVkZJTyBBUEkuICBUaGUgY29yZSBWRklPCmNv ZGUgaW4gdGhlIExpbnV4IGtlcm5lbCBpcyBqdXN0IGFzIGF2YWlsYWJsZSBmb3IgdXNlIGluIFhl biBkb20wIGFzIGl0CmlzIGZvciBhIEtWTSBob3N0LiAgVkZJTyBpbiBRRU1VIGNlcnRhaW5seSBr bm93cyBhYm91dCBzb21lCmFjY2VsZXJhdGlvbnMgZm9yIEtWTSwgYnV0IHRoZXNlIGFyZSBhbG1v c3QgZW50aXJlbHkgYXJvdW5kIGFsbG93aW5nCmV2ZW50ZmQgYmFzZWQgaW50ZXJydXB0cyB0byBi ZSBpbmplY3RlZCB0aHJvdWdoIEtWTSwgd2hpY2ggaXMgc29tZXRoaW5nCkknbSBzdXJlIFhlbiBj b3VsZCBwcm92aWRlIGFzIHdlbGwuICBUaGVzZSBhY2NlbGVyYXRpb25zIGFyZSBhbHNvIG5vdApy ZXF1aXJlZCwgVkZJTyBiYXNlZCBkZXZpY2UgYXNzaWdubWVudCBpbiBRRU1VIHdvcmtzIHdpdGgg b3Igd2l0aG91dApLVk0uICBMaWtld2lzZSwgdGhlIFZGSU8ga2VybmVsIGludGVyZmFjZSBrbm93 cyBub3RoaW5nIGFib3V0IEtWTSBhbmQKaGFzIG5vIGRlcGVuZGVuY2llcyBvbiBpdC4KClRoZXJl IGFyZSB0d28gY29tcG9uZW50cyB0byB0aGUgVkZJTyBBUEksIG9uZSBpcyB0aGUgdHlwZTEgY29t cGxpYW50CklPTU1VIGludGVyZmFjZSwgd2hpY2ggZm9yIHRoaXMgcHJvcG9zYWwgaXMgcmVhbGx5 IGRvaW5nIG5vdGhpbmcgbW9yZQp0aGFuIHRyYWNraW5nIHRoZSBIVkEgdG8gR1BBIG1hcHBpbmdz IGZvciB0aGUgVk0uICBUaGlzIG11Y2ggc2VlbXMKZW50aXJlbHkgY29tbW9uIHJlZ2FyZGxlc3Mg b2YgdGhlIGh5cGVydmlzb3IuICBUaGUgb3RoZXIgcGFydCBpcyB0aGUKZGV2aWNlIGludGVyZmFj ZS4gIFRoZSBsaWZlY3ljbGUgb2YgdGhlIHZpcnR1YWwgZGV2aWNlIHNlZW1zIGxpa2UgaXQKd291 bGQgYmUgZW50aXJlbHkgc2hhcmVkLCBhcyBkb2VzIG11Y2ggb2YgdGhlIGVtdWxhdGlvbiBjb21w b25lbnRzIG9mCnRoZSBkZXZpY2UuICBXaGVuIHdlIGdldCB0byBwaW5uaW5nIHBhZ2VzLCBwcm92 aWRpbmcgZGlyZWN0IGFjY2VzcyB0bwptZW1vcnkgcmFuZ2VzIGZvciBhIFZNLCBhbmQgYWNjZWxl cmF0aW5nIGludGVycnVwdHMsIHRoZSB2R1BVIGRyaXZlcnMKd2lsbCBsaWtlbHkgbmVlZCBzb21l IHBlciBoeXBlcnZpc29yIGJyYW5jaGVzLCBidXQgdGhlc2UgYXJlIGFyZWFzIHdoZXJlCnRoYXQn cyB0cnVlIG5vIG1hdHRlciB3aGF0IHRoZSBpbnRlcmZhY2UuICBJJ20gcHJvYmFibHkgb3Zlcgpz aW1wbGlmeWluZywgYnV0IGhvcGVmdWxseSBub3QgdG9vIG11Y2gsIGNvcnJlY3QgbWUgaWYgSSdt IHdyb25nLgoKVGhlIGJlbmVmaXQgb2YgY291cnNlIGlzIHRoYXQgYXNpZGUgZnJvbSBzb21lIGV4 dGVuc2lvbnMgdG8gdGhlIEFQSSwgdGhlClFFTVUgY29tcG9uZW50cyBhcmUgYWxyZWFkeSBpbiBw bGFjZSBhbmQgdGhlcmUncyBhIGxvdCBtb3JlIGxldmVyYWdlIGZvcgpnZXR0aW5nIGJvdGggUUVN VSBhbmQgbGlidmlydCBzdXBwb3J0IHVwc3RyZWFtIGluIGJlaW5nIGFibGUgdG8gc3VwcG9ydApt dWx0aXBsZSB2ZW5kb3JzLCBwZXJoYXBzIG11bHRpcGxlIGh5cGVydmlzb3JzLCB3aXRoIHRoZSBz YW1lIGNvZGUuCkFsc28sIEknbSBub3Qgc3VyZSBob3cgdXNlZnVsIGl0IGlzLCBidXQgVkZJTyBp cyBhIHVzZXJzcGFjZSBkcml2ZXIKaW50ZXJmYWNlLCB3aGVyZSBoZXJlIHdlJ3JlIHByZWRvbWlu YW50bHkgdGFsa2luZyBhYm91dCB0aGF0IHVzZXJzcGFjZQpkcml2ZXIgYmVpbmcgUUVNVS4gIEl0 J3Mgbm90IGxpbWl0ZWQgdG8gdGhhdCB0aG91Z2guICBBIHVzZXJzcGFjZQpjb21wdXRlIGFwcGxp Y2F0aW9uIGNvdWxkIGhhdmUgZGlyZWN0IGFjY2VzcyB0byBhIHZHUFUgdGhyb3VnaCB0aGlzCm1v ZGVsLiAgVGhhbmtzLAoKQWxleAoKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX18KSW50ZWwtZ2Z4IG1haWxpbmcgbGlzdApJbnRlbC1nZnhAbGlzdHMuZnJlZWRl c2t0b3Aub3JnCmh0dHA6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9p bnRlbC1nZngK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1759271AbbKTEW4 (ORCPT ); Thu, 19 Nov 2015 23:22:56 -0500 Received: from mx1.redhat.com ([209.132.183.28]:52743 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1757944AbbKTEWz (ORCPT ); Thu, 19 Nov 2015 23:22:55 -0500 Message-ID: <1447993371.4697.257.camel@redhat.com> Subject: Re: [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel From: Alex Williamson To: Jike Song Cc: Stefano Stabellini , "Tian, Kevin" , "xen-devel@lists.xen.org" , "igvt-g@ml01.01.org" , "intel-gfx@lists.freedesktop.org" , "linux-kernel@vger.kernel.org" , "White, Michael L" , "Dong, Eddie" , "Li, Susie" , "Cowperthwaite, David J" , "Reddy, Raghuveer" , "Zhu, Libo" , "Zhou, Chao" , "Wang, Hongbo" , "Lv, Zhiyuan" , qemu-devel , Paolo Bonzini , Gerd Hoffmann Date: Thu, 19 Nov 2015 21:22:51 -0700 In-Reply-To: <564E8C51.6070706@intel.com> References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 2015-11-20 at 10:58 +0800, Jike Song wrote: > On 11/19/2015 11:52 PM, Alex Williamson wrote: > > On Thu, 2015-11-19 at 15:32 +0000, Stefano Stabellini wrote: > >> On Thu, 19 Nov 2015, Jike Song wrote: > >>> Hi Alex, thanks for the discussion. > >>> > >>> In addition to Kevin's replies, I have a high-level question: can VFIO > >>> be used by QEMU for both KVM and Xen? > >> > >> No. VFIO cannot be used with Xen today. When running on Xen, the IOMMU > >> is owned by Xen. > > > > Right, but in this case we're talking about device MMUs, which are owned > > by the device driver which I think is running in dom0, right? This > > proposal doesn't require support of the system IOMMU, the dom0 driver > > maps IOVA translations just as it would for itself. We're largely > > proposing use of the VFIO API to provide a common interface to expose a > > PCI(e) device to QEMU, but what happens in the vGPU vendor device and > > IOMMU backends is specific to the device and perhaps even specific to > > the hypervisor. Thanks, > > Let me conclude this, and please correct me in case of any misread: the > vGPU interface between kernel and QEMU will be through VFIO, with a new > VFIO backend (instead of the existing type1), for both KVMGT and XenGT? My primary concern is KVM and QEMU upstream, the proposal is not specifically directed at XenGT, but does not exclude it either. Xen is welcome to adopt this proposal as well, it simply defines the channel through which vGPUs are exposed to QEMU as the VFIO API. The core VFIO code in the Linux kernel is just as available for use in Xen dom0 as it is for a KVM host. VFIO in QEMU certainly knows about some accelerations for KVM, but these are almost entirely around allowing eventfd based interrupts to be injected through KVM, which is something I'm sure Xen could provide as well. These accelerations are also not required, VFIO based device assignment in QEMU works with or without KVM. Likewise, the VFIO kernel interface knows nothing about KVM and has no dependencies on it. There are two components to the VFIO API, one is the type1 compliant IOMMU interface, which for this proposal is really doing nothing more than tracking the HVA to GPA mappings for the VM. This much seems entirely common regardless of the hypervisor. The other part is the device interface. The lifecycle of the virtual device seems like it would be entirely shared, as does much of the emulation components of the device. When we get to pinning pages, providing direct access to memory ranges for a VM, and accelerating interrupts, the vGPU drivers will likely need some per hypervisor branches, but these are areas where that's true no matter what the interface. I'm probably over simplifying, but hopefully not too much, correct me if I'm wrong. The benefit of course is that aside from some extensions to the API, the QEMU components are already in place and there's a lot more leverage for getting both QEMU and libvirt support upstream in being able to support multiple vendors, perhaps multiple hypervisors, with the same code. Also, I'm not sure how useful it is, but VFIO is a userspace driver interface, where here we're predominantly talking about that userspace driver being QEMU. It's not limited to that though. A userspace compute application could have direct access to a vGPU through this model. Thanks, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:47606) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZzdDw-0006n8-KH for qemu-devel@nongnu.org; Thu, 19 Nov 2015 23:23:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZzdDr-00063y-OH for qemu-devel@nongnu.org; Thu, 19 Nov 2015 23:23:00 -0500 Received: from mx1.redhat.com ([209.132.183.28]:52351) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZzdDr-00063u-Gk for qemu-devel@nongnu.org; Thu, 19 Nov 2015 23:22:55 -0500 Message-ID: <1447993371.4697.257.camel@redhat.com> From: Alex Williamson Date: Thu, 19 Nov 2015 21:22:51 -0700 In-Reply-To: <564E8C51.6070706@intel.com> References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jike Song Cc: "igvt-g@ml01.01.org" , "Tian, Kevin" , "Reddy, Raghuveer" , qemu-devel , Stefano Stabellini , "White, Michael L" , "Cowperthwaite, David J" , "intel-gfx@lists.freedesktop.org" , "Li, Susie" , "Dong, Eddie" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xen.org" , Gerd Hoffmann , "Zhou, Chao" , Paolo Bonzini , "Zhu, Libo" , "Wang, Hongbo" , "Lv, Zhiyuan" On Fri, 2015-11-20 at 10:58 +0800, Jike Song wrote: > On 11/19/2015 11:52 PM, Alex Williamson wrote: > > On Thu, 2015-11-19 at 15:32 +0000, Stefano Stabellini wrote: > >> On Thu, 19 Nov 2015, Jike Song wrote: > >>> Hi Alex, thanks for the discussion. > >>> > >>> In addition to Kevin's replies, I have a high-level question: can VFIO > >>> be used by QEMU for both KVM and Xen? > >> > >> No. VFIO cannot be used with Xen today. When running on Xen, the IOMMU > >> is owned by Xen. > > > > Right, but in this case we're talking about device MMUs, which are owned > > by the device driver which I think is running in dom0, right? This > > proposal doesn't require support of the system IOMMU, the dom0 driver > > maps IOVA translations just as it would for itself. We're largely > > proposing use of the VFIO API to provide a common interface to expose a > > PCI(e) device to QEMU, but what happens in the vGPU vendor device and > > IOMMU backends is specific to the device and perhaps even specific to > > the hypervisor. Thanks, > > Let me conclude this, and please correct me in case of any misread: the > vGPU interface between kernel and QEMU will be through VFIO, with a new > VFIO backend (instead of the existing type1), for both KVMGT and XenGT? My primary concern is KVM and QEMU upstream, the proposal is not specifically directed at XenGT, but does not exclude it either. Xen is welcome to adopt this proposal as well, it simply defines the channel through which vGPUs are exposed to QEMU as the VFIO API. The core VFIO code in the Linux kernel is just as available for use in Xen dom0 as it is for a KVM host. VFIO in QEMU certainly knows about some accelerations for KVM, but these are almost entirely around allowing eventfd based interrupts to be injected through KVM, which is something I'm sure Xen could provide as well. These accelerations are also not required, VFIO based device assignment in QEMU works with or without KVM. Likewise, the VFIO kernel interface knows nothing about KVM and has no dependencies on it. There are two components to the VFIO API, one is the type1 compliant IOMMU interface, which for this proposal is really doing nothing more than tracking the HVA to GPA mappings for the VM. This much seems entirely common regardless of the hypervisor. The other part is the device interface. The lifecycle of the virtual device seems like it would be entirely shared, as does much of the emulation components of the device. When we get to pinning pages, providing direct access to memory ranges for a VM, and accelerating interrupts, the vGPU drivers will likely need some per hypervisor branches, but these are areas where that's true no matter what the interface. I'm probably over simplifying, but hopefully not too much, correct me if I'm wrong. The benefit of course is that aside from some extensions to the API, the QEMU components are already in place and there's a lot more leverage for getting both QEMU and libvirt support upstream in being able to support multiple vendors, perhaps multiple hypervisors, with the same code. Also, I'm not sure how useful it is, but VFIO is a userspace driver interface, where here we're predominantly talking about that userspace driver being QEMU. It's not limited to that though. A userspace compute application could have direct access to a vGPU through this model. Thanks, Alex