From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jike Song Subject: Re: [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel Date: Fri, 20 Nov 2015 13:51:46 +0800 Message-ID: <564EB4F2.9080605@intel.com> References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> <1447993371.4697.257.camel@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by gabe.freedesktop.org (Postfix) with ESMTP id 732166E5CF for ; Thu, 19 Nov 2015 21:52:20 -0800 (PST) In-Reply-To: <1447993371.4697.257.camel@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: Alex Williamson Cc: "igvt-g@ml01.01.org" , "Reddy, Raghuveer" , Stefano Stabellini , "White, Michael L" , "Cowperthwaite, David J" , "intel-gfx@lists.freedesktop.org" , "Li, Susie" , "Dong, Eddie" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xen.org" , qemu-devel , "Zhou, Chao" , Paolo Bonzini , "Zhu, Libo" , "Wang, Hongbo" List-Id: intel-gfx@lists.freedesktop.org T24gMTEvMjAvMjAxNSAxMjoyMiBQTSwgQWxleCBXaWxsaWFtc29uIHdyb3RlOgo+IE9uIEZyaSwg MjAxNS0xMS0yMCBhdCAxMDo1OCArMDgwMCwgSmlrZSBTb25nIHdyb3RlOgo+PiBPbiAxMS8xOS8y MDE1IDExOjUyIFBNLCBBbGV4IFdpbGxpYW1zb24gd3JvdGU6Cj4+PiBPbiBUaHUsIDIwMTUtMTEt MTkgYXQgMTU6MzIgKzAwMDAsIFN0ZWZhbm8gU3RhYmVsbGluaSB3cm90ZToKPj4+PiBPbiBUaHUs IDE5IE5vdiAyMDE1LCBKaWtlIFNvbmcgd3JvdGU6Cj4+Pj4+IEhpIEFsZXgsIHRoYW5rcyBmb3Ig dGhlIGRpc2N1c3Npb24uCj4+Pj4+Cj4+Pj4+IEluIGFkZGl0aW9uIHRvIEtldmluJ3MgcmVwbGll cywgSSBoYXZlIGEgaGlnaC1sZXZlbCBxdWVzdGlvbjogY2FuIFZGSU8KPj4+Pj4gYmUgdXNlZCBi eSBRRU1VIGZvciBib3RoIEtWTSBhbmQgWGVuPwo+Pj4+Cj4+Pj4gTm8uIFZGSU8gY2Fubm90IGJl IHVzZWQgd2l0aCBYZW4gdG9kYXkuIFdoZW4gcnVubmluZyBvbiBYZW4sIHRoZSBJT01NVQo+Pj4+ IGlzIG93bmVkIGJ5IFhlbi4KPj4+Cj4+PiBSaWdodCwgYnV0IGluIHRoaXMgY2FzZSB3ZSdyZSB0 YWxraW5nIGFib3V0IGRldmljZSBNTVVzLCB3aGljaCBhcmUgb3duZWQKPj4+IGJ5IHRoZSBkZXZp Y2UgZHJpdmVyIHdoaWNoIEkgdGhpbmsgaXMgcnVubmluZyBpbiBkb20wLCByaWdodD8gIFRoaXMK Pj4+IHByb3Bvc2FsIGRvZXNuJ3QgcmVxdWlyZSBzdXBwb3J0IG9mIHRoZSBzeXN0ZW0gSU9NTVUs IHRoZSBkb20wIGRyaXZlcgo+Pj4gbWFwcyBJT1ZBIHRyYW5zbGF0aW9ucyBqdXN0IGFzIGl0IHdv dWxkIGZvciBpdHNlbGYuICBXZSdyZSBsYXJnZWx5Cj4+PiBwcm9wb3NpbmcgdXNlIG9mIHRoZSBW RklPIEFQSSB0byBwcm92aWRlIGEgY29tbW9uIGludGVyZmFjZSB0byBleHBvc2UgYQo+Pj4gUENJ KGUpIGRldmljZSB0byBRRU1VLCBidXQgd2hhdCBoYXBwZW5zIGluIHRoZSB2R1BVIHZlbmRvciBk ZXZpY2UgYW5kCj4+PiBJT01NVSBiYWNrZW5kcyBpcyBzcGVjaWZpYyB0byB0aGUgZGV2aWNlIGFu ZCBwZXJoYXBzIGV2ZW4gc3BlY2lmaWMgdG8KPj4+IHRoZSBoeXBlcnZpc29yLiAgVGhhbmtzLAo+ Pgo+PiBMZXQgbWUgY29uY2x1ZGUgdGhpcywgYW5kIHBsZWFzZSBjb3JyZWN0IG1lIGluIGNhc2Ug b2YgYW55IG1pc3JlYWQ6IHRoZQo+PiB2R1BVIGludGVyZmFjZSBiZXR3ZWVuIGtlcm5lbCBhbmQg UUVNVSB3aWxsIGJlIHRocm91Z2ggVkZJTywgd2l0aCBhIG5ldwo+PiBWRklPIGJhY2tlbmQgKGlu c3RlYWQgb2YgdGhlIGV4aXN0aW5nIHR5cGUxKSwgZm9yIGJvdGggS1ZNR1QgYW5kIFhlbkdUPwo+ Cj4gTXkgcHJpbWFyeSBjb25jZXJuIGlzIEtWTSBhbmQgUUVNVSB1cHN0cmVhbSwgdGhlIHByb3Bv c2FsIGlzIG5vdAo+IHNwZWNpZmljYWxseSBkaXJlY3RlZCBhdCBYZW5HVCwgYnV0IGRvZXMgbm90 IGV4Y2x1ZGUgaXQgZWl0aGVyLiAgWGVuIGlzCj4gd2VsY29tZSB0byBhZG9wdCB0aGlzIHByb3Bv c2FsIGFzIHdlbGwsIGl0IHNpbXBseSBkZWZpbmVzIHRoZSBjaGFubmVsCj4gdGhyb3VnaCB3aGlj aCB2R1BVcyBhcmUgZXhwb3NlZCB0byBRRU1VIGFzIHRoZSBWRklPIEFQSS4gIFRoZSBjb3JlIFZG SU8KPiBjb2RlIGluIHRoZSBMaW51eCBrZXJuZWwgaXMganVzdCBhcyBhdmFpbGFibGUgZm9yIHVz ZSBpbiBYZW4gZG9tMCBhcyBpdAo+IGlzIGZvciBhIEtWTSBob3N0LiBWRklPIGluIFFFTVUgY2Vy dGFpbmx5IGtub3dzIGFib3V0IHNvbWUKPiBhY2NlbGVyYXRpb25zIGZvciBLVk0sIGJ1dCB0aGVz ZSBhcmUgYWxtb3N0IGVudGlyZWx5IGFyb3VuZCBhbGxvd2luZwo+IGV2ZW50ZmQgYmFzZWQgaW50 ZXJydXB0cyB0byBiZSBpbmplY3RlZCB0aHJvdWdoIEtWTSwgd2hpY2ggaXMgc29tZXRoaW5nCj4g SSdtIHN1cmUgWGVuIGNvdWxkIHByb3ZpZGUgYXMgd2VsbC4gIFRoZXNlIGFjY2VsZXJhdGlvbnMg YXJlIGFsc28gbm90Cj4gcmVxdWlyZWQsIFZGSU8gYmFzZWQgZGV2aWNlIGFzc2lnbm1lbnQgaW4g UUVNVSB3b3JrcyB3aXRoIG9yIHdpdGhvdXQKPiBLVk0uICBMaWtld2lzZSwgdGhlIFZGSU8ga2Vy bmVsIGludGVyZmFjZSBrbm93cyBub3RoaW5nIGFib3V0IEtWTSBhbmQKPiBoYXMgbm8gZGVwZW5k ZW5jaWVzIG9uIGl0Lgo+Cj4gVGhlcmUgYXJlIHR3byBjb21wb25lbnRzIHRvIHRoZSBWRklPIEFQ SSwgb25lIGlzIHRoZSB0eXBlMSBjb21wbGlhbnQKPiBJT01NVSBpbnRlcmZhY2UsIHdoaWNoIGZv ciB0aGlzIHByb3Bvc2FsIGlzIHJlYWxseSBkb2luZyBub3RoaW5nIG1vcmUKPiB0aGFuIHRyYWNr aW5nIHRoZSBIVkEgdG8gR1BBIG1hcHBpbmdzIGZvciB0aGUgVk0uICBUaGlzIG11Y2ggc2VlbXMK PiBlbnRpcmVseSBjb21tb24gcmVnYXJkbGVzcyBvZiB0aGUgaHlwZXJ2aXNvci4gIFRoZSBvdGhl ciBwYXJ0IGlzIHRoZQo+IGRldmljZSBpbnRlcmZhY2UuICBUaGUgbGlmZWN5Y2xlIG9mIHRoZSB2 aXJ0dWFsIGRldmljZSBzZWVtcyBsaWtlIGl0Cj4gd291bGQgYmUgZW50aXJlbHkgc2hhcmVkLCBh cyBkb2VzIG11Y2ggb2YgdGhlIGVtdWxhdGlvbiBjb21wb25lbnRzIG9mCj4gdGhlIGRldmljZS4g IFdoZW4gd2UgZ2V0IHRvIHBpbm5pbmcgcGFnZXMsIHByb3ZpZGluZyBkaXJlY3QgYWNjZXNzIHRv Cj4gbWVtb3J5IHJhbmdlcyBmb3IgYSBWTSwgYW5kIGFjY2VsZXJhdGluZyBpbnRlcnJ1cHRzLCB0 aGUgdkdQVSBkcml2ZXJzCj4gd2lsbCBsaWtlbHkgbmVlZCBzb21lIHBlciBoeXBlcnZpc29yIGJy YW5jaGVzLCBidXQgdGhlc2UgYXJlIGFyZWFzIHdoZXJlCj4gdGhhdCdzIHRydWUgbm8gbWF0dGVy IHdoYXQgdGhlIGludGVyZmFjZS4gIEknbSBwcm9iYWJseSBvdmVyCj4gc2ltcGxpZnlpbmcsIGJ1 dCBob3BlZnVsbHkgbm90IHRvbyBtdWNoLCBjb3JyZWN0IG1lIGlmIEknbSB3cm9uZy4KPgoKVGhh bmtzIGZvciBjb25maXJtYXRpb24uIEZvciBRRU1VL0tWTSwgSSB0b3RhbGx5IGFncmVlIHlvdXIg cG9pbnQ7IEhvd2V2ZXIsCmlmIHdlIHRha2UgWGVuR1QgdG8gY29uc2lkZXIsIGl0IHdpbGwgYmUg YSBiaXQgbW9yZSBjb21wbGV4OiB3aXRoIFhlbgpoeXBlcnZpc29yIGFuZCBEb20wIGtlcm5lbCBy dW5uaW5nIGluIGRpZmZlcmVudCBsZXZlbCwgaXQncyBub3QgYSBzdHJhaWdodC0KZm9yd2FyZCB3 YXkgZm9yIFFFTVUgdG8gZG8gc29tZXRoaW5nIGxpa2UgbWFwcGluZyBhIHBvcnRpb24gb2YgTU1J TyBCQVIKdmlhIFZGSU8gaW4gRG9tMCBrZXJuZWwsIGluc3RlYWQgb2YgY2FsbGluZyBoeXBlcmNh bGxzIGRpcmVjdGx5LgoKSSBkb24ndCBrbm93IGlmIHRoZXJlIGlzIGEgYmV0dGVyIHdheSB0byBo YW5kbGUgdGhpcy4gQnV0IEkgZG8gYWdyZWUgdGhhdApjaGFubmVscyBiZXR3ZWVuIGtlcm5lbCBh bmQgUWVtdSB2aWEgVkZJTyBpcyBhIGdvb2QgaWRlYSwgZXZlbiB0aG91Z2ggd2UKbWF5IGhhdmUg dG8gc3BsaXQgS1ZNR1QvWGVuR1QgaW4gUWVtdSBhIGJpdC4gIFdlIGFyZSBjdXJyZW50bHkgd29y a2luZyBvbgptb3ZpbmcgYWxsIG9mIFBDSSBDRkcgZW11bGF0aW9uIGZyb20ga2VybmVsIHRvIFFl bXUsIGhvcGVmdWxseSB3ZSBjYW4KcmVsZWFzZSBpdCBieSBlbmQgb2YgdGhpcyB5ZWFyIGFuZCB3 b3JrIHdpdGggeW91IGd1eXMgdG8gYWRqdXN0IGl0IGZvcgp0aGUgYWdyZWVkIG1ldGhvZC4KCgo+ IFRoZSBiZW5lZml0IG9mIGNvdXJzZSBpcyB0aGF0IGFzaWRlIGZyb20gc29tZSBleHRlbnNpb25z IHRvIHRoZSBBUEksIHRoZQo+IFFFTVUgY29tcG9uZW50cyBhcmUgYWxyZWFkeSBpbiBwbGFjZSBh bmQgdGhlcmUncyBhIGxvdCBtb3JlIGxldmVyYWdlIGZvcgo+IGdldHRpbmcgYm90aCBRRU1VIGFu ZCBsaWJ2aXJ0IHN1cHBvcnQgdXBzdHJlYW0gaW4gYmVpbmcgYWJsZSB0byBzdXBwb3J0Cj4gbXVs dGlwbGUgdmVuZG9ycywgcGVyaGFwcyBtdWx0aXBsZSBoeXBlcnZpc29ycywgd2l0aCB0aGUgc2Ft ZSBjb2RlLgo+IEFsc28sIEknbSBub3Qgc3VyZSBob3cgdXNlZnVsIGl0IGlzLCBidXQgVkZJTyBp cyBhIHVzZXJzcGFjZSBkcml2ZXIKPiBpbnRlcmZhY2UsIHdoZXJlIGhlcmUgd2UncmUgcHJlZG9t aW5hbnRseSB0YWxraW5nIGFib3V0IHRoYXQgdXNlcnNwYWNlCj4gZHJpdmVyIGJlaW5nIFFFTVUu ICBJdCdzIG5vdCBsaW1pdGVkIHRvIHRoYXQgdGhvdWdoLiAgQSB1c2Vyc3BhY2UKPiBjb21wdXRl IGFwcGxpY2F0aW9uIGNvdWxkIGhhdmUgZGlyZWN0IGFjY2VzcyB0byBhIHZHUFUgdGhyb3VnaCB0 aGlzCj4gbW9kZWwuICBUaGFua3MsCgoKPgo+IEFsZXgKPgotLQpUaGFua3MsCkppa2UKX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KSW50ZWwtZ2Z4IG1haWxp bmcgbGlzdApJbnRlbC1nZnhAbGlzdHMuZnJlZWRlc2t0b3Aub3JnCmh0dHA6Ly9saXN0cy5mcmVl ZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9pbnRlbC1nZngK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754328AbbKTFwT (ORCPT ); Fri, 20 Nov 2015 00:52:19 -0500 Received: from mga09.intel.com ([134.134.136.24]:53959 "EHLO mga09.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751923AbbKTFwR (ORCPT ); Fri, 20 Nov 2015 00:52:17 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,321,1444719600"; d="scan'208";a="843048358" Message-ID: <564EB4F2.9080605@intel.com> Date: Fri, 20 Nov 2015 13:51:46 +0800 From: Jike Song User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Alex Williamson CC: Stefano Stabellini , "Tian, Kevin" , "xen-devel@lists.xen.org" , "igvt-g@ml01.01.org" , "intel-gfx@lists.freedesktop.org" , "linux-kernel@vger.kernel.org" , "White, Michael L" , "Dong, Eddie" , "Li, Susie" , "Cowperthwaite, David J" , "Reddy, Raghuveer" , "Zhu, Libo" , "Zhou, Chao" , "Wang, Hongbo" , "Lv, Zhiyuan" , qemu-devel , Paolo Bonzini , Gerd Hoffmann Subject: Re: [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> <1447993371.4697.257.camel@redhat.com> In-Reply-To: <1447993371.4697.257.camel@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/20/2015 12:22 PM, Alex Williamson wrote: > On Fri, 2015-11-20 at 10:58 +0800, Jike Song wrote: >> On 11/19/2015 11:52 PM, Alex Williamson wrote: >>> On Thu, 2015-11-19 at 15:32 +0000, Stefano Stabellini wrote: >>>> On Thu, 19 Nov 2015, Jike Song wrote: >>>>> Hi Alex, thanks for the discussion. >>>>> >>>>> In addition to Kevin's replies, I have a high-level question: can VFIO >>>>> be used by QEMU for both KVM and Xen? >>>> >>>> No. VFIO cannot be used with Xen today. When running on Xen, the IOMMU >>>> is owned by Xen. >>> >>> Right, but in this case we're talking about device MMUs, which are owned >>> by the device driver which I think is running in dom0, right? This >>> proposal doesn't require support of the system IOMMU, the dom0 driver >>> maps IOVA translations just as it would for itself. We're largely >>> proposing use of the VFIO API to provide a common interface to expose a >>> PCI(e) device to QEMU, but what happens in the vGPU vendor device and >>> IOMMU backends is specific to the device and perhaps even specific to >>> the hypervisor. Thanks, >> >> Let me conclude this, and please correct me in case of any misread: the >> vGPU interface between kernel and QEMU will be through VFIO, with a new >> VFIO backend (instead of the existing type1), for both KVMGT and XenGT? > > My primary concern is KVM and QEMU upstream, the proposal is not > specifically directed at XenGT, but does not exclude it either. Xen is > welcome to adopt this proposal as well, it simply defines the channel > through which vGPUs are exposed to QEMU as the VFIO API. The core VFIO > code in the Linux kernel is just as available for use in Xen dom0 as it > is for a KVM host. VFIO in QEMU certainly knows about some > accelerations for KVM, but these are almost entirely around allowing > eventfd based interrupts to be injected through KVM, which is something > I'm sure Xen could provide as well. These accelerations are also not > required, VFIO based device assignment in QEMU works with or without > KVM. Likewise, the VFIO kernel interface knows nothing about KVM and > has no dependencies on it. > > There are two components to the VFIO API, one is the type1 compliant > IOMMU interface, which for this proposal is really doing nothing more > than tracking the HVA to GPA mappings for the VM. This much seems > entirely common regardless of the hypervisor. The other part is the > device interface. The lifecycle of the virtual device seems like it > would be entirely shared, as does much of the emulation components of > the device. When we get to pinning pages, providing direct access to > memory ranges for a VM, and accelerating interrupts, the vGPU drivers > will likely need some per hypervisor branches, but these are areas where > that's true no matter what the interface. I'm probably over > simplifying, but hopefully not too much, correct me if I'm wrong. > Thanks for confirmation. For QEMU/KVM, I totally agree your point; However, if we take XenGT to consider, it will be a bit more complex: with Xen hypervisor and Dom0 kernel running in different level, it's not a straight- forward way for QEMU to do something like mapping a portion of MMIO BAR via VFIO in Dom0 kernel, instead of calling hypercalls directly. I don't know if there is a better way to handle this. But I do agree that channels between kernel and Qemu via VFIO is a good idea, even though we may have to split KVMGT/XenGT in Qemu a bit. We are currently working on moving all of PCI CFG emulation from kernel to Qemu, hopefully we can release it by end of this year and work with you guys to adjust it for the agreed method. > The benefit of course is that aside from some extensions to the API, the > QEMU components are already in place and there's a lot more leverage for > getting both QEMU and libvirt support upstream in being able to support > multiple vendors, perhaps multiple hypervisors, with the same code. > Also, I'm not sure how useful it is, but VFIO is a userspace driver > interface, where here we're predominantly talking about that userspace > driver being QEMU. It's not limited to that though. A userspace > compute application could have direct access to a vGPU through this > model. Thanks, > > Alex > -- Thanks, Jike From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33170) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZzecS-00048l-0K for qemu-devel@nongnu.org; Fri, 20 Nov 2015 00:52:25 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZzecM-0001Kt-WB for qemu-devel@nongnu.org; Fri, 20 Nov 2015 00:52:23 -0500 Received: from mga02.intel.com ([134.134.136.20]:55818) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZzecM-0001Kp-Kn for qemu-devel@nongnu.org; Fri, 20 Nov 2015 00:52:18 -0500 Message-ID: <564EB4F2.9080605@intel.com> Date: Fri, 20 Nov 2015 13:51:46 +0800 From: Jike Song MIME-Version: 1.0 References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> <1447993371.4697.257.camel@redhat.com> In-Reply-To: <1447993371.4697.257.camel@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alex Williamson Cc: "igvt-g@ml01.01.org" , "Tian, Kevin" , "Reddy, Raghuveer" , qemu-devel , Stefano Stabellini , "White, Michael L" , "Cowperthwaite, David J" , "intel-gfx@lists.freedesktop.org" , "Li, Susie" , "Dong, Eddie" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xen.org" , Gerd Hoffmann , "Zhou, Chao" , Paolo Bonzini , "Zhu, Libo" , "Wang, Hongbo" , "Lv, Zhiyuan" On 11/20/2015 12:22 PM, Alex Williamson wrote: > On Fri, 2015-11-20 at 10:58 +0800, Jike Song wrote: >> On 11/19/2015 11:52 PM, Alex Williamson wrote: >>> On Thu, 2015-11-19 at 15:32 +0000, Stefano Stabellini wrote: >>>> On Thu, 19 Nov 2015, Jike Song wrote: >>>>> Hi Alex, thanks for the discussion. >>>>> >>>>> In addition to Kevin's replies, I have a high-level question: can VFIO >>>>> be used by QEMU for both KVM and Xen? >>>> >>>> No. VFIO cannot be used with Xen today. When running on Xen, the IOMMU >>>> is owned by Xen. >>> >>> Right, but in this case we're talking about device MMUs, which are owned >>> by the device driver which I think is running in dom0, right? This >>> proposal doesn't require support of the system IOMMU, the dom0 driver >>> maps IOVA translations just as it would for itself. We're largely >>> proposing use of the VFIO API to provide a common interface to expose a >>> PCI(e) device to QEMU, but what happens in the vGPU vendor device and >>> IOMMU backends is specific to the device and perhaps even specific to >>> the hypervisor. Thanks, >> >> Let me conclude this, and please correct me in case of any misread: the >> vGPU interface between kernel and QEMU will be through VFIO, with a new >> VFIO backend (instead of the existing type1), for both KVMGT and XenGT? > > My primary concern is KVM and QEMU upstream, the proposal is not > specifically directed at XenGT, but does not exclude it either. Xen is > welcome to adopt this proposal as well, it simply defines the channel > through which vGPUs are exposed to QEMU as the VFIO API. The core VFIO > code in the Linux kernel is just as available for use in Xen dom0 as it > is for a KVM host. VFIO in QEMU certainly knows about some > accelerations for KVM, but these are almost entirely around allowing > eventfd based interrupts to be injected through KVM, which is something > I'm sure Xen could provide as well. These accelerations are also not > required, VFIO based device assignment in QEMU works with or without > KVM. Likewise, the VFIO kernel interface knows nothing about KVM and > has no dependencies on it. > > There are two components to the VFIO API, one is the type1 compliant > IOMMU interface, which for this proposal is really doing nothing more > than tracking the HVA to GPA mappings for the VM. This much seems > entirely common regardless of the hypervisor. The other part is the > device interface. The lifecycle of the virtual device seems like it > would be entirely shared, as does much of the emulation components of > the device. When we get to pinning pages, providing direct access to > memory ranges for a VM, and accelerating interrupts, the vGPU drivers > will likely need some per hypervisor branches, but these are areas where > that's true no matter what the interface. I'm probably over > simplifying, but hopefully not too much, correct me if I'm wrong. > Thanks for confirmation. For QEMU/KVM, I totally agree your point; However, if we take XenGT to consider, it will be a bit more complex: with Xen hypervisor and Dom0 kernel running in different level, it's not a straight- forward way for QEMU to do something like mapping a portion of MMIO BAR via VFIO in Dom0 kernel, instead of calling hypercalls directly. I don't know if there is a better way to handle this. But I do agree that channels between kernel and Qemu via VFIO is a good idea, even though we may have to split KVMGT/XenGT in Qemu a bit. We are currently working on moving all of PCI CFG emulation from kernel to Qemu, hopefully we can release it by end of this year and work with you guys to adjust it for the agreed method. > The benefit of course is that aside from some extensions to the API, the > QEMU components are already in place and there's a lot more leverage for > getting both QEMU and libvirt support upstream in being able to support > multiple vendors, perhaps multiple hypervisors, with the same code. > Also, I'm not sure how useful it is, but VFIO is a userspace driver > interface, where here we're predominantly talking about that userspace > driver being QEMU. It's not limited to that though. A userspace > compute application could have direct access to a vGPU through this > model. Thanks, > > Alex > -- Thanks, Jike