From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel Date: Fri, 20 Nov 2015 09:40:40 -0700 Message-ID: <1448037640.4697.266.camel@redhat.com> References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> <1447993371.4697.257.camel@redhat.com> <564EB4F2.9080605@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by gabe.freedesktop.org (Postfix) with ESMTPS id 361EC6EB00 for ; Fri, 20 Nov 2015 08:40:43 -0800 (PST) In-Reply-To: <564EB4F2.9080605@intel.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: Jike Song Cc: "igvt-g@ml01.01.org" , "Reddy, Raghuveer" , Stefano Stabellini , "White, Michael L" , "Cowperthwaite, David J" , "intel-gfx@lists.freedesktop.org" , "Li, Susie" , "Dong, Eddie" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xen.org" , qemu-devel , "Zhou, Chao" , Paolo Bonzini , "Zhu, Libo" , "Wang, Hongbo" List-Id: intel-gfx@lists.freedesktop.org T24gRnJpLCAyMDE1LTExLTIwIGF0IDEzOjUxICswODAwLCBKaWtlIFNvbmcgd3JvdGU6Cj4gT24g MTEvMjAvMjAxNSAxMjoyMiBQTSwgQWxleCBXaWxsaWFtc29uIHdyb3RlOgo+ID4gT24gRnJpLCAy MDE1LTExLTIwIGF0IDEwOjU4ICswODAwLCBKaWtlIFNvbmcgd3JvdGU6Cj4gPj4gT24gMTEvMTkv MjAxNSAxMTo1MiBQTSwgQWxleCBXaWxsaWFtc29uIHdyb3RlOgo+ID4+PiBPbiBUaHUsIDIwMTUt MTEtMTkgYXQgMTU6MzIgKzAwMDAsIFN0ZWZhbm8gU3RhYmVsbGluaSB3cm90ZToKPiA+Pj4+IE9u IFRodSwgMTkgTm92IDIwMTUsIEppa2UgU29uZyB3cm90ZToKPiA+Pj4+PiBIaSBBbGV4LCB0aGFu a3MgZm9yIHRoZSBkaXNjdXNzaW9uLgo+ID4+Pj4+Cj4gPj4+Pj4gSW4gYWRkaXRpb24gdG8gS2V2 aW4ncyByZXBsaWVzLCBJIGhhdmUgYSBoaWdoLWxldmVsIHF1ZXN0aW9uOiBjYW4gVkZJTwo+ID4+ Pj4+IGJlIHVzZWQgYnkgUUVNVSBmb3IgYm90aCBLVk0gYW5kIFhlbj8KPiA+Pj4+Cj4gPj4+PiBO by4gVkZJTyBjYW5ub3QgYmUgdXNlZCB3aXRoIFhlbiB0b2RheS4gV2hlbiBydW5uaW5nIG9uIFhl biwgdGhlIElPTU1VCj4gPj4+PiBpcyBvd25lZCBieSBYZW4uCj4gPj4+Cj4gPj4+IFJpZ2h0LCBi dXQgaW4gdGhpcyBjYXNlIHdlJ3JlIHRhbGtpbmcgYWJvdXQgZGV2aWNlIE1NVXMsIHdoaWNoIGFy ZSBvd25lZAo+ID4+PiBieSB0aGUgZGV2aWNlIGRyaXZlciB3aGljaCBJIHRoaW5rIGlzIHJ1bm5p bmcgaW4gZG9tMCwgcmlnaHQ/ICBUaGlzCj4gPj4+IHByb3Bvc2FsIGRvZXNuJ3QgcmVxdWlyZSBz dXBwb3J0IG9mIHRoZSBzeXN0ZW0gSU9NTVUsIHRoZSBkb20wIGRyaXZlcgo+ID4+PiBtYXBzIElP VkEgdHJhbnNsYXRpb25zIGp1c3QgYXMgaXQgd291bGQgZm9yIGl0c2VsZi4gIFdlJ3JlIGxhcmdl bHkKPiA+Pj4gcHJvcG9zaW5nIHVzZSBvZiB0aGUgVkZJTyBBUEkgdG8gcHJvdmlkZSBhIGNvbW1v biBpbnRlcmZhY2UgdG8gZXhwb3NlIGEKPiA+Pj4gUENJKGUpIGRldmljZSB0byBRRU1VLCBidXQg d2hhdCBoYXBwZW5zIGluIHRoZSB2R1BVIHZlbmRvciBkZXZpY2UgYW5kCj4gPj4+IElPTU1VIGJh Y2tlbmRzIGlzIHNwZWNpZmljIHRvIHRoZSBkZXZpY2UgYW5kIHBlcmhhcHMgZXZlbiBzcGVjaWZp YyB0bwo+ID4+PiB0aGUgaHlwZXJ2aXNvci4gIFRoYW5rcywKPiA+Pgo+ID4+IExldCBtZSBjb25j bHVkZSB0aGlzLCBhbmQgcGxlYXNlIGNvcnJlY3QgbWUgaW4gY2FzZSBvZiBhbnkgbWlzcmVhZDog dGhlCj4gPj4gdkdQVSBpbnRlcmZhY2UgYmV0d2VlbiBrZXJuZWwgYW5kIFFFTVUgd2lsbCBiZSB0 aHJvdWdoIFZGSU8sIHdpdGggYSBuZXcKPiA+PiBWRklPIGJhY2tlbmQgKGluc3RlYWQgb2YgdGhl IGV4aXN0aW5nIHR5cGUxKSwgZm9yIGJvdGggS1ZNR1QgYW5kIFhlbkdUPwo+ID4KPiA+IE15IHBy aW1hcnkgY29uY2VybiBpcyBLVk0gYW5kIFFFTVUgdXBzdHJlYW0sIHRoZSBwcm9wb3NhbCBpcyBu b3QKPiA+IHNwZWNpZmljYWxseSBkaXJlY3RlZCBhdCBYZW5HVCwgYnV0IGRvZXMgbm90IGV4Y2x1 ZGUgaXQgZWl0aGVyLiAgWGVuIGlzCj4gPiB3ZWxjb21lIHRvIGFkb3B0IHRoaXMgcHJvcG9zYWwg YXMgd2VsbCwgaXQgc2ltcGx5IGRlZmluZXMgdGhlIGNoYW5uZWwKPiA+IHRocm91Z2ggd2hpY2gg dkdQVXMgYXJlIGV4cG9zZWQgdG8gUUVNVSBhcyB0aGUgVkZJTyBBUEkuICBUaGUgY29yZSBWRklP Cj4gPiBjb2RlIGluIHRoZSBMaW51eCBrZXJuZWwgaXMganVzdCBhcyBhdmFpbGFibGUgZm9yIHVz ZSBpbiBYZW4gZG9tMCBhcyBpdAo+ID4gaXMgZm9yIGEgS1ZNIGhvc3QuIFZGSU8gaW4gUUVNVSBj ZXJ0YWlubHkga25vd3MgYWJvdXQgc29tZQo+ID4gYWNjZWxlcmF0aW9ucyBmb3IgS1ZNLCBidXQg dGhlc2UgYXJlIGFsbW9zdCBlbnRpcmVseSBhcm91bmQgYWxsb3dpbmcKPiA+IGV2ZW50ZmQgYmFz ZWQgaW50ZXJydXB0cyB0byBiZSBpbmplY3RlZCB0aHJvdWdoIEtWTSwgd2hpY2ggaXMgc29tZXRo aW5nCj4gPiBJJ20gc3VyZSBYZW4gY291bGQgcHJvdmlkZSBhcyB3ZWxsLiAgVGhlc2UgYWNjZWxl cmF0aW9ucyBhcmUgYWxzbyBub3QKPiA+IHJlcXVpcmVkLCBWRklPIGJhc2VkIGRldmljZSBhc3Np Z25tZW50IGluIFFFTVUgd29ya3Mgd2l0aCBvciB3aXRob3V0Cj4gPiBLVk0uICBMaWtld2lzZSwg dGhlIFZGSU8ga2VybmVsIGludGVyZmFjZSBrbm93cyBub3RoaW5nIGFib3V0IEtWTSBhbmQKPiA+ IGhhcyBubyBkZXBlbmRlbmNpZXMgb24gaXQuCj4gPgo+ID4gVGhlcmUgYXJlIHR3byBjb21wb25l bnRzIHRvIHRoZSBWRklPIEFQSSwgb25lIGlzIHRoZSB0eXBlMSBjb21wbGlhbnQKPiA+IElPTU1V IGludGVyZmFjZSwgd2hpY2ggZm9yIHRoaXMgcHJvcG9zYWwgaXMgcmVhbGx5IGRvaW5nIG5vdGhp bmcgbW9yZQo+ID4gdGhhbiB0cmFja2luZyB0aGUgSFZBIHRvIEdQQSBtYXBwaW5ncyBmb3IgdGhl IFZNLiAgVGhpcyBtdWNoIHNlZW1zCj4gPiBlbnRpcmVseSBjb21tb24gcmVnYXJkbGVzcyBvZiB0 aGUgaHlwZXJ2aXNvci4gIFRoZSBvdGhlciBwYXJ0IGlzIHRoZQo+ID4gZGV2aWNlIGludGVyZmFj ZS4gIFRoZSBsaWZlY3ljbGUgb2YgdGhlIHZpcnR1YWwgZGV2aWNlIHNlZW1zIGxpa2UgaXQKPiA+ IHdvdWxkIGJlIGVudGlyZWx5IHNoYXJlZCwgYXMgZG9lcyBtdWNoIG9mIHRoZSBlbXVsYXRpb24g Y29tcG9uZW50cyBvZgo+ID4gdGhlIGRldmljZS4gIFdoZW4gd2UgZ2V0IHRvIHBpbm5pbmcgcGFn ZXMsIHByb3ZpZGluZyBkaXJlY3QgYWNjZXNzIHRvCj4gPiBtZW1vcnkgcmFuZ2VzIGZvciBhIFZN LCBhbmQgYWNjZWxlcmF0aW5nIGludGVycnVwdHMsIHRoZSB2R1BVIGRyaXZlcnMKPiA+IHdpbGwg bGlrZWx5IG5lZWQgc29tZSBwZXIgaHlwZXJ2aXNvciBicmFuY2hlcywgYnV0IHRoZXNlIGFyZSBh cmVhcyB3aGVyZQo+ID4gdGhhdCdzIHRydWUgbm8gbWF0dGVyIHdoYXQgdGhlIGludGVyZmFjZS4g IEknbSBwcm9iYWJseSBvdmVyCj4gPiBzaW1wbGlmeWluZywgYnV0IGhvcGVmdWxseSBub3QgdG9v IG11Y2gsIGNvcnJlY3QgbWUgaWYgSSdtIHdyb25nLgo+ID4KPiAKPiBUaGFua3MgZm9yIGNvbmZp cm1hdGlvbi4gRm9yIFFFTVUvS1ZNLCBJIHRvdGFsbHkgYWdyZWUgeW91ciBwb2ludDsgSG93ZXZl ciwKPiBpZiB3ZSB0YWtlIFhlbkdUIHRvIGNvbnNpZGVyLCBpdCB3aWxsIGJlIGEgYml0IG1vcmUg Y29tcGxleDogd2l0aCBYZW4KPiBoeXBlcnZpc29yIGFuZCBEb20wIGtlcm5lbCBydW5uaW5nIGlu IGRpZmZlcmVudCBsZXZlbCwgaXQncyBub3QgYSBzdHJhaWdodC0KPiBmb3J3YXJkIHdheSBmb3Ig UUVNVSB0byBkbyBzb21ldGhpbmcgbGlrZSBtYXBwaW5nIGEgcG9ydGlvbiBvZiBNTUlPIEJBUgo+ IHZpYSBWRklPIGluIERvbTAga2VybmVsLCBpbnN0ZWFkIG9mIGNhbGxpbmcgaHlwZXJjYWxscyBk aXJlY3RseS4KClRoaXMgd291bGQgbmVlZCB0byBiZSBwYXJ0IG9mIHRoZSBzdXBwb3J0IGFkZGVk IGZvciBYZW4uICBUbyBkaXJlY3RseQptYXAgYSBkZXZpY2UgTU1JTyBzcGFjZSB0byB0aGUgVk0s IFZGSU8gcHJvdmlkZXMgYW4gbW1hcCwgUUVNVSByZWdpc3RlcnMKdGhhdCBtbWFwIHdpdGggS1ZN LCBvciBYZW4uICBJdCdzIGFsbCBqdXN0IE1lbW9yeVJlZ2lvbnMgaW4gUUVNVS4KUGVyaGFwcyBp dCdzIGV2ZW4gYWxyZWFkeSBzdXBwb3J0ZWQgYnkgWGVuLgoKPiBJIGRvbid0IGtub3cgaWYgdGhl cmUgaXMgYSBiZXR0ZXIgd2F5IHRvIGhhbmRsZSB0aGlzLiBCdXQgSSBkbyBhZ3JlZSB0aGF0Cj4g Y2hhbm5lbHMgYmV0d2VlbiBrZXJuZWwgYW5kIFFlbXUgdmlhIFZGSU8gaXMgYSBnb29kIGlkZWEs IGV2ZW4gdGhvdWdoIHdlCj4gbWF5IGhhdmUgdG8gc3BsaXQgS1ZNR1QvWGVuR1QgaW4gUWVtdSBh IGJpdC4gIFdlIGFyZSBjdXJyZW50bHkgd29ya2luZyBvbgo+IG1vdmluZyBhbGwgb2YgUENJIENG RyBlbXVsYXRpb24gZnJvbSBrZXJuZWwgdG8gUWVtdSwgaG9wZWZ1bGx5IHdlIGNhbgo+IHJlbGVh c2UgaXQgYnkgZW5kIG9mIHRoaXMgeWVhciBhbmQgd29yayB3aXRoIHlvdSBndXlzIHRvIGFkanVz dCBpdCBmb3IKPiB0aGUgYWdyZWVkIG1ldGhvZC4KCldlbGwsIG1vdmluZyBQQ0kgY29uZmlnIHNw YWNlIGVtdWxhdGlvbiBmcm9tIGtlcm5lbCB0byBRRU1VIGlzIGV4YWN0bHkKdGhlIHdyb25nIGRp cmVjdGlvbiB0byB0YWtlIGZvciB0aGlzIHByb3Bvc2FsLiAgQ29uZmlnIHNwYWNlIGFjY2VzcyB0 bwp0aGUgdkdQVSB3b3VsZCBvY2N1ciB0aHJvdWdoIHRoZSBWRklPIEFQSS4gIFNvIGlmIHlvdSBh bHJlYWR5IGhhdmUKY29uZmlnIHNwYWNlIGVtdWxhdGlvbiBpbiB0aGUga2VybmVsLCB0aGF0J3Mg YWxyZWFkeSBvbmUgbGVzcyBwaWVjZSBvZgp3b3JrIGZvciBhIFZGSU8gbW9kZWwsIGl0IGp1c3Qg bmVlZHMgdG8gYmUgIndpcmVkIHVwIiB0aHJvdWdoIHRoZSBWRklPCkFQSS4gIFRoYW5rcywKCkFs ZXgKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCkludGVs LWdmeCBtYWlsaW5nIGxpc3QKSW50ZWwtZ2Z4QGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRwOi8v bGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vaW50ZWwtZ2Z4Cg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1163089AbbKTQkp (ORCPT ); Fri, 20 Nov 2015 11:40:45 -0500 Received: from mx1.redhat.com ([209.132.183.28]:34094 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1163015AbbKTQkn (ORCPT ); Fri, 20 Nov 2015 11:40:43 -0500 Message-ID: <1448037640.4697.266.camel@redhat.com> Subject: Re: [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel From: Alex Williamson To: Jike Song Cc: Stefano Stabellini , "Tian, Kevin" , "xen-devel@lists.xen.org" , "igvt-g@ml01.01.org" , "intel-gfx@lists.freedesktop.org" , "linux-kernel@vger.kernel.org" , "White, Michael L" , "Dong, Eddie" , "Li, Susie" , "Cowperthwaite, David J" , "Reddy, Raghuveer" , "Zhu, Libo" , "Zhou, Chao" , "Wang, Hongbo" , "Lv, Zhiyuan" , qemu-devel , Paolo Bonzini , Gerd Hoffmann Date: Fri, 20 Nov 2015 09:40:40 -0700 In-Reply-To: <564EB4F2.9080605@intel.com> References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> <1447993371.4697.257.camel@redhat.com> <564EB4F2.9080605@intel.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 2015-11-20 at 13:51 +0800, Jike Song wrote: > On 11/20/2015 12:22 PM, Alex Williamson wrote: > > On Fri, 2015-11-20 at 10:58 +0800, Jike Song wrote: > >> On 11/19/2015 11:52 PM, Alex Williamson wrote: > >>> On Thu, 2015-11-19 at 15:32 +0000, Stefano Stabellini wrote: > >>>> On Thu, 19 Nov 2015, Jike Song wrote: > >>>>> Hi Alex, thanks for the discussion. > >>>>> > >>>>> In addition to Kevin's replies, I have a high-level question: can VFIO > >>>>> be used by QEMU for both KVM and Xen? > >>>> > >>>> No. VFIO cannot be used with Xen today. When running on Xen, the IOMMU > >>>> is owned by Xen. > >>> > >>> Right, but in this case we're talking about device MMUs, which are owned > >>> by the device driver which I think is running in dom0, right? This > >>> proposal doesn't require support of the system IOMMU, the dom0 driver > >>> maps IOVA translations just as it would for itself. We're largely > >>> proposing use of the VFIO API to provide a common interface to expose a > >>> PCI(e) device to QEMU, but what happens in the vGPU vendor device and > >>> IOMMU backends is specific to the device and perhaps even specific to > >>> the hypervisor. Thanks, > >> > >> Let me conclude this, and please correct me in case of any misread: the > >> vGPU interface between kernel and QEMU will be through VFIO, with a new > >> VFIO backend (instead of the existing type1), for both KVMGT and XenGT? > > > > My primary concern is KVM and QEMU upstream, the proposal is not > > specifically directed at XenGT, but does not exclude it either. Xen is > > welcome to adopt this proposal as well, it simply defines the channel > > through which vGPUs are exposed to QEMU as the VFIO API. The core VFIO > > code in the Linux kernel is just as available for use in Xen dom0 as it > > is for a KVM host. VFIO in QEMU certainly knows about some > > accelerations for KVM, but these are almost entirely around allowing > > eventfd based interrupts to be injected through KVM, which is something > > I'm sure Xen could provide as well. These accelerations are also not > > required, VFIO based device assignment in QEMU works with or without > > KVM. Likewise, the VFIO kernel interface knows nothing about KVM and > > has no dependencies on it. > > > > There are two components to the VFIO API, one is the type1 compliant > > IOMMU interface, which for this proposal is really doing nothing more > > than tracking the HVA to GPA mappings for the VM. This much seems > > entirely common regardless of the hypervisor. The other part is the > > device interface. The lifecycle of the virtual device seems like it > > would be entirely shared, as does much of the emulation components of > > the device. When we get to pinning pages, providing direct access to > > memory ranges for a VM, and accelerating interrupts, the vGPU drivers > > will likely need some per hypervisor branches, but these are areas where > > that's true no matter what the interface. I'm probably over > > simplifying, but hopefully not too much, correct me if I'm wrong. > > > > Thanks for confirmation. For QEMU/KVM, I totally agree your point; However, > if we take XenGT to consider, it will be a bit more complex: with Xen > hypervisor and Dom0 kernel running in different level, it's not a straight- > forward way for QEMU to do something like mapping a portion of MMIO BAR > via VFIO in Dom0 kernel, instead of calling hypercalls directly. This would need to be part of the support added for Xen. To directly map a device MMIO space to the VM, VFIO provides an mmap, QEMU registers that mmap with KVM, or Xen. It's all just MemoryRegions in QEMU. Perhaps it's even already supported by Xen. > I don't know if there is a better way to handle this. But I do agree that > channels between kernel and Qemu via VFIO is a good idea, even though we > may have to split KVMGT/XenGT in Qemu a bit. We are currently working on > moving all of PCI CFG emulation from kernel to Qemu, hopefully we can > release it by end of this year and work with you guys to adjust it for > the agreed method. Well, moving PCI config space emulation from kernel to QEMU is exactly the wrong direction to take for this proposal. Config space access to the vGPU would occur through the VFIO API. So if you already have config space emulation in the kernel, that's already one less piece of work for a VFIO model, it just needs to be "wired up" through the VFIO API. Thanks, Alex From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34751) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zzojw-0007NM-QB for qemu-devel@nongnu.org; Fri, 20 Nov 2015 11:40:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zzojs-0004vy-Lz for qemu-devel@nongnu.org; Fri, 20 Nov 2015 11:40:48 -0500 Received: from mx1.redhat.com ([209.132.183.28]:42414) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zzojs-0004vh-6G for qemu-devel@nongnu.org; Fri, 20 Nov 2015 11:40:44 -0500 Message-ID: <1448037640.4697.266.camel@redhat.com> From: Alex Williamson Date: Fri, 20 Nov 2015 09:40:40 -0700 In-Reply-To: <564EB4F2.9080605@intel.com> References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <564D78D0.80904@intel.com> <1447948366.4697.119.camel@redhat.com> <564E8C51.6070706@intel.com> <1447993371.4697.257.camel@redhat.com> <564EB4F2.9080605@intel.com> Content-Type: text/plain; charset="UTF-8" Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jike Song Cc: "igvt-g@ml01.01.org" , "Tian, Kevin" , "Reddy, Raghuveer" , qemu-devel , Stefano Stabellini , "White, Michael L" , "Cowperthwaite, David J" , "intel-gfx@lists.freedesktop.org" , "Li, Susie" , "Dong, Eddie" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xen.org" , Gerd Hoffmann , "Zhou, Chao" , Paolo Bonzini , "Zhu, Libo" , "Wang, Hongbo" , "Lv, Zhiyuan" On Fri, 2015-11-20 at 13:51 +0800, Jike Song wrote: > On 11/20/2015 12:22 PM, Alex Williamson wrote: > > On Fri, 2015-11-20 at 10:58 +0800, Jike Song wrote: > >> On 11/19/2015 11:52 PM, Alex Williamson wrote: > >>> On Thu, 2015-11-19 at 15:32 +0000, Stefano Stabellini wrote: > >>>> On Thu, 19 Nov 2015, Jike Song wrote: > >>>>> Hi Alex, thanks for the discussion. > >>>>> > >>>>> In addition to Kevin's replies, I have a high-level question: can VFIO > >>>>> be used by QEMU for both KVM and Xen? > >>>> > >>>> No. VFIO cannot be used with Xen today. When running on Xen, the IOMMU > >>>> is owned by Xen. > >>> > >>> Right, but in this case we're talking about device MMUs, which are owned > >>> by the device driver which I think is running in dom0, right? This > >>> proposal doesn't require support of the system IOMMU, the dom0 driver > >>> maps IOVA translations just as it would for itself. We're largely > >>> proposing use of the VFIO API to provide a common interface to expose a > >>> PCI(e) device to QEMU, but what happens in the vGPU vendor device and > >>> IOMMU backends is specific to the device and perhaps even specific to > >>> the hypervisor. Thanks, > >> > >> Let me conclude this, and please correct me in case of any misread: the > >> vGPU interface between kernel and QEMU will be through VFIO, with a new > >> VFIO backend (instead of the existing type1), for both KVMGT and XenGT? > > > > My primary concern is KVM and QEMU upstream, the proposal is not > > specifically directed at XenGT, but does not exclude it either. Xen is > > welcome to adopt this proposal as well, it simply defines the channel > > through which vGPUs are exposed to QEMU as the VFIO API. The core VFIO > > code in the Linux kernel is just as available for use in Xen dom0 as it > > is for a KVM host. VFIO in QEMU certainly knows about some > > accelerations for KVM, but these are almost entirely around allowing > > eventfd based interrupts to be injected through KVM, which is something > > I'm sure Xen could provide as well. These accelerations are also not > > required, VFIO based device assignment in QEMU works with or without > > KVM. Likewise, the VFIO kernel interface knows nothing about KVM and > > has no dependencies on it. > > > > There are two components to the VFIO API, one is the type1 compliant > > IOMMU interface, which for this proposal is really doing nothing more > > than tracking the HVA to GPA mappings for the VM. This much seems > > entirely common regardless of the hypervisor. The other part is the > > device interface. The lifecycle of the virtual device seems like it > > would be entirely shared, as does much of the emulation components of > > the device. When we get to pinning pages, providing direct access to > > memory ranges for a VM, and accelerating interrupts, the vGPU drivers > > will likely need some per hypervisor branches, but these are areas where > > that's true no matter what the interface. I'm probably over > > simplifying, but hopefully not too much, correct me if I'm wrong. > > > > Thanks for confirmation. For QEMU/KVM, I totally agree your point; However, > if we take XenGT to consider, it will be a bit more complex: with Xen > hypervisor and Dom0 kernel running in different level, it's not a straight- > forward way for QEMU to do something like mapping a portion of MMIO BAR > via VFIO in Dom0 kernel, instead of calling hypercalls directly. This would need to be part of the support added for Xen. To directly map a device MMIO space to the VM, VFIO provides an mmap, QEMU registers that mmap with KVM, or Xen. It's all just MemoryRegions in QEMU. Perhaps it's even already supported by Xen. > I don't know if there is a better way to handle this. But I do agree that > channels between kernel and Qemu via VFIO is a good idea, even though we > may have to split KVMGT/XenGT in Qemu a bit. We are currently working on > moving all of PCI CFG emulation from kernel to Qemu, hopefully we can > release it by end of this year and work with you guys to adjust it for > the agreed method. Well, moving PCI config space emulation from kernel to QEMU is exactly the wrong direction to take for this proposal. Config space access to the vGPU would occur through the VFIO API. So if you already have config space emulation in the kernel, that's already one less piece of work for a VFIO model, it just needs to be "wired up" through the VFIO API. Thanks, Alex