From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jike Song Subject: Re: [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel Date: Mon, 23 Nov 2015 13:05:47 +0800 Message-ID: <56529EAB.4030007@intel.com> References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <1447963356.4697.184.camel@redhat.com> <1448040302.4697.300.camel@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8"; Format="flowed" Content-Transfer-Encoding: base64 Return-path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by gabe.freedesktop.org (Postfix) with ESMTP id 433CB88E9B for ; Sun, 22 Nov 2015 21:06:22 -0800 (PST) In-Reply-To: <1448040302.4697.300.camel@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" To: Alex Williamson Cc: "igvt-g@ml01.01.org" , "Reddy, Raghuveer" , "White, Michael L" , "Cowperthwaite, David J" , "intel-gfx@lists.freedesktop.org" , "Li, Susie" , "Dong, Eddie" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xen.org" , qemu-devel , "Zhou, Chao" , Paolo Bonzini , "Zhu, Libo" , "Wang, Hongbo" List-Id: intel-gfx@lists.freedesktop.org T24gMTEvMjEvMjAxNSAwMToyNSBBTSwgQWxleCBXaWxsaWFtc29uIHdyb3RlOgo+IE9uIEZyaSwg MjAxNS0xMS0yMCBhdCAwODoxMCArMDAwMCwgVGlhbiwgS2V2aW4gd3JvdGU6Cj4+Cj4+IEhlcmUg aXMgYSBtb3JlIGNvbmNyZXRlIGV4YW1wbGU6Cj4+Cj4+IEtWTUdUIGRvZXNuJ3QgcmVxdWlyZSBJ T01NVS4gQWxsIERNQSB0YXJnZXRzIGFyZSBhbHJlYWR5IHJlcGxhY2VkIHdpdGgKPj4gSFBBIHRo cnUgc2hhZG93IEdUVC4gU28gRE1BIHJlcXVlc3RzIGZyb20gR1BVIGFsbCBjb250YWluIEhQQXMu Cj4+Cj4+IFdoZW4gSU9NTVUgaXMgZW5hYmxlZCwgb25lIHNpbXBsZSBhcHByb2FjaCBpcyB0byBo YXZlIHZHUFUgSU9NTVUKPj4gZHJpdmVyIGNvbmZpZ3VyZSBzeXN0ZW0gSU9NTVUgd2l0aCBpZGVu dGl0eSBtYXBwaW5nIChIUEEtPkhQQSkuIFdlCj4+IGNhbid0IHVzZSAoR1BBLT5IUEEpIHNpbmNl IEdQQXMgZnJvbSBtdWx0aXBsZSBWTXMgYXJlIGNvbmZsaWN0aW5nLgo+Pgo+PiBIb3dldmVyLCB3 ZSBzdGlsbCBoYXZlIGhvc3QgZ2Z4IGRyaXZlciBydW5uaW5nLiBXaGVuIElPTU1VIGlzIGVuYWJs ZWQsCj4+IGRtYV9hbGxvY18qKiogd2lsbCByZXR1cm4gSU9WQSAoZHJ2ZXJzL2lvbW11L2lvdmEu YykgaW4gaG9zdCBnZnggZHJpdmVyLAo+PiB3aGljaCB3aWxsIGhhdmUgSU9WQS0+SFBBIHByb2dy YW1tZWQgdG8gc3lzdGVtIElPTU1VLgo+Pgo+PiBPbmUgSU9NTVUgZGV2aWNlIGVudHJ5IGNhbiBv bmx5IHRyYW5zbGF0ZSBvbmUgYWRkcmVzcyBzcGFjZSwgc28gaGVyZQo+PiBjb21lcyBhIGNvbmZs aWN0IChIUEEtPkhQQSB2cy4gSU9WQS0+SFBBKS4gVG8gc29sdmUgdGhpcywgdkdQVSBJT01NVQo+ PiBkcml2ZXIgbmVlZHMgdG8gYWxsb2NhdGUgSU9WQSBmcm9tIGlvdmEuYyBmb3IgZWFjaCBWTSB3 LyB2R1BVIGFzc2lnbmVkLAo+PiBhbmQgdGhlbiBLVk1HVCB3aWxsIHByb2dyYW0gSU9WQSBpbiBz aGFkb3cgR1RUIGFjY29yZGluZ2x5LiBJdCBhZGRzCj4+IG9uZSBhZGRpdGlvbmFsIG1hcHBpbmcg bGF5ZXIgKEdQQS0+SU9WQS0+SFBBKS4gSW4gdGhpcyB3YXkgdHdvCj4+IHJlcXVpcmVtZW50cyBj YW4gYmUgdW5pZmllZCB0b2dldGhlciBzaW5jZSBvbmx5IElPVkEtPkhQQSBtYXBwaW5nCj4+IG5l ZWRzIHRvIGJlIGJ1aWx0Lgo+Pgo+PiBTbyB1bmxpa2UgZXhpc3RpbmcgdHlwZTEgSU9NTVUgZHJp dmVyIHdoaWNoIGNvbnRyb2xzIElPTU1VIGFsb25lLCB2R1BVCj4+IElPTU1VIGRyaXZlciBuZWVk cyB0byBjb29wZXJhdGUgd2l0aCBvdGhlciBhZ2VudCAoaW92YS5jIGhlcmUpIHRvCj4+IGNvLW1h bmFnZSBzeXN0ZW0gSU9NTVUuIFRoaXMgbWF5IG5vdCBpbXBhY3QgZXhpc3RpbmcgVkZJTyBmcmFt ZXdvcmsuCj4+IEp1c3Qgd2FudCB0byBoaWdobGlnaHQgYWRkaXRpb25hbCB3b3JrIGhlcmUgd2hl biBpbXBsZW1lbnRpbmcgdGhlIHZHUFUKPj4gSU9NTVUgZHJpdmVyLgo+Cj4gUmlnaHQsIHNvIHRo ZSBleGlzdGluZyBpOTE1IGRyaXZlciBuZWVkcyB0byB1c2UgdGhlIERNQSBBUEkgYW5kIGNhbGxz Cj4gbGlrZSBkbWFfbWFwX3BhZ2UoKSB0byBlbmFibGUgdHJhbnNsYXRpb25zIHRocm91Z2ggdGhl IElPTU1VLiAgV2l0aAo+IGRtYV9tYXBfcGFnZSgpLCB0aGUgY2FsbGVyIHByb3ZpZGVzIGEgcGFn ZSBhZGRyZXNzICh+SFBBKSBhbmQgaXMKPiByZXR1cm5lZCBhbiBJT1ZBLiAgU28gdW5mb3J0dW5h dGVseSB5b3UgZG9uJ3QgZ2V0IHRvIHRha2UgdGhlIHNob3J0Y3V0Cj4gb2YgaGF2aW5nIGFuIGlk ZW50aXR5IG1hcHBpbmcgdGhyb3VnaCB0aGUgSU9NTVUgdW5sZXNzIHlvdSB3YW50IHRvCj4gY29u dmVydCBpOTE1IGVudGlyZWx5IHRvIHVzaW5nIHRoZSBJT01NVSBBUEksIGJlY2F1c2Ugd2UgYWxz byBjYW4ndCBoYXZlCj4gdGhlIGNvbmZsaWN0IHRoYXQgYW4gSFBBIGNvdWxkIG92ZXJsYXAgYW4g SU9WQSBmb3IgYSBwcmV2aW91c2x5IG1hcHBlZAo+IHBhZ2UuCj4KPiBUaGUgZG91YmxlIHRyYW5z bGF0aW9uLCBvbmNlIHRocm91Z2ggdGhlIEdQVSBNTVUgYW5kIG9uY2UgdGhyb3VnaCB0aGUKPiBz eXN0ZW0gSU9NTVUgaXMgZ29pbmcgdG8gaGFwcGVuIHJlZ2FyZGxlc3Mgb2Ygd2hldGhlciB3ZSBj YW4gaWRlbnRpdHkKPiBtYXAgdGhyb3VnaCB0aGUgSU9NTVUuICBUaGUgb25seSBzb2x1dGlvbiB0 byB0aGlzIHdvdWxkIGJlIGZvciB0aGUgR1BVCj4gdG8gcGFydGljaXBhdGUgaW4gQVRTIGFuZCBw cm92aWRlIHByZS10cmFuc2xhdGVkIHRyYW5zYWN0aW9ucyBmcm9tIHRoZQo+IEdQVS4gIEFsbCBv ZiB0aGlzIGlzIGludGVybmFsIHRvIHRoZSBpOTE1IGRyaXZlciAob3IgdmZpbyBleHRlbnNpb24g b2YKPiB0aGF0IGRyaXZlcikgYW5kIG5lZWRzIHRvIGJlIGRvbmUgcmVnYXJkbGVzcyBvZiB3aGF0 IHNvcnQgb2YgaW50ZXJmYWNlCj4gd2UncmUgdXNpbmcgdG8gZXhwb3NlIHRoZSB2R1BVIHRvIFFF TVUuICBJdCBqdXN0IHNlZW1zIGxpa2UgVkZJTwo+IHByb3ZpZGVzIGEgY29udmVuaWVudCB3YXkg b2YgZG9pbmcgdGhpcyBzaW5jZSB5b3UnbGwgaGF2ZSByZWFkeSBhY2Nlc3MKPiB0byB0aGUgSFZB LUdQQSBtYXBwaW5ncyBmb3IgdGhlIHVzZXIuCj4KPiBJIHRoaW5rIHRoZSBrZXkgcG9pbnRzIHRo b3VnaCBhcmU6Cj4KPiAgICAgICAgKiB0aGUgVkZJTyB0eXBlMSBJT01NVSBzdG9yZXMgR1BBIHRv IEhWQSB0cmFuc2xhdGlvbnMKPiAgICAgICAgKiBnZXRfdXNlcl9wYWdlcygpIG9uIHRoZSBIVkEg d2lsbCBwaW4gdGhlIHBhZ2UgYW5kIGdpdmUgeW91IGEKPiAgICAgICAgICBwYWdlCj4gICAgICAg ICogZG1hX21hcF9wYWdlKCkgcmVjZWl2ZXMgdGhhdCBwYWdlLCBwcm9ncmFtcyB0aGUgc3lzdGVt IElPTU1VIGFuZAo+ICAgICAgICAgIHByb3ZpZGVzIGFuIElPVkEKPiAgICAgICAgKiB0aGUgR1BV IE1NVSBjYW4gdGhlbiBiZSBwcm9ncmFtbWVkIHdpdGggdGhlIEdQQSB0byBJT1ZBCj4gICAgICAg ICAgdHJhbnNsYXRpb25zCgpUaGFua3MgZm9yIHN1Y2ggYSBuaWNlIGV4YW1wbGUhIEknbGwgZG8g bXkgaG9tZSB3b3JrIGFuZCBnZXQgYmFjayB0byB5b3UKc2hvcnRseSA6KQoKPgo+IFRoYW5rcywK PiBBbGV4Cj4KCi0tClRoYW5rcywKSmlrZQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fXwpJbnRlbC1nZnggbWFpbGluZyBsaXN0CkludGVsLWdmeEBsaXN0cy5m cmVlZGVza3RvcC5vcmcKaHR0cDovL2xpc3RzLmZyZWVkZXNrdG9wLm9yZy9tYWlsbWFuL2xpc3Rp bmZvL2ludGVsLWdmeAo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753284AbbKWFGX (ORCPT ); Mon, 23 Nov 2015 00:06:23 -0500 Received: from mga01.intel.com ([192.55.52.88]:20222 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753172AbbKWFGW (ORCPT ); Mon, 23 Nov 2015 00:06:22 -0500 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,335,1444719600"; d="scan'208";a="605517754" Message-ID: <56529EAB.4030007@intel.com> Date: Mon, 23 Nov 2015 13:05:47 +0800 From: Jike Song User-Agent: Mozilla/5.0 (X11; Linux i686 on x86_64; rv:17.0) Gecko/20130801 Thunderbird/17.0.8 MIME-Version: 1.0 To: Alex Williamson CC: "Tian, Kevin" , "xen-devel@lists.xen.org" , "igvt-g@ml01.01.org" , "intel-gfx@lists.freedesktop.org" , "linux-kernel@vger.kernel.org" , "White, Michael L" , "Dong, Eddie" , "Li, Susie" , "Cowperthwaite, David J" , "Reddy, Raghuveer" , "Zhu, Libo" , "Zhou, Chao" , "Wang, Hongbo" , "Lv, Zhiyuan" , qemu-devel , Paolo Bonzini , Gerd Hoffmann Subject: Re: [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <1447963356.4697.184.camel@redhat.com> <1448040302.4697.300.camel@redhat.com> In-Reply-To: <1448040302.4697.300.camel@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/21/2015 01:25 AM, Alex Williamson wrote: > On Fri, 2015-11-20 at 08:10 +0000, Tian, Kevin wrote: >> >> Here is a more concrete example: >> >> KVMGT doesn't require IOMMU. All DMA targets are already replaced with >> HPA thru shadow GTT. So DMA requests from GPU all contain HPAs. >> >> When IOMMU is enabled, one simple approach is to have vGPU IOMMU >> driver configure system IOMMU with identity mapping (HPA->HPA). We >> can't use (GPA->HPA) since GPAs from multiple VMs are conflicting. >> >> However, we still have host gfx driver running. When IOMMU is enabled, >> dma_alloc_*** will return IOVA (drvers/iommu/iova.c) in host gfx driver, >> which will have IOVA->HPA programmed to system IOMMU. >> >> One IOMMU device entry can only translate one address space, so here >> comes a conflict (HPA->HPA vs. IOVA->HPA). To solve this, vGPU IOMMU >> driver needs to allocate IOVA from iova.c for each VM w/ vGPU assigned, >> and then KVMGT will program IOVA in shadow GTT accordingly. It adds >> one additional mapping layer (GPA->IOVA->HPA). In this way two >> requirements can be unified together since only IOVA->HPA mapping >> needs to be built. >> >> So unlike existing type1 IOMMU driver which controls IOMMU alone, vGPU >> IOMMU driver needs to cooperate with other agent (iova.c here) to >> co-manage system IOMMU. This may not impact existing VFIO framework. >> Just want to highlight additional work here when implementing the vGPU >> IOMMU driver. > > Right, so the existing i915 driver needs to use the DMA API and calls > like dma_map_page() to enable translations through the IOMMU. With > dma_map_page(), the caller provides a page address (~HPA) and is > returned an IOVA. So unfortunately you don't get to take the shortcut > of having an identity mapping through the IOMMU unless you want to > convert i915 entirely to using the IOMMU API, because we also can't have > the conflict that an HPA could overlap an IOVA for a previously mapped > page. > > The double translation, once through the GPU MMU and once through the > system IOMMU is going to happen regardless of whether we can identity > map through the IOMMU. The only solution to this would be for the GPU > to participate in ATS and provide pre-translated transactions from the > GPU. All of this is internal to the i915 driver (or vfio extension of > that driver) and needs to be done regardless of what sort of interface > we're using to expose the vGPU to QEMU. It just seems like VFIO > provides a convenient way of doing this since you'll have ready access > to the HVA-GPA mappings for the user. > > I think the key points though are: > > * the VFIO type1 IOMMU stores GPA to HVA translations > * get_user_pages() on the HVA will pin the page and give you a > page > * dma_map_page() receives that page, programs the system IOMMU and > provides an IOVA > * the GPU MMU can then be programmed with the GPA to IOVA > translations Thanks for such a nice example! I'll do my home work and get back to you shortly :) > > Thanks, > Alex > -- Thanks, Jike From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:48983) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a0jKc-00018u-4e for qemu-devel@nongnu.org; Mon, 23 Nov 2015 00:06:27 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a0jKY-0005Ki-Uk for qemu-devel@nongnu.org; Mon, 23 Nov 2015 00:06:26 -0500 Received: from mga03.intel.com ([134.134.136.65]:33818) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a0jKY-0005Ke-LH for qemu-devel@nongnu.org; Mon, 23 Nov 2015 00:06:22 -0500 Message-ID: <56529EAB.4030007@intel.com> Date: Mon, 23 Nov 2015 13:05:47 +0800 From: Jike Song MIME-Version: 1.0 References: <53D215D3.50608@intel.com> <547FCAAD.2060406@intel.com> <54AF967B.3060503@intel.com> <5527CEC4.9080700@intel.com> <559B3E38.1080707@intel.com> <562F4311.9@intel.com> <1447870341.4697.92.camel@redhat.com> <1447963356.4697.184.camel@redhat.com> <1448040302.4697.300.camel@redhat.com> In-Reply-To: <1448040302.4697.300.camel@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [Intel-gfx] [Announcement] 2015-Q3 release of XenGT - a Mediated Graphics Passthrough Solution from Intel List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Alex Williamson Cc: "igvt-g@ml01.01.org" , "Tian, Kevin" , "Reddy, Raghuveer" , qemu-devel , "White, Michael L" , "Cowperthwaite, David J" , "intel-gfx@lists.freedesktop.org" , "Li, Susie" , "Dong, Eddie" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xen.org" , Gerd Hoffmann , "Zhou, Chao" , Paolo Bonzini , "Zhu, Libo" , "Wang, Hongbo" , "Lv, Zhiyuan" On 11/21/2015 01:25 AM, Alex Williamson wrote: > On Fri, 2015-11-20 at 08:10 +0000, Tian, Kevin wrote: >> >> Here is a more concrete example: >> >> KVMGT doesn't require IOMMU. All DMA targets are already replaced with >> HPA thru shadow GTT. So DMA requests from GPU all contain HPAs. >> >> When IOMMU is enabled, one simple approach is to have vGPU IOMMU >> driver configure system IOMMU with identity mapping (HPA->HPA). We >> can't use (GPA->HPA) since GPAs from multiple VMs are conflicting. >> >> However, we still have host gfx driver running. When IOMMU is enabled, >> dma_alloc_*** will return IOVA (drvers/iommu/iova.c) in host gfx driver, >> which will have IOVA->HPA programmed to system IOMMU. >> >> One IOMMU device entry can only translate one address space, so here >> comes a conflict (HPA->HPA vs. IOVA->HPA). To solve this, vGPU IOMMU >> driver needs to allocate IOVA from iova.c for each VM w/ vGPU assigned, >> and then KVMGT will program IOVA in shadow GTT accordingly. It adds >> one additional mapping layer (GPA->IOVA->HPA). In this way two >> requirements can be unified together since only IOVA->HPA mapping >> needs to be built. >> >> So unlike existing type1 IOMMU driver which controls IOMMU alone, vGPU >> IOMMU driver needs to cooperate with other agent (iova.c here) to >> co-manage system IOMMU. This may not impact existing VFIO framework. >> Just want to highlight additional work here when implementing the vGPU >> IOMMU driver. > > Right, so the existing i915 driver needs to use the DMA API and calls > like dma_map_page() to enable translations through the IOMMU. With > dma_map_page(), the caller provides a page address (~HPA) and is > returned an IOVA. So unfortunately you don't get to take the shortcut > of having an identity mapping through the IOMMU unless you want to > convert i915 entirely to using the IOMMU API, because we also can't have > the conflict that an HPA could overlap an IOVA for a previously mapped > page. > > The double translation, once through the GPU MMU and once through the > system IOMMU is going to happen regardless of whether we can identity > map through the IOMMU. The only solution to this would be for the GPU > to participate in ATS and provide pre-translated transactions from the > GPU. All of this is internal to the i915 driver (or vfio extension of > that driver) and needs to be done regardless of what sort of interface > we're using to expose the vGPU to QEMU. It just seems like VFIO > provides a convenient way of doing this since you'll have ready access > to the HVA-GPA mappings for the user. > > I think the key points though are: > > * the VFIO type1 IOMMU stores GPA to HVA translations > * get_user_pages() on the HVA will pin the page and give you a > page > * dma_map_page() receives that page, programs the system IOMMU and > provides an IOVA > * the GPU MMU can then be programmed with the GPA to IOVA > translations Thanks for such a nice example! I'll do my home work and get back to you shortly :) > > Thanks, > Alex > -- Thanks, Jike