From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jerome Glisse Subject: Re: What differences and relations between SVM, HSA, HMM and Unified Memory? Date: Mon, 17 Jul 2017 10:27:44 -0400 Message-ID: <20170717142743.GA9420@gmail.com> References: <9BD73EA91F8E404F851CF3F519B14AA8CE753F@SZXEMI503-MBS.china.huawei.com> <20c9cdd5-5118-f916-d8ad-70b7c1434d73@arm.com> <1c4f4fb0-7201-ed4c-aa88-4d7e2369238e@huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <1c4f4fb0-7201-ed4c-aa88-4d7e2369238e-hv44wF8Li93QT0dZR+AlfA@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org Errors-To: iommu-bounces-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org To: Yisheng Xie Cc: "linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org" , "oded.gabbay-5C7GfCeVMHo@public.gmane.org" , "iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org" , "Wanzongshun (Vincent)" , "Wuzongyong (Cordius Wu, Euler Dept)" List-Id: iommu@lists.linux-foundation.org T24gTW9uLCBKdWwgMTcsIDIwMTcgYXQgMDc6NTc6MjNQTSArMDgwMCwgWWlzaGVuZyBYaWUgd3Jv dGU6Cj4gSGkgSmVhbi1QaGlsaXBwZe+8jAo+IAo+IE9uIDIwMTcvNi8xMiAxOTozNywgSmVhbi1Q aGlsaXBwZSBCcnVja2VyIHdyb3RlOgo+ID4gSGVsbG8sCj4gPiAKPiA+IE9uIDEwLzA2LzE3IDA1 OjA2LCBXdXpvbmd5b25nIChDb3JkaXVzIFd1LCBFdWxlciBEZXB0KSB3cm90ZToKPiA+PiBIaSwK PiA+Pgo+ID4+IENvdWxkIHNvbWVvbmUgZXhwbGFpbiBkaWZmZXJlbmNlcyBhbmQgcmVsYXRpb25z IGJldHdlZW4gdGhlIFNWTShTaGFyZWQKPiA+PiBWaXJ0dWFsIE1lbW9yeSwgYnkgSW50ZWwpLCBI U0EoSGV0ZXJvZ2VuZW91cyBTeXN0ZW0gQXJjaGl0ZWN0dXJlLCBieSBBTUQpLAo+ID4+IEhNTShI ZXRlcm9nZW5lb3VzIE1lbW9yeSBNYW5hZ2VtZW50LCBieSBHbGlzc2UpIGFuZCBVTShVbmlmaWVk IE1lbW9yeSwgYnkKPiA+PiBOVklESUEpID8gQXJlIHRoZXNlIGluIHRoZSBzdWJzdGl0dXRpb25h bCByZWxhdGlvbj8KPiA+Pgo+ID4+IEFzIEkgdW5kZXJzdGFuZCBpdCwgdGhlc2UgYWltIHRvIHNv bHZlIHRoZSBzYW1lIHRoaW5nLCBzaGFyaW5nIHBvaW50ZXJzCj4gPj4gYmV0d2VlbiBDUFUgYW5k IEdQVShpbXBsZW1lbnQgd2l0aCBBVFMvUEFTSUQvUFJJL0lPTU1VIHN1cHBvcnQpLiBTbyBmYXIs Cj4gPj4gU1ZNIGFuZCBIU0EgY2FuIG9ubHkgYmUgdXNlZCBieSBpbnRlZ3JhdGVkIGdwdS4gQW5k LCBJbnRlbCBkZWNsYXJlIHRoYXQKPiA+PiB0aGUgcm9vdCBwb3J0cyBkb2VzbuKAmXQgbm90IGhh dmUgdGhlIHJlcXVpcmVkIFRMUCBwcmVmaXggc3VwcG9ydCwgcmVzdWx0aW5nCj4gPj4gIHRoYXQg U1ZNIGNhbuKAmXQgYmUgdXNlZCBieSBkaXNjcmV0ZSBkZXZpY2VzLiBTbyBjb3VsZCBzb21lb25l IHRlbGwgbWUgdGhlCj4gPj4gcmVxdWlyZWQgVExQIHByZWZpeCBtZWFucyB3aGF0IHNwZWNpZmlj YWxseT8+Cj4gPj4gV2l0aCBITU0sIHdlIGNhbiB1c2UgYWxsb2NhdG9yIGxpa2UgbWFsbG9jIHRv IG1hbmFnZSBob3N0IGFuZCBkZXZpY2UKPiA+PiBtZW1vcnkuIERvZXMgdGhpcyBtZWFuIHRoYXQg dGhlcmUgaXMgbm8gbmVlZCB0byB1c2UgU1ZNIGFuZCBIU0Egd2l0aCBITU0sCj4gPj4gb3IgSE1N IGlzIHRoZSBiYXNpcyBvZiBTVk0gYW5kIEhBUyB0byBpbXBsZW1lbnQgRmluZS1HcmFpbmVkIHN5 c3RlbSBTVk0KPiA+PiBkZWZpbmVkIGluIHRoZSBvcGVuY2wgc3BlYz8KPiA+IAo+ID4gSSBjYW4n dCBwcm92aWRlIGFuIGV4aGF1c3RpdmUgYW5zd2VyLCBidXQgSSBoYXZlIGRvbmUgc29tZSB3b3Jr IG9uIFNWTS4KPiA+IFRha2UgaXQgd2l0aCBhIGdyYWluIG9mIHNhbHQgdGhvdWdoLCBJIGFtIG5v dCBhbiBleHBlcnQuCj4gPiAKPiA+ICogSFNBIGlzIGFuIGFyY2hpdGVjdHVyZSB0aGF0IHByb3Zp ZGVzIGEgY29tbW9uIHByb2dyYW1taW5nIG1vZGVsIGZvciBDUFVzCj4gPiBhbmQgYWNjZWxlcmF0 b3JzIChHUEdQVXMgZXRjKS4gSXQgZG9lcyBoYXZlIFNWTSByZXF1aXJlbWVudCAoSS9PIHBhZ2UK PiA+IGZhdWx0cywgUEFTSUQgYW5kIGNvbXBhdGlibGUgYWRkcmVzcyBzcGFjZXMpLCB0aG91Z2gg aXQncyBvbmx5IGEgc21hbGwKPiA+IHBhcnQgb2YgaXQuCj4gPiAKPiA+ICogU2ltaWxhcmx5LCBP cGVuQ0wgcHJvdmlkZXMgYW4gQVBJIGZvciBkZWFsaW5nIHdpdGggYWNjZWxlcmF0b3JzLiBPcGVu Q0wKPiA+IDIuMCBpbnRyb2R1Y2VkIHRoZSBjb25jZXB0IG9mIEZpbmUtR3JhaW5lZCBTeXN0ZW0g U1ZNLCB3aGljaCBhbGxvd3MgdG8KPiA+IHBhc3MgdXNlcnNwYWNlIHBvaW50ZXJzIHRvIGRldmlj ZXMuIEl0IGlzIGp1c3Qgb25lIGZsYXZvciBvZiBTVk0sIHRoZXkKPiA+IGFsc28gaGF2ZSBjb2Fy c2UtZ3JhaW5lZCBhbmQgbm9uLXN5c3RlbS4gQnV0IHRoZXkgbWlnaHQgaGF2ZSBjb2luZWQgdGhl Cj4gPiBuYW1lLCBhbmQgSSBiZWxpZXZlIHRoYXQgaW4gdGhlIGNvbnRleHQgb2YgTGludXggSU9N TVUsIHdoZW4gd2UgdGFsayBhYm91dAo+ID4gIlNWTSIgaXQgaXMgT3BlbkNMJ3MgZmluZS1ncmFp bmVkIHN5c3RlbSBTVk0uCj4gPiBbLi4uXQo+ID4gCj4gPiBXaGlsZSBTVk0gaXMgb25seSBhYm91 dCB2aXJ0dWFsIGFkZHJlc3Mgc3BhY2UsCj4gQXMgeW91IG1lbnRpb25lZCwgU1ZNIGlzIG9ubHkg YWJvdXQgdmlydHVhbCBhZGRyZXNzIHNwYWNlLCBJJ2QgbGlrZSB0byBrbm93IGhvdyB0bwo+IG1h bmFnZSB0aGUgcGh5c2ljYWwgYWRkcmVzcyBlc3BlY2lhbGx5IGFib3V0IGRldmljZSdzIFJBTSwg YmVmb3JlIEhNTT8KPiAKPiBXaGVuIE9wZW5DTCBhbGxvYyBhIFNWTSBwb2ludGVyIGxpa2U6Cj4g ICAgIHZvaWQqIHAgPSBjbFNWTUFsbG9jICgKPiAgICAgICAgIGNvbnRleHQsIC8vIGFuIE9wZW5D TCBjb250ZXh0IHdoZXJlIHRoaXMgYnVmZmVyIGlzIGF2YWlsYWJsZQo+ICAgICAgICAgQ0xfTUVN X1JFQURfV1JJVEUgfCBDTF9NRU1fU1ZNX0ZJTkVfR1JBSU5fQlVGRkVSLAo+ICAgICAgICAgc2l6 ZSwgLy8gYW1vdW50IG9mIG1lbW9yeSB0byBhbGxvY2F0ZSAoaW4gYnl0ZXMpCj4gICAgICAgICAw IC8vIGFsaWdubWVudCBpbiBieXRlcyAoMCBtZWFucyBkZWZhdWx0KQo+ICAgICApOwo+IAo+IHdo ZXJlIHRoaXMgUkFNIGNvbWUgZnJvbe+8jCBkZXZpY2UgUkFNIG9yIGhvc3QgUkFNPwo+IAoKRm9y IFNWTSB1c2luZyBBVFMvUEFTSUQgd2l0aCBGSU5FX0dSQUlOIHlvdXIgYWxsb2NhdGlvbiBjYW4g b25seQpiZSBpbnNpZGUgdGhlIHN5c3RlbSBtZW1vcnkgKGhvc3QgUkFNKS4gWW91IG5lZWQgYSBz cGVjaWFsIHN5c3RlbQpidXMgbGlrZSBDQVBJIG9yIENDSVggd2hpY2ggYm90aCBhcmUgc3RlcCBm dXJ0aGVyIHRoYW4gQVRTL1BBU0lECnRvIGJlIGFibGUgdG8gYWxsb3cgZmluZSBncmFpbiB0byB1 c2UgZGV2aWNlIG1lbW9yeS4KCkhvd2V2ZXIgdGhhdCBpcyB3aGVyZSBITU0gY2FuIGJlIHVzZWZ1 bGwgYXMgSE1NIGlzIGEgc29mdHdhcmUKc29sdXRpb24gdG8gdGhpcyBwcm9ibGVtLiBTbyB3aXRo IEhNTSBhbmQgYSBkZXZpY2UgdGhhdCBjYW4gd29yawp3aXRoIEhNTSwgeW91IGNhbiBnZXQgZmlu ZSBncmFpbiBhbGxvY2F0aW9uIHRvIGFsc28gdXNlIGRldmljZQptZW1vcnkgaG93ZXZlciBhbnkg Q1BVIGFjY2VzcyB3aWxsIGhhcHBlbiBpbiBob3N0IFJBTS4KCkrDqXLDtG1lCl9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCmlvbW11IG1haWxpbmcgbGlzdApp b21tdUBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3RzLmxpbnV4Zm91bmRh dGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby9pb21tdQ== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751590AbdGQO2V (ORCPT ); Mon, 17 Jul 2017 10:28:21 -0400 Received: from mail-qt0-f182.google.com ([209.85.216.182]:33041 "EHLO mail-qt0-f182.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751507AbdGQO1y (ORCPT ); Mon, 17 Jul 2017 10:27:54 -0400 Date: Mon, 17 Jul 2017 10:27:44 -0400 From: Jerome Glisse To: Yisheng Xie Cc: Jean-Philippe Brucker , "Wuzongyong (Cordius Wu, Euler Dept)" , "iommu@lists.linux-foundation.org" , "linux-kernel@vger.kernel.org" , "Wanzongshun (Vincent)" , "oded.gabbay@amd.com" , liubo95@huawei.com Subject: Re: What differences and relations between SVM, HSA, HMM and Unified Memory? Message-ID: <20170717142743.GA9420@gmail.com> References: <9BD73EA91F8E404F851CF3F519B14AA8CE753F@SZXEMI503-MBS.china.huawei.com> <20c9cdd5-5118-f916-d8ad-70b7c1434d73@arm.com> <1c4f4fb0-7201-ed4c-aa88-4d7e2369238e@huawei.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <1c4f4fb0-7201-ed4c-aa88-4d7e2369238e@huawei.com> User-Agent: Mutt/1.8.3 (2017-05-23) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jul 17, 2017 at 07:57:23PM +0800, Yisheng Xie wrote: > Hi Jean-Philippe， > > On 2017/6/12 19:37, Jean-Philippe Brucker wrote: > > Hello, > > > > On 10/06/17 05:06, Wuzongyong (Cordius Wu, Euler Dept) wrote: > >> Hi, > >> > >> Could someone explain differences and relations between the SVM(Shared > >> Virtual Memory, by Intel), HSA(Heterogeneous System Architecture, by AMD), > >> HMM(Heterogeneous Memory Management, by Glisse) and UM(Unified Memory, by > >> NVIDIA) ? Are these in the substitutional relation? > >> > >> As I understand it, these aim to solve the same thing, sharing pointers > >> between CPU and GPU(implement with ATS/PASID/PRI/IOMMU support). So far, > >> SVM and HSA can only be used by integrated gpu. And, Intel declare that > >> the root ports doesn’t not have the required TLP prefix support, resulting > >> that SVM can’t be used by discrete devices. So could someone tell me the > >> required TLP prefix means what specifically?> > >> With HMM, we can use allocator like malloc to manage host and device > >> memory. Does this mean that there is no need to use SVM and HSA with HMM, > >> or HMM is the basis of SVM and HAS to implement Fine-Grained system SVM > >> defined in the opencl spec? > > > > I can't provide an exhaustive answer, but I have done some work on SVM. > > Take it with a grain of salt though, I am not an expert. > > > > * HSA is an architecture that provides a common programming model for CPUs > > and accelerators (GPGPUs etc). It does have SVM requirement (I/O page > > faults, PASID and compatible address spaces), though it's only a small > > part of it. > > > > * Similarly, OpenCL provides an API for dealing with accelerators. OpenCL > > 2.0 introduced the concept of Fine-Grained System SVM, which allows to > > pass userspace pointers to devices. It is just one flavor of SVM, they > > also have coarse-grained and non-system. But they might have coined the > > name, and I believe that in the context of Linux IOMMU, when we talk about > > "SVM" it is OpenCL's fine-grained system SVM. > > [...] > > > > While SVM is only about virtual address space, > As you mentioned, SVM is only about virtual address space, I'd like to know how to > manage the physical address especially about device's RAM, before HMM? > > When OpenCL alloc a SVM pointer like: > void* p = clSVMAlloc ( > context, // an OpenCL context where this buffer is available > CL_MEM_READ_WRITE | CL_MEM_SVM_FINE_GRAIN_BUFFER, > size, // amount of memory to allocate (in bytes) > 0 // alignment in bytes (0 means default) > ); > > where this RAM come from， device RAM or host RAM? > For SVM using ATS/PASID with FINE_GRAIN your allocation can only be inside the system memory (host RAM). You need a special system bus like CAPI or CCIX which both are step further than ATS/PASID to be able to allow fine grain to use device memory. However that is where HMM can be usefull as HMM is a software solution to this problem. So with HMM and a device that can work with HMM, you can get fine grain allocation to also use device memory however any CPU access will happen in host RAM. Jérôme